├── .gitignore ├── CODEOWNERS ├── CODE_OF_CONDUCT.MD ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── codeql_bundle ├── __init__.py ├── cli.py └── helpers │ ├── __init__.py │ ├── bundle.py │ └── codeql.py ├── poetry.lock ├── pyproject.toml └── tests ├── __init__.py ├── bundle-diff.py ├── workspace-with-cyclic-dep ├── codeql-workspace.yml ├── x │ └── qlpack.yml ├── y │ └── qlpack.yml └── z │ └── qlpack.yml └── workspace ├── codeql-workspace.yml └── cpp ├── a ├── codeql-pack.lock.yml └── qlpack.yml ├── aa ├── codeql-pack.lock.yml └── qlpack.yml ├── foo-bundle-customizations-tests ├── FooExternalSourceFunction │ ├── Foo.expected │ ├── Foo.ql │ └── test.c ├── codeql-pack.lock.yml └── qlpack.yml ├── foo-customization-tests ├── FooExternalSourceFunction │ ├── Foo.expected │ ├── Foo.ql │ └── test.c ├── codeql-pack.lock.yml └── qlpack.yml └── foo-customizations ├── codeql-pack.lock.yml ├── foo └── cpp_customizations │ └── Customizations.qll └── qlpack.yml /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode/settings.json 2 | __pycache__ -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | # This project is maintained with love by: @advanced-security/es-codeql 2 | 3 | * @advanced-security/es-codeql 4 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.MD: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | We as members, contributors, and leaders pledge to make participation in our 6 | community a harassment-free experience for everyone, regardless of age, body 7 | size, visible or invisible disability, ethnicity, sex characteristics, gender 8 | identity and expression, level of experience, education, socio-economic status, 9 | nationality, personal appearance, race, religion, or sexual identity 10 | and orientation. 11 | 12 | We pledge to act and interact in ways that contribute to an open, welcoming, 13 | diverse, inclusive, and healthy community. 14 | 15 | ## Our Standards 16 | 17 | Examples of behavior that contributes to a positive environment for our 18 | community include: 19 | 20 | * Demonstrating empathy and kindness toward other people 21 | * Being respectful of differing opinions, viewpoints, and experiences 22 | * Giving and gracefully accepting constructive feedback 23 | * Accepting responsibility and apologizing to those affected by our mistakes, 24 | and learning from the experience 25 | * Focusing on what is best not just for us as individuals, but for the 26 | overall community 27 | 28 | Examples of unacceptable behavior include: 29 | 30 | * The use of sexualized language or imagery, and sexual attention or 31 | advances of any kind 32 | * Trolling, insulting or derogatory comments, and personal or political attacks 33 | * Public or private harassment 34 | * Publishing others' private information, such as a physical or email 35 | address, without their explicit permission 36 | * Other conduct which could reasonably be considered inappropriate in a 37 | professional setting 38 | 39 | ## Enforcement Responsibilities 40 | 41 | Community leaders are responsible for clarifying and enforcing our standards of 42 | acceptable behavior and will take appropriate and fair corrective action in 43 | response to any behavior that they deem inappropriate, threatening, offensive, 44 | or harmful. 45 | 46 | Community leaders have the right and responsibility to remove, edit, or reject 47 | comments, commits, code, wiki edits, issues, and other contributions that are 48 | not aligned to this Code of Conduct, and will communicate reasons for moderation 49 | decisions when appropriate. 50 | 51 | ## Scope 52 | 53 | This Code of Conduct applies within all community spaces, and also applies when 54 | an individual is officially representing the community in public spaces. 55 | Examples of representing our community include using an official e-mail address, 56 | posting via an official social media account, or acting as an appointed 57 | representative at an online or offline event. 58 | 59 | ## Enforcement 60 | 61 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 62 | reported to the community leaders responsible for enforcement at 63 | opensource@github.com. 64 | All complaints will be reviewed and investigated promptly and fairly. 65 | 66 | All community leaders are obligated to respect the privacy and security of the 67 | reporter of any incident. 68 | 69 | ## Enforcement Guidelines 70 | 71 | Community leaders will follow these Community Impact Guidelines in determining 72 | the consequences for any action they deem in violation of this Code of Conduct: 73 | 74 | ### 1. Correction 75 | 76 | **Community Impact**: Use of inappropriate language or other behavior deemed 77 | unprofessional or unwelcome in the community. 78 | 79 | **Consequence**: A private, written warning from community leaders, providing 80 | clarity around the nature of the violation and an explanation of why the 81 | behavior was inappropriate. A public apology may be requested. 82 | 83 | ### 2. Warning 84 | 85 | **Community Impact**: A violation through a single incident or series 86 | of actions. 87 | 88 | **Consequence**: A warning with consequences for continued behavior. No 89 | interaction with the people involved, including unsolicited interaction with 90 | those enforcing the Code of Conduct, for a specified period of time. This 91 | includes avoiding interactions in community spaces as well as external channels 92 | like social media. Violating these terms may lead to a temporary or 93 | permanent ban. 94 | 95 | ### 3. Temporary Ban 96 | 97 | **Community Impact**: A serious violation of community standards, including 98 | sustained inappropriate behavior. 99 | 100 | **Consequence**: A temporary ban from any sort of interaction or public 101 | communication with the community for a specified period of time. No public or 102 | private interaction with the people involved, including unsolicited interaction 103 | with those enforcing the Code of Conduct, is allowed during this period. 104 | Violating these terms may lead to a permanent ban. 105 | 106 | ### 4. Permanent Ban 107 | 108 | **Community Impact**: Demonstrating a pattern of violation of community 109 | standards, including sustained inappropriate behavior, harassment of an 110 | individual, or aggression toward or disparagement of classes of individuals. 111 | 112 | **Consequence**: A permanent ban from any sort of public interaction within 113 | the community. 114 | 115 | ## Attribution 116 | 117 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], 118 | version 2.0, available at 119 | https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. 120 | 121 | Community Impact Guidelines were inspired by [Mozilla's code of conduct 122 | enforcement ladder](https://github.com/mozilla/diversity). 123 | 124 | [homepage]: https://www.contributor-covenant.org 125 | 126 | For answers to common questions about this code of conduct, see the FAQ at 127 | https://www.contributor-covenant.org/faq. Translations are available at 128 | https://www.contributor-covenant.org/translations. -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | [fork]: https://github.com/github/REPO/fork 4 | [pr]: https://github.com/github/REPO/compare 5 | [code-of-conduct]: CODE_OF_CONDUCT.md 6 | 7 | Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great. 8 | 9 | Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE.md). 10 | 11 | Please note that this project is released with a [Contributor Code of Conduct][code-of-conduct]. By participating in this project you agree to abide by its terms. 12 | 13 | ## Submitting a pull request 14 | 15 | 0. [Fork][fork] and clone the repository. 16 | 1. Create a new branch: `git checkout -b my-branch-name`. 17 | 2. Make your change, add tests, and make sure the tests pass. 18 | 3. Push to your fork and [submit a pull request][pr]. 19 | 4. Ensure the required checks pass. 20 | 21 | Here are a few things you can do that will increase the likelihood of your pull request being accepted: 22 | 23 | - Write tests. 24 | - Keep your change as focused as possible. If there are multiple changes you would like to make that are not dependent upon each other, consider submitting them as separate pull requests. 25 | - Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). 26 | 27 | ## Resources 28 | 29 | - [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/) 30 | - [Using Pull Requests](https://help.github.com/articles/about-pull-requests/) 31 | - [GitHub Help](https://help.github.com) -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 GitHub 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CodeQL bundle 2 | 3 | A CLI application for building a custom CodeQL bundle. 4 | 5 | ## Introduction 6 | 7 | A CodeQL bundle is an archive containing a CodeQL CLI and compatible [standard library packs](https://github.com/github/codeql) that provides a single deployable artifact for using CodeQL locally or in a CI/CD environment. 8 | 9 | A custom CodeQL bundle contains additional CodeQL query packs, library packs, or customization packs. 10 | CodeQL customization packs are special CodeQL library packs that provide unofficial support to extend the CodeQL standard library packs with *sources*, *sinks*, *data-flow steps*, *barriers*, *sanitizer*, and models that describe frameworks 11 | aimed to increase the coverage and/or precision of the CodeQL queries. 12 | 13 | Customizations are used to add, or extend, support for open-source frameworks or proprietary frameworks. 14 | For more details on CodeQL customization packs see the section [CodeQL customization packs](#codeql-customization-packs). 15 | 16 | ## Installation 17 | 18 | The CodeQL bundle application can be installed using `pip` with the command: 19 | 20 | ```bash 21 | python3.11 -m pip install https://github.com/rvermeulen/codeql-bundle/releases/download/v0.1.8/codeql_bundle-0.1.8-py3-none-any.whl 22 | ``` 23 | 24 | ## Usage 25 | 26 | Before you can use the CodeQL bundle application you must download a bundle you want to customize from the CodeQL Action [releases](https://github.com/github/codeql-action/releases) page. 27 | 28 | The CodeQL bundle application requires a [CodeQL workspace](https://codeql.github.com/docs/codeql-cli/about-codeql-workspaces/) to locate the packs you want to include in a custom bundle. 29 | You can see the packs available in your workspace by running `codeql pack ls -- ` where `` is the root directory of your CodeQL workspace. 30 | 31 | With both a CodeQL bundle and a CodeQL workspace you can create a bundle with the command: 32 | 33 | ```bash 34 | codeql-bundle --bundle --output codeql-custom-bundle.tar.gz --workspace --log INFO 35 | ``` 36 | 37 | If the source bundle is the platform agnostic bundle then you can create platform specific bundles to reduce the size of the used bundle(s). 38 | The following example creates platform specific bundles for all the currently supported platforms. 39 | 40 | ```bash 41 | codeql-bundle --bundle --output --workspace --log INFO -p linux64 -p osx64 -p win64 42 | ``` 43 | 44 | ## CodeQL customization packs 45 | 46 | The CodeQL bundle CLI application provides a development experience for customization packs that mimics the development experience for official CodeQL packs. 47 | 48 | A customization pack is a library pack with the following properties: 49 | 50 | 1. Has a module `Customizations.qll` located in the subdirectory `/` where any character `-` in the `scope` or `package_name` is replaced with `_`. 51 | For example, a library pack with the name `foo/cpp-customizations` must have a `Customizations.qll` at `foo/cpp_customizations/Customizations.qll`. 52 | 1. Has a dependency on the standard library pack it wants to customize. For example, `codeql/cpp-all`. 53 | 54 | If both properties hold, then the CodeQL bundle application will import the customizations pack `Customizations.qll` module in the standard library pack and recompile all 55 | the CodeQL query packs in CodeQL bundle that have a dependency on that standard library pack to ensure the customizations are available to the queries. 56 | 57 | ### Steps to create a customization pack 58 | 59 | This example targets the C/C++ language, but you can use this for any supported language. 60 | 61 | 1. Create a new pack `codeql pack init foo/cpp-customizations`. 62 | 2. Turn the pack into a library pack `sed -i '' -e 's/library: false/library: true/' cpp-customizations/qlpack.yml`. 63 | 3. Add a dependency on `codeql/cpp-all` with `codeql pack add --dir=cpp-customizations codeql/cpp-all` 64 | 4. Implement the customizations module with `mkdir -p cpp-customizations/foo/cpp_customizations && echo "import cpp" > cpp-customizations/foo/cpp_customizations/Customizations.qll` 65 | 66 | ## Limitations 67 | 68 | - The customization pack must directly rely on a CodeQL language pack. 69 | - The customization pack cannot have a dependency besides a CodeQL language pack, otherwise this will result in a cyclic dependency that we cannot resolve. 70 | - All packs must directly be specified. That is if a pack relies on another pack in the workspace, but that pack isn't specified as part of the `packs` argument, the pack will not be implicitly added and `codeql-bundle` will fail. 71 | -------------------------------------------------------------------------------- /codeql_bundle/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/advanced-security/codeql-bundle/c65d9c36496679e444d125aa52f71335025fbb6a/codeql_bundle/__init__.py -------------------------------------------------------------------------------- /codeql_bundle/cli.py: -------------------------------------------------------------------------------- 1 | # Add the parent directory to the path if this module is run directly (i.e. not imported) 2 | # This is necessary to support both the Poetry script invocation and the direct invocation. 3 | if not __package__ and __name__ == "__main__": 4 | import sys 5 | from pathlib import Path 6 | 7 | sys.path.append(str(Path(__file__).parent.parent)) 8 | __package__ = Path(__file__).parent.name 9 | 10 | import click 11 | from pathlib import Path 12 | from codeql_bundle.helpers.codeql import CodeQLException 13 | from codeql_bundle.helpers.bundle import CustomBundle, BundleException, BundlePlatform 14 | from typing import List, Optional 15 | import sys 16 | import logging 17 | import os 18 | 19 | logger = logging.getLogger(__name__) 20 | 21 | @click.command() 22 | @click.option( 23 | "-b", 24 | "--bundle", 25 | "bundle_path", 26 | required=True, 27 | help="Path to a CodeQL bundle downloaded from https://github.com/github/codeql-action/releases", 28 | type=click.Path(exists=True, path_type=Path), 29 | ) 30 | @click.option( 31 | "-o", 32 | "--output", 33 | required=True, 34 | help="Path to store the custom CodeQL bundle. Can be a directory or a non-existing archive ending with the extension '.tar.gz' if there is only a single bundle", 35 | type=click.Path(path_type=Path), 36 | ) 37 | @click.option( 38 | "-w", 39 | "--workspace", 40 | help="Path to a directory containing a 'codeql-workspace.yml' file or a path to a 'codeql-workspace.yml' file", 41 | type=click.Path(exists=True, path_type=Path), 42 | default=Path.cwd(), 43 | ) 44 | @click.option( 45 | "--no-precompile", "-nc", is_flag=True, help="Do not pre-compile the bundle." 46 | ) 47 | @click.option( 48 | "-l", 49 | "--log", 50 | "loglevel", 51 | type=click.Choice( 52 | ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"], case_sensitive=False 53 | ), 54 | default="WARNING", 55 | ) 56 | @click.option( 57 | "-p", 58 | "--platform", 59 | multiple=True, 60 | type=click.Choice(["linux64", "osx64", "win64"], case_sensitive=False), 61 | help="Target platform for the bundle", 62 | ) 63 | @click.option( 64 | "-c", 65 | "--code-scanning-config", 66 | type=click.Path(exists=True, path_type=Path), 67 | help="Path to a Code Scanning configuration file that will be the default for the bundle", 68 | ) 69 | @click.option( 70 | "-a", 71 | "--additional-data-config", 72 | type=click.Path(exists=True, path_type=Path), 73 | help="Path to a JSON file specifying additional data to install into the bundle", 74 | ) 75 | @click.argument("packs", nargs=-1, required=True) 76 | def main( 77 | bundle_path: Path, 78 | output: Path, 79 | workspace: Path, 80 | no_precompile: bool, 81 | loglevel: str, 82 | platform: List[str], 83 | code_scanning_config: Optional[Path], 84 | additional_data_config: Optional[Path], 85 | packs: List[str], 86 | ) -> None: 87 | 88 | if loglevel == "DEBUG": 89 | logging.basicConfig( 90 | format="%(levelname)s:%(asctime)s %(message)s", 91 | level=getattr(logging, loglevel.upper()), 92 | ) 93 | else: 94 | logging.basicConfig( 95 | format="%(levelname)s: %(message)s", 96 | level=getattr(logging, loglevel.upper()), 97 | ) 98 | 99 | workspace = Path(os.path.abspath(workspace)) 100 | 101 | if workspace.name == "codeql-workspace.yml": 102 | workspace = workspace.parent 103 | 104 | logger.info( 105 | f"Creating custom bundle of {bundle_path} using CodeQL pack(s) in workspace {workspace}" 106 | ) 107 | 108 | try: 109 | bundle = CustomBundle(bundle_path, workspace) 110 | # options for custom bundle 111 | bundle.disable_precompilation = no_precompile 112 | 113 | unsupported_platforms = list( 114 | filter( 115 | lambda p: not bundle.supports_platform(BundlePlatform.from_string(p)), 116 | platform, 117 | ) 118 | ) 119 | if len(unsupported_platforms) > 0: 120 | logger.fatal( 121 | f"The provided bundle supports the platform(s) {', '.join(map(str, bundle.platforms))}, but doesn't support the following platform(s): {', '.join(unsupported_platforms)}" 122 | ) 123 | sys.exit(1) 124 | 125 | logger.info(f"Looking for CodeQL packs in workspace {workspace}") 126 | packs_in_workspace = bundle.get_workspace_packs() 127 | logger.info( 128 | f"Found the CodeQL pack(s): {','.join(map(lambda p: p.config.name, packs_in_workspace))}" 129 | ) 130 | 131 | logger.info( 132 | f"Considering the following CodeQL pack(s) for inclusion in the custom bundle: {','.join(packs)}" 133 | ) 134 | 135 | if len(packs) > 0: 136 | selected_packs = [ 137 | available_pack 138 | for available_pack in packs_in_workspace 139 | if available_pack.config.name in packs 140 | ] 141 | else: 142 | selected_packs = packs_in_workspace 143 | 144 | missing_packs = set(packs) - {pack.config.name for pack in selected_packs} 145 | if len(missing_packs) > 0: 146 | logger.fatal( 147 | f"The provided CodeQL workspace doesn't contain the provided pack(s) '{','.join(missing_packs)}'", 148 | ) 149 | sys.exit(1) 150 | 151 | logger.info( 152 | f"Adding the pack(s) {','.join(map(lambda p: p.config.name, selected_packs))} and its workspace dependencies to the custom bundle." 153 | ) 154 | bundle.add_packs(*selected_packs) 155 | if additional_data_config: 156 | logger.info( 157 | f"Installing additions specified in the config file {additional_data_config} to the custom bundle." 158 | ) 159 | bundle.add_files_and_certs(additional_data_config, workspace) 160 | if code_scanning_config: 161 | logger.info( 162 | f"Adding the Code Scanning configuration file {code_scanning_config} to the custom bundle." 163 | ) 164 | bundle.add_code_scanning_config(code_scanning_config) 165 | logger.info(f"Bundling custom bundle(s) at {output}") 166 | platforms = set(map(BundlePlatform.from_string, platform)) 167 | bundle.bundle(output, platforms) 168 | logger.info(f"Completed building of custom bundle(s).") 169 | except CodeQLException as e: 170 | logger.fatal(f"Failed executing CodeQL command with reason: '{e}'") 171 | sys.exit(1) 172 | except BundleException as e: 173 | logger.fatal(f"Failed to build custom bundle with reason: '{e}'") 174 | sys.exit(1) 175 | 176 | 177 | if __name__ == "__main__": 178 | main() 179 | -------------------------------------------------------------------------------- /codeql_bundle/helpers/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/advanced-security/codeql-bundle/c65d9c36496679e444d125aa52f71335025fbb6a/codeql_bundle/helpers/__init__.py -------------------------------------------------------------------------------- /codeql_bundle/helpers/bundle.py: -------------------------------------------------------------------------------- 1 | from .codeql import CodeQL, CodeQLException, CodeQLPack 2 | from pathlib import Path 3 | from tempfile import TemporaryDirectory 4 | import tarfile 5 | from typing import List, cast, Callable, Optional 6 | from collections import defaultdict 7 | import shutil 8 | import yaml 9 | import dataclasses 10 | import logging 11 | import json 12 | import os 13 | import subprocess 14 | from jsonschema import validate, ValidationError 15 | from enum import Enum, verify, UNIQUE 16 | from dataclasses import dataclass 17 | from graphlib import TopologicalSorter 18 | import platform 19 | import concurrent.futures 20 | 21 | logger = logging.getLogger(__name__) 22 | 23 | 24 | @verify(UNIQUE) 25 | class CodeQLPackKind(Enum): 26 | QUERY_PACK = 1 27 | LIBRARY_PACK = 2 28 | CUSTOMIZATION_PACK = 3 29 | 30 | 31 | @dataclass(kw_only=True, frozen=True, eq=True) 32 | class ResolvedCodeQLPack(CodeQLPack): 33 | kind: CodeQLPackKind 34 | dependencies: List["ResolvedCodeQLPack"] = dataclasses.field(default_factory=list) 35 | 36 | def __hash__(self): 37 | return CodeQLPack.__hash__(self) 38 | 39 | def is_customizable(self) -> bool: 40 | return self.get_customizations_module_path().exists() 41 | 42 | def get_module_name(self) -> str: 43 | return self.config.name.replace("-", "_").replace("/", ".") 44 | 45 | def get_customizations_module_path(self) -> Path: 46 | return self.path.parent / "Customizations.qll" 47 | 48 | def get_lock_file_path(self) -> Path: 49 | return self.path.parent / "codeql-pack.lock.yml" 50 | 51 | def get_dependencies_path(self) -> Path: 52 | return self.path.parent / ".codeql" 53 | 54 | def get_cache_path(self) -> Path: 55 | return self.path.parent / ".cache" 56 | 57 | def is_stdlib_module(self) -> bool: 58 | return self.config.get_scope() == "codeql" 59 | 60 | 61 | class BundleException(Exception): 62 | pass 63 | 64 | 65 | class PackResolverException(Exception): 66 | pass 67 | 68 | 69 | def build_pack_resolver( 70 | packs: List[CodeQLPack], already_resolved_packs: List[ResolvedCodeQLPack] = [] 71 | ) -> Callable[[CodeQLPack], ResolvedCodeQLPack]: 72 | def builder() -> Callable[[CodeQLPack], ResolvedCodeQLPack]: 73 | resolved_packs: dict[CodeQLPack, ResolvedCodeQLPack] = { 74 | pack: pack for pack in already_resolved_packs 75 | } 76 | 77 | candidates: dict[str, List[CodeQLPack]] = defaultdict(list) 78 | for pack in packs + already_resolved_packs: 79 | candidates[pack.config.name].append(pack) 80 | 81 | def get_pack_kind(pack: CodeQLPack) -> CodeQLPackKind: 82 | kind = CodeQLPackKind.QUERY_PACK 83 | if pack.config.library: 84 | if ( 85 | pack.path.parent 86 | / pack.config.name.replace("-", "_") 87 | / "Customizations.qll" 88 | ).exists(): 89 | kind = CodeQLPackKind.CUSTOMIZATION_PACK 90 | else: 91 | kind = CodeQLPackKind.LIBRARY_PACK 92 | return kind 93 | 94 | def resolve(pack: CodeQLPack) -> ResolvedCodeQLPack: 95 | def inner(pack_to_be_resolved: CodeQLPack) -> ResolvedCodeQLPack: 96 | logger.debug( 97 | f"Resolving pack {pack_to_be_resolved.config.name}@{pack_to_be_resolved.config.version}" 98 | ) 99 | if pack_to_be_resolved in resolved_packs: 100 | logger.debug( 101 | f"Resolved pack {pack_to_be_resolved.config.name}@{pack_to_be_resolved.config.version}, already resolved." 102 | ) 103 | return resolved_packs[pack_to_be_resolved] 104 | else: 105 | resolved_deps: List[ResolvedCodeQLPack] = [] 106 | for ( 107 | dep_name, 108 | dep_version, 109 | ) in pack_to_be_resolved.config.dependencies.items(): 110 | logger.debug(f"Resolving dependency {dep_name}:{dep_version}.") 111 | resolved_dep = None 112 | for candidate_pack in candidates[dep_name]: 113 | logger.debug( 114 | f"Considering candidate pack {candidate_pack.config.name}@{candidate_pack.config.version}." 115 | ) 116 | if candidate_pack == pack: 117 | raise PackResolverException( 118 | f"Pack {pack.config.name}@{str(pack.config.version)} (transitively) depends on itself via {pack_to_be_resolved.config.name}@{str(pack_to_be_resolved.config.version)}!" 119 | ) 120 | if dep_version.match(candidate_pack.config.version): 121 | logger.debug( 122 | f"Found candidate pack {candidate_pack.config.name}@{candidate_pack.config.version}." 123 | ) 124 | resolved_dep = inner(candidate_pack) 125 | 126 | if not resolved_dep: 127 | raise PackResolverException( 128 | f"Could not resolve dependency {dep_name}@{dep_version} for pack {pack_to_be_resolved.config.name}@{str(pack_to_be_resolved.config.version)}!" 129 | ) 130 | resolved_deps.append(resolved_dep) 131 | 132 | resolved_pack = ResolvedCodeQLPack( 133 | path=pack_to_be_resolved.path, 134 | config=pack_to_be_resolved.config, 135 | kind=get_pack_kind(pack_to_be_resolved), 136 | dependencies=resolved_deps, 137 | ) 138 | resolved_packs[pack_to_be_resolved] = resolved_pack 139 | return resolved_pack 140 | 141 | return inner(pack) 142 | 143 | return resolve 144 | 145 | return builder() 146 | 147 | 148 | @verify(UNIQUE) 149 | class BundlePlatform(Enum): 150 | LINUX = 1 151 | WINDOWS = 2 152 | OSX = 3 153 | 154 | @staticmethod 155 | def from_string(platform: str) -> "BundlePlatform": 156 | if platform.lower() == "linux" or platform.lower() == "linux64": 157 | return BundlePlatform.LINUX 158 | elif platform.lower() == "windows" or platform.lower() == "win64": 159 | return BundlePlatform.WINDOWS 160 | elif platform.lower() == "osx" or platform.lower() == "osx64": 161 | return BundlePlatform.OSX 162 | else: 163 | raise BundleException(f"Invalid platform {platform}") 164 | 165 | def __str__(self): 166 | if self == BundlePlatform.LINUX: 167 | return "linux64" 168 | elif self == BundlePlatform.WINDOWS: 169 | return "win64" 170 | elif self == BundlePlatform.OSX: 171 | return "osx64" 172 | else: 173 | raise BundleException(f"Invalid platform {self}") 174 | 175 | 176 | class Bundle: 177 | def __init__(self, bundle_path: Path) -> None: 178 | self.tmp_dir = TemporaryDirectory() 179 | self.disable_precompilation = False 180 | 181 | if bundle_path.is_dir(): 182 | self.bundle_path = Path(self.tmp_dir.name) / bundle_path.name 183 | shutil.copytree( 184 | bundle_path, 185 | self.bundle_path, 186 | ) 187 | elif bundle_path.is_file() and bundle_path.name.endswith(".tar.gz"): 188 | logging.info( 189 | f"Unpacking provided bundle {bundle_path} to {self.tmp_dir.name}." 190 | ) 191 | file = tarfile.open(bundle_path) 192 | file.extractall(self.tmp_dir.name) 193 | self.bundle_path = Path(self.tmp_dir.name) / "codeql" 194 | else: 195 | raise BundleException("Invalid CodeQL bundle path") 196 | 197 | def supports_linux() -> set[BundlePlatform]: 198 | if (self.bundle_path / "cpp" / "tools" / "linux64").exists(): 199 | return {BundlePlatform.LINUX} 200 | else: 201 | return set() 202 | 203 | def supports_macos() -> set[BundlePlatform]: 204 | if (self.bundle_path / "cpp" / "tools" / "osx64").exists(): 205 | return {BundlePlatform.OSX} 206 | else: 207 | return set() 208 | 209 | def supports_windows() -> set[BundlePlatform]: 210 | if (self.bundle_path / "cpp" / "tools" / "win64").exists(): 211 | return {BundlePlatform.WINDOWS} 212 | else: 213 | return set() 214 | 215 | self.platforms: set[BundlePlatform] = ( 216 | supports_linux() | supports_macos() | supports_windows() 217 | ) 218 | 219 | current_system = platform.system() 220 | if not current_system in ["Linux", "Darwin", "Windows"]: 221 | raise BundleException(f"Unsupported system: {current_system}") 222 | if current_system == "Linux" and BundlePlatform.LINUX not in self.platforms: 223 | raise BundleException("Bundle doesn't support Linux!") 224 | elif current_system == "Darwin" and BundlePlatform.OSX not in self.platforms: 225 | raise BundleException("Bundle doesn't support OSX!") 226 | elif ( 227 | current_system == "Windows" and BundlePlatform.WINDOWS not in self.platforms 228 | ): 229 | raise BundleException("Bundle doesn't support Windows!") 230 | 231 | self.codeql = CodeQL(self.bundle_codeql_exe) 232 | 233 | try: 234 | logging.info(f"Validating the CodeQL CLI version part of the bundle.") 235 | unpacked_location = self.codeql.unpacked_location() 236 | logging.debug(f"Found CodeQL CLI in {str(unpacked_location)}.") 237 | version = self.codeql.version() 238 | logging.info(f"Found CodeQL CLI version {version}.") 239 | 240 | logging.debug(f"Resolving packs in {self.bundle_path}.") 241 | packs: List[CodeQLPack] = self.codeql.pack_ls(self.bundle_path) 242 | resolve = build_pack_resolver(packs) 243 | 244 | self.bundle_packs: list[ResolvedCodeQLPack] = [ 245 | resolve(pack) for pack in packs 246 | ] 247 | 248 | self.languages = self.codeql.resolve_languages() 249 | 250 | except CodeQLException: 251 | raise BundleException("Cannot determine CodeQL version!") 252 | 253 | def __del__(self) -> None: 254 | if self.tmp_dir: 255 | logging.info( 256 | f"Removing temporary directory {self.tmp_dir.name} used to build custom bundle." 257 | ) 258 | self.tmp_dir.cleanup() 259 | 260 | def get_bundle_packs(self) -> List[ResolvedCodeQLPack]: 261 | return self.bundle_packs 262 | 263 | def supports_platform(self, platform: BundlePlatform) -> bool: 264 | return platform in self.platforms 265 | 266 | @property 267 | def bundle_codeql_exe(self): 268 | if platform.system() == "Windows": 269 | return self.bundle_path / "codeql.exe" 270 | 271 | return self.bundle_path / "codeql" 272 | 273 | @property 274 | def disable_precompilation(self): 275 | return self._disable_precompilation 276 | 277 | @disable_precompilation.setter 278 | def disable_precompilation(self, value: bool): 279 | self._disable_precompilation = value 280 | 281 | 282 | class CustomBundle(Bundle): 283 | def __init__(self, bundle_path: Path, workspace_path: Path = Path.cwd()) -> None: 284 | Bundle.__init__(self, bundle_path) 285 | 286 | packs: List[CodeQLPack] = self.codeql.pack_ls(workspace_path) 287 | # Perform a sanity check on the packs in the workspace. 288 | for pack in packs: 289 | if not pack.config.get_scope(): 290 | raise BundleException( 291 | f"Pack '{pack.config.name}' does not have the required scope. This pack cannot be bundled!" 292 | ) 293 | 294 | resolve = build_pack_resolver(packs, self.bundle_packs) 295 | try: 296 | self.workspace_packs: list[ResolvedCodeQLPack] = [ 297 | resolve(pack) for pack in packs 298 | ] 299 | except PackResolverException as e: 300 | raise BundleException(e) 301 | 302 | self.available_packs: dict[str, ResolvedCodeQLPack] = { 303 | pack.config.name: pack for pack in self.bundle_packs + self.workspace_packs 304 | } 305 | # A custom bundle will always need a temp directory for customization work. 306 | # If the bundle didn't create one (there was no need to unpack it), create it here. 307 | if not self.tmp_dir: 308 | self.tmp_dir = TemporaryDirectory() 309 | logging.debug( 310 | f"Bundle doesn't have an associated temporary directory, created {self.tmp_dir.name} for building a custom bundle." 311 | ) 312 | 313 | def get_workspace_packs(self) -> List[ResolvedCodeQLPack]: 314 | return self.workspace_packs 315 | 316 | def add_packs(self, *packs: ResolvedCodeQLPack): 317 | """ 318 | Add packs and their workspace dependencies to the bundle. Standard library packs are customized if needed and standard query packs are recreated. 319 | 320 | The approach taken is to first create a dependency graph from the provided packs and their dependencies. 321 | During the dependency graph construction we track which standard library packs are customized by customization packs and add those and 322 | the standard query packs depending on the customized standard library packs to the graph. 323 | 324 | Once the dependency graph is constructed we use the graph to determine the order in which to process the packs. 325 | For each pack kind we process the pack as necessary. 326 | Library packs are bundled, query packs are (re)created, and customization packs are bundle and added as a dependency to the standard library pack they customize. 327 | Last but not least, the `Customizations.qll` module is updated to import the customization packs whenever we re-bundle a standard library pack. 328 | 329 | During the process a few hacks are applied. The customization packs that are bundled have their dependencies removed to prevent circular dependencies between the 330 | customization packs and the standard library pack they customize. 331 | Languages that do not have a `Customizations.qll` module are provided with one. This process will add the `Customizations.qll` module to the standard library pack 332 | and import as the first module in the language module (eg., `cpp.qll` will import `Customizations.qll` as the first module). 333 | """ 334 | # Keep a map of standard library packs to their customization packs so we know which need to be modified. 335 | std_lib_deps: dict[ResolvedCodeQLPack, List[ResolvedCodeQLPack]] = defaultdict( 336 | list 337 | ) 338 | pack_sorter: TopologicalSorter[ResolvedCodeQLPack] = TopologicalSorter() 339 | 340 | def add_to_graph( 341 | pack: ResolvedCodeQLPack, 342 | processed_packs: set[ResolvedCodeQLPack], 343 | std_lib_deps: dict[ResolvedCodeQLPack, List[ResolvedCodeQLPack]], 344 | ): 345 | # Only process workspace packs in this function 346 | if not pack in self.workspace_packs: 347 | logger.debug( 348 | f"Skipping adding pack {pack.config.name}@{str(pack.config.version)} to dependency graph" 349 | ) 350 | return 351 | if pack.kind == CodeQLPackKind.CUSTOMIZATION_PACK: 352 | logger.debug( 353 | f"Adding customization pack {pack.config.name}@{str(pack.config.version)} to dependency graph" 354 | ) 355 | pack_sorter.add(pack) 356 | std_lib_deps[pack.dependencies[0]].append(pack) 357 | else: 358 | # If the query pack relies on a customization pack (e.g. for tests), add the std lib dependency of 359 | # the customization pack to query pack because the customization pack will no longer have that 360 | # dependency in the bundle. 361 | if pack.kind == CodeQLPackKind.QUERY_PACK: 362 | for customization_pack in [ 363 | dep 364 | for dep in pack.dependencies 365 | if dep.kind == CodeQLPackKind.CUSTOMIZATION_PACK 366 | ]: 367 | std_lib_dep = customization_pack.dependencies[0] 368 | if not std_lib_dep in pack.dependencies: 369 | logger.debug( 370 | f"Adding stdlib dependency {std_lib_dep.config.name}@{str(std_lib_dep.config.version)} to {pack.config.name}@{str(pack.config.version)}" 371 | ) 372 | pack.dependencies.append(std_lib_dep) 373 | logger.debug( 374 | f"Adding pack {pack.config.name}@{str(pack.config.version)} to dependency graph" 375 | ) 376 | # We include standard library packs in the dependency graph to ensure they dictate the correct order 377 | # when we need to customize packs. 378 | # This does mean we will repack them, but that is only small price to pay for simplicity. 379 | pack_sorter.add(pack, *pack.dependencies) 380 | for dep in pack.dependencies: 381 | if dep not in processed_packs: 382 | add_to_graph(dep, processed_packs, std_lib_deps) 383 | processed_packs.add(pack) 384 | 385 | processed_packs: set[ResolvedCodeQLPack] = set() 386 | for pack in packs: 387 | if not pack in processed_packs: 388 | add_to_graph(pack, processed_packs, std_lib_deps) 389 | 390 | def is_dependent_on( 391 | pack: ResolvedCodeQLPack, other: ResolvedCodeQLPack 392 | ) -> bool: 393 | return other in pack.dependencies or any( 394 | map(lambda p: is_dependent_on(p, other), pack.dependencies) 395 | ) 396 | 397 | # Add the stdlib and its dependencies to properly sort the customization packs before the other packs. 398 | for pack, deps in std_lib_deps.items(): 399 | logger.debug( 400 | f"Adding standard library pack {pack.config.name}@{str(pack.config.version)} to dependency graph" 401 | ) 402 | pack_sorter.add(pack, *deps) 403 | # Add the standard query packs that rely transitively on the stdlib. 404 | for query_pack in [ 405 | p 406 | for p in self.bundle_packs 407 | if p.kind == CodeQLPackKind.QUERY_PACK and is_dependent_on(p, pack) 408 | ]: 409 | logger.debug( 410 | f"Adding standard query pack {query_pack.config.name}@{str(query_pack.config.version)} to dependency graph" 411 | ) 412 | pack_sorter.add(query_pack, pack) 413 | 414 | def bundle_customization_pack(customization_pack: ResolvedCodeQLPack): 415 | logging.info( 416 | f"Bundling the customization pack {customization_pack.config.name}." 417 | ) 418 | customization_pack_copy = copy_pack(customization_pack) 419 | 420 | # Remove the target dependency to prevent a circular dependency in the target. 421 | logging.debug( 422 | f"Removing dependency on standard library to prevent circular dependency." 423 | ) 424 | with customization_pack_copy.path.open("r") as fd: 425 | qlpack_spec = yaml.safe_load(fd) 426 | 427 | # Assume there is only one dependency and it is the standard library. 428 | qlpack_spec["dependencies"] = {} 429 | with customization_pack_copy.path.open("w") as fd: 430 | yaml.dump(qlpack_spec, fd) 431 | 432 | logging.debug( 433 | f"Bundling the customization pack {customization_pack_copy.config.name} at {customization_pack_copy.path}" 434 | ) 435 | self.codeql.pack_bundle( 436 | customization_pack_copy, 437 | self.bundle_path / "qlpacks", 438 | disable_precompilation=self.disable_precompilation, 439 | ) 440 | 441 | def copy_pack(pack: ResolvedCodeQLPack) -> ResolvedCodeQLPack: 442 | pack_copy_dir = ( 443 | Path(self.tmp_dir.name) 444 | / "temp" # Add a temp path segment because the standard library packs have scope 'codeql' that collides with the 'codeql' directory in the bundle that is extracted to the temporary directory. 445 | / cast(str, pack.config.get_scope()) 446 | / pack.config.get_pack_name() 447 | / str(pack.config.version) 448 | ) 449 | 450 | logging.debug( 451 | f"Copying {pack.path.parent} to {pack_copy_dir} for modification" 452 | ) 453 | shutil.copytree( 454 | pack.path.parent, 455 | pack_copy_dir, 456 | ) 457 | pack_copy_path = pack_copy_dir / pack.path.name 458 | return dataclasses.replace(pack, path=pack_copy_path) 459 | 460 | def add_customization_support(pack: ResolvedCodeQLPack): 461 | if pack.is_customizable(): 462 | return 463 | 464 | if not pack.config.get_scope() == "codeql" or not pack.config.library: 465 | return 466 | 467 | logging.debug( 468 | f"Standard library CodeQL pack {pack.config.name} does not have a 'Customizations' library, attempting to add one." 469 | ) 470 | # Assume the CodeQL library pack has name `-all`. 471 | target_language = pack.config.get_pack_name().removesuffix("-all") 472 | target_language_library_path = pack.path.parent / f"{target_language}.qll" 473 | logging.debug( 474 | f"Looking for standard library language module {target_language_library_path.name}" 475 | ) 476 | if not target_language_library_path.exists(): 477 | raise BundleException( 478 | f"Unable to customize {pack.config.name}, because it doesn't have a 'Customizations' library and we cannot determine the language library." 479 | ) 480 | logging.debug( 481 | f"Found standard library language module {target_language_library_path.name}, adding import of 'Customizations' library." 482 | ) 483 | with target_language_library_path.open("r") as fd: 484 | target_language_library_lines = fd.readlines() 485 | logging.debug(f"Looking for the first import statement.") 486 | 487 | first_import_idx = None 488 | for idx, line in enumerate(target_language_library_lines): 489 | if line.startswith("import"): 490 | first_import_idx = idx 491 | break 492 | if first_import_idx == None: 493 | raise BundleException( 494 | f"Unable to customize {pack.config.name}, because we cannot determine the first import statement of {target_language_library_path.name}." 495 | ) 496 | logging.debug( 497 | "Found first import statement and prepending import statement importing 'Customizations'" 498 | ) 499 | target_language_library_lines.insert( 500 | first_import_idx, "import Customizations\n" 501 | ) 502 | with target_language_library_path.open("w") as fd: 503 | fd.writelines(target_language_library_lines) 504 | logging.debug( 505 | f"Writing modified language library to {target_language_library_path}" 506 | ) 507 | 508 | target_customization_library_path = pack.path.parent / "Customizations.qll" 509 | logging.debug( 510 | f"Creating Customizations library with import of language {target_language}" 511 | ) 512 | with target_customization_library_path.open("w") as fd: 513 | fd.write(f"import {target_language}\n") 514 | 515 | def bundle_stdlib_pack(pack: ResolvedCodeQLPack): 516 | logging.info(f"Bundling the standard library pack {pack.config.name}.") 517 | 518 | pack_copy = copy_pack(pack) 519 | 520 | with pack_copy.path.open("r") as fd: 521 | qlpack_spec = yaml.safe_load(fd) 522 | if not "dependencies" in qlpack_spec: 523 | qlpack_spec["dependencies"] = {} 524 | for customization_pack in std_lib_deps[pack]: 525 | logging.debug( 526 | f"Adding dependency {customization_pack.config.name} to {pack_copy.config.name}" 527 | ) 528 | qlpack_spec["dependencies"][customization_pack.config.name] = str( 529 | customization_pack.config.version 530 | ) 531 | with pack_copy.path.open("w") as fd: 532 | yaml.dump(qlpack_spec, fd) 533 | 534 | logging.debug( 535 | f"Determining if standard library CodeQL library pack {pack_copy.config.name} is customizable." 536 | ) 537 | if not pack_copy.is_customizable(): 538 | add_customization_support(pack_copy) 539 | 540 | logging.debug( 541 | f"Updating 'Customizations.qll' with imports of customization libraries." 542 | ) 543 | with pack_copy.get_customizations_module_path().open("r") as fd: 544 | contents = fd.readlines() 545 | for customization_pack in std_lib_deps[pack]: 546 | contents.append( 547 | f"import {customization_pack.get_module_name()}.Customizations" 548 | ) 549 | with pack_copy.get_customizations_module_path().open("w") as fd: 550 | fd.writelines(map(lambda content: content + "\n", contents)) 551 | 552 | # Remove the original target library pack 553 | logging.debug( 554 | f"Removing the standard library at {pack.path} in preparation for replacement." 555 | ) 556 | shutil.rmtree(pack.path.parent.parent) 557 | # Bundle the new into the bundle. 558 | logging.debug( 559 | f"Bundling the standard library pack {pack_copy.config.name} at {pack_copy.path}" 560 | ) 561 | self.codeql.pack_bundle( 562 | pack_copy, 563 | self.bundle_path / "qlpacks", 564 | disable_precompilation=self.disable_precompilation, 565 | ) 566 | 567 | def bundle_library_pack(library_pack: ResolvedCodeQLPack): 568 | logging.info(f"Bundling the library pack {library_pack.config.name}.") 569 | 570 | pack_copy = copy_pack(library_pack) 571 | 572 | self.codeql.pack_bundle( 573 | pack_copy, 574 | self.bundle_path / "qlpacks", 575 | disable_precompilation=self.disable_precompilation, 576 | ) 577 | 578 | def bundle_query_pack(pack: ResolvedCodeQLPack): 579 | if pack.config.get_scope() == "codeql": 580 | logging.info(f"Bundling the standard query pack {pack.config.name}.") 581 | pack_copy = copy_pack(pack) 582 | 583 | # Remove the lock file 584 | logging.debug( 585 | f"Removing CodeQL pack lock file {pack_copy.get_lock_file_path()}" 586 | ) 587 | pack_copy.get_lock_file_path().unlink() 588 | # Remove the included dependencies 589 | logging.debug( 590 | f"Removing CodeQL query pack dependencies directory {pack_copy.get_dependencies_path()}" 591 | ) 592 | shutil.rmtree(pack_copy.get_dependencies_path()) 593 | # Remove the query cache, if it exists. 594 | logging.debug( 595 | f"Removing CodeQL query pack cache directory {pack_copy.get_cache_path()}, if it exists." 596 | ) 597 | shutil.rmtree( 598 | pack_copy.get_cache_path(), 599 | ignore_errors=True, 600 | ) 601 | # Remove qlx files 602 | if self.codeql.supports_qlx(): 603 | logging.debug(f"Removing 'qlx' files in {pack_copy.path.parent}.") 604 | for qlx_path in pack_copy.path.parent.glob("**/*.qlx"): 605 | qlx_path.unlink() 606 | 607 | # Remove the original query pack 608 | logging.debug( 609 | f"Removing the standard library query pack directory {pack.path.parent.parent} in preparation for recreation." 610 | ) 611 | shutil.rmtree(pack.path.parent.parent) 612 | logging.debug( 613 | f"Recreating {pack_copy.config.name} at {pack_copy.path} to {self.bundle_path / 'qlpacks'}" 614 | ) 615 | # Recompile the query pack with the assumption that all its dependencies are now in the bundle. 616 | self.codeql.pack_create( 617 | pack_copy, self.bundle_path / "qlpacks", self.bundle_path 618 | ) 619 | else: 620 | logging.info(f"Bundling the query pack {pack.config.name}.") 621 | pack_copy = copy_pack(pack) 622 | # Rewrite the query pack dependencies 623 | with pack_copy.path.open("r") as fd: 624 | qlpack_spec = yaml.safe_load(fd) 625 | 626 | # Assume there is only one dependency and it is the standard library. 627 | qlpack_spec["dependencies"] = { 628 | pack.config.name: str(pack.config.version) 629 | for pack in pack_copy.dependencies 630 | } 631 | logging.debug(f"Rewriting dependencies for {pack.config.name}.") 632 | with pack_copy.path.open("w") as fd: 633 | yaml.dump(qlpack_spec, fd) 634 | 635 | self.codeql.pack_create(pack_copy, self.bundle_path / "qlpacks") 636 | 637 | sorted_packs = list(pack_sorter.static_order()) 638 | logger.debug( 639 | f"Sorted packs: {' -> '.join(map(lambda p: p.config.name, sorted_packs))}" 640 | ) 641 | for pack in sorted_packs: 642 | if pack.kind == CodeQLPackKind.CUSTOMIZATION_PACK: 643 | bundle_customization_pack(pack) 644 | elif pack.kind == CodeQLPackKind.LIBRARY_PACK: 645 | if pack.config.get_scope() == "codeql": 646 | bundle_stdlib_pack(pack) 647 | else: 648 | bundle_library_pack(pack) 649 | elif pack.kind == CodeQLPackKind.QUERY_PACK: 650 | bundle_query_pack(pack) 651 | 652 | def add_files_and_certs(self, config_path: Path, workspace_path: Path): 653 | schema = { 654 | "type": "object", 655 | "properties": { 656 | "CodeQLBundleAdditionalFiles": { 657 | "type": "array", 658 | "items": { 659 | "type": "object", 660 | "properties": { 661 | "Source": {"type": "string"}, 662 | "Destination": {"type": "string"}, 663 | }, 664 | "required": ["Source", "Destination"], 665 | "additionalProperties": False, 666 | }, 667 | "minItems": 1, 668 | }, 669 | "CodeQLBundleAdditionalCertificates": { 670 | "type": "array", 671 | "items": { 672 | "type": "object", 673 | "properties": {"Source": {"type": "string"}}, 674 | "required": ["Source"], 675 | "additionalProperties": False, 676 | }, 677 | "minItems": 1, 678 | }, 679 | }, 680 | "additionalProperties": True, 681 | } 682 | 683 | def validate_config(config) -> bool: 684 | try: 685 | validate(instance=config, schema=schema) 686 | except ValidationError as e: 687 | print("JSON validation error:", e) 688 | return False 689 | return True 690 | 691 | def load_config(file_path): 692 | with open(file_path, "r") as file: 693 | data = json.load(file) 694 | if validate_config(data): 695 | return data 696 | else: 697 | raise BundleException( 698 | f"Installation config {file_path} is not valid." 699 | ) 700 | 701 | def is_unsafe_path(basedir: Path, path: Path) -> bool: 702 | matchpath = os.path.realpath(path) 703 | basedir = os.path.realpath(basedir) 704 | return os.path.commonpath([basedir, matchpath]) != basedir 705 | 706 | if not config_path.exists(): 707 | raise BundleException(f"Installation config {config_path} does not exist.") 708 | if not config_path.is_file(): 709 | raise BundleException(f"Installation config {config_path} is not a file.") 710 | 711 | config = load_config(config_path) 712 | 713 | if platform.system() == "Windows": 714 | keytool = "tools/win64/java/bin/keytool.exe" 715 | elif platform.system() == "Linux": 716 | keytool = "tools/linux64/java/bin/keytool" 717 | elif platform.system() == "Darwin": 718 | keytool = "tools/osx64/java/bin/keytool" 719 | else: 720 | raise BundleException(f"Unsupported platform {platform.system()}") 721 | 722 | keytool = self.bundle_path / keytool 723 | if not keytool.exists(): 724 | raise BundleException(f"Keytool {keytool} does not exist.") 725 | 726 | keystores: list[str] = [ 727 | "tools/win64/java/lib/security/cacerts", 728 | "tools/linux64/java/lib/security/cacerts", 729 | "tools/osx64/java/lib/security/cacerts", 730 | "tools/osx64/java-aarch64/lib/security/cacerts", 731 | ] 732 | 733 | # Add the certificates to the Java keystores 734 | if "CodeQLBundleAdditionalCertificates" in config: 735 | for cert in config["CodeQLBundleAdditionalCertificates"]: 736 | src = workspace_path / Path(cert["Source"]) 737 | src = src.resolve() 738 | if is_unsafe_path(workspace_path, src): 739 | raise BundleException( 740 | f"Certificate file {src} is not in the workspace path." 741 | ) 742 | if not src.exists(): 743 | raise BundleException(f"Certificate file {src} does not exist.") 744 | 745 | for keystore in keystores: 746 | keystore = self.bundle_path / keystore 747 | if not keystore.exists(): 748 | raise BundleException(f"Keystore {keystore} does not exist.") 749 | logging.info(f"Adding certificate {src} to keystore {keystore}") 750 | subprocess.run( 751 | [ 752 | str(keytool), 753 | "-import", 754 | "-trustcacerts", 755 | "-alias", 756 | "root", 757 | "-file", 758 | str(src), 759 | "-keystore", 760 | str(keystore), 761 | "-storepass", 762 | "changeit", 763 | "-noprompt", 764 | ], 765 | check=True, 766 | cwd=self.bundle_path, 767 | ) 768 | 769 | # Add additional files to the bundle 770 | if "CodeQLBundleAdditionalFiles" in config: 771 | for file in config["CodeQLBundleAdditionalFiles"]: 772 | src = (workspace_path / Path(file["Source"])).resolve() 773 | dst = (self.bundle_path / Path(file["Destination"])).resolve() 774 | 775 | if not src.exists(): 776 | raise BundleException(f"Source file {src} does not exist.") 777 | 778 | if is_unsafe_path(workspace_path, src): 779 | raise BundleException( 780 | f"Source file {src} is not in the workspace path." 781 | ) 782 | if is_unsafe_path(self.bundle_path, dst): 783 | print(self.bundle_path) 784 | raise BundleException( 785 | f"Destination path {dst} is not in the bundle path." 786 | ) 787 | 788 | if src.is_dir(): 789 | logging.info(f"Copying directory {src} to {dst}") 790 | shutil.copytree(src, dst, dirs_exist_ok=True) 791 | else: 792 | logging.info(f"Copying file {src} to {dst}") 793 | dst.parent.mkdir(parents=True, exist_ok=True) 794 | shutil.copy(src, dst) 795 | 796 | def add_code_scanning_config(self, default_config: Path): 797 | if not default_config.exists(): 798 | raise BundleException(f"Default config {default_config} does not exist.") 799 | if not default_config.is_file(): 800 | raise BundleException(f"Default config {default_config} is not a file.") 801 | shutil.copy(default_config, self.bundle_path / "default-codeql-config.yml") 802 | 803 | def bundle( 804 | self, 805 | output_path: Path, 806 | platforms: set[BundlePlatform] = set(), 807 | default_config: Optional[Path] = None, 808 | ): 809 | if len(platforms) == 0: 810 | if output_path.is_dir(): 811 | output_path = output_path / "codeql-bundle.tar.gz" 812 | 813 | logging.debug(f"Bundling custom bundle to {output_path}.") 814 | with tarfile.open(output_path, mode="w:gz") as bundle_archive: 815 | bundle_archive.add(self.bundle_path, arcname="codeql") 816 | else: 817 | if not output_path.is_dir(): 818 | raise BundleException( 819 | f"Output path {output_path} must be a directory when bundling for multiple platforms." 820 | ) 821 | 822 | unsupported_platforms = platforms - self.platforms 823 | if len(unsupported_platforms) > 0: 824 | raise BundleException( 825 | f"Unsupported platform(s) {', '.join(map(str,unsupported_platforms))} specified. Use the platform agnostic bundle to bundle for different platforms." 826 | ) 827 | 828 | def create_bundle_for_platform( 829 | bundle_output_path: Path, platform: BundlePlatform 830 | ) -> None: 831 | """Create a bundle for a single platform.""" 832 | 833 | def filter_for_platform( 834 | platform: BundlePlatform, 835 | ) -> Callable[[tarfile.TarInfo], Optional[tarfile.TarInfo]]: 836 | """Create a filter function that will only include files for the specified platform.""" 837 | relative_tools_paths = [ 838 | Path(lang) / "tools" for lang in self.languages 839 | ] + [Path("tools")] 840 | 841 | def get_nonplatform_tool_paths( 842 | platform: BundlePlatform, 843 | ) -> List[Path]: 844 | """Get a list of paths to tools that are not for the specified platform relative to the root of a bundle.""" 845 | specialize_path: Optional[Callable[[Path], List[Path]]] = None 846 | linux64_subpaths = [Path("linux64"), Path("linux")] 847 | osx64_subpaths = [Path("osx64"), Path("macos")] 848 | win64_subpaths = [Path("win64"), Path("windows")] 849 | if platform == BundlePlatform.LINUX: 850 | specialize_path = lambda p: [ 851 | p / subpath 852 | for subpath in osx64_subpaths + win64_subpaths 853 | ] 854 | elif platform == BundlePlatform.WINDOWS: 855 | specialize_path = lambda p: [ 856 | p / subpath 857 | for subpath in osx64_subpaths + linux64_subpaths 858 | ] 859 | elif platform == BundlePlatform.OSX: 860 | specialize_path = lambda p: [ 861 | p / subpath 862 | for subpath in linux64_subpaths + win64_subpaths 863 | ] 864 | else: 865 | raise BundleException(f"Unsupported platform {platform}.") 866 | 867 | return [ 868 | candidate 869 | for candidates in map(specialize_path, relative_tools_paths) 870 | for candidate in candidates 871 | ] 872 | 873 | def filter(tarinfo: tarfile.TarInfo) -> Optional[tarfile.TarInfo]: 874 | tarfile_path = Path(tarinfo.name) 875 | 876 | exclusion_paths = get_nonplatform_tool_paths(platform) 877 | 878 | # Manual exclusions based on diffing the contents of the platform specific bundles and the generated platform specific bundles. 879 | if platform != BundlePlatform.WINDOWS: 880 | exclusion_paths.append(Path("codeql.exe")) 881 | else: 882 | exclusion_paths.append(Path("swift/qltest")) 883 | exclusion_paths.append(Path("swift/resource-dir")) 884 | 885 | if platform == BundlePlatform.LINUX: 886 | exclusion_paths.append(Path("swift/qltest/osx64")) 887 | exclusion_paths.append(Path("swift/resource-dir/osx64")) 888 | 889 | if platform == BundlePlatform.OSX: 890 | exclusion_paths.append(Path("swift/qltest/linux64")) 891 | exclusion_paths.append(Path("swift/resource-dir/linux64")) 892 | 893 | tarfile_path_root = Path(tarfile_path.parts[0]) 894 | exclusion_paths = [ 895 | tarfile_path_root / path for path in exclusion_paths 896 | ] 897 | 898 | if any( 899 | tarfile_path.is_relative_to(path) 900 | for path in exclusion_paths 901 | ): 902 | return None 903 | 904 | return tarinfo 905 | 906 | return filter 907 | 908 | logging.debug( 909 | f"Bundling custom bundle for {platform} to {bundle_output_path}." 910 | ) 911 | with tarfile.open(bundle_output_path, mode="w:gz") as bundle_archive: 912 | bundle_archive.add( 913 | self.bundle_path, 914 | arcname="codeql", 915 | filter=filter_for_platform(platform), 916 | ) 917 | 918 | with concurrent.futures.ThreadPoolExecutor( 919 | max_workers=len(platforms) 920 | ) as executor: 921 | future_to_platform = { 922 | executor.submit( 923 | create_bundle_for_platform, 924 | output_path / f"codeql-bundle-{platform}.tar.gz", 925 | platform, 926 | ): platform 927 | for platform in platforms 928 | } 929 | for future in concurrent.futures.as_completed(future_to_platform): 930 | platform = future_to_platform[future] 931 | try: 932 | future.result() 933 | except Exception as exc: 934 | raise BundleException( 935 | f"Failed to create bundle for platform {platform} with exception: {exc}." 936 | ) 937 | -------------------------------------------------------------------------------- /codeql_bundle/helpers/codeql.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | import json 3 | from semantic_version import Version, NpmSpec 4 | from pathlib import Path 5 | from typing import Dict, Any, Iterable, Self, Optional, List 6 | import yaml 7 | from dataclasses import dataclass, fields, field 8 | import logging 9 | 10 | logger = logging.getLogger(__name__) 11 | 12 | 13 | class CodeQLException(Exception): 14 | pass 15 | 16 | 17 | @dataclass(kw_only=True, frozen=True, eq=True) 18 | class CodeQLPackConfig: 19 | library: bool = False 20 | name: str 21 | version: Version = Version("0.0.0") 22 | dependencies: Dict[str, NpmSpec] = field(default_factory=dict) 23 | extractor: Optional[str] = None 24 | 25 | @classmethod 26 | def from_dict(cls, dict_: Dict[str, Any]) -> Self: 27 | fieldset = {f.name for f in fields(cls) if f.init} 28 | 29 | def _convert_value(k : str, v : Any) -> Any: 30 | if k == "version": 31 | return Version(v) 32 | elif k == "dependencies": 33 | return {k: NpmSpec(v) for k, v in v.items()} 34 | else: 35 | return v 36 | 37 | filtered_dict = {k: _convert_value(k, v) for k, v in dict_.items() if k in fieldset} 38 | return cls(**filtered_dict) 39 | 40 | def get_scope(self) -> Optional[str]: 41 | if "/" in self.name: 42 | return self.name.split("/")[0] 43 | else: 44 | return None 45 | 46 | def get_pack_name(self) -> str: 47 | if self.get_scope() != None: 48 | return self.name.split("/")[1] 49 | else: 50 | return self.name 51 | 52 | def __hash__(self): 53 | return hash(f"{self.name}@{str(self.version)}") 54 | 55 | @dataclass(kw_only=True, frozen=True, eq=True) 56 | class CodeQLPack: 57 | path: Path 58 | config: CodeQLPackConfig 59 | 60 | def __hash__(self) -> int: 61 | return hash(f"{self.path}") 62 | 63 | class CodeQL: 64 | def __init__(self, codeql_path: Path): 65 | self.codeql_path = codeql_path 66 | self._version = None 67 | 68 | @property 69 | def disable_precompilation(self): 70 | return self._disable_precompilation 71 | 72 | @disable_precompilation.setter 73 | def disable_precompilation(self, value: bool): 74 | self._disable_precompilation = value 75 | 76 | def _exec(self, command: str, *args: str) -> subprocess.CompletedProcess[str]: 77 | logger.debug( 78 | f"Running CodeQL command: {command} with arguments: {' '.join(args)}" 79 | ) 80 | return subprocess.run( 81 | [f"{self.codeql_path}", command] + [arg for arg in args], 82 | capture_output=True, 83 | text=True 84 | ) 85 | 86 | def version(self) -> Version: 87 | if self._version != None: 88 | return self._version 89 | else: 90 | cp = self._exec("version", "--format=json") 91 | if cp.returncode == 0: 92 | version_info = json.loads(cp.stdout) 93 | self._version = Version(version_info["version"]) 94 | return self._version 95 | else: 96 | raise CodeQLException(f"Failed to run {cp.args} command!") 97 | 98 | def unpacked_location(self) -> Path: 99 | cp = self._exec("version", "--format=json") 100 | if cp.returncode == 0: 101 | version_info = json.loads(cp.stdout) 102 | return Path(version_info["unpackedLocation"]) 103 | else: 104 | raise CodeQLException(f"Failed to run {cp.args} command!") 105 | 106 | def supports_qlx(self) -> bool: 107 | return self.version() >= Version("2.11.4") 108 | 109 | def pack_ls(self, workspace: Path = Path.cwd()) -> List[CodeQLPack]: 110 | cp = self._exec("pack", "ls", "--format=json", str(workspace)) 111 | if cp.returncode == 0: 112 | packs: Iterable[Path] = map(Path, json.loads(cp.stdout)["packs"].keys()) 113 | 114 | def load(qlpack_yml_path: Path) -> CodeQLPack: 115 | with qlpack_yml_path.open("r") as qlpack_yml_file: 116 | logger.debug(f"Loading CodeQL pack configuration at {qlpack_yml_path}.") 117 | qlpack_yml_as_dict: Dict[str, Any] = yaml.safe_load(qlpack_yml_file) 118 | qlpack_config = CodeQLPackConfig.from_dict(qlpack_yml_as_dict) 119 | qlpack = CodeQLPack(path=qlpack_yml_path, config=qlpack_config) 120 | logger.debug( 121 | f"Loaded {qlpack.config.name} with version {str(qlpack.config.version)} at {qlpack.path}." 122 | ) 123 | return qlpack 124 | 125 | logger.debug(f"Listing CodeQL packs for workspace {workspace}") 126 | return list(map(load, packs)) 127 | else: 128 | raise CodeQLException(f"Failed to run {cp.args} command! {cp.stderr}") 129 | 130 | def pack_bundle( 131 | self, 132 | pack: CodeQLPack, 133 | output_path: Path, 134 | *additional_packs: Path, 135 | disable_precompilation = False 136 | ): 137 | if not pack.config.library: 138 | raise CodeQLException(f"Cannot bundle non-library pack {pack.config.name}!") 139 | 140 | args = ["bundle", "--format=json", f"--pack-path={output_path}"] 141 | if disable_precompilation: 142 | args.append("--no-precompile") 143 | logging.warn( 144 | f"NOTE: Precompilation is disabled for {pack.config.name}! This may result in slower query execution." 145 | ) 146 | 147 | if len(additional_packs) > 0: 148 | args.append(f"--additional-packs={':'.join(map(str,additional_packs))}") 149 | cp = self._exec( 150 | "pack", 151 | *args, 152 | "--", 153 | str(pack.path.parent), 154 | ) 155 | 156 | if cp.returncode != 0: 157 | raise CodeQLException(f"Failed to run {cp.args} command! {cp.stderr}") 158 | 159 | def pack_create( 160 | self, 161 | pack: CodeQLPack, 162 | output_path: Path, 163 | *additional_packs: Path, 164 | disable_precompilation = False 165 | ): 166 | if pack.config.library: 167 | raise CodeQLException(f"Cannot bundle non-query pack {pack.config.name}!") 168 | 169 | args = ["create", "--format=json", f"--output={output_path}", "--threads=0", "--no-default-compilation-cache"] 170 | if disable_precompilation: 171 | args.append("--no-precompile") 172 | logging.warn( 173 | f"NOTE: Precompilation is disabled for {pack.config.name}! This may result in slower query execution." 174 | ) 175 | 176 | if self.supports_qlx(): 177 | args.append("--qlx") 178 | if len(additional_packs) > 0: 179 | args.append(f"--additional-packs={':'.join(map(str,additional_packs))}") 180 | cp = self._exec( 181 | "pack", 182 | *args, 183 | "--", 184 | str(pack.path.parent), 185 | ) 186 | 187 | if cp.returncode != 0: 188 | raise CodeQLException(f"Failed to run {cp.args} command! {cp.stderr}") 189 | 190 | def resolve_languages(self) -> set[str]: 191 | cp = self._exec("resolve", "languages", "--format=json") 192 | if cp.returncode == 0: 193 | return set(json.loads(cp.stdout).keys()) 194 | else: 195 | raise CodeQLException(f"Failed to run {cp.args} command! {cp.stderr}") 196 | -------------------------------------------------------------------------------- /poetry.lock: -------------------------------------------------------------------------------- 1 | # This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand. 2 | 3 | [[package]] 4 | name = "attrs" 5 | version = "23.2.0" 6 | description = "Classes Without Boilerplate" 7 | optional = false 8 | python-versions = ">=3.7" 9 | files = [ 10 | {file = "attrs-23.2.0-py3-none-any.whl", hash = "sha256:99b87a485a5820b23b879f04c2305b44b951b502fd64be915879d77a7e8fc6f1"}, 11 | {file = "attrs-23.2.0.tar.gz", hash = "sha256:935dc3b529c262f6cf76e50877d35a4bd3c1de194fd41f47a2b7ae8f19971f30"}, 12 | ] 13 | 14 | [package.extras] 15 | cov = ["attrs[tests]", "coverage[toml] (>=5.3)"] 16 | dev = ["attrs[tests]", "pre-commit"] 17 | docs = ["furo", "myst-parser", "sphinx", "sphinx-notfound-page", "sphinxcontrib-towncrier", "towncrier", "zope-interface"] 18 | tests = ["attrs[tests-no-zope]", "zope-interface"] 19 | tests-mypy = ["mypy (>=1.6)", "pytest-mypy-plugins"] 20 | tests-no-zope = ["attrs[tests-mypy]", "cloudpickle", "hypothesis", "pympler", "pytest (>=4.3.0)", "pytest-xdist[psutil]"] 21 | 22 | [[package]] 23 | name = "click" 24 | version = "8.1.3" 25 | description = "Composable command line interface toolkit" 26 | optional = false 27 | python-versions = ">=3.7" 28 | files = [ 29 | {file = "click-8.1.3-py3-none-any.whl", hash = "sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48"}, 30 | {file = "click-8.1.3.tar.gz", hash = "sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e"}, 31 | ] 32 | 33 | [package.dependencies] 34 | colorama = {version = "*", markers = "platform_system == \"Windows\""} 35 | 36 | [[package]] 37 | name = "colorama" 38 | version = "0.4.6" 39 | description = "Cross-platform colored terminal text." 40 | optional = false 41 | python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,!=3.6.*,>=2.7" 42 | files = [ 43 | {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, 44 | {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, 45 | ] 46 | 47 | [[package]] 48 | name = "jsonschema" 49 | version = "4.22.0" 50 | description = "An implementation of JSON Schema validation for Python" 51 | optional = false 52 | python-versions = ">=3.8" 53 | files = [ 54 | {file = "jsonschema-4.22.0-py3-none-any.whl", hash = "sha256:ff4cfd6b1367a40e7bc6411caec72effadd3db0bbe5017de188f2d6108335802"}, 55 | {file = "jsonschema-4.22.0.tar.gz", hash = "sha256:5b22d434a45935119af990552c862e5d6d564e8f6601206b305a61fdf661a2b7"}, 56 | ] 57 | 58 | [package.dependencies] 59 | attrs = ">=22.2.0" 60 | jsonschema-specifications = ">=2023.03.6" 61 | referencing = ">=0.28.4" 62 | rpds-py = ">=0.7.1" 63 | 64 | [package.extras] 65 | format = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3987", "uri-template", "webcolors (>=1.11)"] 66 | format-nongpl = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3986-validator (>0.1.0)", "uri-template", "webcolors (>=1.11)"] 67 | 68 | [[package]] 69 | name = "jsonschema-specifications" 70 | version = "2023.12.1" 71 | description = "The JSON Schema meta-schemas and vocabularies, exposed as a Registry" 72 | optional = false 73 | python-versions = ">=3.8" 74 | files = [ 75 | {file = "jsonschema_specifications-2023.12.1-py3-none-any.whl", hash = "sha256:87e4fdf3a94858b8a2ba2778d9ba57d8a9cafca7c7489c46ba0d30a8bc6a9c3c"}, 76 | {file = "jsonschema_specifications-2023.12.1.tar.gz", hash = "sha256:48a76787b3e70f5ed53f1160d2b81f586e4ca6d1548c5de7085d1682674764cc"}, 77 | ] 78 | 79 | [package.dependencies] 80 | referencing = ">=0.31.0" 81 | 82 | [[package]] 83 | name = "pyyaml" 84 | version = "6.0" 85 | description = "YAML parser and emitter for Python" 86 | optional = false 87 | python-versions = ">=3.6" 88 | files = [ 89 | {file = "PyYAML-6.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d4db7c7aef085872ef65a8fd7d6d09a14ae91f691dec3e87ee5ee0539d516f53"}, 90 | {file = "PyYAML-6.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9df7ed3b3d2e0ecfe09e14741b857df43adb5a3ddadc919a2d94fbdf78fea53c"}, 91 | {file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:77f396e6ef4c73fdc33a9157446466f1cff553d979bd00ecb64385760c6babdc"}, 92 | {file = "PyYAML-6.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a80a78046a72361de73f8f395f1f1e49f956c6be882eed58505a15f3e430962b"}, 93 | {file = "PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:f84fbc98b019fef2ee9a1cb3ce93e3187a6df0b2538a651bfb890254ba9f90b5"}, 94 | {file = "PyYAML-6.0-cp310-cp310-win32.whl", hash = "sha256:2cd5df3de48857ed0544b34e2d40e9fac445930039f3cfe4bcc592a1f836d513"}, 95 | {file = "PyYAML-6.0-cp310-cp310-win_amd64.whl", hash = "sha256:daf496c58a8c52083df09b80c860005194014c3698698d1a57cbcfa182142a3a"}, 96 | {file = "PyYAML-6.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:d4b0ba9512519522b118090257be113b9468d804b19d63c71dbcf4a48fa32358"}, 97 | {file = "PyYAML-6.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:81957921f441d50af23654aa6c5e5eaf9b06aba7f0a19c18a538dc7ef291c5a1"}, 98 | {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:afa17f5bc4d1b10afd4466fd3a44dc0e245382deca5b3c353d8b757f9e3ecb8d"}, 99 | {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dbad0e9d368bb989f4515da330b88a057617d16b6a8245084f1b05400f24609f"}, 100 | {file = "PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:432557aa2c09802be39460360ddffd48156e30721f5e8d917f01d31694216782"}, 101 | {file = "PyYAML-6.0-cp311-cp311-win32.whl", hash = "sha256:bfaef573a63ba8923503d27530362590ff4f576c626d86a9fed95822a8255fd7"}, 102 | {file = "PyYAML-6.0-cp311-cp311-win_amd64.whl", hash = "sha256:01b45c0191e6d66c470b6cf1b9531a771a83c1c4208272ead47a3ae4f2f603bf"}, 103 | {file = "PyYAML-6.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:897b80890765f037df3403d22bab41627ca8811ae55e9a722fd0392850ec4d86"}, 104 | {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:50602afada6d6cbfad699b0c7bb50d5ccffa7e46a3d738092afddc1f9758427f"}, 105 | {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:48c346915c114f5fdb3ead70312bd042a953a8ce5c7106d5bfb1a5254e47da92"}, 106 | {file = "PyYAML-6.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:98c4d36e99714e55cfbaaee6dd5badbc9a1ec339ebfc3b1f52e293aee6bb71a4"}, 107 | {file = "PyYAML-6.0-cp36-cp36m-win32.whl", hash = "sha256:0283c35a6a9fbf047493e3a0ce8d79ef5030852c51e9d911a27badfde0605293"}, 108 | {file = "PyYAML-6.0-cp36-cp36m-win_amd64.whl", hash = "sha256:07751360502caac1c067a8132d150cf3d61339af5691fe9e87803040dbc5db57"}, 109 | {file = "PyYAML-6.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:819b3830a1543db06c4d4b865e70ded25be52a2e0631ccd2f6a47a2822f2fd7c"}, 110 | {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:473f9edb243cb1935ab5a084eb238d842fb8f404ed2193a915d1784b5a6b5fc0"}, 111 | {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0ce82d761c532fe4ec3f87fc45688bdd3a4c1dc5e0b4a19814b9009a29baefd4"}, 112 | {file = "PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:231710d57adfd809ef5d34183b8ed1eeae3f76459c18fb4a0b373ad56bedcdd9"}, 113 | {file = "PyYAML-6.0-cp37-cp37m-win32.whl", hash = "sha256:c5687b8d43cf58545ade1fe3e055f70eac7a5a1a0bf42824308d868289a95737"}, 114 | {file = "PyYAML-6.0-cp37-cp37m-win_amd64.whl", hash = "sha256:d15a181d1ecd0d4270dc32edb46f7cb7733c7c508857278d3d378d14d606db2d"}, 115 | {file = "PyYAML-6.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:0b4624f379dab24d3725ffde76559cff63d9ec94e1736b556dacdfebe5ab6d4b"}, 116 | {file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:213c60cd50106436cc818accf5baa1aba61c0189ff610f64f4a3e8c6726218ba"}, 117 | {file = "PyYAML-6.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9fa600030013c4de8165339db93d182b9431076eb98eb40ee068700c9c813e34"}, 118 | {file = "PyYAML-6.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:277a0ef2981ca40581a47093e9e2d13b3f1fbbeffae064c1d21bfceba2030287"}, 119 | {file = "PyYAML-6.0-cp38-cp38-win32.whl", hash = "sha256:d4eccecf9adf6fbcc6861a38015c2a64f38b9d94838ac1810a9023a0609e1b78"}, 120 | {file = "PyYAML-6.0-cp38-cp38-win_amd64.whl", hash = "sha256:1e4747bc279b4f613a09eb64bba2ba602d8a6664c6ce6396a4d0cd413a50ce07"}, 121 | {file = "PyYAML-6.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:055d937d65826939cb044fc8c9b08889e8c743fdc6a32b33e2390f66013e449b"}, 122 | {file = "PyYAML-6.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e61ceaab6f49fb8bdfaa0f92c4b57bcfbea54c09277b1b4f7ac376bfb7a7c174"}, 123 | {file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d67d839ede4ed1b28a4e8909735fc992a923cdb84e618544973d7dfc71540803"}, 124 | {file = "PyYAML-6.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cba8c411ef271aa037d7357a2bc8f9ee8b58b9965831d9e51baf703280dc73d3"}, 125 | {file = "PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl", hash = "sha256:40527857252b61eacd1d9af500c3337ba8deb8fc298940291486c465c8b46ec0"}, 126 | {file = "PyYAML-6.0-cp39-cp39-win32.whl", hash = "sha256:b5b9eccad747aabaaffbc6064800670f0c297e52c12754eb1d976c57e4f74dcb"}, 127 | {file = "PyYAML-6.0-cp39-cp39-win_amd64.whl", hash = "sha256:b3d267842bf12586ba6c734f89d1f5b871df0273157918b0ccefa29deb05c21c"}, 128 | {file = "PyYAML-6.0.tar.gz", hash = "sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a2"}, 129 | ] 130 | 131 | [[package]] 132 | name = "referencing" 133 | version = "0.35.1" 134 | description = "JSON Referencing + Python" 135 | optional = false 136 | python-versions = ">=3.8" 137 | files = [ 138 | {file = "referencing-0.35.1-py3-none-any.whl", hash = "sha256:eda6d3234d62814d1c64e305c1331c9a3a6132da475ab6382eaa997b21ee75de"}, 139 | {file = "referencing-0.35.1.tar.gz", hash = "sha256:25b42124a6c8b632a425174f24087783efb348a6f1e0008e63cd4466fedf703c"}, 140 | ] 141 | 142 | [package.dependencies] 143 | attrs = ">=22.2.0" 144 | rpds-py = ">=0.7.0" 145 | 146 | [[package]] 147 | name = "rpds-py" 148 | version = "0.18.0" 149 | description = "Python bindings to Rust's persistent data structures (rpds)" 150 | optional = false 151 | python-versions = ">=3.8" 152 | files = [ 153 | {file = "rpds_py-0.18.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:5b4e7d8d6c9b2e8ee2d55c90b59c707ca59bc30058269b3db7b1f8df5763557e"}, 154 | {file = "rpds_py-0.18.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c463ed05f9dfb9baebef68048aed8dcdc94411e4bf3d33a39ba97e271624f8f7"}, 155 | {file = "rpds_py-0.18.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:01e36a39af54a30f28b73096dd39b6802eddd04c90dbe161c1b8dbe22353189f"}, 156 | {file = "rpds_py-0.18.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d62dec4976954a23d7f91f2f4530852b0c7608116c257833922a896101336c51"}, 157 | {file = "rpds_py-0.18.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:dd18772815d5f008fa03d2b9a681ae38d5ae9f0e599f7dda233c439fcaa00d40"}, 158 | {file = "rpds_py-0.18.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:923d39efa3cfb7279a0327e337a7958bff00cc447fd07a25cddb0a1cc9a6d2da"}, 159 | {file = "rpds_py-0.18.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:39514da80f971362f9267c600b6d459bfbbc549cffc2cef8e47474fddc9b45b1"}, 160 | {file = "rpds_py-0.18.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a34d557a42aa28bd5c48a023c570219ba2593bcbbb8dc1b98d8cf5d529ab1434"}, 161 | {file = "rpds_py-0.18.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:93df1de2f7f7239dc9cc5a4a12408ee1598725036bd2dedadc14d94525192fc3"}, 162 | {file = "rpds_py-0.18.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:34b18ba135c687f4dac449aa5157d36e2cbb7c03cbea4ddbd88604e076aa836e"}, 163 | {file = "rpds_py-0.18.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:c0b5dcf9193625afd8ecc92312d6ed78781c46ecbf39af9ad4681fc9f464af88"}, 164 | {file = "rpds_py-0.18.0-cp310-none-win32.whl", hash = "sha256:c4325ff0442a12113a6379af66978c3fe562f846763287ef66bdc1d57925d337"}, 165 | {file = "rpds_py-0.18.0-cp310-none-win_amd64.whl", hash = "sha256:7223a2a5fe0d217e60a60cdae28d6949140dde9c3bcc714063c5b463065e3d66"}, 166 | {file = "rpds_py-0.18.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:3a96e0c6a41dcdba3a0a581bbf6c44bb863f27c541547fb4b9711fd8cf0ffad4"}, 167 | {file = "rpds_py-0.18.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:30f43887bbae0d49113cbaab729a112251a940e9b274536613097ab8b4899cf6"}, 168 | {file = "rpds_py-0.18.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fcb25daa9219b4cf3a0ab24b0eb9a5cc8949ed4dc72acb8fa16b7e1681aa3c58"}, 169 | {file = "rpds_py-0.18.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d68c93e381010662ab873fea609bf6c0f428b6d0bb00f2c6939782e0818d37bf"}, 170 | {file = "rpds_py-0.18.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b34b7aa8b261c1dbf7720b5d6f01f38243e9b9daf7e6b8bc1fd4657000062f2c"}, 171 | {file = "rpds_py-0.18.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2e6d75ab12b0bbab7215e5d40f1e5b738aa539598db27ef83b2ec46747df90e1"}, 172 | {file = "rpds_py-0.18.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0b8612cd233543a3781bc659c731b9d607de65890085098986dfd573fc2befe5"}, 173 | {file = "rpds_py-0.18.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:aec493917dd45e3c69d00a8874e7cbed844efd935595ef78a0f25f14312e33c6"}, 174 | {file = "rpds_py-0.18.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:661d25cbffaf8cc42e971dd570d87cb29a665f49f4abe1f9e76be9a5182c4688"}, 175 | {file = "rpds_py-0.18.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:1df3659d26f539ac74fb3b0c481cdf9d725386e3552c6fa2974f4d33d78e544b"}, 176 | {file = "rpds_py-0.18.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:a1ce3ba137ed54f83e56fb983a5859a27d43a40188ba798993812fed73c70836"}, 177 | {file = "rpds_py-0.18.0-cp311-none-win32.whl", hash = "sha256:69e64831e22a6b377772e7fb337533c365085b31619005802a79242fee620bc1"}, 178 | {file = "rpds_py-0.18.0-cp311-none-win_amd64.whl", hash = "sha256:998e33ad22dc7ec7e030b3df701c43630b5bc0d8fbc2267653577e3fec279afa"}, 179 | {file = "rpds_py-0.18.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:7f2facbd386dd60cbbf1a794181e6aa0bd429bd78bfdf775436020172e2a23f0"}, 180 | {file = "rpds_py-0.18.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1d9a5be316c15ffb2b3c405c4ff14448c36b4435be062a7f578ccd8b01f0c4d8"}, 181 | {file = "rpds_py-0.18.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cd5bf1af8efe569654bbef5a3e0a56eca45f87cfcffab31dd8dde70da5982475"}, 182 | {file = "rpds_py-0.18.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5417558f6887e9b6b65b4527232553c139b57ec42c64570569b155262ac0754f"}, 183 | {file = "rpds_py-0.18.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:56a737287efecafc16f6d067c2ea0117abadcd078d58721f967952db329a3e5c"}, 184 | {file = "rpds_py-0.18.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8f03bccbd8586e9dd37219bce4d4e0d3ab492e6b3b533e973fa08a112cb2ffc9"}, 185 | {file = "rpds_py-0.18.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4457a94da0d5c53dc4b3e4de1158bdab077db23c53232f37a3cb7afdb053a4e3"}, 186 | {file = "rpds_py-0.18.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:0ab39c1ba9023914297dd88ec3b3b3c3f33671baeb6acf82ad7ce883f6e8e157"}, 187 | {file = "rpds_py-0.18.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:9d54553c1136b50fd12cc17e5b11ad07374c316df307e4cfd6441bea5fb68496"}, 188 | {file = "rpds_py-0.18.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:0af039631b6de0397ab2ba16eaf2872e9f8fca391b44d3d8cac317860a700a3f"}, 189 | {file = "rpds_py-0.18.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:84ffab12db93b5f6bad84c712c92060a2d321b35c3c9960b43d08d0f639d60d7"}, 190 | {file = "rpds_py-0.18.0-cp312-none-win32.whl", hash = "sha256:685537e07897f173abcf67258bee3c05c374fa6fff89d4c7e42fb391b0605e98"}, 191 | {file = "rpds_py-0.18.0-cp312-none-win_amd64.whl", hash = "sha256:e003b002ec72c8d5a3e3da2989c7d6065b47d9eaa70cd8808b5384fbb970f4ec"}, 192 | {file = "rpds_py-0.18.0-cp38-cp38-macosx_10_12_x86_64.whl", hash = "sha256:08f9ad53c3f31dfb4baa00da22f1e862900f45908383c062c27628754af2e88e"}, 193 | {file = "rpds_py-0.18.0-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:c0013fe6b46aa496a6749c77e00a3eb07952832ad6166bd481c74bda0dcb6d58"}, 194 | {file = "rpds_py-0.18.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e32a92116d4f2a80b629778280103d2a510a5b3f6314ceccd6e38006b5e92dcb"}, 195 | {file = "rpds_py-0.18.0-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e541ec6f2ec456934fd279a3120f856cd0aedd209fc3852eca563f81738f6861"}, 196 | {file = "rpds_py-0.18.0-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:bed88b9a458e354014d662d47e7a5baafd7ff81c780fd91584a10d6ec842cb73"}, 197 | {file = "rpds_py-0.18.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2644e47de560eb7bd55c20fc59f6daa04682655c58d08185a9b95c1970fa1e07"}, 198 | {file = "rpds_py-0.18.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8e8916ae4c720529e18afa0b879473049e95949bf97042e938530e072fde061d"}, 199 | {file = "rpds_py-0.18.0-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:465a3eb5659338cf2a9243e50ad9b2296fa15061736d6e26240e713522b6235c"}, 200 | {file = "rpds_py-0.18.0-cp38-cp38-musllinux_1_2_aarch64.whl", hash = "sha256:ea7d4a99f3b38c37eac212dbd6ec42b7a5ec51e2c74b5d3223e43c811609e65f"}, 201 | {file = "rpds_py-0.18.0-cp38-cp38-musllinux_1_2_i686.whl", hash = "sha256:67071a6171e92b6da534b8ae326505f7c18022c6f19072a81dcf40db2638767c"}, 202 | {file = "rpds_py-0.18.0-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:41ef53e7c58aa4ef281da975f62c258950f54b76ec8e45941e93a3d1d8580594"}, 203 | {file = "rpds_py-0.18.0-cp38-none-win32.whl", hash = "sha256:fdea4952db2793c4ad0bdccd27c1d8fdd1423a92f04598bc39425bcc2b8ee46e"}, 204 | {file = "rpds_py-0.18.0-cp38-none-win_amd64.whl", hash = "sha256:7cd863afe7336c62ec78d7d1349a2f34c007a3cc6c2369d667c65aeec412a5b1"}, 205 | {file = "rpds_py-0.18.0-cp39-cp39-macosx_10_12_x86_64.whl", hash = "sha256:5307def11a35f5ae4581a0b658b0af8178c65c530e94893345bebf41cc139d33"}, 206 | {file = "rpds_py-0.18.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:77f195baa60a54ef9d2de16fbbfd3ff8b04edc0c0140a761b56c267ac11aa467"}, 207 | {file = "rpds_py-0.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:39f5441553f1c2aed4de4377178ad8ff8f9d733723d6c66d983d75341de265ab"}, 208 | {file = "rpds_py-0.18.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:9a00312dea9310d4cb7dbd7787e722d2e86a95c2db92fbd7d0155f97127bcb40"}, 209 | {file = "rpds_py-0.18.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8f2fc11e8fe034ee3c34d316d0ad8808f45bc3b9ce5857ff29d513f3ff2923a1"}, 210 | {file = "rpds_py-0.18.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:586f8204935b9ec884500498ccc91aa869fc652c40c093bd9e1471fbcc25c022"}, 211 | {file = "rpds_py-0.18.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ddc2f4dfd396c7bfa18e6ce371cba60e4cf9d2e5cdb71376aa2da264605b60b9"}, 212 | {file = "rpds_py-0.18.0-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5ddcba87675b6d509139d1b521e0c8250e967e63b5909a7e8f8944d0f90ff36f"}, 213 | {file = "rpds_py-0.18.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:7bd339195d84439cbe5771546fe8a4e8a7a045417d8f9de9a368c434e42a721e"}, 214 | {file = "rpds_py-0.18.0-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:d7c36232a90d4755b720fbd76739d8891732b18cf240a9c645d75f00639a9024"}, 215 | {file = "rpds_py-0.18.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:6b0817e34942b2ca527b0e9298373e7cc75f429e8da2055607f4931fded23e20"}, 216 | {file = "rpds_py-0.18.0-cp39-none-win32.whl", hash = "sha256:99f70b740dc04d09e6b2699b675874367885217a2e9f782bdf5395632ac663b7"}, 217 | {file = "rpds_py-0.18.0-cp39-none-win_amd64.whl", hash = "sha256:6ef687afab047554a2d366e112dd187b62d261d49eb79b77e386f94644363294"}, 218 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-macosx_10_12_x86_64.whl", hash = "sha256:ad36cfb355e24f1bd37cac88c112cd7730873f20fb0bdaf8ba59eedf8216079f"}, 219 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:36b3ee798c58ace201289024b52788161e1ea133e4ac93fba7d49da5fec0ef9e"}, 220 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f8a2f084546cc59ea99fda8e070be2fd140c3092dc11524a71aa8f0f3d5a55ca"}, 221 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e4461d0f003a0aa9be2bdd1b798a041f177189c1a0f7619fe8c95ad08d9a45d7"}, 222 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8db715ebe3bb7d86d77ac1826f7d67ec11a70dbd2376b7cc214199360517b641"}, 223 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:793968759cd0d96cac1e367afd70c235867831983f876a53389ad869b043c948"}, 224 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:66e6a3af5a75363d2c9a48b07cb27c4ea542938b1a2e93b15a503cdfa8490795"}, 225 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6ef0befbb5d79cf32d0266f5cff01545602344eda89480e1dd88aca964260b18"}, 226 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:1d4acf42190d449d5e89654d5c1ed3a4f17925eec71f05e2a41414689cda02d1"}, 227 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-musllinux_1_2_i686.whl", hash = "sha256:a5f446dd5055667aabaee78487f2b5ab72e244f9bc0b2ffebfeec79051679984"}, 228 | {file = "rpds_py-0.18.0-pp310-pypy310_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:9dbbeb27f4e70bfd9eec1be5477517365afe05a9b2c441a0b21929ee61048124"}, 229 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-macosx_10_12_x86_64.whl", hash = "sha256:22806714311a69fd0af9b35b7be97c18a0fc2826e6827dbb3a8c94eac6cf7eeb"}, 230 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-macosx_11_0_arm64.whl", hash = "sha256:b34ae4636dfc4e76a438ab826a0d1eed2589ca7d9a1b2d5bb546978ac6485461"}, 231 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8c8370641f1a7f0e0669ddccca22f1da893cef7628396431eb445d46d893e5cd"}, 232 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:c8362467a0fdeccd47935f22c256bec5e6abe543bf0d66e3d3d57a8fb5731863"}, 233 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:11a8c85ef4a07a7638180bf04fe189d12757c696eb41f310d2426895356dcf05"}, 234 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b316144e85316da2723f9d8dc75bada12fa58489a527091fa1d5a612643d1a0e"}, 235 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:cf1ea2e34868f6fbf070e1af291c8180480310173de0b0c43fc38a02929fc0e3"}, 236 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:e546e768d08ad55b20b11dbb78a745151acbd938f8f00d0cfbabe8b0199b9880"}, 237 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:4901165d170a5fde6f589acb90a6b33629ad1ec976d4529e769c6f3d885e3e80"}, 238 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-musllinux_1_2_i686.whl", hash = "sha256:618a3d6cae6ef8ec88bb76dd80b83cfe415ad4f1d942ca2a903bf6b6ff97a2da"}, 239 | {file = "rpds_py-0.18.0-pp38-pypy38_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:ed4eb745efbff0a8e9587d22a84be94a5eb7d2d99c02dacf7bd0911713ed14dd"}, 240 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-macosx_10_12_x86_64.whl", hash = "sha256:6c81e5f372cd0dc5dc4809553d34f832f60a46034a5f187756d9b90586c2c307"}, 241 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-macosx_11_0_arm64.whl", hash = "sha256:43fbac5f22e25bee1d482c97474f930a353542855f05c1161fd804c9dc74a09d"}, 242 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6d7faa6f14017c0b1e69f5e2c357b998731ea75a442ab3841c0dbbbfe902d2c4"}, 243 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:08231ac30a842bd04daabc4d71fddd7e6d26189406d5a69535638e4dcb88fe76"}, 244 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:044a3e61a7c2dafacae99d1e722cc2d4c05280790ec5a05031b3876809d89a5c"}, 245 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3f26b5bd1079acdb0c7a5645e350fe54d16b17bfc5e71f371c449383d3342e17"}, 246 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:482103aed1dfe2f3b71a58eff35ba105289b8d862551ea576bd15479aba01f66"}, 247 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1374f4129f9bcca53a1bba0bb86bf78325a0374577cf7e9e4cd046b1e6f20e24"}, 248 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:635dc434ff724b178cb192c70016cc0ad25a275228f749ee0daf0eddbc8183b1"}, 249 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-musllinux_1_2_i686.whl", hash = "sha256:bc362ee4e314870a70f4ae88772d72d877246537d9f8cb8f7eacf10884862432"}, 250 | {file = "rpds_py-0.18.0-pp39-pypy39_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:4832d7d380477521a8c1644bbab6588dfedea5e30a7d967b5fb75977c45fd77f"}, 251 | {file = "rpds_py-0.18.0.tar.gz", hash = "sha256:42821446ee7a76f5d9f71f9e33a4fb2ffd724bb3e7f93386150b61a43115788d"}, 252 | ] 253 | 254 | [[package]] 255 | name = "semantic-version" 256 | version = "2.10.0" 257 | description = "A library implementing the 'SemVer' scheme." 258 | optional = false 259 | python-versions = ">=2.7" 260 | files = [ 261 | {file = "semantic_version-2.10.0-py2.py3-none-any.whl", hash = "sha256:de78a3b8e0feda74cabc54aab2da702113e33ac9d9eb9d2389bcf1f58b7d9177"}, 262 | {file = "semantic_version-2.10.0.tar.gz", hash = "sha256:bdabb6d336998cbb378d4b9db3a4b56a1e3235701dc05ea2690d9a997ed5041c"}, 263 | ] 264 | 265 | [package.extras] 266 | dev = ["Django (>=1.11)", "check-manifest", "colorama (<=0.4.1)", "coverage", "flake8", "nose2", "readme-renderer (<25.0)", "tox", "wheel", "zest.releaser[recommended]"] 267 | doc = ["Sphinx", "sphinx-rtd-theme"] 268 | 269 | [metadata] 270 | lock-version = "2.0" 271 | python-versions = "^3.11" 272 | content-hash = "9602b58c5eec044f93ac2563400b8cbeb650c73af658e2ac91733dff25ba5645" 273 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.poetry] 2 | name = "codeql-bundle" 3 | version = "0.4.1" 4 | description = "Tool to create custom CodeQL bundles" 5 | authors = ["Remco Vermeulen "] 6 | readme = "README.md" 7 | packages = [{include = "codeql_bundle"}] 8 | license = "MIT" 9 | repository = "https://github.com/advanced-security/codeql-bundle" 10 | 11 | [tool.poetry.dependencies] 12 | python = "^3.11" 13 | click = "^8.1.3" 14 | semantic-version = "^2.10.0" 15 | pyyaml = "^6.0" 16 | jsonschema = "^4.22.0" 17 | 18 | [tool.poetry.scripts] 19 | codeql-bundle = 'codeql_bundle.cli:main' 20 | 21 | [build-system] 22 | requires = ["poetry-core"] 23 | build-backend = "poetry.core.masonry.api" 24 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/advanced-security/codeql-bundle/c65d9c36496679e444d125aa52f71335025fbb6a/tests/__init__.py -------------------------------------------------------------------------------- /tests/bundle-diff.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | 3 | def main(argv : list[str]) -> int: 4 | if len(argv[1:]) != 2: 5 | print("Usage: bundle-diff.py ") 6 | return 1 7 | 8 | added : set[Path] = set() 9 | removed : set[Path] = set() 10 | 11 | bundle1 = Path(argv[1]) 12 | if not bundle1.is_dir(): 13 | print(f"Error: {bundle1} is not a directory") 14 | return 1 15 | bundle2 = Path(argv[2]) 16 | if not bundle2.is_dir(): 17 | print(f"Error: {bundle2} is not a directory") 18 | return 1 19 | 20 | bundle1 = bundle1.absolute() 21 | bundle2 = bundle2.absolute() 22 | 23 | for p in bundle1.glob("**/*"): 24 | subpath = p.absolute().relative_to(bundle1) 25 | #print(subpath) 26 | if not set(subpath.parents).isdisjoint(removed): 27 | continue 28 | path_in_bundle2 = bundle2 / subpath 29 | #print(path_in_bundle2) 30 | if not path_in_bundle2.exists(): 31 | removed.add(subpath) 32 | 33 | for p in bundle2.glob("**/*"): 34 | subpath = p.absolute().relative_to(bundle2) 35 | if not set(subpath.parents).isdisjoint(added): 36 | continue 37 | path_in_bundle1 = bundle1 / subpath 38 | if not path_in_bundle1.exists(): 39 | added.add(subpath) 40 | 41 | for p in sorted(added): 42 | print(f"+ {p}") 43 | 44 | for p in sorted(removed): 45 | print(f"- {p}") 46 | return 0 47 | 48 | if __name__ == "__main__": 49 | import sys 50 | sys.exit(main(sys.argv)) -------------------------------------------------------------------------------- /tests/workspace-with-cyclic-dep/codeql-workspace.yml: -------------------------------------------------------------------------------- 1 | provide: 2 | - "**/qlpack.yml" -------------------------------------------------------------------------------- /tests/workspace-with-cyclic-dep/x/qlpack.yml: -------------------------------------------------------------------------------- 1 | --- 2 | library: false 3 | warnOnImplicitThis: false 4 | name: cycle/x 5 | version: 0.0.1 6 | dependencies: 7 | "cycle/y": "*" 8 | -------------------------------------------------------------------------------- /tests/workspace-with-cyclic-dep/y/qlpack.yml: -------------------------------------------------------------------------------- 1 | --- 2 | library: false 3 | warnOnImplicitThis: false 4 | name: cycle/y 5 | version: 0.0.1 6 | dependencies: 7 | "cycle/z": "*" -------------------------------------------------------------------------------- /tests/workspace-with-cyclic-dep/z/qlpack.yml: -------------------------------------------------------------------------------- 1 | --- 2 | library: false 3 | warnOnImplicitThis: false 4 | name: cycle/z 5 | version: 0.0.1 6 | dependencies: 7 | "cycle/x": "*" 8 | -------------------------------------------------------------------------------- /tests/workspace/codeql-workspace.yml: -------------------------------------------------------------------------------- 1 | provide: 2 | - "**/qlpack.yml" -------------------------------------------------------------------------------- /tests/workspace/cpp/a/codeql-pack.lock.yml: -------------------------------------------------------------------------------- 1 | --- 2 | lockVersion: 1.0.0 3 | dependencies: 4 | codeql/cpp-all: 5 | version: 0.7.3 6 | codeql/ssa: 7 | version: 0.0.18 8 | codeql/tutorial: 9 | version: 0.0.11 10 | codeql/util: 11 | version: 0.0.11 12 | compiled: false 13 | -------------------------------------------------------------------------------- /tests/workspace/cpp/a/qlpack.yml: -------------------------------------------------------------------------------- 1 | --- 2 | library: true 3 | warnOnImplicitThis: false 4 | name: test/a 5 | version: 0.0.1 6 | dependencies: 7 | test/aa: "*" 8 | -------------------------------------------------------------------------------- /tests/workspace/cpp/aa/codeql-pack.lock.yml: -------------------------------------------------------------------------------- 1 | --- 2 | lockVersion: 1.0.0 3 | dependencies: 4 | codeql/cpp-all: 5 | version: 0.7.3 6 | codeql/ssa: 7 | version: 0.0.18 8 | codeql/tutorial: 9 | version: 0.0.11 10 | codeql/util: 11 | version: 0.0.11 12 | compiled: false 13 | -------------------------------------------------------------------------------- /tests/workspace/cpp/aa/qlpack.yml: -------------------------------------------------------------------------------- 1 | --- 2 | library: true 3 | warnOnImplicitThis: false 4 | name: test/aa 5 | version: 0.0.1 6 | dependencies: 7 | "codeql/cpp-all": "^0.8.0" 8 | -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-bundle-customizations-tests/FooExternalSourceFunction/Foo.expected: -------------------------------------------------------------------------------- 1 | | test.c:4:13:4:15 | call to foo | 2 | -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-bundle-customizations-tests/FooExternalSourceFunction/Foo.ql: -------------------------------------------------------------------------------- 1 | import cpp 2 | import semmle.code.cpp.security.FlowSources 3 | 4 | select any(RemoteFlowSource s) -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-bundle-customizations-tests/FooExternalSourceFunction/test.c: -------------------------------------------------------------------------------- 1 | int foo(); 2 | 3 | int main(int argc, char** argv) { 4 | int i = foo(); 5 | 6 | return i; 7 | } -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-bundle-customizations-tests/codeql-pack.lock.yml: -------------------------------------------------------------------------------- 1 | --- 2 | lockVersion: 1.0.0 3 | dependencies: 4 | codeql/cpp-all: 5 | version: 0.7.3 6 | codeql/ssa: 7 | version: 0.0.18 8 | codeql/tutorial: 9 | version: 0.0.11 10 | codeql/util: 11 | version: 0.0.11 12 | compiled: false 13 | -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-bundle-customizations-tests/qlpack.yml: -------------------------------------------------------------------------------- 1 | library: true 2 | name: foo/bundle-customizations-tests 3 | version: 0.0.1 4 | dependencies: 5 | "test/a": "*" 6 | extractor: cpp -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-customization-tests/FooExternalSourceFunction/Foo.expected: -------------------------------------------------------------------------------- 1 | | test.c:4:13:4:15 | call to foo | 2 | -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-customization-tests/FooExternalSourceFunction/Foo.ql: -------------------------------------------------------------------------------- 1 | import cpp 2 | import foo.cpp_customizations.Customizations 3 | import semmle.code.cpp.security.FlowSources 4 | 5 | select any(RemoteFlowSource s) -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-customization-tests/FooExternalSourceFunction/test.c: -------------------------------------------------------------------------------- 1 | int foo(); 2 | 3 | int main(int argc, char** argv) { 4 | int i = foo(); 5 | 6 | return i; 7 | } -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-customization-tests/codeql-pack.lock.yml: -------------------------------------------------------------------------------- 1 | --- 2 | lockVersion: 1.0.0 3 | dependencies: 4 | codeql/cpp-all: 5 | version: 0.7.3 6 | codeql/ssa: 7 | version: 0.0.18 8 | codeql/tutorial: 9 | version: 0.0.11 10 | codeql/util: 11 | version: 0.0.11 12 | compiled: false 13 | -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-customization-tests/qlpack.yml: -------------------------------------------------------------------------------- 1 | library: true 2 | name: foo/cpp-customizations-tests 3 | version: 0.0.1 4 | dependencies: 5 | "foo/cpp-customizations": "*" 6 | "test/a": "*" 7 | extractor: cpp -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-customizations/codeql-pack.lock.yml: -------------------------------------------------------------------------------- 1 | --- 2 | lockVersion: 1.0.0 3 | dependencies: 4 | codeql/cpp-all: 5 | version: 0.7.3 6 | codeql/ssa: 7 | version: 0.0.18 8 | codeql/tutorial: 9 | version: 0.0.11 10 | codeql/util: 11 | version: 0.0.11 12 | compiled: false 13 | -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-customizations/foo/cpp_customizations/Customizations.qll: -------------------------------------------------------------------------------- 1 | import cpp 2 | private import semmle.code.cpp.security.FlowSources 3 | 4 | module FooSources { 5 | private class FooExternalSourceFunction extends RemoteFlowSourceFunction { 6 | FooExternalSourceFunction() { this.hasName("foo") } 7 | 8 | override predicate hasRemoteFlowSource(FunctionOutput output, string description) { 9 | output.isReturnValue() and 10 | description = "value returned by " + this.getName() 11 | } 12 | } 13 | } 14 | -------------------------------------------------------------------------------- /tests/workspace/cpp/foo-customizations/qlpack.yml: -------------------------------------------------------------------------------- 1 | library: True 2 | name: foo/cpp-customizations 3 | version: 0.0.1 4 | dependencies: 5 | "codeql/cpp-all": "^0.8.0" --------------------------------------------------------------------------------