├── tests ├── __init__.py ├── api │ └── __init__.py ├── cli │ ├── __init__.py │ ├── resources │ │ ├── __init__.py │ │ ├── config.yaml │ │ └── custom_module │ │ │ └── __init__.py │ ├── snapshots │ │ └── test_init │ │ │ ├── test_init_project_exclude_optional │ │ │ ├── defaults.yaml │ │ │ ├── pipeline.yaml │ │ │ └── config.yaml │ │ │ └── test_init_project_include_optional │ │ │ ├── defaults.yaml │ │ │ ├── pipeline.yaml │ │ │ └── config.yaml │ └── test_init.py ├── utils │ ├── __init__.py │ ├── resources │ │ ├── __init__.py │ │ └── nested_base_settings.py │ └── test_yaml.py ├── compiler │ ├── __init__.py │ ├── resources │ │ ├── test.yaml │ │ ├── defaults_development.yaml │ │ ├── erroneous-file.yaml │ │ └── another-test.yaml │ └── test_yaml_loading.py ├── kubernetes │ └── __init__.py ├── manifests │ ├── __init__.py │ └── strimzi │ │ └── __init__.py ├── pipeline │ ├── __init__.py │ ├── resources │ │ ├── symlinked-folder │ │ ├── pipeline-symlinked │ │ │ └── pipeline.yaml │ │ ├── resetter_values │ │ │ ├── pipeline.yaml │ │ │ ├── pipeline_connector_only.yaml │ │ │ └── defaults.yaml │ │ ├── pipeline-folders-with-symlinks │ │ │ ├── pipeline-2 │ │ │ ├── pipeline-1 │ │ │ │ └── pipeline.yaml │ │ │ └── pipeline-3 │ │ │ │ └── pipeline.yaml │ │ ├── kafka-connect-sink-config │ │ │ ├── defaults.yaml │ │ │ ├── pipeline.yaml │ │ │ └── config.yaml │ │ ├── dotenv │ │ │ ├── .env │ │ │ ├── custom.env │ │ │ └── config.yaml │ │ ├── multi-config │ │ │ ├── config_production.yaml │ │ │ ├── config_development.yaml │ │ │ └── config.yaml │ │ ├── simple-pipeline │ │ │ └── pipeline.yaml │ │ ├── pipelines-with-graphs │ │ │ ├── simple-pipeline │ │ │ │ ├── pipeline.yaml │ │ │ │ ├── config.yaml │ │ │ │ └── defaults.yaml │ │ │ └── same-topic-and-component-name │ │ │ │ ├── pipeline.yaml │ │ │ │ ├── config.yaml │ │ │ │ └── defaults.yaml │ │ ├── no-topics-defaults │ │ │ ├── defaults_development.yaml │ │ │ └── defaults.yaml │ │ ├── pipeline-with-env-defaults │ │ │ ├── defaults_development.yaml │ │ │ ├── pipeline.yaml │ │ │ └── defaults.yaml │ │ ├── pipeline-duplicate-step-names │ │ │ └── pipeline.yaml │ │ ├── pipeline-folders │ │ │ ├── pipeline-1 │ │ │ │ └── pipeline.yaml │ │ │ ├── pipeline-2 │ │ │ │ └── pipeline.yaml │ │ │ └── pipeline-3 │ │ │ │ └── pipeline.yaml │ │ ├── pipeline-with-envs │ │ │ ├── pipeline_development.yaml │ │ │ └── pipeline.yaml │ │ ├── temp-trim-release-name │ │ │ ├── pipeline.yaml │ │ │ └── defaults.yaml │ │ ├── no-user-defined-components │ │ │ └── pipeline.yaml │ │ ├── custom-config │ │ │ ├── pipeline.yaml │ │ │ ├── config.yaml │ │ │ └── defaults.yaml │ │ ├── component-type-substitution │ │ │ ├── infinite_pipeline.yaml │ │ │ └── pipeline.yaml │ │ ├── name_prefix_concatenation │ │ │ └── pipeline.yaml │ │ ├── env-specific-config-only │ │ │ └── config_production.yaml │ │ ├── pipeline-component-should-have-prefix │ │ │ ├── pipeline.yaml │ │ │ └── defaults.yaml │ │ ├── pipeline-with-paths │ │ │ └── pipeline.yaml │ │ ├── parallel-pipeline │ │ │ ├── config.yaml │ │ │ ├── defaults.yaml │ │ │ └── pipeline.yaml │ │ ├── pipeline-with-loop │ │ │ ├── defaults.yaml │ │ │ └── pipeline.yaml │ │ ├── kafka-connect-sink │ │ │ └── pipeline.yaml │ │ ├── no-input-topic-pipeline │ │ │ └── pipeline.yaml │ │ ├── pipeline-with-inflate │ │ │ └── pipeline.yaml │ │ ├── pipeline-with-short-topics │ │ │ ├── pipeline.yaml │ │ │ └── defaults.yaml │ │ ├── manifest-pipeline │ │ │ ├── defaults.yaml │ │ │ └── pipeline.yaml │ │ ├── pipeline-with-illegal-kubernetes-name │ │ │ └── pipeline.yaml │ │ ├── first-pipeline │ │ │ └── pipeline.yaml │ │ ├── read-from-component │ │ │ └── pipeline.yaml │ │ └── streams-bootstrap │ │ │ ├── defaults.yaml │ │ │ └── pipeline.yaml │ ├── snapshots │ │ ├── test_manifest │ │ │ ├── test_manifest_clean_argo_mode │ │ │ │ └── manifest.yaml │ │ │ ├── test_manifest_destroy_argo_mode │ │ │ │ └── manifest.yaml │ │ │ ├── test_manifest_destroy_manifest_mode │ │ │ │ └── manifest.yaml │ │ │ └── test_manifest_destroy_python_api │ │ │ │ └── manifest.yaml │ │ └── test_generate │ │ │ ├── test_model_serialization │ │ │ └── pipeline.yaml │ │ │ ├── test_prefix_pipeline_component │ │ │ └── pipeline.yaml │ │ │ ├── test_no_user_defined_components │ │ │ └── pipeline.yaml │ │ │ ├── test_kafka_connect_sink_weave_from_topics │ │ │ └── pipeline.yaml │ │ │ ├── test_with_custom_config_with_absolute_defaults_path │ │ │ └── pipeline.yaml │ │ │ ├── test_with_custom_config_with_relative_defaults_path │ │ │ └── pipeline.yaml │ │ │ ├── test_with_env_defaults │ │ │ └── pipeline.yaml │ │ │ └── test_default_config │ │ │ └── pipeline.yaml │ ├── defaults.yaml │ ├── test_components_without_schema_handler │ │ ├── __init__.py │ │ └── components.py │ ├── test_components │ │ └── __init__.py │ ├── test_example.py │ ├── test_deploy.py │ └── test_destroy.py ├── component_handlers │ ├── __init__.py │ ├── kubernetes │ │ ├── __init__.py │ │ └── model.py │ ├── helm_wrapper │ │ ├── __init__.py │ │ └── test_utils.py │ ├── schema_handler │ │ ├── __init__.py │ │ └── resources │ │ │ ├── __init__.py │ │ │ └── module.py │ ├── kafka_connect │ │ └── __init__.py │ ├── topic │ │ └── __init__.py │ └── resources │ │ ├── defaults_development.yaml │ │ ├── defaults.yaml │ │ └── kafka_rest_proxy_responses │ │ ├── get_topic_response.json │ │ ├── get_default_topic_response.json │ │ ├── cluster-info.json │ │ ├── broker_response.json │ │ └── topic_config_response.json └── components │ ├── streams_bootstrap │ └── __init__.py │ ├── streams_bootstrap_v2 │ └── __init__.py │ ├── resources │ ├── pipelines │ │ └── test-distributed-defaults │ │ │ ├── defaults.yaml │ │ │ ├── defaults_development.yaml │ │ │ ├── defaults_production.yaml │ │ │ └── pipeline-deep │ │ │ ├── defaults_production.yaml │ │ │ ├── pipeline_development.yaml │ │ │ ├── pipeline.yaml │ │ │ └── defaults.yaml │ ├── defaults_development.yaml │ ├── defaults.yaml │ └── pipeline.yaml │ ├── __init__.py │ ├── conftest.py │ └── test_kubernetes_app.py ├── kpops ├── cli │ ├── __init__.py │ └── utils.py ├── core │ ├── __init__.py │ ├── operation.py │ └── exception.py ├── manifests │ ├── __init__.py │ ├── strimzi │ │ └── __init__.py │ └── argo.py ├── components │ ├── common │ │ ├── __init__.py │ │ └── app_type.py │ ├── base_components │ │ ├── models │ │ │ ├── resource.py │ │ │ ├── __init__.py │ │ │ ├── to_section.py │ │ │ └── from_section.py │ │ ├── __init__.py │ │ ├── kafka_app.py │ │ ├── cleaner.py │ │ └── kubernetes_app.py │ ├── streams_bootstrap │ │ ├── producer │ │ │ └── __init__.py │ │ ├── streams │ │ │ └── __init__.py │ │ └── __init__.py │ └── streams_bootstrap_v2 │ │ ├── producer │ │ ├── __init__.py │ │ └── model.py │ │ ├── streams │ │ └── __init__.py │ │ └── __init__.py ├── component_handlers │ ├── topic │ │ ├── __init__.py │ │ └── exception.py │ ├── utils │ │ ├── __init__.py │ │ └── exception.py │ ├── kubernetes │ │ ├── __init__.py │ │ ├── utils.py │ │ └── pvc_handler.py │ ├── helm_wrapper │ │ ├── __init__.py │ │ ├── exception.py │ │ ├── utils.py │ │ └── dry_run_handler.py │ ├── kafka_connect │ │ ├── __init__.py │ │ ├── exception.py │ │ └── timeout.py │ ├── schema_handler │ │ ├── __init__.py │ │ └── schema_provider.py │ └── __init__.py ├── const │ ├── __init__.py │ └── file_type.py ├── utils │ ├── dataclasses.py │ ├── types.py │ ├── json.py │ ├── colorify.py │ ├── __init__.py │ └── environment.py └── api │ ├── options.py │ └── logs.py ├── CODEOWNERS ├── CHANGELOG.md ├── CONTRIBUTING.md ├── docs ├── docs │ ├── index.md │ ├── resources │ │ ├── pipeline-components │ │ │ ├── sections │ │ │ │ ├── namespace.yaml │ │ │ │ ├── version.yaml │ │ │ │ ├── version-kafka-app.yaml │ │ │ │ ├── version-kafka-connector.yaml │ │ │ │ ├── from_-kafka-source-connector.yaml │ │ │ │ ├── values-helm-app.yaml │ │ │ │ ├── values-kubernetes-app.yaml │ │ │ │ ├── from_-producer-app.yaml │ │ │ │ ├── offset_topic-kafka-source-connector.yaml │ │ │ │ ├── resetter_values.yaml │ │ │ │ ├── config-kafka-connector.yaml │ │ │ │ ├── prefix.yaml │ │ │ │ ├── repo_config-kafka-connector.yaml │ │ │ │ ├── values-kafka-app.yaml │ │ │ │ ├── repo_config-helm-app.yaml │ │ │ │ ├── values-producer-app.yaml │ │ │ │ ├── to.yaml │ │ │ │ └── from_.yaml │ │ │ ├── headers │ │ │ │ ├── kafka-connector.yaml │ │ │ │ ├── kubernetes-app.yaml │ │ │ │ ├── kafka-sink-connector.yaml │ │ │ │ ├── helm-app.yaml │ │ │ │ ├── kafka-source-connector.yaml │ │ │ │ ├── kafka-app.yaml │ │ │ │ ├── streams-app.yaml │ │ │ │ └── producer-app.yaml │ │ │ ├── pipeline.md │ │ │ ├── dependencies │ │ │ │ ├── defaults_pipeline_component_dependencies.yaml │ │ │ │ └── pipeline_component_dependencies.yaml │ │ │ ├── kafka-source-connector.yaml │ │ │ ├── kafka-app.yaml │ │ │ └── kubernetes-app.yaml │ │ ├── pipeline-defaults │ │ │ ├── headers │ │ │ │ ├── defaults-kafka-source-connector.yaml │ │ │ │ ├── defaults-kubernetes-app.yaml │ │ │ │ ├── defaults-kafka-app.yaml │ │ │ │ ├── defaults-helm-app.yaml │ │ │ │ ├── defaults-kafka-connector.yaml │ │ │ │ ├── defaults-kafka-sink-connector.yaml │ │ │ │ ├── defaults-streams-app.yaml │ │ │ │ └── defaults-producer-app.yaml │ │ │ ├── defaults-kafka-sink-connector.yaml │ │ │ ├── defaults.md │ │ │ ├── defaults-kafka-source-connector.yaml │ │ │ ├── defaults-helm-app.yaml │ │ │ ├── defaults-producer-app.yaml │ │ │ ├── defaults-kafka-app.yaml │ │ │ └── defaults-kubernetes-app.yaml │ │ ├── editor_integration │ │ │ └── settings.json │ │ ├── examples │ │ │ ├── pipeline.md │ │ │ └── defaults.md │ │ ├── architecture │ │ │ └── components-hierarchy.md │ │ ├── variables │ │ │ ├── variable_substitution.yaml │ │ │ ├── cli_env_vars.env │ │ │ └── cli_env_vars.md │ │ └── pipeline-config │ │ │ └── config.yaml │ ├── images │ │ ├── atm-fraud-pipeline_streams-explorer.png │ │ └── word-count-pipeline_streams-explorer.png │ ├── developer │ │ ├── api.md │ │ ├── getting-started.md │ │ └── contributing.md │ └── user │ │ ├── core-concepts │ │ ├── components │ │ │ ├── overview.md │ │ │ ├── streams-bootstrap.md │ │ │ ├── kafka-connector.md │ │ │ ├── kubernetes-app.md │ │ │ ├── helm-app.md │ │ │ ├── kafka-sink-connector.md │ │ │ ├── producer-app.md │ │ │ ├── streams-app.md │ │ │ └── kafka-source-connector.md │ │ ├── config.md │ │ └── variables │ │ │ └── environment_variables.md │ │ ├── migration-guide │ │ ├── v9-v10.md │ │ ├── v4-v5.md │ │ ├── v5-v6.md │ │ ├── v3-v4.md │ │ └── v8-v9.md │ │ ├── references │ │ └── editor-integration.md │ │ ├── getting-started │ │ └── teardown.md │ │ └── what-is-kpops.md └── README.md ├── .gitmodules ├── .gitignore ├── hooks ├── __init__.py ├── gen_docs │ ├── __init__.py │ └── gen_docs_cli_usage.py └── gen_schema.py ├── config.yaml ├── .github ├── workflows │ ├── _lint-gh-ci.yaml │ ├── add-issue-ci.yaml │ ├── publish.yaml │ ├── update-gh-pages.yaml │ └── release.yaml ├── pyright-matcher.json └── actions │ └── update-docs │ └── action.yml ├── justfile ├── .bumpversion.cfg ├── dprint.jsonc ├── LICENSE └── lefthook.yaml /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/cli/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/core/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/api/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/cli/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /CODEOWNERS: -------------------------------------------------------------------------------- 1 | * @disrupted 2 | -------------------------------------------------------------------------------- /kpops/manifests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/compiler/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/kubernetes/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/manifests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/pipeline/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/cli/resources/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/utils/resources/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/components/common/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/manifests/strimzi/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/component_handlers/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/manifests/strimzi/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | docs/docs/user/changelog.md -------------------------------------------------------------------------------- /kpops/component_handlers/topic/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/component_handlers/utils/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/component_handlers/kubernetes/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/component_handlers/kubernetes/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/components/streams_bootstrap/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | docs/docs/developer/contributing.md -------------------------------------------------------------------------------- /kpops/component_handlers/helm_wrapper/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/component_handlers/kafka_connect/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/component_handlers/schema_handler/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/components/base_components/models/resource.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/component_handlers/helm_wrapper/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/component_handlers/schema_handler/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/components/streams_bootstrap_v2/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/components/streams_bootstrap/producer/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/components/streams_bootstrap/streams/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/components/streams_bootstrap_v2/producer/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /kpops/components/streams_bootstrap_v2/streams/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/pipeline/resources/symlinked-folder: -------------------------------------------------------------------------------- 1 | first-pipeline -------------------------------------------------------------------------------- /tests/component_handlers/schema_handler/resources/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/compiler/resources/test.yaml: -------------------------------------------------------------------------------- 1 | test: 2 | correct: file 3 | -------------------------------------------------------------------------------- /tests/cli/resources/config.yaml: -------------------------------------------------------------------------------- 1 | kafka_brokers: http://127.0.0.1:9092 2 | -------------------------------------------------------------------------------- /tests/compiler/resources/defaults_development.yaml: -------------------------------------------------------------------------------- 1 | producer-app: {} 2 | -------------------------------------------------------------------------------- /tests/compiler/resources/erroneous-file.yaml: -------------------------------------------------------------------------------- 1 | test: [] 2 | wrong: 3 | -------------------------------------------------------------------------------- /docs/docs/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | template: home.html 3 | title: Home 4 | --- 5 | -------------------------------------------------------------------------------- /tests/cli/snapshots/test_init/test_init_project_exclude_optional/defaults.yaml: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/cli/snapshots/test_init/test_init_project_exclude_optional/pipeline.yaml: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/cli/snapshots/test_init/test_init_project_include_optional/defaults.yaml: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/cli/snapshots/test_init/test_init_project_include_optional/pipeline.yaml: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_manifest/test_manifest_clean_argo_mode/manifest.yaml: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/compiler/resources/another-test.yaml: -------------------------------------------------------------------------------- 1 | test: 2 | correct: another-test-file 3 | -------------------------------------------------------------------------------- /tests/pipeline/defaults.yaml: -------------------------------------------------------------------------------- 1 | kubernetes-app: 2 | namespace: "global-namespace" 3 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-symlinked/pipeline.yaml: -------------------------------------------------------------------------------- 1 | ../first-pipeline/pipeline.yaml -------------------------------------------------------------------------------- /tests/pipeline/resources/resetter_values/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: simple-inflate-connectors 2 | -------------------------------------------------------------------------------- /kpops/const/__init__.py: -------------------------------------------------------------------------------- 1 | __version__ = "10.6.1" 2 | KPOPS = "KPOps" 3 | KPOPS_MODULE = "kpops." 4 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-folders-with-symlinks/pipeline-2: -------------------------------------------------------------------------------- 1 | ../pipeline-folders/pipeline-2 -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/namespace.yaml: -------------------------------------------------------------------------------- 1 | namespace: namespace # required 2 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/version.yaml: -------------------------------------------------------------------------------- 1 | version: "1.0.0" # Helm chart version 2 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- 1 | [submodule "examples"] 2 | path = examples 3 | url = https://github.com/bakdata/kpops-examples 4 | -------------------------------------------------------------------------------- /tests/pipeline/resources/kafka-connect-sink-config/defaults.yaml: -------------------------------------------------------------------------------- 1 | streams-bootstrap: 2 | version: "3.6.1" 3 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/version-kafka-app.yaml: -------------------------------------------------------------------------------- 1 | version: "2.12.0" # Helm chart version 2 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea 2 | **/__pycache__ 3 | .DS_Store 4 | .envrc 5 | .vscode 6 | site/ 7 | scratch* 8 | lefthook-local.yaml 9 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/version-kafka-connector.yaml: -------------------------------------------------------------------------------- 1 | version: "1.0.6" # Helm chart version 2 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-folders-with-symlinks/pipeline-1/pipeline.yaml: -------------------------------------------------------------------------------- 1 | ../../pipeline-folders/pipeline-1/pipeline.yaml -------------------------------------------------------------------------------- /hooks/__init__.py: -------------------------------------------------------------------------------- 1 | """KPOps Git hooks.""" 2 | 3 | from pathlib import Path 4 | 5 | ROOT = Path(__file__).parents[1].resolve() 6 | -------------------------------------------------------------------------------- /tests/pipeline/resources/dotenv/.env: -------------------------------------------------------------------------------- 1 | KPOPS_schema_registry__enabled="true" 2 | KPOPS_schema_registry__url="http://localhost:8081" 3 | -------------------------------------------------------------------------------- /tests/pipeline/resources/multi-config/config_production.yaml: -------------------------------------------------------------------------------- 1 | schema_registry: 2 | enabled: true 3 | url: "http://production:8081" 4 | -------------------------------------------------------------------------------- /tests/component_handlers/kafka_connect/__init__.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | 3 | RESOURCES_PATH = Path(__file__).parent / "resources" 4 | -------------------------------------------------------------------------------- /tests/component_handlers/topic/__init__.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | 3 | RESOURCES_PATH = Path(__file__).parent.parent / "resources" 4 | -------------------------------------------------------------------------------- /tests/pipeline/resources/dotenv/custom.env: -------------------------------------------------------------------------------- 1 | KPOPS_schema_registry__enabled="false" 2 | KPOPS_schema_registry__url="http://notlocalhost:8081" 3 | -------------------------------------------------------------------------------- /tests/pipeline/resources/multi-config/config_development.yaml: -------------------------------------------------------------------------------- 1 | schema_registry: 2 | enabled: true 3 | url: "http://development:8081" 4 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/headers/kafka-connector.yaml: -------------------------------------------------------------------------------- 1 | - type: kafka-connector # required 2 | name: kafka-connector # required 3 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/from_-kafka-source-connector.yaml: -------------------------------------------------------------------------------- 1 | # The source connector has no `from` section 2 | # from: 3 | -------------------------------------------------------------------------------- /tests/pipeline/resources/simple-pipeline/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | 3 | - type: streams-app 4 | 5 | - type: helm-app 6 | values: {} 7 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/headers/kubernetes-app.yaml: -------------------------------------------------------------------------------- 1 | # Base Kubernetes App 2 | - type: kubernetes-app 3 | name: kubernetes-app # required 4 | -------------------------------------------------------------------------------- /tests/components/resources/pipelines/test-distributed-defaults/defaults.yaml: -------------------------------------------------------------------------------- 1 | helm-app: 2 | name: ${component.type} 3 | namespace: example-namespace 4 | -------------------------------------------------------------------------------- /docs/docs/images/atm-fraud-pipeline_streams-explorer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakdata/kpops/HEAD/docs/docs/images/atm-fraud-pipeline_streams-explorer.png -------------------------------------------------------------------------------- /docs/docs/images/word-count-pipeline_streams-explorer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakdata/kpops/HEAD/docs/docs/images/word-count-pipeline_streams-explorer.png -------------------------------------------------------------------------------- /tests/components/__init__.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | 3 | PIPELINE_BASE_DIR = Path(__file__).parent 4 | RESOURCES_PATH = PIPELINE_BASE_DIR / "resources" 5 | -------------------------------------------------------------------------------- /kpops/component_handlers/helm_wrapper/exception.py: -------------------------------------------------------------------------------- 1 | class ReleaseNotFoundException(Exception): 2 | pass 3 | 4 | 5 | class ParseError(Exception): 6 | pass 7 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipelines-with-graphs/simple-pipeline/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: app1 3 | 4 | - type: streams-app 5 | name: app2 6 | -------------------------------------------------------------------------------- /tests/cli/snapshots/test_init/test_init_project_exclude_optional/config.yaml: -------------------------------------------------------------------------------- 1 | # Global configuration for KPOps project. 2 | 3 | # Required fields 4 | kafka_brokers: null 5 | -------------------------------------------------------------------------------- /tests/pipeline/resources/no-topics-defaults/defaults_development.yaml: -------------------------------------------------------------------------------- 1 | kubernetes-app: 2 | name: "${component.type}-development" 3 | namespace: development-namespace 4 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/headers/kafka-sink-connector.yaml: -------------------------------------------------------------------------------- 1 | # Kafka sink connector 2 | - type: kafka-sink-connector 3 | name: kafka-sink-connector # required 4 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/headers/defaults-kafka-source-connector.yaml: -------------------------------------------------------------------------------- 1 | # Kafka source connector 2 | # 3 | # Child of: KafkaConnector 4 | kafka-source-connector: 5 | -------------------------------------------------------------------------------- /tests/components/resources/pipelines/test-distributed-defaults/defaults_development.yaml: -------------------------------------------------------------------------------- 1 | helm-app: 2 | name: dev-${component.type} 3 | namespace: dev-example-namespace 4 | -------------------------------------------------------------------------------- /tests/components/resources/pipelines/test-distributed-defaults/defaults_production.yaml: -------------------------------------------------------------------------------- 1 | helm-app: 2 | name: prod-${component.type} 3 | namespace: prod-example-namespace 4 | -------------------------------------------------------------------------------- /tests/components/resources/pipelines/test-distributed-defaults/pipeline-deep/defaults_production.yaml: -------------------------------------------------------------------------------- 1 | helm-app: 2 | name: ${component.type} 3 | namespace: prod-namespace 4 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-env-defaults/defaults_development.yaml: -------------------------------------------------------------------------------- 1 | kubernetes-app: 2 | name: ${component.type}-development 3 | namespace: development-namespace 4 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-duplicate-step-names/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: pipeline-component 2 | name: component 3 | 4 | - type: pipeline-component 5 | name: component 6 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/headers/helm-app.yaml: -------------------------------------------------------------------------------- 1 | # Kubernetes app managed through Helm with an associated Helm chart 2 | - type: helm-app 3 | name: helm-app # required 4 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/headers/defaults-kubernetes-app.yaml: -------------------------------------------------------------------------------- 1 | # Base Kubernetes App 2 | # 3 | # Parent of: HelmApp 4 | # Child of: PipelineComponent 5 | kubernetes-app: 6 | -------------------------------------------------------------------------------- /kpops/utils/dataclasses.py: -------------------------------------------------------------------------------- 1 | from dataclasses import is_dataclass 2 | 3 | 4 | def is_dataclass_instance(obj: object) -> bool: 5 | return is_dataclass(obj) and not isinstance(obj, type) 6 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/headers/kafka-source-connector.yaml: -------------------------------------------------------------------------------- 1 | # Kafka source connector 2 | - type: kafka-source-connector # required 3 | name: kafka-source-connector # required 4 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/values-helm-app.yaml: -------------------------------------------------------------------------------- 1 | values: # required 2 | image: exampleImage # Example 3 | debug: false # Example 4 | commandLine: {} # Example 5 | -------------------------------------------------------------------------------- /config.yaml: -------------------------------------------------------------------------------- 1 | kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092" 2 | pipeline_base_dir: tests/pipeline 3 | strimzi_topic: 4 | label: 5 | strimzi.io/cluster: my-cluster 6 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/values-kubernetes-app.yaml: -------------------------------------------------------------------------------- 1 | values: # required 2 | image: exampleImage # Example 3 | debug: false # Example 4 | commandLine: {} # Example 5 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-folders/pipeline-1/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | values: 3 | commandLine: 4 | FAKE_ARG: "fake-arg-value" 5 | schedule: "30 3/8 * * *" 6 | -------------------------------------------------------------------------------- /tests/components/resources/defaults_development.yaml: -------------------------------------------------------------------------------- 1 | parent: 2 | name: fake-name 3 | value: 2.0 4 | child: 5 | nice: 6 | fake-value: fake 7 | grand-child: 8 | grand_child: grand-child-value 9 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/from_-producer-app.yaml: -------------------------------------------------------------------------------- 1 | # from: # While the producer-app does inherit from kafka-app, it does not need a 2 | # `from` section, hence it does not support it. 3 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/defaults-kafka-sink-connector.yaml: -------------------------------------------------------------------------------- 1 | # Kafka sink connector 2 | # 3 | # Child of: KafkaConnector 4 | kafka-sink-connector: 5 | # No settings differ from `kafka-connector` 6 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/headers/defaults-kafka-app.yaml: -------------------------------------------------------------------------------- 1 | # Base component for Kafka-based components. 2 | # 3 | # Parent of: ProducerApp, StreamsApp 4 | # Child of: KubernetesApp 5 | kafka-app: 6 | -------------------------------------------------------------------------------- /tests/component_handlers/resources/defaults_development.yaml: -------------------------------------------------------------------------------- 1 | parent: 2 | name: fake-name 3 | value: 2.0 4 | child: 5 | nice: 6 | fake-value: fake 7 | grand-child: 8 | grand_child: grand-child-value 9 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/offset_topic-kafka-source-connector.yaml: -------------------------------------------------------------------------------- 1 | # offset.storage.topic 2 | # https://kafka.apache.org/documentation/#connect_running 3 | offset_topic: offset_topic 4 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/resetter_values.yaml: -------------------------------------------------------------------------------- 1 | # Overriding Kafka Connect Resetter Helm values. E.g. to override the 2 | # Image Tag etc. 3 | resetter_values: 4 | imageTag: "1.2.3" 5 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/headers/defaults-helm-app.yaml: -------------------------------------------------------------------------------- 1 | # Kubernetes app managed through Helm with an associated Helm chart 2 | # 3 | # Parent of: KafkaApp 4 | # Child of: KubernetesApp 5 | helm-app: 6 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/headers/defaults-kafka-connector.yaml: -------------------------------------------------------------------------------- 1 | # Kafka connector 2 | # 3 | # Parent of: KafkaSinkConnector, KafkaSourceConnector 4 | # Child of: PipelineComponent 5 | kafka-connector: 6 | -------------------------------------------------------------------------------- /tests/pipeline/test_components_without_schema_handler/__init__.py: -------------------------------------------------------------------------------- 1 | from tests.pipeline.test_components.components import ( 2 | Converter, 3 | Filter, 4 | ScheduledProducer, 5 | ShouldInflate, 6 | ) 7 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/config-kafka-connector.yaml: -------------------------------------------------------------------------------- 1 | # Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs 2 | config: # required 3 | tasks.max: 1 4 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/headers/defaults-kafka-sink-connector.yaml: -------------------------------------------------------------------------------- 1 | # Kafka sink connector 2 | # 3 | # Child of: KafkaConnector 4 | kafka-sink-connector: 5 | # No settings differ from `kafka-connector` 6 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/defaults.md: -------------------------------------------------------------------------------- 1 | # defaults.yaml 2 | 3 | [:material-download:](./defaults.yaml) 4 | 5 | ```yaml 6 | --8<-- 7 | ./docs/resources/pipeline-defaults/defaults.yaml 8 | --8<-- 9 | ``` 10 | -------------------------------------------------------------------------------- /kpops/components/base_components/models/__init__.py: -------------------------------------------------------------------------------- 1 | from typing import NewType 2 | 3 | TopicName = NewType("TopicName", str) 4 | 5 | ModelName = NewType("ModelName", str) 6 | ModelVersion = NewType("ModelVersion", str) 7 | -------------------------------------------------------------------------------- /kpops/core/operation.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | import enum 4 | 5 | 6 | class OperationMode(str, enum.Enum): 7 | ARGO = "argo" 8 | MANIFEST = "manifest" 9 | MANAGED = "managed" 10 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-envs/pipeline_development.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | name: input-producer 3 | values: 4 | commandLine: 5 | FAKE_ARG: override-arg 6 | schedule: 20 3/8 * * * 7 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/pipeline.md: -------------------------------------------------------------------------------- 1 | # pipeline.yaml 2 | 3 | [:material-download:](./pipeline.yaml) 4 | 5 | ```yaml 6 | --8<-- 7 | ./docs/resources/pipeline-components/pipeline.yaml 8 | --8<-- 9 | ``` 10 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/prefix.yaml: -------------------------------------------------------------------------------- 1 | # Pipeline prefix that will prefix every component name. If you wish to not 2 | # have any prefix you can specify an empty string. 3 | prefix: ${pipeline.name}- 4 | -------------------------------------------------------------------------------- /tests/pipeline/resources/resetter_values/pipeline_connector_only.yaml: -------------------------------------------------------------------------------- 1 | - type: kafka-sink-connector 2 | name: es-sink-connector 3 | config: 4 | connector.class: io.confluent.connect.elasticsearch.ElasticsearchSinkConnector 5 | -------------------------------------------------------------------------------- /tests/pipeline/resources/temp-trim-release-name/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: streams-app 2 | name: in-order-to-have-len-fifty-two-name-should-end--here 3 | values: 4 | kafka: 5 | config: 6 | max.poll.records: 100 7 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/headers/kafka-app.yaml: -------------------------------------------------------------------------------- 1 | # Base component for Kafka-based components. 2 | # Producer or streaming apps should inherit from this class. 3 | - type: kafka-app # required 4 | name: kafka-app # required 5 | -------------------------------------------------------------------------------- /tests/component_handlers/resources/defaults.yaml: -------------------------------------------------------------------------------- 1 | parent: 2 | name: fake-name 3 | value: 1.0 4 | child: 5 | name: fake-child-name 6 | nice: 7 | fake-value: must-be-overwritten 8 | env-var-test: 9 | name: $pipeline_name 10 | -------------------------------------------------------------------------------- /tests/components/resources/pipelines/test-distributed-defaults/pipeline-deep/pipeline_development.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | name: input-producer 3 | app: 4 | commandLine: 5 | FAKE_ARG: override-arg 6 | schedule: 20 3/8 * * * 7 | -------------------------------------------------------------------------------- /tests/pipeline/resources/kafka-connect-sink-config/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: kafka-sink-connector 2 | namespace: test-namespace 3 | name: test-connector 4 | state: paused 5 | to: 6 | topics: 7 | ${error_topic_name}: 8 | type: error 9 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipelines-with-graphs/same-topic-and-component-name/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: streams-app 2 | name: app2-processor 3 | values: 4 | image: some-image 5 | to: 6 | topics: 7 | app2-processor: 8 | type: output 9 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-folders/pipeline-2/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: converter 2 | values: 3 | commandLine: 4 | CONVERT_XML: true 5 | resources: 6 | limits: 7 | memory: 2G 8 | requests: 9 | memory: 2G 10 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/headers/defaults-streams-app.yaml: -------------------------------------------------------------------------------- 1 | # StreamsApp component that configures a streams bootstrap app. 2 | # 3 | # Child of: KafkaApp 4 | # More documentation on StreamsApp: https://github.com/bakdata/streams-bootstrap 5 | streams-app: 6 | -------------------------------------------------------------------------------- /tests/components/resources/defaults.yaml: -------------------------------------------------------------------------------- 1 | parent: 2 | name: fake-name 3 | value: 1.0 4 | child: 5 | name: fake-child-name 6 | nice: 7 | fake-value: must-be-overwritten 8 | nested: 9 | foo: foo 10 | env-var-test: 11 | name: ${pipeline_name} 12 | -------------------------------------------------------------------------------- /tests/pipeline/resources/dotenv/config.yaml: -------------------------------------------------------------------------------- 1 | kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092" 2 | kafka_connect: 3 | url: "http://localhost:8083" 4 | kafka_rest: 5 | url: "http://localhost:8082" 6 | helm_config: 7 | api_version: "2.1.1" 8 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/headers/streams-app.yaml: -------------------------------------------------------------------------------- 1 | # StreamsApp component that configures a streams bootstrap app. 2 | # More documentation on StreamsApp: https://github.com/bakdata/streams-bootstrap 3 | - type: streams-app # required 4 | name: streams-app # required 5 | -------------------------------------------------------------------------------- /kpops/components/common/app_type.py: -------------------------------------------------------------------------------- 1 | from enum import Enum 2 | 3 | 4 | class AppType(Enum): 5 | STREAMS_APP = "streams-app" 6 | PRODUCER_APP = "producer-app" 7 | CLEANUP_STREAMS_APP = "streams-app-cleanup-job" 8 | CLEANUP_PRODUCER_APP = "producer-app-cleanup-job" 9 | -------------------------------------------------------------------------------- /kpops/components/streams_bootstrap/__init__.py: -------------------------------------------------------------------------------- 1 | from kpops.components.streams_bootstrap.base import StreamsBootstrap 2 | 3 | from .producer.producer_app import ProducerApp 4 | from .streams.streams_app import StreamsApp 5 | 6 | __all__ = ("ProducerApp", "StreamsApp", "StreamsBootstrap") 7 | -------------------------------------------------------------------------------- /kpops/core/exception.py: -------------------------------------------------------------------------------- 1 | class ValidationError(Exception): 2 | pass 3 | 4 | 5 | class ParsingException(Exception): 6 | pass 7 | 8 | 9 | class ClassNotFoundError(Exception): 10 | """Similar to builtin `ModuleNotFoundError`; class doesn't exist inside module.""" 11 | -------------------------------------------------------------------------------- /tests/pipeline/resources/no-user-defined-components/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: streams-app 2 | from: 3 | topics: 4 | example-topic: 5 | type: input 6 | values: 7 | image: fake-image 8 | to: 9 | topics: 10 | example-output: 11 | type: output 12 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/headers/defaults-producer-app.yaml: -------------------------------------------------------------------------------- 1 | # Holds configuration to use as values for the streams bootstrap producer-app Helm 2 | # chart. 3 | # 4 | # Child of: KafkaApp 5 | # More documentation on ProducerApp: https://github.com/bakdata/streams-bootstrap 6 | producer-app: 7 | -------------------------------------------------------------------------------- /kpops/utils/types.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | from collections.abc import Mapping, Sequence 4 | from typing import TypeAlias 5 | 6 | # JSON values 7 | JsonType: TypeAlias = ( 8 | Mapping[str, "JsonType"] | Sequence["JsonType"] | str | int | float | bool | None 9 | ) 10 | -------------------------------------------------------------------------------- /tests/pipeline/resources/custom-config/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: app1 3 | values: 4 | resources: 5 | limits: 6 | memory: 2G 7 | requests: 8 | memory: 2G 9 | 10 | - type: streams-app 11 | name: app2 12 | values: 13 | image: some-image 14 | -------------------------------------------------------------------------------- /tests/pipeline/test_components/__init__.py: -------------------------------------------------------------------------------- 1 | from tests.pipeline.test_components.components import ( 2 | Converter, 3 | Filter, 4 | MyProducerApp, 5 | MyStreamsApp, 6 | ScheduledProducer, 7 | ShouldInflate, 8 | SimpleInflateConnectors, 9 | TestSchemaProvider, 10 | ) 11 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/headers/producer-app.yaml: -------------------------------------------------------------------------------- 1 | # Holds configuration to use as values for the streams bootstrap producer-app Helm 2 | # chart. 3 | # More documentation on ProducerApp: 4 | # https://github.com/bakdata/streams-bootstrap 5 | - type: producer-app 6 | name: producer-app # required 7 | -------------------------------------------------------------------------------- /tests/pipeline/resources/component-type-substitution/infinite_pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: converter 2 | values: 3 | labels: 4 | l_1: ${component.values.labels.l_2} 5 | l_2: ${component.values.labels.l_3} 6 | l_3: ${component.values.labels.l_1} 7 | infinite_nesting: ${component.values.labels} 8 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipelines-with-graphs/simple-pipeline/config.yaml: -------------------------------------------------------------------------------- 1 | schema_registry: 2 | enabled: false 3 | url: "http://localhost:8081" 4 | 5 | kafka_connect: 6 | url: "http://kafka_connect_url:8083" 7 | kafka_rest: 8 | url: "http://kafka_rest_url:8082" 9 | 10 | kafka_brokers: "broker:9092" 11 | -------------------------------------------------------------------------------- /docs/docs/developer/api.md: -------------------------------------------------------------------------------- 1 | # Python API 2 | 3 | 4 | 5 | ::: kpops.api 6 | options: 7 | filters: 8 | - "!^_" 9 | 10 | ::: kpops.pipeline.Pipeline 11 | options: 12 | filters: 13 | - "!^_" 14 | - "!^model_config$" 15 | 16 | 17 | -------------------------------------------------------------------------------- /kpops/component_handlers/topic/exception.py: -------------------------------------------------------------------------------- 1 | from kpops.component_handlers.utils.exception import HttpxException 2 | 3 | 4 | class TopicNotFoundException(Exception): 5 | pass 6 | 7 | 8 | class TopicTransactionError(Exception): 9 | pass 10 | 11 | 12 | class KafkaRestProxyError(HttpxException): 13 | pass 14 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipelines-with-graphs/same-topic-and-component-name/config.yaml: -------------------------------------------------------------------------------- 1 | schema_registry: 2 | enabled: false 3 | url: "http://localhost:8081" 4 | 5 | kafka_connect: 6 | url: "http://kafka_connect_url:8083" 7 | kafka_rest: 8 | url: "http://kafka_rest_url:8082" 9 | 10 | kafka_brokers: "broker:9092" 11 | -------------------------------------------------------------------------------- /kpops/component_handlers/kafka_connect/exception.py: -------------------------------------------------------------------------------- 1 | from kpops.component_handlers.utils.exception import HttpxException 2 | 3 | 4 | class ConnectorNotFoundException(Exception): 5 | pass 6 | 7 | 8 | class ConnectorStateException(Exception): 9 | pass 10 | 11 | 12 | class KafkaConnectError(HttpxException): 13 | pass 14 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/overview.md: -------------------------------------------------------------------------------- 1 | # Overview 2 | 3 | This section explains the different components of KPOps, their usage and configuration in the pipeline 4 | definition [`pipeline.yaml`](../../../resources/pipeline-components/pipeline.md). 5 | 6 | --8<-- 7 | ./docs/resources/architecture/components-hierarchy.md 8 | --8<-- 9 | -------------------------------------------------------------------------------- /kpops/components/streams_bootstrap_v2/__init__.py: -------------------------------------------------------------------------------- 1 | from kpops.components.streams_bootstrap_v2.base import StreamsBootstrapV2 2 | 3 | from .producer.producer_app import ProducerAppV2 4 | from .streams.streams_app import StreamsAppV2 5 | 6 | __all__ = ( 7 | "ProducerAppV2", 8 | "StreamsAppV2", 9 | "StreamsBootstrapV2", 10 | ) 11 | -------------------------------------------------------------------------------- /tests/pipeline/resources/name_prefix_concatenation/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: streams-app 2 | name: my-streams-app 3 | prefix: my-fake-prefix- 4 | from: 5 | topics: 6 | example-topic: 7 | type: input 8 | values: 9 | image: fake-image 10 | to: 11 | topics: 12 | example-output: 13 | type: output 14 | -------------------------------------------------------------------------------- /tests/pipeline/resources/kafka-connect-sink-config/config.yaml: -------------------------------------------------------------------------------- 1 | kafka_brokers: "broker:9092" 2 | topic_name_config: 3 | default_error_topic_name: ${component.type}-error-topic 4 | default_output_topic_name: ${component.type}-output-topic 5 | kafka_connect: 6 | url: "http://kafka_connect_url:8083" 7 | kafka_rest: 8 | url: "http://kafka_rest_url:8082" 9 | -------------------------------------------------------------------------------- /tests/pipeline/resources/multi-config/config.yaml: -------------------------------------------------------------------------------- 1 | kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092" 2 | kafka_connect: 3 | url: "http://localhost:8083" 4 | kafka_rest: 5 | url: "http://localhost:8082" 6 | schema_registry: 7 | enabled: true 8 | url: "http://localhost:8081" 9 | helm_config: 10 | api_version: "2.1.1" 11 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/defaults-kafka-source-connector.yaml: -------------------------------------------------------------------------------- 1 | # Kafka source connector 2 | # 3 | # Child of: KafkaConnector 4 | kafka-source-connector: 5 | # The source connector has no `from` section 6 | # from: 7 | # offset.storage.topic 8 | # https://kafka.apache.org/documentation/#connect_running 9 | offset_topic: offset_topic 10 | -------------------------------------------------------------------------------- /kpops/utils/json.py: -------------------------------------------------------------------------------- 1 | import json 2 | from typing import Any 3 | 4 | 5 | def is_jsonable(input: Any) -> bool: 6 | """Check whether a value is json-serializable. 7 | 8 | :param input: Value to be checked. 9 | """ 10 | try: 11 | json.dumps(input) 12 | except (TypeError, OverflowError): 13 | return False 14 | else: 15 | return True 16 | -------------------------------------------------------------------------------- /tests/pipeline/resources/env-specific-config-only/config_production.yaml: -------------------------------------------------------------------------------- 1 | kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092" 2 | kafka_connect: 3 | url: "http://localhost:8083" 4 | kafka_rest: 5 | url: "http://localhost:8082" 6 | schema_registry: 7 | enabled: true 8 | url: "http://localhost:8081" 9 | helm_config: 10 | api_version: "2.1.1" 11 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-component-should-have-prefix/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: account-producer 3 | values: 4 | replicaCount: 1 5 | image: ${DOCKER_REGISTRY}/atm-demo-accountproducer 6 | imageTag: 1.0.0 7 | schedule: 0 12 * * * 8 | suspend: true 9 | prometheus: 10 | jmx: 11 | enabled: false 12 | debug: true 13 | -------------------------------------------------------------------------------- /tests/components/resources/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | name: input-producer 3 | app: 4 | commandLine: 5 | FAKE_ARG: fake-arg-value 6 | schedule: 30 3/8 * * * 7 | 8 | - type: converter 9 | app: 10 | commandLine: 11 | CONVERT_XML: true 12 | resources: 13 | limits: 14 | memory: 2G 15 | requests: 16 | memory: 2G 17 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-component-should-have-prefix/defaults.yaml: -------------------------------------------------------------------------------- 1 | pipeline-component: 2 | prefix: "from-pipeline-component-" 3 | 4 | kubernetes-app: 5 | namespace: ${NAMESPACE} 6 | 7 | streams-bootstrap: 8 | version: "3.6.1" 9 | values: 10 | kafka: 11 | bootstrapServers: ${config.kafka_brokers} 12 | schemaRegistryUrl: ${config.schema_registry.url} 13 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-paths/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: account-producer 3 | namespace: test 4 | values: 5 | kafka: 6 | bootstrapServers: test 7 | output_topic: out 8 | repo_config: 9 | repository_name: masked 10 | url: masked 11 | repo_auth_flags: 12 | username: masked 13 | password: $CI_JOB_TOKEN 14 | ca_file: my-cert.cert 15 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/repo_config-kafka-connector.yaml: -------------------------------------------------------------------------------- 1 | # Helm repository configuration for resetter 2 | repo_config: 3 | repository_name: my-repo # required 4 | url: https://bakdata.github.io/kafka-connect-resetter/ # required 5 | repo_auth_flags: 6 | username: user 7 | password: pass 8 | ca_file: /home/user/path/to/ca-file 9 | insecure_skip_tls_verify: false 10 | -------------------------------------------------------------------------------- /tests/pipeline/resources/resetter_values/defaults.yaml: -------------------------------------------------------------------------------- 1 | helm-app: 2 | name: "${component.type}" 3 | namespace: "namespace" 4 | values: 5 | label: ${component.name} 6 | kafka: 7 | bootstrapServers: ${config.kafka_brokers} 8 | 9 | kafka-sink-connector: 10 | config: 11 | connector.class: "io.confluent.connect.jdbc.JdbcSinkConnector" 12 | resetter_values: 13 | imageTag: override-default-image-tag 14 | -------------------------------------------------------------------------------- /tests/components/resources/pipelines/test-distributed-defaults/pipeline-deep/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | name: input-producer 3 | app: 4 | commandLine: 5 | FAKE_ARG: fake-arg-value 6 | schedule: 30 3/8 * * * 7 | 8 | - type: converter 9 | app: 10 | commandLine: 11 | CONVERT_XML: true 12 | resources: 13 | limits: 14 | memory: 2G 15 | requests: 16 | memory: 2G 17 | -------------------------------------------------------------------------------- /tests/pipeline/resources/parallel-pipeline/config.yaml: -------------------------------------------------------------------------------- 1 | topic_name_config: 2 | default_error_topic_name: ${component.name}-dead-letter-topic 3 | default_output_topic_name: ${component.name}-test-topic 4 | 5 | schema_registry: 6 | enabled: true 7 | url: "http://localhost:8081" 8 | 9 | kafka_connect: 10 | url: "http://kafka_connect_url:8083" 11 | kafka_rest: 12 | url: "http://kafka_rest_url:8082" 13 | 14 | kafka_brokers: "broker:9092" 15 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/values-kafka-app.yaml: -------------------------------------------------------------------------------- 1 | values: # required 2 | streams: # required 3 | brokers: ${config.kafka_brokers} # required 4 | schemaRegistryUrl: ${config.schema_registry.url} 5 | nameOverride: override-with-this-name # kafka-app-specific 6 | fullnameOverride: override-with-this-name # kafka-app-specific 7 | imageTag: "1.0.0" # Example values that are shared between streams-app and producer-app 8 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-loop/defaults.yaml: -------------------------------------------------------------------------------- 1 | pipeline-component: 2 | prefix: "" 3 | 4 | kubernetes-app: 5 | namespace: example-namespace 6 | 7 | kafka-connector: 8 | namespace: example-namespace 9 | 10 | streams-bootstrap: 11 | values: 12 | kafka: 13 | bootstrapServers: 127.0.0.1:9092 14 | schemaRegistryUrl: 127.0.0.1:8081 15 | 16 | streams-app: 17 | values: 18 | labels: 19 | pipeline: ${pipeline.name} 20 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-folders/pipeline-3/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: filter 2 | name: "a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name" 3 | values: 4 | commandLine: 5 | TYPE: "nothing" 6 | resources: 7 | requests: 8 | memory: 3G 9 | replicaCount: 4 10 | autoscaling: 11 | minReplicas: 4 12 | maxReplicas: 4 13 | -------------------------------------------------------------------------------- /tests/utils/resources/nested_base_settings.py: -------------------------------------------------------------------------------- 1 | from pydantic import Field 2 | from pydantic_settings import BaseSettings 3 | 4 | 5 | class NestedSettings(BaseSettings): 6 | attr: str = Field("attr") 7 | 8 | 9 | class ParentSettings(BaseSettings): 10 | not_nested_field: str = Field("not_nested_field") 11 | nested_field: NestedSettings = Field(...) 12 | field_with_env_defined: str = Field( 13 | alias="FIELD_WITH_ENV_DEFINED", 14 | ) 15 | -------------------------------------------------------------------------------- /.github/workflows/_lint-gh-ci.yaml: -------------------------------------------------------------------------------- 1 | name: Lint GitHub CI 2 | 3 | on: 4 | pull_request: 5 | branches: 6 | - main 7 | paths: 8 | - .github/** 9 | - actions/** 10 | 11 | jobs: 12 | actionlint: 13 | name: actionlint 14 | runs-on: ubuntu-24.04 15 | steps: 16 | - name: Lint actions and workflows 17 | uses: bakdata/ci-templates/actions/action-lint@v1.14.0 18 | with: 19 | action-lint-version: "v1.7.6" 20 | -------------------------------------------------------------------------------- /kpops/utils/colorify.py: -------------------------------------------------------------------------------- 1 | import typer 2 | 3 | 4 | def yellowify(string: str) -> str: 5 | return typer.style(string, fg=typer.colors.YELLOW) 6 | 7 | 8 | def greenify(string: str) -> str: 9 | return typer.style(string, fg=typer.colors.GREEN) 10 | 11 | 12 | def magentaify(string: str) -> str: 13 | return typer.style(string, fg=typer.colors.MAGENTA) 14 | 15 | 16 | def redify(string: str) -> str: 17 | return typer.style(string, fg=typer.colors.RED) 18 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-folders-with-symlinks/pipeline-3/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: filter 2 | name: "a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name" 3 | values: 4 | commandLine: 5 | TYPE: "nothing" 6 | resources: 7 | requests: 8 | memory: 3G 9 | replicaCount: 4 10 | autoscaling: 11 | minReplicas: 4 12 | maxReplicas: 4 13 | -------------------------------------------------------------------------------- /.github/pyright-matcher.json: -------------------------------------------------------------------------------- 1 | { 2 | "problemMatcher": [ 3 | { 4 | "owner": "pyright", 5 | "pattern": [ 6 | { 7 | "regexp": "^\\s\\s(\\S*\\.py):(\\d+):(\\d+)\\s-\\s(error|warning|information):\\s([\\s\\S]+?)\\s\\(([\\s\\S]+?)(?=(?:\\)|\\d+ errors,))", 8 | "file": 1, 9 | "line": 2, 10 | "column": 3, 11 | "severity": 4, 12 | "message": 5, 13 | "code": 6 14 | } 15 | ] 16 | } 17 | ] 18 | } 19 | -------------------------------------------------------------------------------- /tests/pipeline/resources/kafka-connect-sink/pipeline.yaml: -------------------------------------------------------------------------------- 1 | # Parse Connector topics from previous component output topic 2 | - type: streams-app 3 | from: 4 | topics: 5 | example-topic: 6 | type: input 7 | values: 8 | image: fake-image 9 | to: 10 | topics: 11 | example-output: 12 | type: output 13 | 14 | - type: kafka-sink-connector 15 | name: es-sink-connector 16 | config: 17 | connector.class: io.confluent.connect.elasticsearch.ElasticsearchSinkConnector 18 | -------------------------------------------------------------------------------- /tests/cli/resources/custom_module/__init__.py: -------------------------------------------------------------------------------- 1 | from schema_registry.client.schema import AvroSchema 2 | 3 | from kpops.component_handlers.schema_handler.schema_provider import ( 4 | Schema, 5 | SchemaProvider, 6 | ) 7 | from kpops.components.base_components.models import ModelName, ModelVersion 8 | 9 | 10 | class CustomSchemaProvider(SchemaProvider): 11 | def provide_schema( 12 | self, schema_class: str, models: dict[ModelName, ModelVersion] 13 | ) -> Schema: 14 | return AvroSchema() 15 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-env-defaults/pipeline.yaml: -------------------------------------------------------------------------------- 1 | # Parse Connector topics from previous component output topic 2 | - type: streams-app 3 | from: 4 | topics: 5 | example-topic: 6 | type: input 7 | values: 8 | image: fake-image 9 | to: 10 | topics: 11 | example-output: 12 | type: output 13 | 14 | - type: kafka-sink-connector 15 | name: es-sink-connector 16 | config: 17 | connector.class: io.confluent.connect.elasticsearch.ElasticsearchSinkConnector 18 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/repo_config-helm-app.yaml: -------------------------------------------------------------------------------- 1 | # Helm repository configuration (optional) 2 | # If not set the helm repo add will not be called. Useful when using local Helm charts 3 | repo_config: 4 | repository_name: bakdata-streams-bootstrap # required 5 | url: https://bakdata.github.io/streams-bootstrap/ # required 6 | repo_auth_flags: 7 | username: user 8 | password: pass 9 | ca_file: /home/user/path/to/ca-file 10 | insecure_skip_tls_verify: false 11 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/streams-bootstrap.md: -------------------------------------------------------------------------------- 1 | # Streams Bootstrap 2 | 3 | Subclass of [_HelmApp_](helm-app.md). 4 | 5 | ### Usage 6 | 7 | - Defines a [streams-bootstrap](https://github.com/bakdata/streams-bootstrap){target=_blank} component 8 | 9 | - Often used in `defaults.yaml` 10 | 11 | ### Operations 12 | 13 | #### deploy 14 | 15 | Deploy using Helm. 16 | 17 | #### destroy 18 | 19 | Uninstall Helm release. 20 | 21 | #### reset 22 | 23 | Do nothing. 24 | 25 | #### clean 26 | 27 | Do nothing. 28 | -------------------------------------------------------------------------------- /tests/component_handlers/schema_handler/resources/module.py: -------------------------------------------------------------------------------- 1 | from schema_registry.client.schema import AvroSchema 2 | 3 | from kpops.component_handlers.schema_handler.schema_provider import ( 4 | Schema, 5 | SchemaProvider, 6 | ) 7 | from kpops.components.base_components.models import ModelName, ModelVersion 8 | 9 | 10 | class CustomSchemaProvider(SchemaProvider): 11 | def provide_schema( 12 | self, schema_class: str, models: dict[ModelName, ModelVersion] 13 | ) -> Schema: 14 | return AvroSchema({}) 15 | -------------------------------------------------------------------------------- /.github/workflows/add-issue-ci.yaml: -------------------------------------------------------------------------------- 1 | name: Add opened issues to specific project board 2 | 3 | on: 4 | issues: 5 | types: 6 | - opened 7 | 8 | jobs: 9 | add-issue-to-project: 10 | name: Add issue to project board 11 | runs-on: ubuntu-24.04 12 | steps: 13 | - name: Add the issue to the project 14 | uses: actions/add-to-project@v0.4.0 15 | with: 16 | project-url: https://github.com/orgs/bakdata/projects/5 17 | github-token: ${{ secrets.GH_TOKEN }} 18 | id: add-project 19 | -------------------------------------------------------------------------------- /kpops/components/base_components/__init__.py: -------------------------------------------------------------------------------- 1 | from .helm_app import HelmApp 2 | from .kafka_app import KafkaApp 3 | from .kafka_connector import ( 4 | KafkaConnector, 5 | KafkaSinkConnector, 6 | KafkaSourceConnector, 7 | ) 8 | from .kubernetes_app import KubernetesApp 9 | from .pipeline_component import PipelineComponent 10 | 11 | __all__ = ( 12 | "HelmApp", 13 | "KafkaApp", 14 | "KafkaConnector", 15 | "KafkaSinkConnector", 16 | "KafkaSourceConnector", 17 | "KubernetesApp", 18 | "PipelineComponent", 19 | ) 20 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_generate/test_model_serialization/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: account-producer 3 | prefix: resources-pipeline-with-paths- 4 | namespace: test 5 | values: 6 | image: bakdata-demo-producer-app 7 | kafka: 8 | bootstrapServers: test 9 | schemaRegistryUrl: http://localhost:8081/ 10 | outputTopic: out 11 | version: 3.6.1 12 | helm_release_name: resources-pipeline-with-paths-account-producer 13 | helm_name_override: resources-pipeline-with-paths-account-producer 14 | 15 | -------------------------------------------------------------------------------- /docs/README.md: -------------------------------------------------------------------------------- 1 | # Render the documentation locally 2 | 3 | ## Prerequisites 4 | 5 | - [uv](https://docs.astral.sh/uv/) 6 | - [just 1.5.0+](https://github.com/casey/just) 7 | 8 | ## Serve the docs 9 | 10 | 1. Clone the repository 11 | ```sh 12 | git clone git@github.com:bakdata/kpops.git 13 | ``` 14 | 2. Change directory to `kpops`. 15 | 3. Install the documentation dependencies and use `just` to serve the docs: 16 | ```sh 17 | just serve-docs 18 | ``` 19 | Go to your browser and open [http://localhost:8000/](http://localhost:8000/). 20 | -------------------------------------------------------------------------------- /docs/docs/resources/editor_integration/settings.json: -------------------------------------------------------------------------------- 1 | { 2 | "yaml.schemas": { 3 | "https://bakdata.github.io/kpops/4.0/schema/pipeline.json": [ 4 | "pipeline.yaml", 5 | "pipeline_*.yaml" 6 | ], 7 | "https://bakdata.github.io/kpops/4.0/schema/defaults.json": [ 8 | "defaults.yaml", 9 | "defaults_*.yaml" 10 | ], 11 | "https://bakdata.github.io/kpops/4.0/schema/config.json": [ 12 | "config.yaml", 13 | "config_*.yaml" 14 | ] 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /.github/workflows/publish.yaml: -------------------------------------------------------------------------------- 1 | name: Publish to PyPI 2 | on: 3 | push: 4 | tags: 5 | - "*" 6 | 7 | jobs: 8 | publish: 9 | name: Publish to PyPI 10 | runs-on: ubuntu-24.04 11 | permissions: 12 | id-token: write 13 | steps: 14 | - uses: actions/checkout@v4 15 | 16 | - name: Install uv 17 | uses: astral-sh/setup-uv@v5 18 | with: 19 | python-version: "3.11" 20 | version: "0.5.14" 21 | 22 | - name: Build 23 | run: uv build 24 | 25 | - name: Publish 26 | run: uv publish 27 | -------------------------------------------------------------------------------- /justfile: -------------------------------------------------------------------------------- 1 | set shell := ["bash", "-uc"] 2 | 3 | base-directory := justfile_directory() 4 | 5 | _default: 6 | @just --list --unsorted 7 | 8 | ############################################################################### 9 | ## Documentation 10 | ############################################################################### 11 | 12 | docs-config := base-directory / "docs/mkdocs.yml" 13 | 14 | # Serve current docs located in ./docs/docs 15 | serve-docs port="8000": 16 | uv run --frozen --group=docs mkdocs serve --config-file {{ docs-config }} --dev-addr localhost:{{ port }} 17 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/kafka-connector.md: -------------------------------------------------------------------------------- 1 | # KafkaConnector 2 | 3 | `KafkaConnector` is a component that deploys [Kafka Connectors](https://kafka.apache.org/documentation.html#connect_configuring){target=_blank}. Since a connector cannot be different from sink or source it is not recommended to use `KafkaConnector` for deployment in [`pipeline.yaml`](../../../resources/pipeline-components/pipeline.md). Instead, `KafkaConnector` should be used in [`defaults.yaml`](../defaults.md#kafkaconnector) to set defaults for all connectors in the pipeline as they can share some common settings. 4 | -------------------------------------------------------------------------------- /.bumpversion.cfg: -------------------------------------------------------------------------------- 1 | [bumpversion] 2 | current_version = 10.6.1 3 | parse = (?P\d+)\.(?P\d+)\.(?P\d+)(\-(?Pdev)(?P\d+)?)? 4 | serialize = 5 | {major}.{minor}.{patch}-{release}{timestamp} 6 | {major}.{minor}.{patch}-{release} 7 | {major}.{minor}.{patch} 8 | 9 | [bumpversion:part:release] 10 | first_value = dev 11 | optional_value = release 12 | values = 13 | dev 14 | release 15 | 16 | [bumpversion:file:pyproject.toml] 17 | search = version = "{current_version}" 18 | replace = version = "{new_version}" 19 | 20 | [bumpversion:file:kpops/const/__init__.py] 21 | -------------------------------------------------------------------------------- /tests/pipeline/resources/no-input-topic-pipeline/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: streams-app 2 | name: app1 3 | from: 4 | topics: 5 | ".*": 6 | type: pattern 7 | values: 8 | commandLine: 9 | CONVERT_XML: true 10 | resources: 11 | limits: 12 | memory: 2G 13 | requests: 14 | memory: 2G 15 | to: 16 | topics: 17 | example-output: 18 | type: output 19 | - type: streams-app 20 | name: app2 21 | to: 22 | topics: 23 | example-output-extra: 24 | label: extra 25 | test-output-extra: 26 | label: test-output 27 | -------------------------------------------------------------------------------- /tests/pipeline/resources/parallel-pipeline/defaults.yaml: -------------------------------------------------------------------------------- 1 | pipeline-component: 2 | prefix: "" 3 | 4 | kubernetes-app: 5 | namespace: ${NAMESPACE} 6 | 7 | streams-bootstrap: 8 | values: 9 | kafka: 10 | bootstrapServers: ${config.kafka_brokers} 11 | schemaRegistryUrl: ${config.schema_registry.url} 12 | 13 | streams-app: 14 | values: 15 | labels: 16 | pipeline: ${pipeline.name} 17 | to: 18 | topics: 19 | ${error_topic_name}: 20 | type: error 21 | partitions_count: 1 22 | ${output_topic_name}: 23 | type: output 24 | partitions_count: 3 25 | -------------------------------------------------------------------------------- /kpops/component_handlers/schema_handler/schema_provider.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | from abc import ABC, abstractmethod 4 | from typing import TYPE_CHECKING, TypeAlias 5 | 6 | from schema_registry.client.schema import AvroSchema, JsonSchema 7 | 8 | if TYPE_CHECKING: 9 | from kpops.components.base_components.models import ModelName, ModelVersion 10 | 11 | Schema: TypeAlias = AvroSchema | JsonSchema 12 | 13 | 14 | class SchemaProvider(ABC): 15 | @abstractmethod 16 | def provide_schema( 17 | self, schema_class: str, models: dict[ModelName, ModelVersion] 18 | ) -> Schema: ... 19 | -------------------------------------------------------------------------------- /tests/pipeline/resources/custom-config/config.yaml: -------------------------------------------------------------------------------- 1 | topic_name_config: 2 | default_error_topic_name: "${component.name}-dead-letter-topic" 3 | default_output_topic_name: "${component.name}-test-topic" 4 | kafka_brokers: "http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092" 5 | kafka_connect: 6 | url: "http://localhost:8083" 7 | kafka_rest: 8 | url: "http://localhost:8082" 9 | schema_registry: 10 | enabled: true 11 | url: "http://localhost:8081" 12 | helm_config: 13 | api_version: "2.1.1" 14 | pipeline_base_dir: tests/pipeline 15 | strimzi_topic: 16 | label: 17 | bakdata.com/cluster: my-cluster 18 | -------------------------------------------------------------------------------- /docs/docs/user/migration-guide/v9-v10.md: -------------------------------------------------------------------------------- 1 | # Migrate from V9 to V10 2 | 3 | ## Helm diff config 4 | 5 | - https://github.com/bakdata/kpops/pull/600 6 | - https://github.com/bakdata/kpops/pull/601 7 | 8 | The Helm diff configuration has moved from `config.yaml` to the `HelmApp` component. To ignore specific keypaths we use string arrays now (formerly dot-notation). It is now possible to ignore YAML keys containing `.` or other special characters. 9 | 10 | ### Example 11 | 12 | #### defaults.yaml 13 | 14 | ```yaml 15 | helm-app: 16 | diff_config: 17 | ignore: 18 | - ["metadata", "labels", "helm.sh/chart"] 19 | ``` 20 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-inflate/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | values: 3 | commandLine: 4 | FAKE_ARG: "fake-arg-value" 5 | schedule: "30 3/8 * * *" 6 | - type: converter 7 | values: 8 | commandLine: 9 | CONVERT_XML: true 10 | resources: 11 | limits: 12 | memory: 2G 13 | requests: 14 | memory: 2G 15 | - type: should-inflate 16 | values: 17 | commandLine: 18 | TYPE: "nothing" 19 | resources: 20 | requests: 21 | memory: 3G 22 | replicaCount: 4 23 | autoscaling: 24 | minReplicas: 4 25 | maxReplicas: 4 26 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/kubernetes-app.md: -------------------------------------------------------------------------------- 1 | # KubernetesApp 2 | 3 | ### Usage 4 | 5 | Can be used to create components for any Kubernetes app. 6 | 7 | ### Configuration 8 | 9 | 10 | 11 | ??? example "`pipeline.yaml`" 12 | 13 | ```yaml 14 | --8<-- 15 | ./docs/resources/pipeline-components/kubernetes-app.yaml 16 | --8<-- 17 | ``` 18 | 19 | 20 | 21 | ### Operations 22 | 23 | #### deploy 24 | 25 | Do nothing. 26 | 27 | #### destroy 28 | 29 | Do nothing. 30 | 31 | #### reset 32 | 33 | Do nothing. 34 | 35 | #### clean 36 | 37 | Do nothing. 38 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/values-producer-app.yaml: -------------------------------------------------------------------------------- 1 | # Allowed configs: 2 | # https://github.com/bakdata/streams-bootstrap/tree/master/charts/producer-app 3 | values: # required 4 | kafka: # required, producer-app-specific 5 | bootstrapServers: ${config.kafka_brokers} 6 | schemaRegistryUrl: ${config.schema_registry.url} 7 | outputTopic: output_topic 8 | labeledOutputTopics: 9 | output_role1: output_topic1 10 | output_role2: output_topic2 11 | nameOverride: override-with-this-name # kafka-app-specific 12 | fullnameOverride: override-with-this-name # kafka-app-specific 13 | -------------------------------------------------------------------------------- /tests/components/resources/pipelines/test-distributed-defaults/pipeline-deep/defaults.yaml: -------------------------------------------------------------------------------- 1 | helm-app: 2 | name: ${component.type} 3 | namespace: namespace-to-override-it-all 4 | 5 | streams-bootstrap-v2: 6 | app: 7 | streams: 8 | brokers: "${config.kafka_brokers}" 9 | 10 | producer-app-v2: {} # inherits from streams-bootstrap-v2 11 | 12 | streams-app-v2: # inherits from streams-bootstrap-v2 13 | to: 14 | topics: 15 | ${error_topic_name}: 16 | type: error 17 | value_schema: com.bakdata.kafka.DeadLetter 18 | partitions_count: 1 19 | configs: 20 | cleanup.policy: compact,delete 21 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-envs/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | name: input-producer 3 | values: 4 | commandLine: 5 | FAKE_ARG: fake-arg-value 6 | schedule: 30 3/8 * * * 7 | - type: converter 8 | values: 9 | commandLine: 10 | CONVERT_XML: true 11 | resources: 12 | limits: 13 | memory: 2G 14 | requests: 15 | memory: 2G 16 | - type: filter 17 | values: 18 | commandLine: 19 | TYPE: nothing 20 | resources: 21 | requests: 22 | memory: 3G 23 | replicaCount: 4 24 | autoscaling: 25 | minReplicas: 4 26 | maxReplicas: 4 27 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-short-topics/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: component-input 3 | 4 | - type: producer-app 5 | name: component-extra 6 | 7 | - type: producer-app 8 | name: component-input-pattern 9 | 10 | - type: producer-app 11 | name: component-extra-pattern 12 | 13 | - type: streams-app 14 | name: simple-app 15 | from: 16 | components: 17 | component-input: 18 | type: null 19 | component-extra: 20 | label: role 21 | component-input-pattern: 22 | type: pattern 23 | component-extra-pattern: 24 | type: pattern 25 | label: role 26 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipelines-with-graphs/simple-pipeline/defaults.yaml: -------------------------------------------------------------------------------- 1 | pipeline-component: 2 | prefix: "" 3 | 4 | kubernetes-app: 5 | namespace: example-namespace 6 | 7 | kafka-connector: 8 | namespace: example-namespace 9 | 10 | streams-bootstrap: 11 | values: 12 | kafka: 13 | bootstrapServers: 127.0.0.1:9092 14 | schemaRegistryUrl: 127.0.0.1:8081 15 | 16 | streams-app: 17 | values: 18 | labels: 19 | pipeline: ${pipeline.name} 20 | 21 | producer-app: 22 | to: 23 | topics: 24 | ${output_topic_name}: 25 | type: output 26 | configs: 27 | cleanup.policy: compact,delete 28 | -------------------------------------------------------------------------------- /kpops/utils/__init__.py: -------------------------------------------------------------------------------- 1 | from collections.abc import Callable 2 | from functools import cached_property 3 | from typing import TYPE_CHECKING, Any, Concatenate, Generic, TypeVar 4 | 5 | _T = TypeVar("_T") 6 | _R = TypeVar("_R") 7 | 8 | if TYPE_CHECKING: 9 | 10 | def cached_classproperty( 11 | func: Callable[Concatenate[Any, ...], _R], attrname: str | None = None 12 | ) -> _R: ... 13 | 14 | else: 15 | 16 | class cached_classproperty(cached_property, Generic[_T, _R]): 17 | def __get__(self, owner_obj: _T, owner_cls: type[_T]) -> _R: 18 | self.func: Callable[[type[_T]], _R] 19 | return self.func(owner_cls) 20 | -------------------------------------------------------------------------------- /tests/pipeline/resources/no-topics-defaults/defaults.yaml: -------------------------------------------------------------------------------- 1 | streams-bootstrap: 2 | values: 3 | kafka: 4 | bootstrapServers: "${config.kafka_brokers}" 5 | schemaRegistryUrl: "${config.schema_registry.url}" 6 | 7 | producer-app: 8 | to: 9 | topics: 10 | ${output_topic_name}: 11 | type: output 12 | partitions_count: 3 13 | 14 | streams-app: 15 | values: 16 | labels: 17 | pipeline: ${pipeline.name} 18 | to: 19 | topics: 20 | ${error_topic_name}: 21 | type: error 22 | partitions_count: 1 23 | ${output_topic_name}: 24 | type: output 25 | partitions_count: 3 26 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipelines-with-graphs/same-topic-and-component-name/defaults.yaml: -------------------------------------------------------------------------------- 1 | pipeline-component: 2 | prefix: "" 3 | 4 | kubernetes-app: 5 | namespace: example-namespace 6 | 7 | kafka-connector: 8 | namespace: example-namespace 9 | 10 | streams-bootstrap: 11 | values: 12 | kafka: 13 | bootstrapServers: 127.0.0.1:9092 14 | schemaRegistryUrl: 127.0.0.1:8081 15 | 16 | streams-app: 17 | values: 18 | labels: 19 | pipeline: ${pipeline.name} 20 | 21 | producer: 22 | to: 23 | topics: 24 | ${output_topic_name}: 25 | type: output 26 | configs: 27 | cleanup.policy: compact,delete 28 | -------------------------------------------------------------------------------- /tests/pipeline/resources/manifest-pipeline/defaults.yaml: -------------------------------------------------------------------------------- 1 | streams-bootstrap: 2 | version: "3.6.1" 3 | values: 4 | kafka: 5 | bootstrapServers: ${config.kafka_brokers} 6 | schemaRegistryUrl: ${config.schema_registry.url} 7 | 8 | producer-app: {} # inherits from streams-bootstrap 9 | 10 | streams-app: # inherits from streams-bootstrap 11 | values: 12 | prometheus: 13 | jmx: 14 | enabled: false 15 | to: 16 | topics: 17 | ${error_topic_name}: 18 | type: error 19 | value_schema: com.bakdata.kafka.DeadLetter 20 | partitions_count: 1 21 | configs: 22 | cleanup.policy: compact,delete 23 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-illegal-kubernetes-name/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | values: 3 | commandLine: 4 | FAKE_ARG: "fake-arg-value" 5 | schedule: "30 3/8 * * *" 6 | - type: converter 7 | values: 8 | commandLine: 9 | CONVERT_XML: true 10 | resources: 11 | limits: 12 | memory: 2G 13 | requests: 14 | memory: 2G 15 | - type: filter 16 | name: "illegal_name" 17 | values: 18 | commandLine: 19 | TYPE: "nothing" 20 | resources: 21 | requests: 22 | memory: 3G 23 | replicaCount: 4 24 | autoscaling: 25 | minReplicas: 4 26 | maxReplicas: 4 27 | -------------------------------------------------------------------------------- /docs/docs/user/migration-guide/v4-v5.md: -------------------------------------------------------------------------------- 1 | # Migrate from V4 to V5 2 | 3 | ## [Allow custom timeout for external services](https://github.com/bakdata/kpops/pull/485) 4 | 5 | The global `timeout` setting has been removed. Instead, an individual timeout can be set for each external service. The default is 30 seconds. 6 | 7 | #### config.yaml 8 | 9 | ```diff 10 | - timeout: 300 11 | 12 | kafka_rest: 13 | url: "http://my-custom-rest.url:8082" 14 | + timeout: 30 15 | kafka_connect: 16 | url: "http://my-custom-connect.url:8083" 17 | + timeout: 30 18 | schema_registry: 19 | enabled: true 20 | url: "http://my-custom-sr.url:8081" 21 | + timeout: 30 22 | ``` 23 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/helm-app.md: -------------------------------------------------------------------------------- 1 | # HelmApp 2 | 3 | ### Usage 4 | 5 | Can be used to deploy any app in Kubernetes using Helm, for example, a REST service that serves Kafka data. 6 | 7 | ### Configuration 8 | 9 | 10 | 11 | ??? example "`pipeline.yaml`" 12 | 13 | ```yaml 14 | --8<-- 15 | ./docs/resources/pipeline-components/helm-app.yaml 16 | --8<-- 17 | ``` 18 | 19 | 20 | 21 | ### Operations 22 | 23 | #### deploy 24 | 25 | Deploy using Helm. 26 | 27 | #### destroy 28 | 29 | Uninstall Helm release. 30 | 31 | #### reset 32 | 33 | Do nothing. 34 | 35 | #### clean 36 | 37 | Do nothing. 38 | -------------------------------------------------------------------------------- /tests/pipeline/resources/custom-config/defaults.yaml: -------------------------------------------------------------------------------- 1 | streams-bootstrap: 2 | version: "3.6.1" 3 | namespace: development-namespace 4 | values: 5 | kafka: 6 | bootstrapServers: "${config.kafka_brokers}" 7 | schemaRegistryUrl: "${config.schema_registry.url}" 8 | 9 | producer-app: 10 | to: 11 | topics: 12 | ${output_topic_name}: 13 | type: output 14 | partitions_count: 3 15 | 16 | streams-app: 17 | values: 18 | labels: 19 | pipeline: ${pipeline.name} 20 | to: 21 | topics: 22 | ${error_topic_name}: 23 | type: error 24 | partitions_count: 1 25 | ${output_topic_name}: 26 | type: output 27 | partitions_count: 3 28 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-loop/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: app1 3 | values: 4 | image: producer-image 5 | to: 6 | topics: 7 | my-output-topic: 8 | type: output 9 | 10 | - type: streams-app 11 | name: app2 12 | values: 13 | image: app2-image 14 | from: 15 | topics: 16 | my-output-topic: 17 | type: input 18 | to: 19 | topics: 20 | my-app2-topic: 21 | type: output 22 | 23 | - type: streams-app 24 | name: app3 25 | values: 26 | image: app3-image 27 | from: 28 | topics: 29 | my-app2-topic: 30 | type: input 31 | to: 32 | topics: 33 | my-output-topic: 34 | type: output 35 | -------------------------------------------------------------------------------- /tests/pipeline/resources/temp-trim-release-name/defaults.yaml: -------------------------------------------------------------------------------- 1 | kubernetes-app: 2 | namespace: example-namespace 3 | 4 | streams-bootstrap: 5 | values: 6 | kafka: 7 | bootstrapServers: "${config.kafka_brokers}" 8 | schema_registry_url: "${schema_registry_url}" 9 | version: "3.6.1" 10 | 11 | streams-app: # inherits from streams-bootstrap 12 | values: 13 | kafka: 14 | config: 15 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 16 | to: 17 | topics: 18 | ${error_topic_name}: 19 | type: error 20 | value_schema: com.bakdata.kafka.DeadLetter 21 | partitions_count: 1 22 | configs: 23 | cleanup.policy: compact,delete 24 | -------------------------------------------------------------------------------- /kpops/components/streams_bootstrap_v2/producer/model.py: -------------------------------------------------------------------------------- 1 | from typing import ClassVar 2 | 3 | from pydantic import ConfigDict 4 | 5 | from kpops.components.streams_bootstrap_v2.base import ( 6 | KafkaStreamsConfig, 7 | StreamsBootstrapV2Values, 8 | ) 9 | 10 | 11 | class ProducerStreamsConfig(KafkaStreamsConfig): 12 | """Kafka Streams settings specific to Producer.""" 13 | 14 | 15 | class ProducerAppV2Values(StreamsBootstrapV2Values): 16 | """Settings specific to producers. 17 | 18 | :param streams: Kafka Streams settings 19 | """ 20 | 21 | streams: ProducerStreamsConfig # pyright: ignore[reportIncompatibleVariableOverride] 22 | 23 | model_config: ClassVar[ConfigDict] = ConfigDict(extra="allow") 24 | -------------------------------------------------------------------------------- /.github/workflows/update-gh-pages.yaml: -------------------------------------------------------------------------------- 1 | name: Update GitHub pages 2 | 3 | on: 4 | push: 5 | tags: 6 | # Push events to semver tags 7 | - "[0-9]+.[0-9]+.[0-9]+" 8 | 9 | jobs: 10 | update-docs: 11 | runs-on: ubuntu-24.04 12 | name: Update docs 13 | steps: 14 | - name: checkout 15 | uses: actions/checkout@v4 16 | with: 17 | fetch-depth: 0 # Fetch all tags; Required for doc versioning 18 | 19 | - name: Update gh-pages 20 | uses: ./.github/actions/update-docs 21 | with: 22 | username: ${{ secrets.GH_USERNAME }} 23 | email: ${{ secrets.GH_EMAIL }} 24 | token: ${{ secrets.GH_TOKEN }} 25 | version: ${{ github.ref_name }} 26 | release: true 27 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_generate/test_prefix_pipeline_component/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: account-producer 3 | prefix: from-pipeline-component- 4 | namespace: ${NAMESPACE} 5 | values: 6 | image: ${DOCKER_REGISTRY}/atm-demo-accountproducer 7 | imageTag: 1.0.0 8 | kafka: 9 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 10 | schemaRegistryUrl: http://localhost:8081/ 11 | schedule: 0 12 * * * 12 | suspend: true 13 | replicaCount: 1 14 | prometheus: 15 | jmx: 16 | enabled: false 17 | debug: true 18 | version: 3.6.1 19 | helm_release_name: from-pipeline-component-account-producer 20 | helm_name_override: from-pipeline-component-account-producer 21 | 22 | -------------------------------------------------------------------------------- /tests/pipeline/resources/first-pipeline/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | values: 3 | commandLine: 4 | FAKE_ARG: "fake-arg-value" 5 | schedule: "30 3/8 * * *" 6 | - type: converter 7 | values: 8 | commandLine: 9 | CONVERT_XML: true 10 | resources: 11 | limits: 12 | memory: 2G 13 | requests: 14 | memory: 2G 15 | - type: filter 16 | name: "a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name-a-long-name" 17 | values: 18 | commandLine: 19 | TYPE: "nothing" 20 | resources: 21 | requests: 22 | memory: 3G 23 | replicaCount: 4 24 | autoscaling: 25 | minReplicas: 4 26 | maxReplicas: 4 27 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/defaults-helm-app.yaml: -------------------------------------------------------------------------------- 1 | # Kubernetes app managed through Helm with an associated Helm chart 2 | # 3 | # Parent of: KafkaApp 4 | # Child of: KubernetesApp 5 | helm-app: 6 | values: # required 7 | image: exampleImage # Example 8 | debug: false # Example 9 | commandLine: {} # Example 10 | # Helm repository configuration (optional) 11 | # If not set the helm repo add will not be called. Useful when using local Helm charts 12 | repo_config: 13 | repository_name: bakdata-streams-bootstrap # required 14 | url: https://bakdata.github.io/streams-bootstrap/ # required 15 | repo_auth_flags: 16 | username: user 17 | password: pass 18 | ca_file: /home/user/path/to/ca-file 19 | insecure_skip_tls_verify: false 20 | -------------------------------------------------------------------------------- /tests/cli/snapshots/test_init/test_init_project_include_optional/config.yaml: -------------------------------------------------------------------------------- 1 | # Global configuration for KPOps project. 2 | 3 | # Required fields 4 | kafka_brokers: null 5 | 6 | # Non-required fields 7 | create_namespace: false 8 | helm_config: 9 | api_version: null 10 | context: null 11 | debug: false 12 | kafka_connect: 13 | timeout: 30 14 | url: http://localhost:8083/ 15 | kafka_rest: 16 | timeout: 30 17 | url: http://localhost:8082/ 18 | pipeline_base_dir: . 19 | retain_clean_jobs: false 20 | schema_registry: 21 | enabled: false 22 | timeout: 30 23 | url: http://localhost:8081/ 24 | strimzi_topic: null 25 | topic_name_config: 26 | default_error_topic_name: ${pipeline.name}-${component.name}-error 27 | default_output_topic_name: ${pipeline.name}-${component.name} 28 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/dependencies/defaults_pipeline_component_dependencies.yaml: -------------------------------------------------------------------------------- 1 | helm-app.yaml: 2 | - values-helm-app.yaml 3 | - repo_config-helm-app.yaml 4 | kafka-app.yaml: 5 | - prefix.yaml 6 | - from_.yaml 7 | - to.yaml 8 | kafka-connector.yaml: 9 | - prefix.yaml 10 | - from_.yaml 11 | - to.yaml 12 | - config-kafka-connector.yaml 13 | - resetter_values.yaml 14 | kafka-sink-connector.yaml: [] 15 | kafka-source-connector.yaml: 16 | - from_-kafka-source-connector.yaml 17 | - offset_topic-kafka-source-connector.yaml 18 | kubernetes-app.yaml: 19 | - prefix.yaml 20 | - from_.yaml 21 | - to.yaml 22 | - namespace.yaml 23 | - values-kubernetes-app.yaml 24 | producer-app.yaml: 25 | - from_-producer-app.yaml 26 | - values-producer-app.yaml 27 | streams-app.yaml: 28 | - values-streams-app.yaml 29 | -------------------------------------------------------------------------------- /docs/docs/developer/getting-started.md: -------------------------------------------------------------------------------- 1 | # Getting started 2 | 3 | **Welcome!** We are glad to have you visit our developer guide! If you find any bugs or have suggestions for improvements, please open an [issue](https://github.com/bakdata/kpops/issues/new) and optionally a [pull request (PR)](https://github.com/bakdata/kpops/compare). In the case of a PR, we would appreciate it if you preface it with an issue outlining your goal and means of achieving it. 4 | 5 | Find more about our code-style or insights into KPOps' code base here in our developer guide. 6 | 7 | 8 | 9 | !!! wip "Work in progress" 10 | The developer guide is still under construction. If you have a question left unanswered here, feel free to ask it by opening an issue. 11 | 12 | 13 | -------------------------------------------------------------------------------- /docs/docs/user/migration-guide/v5-v6.md: -------------------------------------------------------------------------------- 1 | # Migrate from V5 to V6 2 | 3 | ## [Deploy multiple pipelines](https://github.com/bakdata/kpops/pull/487) 4 | 5 | KPOps can now deploy multiple pipelines in a single command. It is possible to pass one or many pipeline.yaml files or pass a directory with many pipeline.yaml files within it. 6 | 7 | The environment variable `KPOPS_PIPELINE_PATH` is changed to `KPOPS_PIPELINE_PATHS`. 8 | 9 | Read more: 10 | 11 | - [CLI Usage](https://bakdata.github.io/kpops/6.0/user/references/cli-commands/) 12 | - [Environment variables](https://bakdata.github.io/kpops/6.0/user/core-concepts/variables/environment_variables/) 13 | 14 | ## [Separate KPOps API from the CLI](https://github.com/bakdata/kpops/pull/489) 15 | 16 | KPops Python API is now stable and separated from the CLI! 🎉 17 | -------------------------------------------------------------------------------- /docs/docs/resources/examples/pipeline.md: -------------------------------------------------------------------------------- 1 | # Example `pipeline.yaml` files 2 | 3 | ## [ATM Fraud Pipeline](https://github.com/bakdata/kpops-examples/tree/main/atm-fraud){target=_blank} 4 | 5 | 6 | 7 | ??? example "pipeline.yaml" 8 | ```yaml 9 | --8<-- 10 | https://raw.githubusercontent.com/bakdata/kpops-examples/main/atm-fraud/pipeline.yaml 11 | --8<-- 12 | ``` 13 | 14 | 15 | 16 | ## [Word-count Pipeline](https://github.com/bakdata/kpops-examples/tree/main/word-count){target=_blank} 17 | 18 | 19 | 20 | ??? example "pipeline.yaml" 21 | ```yaml 22 | --8<-- 23 | https://raw.githubusercontent.com/bakdata/kpops-examples/main/word-count/pipeline.yaml 24 | --8<-- 25 | ``` 26 | 27 | 28 | -------------------------------------------------------------------------------- /kpops/manifests/argo.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | import enum 4 | from typing import Any 5 | 6 | from pydantic import BaseModel 7 | 8 | 9 | def enrich_annotations( 10 | helm_values: dict[str, Any], key: str, value: str 11 | ) -> dict[str, Any]: 12 | annotations = helm_values.setdefault("annotations", {}) 13 | annotations[key] = value 14 | return helm_values 15 | 16 | 17 | class ArgoHook(str, enum.Enum): 18 | POST_DELETE = "PostDelete" 19 | 20 | @property 21 | def key(self) -> str: 22 | return "argocd.argoproj.io/hook" 23 | 24 | 25 | class ArgoSyncWave(BaseModel): 26 | sync_wave: int = 0 27 | 28 | @property 29 | def key(self) -> str: 30 | return "argocd.argoproj.io/sync-wave" 31 | 32 | @property 33 | def value(self) -> str: 34 | return str(self.sync_wave) 35 | -------------------------------------------------------------------------------- /docs/docs/resources/examples/defaults.md: -------------------------------------------------------------------------------- 1 | # Example `defaults.yaml` files 2 | 3 | ## [KPOps examples](https://github.com/bakdata/kpops-examples){target=_blank} 4 | 5 | Defaults for [ATM Fraud Pipeline](https://github.com/bakdata/kpops-examples/tree/main/atm-fraud){target=_blank} 6 | 7 | 8 | 9 | ??? example "defaults.yaml" 10 | ```yaml 11 | --8<-- 12 | https://raw.githubusercontent.com/bakdata/kpops-examples/main/atm-fraud/defaults.yaml 13 | --8<-- 14 | ``` 15 | 16 | 17 | 18 | Defaults for [Word-count Pipeline](https://github.com/bakdata/kpops-examples/tree/main/word-count){target=_blank} 19 | 20 | ??? example "defaults.yaml" 21 | `yaml --8<-- https://raw.githubusercontent.com/bakdata/kpops-examples/main/word-count/defaults.yaml --8<--` 22 | 23 | 24 | -------------------------------------------------------------------------------- /kpops/component_handlers/utils/exception.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | 4 | import httpx 5 | 6 | log = logging.getLogger("HttpxException") 7 | 8 | 9 | class HttpxException(Exception): 10 | def __init__(self, response: httpx.Response) -> None: 11 | self.error_code: int = response.status_code 12 | self.error_msg: str = "Something went wrong!" 13 | log_lines = [f"The request responded with the code {self.error_code}."] 14 | if response.headers.get("Content-Type") == "application/json": 15 | log_lines.append("Error body:") 16 | log_lines.append(json.dumps(response.json(), indent=2)) 17 | try: 18 | response.raise_for_status() 19 | except httpx.HTTPError as e: 20 | log_lines.append(str(e)) 21 | log.exception(" ".join(log_lines)) 22 | super().__init__() 23 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/to.yaml: -------------------------------------------------------------------------------- 1 | # Topic(s) into which the component will write output 2 | to: 3 | topics: 4 | ${pipeline.name}-output-topic: 5 | type: output # Implied when role is NOT specified 6 | ${pipeline.name}-extra-topic: 7 | role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined 8 | ${pipeline.name}-error-topic: 9 | type: error 10 | # Currently KPOps supports Avro and JSON schemas. 11 | key_schema: key-schema # must implement SchemaProvider to use 12 | value_schema: value-schema 13 | partitions_count: 1 14 | replication_factor: 1 15 | configs: # https://kafka.apache.org/documentation/#topicconfigs 16 | cleanup.policy: compact 17 | models: # SchemaProvider is initiated with the values given here 18 | model: model 19 | -------------------------------------------------------------------------------- /hooks/gen_docs/__init__.py: -------------------------------------------------------------------------------- 1 | """Documentation generation.""" 2 | 3 | from collections.abc import Iterator 4 | from enum import StrEnum 5 | 6 | 7 | class IterableStrEnum(StrEnum): 8 | """Polyfill that also introduces dict-like behavior. 9 | 10 | Introduces constructors that return a ``Iterator`` object 11 | either containing all items, only their names or their values. 12 | """ 13 | 14 | @classmethod 15 | def items(cls) -> Iterator[tuple[str, str]]: 16 | """Return all item names and values in tuples.""" 17 | return ((e.name, e.value) for e in cls) 18 | 19 | @classmethod 20 | def keys(cls) -> Iterator[str]: 21 | """Return all item names.""" 22 | return (e.name for e in cls) 23 | 24 | @classmethod 25 | def values(cls) -> Iterator[str]: 26 | """Return all item values.""" 27 | return (e.value for e in cls) 28 | -------------------------------------------------------------------------------- /.github/workflows/release.yaml: -------------------------------------------------------------------------------- 1 | name: Release to GitHub and push tag 2 | 3 | on: 4 | workflow_dispatch: 5 | inputs: 6 | release-type: 7 | description: "SemVer scope of the release" 8 | type: choice 9 | required: true 10 | default: patch 11 | options: 12 | - patch 13 | - minor 14 | - major 15 | 16 | jobs: 17 | create-github-release-push-tag: 18 | uses: bakdata/ci-templates/.github/workflows/python-uv-release.yaml@1.69.1 19 | name: Release 20 | with: 21 | release-type: ${{ inputs.release-type }} 22 | python-version: "3.11" 23 | uv-version: "0.5.14" 24 | changelog: true 25 | changelog-file: "./docs/docs/user/changelog.md" 26 | secrets: 27 | github-username: "${{ secrets.GH_USERNAME }}" 28 | github-email: "${{ secrets.GH_EMAIL }}" 29 | github-token: "${{ secrets.GH_TOKEN }}" 30 | -------------------------------------------------------------------------------- /docs/docs/resources/architecture/components-hierarchy.md: -------------------------------------------------------------------------------- 1 | ```mermaid 2 | flowchart BT 3 | KubernetesApp --> PipelineComponent 4 | HelmApp --> KubernetesApp 5 | StreamsBootstrap --> HelmApp 6 | StreamsApp --> StreamsBootstrap 7 | ProducerApp --> StreamsBootstrap 8 | KafkaConnector --> PipelineComponent 9 | KafkaSourceConnector --> KafkaConnector 10 | KafkaSinkConnector --> KafkaConnector 11 | 12 | click KubernetesApp "./../kubernetes-app" 13 | click HelmApp "./../helm-app" 14 | click StreamsBootstrap "./../streams-bootstrap" 15 | click StreamsApp "./../streams-app" 16 | click ProducerApp "./../producer-app" 17 | click KafkaConnector "./../kafka-connector" 18 | click KafkaSourceConnector "./../kafka-source-connector" 19 | click KafkaSinkConnector "./../kafka-sink-connector" 20 | ``` 21 | 22 |

KPOps component hierarchy

23 | -------------------------------------------------------------------------------- /kpops/components/base_components/kafka_app.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | import logging 4 | from abc import ABC 5 | 6 | from typing_extensions import override 7 | 8 | from kpops.component_handlers import get_handlers 9 | from kpops.components.base_components.pipeline_component import PipelineComponent 10 | 11 | log = logging.getLogger("KafkaApp") 12 | 13 | 14 | class KafkaApp(PipelineComponent, ABC): 15 | """Base component for Kafka-based components.""" 16 | 17 | @override 18 | async def deploy(self, dry_run: bool) -> None: 19 | if self.to: 20 | for topic in self.to.kafka_topics: 21 | await get_handlers().topic_handler.create_topic(topic, dry_run=dry_run) 22 | 23 | if schema_handler := get_handlers().schema_handler: 24 | await schema_handler.submit_schemas(to_section=self.to, dry_run=dry_run) 25 | 26 | await super().deploy(dry_run) 27 | -------------------------------------------------------------------------------- /kpops/component_handlers/kafka_connect/timeout.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import builtins 3 | import logging 4 | from collections.abc import Coroutine 5 | from typing import Any, TypeVar 6 | 7 | log = logging.getLogger("Timeout") 8 | 9 | T = TypeVar("T") 10 | 11 | 12 | async def timeout(coro: Coroutine[Any, Any, T], *, secs: int = 0) -> T | None: 13 | """Set a timeout for a given lambda function. 14 | 15 | :param coro: The callable function 16 | :param secs: The timeout in seconds. 17 | """ 18 | try: 19 | task = asyncio.create_task(coro) 20 | if secs == 0: 21 | return await task 22 | else: 23 | return await asyncio.wait_for(task, timeout=secs) 24 | except builtins.TimeoutError: 25 | log.exception( 26 | f"Kafka Connect operation {coro.__name__} timed out after {secs} seconds. To increase the duration, set the `timeout` option in config.yaml." 27 | ) 28 | -------------------------------------------------------------------------------- /tests/component_handlers/resources/kafka_rest_proxy_responses/get_topic_response.json: -------------------------------------------------------------------------------- 1 | { 2 | "kind": "KafkaTopic", 3 | "metadata": { 4 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-X", 5 | "resource_name": "crn:///kafka=cluster-1/topic=topic-X" 6 | }, 7 | "cluster_id": "cluster-1", 8 | "topic_name": "topic-X", 9 | "is_internal": "false", 10 | "replication_factor": 3, 11 | "partitions_count": 10, 12 | "partitions": { 13 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/partitions" 14 | }, 15 | "configs": { 16 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/configs" 17 | }, 18 | "partition_reassignments": { 19 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/partitions/-/reassignments" 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /dprint.jsonc: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "https://dprint.dev/schemas/v0.json", 3 | "incremental": true, 4 | "json": {}, 5 | "yaml": {}, 6 | "markdown": {}, 7 | "toml": {}, 8 | "malva": {}, 9 | "includes": ["**/*.{json,jsonc,yaml,yml,md,toml,css}"], 10 | "excludes": [ 11 | ".pytest_cache/**", 12 | ".mypy_cache/**", 13 | "**/.venv/**", 14 | "CHANGELOG.md", 15 | "**/cli-commands.md", 16 | "**/cli_env_vars.md", 17 | "**/config_env_vars.md", 18 | "tests/compiler/resources/erroneous-file.yaml", 19 | "tests/**/snapshots/**", 20 | "docs/docs/resources/**", 21 | "docs/docs/schema/**", 22 | ], 23 | "plugins": [ 24 | "https://plugins.dprint.dev/json-0.19.4.wasm", 25 | "https://plugins.dprint.dev/g-plane/pretty_yaml-v0.5.0.wasm", 26 | "https://plugins.dprint.dev/markdown-0.17.8.wasm", 27 | "https://plugins.dprint.dev/toml-0.6.4.wasm", 28 | "https://plugins.dprint.dev/g-plane/malva-v0.11.1.wasm", 29 | ], 30 | } 31 | -------------------------------------------------------------------------------- /tests/component_handlers/resources/kafka_rest_proxy_responses/get_default_topic_response.json: -------------------------------------------------------------------------------- 1 | { 2 | "kind": "KafkaTopic", 3 | "metadata": { 4 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-X", 5 | "resource_name": "crn:///kafka=cluster-1/topic=topic-X" 6 | }, 7 | "cluster_id": "cluster-1", 8 | "topic_name": "topic-X", 9 | "is_internal": "false", 10 | "replication_factor": 1, 11 | "partitions_count": 1, 12 | "partitions": { 13 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/partitions" 14 | }, 15 | "configs": { 16 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/configs" 17 | }, 18 | "partition_reassignments": { 19 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/partitions/-/reassignments" 20 | } 21 | } 22 | -------------------------------------------------------------------------------- /tests/components/conftest.py: -------------------------------------------------------------------------------- 1 | from unittest import mock 2 | 3 | import pytest 4 | 5 | from kpops.component_handlers import ComponentHandlers 6 | from kpops.config import KpopsConfig, TopicNameConfig, set_config 7 | from tests.components import PIPELINE_BASE_DIR 8 | 9 | 10 | @pytest.fixture(autouse=True, scope="module") 11 | def config() -> None: 12 | config = KpopsConfig( 13 | topic_name_config=TopicNameConfig( 14 | default_error_topic_name="${component.type}-error-topic", 15 | default_output_topic_name="${component.type}-output-topic", 16 | ), 17 | kafka_brokers="broker:9092", 18 | pipeline_base_dir=PIPELINE_BASE_DIR, 19 | ) 20 | set_config(config) 21 | 22 | 23 | @pytest.fixture(autouse=True, scope="module") 24 | def handlers() -> None: 25 | ComponentHandlers( 26 | schema_handler=mock.AsyncMock(), 27 | connector_handler=mock.AsyncMock(), 28 | topic_handler=mock.AsyncMock(), 29 | ) 30 | -------------------------------------------------------------------------------- /kpops/components/base_components/models/to_section.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | from typing import ClassVar 4 | 5 | from pydantic import ConfigDict 6 | 7 | from kpops.components.base_components.models import ModelName, ModelVersion, TopicName 8 | from kpops.components.common.topic import KafkaTopic, TopicConfig 9 | from kpops.utils.pydantic import DescConfigModel 10 | 11 | 12 | class ToSection(DescConfigModel): 13 | """Holds multiple output topics. 14 | 15 | :param topics: Output topics 16 | :param models: Data models 17 | """ 18 | 19 | topics: dict[TopicName, TopicConfig] = {} 20 | models: dict[ModelName, ModelVersion] = {} 21 | 22 | model_config: ClassVar[ConfigDict] = ConfigDict(extra="forbid") 23 | 24 | @property 25 | def kafka_topics(self) -> list[KafkaTopic]: 26 | return [ 27 | KafkaTopic(name=topic_name, config=topic_config) 28 | for topic_name, topic_config in self.topics.items() 29 | ] 30 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-short-topics/defaults.yaml: -------------------------------------------------------------------------------- 1 | kubernetes-app: 2 | name: "${component_type}" 3 | namespace: example-namespace 4 | 5 | streams-bootstrap: 6 | values: 7 | kafka: 8 | bootstrapServers: "${config.kafka_brokers}" 9 | schema_registry_url: "${config.schema_registry.url}" 10 | version: "3.6.1" 11 | 12 | producer-app: 13 | to: 14 | topics: 15 | ${output_topic_name}: 16 | partitions_count: 3 17 | 18 | streams-app: # inherits from streams-bootstrap 19 | values: 20 | kafka: 21 | bootstrapServers: test 22 | to: 23 | topics: 24 | output-topic: 25 | type: 26 | error-topic: 27 | type: error 28 | extra-topic-output: 29 | label: role 30 | from: 31 | topics: 32 | input-topic: 33 | type: input 34 | extra-topic: 35 | label: role 36 | input-pattern: 37 | type: pattern 38 | extra-pattern: 39 | type: pattern 40 | label: role 41 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/defaults-producer-app.yaml: -------------------------------------------------------------------------------- 1 | # Holds configuration to use as values for the streams bootstrap producer-app Helm 2 | # chart. 3 | # 4 | # Child of: KafkaApp 5 | # More documentation on ProducerApp: https://github.com/bakdata/streams-bootstrap 6 | producer-app: 7 | # from: # While the producer-app does inherit from kafka-app, it does not need a 8 | # `from` section, hence it does not support it. 9 | # Allowed configs: 10 | # https://github.com/bakdata/streams-bootstrap/tree/master/charts/producer-app 11 | values: # required 12 | kafka: # required, producer-app-specific 13 | bootstrapServers: ${config.kafka_brokers} 14 | schemaRegistryUrl: ${config.schema_registry.url} 15 | outputTopic: output_topic 16 | labeledOutputTopics: 17 | output_role1: output_topic1 18 | output_role2: output_topic2 19 | nameOverride: override-with-this-name # kafka-app-specific 20 | fullnameOverride: override-with-this-name # kafka-app-specific 21 | -------------------------------------------------------------------------------- /hooks/gen_docs/gen_docs_cli_usage.py: -------------------------------------------------------------------------------- 1 | """Generates the documentation on KPOps CLI usage.""" 2 | 3 | import subprocess 4 | 5 | from hooks import ROOT 6 | 7 | PATH_KPOPS_MAIN = ROOT / "kpops/cli/main.py" 8 | PATH_CLI_COMMANDS_DOC = ROOT / "docs/docs/user/references/cli-commands.md" 9 | 10 | # TODO(Ivan Yordanov): try to use typer_cli.main.docs here instead 11 | # https://github.com/bakdata/kpops/issues/297 12 | 13 | if __name__ == "__main__": 14 | typer_args: list[str] = [ 15 | "typer", 16 | str(PATH_KPOPS_MAIN), 17 | "utils", 18 | "docs", 19 | "--name", 20 | "kpops", 21 | "--output", 22 | str(PATH_CLI_COMMANDS_DOC), 23 | ] 24 | subprocess.run(typer_args, check=True, capture_output=True) 25 | 26 | # Replace wrong title in CLI Usage doc 27 | with PATH_CLI_COMMANDS_DOC.open("r") as f: 28 | text = f.readlines() 29 | text[0] = "# CLI Usage\n" 30 | with PATH_CLI_COMMANDS_DOC.open("w") as f: 31 | f.writelines(text) 32 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/config.md: -------------------------------------------------------------------------------- 1 | # Configuration 2 | 3 | KPOps reads its global configuration that is unrelated to a pipeline's [components](./components/overview.md) from [`config.yaml`](#__codelineno-0-1). 4 | 5 | Consider enabling [KPOps' editor integration](../references/editor-integration.md) feature to enjoy the benefits of autocompletion and validation when configuring your pipeline. 6 | 7 | To learn about any of the available settings, take a look at the example below. 8 | 9 | 10 | 11 | ??? example "`config.yaml`" 12 | 13 | ```yaml 14 | --8<-- 15 | ./docs/resources/pipeline-config/config.yaml 16 | --8<-- 17 | ``` 18 | 19 | !!! note "Environment-specific pipeline definitions" 20 | Similarly to [defaults](defaults.md#configuration), it is possible to have an unlimited amount of additional environment-specific pipeline definitions. The naming convention is the same: add a suffix of the form `_{environment}` to the filename. 21 | 22 | 23 | -------------------------------------------------------------------------------- /kpops/api/options.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | from enum import StrEnum 4 | from typing import TYPE_CHECKING 5 | 6 | if TYPE_CHECKING: 7 | from kpops.components.base_components.pipeline_component import PipelineComponent 8 | from kpops.pipeline import ComponentFilterPredicate 9 | 10 | 11 | class FilterType(StrEnum): 12 | INCLUDE = "include" 13 | EXCLUDE = "exclude" 14 | 15 | @staticmethod 16 | def is_in_steps(component: PipelineComponent, component_names: set[str]) -> bool: 17 | return component.name in component_names 18 | 19 | def create_default_step_names_filter_predicate( 20 | self, component_names: set[str] 21 | ) -> ComponentFilterPredicate: 22 | def predicate(component: PipelineComponent) -> bool: 23 | match self, FilterType.is_in_steps(component, component_names): 24 | case (FilterType.INCLUDE, False) | (FilterType.EXCLUDE, True): 25 | return False 26 | case _: 27 | return True 28 | 29 | return predicate 30 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/sections/from_.yaml: -------------------------------------------------------------------------------- 1 | from: # Must not be null 2 | topics: # read from topic 3 | ${pipeline.name}-input-topic: 4 | type: input # Implied when role is NOT specified 5 | ${pipeline.name}-extra-topic: 6 | role: topic-role # Implies `type` to be extra 7 | ${pipeline.name}-input-pattern-topic: 8 | type: pattern # Implied to be an input pattern if `role` is undefined 9 | ${pipeline.name}-extra-pattern-topic: 10 | type: pattern # Implied to be an extra pattern if `role` is defined 11 | role: some-role 12 | components: # read from specific component 13 | account-producer: 14 | type: input # Implied when role is NOT specified 15 | other-producer: 16 | role: some-role # Implies `type` to be extra 17 | component-as-input-pattern: 18 | type: pattern # Implied to be an input pattern if `role` is undefined 19 | component-as-extra-pattern: 20 | type: pattern # Implied to be an extra pattern if `role` is defined 21 | role: some-role 22 | -------------------------------------------------------------------------------- /tests/pipeline/resources/manifest-pipeline/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: my-producer-app 2 | values: 3 | image: "my-registry/my-producer-image" 4 | imageTag: "1.0.0" 5 | 6 | to: 7 | topics: 8 | my-producer-app-output-topic: 9 | type: output 10 | my-labeled-producer-app-topic-output: 11 | label: my-producer-app-output-topic-label 12 | 13 | - type: my-streams-app 14 | values: 15 | image: "my-registry/my-streams-app-image" 16 | imageTag: "1.0.0" 17 | kafka: 18 | applicationId: "my-streams-app-id" 19 | 20 | from: 21 | topics: 22 | my-input-topic: 23 | type: input 24 | my-labeled-input-topic: 25 | label: my-input-topic-label 26 | my-input-pattern: 27 | type: pattern 28 | my-labeled-input-pattern: 29 | type: pattern 30 | label: my-input-topic-labeled-pattern 31 | 32 | to: 33 | topics: 34 | my-output-topic: 35 | type: output 36 | my-error-topic: 37 | type: error 38 | my-labeled-topic-output: 39 | label: my-output-topic-label 40 | -------------------------------------------------------------------------------- /kpops/component_handlers/helm_wrapper/utils.py: -------------------------------------------------------------------------------- 1 | from kpops.component_handlers.kubernetes.utils import trim 2 | from kpops.manifests.kubernetes import K8S_LABEL_MAX_LEN 3 | 4 | RELEASE_NAME_MAX_LEN = 53 5 | 6 | 7 | def create_helm_release_name(name: str, suffix: str = "") -> str: 8 | """Shortens the long Helm release name. 9 | 10 | :param name: The Helm release name to be shortened. 11 | :param suffix: The release suffix to preserve 12 | :return: Trimmed + hashed version of the release name if it exceeds the Helm release character length, otherwise the input name 13 | """ 14 | return trim(RELEASE_NAME_MAX_LEN, name, suffix) 15 | 16 | 17 | def create_helm_name_override(name: str, suffix: str = "") -> str: 18 | """Create Helm chart name override. 19 | 20 | :param name: The Helm name override to be shortened. 21 | :param suffix: The release suffix to preserve 22 | :return: Trimmed + hashed version of the name override if it exceeds the Kubernetes character length, otherwise the input name 23 | """ 24 | return trim(K8S_LABEL_MAX_LEN, name, suffix) 25 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/kafka-sink-connector.md: -------------------------------------------------------------------------------- 1 | # KafkaSinkConnector 2 | 3 | Subclass of [_KafkaConnector_](./kafka-connector.md). 4 | 5 | ### Usage 6 | 7 | Lets other systems pull data from Apache Kafka. 8 | 9 | ### Configuration 10 | 11 | 12 | 13 | ??? example "`pipeline.yaml`" 14 | 15 | ```yaml 16 | --8<-- 17 | ./docs/resources/pipeline-components/kafka-sink-connector.yaml 18 | --8<-- 19 | ``` 20 | 21 | 22 | 23 | ### Operations 24 | 25 | #### deploy 26 | 27 | - Add the sink connector to the Kafka Connect cluster 28 | - Create the output topics if provided (optional) 29 | - Register schemas in the Schema Registry if provided (optional) 30 | 31 | #### destroy 32 | 33 | The associated sink connector is removed from the Kafka Connect cluster. 34 | 35 | #### reset 36 | 37 | Reset the consumer group offsets using 38 | [bakdata's sink resetter](https://github.com/bakdata/kafka-connect-resetter/#sink-resetter){target=_blank}. 39 | 40 | #### clean 41 | 42 | - Delete associated consumer group 43 | - Delete configured error topics 44 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/producer-app.md: -------------------------------------------------------------------------------- 1 | # ProducerApp 2 | 3 | Subclass of [_StreamsBootstrap_](streams-bootstrap.md). 4 | 5 | ### Usage 6 | 7 | Configures a [streams-bootstrap](https://github.com/bakdata/streams-bootstrap){target=_blank} [Kafka producer app](https://github.com/bakdata/streams-bootstrap#kafka-producer){target=_blank} 8 | 9 | ### Configuration 10 | 11 | 12 | 13 | ??? example "`pipeline.yaml`" 14 | 15 | ```yaml 16 | --8<-- 17 | ./docs/resources/pipeline-components/producer-app.yaml 18 | --8<-- 19 | ``` 20 | 21 | 22 | 23 | ### Operations 24 | 25 | #### deploy 26 | 27 | In addition to [KubernetesApp's `deploy`](kubernetes-app.md#deploy): 28 | 29 | - Create topics if provided (optional) 30 | - Submit Avro schemas to the registry if provided (optional) 31 | 32 | #### destroy 33 | 34 | Uninstall Helm release. 35 | 36 | #### reset 37 | 38 | Do nothing, producers are stateless. 39 | 40 | #### clean 41 | 42 | - Delete the output topics of the Kafka producer 43 | - Delete all associated schemas in the Schema Registry 44 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2022 bakdata 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /docs/docs/resources/variables/variable_substitution.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | values: 3 | labels: 4 | app_type: "${component.type}" 5 | app_name: "${component.name}" 6 | app_schedule: "${component.values.schedule}" 7 | commandLine: 8 | FAKE_ARG: "fake-arg-value" 9 | schedule: "30 3/8 * * *" 10 | - type: converter 11 | values: 12 | commandLine: 13 | CONVERT_XML: true 14 | resources: 15 | limits: 16 | memory: 2G 17 | requests: 18 | memory: 2G 19 | - type: filter 20 | name: "filter-app" 21 | values: 22 | labels: 23 | app_type: "${component.type}" 24 | app_name: "${component.name}" 25 | app_resources_requests_memory: "${component.values.resources.requests.memory}" 26 | ${component.type}: "${component.values.labels.app_name}-${component.values.labels.app_type}" 27 | test_placeholder_in_placeholder: "${component.values.labels.${component.type}}" 28 | commandLine: 29 | TYPE: "nothing" 30 | resources: 31 | requests: 32 | memory: 3G 33 | replicaCount: 4 34 | autoscaling: 35 | minReplicas: 4 36 | maxReplicas: 4 37 | -------------------------------------------------------------------------------- /docs/docs/resources/variables/cli_env_vars.env: -------------------------------------------------------------------------------- 1 | # CLI Environment variables 2 | # 3 | # The default setup is shown. These variables take precedence over the 4 | # commands' flags. If a variable is set, the corresponding flag does 5 | # not have to be specified in commands. Variables marked as required 6 | # can instead be set as flags. 7 | # 8 | # Path to the dir containing config.yaml files 9 | KPOPS_CONFIG_PATH=. 10 | # Path to dotenv file. Multiple files can be provided. The files will 11 | # be loaded in order, with each file overriding the previous one. 12 | KPOPS_DOTENV_PATH # No default value, not required 13 | # The environment you want to generate and deploy the pipeline to. 14 | # Suffix your environment files with this value (e.g. 15 | # defaults_development.yaml for environment=development). 16 | KPOPS_ENVIRONMENT # No default value, not required 17 | # How KPOps should operate. 18 | KPOPS_OPERATION_MODE=managed 19 | # Paths to dir containing 'pipeline.yaml' or files named 20 | # 'pipeline.yaml'. 21 | KPOPS_PIPELINE_PATHS # No default value, required 22 | # Comma separated list of steps to apply the command on 23 | KPOPS_PIPELINE_STEPS # No default value, not required 24 | -------------------------------------------------------------------------------- /tests/pipeline/resources/component-type-substitution/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: scheduled-producer 2 | values: 3 | labels: 4 | app_type: "${component.type}" 5 | app_name: "${component.name}" 6 | app_schedule: "${component.values.schedule}" 7 | commandLine: 8 | FAKE_ARG: "fake-arg-value" 9 | schedule: "30 3/8 * * *" 10 | - type: converter 11 | values: 12 | commandLine: 13 | CONVERT_XML: true 14 | resources: 15 | limits: 16 | memory: 2G 17 | requests: 18 | memory: 2G 19 | - type: filter 20 | name: "filter-app" 21 | values: 22 | labels: 23 | app_type: "${component.type}" 24 | app_name: "${component.name}" 25 | app_resources_requests_memory: "${component.values.resources.requests.memory}" 26 | ${component.type}: "${component.values.labels.app_name}-${component.values.labels.app_type}" 27 | test_placeholder_in_placeholder: "${component.values.labels.${component.type}}" 28 | commandLine: 29 | TYPE: "nothing" 30 | resources: 31 | requests: 32 | memory: 3G 33 | replicaCount: 4 34 | autoscaling: 35 | minReplicas: 4 36 | maxReplicas: 4 37 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/streams-app.md: -------------------------------------------------------------------------------- 1 | # StreamsApp 2 | 3 | Subclass of and [_StreamsBootstrap_](streams-bootstrap.md). 4 | 5 | ### Usage 6 | 7 | Configures a 8 | [streams-bootstrap](https://github.com/bakdata/streams-bootstrap){target=_blank} 9 | [Kafka Streams app](https://github.com/bakdata/streams-bootstrap#kafka-streams){target=_blank} 10 | 11 | ### Configuration 12 | 13 | 14 | 15 | ??? example "`pipeline.yaml`" 16 | 17 | ```yaml 18 | --8<-- 19 | ./docs/resources/pipeline-components/streams-app.yaml 20 | --8<-- 21 | ``` 22 | 23 | 24 | 25 | ### Operations 26 | 27 | #### deploy 28 | 29 | In addition to [KubernetesApp's `deploy`](kubernetes-app.md#deploy): 30 | 31 | - Create topics if provided (optional) 32 | - Submit Avro schemas to the registry if provided (optional) 33 | 34 | #### destroy 35 | 36 | Uninstall Helm release. 37 | 38 | #### reset 39 | 40 | - Delete the consumer group offsets 41 | - Delete Kafka Streams state 42 | 43 | #### clean 44 | 45 | Similar to [`reset`](#reset) with to additional steps: 46 | 47 | - Delete the app's output topics 48 | - Delete all associated schemas in the Schema Registry 49 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/components/kafka-source-connector.md: -------------------------------------------------------------------------------- 1 | # KafkaSourceConnector 2 | 3 | Subclass of [_KafkaConnector_](./kafka-connector.md). 4 | 5 | ### Usage 6 | 7 | Manages source connectors in your Kafka Connect cluster. 8 | 9 | ### Configuration 10 | 11 | 12 | 13 | ??? example "`pipeline.yaml`" 14 | 15 | ```yaml 16 | --8<-- 17 | ./docs/resources/pipeline-components/kafka-source-connector.yaml 18 | --8<-- 19 | ``` 20 | 21 | 22 | 23 | ### Operations 24 | 25 | #### deploy 26 | 27 | - Add the source connector to the Kafka Connect cluster 28 | - Create the output topics if provided (optional) 29 | - Register schemas in the Schema registry if provided (optional) 30 | 31 | #### destroy 32 | 33 | Remove the source connector from the Kafka Connect cluster. 34 | 35 | #### reset 36 | 37 | Delete state associated with the connector using 38 | [bakdata's source resetter](https://github.com/bakdata/kafka-connect-resetter/#source-resetter){target=_blank}. 39 | 40 | #### clean 41 | 42 | - Delete all associated output topics 43 | - Delete all associated schemas in the Schema Registry 44 | - Delete state associated with the connector 45 | -------------------------------------------------------------------------------- /kpops/component_handlers/kubernetes/utils.py: -------------------------------------------------------------------------------- 1 | import hashlib 2 | import logging 3 | 4 | log = logging.getLogger("K8sUtils") 5 | 6 | 7 | def trim(max_len: int, name: str, suffix: str) -> str: 8 | """Shortens long K8s identifiers. 9 | 10 | Kubernetes has a character limit for certain identifiers. 11 | If the name exceeds the character limit: 12 | 1. trim the string and fetch the first ``max_len - len(suffix)`` characters. 13 | 2. replace the last 6 characters with the SHA-1 encoded string (with "-") to avoid collision 14 | 3. append the suffix if given 15 | 16 | :param name: The name to be shortened. 17 | :param suffix: The release suffix to preserve 18 | :return: Trimmed + hashed version of the name if it exceeds the character length otherwise the actual name 19 | """ 20 | if len(name) > max_len: 21 | exact_name = name[: max_len - len(suffix)] 22 | hash_name = hashlib.sha1(name.encode()).hexdigest() 23 | new_name = exact_name[:-6] + "-" + hash_name[:5] + suffix 24 | log.critical( 25 | f"Kubernetes identifier '{name}' exceeds character limit. Truncating and hashing to {max_len} characters: \n {name} --> {new_name}" 26 | ) 27 | return new_name 28 | return name 29 | -------------------------------------------------------------------------------- /tests/pipeline/resources/pipeline-with-env-defaults/defaults.yaml: -------------------------------------------------------------------------------- 1 | kubernetes-app: 2 | name: ${component.type} 3 | namespace: example-namespace 4 | 5 | stream-bootstrap: 6 | values: 7 | kafka: 8 | bootstrapServers: "${config.kafka_brokers}" 9 | schemaRegistryUrl: "${config.schema_registry.url}" 10 | 11 | producer-app: {} # inherits from streams-bootstrap 12 | 13 | streams-app: # inherits from streams-bootstrap 14 | version: "3.6.1" 15 | values: 16 | kafka: 17 | config: 18 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 19 | to: 20 | topics: 21 | ${error_topic_name}: 22 | type: error 23 | value_schema: com.bakdata.kafka.DeadLetter 24 | partitions_count: 1 25 | configs: 26 | cleanup.policy: compact,delete 27 | 28 | kafka-connector: 29 | name: sink-connector 30 | config: 31 | batch.size: "2000" 32 | behavior.on.malformed.documents: "warn" 33 | behavior.on.null.values: "delete" 34 | connection.compression: "true" 35 | connector.class: "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector" 36 | key.ignore: "false" 37 | linger.ms: "5000" 38 | max.buffered.records: "20000" 39 | read.timeout.ms: "120000" 40 | tasks.max: "1" 41 | -------------------------------------------------------------------------------- /docs/docs/developer/contributing.md: -------------------------------------------------------------------------------- 1 | # How to contribute 2 | 3 | **Welcome!** We are glad to have you visit our contributing guide! 4 | 5 | If you find any bugs or have suggestions for improvements, please open an [issue](https://github.com/bakdata/kpops/issues/new) and optionally a [pull request (PR)](https://github.com/bakdata/kpops/compare). In the case of a PR, we would appreciate it if you preface it with an issue outlining your goal and means of achieving it. 6 | 7 | ### git 8 | 9 | We are using [git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) to import the [KPOps examples](https://github.com/bakdata/kpops-examples) repository. You need to fetch the repository locally on your machine. To do so use this command: 10 | 11 | ```bash 12 | git submodule init 13 | git submodule update --recursive 14 | ``` 15 | 16 | This will fetch the resources under the `examples` folder. 17 | 18 | ## Style 19 | 20 | We advise that you stick to our Git hooks for code linting, formatting, and auto-generation of documentation. After you install them using `lefthook install` they're triggered automatically during `git` operations, such as commit or checkout. Additionally, you can manually invoke them with `lefthook run pre-commit --all-files`. Please also install the [`dprint`](https://dprint.dev/) formatter. 21 | -------------------------------------------------------------------------------- /hooks/gen_schema.py: -------------------------------------------------------------------------------- 1 | """Generates the stock KPOps editor integration schemas.""" 2 | 3 | from contextlib import redirect_stdout 4 | from io import StringIO 5 | from pathlib import Path 6 | 7 | from hooks import ROOT 8 | from kpops.const.file_type import KpopsFileType 9 | from kpops.utils.gen_schema import ( 10 | gen_config_schema, 11 | gen_defaults_schema, 12 | gen_pipeline_schema, 13 | ) 14 | 15 | PATH_TO_SCHEMA = ROOT / "docs/docs/schema" 16 | 17 | 18 | def gen_schema(scope: KpopsFileType): 19 | """Generate the specified schema and save it to a file. 20 | 21 | The file is located in docs/docs/schema and is named ``.json`` 22 | 23 | :param scope: Scope of the generated schema 24 | """ 25 | with redirect_stdout(StringIO()) as f: 26 | match scope: 27 | case KpopsFileType.PIPELINE: 28 | gen_pipeline_schema() 29 | case KpopsFileType.DEFAULTS: 30 | gen_defaults_schema() 31 | case KpopsFileType.CONFIG: 32 | gen_config_schema() 33 | Path(PATH_TO_SCHEMA / f"{scope.value}.json").write_text(f.getvalue()) 34 | 35 | 36 | if __name__ == "__main__": 37 | gen_schema(KpopsFileType.PIPELINE) 38 | gen_schema(KpopsFileType.DEFAULTS) 39 | gen_schema(KpopsFileType.CONFIG) 40 | -------------------------------------------------------------------------------- /kpops/components/base_components/cleaner.py: -------------------------------------------------------------------------------- 1 | from abc import ABC 2 | 3 | from typing_extensions import override 4 | 5 | from kpops.component_handlers.helm_wrapper.model import HelmFlags 6 | from kpops.component_handlers.helm_wrapper.utils import ( 7 | create_helm_name_override, 8 | create_helm_release_name, 9 | ) 10 | from kpops.components.base_components.helm_app import HelmApp 11 | from kpops.config import get_config 12 | 13 | 14 | class Cleaner(HelmApp, ABC): 15 | """Generic Helm app for cleaning or resetting.""" 16 | 17 | suffix: str = "-clean" 18 | 19 | @property 20 | @override 21 | def full_name(self) -> str: 22 | return super().full_name + self.suffix 23 | 24 | @property 25 | @override 26 | def helm_release_name(self) -> str: 27 | return create_helm_release_name(self.full_name, self.suffix) 28 | 29 | @property 30 | @override 31 | def helm_name_override(self) -> str: 32 | return create_helm_name_override(self.full_name, self.suffix) 33 | 34 | @property 35 | @override 36 | def helm_flags(self) -> HelmFlags: 37 | return HelmFlags( 38 | create_namespace=get_config().create_namespace, 39 | version=self.version, 40 | wait=True, 41 | wait_for_jobs=True, 42 | ) 43 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/dependencies/pipeline_component_dependencies.yaml: -------------------------------------------------------------------------------- 1 | helm-app.yaml: 2 | - prefix.yaml 3 | - from_.yaml 4 | - to.yaml 5 | - namespace.yaml 6 | - values-helm-app.yaml 7 | - repo_config-helm-app.yaml 8 | - version.yaml 9 | kafka-app.yaml: 10 | - prefix.yaml 11 | - from_.yaml 12 | - to.yaml 13 | kafka-connector.yaml: 14 | - prefix.yaml 15 | - from_.yaml 16 | - to.yaml 17 | - config-kafka-connector.yaml 18 | - resetter_values.yaml 19 | kafka-sink-connector.yaml: 20 | - prefix.yaml 21 | - from_.yaml 22 | - to.yaml 23 | - config-kafka-connector.yaml 24 | - resetter_values.yaml 25 | kafka-source-connector.yaml: 26 | - prefix.yaml 27 | - from_-kafka-source-connector.yaml 28 | - to.yaml 29 | - config-kafka-connector.yaml 30 | - resetter_values.yaml 31 | - offset_topic-kafka-source-connector.yaml 32 | kubernetes-app.yaml: 33 | - prefix.yaml 34 | - from_.yaml 35 | - to.yaml 36 | - namespace.yaml 37 | - values-kubernetes-app.yaml 38 | producer-app.yaml: 39 | - prefix.yaml 40 | - from_-producer-app.yaml 41 | - to.yaml 42 | - namespace.yaml 43 | - values-producer-app.yaml 44 | - repo_config-helm-app.yaml 45 | - version-kafka-app.yaml 46 | streams-app.yaml: 47 | - prefix.yaml 48 | - from_.yaml 49 | - to.yaml 50 | - namespace.yaml 51 | - values-streams-app.yaml 52 | - repo_config-helm-app.yaml 53 | - version-kafka-app.yaml 54 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_generate/test_no_user_defined_components/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: streams-app 2 | name: streams-app 3 | prefix: resources-no-user-defined-components- 4 | from: 5 | topics: 6 | example-topic: 7 | type: input 8 | components: {} 9 | to: 10 | topics: 11 | example-output: 12 | type: output 13 | configs: {} 14 | resources-no-user-defined-components-streams-app-error: 15 | type: error 16 | value_schema: com.bakdata.kafka.DeadLetter 17 | partitions_count: 1 18 | configs: 19 | cleanup.policy: compact,delete 20 | models: {} 21 | namespace: example-namespace 22 | values: 23 | image: fake-image 24 | kafka: 25 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 26 | schemaRegistryUrl: http://localhost:8081/ 27 | outputTopic: example-output 28 | inputTopics: 29 | - example-topic 30 | errorTopic: resources-no-user-defined-components-streams-app-error 31 | config: 32 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 33 | statefulSet: false 34 | version: 3.6.1 35 | helm_release_name: resources-no-user-defined-components-streams-app 36 | helm_name_override: resources-no-user-defined-components-streams-app 37 | 38 | -------------------------------------------------------------------------------- /tests/component_handlers/kubernetes/model.py: -------------------------------------------------------------------------------- 1 | from textwrap import dedent 2 | 3 | import pytest 4 | 5 | from kpops.manifests.kubernetes import KubernetesManifest 6 | 7 | 8 | class TestKubernetesManifest: 9 | @pytest.mark.parametrize( 10 | ("helm_template", "expected_manifest"), 11 | [ 12 | pytest.param( 13 | dedent( 14 | """ 15 | --- 16 | # Source: chart/templates/test2.yaml 17 | apiVersion: v1 18 | kind: ServiceAccount 19 | metadata: 20 | labels: 21 | foo: bar 22 | """ 23 | ), 24 | [ 25 | KubernetesManifest.model_validate( 26 | { 27 | "apiVersion": "v1", 28 | "kind": "ServiceAccount", 29 | "metadata": {"labels": {"foo": "bar"}}, 30 | } 31 | ) 32 | ], 33 | ) 34 | ], 35 | ) 36 | def test_from_yaml(self, helm_template: str, expected_manifest: KubernetesManifest): 37 | manifests = KubernetesManifest.from_yaml(helm_template) 38 | assert list(manifests) == expected_manifest 39 | -------------------------------------------------------------------------------- /kpops/component_handlers/helm_wrapper/dry_run_handler.py: -------------------------------------------------------------------------------- 1 | from logging import Logger 2 | from typing import final 3 | 4 | from kpops.component_handlers.helm_wrapper.helm import Helm 5 | from kpops.component_handlers.helm_wrapper.helm_diff import HelmDiff 6 | 7 | 8 | @final 9 | class DryRunHandler: 10 | def __init__(self, helm: Helm, helm_diff: HelmDiff, namespace: str) -> None: 11 | self._helm = helm 12 | self._helm_diff = helm_diff 13 | self.namespace = namespace 14 | 15 | def print_helm_diff(self, stdout: str, helm_release_name: str, log: Logger) -> None: 16 | """Print the diff of the last and current release of this component. 17 | 18 | :param stdout: The output of a Helm command that installs or upgrades the release 19 | :param helm_release_name: The Helm release name 20 | :param log: The Logger object of the component class 21 | """ 22 | current_release = list( 23 | self._helm.get_manifest(helm_release_name, self.namespace) 24 | ) 25 | if current_release: 26 | log.info(f"Helm release {helm_release_name} already exists") 27 | else: 28 | log.info(f"Helm release {helm_release_name} does not exist") 29 | new_release = Helm.load_manifest(stdout) 30 | self._helm_diff.log_helm_diff(log, current_release, new_release) 31 | -------------------------------------------------------------------------------- /tests/component_handlers/helm_wrapper/test_utils.py: -------------------------------------------------------------------------------- 1 | from kpops.component_handlers.helm_wrapper.utils import ( 2 | create_helm_release_name, 3 | ) 4 | 5 | 6 | def test_helm_release_name_for_long_names(): 7 | long_release_name = "example-component-name-too-long-fake-fakefakefakefakefake" 8 | 9 | actual_release_name = create_helm_release_name(long_release_name) 10 | 11 | expected_helm_release_name = "example-component-name-too-long-fake-fakefakefa-0a7fc" 12 | assert expected_helm_release_name == actual_release_name 13 | assert len(expected_helm_release_name) == 53 14 | 15 | 16 | def test_helm_release_name_for_install_and_clean_must_be_different(): 17 | long_release_name = "example-component-name-too-long-fake-fakefakefakefakefake" 18 | 19 | helm_clean_release_name = create_helm_release_name(long_release_name, "-clean") 20 | expected_helm_release_name = ( 21 | "example-component-name-too-long-fake-fakefakef-0a7fc-clean" 22 | ) 23 | 24 | assert expected_helm_release_name != helm_clean_release_name 25 | 26 | 27 | def test_helm_release_name_for_short_names(): 28 | short_release_name = "example-component-name" 29 | 30 | actual_helm_release_name = create_helm_release_name(short_release_name) 31 | 32 | assert actual_helm_release_name == short_release_name 33 | assert len(actual_helm_release_name) <= 53 34 | -------------------------------------------------------------------------------- /kpops/utils/environment.py: -------------------------------------------------------------------------------- 1 | import os 2 | from collections import UserDict 3 | from collections.abc import ItemsView, KeysView, MutableMapping, ValuesView 4 | 5 | PIPELINE_PATH = "pipeline.path" 6 | 7 | 8 | class Environment(UserDict[str, str]): 9 | """Internal environment wrapping OS environment.""" 10 | 11 | def __init__( 12 | self, mapping: MutableMapping[str, str] | None = None, /, **kwargs: str 13 | ) -> None: 14 | self._global = os.environ 15 | if mapping is None: 16 | mapping = {} 17 | if kwargs: 18 | mapping.update(**kwargs) 19 | super().__init__(mapping) 20 | 21 | def __getitem__(self, key: str) -> str: 22 | try: 23 | return self.data[key] 24 | except KeyError: 25 | return self._global[key] 26 | 27 | def __contains__(self, key: object) -> bool: 28 | return super().__contains__(key) or self._global.__contains__(key) 29 | 30 | @property 31 | def _dict(self) -> dict[str, str]: 32 | return {**self._global, **self.data} 33 | 34 | def keys(self) -> KeysView[str]: 35 | return KeysView(self._dict) 36 | 37 | def values(self) -> ValuesView[str]: 38 | return ValuesView(self._dict) 39 | 40 | def items(self) -> ItemsView[str, str]: 41 | return ItemsView(self._dict) 42 | 43 | 44 | ENV = Environment() 45 | -------------------------------------------------------------------------------- /tests/pipeline/resources/read-from-component/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: producer1 3 | to: 4 | topics: 5 | ${output_topic_name}: 6 | type: output 7 | 8 | - type: producer-app 9 | name: producer2 10 | prefix: "" 11 | to: 12 | topics: 13 | ${output_topic_name}: 14 | type: output 15 | 16 | - type: should-inflate 17 | name: inflate-step 18 | 19 | - type: should-inflate 20 | name: inflate-step-without-prefix 21 | prefix: "" 22 | 23 | - type: streams-app 24 | name: consumer1 25 | from: 26 | components: 27 | producer1: 28 | type: input 29 | to: 30 | topics: 31 | ${output_topic_name}: 32 | type: output 33 | 34 | - type: streams-app 35 | name: consumer2 36 | from: 37 | components: 38 | producer1: 39 | type: input 40 | consumer1: 41 | type: input 42 | 43 | - type: streams-app 44 | name: consumer3 45 | from: 46 | topics: 47 | ${pipeline.name}-producer1: 48 | type: input 49 | components: 50 | producer2: 51 | type: input 52 | 53 | - type: streams-app 54 | name: consumer4 55 | from: 56 | components: 57 | inflate-step: 58 | type: input 59 | 60 | - type: streams-app 61 | name: consumer5 62 | from: 63 | components: 64 | inflate-step-without-prefix: 65 | type: input 66 | -------------------------------------------------------------------------------- /docs/docs/user/references/editor-integration.md: -------------------------------------------------------------------------------- 1 | # Editor integration 2 | 3 | ## Native 4 | 5 | We are working towards first-class editor support by providing plugins that work out of the box. 6 | 7 | - Neovim: [kpops.nvim](https://github.com/disrupted/kpops.nvim) 8 | - Visual Studio Code: _planned_ 9 | 10 | ## Manual (for unsupported editors with LSP) 11 | 12 | 1. Install the [yaml-language-server](https://github.com/redhat-developer/yaml-language-server#clients){target=_blank} in your editor of choice. (requires LSP support) 13 | 2. Configure the extension with the settings below. 14 | 15 | 16 | 17 | ??? note "`settings.json`" 18 | 19 | ```json 20 | --8<-- 21 | ./docs/resources/editor_integration/settings.json 22 | --8<-- 23 | ``` 24 | 25 | !!! tip "Advanced usage" 26 | It is possible to generate schemas with the [`kpops schema`](./cli-commands.md#kpops-schema) command. Useful for including custom components or when using a pre-release version of KPOps. 27 | 28 | 29 | 30 | --- 31 | 32 | ## Concepts 33 | 34 | KPOps provides JSON schemas that enable autocompletion and validation for all YAML files that the user must work with. 35 | 36 | ### Supported files 37 | 38 | - [`pipeline.yaml`](../../resources/pipeline-components/pipeline.md) 39 | - [`defaults.yaml`](../core-concepts/defaults.md) 40 | - [`config.yaml`](../core-concepts/config.md) 41 | -------------------------------------------------------------------------------- /kpops/const/file_type.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | from enum import StrEnum 4 | 5 | FILE_EXTENSION = ".yaml" 6 | 7 | 8 | class KpopsFileType(StrEnum): 9 | """Enum representing different types of KPOps file naming conventions. 10 | 11 | Attributes: 12 | PIPELINE (str): Represents a pipeline YAML file type. 13 | DEFAULTS (str): Represents a defaults YAML file type. 14 | CONFIG (str): Represents a config YAML file type. 15 | """ 16 | 17 | PIPELINE = "pipeline" 18 | DEFAULTS = "defaults" 19 | CONFIG = "config" 20 | 21 | def as_yaml_file(self, prefix: str = "", suffix: str = "") -> str: 22 | """Generate a YAML file name based on the enum value with optional prefix and suffix. 23 | 24 | Args: 25 | prefix (str): An optional prefix for the file name. Default is an empty string. 26 | suffix (str): An optional suffix for the file name. Default is an empty string. 27 | 28 | Returns: 29 | str: The constructed file name in the format ''. 30 | 31 | Example: 32 | >>> KpopsFileType.PIPELINE.as_yaml_file(prefix="pre_", suffix="_suf") 33 | 'pre_pipeline_suf.yaml' 34 | """ 35 | return prefix + self.value + suffix + FILE_EXTENSION 36 | 37 | 38 | PIPELINE_YAML = KpopsFileType.PIPELINE.as_yaml_file() 39 | DEFAULTS_YAML = KpopsFileType.DEFAULTS.as_yaml_file() 40 | CONFIG_YAML = KpopsFileType.CONFIG.as_yaml_file() 41 | -------------------------------------------------------------------------------- /kpops/component_handlers/kubernetes/pvc_handler.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from collections.abc import AsyncIterable 3 | from typing import final 4 | 5 | from lightkube.core.async_client import AsyncClient 6 | from lightkube.resources.core_v1 import PersistentVolumeClaim 7 | 8 | log = logging.getLogger("PVC_handler") 9 | 10 | 11 | @final 12 | class PVCHandler: 13 | def __init__(self, app_name: str, namespace: str) -> None: 14 | self.app_name = app_name 15 | self.namespace = namespace 16 | self._client = AsyncClient(namespace=namespace) 17 | 18 | async def list_pvcs(self) -> AsyncIterable[PersistentVolumeClaim]: 19 | return self._client.list( 20 | PersistentVolumeClaim, labels={"app.kubernetes.io/name": self.app_name} 21 | ) 22 | 23 | async def delete_pvcs(self, dry_run: bool) -> None: 24 | pvc_names: list[str] = [ 25 | pvc.metadata.name 26 | async for pvc in await self.list_pvcs() 27 | if pvc.metadata and pvc.metadata.name 28 | ] 29 | if not pvc_names: 30 | log.warning( 31 | f"No PVCs found for app '{self.app_name}', in namespace '{self.namespace}'" 32 | ) 33 | return 34 | log.debug( 35 | f"Deleting in namespace '{self.namespace}' StatefulSet '{self.app_name}' PVCs {pvc_names}" 36 | ) 37 | if dry_run: 38 | return 39 | for pvc_name in pvc_names: 40 | await self._client.delete(PersistentVolumeClaim, pvc_name) 41 | -------------------------------------------------------------------------------- /tests/pipeline/resources/streams-bootstrap/defaults.yaml: -------------------------------------------------------------------------------- 1 | cleaner: 2 | diff_config: 3 | ignore: 4 | - ["cleaner"] 5 | 6 | streams-bootstrap: 7 | version: "3.6.1" 8 | values: 9 | imagePullPolicy: IfNotPresent 10 | files: 11 | log4j2.xml: 12 | mountPath: app/resources 13 | content: | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | kafka: 28 | bootstrapServers: ${config.kafka_brokers} 29 | schemaRegistryUrl: ${config.schema_registry.url} 30 | 31 | producer-app: {} # inherits from streams-bootstrap 32 | 33 | streams-app: # inherits from streams-bootstrap 34 | diff_config: 35 | ignore: 36 | - ["foo", "bar"] 37 | values: 38 | kafka: 39 | config: 40 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 41 | to: 42 | topics: 43 | ${error_topic_name}: 44 | type: error 45 | value_schema: com.bakdata.kafka.DeadLetter 46 | partitions_count: 1 47 | configs: 48 | cleanup.policy: compact,delete 49 | -------------------------------------------------------------------------------- /kpops/component_handlers/__init__.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | from typing import TYPE_CHECKING, Any, final 4 | 5 | if TYPE_CHECKING: 6 | from kpops.component_handlers.kafka_connect.kafka_connect_handler import ( 7 | KafkaConnectHandler, 8 | ) 9 | from kpops.component_handlers.schema_handler.schema_handler import SchemaHandler 10 | from kpops.component_handlers.topic.handler import TopicHandler 11 | 12 | 13 | @final 14 | class ComponentHandlers: 15 | _instance: ComponentHandlers | None = None 16 | 17 | def __new__( 18 | cls, 19 | schema_handler: SchemaHandler | None, 20 | connector_handler: KafkaConnectHandler, 21 | topic_handler: TopicHandler, 22 | *args: Any, 23 | **kwargs: Any, 24 | ) -> ComponentHandlers: 25 | if not cls._instance: 26 | cls._instance = super().__new__(cls, *args, **kwargs) 27 | return cls._instance 28 | 29 | def __init__( 30 | self, 31 | schema_handler: SchemaHandler | None, 32 | connector_handler: KafkaConnectHandler, 33 | topic_handler: TopicHandler, 34 | ) -> None: 35 | self.schema_handler = schema_handler 36 | self.connector_handler = connector_handler 37 | self.topic_handler = topic_handler 38 | 39 | 40 | def get_handlers() -> ComponentHandlers: 41 | if not ComponentHandlers._instance: 42 | msg = f"{ComponentHandlers.__name__} has not been initialized" 43 | raise RuntimeError(msg) 44 | return ComponentHandlers._instance 45 | -------------------------------------------------------------------------------- /kpops/cli/utils.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | from collections.abc import Iterable, Iterator 4 | from glob import glob 5 | from pathlib import Path 6 | 7 | from kpops.const.file_type import PIPELINE_YAML 8 | 9 | 10 | def collect_pipeline_paths(pipeline_paths: Iterable[Path]) -> Iterator[Path]: 11 | """Generate paths to pipeline files. 12 | 13 | :param pipeline_paths: The list of paths to the pipeline files or directories. 14 | 15 | :yields: Path: Paths to pipeline files. If `pipeline_path` file yields the given path. 16 | For a directory it yields all the pipeline.yaml paths. 17 | 18 | :raises: ValueError: If `pipeline_path` is neither a file nor a directory. 19 | """ 20 | for pipeline_path in pipeline_paths: 21 | if pipeline_path.is_file(): 22 | yield pipeline_path 23 | elif pipeline_path.is_dir(): 24 | # TODO: In python 3.13 symbolic links become supported by Path.glob 25 | # docs.python.org/3.13#pathlib.Path.glob, probably it make sense to use it after ugprading, 26 | # likely the code will look like: 27 | # yield from sorted(pipeline_path.glob(f"**/{PIPELINE_YAML}", recurse_symlinks=True)) 28 | yield from sorted( 29 | Path(p).resolve() 30 | for p in glob(f"{pipeline_path}/**/{PIPELINE_YAML}", recursive=True) # noqa: PTH207 31 | ) 32 | else: 33 | msg = f"The entered pipeline path '{pipeline_path}' should be a directory or file." 34 | raise ValueError(msg) 35 | -------------------------------------------------------------------------------- /tests/pipeline/test_components_without_schema_handler/components.py: -------------------------------------------------------------------------------- 1 | from typing_extensions import override 2 | 3 | from kpops.component_handlers.kafka_connect.model import KafkaConnectorConfig 4 | from kpops.components.base_components.kafka_connector import KafkaSinkConnector 5 | from kpops.components.base_components.pipeline_component import PipelineComponent 6 | from kpops.components.common.topic import OutputTopicTypes 7 | from kpops.components.streams_bootstrap.producer.producer_app import ProducerApp 8 | from kpops.components.streams_bootstrap.streams.streams_app import StreamsApp 9 | 10 | 11 | class ScheduledProducer(ProducerApp): ... 12 | 13 | 14 | class Converter(StreamsApp): ... 15 | 16 | 17 | class ShouldInflate(StreamsApp): 18 | @override 19 | def inflate(self) -> list[PipelineComponent]: 20 | inflate_steps = super().inflate() 21 | if self.to: 22 | for topic_name, topic_config in self.to.topics.items(): 23 | if topic_config.type == OutputTopicTypes.OUTPUT: 24 | kafka_connector = KafkaSinkConnector( 25 | name="sink-connector", 26 | config=KafkaConnectorConfig.model_validate( 27 | { 28 | "topics": topic_name, 29 | "transforms.changeTopic.replacement": f"{topic_name}-index-v1", 30 | } 31 | ), 32 | ) 33 | inflate_steps.append(kafka_connector) 34 | 35 | return inflate_steps 36 | -------------------------------------------------------------------------------- /tests/component_handlers/resources/kafka_rest_proxy_responses/cluster-info.json: -------------------------------------------------------------------------------- 1 | { 2 | "kind": "KafkaClusterList", 3 | "metadata": { 4 | "self": "https://fake-cluster/kafka/v3/clusters", 5 | "next": null 6 | }, 7 | "data": [ 8 | { 9 | "kind": "KafkaCluster", 10 | "metadata": { 11 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1", 12 | "resource_name": "crn:///kafka=cluster-1" 13 | }, 14 | "cluster_id": "cluster-1", 15 | "controller": { 16 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/brokers/1" 17 | }, 18 | "acls": { 19 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/acls" 20 | }, 21 | "brokers": { 22 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/brokers" 23 | }, 24 | "broker_configs": { 25 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/broker-configs" 26 | }, 27 | "consumer_groups": { 28 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/consumer-groups" 29 | }, 30 | "topics": { 31 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics" 32 | }, 33 | "partition_reassignments": { 34 | "related": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/-/partitions/-/reassignment" 35 | } 36 | } 37 | ] 38 | } 39 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_manifest/test_manifest_destroy_argo_mode/manifest.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kafka.strimzi.io/v1beta2 3 | kind: KafkaTopic 4 | metadata: 5 | labels: 6 | strimzi.io/cluster: my-cluster 7 | name: my-producer-app-output-topic 8 | spec: 9 | config: {} 10 | partitions: 1 11 | replicas: 1 12 | 13 | --- 14 | apiVersion: kafka.strimzi.io/v1beta2 15 | kind: KafkaTopic 16 | metadata: 17 | labels: 18 | strimzi.io/cluster: my-cluster 19 | name: my-labeled-producer-app-topic-output 20 | spec: 21 | config: {} 22 | partitions: 1 23 | replicas: 1 24 | 25 | --- 26 | apiVersion: kafka.strimzi.io/v1beta2 27 | kind: KafkaTopic 28 | metadata: 29 | labels: 30 | strimzi.io/cluster: my-cluster 31 | name: my-output-topic 32 | spec: 33 | config: {} 34 | partitions: 1 35 | replicas: 1 36 | 37 | --- 38 | apiVersion: kafka.strimzi.io/v1beta2 39 | kind: KafkaTopic 40 | metadata: 41 | labels: 42 | strimzi.io/cluster: my-cluster 43 | name: my-error-topic 44 | spec: 45 | config: {} 46 | partitions: 1 47 | replicas: 1 48 | 49 | --- 50 | apiVersion: kafka.strimzi.io/v1beta2 51 | kind: KafkaTopic 52 | metadata: 53 | labels: 54 | strimzi.io/cluster: my-cluster 55 | name: my-labeled-topic-output 56 | spec: 57 | config: {} 58 | partitions: 1 59 | replicas: 1 60 | 61 | --- 62 | apiVersion: kafka.strimzi.io/v1beta2 63 | kind: KafkaTopic 64 | metadata: 65 | labels: 66 | strimzi.io/cluster: my-cluster 67 | name: resources-manifest-pipeline-my-streams-app-error 68 | spec: 69 | config: 70 | cleanup.policy: compact,delete 71 | partitions: 1 72 | replicas: 1 73 | 74 | -------------------------------------------------------------------------------- /tests/pipeline/resources/parallel-pipeline/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: transaction-avro-producer-1 3 | to: 4 | topics: 5 | my-output-topic-with-multiple-producers: 6 | type: output 7 | partitions_count: 3 8 | 9 | - type: producer-app 10 | name: transaction-avro-producer-2 11 | to: 12 | topics: 13 | my-output-topic-with-multiple-producers: 14 | type: output 15 | partitions_count: 3 16 | 17 | - type: producer-app 18 | name: transaction-avro-producer-3 19 | to: 20 | topics: 21 | my-output-topic-with-multiple-producers: 22 | type: output 23 | partitions_count: 3 24 | 25 | - type: streams-app 26 | name: transaction-joiner 27 | 28 | - type: streams-app 29 | name: fraud-detector 30 | 31 | - type: streams-app 32 | name: account-linker 33 | from: 34 | components: 35 | fraud-detector: 36 | type: input 37 | 38 | - type: kafka-sink-connector 39 | name: s3-connector-1 40 | from: 41 | topics: 42 | account-linker-test-topic: 43 | type: input 44 | config: 45 | connector.class: io.confluent.connect.s3.S3SinkConnector 46 | 47 | - type: kafka-sink-connector 48 | name: s3-connector-2 49 | from: 50 | topics: 51 | account-linker-test-topic: 52 | type: input 53 | config: 54 | connector.class: io.confluent.connect.s3.S3SinkConnector 55 | 56 | - type: kafka-sink-connector 57 | name: s3-connector-3 58 | from: 59 | topics: 60 | account-linker-test-topic: 61 | type: input 62 | config: 63 | connector.class: io.confluent.connect.s3.S3SinkConnector 64 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_manifest/test_manifest_destroy_manifest_mode/manifest.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kafka.strimzi.io/v1beta2 3 | kind: KafkaTopic 4 | metadata: 5 | labels: 6 | strimzi.io/cluster: my-cluster 7 | name: my-producer-app-output-topic 8 | spec: 9 | config: {} 10 | partitions: 1 11 | replicas: 1 12 | 13 | --- 14 | apiVersion: kafka.strimzi.io/v1beta2 15 | kind: KafkaTopic 16 | metadata: 17 | labels: 18 | strimzi.io/cluster: my-cluster 19 | name: my-labeled-producer-app-topic-output 20 | spec: 21 | config: {} 22 | partitions: 1 23 | replicas: 1 24 | 25 | --- 26 | apiVersion: kafka.strimzi.io/v1beta2 27 | kind: KafkaTopic 28 | metadata: 29 | labels: 30 | strimzi.io/cluster: my-cluster 31 | name: my-output-topic 32 | spec: 33 | config: {} 34 | partitions: 1 35 | replicas: 1 36 | 37 | --- 38 | apiVersion: kafka.strimzi.io/v1beta2 39 | kind: KafkaTopic 40 | metadata: 41 | labels: 42 | strimzi.io/cluster: my-cluster 43 | name: my-error-topic 44 | spec: 45 | config: {} 46 | partitions: 1 47 | replicas: 1 48 | 49 | --- 50 | apiVersion: kafka.strimzi.io/v1beta2 51 | kind: KafkaTopic 52 | metadata: 53 | labels: 54 | strimzi.io/cluster: my-cluster 55 | name: my-labeled-topic-output 56 | spec: 57 | config: {} 58 | partitions: 1 59 | replicas: 1 60 | 61 | --- 62 | apiVersion: kafka.strimzi.io/v1beta2 63 | kind: KafkaTopic 64 | metadata: 65 | labels: 66 | strimzi.io/cluster: my-cluster 67 | name: resources-manifest-pipeline-my-streams-app-error 68 | spec: 69 | config: 70 | cleanup.policy: compact,delete 71 | partitions: 1 72 | replicas: 1 73 | 74 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_manifest/test_manifest_destroy_python_api/manifest.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | apiVersion: kafka.strimzi.io/v1beta2 3 | kind: KafkaTopic 4 | metadata: 5 | labels: 6 | strimzi.io/cluster: my-cluster 7 | name: my-producer-app-output-topic 8 | spec: 9 | config: {} 10 | partitions: 1 11 | replicas: 1 12 | 13 | --- 14 | apiVersion: kafka.strimzi.io/v1beta2 15 | kind: KafkaTopic 16 | metadata: 17 | labels: 18 | strimzi.io/cluster: my-cluster 19 | name: my-labeled-producer-app-topic-output 20 | spec: 21 | config: {} 22 | partitions: 1 23 | replicas: 1 24 | 25 | --- 26 | apiVersion: kafka.strimzi.io/v1beta2 27 | kind: KafkaTopic 28 | metadata: 29 | labels: 30 | strimzi.io/cluster: my-cluster 31 | name: my-output-topic 32 | spec: 33 | config: {} 34 | partitions: 1 35 | replicas: 1 36 | 37 | --- 38 | apiVersion: kafka.strimzi.io/v1beta2 39 | kind: KafkaTopic 40 | metadata: 41 | labels: 42 | strimzi.io/cluster: my-cluster 43 | name: my-error-topic 44 | spec: 45 | config: {} 46 | partitions: 1 47 | replicas: 1 48 | 49 | --- 50 | apiVersion: kafka.strimzi.io/v1beta2 51 | kind: KafkaTopic 52 | metadata: 53 | labels: 54 | strimzi.io/cluster: my-cluster 55 | name: my-labeled-topic-output 56 | spec: 57 | config: {} 58 | partitions: 1 59 | replicas: 1 60 | 61 | --- 62 | apiVersion: kafka.strimzi.io/v1beta2 63 | kind: KafkaTopic 64 | metadata: 65 | labels: 66 | strimzi.io/cluster: my-cluster 67 | name: resources-manifest-pipeline-my-streams-app-error 68 | spec: 69 | config: 70 | cleanup.policy: compact,delete 71 | partitions: 1 72 | replicas: 1 73 | 74 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-config/config.yaml: -------------------------------------------------------------------------------- 1 | # CONFIGURATION 2 | # 3 | # Base directory to the pipelines (default is current working directory) 4 | pipeline_base_dir: . 5 | # The Kafka brokers address. 6 | # REQUIRED 7 | kafka_brokers: "http://broker1:9092,http://broker2:9092" 8 | # Configure the topic name variables you can use in the pipeline definition. 9 | topic_name_config: 10 | # Configures the value for the variable ${output_topic_name} 11 | default_output_topic_name: ${pipeline.name}-${component.name} 12 | # Configures the value for the variable ${error_topic_name} 13 | default_error_topic_name: ${pipeline.name}-${component.name}-error 14 | # Configuration for Schema Registry. 15 | schema_registry: 16 | # Whether the Schema Registry handler should be initialized. 17 | enabled: false 18 | # Address of the Schema Registry. 19 | url: "http://localhost:8081" 20 | # Configuration for the Kafka REST Proxy. 21 | kafka_rest: 22 | # Address of the Kafka REST Proxy. 23 | url: "http://localhost:8082" 24 | # Configuration for Kafka Connect. 25 | kafka_connect: 26 | # Address of Kafka Connect. 27 | url: "http://localhost:8083" 28 | # Flag for `helm upgrade --install`. 29 | # Create the release namespace if not present. 30 | create_namespace: false 31 | # Global flags for Helm. 32 | helm_config: 33 | # Name of kubeconfig context (`--kube-context`) 34 | context: name 35 | # Run Helm in Debug mode. 36 | debug: false 37 | # Kubernetes API version used for Capabilities.APIVersions 38 | api_version: null 39 | # Whether to retain clean up jobs in the cluster or uninstall the, after 40 | # completion. 41 | retain_clean_jobs: false 42 | -------------------------------------------------------------------------------- /docs/docs/user/core-concepts/variables/environment_variables.md: -------------------------------------------------------------------------------- 1 | # Environment variables 2 | 3 | Environment variables can be set by using the [export](https://www.unix.com/man-page/linux/1/export/){target=_blank} command in Linux or the [set](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/set_1){target=_blank} command in Windows. 4 | 5 | 6 | 7 | !!! tip "dotenv files" 8 | 9 | KPOps currently supports `.env` files only for variables related to the [config](../config.md). Full support for `.env` files is on the [roadmap](https://github.com/bakdata/kpops/issues/20). One of the possible ways to use one and export the contents manually is with the following command: `#!sh export $(xargs < .env)`. This would work in `bash` suppose there are no spaces inside the values. 10 | 11 | 12 | 13 | ## Config 14 | 15 | --8<-- 16 | ./docs/resources/variables/config_env_vars.md 17 | --8<-- 18 | 19 | 20 | 21 | ??? example "config_env_vars.env" 22 | 23 | ```py title="Exhaustive list of all config-related environment variables" 24 | --8<-- 25 | ./docs/resources/variables/config_env_vars.env 26 | --8<-- 27 | ``` 28 | 29 | 30 | 31 | ## CLI 32 | 33 | --8<-- 34 | ./docs/resources/variables/cli_env_vars.md 35 | --8<-- 36 | 37 | 38 | 39 | ??? example "cli_env_vars.env" 40 | 41 | ```py title="Exhaustive list of all cli-related environment variables" 42 | --8<-- 43 | ./docs/resources/variables/cli_env_vars.env 44 | --8<-- 45 | ``` 46 | 47 | 48 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/kafka-source-connector.yaml: -------------------------------------------------------------------------------- 1 | # Kafka source connector 2 | - type: kafka-source-connector # required 3 | name: kafka-source-connector # required 4 | # Pipeline prefix that will prefix every component name. If you wish to not 5 | # have any prefix you can specify an empty string. 6 | prefix: ${pipeline.name}- 7 | # The source connector has no `from` section 8 | # from: 9 | # Topic(s) into which the component will write output 10 | to: 11 | topics: 12 | ${pipeline.name}-output-topic: 13 | type: output # Implied when role is NOT specified 14 | ${pipeline.name}-extra-topic: 15 | role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined 16 | ${pipeline.name}-error-topic: 17 | type: error 18 | # Currently KPOps supports Avro and JSON schemas. 19 | key_schema: key-schema # must implement SchemaProvider to use 20 | value_schema: value-schema 21 | partitions_count: 1 22 | replication_factor: 1 23 | configs: # https://kafka.apache.org/documentation/#topicconfigs 24 | cleanup.policy: compact 25 | models: # SchemaProvider is initiated with the values given here 26 | model: model 27 | # Full documentation on connectors: https://kafka.apache.org/documentation/#connectconfigs 28 | config: # required 29 | tasks.max: 1 30 | # Overriding Kafka Connect Resetter Helm values. E.g. to override the 31 | # Image Tag etc. 32 | resetter_values: 33 | imageTag: "1.2.3" 34 | # offset.storage.topic 35 | # https://kafka.apache.org/documentation/#connect_running 36 | offset_topic: offset_topic 37 | -------------------------------------------------------------------------------- /kpops/api/logs.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | import logging 4 | from typing import TYPE_CHECKING 5 | 6 | import typer 7 | from typing_extensions import override 8 | 9 | if TYPE_CHECKING: 10 | from kpops.components.base_components.pipeline_component import PipelineComponent 11 | 12 | 13 | class CustomFormatter(logging.Formatter): 14 | @override 15 | def format(self, record: logging.LogRecord) -> str: 16 | message_format = "%(name)s - %(message)s" 17 | 18 | if record.name == "root": 19 | message_format = "%(message)s" 20 | 21 | formats = { 22 | logging.DEBUG: message_format, 23 | logging.INFO: message_format, 24 | logging.WARNING: typer.style(message_format, fg=typer.colors.YELLOW), 25 | logging.ERROR: typer.style(message_format, fg=typer.colors.RED), 26 | logging.CRITICAL: typer.style( 27 | message_format, fg=typer.colors.RED, bold=True 28 | ), 29 | } 30 | 31 | log_fmt = formats.get(record.levelno) 32 | formatter = logging.Formatter(log_fmt) 33 | return formatter.format(record) 34 | 35 | 36 | logger = logging.getLogger() 37 | logging.getLogger("httpx").setLevel(logging.WARNING) 38 | stream_handler = logging.StreamHandler() 39 | stream_handler.setFormatter(CustomFormatter()) 40 | logger.addHandler(stream_handler) 41 | 42 | log = logging.getLogger("") 43 | LOG_DIVIDER = "#" * 100 44 | 45 | 46 | def log_action(action: str, pipeline_component: PipelineComponent): 47 | log.info("\n") 48 | log.info(LOG_DIVIDER) 49 | log.info(f"{action} {pipeline_component.name}") 50 | log.info(LOG_DIVIDER) 51 | log.info("\n") 52 | -------------------------------------------------------------------------------- /docs/docs/user/migration-guide/v3-v4.md: -------------------------------------------------------------------------------- 1 | # Migrate from V3 to V4 2 | 3 | ## [Distribute defaults across multiple files](https://github.com/bakdata/kpops/pull/438/) 4 | 5 | 6 | 7 | !!! warning inline end 8 | **The `--defaults` flag is removed** 9 | 10 | 11 | 12 | It is possible now to use multiple default values. The `defaults.yaml` (or `defaults_.yaml`) files can be distributed across multiple files. These will be picked up by KPOps and get merged into a single `pipeline.yaml` file. 13 | KPOps starts from reading the default files from where the pipeline path is defined and picks up every defaults file on its way to where the `pipeline_base_dir` is defined. 14 | 15 | For example, imagine the following folder structure: 16 | 17 | ``` 18 | └─ pipelines 19 | └── distributed-defaults 20 | ├── defaults.yaml 21 | ├── defaults_dev.yaml 22 | └── pipeline-deep 23 | ├── defaults.yaml 24 | └── pipeline.yaml 25 | ``` 26 | 27 | The `pipeline_base_dir` is configured to `pipelines`. Now if we generate this pipeline with the following command: 28 | 29 | ```bash 30 | kpops generate \ 31 | --environment dev 32 | ./pipelines/distributed-defaults/pipeline-deep/pipeline.yaml 33 | ``` 34 | 35 | The defaults would be picked in the following order (high to low priority): 36 | 37 | - `./pipelines/distributed-defaults/pipeline-deep/defaults.yaml` 38 | - `./pipelines/distributed-defaults/defaults_dev.yaml` 39 | - `./pipelines/distributed-defaults/defaults.yaml` 40 | 41 | The deepest `defaults.yaml` file in the folder hierarchy (i.e., the closest one to the `pipeline.yaml`) overwrites the higher-level defaults' values. 42 | -------------------------------------------------------------------------------- /tests/pipeline/resources/streams-bootstrap/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: my-producer-app 2 | values: 3 | image: "my-registry/my-producer-image" 4 | imageTag: "1.0.0" 5 | commandLine: 6 | STR: "fake-arg-value" 7 | BOOL: true 8 | INT: 1 9 | FLOAT: 0.1 10 | schedule: "30 3/8 * * *" 11 | 12 | to: 13 | topics: 14 | my-producer-app-output-topic: 15 | type: output 16 | my-labeled-producer-app-topic-output: 17 | label: my-producer-app-output-topic-label 18 | 19 | - type: my-streams-app 20 | values: 21 | image: "my-registry/my-streams-app-image" 22 | imageTag: "1.0.0" 23 | kafka: 24 | applicationId: "my-streams-app-id" 25 | affinity: 26 | nodeAffinity: 27 | preferredDuringSchedulingIgnoredDuringExecution: 28 | - preference: 29 | matchExpressions: 30 | - key: foo 31 | operator: Exists 32 | weight: 2 33 | - preference: 34 | matchExpressions: 35 | - key: bar 36 | operator: DoesNotExist 37 | weight: 1 38 | commandLine: 39 | CONVERT_XML: true 40 | javaOptions: 41 | maxRAMPercentage: 85 42 | 43 | from: 44 | topics: 45 | my-input-topic: 46 | type: input 47 | my-labeled-input-topic: 48 | label: my-input-topic-label 49 | my-input-pattern: 50 | type: pattern 51 | my-labeled-input-pattern: 52 | type: pattern 53 | label: my-input-topic-labeled-pattern 54 | 55 | to: 56 | topics: 57 | my-output-topic: 58 | type: output 59 | my-error-topic: 60 | type: error 61 | my-labeled-topic-output: 62 | label: my-output-topic-label 63 | -------------------------------------------------------------------------------- /tests/compiler/test_yaml_loading.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | from unittest.mock import patch 3 | 4 | import pytest 5 | import yaml 6 | 7 | from kpops.utils.yaml import load_yaml_file 8 | 9 | RESOURCE_PATH = Path(__file__).parent / "resources" 10 | 11 | 12 | def test_load_yaml(): 13 | content = load_yaml_file(RESOURCE_PATH / "test.yaml") 14 | assert isinstance(content, dict) 15 | assert content["test"] == {"correct": "file"} 16 | 17 | 18 | def test_fail_load_yaml(): 19 | with pytest.raises(yaml.YAMLError): 20 | load_yaml_file(RESOURCE_PATH / "erroneous-file.yaml") 21 | 22 | 23 | @patch("yaml.safe_load") 24 | def test_caching_load_yaml(mocked_func): 25 | load_yaml_file( 26 | RESOURCE_PATH / "test.yaml", 27 | substitution={"example": "test", "another": "field"}, 28 | ) 29 | mocked_func.assert_called_once() 30 | mocked_func.reset_mock() 31 | 32 | # load the same yaml with the same substitution 33 | load_yaml_file( 34 | RESOURCE_PATH / "test.yaml", 35 | substitution={"example": "test", "another": "field"}, 36 | ) 37 | mocked_func.assert_not_called() 38 | mocked_func.reset_mock() 39 | 40 | # load the same yaml with the same substitution 41 | load_yaml_file( 42 | RESOURCE_PATH / "test.yaml", 43 | substitution={"another": "field", "example": "test"}, 44 | ) 45 | mocked_func.assert_not_called() 46 | mocked_func.reset_mock() 47 | 48 | # load the same yaml with the another substitution 49 | load_yaml_file( 50 | RESOURCE_PATH / "test.yaml", 51 | substitution={"another": "test"}, 52 | ) 53 | mocked_func.assert_called_once() 54 | mocked_func.reset_mock() 55 | 56 | # load the another yaml 57 | load_yaml_file(RESOURCE_PATH / "another-test.yaml") 58 | mocked_func.assert_called_once() 59 | -------------------------------------------------------------------------------- /kpops/components/base_components/models/from_section.py: -------------------------------------------------------------------------------- 1 | from enum import StrEnum 2 | from typing import Any, ClassVar, NewType 3 | 4 | from pydantic import ConfigDict, model_validator 5 | 6 | from kpops.components.base_components.models import TopicName 7 | from kpops.utils.pydantic import DescConfigModel 8 | 9 | 10 | class InputTopicTypes(StrEnum): 11 | """Input topic types. 12 | 13 | - INPUT: input topic 14 | - PATTERN: extra-topic-pattern or input-topic-pattern 15 | """ 16 | 17 | INPUT = "input" 18 | PATTERN = "pattern" 19 | 20 | 21 | class FromTopic(DescConfigModel): 22 | """Input topic. 23 | 24 | :param type: Topic type, defaults to None 25 | :param label: Custom identifier belonging to a topic; 26 | define only if `type` is `pattern` or `None`, defaults to None 27 | """ 28 | 29 | type: InputTopicTypes | None = None 30 | label: str | None = None 31 | 32 | model_config: ClassVar[ConfigDict] = ConfigDict( 33 | extra="forbid", 34 | use_enum_values=True, 35 | ) 36 | 37 | @model_validator(mode="after") 38 | def extra_topic_label(self) -> Any: 39 | """Ensure that `cls.label` is used correctly, assign type if needed.""" 40 | if self.type == InputTopicTypes.INPUT and self.label: 41 | msg = "Define label only if `type` is `pattern` or `None`" 42 | raise ValueError(msg) 43 | return self 44 | 45 | 46 | ComponentName = NewType("ComponentName", str) 47 | 48 | 49 | class FromSection(DescConfigModel): 50 | """Holds multiple input topics. 51 | 52 | :param topics: Input topics 53 | :param components: Components to read from 54 | """ 55 | 56 | topics: dict[TopicName, FromTopic] = {} 57 | components: dict[ComponentName, FromTopic] = {} 58 | 59 | model_config: ClassVar[ConfigDict] = ConfigDict(extra="forbid") 60 | -------------------------------------------------------------------------------- /tests/component_handlers/resources/kafka_rest_proxy_responses/broker_response.json: -------------------------------------------------------------------------------- 1 | { 2 | "kind": "KafkaBrokerConfigList", 3 | "metadata": { 4 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/brokers/-/configs", 5 | "next": "null" 6 | }, 7 | "data": [ 8 | { 9 | "kind": "KafkaBrokerConfig", 10 | "metadata": { 11 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/brokers/0/configs/default.replication.factor", 12 | "resource_name": "crn:///kafka=cluster-1/broker=0/config=default.replication.factor" 13 | }, 14 | "cluster_id": "cluster-1", 15 | "broker_id": "0", 16 | "name": "default.replication.factor", 17 | "value": "1", 18 | "is_default": "true", 19 | "is_read_only": "false", 20 | "is_sensitive": "false", 21 | "source": "DEFAULT_CONFIG", 22 | "synonyms": [ 23 | { 24 | "name": "default.replication.factor", 25 | "value": "1", 26 | "source": "DEFAULT_CONFIG" 27 | } 28 | ] 29 | }, 30 | { 31 | "kind": "KafkaBrokerConfig", 32 | "metadata": { 33 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/brokers/0/configs/num.partitions", 34 | "resource_name": "crn:///kafka=cluster-1/broker=0/config=num.partitions" 35 | }, 36 | "cluster_id": "cluster-1", 37 | "broker_id": "0", 38 | "name": "num.partitions", 39 | "value": "1", 40 | "is_default": "true", 41 | "is_read_only": "false", 42 | "is_sensitive": "false", 43 | "source": "DEFAULT_CONFIG", 44 | "synonyms": [ 45 | { 46 | "name": "num.partitions", 47 | "value": "1", 48 | "source": "DEFAULT_CONFIG" 49 | } 50 | ] 51 | } 52 | ] 53 | } 54 | -------------------------------------------------------------------------------- /tests/pipeline/test_example.py: -------------------------------------------------------------------------------- 1 | import os 2 | from pathlib import Path 3 | 4 | import pytest 5 | from pytest_snapshot.plugin import Snapshot 6 | from typer.testing import CliRunner 7 | 8 | import kpops.api as kpops 9 | 10 | runner = CliRunner() 11 | 12 | EXAMPLES_PATH = Path("examples").absolute() 13 | 14 | 15 | @pytest.mark.usefixtures("mock_env", "load_yaml_file_clear_cache", "clear_kpops_config") 16 | class TestExample: 17 | @pytest.fixture(scope="class", autouse=True) 18 | def cd(self): 19 | cwd = Path.cwd().absolute() 20 | os.chdir(EXAMPLES_PATH) 21 | yield 22 | os.chdir(cwd) 23 | 24 | def test_cwd(self): 25 | assert Path.cwd() == EXAMPLES_PATH 26 | 27 | @pytest.fixture(scope="session") 28 | def test_submodule(self): 29 | assert any(EXAMPLES_PATH.iterdir()), ( 30 | "examples directory is empty, please initialize and update the git submodule (see contributing guide)" 31 | ) 32 | 33 | @pytest.mark.usefixtures("test_submodule") 34 | @pytest.mark.parametrize( 35 | "pipeline_name", 36 | [ 37 | pytest.param("word-count"), 38 | pytest.param( 39 | "atm-fraud", 40 | marks=( 41 | # NOTE: remove after pipeline has been updated to streams-bootstrap v3 42 | # depends on https://github.com/bakdata/pipeline-atm-fraud/issues/4 43 | pytest.mark.filterwarnings( 44 | "ignore:.*StreamsBootstrapV2|(Producer|Streams)AppV2.*:DeprecationWarning" 45 | ) 46 | ), 47 | ), 48 | ], 49 | ) 50 | def test_generate(self, pipeline_name: str, snapshot: Snapshot): 51 | pipeline = kpops.generate(Path(f"{pipeline_name}/pipeline.yaml")) 52 | snapshot.assert_match(pipeline.to_yaml(), "pipeline.yaml") 53 | -------------------------------------------------------------------------------- /kpops/components/base_components/kubernetes_app.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | import logging 4 | import re 5 | from abc import ABC 6 | from typing import ClassVar 7 | 8 | from pydantic import ConfigDict 9 | from typing_extensions import override 10 | 11 | from kpops.components.base_components.pipeline_component import PipelineComponent 12 | from kpops.utils.pydantic import CamelCaseConfigModel, DescConfigModel 13 | 14 | log = logging.getLogger("KubernetesApp") 15 | 16 | KUBERNETES_NAME_CHECK_PATTERN = re.compile( 17 | r"^(?![0-9]+$)(?!.*-$)(?!-)[a-z0-9-.]{1,253}(? None: 43 | super()._validate_custom() 44 | self.validate_kubernetes_name(self.name) 45 | 46 | @staticmethod 47 | def validate_kubernetes_name(name: str) -> None: 48 | """Check if a name is valid for a Kubernetes resource. 49 | 50 | :param name: Name that is to be used for the resource 51 | :raises ValueError: The component name {name} is invalid for Kubernetes. 52 | """ 53 | if not bool(KUBERNETES_NAME_CHECK_PATTERN.match(name)): 54 | msg = f"The component name {name} is invalid for Kubernetes." 55 | raise ValueError(msg) 56 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_generate/test_kafka_connect_sink_weave_from_topics/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: streams-app 2 | name: streams-app 3 | prefix: resources-kafka-connect-sink- 4 | from: 5 | topics: 6 | example-topic: 7 | type: input 8 | components: {} 9 | to: 10 | topics: 11 | example-output: 12 | type: output 13 | configs: {} 14 | resources-kafka-connect-sink-streams-app-error: 15 | type: error 16 | value_schema: com.bakdata.kafka.DeadLetter 17 | partitions_count: 1 18 | configs: 19 | cleanup.policy: compact,delete 20 | models: {} 21 | namespace: example-namespace 22 | values: 23 | image: fake-image 24 | kafka: 25 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 26 | schemaRegistryUrl: http://localhost:8081/ 27 | outputTopic: example-output 28 | inputTopics: 29 | - example-topic 30 | errorTopic: resources-kafka-connect-sink-streams-app-error 31 | config: 32 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 33 | statefulSet: false 34 | version: 3.6.1 35 | helm_release_name: resources-kafka-connect-sink-streams-app 36 | helm_name_override: resources-kafka-connect-sink-streams-app 37 | - type: kafka-sink-connector 38 | name: es-sink-connector 39 | prefix: resources-kafka-connect-sink- 40 | config: 41 | connector.class: io.confluent.connect.elasticsearch.ElasticsearchSinkConnector 42 | name: resources-kafka-connect-sink-es-sink-connector 43 | topics: example-output 44 | batch.size: '2000' 45 | behavior.on.malformed.documents: warn 46 | behavior.on.null.values: delete 47 | connection.compression: 'true' 48 | key.ignore: 'false' 49 | linger.ms: '5000' 50 | max.buffered.records: '20000' 51 | read.timeout.ms: '120000' 52 | tasks.max: '1' 53 | resetter_values: {} 54 | 55 | -------------------------------------------------------------------------------- /docs/docs/user/migration-guide/v8-v9.md: -------------------------------------------------------------------------------- 1 | # Migrate from V8 to V9 2 | 3 | ## [Introduce KPOps operation and manifest resources for deployment](https://github.com/bakdata/kpops/pull/541) 4 | 5 | The `kpops manifest` command and `kpops.manifest()` API have been **removed**. 6 | 7 | Resource manifesting is now integrated into the _operation_ commands (`deploy`, `destroy`, `reset`, `clean`) through the new **operation mode** feature. 8 | 9 | To manifest resources, you can: 10 | 11 | - Pass `--operation-mode manifest` when executing `kpops` commands. 12 | - Set the operation mode by defining the `KPOPS_OPERATION_MODE` environment variable. 13 | 14 | ## [Manifest toSection with Strimzi KafkaTopic](https://github.com/bakdata/kpops/pull/545) 15 | 16 | KPOps now supports generating valid Kubernetes KafkaTopic resources compatible with [Strimzi](https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/topic/kafka-topic.yaml). When using manifest or argo as the operation_mode, you must specify the Strimzi cluster label to ensure the topics are recognized by the deployed Strimzi Topic Operator. 17 | 18 | ```diff 19 | operation_mode: manifest 20 | 21 | + strimzi_topic: 22 | + label: 23 | + strimzi.io/cluster: my-cluster 24 | 25 | # rest of your config 26 | ``` 27 | 28 | 29 | 30 | !!! info Standalone topic operator deployment 31 | Refer to the [Strimzi documentation on deploying a standalone topic operator](https://strimzi.io/docs/operators/latest/deploying#deploying-the-topic-operator-standalone-str) for more details. 32 | 33 | 34 | 35 | ## [Drop support for Python 3.10](https://github.com/bakdata/kpops/pull/561) 36 | 37 | KPOps V9 no longer supports Python 3.10. Ensure your environment is running Python 3.11 to 3.12. 38 | 39 | #### Action Required: 40 | 41 | Upgrade your Python version to a supported version (3.11 or 3.12). 42 | Update your virtual environments and CI pipelines to reflect this change. 43 | -------------------------------------------------------------------------------- /docs/docs/resources/variables/cli_env_vars.md: -------------------------------------------------------------------------------- 1 | These variables take precedence over the commands' flags. If a variable is set, the corresponding flag does not have to be specified in commands. Variables marked as required can instead be set as flags. 2 | 3 | | Name |Default Value|Required| Description | 4 | |--------------------|-------------|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| 5 | |KPOPS_CONFIG_PATH |. |False |Path to the dir containing config.yaml files | 6 | |KPOPS_DOTENV_PATH | |False |Path to dotenv file. Multiple files can be provided. The files will be loaded in order, with each file overriding the previous one. | 7 | |KPOPS_ENVIRONMENT | |False |The environment you want to generate and deploy the pipeline to. Suffix your environment files with this value (e.g. defaults_development.yaml for environment=development).| 8 | |KPOPS_OPERATION_MODE|managed |False |How KPOps should operate. | 9 | |KPOPS_PIPELINE_PATHS| |True |Paths to dir containing 'pipeline.yaml' or files named 'pipeline.yaml'. | 10 | |KPOPS_PIPELINE_STEPS| |False |Comma separated list of steps to apply the command on | 11 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_generate/test_with_custom_config_with_absolute_defaults_path/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: app1 3 | prefix: resources-custom-config- 4 | to: 5 | topics: 6 | app1-test-topic: 7 | type: output 8 | partitions_count: 3 9 | configs: {} 10 | models: {} 11 | namespace: development-namespace 12 | values: 13 | image: bakdata-demo-producer-app 14 | kafka: 15 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 16 | schemaRegistryUrl: http://localhost:8081/ 17 | outputTopic: app1-test-topic 18 | resources: 19 | requests: 20 | memory: 2G 21 | limits: 22 | memory: 2G 23 | version: 3.6.1 24 | helm_release_name: resources-custom-config-app1 25 | helm_name_override: resources-custom-config-app1 26 | - type: streams-app 27 | name: app2 28 | prefix: resources-custom-config- 29 | to: 30 | topics: 31 | app2-dead-letter-topic: 32 | type: error 33 | value_schema: com.bakdata.kafka.DeadLetter 34 | partitions_count: 1 35 | configs: 36 | cleanup.policy: compact,delete 37 | app2-test-topic: 38 | type: output 39 | partitions_count: 3 40 | configs: {} 41 | models: {} 42 | namespace: development-namespace 43 | values: 44 | image: some-image 45 | kafka: 46 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 47 | schemaRegistryUrl: http://localhost:8081/ 48 | outputTopic: app2-test-topic 49 | inputTopics: 50 | - app1-test-topic 51 | errorTopic: app2-dead-letter-topic 52 | config: 53 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 54 | statefulSet: false 55 | labels: 56 | pipeline: resources-custom-config 57 | version: 3.6.1 58 | helm_release_name: resources-custom-config-app2 59 | helm_name_override: resources-custom-config-app2 60 | 61 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_generate/test_with_custom_config_with_relative_defaults_path/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: app1 3 | prefix: resources-custom-config- 4 | to: 5 | topics: 6 | app1-test-topic: 7 | type: output 8 | partitions_count: 3 9 | configs: {} 10 | models: {} 11 | namespace: development-namespace 12 | values: 13 | image: bakdata-demo-producer-app 14 | kafka: 15 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 16 | schemaRegistryUrl: http://localhost:8081/ 17 | outputTopic: app1-test-topic 18 | resources: 19 | requests: 20 | memory: 2G 21 | limits: 22 | memory: 2G 23 | version: 3.6.1 24 | helm_release_name: resources-custom-config-app1 25 | helm_name_override: resources-custom-config-app1 26 | - type: streams-app 27 | name: app2 28 | prefix: resources-custom-config- 29 | to: 30 | topics: 31 | app2-dead-letter-topic: 32 | type: error 33 | value_schema: com.bakdata.kafka.DeadLetter 34 | partitions_count: 1 35 | configs: 36 | cleanup.policy: compact,delete 37 | app2-test-topic: 38 | type: output 39 | partitions_count: 3 40 | configs: {} 41 | models: {} 42 | namespace: development-namespace 43 | values: 44 | image: some-image 45 | kafka: 46 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 47 | schemaRegistryUrl: http://localhost:8081/ 48 | outputTopic: app2-test-topic 49 | inputTopics: 50 | - app1-test-topic 51 | errorTopic: app2-dead-letter-topic 52 | config: 53 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 54 | statefulSet: false 55 | labels: 56 | pipeline: resources-custom-config 57 | version: 3.6.1 58 | helm_release_name: resources-custom-config-app2 59 | helm_name_override: resources-custom-config-app2 60 | 61 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_generate/test_with_env_defaults/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: streams-app 2 | name: streams-app-development 3 | prefix: resources-pipeline-with-env-defaults- 4 | from: 5 | topics: 6 | example-topic: 7 | type: input 8 | components: {} 9 | to: 10 | topics: 11 | example-output: 12 | type: output 13 | configs: {} 14 | resources-pipeline-with-env-defaults-streams-app-development-error: 15 | type: error 16 | value_schema: com.bakdata.kafka.DeadLetter 17 | partitions_count: 1 18 | configs: 19 | cleanup.policy: compact,delete 20 | models: {} 21 | namespace: development-namespace 22 | values: 23 | image: fake-image 24 | kafka: 25 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 26 | schemaRegistryUrl: http://localhost:8081/ 27 | outputTopic: example-output 28 | inputTopics: 29 | - example-topic 30 | errorTopic: resources-pipeline-with-env-defaults-streams-app-development-error 31 | config: 32 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 33 | statefulSet: false 34 | version: 3.6.1 35 | helm_release_name: resources-pipeline-with-env-defaults-streams-ap-49439 36 | helm_name_override: resources-pipeline-with-env-defaults-streams-app-development 37 | - type: kafka-sink-connector 38 | name: es-sink-connector 39 | prefix: resources-pipeline-with-env-defaults- 40 | config: 41 | connector.class: io.confluent.connect.elasticsearch.ElasticsearchSinkConnector 42 | name: resources-pipeline-with-env-defaults-es-sink-connector 43 | topics: example-output 44 | batch.size: '2000' 45 | behavior.on.malformed.documents: warn 46 | behavior.on.null.values: delete 47 | connection.compression: 'true' 48 | key.ignore: 'false' 49 | linger.ms: '5000' 50 | max.buffered.records: '20000' 51 | read.timeout.ms: '120000' 52 | tasks.max: '1' 53 | resetter_values: {} 54 | 55 | -------------------------------------------------------------------------------- /tests/pipeline/snapshots/test_generate/test_default_config/pipeline.yaml: -------------------------------------------------------------------------------- 1 | - type: producer-app 2 | name: app1 3 | prefix: resources-custom-config- 4 | to: 5 | topics: 6 | resources-custom-config-app1: 7 | type: output 8 | partitions_count: 3 9 | configs: {} 10 | models: {} 11 | namespace: development-namespace 12 | values: 13 | image: bakdata-demo-producer-app 14 | kafka: 15 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 16 | schemaRegistryUrl: http://localhost:8081/ 17 | outputTopic: resources-custom-config-app1 18 | resources: 19 | requests: 20 | memory: 2G 21 | limits: 22 | memory: 2G 23 | version: 3.6.1 24 | helm_release_name: resources-custom-config-app1 25 | helm_name_override: resources-custom-config-app1 26 | - type: streams-app 27 | name: app2 28 | prefix: resources-custom-config- 29 | to: 30 | topics: 31 | resources-custom-config-app2-error: 32 | type: error 33 | value_schema: com.bakdata.kafka.DeadLetter 34 | partitions_count: 1 35 | configs: 36 | cleanup.policy: compact,delete 37 | resources-custom-config-app2: 38 | type: output 39 | partitions_count: 3 40 | configs: {} 41 | models: {} 42 | namespace: development-namespace 43 | values: 44 | image: some-image 45 | kafka: 46 | bootstrapServers: http://k8kafka-cp-kafka-headless.kpops.svc.cluster.local:9092 47 | schemaRegistryUrl: http://localhost:8081/ 48 | outputTopic: resources-custom-config-app2 49 | inputTopics: 50 | - resources-custom-config-app1 51 | errorTopic: resources-custom-config-app2-error 52 | config: 53 | large.message.id.generator: com.bakdata.kafka.MurmurHashIdGenerator 54 | statefulSet: false 55 | labels: 56 | pipeline: resources-custom-config 57 | version: 3.6.1 58 | helm_release_name: resources-custom-config-app2 59 | helm_name_override: resources-custom-config-app2 60 | 61 | -------------------------------------------------------------------------------- /.github/actions/update-docs/action.yml: -------------------------------------------------------------------------------- 1 | name: "Update documentation in gh-pages" 2 | description: | 3 | Compile markdown documents to html and deploy to docs branch. If a semver version is given, this action strips the patch. 4 | It then pushes the . as the latest alias to the documentation branch with mike. 5 | 6 | inputs: 7 | username: 8 | description: "GitHub username" 9 | required: true 10 | email: 11 | description: "GitHub email" 12 | required: true 13 | token: 14 | description: "GitHub Token (must be a PAT for repository dispatch)" 15 | required: true 16 | version: 17 | description: "Version name to be deployed by mike" 18 | required: true 19 | release: 20 | description: "Determines if the set version is a stable and latest version, otherwise it is a dev version. (Default false)" 21 | default: "false" 22 | required: false 23 | 24 | runs: 25 | using: "composite" 26 | steps: 27 | - name: Install uv 28 | uses: astral-sh/setup-uv@v5 29 | with: 30 | python-version: "3.11" 31 | version: "0.5.14" 32 | 33 | - name: Update gh-pages branch 34 | shell: bash 35 | run: | 36 | git config --local user.name ${{ inputs.username }} 37 | git config --local user.email ${{ inputs.email }} 38 | git config --local user.password ${{ inputs.token }} 39 | git fetch origin gh-pages 40 | 41 | - name: Deploy ${{ inputs.version }} version of the documentation with mike 42 | shell: bash 43 | if: ${{ inputs.release == 'false' }} 44 | run: | 45 | uv run --frozen --group=docs mike deploy ${{ inputs.version }} --push --config-file ./docs/mkdocs.yml 46 | 47 | - name: Deploy ${{ inputs.version }} version (latest) of the documentation with mike 48 | shell: bash 49 | if: ${{ inputs.release == 'true' }} 50 | run: | 51 | sem_version=${{ inputs.version }} 52 | major_minor_version=${sem_version%.*} 53 | uv run --frozen --group=docs mike deploy "$major_minor_version" latest --update-aliases --push --config-file ./docs/mkdocs.yml 54 | -------------------------------------------------------------------------------- /tests/component_handlers/resources/kafka_rest_proxy_responses/topic_config_response.json: -------------------------------------------------------------------------------- 1 | { 2 | "kind": "KafkaTopicConfigList", 3 | "metadata": { 4 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/configs", 5 | "next": "null" 6 | }, 7 | "data": [ 8 | { 9 | "kind": "KafkaTopicConfig", 10 | "metadata": { 11 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/configs/cleanup.policy", 12 | "resource_name": "crn:///kafka=cluster-1/topic=topic-1/config=cleanup.policy" 13 | }, 14 | "cluster_id": "cluster-1", 15 | "topic_name": "topic-1", 16 | "name": "cleanup.policy", 17 | "value": "compact", 18 | "is_default": "false", 19 | "is_read_only": "false", 20 | "is_sensitive": "false", 21 | "source": "DYNAMIC_TOPIC_CONFIG", 22 | "synonyms": [ 23 | { 24 | "name": "cleanup.policy", 25 | "value": "compact", 26 | "source": "DYNAMIC_TOPIC_CONFIG" 27 | }, 28 | { 29 | "name": "cleanup.policy", 30 | "value": "delete", 31 | "source": "DEFAULT_CONFIG" 32 | } 33 | ] 34 | }, 35 | { 36 | "kind": "KafkaTopicConfig", 37 | "metadata": { 38 | "self": "https://pkc-00000.region.provider.confluent.cloud/kafka/v3/clusters/cluster-1/topics/topic-1/configs/compression.type", 39 | "resource_name": "crn:///kafka=cluster-1/topic=topic-1/config=compression.type" 40 | }, 41 | "cluster_id": "cluster-1", 42 | "topic_name": "topic-1", 43 | "name": "compression.type", 44 | "value": "gzip", 45 | "is_default": "false", 46 | "is_read_only": "false", 47 | "is_sensitive": "false", 48 | "source": "DYNAMIC_TOPIC_CONFIG", 49 | "synonyms": [ 50 | { 51 | "name": "compression.type", 52 | "value": "gzip", 53 | "source": "DYNAMIC_TOPIC_CONFIG" 54 | }, 55 | { 56 | "name": "compression.type", 57 | "value": "producer", 58 | "source": "DEFAULT_CONFIG" 59 | } 60 | ] 61 | } 62 | ] 63 | } 64 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/defaults-kafka-app.yaml: -------------------------------------------------------------------------------- 1 | # Base component for Kafka-based components. 2 | # 3 | # Parent of: ProducerApp, StreamsApp 4 | # Child of: KubernetesApp 5 | kafka-app: 6 | # Pipeline prefix that will prefix every component name. If you wish to not 7 | # have any prefix you can specify an empty string. 8 | prefix: ${pipeline.name}- 9 | from: # Must not be null 10 | topics: # read from topic 11 | ${pipeline.name}-input-topic: 12 | type: input # Implied when role is NOT specified 13 | ${pipeline.name}-extra-topic: 14 | role: topic-role # Implies `type` to be extra 15 | ${pipeline.name}-input-pattern-topic: 16 | type: pattern # Implied to be an input pattern if `role` is undefined 17 | ${pipeline.name}-extra-pattern-topic: 18 | type: pattern # Implied to be an extra pattern if `role` is defined 19 | role: some-role 20 | components: # read from specific component 21 | account-producer: 22 | type: input # Implied when role is NOT specified 23 | other-producer: 24 | role: some-role # Implies `type` to be extra 25 | component-as-input-pattern: 26 | type: pattern # Implied to be an input pattern if `role` is undefined 27 | component-as-extra-pattern: 28 | type: pattern # Implied to be an extra pattern if `role` is defined 29 | role: some-role 30 | # Topic(s) into which the component will write output 31 | to: 32 | topics: 33 | ${pipeline.name}-output-topic: 34 | type: output # Implied when role is NOT specified 35 | ${pipeline.name}-extra-topic: 36 | role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined 37 | ${pipeline.name}-error-topic: 38 | type: error 39 | # Currently KPOps supports Avro and JSON schemas. 40 | key_schema: key-schema # must implement SchemaProvider to use 41 | value_schema: value-schema 42 | partitions_count: 1 43 | replication_factor: 1 44 | configs: # https://kafka.apache.org/documentation/#topicconfigs 45 | cleanup.policy: compact 46 | models: # SchemaProvider is initiated with the values given here 47 | model: model 48 | -------------------------------------------------------------------------------- /lefthook.yaml: -------------------------------------------------------------------------------- 1 | # yaml-language-server: $schema=https://raw.githubusercontent.com/evilmartians/lefthook/master/schema.json 2 | 3 | pre-commit: 4 | parallel: true 5 | commands: 6 | uv-lock: 7 | glob: "{pyproject.toml,uv.lock}" 8 | run: uv lock --check 9 | ruff-check: 10 | glob: "*.py" 11 | run: uv run ruff check --force-exclude --config pyproject.toml --fix --show-fixes {staged_files} 12 | stage_fixed: true 13 | ruff-format: 14 | glob: "*.py" 15 | run: uv run ruff format --force-exclude --config pyproject.toml {staged_files} 16 | stage_fixed: true 17 | basedpyright: 18 | glob: "*.py" 19 | run: uv run basedpyright 20 | dprint: 21 | glob: "*.{json,jsonc,yaml,yml,md,toml,css}" 22 | run: dprint fmt 23 | kpops-schema: 24 | glob: "{uv.lock,kpops/**/*.py,hooks/gen_schema.py}" 25 | run: python -m hooks.gen_schema && git add -u docs/docs/schema 26 | kpops-docs-cli: 27 | glob: "{uv.lock,kpops/cli/main.py,hooks/gen_docs/gen_docs_cli_usage.py}" 28 | run: python -m hooks.gen_docs.gen_docs_cli_usage 29 | kpops-docs-env-vars: 30 | glob: "{uv.lock,kpops/**/*.py,hooks/gen_docs/gen_docs_env_vars.py}" 31 | run: python -m hooks.gen_docs.gen_docs_env_vars 32 | kpops-docs-components: 33 | glob: "{\ 34 | .gitignore,\ 35 | uv.lock,\ 36 | hooks/gen_docs/gen_docs_components.py,\ 37 | docs/docs/resources/pipeline-defaults/headers/*.yaml,\ 38 | docs/docs/resources/pipeline-components/headers/*.yaml,\ 39 | docs/docs/resources/pipeline-components/sections/*.yaml,\ 40 | docs/docs/resources/pipeline-components/dependencies/*.yaml,\ 41 | kpops/components/*.py\ 42 | }" 43 | # .gitignore -- indicates whether the script is ran with `--all-files` 44 | # hooks/gen_docs/gen_docs_components.py -- hook's behavior is possibly changed 45 | # docs/.../.yaml -- examples changed or dependencies edited 46 | # kpops/components/... -- KPOps components possibly changed 47 | run: python -m hooks.gen_docs.gen_docs_components {staged_files} 48 | 49 | post-checkout: 50 | commands: 51 | dependencies: 52 | glob: "uv.lock" 53 | run: uv sync 54 | -------------------------------------------------------------------------------- /docs/docs/user/getting-started/teardown.md: -------------------------------------------------------------------------------- 1 | # Teardown resources 2 | 3 | ## KPOps teardown commands 4 | 5 | - `destroy`: Removes Kubernetes resources. 6 | - `reset`: Runs `destroy`, resets the states of Kafka Streams apps and resets offsets to zero. 7 | - `clean`: Runs `reset` and removes all Kafka resources. 8 | 9 | ## KPOps-deployed pipeline 10 | 11 | The [`kpops` CLI](../references/cli-commands.md) can be used to destroy a pipeline that was previously deployed with KPOps. 12 | In case that doesn't work, the pipeline can always be taken down manually with `helm` (see section [Infrastructure](#infrastructure)). 13 | 14 | 15 | 16 | 1. Export environment variables. 17 | 18 | ```shell 19 | export DOCKER_REGISTRY=bakdata && \ 20 | export NAMESPACE=kpops 21 | ``` 22 | 23 | 2. Navigate to the `examples` folder. 24 | Replace the `` with the example you want to tear down. 25 | For example the `atm-fraud-detection`. 26 | 27 | 3. Remove the pipeline 28 | 29 | ```shell 30 | # Uncomment 1 line to either destroy, reset or clean. 31 | 32 | # kpops destroy /pipeline.yaml \ 33 | # kpops reset /pipeline.yaml \ 34 | # kpops clean /pipeline.yaml \ 35 | --config /config.yaml \ 36 | --execute 37 | ``` 38 | 39 | 40 | 41 | ## Infrastructure 42 | 43 | Delete namespace: 44 | 45 | ```shell 46 | kubectl delete namespace kpops 47 | ``` 48 | 49 | 50 | 51 | !!! Note 52 | In case `kpops destroy` is not working one can uninstall the pipeline services one by one. 53 | This is equivalent to running `kpops destroy`. In case a clean uninstall (like the one `kpops clean` does) 54 | is needed, one needs to also delete the topics and schemas created by deployment of the pipeline. 55 | 56 | 57 | 58 | ## Local cluster 59 | 60 | Delete local cluster: 61 | 62 | ```shell 63 | k3d cluster delete kpops 64 | ``` 65 | 66 | ## Local image registry 67 | 68 | Delete local registry: 69 | 70 | ```shell 71 | k3d registry delete k3d-kpops-registry.localhost 72 | ``` 73 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/kafka-app.yaml: -------------------------------------------------------------------------------- 1 | # Base component for Kafka-based components. 2 | # Producer or streaming apps should inherit from this class. 3 | - type: kafka-app # required 4 | name: kafka-app # required 5 | # Pipeline prefix that will prefix every component name. If you wish to not 6 | # have any prefix you can specify an empty string. 7 | prefix: ${pipeline.name}- 8 | from: # Must not be null 9 | topics: # read from topic 10 | ${pipeline.name}-input-topic: 11 | type: input # Implied when role is NOT specified 12 | ${pipeline.name}-extra-topic: 13 | role: topic-role # Implies `type` to be extra 14 | ${pipeline.name}-input-pattern-topic: 15 | type: pattern # Implied to be an input pattern if `role` is undefined 16 | ${pipeline.name}-extra-pattern-topic: 17 | type: pattern # Implied to be an extra pattern if `role` is defined 18 | role: some-role 19 | components: # read from specific component 20 | account-producer: 21 | type: input # Implied when role is NOT specified 22 | other-producer: 23 | role: some-role # Implies `type` to be extra 24 | component-as-input-pattern: 25 | type: pattern # Implied to be an input pattern if `role` is undefined 26 | component-as-extra-pattern: 27 | type: pattern # Implied to be an extra pattern if `role` is defined 28 | role: some-role 29 | # Topic(s) into which the component will write output 30 | to: 31 | topics: 32 | ${pipeline.name}-output-topic: 33 | type: output # Implied when role is NOT specified 34 | ${pipeline.name}-extra-topic: 35 | role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined 36 | ${pipeline.name}-error-topic: 37 | type: error 38 | # Currently KPOps supports Avro and JSON schemas. 39 | key_schema: key-schema # must implement SchemaProvider to use 40 | value_schema: value-schema 41 | partitions_count: 1 42 | replication_factor: 1 43 | configs: # https://kafka.apache.org/documentation/#topicconfigs 44 | cleanup.policy: compact 45 | models: # SchemaProvider is initiated with the values given here 46 | model: model 47 | -------------------------------------------------------------------------------- /tests/pipeline/test_deploy.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | from unittest.mock import AsyncMock 3 | 4 | import pytest 5 | from pytest_mock import MockerFixture 6 | from typer.testing import CliRunner 7 | 8 | from kpops.cli.main import app 9 | from kpops.components.base_components import HelmApp 10 | from kpops.components.streams_bootstrap import ProducerApp, StreamsApp 11 | 12 | runner = CliRunner() 13 | 14 | RESOURCE_PATH = Path(__file__).parent / "resources" 15 | 16 | 17 | @pytest.mark.usefixtures("mock_env", "load_yaml_file_clear_cache", "clear_kpops_config") 18 | class TestDeploy: 19 | @pytest.fixture(autouse=True) 20 | def mock_helm(self, mocker: MockerFixture) -> AsyncMock: 21 | return mocker.patch( 22 | "kpops.components.base_components.helm_app.Helm", 23 | return_value=AsyncMock(), 24 | ).return_value 25 | 26 | def test_order(self, mocker: MockerFixture): 27 | producer_app_mock_deploy = mocker.patch.object(ProducerApp, "deploy") 28 | streams_app_mock_deploy = mocker.patch.object(StreamsApp, "deploy") 29 | helm_app_mock_deploy = mocker.patch.object(HelmApp, "deploy") 30 | mock_deploy = mocker.AsyncMock() 31 | mock_deploy.attach_mock(producer_app_mock_deploy, "producer_app_mock_deploy") 32 | mock_deploy.attach_mock(streams_app_mock_deploy, "streams_app_mock_deploy") 33 | mock_deploy.attach_mock(helm_app_mock_deploy, "helm_app_mock_deploy") 34 | 35 | result = runner.invoke( 36 | app, 37 | [ 38 | "deploy", 39 | str(RESOURCE_PATH / "simple-pipeline" / "pipeline.yaml"), 40 | ], 41 | catch_exceptions=False, 42 | ) 43 | 44 | assert result.exit_code == 0, result.stdout 45 | 46 | # check called 47 | producer_app_mock_deploy.assert_called_once_with(True) 48 | streams_app_mock_deploy.assert_called_once_with(True) 49 | helm_app_mock_deploy.assert_called_once_with(True) 50 | 51 | # check order 52 | assert mock_deploy.mock_calls == [ 53 | mocker.call.producer_app_mock_deploy(True), 54 | mocker.call.streams_app_mock_deploy(True), 55 | mocker.call.helm_app_mock_deploy(True), 56 | ] 57 | -------------------------------------------------------------------------------- /tests/cli/test_init.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | 3 | import pytest 4 | from pytest_snapshot.plugin import Snapshot 5 | from typer.testing import CliRunner 6 | 7 | import kpops.api as kpops 8 | from kpops.cli.main import app 9 | from kpops.utils.cli_commands import create_config 10 | 11 | runner = CliRunner() 12 | 13 | 14 | def test_create_config(tmp_path: Path): 15 | opt_conf_name = "config_with_non_required" 16 | req_conf_name = "config_with_only_required" 17 | create_config(opt_conf_name, tmp_path, True) 18 | create_config(req_conf_name, tmp_path, False) 19 | opt_conf = (tmp_path / opt_conf_name).with_suffix(".yaml") 20 | assert opt_conf.exists() 21 | req_conf = (tmp_path / req_conf_name).with_suffix(".yaml") 22 | assert req_conf.exists() 23 | assert len(opt_conf.read_text()) > len(req_conf.read_text()) 24 | 25 | 26 | @pytest.mark.usefixtures("mock_env", "load_yaml_file_clear_cache", "clear_kpops_config") 27 | def test_init_project_exclude_optional(tmp_path: Path, snapshot: Snapshot): 28 | req_path = tmp_path / "req" 29 | req_path.mkdir() 30 | kpops.init(req_path, config_include_optional=False) 31 | snapshot.assert_match(Path(req_path / "config.yaml").read_text(), "config.yaml") 32 | snapshot.assert_match(Path(req_path / "pipeline.yaml").read_text(), "pipeline.yaml") 33 | snapshot.assert_match(Path(req_path / "defaults.yaml").read_text(), "defaults.yaml") 34 | 35 | 36 | def test_init_project_include_optional(tmp_path: Path, snapshot: Snapshot): 37 | opt_path = tmp_path / "opt" 38 | opt_path.mkdir() 39 | kpops.init(opt_path, config_include_optional=True) 40 | snapshot.assert_match(Path(opt_path / "config.yaml").read_text(), "config.yaml") 41 | snapshot.assert_match(Path(opt_path / "pipeline.yaml").read_text(), "pipeline.yaml") 42 | snapshot.assert_match(Path(opt_path / "defaults.yaml").read_text(), "defaults.yaml") 43 | 44 | 45 | def test_init_project_from_cli_with_bad_path(tmp_path: Path): 46 | bad_path = Path(tmp_path / "random_file.yaml") 47 | bad_path.touch() 48 | result = runner.invoke( 49 | app, 50 | [ 51 | "init", 52 | str(bad_path), 53 | ], 54 | catch_exceptions=False, 55 | ) 56 | assert result.exit_code == 2 57 | -------------------------------------------------------------------------------- /tests/pipeline/test_destroy.py: -------------------------------------------------------------------------------- 1 | from pathlib import Path 2 | from unittest.mock import AsyncMock 3 | 4 | import pytest 5 | from pytest_mock import MockerFixture 6 | from typer.testing import CliRunner 7 | 8 | from kpops.cli.main import app 9 | from kpops.components.base_components import HelmApp 10 | from kpops.components.streams_bootstrap import ProducerApp, StreamsApp 11 | 12 | runner = CliRunner() 13 | 14 | RESOURCE_PATH = Path(__file__).parent / "resources" 15 | 16 | 17 | @pytest.mark.usefixtures("mock_env", "load_yaml_file_clear_cache", "clear_kpops_config") 18 | class TestDestroy: 19 | @pytest.fixture(autouse=True) 20 | def mock_helm(self, mocker: MockerFixture) -> AsyncMock: 21 | return mocker.patch( 22 | "kpops.components.base_components.helm_app.Helm", 23 | return_value=AsyncMock(), 24 | ).return_value 25 | 26 | def test_order(self, mocker: MockerFixture): 27 | producer_app_mock_destroy = mocker.patch.object(ProducerApp, "destroy") 28 | streams_app_mock_destroy = mocker.patch.object(StreamsApp, "destroy") 29 | helm_app_mock_destroy = mocker.patch.object(HelmApp, "destroy") 30 | mock_destroy = mocker.AsyncMock() 31 | mock_destroy.attach_mock(producer_app_mock_destroy, "producer_app_mock_destroy") 32 | mock_destroy.attach_mock(streams_app_mock_destroy, "streams_app_mock_destroy") 33 | mock_destroy.attach_mock(helm_app_mock_destroy, "helm_app_mock_destroy") 34 | 35 | result = runner.invoke( 36 | app, 37 | [ 38 | "destroy", 39 | str(RESOURCE_PATH / "simple-pipeline" / "pipeline.yaml"), 40 | ], 41 | catch_exceptions=False, 42 | ) 43 | 44 | assert result.exit_code == 0, result.stdout 45 | 46 | # check called 47 | producer_app_mock_destroy.assert_called_once_with(True) 48 | streams_app_mock_destroy.assert_called_once_with(True) 49 | helm_app_mock_destroy.assert_called_once_with(True) 50 | 51 | # check reverse order 52 | assert mock_destroy.mock_calls == [ 53 | mocker.call.helm_app_mock_destroy(True), 54 | mocker.call.streams_app_mock_destroy(True), 55 | mocker.call.producer_app_mock_destroy(True), 56 | ] 57 | -------------------------------------------------------------------------------- /tests/utils/test_yaml.py: -------------------------------------------------------------------------------- 1 | from collections.abc import Mapping 2 | from textwrap import dedent 3 | from typing import Any 4 | 5 | import pytest 6 | 7 | from kpops.utils.yaml import print_yaml, substitute_nested 8 | 9 | 10 | @pytest.mark.parametrize( 11 | ("input", "substitution", "expected"), 12 | [ 13 | pytest.param( 14 | '"a": "b", "${a}": "c", "a_${b}": "d", "e": "${a_${b}}"', 15 | {"a": "b", "${a}": "c", "a_${b}": "d", "e": "${a_${b}}"}, 16 | '"a": "b", "b": "c", "a_c": "d", "e": "d"', 17 | id="requires-multiple-passes", 18 | ), 19 | pytest.param( 20 | "${a}, ${b}, ${c}, ${d}", 21 | { 22 | "a": "0", 23 | "b": "${a}", 24 | "c": "${b}", 25 | "d": "${a}", 26 | }, 27 | "0, 0, 0, 0", 28 | id="chained-references", 29 | ), 30 | ], 31 | ) 32 | def test_substitute_nested(input: str, substitution: dict[str, str], expected: str): 33 | assert substitute_nested(input, **substitution) == expected 34 | 35 | 36 | @pytest.mark.parametrize( 37 | ("data", "expected_stdout"), 38 | [ 39 | pytest.param( 40 | {"foo": "bar"}, 41 | dedent( 42 | """\ 43 | --- 44 | foo: bar 45 | 46 | """ 47 | ), 48 | ), 49 | pytest.param( 50 | {"foo": "bar\nbaz"}, 51 | dedent( 52 | """\ 53 | --- 54 | foo: |- 55 | bar 56 | baz 57 | 58 | """ 59 | ), 60 | ), 61 | pytest.param( 62 | {"foo": ["bar", "baz"]}, 63 | dedent( 64 | """\ 65 | --- 66 | foo: 67 | - bar 68 | - baz 69 | 70 | """ 71 | ), 72 | ), 73 | ], 74 | ) 75 | def test_print_yaml( 76 | capsys: pytest.CaptureFixture[str], 77 | data: Mapping[str, Any] | str, 78 | expected_stdout: str, 79 | ): 80 | print_yaml(data) 81 | captured = capsys.readouterr() 82 | assert captured.out == expected_stdout 83 | assert not captured.err 84 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-components/kubernetes-app.yaml: -------------------------------------------------------------------------------- 1 | # Base Kubernetes App 2 | - type: kubernetes-app 3 | name: kubernetes-app # required 4 | # Pipeline prefix that will prefix every component name. If you wish to not 5 | # have any prefix you can specify an empty string. 6 | prefix: ${pipeline.name}- 7 | from: # Must not be null 8 | topics: # read from topic 9 | ${pipeline.name}-input-topic: 10 | type: input # Implied when role is NOT specified 11 | ${pipeline.name}-extra-topic: 12 | role: topic-role # Implies `type` to be extra 13 | ${pipeline.name}-input-pattern-topic: 14 | type: pattern # Implied to be an input pattern if `role` is undefined 15 | ${pipeline.name}-extra-pattern-topic: 16 | type: pattern # Implied to be an extra pattern if `role` is defined 17 | role: some-role 18 | components: # read from specific component 19 | account-producer: 20 | type: input # Implied when role is NOT specified 21 | other-producer: 22 | role: some-role # Implies `type` to be extra 23 | component-as-input-pattern: 24 | type: pattern # Implied to be an input pattern if `role` is undefined 25 | component-as-extra-pattern: 26 | type: pattern # Implied to be an extra pattern if `role` is defined 27 | role: some-role 28 | # Topic(s) into which the component will write output 29 | to: 30 | topics: 31 | ${pipeline.name}-output-topic: 32 | type: output # Implied when role is NOT specified 33 | ${pipeline.name}-extra-topic: 34 | role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined 35 | ${pipeline.name}-error-topic: 36 | type: error 37 | # Currently KPOps supports Avro and JSON schemas. 38 | key_schema: key-schema # must implement SchemaProvider to use 39 | value_schema: value-schema 40 | partitions_count: 1 41 | replication_factor: 1 42 | configs: # https://kafka.apache.org/documentation/#topicconfigs 43 | cleanup.policy: compact 44 | models: # SchemaProvider is initiated with the values given here 45 | model: model 46 | namespace: namespace # required 47 | values: # required 48 | image: exampleImage # Example 49 | debug: false # Example 50 | commandLine: {} # Example 51 | -------------------------------------------------------------------------------- /tests/components/test_kubernetes_app.py: -------------------------------------------------------------------------------- 1 | from unittest.mock import MagicMock 2 | 3 | import pytest 4 | from pytest_mock import MockerFixture 5 | 6 | from kpops.component_handlers.helm_wrapper.model import ( 7 | HelmRepoConfig, 8 | ) 9 | from kpops.component_handlers.helm_wrapper.utils import create_helm_release_name 10 | from kpops.components.base_components.kubernetes_app import ( 11 | KubernetesApp, 12 | KubernetesAppValues, 13 | ) 14 | 15 | HELM_RELEASE_NAME = create_helm_release_name("${pipeline.name}-test-kubernetes-app") 16 | 17 | 18 | class KubernetesTestValues(KubernetesAppValues): 19 | foo: str 20 | 21 | 22 | class TestKubernetesApp: 23 | @pytest.fixture() 24 | def log_info_mock(self, mocker: MockerFixture) -> MagicMock: 25 | return mocker.patch("kpops.components.base_components.kubernetes_app.log.info") 26 | 27 | @pytest.fixture() 28 | def app_values(self) -> KubernetesTestValues: 29 | return KubernetesTestValues(foo="foo") 30 | 31 | @pytest.fixture() 32 | def repo_config(self) -> HelmRepoConfig: 33 | return HelmRepoConfig(repository_name="test", url="https://bakdata.com") 34 | 35 | @pytest.fixture() 36 | def kubernetes_app(self, app_values: KubernetesTestValues) -> KubernetesApp: 37 | return KubernetesApp( 38 | name="test-kubernetes-app", 39 | values=app_values, 40 | namespace="test-namespace", 41 | ) 42 | 43 | def test_should_raise_value_error_when_name_is_not_valid( 44 | self, app_values: KubernetesTestValues 45 | ): 46 | with pytest.raises( 47 | ValueError, match=r"The component name .* is invalid for Kubernetes." 48 | ): 49 | KubernetesApp( 50 | name="Not-Compatible*", 51 | values=app_values, 52 | namespace="test-namespace", 53 | ) 54 | 55 | with pytest.raises( 56 | ValueError, match=r"The component name .* is invalid for Kubernetes." 57 | ): 58 | KubernetesApp( 59 | name="snake_case*", 60 | values=app_values, 61 | namespace="test-namespace", 62 | ) 63 | 64 | assert KubernetesApp( 65 | name="valid-name", 66 | values=app_values, 67 | namespace="test-namespace", 68 | ) 69 | -------------------------------------------------------------------------------- /docs/docs/resources/pipeline-defaults/defaults-kubernetes-app.yaml: -------------------------------------------------------------------------------- 1 | # Base Kubernetes App 2 | # 3 | # Parent of: HelmApp 4 | # Child of: PipelineComponent 5 | kubernetes-app: 6 | # Pipeline prefix that will prefix every component name. If you wish to not 7 | # have any prefix you can specify an empty string. 8 | prefix: ${pipeline.name}- 9 | from: # Must not be null 10 | topics: # read from topic 11 | ${pipeline.name}-input-topic: 12 | type: input # Implied when role is NOT specified 13 | ${pipeline.name}-extra-topic: 14 | role: topic-role # Implies `type` to be extra 15 | ${pipeline.name}-input-pattern-topic: 16 | type: pattern # Implied to be an input pattern if `role` is undefined 17 | ${pipeline.name}-extra-pattern-topic: 18 | type: pattern # Implied to be an extra pattern if `role` is defined 19 | role: some-role 20 | components: # read from specific component 21 | account-producer: 22 | type: input # Implied when role is NOT specified 23 | other-producer: 24 | role: some-role # Implies `type` to be extra 25 | component-as-input-pattern: 26 | type: pattern # Implied to be an input pattern if `role` is undefined 27 | component-as-extra-pattern: 28 | type: pattern # Implied to be an extra pattern if `role` is defined 29 | role: some-role 30 | # Topic(s) into which the component will write output 31 | to: 32 | topics: 33 | ${pipeline.name}-output-topic: 34 | type: output # Implied when role is NOT specified 35 | ${pipeline.name}-extra-topic: 36 | role: topic-role # Implies `type` to be extra; Will throw an error if `type` is defined 37 | ${pipeline.name}-error-topic: 38 | type: error 39 | # Currently KPOps supports Avro and JSON schemas. 40 | key_schema: key-schema # must implement SchemaProvider to use 41 | value_schema: value-schema 42 | partitions_count: 1 43 | replication_factor: 1 44 | configs: # https://kafka.apache.org/documentation/#topicconfigs 45 | cleanup.policy: compact 46 | models: # SchemaProvider is initiated with the values given here 47 | model: model 48 | namespace: namespace # required 49 | values: # required 50 | image: exampleImage # Example 51 | debug: false # Example 52 | commandLine: {} # Example 53 | -------------------------------------------------------------------------------- /docs/docs/user/what-is-kpops.md: -------------------------------------------------------------------------------- 1 | # What is KPOps? 2 | 3 | With a couple of easy commands in the shell, and a [`pipeline.yaml`](#example) of under 30 lines, KPOps can not only [`deploy`](./references/cli-commands.md#kpops-deploy) a Kafka pipeline[^1] to a Kubernetes cluster, but also [`reset`](./references/cli-commands.md#kpops-reset), [`clean`](./references/cli-commands.md#kpops-clean) or [`destroy`](./references/cli-commands.md#kpops-destroy) it! 4 | 5 | [^1]: A Kafka pipeline can consist of consecutive [streaming applications](./core-concepts/components/streams-app.md), [producers](./core-concepts/components/producer-app.md), and [connectors](./core-concepts/components/kafka-connector.md). 6 | 7 | ## Key features 8 | 9 | - **Deploy Kafka apps to Kubernetes**: KPOps allows to deploy consecutive Kafka Streams applications and producers using an easy-to-read and -write pipeline definition. 10 | - **Manage Kafka Connectors**: KPOps connects with your Kafka Connect cluster and deploys, validates, and deletes your connectors. 11 | - **Configure multiple pipelines and steps**: KPOps has various abstractions that simplify configuring multiple pipelines and steps within pipelines by sharing common configuration between different components, such as producers or streaming applications. 12 | - **Handle your topics and schemas**: KPOps not only creates and deletes your topics but also registers and deletes your schemas. 13 | - **Clean termination of Kafka components**: KPOps removes your pipeline components (i.e., Kafka Streams applications) from the Kubernetes cluster _and_ cleans up the component-related states (i.e., removing/resetting offset of Kafka consumer groups). 14 | - **Preview your pipeline changes**: With the KPOps dry-run, you can ensure your pipeline definition is set up correctly. This helps to minimize downtime and prevent potential errors or issues that could impact your production environment. 15 | 16 | ## Example 17 | 18 |
19 | ![atm-fraud-pipeline](../images/word-count-pipeline_streams-explorer.png) 20 |
An overview of Word-count pipeline shown in Streams Explorer
21 |
22 | 23 | ```yaml title="Word-count pipeline.yaml" 24 | --8<-- 25 | https://raw.githubusercontent.com/bakdata/kpops-examples/main/word-count/pipeline.yaml 26 | --8<-- 27 | ``` 28 | --------------------------------------------------------------------------------