├── .gitignore ├── README.md └── kfp-cloudbuild ├── README.md ├── build ├── Dockerfile ├── build.sh └── cloudbuild.yaml ├── components ├── my_add │ ├── Dockerfile │ ├── component.yaml │ ├── my_add.py │ └── test_my_add.py ├── my_divide │ ├── Dockerfile │ ├── component.yaml │ ├── my_divide.py │ └── test_my_divide.py └── tests.sh ├── notebooks └── interactive.ipynb ├── pipeline ├── helper.py ├── settings.yaml └── workflow.py └── resources ├── .DS_Store ├── cloudbuild-exec.png ├── cloudbuild-permission.png └── cloudbuild-steps.png /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | develop-eggs/ 12 | dist/ 13 | downloads/ 14 | eggs/ 15 | .eggs/ 16 | lib/ 17 | lib64/ 18 | parts/ 19 | sdist/ 20 | var/ 21 | wheels/ 22 | *.egg-info/ 23 | .installed.cfg 24 | *.egg 25 | MANIFEST 26 | 27 | # PyInstaller 28 | # Usually these files are written by a python script from a template 29 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 30 | *.manifest 31 | *.spec 32 | 33 | # Installer logs 34 | pip-log.txt 35 | pip-delete-this-directory.txt 36 | 37 | # Unit test / coverage reports 38 | htmlcov/ 39 | .tox/ 40 | .coverage 41 | .coverage.* 42 | .cache 43 | nosetests.xml 44 | coverage.xml 45 | *.cover 46 | .hypothesis/ 47 | .pytest_cache/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | db.sqlite3 57 | 58 | # Flask stuff: 59 | instance/ 60 | .webassets-cache 61 | 62 | # Scrapy stuff: 63 | .scrapy 64 | 65 | # Sphinx documentation 66 | docs/_build/ 67 | 68 | # PyBuilder 69 | target/ 70 | 71 | # Jupyter Notebook 72 | .ipynb_checkpoints 73 | 74 | # pyenv 75 | .python-version 76 | 77 | # celery beat schedule file 78 | celerybeat-schedule 79 | 80 | # SageMath parsed files 81 | *.sage.py 82 | 83 | # Environments 84 | .env 85 | .venv 86 | env/ 87 | venv/ 88 | ENV/ 89 | env.bak/ 90 | venv.bak/ 91 | 92 | # Spyder project settings 93 | .spyderproject 94 | .spyproject 95 | 96 | # Rope project settings 97 | .ropeproject 98 | 99 | # mkdocs documentation 100 | /site 101 | 102 | # mypy 103 | .mypy_cache/ 104 | .idea/ 105 | .DS_Store -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Kubeflow Examples 2 | This repository includes examples for Kubeflow and Kubeflow pipelines. 3 | -------------------------------------------------------------------------------- /kfp-cloudbuild/README.md: -------------------------------------------------------------------------------- 1 | # A Simple CI/CD Example with Kubeflow Pipelines and Cloud Build 2 | 3 | This repository shows a simple example for Kubeflow Pipelines (KFP) and 4 | Cloud Build. The aim from this example is to: 5 | 1. Create KFP components with inputs and outputs. 6 | 2. Show how to **organise the repository structure** for implementing your 7 | components (source code, unit tests, Dockerfiles, specs, and pipeline workflow and settings). 8 | 2. How to **perform build and deploy** for your pipeline using Cloud Build. 9 | 10 | ## Requirements 11 | 12 | You need to have your [GCP Project](https://cloud.google.com/resource-manager/docs/creating-managing-projects). 13 | You can use [Cloud Shell](https://cloud.google.com/shell/docs/quickstart) 14 | or [gcloud CLI](https://cloud.google.com/sdk/) to run all the commands in this 15 | guideline. 16 | 17 | ## Setup 18 | 19 | ### 1. GCP project setup 20 | 21 | Follow the [instruction](https://cloud.google.com/resource-manager/docs/creating-managing-projects) and create a GCP project. 22 | Once created, enable the Dataflow API, BigQuery API in this [page](https://console.developers.google.com/apis/enabled). 23 | You can also find more details about enabling the [billing](https://cloud.google.com/billing/docs/how-to/modify-project?#enable-billing). 24 | 25 | We recommend to use CloudShell from the GCP console to run the below commands. 26 | CloudShell starts with an environment already logged in to your account and set 27 | to the currently selected project. The following commands are required only in a 28 | workstation shell environment, they are not needed in the CloudShell. 29 | 30 | ```bash 31 | gcloud auth login 32 | gcloud auth application-default login 33 | gcloud config set project [your-project-id] 34 | gcloud 35 | ``` 36 | 37 | ### 2. Kubeflow Setup 38 | You can deploy a Kubeflow instance on Google Cloud Platform (GCP) 39 | by following the [instructions](https://www.kubeflow.org/docs/gke/deploy/) 40 | in Kubeflow documentations. 41 | 42 | ### 3. Cloud Build Setup 43 | Cloud Build executes builds with the permissions granted to the Cloud Build 44 | service account tied to the project. 45 | You need to grant the service account the [Kubernetes Engine Developer](https://console.cloud.google.com/iam-admin/roles/details/roles**. In this example, 60 | there are two components: [my_add](components/my_add), 61 | and [my_divide](components/my_divide). 62 | 63 | Each component directory includes the following files: 64 | * **[component-name].py**: This includes the source code of the components. 65 | For example, [my_add.py](components/my_add/my_add.py). 66 | * **test_[component-name].py**: This includes the unit tests for the functions 67 | in the component source code. For example, [my_add.py](components/my_add/may_add.py). 68 | * **Dockerfile**: This defines the container image to run the component code. 69 | For example, see the [Dockerfile](components/my_add/Dockerfile) for the 70 | my_add component. 71 | * **component.yaml**: This defines the [component specs](https://www.kubeflow.org/docs/pipelines/reference/component-spec/) 72 | for the pipeline. The specs includes component **inputs**, **outputs**, and **image**. 73 | For example, see the [component.yaml](components/my_add/component.yaml) for the 74 | my_add component. 75 | 76 | In addition, the [test.sh](components/tests.sh) includes the scripts to run 77 | unit test modules in all the components. 78 | 79 | ### 2. [pipeline](pipeline) 80 | 81 | The pipeline directory includes the following fies: 82 | * [workflow.py](pipeline/workflow.py): This includes the definition of the Kubeflow Pipeline. 83 | More specifically: 84 | 1. Component ops are created from the **component.yaml** files, 85 | using the [kfp.components.ComponentStore](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.components.html#kfp.components.ComponentStore) 86 | API. 87 | 2. A pipeline function is created, with **@dsl_pipline** decorator, 88 | where the component ops are executed and chained together to compose the workflow of the pipeline. 89 | * [settings.yaml](pipeline/settings.yaml): This includes are the **parameter values** 90 | for the pipeline to run with. 91 | * [helper.py](pipeline/helper.py): This implements the following functionality: 92 | 1. **update_component_spec()**: This change the **image** 93 | property of all of the **component.yaml** with respect to the given docker 94 | container image repository and image tag. This is executed when new container 95 | image are created, as part of the build process, and new tags are generated. 96 | 2. **read_settings()**: This reads all the parameter values from the **settings.yaml** 97 | file, to invoke the pipeline run with. 98 | 3. **run_pipeline()**: This creates and experiment and run a given pipeline 99 | package using the [kfp.Client](https://kubeflow-pipelines.readthedocs.io/en/latest/source/kfp.client.html) 100 | API. 101 | ### 3. [build](build) 102 | The build directory includes the following files: 103 | 1. [Dockerfile](build/Dockerfile): This is a definition for the **kfp-util** container image required 104 | to deploy and run the Kubeflow pipelines: 105 | 1. It is based on [gcr.io/cloud-builders/kubectl](https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/kubectl) 106 | image, which has the [gcloud SDK](https://cloud.google.com/sdk/install) installed and sets the **kubeconfig**. 107 | 2. It installs **Python 3.6.8**. 108 | 3. It installs the [Kubeflow Pipeline SDK](https://www.kubeflow.org/docs/pipelines/sdk/install-sdk/) and other required Python packages. 109 | 2. [cloudbuild.yaml](build/cloudbuild.yaml): This is the Cloud Build configuration file 110 | the defines the various build steps. This file is designed for [manual execution](https://cloud.google.com/cloud-build/docs/running-builds/start-build-manually). 111 | 3. [build.sh](build/build.sh): This is the entry point to execute the build. 112 | 113 | ### 4. [notebooks](notebooks) 114 | This directory includes the notebook(s) needed for interactive testing 115 | and experimentation of the pipeline components, as well as submitting 116 | the pipeline to KFP 117 | 118 | ## Cloud Build Steps 119 | 120 | The following diagram shows the build steps: 121 | 122 | ![Cloud Build Execution](resources/cloudbuild-steps.png) 123 | 124 | 1. **Clone Repo**: When executing the [build.sh](build/build.sh) script with the necessary 125 | substitutions, the first step is to clone the code repository to the Cloud Build 126 | local environment. 127 | * This step uses [gcr.io/cloud-builders/git](https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/git) image. 128 | 2. **Run Unit Tests**: This step executes the [tests.sh](components/tests.sh) 129 | script that run the unit test for the various components. 130 | * This step uses the custom **kfp-util** image defines in this [Dockefile](build/Dockerfile). 131 | 3. **Build Component Images**: In this example, we build Docker container images 132 | for two components: [my_add](components/my_add/Dockerfile) 133 | and [my_divide](components/my_divide/Dockerfile), 134 | and tag with the **${_TAG}** value passed in the substitutions. 135 | * This step uses [gcr.io/cloud-builders/docker](https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/docker) image. 136 | 4. **Push Images to Registry**: This is performed automatically when you define 137 | the **images** section in the [cloudbuild.yaml](build/cloudbuild.yaml) file. 138 | 139 | 5. **Update Component Specs**: This executes the **update_component_specs** method 140 | in the [helper.py](pipeline/helper.py) module. The method uses the recently created and tagged image urls 141 | and update them to the **component.yaml** files. Note that these component.yaml specs 142 | are loaded - with the updated specs - by the pipeline before it is complied. 143 | * This step uses the custom **kfp-util** image defines in this [Dockefile](build/Dockerfile). 144 | 145 | 6. **Compile Pipeline**: This executes **dsl-compile** to compile the pipeline 146 | defined in [workflow.py](pipeline/workflow.py), to generate **pipeline.tar.gz** package. 147 | * This step uses the custom **kfp-util** image defines in this [Dockefile](build/Dockerfile). 148 | 149 | 7. **Upload to GCS**: This uploads the **pipeline.tar.gz** package to GCS. 150 | * This step uses [gcr.io/cloud-builders/gsutil](https://github.com/GoogleCloudPlatform/cloud-builders/tree/master/gsutil) image. 151 | 152 | 8. **Deploy & Run Pipeline**: This executes the **deploy_pipeline** method 153 | in the [helper.py](pipeline/helper.py) module, given the **pipeline.tar.gz** package and pipeline **version**. 154 | The method deploys the pipeline package to KFP. If an optional **--run** argument is passed, 155 | the method also creates and experiment in KFP, and run pipeline given the 156 | created experiment, compiled pipeline package, and loaded parameter values from the [settings.yaml](pipeline/settings.yaml) file. 157 | * This step uses the custom **kfp-util** image defines in this [Dockefile](build/Dockerfile). 158 | 159 | ## Executing Cloud Build 160 | 161 | To run the build routine defined in the [build/cloudbuild.yaml](build/cloudbuild.yaml) file, 162 | you can execute the [build/build.sh](build/build.sh) script file, 163 | after setting the substitution values. 164 | 165 | | Substitution | Description | 166 | |-----------------|--------------| 167 | | _REPO_URL | Your source code repository | 168 | | _PROJECT_ID | GCP project Id where you container registry lives | 169 | | _COMPUTE_ZONE | Compute zone of your GKE cluster of your Kubeflow deployment | 170 | | _CLUSTER_NAME | Your GKE cluster name | 171 | | _GCS_LOCATION | The GCS location where the compiled pipeline is uploaded | 172 | | _EXPERIMENT_NAME| The name of the experiment to use for running the pipeline | 173 | | _TAG | The label to use to tag the component images when built | 174 | 175 | The folowing screenshot shows the output of a build process: 176 | 177 | ![Cloud Build Execution](resources/cloudbuild-exec.png) 178 | -------------------------------------------------------------------------------- /kfp-cloudbuild/build/Dockerfile: -------------------------------------------------------------------------------- 1 | #FROM python:3.6-slim-jessie 2 | 3 | FROM gcr.io/cloud-builders/kubectl 4 | 5 | #RUN apt update && \ 6 | # apt-get install -y python3 python-dev python3-dev build-essential \ 7 | # libssl-dev libffi-dev libxml2-dev libxslt1-dev zlib1g-dev python-pip && \ 8 | # apt install -y python3-pip 9 | 10 | ### Install Python 3.6 ####################################################### 11 | # Taken from python:3.6-slim-jessie 12 | # https://github.com/docker-library/python/blob/master/3.6/jessie/slim/Dockerfile 13 | 14 | # ensure local python is preferred over distribution python 15 | ENV PATH /usr/local/bin:$PATH 16 | 17 | # http://bugs.python.org/issue19846 18 | # > At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK. 19 | ENV LANG C.UTF-8 20 | 21 | # runtime dependencies 22 | RUN apt-get update && apt-get install -y --no-install-recommends \ 23 | ca-certificates \ 24 | netbase \ 25 | && rm -rf /var/lib/apt/lists/* 26 | 27 | ENV GPG_KEY 0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D 28 | ENV PYTHON_VERSION 3.6.8 29 | 30 | RUN set -ex \ 31 | \ 32 | && savedAptMark="$(apt-mark showmanual)" \ 33 | && apt-get update && apt-get install -y --no-install-recommends \ 34 | dpkg-dev \ 35 | gcc \ 36 | libbz2-dev \ 37 | libc6-dev \ 38 | libexpat1-dev \ 39 | libffi-dev \ 40 | libgdbm-dev \ 41 | liblzma-dev \ 42 | libncursesw5-dev \ 43 | libreadline-dev \ 44 | libsqlite3-dev \ 45 | libssl-dev \ 46 | make \ 47 | tk-dev \ 48 | wget \ 49 | xz-utils \ 50 | zlib1g-dev \ 51 | # as of Stretch, "gpg" is no longer included by default 52 | $(command -v gpg > /dev/null || echo 'gnupg dirmngr') \ 53 | \ 54 | && wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz" \ 55 | && wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc" \ 56 | && export GNUPGHOME="$(mktemp -d)" \ 57 | && gpg --batch --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEY" \ 58 | && gpg --batch --verify python.tar.xz.asc python.tar.xz \ 59 | && { command -v gpgconf > /dev/null && gpgconf --kill all || :; } \ 60 | && rm -rf "$GNUPGHOME" python.tar.xz.asc \ 61 | && mkdir -p /usr/src/python \ 62 | && tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \ 63 | && rm python.tar.xz \ 64 | \ 65 | && cd /usr/src/python \ 66 | && gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \ 67 | && ./configure \ 68 | --build="$gnuArch" \ 69 | --enable-loadable-sqlite-extensions \ 70 | --enable-shared \ 71 | --with-system-expat \ 72 | --with-system-ffi \ 73 | --without-ensurepip \ 74 | && make -j "$(nproc)" \ 75 | && make install \ 76 | && ldconfig \ 77 | \ 78 | && apt-mark auto '.*' > /dev/null \ 79 | && apt-mark manual $savedAptMark \ 80 | && find /usr/local -type f -executable -not \( -name '*tkinter*' \) -exec ldd '{}' ';' \ 81 | | awk '/=>/ { print $(NF-1) }' \ 82 | | sort -u \ 83 | | xargs -r dpkg-query --search \ 84 | | cut -d: -f1 \ 85 | | sort -u \ 86 | | xargs -r apt-mark manual \ 87 | && apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \ 88 | && rm -rf /var/lib/apt/lists/* \ 89 | \ 90 | && find /usr/local -depth \ 91 | \( \ 92 | \( -type d -a \( -name test -o -name tests \) \) \ 93 | -o \ 94 | \( -type f -a \( -name '*.pyc' -o -name '*.pyo' \) \) \ 95 | \) -exec rm -rf '{}' + \ 96 | && rm -rf /usr/src/python \ 97 | \ 98 | && python3 --version 99 | 100 | # make some useful symlinks that are expected to exist 101 | RUN cd /usr/local/bin \ 102 | && ln -s idle3 idle \ 103 | && ln -s pydoc3 pydoc \ 104 | && ln -s python3 python \ 105 | && ln -s python3-config python-config 106 | 107 | # if this is called "PIP_VERSION", pip explodes with "ValueError: invalid truth value ''" 108 | ENV PYTHON_PIP_VERSION 19.1.1 109 | 110 | RUN set -ex; \ 111 | \ 112 | savedAptMark="$(apt-mark showmanual)"; \ 113 | apt-get update; \ 114 | apt-get install -y --no-install-recommends wget; \ 115 | \ 116 | wget -O get-pip.py 'https://bootstrap.pypa.io/get-pip.py'; \ 117 | \ 118 | apt-mark auto '.*' > /dev/null; \ 119 | [ -z "$savedAptMark" ] || apt-mark manual $savedAptMark; \ 120 | apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \ 121 | rm -rf /var/lib/apt/lists/*; \ 122 | \ 123 | python get-pip.py \ 124 | --disable-pip-version-check \ 125 | --no-cache-dir \ 126 | "pip==$PYTHON_PIP_VERSION" \ 127 | ; \ 128 | pip --version; \ 129 | \ 130 | find /usr/local -depth \ 131 | \( \ 132 | \( -type d -a \( -name test -o -name tests \) \) \ 133 | -o \ 134 | \( -type f -a \( -name '*.pyc' -o -name '*.pyo' \) \) \ 135 | \) -exec rm -rf '{}' +; \ 136 | rm -f get-pip.py 137 | 138 | 139 | ### Install the required Python packages ###################################### 140 | 141 | RUN pip3 install fire pyyaml pathlib 142 | RUN pip3 install --upgrade kfp 143 | 144 | ############################################################################### 145 | 146 | 147 | -------------------------------------------------------------------------------- /kfp-cloudbuild/build/build.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # Copyright 2019 Google LLC 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | # Create the kfp-util docker container image 17 | docker build -t gcr.io/ml-cicd-template/kfp-util:latest . 18 | gcloud docker -- push gcr.io/ml-cicd-template/kfp-util:latest 19 | 20 | # Set substitutions 21 | SUBSTITUTIONS=\ 22 | _REPO_URL='https://github.com/ksalama/kubeflow-examples.git',\ 23 | _PROJECT_ID='ml-cicd-template',\ 24 | _COMPUTE_ZONE='europe-west1-b',\ 25 | _CLUSTER_NAME='kubeflow-cluster',\ 26 | _GCS_LOCATION='ml-cicd-template/helloworld/pipelines',\ 27 | _EXPERIMENT_NAME='helloworld-dev',\ 28 | _TAG='latest' 29 | 30 | # Submit the build job 31 | gcloud builds submit --no-source --config cloudbuild.yaml \ 32 | --substitutions $SUBSTITUTIONS -------------------------------------------------------------------------------- /kfp-cloudbuild/build/cloudbuild.yaml: -------------------------------------------------------------------------------- 1 | steps: 2 | 3 | # Clone repo to Cloud Build environment 4 | - name: 'gcr.io/cloud-builders/git' 5 | args: ['clone', 6 | '${_REPO_URL}', '.', 7 | '--depth', '1', 8 | '--verbose'] 9 | id: 'Clone Repo' 10 | 11 | # Run unit tests 12 | - name: 'python:3.6-slim-jessie' 13 | entrypoint: 'bash' 14 | args: ['components/tests.sh'] 15 | dir: 'kfp-cloudbuild' 16 | id: 'Run Unit Tests' 17 | 18 | # Create the image for the my_add component 19 | - name: 'gcr.io/cloud-builders/docker' 20 | args: ['build', 21 | '-t', 'gcr.io/$_PROJECT_ID/my_add:$_TAG', '.'] 22 | dir: 'kfp-cloudbuild/components/my_add' 23 | id: 'Build my_add Image' 24 | waitFor: ['Run Unit Tests'] 25 | 26 | # Create the image for the my_divide component 27 | - name: 'gcr.io/cloud-builders/docker' 28 | args: ['build', 29 | '-t', 'gcr.io/$_PROJECT_ID/my_divide:$_TAG', '.'] 30 | dir: 'kfp-cloudbuild/components/my_divide' 31 | id: 'Build my_divide Image' 32 | waitFor: ['Run Unit Tests'] 33 | 34 | # Update component specs images 35 | - name: 'gcr.io/ml-cicd-template/kfp-util:latest' 36 | entrypoint: 'python3' 37 | args: ['pipeline/helper.py','update-specs', 38 | '--repo_url', 'gcr.io/${_PROJECT_ID}', 39 | '--image_tage', '{_TAG}'] 40 | dir: 'kfp-cloudbuild' 41 | id: 'Update Component Spec Images' 42 | 43 | # Compile pipeline 44 | - name: 'gcr.io/ml-cicd-template/kfp-util:latest' 45 | entrypoint: 'dsl-compile' 46 | args: ['--py', 'pipeline/workflow.py', 47 | '--output', 'pipeline/pipeline.tar.gz', 48 | '--disable-type-check'] 49 | dir: 'kfp-cloudbuild' 50 | id: 'Compile Pipeline' 51 | 52 | # Upload compiled pipeline to GCS 53 | - name: 'gcr.io/cloud-builders/gsutil' 54 | args: ['cp', 'pipeline.tar.gz', 'settings.yaml', 'gs://${_GCS_LOCATION}/${_TAG}/'] 55 | dir: 'kfp-cloudbuild/pipeline' 56 | id: 'Upload Pipeline to GCS' 57 | 58 | # Deploy pipeline in KFP 59 | - name: 'gcr.io/ml-cicd-template/kfp-util:latest' 60 | entrypoint: '/bin/sh' 61 | args: ['-c', '/builder/kubectl.bash; 62 | python3 helper.py deploy-pipeline 63 | --package_path=pipeline.tar.gz 64 | --version=\"$_TAG\" 65 | --experiment=\"$_EXPERIMENT_NAME\" 66 | --run'] 67 | dir: 'kfp-cloudbuild/pipeline' 68 | env: 69 | - 'CLOUDSDK_COMPUTE_ZONE=${_COMPUTE_ZONE}' 70 | - 'CLOUDSDK_CONTAINER_CLUSTER=${_CLUSTER_NAME}' 71 | id: 'Deploy & Run Pipeline' 72 | waitFor: ['Compile Pipeline'] 73 | 74 | images: 75 | - 'gcr.io/${_PROJECT_ID}/my_add:${_TAG}' 76 | - 'gcr.io/${_PROJECT_ID}/my_divide:${_TAG}' -------------------------------------------------------------------------------- /kfp-cloudbuild/components/my_add/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3 2 | 3 | COPY my_add.py . -------------------------------------------------------------------------------- /kfp-cloudbuild/components/my_add/component.yaml: -------------------------------------------------------------------------------- 1 | description: Adds to float values 2 | implementation: 3 | container: 4 | command: ["python", "/my_add.py"] 5 | args: 6 | - --x-value 7 | - inputValue: x-value 8 | - --y-value 9 | - inputValue: y-value 10 | - --result-path 11 | - outputPath: sum 12 | image: gcr.io 13 | inputs: 14 | - default: 0 15 | description: Input x value 16 | name: x-value 17 | type: Integer 18 | - default: 0 19 | description: Input y value 20 | name: y-value 21 | type: Integer 22 | outputs: 23 | - name: sum 24 | name: my_add 25 | -------------------------------------------------------------------------------- /kfp-cloudbuild/components/my_add/my_add.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """My_add component source code.""" 15 | 16 | import argparse 17 | from pathlib import Path 18 | 19 | 20 | def add(x: int, y: int) -> int: 21 | """Returns the sum of x and y.""" 22 | result = x + y 23 | return result 24 | 25 | 26 | def main(args): 27 | x = int(args.x_value) 28 | y = int(args.y_value) 29 | result = add(x, y) 30 | print("Result: {}".format(result)) 31 | 32 | # Write output to file 33 | Path(args.result_path).parent.mkdir(parents=True, exist_ok=True) 34 | Path(args.result_path).write_text(str(result)) 35 | 36 | 37 | 38 | if __name__ == "__main__": 39 | parser = argparse.ArgumentParser() 40 | parser.add_argument("--x-value", type=int) 41 | parser.add_argument("--y-value", type=int) 42 | parser.add_argument("--result-path", type=str) 43 | args = parser.parse_args() 44 | main(args) -------------------------------------------------------------------------------- /kfp-cloudbuild/components/my_add/test_my_add.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """My_add component unit tests.""" 15 | 16 | import unittest 17 | import my_add 18 | 19 | 20 | class TestMyAdd(unittest.TestCase): 21 | 22 | def test_add(self): 23 | self.assertEqual(my_add.add(5, 5), 10) 24 | 25 | 26 | if __name__ == '__main__': 27 | unittest.main() 28 | -------------------------------------------------------------------------------- /kfp-cloudbuild/components/my_divide/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3 2 | 3 | COPY my_divide.py . 4 | -------------------------------------------------------------------------------- /kfp-cloudbuild/components/my_divide/component.yaml: -------------------------------------------------------------------------------- 1 | description: Divides to float values 2 | implementation: 3 | container: 4 | command: ["python", "/my_divide.py"] 5 | args: 6 | - --x-value 7 | - inputValue: x-value 8 | - --y-value 9 | - inputValue: y-value 10 | - --quotient-path 11 | - outputPath: quotient 12 | - --remainder-path 13 | - outputPath: remainder 14 | image: gcr.io/ 15 | inputs: 16 | - default: 0 17 | description: Input x value 18 | name: x-value 19 | type: Integer 20 | - default: 0 21 | description: Input y value 22 | name: y-value 23 | type: Integer 24 | outputs: 25 | - name: quotient 26 | - name: remainder 27 | name: my_divide 28 | -------------------------------------------------------------------------------- /kfp-cloudbuild/components/my_divide/my_divide.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """My_divide component source code.""" 15 | 16 | import argparse 17 | from pathlib import Path 18 | from typing import NamedTuple 19 | from collections import namedtuple 20 | 21 | 22 | def divide(x: int, y: int) -> NamedTuple('MyDivmodOutput', [('quotient', float), ('remainder', float)]): 23 | """Returns the quotient and the remainder of dividing x on y.""" 24 | quotient = int(x / y) 25 | remainder = int(x) % int(y) 26 | 27 | myDivmodOutput = namedtuple('MyDivmodOutput', ['quotient', 'remainder']) 28 | result = myDivmodOutput(quotient=quotient, remainder=remainder) 29 | 30 | return result 31 | 32 | 33 | def main(args): 34 | x = int(args.x_value) 35 | y = int(args.y_value) 36 | result = divide(x, y) 37 | print("Result: {}".format(result)) 38 | 39 | # Write output to file 40 | Path(args.quotient_path).parent.mkdir(parents=True, exist_ok=True) 41 | Path(args.quotient_path).write_text(str(result.quotient)) 42 | 43 | Path(args.remainder_path).parent.mkdir(parents=True, exist_ok=True) 44 | Path(args.remainder_path).write_text(str(result.remainder)) 45 | 46 | 47 | if __name__ == "__main__": 48 | parser = argparse.ArgumentParser() 49 | parser.add_argument("--x-value", type=int) 50 | parser.add_argument("--y-value", type=int) 51 | parser.add_argument("--quotient-path", type=str) 52 | parser.add_argument("--remainder-path", type=str) 53 | args = parser.parse_args() 54 | main(args) -------------------------------------------------------------------------------- /kfp-cloudbuild/components/my_divide/test_my_divide.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """My_divide component unit tests.""" 15 | 16 | import unittest 17 | import my_divide 18 | 19 | 20 | class TestMyDivide(unittest.TestCase): 21 | 22 | def test_divide(self): 23 | result = my_divide.divide(10, 3) 24 | self.assertEqual(result.quotient, 3) 25 | self.assertEqual(result.remainder, 1) 26 | 27 | 28 | if __name__ == '__main__': 29 | unittest.main() 30 | -------------------------------------------------------------------------------- /kfp-cloudbuild/components/tests.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # Copyright 2019 Google LLC 3 | # 4 | # Licensed under the Apache License, Version 2.0 (the "License"); 5 | # you may not use this file except in compliance with the License. 6 | # You may obtain a copy of the License at 7 | # 8 | # http://www.apache.org/licenses/LICENSE-2.0 9 | # 10 | # Unless required by applicable law or agreed to in writing, software 11 | # distributed under the License is distributed on an "AS IS" BASIS, 12 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | # See the License for the specific language governing permissions and 14 | # limitations under the License. 15 | 16 | python components/my_add/test_my_add.py 17 | python components/my_divide/test_my_divide.py -------------------------------------------------------------------------------- /kfp-cloudbuild/notebooks/interactive.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# implement you experimentation code...." 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Import the required libraries" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 1, 22 | "metadata": {}, 23 | "outputs": [], 24 | "source": [ 25 | "import kfp\n", 26 | "import kfp.dsl as dsl\n", 27 | "from datetime import datetime\n", 28 | "import kfp.compiler as compiler" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Load the pipeline components" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [ 44 | "component_store = kfp.components.ComponentStore(\n", 45 | " local_search_paths=['components'])" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "## Create pipeline ops" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "add_op = component_store.load_component('my_add')\n", 62 | "divide_op = component_store.load_component('my_divide')" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": {}, 68 | "source": [ 69 | "## Define pipeline" 70 | ] 71 | }, 72 | { 73 | "cell_type": "code", 74 | "execution_count": null, 75 | "metadata": {}, 76 | "outputs": [], 77 | "source": [ 78 | "def pipeline(\n", 79 | " x_value: int=1,\n", 80 | " y_value: int=1,\n", 81 | " z_value: int=1,\n", 82 | "):\n", 83 | "\n", 84 | " add_step = add_op(x_value=x_value, y_value=y_value)\n", 85 | " add_step.set_display_name('Add x and y')\n", 86 | " add_result = add_step.outputs\n", 87 | " sum_value = add_result['sum']\n", 88 | " is_even = sum_value != 0\n", 89 | " with kfp.dsl.Condition(is_even):\n", 90 | " divide_step = divide_op(x_value=sum_value, y_value=z_value)\n", 91 | " divide_step.set_display_name('Divide sum by z')\n", 92 | " add_step2 = add_op(\n", 93 | " x_value=divide_step.outputs['quotient'],\n", 94 | " y_value=divide_step.outputs['remainder'])\n", 95 | " add_step2.set_display_name('Add quotient and remainder')" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "metadata": {}, 101 | "source": [ 102 | "## Compile pipeline" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "metadata": {}, 109 | "outputs": [], 110 | "source": [ 111 | "compiler.Compiler().compile(\n", 112 | " pipeline, 'pipeline.tar.gz')" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "metadata": {}, 118 | "source": [ 119 | "## Run pipeline" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": {}, 126 | "outputs": [], 127 | "source": [ 128 | "args = {\n", 129 | " 'x_value': 1,\n", 130 | " 'y_value': 1,\n", 131 | " 'z_value': 2,\n", 132 | " \n", 133 | "}\n", 134 | "\n", 135 | "client = kfp.Client(namespace='kubeflow')\n", 136 | "run_id = 'run-' + datetime.now().strftime('%Y%m%d-%H%M%S')\n", 137 | "experiment = client.create_experiment(name='exp-dev')\n", 138 | "\n", 139 | "client.run_pipeline(\n", 140 | " experiment.id,\n", 141 | " job_name=run_id,\n", 142 | " pipeline_package_path=pipeline_package,\n", 143 | " params=args\n", 144 | ")" 145 | ] 146 | } 147 | ], 148 | "metadata": { 149 | "kernelspec": { 150 | "display_name": "py36-venv", 151 | "language": "python", 152 | "name": "py36-venv" 153 | }, 154 | "language_info": { 155 | "codemirror_mode": { 156 | "name": "ipython", 157 | "version": 3 158 | }, 159 | "file_extension": ".py", 160 | "mimetype": "text/x-python", 161 | "name": "python", 162 | "nbconvert_exporter": "python", 163 | "pygments_lexer": "ipython3", 164 | "version": "3.6.4" 165 | } 166 | }, 167 | "nbformat": 4, 168 | "nbformat_minor": 2 169 | } 170 | -------------------------------------------------------------------------------- /kfp-cloudbuild/pipeline/helper.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Helper module to deploy and run pipelines.""" 15 | 16 | import yaml 17 | import os 18 | import pathlib 19 | import kfp 20 | import fire 21 | from datetime import datetime 22 | 23 | SETTINGS_FILENAME = 'settings.yaml' 24 | HOST_URL = 'https://{}.endpoints.{}.cloud.goog/pipeline' 25 | OUTPUT_PACKAGE_PATH = 'pipeline.tar.gz' 26 | NAMESPACE = 'kubeflow' 27 | EXPERIMENT_NAME = 'default-experiment' 28 | PIPELINE_NAME = 'helloworld-pipeline' 29 | 30 | 31 | def update_component_spec(repo_url, image_tag): 32 | """Update the image in the component.yaml files given the repo_url and image_tag.""" 33 | 34 | for spec_path in pathlib.Path('components').glob('*/component.yaml'): 35 | spec = yaml.safe_load(pathlib.Path(spec_path).read_text()) 36 | image_name = str(spec_path).split('/')[1] 37 | full_image_name = '{}/{}:{}'.format(repo_url, image_name, image_tag) 38 | spec['implementation']['container']['image'] = full_image_name 39 | pathlib.Path(spec_path).write_text(yaml.dump(spec)) 40 | print('Component {} specs updated. Image:{}'.format( 41 | image_name, full_image_name)) 42 | 43 | def read_settings(): 44 | """Read all the parameter values from the settings.yaml file.""" 45 | settings_file = os.path.join(os.path.dirname(__file__), SETTINGS_FILENAME) 46 | flat_settings = dict() 47 | setting_sections = yaml.safe_load(pathlib.Path(settings_file).read_text()) 48 | for sections in setting_sections: 49 | flat_settings.update(setting_sections[sections]) 50 | return flat_settings 51 | 52 | 53 | def deploy_pipeline( 54 | kfp_package_path, version, experiment_name, namespace, run): 55 | """Deploy and run the givne kfp_package_path.""" 56 | 57 | pipeline_name = PIPELINE_NAME+"-"+version 58 | 59 | client = kfp.Client(namespace=namespace) 60 | 61 | pipeline = client.upload_pipeline( 62 | pipeline_package_path=kfp_package_path, 63 | pipeline_name=pipeline_name) 64 | pipeline_id = pipeline.id 65 | 66 | if run: 67 | run_id = 'run-' + datetime.now().strftime('%Y%m%d-%H%M%S') 68 | experiment = client.create_experiment(name=experiment_name) 69 | settings=read_settings() 70 | client.run_pipeline( 71 | experiment.id, 72 | job_name=run_id, 73 | pipeline_id=pipeline_id, 74 | params=settings) 75 | 76 | 77 | def main(operation, **args): 78 | 79 | # Update Component Specs 80 | if operation == 'update-specs': 81 | print('Setting images to the component spec...') 82 | if 'repo_url' not in args: 83 | raise ValueError('repo_url has to be supplied.') 84 | repo_url = args['repo_url'] 85 | image_tag = "latest" 86 | if 'image_tag' in args: 87 | image_tag = args['image_tage'] 88 | update_component_spec(repo_url, image_tag) 89 | 90 | # Deploy Pipeline 91 | elif operation == 'deploy-pipeline': 92 | print('Running Kubeflow pipeline...') 93 | if 'package_path' not in args: 94 | raise ValueError('package_path has to be supplied.') 95 | package_path = args['package_path'] 96 | 97 | if 'version' not in args: 98 | raise ValueError('version has to be supplied.') 99 | version = args['version'] 100 | 101 | namespace = NAMESPACE 102 | if 'namespace' in args: 103 | namespace = args['namespace'] 104 | 105 | experiment_name = EXPERIMENT_NAME 106 | if 'experiment' in args: 107 | experiment_name = args['experiment'] 108 | 109 | run = 'run' in args 110 | 111 | deploy_pipeline(package_path, version, experiment_name, namespace, run) 112 | 113 | else: 114 | raise ValueError( 115 | 'Invalid operation name: {}. Valid operations: update-specs | deploy-pipeline'.format(operation)) 116 | 117 | 118 | if __name__ == '__main__': 119 | fire.Fire(main) 120 | -------------------------------------------------------------------------------- /kfp-cloudbuild/pipeline/settings.yaml: -------------------------------------------------------------------------------- 1 | arguments: 2 | x_value: 10 3 | y_value: 10 4 | z_value: 3 5 | -------------------------------------------------------------------------------- /kfp-cloudbuild/pipeline/workflow.py: -------------------------------------------------------------------------------- 1 | # Copyright 2019 Google LLC 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | """Pipeline workflow definition.""" 15 | 16 | import kfp 17 | import kfp.dsl as dsl 18 | 19 | # Initialize component store 20 | component_store = kfp.components.ComponentStore( 21 | local_search_paths=['components']) 22 | 23 | # Create component factories 24 | add_op = component_store.load_component('my_add') 25 | divide_op = component_store.load_component('my_divide') 26 | 27 | # Define pipeline 28 | @dsl.pipeline( 29 | name='A Simple CI pipeline', 30 | description='Basic sample to show how to do CI with KFP using CloudBuild' 31 | ) 32 | def pipeline( 33 | x_value: int=1, 34 | y_value: int=1, 35 | z_value: int=1, 36 | ): 37 | 38 | add_step = add_op(x_value=x_value, y_value=y_value) 39 | add_step.set_display_name('Add x and y') 40 | add_result = add_step.outputs 41 | sum_value = add_result['sum'] 42 | with kfp.dsl.Condition(sum_value != 0): 43 | divide_step = divide_op(x_value=sum_value, y_value=z_value) 44 | divide_step.set_display_name('Divide sum by z') 45 | add_step2 = add_op( 46 | x_value=divide_step.outputs['quotient'], 47 | y_value=divide_step.outputs['remainder']) 48 | add_step2.set_display_name('Add quotient and remainder') 49 | 50 | 51 | 52 | -------------------------------------------------------------------------------- /kfp-cloudbuild/resources/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ksalama/kubeflow-examples/f20a8167d37588bd60545f8f950903e5bc64e60a/kfp-cloudbuild/resources/.DS_Store -------------------------------------------------------------------------------- /kfp-cloudbuild/resources/cloudbuild-exec.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ksalama/kubeflow-examples/f20a8167d37588bd60545f8f950903e5bc64e60a/kfp-cloudbuild/resources/cloudbuild-exec.png -------------------------------------------------------------------------------- /kfp-cloudbuild/resources/cloudbuild-permission.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ksalama/kubeflow-examples/f20a8167d37588bd60545f8f950903e5bc64e60a/kfp-cloudbuild/resources/cloudbuild-permission.png -------------------------------------------------------------------------------- /kfp-cloudbuild/resources/cloudbuild-steps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ksalama/kubeflow-examples/f20a8167d37588bd60545f8f950903e5bc64e60a/kfp-cloudbuild/resources/cloudbuild-steps.png --------------------------------------------------------------------------------