├── .gitignore ├── README.md ├── agent ├── Dockerfile └── docker-compose.yml ├── agent_docker ├── Dockerfile └── docker-compose.yml ├── client ├── Dockerfile ├── app │ ├── .prefectignore │ └── weather.py └── docker-compose.yml ├── client_docker ├── Dockerfile ├── app │ ├── .prefectignore │ └── weather.py ├── docker-compose.yml ├── entrypoint.sh ├── execution.Dockerfile └── registry │ ├── .gitignore │ └── auth │ └── .gitkeep ├── client_s3 ├── .env ├── Dockerfile ├── app │ ├── .prefectignore │ └── weather.py └── docker-compose.yml ├── docker_interface.png ├── prefect_schema_principle.drawio ├── prefect_schema_principle.jpg └── server ├── .env ├── Dockerfile └── docker-compose.yml /.gitignore: -------------------------------------------------------------------------------- 1 | .venv 2 | .vscode 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Prefect - Docker Compose 2 | 3 | A simple guide to understand and make Prefect **2.x** work with your own docker-compose configuration. 4 | 5 | Interested in examples for Prefect **1.x** ? Switch to the [last 1.x branch of this repo](https://github.com/flavienbwk/prefect-docker-compose/tree/e758a498d5819550a9b926b0bf9bb4e9c85574d1). 6 | 7 | This allows you to package your Prefect instance for fully-containerized environments (e.g: docker-compose, Kubernetes) or offline use. 8 | 9 | ![Operating principle of Prefect](./prefect_schema_principle.jpg) 10 | 11 | - [Prefect - Docker Compose](#prefect---docker-compose) 12 | - [Run the server](#run-the-server) 13 | - [Run one or multiple agents](#run-one-or-multiple-agents) 14 | - [Run your first flow via the Prefect API](#run-your-first-flow-via-the-prefect-api) 15 | - [Principles to understand](#principles-to-understand) 16 | - [Flow with Local storage (easiest)](#flow-with-local-storage-easiest) 17 | - [Flow with S3 Storage (recommended)](#flow-with-s3-storage-recommended) 18 | - [Flow with Docker storage](#flow-with-docker-storage) 19 | - [Preparing the Registry](#preparing-the-registry) 20 | - [Start the Docker in Docker agent](#start-the-docker-in-docker-agent) 21 | - [Registering the flow](#registering-the-flow) 22 | 23 | ## Run the server 24 | 25 | 1. Optionally open and edit the [`server/.env`](./server/.env) file 26 | 27 | :information_source: All `PREFECT_*` env variables can be [found in the Prefect settings.py file](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/settings.py#L238). 28 | 29 | 2. Start the server : 30 | 31 | ```bash 32 | cd server && docker-compose up --build -d && cd - 33 | ``` 34 | 35 | 3. Access the Orion UI at [localhost:4200](http://localhost:4200) 36 | 37 | ## Run one or multiple agents 38 | 39 | Agents are services that run your scheduled flows. 40 | 41 | 1. Optionally open and edit the [`agent/docker-compose.yml`](./agent/docker-compose.yml) file. 42 | 43 | > :information_source: In each `docker-compose.yml`, you will find the `PREFECT_API_URL` env variable including the `172.17.0.1` IP address. This is the IP of the Docker daemon on which are exposed all exposed ports of your containers. This allows containers launched from different docker-compose networks to communicate. Change it if yours is different (check your daemon IP by typing `ip a | grep docker0`). 44 | > 45 | > ![Docker interface IP](./docker_interface.png) 46 | > 47 | > Here, mine is `192.168.254.1` but the default is generally to `172.17.0.1`. 48 | 49 | 2. Start the agent : 50 | 51 | ```bash 52 | docker-compose -f agent/docker-compose.yml up -d 53 | ``` 54 | 55 | > :information_source: You can run the agent on another machine than the one with the Prefect server. Edit the `PREFECT_API_URL` env variable for that. 56 | 57 | Maybe you want to instanciate multiple agents ? 58 | 59 | ```bash 60 | docker-compose -f agent/docker-compose.yml up -d --scale agent=3 agent 61 | ``` 62 | 63 | 3. Our agents are now starting listening the Orion server on the `flows-example-queue` queue ([see the `--work-queue` option](./agent/docker-compose.yml#L7)). 64 | 65 | ## Run your first flow via the Prefect API 66 | 67 | ### Principles to understand 68 | 69 | > :speech_balloon: [Execution in your cloud; orchestration in ours](https://medium.com/the-prefect-blog/the-prefect-hybrid-model-1b70c7fd296) 70 | 71 | This means the Prefect server never stores your code. It just orchestrates the running (optionally the scheduling) of it. 72 | 73 | 1. After developing your flow, Prefect will register it to the Orion server [through a Deployment](./client/app/weather.py#L49). In that script, you may ask the server to run your flow 3 times a day, for example. 74 | 2. Your code never lies on the Prefect server : this means the code has to be stored somewhere accessible to the agents in order to be executed. 75 | 76 | Prefect has [a lot of storage options](https://docs.prefect.io/tutorials/storage) but the most famous are : Local, S3, Docker and git. 77 | - Local : saves the flows to be run on disk. So the volume where you save the flows must be [shared among your client and your agent(s)](./client/docker-compose.yml#L9). Requires your agent to have the same environment than your client (Python modules, packages installed etc... (the same Dockerfile if your agent and client are containers)) 78 | - S3 : similar to local, but saves the flows to be run in S3 objects. 79 | - Docker : saves the flows to be run as Docker images to your Docker Registry so your agents can easily run the code. 80 | 81 | ### Flow with Local storage (easiest) 82 | 83 | :information_source: If your agents are installed among multiple machines, I recommend you to mount a shared directory with SSHFS. 84 | 85 | 1. Run the following command to register your deployment and run the flow : 86 | 87 | ```bash 88 | docker-compose -f client/docker-compose.yml up # Executes weather.py 89 | ``` 90 | 91 | 2. Access the UI to see your flow correctly run 92 | 93 | ### Flow with S3 Storage (recommended) 94 | 95 |
96 | Tutorial for S3 Storage 97 |
98 | 99 | We will use [MinIO](https://www.github.com/minio/minio) as our S3 server. 100 | 101 | 1. Optionally open and edit the [`client_s3/.env`](./client_s3/.env) file and start MinIO 102 | 103 | ```bash 104 | docker-compose -f client_s3/docker-compose.yml up -d minio # Starts MinIO 105 | ``` 106 | 107 | 2. Register the flow : 108 | 109 | ```bash 110 | docker-compose -f client_s3/docker-compose.yml up weather # Executes weather.py 111 | ``` 112 | 113 | Now your flow is registered. You can access the UI to run it. 114 | 115 |
116 | 117 | ### Flow with Docker storage 118 | 119 | This method requires our client AND agent containers to have access to Docker so they can package or load the image in which the flow will be executed. We use _Docker in Docker_ for that. 120 | 121 |
122 | Tutorial for (secure) Docker Storage 123 | 124 | #### Preparing the Registry 125 | 126 | A Docker Registry is needed in order to save images that are going to be used by our agents. 127 | 128 | 1. Generate the authentication credentials for our registry 129 | 130 | ```bash 131 | sudo apt install apache2-utils # required to generate basic_auth credentials 132 | cd client_docker/registry/auth && htpasswd -B -c .htpasswd myusername && cd - 133 | ``` 134 | 135 | > To add more users, re-run the previous command **without** the -c option 136 | 137 | 2. Start the registry 138 | 139 | ```bash 140 | docker-compose -f client_docker/docker-compose.yml up -d registry 141 | ``` 142 | 143 | 3. Login to the registry 144 | 145 | You need to allow your Docker daemon to push to this registry. Insert this in your `/etc/docker/daemon.json` (create if needed) : 146 | 147 | ```json 148 | { 149 | "insecure-registries": ["172.17.0.1:5000"] 150 | } 151 | ``` 152 | 153 | 4. Start the registry 154 | 155 | ```bash 156 | docker login http://172.17.0.1:5000 # with myusername and the password you typed 157 | ``` 158 | 159 | You should see : _Login Succeeded_ 160 | 161 | #### Start the Docker in Docker agent 162 | 163 | Optionally edit registry credentials in [`./agent_docker/docker-compose.yml`](./agent_docker/docker-compose.yml) and run : 164 | 165 | ```bash 166 | docker-compose -f agent_docker/docker-compose.yml up --build -d 167 | ``` 168 | 169 | #### Registering the flow 170 | 171 | We're going to push our Docker image with Python dependencies and register our flow. 172 | 173 | 1. Build, tag and push the image 174 | 175 | ```bash 176 | docker build . -f ./client_docker/execution.Dockerfile -t 172.17.0.1:5000/weather/base_image:latest 177 | ``` 178 | 179 | > You **must** prefix your image with the registry URI `172.17.0.1:5000` to push it 180 | 181 | ```bash 182 | docker push 172.17.0.1:5000/weather/base_image:latest 183 | ``` 184 | 185 | 2. Register the flow 186 | 187 | Optionally edit registry credentials in `./client_docker/docker-compose.yml` and run : 188 | 189 | ```bash 190 | docker-compose -f ./client_docker/docker-compose.yml up --build weather 191 | ``` 192 | 193 | Now your flow is registered. You can access the UI to run it. 194 | 195 |
196 | -------------------------------------------------------------------------------- /agent/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.10 2 | 3 | RUN apt update && apt install uuid -y 4 | RUN pip install prefect==2.4.2 psycopg2-binary==2.9.3 s3fs==2022.8.2 minio==7.1.11 5 | -------------------------------------------------------------------------------- /agent/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.7" 2 | 3 | services: 4 | 5 | agent: 6 | build: . 7 | command: bash -c "prefect agent start --work-queue flows-example-queue" 8 | environment: 9 | PREFECT_API_URL: "http://172.17.0.1:4200/api" 10 | volumes: 11 | - /srv/docker/prefect/flows:/root/.prefect/flows 12 | - /srv/docker/prefect/flows:/flows 13 | -------------------------------------------------------------------------------- /agent_docker/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM docker:20.10-dind-rootless 2 | ENV TZ="Europe/Paris" 3 | 4 | USER root 5 | RUN apk update && apk add build-base libffi-dev python3 python3-dev py3-pip gcc linux-headers musl-dev util-linux 6 | 7 | RUN pip3 install --upgrade pip && pip3 install prefect==2.4.2 8 | -------------------------------------------------------------------------------- /agent_docker/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.7" 2 | 3 | services: 4 | 5 | agent: 6 | build: . 7 | privileged: true # forced for Docker DinD 8 | entrypoint: ["/bin/sh", "-c"] 9 | command: 10 | - | 11 | docker login -u=myusername -p=test http://172.17.0.1:5000 12 | mkdir -p /opt/prefect/flows 13 | prefect agent start --work-queue flows-example-queue-docker 14 | environment: 15 | PREFECT_API_URL: "http://172.17.0.1:4200/api" 16 | volumes: 17 | - type: bind 18 | source: /var/run/docker.sock 19 | target: /var/run/docker.sock 20 | read_only: true 21 | -------------------------------------------------------------------------------- /client/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.10 2 | 3 | RUN apt update && apt install uuid -y 4 | RUN pip install prefect==2.4.2 5 | 6 | WORKDIR /usr/app 7 | -------------------------------------------------------------------------------- /client/app/.prefectignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | -------------------------------------------------------------------------------- /client/app/weather.py: -------------------------------------------------------------------------------- 1 | # A simple example to demonstrate Prefect is working as expected 2 | # Works with a local folder shared with the agents (/root/.prefect/flows by default). 3 | 4 | import json 5 | 6 | import requests 7 | from prefect import flow, task, get_run_logger 8 | from prefect.deployments import Deployment 9 | from prefect.filesystems import LocalFileSystem 10 | 11 | # --- Flow definition 12 | 13 | @task 14 | def get_city_coordinates(city: str): 15 | logger = get_run_logger() 16 | cities = { 17 | "Paris": (2.3510, 48.8567) 18 | } 19 | logger.info("Getting {}'s coordinates".format(city)) 20 | return cities[city] 21 | 22 | 23 | @task 24 | def get_weather(longitude: float, latitude: float): 25 | logger = get_run_logger() 26 | logger.info(f"Getting weather of latitude={latitude} and longitude={longitude}") 27 | api_endpoint = f"https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}&hourly=temperature_2m" 28 | response = requests.get(api_endpoint) 29 | if response.status_code == 200: 30 | weather_data = json.loads(response.text) 31 | logger.debug(weather_data) 32 | return weather_data 33 | else: 34 | raise Exception("Failed to query " + api_endpoint) 35 | 36 | 37 | @flow(name="get_paris_weather") 38 | def get_paris_weather(): 39 | city_coordinates = get_city_coordinates("Paris") 40 | return get_weather(city_coordinates[0], city_coordinates[1]) 41 | 42 | # --- Deployment definition 43 | 44 | if __name__ == '__main__': 45 | 46 | block_storage = LocalFileSystem(basepath="/flows") 47 | block_storage.save("local-storage", overwrite=True) 48 | 49 | deployment = Deployment.build_from_flow( 50 | name="get_weather_local_example", 51 | flow=get_paris_weather, 52 | storage=LocalFileSystem.load("local-storage"), 53 | work_queue_name="flows-example-queue" 54 | ) 55 | deployment.apply() 56 | 57 | # --- Execute the flow 58 | 59 | get_paris_weather() 60 | -------------------------------------------------------------------------------- /client/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.7" 2 | 3 | services: 4 | 5 | weather: 6 | build: . 7 | command: python3 /usr/app/weather.py 8 | volumes: 9 | - /srv/docker/prefect/flows:/flows 10 | - ./app:/usr/app:ro 11 | environment: 12 | PREFECT_API_URL: "http://172.17.0.1:4200/api" 13 | -------------------------------------------------------------------------------- /client_docker/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM docker:20.10-dind-rootless 2 | ENV TZ="Europe/Paris" 3 | 4 | USER root 5 | RUN apk update && apk add build-base libffi-dev python3 python3-dev py3-pip gcc linux-headers musl-dev util-linux 6 | 7 | RUN pip3 install --upgrade pip && pip3 install prefect==2.4.2 docker==6.0.0 8 | 9 | COPY ./entrypoint.sh /entrypoint.sh 10 | ENTRYPOINT [ "sh", "/entrypoint.sh" ] 11 | -------------------------------------------------------------------------------- /client_docker/app/.prefectignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | -------------------------------------------------------------------------------- /client_docker/app/weather.py: -------------------------------------------------------------------------------- 1 | # A simple example to demonstrate Prefect is working as expected 2 | # Works with a local folder shared with the agents (/root/.prefect/flows by default). 3 | 4 | import json 5 | import os 6 | import shutil 7 | import tarfile 8 | import tempfile 9 | import uuid 10 | from datetime import datetime 11 | 12 | import requests 13 | from prefect import flow, get_run_logger, task 14 | from prefect.deployments import Deployment 15 | from prefect.infrastructure.docker import DockerContainer, DockerRegistry 16 | 17 | # --- Flow definition 18 | 19 | 20 | @task 21 | def get_city_coordinates(city: str): 22 | logger = get_run_logger() 23 | cities = {"Paris": (2.3510, 48.8567)} 24 | logger.info("Getting {}'s coordinates".format(city)) 25 | return cities[city] 26 | 27 | 28 | @task 29 | def get_weather(longitude: float, latitude: float): 30 | logger = get_run_logger() 31 | logger.info(f"Getting weather of latitude={latitude} and longitude={longitude}") 32 | api_endpoint = f"https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}&hourly=temperature_2m" 33 | response = requests.get(api_endpoint) 34 | if response.status_code == 200: 35 | weather_data = json.loads(response.text) 36 | logger.debug(weather_data) 37 | return weather_data 38 | else: 39 | raise Exception("Failed to query " + api_endpoint) 40 | 41 | 42 | @flow(name="get_paris_weather_docker") 43 | def get_paris_weather(): 44 | city_coordinates = get_city_coordinates("Paris") 45 | weather_content = get_weather(city_coordinates[0], city_coordinates[1]) 46 | logger = get_run_logger() 47 | logger.info(weather_content) 48 | return True 49 | 50 | 51 | # --- Deployment definition 52 | 53 | if __name__ == "__main__": 54 | from io import BytesIO 55 | from docker import APIClient 56 | 57 | flow_identifier = datetime.today().strftime("%Y%m%d%H%M%S-") + str(uuid.uuid4()) 58 | 59 | # Mimicking Prefect 1.x image build for Prefect 2.x 60 | 61 | ## 1. Creating Docker build context (including flow files) 62 | base_image = f"{os.environ.get('REGISTRY_ENDPOINT')}/weather/base_image:latest" 63 | flow_image = ( 64 | f"{os.environ.get('REGISTRY_ENDPOINT')}/weather/flow_image:{flow_identifier}" 65 | ) 66 | dockerfile = f""" 67 | FROM {base_image} 68 | RUN mkdir -p /usr/app 69 | COPY ./flow /usr/app 70 | """ 71 | with tempfile.TemporaryDirectory() as tmp_path: 72 | ### a. Creating archive with context (flow files + Dockerfile) for Docker build API 73 | os.makedirs(f"{tmp_path}/build") 74 | with open(f"{tmp_path}/build/Dockerfile", "w+") as the_file: 75 | the_file.write(dockerfile) 76 | shutil.copytree("/usr/app", f"{tmp_path}/build/flow") 77 | with tarfile.open(f"{tmp_path}/flow.tar", "w") as tar: 78 | tar.add(f"{tmp_path}/build", arcname=".") 79 | ### b. Build image with context 80 | with open(f"{tmp_path}/flow.tar", "rb") as fh: 81 | docker_build_archive = BytesIO(fh.read()) 82 | cli = APIClient(base_url="unix:///var/run/docker.sock") 83 | for line in cli.build( 84 | fileobj=docker_build_archive, custom_context=True, rm=True, tag=flow_image 85 | ): 86 | print(line, flush=True) 87 | for line in cli.push(flow_image, stream=True, decode=True): 88 | print(line, flush=True) 89 | 90 | ## 2. Registering flow 91 | dockerhub = DockerRegistry( 92 | username=os.environ.get("REGISTRY_USERNAME"), 93 | password=os.environ.get("REGISTRY_PASSWORD"), 94 | reauth=True, 95 | registry_url=f"{os.environ.get('REGISTRY_SCHEME')}://{os.environ.get('REGISTRY_ENDPOINT')}", 96 | ) 97 | dockerhub.save("docker-storage", overwrite=True) 98 | docker_block = DockerContainer( 99 | image=flow_image, 100 | image_registry=dockerhub, 101 | ) 102 | docker_block.save("docker-storage", overwrite=True) 103 | 104 | deployment = Deployment.build_from_flow( 105 | name="get_weather_docker_example", 106 | flow=get_paris_weather, 107 | infrastructure=docker_block, # storage block is automatically detected from https://github.com/PrefectHQ/prefect/pull/6574/files 108 | work_queue_name="flows-example-queue-docker", 109 | path="/usr/app", 110 | ) 111 | deployment.apply() 112 | -------------------------------------------------------------------------------- /client_docker/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.8" 2 | 3 | services: 4 | 5 | registry: 6 | restart: always 7 | image: registry:2.8.1 8 | ports: 9 | - 5000:5000 10 | volumes: 11 | - ./registry/auth:/auth 12 | - registry_data:/data 13 | environment: 14 | REGISTRY_AUTH: htpasswd 15 | REGISTRY_AUTH_HTPASSWD_REALM: Registry 16 | REGISTRY_AUTH_HTPASSWD_PATH: /auth/.htpasswd 17 | REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data 18 | 19 | # The script should have access to Docker in order to register 20 | # the Dockerized script (`execution.Dockerfile`) in a 21 | # Docker image later used by the Prefect agent. 22 | weather: 23 | build: . 24 | privileged: true # forced for Docker DinD 25 | environment: 26 | PREFECT_API_URL: "http://172.17.0.1:4200/api" 27 | REGISTRY_ENDPOINT: "172.17.0.1:5000" 28 | REGISTRY_SCHEME: "http" 29 | REGISTRY_USERNAME: myusername 30 | REGISTRY_PASSWORD: test 31 | volumes: 32 | - ./app:/usr/app:ro 33 | - type: bind 34 | source: /var/run/docker.sock 35 | target: /var/run/docker.sock 36 | read_only: true 37 | depends_on: 38 | - registry 39 | 40 | volumes: 41 | registry_data: 42 | flows_docker: 43 | -------------------------------------------------------------------------------- /client_docker/entrypoint.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | docker login -u=${REGISTRY_USERNAME} -p=${REGISTRY_PASSWORD} "${REGISTRY_SCHEME}://${REGISTRY_ENDPOINT}" 3 | python3 /usr/app/weather.py 4 | -------------------------------------------------------------------------------- /client_docker/execution.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.10 2 | 3 | RUN apt update && apt install uuid -y 4 | RUN pip3 install --upgrade pip && pip3 install prefect==2.4.2 5 | -------------------------------------------------------------------------------- /client_docker/registry/.gitignore: -------------------------------------------------------------------------------- 1 | .htpasswd -------------------------------------------------------------------------------- /client_docker/registry/auth/.gitkeep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/flavienbwk/prefect-docker-compose/ab3cb5a2f0714bd6f351df60e7096d814453fa3c/client_docker/registry/auth/.gitkeep -------------------------------------------------------------------------------- /client_s3/.env: -------------------------------------------------------------------------------- 1 | MINIO_ENDPOINT=172.17.0.1:9000 2 | MINIO_USE_SSL=false 3 | MINIO_ACCESS_KEY=minio 4 | MINIO_SECRET_KEY=minio123 5 | MINIO_PREFECT_FLOWS_BUCKET_NAME=prefect-flows 6 | MINIO_PREFECT_ARTIFACTS_BUCKET_NAME=prefect-artifacts 7 | -------------------------------------------------------------------------------- /client_s3/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.10 2 | 3 | RUN apt update && apt install uuid -y 4 | RUN pip install prefect==2.4.2 psycopg2-binary==2.9.3 s3fs==2022.8.2 minio==7.1.11 5 | 6 | WORKDIR /usr/app 7 | -------------------------------------------------------------------------------- /client_s3/app/.prefectignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | -------------------------------------------------------------------------------- /client_s3/app/weather.py: -------------------------------------------------------------------------------- 1 | # A simple example to demonstrate Prefect is working as expected 2 | # Works with a local folder shared with the agents (/root/.prefect/flows by default). 3 | 4 | import io 5 | import json 6 | import os 7 | import uuid 8 | from datetime import datetime 9 | 10 | import requests 11 | from minio import Minio 12 | from minio.error import S3Error 13 | from prefect import flow, get_run_logger, task 14 | from prefect.deployments import Deployment 15 | from prefect.filesystems import RemoteFileSystem 16 | 17 | # --- Flow definition 18 | 19 | 20 | @task 21 | def create_bucket( 22 | minio_endpoint, 23 | minio_access_key, 24 | minio_secret_key, 25 | minio_use_ssl, 26 | bucket_name, 27 | ): 28 | client = Minio( 29 | minio_endpoint, minio_access_key, minio_secret_key, secure=minio_use_ssl 30 | ) 31 | try: 32 | client.make_bucket(bucket_name) 33 | except S3Error as ex: 34 | if ex.code != "BucketAlreadyOwnedByYou": 35 | raise ex 36 | print("Flows bucket already exist, skipping.", flush=True) 37 | 38 | 39 | @task 40 | def get_city_coordinates(city: str): 41 | logger = get_run_logger() 42 | cities = {"Paris": (2.3510, 48.8567)} 43 | logger.info("Getting {}'s coordinates".format(city)) 44 | return cities[city] 45 | 46 | 47 | @task 48 | def get_weather(longitude: float, latitude: float): 49 | logger = get_run_logger() 50 | logger.info(f"Getting weather of latitude={latitude} and longitude={longitude}") 51 | api_endpoint = f"https://api.open-meteo.com/v1/forecast?latitude={latitude}&longitude={longitude}&hourly=temperature_2m" 52 | response = requests.get(api_endpoint) 53 | if response.status_code == 200: 54 | weather_data = json.loads(response.text) 55 | logger.debug(weather_data) 56 | return weather_data 57 | else: 58 | raise Exception("Failed to query " + api_endpoint) 59 | 60 | 61 | @task 62 | def add_text_to_bucket( 63 | content: str, 64 | minio_endpoint, 65 | minio_access_key, 66 | minio_secret_key, 67 | minio_use_ssl, 68 | bucket_name, 69 | ): 70 | client = Minio( 71 | minio_endpoint, minio_access_key, minio_secret_key, secure=minio_use_ssl 72 | ) 73 | content_json = json.dumps(content) 74 | client.put_object( 75 | object_name=f"{datetime.today().strftime('%Y%m%d%H%M%S')}/weather.txt", 76 | bucket_name=bucket_name, 77 | data=io.BytesIO(str.encode(content_json)), 78 | length=len(content_json), 79 | content_type="text/plain", 80 | ) 81 | 82 | 83 | @flow(name="get_paris_weather_s3") 84 | def get_paris_weather( 85 | minio_endpoint: str, 86 | minio_access_key: str, 87 | minio_secret_key: str, 88 | minio_use_ssl: bool, 89 | artifacts_bucket_name: str, 90 | ): 91 | city_coordinates = get_city_coordinates("Paris") 92 | weather_content = get_weather(city_coordinates[0], city_coordinates[1]) 93 | create_bucket( 94 | minio_endpoint, 95 | minio_access_key, 96 | minio_secret_key, 97 | minio_use_ssl, 98 | artifacts_bucket_name, 99 | ) 100 | add_text_to_bucket( 101 | weather_content, 102 | minio_endpoint, 103 | minio_access_key, 104 | minio_secret_key, 105 | minio_use_ssl, 106 | artifacts_bucket_name, 107 | ) 108 | return True 109 | 110 | 111 | # --- Deployment definition 112 | 113 | if __name__ == "__main__": 114 | bucket_name = os.environ.get("MINIO_PREFECT_FLOWS_BUCKET_NAME") 115 | artifacts_bucket_name = os.environ.get("MINIO_PREFECT_ARTIFACTS_BUCKET_NAME") 116 | minio_endpoint = os.environ.get("MINIO_ENDPOINT") 117 | minio_use_ssl = os.environ.get("MINIO_USE_SSL") == "true" 118 | minio_scheme = "https" if minio_use_ssl else "http" 119 | minio_access_key = os.environ.get("MINIO_ACCESS_KEY") 120 | minio_secret_key = os.environ.get("MINIO_SECRET_KEY") 121 | 122 | flow_identifier = datetime.today().strftime("%Y%m%d%H%M%S-") + str(uuid.uuid4()) 123 | block_storage = RemoteFileSystem( 124 | basepath=f"s3://{bucket_name}/{flow_identifier}", 125 | key_type="hash", 126 | settings=dict( 127 | use_ssl=minio_use_ssl, 128 | key=minio_access_key, 129 | secret=minio_secret_key, 130 | client_kwargs=dict(endpoint_url=f"{minio_scheme}://{minio_endpoint}"), 131 | ), 132 | ) 133 | block_storage.save("s3-storage", overwrite=True) 134 | 135 | deployment = Deployment.build_from_flow( 136 | name="get_weather_s3_example", 137 | flow=get_paris_weather, 138 | storage=RemoteFileSystem.load("s3-storage"), 139 | work_queue_name="flows-example-queue", 140 | parameters={ 141 | "minio_endpoint": minio_endpoint, 142 | "minio_access_key": minio_access_key, 143 | "minio_secret_key": minio_secret_key, 144 | "minio_use_ssl": minio_use_ssl, 145 | "artifacts_bucket_name": artifacts_bucket_name, 146 | }, 147 | ) 148 | deployment.apply() 149 | -------------------------------------------------------------------------------- /client_s3/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.7" 2 | 3 | services: 4 | 5 | minio: 6 | image: minio/minio:RELEASE.2022-09-07T22-25-02Z 7 | command: server --console-address ":9001" /data 8 | ports: 9 | - 9000:9000 10 | - 9001:9001 11 | volumes: 12 | - /srv/docker/prefect/minio:/data 13 | environment: 14 | MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY} 15 | MINIO_SECRET_KEY: ${MINIO_SECRET_KEY} 16 | healthcheck: 17 | test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] 18 | interval: 5s 19 | timeout: 3s 20 | retries: 10 21 | 22 | weather: 23 | build: . 24 | command: python3 /usr/app/weather.py 25 | volumes: 26 | - ./app:/usr/app:ro 27 | environment: 28 | PREFECT_API_URL: "http://172.17.0.1:4200/api" 29 | MINIO_USE_SSL: ${MINIO_USE_SSL} 30 | MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY} 31 | MINIO_SECRET_KEY: ${MINIO_SECRET_KEY} 32 | MINIO_ENDPOINT: ${MINIO_ENDPOINT} # Must be accessible from Agent 33 | MINIO_PREFECT_FLOWS_BUCKET_NAME: ${MINIO_PREFECT_FLOWS_BUCKET_NAME} 34 | MINIO_PREFECT_ARTIFACTS_BUCKET_NAME: ${MINIO_PREFECT_ARTIFACTS_BUCKET_NAME} 35 | depends_on: 36 | minio: 37 | condition: service_healthy 38 | -------------------------------------------------------------------------------- /docker_interface.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/flavienbwk/prefect-docker-compose/ab3cb5a2f0714bd6f351df60e7096d814453fa3c/docker_interface.png -------------------------------------------------------------------------------- /prefect_schema_principle.drawio: -------------------------------------------------------------------------------- 1 | 7VxZc+K4Fv41VM08hJI3lkeWsOSypAPpJLxMGVsYgReubbDh18+RN2wZEpKYTLqrSapiHclHss53VouUhJbhd215sxxaKtZLPFL9ktAu8Twn8nyJ/iJ1H1J4hKSQotlEjUYdCRNywBERRdQtUbGTGehalu6STZaoWKaJFTdDk23b8rLDFpaenXUjazhHmCiynqc+EdVdhtSahI70HibaMp6ZQ1GPIceDI4KzlFXLy5Cw73Ys042WOKSX2LZltyTdLl2XPmGjxHfgd0G7ypplaTqWN8QpK5YBZMWBIZ2FbBCd7m+KAbAXbktCy7YsN7wy/BbWqXDibQ/X0DnTmzy3jU33khvcquNXDt3K46o65iqqsv1fz78RIjY7Wd9GGxpthruPdxibaoMKClqKLjsOUUpCc+kaOhA4uKTP3rJ0yw6GC/TTgUU0YV32/hmIqFyV4vZLdFPQaFMYoqS1j1rK1t5hNRqYf8wYZbKtYfeVZ6tFMlQzAIo2p4stA8OkMMA7wiZGzTKFmJhmY112yS4LOzmChpawS2a4twismEeRqlVi3MWKFmMsZuFYW1vB0V1pOTKMBInPMOJ4sVxPfarVLN9wn3J8QaTyPjVsQwc459cv1Jj1Sxm4wUXIMW6ltvhICiD5Dnjy9T/w/Bp4ShwjXvEXg2f19PqvCk/hBDorOgimOYcLzQ3wERIoCgMvGAOx8v+tFQ4QRJX+pEnhvS2dUFRFHCIvwzIGcm6yuR1TPALYoWB01k5Oc2xra6oJkr0lcfFkIyu014NogVEjouspNVoscEWhmua4trXGqR61Wp8j9Jpq7LDtYv9V0Ee9rM0Ro9gkpROVEzpRYbCbhn8KEe8XuPi2OcpYh7dsE/aJG5ogKWq9xCYHro/2hzZi83N2W0MNfRuuv5hl4ipZEHAV6WOWKUFPxKjO8CnIEuXWy0nXt0RSDpgPWCOOi22HLp0G1jrAkMYcrIn4SwM7sIEuaxFbir9zqAZ9dbPQlXWimRTXAEMMqt+kWk0gLm9EHQZRVXp708YOOcjzgBUFcLSBwFdqlqQ2UHR5jvWmrKy1wCRlzAz9RPZzEq3nhFdHDfpzUjteVWTWEiV5SrTeUhSmnbVQN6gsoHo9I/Gb+ueAHw+xFgsHuwx4CoFL9bqOa+JaNk3bPuO58pQOAZEAKvcAa+P8DSQmWHZqJnKeb9tS1oBgKi+qMlQkhfGeCDlVCh1mnKtyWa2CHHRDxxm+RvP1MtVaZSnbblmVAZOyg8946gKcLV/JRm4CynvbJHtOG3qOTWUK87e1szj9CCxTGE8JJB5FTdxNZNUalHVo2HJc7m28oIUM4IHB1dtvwfkTU72hQM5GNj/F/6+xTSxgUZENiiFz7myys4YzFDwr9Io8Qn+fnYdRmKy/aALWWoj6Db5FW2VeYghsu5olcPkW5ZElsO1qlsCx7Dlmfo5dYIqQa2XYI2Z+lFpg4CytrasTEyLuuJ6GsibklHUIvGdYuuN4xpuiE2ZH9hyxHMYFQlOz++AUT/b+Y5n/bGxskMAusRGAa21SgYKOFxF8FGJqg6DVFujsquwsg5C5oKyBZ1LQOCBL2zH+VMAqXcmMcacSxSLyhvg6yBqiMsbZxOFY8EhXO6LU45V6R9C6xzaBzaCR3hdlIdXvlYXUs1G9yH2wPpIUVmIvyyarRaUhTO4sfEVBhMsnyDQWxE6SgLABEg1roPMvxVLx7592JIbg03kHKnNiNZt1CIUAHdZY5pl8hquXIV1OF/GyXK+XqHD5xJaJAPNxCO24cQJvR8MQXtz4+SBkvME2bJOpUbHZxFTIJsgugiQ4CfHOhF9vojRXFEPBhymjmZaJ3wXpU679WMRD0bN3Trxuguxpug9cvFaMj2Wsy00Q9jBOVqqdsMdsXbk4J1u5blLb0IJiLOLeldZ+puSazxQLEJzI+B8e5aOjL62pclcuRsRy439tueVe0Pzncrtacp6Vm/Bagv2Waf4GcmPfYfzncrvgnWoByciluUg58+71mH2czkXOCuTNtCNC65tpR+xGvkneIUpZ+CRh/XvzDok5f8AJ18k7ROZFLMd9xTkB9AfTF7jY74JpIQuRqlD+4Cs9ia9ljavALKYgUEvMiuMy+HVBnc99JoBbJxDAAgNclCRjcZc4SbGRa1FZ+1jZujinBr9bSp2ofgEpNZ99QRyXoD578itz/qVez0JWlMo1huX10ukkfPljJ18Lab+LnaxwBdnJipCt6FzNTjLBbq32BWaSvxKms4dyuMswzWUQ/bHaugPScPNLDMgdQnfnk/oQx7i/WA1eZEvaH63BS1XGBleqrGZdKRqOq/3X1YhrvW56j0ZcCOJCFecTGlH9TTTigw5CYA6tCVfKDtkqVnxs97r6kK8Md2WImO3jaylE3z/FofMcH+NnNac5v10AXdg7qRtUrtZq2Zgljr0+CvQ44aqX2YPjV4ySv+aY728VUVwaYn8v+1lhzd5Hq2usnxfRdeIJSTq94Ovaz1NvaBh9+PJv4SQ6lck83zgCc9lXd4qLn79XPimwr5CkD6KdjRaSdxpFf6mHnYf/CrTn32vFZbc5+OXSuZMsYEzdbXDa3oSgAemWlv/GzbeIHVL0TqcWHUvIxBQXBg6xWSgicEDJ946LPc7CcWW+ilIfITsLXylL6ER17vpRhnDqaAuDl/hcJjGCb0En4BhQSd9bDnFJcDJzbrmuZVwGgTOnNJ1NeJp0QXxqIZvBlI2YimIKXAdHtoRG2OQ7zk4r8U0fEM237nsjfrZvivMnf6scEJF7D0hpW7uBoArqXhKGe2mnGMpuuGp4w1b9oBoK6feW7rwrHcbm0pGfJPt+cmepvQdvTGo7uEsYmMphYNT3s33NH0/X0kAIx/VJk5893x3kp/r2ftL3B6tb0u/qa7i//fLcXD4foN3T6v1V08OtvoaniAx7TRHuE+SnByS3ERlNbz2lp8F9S11+Ui01TevNNrNntTUXKI+GNmw1uOG0odF+4HFQe7ozA56zJ2n5Yvj64Olu+cK7pmLUubnxozI3fgr9dva+8R7W0eWcsTnyXp5GOqzVU4yfB3gOfjbpwz7ekZH4MnVqs4No9IXlcuyta7CjmxmsTOFH1uzpgZt3H8k9eVnh7m219WNdU7odJLdSKzJ0Z97idkBfyc8P0njV2I6nDjzlj3p/jfaj6aMzvB36w+mP7egA1y3PG7Rv/VFLFEYTet3nRs+p/tUjF9AmHtz78xC0p8l4ys+ltMfDUBpNxH3CA/5i0tD6rYY2nQZtNJrqK3rvkIj7QbsB9zXQYLoG6WnRdd8bdYb+qN3fjlseNwz5O6Np5xD0t4OxTngdjD2MVsp2SCjvdbTWH9z42SKD4x7e3feau3lPN+e8qCnCgwR7WO8bs82867lzsMgvvI7w1PIGq3UVpOvNeX2r9oYg+T4/2gfP6Q9hntGksR+u1tsT7f243aBtWE9/C+uO/jYOY3gW6If1ayCHR224ehEGK4XuKT9siT7dnz6pHaV/d9+qm7GEM9ddJYWRAa+cQszB2x2v/T98PsIntE1wdzGHQARJzLid+ABl+kS6lI/72GrOBWdAoHn8lxKhqzr+4w/h9l8= -------------------------------------------------------------------------------- /prefect_schema_principle.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/flavienbwk/prefect-docker-compose/ab3cb5a2f0714bd6f351df60e7096d814453fa3c/prefect_schema_principle.jpg -------------------------------------------------------------------------------- /server/.env: -------------------------------------------------------------------------------- 1 | POSTGRES_DB="prefect_db" 2 | POSTGRES_USER="prefect_user" 3 | POSTGRES_PASSWORD="prefect_password" # PLEASE CHANGE ! 4 | PREFECT_API_URL=http://172.17.0.1:4200/api 5 | DB_CONNECTION_URL="postgresql+asyncpg://prefect_user:prefect_password@postgres:5432/prefect_db" # PLEASE EDIT ! 6 | -------------------------------------------------------------------------------- /server/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM prefecthq/prefect:2.4.2-python3.10 2 | 3 | RUN apt update && \ 4 | pip install psycopg2-binary==2.9.3 s3fs==2022.8.2 5 | -------------------------------------------------------------------------------- /server/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3.7" 2 | 3 | services: 4 | 5 | postgres: 6 | command: 7 | - postgres 8 | - -c 9 | - max_connections=150 10 | environment: 11 | POSTGRES_USER: ${POSTGRES_USER} 12 | POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} 13 | POSTGRES_DB: ${POSTGRES_DB} 14 | volumes: 15 | - postgres:/var/lib/postgresql/data 16 | healthcheck: 17 | interval: 10s 18 | retries: 60 19 | start_period: 2s 20 | test: pg_isready -q -d $${POSTGRES_DB} -U $${POSTGRES_USER} | grep "accepting connections" || exit 1 21 | timeout: 2s 22 | image: postgres:14 23 | restart: always 24 | 25 | orion: 26 | restart: always 27 | build: . 28 | 29 | command: prefect orion start 30 | ports: 31 | - 4200:4200 32 | depends_on: 33 | postgres: 34 | condition: service_started 35 | volumes: 36 | - prefect_data:/root/.prefect 37 | - prefect_flows:/flows 38 | environment: 39 | PREFECT_ORION_API_HOST: 0.0.0.0 40 | PREFECT_ORION_DATABASE_CONNECTION_URL: ${DB_CONNECTION_URL} 41 | PREFECT_ORION_ANALYTICS_ENABLED: "false" 42 | PREFECT_LOGGING_SERVER_LEVEL: WARNING 43 | PREFECT_API_URL: ${PREFECT_API_URL} 44 | 45 | volumes: 46 | postgres: 47 | prefect_data: 48 | prefect_flows: --------------------------------------------------------------------------------