├── README.md ├── TUTORIAL.md ├── assets └── flow_runs_radar.png ├── flows ├── 1_github_stars.py ├── 2_basic_flow.py ├── 3_flows_with_parameters.py ├── 4_basic_flow_with_tasks.py ├── 5_cascading_tasks.py ├── 6_flow_within_flow.py ├── 7_failing_flow_with_retries.py ├── 8_task_caching.py ├── 9_github_stars_deployment.py └── __init__.py └── requirements.txt /README.md: -------------------------------------------------------------------------------- 1 | # prefect-tutorial 2 | A quick and easy way to start learning about Prefect 3 | 4 | ## Getting started 5 | 6 | - Make sure you have python 3.9 installed. Ideally, set up a python virtual environment before starting. See the [Useful commands](#useful-commands) section for how to create and activate a virtual environment. 7 | - Once you have you virtual environment activated, run the following command to install dependencies required for this tutorial 8 | `pip install -r requirement.txt` 9 | - Now that your setup is completed, simply go to the [tutorial](https://github.com/jmprovencher/prefect-tutorial/blob/main/TUTORIAL.md) page to get started. 10 | 11 | ## Useful commands 12 | |command|description| 13 | |---|---| 14 | |`python3 -m venv `|Create a virtual environment| 15 | |`source /bin/activate`|Activate the virtual environment 16 | |`pip install -r requirements.txt`|Install dependencies| 17 | |`deactivate`|Deactivate the virtual environment| 18 | -------------------------------------------------------------------------------- /TUTORIAL.md: -------------------------------------------------------------------------------- 1 | # Prefect tutorial 2 | ## Follow this markdown file to go through different features of Prefect 2 3 | 4 | 5 | Run a flow locally 6 | ``` 7 | python flows/1_github_stars.py 8 | ``` 9 | 10 | Start the prefect local UI 11 | ``` 12 | prefect orion start 13 | ``` 14 | 15 | Go and explore the flow run we just created. Go and check the logs, the parameters that were passed, etc. 16 | 17 | In the flow runs panel, try to find the radar view and observe the execution in parallel of the different star processes. 18 | 19 | ![radar](assets/flow_runs_radar.png) 20 | 21 | Now let's start with a basic flow! 22 | ``` 23 | python flows/2_basic_flow.py 24 | ``` 25 | 26 | Run flows using parameters 27 | 28 | ``` 29 | python flows/3_flows_with_parameters.py 30 | ``` 31 | 32 | Run a basic flow with a task 33 | 34 | ``` 35 | python flows/4_basic_flow_with_tasks.py 36 | ``` 37 | 38 | Run a flow with cascading tasks. Make sure to go in the UI and look at the resulting radar! 39 | 40 | ``` 41 | python flows/5_cascading_tasks.py 42 | ``` 43 | 44 | Now, let's have a look at how we can launch flows within flows. This is useful when you might have flows that can fail and that should be relaunched independently. 45 | 46 | ``` 47 | python flows/6_flow_withing_flow.py 48 | ``` 49 | 50 | It's easy to leverage retries with tasks, run the following flow that is expected to fail, and go have a look in the Orion UI to see the result! 51 | 52 | ``` 53 | python flows/7_failing_flow_with_retries.py 54 | ``` 55 | 56 | To see how caching works for tasks, run the following flow a few times, and notice how Mario Bros in only welcomed once. 57 | 58 | ``` 59 | python flows/8_task_caching.py 60 | ``` 61 | 62 | Now that we have tried a few of Prefect 2 features, let's try to build a deployment that will allow us to run flows in multiple infrastructure 63 | 64 | ``` 65 | prefect deployment build ./flows/9_github_stars_deployment.py:github_stars -n github-stars -q tutorial 66 | ``` 67 | 68 | You'll notice this created a github_stars-deployment.yaml file at the root of the repository. Open it and go modify the parameters and the schedule to setup a deployment that would print the number of stars for the https://github.com/PrefectHQ/prefect every minutes. 69 | 70 | ``` 71 | parameters: {} 72 | schedule: null 73 | ``` 74 | 75 | Modify these 2 lines to the following: 76 | 77 | ``` 78 | parameters: { 79 | "repos": ["PrefectHQ/prefect"] 80 | } 81 | schedule: 82 | interval: 300 83 | ``` 84 | 85 | Now, let's apply this deployment so that Prefect Orion registers it. 86 | 87 | ``` 88 | prefect deployment apply github_stars-deployment.yaml 89 | ``` 90 | 91 | Note that this deployment will be trying to run the flow every minute using the 'tutorial' work queue. You will notice in the 'Runs' tab that a some of tje runs will be tagged as 'Late' if you don't start an agent to consume the work queue. 92 | 93 | If you go in the Work Queues navigation in the UI, you'll see that that the 'tutorial' work queue is unhealthy because no agent is consuming it. 94 | 95 | To make sure we can run this flow on schedule properly, let's create start an agent! 96 | 97 | ``` 98 | prefect agent start -q 'tutorial' 99 | ``` 100 | 101 | Now, go in the Deployments navigation in the UI, select the 'github-stars' deployment and run it using the defaults. 102 | You should see the local agent on your computer picking up the flow, and running it. Go in the UI, and take a look at the execution logs. 103 | 104 | Now, go and click on the 'Run' button again, and try to find the numbers of stars of 'freeCodeCamp/freeCodeCamp', which is the most popular repository on Github. 105 | 106 | 107 | -------------------------------------------------------------------------------- /assets/flow_runs_radar.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jmprovencher/prefect-tutorial/7194d41f7352bb92e7d736d6d805bfbf2e23e256/assets/flow_runs_radar.png -------------------------------------------------------------------------------- /flows/1_github_stars.py: -------------------------------------------------------------------------------- 1 | from prefect import flow, task 2 | import httpx 3 | 4 | @task(retries=3) 5 | def get_stars(repo): 6 | url = f"https://api.github.com/repos/{repo}" 7 | count = httpx.get(url).json()["stargazers_count"] 8 | print(f"{repo} has {count} stars!") 9 | 10 | @flow 11 | def github_stars(repos): 12 | for repo in repos: 13 | get_stars(repo) 14 | 15 | # call the flow! 16 | github_stars(["coveo/ui-kit", "coveo/platform-client", "coveo/plasma"]) -------------------------------------------------------------------------------- /flows/2_basic_flow.py: -------------------------------------------------------------------------------- 1 | from prefect import flow 2 | 3 | @flow 4 | def my_favorite_function(): 5 | print("What is your favorite number?") 6 | return 42 7 | 8 | print(my_favorite_function()) -------------------------------------------------------------------------------- /flows/3_flows_with_parameters.py: -------------------------------------------------------------------------------- 1 | import requests 2 | from prefect import flow 3 | 4 | @flow 5 | def call_api(url): 6 | return requests.get(url).json() 7 | 8 | api_result = call_api("http://time.jsontest.com/") 9 | print(api_result) -------------------------------------------------------------------------------- /flows/4_basic_flow_with_tasks.py: -------------------------------------------------------------------------------- 1 | import requests 2 | from prefect import flow, task 3 | 4 | @task 5 | def call_api(url): 6 | response = requests.get(url) 7 | print(response.status_code) 8 | return response.json() 9 | 10 | @flow 11 | def api_flow(url): 12 | fact_json = call_api(url) 13 | return fact_json 14 | 15 | print(api_flow("https://catfact.ninja/fact")) -------------------------------------------------------------------------------- /flows/5_cascading_tasks.py: -------------------------------------------------------------------------------- 1 | import requests 2 | from prefect import flow, task 3 | 4 | @task 5 | def call_api(url): 6 | response = requests.get(url) 7 | print(response.status_code) 8 | return response.json() 9 | 10 | @task 11 | def parse_fact(response): 12 | fact = response["fact"] 13 | print(fact) 14 | return fact 15 | 16 | @flow 17 | def api_flow(url): 18 | fact_json = call_api(url) 19 | fact_text = parse_fact(fact_json) 20 | return fact_text 21 | 22 | api_flow("https://catfact.ninja/fact") -------------------------------------------------------------------------------- /flows/6_flow_within_flow.py: -------------------------------------------------------------------------------- 1 | from prefect import flow 2 | 3 | @flow 4 | def common_flow(config: dict): 5 | print("I am Mansour and I love flows of flows!") 6 | intermediate_result = 42 7 | return intermediate_result 8 | 9 | @flow 10 | def main_flow(): 11 | # do some things 12 | # then call another flow function 13 | data = common_flow(config={}) 14 | data = common_flow(config={}) 15 | data = common_flow(config={}) 16 | # do more things 17 | 18 | main_flow() -------------------------------------------------------------------------------- /flows/7_failing_flow_with_retries.py: -------------------------------------------------------------------------------- 1 | from prefect import flow, task 2 | 3 | # this tasks runs 3 times before the flow fails 4 | @task(retries=2, retry_delay_seconds=5) 5 | def failure(): 6 | print('running') 7 | raise ValueError("bad code") 8 | 9 | @flow 10 | def test_retries(): 11 | return failure() 12 | 13 | test_retries() -------------------------------------------------------------------------------- /flows/8_task_caching.py: -------------------------------------------------------------------------------- 1 | from prefect import flow, task 2 | from prefect.tasks import task_input_hash 3 | from datetime import timedelta 4 | 5 | @task(cache_key_fn=task_input_hash, cache_expiration=timedelta(minutes=1)) 6 | def hello_task(name_input): 7 | # Doing some work 8 | print(f"Welcome {name_input}!") 9 | return "Welcome " + name_input 10 | 11 | @flow 12 | def hello_flow(name_input): 13 | hello_task(name_input) 14 | 15 | hello_flow("Mario Bros") -------------------------------------------------------------------------------- /flows/9_github_stars_deployment.py: -------------------------------------------------------------------------------- 1 | from prefect import flow, task, get_run_logger 2 | import httpx 3 | 4 | @task(retries=3) 5 | def get_stars(repo): 6 | url = f"https://api.github.com/repos/{repo}" 7 | count = httpx.get(url).json()["stargazers_count"] 8 | logger = get_run_logger() 9 | logger.info(f"{repo} has {count} stars!") 10 | 11 | @flow 12 | def github_stars(repos): 13 | for repo in repos: 14 | get_stars(repo) 15 | -------------------------------------------------------------------------------- /flows/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jmprovencher/prefect-tutorial/7194d41f7352bb92e7d736d6d805bfbf2e23e256/flows/__init__.py -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | prefect --------------------------------------------------------------------------------