├── .gitignore ├── README.md ├── curl.sh ├── fastapi-server.py ├── images ├── deployment_logs.png ├── wandb_create_secret.png ├── wandb_secrets.png ├── wandb_webhook_auto.png └── wandb_webhook_payload.png ├── img ├── 2023-08-03-14-12-32.png └── 2023-08-03-14-13-58.png └── server.py /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__ 2 | .ipynb_checkpoints 3 | .virtual_documents -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # W&B Modal Web Hooks 3 | > Setup a webhook that integrates the W&B model registry w/ [Modal Labs](https://modal.com/). 4 | 5 | Webhooks are a nice way to trigger external applications to perform some action. This is an example that receives a webhook when you change a tag on a model in a Weights & Biases model registry. [This document](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw) describes how to setup W&B webhooks with GitHub Actions. However, I think it is more useful to show how this might work with Modal Labs, because Modal is: 6 | 7 | 1. Faster 8 | 2. You have access to GPUs 9 | 3. More ergonomic (you can locally debug, easier to use) 10 | 11 | ## Instructions 12 | 13 | 1. Setup 14 | 15 | ```bash 16 | pip install modal 17 | pip install -U modal-client 18 | ``` 19 | 20 | 1. Setup a [modal secret](https://modal.com/secrets) with the name `my-random-secret` with the following fields: 21 | 22 | - Key: `AUTH_TOKEN` 23 | - Value: `secret-random-token` 24 | 25 | 1. Create a [modal webhook](https://modal.com/docs/guide/webhooks) by running the following command from the root of this repo: 26 | 27 | ```bash 28 | modal deploy server.py 29 | ``` 30 | 31 | You will get an endpoint that you need to use in the next step. This url will look something like ` https://hamelsmu--wandb-hook-f.modal.run` 32 | 33 | Also you will get a deployment url that will look something like `https://modal.com/apps/hamelsmu/wandb-hook` - you can use this to see the logs of your webhook. 34 | 35 | 2. Test the webhook by running the following command: 36 | 37 | ``` 38 | ./curl.sh 39 | ``` 40 | You should see the following response locally: 41 | 42 | ``` 43 | {"message":"Event processed successfully"} 44 | ``` 45 | Open your modal deployment url and you should see something like this on the logs tab: 46 | 47 | ![](images/deployment_logs.png) 48 | 49 | 3. Create a secret in your wandb team settings with the name `AUTH_TOKEN` and value `secret-random-token` 50 | 51 | ![](images/wandb_create_secret.png) 52 | 53 | 5. Create a new [W&B Webhook](https://docs.wandb.ai/guides/model_registry/automation#configure-a-webhook) in your team settings. Set the url to the one you got in the previous step, and the `Access token` to the `AUTH_TOKEN`. 54 | 55 | ![](images/wandb_secrets.png) 56 | 57 | 6. Create a new [webhook automation](https://docs.wandb.ai/guides/model_registry/automation#add-a-webhook) in your team's model registry: Set the trigger to `an artificat alias is added` and set the Alias regex to `candidate`. 58 | 59 | ![](images/wandb_webhook_auto.png) 60 | 61 | 62 | Next, set the following payload: 63 | 64 | ```json 65 | { 66 | "event_type": "${event_type}", 67 | "event_author": "${event_author}", 68 | "alias": "${alias}", 69 | "artifact_version": "${artifact_version}", 70 | "artifact_version_string": "${artifact_version_string}", 71 | "artifact_collection_name": "${artifact_collection_name}", 72 | "project_name": "${project_name}", 73 | "entity_name": "${entity_name}" 74 | } 75 | ``` 76 | 77 | ![](2023-12-24-13-29-02.png) 78 | 79 | 7. Trigger the payload 80 | 81 | The easiest way to trigger the payload is to go to the model registry and add an alias 82 | 83 | ![](img/2023-08-03-14-12-32.png) 84 | 85 | type in `candidate`. This will trigger the webhook. 86 | 87 | > [!IMPORTANT] 88 | > Make sure you spell `candidate` correctly, otherwise the webhook won't trigger! 89 | 90 | 8. Check the logs. Go to the [logs for your modal webhook](https://modal.com/logs) and you should see something like this: 91 | 92 | ![](img/2023-08-03-14-13-58.png) 93 | 94 | 9. Do stuff! You can do tasks like: 95 | - Download the appropriate model from the registry based on the webhook payload and test/run it. 96 | - Push the model to an inference server or update a Modal deployment. 97 | - Send a [Slack message](https://modal.com/docs/guide/ex/stable_diffusion_slackbot#slack-webhook) to a channel. 98 | 99 | ## Further reading 100 | 101 | ### Modal 102 | 103 | 1. [Webhooks](https://modal.com/docs/guide/webhooks) 104 | 2. [Web endpoint](https://modal.com/docs/guide/webhook-urls) 105 | 3. [Slack web hook](https://modal.com/docs/guide/ex/stable_diffusion_slackbot#slack-webhook) 106 | 107 | 108 | ### W&B Webhooks 109 | 110 | [W&B Webhooks](https://wandb.ai/wandb/wandb-model-cicd/reports/Model-CI-CD-with-W-B--Vmlldzo0OTcwNDQw) 111 | 112 | 113 | -------------------------------------------------------------------------------- /curl.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Default URL 4 | URL="http://localhost:8000/" 5 | 6 | # Check if a URL is provided as the first argument 7 | if [ "$1" != "" ]; then 8 | URL=$1 9 | fi 10 | 11 | # Curl command 12 | curl -X POST "$URL" \ 13 | -H "Authorization: Bearer secret-random-token" \ 14 | -H "Content-Type: application/json" \ 15 | -d '{ 16 | "event_type": "test_event", 17 | "event_author": "JohnDoe", 18 | "alias": "alias123", 19 | "artifact_version": "v1.0", 20 | "artifact_version_string": "1.0.0", 21 | "artifact_collection_name": "collection1", 22 | "project_name": "projectX", 23 | "entity_name": "entityA" 24 | }' 25 | -------------------------------------------------------------------------------- /fastapi-server.py: -------------------------------------------------------------------------------- 1 | from fastapi import Depends, HTTPException, status, FastAPI 2 | from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials 3 | from pydantic import BaseModel 4 | 5 | auth_scheme = HTTPBearer() 6 | app = FastAPI() 7 | 8 | class Event(BaseModel): 9 | "Defines the payload your webhook will send." 10 | event_type: str 11 | event_author: str 12 | alias: str 13 | artifact_version: str 14 | artifact_version_string: str 15 | artifact_collection_name: str 16 | project_name: str 17 | entity_name: str 18 | 19 | def __str__(self): 20 | msg = 'Payload:\n========\n' 21 | for k,v in self.model_dump().items(): 22 | msg += f'{k}={v}\n' 23 | return msg 24 | 25 | @app.post("/") 26 | def webhook(event: Event, 27 | token: HTTPAuthorizationCredentials = Depends(auth_scheme)): 28 | "Receive the webhook and print the payload. Uses a token to authenticate." 29 | print(event) 30 | if token.credentials != 'secret-random-token': 31 | raise HTTPException( 32 | status_code=status.HTTP_401_UNAUTHORIZED, 33 | detail="Incorrect bearer token", 34 | headers={"WWW-Authenticate": "Bearer"}, 35 | ) 36 | return {"message": "Event processed successfully"} 37 | -------------------------------------------------------------------------------- /images/deployment_logs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hamelsmu/wandb-modal-webhook/62c91b055a343f802410ee328e6d70ec602c7eeb/images/deployment_logs.png -------------------------------------------------------------------------------- /images/wandb_create_secret.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hamelsmu/wandb-modal-webhook/62c91b055a343f802410ee328e6d70ec602c7eeb/images/wandb_create_secret.png -------------------------------------------------------------------------------- /images/wandb_secrets.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hamelsmu/wandb-modal-webhook/62c91b055a343f802410ee328e6d70ec602c7eeb/images/wandb_secrets.png -------------------------------------------------------------------------------- /images/wandb_webhook_auto.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hamelsmu/wandb-modal-webhook/62c91b055a343f802410ee328e6d70ec602c7eeb/images/wandb_webhook_auto.png -------------------------------------------------------------------------------- /images/wandb_webhook_payload.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hamelsmu/wandb-modal-webhook/62c91b055a343f802410ee328e6d70ec602c7eeb/images/wandb_webhook_payload.png -------------------------------------------------------------------------------- /img/2023-08-03-14-12-32.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hamelsmu/wandb-modal-webhook/62c91b055a343f802410ee328e6d70ec602c7eeb/img/2023-08-03-14-12-32.png -------------------------------------------------------------------------------- /img/2023-08-03-14-13-58.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hamelsmu/wandb-modal-webhook/62c91b055a343f802410ee328e6d70ec602c7eeb/img/2023-08-03-14-13-58.png -------------------------------------------------------------------------------- /server.py: -------------------------------------------------------------------------------- 1 | # see https://modal.com/docs/guide/webhooks 2 | from fastapi import Depends, HTTPException, status 3 | from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials 4 | from pydantic import BaseModel 5 | from modal import Secret, Stub, Image, web_endpoint 6 | 7 | auth_scheme = HTTPBearer() 8 | img = Image.debian_slim().pip_install("fastapi==0.101.0", "pydantic==2.1.1") 9 | 10 | stub = Stub("wandb-hook") 11 | class Event(BaseModel): 12 | "Defines the payload your webhook will send." 13 | event_type: str 14 | event_author: str 15 | alias: str 16 | artifact_version: str 17 | artifact_version_string: str 18 | artifact_collection_name: str 19 | project_name: str 20 | entity_name: str 21 | 22 | def __str__(self): 23 | msg = 'Payload:\n========\n' 24 | for k,v in self.model_dump().items(): 25 | msg += f'{k}={v}\n' 26 | return msg 27 | 28 | @stub.function(secret=Secret.from_name("my-random-secret"), image=img) 29 | @web_endpoint(method="POST") 30 | async def f(event: Event, token: HTTPAuthorizationCredentials = Depends(auth_scheme)): 31 | import os 32 | 33 | print(event) 34 | if token.credentials != os.environ["AUTH_TOKEN"]: 35 | raise HTTPException( 36 | status_code=status.HTTP_401_UNAUTHORIZED, 37 | detail="Incorrect bearer token", 38 | headers={"WWW-Authenticate": "Bearer"}, 39 | ) 40 | return {"message": "Event processed successfully"} 41 | --------------------------------------------------------------------------------