├── .gitignore ├── .dockerignore ├── .travis ├── docker_script.sh └── docker_push.sh ├── docs ├── mlflow_tracking_server_architecture.jpg └── mlflow_tracking_server_architecture.xml ├── .travis.yml ├── mlflow_tracking_server ├── Dockerfile └── files │ └── run.sh ├── mlflow_client_examples └── test.py ├── LICENSE └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | *.DS_Store 2 | -------------------------------------------------------------------------------- /.dockerignore: -------------------------------------------------------------------------------- 1 | * 2 | !Dockerfile 3 | !files/run.sh 4 | !.travis/docker_push.sh 5 | !.travis.yml 6 | -------------------------------------------------------------------------------- /.travis/docker_script.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | cd mlflow_tracking_server && docker build --no-cache -t mlflow-tracking-server:0.5.0 . 3 | -------------------------------------------------------------------------------- /docs/mlflow_tracking_server_architecture.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/flmu/mlflow-tracking-server/HEAD/docs/mlflow_tracking_server_architecture.jpg -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | --- 2 | sudo: required 3 | 4 | language: bash 5 | 6 | services: 7 | - docker 8 | 9 | script: 10 | - .travis/docker_script.sh 11 | 12 | deploy: 13 | provider: script 14 | script: bash .travis/docker_push.sh 15 | on: 16 | branch: master 17 | -------------------------------------------------------------------------------- /.travis/docker_push.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin 3 | cd mlflow_tracking_server && docker build --no-cache -t mlflow-tracking-server:0.5.0 . 4 | docker tag mlflow-tracking-server:0.5.0 foxrider/mlflow-tracking-server:0.5.0 5 | docker push foxrider/mlflow-tracking-server:0.5.0 6 | -------------------------------------------------------------------------------- /mlflow_tracking_server/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.9 2 | 3 | LABEL maintainer "Florian Muchow " 4 | 5 | RUN pip install --upgrade pip && \ 6 | pip install awscli --upgrade --user && \ 7 | pip install boto3==1.34.158 && \ 8 | pip install mlflow==2.15.1 9 | 10 | ENV PORT 5000 11 | 12 | COPY files/run.sh / 13 | 14 | CMD ["/run.sh"] 15 | -------------------------------------------------------------------------------- /mlflow_tracking_server/files/run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | set -e 4 | 5 | if [ -z "$FILE_DIR" ]; then 6 | echo >&2 "FILE_DIR must be set" 7 | exit 1 8 | fi 9 | 10 | if [ -z "$AWS_BUCKET" ]; then 11 | echo >&2 "AWS_BUCKET must be set" 12 | exit 1 13 | fi 14 | 15 | mkdir -p "$FILE_DIR" && mlflow server \ 16 | --backend-store-uri sqlite:///${FILE_DIR}/sqlite.db \ 17 | --default-artifact-root s3://${AWS_BUCKET}/artifacts \ 18 | --host 0.0.0.0 \ 19 | --port $PORT 20 | -------------------------------------------------------------------------------- /mlflow_client_examples/test.py: -------------------------------------------------------------------------------- 1 | import os 2 | from random import random, randint 3 | import mlflow 4 | 5 | if __name__ == "__main__": 6 | print("Running the test script ...") 7 | 8 | #Connect to tracking server 9 | mlflow.set_tracking_uri("http://127.0.0.1:5000") 10 | 11 | #Create directory for artifacts 12 | if not os.path.exists("artifact_folder"): 13 | os.makedirs("artifact_folder") 14 | 15 | #Test parametes 16 | mlflow.log_param("param1", randint(0, 100)) 17 | 18 | #Test metrics 19 | mlflow.log_metric("metric1", random()) 20 | mlflow.log_metric("metric1", random()) 21 | mlflow.log_metric("metric1", random()) 22 | mlflow.log_metric("metric1", random()) 23 | 24 | #Test artifact 25 | with open("artifact_folder/test.txt", "w") as f: 26 | f.write("hello world!") 27 | mlflow.log_artifacts("artifact_folder") 28 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Florian Muchow 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /docs/mlflow_tracking_server_architecture.xml: -------------------------------------------------------------------------------- 1 | 7Vrfj+I2EP5rIrUvp8SGBB6B3W0fdqWVqHS3T5U3MYlvnThKzAL96zsGmxBsuts2AcQdDygZ/4j9zefxzNgenuXr3ypSZk8iodxDfrL28J2HUDRA8K8Em51ggPBOkFYs2YmCRjBnf1Et9LV0yRJatypKIbhkZVsYi6KgsWzJSFWJVbvaQvD2V0uSUkswjwm3pV9ZIrOddISiRv47ZWlmvhyE413JK4nf0kosC/09D+HF9rcrzonpS0+0zkgiVgcifO/hWSWE3D3l6xnlCloD267dw4nS/bgrWsjPNBjsGrwTvqRmxNtxyY3BYjsbqur7Hp6uMibpvCSxKl2B8kGWyZzDWwCPujtaSbo+OaRgP1HgDxU5ldUGqugGeKix2RhwdRerRhPYECU71ALWQqK1n+77bhCABw2CG5CRNX+aADX0q6hkJlJREH7fSKdthA7QoGsmvx08v6gqX4bqrYBxfdMtti9N2Xcq5UavCbKUAkTNdx+FKHWPtSSVnCi2gyDmpK5ZbMQPjLdVombxzwqBSYtlFetamgfQV0p1LexWW0U5key93fv/UQGyODn5OgfBHAbgT5fxGwzo0iQdH3F0aHM0QA6Ohh1Q1Fihvjn6SX59lq7/jYfY5iEKz0REbBHx6fGBAxzI/6MCQ8+KVNGSVkClizNy4I9Aaxcj5dDC6k7ASrVxgRnK9uRrWYk3OhNcVCApRKH4ugCCHYkIZ2mhqAiIQMd4qvBisGlPdEHOkmRLdhfabX10AHiIPt6nBg60UQdoh1e4bbfRwA40Qt+1a0cd4BFZeLyADQHJjDM1/psloXHgjJ8d2qiPeuLg2N6mAYkFiWV9w3iPjha9A/C+Fr3xea9p1QcRbi97x54zwJENyLgLQAILkOcNeB6F2pXjipWw8P1fVIO8BJfEQyHJFQTFa11upx9yxcq6JNAiTNVzzhfbDd5dEwZ0WPnX2+X5wETw+wDe4UuEDr12QvReHFz/dBAW2EHYVTi/gSMKC87l/Qa9hMIaeaOHVijcKOXF+1ch7vm1gM6lBDsYPm22vpywU64WXKR/kma/PtHu0rvL+Gh3GTmskCsV1EUmaE+FH57/A5v/58oFBa4EZVf8V6MBPK+U/Md50POSH/8kv06iXJD8dlKlO/LDvEl+rdwfRO1U1lm5H9jZlUcBnjiI7phKKiyE+j8d7Z4brWNDEYYWWK4kQBdpv+CHzAIEfthCHLmi3p6yAI4juycqSUIkUdHuk97ToB//WS3xWw5Sh+hDNQQj1JMePmclGt1c2kiEHx9Z9WUlTK6wI2/i2n0Gc4jcCtt9t3a6Pz0dWby8o++Ui9JxGFNnpFSPy5xPYikOl/QjeaX8WdRMMqGW9quQUuTKeVAF0/09B2MimpsOtlmQ4ojPYik5K8DimOsbHVkEHEUtkg+Nj3B4d6AnkuNO48WrILKB6hLOL7az3ROY06a2T3NujcXHtwtQbySG1+au0bbs4D4Xvv8b -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MLFLow Tracking Server Based on Docker and AWS S3 2 | 3 | [![Build Status](https://travis-ci.com/flmu/mlflow-tracking-server.svg?branch=master)](https://travis-ci.com/github/flmu/mlflow-tracking-server) 4 | [![Code Score](https://api.codiga.io/project/17725/score/svg)](https://api.codiga.io/project/17725/score/svg) 5 | 6 | This repo provides a docker image of [MLFLow Tracking Server](https://www.mlflow.org/docs/latest/tracking.html) which is based on sqlite, an internal file system for metadata (e.g. parameters, metrics) and an [AWS S3 Bucket](https://aws.amazon.com/s3/) for [files and artifacts](https://www.mlflow.org/docs/latest/tracking.html#storage). 7 | 8 | ![Architecture](docs/mlflow_tracking_server_architecture.jpg) 9 | 10 | ## Prerequisites 11 | Before you start the MLFlow Tracking Server, you must create an AWS S3 Bucket and the corresponding credentials. 12 | 1. [AWS Account](https://aws.amazon.com/?nc2=h_lg) 13 | 2. [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) 14 | 3. Create a new AWS [S3 Bucket](https://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html) 15 | 4. Add a new [AWS User](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console) to to the system 16 | 5. Assign the following policy to the user: 17 | ```json 18 | { 19 | "Version": "2012-10-17", 20 | "Statement": [ 21 | { 22 | "Sid": "VisualEditor0", 23 | "Effect": "Allow", 24 | "Action": [ 25 | "s3:ListAllMyBuckets", 26 | "s3:HeadBucket" 27 | ], 28 | "Resource": "*" 29 | }, 30 | { 31 | "Sid": "VisualEditor1", 32 | "Effect": "Allow", 33 | "Action": "s3:*", 34 | "Resource": [ 35 | "arn:aws:s3:::<_AWS_BUCKET_NAME_>", 36 | "arn:aws:s3:::<_AWS_BUCKET_NAME_>/*" 37 | ] 38 | } 39 | ] 40 | } 41 | ``` 42 | 43 | ## Run the MLFlow Tracking Server 44 | 45 | ```bash 46 | $ docker run \ 47 | --rm \ 48 | --name mlflow-tracking-server \ 49 | -p 5000:5000 \ 50 | -e PORT=5000 \ 51 | -e FILE_DIR=/mlflow \ 52 | -e AWS_BUCKET= \ 53 | -e AWS_ACCESS_KEY_ID= \ 54 | -e AWS_SECRET_ACCESS_KEY= \ 55 | foxrider/mlflow-tracking-server:0.2.0 56 | ``` 57 | 58 | Access to http://127.0.0.1:5000 59 | 60 | ## Environment Variables for the MLFlow Tracking Server 61 | 62 | ### Required 63 | 64 | |Key|Description| 65 | |---|---| 66 | |`FILE_DIR`|Directory for artifacts and metadata (e.g. parameters, metrics)| 67 | |`AWS_BUCKET`|Name of AWS S3 Bucket that will contain the artifacts| 68 | |`AWS_ACCESS_KEY_ID`|AWS-Access-Key that you have created in the `Prerequisite` section| 69 | |`AWS_SECRET_ACCESS_KEY`|AWS-Secret-Access-Key that you have created in the `Prerequisite` section| 70 | 71 | ### Optional 72 | 73 | |Key|Description|Default| 74 | |---|---|---| 75 | |`PORT`|Value for `listen` directive|`5000`| 76 | 77 | ## Test the MLFlow Tracking Server 78 | ### Prerequisite 79 | 80 | 1. Install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/installing.html) 81 | 2. Add the AWS credentials to the client. You can either [set the environment variables](https://www.schrodinger.com/kb/1842) `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` on your client or use the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) command `aws configure`. 82 | 83 | ### Simple Python Script 84 | 85 | See this [link](mlflow_client_examples/test.py) 86 | 87 | # Contribution 88 | 89 | Pull requests are welcome :) 90 | 91 | ### Project Initiator 92 | 93 | [Florian Muchow](https://www.linkedin.com/in/florian-muchow/) 94 | 95 | Buy Me A Coffee 96 | 97 | ### Contributors 98 | 99 | [Thanks to all the contributors](https://github.com/flmu/mlflow-tracking-server/graphs/contributors) 100 | --------------------------------------------------------------------------------