├── .dockerignore ├── .github └── workflows │ └── push-docker-image.yaml ├── .gitignore ├── README.md ├── api └── swagger.yaml ├── build └── Dockerfile ├── database ├── migrations │ ├── down-migration-001-urls_table.sql │ ├── down-migration-002-address_column.sql │ ├── down-migration-003-enable_column.sql │ ├── down-migration-004-expire_column.sql │ ├── down-migration-005-updated_at_column.sql │ ├── up-migration-001-urls_table.sql │ ├── up-migration-002-address_column.sql │ ├── up-migration-003-enable_column.sql │ ├── up-migration-004-expire_column.sql │ └── up-migration-005-updated_at_column.sql └── model │ ├── __init__.py │ └── url.py ├── internal ├── config │ ├── __init__.py │ ├── config.py │ └── env.py ├── http │ ├── __init__.py │ ├── api.py │ ├── handler │ │ ├── __init__.py │ │ └── handler.py │ └── views.py ├── storage │ ├── __init__.py │ ├── minio.py │ └── mysql.py └── utils │ ├── __init__.py │ └── migrate.py ├── main.py ├── requirements.txt └── web ├── static ├── css │ └── main.css └── js │ └── app.js └── template ├── base.html ├── bucket.j2 ├── download.j2 ├── help.j2 └── index.j2 /.dockerignore: -------------------------------------------------------------------------------- 1 | assets/ 2 | build/ 3 | api/ 4 | README.md 5 | .gitignore 6 | .dockerignore -------------------------------------------------------------------------------- /.github/workflows/push-docker-image.yaml: -------------------------------------------------------------------------------- 1 | name: release 2 | 3 | on: 4 | push: 5 | tags: 6 | - 'v*.*.*' 7 | 8 | env: 9 | DOCKER_USER: ${{ secrets.DOCKER_USER }} 10 | DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }} 11 | REPO_NAME: ${{ secrets.REPO_NAME }} 12 | 13 | jobs: 14 | push_to_registry: 15 | name: Push Docker image to Docker Hub 16 | runs-on: ubuntu-latest 17 | steps: 18 | - name: Check out the repo 19 | uses: actions/checkout@v3 20 | 21 | - name: Set output 22 | id: vars 23 | run: echo "tag=${GITHUB_REF#refs/*/}" >> $GITHUB_OUTPUT 24 | 25 | - name: Log in to Docker Hub 26 | uses: docker/login-action@v2 27 | with: 28 | username: ${{ env.DOCKER_USER }} 29 | password: ${{ env.DOCKER_PASSWORD }} 30 | 31 | - name: Set up Docker Buildx 32 | uses: docker/setup-buildx-action@v2 33 | 34 | - name: Build and push Docker image 35 | uses: docker/build-push-action@v2 36 | with: 37 | context: . 38 | file: ./build/Dockerfile 39 | push: true 40 | tags: ${{ env.DOCKER_USER }}/${{ env.REPO_NAME }}:${{ steps.vars.outputs.tag }} 41 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Prerequisites 2 | *.d 3 | 4 | .idea/ 5 | .DS_Store 6 | 7 | # Object files 8 | *.o 9 | *.ko 10 | *.obj 11 | *.elf 12 | 13 | # Linker output 14 | *.ilk 15 | *.map 16 | *.exp 17 | 18 | # Precompiled Headers 19 | *.gch 20 | *.pch 21 | 22 | # Libraries 23 | *.lib 24 | *.a 25 | *.la 26 | *.lo 27 | 28 | # Shared objects (inc. Windows DLLs) 29 | *.dll 30 | *.so 31 | *.so.* 32 | *.dylib 33 | 34 | # Executables 35 | *.exe 36 | *.out 37 | *.app 38 | *.i*86 39 | *.x86_64 40 | *.hex 41 | 42 | # Debug files 43 | *.dSYM/ 44 | *.su 45 | *.idb 46 | *.pdb 47 | 48 | # Kernel Module Compile Results 49 | *.mod* 50 | *.cmd 51 | .tmp_versions/ 52 | modules.order 53 | Module.symvers 54 | Mkfile.old 55 | dkms.conf 56 | 57 | 58 | .idea/ 59 | .DS_Store 60 | .vendor 61 | __pycache__/ 62 | test.txt -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Minio URL Operator 2 | 3 | ![](https://img.shields.io/badge/Language-Python-blue) 4 | ![](https://img.shields.io/badge/Storage-Minio-lightgrey) 5 | ![GitHub release (with filter)](https://img.shields.io/github/v/release/amirhnajafiz/minio-url-operator) 6 | 7 | 8 | URL operator for Minio shared objects links. Since Minio object storage provides 9 | urls for sharing objects with a maximum 7 days limit, you can use this operator 10 | in order to have live urls for every object that you want. 11 | With this operator, you can create a persistent url for your objects in Minio. 12 | 13 | ## Start 14 | 15 | You can use ```docker``` image of MUP in order to setup the operator on ```Docker``` or ```Kubernetes```. 16 | 17 | ### image 18 | 19 | ```shell 20 | docker pull amirhossein21/muo:v0.4.0 21 | ``` 22 | 23 | ### environment variables 24 | 25 | | Name | Type | Example | Description | 26 | |:--------------------:|:----------:|----------------------|-------------------------| 27 | | ```HTTP_PORT``` | ```int``` | ```8080``` | HTTP port of MUO API | 28 | | ```HTTP_DEBUG``` | ```bool``` | ```true``` | Debug flag for logging | 29 | | ```HTTP_HOST``` | ```str``` | ```127.0.0.1``` | Container host name | 30 | | ```HTTP_PRIVATE``` | ```bool``` | ```false``` | Private host or not | 31 | | ```MYSQL_HOST``` | ```str``` | ```127.0.0.1``` | MySQL cluster host | 32 | | ```MYSQL_PORT``` | ```int``` | ```3306``` | MySQL cluster port | 33 | | ```MYSQL_USER``` | ```str``` | ```root``` | MySQL user | 34 | | ```MYSQL_PASSWORD``` | ```str``` | ```pa$$word``` | MySQL pass | 35 | | ```MYSQL_DB``` | ```str``` | ```minio-db``` | MySQL database | 36 | | ```MYSQL_MIGRATE``` | ```bool``` | ```false``` | Database migration | 37 | | ```MINIO_HOST``` | ```str``` | ```localhost:9000``` | Minio cluster host | 38 | | ```MINIO_SECURE``` | ```bool``` | ```false``` | Secure Minio connection | 39 | | ```MINIO_ACCESS``` | ```str``` | - | Minio access token | 40 | | ```MINIO_SECRET``` | ```str``` | - | Minio secret token | 41 | 42 | 43 | ### start 44 | 45 | ```shell 46 | docker run -d -it \ 47 | -e HTTP_PORT=80 -e HTTP_DEBUG=0 -e MINIO_HOST=localhost:9000 \ 48 | -e MINIO_SECURE=0 -e MINIO_ACCESS=9iWKawYzq68iNMN7MsiU \ 49 | -e MINIO_SECRET=zWwZlmTX56Hr8NYBOpN4ga2zV8oO2ECIjjPHPF20 \ 50 | -e HTTP_HOST=localhost -e HTTP_PRIVATE=1 \ 51 | amirhossein21/muo:v0.4.0 52 | ``` 53 | 54 | ## API 55 | 56 | In order to use the operator APIs, you can read the ```swagger``` documents in [docs](./api/swagger.yaml). 57 | -------------------------------------------------------------------------------- /api/swagger.yaml: -------------------------------------------------------------------------------- 1 | openapi: 3.0.3 2 | info: 3 | title: MUO API - OpenAPI 3.0 4 | description: |- 5 | This is a document for using Minio URL operator APIs in order to 6 | manage links of the objects. 7 | version: 0.4.0 8 | servers: 9 | - url: http://localhost/api 10 | tags: 11 | - name: objects 12 | description: Manages the objects of Minio cluster 13 | paths: 14 | /objects: 15 | get: 16 | tags: 17 | - objects 18 | summary: Get a list of Minio cluster objects meta-data 19 | description: Get a list of all available Minio cluster objects meta-data 20 | parameters: 21 | - in: query 22 | name: prefix 23 | description: Objects name prefix 24 | schema: 25 | type: string 26 | - in: query 27 | name: bucket 28 | required: true 29 | description: Objects bucket in Minio 30 | schema: 31 | type: string 32 | responses: 33 | '200': 34 | description: Successful operation 35 | content: 36 | application/json: 37 | schema: 38 | type: array 39 | items: 40 | type: object 41 | properties: 42 | name: 43 | type: string 44 | example: "image.jpeg" 45 | address: 46 | type: string 47 | example: "8jr2i5mt01" 48 | status: 49 | type: integer 50 | example: 0 51 | enum: [0,1] 52 | updated_at: 53 | type: string 54 | example: "2023-06-01 09:45:46.4501" 55 | format: Date 56 | expires_at: 57 | type: string 58 | example: "2023-06-01 09:45:46.4501" 59 | format: Date 60 | created_at: 61 | type: string 62 | example: "2023-06-01 09:45:46.4501" 63 | format: Date 64 | '400': 65 | description: Empty bucket in input 66 | /objects/{address}: 67 | get: 68 | tags: 69 | - objects 70 | summary: Get object by address 71 | description: Redirect to Minio object by given address 72 | parameters: 73 | - name: address 74 | in: path 75 | description: Object address 76 | schema: 77 | type: string 78 | required: True 79 | example: 'p0mm8oee34' 80 | responses: 81 | '404': 82 | description: Address not found 83 | '303': 84 | description: Object found successfully 85 | /objects/{bucket}/{key}: 86 | get: 87 | tags: 88 | - objects 89 | summary: Get an Object address 90 | description: Return the object address in operator 91 | parameters: 92 | - name: bucket 93 | in: path 94 | description: Bucket name 95 | schema: 96 | type: string 97 | required: True 98 | example: 'snapp' 99 | - name: key 100 | in: path 101 | description: Object name 102 | schema: 103 | type: string 104 | required: True 105 | example: 'image.jpeg' 106 | responses: 107 | '200': 108 | description: Successful operation 109 | content: 110 | application/json: 111 | schema: 112 | type: object 113 | properties: 114 | address: 115 | type: string 116 | example: 'http://localhost/api/objects/p0mm8oee34' 117 | post: 118 | tags: 119 | - objects 120 | summary: Update object status 121 | description: Set new status for object in order to manage the access 122 | parameters: 123 | - name: bucket 124 | in: path 125 | description: Bucket name 126 | schema: 127 | type: string 128 | required: True 129 | example: 'snapp' 130 | - name: key 131 | in: path 132 | description: Object name 133 | schema: 134 | type: string 135 | required: True 136 | example: 'image.jpeg' 137 | requestBody: 138 | content: 139 | application/json: 140 | schema: 141 | type: object 142 | properties: 143 | status: 144 | type: integer 145 | enum: [0,1] 146 | description: "Object enable or disable" 147 | expires: 148 | type: string 149 | example: "2023-06-01 09:45:46.4501" 150 | format: Date 151 | responses: 152 | '200': 153 | description: Successful operation 154 | -------------------------------------------------------------------------------- /build/Dockerfile: -------------------------------------------------------------------------------- 1 | # from base image 2 | FROM python:3.10-slim-buster 3 | 4 | # src/app directory 5 | WORKDIR /src/app 6 | 7 | LABEL app=python 8 | LABEL function=http 9 | LABEL name=minio-operator 10 | 11 | # copy requirement file 12 | COPY requirements.txt . 13 | 14 | # install them 15 | RUN pip3 install -r requirements.txt 16 | 17 | # copy all files 18 | COPY . . 19 | 20 | # start script 21 | CMD python3 main.py 22 | -------------------------------------------------------------------------------- /database/migrations/down-migration-001-urls_table.sql: -------------------------------------------------------------------------------- 1 | DROP TABLE urls; 2 | -------------------------------------------------------------------------------- /database/migrations/down-migration-002-address_column.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE urls 2 | DROP COLUMN address; 3 | -------------------------------------------------------------------------------- /database/migrations/down-migration-003-enable_column.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE urls 2 | DROP COLUMN status; 3 | -------------------------------------------------------------------------------- /database/migrations/down-migration-004-expire_column.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE urls 2 | DROP COLUMN expires_at; 3 | -------------------------------------------------------------------------------- /database/migrations/down-migration-005-updated_at_column.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE urls 2 | DROP COLUMN updated_at; 3 | -------------------------------------------------------------------------------- /database/migrations/up-migration-001-urls_table.sql: -------------------------------------------------------------------------------- 1 | CREATE TABLE urls ( 2 | id INTEGER PRIMARY KEY AUTO_INCREMENT, 3 | bucket VARCHAR(1024) NOT NULL, 4 | object_key VARCHAR(1024) NOT NULL, 5 | url VARCHAR(2048) NOT NULL, 6 | created_at DATETIME 7 | ); 8 | -------------------------------------------------------------------------------- /database/migrations/up-migration-002-address_column.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE urls 2 | ADD COLUMN address VARCHAR(1024) NOT NULL; 3 | -------------------------------------------------------------------------------- /database/migrations/up-migration-003-enable_column.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE urls 2 | ADD COLUMN status INTEGER; 3 | -------------------------------------------------------------------------------- /database/migrations/up-migration-004-expire_column.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE urls 2 | ADD COLUMN expires_at DATETIME; 3 | -------------------------------------------------------------------------------- /database/migrations/up-migration-005-updated_at_column.sql: -------------------------------------------------------------------------------- 1 | ALTER TABLE urls 2 | ADD COLUMN updated_at DATETIME; 3 | -------------------------------------------------------------------------------- /database/model/__init__.py: -------------------------------------------------------------------------------- 1 | # Module: Model 2 | # Purpose: Database models and queries 3 | -------------------------------------------------------------------------------- /database/model/url.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | 3 | 4 | class URL(object): 5 | """URL is our url database model""" 6 | 7 | def __init__(self, bucket="", key="", url="", address=""): 8 | self.id = None 9 | self.bucket = bucket 10 | self.key = key 11 | self.url = url 12 | self.createdAt = datetime.now() 13 | self.updatedAt = self.createdAt 14 | self.address = address 15 | self.status = 1 16 | self.expiresAt = None 17 | 18 | def read(self, row: tuple): 19 | """read values from database row 20 | 21 | :param row: database row 22 | :return: None 23 | """ 24 | self.id = row[0] 25 | self.bucket = row[1] 26 | self.key = row[2] 27 | self.url = row[3] 28 | self.createdAt = row[4] 29 | self.address = row[5] 30 | self.status = row[6] 31 | self.expiresAt = row[7] 32 | self.updatedAt = row[8] 33 | 34 | def write(self) -> list: 35 | """write values into a list 36 | 37 | :return: list 38 | """ 39 | return [ 40 | self.bucket, 41 | self.key, 42 | self.url, 43 | self.createdAt, 44 | self.address, 45 | self.status, 46 | self.expiresAt, 47 | self.updatedAt 48 | ] 49 | -------------------------------------------------------------------------------- /internal/config/__init__.py: -------------------------------------------------------------------------------- 1 | # Module: config 2 | # Purpose: reading config variables from env 3 | -------------------------------------------------------------------------------- /internal/config/config.py: -------------------------------------------------------------------------------- 1 | from .env import read_value_from_env 2 | 3 | 4 | class Config(object): 5 | """Config object is used to store config variables of application""" 6 | 7 | def __init__(self): 8 | self.host = None 9 | self.private = False 10 | self.port = None 11 | self.debug = True 12 | 13 | self.minio = {} 14 | self.mysql = {} 15 | 16 | def load(self) -> (str, bool): 17 | """"load configs into class fields 18 | 19 | :returns: (error type, error flag) 20 | """ 21 | try: 22 | self.host = read_value_from_env("HTTP_HOST") 23 | 24 | self.private = read_value_from_env("HTTP_PRIVATE") 25 | self.private = True if self.private == "true" else False 26 | 27 | self.port = int(read_value_from_env("HTTP_PORT")) 28 | 29 | self.debug = read_value_from_env("HTTP_DEBUG") 30 | self.debug = True if self.debug == "true" else False 31 | 32 | self.mysql['host'] = read_value_from_env("MYSQL_HOST") 33 | self.mysql['port'] = int(read_value_from_env("MYSQL_PORT")) 34 | self.mysql['user'] = read_value_from_env("MYSQL_USER") 35 | self.mysql['pass'] = read_value_from_env("MYSQL_PASSWORD") 36 | self.mysql['name'] = read_value_from_env("MYSQL_DB") 37 | 38 | self.mysql['migrate'] = read_value_from_env("MYSQL_MIGRATE") 39 | self.mysql['migrate'] = True if self.mysql['migrate'] == "true" else False 40 | 41 | self.minio['host'] = read_value_from_env("MINIO_HOST") 42 | self.minio['access'] = read_value_from_env("MINIO_ACCESS") 43 | self.minio['secret'] = read_value_from_env("MINIO_SECRET") 44 | 45 | self.minio['secure'] = read_value_from_env("MINIO_SECURE") 46 | self.minio['secure'] = True if self.minio['secure'] == "true" else False 47 | 48 | except Exception as e: 49 | return f"[config.load] failed to read params error={e}", True 50 | 51 | return "OK", False 52 | -------------------------------------------------------------------------------- /internal/config/env.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | 4 | def read_value_from_env(key: str) -> str: 5 | """Read environment variable. 6 | 7 | :param key: name of the variable 8 | :return: value of the variable 9 | """ 10 | return os.getenv(key) 11 | -------------------------------------------------------------------------------- /internal/http/__init__.py: -------------------------------------------------------------------------------- 1 | # Module: http 2 | # Purpose: Creating a http server 3 | -------------------------------------------------------------------------------- /internal/http/api.py: -------------------------------------------------------------------------------- 1 | from flask import Blueprint, request, jsonify, render_template 2 | 3 | from ..storage.mysql import MySQL 4 | from ..storage.minio import MinioConnector 5 | from .handler.handler import Handler 6 | 7 | 8 | class API(object): 9 | """API manages the backend rest api""" 10 | 11 | def __init__(self, database: MySQL, minio_connection: MinioConnector, host="", private=False): 12 | # create a blueprint for application apis 13 | self.blueprint = Blueprint('api_blueprint', __name__, url_prefix="/api") 14 | 15 | # create a new handler 16 | api = Handler(database, minio_connection) 17 | 18 | @self.blueprint.route("/objects", methods=['GET']) 19 | def get_objects(): 20 | """get objects metadata 21 | 22 | :return: list of objects of a bucket with prefix 23 | """ 24 | bucket = request.args.get("bucket", "") 25 | if bucket == "": 26 | return "Bucket cannot be empty", 400 27 | 28 | return jsonify(api.get_objects_metadata(bucket, request.args.get("prefix", ""))), 200 29 | 30 | @self.blueprint.route("/objects/
", methods=['GET']) 31 | def redirect_address(address): 32 | """redirect to shared storage if enabled 33 | 34 | :param address: object address in our system 35 | """ 36 | url = api.get_object_url_by_address(address) 37 | if len(url) == 0: 38 | return "Address does not exists", 404 39 | 40 | return render_template("download.j2", address=url), 200 41 | 42 | @self.blueprint.route("/objects//", methods=['POST']) 43 | def update_object(bucket, key): 44 | """update object enable or disable 45 | 46 | :param bucket: object bucket 47 | :param key: object key 48 | """ 49 | content = request.json 50 | 51 | api.update_object(bucket, key, content['status'], content['expires']) 52 | 53 | return "OK", 200 54 | 55 | @self.blueprint.route("/objects//", methods=['GET']) 56 | def get_object_address(bucket, key): 57 | """get object url 58 | 59 | :param bucket: object bucket 60 | :param key: object key 61 | :return: object url 62 | """ 63 | address = api.get_object_address(bucket, key) 64 | 65 | if private: 66 | uri = f"https://{host}/api/objects/{address}" 67 | else: 68 | uri = f"http://{host}/api/objects/{address}" 69 | 70 | return { 71 | 'address': uri 72 | } 73 | 74 | def get_blue_print(self) -> Blueprint: 75 | """get api blueprint 76 | 77 | :return: flask blueprint 78 | """ 79 | return self.blueprint 80 | -------------------------------------------------------------------------------- /internal/http/handler/__init__.py: -------------------------------------------------------------------------------- 1 | # Module: Handler 2 | # Purpose: handle the http requests and the logic of our program 3 | -------------------------------------------------------------------------------- /internal/http/handler/handler.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime, timedelta 2 | import random 3 | import string 4 | 5 | from ...storage.mysql import MySQL 6 | from ...storage.minio import MinioConnector 7 | from database.model.url import URL 8 | 9 | 10 | def get_random_string(length: int) -> str: 11 | """generate a random string for address 12 | 13 | :param length: size of string 14 | :return: random string 15 | """ 16 | return ''.join(random.choice(string.ascii_lowercase) for _ in range(length)) 17 | 18 | 19 | class Handler(object): 20 | """Handler manages the logic of our backend""" 21 | 22 | def __init__(self, database: MySQL, minio_connection: MinioConnector): 23 | self.database = database 24 | self.minio_connection = minio_connection 25 | self.time_factor = 3600 * 24 * 6 26 | self.time_limit = 7 27 | 28 | def get_objects_metadata(self, bucket, prefix="") -> list: 29 | """get objects of a bucket based on prefix 30 | 31 | :param bucket: minio bucket 32 | :param prefix: objects prefix 33 | :return: list of objects metadata 34 | """ 35 | client = self.minio_connection.get_connection() 36 | 37 | objects = client.list_objects(bucket, prefix=prefix) 38 | 39 | objects_list = [] 40 | 41 | for item in objects: 42 | tmp = self.__get_object__(bucket, item.object_name) 43 | 44 | object_pack = { 45 | 'name': item.object_name, 46 | 'address': tmp.address, 47 | 'status': tmp.status, 48 | 'created_at': tmp.createdAt, 49 | 'updated_at': tmp.updatedAt, 50 | 'expires_at': tmp.expiresAt 51 | } 52 | 53 | objects_list.append(object_pack) 54 | 55 | return objects_list 56 | 57 | def __get_object__(self, bucket: str, key: str) -> URL: 58 | """get object from database 59 | 60 | :param bucket: object bucket 61 | :param key: object key 62 | :return: URL if exists, None if not exists 63 | """ 64 | # get a new cursor 65 | cursor = self.database.get_cursor() 66 | 67 | # select a url from database 68 | cursor.execute(f'SELECT * FROM `urls` WHERE `bucket` = %s AND `object_key` = %s', [bucket, key]) 69 | 70 | # fetch the first item 71 | record = cursor.fetchone() 72 | if record is None: # if not found then we register it 73 | self.__register_object(bucket, key) 74 | 75 | # now we read it again 76 | cursor.execute(f'SELECT * FROM `urls` WHERE `bucket` = %s AND `object_key` = %s', [bucket, key]) 77 | record = cursor.fetchone() 78 | 79 | # create the url 80 | url = URL() 81 | url.read(record) 82 | 83 | cursor.close() 84 | 85 | return url 86 | 87 | def __check_url_time__(self, url: URL) -> bool: 88 | """check if the url is expired or not 89 | 90 | :param url: input url object 91 | :return: true or false 92 | """ 93 | t1 = url.createdAt.date() 94 | t2 = datetime.now().date() 95 | 96 | return ((t2 - t1).total_seconds() / self.time_factor) < self.time_limit 97 | 98 | def __create_url_for_object__(self, bucket: str, key: str) -> str: 99 | """create url for object in minio 100 | 101 | :param bucket: object bucket 102 | :param key: object name 103 | :return: url of object 104 | """ 105 | client = self.minio_connection.get_connection() 106 | 107 | return client.presigned_get_object( 108 | bucket, key, expires=timedelta(days=self.time_limit), 109 | ) 110 | 111 | def __create_object__(self, url: URL): 112 | """create a new object in database 113 | 114 | :param url: url object 115 | """ 116 | # get a new cursor 117 | cursor = self.database.get_cursor() 118 | 119 | cursor.execute( 120 | '''INSERT INTO `urls` (bucket, object_key, url, created_at, address, status, expires_at, updated_at) 121 | VALUES (%s,%s,%s,%s,%s,%s,%s,%s);''', 122 | url.write() 123 | ) 124 | 125 | self.database.commit() 126 | 127 | cursor.close() 128 | 129 | def __update_object_url__(self, url: URL): 130 | """update url for an object 131 | 132 | :param url: url object 133 | """ 134 | # get a new cursor 135 | cursor = self.database.get_cursor() 136 | 137 | cursor.execute( 138 | '''UPDATE `urls` SET `url` = %s, `created_at` = %s WHERE `id` = %s''', 139 | [url.url, url.createdAt, url.id] 140 | ) 141 | 142 | self.database.commit() 143 | 144 | cursor.close() 145 | 146 | def update_object(self, bucket: str, key: str, status: int, expires: datetime): 147 | """update url status to set enable value 148 | 149 | :param bucket: object bucket 150 | :param key: object key 151 | :param status: object status 152 | :param expires: expire time 153 | """ 154 | # get a new cursor 155 | cursor = self.database.get_cursor() 156 | 157 | cursor.execute( 158 | '''UPDATE `urls` 159 | SET `status` = %s, `expires_at` = %s, `updated_at` = %s 160 | WHERE `bucket` = %s AND `object_key` = %s''', 161 | [status, expires, datetime.now(), bucket, key] 162 | ) 163 | self.database.commit() 164 | 165 | cursor.close() 166 | 167 | def get_object_address(self, bucket: str, key: str) -> str: 168 | """get address of an object 169 | 170 | :param bucket: object bucket 171 | :param key: object key 172 | :return: address 173 | """ 174 | # get a new cursor 175 | cursor = self.database.get_cursor() 176 | 177 | cursor.execute("SELECT `address` FROM `urls` WHERE `bucket` = %s AND `object_key` = %s", [bucket, key]) 178 | 179 | url = cursor.fetchone() 180 | 181 | cursor.close() 182 | 183 | return url[0] 184 | 185 | def __register_object(self, bucket, key): 186 | """register an object into our system 187 | 188 | :param bucket: bucket name 189 | :param key: object key 190 | """ 191 | address = get_random_string(10) 192 | url = URL(bucket, key, self.__create_url_for_object__(bucket, key), address) 193 | url.createdAt = datetime.now() 194 | 195 | self.__create_object__(url) 196 | 197 | def get_object_url_by_address(self, address: str) -> str: 198 | """get object url by its address 199 | 200 | :param address: object address 201 | :return: url of that object 202 | """ 203 | # get a new cursor 204 | cursor = self.database.get_cursor() 205 | 206 | cursor.execute( 207 | '''SELECT * FROM `urls` 208 | WHERE `address` = %s AND `status` = 1 AND `expires_at` > NOW()''', 209 | [address] 210 | ) 211 | 212 | row = cursor.fetchone() 213 | if row is None: 214 | return "" 215 | 216 | url = URL() 217 | url.read(row) 218 | 219 | cursor.close() 220 | 221 | url = self.__get_object__(url.bucket, url.key) 222 | 223 | if not self.__check_url_time__(url): # if it was expired create new one 224 | url.url = self.__create_url_for_object__(url.bucket, url.key) 225 | url.createdAt = datetime.now() 226 | 227 | self.__update_object_url__(url) 228 | 229 | return url.url 230 | -------------------------------------------------------------------------------- /internal/http/views.py: -------------------------------------------------------------------------------- 1 | from flask import Blueprint, render_template 2 | 3 | 4 | class Views(object): 5 | def __init__(self): 6 | # create a blueprint for views 7 | self.blueprint = Blueprint('views_blueprint', __name__) 8 | 9 | @self.blueprint.route('/', methods=['GET']) 10 | def index_page(): 11 | """home page of application 12 | 13 | :return: index.html template 14 | """ 15 | return render_template('index.j2', title="MUO: Home") 16 | 17 | @self.blueprint.route('/search', methods=['GET']) 18 | def search_page(): 19 | return render_template('bucket.j2', title="MUO: Buckets") 20 | 21 | @self.blueprint.route('/docs', methods=['GET']) 22 | def help_page(): 23 | return render_template('help.j2', title="MUO: Docs") 24 | 25 | def get_blue_print(self) -> Blueprint: 26 | """get views blueprint 27 | 28 | :return: flask blueprint 29 | """ 30 | return self.blueprint 31 | -------------------------------------------------------------------------------- /internal/storage/__init__.py: -------------------------------------------------------------------------------- 1 | # Module: Storage 2 | # Purpose: Methods and classes in order to connect to SQL server and Minio cluster 3 | -------------------------------------------------------------------------------- /internal/storage/minio.py: -------------------------------------------------------------------------------- 1 | from minio import Minio 2 | 3 | 4 | class MinioConnector(object): 5 | """MinioConnector is used to connect to Minio cluster""" 6 | 7 | def __init__(self, host: str, access: str, secret: str, secure: bool): 8 | self.conn = Minio( 9 | host, 10 | access_key=access, 11 | secret_key=secret, 12 | secure=secure, 13 | ) 14 | 15 | def ping(self) -> (str, bool): 16 | """ping minio cluster 17 | 18 | :return: (error message, error flag) 19 | """ 20 | try: 21 | self.conn.bucket_exists("test_bucket") 22 | except Exception as e: 23 | return f"[minioConnector.ping]' failed to connect to minio cluster error={e}", True 24 | 25 | return "OK", False 26 | 27 | def get_connection(self) -> Minio: 28 | """get minio connection 29 | 30 | :return: minio connection 31 | """ 32 | return self.conn 33 | -------------------------------------------------------------------------------- /internal/storage/mysql.py: -------------------------------------------------------------------------------- 1 | import mysql.connector 2 | 3 | 4 | class MySQL(object): 5 | """MySQL manages the connection to database""" 6 | 7 | def __init__(self, host: str, port: int, user: str, password: str, database: str): 8 | # opening a connection to mysql server 9 | self.connection = mysql.connector.connect( 10 | host=host, 11 | port=port, 12 | user=user, 13 | password=password, 14 | database=database 15 | ) 16 | 17 | def ping(self) -> bool: 18 | """return a boolean to check database connection""" 19 | return self.connection.is_connected() 20 | 21 | def get_cursor(self) -> mysql.connector.connection.MySQLCursor: 22 | """returns a cursor of connection""" 23 | return self.connection.cursor() 24 | 25 | def commit(self): 26 | """commit updates on connection""" 27 | self.connection.commit() 28 | 29 | def close_connection(self): 30 | """closes the database connection""" 31 | self.connection.close() 32 | -------------------------------------------------------------------------------- /internal/utils/__init__.py: -------------------------------------------------------------------------------- 1 | # Module: utils 2 | # Purpose: extra functions in our system 3 | -------------------------------------------------------------------------------- /internal/utils/migrate.py: -------------------------------------------------------------------------------- 1 | import mysql.connector 2 | import os 3 | import logging 4 | 5 | 6 | DIRECTORY = "./database/migrations" 7 | 8 | 9 | def migrate(connection: mysql.connector.connection.MySQLCursor): 10 | """migrate database sql files 11 | 12 | :param connection: mysql connection cursor 13 | """ 14 | files = [filename for filename in os.listdir(DIRECTORY) if filename.startswith('up')] 15 | files.sort() 16 | 17 | for file in files: 18 | with open(DIRECTORY+"/"+file, 'r') as f: 19 | query = f.read() 20 | connection.execute(query) 21 | 22 | logging.info(f"migrated: {file}") 23 | 24 | connection.close() 25 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | import sys 3 | import logging 4 | 5 | # api modules 6 | from internal.http.api import API 7 | from internal.http.views import Views 8 | 9 | # storage connections 10 | from internal.storage.minio import MinioConnector 11 | from internal.storage.mysql import MySQL 12 | 13 | # config module 14 | from internal.config.config import Config 15 | 16 | 17 | # set logging module 18 | logging.basicConfig( 19 | stream=sys.stdout, 20 | level=logging.DEBUG, 21 | format='[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s', 22 | ) 23 | 24 | # load app configs 25 | cfg = Config() 26 | 27 | error, flag = cfg.load() 28 | if flag: # if error occurs 29 | logging.error(error) 30 | sys.exit(-1) 31 | 32 | 33 | # open connection to storages 34 | # mysql 35 | dbConnection = MySQL( 36 | host=cfg.mysql['host'], 37 | port=cfg.mysql['port'], 38 | user=cfg.mysql['user'], 39 | password=cfg.mysql['pass'], 40 | database=cfg.mysql['name'] 41 | ) 42 | 43 | # minio 44 | minioConnection = MinioConnector( 45 | host=cfg.minio['host'], 46 | access=cfg.minio['access'], 47 | secret=cfg.minio['secret'], 48 | secure=False, 49 | ) 50 | 51 | 52 | # create a new flask application 53 | app = Flask(__name__, 54 | static_url_path='/', 55 | static_folder='web/static', 56 | template_folder='web/template') 57 | 58 | # register blueprints 59 | app.register_blueprint(API(dbConnection, minioConnection, f'{cfg.host}:{cfg.port}', cfg.private).get_blue_print()) 60 | app.register_blueprint(Views().get_blue_print()) 61 | 62 | 63 | if __name__ == "__main__": 64 | # check mysql connection 65 | if not dbConnection.ping(): 66 | logging.error("mysql connection failed!") 67 | sys.exit(-2) 68 | 69 | # check minio connection 70 | errorM, flag = minioConnection.ping() 71 | if flag: 72 | logging.error(errorM) 73 | sys.exit(-3) 74 | 75 | # migrate database if needed 76 | if cfg.mysql['migrate']: 77 | from internal.utils.migrate import migrate 78 | migrate(dbConnection.get_cursor()) 79 | 80 | logging.info(f"operator started on port: {cfg.port} ...") 81 | app.run("127.0.0.1", cfg.port, debug=cfg.debug) 82 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | flask~=2.3.2 2 | minio~=7.1.14 3 | mysql-connector-python~=8.0.33 4 | -------------------------------------------------------------------------------- /web/static/css/main.css: -------------------------------------------------------------------------------- 1 | * { 2 | box-sizing: border-box; 3 | margin: 0; 4 | padding: 0; 5 | font-family: "Apple SD Gothic Neo", monospace; 6 | } 7 | 8 | .container { 9 | margin: 50px auto; 10 | width: 80%; 11 | min-height: 550px; 12 | border: 1px solid #e7e7e7; 13 | border-radius: 20px; 14 | box-shadow: 0 2px 10px rgba(0,0,0,0.3); 15 | 16 | display: grid; 17 | grid-template-columns: 20% 80%; 18 | } 19 | 20 | .navigation { 21 | padding: 15px; 22 | background-color: #e7e7e7; 23 | border-top-left-radius: 15px; 24 | border-bottom-left-radius: 15px; 25 | } 26 | 27 | .nav-item { 28 | color: #000000; 29 | margin: 15px 20px; 30 | padding: 7px 30px; 31 | border-radius: 15px; 32 | display: flex; 33 | align-items: center; 34 | text-decoration: none; 35 | border: 1px solid #000000; 36 | } 37 | 38 | .nav-item:hover, .active-item { 39 | color: #ffffff; 40 | background-color: #333333; 41 | } 42 | 43 | .nav-item > span { 44 | margin-left: 10px; 45 | } 46 | 47 | .context { 48 | border-bottom-right-radius: 20px; 49 | border-top-right-radius: 20px; 50 | border-left: 2px solid #e7e7e7; 51 | padding: 25px; 52 | } 53 | 54 | .page-header { 55 | margin-bottom: 20px; 56 | border-radius: 15px; 57 | background-color: #ffff; 58 | padding: 25px; 59 | } 60 | 61 | .page-description { 62 | border-radius: 15px; 63 | background-color: #ffff; 64 | padding: 25px; 65 | text-align: justify; 66 | } 67 | 68 | .row { 69 | margin: 20px 0; 70 | border-radius: 15px; 71 | background-color: #ffff; 72 | padding: 25px; 73 | } 74 | 75 | .input-row { 76 | display: flex; 77 | justify-content: space-between; 78 | padding-bottom: 25px; 79 | } 80 | 81 | .in-row { 82 | margin: 20px 0; 83 | } 84 | 85 | .in-row > table { 86 | width: 100%; 87 | border-collapse: collapse; 88 | } 89 | 90 | .in-row > table > th, .in-row > table > td { 91 | padding: 15px 0; 92 | margin: 0; 93 | } 94 | 95 | .in-row > table > th { 96 | border-bottom: 1px solid #000000; 97 | } 98 | 99 | .in-row > table > td { 100 | text-align: center; 101 | border-bottom: 1px solid #000000; 102 | } 103 | 104 | .input { 105 | padding: 8px 12px; 106 | border: 2px solid #F0F8FFFF; 107 | border-radius: 3px; 108 | } 109 | 110 | .btn { 111 | padding: 5px 10px; 112 | background-color: #008147; 113 | color: #ffff; 114 | border-radius: 5px; 115 | outline: none; 116 | border: 0 solid black; 117 | } 118 | 119 | .btn:hover { 120 | background-color: #00c77c; 121 | } 122 | 123 | .search-btn { 124 | display: flex; 125 | align-items: center; 126 | align-content: center; 127 | } 128 | 129 | .red-color { 130 | color: #ff310d; 131 | } 132 | 133 | .url-btn { 134 | background-color: #008147; 135 | color: #ffff; 136 | display: inline-flex; 137 | align-items: center; 138 | } 139 | 140 | .url-btn:hover { 141 | background-color: #00c77c; 142 | } 143 | 144 | .enable-btn { 145 | background-color: #c2c2c2; 146 | color: #1e1e1e; 147 | display: inline-flex; 148 | align-items: center; 149 | } 150 | 151 | .enable-btn:hover { 152 | background-color: #a1a1a1; 153 | } 154 | 155 | .disable-btn { 156 | background-color: #ff2c00; 157 | color: #ffff; 158 | display: inline-flex; 159 | align-items: center; 160 | } 161 | 162 | .disable-btn:hover { 163 | background-color: #da2500; 164 | } 165 | 166 | .register-btn { 167 | background-color: #ffba00; 168 | color: #000000; 169 | display: inline-flex; 170 | align-items: center; 171 | } 172 | 173 | .register-btn:hover { 174 | background-color: #cb9200; 175 | } 176 | 177 | .border-right { 178 | border-right: 1px solid #000000; 179 | } -------------------------------------------------------------------------------- /web/static/js/app.js: -------------------------------------------------------------------------------- 1 | const downloadIcon = '\n' + 2 | ' \n' + 3 | ''; 4 | 5 | 6 | // create tables from object names. 7 | function generateTable(bucket, data) { 8 | let responseDiv = document.getElementById("response-div"); 9 | let mainTable = document.createElement("table"); 10 | 11 | let numberHeader = document.createElement("th"); 12 | 13 | let nameHeader = document.createElement("th"); 14 | nameHeader.innerText = "Object name"; 15 | 16 | let dateHeader = document.createElement("th"); 17 | dateHeader.innerText = "Created Time"; 18 | 19 | let linkHeader = document.createElement("th"); 20 | linkHeader.innerText = "Object link"; 21 | 22 | mainTable.appendChild(numberHeader); 23 | mainTable.appendChild(nameHeader); 24 | mainTable.appendChild(dateHeader); 25 | mainTable.appendChild(linkHeader); 26 | mainTable.appendChild(document.createElement("tr")); 27 | 28 | data.forEach((item, index) => { 29 | let numberField = document.createElement("td"); 30 | numberField.innerText = index+1; 31 | numberField.style.padding = "10px"; 32 | numberField.classList.add("border-right"); 33 | 34 | let nameField = document.createElement("td"); 35 | nameField.innerText = item['name']; 36 | nameField.style.textAlign = 'center'; 37 | nameField.style.padding = "8px"; 38 | nameField.classList.add("border-right"); 39 | 40 | let dateField = document.createElement("td"); 41 | dateField.innerText = item['created_at'] || "-"; 42 | dateField.style.padding = "8px"; 43 | dateField.classList.add("border-right"); 44 | 45 | let linkField = document.createElement("td"); 46 | let linkButton = document.createElement("button"); 47 | linkButton.onclick = function () { 48 | if (item['status'] !== -1) { 49 | getObjectURL(item['name']); 50 | } else { 51 | register(bucket, item['name']); 52 | } 53 | }; 54 | if (item['status'] !== -1) { 55 | linkButton.classList.add("btn", "url-btn"); 56 | } else { 57 | linkButton.classList.add("btn", "register-btn"); 58 | } 59 | linkButton.innerHTML = downloadIcon; 60 | 61 | let linkButtonText = document.createElement("span"); 62 | if (item['status'] !== -1) { 63 | linkButtonText.innerText = "Copy URL"; 64 | } else { 65 | linkButtonText.innerText = "Register Object"; 66 | } 67 | linkButtonText.style.marginLeft = "10px"; 68 | 69 | linkButton.appendChild(linkButtonText); 70 | linkField.appendChild(linkButton); 71 | 72 | mainTable.appendChild(numberField); 73 | mainTable.appendChild(nameField); 74 | mainTable.appendChild(dateField); 75 | mainTable.appendChild(linkField); 76 | mainTable.appendChild(document.createElement("tr")); 77 | }); 78 | 79 | responseDiv.innerHTML = ""; 80 | responseDiv.appendChild(mainTable); 81 | } 82 | 83 | // get object of a bucket 84 | function getObjects() { 85 | let bucket = document.getElementById("bucket").value; 86 | let prefix = document.getElementById("prefix").value; 87 | 88 | let responseDiv = document.getElementById("response-div"); 89 | 90 | fetch(`/api/objects?bucket=${bucket}&prefix=${prefix}`) 91 | .then((response) => response.json()) 92 | .then((data) => { 93 | generateTable(bucket, data); 94 | }) 95 | .catch((e) => { 96 | console.error(e); 97 | responseDiv.innerText = "Error in reading objects!"; 98 | 99 | alert("Failed to read objects!"); 100 | }); 101 | } 102 | 103 | // update object url. 104 | function updateObject(bucket, key, status) { 105 | fetch(`/api/objects/${bucket}/${key}?status=${status}`, { 106 | method: 'POST' 107 | }) 108 | .then(() => { 109 | console.log("update"); 110 | 111 | alert("Update successfully!"); 112 | }) 113 | .catch((e) => { 114 | console.error(e); 115 | 116 | alert("Failed to update!"); 117 | }) 118 | } 119 | 120 | // get url of an object 121 | function getObjectURL(key) { 122 | let bucket = document.getElementById("bucket").value; 123 | 124 | fetch(`/api/objects/${bucket}/${key}`) 125 | .then((response) => response.json()) 126 | .then((data) => { 127 | let text = data['address']; 128 | 129 | navigator.clipboard.writeText(text).then(function() { 130 | alert('URL copied to clipboard.') 131 | }, function(err) { 132 | console.error(`could not copy text error=${err}`); 133 | }); 134 | }) 135 | .catch((e) => { 136 | console.error(e); 137 | 138 | alert("Failed to get object link!"); 139 | }); 140 | } 141 | 142 | // register an object 143 | function register(bucket, key) { 144 | fetch(`/api/objects/${bucket}/${key}/register`) 145 | .then(() => { 146 | getObjects(); 147 | }) 148 | .catch((e) => { 149 | console.log(e); 150 | 151 | alert("Failed to register!"); 152 | }) 153 | } 154 | 155 | // get active link 156 | function active() { 157 | let path = window.location.pathname; 158 | 159 | switch (path) { 160 | case '/': 161 | document.getElementById("home-link").classList.add("active-item"); 162 | break; 163 | case '/search': 164 | document.getElementById("search-link").classList.add("active-item"); 165 | break 166 | case '/docs': 167 | document.getElementById("docs-link").classList.add("active-item"); 168 | break; 169 | } 170 | } 171 | 172 | active(); 173 | -------------------------------------------------------------------------------- /web/template/base.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | {% block title %} 12 | {{ title }} 13 | {% endblock title %} 14 | 15 | 16 | 17 |
18 |
19 | 20 | 21 | 22 | 23 | 24 | Home 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | Buckets 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | API 41 | 42 | 43 |
44 |
45 | 46 | {% block content %} 47 | {% endblock content %} 48 |
49 |
50 | 51 | 52 | 53 | 54 | 55 | -------------------------------------------------------------------------------- /web/template/bucket.j2: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block content %} 4 | 5 |
6 |
7 |
8 | 9 | 10 |
11 |
12 | 13 | 14 |
15 | 23 |
24 |
25 |
26 | 27 | {% endblock content %} 28 | -------------------------------------------------------------------------------- /web/template/download.j2: -------------------------------------------------------------------------------- 1 |

2 | downloading file ... 3 |

4 | 5 | 21 | -------------------------------------------------------------------------------- /web/template/help.j2: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block content %} 4 | 5 |

6 | Use Minio URL Operator APIs 7 |

8 | 9 |
10 | MUO provides APIs to work with from your application. You can use them 11 | in any application that you want. With this APIs you can control 12 | your Minio shared objects links and generate human readable and short 13 | url addresses for your objects rather than using Minio long urls. 14 | In order to use MUO APIs you need to make HTTP requests as follow:

15 |
16 |
17 | List of objects: (Bucket and Prefix should be placed in query param)

18 | 
19 |         curl -X GET "localhost:8080/api/objects?bucket=snapp&prefix=list"
20 |
21 | Response:

22 | 
23 |         [
24 |             {
25 |                 'name': 'image.jpg',
26 |                 'status': 0,
27 |                 'created_at': '2023-06-01 19:00:02'
28 |             },
29 |             {
30 |                 'name': 'file.pdf',
31 |                 'status': -1,
32 |             }
33 |         ]
34 |
35 |
36 |
37 |
38 | Register an object in order to get URL: (Bucket and Key should be placed in URL Param)

39 | 
40 |         curl -X GET "localhost:8080/api/objects/:bucket/:key/register"
41 |
42 | Response:

43 | 
44 |         "OK"
45 |
46 |
47 |
48 |
49 | Get URL of an object: (Bucket and Key should be placed in URL Param)

50 | 
51 |         curl -X GET "localhost:8080/api/objects/:bucket/:key"
52 |
53 | Response:

54 | 
55 |         {
56 |             'address': 'http://localhost/api/objects/m889wierop'
57 |         }
58 |
59 |
60 |
61 |
62 | Update URL access: (Status should be placed in query param)
63 | Status can be 0 or 1. 64 |

65 | 
66 |         curl -X POST "localhost:8080/api/objects/:bucket/:key?status=0"
67 |
68 | Response:

69 | 
70 |         "OK"
71 |
72 |
73 | 74 | {% endblock content %} 75 | -------------------------------------------------------------------------------- /web/template/index.j2: -------------------------------------------------------------------------------- 1 | {% extends "base.html" %} 2 | 3 | {% block content %} 4 | 5 |

6 | Welcome to Minio URL Operator 7 |

8 | 9 |
10 | By using MUO you can get user readable addresses for your objects 11 | in Minio cluster. These addresses are public and you can share them 12 | with anyone the your like. Since Minio provides shared urls with a 13 | maximum 7 days limit, you can use this operator to get fresh links 14 | everytime you want. You can block access to Minio objects just 15 | by clicking a button. In order to use the application APIs, visit the 16 | documents page. 17 |
18 | 19 | {% endblock content %} 20 | --------------------------------------------------------------------------------