├── coworks ├── py.typed ├── cws │ ├── __init__.py │ ├── project_templates │ │ ├── template.env │ │ ├── README.md │ │ └── tech │ │ │ └── app.py │ ├── exception.py │ ├── templates │ │ ├── project.cws.yml │ │ ├── deploy.j2 │ │ ├── terraform │ │ │ ├── output.j2 │ │ │ ├── local.j2 │ │ │ ├── provider.j2 │ │ │ ├── response_templates.j2 │ │ │ ├── role.j2 │ │ │ ├── lambda.j2 │ │ │ └── request_templates.j2 │ │ └── terraform.j2 │ ├── command.py │ ├── new.py │ └── utils.py ├── globals.py ├── const.py ├── sensors.py ├── operators.py ├── version.py ├── blueprint │ ├── profiler_blueprint.py │ ├── test_blueprint.py │ ├── okta_blueprint.py │ └── mail_blueprint.py ├── biz │ ├── __init__.py │ ├── sensors.py │ └── group.py ├── extension │ └── jsonapi │ │ ├── __init__.py │ │ ├── pydantic.py │ │ └── query.py ├── __init__.py ├── utils.py └── aws.py ├── tests ├── __init__.py ├── cws │ ├── __init__.py │ ├── src │ │ ├── __init__.py │ │ ├── .env.dev │ │ ├── project.cws.yml │ │ ├── project.wrong.yml │ │ ├── app.py │ │ └── command.py │ ├── test_project_file.py │ ├── test_cmd.py │ └── test_environment.py ├── docs │ ├── __init__.py │ ├── test_client.py │ ├── test_hello.py │ ├── test_complete.py │ └── test_first.py ├── coworks │ ├── __init__.py │ ├── tech │ │ ├── __init__.py │ │ ├── test_async.py │ │ ├── test_auth.py │ │ ├── test_content_type.py │ │ └── test_type.py │ ├── blueprint │ │ ├── __init__.py │ │ ├── blueprint.py │ │ ├── test_overload.py │ │ ├── test_empty.py │ │ ├── test_blueprint.py │ │ ├── test_admin.py │ │ └── test_mail.py │ ├── ms.py │ └── event.py └── conftest.py ├── .actrc ├── bin └── act ├── NOTICE ├── docs ├── img │ ├── coworks.png │ ├── coworks_small.png │ └── resource_oriented.png ├── _static │ └── coworks.css ├── _templates │ └── layout.html ├── requirements.txt ├── api.rst ├── contributing.rst ├── Makefile ├── samples.rst ├── faq.rst ├── changelog.rst ├── conf.py ├── installation.rst ├── biz_quickstart.rst ├── code_of_conduct.md ├── command.rst ├── biz.rst ├── configuration.rst ├── tech_quickstart.rst └── index.rst ├── samples ├── website │ ├── input.css │ ├── tech │ │ ├── assets │ │ │ ├── plugins.zip │ │ │ └── img │ │ │ │ ├── coworks.ico │ │ │ │ └── coworks.png │ │ ├── util.py │ │ ├── app.py │ │ ├── templates │ │ │ ├── news.j2 │ │ │ ├── home.j2 │ │ │ ├── account │ │ │ │ └── login.j2 │ │ │ └── base.j2 │ │ ├── account.py │ │ └── website.py │ ├── tailwind.config.js │ ├── project.cws.yml │ ├── package.json │ └── README.md ├── docs │ ├── README.md │ ├── tech │ │ ├── hello.py │ │ ├── first.py │ │ └── complete.py │ └── project.cws.yml ├── directory │ ├── tech │ │ └── app.py │ ├── project.cws.yml │ ├── tests.http │ └── README.md ├── layers │ ├── tests.http │ ├── tech │ │ ├── templates │ │ │ ├── layers.j2 │ │ │ └── home.j2 │ │ └── app.py │ └── project.cws.yml └── Pipfile ├── MANIFEST.in ├── .github ├── pull_request_template.md ├── feature_request.md ├── ISSUE_TEMPLATE │ ├── feature_request.md │ └── bug_report.md ├── workflows │ └── pypi-publish.yml └── bug_report.md ├── .readthedocs.yml ├── .pypirc ├── .coveragerc ├── .gitignore ├── LICENSE.txt ├── pyproject.toml └── README.rst /coworks/py.typed: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /coworks/cws/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/cws/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/docs/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/coworks/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/cws/src/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/coworks/tech/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /tests/coworks/blueprint/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.actrc: -------------------------------------------------------------------------------- 1 | -P ubuntu-latest=catthehacker/ubuntu:full-latest 2 | -------------------------------------------------------------------------------- /bin/act: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gdoumenc/coworks/HEAD/bin/act -------------------------------------------------------------------------------- /tests/cws/src/.env.dev: -------------------------------------------------------------------------------- 1 | TEST="test dev environment variable" 2 | STAGE=dev 3 | -------------------------------------------------------------------------------- /NOTICE: -------------------------------------------------------------------------------- 1 | Coworks 2 | Copyright 2019/2024 Guillaume Doumenc/FPR. All Rights Reserved. -------------------------------------------------------------------------------- /coworks/cws/project_templates/template.env: -------------------------------------------------------------------------------- 1 | TOKEN=token 2 | USER_KEYS=mine,yours 3 | 4 | -------------------------------------------------------------------------------- /docs/img/coworks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gdoumenc/coworks/HEAD/docs/img/coworks.png -------------------------------------------------------------------------------- /samples/website/input.css: -------------------------------------------------------------------------------- 1 | @tailwind base; 2 | @tailwind components; 3 | @tailwind utilities; 4 | -------------------------------------------------------------------------------- /docs/img/coworks_small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gdoumenc/coworks/HEAD/docs/img/coworks_small.png -------------------------------------------------------------------------------- /docs/img/resource_oriented.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gdoumenc/coworks/HEAD/docs/img/resource_oriented.png -------------------------------------------------------------------------------- /samples/docs/README.md: -------------------------------------------------------------------------------- 1 | ### Description 2 | 3 | This project contains all samples defined in the documentation. 4 | 5 | -------------------------------------------------------------------------------- /samples/website/tech/assets/plugins.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gdoumenc/coworks/HEAD/samples/website/tech/assets/plugins.zip -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | graft docs/img 2 | graft coworks/cws/project_templates 3 | graft coworks/cws/templates 4 | prune tests 5 | prune samples 6 | -------------------------------------------------------------------------------- /samples/website/tech/assets/img/coworks.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gdoumenc/coworks/HEAD/samples/website/tech/assets/img/coworks.ico -------------------------------------------------------------------------------- /samples/website/tech/assets/img/coworks.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/gdoumenc/coworks/HEAD/samples/website/tech/assets/img/coworks.png -------------------------------------------------------------------------------- /docs/_static/coworks.css: -------------------------------------------------------------------------------- 1 | 2 | #coworks { 3 | overflow: auto; 4 | } 5 | 6 | #coworks::after { 7 | content: ""; 8 | clear: both; 9 | display: table; 10 | } 11 | -------------------------------------------------------------------------------- /coworks/cws/exception.py: -------------------------------------------------------------------------------- 1 | import click 2 | 3 | 4 | class ExitCommand(click.exceptions.Exit): 5 | def __init__(self, msg): 6 | super().__init__() 7 | self.msg = msg 8 | -------------------------------------------------------------------------------- /docs/_templates/layout.html: -------------------------------------------------------------------------------- 1 | {# layout.html #} 2 | {# Import the layout of the theme. #} 3 | {% extends "!layout.html" %} 4 | 5 | {% set css_files = css_files + ['_static/coworks.css'] %} -------------------------------------------------------------------------------- /coworks/globals.py: -------------------------------------------------------------------------------- 1 | import typing as t 2 | 3 | from flask import request as flask_request 4 | 5 | from .wrappers import CoworksRequest 6 | 7 | request: CoworksRequest = t.cast(CoworksRequest, flask_request) 8 | -------------------------------------------------------------------------------- /.github/pull_request_template.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | Issue #, if available: 4 | 5 | Description of changes: 6 | 7 | By submitting this pull request, I confirm that my contribution is made under the terms of the MIT license. 8 | -------------------------------------------------------------------------------- /samples/website/tech/util.py: -------------------------------------------------------------------------------- 1 | from flask import render_template 2 | 3 | 4 | def render_html_template(*args, **kwargs): 5 | return render_template(*args, **kwargs), 200, {'content-type': 'text/html; charset=utf-8'} 6 | -------------------------------------------------------------------------------- /coworks/cws/templates/project.cws.yml: -------------------------------------------------------------------------------- 1 | version: 3 2 | commands: 3 | deploy: 4 | class: coworks.cws.deploy.deploy_command 5 | layers: 6 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-{{ version }} 7 | 8 | 9 | -------------------------------------------------------------------------------- /docs/requirements.txt: -------------------------------------------------------------------------------- 1 | docutils >=0.16 2 | sphinx >=4.4.0 3 | flask >=2.2.2 4 | boto3 >=1.26.20 5 | apache-airflow >=2.2.4 6 | apache-airflow-providers-amazon >=3.0.0 7 | python-dotenv >=0.21.0 8 | coworks >= 0.9.0 9 | 10 | -------------------------------------------------------------------------------- /samples/website/tech/app.py: -------------------------------------------------------------------------------- 1 | from aws_xray_sdk.core import xray_recorder 2 | 3 | from coworks.extension.xray import XRay 4 | from website import WebsiteMicroService 5 | 6 | app = WebsiteMicroService() 7 | XRay(app, xray_recorder) 8 | -------------------------------------------------------------------------------- /samples/directory/tech/app.py: -------------------------------------------------------------------------------- 1 | from coworks.blueprint.admin_blueprint import Admin 2 | from coworks.tech.directory import DirectoryMicroService 3 | 4 | app = DirectoryMicroService() 5 | app.register_blueprint(Admin(), url_prefix='/admin') 6 | -------------------------------------------------------------------------------- /coworks/const.py: -------------------------------------------------------------------------------- 1 | DEFAULT_DEV_STAGE = "dev" 2 | DEFAULT_LOCAL_STAGE = "local" 3 | DEFAULT_PROJECT_DIR = "tech" 4 | 5 | PROJECT_CONFIG_VERSION = 3 6 | 7 | BIZ_BUCKET_HEADER_KEY: str = 'X-CWS-S3Bucket' 8 | BIZ_KEY_HEADER_KEY: str = 'X-CWS-S3Key' 9 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | build: 4 | os: ubuntu-20.04 5 | tools: 6 | python: "3.11" 7 | 8 | python: 9 | install: 10 | - method: pip 11 | path: . 12 | 13 | 14 | sphinx: 15 | configuration: docs/conf.py 16 | -------------------------------------------------------------------------------- /samples/layers/tests.http: -------------------------------------------------------------------------------- 1 | GET https://2kb9hn4bs4.execute-api.eu-west-1.amazonaws.com/v1/admin 2 | Accept: text/html 3 | 4 | ### 5 | 6 | GET https://2kb9hn4bs4.execute-api.eu-west-1.amazonaws.com/v1 7 | Accept: application/json 8 | 9 | ### 10 | -------------------------------------------------------------------------------- /tests/cws/src/project.cws.yml: -------------------------------------------------------------------------------- 1 | version: 3 2 | commands: 3 | test: 4 | class: command.cmd 5 | b: value 6 | a: default 7 | workspaces: 8 | v1: 9 | commands: 10 | test: 11 | class: command.cmd1 12 | b: value1 13 | -------------------------------------------------------------------------------- /.pypirc: -------------------------------------------------------------------------------- 1 | [distutils] 2 | index-servers = 3 | pypi 4 | testpypi 5 | 6 | [pypi] 7 | repository = https://upload.pypi.org/legacy/ 8 | username = gdoumenc 9 | 10 | [testpypi] 11 | repository = https://test.pypi.org/legacy/ 12 | username = gdoumenc 13 | -------------------------------------------------------------------------------- /coworks/sensors.py: -------------------------------------------------------------------------------- 1 | import warnings 2 | 3 | from coworks.biz.sensors import * # legacy 4 | 5 | warnings.warn( 6 | "This module is deprecated. Please use `coworks.biz` or `coworks.biz.sensors` instead.", 7 | DeprecationWarning, 8 | stacklevel=2, 9 | ) 10 | -------------------------------------------------------------------------------- /coworks/operators.py: -------------------------------------------------------------------------------- 1 | import warnings 2 | 3 | from coworks.biz.operators import * # legacy 4 | 5 | warnings.warn( 6 | "This module is deprecated. Please use `coworks.biz` or `coworks.biz.operators` instead.", 7 | DeprecationWarning, 8 | stacklevel=2, 9 | ) 10 | -------------------------------------------------------------------------------- /samples/docs/tech/hello.py: -------------------------------------------------------------------------------- 1 | from coworks import TechMicroService 2 | from coworks import entry 3 | 4 | 5 | class HelloMicroService(TechMicroService): 6 | 7 | @entry(no_auth=True) 8 | def get(self): 9 | return "Hello world.\n" 10 | 11 | 12 | app = HelloMicroService() 13 | -------------------------------------------------------------------------------- /.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | branch = True 3 | source = coworks 4 | 5 | [report] 6 | exclude_lines = 7 | if self.debug: 8 | pragma: no cover 9 | raise NotImplementedError 10 | if __name__ == .__main__.: 11 | ignore_errors = True 12 | omit = 13 | docs/* 14 | tests/* 15 | coworks/cws/templates/* 16 | coworks/tech/* 17 | -------------------------------------------------------------------------------- /samples/Pipfile: -------------------------------------------------------------------------------- 1 | [[source]] 2 | name = "pypi" 3 | url = "https://pypi.org/simple" 4 | verify_ssl = true 5 | 6 | [packages] 7 | coworks = {editable = true, path = "/home/gdo/workspace/coworks"} 8 | aws-xray-sdk = "*" 9 | email-validator = "*" 10 | flask-login = "*" 11 | flask-wtf = "*" 12 | requests = "*" 13 | 14 | [dev-packages] 15 | -------------------------------------------------------------------------------- /samples/layers/tech/templates/layers.j2: -------------------------------------------------------------------------------- 1 |
2 | The available CoWorks layers are: 3 |
4 | 9 |
10 | Last updated on {{ now }} - Powered by CoWorks. 11 |
12 | -------------------------------------------------------------------------------- /coworks/version.py: -------------------------------------------------------------------------------- 1 | import tomllib 2 | from pathlib import Path 3 | 4 | try: 5 | pyproject = Path(__file__).resolve().parent.parent / "pyproject.toml" 6 | with open(pyproject, "rb") as f: 7 | _META = tomllib.load(f) 8 | 9 | __version__ = _META["project"]["version"] 10 | except FileNotFoundError: 11 | __version__ = 'deployed' 12 | -------------------------------------------------------------------------------- /docs/api.rst: -------------------------------------------------------------------------------- 1 | .. _api: 2 | 3 | Developer Interface 4 | =================== 5 | 6 | This part of the documentation presents all the classes of Coworks. 7 | 8 | 9 | Microservices 10 | ------------- 11 | 12 | .. module:: coworks 13 | 14 | .. autoclass:: TechMicroService 15 | :members: 16 | 17 | .. autoclass:: Blueprint 18 | :members: 19 | 20 | 21 | 22 | 23 | -------------------------------------------------------------------------------- /coworks/cws/templates/deploy.j2: -------------------------------------------------------------------------------- 1 | {% include 'terraform/provider.j2' with context %} 2 | 3 | {% include 'terraform/local.j2' with context %} 4 | 5 | {% include 'terraform/role.j2' with context %} 6 | 7 | {% include 'terraform/rest_api.j2' with context %} 8 | 9 | {% include 'terraform/lambda.j2' with context %} 10 | 11 | {% include 'terraform/output.j2' with context %} 12 | -------------------------------------------------------------------------------- /samples/docs/project.cws.yml: -------------------------------------------------------------------------------- 1 | version: 3 2 | commands: 3 | deploy: 4 | bucket: coworks-microservice 5 | profile_name: fpr-customer 6 | layers: 7 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-0_8_0 8 | workspaces: 9 | dev: 10 | commands: 11 | deploy: 12 | layers: 13 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-dev1 14 | -------------------------------------------------------------------------------- /samples/layers/project.cws.yml: -------------------------------------------------------------------------------- 1 | version: 3 2 | commands: 3 | deploy: 4 | bucket: coworks-microservice 5 | profile_name: fpr-customer 6 | layers: 7 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-0_8_0 8 | workspaces: 9 | dev: 10 | commands: 11 | deploy: 12 | layers: 13 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-dev1 14 | -------------------------------------------------------------------------------- /samples/directory/project.cws.yml: -------------------------------------------------------------------------------- 1 | version: 3 2 | commands: 3 | deploy: 4 | bucket: coworks-microservice 5 | profile_name: fpr-customer 6 | layers: 7 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-0_8_0 8 | workspaces: 9 | dev: 10 | commands: 11 | deploy: 12 | layers: 13 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-dev1 14 | -------------------------------------------------------------------------------- /tests/cws/src/project.wrong.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | commands: 3 | test: 4 | class: cmd.cmd 5 | b: value 6 | a: default 7 | services: 8 | - module: example 9 | service: project2 10 | a: project2 11 | workspaces: 12 | - workspace: prod 13 | a: prod2 14 | - module: example 15 | service: autre 16 | a: autre1 17 | -------------------------------------------------------------------------------- /coworks/cws/templates/terraform/output.j2: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------------------------------------------------- 2 | # OUTPUT 3 | # --------------------------------------------------------------------------------------------------------------------- 4 | 5 | {% if workspace == "common" %} 6 | output "{{ ms_name }}_id" { 7 | value = aws_api_gateway_rest_api.{{ ms_name }}.id 8 | } 9 | {% endif %} 10 | -------------------------------------------------------------------------------- /docs/contributing.rst: -------------------------------------------------------------------------------- 1 | Contributing 2 | ------------ 3 | 4 | We work hard to provide a high-quality and useful framework, and we greatly value 5 | feedback and contributions from our community. Whether it's a new feature, 6 | correction, or additional documentation, we welcome your pull requests. Please 7 | submit any `issues `__ 8 | or `pull requests `__ through GitHub. 9 | 10 | -------------------------------------------------------------------------------- /samples/directory/tests.http: -------------------------------------------------------------------------------- 1 | GET https://mjrafq1qlg.execute-api.eu-west-1.amazonaws.com/dev/admin 2 | Accept: text/html 3 | 4 | ### 5 | 6 | GET https://mjrafq1qlg.execute-api.eu-west-1.amazonaws.com/dev/name/mjrafq1qlg 7 | Accept: application/json 8 | Authorization: {{ token }} 9 | 10 | ### 11 | 12 | GET https://mjrafq1qlg.execute-api.eu-west-1.amazonaws.com/dev/url/directory 13 | Accept: application/json 14 | Authorization: {{ token }} 15 | 16 | ### 17 | -------------------------------------------------------------------------------- /coworks/cws/templates/terraform/local.j2: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------------------------------------------------- 2 | # LOCALS FOR THIS MICROSERVICE 3 | # --------------------------------------------------------------------------------------------------------------------- 4 | 5 | locals { 6 | {{ ms_name }}_security_group_ids = {{ security_groups | list | replace("'", '"') }} 7 | {{ ms_name }}_subnet_ids = {{ subnets | list | replace("'", '"')}} 8 | } 9 | -------------------------------------------------------------------------------- /samples/website/tailwind.config.js: -------------------------------------------------------------------------------- 1 | /** @type {import('tailwindcss').Config} */ 2 | module.exports = { 3 | content: ["./tech/**/*.{j2,js}"], 4 | theme: { 5 | extend: { 6 | fontFamily: {sans: ['Inter var']}, 7 | }, 8 | }, 9 | plugins: [ 10 | require('tailwindcss'), 11 | require('autoprefixer'), 12 | require('@tailwindcss/forms'), 13 | require('@tailwindcss/line-clamp'), 14 | require('@tailwindcss/aspect-ratio'), 15 | ], 16 | } 17 | -------------------------------------------------------------------------------- /coworks/cws/templates/terraform/provider.j2: -------------------------------------------------------------------------------- 1 | provider "aws" { 2 | alias = "{{ ms_name }}" 3 | {% if terraform_cloud %} 4 | access_key = var.AWS_ACCESS_KEY_ID 5 | secret_key = var.AWS_SECRET_ACCESS_KEY 6 | {% else %} 7 | profile = "{{ profile_name }}" 8 | {% endif %} 9 | region = {{ '"{}"'.format(aws_region) if aws_region else 'var.AWS_REGION'}} 10 | 11 | default_tags { 12 | tags = { 13 | Creator = "terraform" 14 | MicroService = "{{ ms_name }}" 15 | Stage = "{{ stage }}" 16 | } 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Project 2 | coworks.egg-info/ 3 | build/ 4 | dist/ 5 | **/.terraform 6 | **/*.tfstate* 7 | .venv 8 | .env* 9 | **/http-client.private.env.json 10 | 11 | # pdm 12 | pdm.lock 13 | .pdm-python 14 | .pdm-build 15 | 16 | # Samples 17 | **/*.secret.json 18 | samples/*/terraform 19 | samples/Pipfile.lock 20 | samples/*/node_modules/ 21 | 22 | # Tests example 23 | tests/example/*.zip 24 | 25 | # Doc tools 26 | coworks/cattr.py 27 | 28 | # Dev tools 29 | .idea/ 30 | **/__pycache__/ 31 | docs/_build/ 32 | .pytest_cache/ 33 | **/*.log 34 | bin/act 35 | /.actrc 36 | -------------------------------------------------------------------------------- /coworks/blueprint/profiler_blueprint.py: -------------------------------------------------------------------------------- 1 | import io 2 | 3 | from werkzeug.middleware.profiler import ProfilerMiddleware 4 | 5 | from coworks import Blueprint 6 | from coworks import entry 7 | 8 | 9 | class Profiler(Blueprint): 10 | 11 | def __init__(self, app, output=None, **kwargs): 12 | super().__init__(**kwargs) 13 | self.output = output or io.StringIO() 14 | app.wsgi_app = ProfilerMiddleware(app.wsgi_app, stream=self.output) 15 | 16 | @entry 17 | def get(self): 18 | profile = self.output.getvalue() 19 | self.output.seek(0) 20 | return profile 21 | -------------------------------------------------------------------------------- /coworks/biz/__init__.py: -------------------------------------------------------------------------------- 1 | from coworks.biz.group import TechMicroServiceAsyncGroup 2 | from coworks.biz.operators import AsyncTechServicePullOperator 3 | from coworks.biz.operators import BranchTechMicroServiceOperator 4 | from coworks.biz.operators import TechMicroServiceOperator 5 | from coworks.biz.sensors import AsyncTechMicroServiceSensor 6 | from coworks.biz.sensors import TechMicroServiceSensor 7 | 8 | _all__ = ( 9 | AsyncTechServicePullOperator, BranchTechMicroServiceOperator, TechMicroServiceOperator, 10 | AsyncTechMicroServiceSensor, TechMicroServiceSensor, 11 | TechMicroServiceAsyncGroup, 12 | ) 13 | -------------------------------------------------------------------------------- /samples/website/project.cws.yml: -------------------------------------------------------------------------------- 1 | version: 3 2 | commands: 3 | deploy: 4 | bucket: coworks-microservice 5 | profile_name: fpr-customer 6 | text_types: 7 | - text/css 8 | binary_types: 9 | - application/zip 10 | layers: 11 | - arn:aws:lambda:eu-west-1:935392763270:layer:website-0_3 12 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-0_8_3 13 | workspaces: 14 | dev: 15 | commands: 16 | deploy: 17 | layers: 18 | - arn:aws:lambda:eu-west-1:935392763270:layer:website-0_3 19 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-dev1 20 | -------------------------------------------------------------------------------- /samples/website/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "coworks_sample", 3 | "version": "0.1", 4 | "description": "Simple serverless website.", 5 | "main": "index.js", 6 | "license": "MIT", 7 | "scripts": { 8 | "tailwind": "npx tailwindcss build -i ./input.css -o ./tech/assets/css/output.css" 9 | }, 10 | "devDependencies": { 11 | "@tailwindcss/aspect-ratio": "^0.4.2", 12 | "@tailwindcss/forms": "^0.5.3", 13 | "@tailwindcss/line-clamp": "^0.4.2", 14 | "@tailwindcss/typography": "^0.5.7", 15 | "autoprefixer": "^10.4.12", 16 | "postcss": "^8.4.16", 17 | "tailwindcss": "^3.1.8" 18 | } 19 | } 20 | -------------------------------------------------------------------------------- /samples/docs/tech/first.py: -------------------------------------------------------------------------------- 1 | from coworks import TechMicroService 2 | from coworks import entry 3 | 4 | 5 | class SimpleMicroService(TechMicroService): 6 | 7 | def __init__(self, **kwargs): 8 | super().__init__(**kwargs) 9 | self.value = 0 10 | 11 | def token_authorizer(self, token): 12 | return token == "token" 13 | 14 | @entry 15 | def get(self): 16 | return f"Stored value {self.value}.\n" 17 | 18 | @entry 19 | def post(self, value=None): 20 | if value is not None: 21 | self.value = value 22 | return f"Value stored ({value}).\n" 23 | 24 | 25 | app = SimpleMicroService(name="sample-first-microservice") 26 | -------------------------------------------------------------------------------- /.github/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line, and also 5 | # from the environment for the first two. 6 | SPHINXOPTS ?= 7 | SPHINXBUILD ?= sphinx-build 8 | SOURCEDIR = . 9 | BUILDDIR = _build 10 | 11 | # Put it first so that "make" without argument is like "make help". 12 | help: 13 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 14 | 15 | .PHONY: help Makefile 16 | 17 | # Catch-all target: route all unknown targets to Sphinx using the new 18 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 19 | %: Makefile 20 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 21 | -------------------------------------------------------------------------------- /.github/workflows/pypi-publish.yml: -------------------------------------------------------------------------------- 1 | name: Upload CoWorks Python Package 2 | 3 | on: push 4 | 5 | jobs: 6 | pypi-publish: 7 | name: upload release to PyPI 8 | runs-on: ubuntu-latest 9 | environment: release 10 | permissions: 11 | id-token: write 12 | 13 | steps: 14 | - uses: actions/checkout@v4 15 | - uses: pdm-project/setup-pdm@v4 16 | 17 | - name: Install dependencies 18 | run: pdm install -G dev 19 | 20 | - name: Display Python version 21 | run: python -c "import sys; print(sys.version)" 22 | 23 | - name: Run tests 24 | run: pdm run on_github 25 | 26 | - name: Publish package distributions to PyPI 27 | if: github.ref_name == 'master' 28 | run: pdm publish 29 | -------------------------------------------------------------------------------- /coworks/cws/command.py: -------------------------------------------------------------------------------- 1 | from click import Command 2 | from click import UsageError 3 | 4 | 5 | def no_project_context(f): 6 | """Decorator to allow command without need to have a project dir defined.""" 7 | setattr(f, '__need_project_context', False) 8 | return f 9 | 10 | 11 | class CwsCommand(Command): 12 | 13 | def invoke(self, ctx): 14 | if getattr(self.callback, '__need_project_context', True) and not self._context_project_dir(ctx): 15 | raise UsageError(f"Project dir {self._context_project_dir(ctx)} not defined.") 16 | return super().invoke(ctx) 17 | 18 | @staticmethod 19 | def _context_project_dir(ctx): 20 | if ctx.parent is None: 21 | return ctx.params.get('project_dir') 22 | return CwsCommand._context_project_dir(ctx.parent) 23 | -------------------------------------------------------------------------------- /samples/layers/tech/templates/home.j2: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 19 | 20 | 21 | 22 |
23 | 25 |
26 |
27 | 28 | 29 | -------------------------------------------------------------------------------- /coworks/extension/jsonapi/__init__.py: -------------------------------------------------------------------------------- 1 | from .data import CursorPagination 2 | from .data import JsonApiDataMixin 3 | from .data import JsonApiDict 4 | from .fetching import FetchingContext 5 | from .fetching import fetching_context 6 | from .jsonapi import JsonApi 7 | from .jsonapi import JsonApiError 8 | from .jsonapi import jsonapi 9 | from .jsonapi import toplevel_from_data 10 | from .jsonapi import toplevel_from_pagination 11 | from .query import ListQuery 12 | from .query import Pagination 13 | from .query import Query 14 | 15 | __all__ = [ 16 | 'CursorPagination', 17 | 'JsonApiDataMixin', 18 | 'JsonApiDict', 19 | 'FetchingContext', 20 | 'fetching_context', 21 | 'JsonApi', 22 | 'JsonApiError', 23 | 'jsonapi', 24 | 'toplevel_from_data', 25 | 'toplevel_from_pagination', 26 | 'Pagination', 27 | 'Query', 28 | 'ListQuery' 29 | ] 30 | -------------------------------------------------------------------------------- /tests/coworks/blueprint/blueprint.py: -------------------------------------------------------------------------------- 1 | from unittest.mock import Mock 2 | 3 | from coworks import Blueprint 4 | from coworks import entry 5 | 6 | 7 | class BP(Blueprint): 8 | 9 | @entry 10 | def get_test(self, index): 11 | return f"blueprint BP {index}" 12 | 13 | @entry 14 | def get_extended_test(self, index): 15 | return f"blueprint extended test {index}" 16 | 17 | 18 | class InitBP(BP): 19 | 20 | def __init__(self, **kwargs): 21 | super().__init__(**kwargs) 22 | self.do_before_activation = Mock() 23 | self.do_after_activation = Mock() 24 | 25 | @self.before_app_request 26 | def before_activation(): 27 | self.do_before_activation() 28 | 29 | @self.after_app_request 30 | def after_activation(response): 31 | self.do_after_activation(response) 32 | return response 33 | -------------------------------------------------------------------------------- /coworks/__init__.py: -------------------------------------------------------------------------------- 1 | import typing as t 2 | 3 | from .coworks import Blueprint 4 | from .coworks import TechMicroService 5 | from .coworks import entry 6 | from .globals import request 7 | from .version import __version__ 8 | 9 | type StrList = list[str] 10 | type StrSet = set[str] 11 | type StrDict[T] = dict[str, T] 12 | 13 | __all__ = ( 14 | 'TechMicroService', 'Blueprint', 'entry', 15 | 'StrList', 'StrSet', 'StrDict', 16 | 'request', '__version__', 17 | ) 18 | 19 | 20 | def __getattr__(name: str) -> t.Any: 21 | if name == "__version__": 22 | import importlib.metadata 23 | import warnings 24 | 25 | warnings.warn( 26 | "The '__version__' attribute is deprecated and will be removed.", 27 | DeprecationWarning, 28 | stacklevel=2, 29 | ) 30 | return importlib.metadata.version("coworks") 31 | 32 | raise AttributeError(name) 33 | -------------------------------------------------------------------------------- /tests/coworks/blueprint/test_overload.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from coworks import Blueprint 4 | from coworks import TechMicroService 5 | from coworks import entry 6 | 7 | 8 | class BP(Blueprint): 9 | 10 | @entry 11 | def get_entry(self): 12 | return f"blueprint test entry" 13 | 14 | 15 | class MS(TechMicroService): 16 | 17 | @entry 18 | def get_bp_entry(self): 19 | return f"blueprint test entry" 20 | 21 | 22 | class TestClass: 23 | 24 | def test_overload(self): 25 | app = MS() 26 | app.register_blueprint(BP(), url_prefix='bp') 27 | with pytest.raises(AssertionError) as pytest_wrapped_e: 28 | with app.test_client() as c: 29 | response = c.get('/', headers={'Authorization': 'token'}) 30 | assert pytest_wrapped_e.type == AssertionError 31 | assert pytest_wrapped_e.value.args[0] == 'Duplicate route /bp/entry' 32 | -------------------------------------------------------------------------------- /samples/directory/README.md: -------------------------------------------------------------------------------- 1 | ### Instructions 2 | 3 | Define an user which must have the following policy: 4 | 5 | ```yaml 6 | { 7 | "Version": "2012-10-17", 8 | "Statement": [ 9 | { 10 | "Sid": "APIGatewayAccess", 11 | "Effect": "Allow", 12 | "Action": [ 13 | "execute-api:ManageConnections", 14 | "execute-api:Invoke", 15 | "execute-api:InvalidateCache", 16 | "apigateway:GET" 17 | ], 18 | "Resource": [ 19 | "arn:aws:apigateway:eu-west-1::/restapis/*", 20 | "arn:aws:apigateway:eu-west-1::/restapis" 21 | ] 22 | } 23 | ] 24 | } 25 | ``` 26 | 27 | Then add a file ``vars.secret.json`` in the ``env_vars`` folder: 28 | 29 | ```yaml 30 | { 31 | "AWS_USER_ACCESS_KEY_ID": AWS_USER_ACCESS_KEY_ID_VALUE, 32 | "AWS_USER_SECRET_ACCESS_KEY": AWS_USER_SECRET_ACCESS_KEY_VALUE 33 | } 34 | ``` 35 | -------------------------------------------------------------------------------- /tests/docs/test_client.py: -------------------------------------------------------------------------------- 1 | 2 | import os 3 | from unittest import mock 4 | from unittest.mock import Mock 5 | 6 | import click 7 | 8 | from coworks.cws.client import client 9 | 10 | 11 | class TestClass: 12 | 13 | def test_default_command(self): 14 | assert 'run' in client.commands 15 | assert 'shell' in client.commands 16 | assert 'routes' in client.commands 17 | 18 | @mock.patch.dict(os.environ, {"FLASK_APP": "complete:app"}) 19 | def test_routes_command(self, monkeypatch, samples_docs_dir): 20 | mclick = Mock() 21 | monkeypatch.setattr(click, "echo", mclick) 22 | client.main(['routes'], 'cws', standalone_mode=False) 23 | mclick.assert_called() 24 | assert len(mclick.mock_calls) == 8 25 | out = [call.args[0].split(' ')[0] for call in mclick.mock_calls] 26 | assert 'Endpoint' in out 27 | assert 'admin.get_route' in out 28 | assert 'get' in out 29 | assert 'post' in out 30 | -------------------------------------------------------------------------------- /.github/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /tests/cws/src/app.py: -------------------------------------------------------------------------------- 1 | import os 2 | from collections import defaultdict 3 | 4 | from coworks import TechMicroService 5 | from coworks import entry 6 | 7 | 8 | class EnvTechMS(TechMicroService): 9 | values = defaultdict(int) 10 | init_value = None 11 | 12 | def token_authorizer(self, token): 13 | return True 14 | 15 | @entry 16 | def get(self, usage="test"): 17 | """Entrypoint for testing named parameter.""" 18 | return f"Simple microservice for {usage}.\n" 19 | 20 | @entry 21 | def get_value(self, index): 22 | """Entrypoint for testing positional parameter.""" 23 | return f"{self.values[index]}\n" 24 | 25 | @entry 26 | def put_value(self, index, value=0): 27 | self.values[index] = value 28 | return value 29 | 30 | @entry 31 | def get_init(self): 32 | return f"Initial value is {self.init_value}.\n" 33 | 34 | @entry 35 | def get_env(self): 36 | return f"Value of environment variable test is : {os.getenv('TEST')}." 37 | -------------------------------------------------------------------------------- /coworks/blueprint/test_blueprint.py: -------------------------------------------------------------------------------- 1 | from coworks import Blueprint 2 | from coworks import entry 3 | 4 | 5 | class TestBlueprint(Blueprint): 6 | """Test blueprint. 7 | This blueprint proposes a template blueprint for CI/CD purpose. 8 | 9 | It has a single get entry for testing purpose. 10 | 11 | .. versionchanged:: 0.7.3 12 | Added. 13 | """ 14 | 15 | def __init__(self, name='test', **kwargs): 16 | super().__init__(name=name, **kwargs) 17 | 18 | @property 19 | def test_workspaces(self): 20 | return ['dev'] 21 | 22 | @entry(stage='dev') 23 | def post_reset(self): 24 | """Entry to reset the test environment.""" 25 | return 'ok' 26 | 27 | @entry(stage='dev') 28 | def get(self): 29 | """Test entry. 30 | 31 | Returns the next entry test case if needed. {'path':..., 'expected value':...} 32 | """ 33 | self.post_reset() 34 | self.test() 35 | self.post_reset() 36 | 37 | def test(self): 38 | return 'ok' 39 | -------------------------------------------------------------------------------- /tests/cws/src/command.py: -------------------------------------------------------------------------------- 1 | import click 2 | from flask.cli import pass_script_info 3 | 4 | from tests.cws.src.app import EnvTechMS 5 | 6 | 7 | @click.command("test", short_help="Test custom command.") 8 | @click.option('-a', required=True) 9 | @click.option('--b') 10 | @click.pass_context 11 | @pass_script_info 12 | def cmd(info, ctx, a, b): 13 | cws_app = info.load_app() 14 | assert cws_app is not None 15 | assert cws_app.config is not None 16 | if a: 17 | print(f"test command with a={a}/", end='') 18 | print(f"test command with b={b}", end='', flush=True) 19 | 20 | 21 | @click.command("test", short_help="Test custom command.") 22 | @click.option('-a', required=True) 23 | @click.option('--b') 24 | @click.pass_context 25 | @pass_script_info 26 | def cmd1(info, ctx, a, b): 27 | cws_app = info.load_app() 28 | assert cws_app is not None 29 | assert cws_app.config is not None 30 | if a: 31 | print(f"test command v1 with a={a}/", end='') 32 | print(f"test command v1 with b={b}", end='', flush=True) 33 | 34 | 35 | app = EnvTechMS() 36 | -------------------------------------------------------------------------------- /tests/coworks/blueprint/test_empty.py: -------------------------------------------------------------------------------- 1 | from coworks import Blueprint, entry 2 | from tests.coworks.ms import TechMS 3 | 4 | 5 | class EmptyBlueprint(Blueprint): 6 | 7 | @entry 8 | def get(self): 9 | return f"blueprint test" 10 | 11 | 12 | class TestClass: 13 | 14 | def test_as_root(self): 15 | app = TechMS() 16 | app.register_blueprint(EmptyBlueprint("bp")) 17 | with app.test_client() as c: 18 | response = c.get('/', headers={'Authorization': 'token'}) 19 | assert response.status_code == 200 20 | assert response.get_data(as_text=True) == 'blueprint test' 21 | assert app.routes == ['/'] 22 | 23 | def test_with_prefix(self): 24 | app = TechMS() 25 | app.register_blueprint(EmptyBlueprint("bp"), url_prefix='bp') 26 | with app.test_client() as c: 27 | response = c.get('/bp', headers={'Authorization': 'token'}) 28 | assert response.status_code == 200 29 | assert response.get_data(as_text=True) == 'blueprint test' 30 | assert app.routes == ['/bp'] 31 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 FPR-Coworks 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /tests/docs/test_hello.py: -------------------------------------------------------------------------------- 1 | import multiprocessing 2 | import os 3 | import time 4 | from unittest import mock 5 | 6 | import requests 7 | 8 | from coworks.cws.client import CwsScriptInfo 9 | 10 | 11 | class TestClass: 12 | 13 | @mock.patch.dict(os.environ, {"FLASK_RUN_FROM_CLI": "false"}) 14 | def test_run_simple(self, samples_docs_dir, unused_tcp_port, monkeypatch): 15 | info = CwsScriptInfo(project_dir='tech') 16 | info.app_import_path = "hello:app" 17 | app = info.load_app() 18 | server = multiprocessing.Process(target=run_server, args=(app, unused_tcp_port), daemon=True) 19 | server.start() 20 | counter = 1 21 | time.sleep(counter) 22 | while not server.is_alive() and counter < 3: 23 | time.sleep(counter) 24 | counter += 1 25 | response = requests.get(f'http://localhost:{unused_tcp_port}/', headers={'Authorization': "token"}) 26 | assert response.text == "Hello world.\n" 27 | server.terminate() 28 | 29 | 30 | def run_server(app, port): 31 | print(f"Server starting on port {port}") 32 | app.run(host='localhost', port=port, use_reloader=False, debug=False) 33 | -------------------------------------------------------------------------------- /samples/website/tech/templates/news.j2: -------------------------------------------------------------------------------- 1 | {% macro news(title) %} 2 |
  • 3 |
    4 |

    5 | {{ title }} 6 |

    7 |

    {{ caller() }}

    8 |
    9 |
    • 10 | {% endmacro %} 11 | 12 | 31 | -------------------------------------------------------------------------------- /docs/samples.rst: -------------------------------------------------------------------------------- 1 | .. _samples: 2 | 3 | Samples 4 | ======== 5 | 6 | Impatient developers often love samples to learn quickly. In this part, we will show you how to use CoWorks to : 7 | 8 | * Understand the CoWorks layer service. 9 | * Create a directory service to call technical microservice by there name. 10 | * Create a website with content defined in the CosmisJS headless tool. 11 | 12 | .. _layers: 13 | 14 | CoWorks layers 15 | -------------- 16 | 17 | Very simple microservice defining a simple HTML page and a call thru javascript. 18 | 19 | .. literalinclude:: ../samples/layers/tech/app.py 20 | 21 | .. _headless: 22 | 23 | Website 24 | ------- 25 | 26 | Very simple microservice defining a simple HTML website. 27 | 28 | .. literalinclude:: ../samples/website/tech/website.py 29 | 30 | .. _directory: 31 | 32 | Directory 33 | --------- 34 | 35 | This microservice is usefull for the ``BizMicroservice``. 36 | 37 | .. literalinclude:: ../samples/directory/tech/app.py 38 | 39 | To create your directory service, you just have to define a file ``env_vars/vars.secret.json`` like :: 40 | 41 | { 42 | "AWS_USER_ACCESS_KEY_ID": XXXX, 43 | "AWS_USER_SECRET_ACCESS_KEY": YYY 44 | } 45 | 46 | -------------------------------------------------------------------------------- /coworks/utils.py: -------------------------------------------------------------------------------- 1 | from __future__ import annotations 2 | 3 | import os 4 | 5 | import dotenv 6 | 7 | from .const import DEFAULT_DEV_STAGE 8 | 9 | 10 | def is_json(mt): 11 | """Checks if a mime type is json. 12 | """ 13 | return ( 14 | mt == "application/json" 15 | or isinstance(mt, str) 16 | and mt.startswith("application/") 17 | and mt.endswith("+json") 18 | ) 19 | 20 | 21 | def to_bool(val: str | bool) -> bool: 22 | if isinstance(val, bool): 23 | return val 24 | if isinstance(val, str): 25 | return val.lower() in ['true', '1', 'yes'] 26 | return False 27 | 28 | 29 | def get_app_stage(): 30 | """Defined only on deployed microservice or should be set manually.""" 31 | return os.getenv('CWS_STAGE', DEFAULT_DEV_STAGE) 32 | 33 | 34 | def load_dotenv(stage: str): 35 | values = {} 36 | for env_filename in get_env_filenames(stage): 37 | path = dotenv.find_dotenv(env_filename, usecwd=True) 38 | if path: 39 | values.update(dotenv.dotenv_values(path)) 40 | return values 41 | 42 | 43 | def get_env_filenames(stage): 44 | return [".env", ".flaskenv", f".env.{stage}", f".flaskenv.{stage}"] 45 | -------------------------------------------------------------------------------- /samples/docs/tech/complete.py: -------------------------------------------------------------------------------- 1 | from aws_xray_sdk.core import xray_recorder 2 | 3 | from coworks import TechMicroService 4 | from coworks import entry 5 | from coworks.blueprint.admin_blueprint import Admin 6 | from coworks.blueprint.profiler_blueprint import Profiler 7 | from coworks.extension.xray import XRay 8 | 9 | 10 | class SimpleMicroService(TechMicroService): 11 | DOC_MD = """ 12 | #### Microservice Documentation 13 | You can document your CoWorks MicroService using the class attributes `DOC_MD` (markdown) or 14 | the instance attributes `doc_md` (markdown) which gets rendered from the '/' entry of the admin blueprint. 15 | """ 16 | 17 | def __init__(self, **kwargs): 18 | super().__init__(**kwargs) 19 | self.register_blueprint(Admin(), url_prefix='/admin') 20 | self.register_blueprint(Profiler(self), url_prefix='/profile') 21 | self.value = 0 22 | 23 | @entry 24 | def get(self): 25 | return f"Stored value {self.value}.\n" 26 | 27 | @entry 28 | def post(self, value=None): 29 | if value is not None: 30 | self.value = value 31 | return f"Value stored ({value}).\n" 32 | 33 | 34 | app = SimpleMicroService(name="sample-complete-microservice") 35 | 36 | XRay(app, xray_recorder) 37 | -------------------------------------------------------------------------------- /coworks/cws/project_templates/README.md: -------------------------------------------------------------------------------- 1 | ### Instructions 2 | 3 | Info : *This project was created by the cws new command.* 4 | 5 | #### Run 6 | 7 | To run the microservice locally : 8 | 9 | ``` 10 | $> CWS_STAGE=dev cws run 11 | * Stage: dev 12 | * Debug mode: off 13 | * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 14 | ``` 15 | Then on another terminal, enter : 16 | 17 | ``` 18 | $> curl -H "Authorization:to_be_set" -H "USER_KEY:test" http://localhost:5000/ 19 | project ready! 20 | ``` 21 | 22 | If you are testing your microservice in an IDE then set the root path to src/tech and 23 | define the INSTANCE_RELATIVE_PATH to .. (there env_vars are defined in fact) 24 | 25 | #### Deploy : 26 | 27 | ``` 28 | $> CWS_STAGE=dev cws -p tech deploy -b bucket_name -pn aws_profile 29 | * Workspace: dev 30 | Copy files to S3 [####################################] 100% 0% 31 | Terraform apply (Create API routes) 32 | Terraform apply (Deploy API and Lambda for the dev stage) 33 | Deploy microservice [####################################] 100% 34 | terraform output 35 | coworks_layers_id = "xxxx" 36 | 37 | $> curl -H "Authorization:to_be_set" -H "USER_KEY:test" https://xxxx.execute-api.eu-west-1.amazonaws.com/dev 38 | project ready! 39 | ``` 40 | -------------------------------------------------------------------------------- /samples/website/README.md: -------------------------------------------------------------------------------- 1 | ### Instructions 2 | 3 | Info : *This project was created by the cws new command.* 4 | 5 | #### Run 6 | 7 | To run the microservice locally : 8 | 9 | ``` 10 | $> CWS_STAGE=dev cws -p tech run 11 | * Environment: dev 12 | * Debug mode: off 13 | * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 14 | ``` 15 | Then on another terminal, enter : 16 | 17 | ``` 18 | $> curl -H "Authorization:to_be_set" -H "USER_KEY:test" http://localhost:5000/ 19 | project ready! 20 | ``` 21 | 22 | If you are testing your microservice in an IDE then set the root path to src/tech and 23 | define the INSTANCE_RELATIVE_PATH to .. (there env_vars are defined in fact) 24 | 25 | #### Deploy : 26 | 27 | ``` 28 | $> CWS_STAGE=dev cws -p tech deploy -b bucket_name -pn aws_profile 29 | * Workspace: dev 30 | Copy files to S3 [####################################] 100% 0% 31 | Terraform apply (Create API routes) 32 | Terraform apply (Deploy API and Lambda for the dev stage) 33 | Deploy microservice [####################################] 100% 34 | terraform output 35 | coworks_layers_id = "xxxx" 36 | 37 | $> curl -H "Authorization:to_be_set" -H "USER_KEY:test" https://xxxx.execute-api.eu-west-1.amazonaws.com/dev 38 | project ready! 39 | ``` 40 | -------------------------------------------------------------------------------- /tests/coworks/tech/test_async.py: -------------------------------------------------------------------------------- 1 | from unittest import mock 2 | from unittest.mock import MagicMock 3 | 4 | import pytest 5 | 6 | from coworks import TechMicroService 7 | from coworks import entry 8 | from coworks.const import BIZ_BUCKET_HEADER_KEY 9 | from coworks.const import BIZ_KEY_HEADER_KEY 10 | from ..event import get_event 11 | 12 | 13 | class AsyncMS(TechMicroService): 14 | 15 | @entry 16 | def get(self): 17 | return "ok" 18 | 19 | 20 | @pytest.mark.skip 21 | class TestClass: 22 | def test_async_store(self, empty_aws_context): 23 | with mock.patch('boto3.session.Session') as session: 24 | session.side_effect = lambda: session 25 | session.client = MagicMock(side_effect=lambda _: session.client) 26 | app = AsyncMS() 27 | event = get_event('/', 'get') 28 | with app.cws_client(event, empty_aws_context) as c: 29 | event['headers']['InvocationType'.lower()] = 'Event' 30 | event['headers'][BIZ_BUCKET_HEADER_KEY.lower()] = 'bucket' 31 | event['headers'][BIZ_KEY_HEADER_KEY.lower()] = 'specific/key' 32 | app(event, empty_aws_context) 33 | session.assert_called_once() 34 | session.client.assert_called_once_with('s3') 35 | session.client.upload_fileobj.assert_called_once() 36 | -------------------------------------------------------------------------------- /docs/faq.rst: -------------------------------------------------------------------------------- 1 | .. _faq: 2 | 3 | FAQ 4 | === 5 | 6 | This is a list of Frequently Asked Questions about ``CoWorks``. Feel free to 7 | suggest new entries! 8 | 9 | Is CoWorks ... 10 | -------------- 11 | 12 | ... a complete unified approach? 13 | The hardest part of using the microservice approach is to be able to control, deploy, maintain and debug composition 14 | of many microservices. Having a compositional approach is the key for production usage. 15 | CoWorks integrates all information and convention to simplify such composition. 16 | ... independant of any specific IaaS? 17 | No, it relies only on AWS solutions. There are already performant solutions to abstract any cloud providers such as 18 | `Zappa `_, `Serverless `_... 19 | The aim of the ``CoWorks`` project is to simplify the experience of using microservices in production with AWS technologies 20 | not to provide a new model. 21 | ... complicated? 22 | No, the ``CoWorks`` framework is based on twos kinds of services: 23 | 24 | * Simple atomic microservices called ``TechMicroservices``. 25 | * Composed microservices called ``BizMicroservices``. 26 | 27 | The model uses mainly Lambda and Airflow technologies but those complex but awesome technologies are used 28 | in asimplified manner for users. 29 | 30 | We welcome any contributions that improve the quality of our projects. 31 | 32 | 33 | -------------------------------------------------------------------------------- /coworks/cws/templates/terraform/response_templates.j2: -------------------------------------------------------------------------------- 1 | {% macro json_response(mimetype) %} 2 | "{{ mimetype }}" = <<-EOF 3 | #set($inputRoot = $input.path('$')) 4 | #set($context.responseOverride.status=$inputRoot.statusCode) 5 | #set($context.responseOverride.header=$inputRoot.headers) 6 | $input.json('$.body') 7 | EOF 8 | {%- endmacro %} 9 | {% macro text_response(mimetype) %} 10 | "{{ mimetype }}" = <<-EOF 11 | #set($inputRoot = $input.path('$')) 12 | #set($context.responseOverride.status=$inputRoot.statusCode) 13 | #set($context.responseOverride.header=$inputRoot.headers) 14 | $inputRoot.body 15 | EOF 16 | {%- endmacro %} 17 | {% macro byte_response(mimetype) %} 18 | "{{ mimetype }}" = <<-EOF 19 | #set($inputRoot = $input.path('$')) 20 | #set($context.responseOverride.status=$inputRoot.statusCode) 21 | #set($context.responseOverride.header=$inputRoot.headers) 22 | $util.base64Decode($inputRoot.body) 23 | EOF 24 | {%- endmacro %} 25 | {{ json_response("application/json") }} 26 | {{ json_response("text/x-json") }} 27 | {{ json_response("application/javascript") }} 28 | {{ json_response("application/x-javascript") }} 29 | {{ json_response("application/vnd.api+json") }} 30 | {% for type in json_types %} 31 | {{ json_response(type) }} 32 | {% endfor %} 33 | {{ text_response("text/plain") }} 34 | {{ text_response("text/html") }} 35 | {% for type in text_types %} 36 | {{ text_response(type) }} 37 | {% endfor %} 38 | -------------------------------------------------------------------------------- /tests/docs/test_complete.py: -------------------------------------------------------------------------------- 1 | import multiprocessing 2 | import os 3 | import time 4 | from unittest import mock 5 | 6 | import pytest 7 | import requests 8 | 9 | from coworks.cws.client import CwsScriptInfo 10 | from coworks.cws.client import import_attr 11 | 12 | 13 | @pytest.mark.not_on_github 14 | class TestClass: 15 | @mock.patch.dict(os.environ, {"FLASK_RUN_FROM_CLI": "false"}) 16 | def test_run_complete(self, samples_docs_dir, unused_tcp_port): 17 | info = CwsScriptInfo(project_dir='tech') 18 | info.app_import_path = "complete:app" 19 | app = info.load_app() 20 | app = import_attr('complete', 'app') 21 | server = multiprocessing.Process(target=run_server, args=(app, unused_tcp_port), daemon=False) 22 | server.start() 23 | counter = 1 24 | time.sleep(counter) 25 | while not server.is_alive() and counter < 3: 26 | time.sleep(counter) 27 | counter += 1 28 | response = requests.get(f'http://localhost:{unused_tcp_port}/', headers={'Authorization': "token"}) 29 | assert response.text == "Stored value 0.\n" 30 | response = requests.get(f'http://localhost:{unused_tcp_port}/admin/route', 31 | headers={'Authorization': "token"}) 32 | assert response.status_code == 200 33 | server.terminate() 34 | 35 | 36 | def run_server(app, port): 37 | print(f"Server starting on port {port}") 38 | app.run(host='localhost', port=port, use_reloader=False, debug=False) 39 | -------------------------------------------------------------------------------- /coworks/cws/project_templates/tech/app.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from coworks import TechMicroService 4 | from coworks import entry 5 | from coworks.blueprint.admin_blueprint import Admin 6 | from coworks.blueprint.profiler_blueprint import Profiler 7 | 8 | 9 | # from aws_xray_sdk.core import xray_recorder 10 | # from coworks.extension.xray import XRay 11 | 12 | 13 | class MyMicroService(TechMicroService): 14 | DOC_MD = """ 15 | ## Microservice for ... 16 | """ 17 | 18 | def __init__(self, **kwargs): 19 | super().__init__(**kwargs) 20 | 21 | self.register_blueprint(Admin(), url_prefix='/admin') 22 | if os.getenv('CWS_STAGE') == "dev": 23 | self.register_blueprint(Profiler(self), url_prefix='/profiler') 24 | 25 | @self.before_request 26 | def before(): 27 | ... 28 | 29 | @self.errorhandler(500) 30 | def handle(e): 31 | ... 32 | 33 | def init_cli(self): 34 | """ 35 | from flask_migrate import Migrate 36 | Migrate(self, db) 37 | """ 38 | 39 | def token_authorizer(self, token): 40 | # Simple authorization process. 41 | # If you want to access AWS event or context for a more complex case, override the function _token_handler. 42 | return token in os.getenv('USER_KEYS', '').split(',') 43 | 44 | @entry 45 | def get(self): 46 | return 'project ready!\n' 47 | 48 | 49 | app = MyMicroService() 50 | 51 | # For this extension you need to install 'aws_xray_sdk'. 52 | # XRay(app, xray_recorder) 53 | -------------------------------------------------------------------------------- /tests/cws/test_project_file.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | import pytest 5 | from click import UsageError 6 | 7 | from coworks.cws.client import client 8 | 9 | 10 | class TestClass: 11 | @mock.patch.dict(os.environ, {"FLASK_APP": "command:app"}) 12 | def test_no_project_file_no_module(self, capsys): 13 | with pytest.raises(SystemExit) as pytest_wrapped_e: 14 | client(prog_name='cws', args=['--project-dir', '.', 'test'], obj={}) 15 | assert pytest_wrapped_e.type == SystemExit 16 | assert pytest_wrapped_e.value.code == 0 17 | 18 | @mock.patch.dict(os.environ, {"FLASK_APP": "command:app"}) 19 | def test_wrong_project_dir(self, example_dir): 20 | with pytest.raises(UsageError) as pytest_wrapped_e: 21 | client.main(['--project-dir', 'doesntexist', 'test'], 'cws', standalone_mode=False) 22 | msg = "tests/cws/src/doesntexist not found." 23 | assert pytest_wrapped_e.value.args[0].endswith(msg) 24 | with pytest.raises(SystemExit) as pytest_wrapped_e: 25 | client(prog_name='cws', args=['-p', 'doesntexist', 'test']) 26 | 27 | def test_wrong_project_config_version(self, example_dir): 28 | with pytest.raises(RuntimeError) as pytest_wrapped_e: 29 | client.main(['--project-dir', '.', '--config-file-suffix', '.wrong.yml', 'cmd'], 'cws', 30 | standalone_mode=False) 31 | assert pytest_wrapped_e.type == RuntimeError 32 | assert pytest_wrapped_e.value.args[0] == "Wrong project file version (should be 3).\n" 33 | -------------------------------------------------------------------------------- /tests/conftest.py: -------------------------------------------------------------------------------- 1 | import contextlib 2 | import os 3 | import socket 4 | from unittest.mock import MagicMock 5 | 6 | import pytest 7 | 8 | fixture_example_dir = os.getenv('EXAMPLE_DIR', 'tests/cws/src') 9 | fixture_samples_docs_dir = os.getenv('SAMPLES_DOCS_DIR', 'samples/docs') 10 | 11 | 12 | @pytest.fixture 13 | def example_dir(monkeypatch): 14 | monkeypatch.syspath_prepend(fixture_example_dir) 15 | monkeypatch.chdir(fixture_example_dir) 16 | yield fixture_example_dir 17 | 18 | 19 | @pytest.fixture 20 | def samples_docs_dir(monkeypatch): 21 | monkeypatch.syspath_prepend(fixture_example_dir) 22 | monkeypatch.chdir(fixture_samples_docs_dir) 23 | yield fixture_samples_docs_dir 24 | 25 | 26 | @pytest.fixture 27 | def progressbar(): 28 | yield MagicMock() 29 | 30 | 31 | @pytest.fixture 32 | def unused_tcp_port(): 33 | with contextlib.closing(socket.socket()) as sock: 34 | sock.bind(('localhost', 0)) 35 | return sock.getsockname()[1] 36 | 37 | 38 | @pytest.fixture 39 | def auth_headers(): 40 | yield { 41 | 'authorization': 'pytest', 42 | } 43 | 44 | 45 | @pytest.fixture 46 | def empty_aws_context(): 47 | return object() 48 | 49 | 50 | def pytest_sessionstart(): 51 | if not os.path.exists(fixture_samples_docs_dir): 52 | msg = "Undefined samples folder: (environment variable 'SAMPLES_DOCS_DIR' must be redefined)." 53 | raise pytest.UsageError(msg) 54 | if not os.path.exists(fixture_example_dir): 55 | msg = "Undefined example folder: (environment variable 'EXAMPLE_DIR' must be redefined)." 56 | raise pytest.UsageError(msg) 57 | -------------------------------------------------------------------------------- /coworks/cws/new.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import typing as t 3 | from pathlib import Path 4 | from shutil import copytree 5 | 6 | import click 7 | from jinja2 import Environment 8 | from jinja2 import PackageLoader 9 | from jinja2 import select_autoescape 10 | 11 | from coworks.version import __version__ 12 | from .command import CwsCommand 13 | from .command import no_project_context 14 | 15 | 16 | @click.command("new", CwsCommand, short_help="Creates a new CoWorks project.") 17 | @click.option('--dir', '-d', default='.', help="Directory where the projet will be created.") 18 | @click.option('--force', '-f', is_flag=True, help="Force creation even if already created.") 19 | @no_project_context 20 | def new_command(dir, force) -> None: 21 | project_templates = Path(__file__).parent / 'project_templates' 22 | 23 | # Copy project configuration file 24 | dest = Path(dir) 25 | project_conf = dest / "project.cws.yml" 26 | 27 | if project_conf.exists() and not force: 28 | click.echo(f"Project was already created in {dest}. Set 'force' option for recreation.") 29 | return 30 | 31 | # Copy files 32 | copytree(src=project_templates.as_posix(), dst=dest, dirs_exist_ok=True) 33 | (dest / 'template.env').rename('.env') 34 | 35 | # Render project configuration file 36 | template_loader = PackageLoader(t.cast(str, sys.modules[__name__].__package__)) 37 | jinja_env = Environment(loader=template_loader, autoescape=select_autoescape(['html', 'xml'])) 38 | template = jinja_env.get_template('project.cws.yml') 39 | output = project_conf 40 | with output.open("w") as f: 41 | f.write(template.render({'version': __version__.replace('.', '_')})) 42 | 43 | click.echo('New project created.') 44 | -------------------------------------------------------------------------------- /docs/changelog.rst: -------------------------------------------------------------------------------- 1 | .. _changelog: 2 | 3 | Changelogs 4 | ========== 5 | 6 | 7 | Version 0.8 8 | ^^^^^^^^^^^ 9 | 10 | First released 11 | 12 | * Transformer added to asynchronous group 13 | * Using dotenv for stage environment variables 14 | * Full Flask 2.2 compatibility 15 | * Apache Airflow asynchronous group defined 16 | 17 | Version 0.7 18 | ^^^^^^^^^^^ 19 | 20 | First released 2022-03-28 21 | 22 | * Apache Airflow operators and sensor defined. 23 | * Better client itegration. 24 | * Doc in progress. 25 | * Workspace command option replaced by FLASK_ENV 26 | * Proxy class defined 27 | 28 | Version 0.6 29 | ^^^^^^^^^^^ 30 | 31 | First released 2021-11-02 32 | 33 | * Moved on Flask framework. 34 | * Middlewares defined. 35 | * Remote terraform. 36 | 37 | Version 0.5 38 | ^^^^^^^^^^^ 39 | 40 | First released 2021-02-03 41 | 42 | * Configuration files search in root before project dir. 43 | * Deploy/destroy comand enhanced. 44 | * More blueprints. 45 | 46 | Version 0.4 47 | ^^^^^^^^^^^ 48 | 49 | First released 2021-01-06 50 | 51 | * Global project configuration file. 52 | * Middleware renamed as Context Manager. 53 | * Deployer command using terraform. 54 | * Samples defined. 55 | 56 | Version 0.3 57 | ^^^^^^^^^^^ 58 | 59 | First released 2020-04-26 60 | 61 | * BizMicroservice YAML description language defined. 62 | * BizMicroservice service added. 63 | * BizFactory service added. 64 | * Generic command added 65 | 66 | Version 0.2 67 | ^^^^^^^^^^^ 68 | 69 | First released 2019-12-27 70 | 71 | * Authorization service added. 72 | * Admin blueprint added. 73 | * Documentation created. 74 | 75 | Version 0.1 76 | ^^^^^^^^^^^ 77 | 78 | First released 2019-12-17 79 | 80 | * First public preview release. 81 | * Microservice and Blueprint released. 82 | 83 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file only contains a selection of the most common options. For a full 4 | # list see the documentation: 5 | # https://www.sphinx-doc.org/en/master/usage/configuration.html 6 | 7 | # -- Path setup -------------------------------------------------------------- 8 | 9 | from coworks import __version__ 10 | 11 | master_doc = 'index' 12 | 13 | # -- Project information ----------------------------------------------------- 14 | 15 | project = 'CoWorks' 16 | copyright = '2024, gdoumenc - FPR' 17 | author = 'gdoumenc' 18 | 19 | # The full version, including alpha/beta/rc tags 20 | release = __version__ 21 | 22 | # -- General configuration --------------------------------------------------- 23 | 24 | extensions = [ 25 | 'sphinx.ext.autodoc', 26 | ] 27 | 28 | # Add any paths that contain templates here, relative to this directory. 29 | templates_path = ['_templates'] 30 | 31 | # List of patterns, relative to source directory, that match files and 32 | # directories to ignore when looking for source files. 33 | # This pattern also affects html_static_path and html_extra_path. 34 | exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] 35 | 36 | # -- Options for HTML output ------------------------------------------------- 37 | 38 | # The theme to use for HTML and HTML Help pages. See the documentation for 39 | # a list of builtin themes. 40 | # 41 | html_theme = 'alabaster' 42 | html_theme_options = { 43 | 'logo': 'img/coworks.png', 44 | } 45 | html_sidebars = { 46 | '**': [ 47 | 'globaltoc.html', 48 | ] 49 | } 50 | # Add any paths that contain custom static files (such as style sheets) here, 51 | # relative to this directory. They are copied after the builtin static files, 52 | # so a file named "default.css" will overwrite the builtin "default.css". 53 | html_static_path = ['_static'] 54 | -------------------------------------------------------------------------------- /coworks/extension/jsonapi/pydantic.py: -------------------------------------------------------------------------------- 1 | from datetime import datetime 2 | 3 | from werkzeug.exceptions import UnprocessableEntity 4 | 5 | from coworks.utils import to_bool 6 | from .data import JsonApiBaseModel 7 | from .fetching import fetching_context 8 | 9 | 10 | def pydantic_filter(base_model: JsonApiBaseModel): 11 | _base_model_filters: list[bool] = [] 12 | for filter in fetching_context.get_filter_parameters(base_model.jsonapi_type): 13 | for key, oper, value in filter: 14 | if '.' in key: 15 | _, key = key.split('.', 1) 16 | if not hasattr(base_model, key): 17 | msg = f"Wrong '{key}' key for '{base_model.jsonapi_type}' in filters parameters" 18 | raise UnprocessableEntity(msg) 19 | column = getattr(base_model, key) 20 | 21 | if oper == 'null': 22 | if to_bool(value): 23 | _base_model_filters.append(column is None) 24 | else: 25 | _base_model_filters.append(column is not None) 26 | continue 27 | 28 | _type = base_model.model_fields.get(key).annotation # type: ignore[union-attr] 29 | if _type is bool: 30 | _base_model_filters.append(base_model_filter(column, oper, to_bool(value))) 31 | elif _type is int: 32 | _base_model_filters.append(base_model_filter(column, oper, int(value))) 33 | elif _type is datetime: 34 | _base_model_filters.append( 35 | base_model_filter(datetime.fromisoformat(column), oper, datetime.fromisoformat(value)) 36 | ) 37 | else: 38 | _base_model_filters.append(base_model_filter(str(column), oper, value)) 39 | 40 | return all(_base_model_filters) 41 | 42 | 43 | def base_model_filter(column, oper, value) -> bool: 44 | """String filter.""" 45 | oper = oper or 'eq' 46 | if oper == 'eq': 47 | return column == value 48 | if oper == 'neq': 49 | return column != value 50 | if oper == 'contains': 51 | return value in column 52 | if oper == 'ncontains': 53 | return value not in column 54 | msg = f"Undefined operator '{oper}' for string value" 55 | raise UnprocessableEntity(msg) 56 | -------------------------------------------------------------------------------- /coworks/extension/jsonapi/query.py: -------------------------------------------------------------------------------- 1 | import typing as t 2 | 3 | from pydantic import BaseModel 4 | from pydantic import ConfigDict 5 | from sqlalchemy.exc import MultipleResultsFound 6 | from sqlalchemy.exc import NoResultFound 7 | 8 | from .data import CursorPagination 9 | from .data import JsonApiDataMixin 10 | 11 | 12 | class Pagination(t.Protocol, t.Iterable): 13 | total: int 14 | page: int 15 | pages: int 16 | per_page: int 17 | has_prev: bool 18 | prev_num: int | None 19 | has_next: bool 20 | next_num: int | None 21 | 22 | 23 | @t.runtime_checkable 24 | class Scalar(t.Protocol): 25 | def one(self) -> JsonApiDataMixin: 26 | ... 27 | 28 | 29 | @t.runtime_checkable 30 | class Query(t.Protocol): 31 | def paginate(self, *, page, per_page, max_per_page) -> type[Pagination]: 32 | ... 33 | 34 | def all(self) -> list[JsonApiDataMixin]: 35 | ... 36 | 37 | def one(self) -> JsonApiDataMixin: 38 | ... 39 | 40 | 41 | class ListPagination(CursorPagination): 42 | total: int = 0 43 | values: list[t.Any] 44 | 45 | def __init__(self, **data): 46 | super().__init__(**data) 47 | self.total = len(self.values) 48 | 49 | def __iter__(self): 50 | assert self.page is not None # by the validator 51 | assert self.per_page is not None # by the validator 52 | return self.values[(self.page - 1) * self.per_page: self.page * self.per_page].__iter__() 53 | 54 | @classmethod 55 | def paginate(cls, values) -> Pagination: 56 | pagination = cls(values=values, page=1, per_page=len(values)) 57 | return t.cast(Pagination, pagination) 58 | 59 | 60 | class ListQuery(BaseModel): 61 | model_config = ConfigDict(arbitrary_types_allowed=True) 62 | values: list[JsonApiDataMixin] 63 | 64 | def paginate(self, *, page, per_page, max_per_page) -> Pagination: 65 | return ListPagination(values=self.values, page=page, per_page=per_page) # type: ignore[return-value] 66 | 67 | def all(self) -> list[JsonApiDataMixin]: 68 | return self.values 69 | 70 | def one(self) -> JsonApiDataMixin: 71 | if len(self.values) == 0: 72 | raise NoResultFound() 73 | if len(self.values) > 1: 74 | raise MultipleResultsFound() 75 | return self.values[0] 76 | -------------------------------------------------------------------------------- /coworks/cws/templates/terraform.j2: -------------------------------------------------------------------------------- 1 | {% if terraform_cloud -%} 2 | # --------------------------------------------------------------------------------------------------------------------- 3 | # TERRAFORM ON CLOUD 4 | # --------------------------------------------------------------------------------------------------------------------- 5 | 6 | terraform { 7 | backend "remote" { 8 | organization = "{{ terraform_organization }}" 9 | 10 | workspaces { 11 | name = "{{ ms_name }}_{{ stage }}" 12 | } 13 | } 14 | required_version = ">= 1.2.0" 15 | required_providers { 16 | aws = { 17 | source = "hashicorp/aws" 18 | version = ">= 5.2.0" 19 | } 20 | } 21 | } 22 | 23 | variable "AWS_ACCESS_KEY_ID" {} 24 | variable "AWS_SECRET_ACCESS_KEY" {} 25 | variable "AWS_REGION" {default="eu-west-1"} 26 | variable "TFC_WORKSPACE_NAME" {} 27 | 28 | provider "aws" { 29 | access_key = var.AWS_ACCESS_KEY_ID 30 | secret_key = var.AWS_SECRET_ACCESS_KEY 31 | region = var.AWS_REGION 32 | 33 | default_tags { 34 | tags = { 35 | Creator = "terraform" 36 | Stage = "{{ stage }}" 37 | } 38 | } 39 | } 40 | 41 | locals { 42 | {%- if profile_name %} 43 | profile = "{{ profile_name }}" 44 | {% else %} 45 | account_id = data.aws_caller_identity.current.account_id 46 | {%- endif %} 47 | region = {{ '"{}"'.format(aws_region) if aws_region else 'var.AWS_REGION'}} 48 | } 49 | {% else -%} 50 | # --------------------------------------------------------------------------------------------------------------------- 51 | # TERRAFORM ON S3 52 | # --------------------------------------------------------------------------------------------------------------------- 53 | 54 | variable "AWS_REGION" {default="eu-west-1"} 55 | 56 | terraform { 57 | backend "s3" { 58 | bucket = "{{ tf_bucket }}" 59 | key = "{{ tf_key }}/tfstate_{{ workspace }}.json" 60 | region = {{ '"{}"'.format(aws_region) if aws_region else 'var.CUSTOMER_AWS_REGION'}} 61 | profile = "{{ profile_name }}" 62 | } 63 | required_version = ">= 1.2.0" 64 | required_providers { 65 | local = { 66 | source = "hashicorp/local" 67 | } 68 | } 69 | } 70 | 71 | locals { 72 | account_id = "{{ aws_account }}" 73 | region = {{ '"{}"'.format(aws_region) if aws_region else 'var.AWS_REGION'}} 74 | } 75 | {%- endif %} 76 | 77 | data "aws_caller_identity" "current" {} 78 | -------------------------------------------------------------------------------- /coworks/cws/templates/terraform/role.j2: -------------------------------------------------------------------------------- 1 | locals { 2 | {% if workspace != "common" %} 3 | {{ ms_name }}_role_arn = data.aws_iam_role.{{ ms_name }}_cws.arn 4 | {% else %} 5 | {{ ms_name }}_role_arn = aws_iam_role.{{ ms_name }}_cws.arn 6 | {% endif %} 7 | } 8 | 9 | # --------------------------------------------------------------------------------------------------------------------- 10 | # ROLE 11 | # --------------------------------------------------------------------------------------------------------------------- 12 | 13 | {% if workspace != "common" %} 14 | data "aws_iam_role" "{{ ms_name }}_cws" { 15 | provider = aws.{{ ms_name }} 16 | name = "{{ ms_name }}_cws_role" 17 | } 18 | {% else %} 19 | resource "aws_iam_role" "{{ ms_name }}_cws" { 20 | provider = aws.{{ ms_name }} 21 | name = "{{ ms_name }}_cws_role" 22 | 23 | assume_role_policy = jsonencode({ 24 | Version: "2012-10-17", 25 | Statement: [ 26 | { 27 | Action: "sts:AssumeRole", 28 | Principal: { 29 | Service: ["lambda.amazonaws.com"] 30 | }, 31 | Effect: "Allow", 32 | Sid: "" 33 | } 34 | ] 35 | }) 36 | } 37 | 38 | resource "aws_iam_role_policy_attachment" "{{ ms_name }}_s3" { 39 | provider = aws.{{ ms_name }} 40 | role = aws_iam_role.{{ ms_name }}_cws.name 41 | policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess" 42 | } 43 | 44 | resource "aws_iam_role_policy_attachment" "{{ ms_name }}_cloud_watch" { 45 | provider = aws.{{ ms_name }} 46 | role = aws_iam_role.{{ ms_name }}_cws.name 47 | policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs" 48 | } 49 | 50 | resource "aws_iam_role_policy_attachment" "{{ ms_name }}_xray" { 51 | provider = aws.{{ ms_name }} 52 | role = aws_iam_role.{{ ms_name }}_cws.name 53 | policy_arn = "arn:aws:iam::aws:policy/AWSXrayFullAccess" 54 | } 55 | 56 | resource "aws_iam_role_policy" "{{ ms_name }}_vpc" { 57 | provider = aws.{{ ms_name }} 58 | name = "{{ ms_name }}_vpc" 59 | role = aws_iam_role.{{ ms_name }}_cws.name 60 | policy = jsonencode({ 61 | Version: "2012-10-17", 62 | Statement: [ 63 | { 64 | Effect: "Allow", 65 | Action: [ 66 | "ec2:CreateNetworkInterface", 67 | "ec2:DescribeNetworkInterfaces", 68 | "ec2:DeleteNetworkInterface" 69 | ], 70 | Resource: "*" 71 | } 72 | ] 73 | }) 74 | } 75 | {% endif %} 76 | 77 | -------------------------------------------------------------------------------- /samples/website/tech/templates/home.j2: -------------------------------------------------------------------------------- 1 | {% extends "base.j2" %} 2 | 3 | {% macro info(title, a_link=None) %} 4 |
    6 |
    7 | 8 | 13 | 14 |
    15 |
    16 |

    17 | 18 | 19 | 20 | {{ title }} 21 | 22 |

    23 |

    {{ caller() }}

    24 |
    25 | 32 |
    33 | {% endmacro %} 34 | 35 | {% block content %} 36 |
    37 |
    39 | 40 | 41 | {% call info("Last AWS layer", a_link=last_layer_url) %} 42 | {{ last_layer }} 43 | {% endcall %} 44 | 45 | {% call info("AirFlow plugins", a_link = url_for("get_zip")) %} 46 | The AirFlow plugins containing the CoWorks operators and sensors. 47 | {% endcall %} 48 |
    49 |
    50 | {% endblock content %} 51 | 52 | {% block news %} 53 | {% include "news.j2" %} 54 | {% endblock news %} 55 | -------------------------------------------------------------------------------- /samples/website/tech/templates/account/login.j2: -------------------------------------------------------------------------------- 1 | {% extends "base.j2" %} 2 | 3 | {% block header %} 4 |
    5 |

    Sign In

    6 |

    7 | Example of a form calling a CWS Microservices.

    8 |
    9 | {% endblock header %} 10 | 11 | {% block content %} 12 |
    13 |
    14 |
    15 | {{ form.csrf_token }} 16 | 17 |
    18 | {{ form.email.label }} 19 | 23 |
    24 |
    25 | {{ form.password.label }} 26 | 30 |
    31 |
    32 | 34 | Back to Home 35 | 39 |
    40 |
    41 |
    42 |
    43 | {% endblock content %} 44 | 45 | {% block news %} 46 | {% include "news.j2" %} 47 | {% endblock news %} 48 | -------------------------------------------------------------------------------- /tests/coworks/blueprint/test_blueprint.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | from flask import url_for 5 | 6 | from ..blueprint.blueprint import BP, InitBP 7 | from ..ms import SimpleMS 8 | 9 | 10 | @mock.patch.dict(os.environ, {"TOKEN": "token"}) 11 | class TestClass: 12 | 13 | def test_request(self): 14 | app = SimpleMS() 15 | app.register_blueprint(BP()) 16 | with app.test_client() as c: 17 | response = c.get('/', headers={'Authorization': 'token'}) 18 | assert response.status_code == 200 19 | assert response.get_data(as_text=True) == 'get' 20 | response = c.get('/test/3', headers={'Authorization': 'token'}) 21 | assert response.status_code == 200 22 | assert response.get_data(as_text=True) == "blueprint BP 3" 23 | response = c.get('/extended/test/3', headers={'Authorization': 'token'}) 24 | assert response.status_code == 200 25 | assert response.get_data(as_text=True) == 'blueprint extended test 3' 26 | 27 | def test_prefix(self): 28 | app = SimpleMS() 29 | app.register_blueprint(BP(), url_prefix="/prefix") 30 | with app.test_client() as c: 31 | response = c.get('/prefix/test/3', headers={'Authorization': 'token'}) 32 | assert response.status_code == 200 33 | assert response.get_data(as_text=True) == "blueprint BP 3" 34 | response = c.get('/prefix/extended/test/3', headers={'Authorization': 'token'}) 35 | assert response.status_code == 200 36 | assert response.get_data(as_text=True) == 'blueprint extended test 3' 37 | 38 | def test_url_for(self): 39 | app = SimpleMS() 40 | app.register_blueprint(BP(), url_prefix="/prefix") 41 | with app.test_client() as c: 42 | response = c.get('/prefix/test/3', headers={'Authorization': 'token'}) 43 | assert url_for('get') == '/' 44 | assert url_for('bp.get_test', index=2) == '/prefix/test/2' 45 | 46 | def test_before_activation(self): 47 | app = SimpleMS() 48 | init_bp = InitBP() 49 | app.register_blueprint(init_bp, url_prefix="/prefix") 50 | with app.test_client() as c: 51 | response = c.get('/prefix/test/3', headers={'Authorization': 'token'}) 52 | assert response.status_code == 200 53 | assert response.get_data(as_text=True) == "blueprint BP 3" 54 | init_bp.do_before_activation.assert_called_once() 55 | init_bp.do_after_activation.assert_called_once() 56 | -------------------------------------------------------------------------------- /tests/coworks/ms.py: -------------------------------------------------------------------------------- 1 | import io 2 | from unittest.mock import MagicMock 3 | 4 | from coworks import TechMicroService 5 | from coworks import entry 6 | from coworks import request 7 | 8 | 9 | class TechMS(TechMicroService): 10 | def __init__(self, **kwargs): 11 | super().__init__('test', **kwargs) 12 | 13 | def token_authorizer(self, token): 14 | return True 15 | 16 | 17 | class S3MockTechMS(TechMicroService): 18 | def __init__(self): 19 | super().__init__('test') 20 | session = MagicMock() 21 | session.client = MagicMock() 22 | s3_object = {'Body': io.BytesIO(b'test'), 'ContentType': 'text/plain'} 23 | session.client.get_object = MagicMock(return_value=s3_object) 24 | self.aws_s3_run_session = session 25 | 26 | 27 | class SimpleMS(TechMicroService): 28 | 29 | @entry 30 | def get(self): 31 | """Root access.""" 32 | return "get" 33 | 34 | def get1(self): 35 | """Not recognized.""" 36 | return "get1" 37 | 38 | @entry 39 | def get_content(self): 40 | return "get content" 41 | 42 | @entry 43 | def get__content(self, value): 44 | return f"get content with {value}" 45 | 46 | @entry 47 | def get_content__(self, value, other): 48 | return f"get content with {value} and {other}" 49 | 50 | @entry 51 | def post_content(self, other="none"): 52 | return f"post content without value but {other}" 53 | 54 | @entry 55 | def post_content_(self, value, other="none"): 56 | return f"post content with {value} and {other}" 57 | 58 | # **param 59 | @entry 60 | def get_kwparam1(self, value=0): 61 | return f"get **param with only {value}" 62 | 63 | @entry 64 | def get_kwparam2(self, value=0, **kwargs: dict): 65 | return f"get **param with {value} and {list(kwargs.keys())}" 66 | 67 | @entry 68 | def put_kwparam2(self, value=0, **kwargs): 69 | return f"get **param with {value} and {list(kwargs.keys())}" 70 | 71 | @entry 72 | def post_kwparam2(self, value, **kwargs): 73 | return f"get **param with {value} and {list(kwargs.keys())}" 74 | 75 | # composed path 76 | @entry 77 | def get_extended_content(self): 78 | return "hello world" 79 | 80 | @entry 81 | def get_dir_content(self): 82 | return {'msg': "hello world"} 83 | 84 | 85 | class GlobalMS(TechMicroService): 86 | 87 | def token_authorizer(self, token): 88 | return True 89 | 90 | @entry 91 | def get_event_method(self): 92 | return request.aws_event['httpMethod'] 93 | -------------------------------------------------------------------------------- /coworks/biz/sensors.py: -------------------------------------------------------------------------------- 1 | from airflow.providers.amazon.aws.hooks.s3 import S3Hook 2 | from airflow.sensors.base import BaseSensorOperator 3 | from airflow.sensors.base import poke_mode_only 4 | from airflow.utils.decorators import apply_defaults 5 | 6 | from coworks.biz.operators import TechMicroServiceOperator 7 | 8 | 9 | @apply_defaults 10 | @poke_mode_only 11 | class TechMicroServiceSensor(BaseSensorOperator, TechMicroServiceOperator): 12 | """Sensor to call a TechMicroservice until call succeed. 13 | 14 | Same parameters as TechMicroServiceOperator except 'asynchronous' and 'raise_400_errors' 15 | """ 16 | 17 | def __init__(self, poke_interval: float = 30, **kwargs): 18 | super().__init__(asynchronous=False, raise_400_errors=False, poke_interval=poke_interval, **kwargs) 19 | 20 | def poke(self, context): 21 | """Set microservice context and execute it eache time.""" 22 | self.pre_execute(context) 23 | res = self._call_cws(context) 24 | return res.ok 25 | 26 | 27 | @apply_defaults 28 | class AsyncTechMicroServiceSensor(BaseSensorOperator): 29 | """Sensor to wait until an asynchronous TechMicroservice call is ended. 30 | 31 | :param cws_task_id: the tech microservice task_id awaited. 32 | :param aws_conn_id: AWS S3 connection. 33 | """ 34 | 35 | def __init__(self, *, cws_task_id: str, aws_conn_id: str = 'aws_s3', **kwargs): 36 | super().__init__(**kwargs) 37 | self.cws_task_id = cws_task_id 38 | self.aws_conn_id = aws_conn_id 39 | 40 | def poke(self, context): 41 | self.log.info(f"Waiting for {self.cws_task_id} result") 42 | 43 | # Get bucket information to poke 44 | ti = context['ti'] 45 | bucket_name = ti.xcom_pull(task_ids=self.cws_task_id, key='bucket') 46 | self.log.info(f"bucket_name: {bucket_name}") 47 | bucket_key = ti.xcom_pull(task_ids=self.cws_task_id, key='key') 48 | self.log.info(f"bucket_key: {bucket_key}") 49 | 50 | if not bucket_name or not bucket_key: 51 | return False 52 | 53 | # For dynamic tasks, the xcom are stored in a list 54 | self.log.info(f"Map index: {ti.map_index}") 55 | if ti.map_index >= 0: 56 | try: 57 | bucket_name = bucket_name[ti.map_index] 58 | bucket_key = bucket_key[ti.map_index] 59 | except IndexError: 60 | # May occurs when all the mapped results are not completed 61 | return False 62 | 63 | self.log.info(f"Poking for key : s3://{bucket_name}/{bucket_key}") 64 | hook = S3Hook(aws_conn_id=self.aws_conn_id) 65 | return hook.check_for_key(bucket_key, bucket_name) 66 | -------------------------------------------------------------------------------- /samples/website/tech/account.py: -------------------------------------------------------------------------------- 1 | import traceback 2 | from dataclasses import dataclass 3 | 4 | from flask import flash 5 | from flask import redirect 6 | from flask import session 7 | from flask import url_for 8 | from flask_login import UserMixin 9 | from flask_login import login_user 10 | from flask_login import logout_user 11 | from flask_wtf import FlaskForm 12 | from wtforms import PasswordField 13 | from wtforms import StringField 14 | from wtforms.validators import DataRequired 15 | from wtforms.validators import Email 16 | 17 | from coworks import Blueprint, request 18 | from coworks import entry 19 | from util import render_html_template 20 | 21 | 22 | class LoginForm(FlaskForm): 23 | """ Form for the login connection. 24 | """ 25 | email = StringField("Email (login)", default='', validators=[DataRequired(), Email()]) 26 | password = PasswordField("Mot de passe", default='', validators=[DataRequired()]) 27 | 28 | 29 | @dataclass 30 | class User(UserMixin): 31 | email: str = None 32 | 33 | # def is_authenticated(self): 34 | # return self.email 35 | 36 | def get_id(self): 37 | return self.email 38 | 39 | 40 | class AccountBlueprint(Blueprint): 41 | 42 | def __init__(self, login_manager): 43 | super().__init__(name="account") 44 | 45 | @login_manager.user_loader 46 | def load_user(user_id): 47 | user = session.get('user') 48 | return User(**user) if user else None 49 | 50 | @entry(no_auth=True) 51 | def get_login(self): 52 | """Get login form.""" 53 | form = LoginForm() 54 | return render_html_template("account/login.j2", form=form) 55 | 56 | @entry(no_auth=True) 57 | def post_login(self, email=None, password=None, next_url=None, remember=False, **kwargs): 58 | """Sign in.""" 59 | try: 60 | form = LoginForm() 61 | if not form.validate_on_submit(): 62 | flash('Connexion refusée.', 'error') 63 | return render_html_template('account/login.j2', form=form) 64 | 65 | try: 66 | user = User(email) 67 | login_user(user, remember=remember) 68 | session['user'] = user 69 | 70 | flash('Logged in successfully.') 71 | return redirect(url_for('get')) 72 | except (Exception,) as e: 73 | flash(e, 'error') 74 | except (Exception,) as exc_obj: 75 | tb_str = ''.join(traceback.format_exception(None, exc_obj, exc_obj.__traceback__)) 76 | print(tb_str) 77 | 78 | @entry(no_auth=True) 79 | def get_logout(self): 80 | logout_user() 81 | return redirect(url_for('get')) 82 | -------------------------------------------------------------------------------- /tests/coworks/tech/test_auth.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from coworks import entry 4 | from tests.coworks.blueprint.blueprint import BP 5 | from tests.coworks.ms import SimpleMS 6 | 7 | 8 | def get_event(token): 9 | return { 10 | 'type': 'TOKEN', 11 | 'methodArn': 'arn:aws:execute-api:eu-west-1:935392763270:htzd2rneg1/dev/GET/', 12 | 'authorizationToken': token 13 | } 14 | 15 | 16 | class AuthorizeAll(SimpleMS): 17 | 18 | def __init__(self, **kwargs): 19 | super().__init__(**kwargs) 20 | self.register_blueprint(BP(), url_prefix="/blueprint") 21 | 22 | def token_authorizer(self, token): 23 | return True 24 | 25 | @entry 26 | def get_product(self, ref): 27 | return ref 28 | 29 | 30 | class AuthorizeNothing(AuthorizeAll): 31 | 32 | def token_authorizer(self, token): 33 | return False 34 | 35 | 36 | class AuthorizedMS(AuthorizeAll): 37 | 38 | def token_authorizer(self, token): 39 | return token == 'token' 40 | 41 | 42 | class AuthorizeExceptionMS(AuthorizeAll): 43 | 44 | def token_authorizer(self, token): 45 | raise Exception() 46 | 47 | 48 | @pytest.mark.skip 49 | class TestClass: 50 | 51 | def test_authorize_all(self, empty_aws_context): 52 | app = AuthorizeAll() 53 | with app.app_context() as c: 54 | response = app(get_event('/token'), empty_aws_context) 55 | assert response['principalId'] == 'user' 56 | assert response['policyDocument']['Statement'][0]['Effect'] == 'Allow' 57 | 58 | def test_authorize_nothing(self, empty_aws_context): 59 | app = AuthorizeNothing() 60 | with app.app_context() as c: 61 | response = app(get_event('token'), empty_aws_context) 62 | assert response['principalId'] == 'user' 63 | assert response['policyDocument']['Statement'][0]['Effect'] == 'Deny' 64 | 65 | def test_authorized(self, empty_aws_context): 66 | app = AuthorizedMS() 67 | with app.app_context() as c: 68 | response = app(get_event('wrong'), empty_aws_context) 69 | assert response['principalId'] == 'user' 70 | assert response['policyDocument']['Statement'][0]['Effect'] == 'Deny' 71 | response = app(get_event('token'), empty_aws_context) 72 | assert response['principalId'] == 'user' 73 | assert response['policyDocument']['Statement'][0]['Effect'] == 'Allow' 74 | 75 | def test_authorize_exception(self, empty_aws_context): 76 | app = AuthorizeExceptionMS() 77 | with app.app_context() as c: 78 | response = app(get_event('token'), empty_aws_context) 79 | assert response['principalId'] == 'user' 80 | assert response['policyDocument']['Statement'][0]['Effect'] == 'Deny' 81 | -------------------------------------------------------------------------------- /samples/layers/tech/app.py: -------------------------------------------------------------------------------- 1 | import os 2 | import re 3 | from datetime import datetime 4 | 5 | import boto3 6 | from aws_xray_sdk.core import xray_recorder 7 | from flask import render_template 8 | from flask import url_for 9 | from werkzeug.exceptions import BadRequest 10 | 11 | from coworks import TechMicroService 12 | from coworks import entry 13 | from coworks import request 14 | from coworks.blueprint.admin_blueprint import Admin 15 | from coworks.extension.xray import XRay 16 | 17 | 18 | class CoworksLayersMicroService(TechMicroService): 19 | DOC_MD = """ 20 | ## Layers service 21 | 22 | Microservice to get all available CoWorks layers. 23 | """ 24 | 25 | def __init__(self, **kwargs): 26 | super().__init__(name="cws_layers", **kwargs) 27 | self.register_blueprint(Admin(), url_prefix='/admin') 28 | 29 | access_key = os.getenv("KEY_ID") 30 | secret_key = os.getenv("SECRET_KEY") 31 | if not access_key or not secret_key: 32 | raise BadRequest("Something wrong in your environment : no AWS credentials defined!") 33 | session = boto3.Session(access_key, secret_key, region_name='eu-west-1') 34 | self.lambda_client = session.client('lambda') 35 | 36 | @entry(no_auth=True) 37 | def get_home(self): 38 | """HTML page to get the layers.""" 39 | headers = {'Content-Type': 'text/html; charset=utf-8'} 40 | return render_template('home.j2', url=url_for('get')), 200, headers 41 | 42 | @entry(no_auth=True) 43 | def get(self, full: bool = False): 44 | """Layers in json or text format.""" 45 | version_pattern = re.compile(r"coworks-[_\d]*") 46 | res = self.lambda_clienlist_layers() 47 | layers = {x['LayerName']: x for x in filter(lambda x: version_pattern.fullmatch(x['LayerName']), res['Layers'])} 48 | if full: 49 | return layers 50 | versions = [*filter(lambda x: 'dev' not in x, map(lambda x: x[8:].replace('_', '.'), layers))] 51 | versions.sort(key=lambda s: list(map(int, s.split('.')))) 52 | last_version = layers[f"coworks-{versions[-1].replace('.', '_')}"] 53 | layers = map(lambda x: x['LayerArn'], layers.values()) 54 | if request.accept_mimetypes['text/html']: 55 | return to_html(layers, last_version) 56 | return to_json(layers, last_version) 57 | 58 | 59 | def to_json(layers, last_version): 60 | return { 61 | 'last': last_version['LayerArn'], 62 | 'layers': [*layers], 63 | } 64 | 65 | 66 | def to_html(layers, last_version): 67 | data = { 68 | 'layers': layers, 69 | 'now': datetime.now(), 70 | } 71 | return render_template('layers.j2', **data), 200, {'Content-Type': 'text/html; charset=utf-8'} 72 | 73 | 74 | app = CoworksLayersMicroService() 75 | XRay(app, xray_recorder) 76 | -------------------------------------------------------------------------------- /coworks/aws.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | import boto3 4 | 5 | 6 | class Boto3Mixin: 7 | 8 | def __init__(self, service, profile_name=None, env_var_access_key='aws_access_key_id', 9 | env_var_secret_key='aws_secret_access_key', env_var_region='aws_region', 10 | **kwargs): 11 | self.__session__ = self.__client__ = None 12 | self.__service = service 13 | self.__env_var_access_key = env_var_access_key 14 | self.__env_var_secret_key = env_var_secret_key 15 | self.__env_var_region = env_var_region 16 | 17 | self.__profile_name = profile_name 18 | 19 | @property 20 | def aws_access_key(self): 21 | value = os.getenv(self.__env_var_access_key) 22 | if not value: 23 | raise RuntimeError(f"{self.__env_var_access_key} not defined in environment") 24 | return value 25 | 26 | @property 27 | def aws_secret_access_key(self): 28 | value = os.getenv(self.__env_var_secret_key) 29 | if not value: 30 | raise RuntimeError(f"{self.__env_var_secret_key} not defined in environment") 31 | return value 32 | 33 | @property 34 | def region_name(self): 35 | if self.__session__: 36 | return self.__session__.region_name 37 | 38 | value = os.getenv(self.__env_var_region) 39 | if not value and self.aws_access_key and self.aws_secret_access_key: 40 | raise RuntimeError(f"{self.__env_var_region} not defined in environment") 41 | return value 42 | 43 | @property 44 | def client(self): 45 | if self.__client__ is None: 46 | self.__client__ = self.__session.client(self.__service) 47 | return self.__client__ 48 | 49 | @property 50 | def __session(self): 51 | if self.__session__ is None: 52 | if self.__profile_name is not None: 53 | try: 54 | self.__session__ = boto3.Session(profile_name=self.__profile_name) 55 | except Exception: 56 | raise RuntimeError(f"Cannot create session for profile {self.__profile_name}.") 57 | else: 58 | access_key = self.aws_access_key 59 | secret_key = self.aws_secret_access_key 60 | region_name = self.region_name 61 | try: 62 | self.__session__ = boto3.Session(access_key, secret_key, region_name=region_name) 63 | except Exception: 64 | raise RuntimeError( 65 | f"Cannot create session for key {access_key}, secret {secret_key} in {region_name}." 66 | ) 67 | return self.__session__ 68 | 69 | 70 | class AwsS3Session(Boto3Mixin): 71 | 72 | def __init__(self, **kwargs): 73 | super().__init__('s3', **kwargs) 74 | -------------------------------------------------------------------------------- /tests/cws/test_cmd.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | import pytest 5 | from click import NoSuchOption 6 | from click import UsageError 7 | 8 | from coworks.cws.client import client 9 | 10 | 11 | class TestClass: 12 | 13 | def test_wrong_cmd(self, example_dir): 14 | with pytest.raises(UsageError) as pytest_wrapped_e: 15 | client.main(['--project-dir', '.', 'cmd'], 'cws', standalone_mode=False) 16 | assert pytest_wrapped_e.type == UsageError 17 | assert pytest_wrapped_e.value.args[0] == "No such command 'cmd'." 18 | 19 | @mock.patch.dict(os.environ, {"FLASK_APP": "command:app"}) 20 | def test_cmd(self, example_dir, capsys): 21 | client.main(['--project-dir', '.', 'test'], 'cws', standalone_mode=False) 22 | captured = capsys.readouterr() 23 | assert captured.out == "test command with a=default/test command with b=value" 24 | 25 | def test_cmd_wrong_option(self, example_dir): 26 | with pytest.raises(NoSuchOption) as pytest_wrapped_e: 27 | client.main(['--project-dir', '.', 'test', '-t', 'wrong'], 'cws', standalone_mode=False) 28 | assert pytest_wrapped_e.type == NoSuchOption 29 | assert pytest_wrapped_e.value.args[0] == "No such option: -t" 30 | 31 | @mock.patch.dict(os.environ, {"FLASK_APP": "command:app"}) 32 | def test_cmd_right_option(self, example_dir, capsys): 33 | client.main(['--project-dir', '.', 'test', '-a', 'right'], 'cws', standalone_mode=False) 34 | captured = capsys.readouterr() 35 | assert captured.out == "test command with a=right/test command with b=value" 36 | 37 | @mock.patch.dict(os.environ, {"FLASK_APP": "command:app"}) 38 | def test_cmd_wrong_b_option(self, example_dir, capsys): 39 | with pytest.raises(NoSuchOption) as pytest_wrapped_e: 40 | client.main(['--project-dir', '.', 'test', '-b', 'right'], 'cws', standalone_mode=False) 41 | assert pytest_wrapped_e.type == NoSuchOption 42 | 43 | @mock.patch.dict(os.environ, {"FLASK_APP": "command:app"}) 44 | def test_cmd_right_b_option(self, example_dir, capsys): 45 | try: 46 | client.main(['--project-dir', '.', 'test', '--b', 'right'], 'cws', standalone_mode=False) 47 | finally: 48 | os.unsetenv("FLASK_RUN_FROM_CLI") 49 | captured = capsys.readouterr() 50 | assert captured.out == "test command with a=default/test command with b=right" 51 | 52 | @mock.patch.dict(os.environ, {"FLASK_APP": "command:app"}) 53 | def test_v1_cmd(self, example_dir, capsys): 54 | client.main(['--project-dir', '.', '-s', 'v1', 'test', '-a', 'right'], 'cws', 55 | standalone_mode=False) 56 | captured = capsys.readouterr() 57 | assert captured.out == "test command v1 with a=right/test command v1 with b=value1" 58 | -------------------------------------------------------------------------------- /samples/website/tech/website.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | import requests 4 | from flask import send_from_directory 5 | from flask_login import LoginManager 6 | 7 | from account import AccountBlueprint 8 | from coworks import TechMicroService 9 | from coworks import __version__ as coworks_version 10 | from coworks import entry 11 | from coworks.blueprint.admin_blueprint import Admin 12 | from util import render_html_template 13 | 14 | 15 | class WebsiteMicroService(TechMicroService): 16 | DOC_MD = """ 17 | ## Simple Flask website 18 | 19 | Microservice to implement a small website with session. 20 | """ 21 | 22 | def __init__(self, **kwargs): 23 | super().__init__(**kwargs) 24 | 25 | self.config['SECRET_KEY'] = os.getenv('SECRET_KEY') 26 | 27 | self.register_blueprint(Admin(), url_prefix='/admin') 28 | self.register_blueprint(AccountBlueprint(LoginManager(self)), url_prefix='/account') 29 | 30 | @self.context_processor 31 | def inject_context(): 32 | context = { 33 | "version": coworks_version 34 | } 35 | 36 | headers = {'Authorization': os.getenv('GITHUB_TOKEN')} 37 | resp = requests.get(os.getenv('COWORKS_GITHUB_URL'), headers=headers) 38 | if resp.ok: 39 | context["stargazers_count"] = resp.json()["stargazers_count"] 40 | 41 | return context 42 | 43 | @entry(no_auth=True) 44 | def get(self): 45 | """Entry for the home page.""" 46 | data = {} 47 | headers = {'Accept': "application/json"} 48 | resp = requests.get(os.getenv('COWORKS_LAYERS_URL'), headers=headers) 49 | if resp.ok: 50 | layers_resp = resp.json() 51 | data['last_layer'] = layers_resp["last"] 52 | zip_name = layers_resp["last"].split(':')[-1].replace('_', '.') 53 | data['last_layer_url'] = f"https://coworks-layer.s3.eu-west-1.amazonaws.com/{zip_name}.zip" 54 | 55 | return render_html_template("home.j2", **data) 56 | 57 | @entry(no_auth=True) 58 | def get_assets_css(self, filename): 59 | """Access for all css files.""" 60 | return send_from_directory('assets', f"css/{filename}", as_attachment=False, conditional=False) 61 | 62 | @entry(no_auth=True, binary_headers={'Content-Type': 'image/webp', 'Content-Disposition': 'inline'}) 63 | def get_assets_img(self, filename): 64 | """Access for all images.""" 65 | return send_from_directory('assets', f"img/{filename}", conditional=False) 66 | 67 | @entry(no_auth=True, binary_headers={'Content-Type': 'application/zip', 68 | 'Content-Disposition': 'attachment; filename=plugins.zip'}) 69 | def get_zip(self): 70 | """Access to the AirFlow plugins zip.""" 71 | return send_from_directory('assets', "plugins.zip", conditional=False) 72 | -------------------------------------------------------------------------------- /docs/installation.rst: -------------------------------------------------------------------------------- 1 | .. _installation: 2 | 3 | Installation 4 | ============ 5 | 6 | Python Version 7 | -------------- 8 | 9 | CoWorks supports only AWS Lambda python version >= 3.7. 10 | 11 | Install Coworks 12 | --------------- 13 | 14 | Use a virtual environment to install CoWorks. We recommend using ``pipenv``:: 15 | 16 | $ mkdir project && cd "$_" 17 | $ touch Pipfile 18 | $ pipenv install coworks 19 | $ pipenv shell 20 | 21 | You can then verify the installation by running:: 22 | 23 | (project) $ cws --version 24 | 25 | 26 | CoWorks is now ready for use. 27 | 28 | Create a project 29 | ---------------- 30 | 31 | To create a new project, enter:: 32 | 33 | (project) $ cws new 34 | New project created. 35 | 36 | You now have everything you need to create your first micro-service by following :ref:`tech_quickstart`. 37 | 38 | Other tools 39 | ----------- 40 | 41 | .. note:: Please see below sections on AWS and Terraform setup prior to deployment. 42 | 43 | Prior to use, please ensure that you also have the AWS CLI and Terrraform binary installed. You can check by running:: 44 | 45 | $ aws --version 46 | $ terraform --version 47 | 48 | 49 | AWS Credentials 50 | *************** 51 | 52 | *If you have previously configured your machine to run boto3 (the AWS SDK for Python) or the 53 | AWS CLI then you can skip this section.* 54 | 55 | Before you can deploy an application, make sure you have an 56 | `AWS account `_ 57 | and configured the 58 | `AWS credentials `_. 59 | 60 | Terraform 61 | ********* 62 | 63 | *If you have previously installed terraform then you can skip this section.* 64 | 65 | For deployment, for the command ``deploy`` we are using ``terraform``. We can use it locally or on 66 | online cloud plateform ``terraform.io``. 67 | 68 | Follow these `instructions `_ to install terraform. Check installation with:: 69 | 70 | (project) $ terraform --version 71 | 72 | Terraform can also be used `online `_. 73 | 74 | Apache Airflow 75 | ************** 76 | 77 | `Apache Airflow `_ may be installed 78 | `manually `_, or provided by 79 | `Astronomer `_ or 80 | `AWS `_. 81 | 82 | In this documentation we will describe how to use ``CoWorks`` with AWS MWAA. 83 | 84 | From the coworks source code, create the zip plugins file by :: 85 | 86 | $ make plugins.zip 87 | 88 | And upload it to S3, then attached this file to the MWAA environment used. Now the coworks operators and sensors are 89 | accessible defined in your environment. 90 | 91 | More information : 92 | `Installing custom plugins `_. 93 | 94 | 95 | -------------------------------------------------------------------------------- /tests/docs/test_first.py: -------------------------------------------------------------------------------- 1 | import multiprocessing 2 | import os 3 | import time 4 | from unittest import mock 5 | 6 | import requests 7 | 8 | from coworks.cws.client import CwsScriptInfo 9 | 10 | 11 | class TestClass: 12 | 13 | @mock.patch.dict(os.environ, {"FLASK_RUN_FROM_CLI": "false"}) 14 | def test_run_first_no_token(self, samples_docs_dir, unused_tcp_port): 15 | info = CwsScriptInfo(project_dir='tech') 16 | info.app_import_path = "first:app" 17 | app = info.load_app() 18 | server = multiprocessing.Process(target=run_server, args=(app, unused_tcp_port), daemon=True) 19 | server.start() 20 | counter = 1 21 | time.sleep(counter) 22 | while not server.is_alive() and counter < 10: 23 | time.sleep(counter) 24 | counter += 1 25 | response = requests.get(f'http://localhost:{unused_tcp_port}/') 26 | assert response.status_code == 403 27 | server.terminate() 28 | 29 | @mock.patch.dict(os.environ, {"FLASK_RUN_FROM_CLI": "false"}) 30 | def test_run_first_wrong_token(self, samples_docs_dir, unused_tcp_port): 31 | info = CwsScriptInfo(project_dir='tech') 32 | info.app_import_path = "first:app" 33 | app = info.load_app() 34 | server = multiprocessing.Process(target=run_server, args=(app, unused_tcp_port), daemon=True) 35 | server.start() 36 | counter = 1 37 | time.sleep(counter) 38 | while not server.is_alive() and counter < 10: 39 | time.sleep(counter) 40 | counter += 1 41 | response = requests.get(f'http://localhost:{unused_tcp_port}/', headers={'Authorization': 'wrong'}) 42 | assert response.status_code == 403 43 | 44 | @mock.patch.dict(os.environ, {"FLASK_RUN_FROM_CLI": "false"}) 45 | def test_run_first(self, samples_docs_dir, unused_tcp_port): 46 | info = CwsScriptInfo(project_dir='tech') 47 | info.app_import_path = "first:app" 48 | app = info.load_app() 49 | server = multiprocessing.Process(target=run_server, args=(app, unused_tcp_port), daemon=True) 50 | server.start() 51 | counter = 1 52 | time.sleep(counter) 53 | while not server.is_alive() and counter < 10: 54 | time.sleep(counter) 55 | counter += 1 56 | response = requests.get(f'http://localhost:{unused_tcp_port}/', headers={'Authorization': "token"}) 57 | assert response.text == "Stored value 0.\n" 58 | response = requests.post(f'http://localhost:{unused_tcp_port}/', json={'value': 1}, 59 | headers={'Authorization': "token"}) 60 | assert response.text == "Value stored (1).\n" 61 | response = requests.get(f'http://localhost:{unused_tcp_port}/', headers={'Authorization': "token"}) 62 | assert response.text == "Stored value 1.\n" 63 | server.terminate() 64 | 65 | 66 | def run_server(app, port): 67 | print(f"Server starting on port {port}") 68 | app.run(host='localhost', port=port, use_reloader=False, debug=False) 69 | -------------------------------------------------------------------------------- /docs/biz_quickstart.rst: -------------------------------------------------------------------------------- 1 | .. _biz_quickstart: 2 | 3 | Biz Quickstart 4 | ============== 5 | 6 | This page gives a quick and partial introduction to CoWorks Business Microservices. 7 | 8 | *Beware* : you must have understand first how to deploy technical microservice :doc:`tech`. 9 | 10 | Airflow concepts defined 11 | ------------------------ 12 | 13 | The `Apache Airflow `_ plateform is the core tool to implement the business model. 14 | As Flask for TechMicroService, the knowledge of Airflow is needed to fully understand the BizMicroService concept and 15 | benefits. 16 | 17 | Few concepts are needed to understand how ``BizMicroService`` works. 18 | 19 | A ``BizMicroService`` is defined as an airflow DAG. It allows to trigger it manually or get running information from 20 | it. Instead of ``TechMicroService`` which may be considered as stateless, ``BizMicroService`` are mainly stateful. 21 | At last, but certainly the more usefull and powerfull feature, the call to a microservice may be done **asynchronously**. 22 | 23 | To interact with DAG defined in airflow, two main operators have been defined : ``TechMicroServiceOperator`` and 24 | ``TechMicroServiceAsyncGroup``; more precisely the mast one is a group of tasks. 25 | The first operator allows to call a technical microservice. The second one is a group of tasks to call the technical 26 | microservice in an asynchronous way, waiting for the execution end and reading the result. 27 | 28 | In the asynchronous call, the called microservice stores automatically its result in 29 | a S3 file, and a ``Sensor`` on this S3 file is then created to allow resynchronisation. 30 | 31 | Start 32 | ----- 33 | 34 | To create your first simple business microservice, create a file ``simple.py`` with the following content:: 35 | 36 | from datetime import datetime 37 | from datetime import timedelta 38 | 39 | from coworks.biz import TechMicroServiceAsyncGroup 40 | from coworks.biz import TechMicroServiceOperator 41 | from coworks.biz import biz 42 | 43 | DEFAULT_ARGS = { 44 | 'retries': 1, 45 | 'retry_delay': timedelta(minutes=5), 46 | 'email': "gdoumenc@neorezo.io", 47 | 'email_on_failure': True, 48 | 'email_on_retry': False, 49 | } 50 | 51 | 52 | # noinspection PyTypeChecker,PyArgumentList 53 | @biz( 54 | default_args=DEFAULT_ARGS, 55 | tags=['air-et-sante', 'send', 'orders'], 56 | start_date=datetime(2023, 1, 1), 57 | schedule_interval='@daily', 58 | catchup=False, 59 | ) 60 | def first(): 61 | simple = TechMicroServiceAsyncGroup( 62 | 'simple', 63 | api_id="xxxx", 64 | stage="dev", 65 | token="token", 66 | method='GET', 67 | entry='/', 68 | ) 69 | 70 | @task() 71 | def print_result(): 72 | context = get_current_context() 73 | ti = context["ti"] 74 | result = ti.xcom_pull(task_ids='simple.read') 75 | print(result) 76 | 77 | simple >> print_result 78 | 79 | 80 | first = first() 81 | 82 | 83 | You microservice is called every day and traces are print in logs. 84 | -------------------------------------------------------------------------------- /coworks/cws/utils.py: -------------------------------------------------------------------------------- 1 | import platform 2 | import sys 3 | import typing as t 4 | from contextlib import contextmanager 5 | from importlib.metadata import version 6 | from threading import Thread 7 | from time import sleep 8 | 9 | import click 10 | 11 | 12 | def get_system_info(): 13 | flask_version = version("flask") 14 | 15 | flask_info = f"flask {flask_version}" 16 | python_info = f"python {sys.version_info[0]}.{sys.version_info[1]}.{sys.version_info[2]}" 17 | platform_system = platform.system().lower() 18 | platform_release = platform.release() 19 | platform_info = f"{platform_system} {platform_release}" 20 | return f"{flask_info}, {python_info}, {platform_info}" 21 | 22 | 23 | def show_stage_banner(stage: str = 'dev'): 24 | click.secho(f" * Stage: {stage}", fg="green") 25 | 26 | 27 | def show_terraform_banner(cloud: bool, refresh: bool): 28 | click.secho(f" * Using terraform backend {'cloud' if cloud else 's3'} (refresh={refresh})", fg="green") 29 | 30 | 31 | class ProgressBar: 32 | 33 | def __init__(self, bar): 34 | self.bar = bar 35 | self.stop = False 36 | self.spin_thread = None 37 | 38 | def echo(self, msg): 39 | swap = self.bar.format_progress_line 40 | self.bar.format_progress_line = lambda: msg 41 | self.bar.render_progress() 42 | click.echo() 43 | self.bar.format_progress_line = swap 44 | 45 | def update(self, msg: str | None = None): 46 | if msg: 47 | self.echo(msg) 48 | self.bar.update(1) 49 | 50 | def terminate(self, msg: str | None = None): 51 | self.stop = True 52 | if self.spin_thread: 53 | self.spin_thread.join() 54 | self.bar.finish() 55 | self.bar.render_progress() 56 | if msg: 57 | self.echo(msg) 58 | 59 | 60 | class DebugProgressBar: 61 | def update(self, msg: str): 62 | if msg: 63 | click.echo("==> " + msg) 64 | 65 | def echo(self, msg: str): 66 | if msg: 67 | click.echo("==> " + msg) 68 | 69 | 70 | @contextmanager # type: ignore[arg-type] 71 | def progressbar(length=200, *, label: str, threaded: bool = False) -> t.ContextManager[ProgressBar]: # type: ignore 72 | """Spinner progress bar. 73 | Creates it with a task label and updates it with progress messages using the 'update' function. 74 | """ 75 | if threaded: 76 | try: 77 | with click.progressbar(range(length - 1), label=label.ljust(40), show_eta=False) as bar: 78 | pb = ProgressBar(bar) 79 | 80 | def display_spinning_cursor(): 81 | while not pb.stop: 82 | sleep(1) 83 | if not pb.stop: 84 | pb.update() 85 | 86 | spin_thread = Thread(target=display_spinning_cursor) 87 | spin_thread.start() 88 | pb.spin_thread = spin_thread 89 | yield pb 90 | if not pb.stop: 91 | pb.terminate() 92 | except (Exception,) as e: 93 | click.echo(f"{type(e).__name__}: {str(e)}") 94 | finally: 95 | pb.stop = True 96 | else: 97 | yield DebugProgressBar() 98 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = ["pdm-backend"] 3 | build-backend = "pdm.backend" 4 | 5 | [project] 6 | name = "coworks" 7 | version = "2.0.8" 8 | description = "CoWorks is a unified compositional microservices framework using Flask/Airflow on AWS serverless technologies." 9 | readme = "README.rst" 10 | requires-python = ">= 3.12" 11 | license = { "file" = "LICENSE.txt" } 12 | authors = [ 13 | { name = "Guillaume Doumenc", email = "gdoumenc@fpr-coworks.com" } 14 | ] 15 | keywords = ["python3", "serverless", "microservice", "flask", "airflow", "aws-lambda", "aws"] 16 | classifiers = [ 17 | "License :: OSI Approved :: MIT License", 18 | "Development Status :: 5 - Production/Stable", 19 | "Intended Audience :: Developers", 20 | "Topic :: System :: Distributed Computing", 21 | "License :: OSI Approved :: MIT License", 22 | "Programming Language :: Python", 23 | "Programming Language :: Python :: 3.12" 24 | ] 25 | dependencies = [ 26 | "boto3>=1.19", 27 | "flask>=2.3.0", 28 | "jsonapi-pydantic>=0.2.3", 29 | "markdown>=3.3.6", 30 | "pydantic>=2.5.2", 31 | "pyyaml>=6.0", 32 | "python-dotenv>=0.21.0", 33 | "requests-toolbelt>=0.10", 34 | ] 35 | 36 | [project.urls] 37 | Documentation = "https://coworks.readthedocs.io" 38 | Source = "https://github.com/gdoumenc/coworks" 39 | 40 | [project.scripts] 41 | cws = "coworks.cws.client:client" 42 | 43 | [tool.pdm.dev-dependencies] 44 | dev = [ 45 | "aws_xray_sdk>=2.12", 46 | "boto3-stubs", 47 | "mypy>=1.5", 48 | "pytest>=7.4", 49 | "ruff>=0.0.284", 50 | "sphinx>=7.1", 51 | "sqlalchemy>=2.0", 52 | "types-Markdown>=3.6.0.20240316", 53 | "types-PyYAML>=6.0.12.20240311", 54 | "types-requests>=2.31.0.20240406", 55 | ] 56 | 57 | [tool.ruff.lint] 58 | ignore = ["F403"] 59 | 60 | [tool.mypy] 61 | python_version = "3.12" 62 | disable_error_code = ["method-assign"] 63 | exclude = [ 64 | "biz", # airflow using not compatible flask version 65 | "coworks/tech/directory.py", # will be removed 66 | "coworks/blueprint/mail_blueprint.py", # wip 67 | "coworks/blueprint/gsheets_blueprint.py", # wip 68 | ] 69 | check_untyped_defs = true 70 | 71 | [[tool.mypy.overrides]] 72 | module = ["requests_toolbelt.*", "aws_xray_sdk.*", "jsonapi_pydantic.*", "okta.*"] 73 | ignore_missing_imports = true 74 | 75 | [tool.pytest.ini_options] 76 | markers = [ 77 | "wip", # work in process. 78 | "not_on_github", # should not be performed on Github. 79 | ] 80 | testpaths = [ 81 | "tests" 82 | ] 83 | 84 | [tool.pdm.scripts] 85 | ruff = { cmd = "ruff check coworks", help = "Linting using Ruff" } 86 | mypy = { cmd = "mypy coworks", help = "Type checking using mypy" } 87 | pytest = { cmd = "pytest", help = "Unit testing using pytest" } 88 | github_pytest = { cmd = "pytest -m 'not not_on_github'", help = "Unit testing using pytest" } 89 | test = { composite = ["ruff", "mypy", "pytest"] } 90 | on_github = { composite = ["ruff", "mypy", "github_pytest"] } 91 | plugins = { shell = "mkdir -p dist; zip -r dist/plugins.zip coworks/operators.py coworks/sensors.py coworks/biz/*" } 92 | pre_build = { cmd = "pdm test" } 93 | pre_deploy = {cmd = "pdm build"} 94 | deploy.shell = "cloudsmith push python neorezo/neorezo dist/*.whl" 95 | 96 | [bdist_wheel] 97 | universal = 1 98 | -------------------------------------------------------------------------------- /coworks/cws/templates/terraform/lambda.j2: -------------------------------------------------------------------------------- 1 | {% if workspace != "common" %} 2 | # --------------------------------------------------------------------------------------------------------------------- 3 | # LAMBDA 4 | # --------------------------------------------------------------------------------------------------------------------- 5 | 6 | locals { 7 | {{ ms_name }}_lambda_name = "{{ ms_name }}-{{ stage }}" 8 | {% if environment_variable_files %} 9 | {{ ms_name }}_environment_variables = [ 10 | for envars_string in data.local_file.{{ ms_name }}_environment_variables_files: jsondecode(envars_string.content) 11 | ] 12 | {% endif %} 13 | } 14 | 15 | {% if layers %} 16 | data "aws_lambda_layer_version" "{{ ms_name }}" { 17 | provider = aws.{{ ms_name }} 18 | for_each = toset([{% for layer in layers %}"{{ layer }}",{% endfor %}]) 19 | layer_name = each.value 20 | } 21 | {% endif %} 22 | 23 | {% if environment_variable_files %} 24 | data "local_file" "{{ ms_name }}_environment_variables_files" { 25 | for_each = {for envar_file in {{ environment_variable_files | tojson }}: envar_file => envar_file} 26 | filename = each.value 27 | } 28 | {% endif %} 29 | 30 | resource "aws_lambda_function" "{{ ms_name }}" { 31 | provider = aws.{{ ms_name }} 32 | {% block aws_lambda_function %} 33 | function_name = local.{{ ms_name }}_lambda_name 34 | s3_bucket = "{{ bucket }}" 35 | s3_key = "{{ key }}" 36 | source_code_hash = "{{ source_code_hash }}" 37 | role = local.{{ ms_name }}_role_arn 38 | handler = "{{ app_import_path }}" 39 | description = "{{ description |truncate(256) | replace("\n", "\\n") }}" 40 | {% if layers %} 41 | layers = [for layer in data.aws_lambda_layer_version.{{ ms_name }} : layer.arn] 42 | {% endif %} 43 | runtime = "python{{ python }}" 44 | timeout = {{ timeout }} 45 | memory_size = {{ memory_size }} 46 | environment { 47 | variables = merge( 48 | {{ environment_variables | tojson }}, 49 | {"FLASK_DEBUG":"{{ '1' if stage == "dev" else '0' }}","CWS_STAGE":"{{ stage }}"}, 50 | {"CWS_DATETIME": "{{ now }}","CWS_LAMBDA": "{{ ms_name }}","CWS_BUCKET": "{{ bucket }}","CWS_KEY": "{{ key }}"} 51 | ) 52 | } 53 | tracing_config { 54 | mode = "Active" 55 | } 56 | {% block vpc_config %} 57 | vpc_config { 58 | security_group_ids = local.{{ ms_name }}_security_group_ids 59 | subnet_ids = local.{{ ms_name }}_subnet_ids 60 | } 61 | {% endblock %} 62 | depends_on = [ 63 | aws_cloudwatch_log_group.lambda_{{ ms_name }}, 64 | ] 65 | {% endblock %} 66 | } 67 | 68 | resource "aws_lambda_permission" "{{ ms_name }}_allow_apigateway" { 69 | provider = aws.{{ ms_name }} 70 | {% block aws_lambda_permission %} 71 | statement_id_prefix = "AllowExecutionFromAPIGateway" 72 | action = "lambda:InvokeFunction" 73 | function_name = local.{{ ms_name }}_lambda_name 74 | principal = "apigateway.amazonaws.com" 75 | source_arn = "${data.aws_api_gateway_rest_api.{{ ms_name }}.execution_arn}/*" 76 | depends_on = [aws_lambda_function.{{ ms_name }}] 77 | {% endblock %} 78 | } 79 | 80 | resource "aws_cloudwatch_log_group" "lambda_{{ ms_name }}" { 81 | provider = aws.{{ ms_name }} 82 | {% block aws_cloudwatch_log_group %} 83 | name = "/aws/lambda/${local.{{ ms_name }}_lambda_name}" 84 | retention_in_days = 7 85 | {% endblock %} 86 | } 87 | {% endif %} 88 | -------------------------------------------------------------------------------- /coworks/cws/templates/terraform/request_templates.j2: -------------------------------------------------------------------------------- 1 | {% macro event(body, is_base64_encoded) %} 2 | #set($allParams = $input.params()) 3 | #set($headers = $allParams.get("header")) 4 | #set($path = $allParams.get("path")) 5 | #set($queries = $allParams.get("querystring")) 6 | { 7 | "type" : "LAMBDA", 8 | "resource" : "$context.resourcePath", 9 | "path" : "$context.resourcePath", 10 | "httpMethod": "$context.httpMethod", 11 | "headers": { 12 | #foreach($key in $headers.keySet()) 13 | "$key.toLowerCase()" : "$util.escapeJavaScript($headers.get($key))"#if($foreach.hasNext),#end 14 | #end 15 | }, 16 | "multiValueHeaders": {}, 17 | {{ body }}, 18 | "queryStringParameters": null, 19 | "multiValueQueryStringParameters": { 20 | #foreach($key in $method.request.multivaluequerystring.keySet()) 21 | "$key" : [ 22 | #foreach($val in $method.request.multivaluequerystring.get($key)) 23 | "$util.escapeJavaScript($val)"#if($foreach.hasNext),#end 24 | #end 25 | ]#if($foreach.hasNext),#end 26 | #end 27 | }, 28 | "entryPathParameters": { 29 | #foreach($key in $path.keySet()) 30 | "$key": "$util.escapeJavaScript($path.get($key))" 31 | #if($foreach.hasNext),#end 32 | #end 33 | }, 34 | "stageVariables": null, 35 | {{ is_base64_encoded }}, 36 | "requestContext": { 37 | "httpMethod": "$context.httpMethod", 38 | "resourceId": "$context.resourceId", 39 | "entryPath": "$context.resourcePath", 40 | "extendedRequestId": "$context.extendedRequestId", 41 | "requestTime": "$context.requestTime", 42 | "path": "$context.path", 43 | "accountId": "$context.accountId", 44 | "protocol": "$context.protocol", 45 | "stage": "$context.stage", 46 | "domainPrefix": "$context.domainPrefix", 47 | "requestTimeEpoch": $context.requestTimeEpoch, 48 | "requestId": "$context.requestId", 49 | "domainName": "$context.domainName", 50 | "apiId": "$context.apiId" 51 | }, 52 | "params" : { 53 | #foreach($type in $allParams.keySet()) 54 | #set($params = $allParams.get($type)) 55 | "$type" : { 56 | #foreach($paramName in $params.keySet()) 57 | "$paramName" : "$util.escapeJavaScript($params.get($paramName))" 58 | #if($foreach.hasNext),#end 59 | #end 60 | } 61 | #if($foreach.hasNext),#end 62 | #end 63 | }, 64 | "context" : { 65 | #foreach($key in $context.keySet()) 66 | "$key" : "$util.escapeJavaScript($context.get($key))" 67 | #if($foreach.hasNext),#end 68 | #end 69 | } 70 | } 71 | {%- endmacro %} 72 | "application/json" = <<-EOT 73 | {{ event("\"body\": $input.json('$')", "\"isBase64Encoded\": false") }} 74 | EOT 75 | "text/html" = <<-EOT 76 | {{ event("\"body\": $input.json('$')", "\"isBase64Encoded\": false") }} 77 | EOT 78 | "application/x-www-form-urlencoded" = <<-EOT 79 | {{ event("\"body\": \"$input.body\"", "\"isBase64Encoded\": false") }} 80 | EOT 81 | "multipart/form-data" = <<-EOT 82 | {{ event("\"body\": \"$input.body\"", "\"isBase64Encoded\": true") }} 83 | EOT 84 | 85 | -------------------------------------------------------------------------------- /docs/code_of_conduct.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | In the interest of fostering an open and welcoming environment, we as 6 | contributors and maintainers pledge to making participation in our project, 7 | and our community a harassment-free experience for everyone, regardless of age, body 8 | size, disability, ethnicity, sex characteristics, gender identity and expression, 9 | level of experience, education, socio-economic status, nationality, personal 10 | appearance, race, religion, or sexual identity and orientation. 11 | 12 | ## Our Standards 13 | 14 | Examples of behavior that contributes to creating a positive environment 15 | include: 16 | 17 | * Using welcoming and inclusive language 18 | * Being respectful of differing viewpoints and experiences 19 | * Gracefully accepting constructive criticism 20 | * Focusing on what is best for the community 21 | * Showing empathy towards other community members 22 | 23 | Examples of unacceptable behavior by participants include: 24 | 25 | * The use of sexualized language or imagery and unwelcome sexual attention or 26 | advances 27 | * Trolling, insulting/derogatory comments, and personal or political attacks 28 | * Public or private harassment 29 | * Publishing others' private information, such as a physical or electronic 30 | address, without explicit permission 31 | * Other conduct which could reasonably be considered inappropriate in a 32 | professional setting 33 | 34 | ## Our Responsibilities 35 | 36 | Project maintainers are responsible for clarifying the standards of acceptable 37 | behavior and are expected to take appropriate and fair corrective action in 38 | response to any instances of unacceptable behavior. 39 | 40 | Project maintainers have the right and responsibility to remove, edit, or 41 | reject comments, commits, code, wiki edits, issues, and other contributions 42 | that are not aligned to this Code of Conduct, or to ban temporarily or 43 | permanently any contributor for other behaviors that they deem inappropriate, 44 | threatening, offensive, or harmful. 45 | 46 | ## Scope 47 | 48 | This Code of Conduct applies both within project spaces and in public spaces 49 | when an individual is representing the project or its community. Examples of 50 | representing a project or community include using an official project e-mail 51 | address, posting via an official social media account, or acting as an appointed 52 | representative at an online or offline event. Representation of a project may be 53 | further defined and clarified by project maintainers. 54 | 55 | ## Enforcement 56 | 57 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 58 | reported by contacting the project team at gdoumenc@fpr-coworks.com. All 59 | complaints will be reviewed and investigated and will result in a response that 60 | is deemed necessary and appropriate to the circumstances. The project team is 61 | obligated to maintain confidentiality with regard to the reporter of an incident. 62 | Further details of specific enforcement policies may be posted separately. 63 | 64 | Project maintainers who do not follow or enforce the Code of Conduct in good 65 | faith may face temporary or permanent repercussions as determined by other 66 | members of the project's leadership. 67 | 68 | ## Attribution 69 | 70 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, 71 | available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html 72 | 73 | [homepage]: https://www.contributor-covenant.org 74 | 75 | For answers to common questions about this code of conduct, see 76 | https://www.contributor-covenant.org/faq 77 | -------------------------------------------------------------------------------- /coworks/blueprint/okta_blueprint.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | from aws_xray_sdk.core import xray_recorder 4 | from flask import request 5 | from okta.client import Client 6 | from okta.okta_object import OktaObject 7 | from werkzeug.exceptions import InternalServerError 8 | 9 | from coworks import Blueprint 10 | from coworks import entry 11 | from coworks.extension.xray import XRay 12 | 13 | 14 | class OktaClient(Client): 15 | """Okta client extended to allow next call event on new client defined.""" 16 | 17 | @XRay.capture(xray_recorder) 18 | async def next(self, next): 19 | req, error = await self._request_executor.create_request("GET", next, {}, {}) 20 | if error: 21 | return None, error 22 | response, error = await self._request_executor.execute(req) 23 | try: 24 | result = [] 25 | for item in response.get_body(): 26 | result.append(OktaDict(self.form_response_body(item))) 27 | except Exception as error: 28 | return None, error 29 | return result, response, None 30 | 31 | 32 | class OktaResponse: 33 | """Class to manipulate results from okta client. 34 | value: value returned as a list of dict. 35 | err: Not None if error. 36 | next: Next url to be called if wanted more results. 37 | 38 | OktaRespons are used as global variables from asynchronous functions. 39 | 40 | To set the value from an asynchronous function: 41 | resp.set(await self.okta_client.function(query_parameters)) 42 | 43 | The property response must be used as microservice returned value: 44 | return resp.response 45 | 46 | To combine results in microservice: 47 | return OktaResponse.combine({'user': resp_user, 'groups': resp_groups}) 48 | """ 49 | 50 | def __init__(self): 51 | self.api_resp = self.next_url = self.error = None 52 | 53 | @XRay.capture(xray_recorder) 54 | def set(self, await_result, fields=None): 55 | """Set the values from the result. Keep only specific fieds if defined.""" 56 | 57 | if len(await_result) == 3: 58 | _, self.api_resp, self.error = await_result 59 | else: 60 | self.api_resp, self.error = await_result 61 | 62 | # noinspection PyProtectedMember 63 | self.next_url = self.api_resp._next if self.api_resp else None 64 | 65 | @property 66 | def body(self): 67 | """Get OKTA body response.""" 68 | return self.api_resp.get_body() if self.api_resp else None 69 | 70 | @property 71 | def response(self): 72 | """Cast the Okta response as microservice response.""" 73 | if self.error: 74 | return self.error.message, self.error.status 75 | return {'value': self.body, 'next': self.next_url} 76 | 77 | 78 | class Okta(Blueprint): 79 | 80 | def __init__(self, name: str = 'okta', 81 | env_url_var_name: str = '', env_token_var_name: str = '', 82 | env_var_prefix: str = "OKTA", **kwargs): 83 | super().__init__(name=name, **kwargs) 84 | if env_var_prefix: 85 | self.env_url_var_name = f"{env_var_prefix}_URL" 86 | self.env_token_var_name = f"{env_var_prefix}_TOKEN" 87 | else: 88 | self.env_url_var_name = env_url_var_name 89 | self.env_token_var_name = env_token_var_name 90 | 91 | self.org_url = os.getenv(self.env_url_var_name) 92 | if not self.org_url: 93 | raise InternalServerError(f"Environment var {self.env_url_var_name} undefined.") 94 | config = { 95 | 'orgUrl': self.org_url, 96 | 'token': os.getenv(self.env_token_var_name) 97 | } 98 | self.okta_client = OktaClient(config) 99 | 100 | @entry 101 | def get_event_verify(self): 102 | """Entry for Okta webhook verification.""" 103 | test_value = request.headers.get('x-okta-verification-challenge') 104 | return {"verification": test_value} 105 | 106 | 107 | class OktaDict(OktaObject): 108 | """Simplified generic okta object.""" 109 | 110 | def __init__(self, values): 111 | super().__init__() 112 | self.values = values 113 | 114 | def as_dict(self): 115 | return {k: v for k, v in self.values.items() if k != "links"} 116 | -------------------------------------------------------------------------------- /tests/coworks/blueprint/test_admin.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | from flask import json 5 | 6 | from coworks import Blueprint 7 | from coworks import TechMicroService 8 | from coworks import entry 9 | from coworks.blueprint.admin_blueprint import Admin 10 | 11 | 12 | class DocumentedMS(TechMicroService): 13 | 14 | def __init__(self, **kwargs): 15 | super().__init__(**kwargs) 16 | self.register_blueprint(Admin(), url_prefix="/admin") 17 | 18 | def token_authorizer(self, token): 19 | return True 20 | 21 | @entry 22 | def get(self): 23 | """Root access.""" 24 | return "get" 25 | 26 | @entry 27 | def post(self): 28 | """Root access.""" 29 | return "post" 30 | 31 | @entry 32 | def post_content(self, value, other="none"): 33 | """Add content.""" 34 | return f"post_content {value}{other}" 35 | 36 | @entry 37 | def post_contentannotated(self, value: int, other: str = "none"): 38 | """Add content.""" 39 | return f"post_content {value}{other}" 40 | 41 | @entry(no_auth=True) 42 | def get_list(self, values: [int]): 43 | return "ok" 44 | 45 | 46 | class HiddenBlueprint(Blueprint): 47 | 48 | @entry 49 | def get(self): 50 | """Test not in routes.""" 51 | return "ok" 52 | 53 | 54 | @mock.patch.dict(os.environ, {"TOKEN": "token"}) 55 | class TestClass: 56 | 57 | def test_routes(self): 58 | app = DocumentedMS() 59 | with app.test_request_context(): 60 | assert '/' in app.routes 61 | assert '/content/{value}' in app.routes 62 | assert '/contentannotated/{value}' in app.routes 63 | assert '/list/{values}' in app.routes 64 | 65 | def test_documentation(self): 66 | app = DocumentedMS() 67 | with app.test_client() as c: 68 | response = c.get('/admin/route?blueprint=test', headers={'Authorization': 'token'}) 69 | assert response.status_code == 404 70 | 71 | response = c.get('/admin/route?blueprint=__all__', headers={'Authorization': 'token'}) 72 | assert response.status_code == 200 73 | routes = json.loads(response.get_data(as_text=True)) 74 | assert routes["/"]['GET'] == { 75 | "doc": "Root access.", "signature": "()", "endpoint": "get", 76 | 'binary_headers': None, 'no_auth': False 77 | } 78 | assert routes["/"]['POST'] == { 79 | "doc": "Root access.", "signature": "()", "endpoint": "post", 80 | 'binary_headers': None, 'no_auth': False 81 | } 82 | assert routes["/content/"]['POST'] == { 83 | "doc": "Add content.", "signature": "(value, other=none)", "endpoint": "post_content", 84 | 'binary_headers': None, 'no_auth': False 85 | } 86 | assert routes["/contentannotated/"]['POST'] == { 87 | "doc": "Add content.", "signature": "(value:, other:=none)", 88 | 'endpoint': "post_contentannotated", 89 | 'binary_headers': None, 'no_auth': False 90 | } 91 | assert routes["/admin/route"]['GET']['signature'] == "(prefix=None, blueprint=None)" 92 | assert routes["/admin/route"]['GET']['endpoint'] == "admin.get_route" 93 | assert routes["/list/"]['GET'] == { 94 | "signature": "(values:[])", 'endpoint': 'get_list', 95 | 'binary_headers': None, 'no_auth': True 96 | } 97 | 98 | response = c.get('/admin/route', headers={'Authorization': 'token'}) 99 | assert response.status_code == 200 100 | routes = json.loads(response.get_data(as_text=True)) 101 | assert "/admin/routes" not in routes 102 | 103 | def test_documentation_with_hidden_blueprints(self): 104 | app = DocumentedMS() 105 | app.register_blueprint(HiddenBlueprint(), url_prefix="/hidden", hide_routes=True) 106 | with app.test_client() as c: 107 | response = c.get('/admin/route', headers={'Authorization': 'token'}) 108 | assert response.status_code == 200 109 | routes = json.loads(response.get_data(as_text=True)) 110 | assert '/hidden' not in routes 111 | -------------------------------------------------------------------------------- /tests/cws/test_environment.py: -------------------------------------------------------------------------------- 1 | import multiprocessing 2 | import os 3 | import time 4 | from unittest import mock 5 | 6 | import pytest 7 | import requests 8 | from flask.cli import ScriptInfo 9 | 10 | from coworks.cws.client import client 11 | from coworks.utils import load_dotenv 12 | from tests.coworks.event import get_event 13 | from tests.cws.src.app import EnvTechMS 14 | 15 | 16 | @pytest.mark.not_on_github 17 | class TestClass: 18 | 19 | @mock.patch.dict(os.environ, {"FLASK_ENV_FILE": ".env.no", "FLASK_RUN_FROM_CLI": "true"}) 20 | def test_no_env(self, example_dir, empty_aws_context): 21 | app = EnvTechMS() 22 | event = get_event('/', 'get') 23 | with app.cws_client(event, empty_aws_context) as c: 24 | response = c.get('/', headers={'Authorization': 'token'}) 25 | 26 | @mock.patch.dict(os.environ, {"FLASK_ENV_FILE": '.env.dev', "FLASK_RUN_FROM_CLI": "true"}) 27 | def test_run_dev_env(self, example_dir, unused_tcp_port): 28 | server = multiprocessing.Process(target=run_server, args=(example_dir, unused_tcp_port), daemon=True) 29 | server.start() 30 | counter = 1 31 | time.sleep(counter) 32 | while not server.is_alive() and counter < 3: 33 | time.sleep(counter) 34 | counter += 1 35 | response = requests.get(f'http://localhost:{unused_tcp_port}/env', headers={'Authorization': "token"}) 36 | assert response.text == "Value of environment variable test is : test dev environment variable." 37 | server.terminate() 38 | 39 | @mock.patch.dict(os.environ, {"FLASK_ENV_FILE": '.env.v1', "FLASK_RUN_FROM_CLI": "true"}) 40 | def test_run_prod_env(self, example_dir, unused_tcp_port): 41 | server = multiprocessing.Process(target=run_server, args=(example_dir, unused_tcp_port), daemon=True) 42 | server.start() 43 | counter = 1 44 | time.sleep(counter) 45 | while not server.is_alive() and counter < 3: 46 | time.sleep(counter) 47 | counter += 1 48 | response = requests.get(f'http://localhost:{unused_tcp_port}/env', headers={'Authorization': "token"}) 49 | assert response.text == "Value of environment variable test is : test prod environment variable." 50 | server.terminate() 51 | 52 | def test_run_dev_stage(self, example_dir, unused_tcp_port): 53 | server = multiprocessing.Process(target=run_server_with_stage, 54 | args=(example_dir, unused_tcp_port, "dev"), 55 | daemon=True) 56 | server.start() 57 | counter = 1 58 | time.sleep(counter) 59 | while not server.is_alive() and counter < 3: 60 | time.sleep(counter) 61 | counter += 1 62 | response = requests.get(f'http://localhost:{unused_tcp_port}/env', headers={'Authorization': "token"}) 63 | assert response.text == "Value of environment variable test is : test dev environment variable." 64 | server.terminate() 65 | 66 | def test_run_prod_stage(self, example_dir, unused_tcp_port): 67 | server = multiprocessing.Process(target=run_server_with_stage, 68 | args=(example_dir, unused_tcp_port, "v1"), 69 | daemon=True) 70 | server.start() 71 | counter = 1 72 | time.sleep(counter) 73 | while not server.is_alive() and counter < 3: 74 | time.sleep(counter) 75 | counter += 1 76 | response = requests.get(f'http://localhost:{unused_tcp_port}/env', headers={'Authorization': "token"}) 77 | assert response.text == "Value of environment variable test is : test prod environment variable." 78 | server.terminate() 79 | 80 | def test_load_env(self): 81 | variables = load_dotenv("dev") 82 | assert True 83 | 84 | 85 | def run_server(project_dir, port): 86 | obj = ScriptInfo(create_app=lambda: EnvTechMS(), set_debug_flag=False) 87 | client.main(['--project-dir', '.', 'run', '--port', port], 'cws', obj=obj, standalone_mode=False) 88 | 89 | 90 | def run_server_with_stage(project_dir, port, stage): 91 | @mock.patch.dict(os.environ, {"FLASK_ENV_FILE": f".env.{stage}", "FLASK_RUN_FROM_CLI": "true"}) 92 | def run(): 93 | obj = ScriptInfo(create_app=lambda: EnvTechMS(), set_debug_flag=False) 94 | client.main(['-p', '.', 'run', '--port', port], 'cws', obj=obj, standalone_mode=False) 95 | 96 | run() 97 | -------------------------------------------------------------------------------- /tests/coworks/blueprint/test_mail.py: -------------------------------------------------------------------------------- 1 | import os 2 | import smtplib 3 | from email import message 4 | from io import BytesIO 5 | from unittest import mock 6 | 7 | import pytest 8 | 9 | from coworks import TechMicroService 10 | from coworks.blueprint.mail_blueprint import Mail 11 | 12 | smtp_mock = mock.MagicMock() 13 | smtp_mock.return_value.__enter__.return_value.login = login_mock = mock.Mock() 14 | smtp_mock.return_value.__enter__.return_value.send_message = send_mock = mock.Mock() 15 | 16 | email_mock = mock.MagicMock() 17 | email_mock.return_value.add_attachment = add_mock = mock.Mock() 18 | 19 | 20 | class MailMS(TechMicroService): 21 | 22 | def __init__(self, **mail_names): 23 | super().__init__('mail') 24 | self.register_blueprint(Mail(**mail_names)) 25 | 26 | def _check_token(self): 27 | return True 28 | 29 | 30 | class TestClass: 31 | 32 | def test_wrong_init(self): 33 | with pytest.raises(RuntimeError) as pytest_wrapped_e: 34 | app = MailMS() 35 | with app.test_client() as c: 36 | response = c.post('/send') 37 | assert pytest_wrapped_e.type == RuntimeError 38 | 39 | @mock.patch.dict(os.environ, { 40 | "SMTP_SERVER": "mail.test.com:587", 41 | "SMTP_LOGIN": "myself@test.com", 42 | "SMTP_PASSWD": "passwd" 43 | }) 44 | def test_wrong_params(self): 45 | mail_names = { 46 | 'env_server_var_name': 'SMTP_SERVER', 47 | 'env_login_var_name': 'SMTP_LOGIN', 48 | 'env_passwd_var_name': 'SMTP_PASSWD', 49 | } 50 | app = MailMS(**mail_names) 51 | with app.test_client() as c: 52 | data = { 53 | 'subject': "Test", 54 | } 55 | response = c.post('/send', data=data) 56 | assert response.status_code == 400 57 | assert response.get_data(as_text=True) == "From address not defined (from_addr:str)" 58 | 59 | @mock.patch.dict(os.environ, { 60 | "SMTP_SERVER": "mail.test.com:587", 61 | "SMTP_LOGIN": "myself@test.com", 62 | "SMTP_PASSWD": "passwd" 63 | }) 64 | def test_wrong_params(self, auth_headers): 65 | app = MailMS(env_var_prefix='SMTP') 66 | with app.test_client() as c: 67 | data = { 68 | 'subject': "Test", 69 | } 70 | response = c.post('/send', data=data, headers=auth_headers) 71 | assert response.status_code == 400 72 | assert "From address not defined (from_addr:str)" in response.get_data(as_text=True) 73 | 74 | @mock.patch.dict(os.environ, { 75 | "SMTP_SERVER": "mail.test.com:587", 76 | "SMTP_LOGIN": "myself@test.com", 77 | "SMTP_PASSWD": "passwd" 78 | }) 79 | @mock.patch.object(smtplib, 'SMTP', smtp_mock) 80 | def test_send_text(self, auth_headers): 81 | app = MailMS(env_var_prefix='SMTP') 82 | with app.test_client() as c: 83 | data = { 84 | 'subject': "Test", 85 | 'from_addr': "from@test.fr", 86 | 'to_addrs': "to@test.fr", 87 | } 88 | response = c.post('/send', data=data, headers=auth_headers) 89 | assert response.status_code == 200 90 | assert response.get_data(as_text=True) == "Mail sent to to@test.fr" 91 | login_mock.assert_called_with('myself@test.com', 'passwd') 92 | send_mock.assert_called_once() 93 | 94 | @mock.patch.dict(os.environ, { 95 | "SMTP_SERVER": "mail.test.com:587", 96 | "SMTP_LOGIN": "myself@test.com", 97 | "SMTP_PASSWD": "passwd" 98 | }) 99 | @mock.patch.object(smtplib, 'SMTP', smtp_mock) 100 | @mock.patch.object(message, 'EmailMessage', email_mock) 101 | def test_send_attachment(self, auth_headers): 102 | app = MailMS(env_var_prefix='SMTP') 103 | with app.test_client() as c: 104 | file = BytesIO(b"hello {{ world_name }}") 105 | data = { 106 | 'subject': "Test", 107 | 'from_addr': "from@test.fr", 108 | 'to_addrs': "to@test.fr", 109 | 'attachments': [(file, 'file.txt')] 110 | } 111 | response = c.post('/send', data=data, headers=auth_headers) 112 | assert response.status_code == 200 113 | login_mock.assert_called_with('myself@test.com', 'passwd') 114 | add_mock.assert_called_once() 115 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | .. image:: https://github.com/gdoumenc/coworks/raw/dev/docs/img/coworks.png 2 | :height: 80px 3 | :alt: CoWorks Logo 4 | 5 | |Maintenance| |Build Status| |Documentation Status| |Coverage| |Python versions| |Licence| 6 | 7 | .. |Maintenance| image:: https://img.shields.io/badge/Maintained%3F-yes-green.svg?style=plastic 8 | :alt: Maintenance 9 | .. |Build Status| image:: https://img.shields.io/travis/com/gdoumenc/coworks?style=plastic 10 | :alt: Build Status 11 | .. |Documentation Status| image:: https://readthedocs.org/projects/coworks/badge/?version=master&style=plastic 12 | :alt: Documentation Status 13 | .. |Coverage| image:: https://img.shields.io/codecov/c/github/gdoumenc/coworks?style=plastic 14 | :alt: Codecov 15 | .. |Python versions| image:: https://img.shields.io/pypi/pyversions/coworks?style=plastic 16 | :alt: Python Versions 17 | .. |Licence| image:: https://img.shields.io/github/license/gdoumenc/coworks?style=plastic 18 | :alt: Licence 19 | 20 | CoWorks is a unified serverless microservices framework based on AWS technologies 21 | (`API Gateway `_, `AWS Lambda `_), 22 | the Flask framework (`Flask `_/`Click `_) and 23 | the `Airflow `_ platform. 24 | 25 | The aim of this project is to offer a very simplified experience of microservices. For such purpose, we divided the 26 | CoWorks framework in two levels: 27 | 28 | **Small technical microservice** 29 | 30 | ``TechMicroservice`` are each composed of simple python `Flask `_ application and deployed as a serverless Lambda. Each ``TechMicroService`` is an ``atomic component`` or `atomic microservice `_. These microservices may be called synchronously or asynchronously. 31 | 32 | **Functional business service** 33 | 34 | ``biz`` are `composite business services `_, which are `Airflow `_ DAGs providing orchestration of atomic microservices or components (aka: ``TechMicroService``). 35 | 36 | To get started with CoWorks, first follow the `Installation Guide `_. Then you can get a quickstart on `TechMicroService Quickstart `_. 37 | Once familiar with ``TechMicroService``, you can continue with `BizMicroService Quickstart `_. 38 | 39 | **Data model** 40 | 41 | The data model shared between those services may be structured with ``pydantic`` and using the JSON:API specification. 42 | You can install this data protocol for CoWorks with: pip install coworks[jsonapi-sqlalchemy]. 43 | 44 | Documentation 45 | ------------- 46 | 47 | * Setup and installation: `Installation `_ 48 | * Complete reference guide: `Documentation `_. 49 | * Samples: 50 | * layers : Get available CoWorks lambda layers: `CoWorks layers `_. 51 | * website : Very simple website done as a simple microservice: `Website `_. 52 | * Read `FAQ `_ for other information. 53 | 54 | Contributing 55 | ------------ 56 | 57 | We work hard to provide a high-quality and useful framework, and we greatly value 58 | feedback and contributions from our community. Whether it's a new feature, 59 | correction, or additional documentation, we welcome your pull requests. Please 60 | submit any `issues `__ 61 | or `pull requests `__ through GitHub. 62 | 63 | Related Projects 64 | ---------------- 65 | 66 | * `Flask `_ - Lightweight WSGI web application framework (`Donate to Pallets `_). 67 | * `Airflow `_ - A platform to programmatically author, schedule, and monitor workflows. 68 | * `Terraform `_ - Infrastructure configuration management tool. 69 | * `Pydantic `_ - Data validation using Python type hints. 70 | 71 | Some ideas guiding this project were found in : 72 | 73 | * `Flask-Classy `_ 74 | * `PyDANJA `_ 75 | -------------------------------------------------------------------------------- /tests/coworks/event.py: -------------------------------------------------------------------------------- 1 | from flask import json 2 | 3 | 4 | def get_event(entry_path, method, entry_path_parameters=None, params=None, body=None, headers=None): 5 | headers = headers or { 6 | 'accept': '*/*', 7 | 'authorization': 'token', 8 | 'content-type': 'application/json' 9 | } 10 | return { 11 | 'type': 'LAMBDA', 12 | 'resource': entry_path, 13 | 'path': entry_path, 14 | 'httpMethod': method.upper(), 15 | 'headers': { 16 | 'accept-encoding': 'gzip, deflate, br', 17 | 'cache-control': 'max-age=0', 18 | 'cookie': 'session=70b58773-af3e-4153-a5ab-5356481ea87e', 19 | 'host': 'zb53tzphr8.execute-api.eu-west-1.amazonaws.com', 20 | 'origin': 'https://neorezo.io', 21 | 'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"', 22 | 'sec-ch-ua-mobile': '?0', 23 | 'sec-ch-ua-platform': '"Linux"', 24 | 'sec-fetch-dest': 'document', 25 | 'sec-fetch-mode': 'navigate', 26 | 'sec-fetch-site': 'same-origin', 27 | 'sec-fetch-user': '?1', 28 | 'upgrade-insecure-requests': '1', 29 | 'user-agent': 'Amazon CloudFront', 30 | 'via': '2.0 4e4ca876a59e9f2e22ec751bbab5f282.cloudfront.net (CloudFront)', 31 | 'x-amz-cf-id': '0gvkeT2_kuutjZHh075zWXA0pfqm-afERp4UltS3XBAguLOI4tdckw==', 32 | 'x-amzn-trace-id': 'Root=1-620f51ee-3f8b28746271cbea29b0084e', 33 | 'x-forwarded-for': '91.170.42.203, 130.176.185.118', 34 | 'x-forwarded-port': '443', 35 | 'x-forwarded-proto': 'https', 36 | **headers 37 | }, 38 | 'multiValueHeaders': {}, 39 | 'body': json.dumps(body or {}), 40 | 'queryStringParameters': {}, 41 | 'multiValueQueryStringParameters': params or {}, 42 | 'pathParameters': {}, 43 | 'stageVariables': None, 44 | 'isBase64Encoded': False, 45 | 'entryPathParameters': entry_path_parameters or {}, 46 | 'requestContext': { 47 | 'httpMethod': method.upper(), 48 | 'resourceId': 'fp5ol74tr7', 49 | 'entryPath': entry_path, 50 | 'extendedRequestId': 'EktgyFweDoEFabw=', 51 | 'requestTime': '24/Aug/2021:13:37:08 +0000', 52 | 'path': f"/dev{entry_path}", 53 | 'accountId': '935392763270', 54 | 'protocol': 'HTTP/1.1', 55 | 'stage': 'dev', 56 | 'domainPrefix': 'htzd2rneg1', 57 | 'requestTimeEpoch': 1629812228818, 58 | 'requestId': '2fa7f00a-58fe-4f46-a829-1fee00898e42', 59 | 'domainName': 'htzd2rneg1.execute-api.eu-west-1.amazonaws.com', 60 | 'apiId': 'htzd2rneg1' 61 | }, 62 | 'params': { 63 | 'path': {}, 64 | 'querystring': {}, 65 | 'header': {'Accept': '*/*', 'Authorization': 'token', 'CloudFront-Forwarded-Proto': 'https', 66 | 'CloudFront-Is-Desktop-Viewer': 'true', 'CloudFront-Is-Mobile-Viewer': 'false', 67 | 'CloudFront-Is-SmartTV-Viewer': 'false', 'CloudFront-Is-Tablet-Viewer': 'false', 68 | 'CloudFront-Viewer-Country': 'FR', 'content-type': 'application/json', 69 | 'Cookie': 'session=70b58773-af3e-4153-a5ab-5356481ea87e', 70 | 'Host': 'htzd2rneg1.execute-api.eu-west-1.amazonaws.com', 'User-Agent': 'insomnia/2021.4.1', 71 | 'Via': '2.0 4dd111c814b0b5cf8bf82e59008da625.cloudfront.net (CloudFront)', 72 | 'X-Amz-Cf-Id': 'ka1hbQCSUOZ-d0VQYuE_gtF4icy443t7kP3UGsDLZDF_5QyTX13FoQ==', 73 | 'X-Amzn-Trace-Id': 'Root=1-6124f604-3fb9457c7489ebf14ed0f8f6', 74 | 'X-Forwarded-For': '78.234.174.193, 130.176.152.165', 'X-Forwarded-Port': '443', 75 | 'X-Forwarded-Proto': 'https'}}, 76 | 'context': { 77 | 'resourceId': 'fp5ol74tr7', 78 | 'authorizer': '', 79 | 'resourcePath': '/', 80 | 'httpMethod': 'GET', 81 | 'extendedRequestId': 'EktgyFweDoEFabw=', 82 | 'requestTime': '24/Aug/2021:13:37:08 +0000', 83 | 'path': '/dev/', 84 | 'accountId': '935392763270', 85 | 'protocol': 'HTTP/1.1', 86 | 'requestOverride': '', 87 | 'stage': 'dev', 88 | 'domainPrefix': 'htzd2rneg1', 89 | 'requestTimeEpoch': '1629812228818', 90 | 'requestId': '2fa7f00a-58fe-4f46-a829-1fee00898e42', 91 | 'identity': '', 92 | 'domainName': 'htzd2rneg1.execute-api.eu-west-1.amazonaws.com', 93 | 'responseOverride': '', 94 | 'apiId': 'htzd2rneg1' 95 | } 96 | } 97 | -------------------------------------------------------------------------------- /tests/coworks/tech/test_content_type.py: -------------------------------------------------------------------------------- 1 | import os 2 | from unittest import mock 3 | 4 | import pytest 5 | from flask import json 6 | 7 | from coworks import TechMicroService 8 | from coworks import entry 9 | from ..event import get_event 10 | 11 | 12 | class ContentMS(TechMicroService): 13 | 14 | @entry 15 | def get(self): 16 | return "test" 17 | 18 | @entry 19 | def get_json(self): 20 | return {'text': 'value', 'int': 1} 21 | 22 | @entry 23 | def post(self, text=None, context=None, files=None): 24 | if files: 25 | if type(files) is not list: 26 | files = [files] 27 | return f"post {text}, {context} and {[f.file.name for f in files]}" 28 | return f"post {text}, {context}" 29 | 30 | @entry(binary_headers={'content-type': 'application/octet'}, no_auth=True) 31 | def get_binary(self): 32 | return b"test" 33 | 34 | @entry(binary_headers={'content-type': 'application/pdf'}, no_auth=True) 35 | def get_content_type(self): 36 | return b"test" 37 | 38 | @entry(binary_headers={'content-type': 'application/octet'}, no_auth=True) 39 | def get_no_auth(self): 40 | return b"test" 41 | 42 | 43 | @pytest.mark.skip 44 | @mock.patch.dict(os.environ, {"TOKEN": "token"}) 45 | class TestClass: 46 | def test_default_content_type(self): 47 | app = ContentMS() 48 | with app.test_client() as c: 49 | headers = {'Authorization': 'token'} 50 | response = c.get('/', headers=headers) 51 | assert response.status_code == 200 52 | assert response.is_json 53 | assert response.headers['Content-Type'] == 'application/json' 54 | assert response.get_data(as_text=True) == 'test' 55 | 56 | def test_json_content_type(self): 57 | app = ContentMS() 58 | with app.test_client() as c: 59 | headers = {'Accept': 'application/json', 'Authorization': 'token'} 60 | response = c.get('/', headers=headers) 61 | assert response.status_code == 200 62 | assert response.is_json 63 | assert response.headers['Content-Type'] == 'application/json' 64 | assert response.get_data(as_text=True) == 'test' 65 | 66 | def test_text_content_type(self): 67 | app = ContentMS() 68 | with app.test_client() as c: 69 | headers = {'Accept': 'text/plain', 'Authorization': 'token'} 70 | response = c.get('/', headers=headers) 71 | assert response.status_code == 200 72 | assert response.is_json 73 | assert response.headers['Content-Type'] == 'application/json' 74 | assert response.get_data(as_text=True) == 'test' 75 | 76 | def test_text_api(self): 77 | app = ContentMS() 78 | with app.test_client() as c: 79 | headers = {'Authorization': 'token'} 80 | response = c.get('/json', headers=headers) 81 | assert response.status_code == 200 82 | assert response.is_json 83 | assert response.headers['Content-Type'] == 'application/json' 84 | assert response.json == {"int": 1, "text": "value"} 85 | 86 | headers = {'Accept': 'application/json', 'Authorization': 'token'} 87 | response = c.get('/json', headers=headers) 88 | assert response.status_code == 200 89 | assert response.is_json 90 | assert response.headers['Content-Type'] == 'application/json' 91 | assert response.json == {"int": 1, "text": "value"} 92 | 93 | headers = {'Accept': 'text/plain', 'Authorization': 'token'} 94 | response = c.get('/json', headers=headers) 95 | assert response.status_code == 200 96 | assert response.is_json 97 | assert response.headers['Content-Type'] == 'application/json' 98 | assert json.loads(response.get_data(as_text=True)) == {"text": "value", "int": 1} 99 | 100 | def test_binary_content_type(self, empty_aws_context): 101 | app = ContentMS() 102 | with app.test_client() as c: 103 | headers = {'Accept': 'img/webp', 'Authorization': 'token'} 104 | response = app(get_event('/binary', 'get', headers=headers), empty_aws_context) 105 | # assert type(response) == str 106 | # assert base64.b64decode(str(response)) == b"test" 107 | 108 | def test_content_type(self, empty_aws_context): 109 | app = ContentMS() 110 | with app.test_client() as c: 111 | headers = {'Accept': 'img/webp', 'Authorization': 'token'} 112 | response = app(get_event('/content/type', 'get', headers=headers), empty_aws_context) 113 | # assert type(response) == str 114 | # assert base64.b64decode(str(response)) == b"test" 115 | -------------------------------------------------------------------------------- /tests/coworks/tech/test_type.py: -------------------------------------------------------------------------------- 1 | import typing as t 2 | 3 | from coworks import TechMicroService 4 | from coworks import entry 5 | 6 | 7 | class TypedMS(TechMicroService): 8 | 9 | def _check_token(self): 10 | """No check.""" 11 | 12 | @entry 13 | def get(self, i: int): 14 | return ("ok", 200) if type(i) is int else ("not ok", 400) 15 | 16 | @entry 17 | def get_(self, i: int = 0): 18 | return ("ok", 200) if type(i) is int else ("not ok", 400) 19 | 20 | @entry 21 | def post_str(self, s: str = None): 22 | return s 23 | 24 | @entry 25 | def get_bool(self, i: bool): 26 | if type(i) is bool: 27 | return ("true", 200) if i else ("false", 200) 28 | return "not ok", 400 29 | 30 | @entry 31 | def get_bool_(self, i: bool = None): 32 | if type(i) is bool: 33 | return ("true", 200) if i else ("false", 200) 34 | return "not ok", 400 35 | 36 | @entry 37 | def get_union(self, i: t.Union[int, int] = 0): 38 | return ("ok", 200) if type(i) is int else ("not ok", 400) 39 | 40 | @entry 41 | def get_list(self, i: list[int] = 0): 42 | return ("ok", 200) if type(i) is list else ("not ok", 400) 43 | 44 | @entry 45 | def post(self, i: t.Union[int, int] = 0): 46 | return ("ok", 200) if type(i) is int else ("not ok", 400) 47 | 48 | 49 | class TestClass: 50 | 51 | def test_base_type(self): 52 | app = TypedMS() 53 | 54 | with app.test_client() as c: 55 | response = c.get('/1') 56 | assert response.status_code == 200 57 | 58 | response = c.get('/?i=1') 59 | assert response.status_code == 200 60 | 61 | response = c.post('/', json={'i': 1}) 62 | assert response.status_code == 200 63 | 64 | response = c.post('/', json={'i': '1'}) 65 | assert response.status_code == 200 66 | 67 | response = c.post('/', json={'i': 'abc'}) 68 | assert response.status_code == 422 69 | 70 | def test_str(self): 71 | app = TypedMS() 72 | 73 | with app.test_client() as c: 74 | response = c.post('/str') 75 | assert response.status_code == 204 76 | 77 | response = c.post('/str', json={'s': None}) 78 | assert response.status_code == 204 79 | 80 | response = c.post('/str', json={'s': ''}) 81 | assert response.status_code == 200 82 | assert response.get_data(as_text=True) == "" 83 | 84 | def test_bool(self): 85 | app = TypedMS() 86 | 87 | with app.test_client() as c: 88 | response = c.get('/bool/true') 89 | assert response.status_code == 200 90 | assert response.get_data(as_text=True) == "true" 91 | 92 | response = c.get('/bool/1') 93 | assert response.status_code == 200 94 | assert response.get_data(as_text=True) == "true" 95 | 96 | response = c.get('/bool/false') 97 | assert response.status_code == 200 98 | assert response.get_data(as_text=True) == "false" 99 | 100 | response = c.get('/bool/0') 101 | assert response.status_code == 200 102 | assert response.get_data(as_text=True) == "false" 103 | 104 | response = c.get('/bool?i=true') 105 | assert response.status_code == 200 106 | assert response.get_data(as_text=True) == "true" 107 | 108 | response = c.get('/bool?i=1') 109 | assert response.status_code == 200 110 | assert response.get_data(as_text=True) == "true" 111 | 112 | response = c.get('/bool?i=false') 113 | assert response.status_code == 200 114 | assert response.get_data(as_text=True) == "false" 115 | 116 | response = c.get('/bool?i=0') 117 | assert response.status_code == 200 118 | assert response.get_data(as_text=True) == "false" 119 | 120 | def test_union_type(self): 121 | app = TypedMS() 122 | 123 | with app.test_client() as c: 124 | response = c.get('/union?i=1') 125 | assert response.status_code == 200 126 | 127 | response = c.get('/union?i=1&i=2') 128 | assert response.status_code == 422 129 | 130 | response = c.get('/union?i=abc') 131 | assert response.status_code == 422 132 | 133 | def test_list_type(self): 134 | app = TypedMS() 135 | 136 | with app.test_client() as c: 137 | response = c.get('/list?i=1') 138 | assert response.status_code == 200 139 | 140 | response = c.get('/list?i=1&i=2') 141 | assert response.status_code == 200 142 | 143 | response = c.get('/list?i=abc') 144 | assert response.status_code == 422 145 | -------------------------------------------------------------------------------- /docs/command.rst: -------------------------------------------------------------------------------- 1 | .. _command: 2 | 3 | Commands 4 | ======== 5 | 6 | As for Flask, CoWorks allows you to extend the ``cws`` application with ``click`` commands. 7 | This powerfull extension is very usefull for complex deployment, testing or documentation. 8 | 9 | As explained before, the microservice architecture needs to be completed by tools. The ``cws`` command line extends 10 | the ``flask`` command for that purpose. 11 | 12 | .. _cli: 13 | 14 | CWS : Command Line Interface 15 | ---------------------------- 16 | 17 | ``cws`` is an extension of the Flask command-line shell program that provides convenience and productivity 18 | features to help user to : 19 | 20 | * Get microservices informations, 21 | * Export microservices to another formats, 22 | * Deploy or update deployed microservices, 23 | * ... 24 | 25 | It is a generic client interface on which commands may be defined. 26 | 27 | Usage 28 | ^^^^^ 29 | 30 | To view a list of the available commands at any time, just run `cws` with no arguments:: 31 | 32 | $ cws --help 33 | Usage: cws [OPTIONS] COMMAND [ARGS]... 34 | 35 | Options: 36 | --version Show the version and exit. 37 | -p, --project-dir TEXT The project directory path (absolute or relative) 38 | [default to 'tech']. 39 | -c, --config-file TEXT Configuration file path [relative from project 40 | dir]. 41 | --config-file-suffix TEXT Configuration file suffix. 42 | -e, --env-file FILE Load environment variables from this file. 43 | python-dotenv must be installed. 44 | -A, --app IMPORT The Flask application or factory function to 45 | load, in the form 'module:name'. Module can be a 46 | dotted import or file path. Name is not required 47 | if it is 'app', 'application', 'create_app', or 48 | 'make_app', and can be 'name(args)' to pass 49 | arguments. 50 | --debug / --no-debug Set debug mode. 51 | --help Show this message and exit. 52 | 53 | 54 | Commands: 55 | deploy Deploy the CoWorks microservice on AWS Lambda. 56 | deployed Retrieve the microservices deployed for this project. 57 | new Creates a new CoWorks project. 58 | routes Show the routes for the app. 59 | run Run a development server. 60 | shell Run a shell in the app context. 61 | zip Zip all source files to create a Lambda file source. 62 | 63 | 64 | As you can see, the default Flask commands as shell, routes or shell are predefined. 65 | Some new commands as ``deploy`` have been defined. 66 | 67 | **Note**: As the ``project_dir`` is not defined when you run the microservice without the ``run`` command, 68 | for example in your IDE, you can define the environment variable ``INSTANCE_RELATIVE_PATH`` to be able to retrieve 69 | the environment variable file. The value is a relative path from ``project_dir``. 70 | 71 | At last the usefull variables: 72 | 73 | * ``CWS_STAGE``: to determine which stage will be used (environment file). 74 | * ``FLASK_DEBUG``: may be used same way as for Flask. 75 | 76 | CoWorks Commands 77 | ------------------- 78 | 79 | new 80 | ^^^ 81 | 82 | The ``new`` command creates an empty CoWorks project. 83 | 84 | deploy 85 | ^^^^^^ 86 | 87 | The ``deploy`` command allows to deploy a ``TechMicroService`` on the AWS plateform. 88 | 89 | This is done by creating terraform files from jinja template files. You can override those templates or add new files 90 | if you needed to enhance the deployment process. 91 | 92 | This command may be used to deal with complex deployments, mainly for staging or respecting infrastucture constraints 93 | or processes. 94 | 95 | How to change the deployment command? 96 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 97 | 98 | This is done by simply creating a new click command. 99 | Find an example below:: 100 | 101 | from coworks.cws.client import client 102 | 103 | # Get cws default deploy command 104 | deploy = client.commands['deploy'] 105 | 106 | # Redefines new deploy command 107 | @pass_script_info 108 | @with_appcontext 109 | def new_deploy_callback(info, *args, **kwargs): 110 | app = info.load_app() 111 | kwargs['key'] = f"tech/{app.__module__}-{app.name}/archive.zip" 112 | kwargs['terraform_cloud'] = True 113 | return deploy.callback(*args, terraform_class=FprTerraformCloud, **kwargs) 114 | 115 | 116 | # Adds options 117 | deploy = client.commands['deploy'] 118 | new_deploy = CwsCommand(deploy.name, callback=deploy_callback, params=deploy.params) 119 | click.option('--vpc', is_flag=True, default=True, help="Set lambda in VPC.")(new_deploy) 120 | -------------------------------------------------------------------------------- /docs/biz.rst: -------------------------------------------------------------------------------- 1 | .. _biz: 2 | 3 | BizMicroservices 4 | ================ 5 | 6 | BizMicroService is the orchestration of TechMicroServices. This orchestration is defined with the 7 | `Airflow `_ plateform. 8 | 9 | **Notice** : We recomand to create first a directory service as described in the directory :ref:`samples`. 10 | 11 | 12 | DAG 13 | --- 14 | 15 | The definition of the DAG, the BizMicroService, is done by thru ``biz`` decorator, which is simply a renaming 16 | of the ``dag`` decorator of Airflow. 17 | 18 | *Notice* : It seems stupid to just rename a decorator, but we have in mind to use this decorator in future for 19 | creating relation dependencies between microservices. 20 | 21 | .. code-block:: python 22 | 23 | from coworks.biz import biz 24 | 25 | DEFAULT_ARGS = { 26 | 'retries': 1, 27 | 'retry_delay': timedelta(minutes=5), 28 | 'email': "gdoumenc@neorezo.io", 29 | 'email_on_failure': True, 30 | 'email_on_retry': False, 31 | } 32 | 33 | 34 | @biz( 35 | default_args=DEFAULT_ARGS, 36 | tags=['coworks', 'sample'], 37 | start_date=datetime(2022, 1, 1), 38 | schedule_interval=None, 39 | catchup=False, 40 | render_template_as_native_obj=True, 41 | ) 42 | def my_first_biz(data): 43 | ... 44 | 45 | Operators 46 | --------- 47 | 48 | TechMicroServiceOperator 49 | ^^^^^^^^^^^^^^^^^^^^^^^^ 50 | 51 | This first operator is for calling a TechMicroService. 52 | 53 | .. code-block:: python 54 | 55 | create_invoice = TechMicroServiceOperator( 56 | task_id='create_invoice', 57 | cws_name='neorezo-billing_invoice-eshop', 58 | method='POST', 59 | entry="/invoice", 60 | json="{{ dag_run.conf['data'] }}", 61 | ) 62 | 63 | The arguments are : 64 | 65 | * ``cws_name`` : allows to call the microservice by its name thanks to the directory service, 66 | * ``method`` and ``entry`` : route to the service, 67 | * ``data`` or ``json`` :service parameters for ``GET`` and ``POST`` method respectively. 68 | 69 | 70 | Other main arguments are needed to be understood : 71 | 72 | * ``directory_conn_id``: the airflow connection id used to call the directory microservice. By default 'coworks_directory'. 73 | * ``asynchronous``: Asynchronous status. By default 'False'. 74 | 75 | ``cws_name``, ``entry``, ``data``, ``json``, ``asynchronous`` arguments are templated. 76 | 77 | If you don't want to use the directory microservice: 78 | 79 | .. code-block:: python 80 | 81 | create_invoice = TechMicroServiceOperator( 82 | task_id='create_invoice', 83 | api_id='xxxxx', 84 | stage='v1', 85 | token='yyyy', 86 | method='POST', 87 | entry="/invoice", 88 | json="{{ dag_run.conf['data'] }}", 89 | ) 90 | 91 | BranchTechMicroServiceOperator 92 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 93 | 94 | This branching operator allows to test microservice status code or result content 95 | 96 | .. code-block:: python 97 | 98 | check_invoice = BranchTechMicroServiceOperator( 99 | task_id='check_invoice', 100 | cws_task_id='neorezo-billing_invoice-eshop', 101 | on_success = "sent_to_customer" 102 | on_failure = "mail_error" 103 | ) 104 | 105 | The arguments are : 106 | 107 | * ``cws_task_id`` : calling task id used to retrieve XCOM values, 108 | * ``on_success`` : branch task id on success, 109 | * ``on_failure`` :branch task id on failure. 110 | 111 | Sensors 112 | ------- 113 | 114 | This sensor is defined to wait until an asynchronous call is finished. 115 | 116 | .. code-block:: python 117 | 118 | await_invoice = AsyncTechMicroServiceSensor( 119 | task_id='await_invoice', 120 | cws_task_id='neorezo-billing_invoice-eshop', 121 | ) 122 | 123 | This sensor will await the microservice ``billing_invoice-eshop`` will terminate its asynchronous execution. 124 | 125 | The arguments are : 126 | 127 | * ``cws_task_id`` : the microserrvice call task awaited, 128 | 129 | Other main arguments are needed to be understood : 130 | 131 | * ``aws_conn_id`` : the airflow connection id used to observe S3 result. By default 'aws_s3'. 132 | 133 | Asynchronous task 134 | ----------------- 135 | 136 | The sequence of a calling task, a waiting task and a reading result task for an asynchronous call is done by: 137 | 138 | .. code-block:: python 139 | 140 | invoice = TechMicroServiceAsyncGroup( 141 | 'invoice', 142 | cws_name='neorezo-billing_invoice-eshop', 143 | method='POST', 144 | entry="/invoice", 145 | json="{{ dag_run.conf['data'] }}", 146 | ) 147 | 148 | The result is then accessible in ``invoice.output`` in python code, or thru the ``invoice.read`` task id:: 149 | 150 | invoice >> send_mail(invoice.output) 151 | 152 | or:: 153 | 154 | ti.xcom_pull(task_ids='invoice.read') 155 | 156 | -------------------------------------------------------------------------------- /coworks/biz/group.py: -------------------------------------------------------------------------------- 1 | import typing as t 2 | 3 | from airflow.models import BaseOperator 4 | from airflow.operators.python import PythonOperator 5 | from airflow.utils.task_group import TaskGroup 6 | from airflow.utils.trigger_rule import TriggerRule 7 | 8 | from coworks.biz.operators import AsyncTechServicePullOperator 9 | from coworks.biz.operators import TechMicroServiceOperator 10 | from coworks.biz.sensors import AsyncTechMicroServiceSensor 11 | 12 | 13 | class CoworksTaskGroup(TaskGroup): 14 | """ Asynchronous tasks group. 15 | 16 | .. versionchanged:: 0.8.4 17 | Added the ``start_id``, ``end_id`` properties. 18 | """ 19 | 20 | def __init__(self, *args, **kwargs): 21 | super().__init__(*args, **kwargs) 22 | self.transformer_task: BaseOperator | None = None 23 | self.call_task: BaseOperator | None = None 24 | self.wait_task: BaseOperator | None = None 25 | self.read_task: BaseOperator | None = None 26 | 27 | @property 28 | def start_id(self): 29 | return f'{self._group_id}.transformer' if self.transformer_task else f'{self._group_id}.call' 30 | 31 | @property 32 | def end_id(self): 33 | return f'{self._group_id}.read' if self.read_task else f'{self._group_id}.wait' 34 | 35 | @property 36 | def output(self): 37 | return self.read_task.output 38 | 39 | 40 | def TechMicroServiceAsyncGroup(group_id: str, transformer: t.Callable = None, read: bool = True, 41 | op_args: t.Collection[t.Any] | None = None, 42 | op_kwargs: t.Mapping[str, t.Any] | None = None, 43 | method: str = 'get', timeout: int = 900, 44 | raise_errors: bool = True, raise_400_errors: bool = True, 45 | xcom_push: bool = True, trigger_rule=TriggerRule.ALL_SUCCESS, 46 | **tech_kwargs): 47 | """Task group to allow asynchronous call of a TechMicroService. 48 | 49 | The returned value is defined in the task_id : '{group_id}.read'. 50 | 51 | :param group_id: group id. 52 | :param transformer: python transformer function. 53 | :param read: do read task. 54 | :param op_args: python transformer args. 55 | :param op_kwargs: python transformer kwargs. 56 | :param method: microservice method called. 57 | :param raise_errors: raise error on client errors (default True). 58 | :param raise_400_errors: raise error on client 400 errors (default True). 59 | :param timeout: asynchronous call timeout. 60 | :param trigger_rule: trigger rule to be set on first task. 61 | :param xcom_push: pushes result in XCom (default True). 62 | 63 | .. versionchanged:: 0.8.4 64 | Added the ``read`` parameter. 65 | .. versionchanged:: 0.8.4 66 | Added the ``trigger_rule`` parameter. 67 | .. versionchanged:: 0.8.0 68 | Added the ``transformer`` parameter. 69 | """ 70 | with CoworksTaskGroup(group_id=group_id) as tg: 71 | if transformer: 72 | tg.transformer_task = PythonOperator( 73 | task_id='transformer', 74 | python_callable=transformer, 75 | op_args=op_args, 76 | op_kwargs=op_kwargs, 77 | trigger_rule=trigger_rule 78 | ) 79 | 80 | if method.lower() == 'get': 81 | if 'query_params' in tech_kwargs: 82 | msg = "Calling transformer method with already query_params parameter call" 83 | tg.transformer_task.log.warning(msg) 84 | tech_kwargs['query_params'] = tg.transformer_task.output 85 | else: 86 | if 'json' in tech_kwargs: 87 | tg.transformer_task.log.warning("Calling transformer method with already json parameter call") 88 | tech_kwargs['json'] = tg.transformer_task.output 89 | 90 | tg.call_task = TechMicroServiceOperator( 91 | task_id="call", 92 | asynchronous=True, 93 | method=method, 94 | raise_errors=raise_errors, 95 | raise_400_errors=raise_400_errors, 96 | trigger_rule=trigger_rule if not transformer else TriggerRule.ALL_SUCCESS, 97 | **tech_kwargs, 98 | ) 99 | 100 | tg.wait_task = AsyncTechMicroServiceSensor( 101 | task_id='wait', 102 | cws_task_id=f'{group_id}.call', 103 | timeout=timeout, 104 | ) 105 | 106 | if read: 107 | tg.read_task = AsyncTechServicePullOperator( 108 | task_id='read', 109 | cws_task_id=f'{group_id}.call', 110 | raise_errors=raise_errors, 111 | raise_400_errors=raise_400_errors, 112 | xcom_push=xcom_push, 113 | ) 114 | 115 | tg.call_task >> tg.wait_task 116 | 117 | if transformer: 118 | tg.transformer_task >> tg.call_task 119 | if read: 120 | tg.wait_task >> tg.read_task 121 | 122 | return tg 123 | -------------------------------------------------------------------------------- /docs/configuration.rst: -------------------------------------------------------------------------------- 1 | .. _configuration: 2 | 3 | Configuration 4 | ============= 5 | 6 | Configuration versus Environment variable 7 | ----------------------------------------- 8 | 9 | There are three configuration levels: 10 | 11 | * Project config, 12 | * Execution config, 13 | * Application config. 14 | 15 | **Project configuration** 16 | Project configuration is related to how the team works and how deployment should be done. This description 17 | is done by a project configuration file: ``project.cws.yml``. This project configuration file describes 18 | the commands and options associated to the project. 19 | 20 | **Execution configuration** 21 | As for the `Twelve-Factor App `_ : *"The twelve-factor app stores config in environment variables. 22 | Env vars are easy to change between deploys without changing any code;"*. Using environment variables is highly 23 | recommanded to enable easy code deployments to differents systems: 24 | Changing configuration is just updating variables in the configuration in the CI/CD process. 25 | 26 | **Application configuration** 27 | At last : *"application config does not vary between deploys, and so is best done in the code."* 28 | 29 | That's why entries are defined in the code. The link from entry to function is always the same. 30 | 31 | In Flask, there is only the concept of application configuration as the excution configuration if out of is scope 32 | (mainly associated to the USWGI server). 33 | 34 | In CoWorks, we use the concept of *stage* for execution configuration deployed in lambda variables. 35 | 36 | 37 | Project configuration 38 | --------------------- 39 | 40 | Stage is a key concept for the deployment. Stages are defined thru the concept of *workspace* in terraform, *stage* for 41 | AWS API Gateway and *variables* of AWS Lambda. 42 | 43 | 44 | Workspace definition 45 | ^^^^^^^^^^^^^^^^^^^^ 46 | 47 | The ``workspace`` value will correspond to the ``CWS_STAGE`` variable value. 48 | 49 | You certainly may need to attach environment variables to your project. Of course thoses variables may depend on the 50 | stage status. How? You just need to create and specify custom environment files. 51 | 52 | CoWorks uses dotenv files to allow you to define your environment variables for stages. 53 | Dotenv file are named ``.env`` and ``.env_{CWS_STAGE}``. 54 | 55 | As example you can deploy the specific stage ``dev`` of the microservice ``app`` defined in the ``app`` python file 56 | in the folder ``tech``:: 57 | 58 | $ CWS_STAGE=dev deploy 59 | 60 | The environment variables accessible from the lambda must be defined in ``.env`` and ``.env_dev``. 61 | 62 | Project configuration file 63 | ^^^^^^^^^^^^^^^^^^^^^^^^^^ 64 | 65 | A project configuration file is a YAML file containg the command and options defined for the project. 66 | 67 | Example 68 | ******* 69 | 70 | Example of a `project.cws.yml` file: 71 | 72 | .. code-block:: yaml 73 | 74 | version: 3 75 | commands: 76 | run: 77 | host: localhost 78 | port: 5000 79 | deploy: 80 | class: fpr.cws.deploy.fpr_deploy 81 | profile_name: fpr-customer 82 | bucket: coworks-microservice 83 | customer: neorezo 84 | project: cws_utils_mail 85 | layers: 86 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-0.6.8 87 | workspaces: 88 | dev: 89 | commands: 90 | run: 91 | port: 8000 92 | deploy: 93 | layers: 94 | - arn:aws:lambda:eu-west-1:935392763270:layer:coworks-dev 95 | 96 | Structure 97 | ********* 98 | 99 | .. list-table:: **Project Configuration File Structure** 100 | :widths: 10 20 20 101 | :header-rows: 1 102 | 103 | * - Field 104 | - Value 105 | - Description 106 | * - version 107 | - 3 108 | - YAML syntax version 109 | * - commands 110 | - Command Structure List (below) 111 | - List of commands 112 | * - workspaces 113 | - Workspace Structure List (below) 114 | - List of workspaces where commands are redefined 115 | 116 | .. list-table:: **Command Structure** 117 | :widths: 10 10 10 118 | :header-rows: 1 119 | 120 | * - Command Name 121 | - Command Option 122 | - Project Value 123 | * - run 124 | - 125 | - 126 | * - 127 | - host 128 | - localhost 129 | * - 130 | - port 131 | - 5000 132 | 133 | .. list-table:: **Workspace Structure** 134 | :widths: 10 10 10 10 135 | :header-rows: 1 136 | 137 | * - Workspace Name 138 | - Command Name 139 | - Command Option 140 | - Project Value 141 | * - dev 142 | - 143 | - 144 | - 145 | * - 146 | - run 147 | - 148 | - 149 | * - 150 | - 151 | - port 152 | - 8000 153 | 154 | .. _auth: 155 | 156 | Authorization 157 | ------------- 158 | 159 | By default all ``TechMicroService`` have access protection defined in the microservice itself and defined thru 160 | a token basic authentication protocol based on 161 | `HTTP Authentification `_ 162 | 163 | Class control 164 | ^^^^^^^^^^^^^ 165 | 166 | For simplicity, we can define only one simple authorizer on a class. The authorizer may be defined by the method 167 | ``token_authorizer``. 168 | 169 | .. code-block:: python 170 | 171 | from coworks import TechMicroService 172 | 173 | class SimpleExampleMicroservice(TechMicroService): 174 | 175 | def token_authorizer(self, token): 176 | return True 177 | 178 | If the method returns ``True`` all the routes are allowed. If it returns ``False`` all routes are denied. 179 | 180 | Using the APIGateway model, the authorization protocol is defined by passing a token 'Authorization'. 181 | The API client must include it in the header to send the authorization token to the Lambda authorizer. 182 | 183 | .. code-block:: python 184 | 185 | from coworks import TechMicroService 186 | 187 | class SimpleExampleMicroservice(TechMicroService): 188 | 189 | def token_authorizer(self, token): 190 | return token == os.getenv('TOKEN') 191 | 192 | To call this microservice, we have to put the right token in headers:: 193 | 194 | curl https://zzzzzzzzz.execute-api.eu-west-1.amazonaws.com/my/route -H 'Authorization: thetokendefined' 195 | 196 | -------------------------------------------------------------------------------- /samples/website/tech/templates/base.j2: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | CoWorks Website Sample 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 22 | 23 | 24 |
    25 |
    26 |
    27 |
    28 | 29 |
    30 | 31 | CoWorks 32 | 33 |
    34 |
    35 |
    36 |
    37 |
    38 |
    39 | 40 |
    41 | 42 |
    43 | 44 | {% with messages = get_flashed_messages(with_categories=true) %} 45 | {# category in "message", "error", and "warning"#} 46 | {% if messages %} 47 |
    48 |
    49 |
      50 | 51 | {% for category, message in messages %} 52 |
    • 53 |
      54 | {{ message }} 55 |
      56 |
      • 57 | {% endfor %} 58 |
    59 |
    60 |
    61 | {% endif %} 62 | {% endwith %} 63 | 64 | 65 |
    66 |
    67 | {% block header %} 68 |

    Profile Overview

    69 |
    70 |
    71 |
    72 |
    73 | CoWorks 75 |
    76 |
    77 |

    Welcome {{ current_user.email }} and enjoy

    78 |

    CoWorks framework

    79 |

    Serverless Microservices Solution

    80 |
    81 |
    82 |
    83 | {% if current_user.is_authenticated %} 84 | 86 | Sign Out 87 | {% else %} 88 | 90 | Sign In 91 | {% endif %} 92 |
    93 |
    94 |
    95 |
    97 |
    98 | Last version: 99 | {{ version }} 100 |
    101 | 102 |
    103 | Github stars: 104 | {{ stargazers_count }} 105 | 110 | 111 |
    112 | 113 |
    114 | In use: 115 | {{ tech_in_use }} 116 |
    117 |
    118 | {% endblock header %} 119 |
    120 |
    121 | 122 | {% block content %} 123 | {% endblock content %} 124 |
    125 | 126 | 127 |
    128 | 129 |
    130 |
    131 |
    132 |

    Announcements

    133 |
    134 | {% block news %} 135 | {% endblock news %} 136 |
    137 |
    138 |
    139 |
    140 | 141 |
    142 |
    143 |
    144 |
    145 |
    146 | 147 | 148 | 149 | 150 | -------------------------------------------------------------------------------- /docs/tech_quickstart.rst: -------------------------------------------------------------------------------- 1 | .. _tech_quickstart: 2 | 3 | Tech Quickstart 4 | =============== 5 | 6 | This page gives a quick and partial introduction to CoWorks Technical Microservices. 7 | Follow :doc:`installation` to install CoWorks and set up a new project. 8 | 9 | CoWorks Technical Microservices are `atomic microservices`, meaning that they are single `atomic` components 10 | (i.e: singular blobs of code with a few inputs and outputs). 11 | 12 | A tech microservice is simply defined by a single python class which looks like this: 13 | 14 | .. code-block:: python 15 | 16 | class SimpleMicroService(TechMicroService): 17 | 18 | def get(self): 19 | return f"Simple microservice ready.\n" 20 | 21 | Start 22 | ----- 23 | 24 | To create your first complete technical microservice, create a file ``hello.py`` in the ``tech`` folder 25 | with the following content: 26 | 27 | .. literalinclude:: ../samples/docs/tech/hello.py 28 | 29 | This first example defines the very classical ``hello`` microservice ``app`` with a simple ``GET`` entry ``/`` 30 | (see :ref:`routing` for more details on entry) 31 | 32 | We set the attribute ``no_auth`` to ``True`` to allow access without authorization. 33 | This effectively disables the token authorizer. 34 | For security reason the default value is ``False`` (see :ref:`auth` for more details on authorizer). 35 | 36 | We now can launch the ``run`` command defined by the ``Flask`` framework. So to test this microservice locally 37 | (see `Flask `_ for more details):: 38 | 39 | (project) $ cws --app hello run 40 | * Stage: dev 41 | * Serving Flask app 'hello' 42 | * Debug mode: off 43 | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. 44 | * Running on http://127.0.0.1:5000 45 | Press CTRL+C to quit 46 | 47 | To test this example, open another terminal window and enter:: 48 | 49 | (project) $ curl http://127.0.0.1:5000/ 50 | Hello world. 51 | 52 | If you remove the argument ``no_auth=True`` from our ``@entry`` decorator, you should instead receive a 403 response. 53 | 54 | First 55 | ----- 56 | 57 | To add more elements, complete your first try with the following content: 58 | 59 | .. literalinclude:: ../samples/docs/tech/first.py 60 | 61 | We have added a dedicated function ``token_authorizer`` to define an authorizer 62 | (see :ref:`auth` for more details on authorizer). 63 | For this simple try, the authorizer validates the request only if a token is defined on header : 64 | ``Authorization`` key with ``token`` as value. 65 | 66 | Then we have defined two entries on same path : ``GET`` and ``POST`` on root path. 67 | These enable reading and writing of our attribute ``value``. 68 | 69 | To test this example, open another terminal window and enter:: 70 | 71 | (project) $ curl -I http://127.0.0.1:5000/ 72 | HTTP/1.0 401 UNAUTHORIZED 73 | ... 74 | 75 | (project) $ curl -H "Authorization:token" http://127.0.0.1:5000/ 76 | Stored value 0. 77 | 78 | (project) $ curl -X POST -d '{"value":20}' -H "Content-Type: application/json" -H "Authorization:token" http://127.0.0.1:5000/ 79 | Value stored (20). 80 | 81 | (project) $ curl -H "Authorization:token" http://127.0.0.1:5000/ 82 | Stored value 20. 83 | 84 | *Beware* : the ``value`` is stored in memory just for this example, if the lambda is redeployed or another lambda instance 85 | is used the value stored is lost. 86 | 87 | Complete 88 | -------- 89 | 90 | We can create and test a more complete case by leveraging blueprints and adding middlewares. 91 | We will also use `StringIO `_ to write our output to a string buffer. 92 | 93 | For more information on how CoWorks uses blueprints, see `TechMS Blueprints `_. 94 | For more information on how CoWorks uses WSGI middlewares, see `Middlewares `_. 95 | 96 | First, ensure that `aws_xray_sdk` is installed in your python environment:: 97 | 98 | $ pip install aws-xray-sdk 99 | 100 | Then, enter the following content: 101 | 102 | .. literalinclude:: ../samples/docs/tech/complete.py 103 | 104 | *Note* : `aws_xray_sdk` must be installed in your python environment or you will get an ``ImportError``. 105 | If you receive this error, follow the step above to install. 106 | 107 | By default the token value should be defined in the ``TOKEN`` environment variable ; the simpliest way to declare it 108 | is to create a dotenv file (``.env``) in the project folder with this token value defined in it:: 109 | 110 | TOKEN=mytoken 111 | 112 | The ``Admin`` blueprint `adds several routes `_ but 113 | for the purposes of this example we're interested in the root one (``/admin`` as prefixed): 114 | 115 | This endpoint gives documentation and all the routes of the microservice with the signature extracted from its associated function. 116 | 117 | We have also a WSGI middleware ``ProfilerMiddleware`` to profile the last request:: 118 | 119 | (project) $ curl -H "Authorization:mytoken" http://127.0.0.1:5000/profile 120 | -------------------------------------------------------------------------------- 121 | PATH: '/profile' 122 | 441 function calls (436 primitive calls) in 0.001 seconds 123 | 124 | Ordered by: internal time, call count 125 | 126 | ncalls tottime percall cumtime percall filename:lineno(function) 127 | 1 0.000 0.000 0.000 0.000 {method 'getvalue' of '_io.StringIO' objects} 128 | 1 0.000 0.000 0.000 0.000 /home/gdo/.local/share/virtualenvs/samples-G9jKBMQA/lib/python3.10/site-packages/werkzeug/routing/map.py:246(bind_to_environ) 129 | 11 0.000 0.000 0.000 0.000 /home/gdo/.local/share/virtualenvs/samples-G9jKBMQA/lib/python3.10/site-packages/werkzeug/local.py:308(__get__) 130 | ... 131 | 132 | And at last we have a CoWorks middleware to add `XRay traces `_ 133 | (available only for deployed microservices). 134 | 135 | Deploy 136 | ------ 137 | 138 | And now we can upload the sources files to AWS S3 and apply predefined terraform planifications (options may be defined 139 | in project file to avoid given then on command line see :ref:`configuration` ):: 140 | 141 | (project) $ cws deploy --bucket XXX --profile-name YYY --layers arn:aws:lambda:eu-west-1:935392763270:layer:coworks-ZZZ 142 | Terraform apply (Create API routes) 143 | Terraform apply (Deploy API and Lambda for the dev stage) 144 | terraform output : 145 | classical_id = "xxxxxxxx" 146 | (project) $ 147 | 148 | **Notice**: To get the available coworks layer versions, just call this public microservice 149 | (source code available in ``samples/layers``):: 150 | 151 | curl -H 'Accept:application/json' https://2kb9hn4bs4.execute-api.eu-west-1.amazonaws.com/v1 152 | 153 | Now we can test our first deployed microservice:: 154 | 155 | (project) $ curl -H "Authorization:mytoken" https://xxxxxxxx.execute-api.eu-west-1.amazonaws.com/dev 156 | Stored value 0. 157 | 158 | **Notice**: The deploy parameters can be defined once in the project configuration file (``project.cws.yml``) 159 | 160 | **Notice**: You can set the debug option of Flask to get more information on the deploy process 161 | (``FLASK_DEBUG=1 cws deploy``) 162 | 163 | -------------------------------------------------------------------------------- /coworks/blueprint/mail_blueprint.py: -------------------------------------------------------------------------------- 1 | import os 2 | import smtplib 3 | import typing as t 4 | from email import message 5 | from email.utils import formataddr 6 | from email.utils import formatdate 7 | from email.utils import make_msgid 8 | 9 | import requests 10 | from flask import current_app 11 | from flask import render_template_string 12 | from werkzeug.datastructures import FileStorage 13 | from werkzeug.exceptions import BadRequest 14 | from werkzeug.exceptions import InternalServerError 15 | 16 | from coworks import Blueprint 17 | from coworks import entry 18 | 19 | 20 | # 21 | # BLUEPRINT PART 22 | # 23 | 24 | class Mail(Blueprint): 25 | """Mail blueprint. 26 | Initialization parameters must be in environment (Twelve-Factor App). 27 | Environment variables needed: 28 | - env_server_var_name: Variable name for the SMTP server. 29 | - env_port_var_name: Variable name for the SMPT port. 30 | - env_login_var_name: Variable name for the login. 31 | - env_passwd_var_name: Variable name for the password. 32 | """ 33 | 34 | def __init__(self, name: str = "mail", 35 | env_server_var_name: str = '', env_port_var_name: str = '', 36 | env_login_var_name: str = '', env_passwd_var_name: str = '', 37 | env_var_prefix: str = '', **kwargs: dict): 38 | super().__init__(name=name, **kwargs) 39 | 40 | # global prefixed veriables 41 | if env_var_prefix: 42 | env_server_var_name = f"{env_var_prefix}_SERVER" 43 | env_port_var_name = f"{env_var_prefix}_PORT" 44 | env_login_var_name = f"{env_var_prefix}_LOGIN" 45 | env_passwd_var_name = f"{env_var_prefix}_PASSWD" 46 | 47 | self.smtp_server = os.getenv(env_server_var_name) 48 | if not self.smtp_server: 49 | raise RuntimeError(f'{env_server_var_name} not defined in environment.') 50 | self.smtp_port = int(os.getenv(env_port_var_name, 587)) 51 | self.smtp_login = os.getenv(env_login_var_name) 52 | if not self.smtp_login: 53 | raise RuntimeError(f'{env_login_var_name} not defined in environment.') 54 | self.smtp_passwd = os.getenv(env_passwd_var_name) 55 | if not self.smtp_passwd: 56 | raise RuntimeError(f'{env_passwd_var_name} not defined in environment.') 57 | 58 | @entry 59 | def post_send(self, subject: str = "", from_addr: str | None = None, from_name: str = '', 60 | reply_to: str | None = None, body: str = "", body_template: str | None = None, body_type="plain", 61 | to_addrs: list[str] | None = None, cc_addrs: list[str] | None = None, 62 | bcc_addrs: list[str] | None = None, 63 | attachments: FileStorage | t.Iterator[FileStorage] | None = None, attachment_urls: dict | None = None, 64 | starttls: bool = True, data: dict | None = None): 65 | """ Send mail. 66 | To send attachments, add files in the body of the request as multipart/form-data. 67 | 68 | :param subject: Email's subject (required). 69 | :param from_addr: From recipient. 70 | :param from_name: Name besides from_address. 71 | :param reply_to: Reply recipient. 72 | :param to_addrs: Email's recipients. Accept one or several email addresses separated by commas. 73 | :param cc_addrs: Email's cc recipients. Accept one or several email addresses separated by commas. 74 | :param bcc_addrs: Email's bcc recipients. Accept one or several email addresses separated by commas. 75 | :param body: Email's body (required if body_template not defined). 76 | :param body_template: Email's body template (required if body not defined). 77 | :param body_type: Email's body type. 78 | :param attachments: File storage. 79 | :param attachment_urls: File url. 80 | :param starttls: Puts the connection to the SMTP server into TLS mode. 81 | :param data: Jinnja context for the templating. 82 | """ 83 | 84 | from_addr = from_addr or os.getenv('from_addr') 85 | if not from_addr: 86 | raise BadRequest("From address not defined (from_addr:str)") 87 | to_addrs = to_addrs or os.getenv('to_addrs') 88 | if not to_addrs: 89 | raise BadRequest("To addresses not defined (to_addrs:[str])") 90 | 91 | if body_template is not None: 92 | if body is not None: 93 | raise BadRequest("Body and body_template parameters both defined.") 94 | body = render_template_string(body_template, **data) 95 | 96 | # Creates email 97 | try: 98 | from_ = formataddr((from_name if from_name else False, from_addr)) 99 | msg = message.EmailMessage() 100 | msg["Date"] = formatdate(localtime=True) 101 | msg["Message-ID"] = make_msgid() 102 | msg['Subject'] = subject 103 | msg['From'] = from_ 104 | if reply_to is not None: 105 | msg['Reply-To'] = reply_to 106 | msg['To'] = to_addrs if isinstance(to_addrs, str) else ', '.join(to_addrs) 107 | if cc_addrs: 108 | msg['Cc'] = cc_addrs if isinstance(cc_addrs, str) else ', '.join(cc_addrs) 109 | if bcc_addrs: 110 | msg['Bcc'] = bcc_addrs if isinstance(bcc_addrs, str) else ', '.join(bcc_addrs) 111 | msg.set_content(body, subtype=body_type) 112 | 113 | if attachments: 114 | current_app.logger.info("Attachments defined") 115 | if not isinstance(attachments, list): 116 | attachments = [attachments] 117 | for attachment in attachments: 118 | msg.add_attachment(attachment.stream.read(), maintype='multipart', subtype=attachment.content_type) 119 | current_app.logger.info(f"Add attachment {attachment}") 120 | 121 | if attachment_urls: 122 | current_app.logger.info("Attachments urls defined") 123 | for attachment_name, attachment_url in attachment_urls.items(): 124 | response = requests.get(attachment_url) 125 | if response.status_code == 200: 126 | attachment = response.content 127 | maintype, subtype = response.headers['Content-Type'].split('/') 128 | msg.add_attachment(attachment, maintype=maintype, subtype=subtype, 129 | filename=attachment_name) 130 | current_app.logger.info(f"Add attachment {attachment_name} - size {len(attachment)}") 131 | else: 132 | return f"Failed to download attachment, error {response.status_code}.", 400 133 | 134 | except Exception as e: 135 | raise BadRequest(f"Cannot create email message (Error: {str(e)}).") 136 | 137 | # Send email 138 | try: 139 | with smtplib.SMTP(self.smtp_server, port=self.smtp_port) as server: 140 | if starttls: 141 | server.starttls() 142 | server.login(self.smtp_login, self.smtp_passwd) 143 | server.send_message(msg) 144 | 145 | resp = f"Mail sent to {msg['To']}" 146 | current_app.logger.info(resp) 147 | return resp 148 | except smtplib.SMTPAuthenticationError: 149 | raise ConnectionError("Wrong username/password : cannot connect.") 150 | except Exception as e: 151 | raise InternalServerError(f"Cannot send email message (Error: {str(e)}).") 152 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. figure:: ./img/coworks.png 2 | :height: 100px 3 | :alt: Coworks Logo 4 | :target: https://coworks.readthedocs.io/en/latest/?badge=latest 5 | 6 | |Maintenance| |Build Status| |Documentation Status| |Coverage| |Python versions| |Licence| 7 | 8 | .. |Maintenance| image:: https://img.shields.io/badge/Maintained%3F-yes-green.svg?style=plastic 9 | :alt: Maintenance 10 | .. |Documentation Status| image:: https://readthedocs.org/projects/coworks/badge/?version=master&style=plastic 11 | :alt: Documentation Status 12 | .. |Coverage| image:: https://img.shields.io/codecov/c/github/gdoumenc/coworks?style=plastic 13 | :alt: Codecov 14 | .. |Python versions| image:: https://img.shields.io/pypi/pyversions/coworks?style=plastic 15 | :alt: Python Versions 16 | .. |Licence| image:: https://img.shields.io/github/license/gdoumenc/coworks?style=plastic 17 | :alt: Licence 18 | 19 | Introduction 20 | ============ 21 | 22 | CoWorks is a unified serverless microservices framework based on AWS technologies 23 | (`API Gateway `_, `AWS Lambda `_), 24 | the `Flask `_ framework and the `Airflow `_ 25 | plateform. 26 | 27 | **Small technical microservice** 28 | 29 | Each atomic microservice (defined as ``class TechMicroService``) is a simple python class deployed as a serverless 30 | AWS Lambda and can be called synchronously and asynchrously. 31 | 32 | **Functional business service** 33 | 34 | Composition of microservices (defined with the ``@biz`` decorator) is performed over the tech microservices and 35 | constructed by Airflow workflows. 36 | 37 | Technical documentation : 38 | 39 | * Get started: :ref:`installation`. 40 | * Quick overview: :ref:`tech_quickstart` then :ref:`biz_quickstart`. 41 | * The Command Line Interface :ref:`cli`. 42 | * Full documentation: :ref:`doc`. 43 | * At least :ref:`faq` if not enough... 44 | 45 | Using and derived from `Flask `_ 46 | (`Donate to Pallets `_). 47 | Main other tools used: 48 | 49 | * `Click `_ - Command Line Interface Creation Kit. 50 | * `Terraform `_ - Infrastructure Configuration Management Tool. 51 | 52 | Other AWS or Terraform technologies are used for logging, administration, … 53 | 54 | What does microservice mean in CoWorks? 55 | --------------------------------------- 56 | 57 | In short, the microservice architectural style is an approach to developing a single application as a suite of small services, 58 | each running in its own process and communicating with lightweight mechanisms. 59 | 60 | In Microservice Architecture (Aligning Principles, Practices, and Culture), 61 | authors M. Amundsen, I. Nadareishvili, R. Mitra, and M. McLarty add detail to the definition 62 | by outlining traits microservice applications share: 63 | 64 | * Small in size 65 | * Messaging enabled 66 | * Bounded by contexts 67 | * Autonomously developed 68 | * Independently deployable 69 | * Decentralized 70 | * Built and released with automated processes 71 | 72 | In CoWorks, microservices are serverless functions over APIs. 73 | 74 | Small in size 75 | Simply implemented as a Flask python class. 76 | 77 | Messaging enabled 78 | API Gateway request-response managed services. 79 | 80 | Service oriented 81 | Technological service by Flask entry, biz service by Airflow workflow. 82 | 83 | Independently deployable 84 | Serverless component accessed thru API. 85 | 86 | Decentralized 87 | Serverless components managed by workflows. 88 | 89 | Smart endpoints 90 | Deriving directly from class methods. 91 | 92 | 93 | What are CoWorks main benefits? 94 | ------------------------------- 95 | 96 | Two levels of microservice 97 | ************************** 98 | 99 | CoWorks microservices are divided in two categories : 100 | 101 | **Small technical microservice** 102 | 103 | Implemented as a simple AWS Lambda function, this kind of microservice is dedicated to technical 104 | operations over a specific service. Technical miscroservice should be stateless. 105 | 106 | 107 | **Functional business service** 108 | 109 | Implemented by Airflow workflow, this kind of microservice allows non programmer to construct 110 | functional business workflows. 111 | 112 | 113 | Distinction between technical microservice and business service is based not only on granularity size but also: 114 | 115 | * A ``TechMicroservice`` should mainly be used as receivers of orders coming from ``@biz``. 116 | * A ``@biz`` represents a logical workflow of actions while a ``TechMicroservice`` represents a simple concrete action. 117 | * A ``TechMicroservice`` is an independant microservice while a ``@biz`` is connected to event handlers (cron, notification, event, ...). 118 | * A ``TechMicroservice`` is more a handler pattern and ``@biz`` a reactor pattern. 119 | 120 | Code oriented tools 121 | ******************* 122 | 123 | Like any model of software architecture, it is very usefull to have complementary tools for programming, testing, 124 | documenting or deploying over it. 125 | 126 | The main advantage of using CoWorks is its ability to defined those tools, called `commands`, directly in 127 | the microservice code. 128 | Predefined commands like ``run`` (defined by the Flask framework) or ``deploy`` are provided, 129 | but you can redefined them or creates new ones like for documentation or testing. 130 | 131 | For more details, see: :ref:`command`. 132 | 133 | Microservice architecture structuration 134 | *************************************** 135 | 136 | The CoWorks microservice architecture provides some best pratices for code organization and directory structure. 137 | Indeed it's so easy to start in serverless project, it's also easy to start moving the wrong direction. 138 | 139 | **API and Lambda organization** 140 | 141 | With AWS API a single Lambda function handles a single HTTP verb/path combinaison. For Rest API it is better to have 142 | a single lambda function to handle all HTTP verbs for a particular resource. 143 | 144 | CoWorks regroups all microservice entrypoints into one single class. And a class is the resource granularity 145 | for the API. 146 | 147 | For example, following the CRUD design : 148 | 149 | .. figure:: ./img/resource_oriented.png 150 | :width: 800px 151 | 152 | The significant benefit of this architecture is that the number of Lambda functions is drastically reduced over a 153 | one to one CRUD event mapping. 154 | 155 | **Configuration** 156 | 157 | CoWorks differenciates two kind of configurations: 158 | 159 | * Automation and command configuraton 160 | * Execution configuration 161 | 162 | For those who are familiar with the `Twelve-Factor App `_ methodology, 163 | the CoWorks configuration model correspond exactly 164 | with the strict separation of config from code. 165 | 166 | More precisely: 167 | 168 | * The project configuration file : *Use a declarative format for setup automation, to minimize time and cost for new developers joining the project* 169 | * The dotenv file : *Env vars are easy to change between deploys without changing any code* 170 | 171 | .. _doc: 172 | 173 | Documentation 174 | ------------- 175 | 176 | .. toctree:: 177 | :maxdepth: 2 178 | :caption: Contents: 179 | 180 | installation 181 | tech_quickstart 182 | tech 183 | command 184 | configuration 185 | biz_quickstart 186 | biz 187 | samples 188 | api 189 | faq 190 | contributing 191 | changelog 192 | 193 | 194 | Taking part in the project 195 | -------------------------- 196 | 197 | If you want to contribute to this project in any kind, your help will be very welcome. 198 | Don't hesitate to contact any project's member. 199 | 200 | 201 | Indices and tables 202 | ------------------ 203 | 204 | * :ref:`genindex` 205 | * :ref:`modindex` 206 | * :ref:`search` 207 | --------------------------------------------------------------------------------