├── .gitignore ├── README.MD ├── assets ├── alarm_start.png ├── alarm_stop.png ├── api_gateway.png ├── aws_instance_scheduler.png ├── bot_error.png ├── bot_info.png ├── bot_start_stop.png ├── bot_status.png ├── cloud_watch_cron.png ├── dynamo_tables.png ├── incoming_webhook.png ├── instance_tag.png ├── lambda_env.png ├── outgoing_webhook.png └── scheduler_bot.png ├── functions └── awsInstanceScheduler │ ├── function.json │ ├── main.py │ └── requirements.txt └── project.json /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/* 2 | -------------------------------------------------------------------------------- /README.MD: -------------------------------------------------------------------------------- 1 | # AWS Instance Scheduler 2 | 3 | Instance Scheduler는 [EC2 Scheduler](https://aws.amazon.com/ko/answers/infrastructure-management/ec2-scheduler/)의 문제점을 보안한 솔루션으로 사용자가 EC2와 RDS 인스턴스에 대한 사용자 정의 스케쥴을 구성할 수 있게 해줍니다. 4 | 이 솔루션은 AWS 서버 운영비용을 절감하는데 많은 도움을 줍니다. 월~금요일 하루 10시간만 운용한다고 했을 경우 70%의 서버 운영비용을 절감할 수 있습니다. 5 | 6 | ![Instance Scheduler Architecture](assets/aws_instance_scheduler.png) 7 | 8 | ## Features 9 | - 스케쥴링 시간에 따른 Instance Start/Stop 10 | - 스케쥴 Timezone 설정 11 | - 서버 그룹 의존관계 설정 12 | - 스케쥴 예외설정 13 | - 서버 강제 실행 / 중지 14 | - Jandi 메신저로 상태 알람 15 | - 스케쥴 봇 16 | 17 | ## TimeZone 설정 18 | 19 | 기본설정은 Asia/Seoul로 설정되어 있으며 다른 Timezone을 원하면 main.py 파일의 15라인의 타임존 설정값을 변경하세요. 20 | 21 | ``` 22 | os.environ['TZ'] = 'Asia/Seoul' 23 | time.tzset() 24 | ``` 25 | 26 | ## Dynamo DB 설정 27 | 28 | Dynamo DB에 스케쥴 정보를 정의하는 3개의 Table을 생성합니다. 29 | 30 | ![Dynamo Table List](assets/dynamo_tables.png) 31 | 32 | ### 1. Schedule 33 | Schedule Table에는 스케쥴의 기본 정보를 설정한다. 34 | 35 | ```json 36 | { 37 | "ScheduleName": "SampleSchedule", 38 | "TagValue": "SampleScheduleTag", 39 | "DaysActive": "weekdays", 40 | "Enabled": true, 41 | "StartTime": "09:00", 42 | "StopTime": "18:00", 43 | "ForceStart": false 44 | } 45 | ``` 46 | 1. ScheduleName : 스케쥴명 47 | 2. TagValue : Instance Tag 값 48 | 3. DaysActive : 적용 요일 49 | * all : 매일 50 | * weekdays : 월~금 51 | * mon,wed,fri : 월,수,금 (특정 요일 , 구분으로 지정) 52 | 4. Enabled : 스케쥴 활성여부 53 | 5. StartTime : 시작시간 (None, H:M) 54 | 6. StopTime : 중지시간 (None, H:M) 55 | 7. ForceStart : 강제시작여부 56 | 57 | StarTime과 StopTime은 24시간제로 표시하며 None으로 설정시 작동시키지 않습니다. 58 | ForceStart가 true로 설정시 스케쥴 시간이나 Enabled 여부와 상관없이 다음 Lambda가 Trigger 되는 시점에 서버를 시작시키며 서버가 모두 시작되면 자동으로 false로 변경됩니다. 59 | 60 | ### 2. ScheduleServerGroup 61 | ScheduleServerGroup에는 서버그룹과 의존관계를 설정합니다. 62 | 63 | ```json 64 | { 65 | "Dependency": [], 66 | "GroupName": "GROUP1", 67 | "InstanceType": "RDS", 68 | "ScheduleName": "SampleSchedule" 69 | }, 70 | { 71 | "Dependency": [], 72 | "GroupName": "GROUP2", 73 | "InstanceType": "EC2", 74 | "ScheduleName": "SampleSchedule" 75 | }, 76 | { 77 | "Dependency": [ 78 | "GROUP1" 79 | "GROUP2" 80 | ], 81 | "GroupName": "GROUP3", 82 | "InstanceType": "EC2", 83 | "ScheduleName": "SampleSchedule" 84 | } 85 | ``` 86 | 1. Dependency : 서버 그룹 의존관계 87 | 2. GroupName : 서버 그룹명 88 | 3. InstanceType : 인스턴스 타입 (EC2, RDS) 89 | 4. ScheduleName : 스케쥴명 90 | 91 | 위와 같이 설정할 경우 GROUP1, GROUP2 -> GROUP3 순서로 시작하게 됩니다. 92 | GROUP1과 GROUP2는 의존관계가 없기 때문에 처음에 시작하게 되고 GROUP3는 GROUP1과 GROUP2과 시작된 후에 시작하게 됩니다. 93 | 94 | ### 3. ScheduleException 95 | ScheduleException에는 특정일의 스케쥴 변경사항을 설정합니다. 96 | ```json 97 | { 98 | "ExceptionUuid": "414faf09-5f6a-4182-b8fd-65522d7612b2", 99 | "ScheduleName": "SampleSchedule", 100 | "ExceptionDate": "2017-07-10", 101 | "ExceptionType": "stop", 102 | "ExceptionValue": "21:00" 103 | } 104 | ``` 105 | 1. ExceptionUuid : 스케쥴 예외 고유번호 106 | 2. ScheduleName : 스케쥴명 107 | 3. ExceptionDate : 예외발생일 108 | 4. ExceptionType : 예외타입 (start, stop) 109 | 5. ExceptionValue : 시간 (None, H:M) 110 | 111 | ## Instance Tagging 112 | 설정된 스케쥴에 포함시킬 인스턴스를 설정하기 위해서는 각 Instance Tag에 아래와 같이 Tagging을 합니다. 113 | 114 | ![Instance Tagging](assets/instance_tag.png) 115 | 116 | ScheduleName에는 Schedule Table에 설정된 TagValue 값을 Tagging합니다. 117 | 서버 그룹이 존재한다면 ScheduleGroupName 을 추가로 Tagging 합니다. 118 | 서버 그룹이 없으면 ScheduleName 으로 Tagging된 모든 Instance를 동시에 시작 시킵니다. 119 | 120 | ## Lambda Deploy 121 | Apex을 이용하여 배포를 합니다. 자세한 설정은 [Apex Github](https://github.com/apex/apex) 을 참조하세요. 122 | 123 | ``` 124 | $ apex delpoy 125 | ``` 126 | 127 | ## Cloudwatch Event 설정 128 | 129 | Cloudwatch Event를 통해 배포된 Instance Scheduler Lambda 함수를 주기적으로 실행하게 됩니다. 130 | 131 | ![Cloudwatch Event](assets/cloud_watch_cron.png) 132 | 133 | 5분마다 작동되게 설정하였으며 상황에 따라 설정값을 바꾸셔도 됩니다. 134 | 135 | ## Schedule Alarm 136 | 137 | 스케쥴의 알람은 Jandi 메신저의 Incoming Webhook을 이용하여 특정 토픽으로 메시지를 전송합니다. 138 | 139 | ### Incoming Webhook Connect 설정 140 | Jandi 메신저에서 알람을 받을 토픽에서 커넥트 연동하기에서 Incoming Webhook을 추가하고 Webhook Url을 복사합니다. 141 | 142 | ![Jandi Incoming Webhook](assets/incoming_webhook.png) 143 | 144 | ### Lambda Environment 설정 145 | ``` 146 | { 147 | "type": "python3.6", 148 | "description": "Aws Instance Scheduler", 149 | "hooks":{ 150 | "build": "pip install -r requirements.txt -t python_modules" 151 | }, 152 | "environment": { 153 | "WEBHOOK_URL": "<>", 154 | ... 155 | } 156 | } 157 | 158 | ``` 159 | /functions/awsInstanceScheduler/function.json 파일의 environment의 WEBHOOK_URL 값을 설정 후 Lambda를 배포하거나 AWS Console 에서 Lambda 함수의 Code 탭에서 Environment variables 을 직접 설정할 수 있다. 160 | 161 | 1. 서버가 시작될때 알람 162 | 163 | ![Alarm Start](assets/alarm_start.png) 164 | 165 | 2. 서버 중지 10분전 알람 166 | 167 | ![Alarm Stop](assets/alarm_stop.png) 168 | 169 | 170 | ## Schedule Bot 171 | 172 | Schedule Bot은 스케쥴의 예외설정, 강제 서버 실행/중지, 서버 상태 조회 등의 기능을 수행합니다. 173 | Scheduler Bot은 Jandi의 Outgoing Webhook Connect를 이용하여 Api Gateway의 Endpoint를 호출하게되고 Api Gateway와 연결된 Lambda 함수를 작동시켜 명령을 주고 받습니다. 174 | Instance Scheduler Lambda는 Scheduler와 Bot이 하나의 함수로 되어 있으며 Lambda가 Trigger 될때 Api Gateway를 통해서 실행되면 Bot이 작동하며 Cloudwatch event를 통해서 작동하게 되면 Scheduler가 작동하게 된다. 175 | 176 | ![Jandi Incoming Webhook](assets/scheduler_bot.png) 177 | 178 | ### Api Gateway 와 Lambda 연결 179 | 180 | Jandi 메신저를 통해 Lambda를 실행시키기 위해서는 Lambda함수와 API Gateway를 연결하여 Http Endpoint를 생성해야 합니다. 181 | 182 | ![Api Gateway](assets/api_gateway.png) 183 | 184 | Endpoint에 POST Method와 Lambda함수를 연결한뒤 단계를 생성한뒤 해당 Method의 Endpoint URL을 복사합니다. 185 | 186 | ### Outgoing Webhook Connect 설정 187 | 188 | ![Outgoing Webhook](assets/outgoing_webhook.png) 189 | 190 | Jandi 메신저의 Outgoing Webhook Connect를 생성 후 위 Api Gateway에서 발급받은 Endpoint URL을 Outgoing Webhook URL에 붙여넣기 합니다. 191 | 그리고 토큰값을 복사하여 Lambda Environment의 OUTGOING_WEBHOOK_TOKEN에 설정합니다. 192 | 193 | ``` 194 | { 195 | "type": "python3.6", 196 | "description": "Aws Instance Scheduler", 197 | "hooks":{ 198 | "build": "pip install -r requirements.txt -t python_modules" 199 | }, 200 | "environment": { 201 | ... 202 | "OUTGOING_WEBHOOK_TOKEN": "<>" 203 | } 204 | } 205 | 206 | ``` 207 | 208 | ### Bot Command 209 | 210 | Schedule Bot은 다음과 같은 명령어를 수행합니다. 211 | 212 | ``` 213 | /서버 help : 도움말 214 | /서버 [스케쥴명] status : 현재 서버 상태 조회 215 | /서버 [스케쥴명] info : 오늘의 스케쥴 조회 216 | /서버 [스케쥴명] info [YYYY-MM-DD] : 특정일 스케쥴 조회 217 | /서버 [스케쥴명] exception info : 오늘의 스케쥴 예외 조회 218 | /서버 [스케쥴명] exception info [YYYY-MM-DD] : 특정일 스케쥴 예외 조회 219 | /서버 [스케쥴명] exception set [YYYY-MM-DD] [start|stop] [h:m] : 예외 설정 220 | /서버 [스케쥴명] exception del [YYYY-MM-DD] [start|stop] : 예외 삭제 221 | /서버 [스케쥴명] force_start : 서버 강제실행 222 | /서버 [스케쥴명] force_stop : 서버 강제중지 223 | ``` 224 | 225 | 1. 스케쥴 정보 조회 226 | 227 | ![Bot Info](assets/bot_info.png) 228 | 229 | 2. 서버 상태 조회 230 | 231 | ![Bot Status](assets/bot_status.png) 232 | 233 | 3. 서버 강제 시작 / 중지 234 | 235 | ![Bot Start Stop](assets/bot_start_stop.png) 236 | 237 | 4. 명령어 오류 238 | 239 | ![Bot Error](assets/bot_error.png) 240 | -------------------------------------------------------------------------------- /assets/alarm_start.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/alarm_start.png -------------------------------------------------------------------------------- /assets/alarm_stop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/alarm_stop.png -------------------------------------------------------------------------------- /assets/api_gateway.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/api_gateway.png -------------------------------------------------------------------------------- /assets/aws_instance_scheduler.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/aws_instance_scheduler.png -------------------------------------------------------------------------------- /assets/bot_error.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/bot_error.png -------------------------------------------------------------------------------- /assets/bot_info.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/bot_info.png -------------------------------------------------------------------------------- /assets/bot_start_stop.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/bot_start_stop.png -------------------------------------------------------------------------------- /assets/bot_status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/bot_status.png -------------------------------------------------------------------------------- /assets/cloud_watch_cron.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/cloud_watch_cron.png -------------------------------------------------------------------------------- /assets/dynamo_tables.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/dynamo_tables.png -------------------------------------------------------------------------------- /assets/incoming_webhook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/incoming_webhook.png -------------------------------------------------------------------------------- /assets/instance_tag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/instance_tag.png -------------------------------------------------------------------------------- /assets/lambda_env.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/lambda_env.png -------------------------------------------------------------------------------- /assets/outgoing_webhook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/outgoing_webhook.png -------------------------------------------------------------------------------- /assets/scheduler_bot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tosslab/aws_instance_scheduler/9d1a3197a39e19bf683624a423912d8a06721fa1/assets/scheduler_bot.png -------------------------------------------------------------------------------- /functions/awsInstanceScheduler/function.json: -------------------------------------------------------------------------------- 1 | { 2 | "type": "python3.6", 3 | "description": "Aws Instance Scheduler", 4 | "hooks":{ 5 | "build": "pip install -r requirements.txt -t python_modules" 6 | }, 7 | "environment": { 8 | "WEBHOOK_URL": "<>", 9 | "STOP_ALERT_BEFORE_TIME_MINUTE": "10", 10 | "OUTGOING_WEBHOOK_TOKEN": "<>" 11 | } 12 | } 13 | -------------------------------------------------------------------------------- /functions/awsInstanceScheduler/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | sys.path.append('./python_modules') 3 | 4 | import os 5 | import time 6 | import uuid 7 | import traceback 8 | import requests 9 | import json 10 | from datetime import datetime, timedelta 11 | 12 | import boto3 13 | from boto3.dynamodb.conditions import Attr 14 | 15 | os.environ['TZ'] = 'Asia/Seoul' 16 | time.tzset() 17 | 18 | WEBHOOK_URL = os.environ['WEBHOOK_URL'] 19 | OUTGOING_WEBHOOK_TOKEN = os.environ['OUTGOING_WEBHOOK_TOKEN'] 20 | STOP_ALERT_BEFORE_TIME_MINUTE = int(os.environ['STOP_ALERT_BEFORE_TIME_MINUTE']) 21 | 22 | 23 | class ScheduleUtil: 24 | @staticmethod 25 | def get_ec2_instance_list_by_status(ec2_instance_list, status) -> list: 26 | filtered_ec2_instance_list = [] 27 | 28 | for ec2_instance in ec2_instance_list: 29 | if ScheduleUtil.get_ec2_instance_status(ec2_instance) == status: 30 | filtered_ec2_instance_list.append(ec2_instance) 31 | 32 | return filtered_ec2_instance_list 33 | 34 | @staticmethod 35 | def get_ec2_instance_ids(ec2_instance_list) -> list: 36 | ec2_instance_ids = [] 37 | 38 | for ec2_instance in ec2_instance_list: 39 | ec2_instance_ids.append(ec2_instance['InstanceId']) 40 | 41 | return ec2_instance_ids 42 | 43 | @staticmethod 44 | def get_ec2_instance_status(ec2_instance) -> str: 45 | return ec2_instance['State']['Name'] 46 | 47 | @staticmethod 48 | def get_ec2_instance_name(ec2_instance, default_name='EC2') -> str: 49 | tag_list = ec2_instance['Tags'] 50 | 51 | for tag in tag_list: 52 | if tag['Key'] == 'Name': 53 | return tag['Value'] 54 | 55 | return default_name 56 | 57 | @staticmethod 58 | def get_rds_instance_list_by_status(rds_instance_list, status) -> list: 59 | filtered_rds_instance_list = [] 60 | 61 | for rds_instance in rds_instance_list: 62 | if ScheduleUtil.get_rds_instance_status(rds_instance) == status: 63 | filtered_rds_instance_list.append(rds_instance) 64 | 65 | return filtered_rds_instance_list 66 | 67 | @staticmethod 68 | def get_rds_instance_ids(rds_instance_list) -> list: 69 | rds_instance_ids = [] 70 | 71 | for rds_instance in rds_instance_list: 72 | rds_instance_ids.append(rds_instance['DBInstanceIdentifier']) 73 | 74 | return rds_instance_ids 75 | 76 | @staticmethod 77 | def get_rds_instance_status(rds_instance) -> str: 78 | return rds_instance['DBInstanceStatus'] 79 | 80 | @staticmethod 81 | def get_rds_instance_name(rds_instance, default_name='RDS') -> str: 82 | name = rds_instance['DBInstanceIdentifier'] 83 | 84 | return default_name if not name else name 85 | 86 | @staticmethod 87 | def equals_rds_schedule_name(rds_tags, tag_value) -> bool: 88 | for t in rds_tags['TagList']: 89 | if t['Key'] != 'ScheduleName': 90 | continue 91 | if t['Value'] == tag_value: 92 | return True 93 | 94 | return False 95 | 96 | @staticmethod 97 | def get_diff_minute(d1, d2) -> float: 98 | diff = d1 - d2 99 | return (diff.days * 24 * 60) + (diff.seconds / 60) 100 | 101 | @staticmethod 102 | def hm_to_date_time(hm): 103 | cur = datetime.now() 104 | hour = hm.split(':')[0] 105 | minute = hm.split(':')[1] 106 | date_time = cur.replace(hour=int(hour), minute=int(minute), second=0, microsecond=0) 107 | 108 | return date_time 109 | 110 | @staticmethod 111 | def replace_time(date_time, hm): 112 | split_time = hm.split(':') 113 | hour = split_time[0] 114 | minute = split_time[1] 115 | return date_time.replace(hour=int(hour), minute=int(minute)) 116 | 117 | @staticmethod 118 | def is_valid_time(hm): 119 | try: 120 | datetime.strptime(hm, '%H:%M') 121 | return True 122 | except ValueError: 123 | return False 124 | 125 | @staticmethod 126 | def is_valid_date(ymd): 127 | try: 128 | datetime.strptime(ymd, '%Y-%m-%d') 129 | return True 130 | except ValueError: 131 | return False 132 | 133 | @staticmethod 134 | def equals_rds_schedule_group_name(rds_tags, server_group) -> bool: 135 | for t in rds_tags['TagList']: 136 | if t['Key'] != 'ScheduleGroupName': 137 | continue 138 | if t['Value'] == server_group['GroupName']: 139 | return True 140 | 141 | return False 142 | 143 | 144 | class JandiWebhook: 145 | color_err = '#FF0000' 146 | color_ok = '#1DDB16' 147 | color_warning = '#FFBB00' 148 | 149 | @staticmethod 150 | def build_connect_info(title, description=None, image_url=None): 151 | connect_info = { 152 | 'title': title 153 | } 154 | 155 | if description is not None: 156 | connect_info['description'] = description 157 | 158 | if image_url is not None: 159 | connect_info['image_url'] = image_url 160 | 161 | return connect_info 162 | 163 | @staticmethod 164 | def build_message(msg, color, connect_info_list=None): 165 | payload = { 166 | 'body': msg, 167 | 'connectColor': color, 168 | } 169 | 170 | if connect_info_list is not None: 171 | payload['connectInfo'] = connect_info_list 172 | 173 | return payload 174 | 175 | @staticmethod 176 | def send_message(msg, color, connect_info_list=None): 177 | try: 178 | print("Send Webhook Message") 179 | 180 | headers = { 181 | 'Accept': 'application/vnd.tosslab.jandi-v2+json', 182 | 'Content-Type': 'application/json' 183 | } 184 | 185 | payload = json.dumps(JandiWebhook.build_message(msg, color, connect_info_list)) 186 | 187 | print('body : ' + payload) 188 | 189 | res = requests.post(WEBHOOK_URL, headers=headers, data=payload) 190 | 191 | print('response : ' + str(res.status_code)) 192 | res.raise_for_status() 193 | except Exception as e: 194 | print(e) 195 | 196 | @staticmethod 197 | def send_err_message(msg, connect_info_list=None): 198 | JandiWebhook.send_message(msg, JandiWebhook.color_err, connect_info_list) 199 | 200 | @staticmethod 201 | def send_warning_message(msg, connect_info_list=None): 202 | JandiWebhook.send_message(msg, JandiWebhook.color_warning, connect_info_list) 203 | 204 | @staticmethod 205 | def send_ok_message(msg, connect_info_list=None): 206 | JandiWebhook.send_message(msg, JandiWebhook.color_ok, connect_info_list) 207 | 208 | @staticmethod 209 | def send_start_rds_server_message(schedule, start_rds_instance_list): 210 | JandiWebhook.send_ok_message( 211 | '{0} 스케쥴 RDS 서버를 시작 합니다'.format(schedule['ScheduleName']), 212 | JandiWebhook.get_rds_server_connect_info_list('RDS 시작 서버 목록', start_rds_instance_list)) 213 | 214 | @staticmethod 215 | def send_start_ec2_server_message(schedule, start_ec2_instance_list): 216 | JandiWebhook.send_ok_message( 217 | '{0} 스케쥴 EC2 서버를 시작 합니다'.format(schedule['ScheduleName']), 218 | JandiWebhook.get_ec2_server_connect_info_list('EC2 시작 서버 목록', start_ec2_instance_list)) 219 | 220 | @staticmethod 221 | def send_start_rds_server_group_message(schedule, server_group, start_rds_instance_list): 222 | JandiWebhook.send_ok_message( 223 | '{0} 스케쥴 {1} 서버 그룹을 시작합니다'.format(schedule['ScheduleName'], server_group['GroupName']), 224 | JandiWebhook.get_rds_server_connect_info_list('RDS 시작 서버 목록', start_rds_instance_list)) 225 | 226 | @staticmethod 227 | def send_start_ec2_server_group_message(schedule, server_group, start_ec2_instance_list): 228 | JandiWebhook.send_ok_message( 229 | '{0} 스케쥴의 {1} 서버 그룹을 시작합니다'.format(schedule['ScheduleName'], server_group['GroupName']), 230 | JandiWebhook.get_ec2_server_connect_info_list('EC2 시작 서버 목록', start_ec2_instance_list)) 231 | 232 | @staticmethod 233 | def send_stop_ec2_server_message(schedule, stop_ec2_instance_list): 234 | JandiWebhook.send_ok_message( 235 | '{0} 스케쥴의 EC2 서버를 중지합니다'.format(schedule['ScheduleName']), 236 | JandiWebhook.get_ec2_server_connect_info_list('EC2 중지 서버 목록', stop_ec2_instance_list)) 237 | 238 | @staticmethod 239 | def send_stop_rds_server_message(schedule, stop_rds_instance_list): 240 | JandiWebhook.send_ok_message( 241 | '{0} 스케쥴의 RDS 서버를 중지합니다'.format(schedule['ScheduleName']), 242 | JandiWebhook.get_rds_server_connect_info_list('RDS 중지 서버 목록', stop_rds_instance_list)) 243 | 244 | @staticmethod 245 | def send_stop_alert_message(schedule_name, stop_date_time, remain): 246 | stop_alert_msg = '잠시후 {0} 스케쥴의 모든 서버가 중지 됩니다.'.format(schedule_name) 247 | stop_time_msg = JandiWebhook.build_connect_info('중지 시간', str(stop_date_time)) 248 | remain_time_msg = JandiWebhook.build_connect_info('남은 시간', str(remain) + '분') 249 | JandiWebhook.send_warning_message(stop_alert_msg, [stop_time_msg, remain_time_msg]) 250 | 251 | @staticmethod 252 | def send_exception_err_message(schedule, e, error_stack): 253 | connect_info = JandiWebhook.build_connect_info(str(e), error_stack) 254 | JandiWebhook.send_err_message('{0} 스케쥴에서 에러가 발생하였습니다'.format(schedule['ScheduleName']), [connect_info]) 255 | 256 | @staticmethod 257 | def get_rds_server_connect_info_list(title, start_rds_instance_list) -> list: 258 | 259 | server_list = [] 260 | 261 | for start_rds_instance in start_rds_instance_list: 262 | instance_name = start_rds_instance['DBInstanceIdentifier'] 263 | engine = start_rds_instance['Engine'] 264 | availability_zone = start_rds_instance['AvailabilityZone'] 265 | server_list.append(instance_name + ' (' + engine + ') : ' + availability_zone) 266 | 267 | connect_info = JandiWebhook.build_connect_info(title, '\n'.join(server_list)) 268 | 269 | return [connect_info] 270 | 271 | @staticmethod 272 | def get_ec2_server_connect_info_list(title, start_ec2_instance_list) -> list: 273 | 274 | server_list = [] 275 | 276 | for start_ec2_instance in start_ec2_instance_list: 277 | instance_name = ScheduleUtil.get_ec2_instance_name(start_ec2_instance) 278 | availability_zone = start_ec2_instance['Placement']['AvailabilityZone'] 279 | instance_id = start_ec2_instance['InstanceId'] 280 | server_list.append(instance_name + ' (' + instance_id + ') : ' + availability_zone) 281 | 282 | connect_info = JandiWebhook.build_connect_info(title, '\n'.join(server_list)) 283 | 284 | return [connect_info] 285 | 286 | 287 | class Schedule: 288 | schedule_name = None 289 | schedule_data = {} 290 | 291 | db = boto3.resource('dynamodb') 292 | ec2 = boto3.client('ec2') 293 | rds = boto3.client('rds') 294 | 295 | def __init__(self, schedule_name): 296 | self.schedule_name = schedule_name 297 | 298 | def load_schedule_item_from_db(self): 299 | table = self.db.Table('Schedule') 300 | response = table.get_item( 301 | Key={ 302 | 'ScheduleName': self.schedule_name 303 | } 304 | ) 305 | 306 | return response['Item'] if 'Item' in response else {} 307 | 308 | def get_schedule_property(self, property_name, default_value=None): 309 | if property_name in self.get_schedule(): 310 | return self.get_schedule()[property_name] 311 | else: 312 | return default_value 313 | 314 | def get_schedule(self): 315 | if not self.schedule_data: 316 | self.schedule_data = self.load_schedule_item_from_db() 317 | 318 | return self.schedule_data 319 | 320 | def get_start_date_time(self) -> datetime: 321 | start_time = self.get_schedule_property('StartTime') 322 | 323 | if start_time == 'None': 324 | return None 325 | 326 | start_date_time = ScheduleUtil.hm_to_date_time(start_time) 327 | 328 | return start_date_time 329 | 330 | def get_stop_date_time(self) -> datetime: 331 | stop_time = self.get_schedule_property('StopTime') 332 | 333 | if stop_time == 'None': 334 | return None 335 | 336 | stop_date_time = ScheduleUtil.hm_to_date_time(stop_time) 337 | 338 | return stop_date_time 339 | 340 | def set_schedule_force_start(self, flag): 341 | return self.db.Table('Schedule').update_item( 342 | Key={ 343 | 'ScheduleName': self.schedule_name, 344 | }, 345 | UpdateExpression='set ForceStart=:c', 346 | ExpressionAttributeValues={ 347 | ':c': flag 348 | }, 349 | ReturnValues='UPDATED_NEW' 350 | ) 351 | 352 | def is_enable(self) -> bool: 353 | return self.get_schedule_property('Enabled') 354 | 355 | def is_active_day(self) -> bool: 356 | days_active = self.get_schedule_property('DaysActive') 357 | current_week_day = datetime.now().strftime('%a').lower() 358 | 359 | # 매일 작동 360 | if days_active == 'all': 361 | return True 362 | # 월~금만 작동 363 | elif days_active == 'weekdays': 364 | weekdays = ['mon', 'tue', 'wed', 'thu', 'fri'] 365 | if current_week_day in weekdays: 366 | return True 367 | # 지정요일만 작동 (ex : mon,tue) 368 | else: 369 | days = days_active.split(',') 370 | for d in days: 371 | if d.lower().strip() == current_week_day: 372 | return True 373 | 374 | return False 375 | 376 | def is_start_time(self) -> bool: 377 | if not self.is_active_day(): 378 | return False 379 | 380 | start = self.get_start_date_time() 381 | 382 | if start is None: 383 | return False 384 | 385 | now = datetime.now() 386 | now_max = now - timedelta(minutes=59) 387 | 388 | return now_max <= start <= now 389 | 390 | def is_stop_time(self) -> bool: 391 | if not self.is_active_day(): 392 | return False 393 | 394 | stop = self.get_stop_date_time() 395 | 396 | if stop is None: 397 | return False 398 | 399 | now = datetime.now() 400 | now_max = now - timedelta(minutes=59) 401 | 402 | return now_max <= stop <= now 403 | 404 | def is_force_start(self) -> bool: 405 | return self.get_schedule_property('ForceStart', False) 406 | 407 | def has_running_instance(self): 408 | ec2_instance_list = self.get_ec2_instance_list() 409 | rds_instance_list = self.get_rds_instance_list() 410 | 411 | running_ec2_instance_list = ScheduleUtil.get_ec2_instance_list_by_status(ec2_instance_list, 'running') 412 | running_rds_instance_list = ScheduleUtil.get_rds_instance_list_by_status(rds_instance_list, 'available') 413 | 414 | return False if len(running_ec2_instance_list) == 0 and len(running_rds_instance_list) == 0 else True 415 | 416 | def get_ec2_instance_list(self) -> list: 417 | 418 | schedule_tag_name = self.get_schedule_property('TagValue') 419 | 420 | ec2_schedule_filter = [{ 421 | 'Name': 'tag:ScheduleName', 422 | 'Values': [schedule_tag_name] 423 | }] 424 | 425 | instances = self.ec2.describe_instances(Filters=ec2_schedule_filter) 426 | 427 | reservation_list = instances['Reservations'] 428 | 429 | instance_list = [] 430 | 431 | for reservation in reservation_list: 432 | instance_list = instance_list + reservation['Instances'] 433 | 434 | return instance_list 435 | 436 | def get_rds_instance_list(self) -> list: 437 | 438 | schedule_tag_value = self.get_schedule_property('TagValue') 439 | instances = self.rds.describe_db_instances() 440 | 441 | schedule_instances_list = [] 442 | 443 | for instance in instances['DBInstances']: 444 | arn = instance['DBInstanceArn'] 445 | tags = self.rds.list_tags_for_resource(ResourceName=arn) 446 | 447 | if ScheduleUtil.equals_rds_schedule_name(tags, schedule_tag_value): 448 | schedule_instances_list.append(instance) 449 | 450 | return schedule_instances_list 451 | 452 | def start_ec2_instances(self, ec2_instance_list): 453 | ec2_instance_ids = ScheduleUtil.get_ec2_instance_ids(ec2_instance_list) 454 | return self.ec2.start_instances(InstanceIds=ec2_instance_ids) 455 | 456 | def stop_ec2_instances(self, ec2_instance_list): 457 | ec2_instance_ids = ScheduleUtil.get_ec2_instance_ids(ec2_instance_list) 458 | return self.ec2.stop_instances(InstanceIds=ec2_instance_ids, Force=True) 459 | 460 | def start_rds_instances(self, rds_instance_list): 461 | response_list = [] 462 | rds_instance_ids = ScheduleUtil.get_rds_instance_ids(rds_instance_list) 463 | 464 | for rdb_instance_id in rds_instance_ids: 465 | res = self.rds.start_db_instance(DBInstanceIdentifier=rdb_instance_id) 466 | print('Start RDS :' + str(rdb_instance_id)) 467 | response_list.append(res) 468 | 469 | return response_list 470 | 471 | def stop_rds_instances(self, rds_instance_list): 472 | response_list = [] 473 | rds_instance_ids = ScheduleUtil.get_rds_instance_ids(rds_instance_list) 474 | 475 | for rdb_instance_id in rds_instance_ids: 476 | res = self.rds.stop_db_instance(DBInstanceIdentifier=rdb_instance_id) 477 | print('Stop RDS : ' + str(rdb_instance_id)) 478 | response_list.append(res) 479 | 480 | return response_list 481 | 482 | def check_remain_stop_time(self): 483 | if not self.is_active_day(): 484 | return 485 | 486 | stop_date_time = self.get_stop_date_time() 487 | 488 | if stop_date_time is None: 489 | return 490 | 491 | now = datetime.now() 492 | remain = int(round(ScheduleUtil.get_diff_minute(stop_date_time, now))) 493 | 494 | if not self.has_running_instance(): 495 | return 496 | 497 | if 0 < remain <= STOP_ALERT_BEFORE_TIME_MINUTE: 498 | JandiWebhook.send_stop_alert_message(self.schedule_name, stop_date_time, remain) 499 | 500 | def start(self, is_force=False): 501 | 502 | if not self.is_start_time() and not is_force: 503 | return 504 | 505 | ec2_instance_list = self.get_ec2_instance_list() 506 | rds_instance_list = self.get_rds_instance_list() 507 | 508 | start_ec2_instance_list = ScheduleUtil.get_ec2_instance_list_by_status(ec2_instance_list, 'stopped') 509 | start_rds_instance_list = ScheduleUtil.get_rds_instance_list_by_status(rds_instance_list, 'stopped') 510 | 511 | if len(start_ec2_instance_list) > 0: 512 | try: 513 | self.start_ec2_instances(start_ec2_instance_list) 514 | JandiWebhook.send_start_ec2_server_message(self.get_schedule(), start_ec2_instance_list) 515 | except Exception as e: 516 | JandiWebhook.send_exception_err_message(self.get_schedule(), e, traceback.format_exc()) 517 | 518 | if len(start_rds_instance_list) > 0: 519 | try: 520 | self.start_rds_instances(start_rds_instance_list) 521 | JandiWebhook.send_start_rds_server_message(self.get_schedule(), start_rds_instance_list) 522 | except Exception as e: 523 | JandiWebhook.send_exception_err_message(self.get_schedule(), e, traceback.format_exc()) 524 | 525 | if is_force and len(start_ec2_instance_list) == 0 and len(start_rds_instance_list) == 0: 526 | self.set_schedule_force_start(False) 527 | 528 | def stop(self, is_force=False): 529 | 530 | if not self.is_stop_time() and not is_force: 531 | return 532 | 533 | ec2_instance_list = self.get_ec2_instance_list() 534 | rds_instance_list = self.get_rds_instance_list() 535 | 536 | stop_ec2_instance_list = ScheduleUtil.get_ec2_instance_list_by_status(ec2_instance_list, 'running') 537 | stop_rds_instance_list = ScheduleUtil.get_rds_instance_list_by_status(rds_instance_list, 'available') 538 | 539 | if len(stop_ec2_instance_list) > 0: 540 | try: 541 | self.stop_ec2_instances(stop_ec2_instance_list) 542 | JandiWebhook.send_stop_ec2_server_message(self.get_schedule(), stop_ec2_instance_list) 543 | except Exception as e: 544 | JandiWebhook.send_exception_err_message(self.get_schedule(), e, traceback.format_exc()) 545 | 546 | if len(stop_rds_instance_list) > 0: 547 | try: 548 | JandiWebhook.send_stop_rds_server_message(self.get_schedule(), stop_rds_instance_list) 549 | self.stop_rds_instances(stop_rds_instance_list) 550 | except Exception as e: 551 | JandiWebhook.send_exception_err_message(self.get_schedule(), e, traceback.format_exc()) 552 | 553 | def run(self): 554 | 555 | if self.is_force_start(): 556 | self.start(True) 557 | return 558 | 559 | if not self.is_enable(): 560 | return 561 | 562 | self.check_remain_stop_time() 563 | 564 | self.start() 565 | self.stop() 566 | 567 | 568 | class GroupSchedule(Schedule): 569 | schedule_server_group_list = [] 570 | 571 | def __init__(self, schedule_name): 572 | super().__init__(schedule_name) 573 | 574 | def load_schedule_server_group_list_from_db(self) -> list: 575 | table = self.db.Table('ScheduleServerGroup') 576 | 577 | response = table.scan( 578 | Select='ALL_ATTRIBUTES', 579 | FilterExpression=Attr('ScheduleName').eq(self.schedule_name) 580 | ) 581 | 582 | return response['Items'] 583 | 584 | def get_schedule_server_group_list(self) -> list: 585 | if not self.schedule_server_group_list: 586 | self.schedule_server_group_list = self.load_schedule_server_group_list_from_db() 587 | 588 | return self.schedule_server_group_list 589 | 590 | def get_schedule_server_group(self, group_name): 591 | server_group_list = self.get_schedule_server_group_list() 592 | 593 | for server_group in server_group_list: 594 | if server_group['GroupName'] == group_name: 595 | return server_group 596 | 597 | return None 598 | 599 | def get_server_group_ec2_instance_list(self, server_group) -> list: 600 | schedule_tag_name = self.get_schedule_property('TagValue') 601 | 602 | ec2_schedule_filter = [{ 603 | 'Name': 'tag:ScheduleName', 604 | 'Values': [schedule_tag_name] 605 | }, { 606 | 'Name': 'tag:ScheduleGroupName', 607 | 'Values': [server_group['GroupName']] 608 | }] 609 | instances = self.ec2.describe_instances(Filters=ec2_schedule_filter) 610 | 611 | reservation_list = instances['Reservations'] 612 | 613 | instance_list = [] 614 | 615 | for reservation in reservation_list: 616 | instance_list = instance_list + reservation['Instances'] 617 | 618 | return instance_list 619 | 620 | def get_server_group_rds_instance_list(self, server_group) -> list: 621 | schedule_tag_value = self.get_schedule_property('TagValue') 622 | instances = self.rds.describe_db_instances() 623 | 624 | schedule_instances_list = [] 625 | 626 | for instance in instances['DBInstances']: 627 | arn = instance['DBInstanceArn'] 628 | tags = self.rds.list_tags_for_resource(ResourceName=arn) 629 | 630 | if False not in \ 631 | (ScheduleUtil.equals_rds_schedule_name(tags, schedule_tag_value), 632 | ScheduleUtil.equals_rds_schedule_group_name(tags, server_group)): 633 | schedule_instances_list.append(instance) 634 | 635 | return schedule_instances_list 636 | 637 | def get_server_group_instance_list(self, server_group) -> list: 638 | if server_group is None: 639 | return [] 640 | 641 | if server_group['InstanceType'] == 'RDS': 642 | return self.get_server_group_rds_instance_list(server_group) 643 | elif server_group['InstanceType'] == 'EC2': 644 | return self.get_server_group_ec2_instance_list(server_group) 645 | else: 646 | return [] 647 | 648 | def has_server_group(self) -> bool: 649 | return False if not self.get_schedule_server_group_list() else True 650 | 651 | def is_server_group_running(self, server_group) -> bool: 652 | instance_list = self.get_server_group_instance_list(server_group) 653 | 654 | if len(instance_list) is 0: 655 | return True 656 | else: 657 | for instance in instance_list: 658 | if server_group['InstanceType'] == 'EC2': 659 | if ScheduleUtil.get_ec2_instance_status(instance) != 'running': 660 | return False 661 | elif server_group['InstanceType'] == 'RDS': 662 | if ScheduleUtil.get_rds_instance_status(instance) != 'available': 663 | return False 664 | 665 | return True 666 | 667 | def is_dependency_server_group_all_running(self, server_group) -> bool: 668 | dependency_list = server_group['Dependency'] 669 | 670 | if len(dependency_list) > 0: 671 | for dependency in dependency_list: 672 | dependency_server_group = self.get_schedule_server_group(dependency) 673 | 674 | if not self.is_server_group_running(dependency_server_group): 675 | return False 676 | 677 | return True 678 | 679 | def start_server_group_instance(self, server_group): 680 | if server_group['InstanceType'] == 'EC2': 681 | ec2_instance_list = self.get_server_group_ec2_instance_list(server_group) 682 | start_ec2_instance_list = ScheduleUtil.get_ec2_instance_list_by_status(ec2_instance_list, 'stopped') 683 | 684 | if len(start_ec2_instance_list) > 0: 685 | try: 686 | JandiWebhook.send_start_ec2_server_group_message(self.get_schedule(), server_group, 687 | ec2_instance_list) 688 | start_ec2_response = self.start_ec2_instances(start_ec2_instance_list) 689 | return start_ec2_response 690 | except Exception as e: 691 | JandiWebhook.send_exception_err_message(self.get_schedule(), e, traceback.format_exc()) 692 | 693 | elif server_group['InstanceType'] == 'RDS': 694 | rds_instance_list = self.get_server_group_rds_instance_list(server_group) 695 | start_rds_instance_list = ScheduleUtil.get_rds_instance_list_by_status(rds_instance_list, 'stopped') 696 | 697 | if len(start_rds_instance_list) > 0: 698 | try: 699 | JandiWebhook.send_start_rds_server_group_message(self.get_schedule(), server_group, 700 | start_rds_instance_list) 701 | start_rds_response_list = self.start_rds_instances(start_rds_instance_list) 702 | return start_rds_response_list 703 | except Exception as e: 704 | JandiWebhook.send_exception_err_message(self.get_schedule(), e, traceback.format_exc()) 705 | 706 | def stop_server_group_instance(self, server_group): 707 | if server_group['InstanceType'] == 'EC2': 708 | ec2_instance_list = self.get_server_group_ec2_instance_list(server_group) 709 | stop_ec2_instance_list = ScheduleUtil.get_ec2_instance_list_by_status(ec2_instance_list, 'running') 710 | self.stop_ec2_instances(stop_ec2_instance_list) 711 | 712 | elif server_group['InstanceType'] == 'RDS': 713 | rds_instance_list = self.get_server_group_ec2_instance_list(server_group) 714 | stop_rds_instance_list = ScheduleUtil.get_rds_instance_list_by_status(rds_instance_list, 'available') 715 | self.stop_rds_instances(stop_rds_instance_list) 716 | 717 | def start(self, is_force=False): 718 | 719 | if not self.get_schedule_server_group_list(): 720 | super().start(is_force) 721 | return 722 | 723 | if not self.is_start_time() and not is_force: 724 | return 725 | 726 | is_all_server_group_running = True 727 | server_group_list = self.get_schedule_server_group_list() 728 | 729 | for server_group in server_group_list: 730 | is_dependency_started = self.is_dependency_server_group_all_running(server_group) 731 | 732 | if is_dependency_started: 733 | try: 734 | self.start_server_group_instance(server_group) 735 | except Exception as e: 736 | JandiWebhook.send_exception_err_message(self.get_schedule(), e, traceback.format_exc()) 737 | else: 738 | is_all_server_group_running = False 739 | print(server_group['GroupName'] + ' 의 의존관계 ' + str(server_group['Dependency']) + ' 가 아직 시작하지 않았습니다') 740 | 741 | if is_force and is_all_server_group_running: 742 | self.set_schedule_force_start(False) 743 | 744 | 745 | class ExceptionSchedule(GroupSchedule): 746 | schedule_exception_list = None 747 | 748 | def __init__(self, schedule_name): 749 | super().__init__(schedule_name) 750 | 751 | def load_schedule_exception_list_from_db(self): 752 | schedule_date_ymd = self.get_exception_date_ymd() 753 | table = self.db.Table('ScheduleException') 754 | response = table.scan( 755 | Select='ALL_ATTRIBUTES', 756 | FilterExpression=Attr('ScheduleName').eq(self.schedule_name) & Attr('ExceptionDate').eq(schedule_date_ymd) 757 | ) 758 | 759 | return response['Items'] 760 | 761 | def get_exception_date_ymd(self) -> str: 762 | return datetime.now().strftime('%Y-%m-%d') 763 | 764 | def get_schedule_exception_list(self) -> list: 765 | if self.schedule_exception_list is None: 766 | self.schedule_exception_list = self.load_schedule_exception_list_from_db() 767 | 768 | return [] if self.schedule_exception_list is None else self.schedule_exception_list 769 | 770 | def get_exception_value(self, exception_type): 771 | exception_list = self.get_schedule_exception_list() 772 | 773 | for exception in exception_list: 774 | if exception['ExceptionType'] == exception_type: 775 | return exception['ExceptionValue'] 776 | 777 | return None 778 | 779 | def get_start_date_time(self): 780 | origin_start_date_time = super(ExceptionSchedule, self).get_start_date_time() 781 | 782 | start_exception_value = self.get_exception_value('start') 783 | 784 | if start_exception_value is None: 785 | return origin_start_date_time 786 | elif start_exception_value == 'None': 787 | return None 788 | else: 789 | if origin_start_date_time is None: 790 | return ScheduleUtil.hm_to_date_time(start_exception_value) 791 | else: 792 | return ScheduleUtil.replace_time(origin_start_date_time, start_exception_value) 793 | 794 | def get_stop_date_time(self): 795 | origin_stop_date_time = super(ExceptionSchedule, self).get_stop_date_time() 796 | 797 | stop_exception_value = self.get_exception_value('stop') 798 | 799 | if stop_exception_value is None: 800 | return origin_stop_date_time 801 | elif stop_exception_value == 'None': 802 | return None 803 | else: 804 | if origin_stop_date_time is None: 805 | return ScheduleUtil.hm_to_date_time(stop_exception_value) 806 | else: 807 | return ScheduleUtil.replace_time(origin_stop_date_time, stop_exception_value) 808 | 809 | def set_schedule_exception(self, exception_date, exception_type, exception_value): 810 | scan_items = self.scan_schedule_exception(exception_date, exception_type) 811 | is_already = True if len(scan_items) > 0 else False 812 | 813 | if is_already: 814 | self.update_schedule_exception(scan_items[0], exception_value) 815 | else: 816 | self.insert_schedule_exception(exception_date, exception_type, exception_value) 817 | 818 | def remove_schedule_exception(self, exception_date, exception_type): 819 | scan_items = self.scan_schedule_exception(exception_date, exception_type) 820 | 821 | for item in scan_items: 822 | self.delete_schedule_exception(item) 823 | 824 | def scan_schedule_exception(self, exception_date, exception_type): 825 | scan_result = self.db.Table('ScheduleException').scan( 826 | Select='ALL_ATTRIBUTES', 827 | FilterExpression=Attr('ScheduleName').eq(self.schedule_name) & Attr('ExceptionDate').eq( 828 | exception_date.strftime('%Y-%m-%d')) & Attr('ExceptionType').eq(exception_type) 829 | ) 830 | 831 | return scan_result['Items'] 832 | 833 | def insert_schedule_exception(self, exception_date, exception_type, exception_value): 834 | new_uuid = str(uuid.uuid4()) 835 | 836 | return self.db.Table('ScheduleException').put_item( 837 | Item={ 838 | 'ExceptionUuid': new_uuid, 839 | 'ExceptionDate': exception_date.strftime('%Y-%m-%d'), 840 | 'ExceptionType': exception_type, 841 | 'ExceptionValue': exception_value, 842 | 'ScheduleName': self.schedule_name 843 | } 844 | ) 845 | 846 | def update_schedule_exception(self, item, value): 847 | return self.db.Table('ScheduleException').update_item( 848 | Key={ 849 | 'ExceptionUuid': item['ExceptionUuid'], 850 | }, 851 | UpdateExpression='set ExceptionValue=:c', 852 | ExpressionAttributeValues={ 853 | ':c': value 854 | }, 855 | ReturnValues='UPDATED_NEW' 856 | ) 857 | 858 | def delete_schedule_exception(self, item): 859 | return self.db.Table('ScheduleException').delete_item( 860 | Key={ 861 | 'ExceptionUuid': item['ExceptionUuid'], 862 | } 863 | ) 864 | 865 | def print_schedule_data(self): 866 | 867 | print('========================================================================') 868 | print('Schedule Name : ' + self.schedule_name) 869 | print('Today : ' + self.get_exception_date_ymd()) 870 | print('Enabled : ' + str(self.is_enable())) 871 | print('ActiveDays : ' + str(self.get_schedule_property('DaysActive'))) 872 | print('Start Time : ' + str(self.get_start_date_time())) 873 | print('Stop Time : ' + str(self.get_stop_date_time())) 874 | print('IsActiveDay : ' + str(self.is_active_day())) 875 | print('IsStartTime : ' + str(self.is_start_time())) 876 | print('IsStopTime : ' + str(self.is_stop_time())) 877 | print('TagValue : ' + str(self.get_schedule_property('TagValue'))) 878 | print('========================================================================') 879 | 880 | def print_schedule_group_data(self): 881 | 882 | group_list = self.get_schedule_server_group_list() 883 | 884 | for group in group_list: 885 | il = self.get_server_group_instance_list(group) 886 | gn = group['GroupName'] 887 | print('------------------------------------------------------------------------') 888 | print(gn + ' -> ' + str(group['Dependency'])) 889 | print('------------------------------------------------------------------------') 890 | for i in il: 891 | if gn == 'RDB': 892 | print(i['DBInstanceIdentifier']) 893 | else: 894 | print(ScheduleUtil.get_ec2_instance_name(i)) 895 | 896 | 897 | class SpecificDateSchedule(ExceptionSchedule): 898 | 899 | today_ymd = None 900 | 901 | def __init__(self, schedule_name, today_ymd): 902 | super().__init__(schedule_name) 903 | self.today_ymd = today_ymd 904 | 905 | def get_exception_date_ymd(self) -> str: 906 | return datetime.strptime(self.today_ymd, '%Y-%m-%d').strftime('%Y-%m-%d') 907 | 908 | 909 | class Scheduler: 910 | 911 | @staticmethod 912 | def run_job(): 913 | db = boto3.resource('dynamodb') 914 | table = db.Table('Schedule') 915 | response = table.scan() 916 | 917 | for item in response['Items']: 918 | schedule = ExceptionSchedule(item['ScheduleName']) 919 | schedule.print_schedule_data() 920 | schedule.run() 921 | Scheduler.print_line() 922 | 923 | @staticmethod 924 | def print_schedules(): 925 | db = boto3.resource('dynamodb') 926 | table = db.Table('Schedule') 927 | response = table.scan() 928 | 929 | for item in response['Items']: 930 | schedule = ExceptionSchedule(item['ScheduleName']) 931 | schedule.print_schedule_data() 932 | schedule.print_schedule_group_data() 933 | Scheduler.print_line() 934 | 935 | @staticmethod 936 | def print_line(): 937 | print('\n----------------------------------------------------------------------------------------------\n') 938 | 939 | 940 | class BotError(Exception): 941 | 942 | bot = None 943 | 944 | def __init__(self, value, bot): 945 | self.bot = bot 946 | self.value = value 947 | 948 | def err_response(self): 949 | connect_info = JandiWebhook.build_connect_info("Error Stack", traceback.format_exc()) 950 | return SchedulerBot.build_http_response( 951 | JandiWebhook.build_message(self.__str__(), JandiWebhook.color_err, [connect_info])) 952 | 953 | 954 | class BotInvalidError(BotError): 955 | def __init__(self, value, bot): 956 | super(BotInvalidError, self).__init__(value, bot) 957 | 958 | def __str__(self): 959 | return str(self.value) 960 | 961 | def err_response(self): 962 | return SchedulerBot.build_http_response( 963 | JandiWebhook.build_message(self.__str__(), JandiWebhook.color_err)) 964 | 965 | 966 | class BotCommandSyntaxError(BotError): 967 | 968 | description = None 969 | 970 | def __init__(self, value, bot, description=None): 971 | super(BotCommandSyntaxError, self).__init__(value, bot) 972 | self.description = description 973 | 974 | def err_response(self): 975 | connect_info = {} if not self.description else \ 976 | JandiWebhook.build_connect_info(str(self), self.description) 977 | 978 | return SchedulerBot.build_http_response( 979 | JandiWebhook.build_message('잘 못들었습니다?', JandiWebhook.color_err, [connect_info])) 980 | 981 | 982 | class SchedulerBot: 983 | 984 | event = None 985 | body = None 986 | schedule = None 987 | 988 | def __init__(self, event) -> None: 989 | super().__init__() 990 | self.event = event 991 | self.body = json.loads(self.event['body']) 992 | print(self.body) 993 | 994 | def run(self): 995 | try: 996 | if not self.is_valid_token(): 997 | return self.wrong_response('손들어 움직이면 쏜다!') 998 | 999 | command_result = self.command() 1000 | 1001 | if type(command_result) == str: 1002 | return self.build_http_response(JandiWebhook.build_message(command_result, JandiWebhook.color_ok)) 1003 | else: 1004 | return self.build_http_response(command_result) 1005 | 1006 | except BotError as be: 1007 | return be.err_response() 1008 | 1009 | except Exception as e: 1010 | print(traceback.format_exc()) 1011 | return self.not_handle_err_response(e, traceback.format_exc()) 1012 | 1013 | def is_valid_token(self) -> bool: 1014 | print('token : ' + self.body['token']) 1015 | print('outgoing : ' + OUTGOING_WEBHOOK_TOKEN) 1016 | return True if self.body['token'] == OUTGOING_WEBHOOK_TOKEN else False 1017 | 1018 | def command(self): 1019 | hook_text = self.body['text'] 1020 | args = hook_text.split()[1:] 1021 | 1022 | if not args: 1023 | raise BotCommandSyntaxError('Command parsing error', self) 1024 | 1025 | # 도움말 1026 | if args[0] == 'help': 1027 | return self.help() 1028 | else: 1029 | return self.command_schedule(args) 1030 | 1031 | def command_schedule(self, args): 1032 | schedule_name = args[0] 1033 | 1034 | self.schedule = ExceptionSchedule(schedule_name) 1035 | 1036 | # schedule 유효성 체크 1037 | if not self.schedule.get_schedule(): 1038 | raise BotInvalidError('**{0}** 은 저희 부대 스케쥴이 아닌거 같지 말입니다'.format(schedule_name), self) 1039 | 1040 | command = args[1] 1041 | 1042 | if command == 'status' or command == 's': 1043 | return self.status() 1044 | elif command == 'info' or command == 'i': 1045 | return self.info(args[2:]) 1046 | elif command == 'exception' or command == 'e': 1047 | return self.exception(args[2:]) 1048 | elif command == 'force_start': 1049 | return self.force_start() 1050 | elif command == 'force_stop': 1051 | return self.force_stop() 1052 | else: 1053 | raise BotCommandSyntaxError('Wrong command', self) 1054 | 1055 | def help(self): 1056 | 1057 | keyword = self.body['keyword'] 1058 | 1059 | help_command_list = [ 1060 | '/{0} help : 도움말'.format(keyword), 1061 | '/{0} [스케쥴명] status : 현재 서버 상태 조회'.format(keyword), 1062 | '/{0} [스케쥴명] info : 오늘의 스케쥴 조회'.format(keyword), 1063 | '/{0} [스케쥴명] info [YYYY-MM-DD] : 특정일 스케쥴 조회'.format(keyword), 1064 | '/{0} [스케쥴명] exception info : 오늘의 스케쥴 예외 조회'.format(keyword), 1065 | '/{0} [스케쥴명] exception info [YYYY-MM-DD] : 특정일 스케쥴 예외 조회'.format(keyword), 1066 | '/{0} [스케쥴명] exception set [YYYY-MM-DD] [start|stop] [h:m] : 예외 설정'.format(keyword), 1067 | '/{0} [스케쥴명] exception del [YYYY-MM-DD] [start|stop] : 예외 삭제'.format(keyword), 1068 | '/{0} [스케쥴명] force_start : 서버 강제실행'.format(keyword), 1069 | '/{0} [스케쥴명] force_stop : 서버 강제중지'.format(keyword)] 1070 | 1071 | connect_info = JandiWebhook.build_connect_info("명령 커멘드", '\n'.join(help_command_list)) 1072 | 1073 | return JandiWebhook.build_message('충성!', JandiWebhook.color_ok, [connect_info]) 1074 | 1075 | def status(self): 1076 | 1077 | has_server_group = self.schedule.has_server_group() 1078 | 1079 | if has_server_group: 1080 | info = self.get_server_group_status_connect_info_list() 1081 | else: 1082 | info = self.get_server_status_connect_info_list() 1083 | 1084 | return JandiWebhook.build_message( 1085 | '보고 합니다! 현재 서버 상태는 총 : **{0}**, 작업 : **{1}**, 열외 : **{2}** 이상!'.format( 1086 | info['on'] + info['off'], info['on'], info['off']), 1087 | JandiWebhook.color_ok, info['connect_info_list']) 1088 | 1089 | def get_server_group_status_connect_info_list(self): 1090 | 1091 | result = {} 1092 | on = 0 1093 | off = 0 1094 | connect_info_list = [] 1095 | server_group_list = self.schedule.get_schedule_server_group_list() 1096 | 1097 | for server_group in server_group_list: 1098 | instance_list = self.schedule.get_server_group_instance_list(server_group) 1099 | 1100 | description_list = [] 1101 | 1102 | for instance in instance_list: 1103 | 1104 | instance_type = server_group['InstanceType'] 1105 | 1106 | if instance_type == 'EC2': 1107 | status = ScheduleUtil.get_ec2_instance_status(instance) 1108 | description_list.append('{0} : {1}'.format( 1109 | ScheduleUtil.get_ec2_instance_name(instance), 1110 | status)) 1111 | 1112 | if status == 'running': 1113 | on += 1 1114 | else: 1115 | off += 1 1116 | 1117 | elif instance_type == 'RDS': 1118 | status = ScheduleUtil.get_rds_instance_status(instance) 1119 | description_list.append('{0} : {1}'.format( 1120 | ScheduleUtil.get_rds_instance_name(instance), 1121 | status)) 1122 | 1123 | if status == 'available': 1124 | on += 1 1125 | else: 1126 | off += 1 1127 | else: 1128 | continue 1129 | 1130 | connect_info_list.append( 1131 | JandiWebhook.build_connect_info( 1132 | '{0} 그룹의 서버 상태'.format(server_group['GroupName']), 1133 | '\n'.join(description_list))) 1134 | 1135 | result['on'] = on 1136 | result['off'] = off 1137 | result['connect_info_list'] = connect_info_list 1138 | 1139 | return result 1140 | 1141 | def get_server_status_connect_info_list(self): 1142 | result = {} 1143 | on = 0 1144 | off = 0 1145 | connect_info_list = [] 1146 | rds_instance_list = self.schedule.get_rds_instance_list() 1147 | ec2_instance_list = self.schedule.get_ec2_instance_list() 1148 | 1149 | if len(rds_instance_list) > 0: 1150 | 1151 | rds_desc_list = [] 1152 | 1153 | for rds_instance in rds_instance_list: 1154 | status = ScheduleUtil.get_rds_instance_status(rds_instance) 1155 | rds_desc_list.append('{0} : {1}'.format( 1156 | ScheduleUtil.get_rds_instance_name(rds_instance), 1157 | status)) 1158 | 1159 | if status == 'available': 1160 | on += 1 1161 | else: 1162 | off += 1 1163 | 1164 | connect_info_list.append( 1165 | JandiWebhook.build_connect_info( 1166 | '{0} 서버 상태'.format('RDS'), 1167 | '\n'.join(rds_desc_list))) 1168 | 1169 | if len(ec2_instance_list) > 0: 1170 | 1171 | ec2_desc_list = [] 1172 | 1173 | for ec2_instance in ec2_instance_list: 1174 | status = ScheduleUtil.get_ec2_instance_status(ec2_instance) 1175 | ec2_desc_list.append('{0} : {1}'.format( 1176 | ScheduleUtil.get_ec2_instance_name(ec2_instance), 1177 | status)) 1178 | 1179 | if status == 'running': 1180 | on += 1 1181 | else: 1182 | off += 1 1183 | 1184 | connect_info_list.append( 1185 | JandiWebhook.build_connect_info( 1186 | '{0} 서버 상태'.format('EC2'), 1187 | '\n'.join(ec2_desc_list))) 1188 | 1189 | result['on'] = on 1190 | result['off'] = off 1191 | result['connect_info_list'] = connect_info_list 1192 | 1193 | return result 1194 | 1195 | def info(self, args): 1196 | if len(args) > 0: 1197 | dt = args[0] 1198 | if not ScheduleUtil.is_valid_date(dt): 1199 | raise BotInvalidError('날짜 형식을 잘못 입력하였습니다', self) 1200 | 1201 | desc = self.build_info_by_schedule(SpecificDateSchedule(self.schedule.schedule_name, dt)) 1202 | 1203 | return JandiWebhook.build_message("**{0}** 스케쥴 정보 입니다!".format(dt), JandiWebhook.color_ok, [desc]) 1204 | 1205 | else: 1206 | desc = self.build_info_by_schedule(self.schedule) 1207 | 1208 | return JandiWebhook.build_message("오늘의 스케쥴 정보 입니다!", JandiWebhook.color_ok, [desc]) 1209 | 1210 | def build_info_by_schedule(self, schedule_obj): 1211 | desc_list = [ 1212 | 'Enabled : {0}'.format(schedule_obj.is_enable()), 1213 | 'DaysActive : {0}'.format(schedule_obj.get_schedule_property('DaysActive')), 1214 | 'Start Time : {0}'.format(schedule_obj.get_start_date_time().strftime('%H:%M') 1215 | if schedule_obj.get_start_date_time() is not None else 'None'), 1216 | 'Stop Time : {0}'.format(schedule_obj.get_stop_date_time().strftime('%H:%M') 1217 | if schedule_obj.get_stop_date_time() is not None else 'None'), 1218 | ] 1219 | 1220 | return JandiWebhook.build_connect_info("상세정보", '\n'.join(desc_list)) 1221 | 1222 | def exception(self, args): 1223 | if len(args) == 0: 1224 | raise BotCommandSyntaxError('Exception command error', self) 1225 | elif args[0] == 'info' or args[0] == 'i': 1226 | return self.exception_info(args[1:]) 1227 | elif args[0] == 'set' or args[0] == 's': 1228 | return self.exception_set(args[1:]) 1229 | elif args[0] == 'del' or args[0] == 'd': 1230 | return self.exception_del(args[1:]) 1231 | else: 1232 | raise BotCommandSyntaxError('Wrong exception command error', self) 1233 | 1234 | def exception_info(self, args): 1235 | if len(args) > 0: 1236 | dt = args[0] 1237 | if not ScheduleUtil.is_valid_date(dt): 1238 | raise BotInvalidError('날짜 형식을 잘못 입력하였습니다', self) 1239 | 1240 | return self.build_exception_info_by_schedule(SpecificDateSchedule(self.schedule.schedule_name, dt), dt) 1241 | else: 1242 | return self.build_exception_info_by_schedule(self.schedule) 1243 | 1244 | def build_exception_info_by_schedule(self, schedule_obj, dt=None): 1245 | exception_list = schedule_obj.get_schedule_exception_list() 1246 | 1247 | date = '오늘' if dt is None else dt 1248 | 1249 | if len(exception_list) == 0: 1250 | return '**{0}** 근무시간 변경이 없습니다'.format(date) 1251 | 1252 | desc_list = [] 1253 | 1254 | for exception in exception_list: 1255 | desc_list.append('**{0}** 시간을 **{1}**으로 변경'.format(exception['ExceptionType'], exception['ExceptionValue'])) 1256 | 1257 | return JandiWebhook.build_message('**{0}** 근무시간 변경표 입니다'.format(date), JandiWebhook.color_ok, 1258 | [JandiWebhook.build_connect_info('변경표', '\n'.join(desc_list))]) 1259 | 1260 | def exception_set(self, args): 1261 | if not len(args) == 3: 1262 | raise BotCommandSyntaxError('Wrong exception set error', self) 1263 | 1264 | exception_date = args[0] 1265 | exception_type = args[1] 1266 | exception_time = args[2] 1267 | 1268 | if not ScheduleUtil.is_valid_date(exception_date): 1269 | raise BotInvalidError('날짜가 잘못되었지 말입니다', self) 1270 | 1271 | exception_date = datetime.strptime(exception_date, '%Y-%m-%d').date() 1272 | 1273 | h = 0 1274 | 1275 | if exception_time != "None": 1276 | if not ScheduleUtil.is_valid_time(exception_time): 1277 | raise BotInvalidError('시간이 잘못되었지 말입니다', self) 1278 | h = int(exception_time.split(':')[0]) 1279 | 1280 | self.schedule.set_schedule_exception(exception_date, exception_type, exception_time) 1281 | 1282 | result = [] 1283 | 1284 | if h > 20: 1285 | result.append("야간 근무 하십니까? 고생하시지 말입니다!") 1286 | 1287 | result.append('**{0}** 일 **{1}** 스케쥴 **{2}** 시간은 **{3}** 으로 설정되었습니다'.format( 1288 | exception_date, 1289 | self.schedule.schedule_name, 1290 | exception_type, 1291 | exception_time)) 1292 | 1293 | return '\n'.join(result) 1294 | 1295 | def exception_del(self, args): 1296 | 1297 | if not len(args) == 2: 1298 | raise BotCommandSyntaxError('Wrong exception del error', self) 1299 | 1300 | exception_date = args[0] 1301 | exception_type = args[1] 1302 | 1303 | if not ScheduleUtil.is_valid_date(exception_date): 1304 | raise BotInvalidError('날짜가 잘못되었지 말입니다', self) 1305 | 1306 | exception_date = datetime.strptime(exception_date, '%Y-%m-%d').date() 1307 | 1308 | self.schedule.remove_schedule_exception(exception_date, exception_type) 1309 | 1310 | result = exception_date.strftime('%Y-%m-%d') + ' 일 ' + exception_type + ' 예외를 삭제 하였습니다.' 1311 | 1312 | return result 1313 | 1314 | def force_start(self): 1315 | 1316 | self.schedule.set_schedule_force_start(True) 1317 | 1318 | return '진돗개 하나 발령! 현 시간부로 모든 서버들은 기상한다!' 1319 | 1320 | def force_stop(self): 1321 | 1322 | self.schedule.set_schedule_force_start(False) 1323 | 1324 | self.schedule.stop(True) 1325 | 1326 | return '취침소등 하겠습니다!' 1327 | 1328 | @staticmethod 1329 | def build_http_response(res=None): 1330 | return { 1331 | 'statusCode': '200', 1332 | 'body': {} if not res else json.dumps(res), 1333 | 'headers': { 1334 | 'Content-Type': 'application/json', 1335 | }, 1336 | } 1337 | 1338 | def wrong_response(self, wrong_message): 1339 | return self.build_http_response(JandiWebhook.build_message(wrong_message, JandiWebhook.color_warning)) 1340 | 1341 | def not_handle_err_response(self, exception, error_stack): 1342 | connect_info = JandiWebhook.build_connect_info(str(exception), error_stack) 1343 | return self.build_http_response(JandiWebhook.build_message('검열한번 하셔야 할거 같지 말입니다', 1344 | JandiWebhook.color_err, [connect_info])) 1345 | 1346 | 1347 | def handle(event, context): 1348 | 1349 | if event and 'httpMethod' in event: 1350 | bot = SchedulerBot(event) 1351 | res = bot.run() 1352 | print(res) 1353 | return res 1354 | 1355 | else: 1356 | return Scheduler.run_job() 1357 | -------------------------------------------------------------------------------- /functions/awsInstanceScheduler/requirements.txt: -------------------------------------------------------------------------------- 1 | boto3==1.4.4 2 | botocore==1.5.75 3 | requests 4 | -------------------------------------------------------------------------------- /project.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "production", 3 | "description": "", 4 | "memory": 512, 5 | "timeout": 300, 6 | "role": "<>", 7 | "nameTemplate": "{{.Project.Name}}-{{.Function.Name}}", 8 | "environment": {} 9 | } 10 | --------------------------------------------------------------------------------