├── .gitignore ├── LICENSE ├── gce ├── cloud_handler.py ├── cron_executor.py ├── logger_sample_task.py └── test_executor.py ├── gcp-scheduler-pubsub-cf-gce └── README.md └── readme.md /.gitignore: -------------------------------------------------------------------------------- 1 | gae/lib/** 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /gce/cloud_handler.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright 2015 Google Inc. All Rights Reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from datetime import datetime 18 | import json 19 | import logging 20 | import sys 21 | from threading import Thread 22 | 23 | from apiclient import discovery 24 | 25 | import httplib2 26 | import oauth2client.contrib.gce as gce_oauth2client 27 | 28 | LOGGING_SCOPES = ["https://www.googleapis.com/auth/logging.admin", 29 | "https://www.googleapis.com/auth/cloud-platform"] 30 | METADATA_SERVER = 'http://metadata/computeMetadata/v1/%s' 31 | 32 | SEVERITY = { 33 | 0: "DEFAULT", 34 | 10: "DEBUG", 35 | 20: "INFO", 36 | 25: "NOTICE", 37 | 30: "WARNING", 38 | 40: "ERROR", 39 | 50: "CRITICAL", 40 | 60: "ALERT", 41 | 70: "EMERGENCY", 42 | } 43 | 44 | 45 | class CloudLoggingHandler(logging.Handler): 46 | """ 47 | A python logging handler that emits to Google Cloud Logging 48 | 49 | NOTE this is for demo purposes only, do not use on high volume production 50 | services. 51 | """ 52 | 53 | def __init__(self, 54 | credentials=None, 55 | logname='python', 56 | labels={}, 57 | project_id=None, 58 | on_gce=False, 59 | async=True): 60 | 61 | super(CloudLoggingHandler, self).__init__() 62 | 63 | self.main_http = http = httplib2.Http() 64 | if not credentials: 65 | if not on_gce: 66 | raise ValueError( 67 | "credentials need to be provided if not on running on GCE") 68 | else: 69 | credentials = gce_oauth2client.AppAssertionCredentials( 70 | scope=LOGGING_SCOPES) 71 | if project_id is None: 72 | if not on_gce: 73 | raise ValueError( 74 | "project_id needs to be provided if not running on GCE") 75 | else: 76 | resp, self.project_id = http.request( 77 | METADATA_SERVER % "project/project-id", 78 | method='GET', 79 | body=None, 80 | headers={'Metadata-Flavor': 'Google'}) 81 | else: 82 | self.project_id = project_id 83 | 84 | if not on_gce: 85 | # The following are placeholder values 86 | self.instance_id = '12345' 87 | self.zone = 'us-central1-a' 88 | else: 89 | resp, self.zone = http.request( 90 | METADATA_SERVER % "instance/zone", 91 | method='GET', 92 | body=None, 93 | headers={'Metadata-Flavor': 'Google'}) 94 | 95 | self.zone = self.zone.decode('UTF-8').split('/')[-1] 96 | resp, self.instance_id = http.request( 97 | METADATA_SERVER % "instance/id", 98 | method='GET', 99 | body=None, 100 | headers={'Metadata-Flavor': 'Google'}) 101 | 102 | credentials = credentials.create_scoped(LOGGING_SCOPES) 103 | credentials.authorize(http) 104 | self.credentials = credentials 105 | self.async = async 106 | self.client = discovery.build("logging", "v1beta3", http=http) 107 | self.logname = logname 108 | self.labels = labels 109 | self.labels["compute.googleapis.com/resource_id"] = self.instance_id 110 | self.labels["compute.googleapis.com/resource_type"] = 'instance' 111 | 112 | def write_log(self, record): 113 | if self.async: 114 | http = httplib2.Http() 115 | self.credentials.authorize(http) 116 | else: 117 | http = self.main_http 118 | 119 | msg = self.format(record) 120 | entry_metadata = { 121 | "timestamp": "2015-03-25T10:20:50.52Z", 122 | "region": "us-central1", 123 | "zone": "us-central1-a", 124 | "serviceName": "compute.googleapis.com", 125 | "severity": "CRITICAL", 126 | "labels": {} 127 | } 128 | entry_metadata['timestamp'] = "%sZ" % datetime.utcfromtimestamp( 129 | record.created).isoformat() 130 | # min(range(len(levels)), key=lambda i: abs(levels[i]-loglevel)) 131 | entry_metadata['severity'] = SEVERITY[min(SEVERITY.keys(), 132 | key=lambda i: 133 | abs(record.levelno - i))] 134 | entry_metadata['zone'] = self.zone 135 | entry_metadata['region'] = '-'.join(self.zone.split('-')[:-1]) 136 | # note - labels are searchable, but not visible in UI 137 | entry_metadata['labels'] = { 138 | 'module': record.module, 139 | 'funcName': record.funcName, 140 | 'filename': record.filename, 141 | 'name': record.name, 142 | } 143 | 144 | body = { 145 | "commonLabels": self.labels, 146 | "entries": [{"metadata": entry_metadata, 147 | "log": self.logname, }] 148 | } 149 | 150 | if isinstance(record.msg, dict): 151 | # TODO - not working as expected 152 | raise NotImplementedError( 153 | "structured logs not supported in this version") 154 | else: 155 | body['entries'][0]['textPayload'] = msg 156 | 157 | try: 158 | resp = self.client.projects().logs().entries().write( 159 | projectsId=self.project_id, 160 | logsId=self.logname, 161 | body=body).execute(http=http) 162 | if resp: 163 | # this would be an error 164 | sys.stderr.write(resp + '\n') 165 | except Exception as e: 166 | sys.stderr.write(e.message + '\n') 167 | 168 | def emit(self, record): 169 | if self.async: 170 | # send the logging event to logging service in a thread so 171 | # main program is not blocked 172 | http_writer = Thread(target=self.write_log, args=(record, )) 173 | http_writer.start() 174 | return 175 | else: 176 | self.write_log(record) 177 | return 178 | -------------------------------------------------------------------------------- /gce/cron_executor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright 2015 Google Inc. All Rights Reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | from datetime import datetime 18 | import logging 19 | from subprocess import PIPE, Popen 20 | import sys 21 | import threading 22 | import time 23 | 24 | try: 25 | from Queue import Queue, Empty 26 | except ImportError: 27 | from queue import Queue, Empty # python 3.x 28 | 29 | from apiclient import discovery 30 | from apiclient.errors import HttpError 31 | import httplib2 32 | from oauth2client.client import GoogleCredentials 33 | 34 | log = logging.getLogger(__name__) 35 | 36 | PUBSUB_SCOPES = ["https://www.googleapis.com/auth/pubsub"] 37 | 38 | 39 | class AsynchronousFileReader(threading.Thread): 40 | ''' 41 | Helper class to implement asynchronous reading of a file 42 | in a separate thread. Pushes read lines on a queue to 43 | be consumed in another thread. 44 | ''' 45 | 46 | def __init__(self, fd, queue): 47 | assert isinstance(queue, Queue) 48 | assert callable(fd.readline) 49 | threading.Thread.__init__(self) 50 | self._fd = fd 51 | self._queue = queue 52 | 53 | def run(self): 54 | '''The body of the tread: read lines and put them on the queue.''' 55 | for line in iter(self._fd.readline, ''): 56 | self._queue.put(line) 57 | 58 | def eof(self): 59 | '''Check whether there is no more content to expect.''' 60 | return not self.is_alive() and self._queue.empty() 61 | 62 | 63 | def get_client(): 64 | """Creates Pub/Sub client and returns it.""" 65 | credentials = GoogleCredentials.get_application_default() 66 | credentials = credentials.create_scoped(PUBSUB_SCOPES) 67 | http = httplib2.Http() 68 | credentials.authorize(http) 69 | return discovery.build('pubsub', 'v1beta2', http=http) 70 | 71 | 72 | class Executor(): 73 | def __init__(self, 74 | topic='mytopic', 75 | project='myproject', 76 | subname='default_sub', 77 | task_cmd="sleep 20", 78 | deadline=60): 79 | self.topic = topic 80 | self.project = project 81 | self.subname = "%s_%s" % (topic, subname) 82 | self.task_cmd = task_cmd 83 | self.client = get_client() 84 | self.sub = self.get_subscription(deadline=deadline) 85 | self.ackdeadline = self.sub['ackDeadlineSeconds'] 86 | self.io_queue = Queue() 87 | self.lease_start = None 88 | self.job_log = logging.getLogger(self.subname) 89 | 90 | def create_subscription(self, deadline=60): 91 | log.debug("creating subscription") 92 | body = { 93 | # The name of the topic from which this subscription receives messages 94 | 'topic': 'projects/{}/topics/{}'.format(self.project, self.topic), 95 | 'ackDeadlineSeconds': deadline 96 | } 97 | 98 | try: 99 | subscription = self.client.projects().subscriptions().create( 100 | name='projects/{}/subscriptions/{}'.format(self.project, 101 | self.subname), 102 | body=body).execute() 103 | except Exception as e: 104 | log.critical("unable to create subscription") 105 | raise 106 | 107 | return subscription 108 | 109 | def get_subscription(self, deadline=60): 110 | sub = None 111 | log.debug("getting subscription") 112 | try: 113 | # note: subscriptions are a flat namespace in a project 114 | # we delete then recreate the subscription if it exists 115 | # so we don't execute old messages 116 | 117 | self.client.projects().subscriptions().delete( 118 | subscription='projects/{}/subscriptions/{}'.format( 119 | self.project, self.subname)).execute() 120 | log.debug("deleted existing subscription") 121 | except HttpError as e: 122 | if e.resp.status == 404: 123 | sub = self.create_subscription(deadline=deadline) 124 | else: 125 | raise 126 | else: 127 | sub = self.create_subscription(deadline=deadline) 128 | log.debug("subscription %s" % sub) 129 | return sub 130 | 131 | def get_messages(self): 132 | # You can fetch multiple messages with a single API call. 133 | batch_size = 1 134 | 135 | # Create a POST body for the Pub/Sub request 136 | body = { 137 | # Setting ReturnImmediately to false instructs the API to wait 138 | # to collect the message up to the size of MaxEvents, or until 139 | # the timeout (approx 90s) 140 | 'returnImmediately': False, 141 | 'maxMessages': batch_size, 142 | } 143 | log.debug("pulling messages") 144 | resp = self.client.projects().subscriptions().pull( 145 | subscription=self.sub['name'], 146 | body=body).execute() 147 | if 'receivedMessages' in resp: 148 | log.debug("number msgs: %s" % len(resp.get('receivedMessages'))) 149 | self.lease_start = datetime.now() 150 | return resp.get('receivedMessages') 151 | else: 152 | return [] 153 | 154 | def extend_lease(self, msg): 155 | body = { 156 | 'ackIds': [msg['ackId']], 157 | 'ackDeadlineSeconds': self.ackdeadline, 158 | } 159 | resp = self.client.projects().subscriptions().modifyAckDeadline( 160 | subscription=self.sub['name'], 161 | body=body).execute() 162 | return resp 163 | 164 | def run_task(self, msg): 165 | proc = Popen(self.task_cmd, stdout=PIPE, shell=True) 166 | stdout_reader = AsynchronousFileReader(proc.stdout, self.io_queue) 167 | stdout_reader.start() 168 | while not stdout_reader.eof(): 169 | # read line without blocking 170 | while True: 171 | try: 172 | # line = self.io_queue.get_nowait() # or q.get(timeout=.1) 173 | line = self.io_queue.get_nowait() # could do timeout=.1 174 | except Empty: 175 | break 176 | else: 177 | self.job_log.info(line) 178 | 179 | lease_age = datetime.now() - self.lease_start 180 | if lease_age.seconds > (self.ackdeadline - 20): 181 | # 10 seconds left in lease, renew 182 | log.debug("extending lease") 183 | try: 184 | resp = self.extend_lease(msg) 185 | self.extend_error_ct = 0 186 | self.lease_start = datetime.now() 187 | except HttpError as e: 188 | if e.resp.status == 503: 189 | # service might return intermitant 503 190 | log.warning("PubSub returned 503") 191 | self.extend_error_ct += 1 192 | if self.extend_error_ct > 5: 193 | log.critical( 194 | "Too many error responses to extend request") 195 | raise 196 | time.sleep(1) 197 | 198 | retcode = proc.poll() 199 | if retcode is not None: 200 | # TODO if error - expire lease immediately? 201 | # process exited 202 | log.debug("process ended") 203 | return retcode 204 | 205 | def process_messages(self, msgs): 206 | for received_message in msgs: 207 | pubsub_message = received_message.get('message') 208 | log.debug("processing %s" % received_message.get('ackId')) 209 | if pubsub_message: 210 | ack_ids = [] 211 | # Process messages 212 | # Note the design here is to run a single task at a time 213 | # print base64.urlsafe_b64decode( 214 | # str(pubsub_message.get('data'))) 215 | # Get the message's ack ID 216 | cmd_retcode = self.run_task(received_message) 217 | # TODO if cmd_retcode == 0, the cmd exited clean 218 | # the retry logic could get complex and is left as an exercise 219 | ack_ids.append(received_message.get('ackId')) 220 | # in this case - should ack per message instead of batch 221 | # as want to make sure task is acked after completion, as 222 | # nothing else will extend 223 | # Create a POST body for the acknowledge request 224 | ack_body = {'ackIds': ack_ids} 225 | if ack_ids: 226 | # Acknowledge the message. 227 | log.debug("acking %s" % ack_ids) 228 | ack_resp = self.client.projects().subscriptions().acknowledge( 229 | subscription=self.sub['name'], 230 | body=ack_body).execute() 231 | 232 | def watch_topic(self): 233 | while True: 234 | msgs = self.get_messages() 235 | if msgs: 236 | self.process_messages(msgs) 237 | # when return immediately is False- there is about a 90second open 238 | # request 239 | -------------------------------------------------------------------------------- /gce/logger_sample_task.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env/ python 2 | 3 | # Copyright 2015 Google Inc. All Rights Reserved. 4 | 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | """ 18 | This is simply an example task, meant to replace an executable bit of code 19 | that does work on your system 20 | """ 21 | import time 22 | import os 23 | import sys 24 | 25 | try: 26 | os.mkdir('/tmp/foo') 27 | except: 28 | pass 29 | 30 | for x in range(20): 31 | print "Doing work... %s" % x 32 | try: 33 | os.mkdir('/tmp/foo/%s' % x) 34 | except: 35 | pass 36 | time.sleep(.5) 37 | 38 | sys.exit(0) 39 | -------------------------------------------------------------------------------- /gce/test_executor.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Copyright 2015 Google Inc. All Rights Reserved. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | """ 18 | 19 | This sample script shows how to use the reusable Executor utility to 20 | watch a topic and execute a command when a message is received 21 | 22 | """ 23 | 24 | import logging 25 | import os 26 | import sys 27 | 28 | from cloud_handler import CloudLoggingHandler 29 | from cron_executor import Executor 30 | 31 | PROJECT = 'your-project-id' # change this to match your project 32 | TOPIC = 'test' 33 | 34 | script_path = os.path.abspath(os.path.join(os.getcwd(), 'logger_sample_task.py')) 35 | 36 | sample_task = "python -u %s" % script_path 37 | 38 | 39 | root_logger = logging.getLogger('cron_executor') 40 | root_logger.setLevel(logging.DEBUG) 41 | ch = logging.StreamHandler(sys.stderr) 42 | ch.setLevel(logging.DEBUG) 43 | formatter = logging.Formatter( 44 | '%(asctime)s - %(name)s - %(levelname)s - %(message)s') 45 | ch.setFormatter(formatter) 46 | root_logger.addHandler(ch) 47 | 48 | cloud_handler = CloudLoggingHandler(on_gce=True, logname="task_runner") 49 | root_logger.addHandler(cloud_handler) 50 | 51 | # create the executor that watches the topic, and will run the job task 52 | test_executor = Executor(topic=TOPIC, project=PROJECT, task_cmd=sample_task, subname='sample_task_task') 53 | 54 | # add a cloud logging handler and stderr logging handler 55 | job_cloud_handler = CloudLoggingHandler(on_gce=True, logname=test_executor.subname) 56 | test_executor.job_log.addHandler(job_cloud_handler) 57 | test_executor.job_log.addHandler(ch) 58 | test_executor.job_log.setLevel(logging.DEBUG) 59 | 60 | 61 | # watches indefinitely 62 | test_executor.watch_topic() 63 | -------------------------------------------------------------------------------- /gcp-scheduler-pubsub-cf-gce/README.md: -------------------------------------------------------------------------------- 1 | This solution has been moved to 2 | [Scheduling Compute Instances with Cloud Scheduler](https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule). 3 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | # Sample: Reliable Task Scheduling on Google Compute Engine with Cloud Scheduler 2 | 3 | In distributed systems, such as a network of Google Compute Engine 4 | instances, it is challenging to reliably schedule tasks because any individual 5 | instance may become unavailable due to autoscaling or network partitioning. 6 | 7 | Google Cloud Platform provides a managed [Cloud Scheduler](https://cloud.google.com/scheduler/) 8 | service. Using this service for scheduling and Google Cloud Pub/Sub for 9 | distributed messaging, you can build an application to reliably schedule tasks 10 | across a fleet of Compute Engine instances. 11 | 12 | This sample illustrates how to build a solution. For a full description of 13 | the design pattern used in this sample, see 14 | [Reliable Task Scheduling on Compute Engine with Cloud Scheduler](http://cloud.google.com/solutions/reliable-task-scheduling-compute-engine). 15 | 16 | For an example of how to start and stop VMs, see [Scheduling Compute Instances with Cloud Scheduler](https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule). 17 | 18 | ## About the sample 19 | 20 | This sample contains two components: 21 | 22 | * Instructions for configuring Cloud Scheduler to send cron messages to Cloud Pub/Sub topics. 23 | 24 | * A utility that runs on Compute Engine. This utility monitors a Cloud Pub/Sub 25 | topic. When it detects a new message, it runs the corresponding command 26 | locally on the server. 27 | 28 | You specify the cron messages to send and their timing in the Cloud Scheduler 29 | configuration. When Cloud Scheduler fires a scheduled event, the cron message is 30 | passed to the corresponding previously created Cloud Pub/Sub topic. 31 | 32 | The utility running on the Compute Engine instances receives cron messages from 33 | Cloud Pub/Sub and runs the specified commands that are normally run by cron. To 34 | do so, it performs the following actions: 35 | 36 | * Creates subscriptions to Cloud Pub/Sub topics. 37 | * Monitors those subscriptions for new messages using a long-polling loop. 38 | * In response to messages, it runs the corresponding command in a subprocess, 39 | during which it: 40 | * Maintains the lease on the Cloud Pub/Sub message and extends the lease 41 | time on a Cloud Pub/Sub message as needed for long-running commands. 42 | * Acknowledges and releases the message on exit of the command. The exit 43 | code is not required to be successful; additional retry logic is left 44 | as an exercise. 45 | 46 | 47 | This sample includes the reusable wrapper code of the utility, an example of its 48 | use, and a sample script that it runs. 49 | 50 | ## How to run the sample 51 | 52 | The overview for configuring and running this sample is as follows: 53 | 54 | 1. Create a project and other cloud resources. 55 | 2. Clone or download the sample code. 56 | 3. Create the Cloud Pub/Sub topic. 57 | 4. Create the Cloud Scheduler job. 58 | 5. Run a utility on Compute Engine that monitors the Cloud Pub/Sub topic for 59 | messages. and, on detecting one, runs a sample script locally on the 60 | instance. 61 | 6. Verify the script ran on schedule by checking the Cloud Logging output. 62 | 63 | ### Prerequisites 64 | 65 | * If you don’t already have one, create a 66 | [Google Account](https://accounts.google.com/SignUp). 67 | 68 | * Create a Developers Console project. 69 | 1. In the [Google Developers Console](https://console.developers.google.com/project), select 70 | **Create Project**. 71 | 2. [Enable the Pub/Sub API](https://console.cloud.google.com/flows/enableapi?apiid=pubsub&redirect=https://console.cloud.google.com) 72 | 3. [Enable the App Engine Admin API](https://console.cloud.google.com/flows/enableapi?apiid=appengine&redirect=https://console.cloud.google.com). This is required by Cloud Scheduler. 73 | 4. Visit the [Compute Engine instances](https://console.cloud.google.com/compute/instances) page, this will activate the API. 74 | 5. [Enable Project Billing](https://support.google.com/cloud/answer/6293499#enable-billing) 75 | 6. Create an App Engine app. This is required by Cloud Scheduler: 76 | 77 | $ gcloud app create --region=us-central 78 | 79 | 7. Enable the Cloud Scheduler API: 80 | 81 | $ gcloud services enable cloudscheduler.googleapis.com 82 | 83 | Ensure that the following is installed if not already on your system: 84 | 85 | * Install [`git`](https://git-scm.com/downloads). 86 | 87 | * Install [Python 2.7](https://www.python.org/download/releases/2.7/). 88 | 89 | * Install [Python `pip`](https://pip.pypa.io/en/latest/installing.html). 90 | 91 | * [Download and install the Google Cloud SDK](http://cloud.google.com/sdk/). 92 | 93 | 94 | Important: This tutorial uses several billable components of Google Cloud 95 | Platform. To estimate the cost of running this sample: 96 | 97 | * Assume the utility runs on a single `f1-micro` Google Compute Instance for 98 | 15 minutes of one day while you test the sample. After which, you delete 99 | the project, releasing all resources. That's **0.25 hours per month**. 100 | * Cloud Scheduler is free for up to **3 jobs per month**. 101 | 102 | Use the [Google Cloud Platform Pricing Calculator](https://cloud.google.com/products/calculator/#id=beb5326f-90c3-4842-9c3f-a3761b40fbe3) 103 | to generate a cost estimate based on this projected usage. New Cloud Platform 104 | users may be eligible for a [free trial](http://cloud.google.com/free-trial). 105 | 106 | ### Clone the sample code 107 | 108 | To clone the GitHub repository to your computer, run the following command: 109 | 110 | $ git clone https://github.com/GoogleCloudPlatform/reliable-task-scheduling-compute-engine-sample 111 | 112 | Change directories to the `reliable-task-scheduling-compute-engine-sample` directory. The exact path 113 | depends on where you placed the directory when you cloned the sample files from 114 | GitHub. 115 | 116 | $ cd reliable-task-scheduling-compute-engine-sample 117 | 118 | ### Create Pub/Sub topic 119 | 120 | 1. Configure the `gcloud` command-line tool to use the project you created in 121 | Prerequisites. 122 | 123 | $ gcloud config set project 124 | 125 | Where you replace `` with the identifier of your cloud 126 | project. 127 | 128 | 1. Create the Pub/Sub topic that you will push messages to. 129 | 130 | $ gcloud pubsub topics create test 131 | 132 | The topic is now listed under `gcloud pubsub topics list`. You can also see the topic 133 | in the console: 134 | 135 | Big Data > Pub/Sub 136 | 137 | ### Create Cloud Scheduler job 138 | 139 | Next, we configure Cloud Scheduler to push a message containing the string `test job` every 140 | minute to the Pub/Sub topic `test` that we just created. 141 | 142 | gcloud scheduler jobs create pubsub test-job --schedule="* * * * *" \ 143 | --topic=test --message-body="test job" 144 | 145 | The `schedule` is specified in [unix-cron format](https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules). 146 | A `*` in every field means the job runs every minute, every hour, every day of the month, 147 | every month, every day of the week. More simply put, it runs once per minute. 148 | 149 | The job is now visible in `gcloud scheduler jobs list`. You can also see the jobs 150 | in the console: 151 | 152 | Tools > Cloud Scheduler 153 | 154 | Execution logs for the job are visible via the Logs link for each job. 155 | 156 | ### How Cloud Pub/Sub subscriptions are specified 157 | 158 | The utility running on a Compute Engine instance monitors a set of Cloud Pub/Sub 159 | topic subscriptions and runs commands on that instance each time it receives a message. 160 | By configuring which topics the utility monitors, you can control the jobs that 161 | run on each instance. Separating the scheduling logic from the utility logic 162 | using Cloud Pub/Sub messaging gives you the ability to schedule all of your 163 | jobs with Cloud Scheduler, and then configure the utility on each instance to 164 | listen to only the job messages that apply to that instance. 165 | 166 | In the sample implementation of the utility, the topic to subscribe to is set as 167 | a variable in `test_executor.py`. 168 | 169 | TOPIC = 'test' 170 | 171 | This value is used when the code creates an `Executor` object to monitor a Cloud Pub/Sub 172 | topic. 173 | 174 | test_executor = Executor(topic=TOPIC, project=PROJECT, task_cmd=logger, subname='logger_sample_task') 175 | 176 | If you only need the utility to monitor a single topic, you can simply change 177 | the value of `TOPIC` in this script. To have the utility monitor multiple topics, 178 | you need to instantiate multiple `Executor` objects. 179 | 180 | For this runthrough of the sample, leave `TOPIC` set to `'test'` so you can 181 | verify your results as described in the following sections. 182 | 183 | 184 | ### Specify the commands to run on the instance 185 | 186 | In this sample, the utility acts as a wrapper to run commands that can be 187 | configured as cron jobs. These commands are specified in the `test_executor.py` 188 | file. For this example, the command runs a script `logger_sample_task.py` that simply 189 | prints output to `stdout`. 190 | 191 | script_path = os.path.abspath(os.path.join(os.getcwd(), 'logger_sample_task.py')) 192 | sample_task = "python -u %s" % script_path 193 | 194 | To modify the sample to run your own tasks, update the command syntax in the 195 | `sample_task` variable. 196 | 197 | For your first time running the sample, leave this set to the `logger_sample_task` 198 | script so you can verify your results as described in the following sections. 199 | 200 | 201 | ### Install the utility script on a Compute Engine instance 202 | 203 | The utility script runs on your Compute Engine instances and subscribes to the 204 | Cloud Pub/Sub topics you specified in Cloud Scheduler. When the utility script 205 | receives a message, it runs the corresponding job locally. To make this 206 | possible, install the utility script on each instance where 207 | you want durable cron jobs to run. The script files are in the `gce` 208 | directory. 209 | 210 | 211 | 1. Create a Compute Engine instance with Cloud Pub/Sub scope. In the following 212 | example, the instance name is `cronworker`. 213 | 214 | $ gcloud compute instances create cronworker \ 215 | --machine-type f1-micro \ 216 | --scopes https://www.googleapis.com/auth/pubsub,https://www.googleapis.com/auth/logging.write \ 217 | --zone us-central1-a 218 | 219 | 220 | 2. Edit `gce/test_executor.py` to change the project constant: 221 | 222 | PROJECT = 'your-project-id' 223 | 224 | Replace `your-project-id` with the identifier of your cloud project. 225 | 226 | 4. Copy the utility script files to the new instance. 227 | 228 | $ gcloud compute scp --recurse gce cronworker:~/ --zone=us-central1-a 229 | 230 | 231 | 5. SSH into the Compute Engine instance. The following steps are run on the instance over the SSH session. 232 | 233 | $ gcloud compute ssh cronworker \ 234 | --zone us-central1-a 235 | 236 | 237 | 6. Update the apt-get package lists on the instance. 238 | 239 | $ sudo apt-get update 240 | 241 | 242 | 7. Install `pip` and the Python development libraries on the instance. 243 | 244 | $ sudo apt-get install -y python-pip python-dev 245 | 246 | 247 | 8. Install the [Python client library for accessing Google APIs](https://github.com/google/google-api-python-client) on the instance with Python Pip. 248 | 249 | $ sudo pip install --upgrade google-api-python-client oauth2client pytz 250 | 251 | 252 | 9. Change directories on the instance to the directory where you uploaded the utility script files. 253 | 254 | $ cd gce 255 | 256 | 257 | 10. Run the utility file script, `test_executor.py` 258 | 259 | $ python test_executor.py 260 | 261 | 262 | 263 | ### Verify cron jobs run on the instance 264 | 265 | After you start the utility, it checks for messages on the specified Cloud 266 | Pub/Sub topic (`'test'`, by default). When it checks the subscription and receives 267 | a message, it runs a sample task that simply prints output to `stdout`. If you are 268 | still connected to the instance using SSH, you’ll see output like the following 269 | when the task runs. 270 | 271 | Doing work... 1 272 | Doing work... 2 273 | Doing work... 3 274 | … 275 | Doing work... 20 276 | 277 | The utility script records its activity using Google Cloud Logging. After a cron 278 | job has had time to run, you can view the `stdout` output of the job in the Logs 279 | Viewer for Google Compute Engine. 280 | 281 | 1. Open the [Developers Console](https://console.developers.google.com) and select your project from the 282 | list. 283 | 284 | 2. From the navigational menu, select **Stackdriver > Logging > Logs**. 285 | 286 | 3. Expand the dropdown box displaying **GAE Application** and select 287 | **GCE VM Instance > cronworker**. 288 | 289 | 4. Expand the dropdown box displaying **All Logs**, and select 290 | **task_runner** to display the logged messages from the executor utility, 291 | and **test_sample_task_task** to display the output of the sample task running 292 | on the Compute Engine instance. Click **Ok** to confirm. 293 | 294 | You can also see the topic in the Pub/Sub console now lists the subscription. 295 | 296 | ### Clean up 297 | 298 | Now that you have tested the sample, delete the cloud resources you created to 299 | prevent further billing for them on your account. 300 | 301 | * Delete the Compute Engine instance. 302 | 303 | $ gcloud compute instances delete cronworker --zone=us-central1-a 304 | 305 | 306 | * Delete the Cloud Scheduler job. 307 | 308 | You can delete the job from the Cloud Scheduler section of the 309 | [Developers Console](https://console.developers.google.com). 310 | 311 | * Delete the Cloud Pub/Sub topic. 312 | You can delete the topic and associated subscriptions from the Cloud Pub/Sub 313 | section of the [Developers Console](https://console.developers.google.com). 314 | 315 | 316 | ## License 317 | 318 | Copyright 2019 Google Inc. All Rights Reserved. 319 | 320 | Licensed under the Apache License, Version 2.0 (the "License"); 321 | you may not use this file except in compliance with the License. 322 | You may obtain a copy of the License at 323 | 324 | http://www.apache.org/licenses/LICENSE-2.0 325 | 326 | Unless required by applicable law or agreed to in writing, software 327 | distributed under the License is distributed on an "AS IS" BASIS, 328 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 329 | See the License for the specific language governing permissions and 330 | limitations under the License. 331 | --------------------------------------------------------------------------------