├── img ├── waze-ccp-1.png └── waze-ccp-2.png ├── requirements.txt ├── cron.yaml ├── app.yaml ├── CONTRIBUTING.md ├── README.md ├── LICENSE └── main.py /img/waze-ccp-1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google/waze-ccp-gcp/HEAD/img/waze-ccp-1.png -------------------------------------------------------------------------------- /img/waze-ccp-2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/google/waze-ccp-gcp/HEAD/img/waze-ccp-2.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | GoogleAppEngineCloudStorageClient 2 | google-cloud-bigquery 3 | unidecode 4 | Flask==2.0.2 5 | gunicorn==20.1.0 6 | appengine-python-standard>=0.3.1 7 | google-cloud-storage -------------------------------------------------------------------------------- /cron.yaml: -------------------------------------------------------------------------------- 1 | #Copyright 2018 Google LLC 2 | # 3 | #Licensed under the Apache License, Version 2.0 (the "License"); 4 | #you may not use this file except in compliance with the License. 5 | #You may obtain a copy of the License at 6 | # 7 | # https://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | #Unless required by applicable law or agreed to in writing, software 10 | #distributed under the License is distributed on an "AS IS" BASIS, 11 | #WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | #See the License for the specific language governing permissions and 13 | #limitations under the License. 14 | 15 | cron: 16 | - description: "Refresh Case Studies" 17 | url: /{guid}/ 18 | schedule: every 2 minutes -------------------------------------------------------------------------------- /app.yaml: -------------------------------------------------------------------------------- 1 | #Copyright 2018 Google LLC 2 | # 3 | #Licensed under the Apache License, Version 2.0 (the "License"); 4 | #you may not use this file except in compliance with the License. 5 | #You may obtain a copy of the License at 6 | # 7 | # https://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | #Unless required by applicable law or agreed to in writing, software 10 | #distributed under the License is distributed on an "AS IS" BASIS, 11 | #WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | #See the License for the specific language governing permissions and 13 | # under the License. 14 | 15 | runtime: python39 16 | 17 | app_engine_apis: true 18 | 19 | entrypoint: gunicorn -b :$PORT main:app 20 | 21 | default_expiration: "300s" 22 | 23 | inbound_services: 24 | - warmup -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # How to Contribute 2 | 3 | We'd love to accept your patches and contributions to this project. There are 4 | just a few small guidelines you need to follow. 5 | 6 | ## Contributor License Agreement 7 | 8 | Contributions to this project must be accompanied by a Contributor License 9 | Agreement. You (or your employer) retain the copyright to your contribution; 10 | this simply gives us permission to use and redistribute your contributions as 11 | part of the project. Head over to to see 12 | your current agreements on file or to sign a new one. 13 | 14 | You generally only need to submit a CLA once, so if you've already submitted one 15 | (even if it was for a different project), you probably don't need to do it 16 | again. 17 | 18 | ## Code reviews 19 | 20 | All submissions, including submissions by project members, require review. We 21 | use GitHub pull requests for this purpose. Consult 22 | [GitHub Help](https://help.github.com/articles/about-pull-requests/) for more 23 | information on using pull requests. 24 | 25 | ## Community Guidelines 26 | 27 | This project follows [Google's Open Source Community 28 | Guidelines](https://opensource.google.com/conduct/). 29 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # Waze CCP on GCP 3 | This is not an officially supported Google product, though support will be provided on a best-effort basis. 4 | 5 | Copyright 2018 Google LLC 6 | 7 | Licensed under the Apache License, Version 2.0 (the "License"); 8 | you may not use this file except in compliance with the License. 9 | You may obtain a copy of the License at 10 | 11 | https://www.apache.org/licenses/LICENSE-2.0 12 | 13 | Unless required by applicable law or agreed to in writing, software 14 | distributed under the License is distributed on an "AS IS" BASIS, 15 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 16 | See the License for the specific language governing permissions and 17 | limitations under the License. 18 | 19 | ### Introduction 20 | 21 | This AppEngine sample application is designed to process your Waze CCP JSON Feed into; BigQuery GIS tables for analysis, Google Cloud Storage as GeoJSON for use in desktop or web GIS applications, and, optionally into [Carto](https://carto.com/) for advanced spatial visualization. 22 | 23 | Join the [Group](https://groups.google.com/d/forum/waze-ccp-gcp) for updates and discussion 24 | 25 | ##### Google Cloud Products Used: 26 | - Google AppEngine 27 | - Google Cloud Datastore 28 | - Google BigQuery 29 | - Google Cloud Storage 30 | 31 | ### Getting Started 32 | 33 | ##### Step 1: Create and Configure your Project 34 | 35 | 36 | 37 | From here on out, we'll refer to your Project ID as **{project-id}** 38 | 39 | ##### Step 2: Configure BigQuery 40 | 41 | ###### 1. Enable the BigQuery API 42 | 43 | 44 | ###### 2. Go to BigQuery UI and Create a Dataset 45 | 46 | 47 | From here on out, we'll refer to your Dataset as **{bqDataset}** 48 | 49 | 50 | ##### Step 3. Go to Cloud Storage UI and Create a Bucket 51 | 52 | 53 | From here on out, we'll refer to your Bucket as **{gcsBucket}** 54 | 55 | ##### Step 4. Download source code, Install Dependencies, and Update Variables to Match your Project. 56 | ###### 1. Clone This Source Code: 57 | ```git clone https://github.com/google/waze-ccp-gcp.git ``` 58 | ###### 2. Update Variables Source Code: 59 | First, generate a [GUID](https://www.guidgenerator.com/). This will be referred to as **{guid}** and its just a way to create a non-guessable URL for the handler that Cron will call to update the Waze data every 10 minutes. 60 | 61 | - In cron.yaml 62 | - Line 17: Change **{guid}** to your **{guid}** 63 | - In main.py 64 | - Line 32: Change **{waze-url}** to your Waze CCP URL 65 | - Line 40: Change **{gcsBucket}** to your **{gcsBucket}** 66 | - Line 46: Change **{bqDataset}** to your **{bqDataset}** 67 | - Line 336: Change **{guid}** to your **{guid}** 68 | 69 | ###### 3. Check dependencies: 70 | This application utilizes various Python libraries, and App Engine will use the requirements.txt file during the build. You are free to check the versions and update accordingly. 71 | 72 | From the terminal, change directories to where you cloned the source code. 73 | ``` cd {your-app-folder} ``` 74 | 75 | 76 | ##### Step 5. Deploy your Application to AppEngine 77 | 78 | 79 | ###### 1. Using gcloud, Switch Project to your New Project: 80 | ``` gcloud config set project {project-id} ``` 81 | ###### 2. Using gcloud, Switch Project to your New Project: 82 | ``` gcloud app create ``` 83 | 84 | 85 | For the next step, you'll need to go to the [Cloud Build settings page](https://console.cloud.google.com/cloud-build/settings/service-account). 86 | 87 |

88 | 89 |

90 | 91 | This will trigger a second prompt to enable the App Engine Admin API. 92 | 93 |

94 | 95 |

96 | 97 | ###### 3. Using gcloud, Deploy your Application and Create a Case: 98 | 99 | Deploy the main application: 100 | ```gcloud beta app deploy {your-app-folder}/app.yaml``` 101 | 102 | Then start the cron job: 103 | ```gcloud app deploy {your-app-folder}/cron.yaml``` 104 | 105 | ###### 4. Secure your Application with Identity Aware Proxy: 106 | Even though you generated a GUID to serve as the URL path that AppEngine's Cron accesses to cause a data update, someone could discover it and maliciously hit that URL, and, they could also hit the /newCase/ endpoint. In order to prevent unwanted use of these URLs, you will enable IAP and lock down access to the application only to approved users (or just you). 107 | 108 | When you go to IAP settings for your project, you'll first have to set up a Credentials Screen (Oath2). 109 | Set the Application Type to "Internal". 110 | 111 | Then, under IAP - turn the IAP on for the AppEngine app: 112 | 113 | You can verify that IAP is working by visiting https://{project-id}.appspot.com in an Incognito browser. You should be redircted to your OAuth2 Credentials Screen, which shows that the IAP is working and protecting the entire application. 114 | 115 | ##### Step 6. Create your New Case Study 116 | 117 | Visit https://{project-id}.appspot.com/newCase/{case-name} to initiate a case, where {case-name} is any name you create for your case (like "Miami" in the example screenshot below). Be careful to just do this once per case you want to create. 118 | 119 | To confirm the Case Study was created, you can visit Datastore and confirm the Entity you expect to see is there. 120 | 121 | 122 | The Cron job described in cron.yaml will call https://{project-id}.appspot.com/{guid}/ and will start populating the tables in BigQuery. *Note - the cron function of AppEngine is internal so it is automatically inscope for IAP purposes. 123 | 124 | ##### Step 7. Investigating the Waze Data 125 | ###### BigQuery: 126 | In BigQuery, you should see the three tables (alerts, jams, irregularities) under your **{bqDataset}** 127 | 128 | 129 | These tables will contain all the **unique** elements from your **{waze-url}** 130 | 131 | 132 | ###### Data Studio: 133 | Once you have a few days worth of data, you can start experimenting with building Data Studio dashboards. 134 | 135 | If you come up with something interesting, be sure to share with the group: 136 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /main.py: -------------------------------------------------------------------------------- 1 | '''Copyright 2018 Google LLC 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | https://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License.''' 14 | 15 | import json 16 | from flask import Flask, request 17 | import datetime 18 | import logging 19 | import google.appengine.api 20 | from google.appengine.api import urlfetch 21 | import urllib 22 | import uuid 23 | from google.appengine.ext import ndb 24 | from google.cloud import storage 25 | from google.cloud import bigquery 26 | import unidecode 27 | 28 | app = Flask(__name__) 29 | app.wsgi_app = google.appengine.api.wrap_wsgi_app(app.wsgi_app) 30 | 31 | # Your Waze CCP URL 32 | wazeURL= '{waze-url}' 33 | 34 | """ **** Remove this line and the quotes here and at the bottom of this block if using Carto *** 35 | cartoURLBase = 'https://{your-carto-server}/user/{your-user}/api/v2/sql?' 36 | cartoAPIKey = '{your-carto-api-key}' 37 | """ 38 | 39 | #GCS Params 40 | bucket_name = '{gcsBucket}' 41 | gcs = storage.Client() 42 | BUCKET = gcs.get_bucket(bucket_name) 43 | gcsPath = '/'+ bucket_name +'/' 44 | 45 | #BigQuery Params 46 | bqDataset = '{bqDataset}' 47 | 48 | #BigQuery Schemas for the three tables that need to be recreated. 49 | #These are also referenced with each write. 50 | 51 | jamsSchema = [ 52 | bigquery.SchemaField('city','STRING',mode='Nullable'), 53 | bigquery.SchemaField('turntype','STRING',mode='Nullable'), 54 | bigquery.SchemaField('level','INT64',mode='Nullable'), 55 | bigquery.SchemaField('country','STRING',mode='Nullable'), 56 | bigquery.SchemaField('speedKMH','FLOAT64',mode='Nullable'), 57 | bigquery.SchemaField('delay','INT64',mode='Nullable'), 58 | bigquery.SchemaField('length','INT64',mode='Nullable'), 59 | bigquery.SchemaField('street','STRING',mode='Nullable'), 60 | bigquery.SchemaField('ms','INT64',mode='Nullable'), 61 | bigquery.SchemaField('ts','TIMESTAMP',mode='Nullable'), 62 | bigquery.SchemaField('endNode','STRING',mode='Nullable'), 63 | bigquery.SchemaField('geo','GEOGRAPHY',mode='Nullable'), 64 | bigquery.SchemaField('geoWKT','STRING',mode='Nullable'), 65 | bigquery.SchemaField('type','STRING',mode='Nullable'), 66 | bigquery.SchemaField('id','INT64',mode='Nullable'), 67 | bigquery.SchemaField('speed','FLOAT64',mode='Nullable'), 68 | bigquery.SchemaField('uuid','STRING',mode='Nullable'), 69 | bigquery.SchemaField('startNode','STRING',mode='Nullable'), 70 | ] 71 | 72 | 73 | alertsSchema = [ 74 | bigquery.SchemaField('city','STRING',mode='Nullable'), 75 | bigquery.SchemaField('confidence','INT64',mode='Nullable'), 76 | bigquery.SchemaField('nThumbsUp','INT64',mode='Nullable'), 77 | bigquery.SchemaField('street','STRING',mode='Nullable'), 78 | bigquery.SchemaField('uuid','STRING',mode='Nullable'), 79 | bigquery.SchemaField('country','STRING',mode='Nullable'), 80 | bigquery.SchemaField('type','STRING',mode='Nullable'), 81 | bigquery.SchemaField('subtype','STRING',mode='Nullable'), 82 | bigquery.SchemaField('roadType','INT64',mode='Nullable'), 83 | bigquery.SchemaField('reliability','INT64',mode='Nullable'), 84 | bigquery.SchemaField('magvar','INT64',mode='Nullable'), 85 | bigquery.SchemaField('reportRating','INT64',mode='Nullable'), 86 | bigquery.SchemaField('ms','INT64',mode='Nullable'), 87 | bigquery.SchemaField('ts','TIMESTAMP',mode='Nullable'), 88 | bigquery.SchemaField('reportDescription','STRING',mode='Nullable'), 89 | bigquery.SchemaField('geo','GEOGRAPHY',mode='Nullable'), 90 | bigquery.SchemaField('geoWKT','STRING',mode='Nullable') 91 | ] 92 | irregularitiesSchema =[ 93 | bigquery.SchemaField('trend','INT64',mode='Nullable'), 94 | bigquery.SchemaField('street','STRING',mode='Nullable'), 95 | bigquery.SchemaField('endNode','STRING',mode='Nullable'), 96 | bigquery.SchemaField('nImages','INT64',mode='Nullable'), 97 | bigquery.SchemaField('speed','FLOAT64',mode='Nullable'), 98 | bigquery.SchemaField('id','STRING',mode='Nullable'), 99 | bigquery.SchemaField('severity','INT64',mode='Nullable'), 100 | bigquery.SchemaField('type','STRING',mode='Nullable'), 101 | bigquery.SchemaField('highway','BOOL',mode='Nullable'), 102 | bigquery.SchemaField('nThumbsUp','INT64',mode='Nullable'), 103 | bigquery.SchemaField('seconds','INT64',mode='Nullable'), 104 | bigquery.SchemaField('alertsCount','INT64',mode='Nullable'), 105 | bigquery.SchemaField('detectionDateMS','INT64',mode='Nullable'), 106 | bigquery.SchemaField('detectionDateTS','TIMESTAMP',mode='Nullable'), 107 | bigquery.SchemaField('driversCount','INT64',mode='Nullable'), 108 | bigquery.SchemaField('geo','GEOGRAPHY',mode='Nullable'), 109 | bigquery.SchemaField('geoWKT','STRING',mode='Nullable'), 110 | bigquery.SchemaField('startNode','STRING',mode='Nullable'), 111 | bigquery.SchemaField('updateDateMS','INT64',mode='Nullable'), 112 | bigquery.SchemaField('updateDateTS','TIMESTAMP',mode='Nullable'), 113 | bigquery.SchemaField('regularSpeed','FLOAT64',mode='Nullable'), 114 | bigquery.SchemaField('country','STRING',mode='Nullable'), 115 | bigquery.SchemaField('length','INT64',mode='Nullable'), 116 | bigquery.SchemaField('delaySeconds','INT64',mode='Nullable'), 117 | bigquery.SchemaField('jamLevel','INT64',mode='Nullable'), 118 | bigquery.SchemaField('nComments','INT64',mode='Nullable'), 119 | bigquery.SchemaField('city','STRING',mode='Nullable'), 120 | bigquery.SchemaField('causeType','STRING',mode='Nullable'), 121 | bigquery.SchemaField('causeAlertUUID','STRING',mode='Nullable'), 122 | 123 | ] 124 | 125 | """ **** Remove this line and the quotes here and at the bottom of this block if using Carto *** 126 | 127 | #cartoSQL Scehmas and Values Strings 128 | 129 | cartoAlertsSchema = "(city text,\ 130 | confidence int,\ 131 | nThumbsUp int,\ 132 | street text,\ 133 | uuid text,\ 134 | country text,\ 135 | type text,\ 136 | subtype text,\ 137 | roadType int,\ 138 | reliability int,\ 139 | magvar int,\ 140 | reportRating int,\ 141 | ms bigint,\ 142 | ts timestamp,\ 143 | reportDescription text,\ 144 | the_geom geometry\ 145 | )" 146 | 147 | cartoAlertsFields = "(city,confidence,nThumbsUp,street,uuid,country,type,subtype,roadType,reliability,magvar,reportRating,ms,ts,reportDescription,the_geom)" 148 | 149 | cartoJamsScehma = "(city text,\ 150 | turntype text,\ 151 | level int,\ 152 | country text,\ 153 | speedKMH real,\ 154 | delay int,\ 155 | length int,\ 156 | street text,\ 157 | ms bigint,\ 158 | ts timestamp,\ 159 | endNode text,\ 160 | the_geom geometry,\ 161 | type text,\ 162 | id bigint,\ 163 | speed real,\ 164 | uuid text,\ 165 | startNode text\ 166 | )" 167 | 168 | cartoJamsFields = "(city,turntype,level,country,speedKMH,delay,length,street,ms,ts,endNode,type,id,speed,uuid,startNode,the_geom)" 169 | 170 | cartoIrregularitiesScehma = "(trend int,\ 171 | street text,\ 172 | endNode text,\ 173 | nImages int,\ 174 | speed real,\ 175 | id text,\ 176 | severity int,\ 177 | type text,\ 178 | highway bool,\ 179 | nThumbsUp int,\ 180 | seconds int,\ 181 | alertsCount int,\ 182 | detectionDateMS bigint,\ 183 | detectionDateTS timestamp,\ 184 | driversCount int,\ 185 | the_geom geometry,\ 186 | startNode text,\ 187 | updateDateMS bigint,\ 188 | updateDateTS timestamp,\ 189 | regularSpeed real,\ 190 | country text,\ 191 | length int,\ 192 | delaySeconds int,\ 193 | jamLevel int,\ 194 | nComments int,\ 195 | city text,\ 196 | causeType text,\ 197 | causeAlertUUID text\ 198 | )" 199 | 200 | cartoIrregularitiesFields = "(trend,street,endNode,nImages,speed,id,severity,type,highway,nThumbsUp,seconds,alertsCount,detectionDateMS,detectionDateTS,driversCount,startNode,updateDateMS,updateDateTS,regularSpeed,country,length,delaySeconds,jamLevel,nComments,city,causeType,causeAlertUUID,the_geom)" 201 | """ 202 | #Define a Datastore ndb Model for each Waze "case" (Study area) that you want to monitor 203 | class caseModel(ndb.Model): 204 | uid = ndb.StringProperty() 205 | name = ndb.StringProperty() 206 | day = ndb.StringProperty() 207 | 208 | #This application will track unique entities for Jams, Alerts, and Irregularities. 209 | #We don't want to write duplicate events to BigQuery if they persist through the refresh window. 210 | 211 | #Define an ndb Model to track unique Jams. 212 | class uniqueJams(ndb.Model): 213 | tableUUID = ndb.StringProperty() 214 | jamsUUID = ndb.StringProperty() 215 | 216 | #Define an ndb Model to track unique Alerts. 217 | class uniqueAlerts(ndb.Model): 218 | tableUUID = ndb.StringProperty() 219 | alertsUUID = ndb.StringProperty() 220 | 221 | #Define an ndb Model to track unique Irregularities. 222 | class uniqueIrregularities(ndb.Model): 223 | tableUUID = ndb.StringProperty() 224 | irregularitiesUUID = ndb.StringProperty() 225 | 226 | #App Request Handler to create a new Case. 227 | #Called ONCE as: {your-app}.appspot.com/newCase/?name={your-case-name} 228 | #This handler can be disabled after you create your first case if you 229 | #only intend to create one. 230 | @app.route("/newCase/", methods=['GET']) 231 | def newCase(): 232 | uid = uuid.uuid4() 233 | name = request.args.get("name") 234 | day = datetime.datetime.now().strftime("%Y-%m-%d") 235 | if not name: 236 | name = "" 237 | 238 | #Write the new Case details to Datastore 239 | wazePut = caseModel(uid=str(uid),day=day,name=name) 240 | wazeKey = wazePut.put() 241 | 242 | #Create the BigQuery Client 243 | client = bigquery.Client() 244 | datasetRef = client.dataset(bqDataset) 245 | tableSuffix = str(uid).replace('-','_') 246 | 247 | #Create the Jams Table 248 | jamsTable = 'jams_' + tableSuffix 249 | tableRef = datasetRef.table(jamsTable) 250 | table = bigquery.Table(tableRef,schema=jamsSchema) 251 | table = client.create_table(table) 252 | assert table.table_id == jamsTable 253 | 254 | 255 | #Create the Alerts Table 256 | alertsTable = 'alerts_' + tableSuffix 257 | tableRef = datasetRef.table(alertsTable) 258 | table = bigquery.Table(tableRef,schema=alertsSchema) 259 | table = client.create_table(table) 260 | assert table.table_id == alertsTable 261 | 262 | 263 | #Create the Irregularities Table 264 | irregularitiesTable = 'irregularities_' + tableSuffix 265 | tableRef = datasetRef.table(irregularitiesTable) 266 | table = bigquery.Table(tableRef,schema=irregularitiesSchema) 267 | table = client.create_table(table) 268 | assert table.table_id == irregularitiesTable 269 | 270 | """ **** Remove this line and the quotes here and at the bottom of this block to also register the Case Study to Carto using CartoSQL *** 271 | 272 | # Create and register Alerts Table in Carto 273 | 274 | url = cartoURLBase + "q=CREATE TABLE " + alertsTable + " " + cartoAlertsSchema + "&api_key=" + cartoAPIKey 275 | logging.info(url) 276 | try: 277 | result = urlfetch.fetch(url.replace(" ", "%20"), validate_certificate=True) 278 | if result.status_code == 200: 279 | url = cartoURLBase + "q=SELECT cdb_cartodbfytable('" + alertsTable + "')&api_key=" + cartoAPIKey 280 | logging.info(url) 281 | try: 282 | result = urlfetch.fetch(url.replace(" ", "%20"), validate_certificate=True) 283 | if result.status_code == 200: 284 | logging.info('Created and Cartodbfyd Table ' + alertsTable) 285 | except urlfetch.Error: 286 | logging.exception('Caught exception Cartodbfying Waze Alerts Table ' + alertsTable) 287 | else: 288 | logging.exception(result.status_code) 289 | except urlfetch.Error: 290 | logging.exception('Caught exception Creating Waze Alerts Table ' + alertsTable) 291 | 292 | # Create and register Jams Table in Carto 293 | 294 | url = cartoURLBase + "q=CREATE TABLE " + jamsTable + " " + cartoJamsScehma + "&api_key=" + cartoAPIKey 295 | logging.info(url) 296 | try: 297 | result = urlfetch.fetch(url.replace(" ", "%20"), validate_certificate=True) 298 | if result.status_code == 200: 299 | url = cartoURLBase + "q=SELECT cdb_cartodbfytable('" + jamsTable + "')&api_key=" + cartoAPIKey 300 | logging.info(url) 301 | try: 302 | result = urlfetch.fetch(url.replace(" ", "%20"), validate_certificate=True) 303 | if result.status_code == 200: 304 | logging.info('Created and Cartodbfyd Table ' + jamsTable) 305 | except urlfetch.Error: 306 | logging.exception('Caught exception Cartodbfying Waze Jams Table ' + jamsTable) 307 | else: 308 | logging.exception(result.status_code) 309 | except urlfetch.Error: 310 | logging.exception('Caught exception Creating Waze Jams Table ' + jamsTable) 311 | 312 | # Create and register Irregularities Table in Carto 313 | 314 | url = cartoURLBase + "q=CREATE TABLE " + irregularitiesTable + " " + cartoIrregularitiesScehma + "&api_key=" + cartoAPIKey 315 | logging.info(url) 316 | try: 317 | result = urlfetch.fetch(url.replace(" ", "%20"), validate_certificate=True) 318 | if result.status_code == 200: 319 | url = cartoURLBase + "q=SELECT cdb_cartodbfytable('" + irregularitiesTable + "')&api_key=" + cartoAPIKey 320 | logging.info(url) 321 | try: 322 | result = urlfetch.fetch(url.replace(" ", "%20"), validate_certificate=True) 323 | if result.status_code == 200: 324 | logging.info('Created and Cartodbfyd Table ' + irregularitiesTable) 325 | except urlfetch.Error: 326 | logging.exception('Caught exception Cartodbfying Waze Jams Table ' + irregularitiesTable) 327 | else: 328 | logging.exception(result.status_code) 329 | except urlfetch.Error: 330 | logging.exception('Caught exception Creating Waze Jams Table ' + irregularitiesTable) 331 | 332 | """ 333 | 334 | #Called at your set cron interval, this function loops through all the cases in datastore 335 | #And adds a task to update the case's tables in Taskqeue 336 | @app.route('/{guid}/', methods=['GET']) 337 | def updateCaseStudies(): 338 | caseStudies = caseModel.query() 339 | for case in caseStudies: 340 | updateCase(case) 341 | return 'OK' 342 | 343 | #For each Case, update each table 344 | def updateCase(case): 345 | url = wazeURL 346 | try: 347 | result = urlfetch.fetch(url) 348 | if result.status_code == 200: 349 | data = json.loads(result.content) 350 | #Get the 3 components from the Waze CCP JSON Response 351 | alerts = data.get('alerts') 352 | jams = data.get('jams') 353 | irregularities = data.get('irregularities') 354 | if alerts is not None: 355 | processAlerts(data['alerts'],case.uid,case.day) 356 | if jams is not None: 357 | processJams(data['jams'],case.uid,case.day) 358 | if irregularities is not None: 359 | processIrregularities(data['irregularities'],case.uid,case.day) 360 | else: 361 | logging.exception(result.status_code) 362 | except urlfetch.Error: 363 | logging.exception('Caught exception fetching Waze URL') 364 | 365 | #Process the Alerts 366 | def processAlerts(alerts,uid,day): 367 | now = datetime.datetime.now().strftime("%s") 368 | features = [] 369 | bqRows = [] 370 | cartoRows =[] 371 | for item in alerts: 372 | ms = item.get('pubMillis') 373 | itemTimeStamp = datetime.datetime.fromtimestamp(ms/1000.0) 374 | caseTimeMinimum = datetime.datetime.strptime(day, "%Y-%m-%d") 375 | if itemTimeStamp >= caseTimeMinimum: 376 | timestamp = datetime.datetime.fromtimestamp(ms/1000.0).strftime("%Y-%m-%d %H:%M:%S") 377 | city = item.get('city') 378 | street = item.get('street') 379 | confidence = item.get('confidence') 380 | nThumbsUp = item.get('nThumbsUp') 381 | uuid = item.get('uuid') 382 | country = item.get('country') 383 | subtype = item.get('subtype') 384 | roadType = item.get('roadType') 385 | reliability = item.get('reliability') 386 | magvar = item.get('magvar') 387 | alertType = item.get('type') 388 | reportRating = item.get('reportRating') 389 | reportDescription = item.get('reportDescription') 390 | longitude = item.get('location').get('x') 391 | latitude = item.get('location').get('y') 392 | #Create GCS GeoJSON Properties 393 | properties = {"city": city, 394 | "street": street, 395 | "confidence": confidence, 396 | "nThumbsUp": nThumbsUp, 397 | "uuid": uuid, 398 | "country": country, 399 | "subtype": subtype, 400 | "roadType": roadType, 401 | "reliability": reliability, 402 | "magvar": magvar, 403 | "type": alertType, 404 | "reportRating": reportRating, 405 | "pubMillis": ms, 406 | "timestamp": timestamp, 407 | "reportDescription": reportDescription, 408 | } 409 | geometry = {"type": "Point", 410 | "coordinates": [longitude, latitude] 411 | } 412 | features.append({"type": "Feature", "properties": properties, "geometry": geometry}) 413 | #BigQuery Row Creation 414 | datastoreQuery = uniqueAlerts.query(uniqueAlerts.tableUUID == uid, uniqueAlerts.alertsUUID == uuid) 415 | datastoreCheck = datastoreQuery.get() 416 | if not datastoreCheck: 417 | bqRow = {"city": city, 418 | "street": street, 419 | "confidence": confidence, 420 | "nThumbsUp": nThumbsUp, 421 | "uuid": uuid, 422 | "country": country, 423 | "subtype": subtype, 424 | "roadType": roadType, 425 | "reliability": reliability, 426 | "magvar": magvar, 427 | "type": alertType, 428 | "reportRating": reportRating, 429 | "ms": ms, 430 | "ts": timestamp, 431 | "reportDescription": reportDescription, 432 | "geo": "Point(" + str(longitude) + " " + str(latitude) + ")", 433 | "geoWKT": "Point(" + str(longitude) + " " + str(latitude) + ")" 434 | } 435 | bqRows.append(bqRow) 436 | """ **** Remove this line and the quotes here and at the bottom of this block to also use Carto *** 437 | 438 | # Create Carto Row 439 | if city is not None: 440 | city = unidecode.unidecode(city) 441 | if street is not None: 442 | street = unidecode.unidecode(street) 443 | if uuid is not None: 444 | uuid = unidecode.unidecode(uuid) 445 | if country is not None: 446 | country = unidecode.unidecode(country) 447 | if alertType is not None: 448 | alertType = unidecode.unidecode(alertType) 449 | if subtype is not None: 450 | subtype = unidecode.unidecode(subtype) 451 | if reportDescription is not None: 452 | reportDescription = unidecode.unidecode(reportDescription) 453 | 454 | cartoRow = "('{city}',{confidence},{nThumbsUp},'{street}','{uuid}','{country}','{type}','{subtype}',{roadType},{reliability},{magvar},{reportRating},{ms},'{ts}','{reportDescription}',ST_GeomFromText({the_geom},4326))".format( 455 | city=city, confidence=confidence, nThumbsUp=nThumbsUp, street=street, uuid=uuid, country=country, 456 | type=alertType, subtype=subtype, roadType=roadType, reliability=reliability, magvar=magvar, 457 | reportRating=reportRating, ms=ms, ts=timestamp, reportDescription=reportDescription, the_geom="'Point(" + str(longitude) + " " + str(latitude) + ")'") 458 | cartoRows.append(cartoRow) 459 | """ 460 | 461 | #Add uuid to Datastore 462 | alertPut = uniqueAlerts(tableUUID=str(uid), alertsUUID=str(uuid)) 463 | alertKey = alertPut.put() 464 | 465 | #Write GeoJSONs to GCS 466 | alertGeoJSON = json.dumps({"type": "FeatureCollection", "features": features}) 467 | writeGeoJSON(alertGeoJSON, uid + '/' + uid + '-alerts.geojson') 468 | writeGeoJSON(alertGeoJSON, uid + '/' + uid + '-' + now +'-alerts.geojson') 469 | 470 | alertsTable = 'alerts_' + str(uid).replace('-','_') 471 | #Stream new Rows to BigQuery 472 | if bqRows: 473 | client = bigquery.Client() 474 | datasetRef = client.dataset(bqDataset) 475 | tableRef = datasetRef.table(alertsTable) 476 | table = bigquery.Table(tableRef,schema=alertsSchema) 477 | errors = client.insert_rows(table, bqRows) 478 | try: 479 | assert errors == [] 480 | logging.info(errors) 481 | except AssertionError as e: 482 | logging.warning(e) 483 | """ **** Remove this line and the quotes here and at the bottom of this block to also use Carto *** 484 | # Load Rows to Carto 485 | 486 | if cartoRows: 487 | cartoQuery = "INSERT INTO " + alertsTable + " " + cartoAlertsFields + "VALUES " + ",".join(cartoRows) 488 | cartoQuery = cartoQuery.replace("'None'","null").replace("None","null") 489 | cartoQueryEncoded = urllib.urlencode({"q":cartoQuery}) 490 | # logging.info(cartoQuery) 491 | # logging.info(cartoQueryEncoded) 492 | 493 | 494 | # url = cartoURLBase + cartoQueryEncoded + "&api_key=" + cartoAPIKey 495 | url = cartoURLBase + "&api_key=" + cartoAPIKey 496 | # logging.info(url) 497 | try: 498 | result = urlfetch.fetch(url, validate_certificate=True,method=urlfetch.POST,payload=cartoQueryEncoded) 499 | if result.status_code == 200: 500 | logging.info('Inserted Alerts Data to Carto') 501 | else: 502 | logging.exception(result.status_code) 503 | logging.exception(cartoQuery) 504 | writeSQLError(cartoQuery,gcsPath + 'carto_errors/' + uid + '-' + now + '-alerts.txt') 505 | except urlfetch.Error: 506 | logging.exception('Caught exception Inserting Data to Alerts Table ' + alertsTable) 507 | """ 508 | #Process the Jams 509 | def processJams(jams,uid,day): 510 | now = datetime.datetime.now().strftime("%s") 511 | features = [] 512 | bqRows =[] 513 | cartoRows = [] 514 | for item in jams: 515 | ms = item.get('pubMillis') 516 | itemTimeStamp = datetime.datetime.fromtimestamp(ms/1000.0) 517 | caseTimeMinimum = datetime.datetime.strptime(day, "%Y-%m-%d") 518 | if itemTimeStamp >= caseTimeMinimum: 519 | timestamp = datetime.datetime.fromtimestamp(ms/1000.0).strftime("%Y-%m-%d %H:%M:%S") 520 | city = item.get('city') 521 | turnType = item.get('turnType') 522 | level = item.get('level') 523 | country = item.get('country') 524 | segments = item.get('segments') 525 | speedKMH = item.get('speedKMH') 526 | roadType = item.get('roadType') 527 | delay = item.get('delay') 528 | length = item.get('length') 529 | street = item.get('street') 530 | endNode = item.get('endNode') 531 | jamType = item.get('type') 532 | iD = item.get('id') 533 | uuid = item.get('uuid') 534 | speed = item.get('speed') 535 | startNode = item.get('startNode') 536 | #Create GCS GeoJSON Properties 537 | properties = {"city": city, 538 | "turnType": turnType, 539 | "level": level, 540 | "country": country, 541 | "segments": segments, 542 | "speedKMH": speedKMH, 543 | "roadType": roadType, 544 | "delay": delay, 545 | "length": length, 546 | "street": street, 547 | "pubMillis": ms, 548 | "timestamp": timestamp, 549 | "endNode": endNode, 550 | "type": jamType, 551 | "id": iD, 552 | "speed": speed, 553 | "uuid": uuid, 554 | "startNode": startNode 555 | } 556 | #Create WKT Polyline for BigQuery 557 | coordinates = [] 558 | bqLineString = '' 559 | for vertex in item.get('line'): 560 | longitude = vertex.get('x') 561 | latitude = vertex.get('y') 562 | coordinate = [longitude, latitude] 563 | bqLineString += str(longitude) + " " + str(latitude) + ', ' 564 | coordinates.append(coordinate) 565 | geometry = {"type": "LineString", 566 | "coordinates": coordinates 567 | } 568 | features.append({"type": "Feature", "properties": properties, "geometry": geometry}) 569 | #BigQuery Row Creation 570 | datastoreQuery = uniqueJams.query(uniqueJams.tableUUID == str(uid), uniqueJams.jamsUUID == str(uuid)) 571 | datastoreCheck = datastoreQuery.get() 572 | if not datastoreCheck: 573 | bqRow = {"city": city, 574 | "turnType": turnType, 575 | "level": level, 576 | "country": country, 577 | "segments": segments, 578 | "speedKMH": speedKMH, 579 | "roadType": roadType, 580 | "delay": delay, 581 | "length": length, 582 | "street": street, 583 | "ms": ms, 584 | "ts": timestamp, 585 | "endNode": endNode, 586 | "type": jamType, 587 | "id": iD, 588 | "speed": speed, 589 | "uuid": uuid, 590 | "startNode": startNode, 591 | "geo": "LineString(" + bqLineString[:-2] + ")", 592 | "geoWKT": "LineString(" + bqLineString[:-2] + ")" 593 | } 594 | bqRows.append(bqRow) 595 | """" **** Remove this line and the quotes here and at the bottom of this block to also use Carto *** 596 | # Create Carto Row 597 | 598 | if city is not None: 599 | city = unidecode.unidecode(city) 600 | if turnType is not None: 601 | turnType = unidecode.unidecode(turnType) 602 | if country is not None: 603 | country = unidecode.unidecode(country) 604 | if street is not None: 605 | street = unidecode.unidecode(street) 606 | if endNode is not None: 607 | endNode = unidecode.unidecode(endNode) 608 | if jamType is not None: 609 | jamType = unidecode.unidecode(jamType) 610 | if startNode is not None: 611 | startNode = unidecode.unidecode(startNode) 612 | 613 | cartoRow = "('{city}','{turntype}',{level},'{country}',{speedKMH},{delay},{length},'{street}',{ms},'{ts}','{endNode}','{type}',{id},{speed},'{uuid}','{startNode}',ST_GeomFromText({the_geom},4326))".format( 614 | city=city, turntype=turnType, level=level, country=country, speedKMH=speedKMH, delay=delay, 615 | length=length, street=street, ms=ms, ts=timestamp, endNode=endNode, 616 | type=jamType, id=iD, speed=speed, uuid=uuid, startNode=startNode, the_geom="'LineString(" + bqLineString[:-2] + ")'") 617 | cartoRows.append(cartoRow) 618 | """ 619 | 620 | #Add uuid to Datastore 621 | jamPut = uniqueJams(tableUUID=str(uid), jamsUUID=str(uuid)) 622 | jamKey = jamPut.put() 623 | jamsGeoJSON = json.dumps({"type": "FeatureCollection", "features": features}) 624 | writeGeoJSON(jamsGeoJSON, uid + '/' + uid + '-jams.geojson') 625 | writeGeoJSON(jamsGeoJSON, uid + '/' + uid + '-' + now +'-jams.geojson') 626 | 627 | #Stream new Rows to BigQuery 628 | if bqRows: 629 | client = bigquery.Client() 630 | datasetRef = client.dataset(bqDataset) 631 | jamsTable = 'jams_' + str(uid).replace('-','_') 632 | tableRef = datasetRef.table(jamsTable) 633 | table = bigquery.Table(tableRef,schema=jamsSchema) 634 | errors = client.insert_rows(table, bqRows) 635 | try: 636 | assert errors == [] 637 | logging.info(errors) 638 | except AssertionError as e: 639 | logging.warning(e) 640 | """ **** Remove this line and the quotes here and at the bottom of this block to also use Carto *** 641 | # Load Rows to Carto 642 | 643 | if cartoRows: 644 | cartoQuery = "INSERT INTO " + jamsTable + " " + cartoJamsFields + "VALUES " + ",".join(cartoRows) 645 | cartoQuery = cartoQuery.replace("'None'","null").replace("None","null") 646 | cartoQueryEncoded = urllib.urlencode({"q":cartoQuery}) 647 | logging.info(len(cartoQueryEncoded)) 648 | # logging.info(cartoQuery) 649 | # logging.info(cartoQueryEncoded) 650 | 651 | 652 | url = cartoURLBase + "&api_key=" + cartoAPIKey 653 | logging.info(url) 654 | try: 655 | result = urlfetch.fetch(url, validate_certificate=True,method=urlfetch.POST,payload=cartoQueryEncoded) 656 | if result.status_code == 200: 657 | logging.info('Inserted Jams Data to Carto') 658 | else: 659 | logging.exception(result.status_code) 660 | logging.exception(cartoQuery) 661 | writeSQLError(cartoQuery,gcsPath + 'carto_errors/' + uid + '-' + now + '-jams.txt') 662 | except urlfetch.Error: 663 | logging.exception('Caught exception Inserting Data to Jams Table ' + jamsTable) 664 | """ 665 | 666 | #Process the Irregularities 667 | def processIrregularities(irregularities,uid,day): 668 | now = datetime.datetime.now().strftime("%s") 669 | features = [] 670 | bqRows = [] 671 | cartoRows = [] 672 | 673 | for item in irregularities: 674 | detectionDateMS = item.get('detectionDateMillis') 675 | itemTimeStamp = datetime.datetime.fromtimestamp(detectionDateMS/1000.0) 676 | caseTimeMinimum = datetime.datetime.strptime(day, "%Y-%m-%d") 677 | if itemTimeStamp >= caseTimeMinimum: 678 | detectionDateTS = datetime.datetime.fromtimestamp(detectionDateMS/1000.0).strftime("%Y-%m-%d %H:%M:%S") 679 | updateDateMS = item.get('updateDateMillis') 680 | updateDateTS = datetime.datetime.fromtimestamp(updateDateMS/1000.0).strftime("%Y-%m-%d %H:%M:%S") 681 | causeAlert = item.get('causeAlert') 682 | if causeAlert is not None: 683 | causeAlertUUID = causeAlert.get('uuid') 684 | else: 685 | causeAlertUUID = None 686 | trend = item.get('trend') 687 | street = item.get('street') 688 | endNode = item.get('endNode') 689 | nImages = item.get('nImages') 690 | speed = item.get('speed') 691 | iD = item.get('id') 692 | severity = item.get('severity') 693 | irregularityType = item.get('type') 694 | highway = item.get('highway') 695 | nThumbsUp = item.get('nThumbsUp') 696 | seconds = item.get('seconds') 697 | alertsCount = item.get('alertsCount') 698 | driversCount = item.get('driversCount') 699 | startNode = item.get('startNode') 700 | regularSpeed = item.get('regularSpeed') 701 | country = item.get('country') 702 | length = item.get('length') 703 | delaySeconds = item.get('delaySeconds') 704 | jamLevel = item.get('jamLevel') 705 | nComments = item.get('nComments') 706 | city = item.get('city') 707 | causeType = item.get('causeType') 708 | #Create GCS GeoJSON Properties 709 | properties = {"trend": trend, 710 | "street": street, 711 | "endNode": endNode, 712 | "nImages": nImages, 713 | "speed": speed, 714 | "id": iD, 715 | "severity": severity, 716 | "type": irregularityType, 717 | "highway": highway, 718 | "nThumbsUp": nThumbsUp, 719 | "seconds": seconds , 720 | "alertsCount": alertsCount, 721 | "driversCount": driversCount , 722 | "startNode": startNode, 723 | "regularSpeed": regularSpeed, 724 | "country": country, 725 | "length": length, 726 | "delaySeconds": delaySeconds, 727 | "jamLevel": jamLevel , 728 | "nComments": nComments, 729 | "city": city, 730 | "causeType": causeType, 731 | "detectionDateMS": detectionDateMS, 732 | "detectionDateTS": detectionDateTS, 733 | "updateDateMS": updateDateMS, 734 | "updateDateTS": updateDateTS, 735 | "causeAlertUUID": causeAlertUUID 736 | } 737 | #Create WKT Polyline for BigQuery 738 | coordinates = [] 739 | bqLineString = '' 740 | for vertex in item.get('line'): 741 | longitude = vertex.get('x') 742 | latitude = vertex.get('y') 743 | coordinate = [longitude, latitude] 744 | bqLineString += str(longitude) + " " + str(latitude) + ', ' 745 | coordinates.append(coordinate) 746 | geometry = {"type": "LineString", 747 | "coordinates": coordinates 748 | } 749 | features.append({"type": "Feature", "properties": properties, "geometry": geometry}) 750 | 751 | #BigQuery Row Creation 752 | datastoreQuery = uniqueIrregularities.query(uniqueIrregularities.tableUUID == str(uid), uniqueIrregularities.irregularitiesUUID == str(iD)+str(updateDateMS)) 753 | datastoreCheck = datastoreQuery.get() 754 | if not datastoreCheck: 755 | bqRow = {"trend": trend, 756 | "street": street, 757 | "endNode": endNode, 758 | "nImages": nImages, 759 | "speed": speed, 760 | "id": iD, 761 | "severity": severity, 762 | "type": irregularityType, 763 | "highway": highway, 764 | "nThumbsUp": nThumbsUp, 765 | "seconds": seconds , 766 | "alertsCount": alertsCount, 767 | "driversCount": driversCount , 768 | "startNode": startNode, 769 | "regularSpeed": regularSpeed, 770 | "country": country, 771 | "length": length, 772 | "delaySeconds": delaySeconds, 773 | "jamLevel": jamLevel , 774 | "nComments": nComments, 775 | "city": city, 776 | "causeType": causeType, 777 | "detectionDateMS": detectionDateMS, 778 | "detectionDateTS": detectionDateTS, 779 | "updateDateMS": updateDateMS, 780 | "updateDateTS": updateDateTS, 781 | "causeAlertUUID": causeAlertUUID, 782 | "geo": "LineString(" + bqLineString[:-2] + ")", 783 | "geoWKT": "LineString(" + bqLineString[:-2] + ")" 784 | } 785 | bqRows.append(bqRow) 786 | """ **** Remove this line and the quotes here and at the bottom of this block to also use Carto *** 787 | # Create Carto Row 788 | 789 | if street is not None: 790 | street = unidecode.unidecode(street) 791 | if endNode is not None: 792 | endNode = unidecode.unidecode(endNode) 793 | if irregularityType is not None: 794 | irregularityType = unidecode.unidecode(irregularityType) 795 | if startNode is not None: 796 | startNode = unidecode.unidecode(startNode) 797 | if city is not None: 798 | city = unidecode.unidecode(city) 799 | if causeType is not None: 800 | causeType = unidecode.unidecode(causeType) 801 | if causeAlertUUID is not None: 802 | causeAlertUUID = unidecode.unidecode(causeAlertUUID) 803 | 804 | cartoRow = "({trend},'{street}','{endNode}',{nImages},{speed},'{id}',{severity},'{type}',{highway},{nThumbsUp},{seconds},{alertsCount},{detectionDateMS},'{detectionDateTS}',{driversCount},'{startNode}',{updateDateMS},'{updateDateTS}',{regularSpeed},'{country}',{length},{delaySeconds},{jamLevel},{nComments},'{city}','{causeType}','{causeAlertUUID}',ST_GeomFromText({the_geom},4326))".format( 805 | trend=trend, street=street, endNode=endNode, nImages=nImages,speed=speed,id=iD,severity=severity,type=irregularityType, 806 | highway=highway, nThumbsUp=nThumbsUp,seconds=seconds,alertsCount=alertsCount,detectionDateMS=detectionDateMS,detectionDateTS=detectionDateTS, 807 | driversCount=driversCount,startNode=startNode,updateDateMS=updateDateMS, updateDateTS=updateDateTS, regularSpeed=regularSpeed, country=country,length=length, 808 | delaySeconds=delaySeconds, jamLevel=jamLevel,nComments=nComments, city=city, causeType=causeType, causeAlertUUID=causeAlertUUID, the_geom="'LineString(" + bqLineString[:-2] + ")'") 809 | cartoRows.append(cartoRow) 810 | """ 811 | 812 | #Add uuid to Datastore 813 | irregularityPut = uniqueIrregularities(tableUUID=str(uid), irregularitiesUUID=str(iD)+str(updateDateMS)) 814 | irregularityKey = irregularityPut.put() 815 | irregularitiesGeoJSON = json.dumps({"type": "FeatureCollection", "features": features}) 816 | writeGeoJSON(irregularitiesGeoJSON, uid + '/' + uid + '-irregularities.geojson') 817 | writeGeoJSON(irregularitiesGeoJSON, uid + '/' + uid + '-' + now +'-irregularities.geojson') 818 | # logging.info(irregularitiesGeoJSON) 819 | 820 | #Stream new Rows to BigQuery 821 | if bqRows: 822 | client = bigquery.Client() 823 | datasetRef = client.dataset(bqDataset) 824 | irregularitiesTable = 'irregularities_' + str(uid).replace('-','_') 825 | tableRef = datasetRef.table(irregularitiesTable) 826 | table = bigquery.Table(tableRef,schema=irregularitiesSchema) 827 | errors = client.insert_rows(table, bqRows) 828 | try: 829 | assert errors == [] 830 | logging.info(errors) 831 | except AssertionError as e: 832 | logging.warning(e) 833 | 834 | """ **** Remove this line and the quotes here and at the bottom of this block to also use Carto *** 835 | # Load Rows to Carto 836 | 837 | if cartoRows: 838 | cartoQuery = "INSERT INTO " + irregularitiesTable + " " + cartoIrregularitiesFields + "VALUES " + ",".join(cartoRows) 839 | cartoQuery = cartoQuery.replace("'None'","null").replace("None","null") 840 | cartoQueryEncoded = urllib.urlencode({"q":cartoQuery}) 841 | # logging.info(cartoQuery) 842 | # logging.info(cartoQueryEncoded) 843 | 844 | 845 | url = cartoURLBase + "&api_key=" + cartoAPIKey 846 | # logging.info(url) 847 | try: 848 | result = urlfetch.fetch(url, validate_certificate=True,method=urlfetch.POST,payload=cartoQueryEncoded) 849 | if result.status_code == 200: 850 | logging.info('Inserted Irregularities Data to Carto') 851 | else: 852 | logging.exception(result.status_code) 853 | logging.exception(cartoQuery) 854 | writeSQLError(cartoQuery,gcsPath + 'carto_errors/' + uid + '-' + now + '-irregularities.txt') 855 | except urlfetch.Error: 856 | logging.exception('Caught exception Inserting Data to Irregularities Table ' + irregularitiesTable) 857 | """ 858 | #Write the GeoJSON to GCS 859 | def writeGeoJSON(geoJSON,filename): 860 | blob = BUCKET.blob(filename) 861 | blob.upload_from_string( 862 | data=json.dumps(geoJSON), 863 | content_type='application/json' 864 | ) 865 | result = filename + ' upload complete' 866 | return {'response' : result} 867 | 868 | """ **** Remove this line and the quotes here and at the bottom of this block to also use Carto *** 869 | def writeSQLError(sql,filename): 870 | blob = BUCKET.blob(filename) 871 | blob.upload_from_string( 872 | data=sql, 873 | content_type='text/html' 874 | ) 875 | """ --------------------------------------------------------------------------------