43 |
44 |
45 | ### Creating the Client ID & Secret
46 |
47 | The Client ID and Secret are specific to the Google Cloud project you are
48 | hosting the solution on and can be created in the *API & Services* panel of
49 | the Cloud Console. The process is
50 | [documented on the Google developers site.](https://developers.google.com/adwords/api/docs/guides/authentication#create_a_client_id_and_client_secret")
51 |
52 | You will need to follow the instructions for a web app. The steps should
53 | be:
54 | 1. Configure the consent screen by clicking the button at the top-right of the
55 | pane and filling out the form.56 |
57 |
58 | 1. Return to the Credentials page and select **Create credentials** and then
59 | **OAuth client ID**
60 | 1. Choose *Web application* as the application type.
61 | 1. Name the client ID
62 | 1. Add the URL to the config service end_config page to the list of authorized
63 | redirect URIs. The URI will have the form `https://control-service-dot-
64 | <defaultHostname>.appspot.com/config_end` If you're not sure of your default
65 | hostname, you can use the command `gcloud app describe` to find it.
66 | 1. Create the **Client ID** and **Client Secret**.
67 | 1. Copy the ID and secret for use in the solution configuration.
68 |
69 | ### Getting a Google Ads Developer Token
70 |
71 | You will require a production developer token to be able to use the Opportunity
72 | Finder with real accounts.
73 |
74 | You can find your developer token in the Google Ads frontend. Select the *API
75 | Center* option from the *Tools & Settings* menu.76 |
77 | 78 | The Developer token should be the first item on the page. If the access level is 79 | *Test Account*, please click the link to apply for basic access. The solution 80 | will not work without a developer token with access to production accounts.
81 |
82 |
83 | ### Requesting a PageSpeed Insights API Key
84 |
85 | To make bulk requests to the Google PageSpeed Insights (PSI) API, you will
86 | require an API key. This is free, and allows you to make a maximum of 25k
87 | requests per day. The key is connected to a Google Cloud project, so you will
88 | need to create one before getting the PSI API key.
89 |
90 | To create an API key, follow the following steps:
91 | 1. From the cloud console side menu, open the **APIs & Services -> Credentials**
92 | page.
93 | 1. Click the *CREATE CREDENTIALS* button and select *API Key*
94 | 1. On the resulting dialog, click *RESTRICT KEY*
95 | 1. On the resutling page, rename the API key to something memorable.
96 | 1. Under API restrictions, choose the radio button labeled *Restrict key*
97 | 1. Using the dropdown, select the PageSpeed Insights API.98 |
99 | 1. Save the changes made to the API key, and copy the key for use with the tool.
--------------------------------------------------------------------------------
/docs/deploy.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: Speed Opportunity Finder - Deployment
3 | ---
4 |
19 |
20 | # Landing Page Speed Opportunity Finder
21 |
22 | ## Deploying the solution
23 |
24 | 1. If you haven't already, create a
25 | [new Google Cloud Project](https://cloud.google.com/resource-manager/docs/creating-managing-projects").
26 | You can also reuse an existing project if you prefer, but you will need to
27 | ensure none of the existing deployed apps and big query tables conflict with
28 | those required by the opportunity finder.
29 |
30 | 1. In the Google cloud console, clone the
31 | [solution repository](https://github.com/google/speed-opportunity-finder)
32 |
33 | 1. Run the installer script with the command `$ bash install.sh`. If you don't
34 | already have an app engine application for this cloud project, you will be asked
35 | to create one. Follow the directions and be sure to select the correct region.
36 |
37 | 1. Enable Firestore in native mode for the project. Please be sure to select
38 | native mode. This cannot be changed, so if you choose Datastore mode, or are
39 | trying to reuse a project where Datastore mode is already in use, you will need
40 | to create a new project.41 |
42 |
43 | ## Finishing the deployment {#finish-deployment}
44 |
45 | 1. Collect all of the [credentials](./credentials) required to use the tool.
46 |
47 | 1. Enter the credentials you gathered on the *config* page of your deployed app.
48 | This should be located at `http://config-service.<defaultHostname>.appspot.com/config`
49 |
50 | 1. Once the credentials are entered, click the *Start OAuth Flow* button and
51 | complete the OAuth flow to provide the solution access to your Ads accounts. If
52 | successful, the page below will be shown. **Please Note**: This step must be
53 | carried out by someone with access to the MCC you are using for the solution.
54 |
55 | 1. To test the solution and start the first round of data collection, ping the
56 | controller service. This should be located at `https://controller-service.<
57 | defaultHostname>.appspot.com`. Please note that the page will likely time out
58 | before you receive a reply. To see if it worked, check the Cloud logs and the
59 | BigQuery tables.
60 |
61 | 1. Set the Cloud project firewall rules to only allow access to the services,
62 | excepting the config service, from the app itself. This will ensure outside
63 | actors are not using project resources or adding unwanted data to the BigQuery
64 | tables.
65 |
66 | ## Attaching the dashboard
67 |
68 | The final step is to attach the Data Studio dashboard to the backend. Follow
69 | the steps below to and be sure to copy the data sources first to ensure the
70 | calculated fields in the data sources are maintained.
71 |
72 | 1. Make a copy of both of the data studio data sources and connect them
73 | to appropriate BigQuery tables in your cloud project. The connectors can
74 | be found here:
75 | * [lh_data data source](https://datastudio.google.com/datasources/62e27e18-338c-4f54-80a7-fe3d43302858)
76 | * [ads_data data source](https://datastudio.google.com/datasources/b689944c-7d8a-4123-8fbf-4eaa5bedd2c9)
77 | Use the copy icon, located next to the *CREATE REPORT* button, to make the copy.78 |
79 |
80 | 1. After confirming you want to copy the data source, a new window should open.
81 | Select your project, the agency_dashboard dataset, and the appropriate table for
82 | the connector being copied.83 |
84 |
85 | 1. Make a copy of the
86 | [dashboard template](https://datastudio.google.com/u/2/reporting/3638a403-30c9-49e9-82a9-9f79ddd8999c/page/X3bCB/preview)
87 | To copy the template, click the *USE TEMPLATE* button at the top right of the page.88 |
89 |
90 | 1. In the resulting dialog, choose the BigQuery data sources you created with
91 | your project tables.92 |
93 |
--------------------------------------------------------------------------------
/install.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # Copyright 2020 Google Inc.
4 | #
5 | # Licensed under the Apache License, Version 2.0 (the "License");
6 | # you may not use this file except in compliance with the License.
7 | # You may obtain a copy of the License at
8 | #
9 | # http://www.apache.org/licenses/LICENSE-2.0
10 | #
11 | # Unless required by applicable law or agreed to in writing, software
12 | # distributed under the License is distributed on an "AS IS" BASIS,
13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14 | # See the License for the specific language governing permissions and
15 | # limitations under the License.
16 | #
17 | # Installation script for the agency dashboard solution. Please see the included
18 | # README file for more information.
19 |
20 | set -eu
21 |
22 | #######################################
23 | # Prints the standard error message and exits.
24 | #######################################
25 | function err() {
26 | echo "[$(date +'%Y-%m-%dT%H:%M:%S%z')]: Error $*. Please check the output
27 | above this and address the issue before trying again." >&2
28 | exit 1
29 | }
30 |
31 | #######################################
32 | # Enables the Google cloud services required for the solution.
33 | #######################################
34 | function enable_gcloud_services() {
35 | declare -a gcloud_services
36 | gcloud_services=("bigquery" "googleads" "cloudtasks" "firestore")
37 | gcloud_services+=("pagespeedonline")
38 |
39 | local gservice
40 | for gservice in "${gcloud_services[@]}"; do
41 | if ! gcloud services enable "${gservice}".googleapis.com; then
42 | err "enabling ${gservice}"
43 | fi
44 | done
45 | }
46 |
47 | #######################################
48 | # Deploys the solution's service to app engine.
49 | #
50 | # The default service must be deployed first.
51 | #######################################
52 | function deploy_solution_services() {
53 | declare -a solution_services
54 | solution_services=("Ads-Task-Handler" "Config-Service" "Controller-Service")
55 | solution_services+=("LH-Task-Handler")
56 |
57 | if ! gcloud app describe; then
58 | if ! gcloud app create; then
59 | echo "This solution requires an app engine project. Please create one." >&2
60 | exit 1
61 | fi
62 | fi
63 | if ! gcloud app deploy -q Default-Service/service.yaml; then
64 | err "deploying Default-Service"
65 | fi
66 | # the location chosen for the service is needed as a environment variable
67 | # in the controller service to add tasks to the task queues.
68 | if ! grep -qF 'APP_LOCATION' Controller-Service/service.yaml; then
69 | app_location="$(gcloud tasks locations list | awk 'FNR==2 {print $1}')"
70 | echo " APP_LOCATION: ${app_location}" >> Controller-Service/service.yaml
71 | fi
72 |
73 | local service
74 | for service in "${solution_services[@]}"; do
75 |
76 | if ! gcloud app deploy -q "${service}"/service.yaml; then
77 | err "deploying ${service} service"
78 | fi
79 | done
80 | }
81 |
82 | #######################################
83 | # Creates the bigquery tables and views required for the solution.
84 | #######################################
85 | function create_bq_tables() {
86 | declare -a solution_tables
87 | solution_tables=("ads_data" "lh_data")
88 |
89 | local bq_datasets
90 | bq_datasets=$(bq ls)
91 |
92 | if ! [[ "${bq_datasets}" =~ agency_dashboard ]]; then
93 | if ! bq mk --dataset \
94 | --description "Agency dashboard data" \
95 | "${project_id}":agency_dashboard; then
96 | err "creating bigquery dataset"
97 | fi
98 | fi
99 |
100 | local bq_tables
101 | bq_tables=$(bq ls agency_dashboard)
102 |
103 | local table
104 | for table in "${solution_tables[@]}"; do
105 | if ! [[ "${bq_tables}" =~ $table ]]; then
106 | if ! bq mk --table agency_dashboard."${table}" schemas/"${table}".json; then
107 | err "creating bigquery table ${table}"
108 | fi
109 | fi
110 | done
111 |
112 | if ! [[ "${bq_tables}" =~ "base_urls" ]]; then
113 | if ! bq mk --use_legacy_sql=false --view \
114 | "SELECT DISTINCT BaseUrl FROM \`${project_id}.agency_dashboard.ads_data\` \
115 | WHERE Cost > 0 \
116 | AND Date = (SELECT MAX(Date) FROM \`${project_id}.agency_dashboard.ads_data\`)" \
117 | agency_dashboard.base_urls; then
118 | err "creating bigquery view"
119 | fi
120 | fi
121 | }
122 |
123 | #######################################
124 | # Deploys the configuration files for the solution.
125 | #
126 | # The dispatch rules must be deployed before the scheduler rules so that the
127 | # controller endpoint exists.
128 | #######################################
129 | function deploy_config_files() {
130 | if ! gcloud app deploy -q queue.yaml; then
131 | err "deploying task queues"
132 | fi
133 |
134 | if ! gcloud app deploy -q dispatch.yaml; then
135 | err "deploying the dispatch rules"
136 | fi
137 |
138 | if ! gcloud app deploy -q cron.yaml; then
139 | err "deploying scheduled jobs"
140 | fi
141 | }
142 |
143 | function main() {
144 | local script_location
145 | local changed_dir
146 | script_location="${0%/*}"
147 | if [[ "${0}" != "${script_location}" ]] && [[ -n "${script_location}" ]]; then
148 | cd "${script_location}" || (echo "Could not cd to script location"; return 1)
149 | changed_dir=$(true)
150 | fi
151 | #get the project ID and set the default project
152 | read -rp 'Please enter your Google Cloud Project ID: ' project_id
153 | if [[ -z "$project_id" ]]; then
154 | echo "A project ID is required to continue." >&2
155 | exit 1
156 | fi
157 | if ! gcloud config set project "$project_id"; then
158 | err "setting the default cloud project"
159 | fi
160 |
161 | echo "Enabling Google cloud services"
162 | enable_gcloud_services
163 | echo "Deploying solution app engine services"
164 | deploy_solution_services
165 | echo "Creating bigquery tables"
166 | create_bq_tables
167 | echo "Deploying final configuration files"
168 | deploy_config_files
169 |
170 | if [[ "${changed_dir}" ]]; then
171 | cd - || echo "Could not change back to original dir."
172 | fi
173 | }
174 |
175 | main "$@"
176 |
177 |
178 |
--------------------------------------------------------------------------------
/LH-Task-Handler/lh-task-handler.js:
--------------------------------------------------------------------------------
1 | /**
2 | * Copyright 2020 Google Inc.
3 | *
4 | * Licensed under the Apache License, Version 2.0 (the "License");
5 | * you may not use this file except in compliance with the License.
6 | * You may obtain a copy of the License at
7 | *
8 | * http://www.apache.org/licenses/LICENSE-2.0
9 | *
10 | * Unless required by applicable law or agreed to in writing, software
11 | * distributed under the License is distributed on an "AS IS" BASIS,
12 | * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | * See the License for the specific language governing permissions and
14 | * limitations under the License.
15 | */
16 |
17 | /**
18 | * @fileoverview A Cloud Tasks task handler for the agency dashboard.
19 | *
20 | * This takes a single url as a query parameter, performs a lighthouse audit on
21 | * it, and then inserts the relevant metrics into bigquery.
22 | */
23 |
24 | const express = require('express');
25 | const {BigQuery} = require('@google-cloud/bigquery');
26 | const request = require('request-promise-native');
27 | const {Logging} = require('@google-cloud/logging');
28 | const Firestore = require('@google-cloud/firestore');
29 |
30 | const app = express();
31 | app.enable('trust proxy');
32 |
33 | const logging = new Logging(process.env.GOOGLE_CLOUD_PROJECT);
34 | const log = logging.log('agency-lh-task');
35 |
36 | const AUDITS = {
37 | 'largest-contentful-paint': 'largest_contentful_paint',
38 | 'total-blocking-time': 'total_blocking_time',
39 | 'cumulative-layout-shift': 'cumulative_layout_shift',
40 | 'first-contentful-paint': 'first_contentful_paint',
41 | 'first-meaningful-paint': 'first_meaningful_paint',
42 | 'speed-index': 'speed_index',
43 | 'estimated-input-latency': 'estimated_input_latency',
44 | 'total-blocking-time': 'total_blocking_time',
45 | 'max-potential-fid': 'max_potential_fid',
46 | 'server-response-time': 'server_response_time',
47 | 'first-cpu-idle': 'first_cpu_idle',
48 | 'interactive': 'interactive',
49 | 'mainthread-work-breakdown': 'mainthread_work_breakdown',
50 | 'bootup-time': 'bootup_time',
51 | 'network-requests': 'network_requests',
52 | 'network-rtt': 'network_rtt',
53 | 'network-server-latency': 'network_server_latency',
54 | 'total-byte-weight': 'total_byte_weight',
55 | 'dom-size': 'dom_size',
56 | };
57 |
58 | /**
59 | * Responds to get requests to handle the case of performing a lighthouse audit
60 | * using the lighthouse audit service and then inserting the results into
61 | * bigquery.
62 | */
63 | app.get('*', async (req, res, next) => {
64 | const testUrl = req.query.url;
65 | if (!testUrl) {
66 | log.error('Missing query parameter');
67 | res.status(200).json({'error': 'Missing query parameter'});
68 | return;
69 | }
70 |
71 | try {
72 | const firestore = new Firestore();
73 | const credentialDoc = firestore.doc('agency_ads/credentials');
74 | const credentialSnapshot = await credentialDoc.get();
75 | const psiApiKey = credentialSnapshot.get('psi_api_token');
76 |
77 | const apiUrl = 'https://www.googleapis.com/pagespeedonline/v5/runPagespeed';
78 | const stdParams = `category=performance&strategy=mobile&key=${psiApiKey}`;
79 | const requestUrl = `${apiUrl}?url=${testUrl}&${stdParams}`;
80 |
81 | let psiResult = undefined;
82 | const row = {};
83 |
84 | try {
85 | psiResult = await request(requestUrl, {json: true});
86 | } catch (error) {
87 | psiError = JSON.parse(error.message.slice(6));
88 | if ('error' in psiError) {
89 | const today = new Date();
90 | row.date = today.toISOString().slice(0, 10);
91 | row.url = testUrl;
92 | // If the page returned an error, we store the error code.
93 | // If PSI returns an error, we set the error code to -2 if the quota was
94 | // exceeeded or to -1 for everything else.
95 | if (psiError.error.message.includes('Status code')) {
96 | row.error_code = psiError.error.message
97 | .match(/Status code: (\d+)/)[1];
98 | } else if (psiError.error.code === 429) {
99 | row.error_code = -2;
100 | } else {
101 | row.error_code = -1;
102 | }
103 | row.error_message = psiError.error.message;
104 |
105 | log.error(`Lighthouse Error (${requestUrl}): ${psiError.error.message}`);
106 | }
107 | }
108 | if (psiResult) {
109 | const lhAudit = psiResult.lighthouseResult;
110 | row.date = lhAudit.fetchTime.slice(0, 10);
111 | row.url = testUrl;
112 | row.lhscore = lhAudit.categories.performance.score;
113 | for (a of Object.keys(AUDITS)) {
114 | try {
115 | row[AUDITS[a]] = lhAudit.audits[a].numericValue;
116 | } catch (e) {
117 | if (e instanceof TypeError) {
118 | log.error(`Problem accessing ${a} in the PSI audit object.`);
119 | } else {
120 | throw e;
121 | }
122 | }
123 | }
124 | // the individual sizes of resources
125 | for (part of lhAudit.audits['resource-summary'].details.items) {
126 | if (part.resourceType === 'total') {
127 | continue;
128 | } else {
129 | const rowName = part.resourceType.replace('-', '_') + '_size';
130 | row[rowName] = part.transferSize;
131 | }
132 | }
133 | }
134 |
135 | const bqClient = new BigQuery();
136 | const table = bqClient.dataset('agency_dashboard').table('lh_data');
137 | await table.insert(row);
138 | log.info(`Success for ${testUrl}`);
139 | res.status(201).json({'url': testUrl});
140 | } catch (err) {
141 | console.error(`LH Task ERROR: ${err.message}`);
142 | if ('errors' in err) {
143 | console.error(`ERROR CAUGHT: ${err.errors[0].row}`);
144 | }
145 | if ('name' in err && err.name === 'PartialFailureError') {
146 | for (e of err.errors) {
147 | for (e2 of e.errors) {
148 | log.error(`BigQuery error: ${e2.message}`);
149 | }
150 | }
151 | }
152 | return next(err);
153 | }
154 | });
155 |
156 | const PORT = process.env.PORT || 8082;
157 | app.listen(PORT, () => {
158 | console.log(`lh-task-handler started on port ${PORT}`);
159 | });
160 |
--------------------------------------------------------------------------------
/Controller-Service/main.py:
--------------------------------------------------------------------------------
1 | """
2 | Copyright 2020 Google Inc.
3 |
4 | Licensed under the Apache License, Version 2.0 (the "License");
5 | you may not use this file except in compliance with the License.
6 | You may obtain a copy of the License at
7 |
8 | http://www.apache.org/licenses/LICENSE-2.0
9 |
10 | Unless required by applicable law or agreed to in writing, software
11 | distributed under the License is distributed on an "AS IS" BASIS,
12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | See the License for the specific language governing permissions and
14 | limitations under the License.
15 | """
16 | """This service drives the agency dashboard solution by triggering landing page
17 |
18 | report requests and lighthouse audits.
19 |
20 | This module runs as a web service and is designed to be targeted by Google Cloud
21 | Scheduler. Using credentials stored in firestore, it first requests all of the
22 | CIDs associated with the stored MCC ID from Ads. Using those CIDs, it creates
23 | Cloud tasks to have landing page reports retrieved and stored in bigquery. Once
24 | the landing page report tasks have been completed, it creates tasks to run
25 | lighthouse audits on all of the URLs in the project's base_urls bigquery table
26 | and have them stored in bigquery.
27 | """
28 |
29 | import datetime
30 | import logging
31 | import os
32 | import time
33 | import urllib
34 |
35 | from bottle import Bottle
36 | from bottle import HTTPError
37 | from googleads import adwords
38 |
39 | import google.cloud.bigquery
40 | import google.cloud.exceptions
41 | import google.cloud.firestore
42 | import google.cloud.logging
43 | import google.cloud.tasks
44 |
45 | app = Bottle()
46 |
47 |
48 | def get_cids(ads_client, mcc_id):
49 | """Fetches all of the cids under the given mcc.
50 |
51 | The cids are placed in a set because it's possible to have a client attached
52 | to multiple mccs.
53 |
54 | Args:
55 | ads_client: an instance of AdwordsClient already authenticated for the MCC
56 | to be used.
57 | mcc_id: the mcc to get the client accounts for.
58 |
59 | Returns:
60 | A set of all of the client ids under the given mcc.
61 | """
62 | cids = set()
63 | mcc_ids = set()
64 | mcc_ids.add(mcc_id)
65 |
66 | while mcc_ids:
67 | current_mcc = mcc_ids.pop()
68 | ads_client.SetClientCustomerId(current_mcc)
69 | mcc_service = ads_client.GetService(
70 | 'ManagedCustomerService', version='v201809')
71 | selector = {'fields': ['CustomerId', 'Name', 'CanManageClients']}
72 | result = mcc_service.get(selector)
73 |
74 | for record in result.entries:
75 | if str(record.customerId) == mcc_id:
76 | continue
77 | elif record.canManageClients:
78 | mcc_ids.add(record.customerId)
79 | else:
80 | cids.add((record.customerId, record.name))
81 |
82 | return cids
83 |
84 |
85 | @app.route('/')
86 | @app.route('/controller')
87 | def start_update():
88 | """This route triggers the process of updating the ads and lighthouse data."""
89 |
90 | logging_client = google.cloud.logging.Client()
91 | logging_handler = logging_client.get_default_handler()
92 | logger = logging.getLogger('Controller-Service')
93 | logger.addHandler(logging_handler)
94 |
95 | project_name = os.environ['GOOGLE_CLOUD_PROJECT']
96 | project_location = os.environ['APP_LOCATION']
97 | today = datetime.date.today().isoformat()
98 |
99 | ads_client = None
100 | task_client = None
101 | ads_queue_path = None
102 | last_run_dates = None
103 | config_doc = None
104 |
105 | try:
106 | storage_client = google.cloud.firestore.Client()
107 | credentials_doc = (
108 | storage_client.collection('agency_ads').document('credentials').get())
109 | mcc_id = credentials_doc.get('mcc_id')
110 | developer_token = credentials_doc.get('developer_token')
111 | client_id = credentials_doc.get('client_id')
112 | client_secret = credentials_doc.get('client_secret')
113 | refresh_token = credentials_doc.get('refresh_token')
114 | ads_credentials = ('adwords:\n' + f' client_customer_id: {mcc_id}\n' +
115 | f' developer_token: {developer_token}\n' +
116 | f' client_id: {client_id}\n' +
117 | f' client_secret: {client_secret}\n' +
118 | f' refresh_token: {refresh_token}')
119 | ads_client = adwords.AdWordsClient.LoadFromString(ads_credentials)
120 | except google.cloud.exceptions.NotFound:
121 | logger.exception('Unable to load ads credentials.')
122 | raise HTTPError(500, 'Unable to load Ads credentials.')
123 |
124 | try:
125 | config_doc = storage_client.collection('agency_ads').document('config')
126 | config_doc_snapshot = config_doc.get()
127 | last_run_dates = config_doc_snapshot.get('last_run')
128 | except google.cloud.exceptions.NotFound:
129 | logger.exception('Exception retrieving last run dates.')
130 | raise HTTPError(500, 'Config document not found in firestore')
131 | except KeyError:
132 | logger.info('Last run dates not in firestore.')
133 | last_run_dates = {}
134 | if last_run_dates is None:
135 | config_doc.create({'last_run': {}})
136 | last_run_dates = {}
137 |
138 | try:
139 | task_client = google.cloud.tasks.CloudTasksClient()
140 | ads_queue_path = task_client.queue_path(project_name, project_location,
141 | 'ads-queue')
142 | except:
143 | logger.exception('Exception creating tasks client')
144 | raise HTTPError(500, 'Exception creating tasks client.')
145 |
146 | try:
147 | cids = get_cids(ads_client, mcc_id.replace('-', ''))
148 | except:
149 | logger.exception('Exception while getting cids')
150 | raise HTTPError(500, 'Exception while getting cids')
151 | for client in cids:
152 | try:
153 | cid = client[0]
154 | client_name = client[1]
155 | task_url = (f'http://ads-task-handler.{project_name}.appspot.com' +
156 | f'?cid={cid}&' + f'name={urllib.parse.quote(client_name)}')
157 | if cid in last_run_dates:
158 | task_url += f'&startdate={last_run_dates[cid]}'
159 | task = {'http_request': {'http_method': 'GET', 'url': task_url}}
160 | except TypeError:
161 | logger.exception('Error creating task_url for record %s - %s', cid,
162 | client_name)
163 | continue
164 | try:
165 | task_client.create_task(ads_queue_path, task)
166 | config_doc.update({f'last_run.{cid}': today})
167 | except (google.api_core.exceptions.GoogleAPICallError,
168 | google.api_core.exceptions.RetryError, ValueError):
169 | logger.exception('Exception queing ads queries (url = %s)', task_url)
170 | except google.cloud.exceptions.NotFound:
171 | logger.exception('Exception updating ads last_run firebase doc.')
172 |
173 | # polling the queue to ensure all the URLs are available before starting the
174 | # lighthouse tests. It would be nice to have a better, parallel way to do
175 | # this.
176 | ads_queue_size = True
177 | while ads_queue_size:
178 | time.sleep(30)
179 | ads_queue_list = list(task_client.list_tasks(ads_queue_path))
180 | ads_queue_size = len(ads_queue_list)
181 |
182 | try:
183 | bigquery_client = google.cloud.bigquery.Client()
184 | url_query = f'''SELECT BaseUrl
185 | FROM `{project_name}.agency_dashboard.base_urls`'''
186 | query_response = bigquery_client.query(url_query)
187 | except:
188 | logger.exception('Exception querying for URLs')
189 | raise HTTPError(500, 'Exception querying for URLs')
190 |
191 | try:
192 | lh_queue_path = task_client.queue_path(project_name, project_location,
193 | 'lh-queue')
194 | for row in query_response:
195 | url = urllib.parse.quote(row['BaseUrl'])
196 | task = {
197 | 'http_request': {
198 | 'http_method':
199 | 'GET',
200 | 'url':
201 | f'http://lh-task-handler.{project_name}.appspot.com?url={url}'
202 | }
203 | }
204 | task_client.create_task(lh_queue_path, task)
205 | except:
206 | logger.exception('Excpetion queue lh tasks.')
207 | raise HTTPError(500, 'Exception queuing lh tasks.')
208 |
209 |
210 | if __name__ == '__main__':
211 | app.run(host='localhost', port=8084)
212 |
--------------------------------------------------------------------------------
/Config-Service/main.py:
--------------------------------------------------------------------------------
1 | """
2 | Copyright 2020 Google Inc.
3 |
4 | Licensed under the Apache License, Version 2.0 (the "License");
5 | you may not use this file except in compliance with the License.
6 | You may obtain a copy of the License at
7 |
8 | http://www.apache.org/licenses/LICENSE-2.0
9 |
10 | Unless required by applicable law or agreed to in writing, software
11 | distributed under the License is distributed on an "AS IS" BASIS,
12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | See the License for the specific language governing permissions and
14 | limitations under the License.
15 | """
16 | """This module provides a service for adding credentials to the agency
17 | dashboard.
18 |
19 | The module is used in conjunction with the rest of the agency dashboard solution
20 | to provide the credentials required to acccess the Ads accounts that will be the
21 | basis for reporting.
22 |
23 | This must be deployed as part of the agency dashboard Google Cloud AppEngine
24 | app. Optionally, this component can be left out of the deployment and the
25 | required credentials can be inserted manually into the project firestore store.
26 |
27 | The required credentials must be located in a document stored at
28 | /agency_ads/credentials and have the following fields:
29 | - mcc_id: the account id of the management account to be used with the app
30 | - client_id: the client ID created in the Google API console
31 | - client_secret: the client secret generated with the above client id
32 | - developer_token: the developer token for the account to be used with the app
33 | - refresh_token: an oauth2 refresh token generated using the client id and
34 | secret above
35 | """
36 |
37 | import logging
38 | import urllib.parse
39 |
40 | from bottle import Bottle
41 | from bottle import HTTPError
42 | from bottle import redirect
43 | from bottle import request
44 | from bottle import template
45 | from bottle import view
46 | from google_auth_oauthlib.flow import Flow
47 | from oauthlib.oauth2.rfc6749.errors import InvalidGrantError
48 |
49 | import google.cloud.exceptions
50 | import google.cloud.firestore
51 | import google.cloud.logging
52 |
53 | app = Bottle()
54 | logging_client = google.cloud.logging.Client()
55 | logging_handler = logging_client.get_default_handler()
56 | logger = logging.getLogger('Config-Service')
57 | logger.addHandler(logging_handler)
58 |
59 |
60 | def client_config_exists():
61 | """Retrives the Adwords client configuation from firestore.
62 |
63 | Retrieves the Adwords client configuration from firestore if it is available.
64 | If the client_config doc is not found in the data store, a NotFound exception
65 | is raised.
66 |
67 | Returns:
68 | A dict containing the web client configuration for Google Adwords
69 |
70 | Raises:
71 | google.cloud.exceptions.NotFound: the client_config document was not in the
72 | agency_ads collection
73 | """
74 | storage_client = google.cloud.firestore.Client()
75 | try:
76 | client_config = (
77 | storage_client.collection('agency_ads').document('credentials').get())
78 | try:
79 | client_config.get('client_id')
80 | except KeyError:
81 | return False
82 | except google.cloud.exceptions.NotFound:
83 | return False
84 |
85 | return True
86 |
87 |
88 | @app.route('/config')
89 | @view('start_config')
90 | def start_ads_config():
91 | """This route starts the oauth authentication flow.
92 |
93 | The route displays a page that allows the user to input the information
94 | required
95 |
96 | The route first checks for the existance of the config. If there is an
97 | existing config, a message is show to the user to help them avoid writing over
98 | it.
99 |
100 | Returns:
101 | An html page with fields for entering credentials.
102 |
103 | """
104 | return {'client_config_exists': client_config_exists()}
105 |
106 |
107 | @app.route('/config_end')
108 | def end_ads_config():
109 | """This route completes the oauth flow and saves the returned refresh token.
110 |
111 | The oauth state saved in the start of the flow is also removed from the
112 | credentials doc.
113 |
114 | Returns:
115 | On success, an HTML page is returned letting the user know the authorization
116 | was successful. On failure, the user is returned the page to enter their
117 | credentials with an error message.
118 | """
119 | storage_client = google.cloud.firestore.Client()
120 | try:
121 | credentials_doc = (
122 | storage_client.collection('agency_ads').document('credentials').get())
123 | client_id = credentials_doc.get('client_id')
124 | client_secret = credentials_doc.get('client_secret')
125 | oauth_state = credentials_doc.get('oauth_state')
126 | except (google.cloud.exceptions.NotFound, KeyError):
127 | logger.exception('Unable to load ads credentials.')
128 | return template(
129 | 'start_config', error='Error loading credentials after oauth')
130 |
131 | auth_code = request.query.get('code')
132 | if not auth_code:
133 | return template('start_config', error='No authorization code in request.')
134 |
135 | client_config = {
136 | 'web': {
137 | 'client_id': client_id,
138 | 'client_secret': client_secret,
139 | 'auth_uri': 'https://accounts.google.com/o/oauth2/auth',
140 | 'token_uri': 'https://accounts.google.com/o/oauth2/token',
141 | }
142 | }
143 | flow = Flow.from_client_config(
144 | client_config,
145 | scopes=['https://www.googleapis.com/auth/adwords'],
146 | state=oauth_state)
147 |
148 | req = urllib.parse.urlparse(request.url)
149 | redirect_uri = f'{req.scheme}://{req.hostname}/config_end'
150 | flow.redirect_uri = redirect_uri
151 | try:
152 | flow.fetch_token(code=auth_code)
153 | except InvalidGrantError:
154 | logger.exception('Error fetching refresh token after oauth')
155 | return template('start_config', error='Error retreiving refresh token.')
156 |
157 | storage_client = google.cloud.firestore.Client()
158 | try:
159 | credentials_doc = storage_client.collection('agency_ads').document(
160 | 'credentials')
161 | credentials_doc.update({
162 | 'refresh_token': flow.credentials.refresh_token,
163 | 'oauth_state': google.cloud.firestore.DELETE_FIELD
164 | })
165 | except google.cloud.exceptions.NotFound:
166 | logger.exception('Error finding or updating credentials in firestore.')
167 | return template(
168 | 'start_config', error='Error updating credentials doc in firestore')
169 |
170 | return template('config_complete')
171 |
172 |
173 | @app.route('/config_upload_client', method='POST')
174 | def save_client_config():
175 | """Route used to save the client configuration JSON file to firestore."""
176 | mcc_id = request.forms.get('mcc_id')
177 | client_id = request.forms.get('client_id')
178 | client_secret = request.forms.get('client_secret')
179 | developer_token = request.forms.get('developer_token')
180 | psi_api_token = request.forms.get('psi_api_token')
181 |
182 | client_config = {
183 | 'web': {
184 | 'client_id': client_id,
185 | 'client_secret': client_secret,
186 | 'auth_uri': 'https://accounts.google.com/o/oauth2/auth',
187 | 'token_uri': 'https://accounts.google.com/o/oauth2/token',
188 | }
189 | }
190 | flow = Flow.from_client_config(
191 | client_config, scopes=['https://www.googleapis.com/auth/adwords'])
192 | req = urllib.parse.urlparse(request.url)
193 | redirect_uri = f'{req.scheme}://{req.hostname}/config_end'
194 | flow.redirect_uri = redirect_uri
195 | auth_url, oauth_state = flow.authorization_url(prompt='consent')
196 |
197 | storage_client = google.cloud.firestore.Client()
198 | try:
199 | credentials_doc = storage_client.collection('agency_ads').document(
200 | 'credentials')
201 | credentials_content = {
202 | 'mcc_id': mcc_id,
203 | 'client_id': client_id,
204 | 'client_secret': client_secret,
205 | 'developer_token': developer_token,
206 | 'psi_api_token': psi_api_token,
207 | 'oauth_state': oauth_state
208 | }
209 | credentials_doc.set(credentials_content)
210 | except google.cloud.exceptions.NotFound:
211 | logger.exception('Unable to find ads credentials.')
212 | raise HTTPError(500, 'Unable to find ads credentials.')
213 |
214 | redirect(auth_url)
215 |
--------------------------------------------------------------------------------
/Ads-Task-Handler/main.py:
--------------------------------------------------------------------------------
1 | """
2 | Copyright 2020 Google Inc.
3 |
4 | Licensed under the Apache License, Version 2.0 (the "License");
5 | you may not use this file except in compliance with the License.
6 | You may obtain a copy of the License at
7 |
8 | http://www.apache.org/licenses/LICENSE-2.0
9 |
10 | Unless required by applicable law or agreed to in writing, software
11 | distributed under the License is distributed on an "AS IS" BASIS,
12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 | See the License for the specific language governing permissions and
14 | limitations under the License.
15 | """
16 | """This service retrieves and forwards landing page reports in CSV format.
17 |
18 | The Ads-Task-Handler downloads the landing page report for the Google Ads
19 | account with the given CID. The report is then enriched with the name provided,
20 | the CID, and a base URL for the landing page. The base URL is the landing page
21 | URL stripped of parameters after {ignore} and any trailing '?' or '/'.
22 |
23 | The enriched landing page report is then loaded into the
24 | agency_dashboard.ads_data biqquery table of the working Google CLoud project.
25 | """
26 |
27 | import csv
28 | from datetime import datetime
29 | import logging
30 | import os
31 |
32 | from bottle import Bottle
33 | from bottle import HTTPError
34 | from bottle import request
35 | from googleads import adwords
36 |
37 | from google.cloud import bigquery
38 | from google.cloud import firestore
39 | import google.cloud.exceptions
40 | import google.cloud.logging
41 |
42 | app = Bottle()
43 |
44 | logging_client = google.cloud.logging.Client()
45 | logging_handler = logging_client.get_default_handler()
46 | logger = logging.getLogger('Ads-Service')
47 | logger.setLevel(logging.INFO)
48 | logger.addHandler(logging_handler)
49 |
50 | # The columns of the landing page report with the name as returned by the API as
51 | # the key and the name used in the select statement as the value.
52 | REPORT_COLS = {
53 | 'Campaign ID': 'CampaignId',
54 | 'Campaign': 'CampaignName',
55 | 'Campaign state': 'CampaignStatus',
56 | 'Landing page': 'UnexpandedFinalUrlString',
57 | 'Day': 'Date',
58 | 'Device': 'Device',
59 | 'Active View avg. CPM': 'ActiveViewCpm',
60 | 'Active View viewable CTR': 'ActiveViewCtr',
61 | 'Active View viewable impressions': 'ActiveViewImpressions',
62 | 'Active View measurable impr. / impr.': 'ActiveViewMeasurability',
63 | 'Active View measurable cost': 'ActiveViewMeasurableCost',
64 | 'Active View measurable impr.': 'ActiveViewMeasurableImpressions',
65 | 'Active View viewable impr. / measurable impr.': 'ActiveViewViewability',
66 | 'All conv.': 'AllConversions',
67 | 'Avg. Cost': 'AverageCost',
68 | 'Avg. CPC': 'AverageCpc',
69 | 'Avg. CPE': 'AverageCpe',
70 | 'Avg. CPM': 'AverageCpm',
71 | 'Avg. CPV': 'AverageCpv',
72 | 'Avg. position': 'AveragePosition',
73 | 'Clicks': 'Clicks',
74 | 'Conv. rate': 'ConversionRate',
75 | 'Conversions': 'Conversions',
76 | 'Total conv. value': 'ConversionValue',
77 | 'Cost': 'Cost',
78 | 'Cost / conv.': 'CostPerConversion',
79 | 'Cross-device conv.': 'CrossDeviceConversions',
80 | 'CTR': 'Ctr',
81 | 'Engagement rate': 'EngagementRate',
82 | 'Engagements': 'Engagements',
83 | 'Impressions': 'Impressions',
84 | 'Interaction Rate': 'InteractionRate',
85 | 'Interactions': 'Interactions',
86 | 'Interaction Types': 'InteractionTypes',
87 | 'Mobile-friendly click rate': 'PercentageMobileFriendlyClicks',
88 | 'Valid AMP click rate': 'PercentageValidAcceleratedMobilePagesClicks',
89 | 'Mobile speed score': 'SpeedScore',
90 | 'Value / conv.': 'ValuePerConversion',
91 | 'View rate': 'VideoViewRate'
92 | }
93 |
94 | PROJECT_NAME = os.environ['GOOGLE_CLOUD_PROJECT']
95 |
96 |
97 | @app.route('/')
98 | def export_landing_page_report():
99 | """This route triggers the download of the Ads landing page report.
100 |
101 | The landing page report for the client is downloaded using credentials stored
102 | in the project firestore datastore. The report is downloaded either for the
103 | last 30 days if never run before, or from the last run date to today if there
104 | is a date stored in firestore. The last run date is updated after the report
105 | is downloaded from Ads.
106 |
107 | Returns:
108 | The landing page report in CSV format
109 |
110 | Raises:
111 | HTTPError: Used to cause bottle to return a 500 error to the client.
112 | """
113 | customer_id = request.params.get('cid')
114 | customer_name = request.params.get('name')
115 | start_date = request.params.get('startdate')
116 | if not customer_id:
117 | logger.error('Client customer id (cid) not included in request')
118 | raise HTTPError(400,
119 | 'Customer client id not provided as cid query parameter.')
120 |
121 | storage_client = firestore.Client()
122 |
123 | try:
124 | credentials_doc = (
125 | storage_client.collection('agency_ads').document('credentials').get())
126 | developer_token = credentials_doc.get('developer_token')
127 | client_id = credentials_doc.get('client_id')
128 | client_secret = credentials_doc.get('client_secret')
129 | refresh_token = credentials_doc.get('refresh_token')
130 | ads_credentials = ('adwords:\n' + f' client_customer_id: {customer_id}\n' +
131 | f' developer_token: {developer_token}\n' +
132 | f' client_id: {client_id}\n' +
133 | f' client_secret: {client_secret}\n' +
134 | f' refresh_token: {refresh_token}')
135 | except google.cloud.exceptions.NotFound:
136 | logger.exception('Unable to load ads credentials.')
137 | raise HTTPError(500, 'Unable to load Ads credentials.')
138 |
139 | ads_client = adwords.AdWordsClient.LoadFromString(ads_credentials)
140 | landing_page_query = adwords.ReportQueryBuilder()
141 | # selecting campaign attributes, unexpanded final url, device,
142 | # date, and all of the landing page metrics.
143 | landing_page_query.Select(','.join(REPORT_COLS.values()))
144 | landing_page_query.From('LANDING_PAGE_REPORT')
145 | if not start_date:
146 | landing_page_query.During(date_range='YESTERDAY')
147 | else:
148 | try:
149 | start_date = datetime.date.fromisoformat(start_date)
150 | today = datetime.date.today()
151 | except ValueError:
152 | logger.info('Invalid date passed in startdate parameter.')
153 | raise HTTPError(400, 'Invalid date in startdate parameter.')
154 | if start_date == datetime.date.today():
155 | landing_page_query.During(date_range='TODAY')
156 | else:
157 | if today < start_date:
158 | logger.error('Last run date in the future (start_date: %s)', start_date)
159 | raise HTTPError(400,
160 | 'startdate in the future (start_date: %s)' % start_date)
161 |
162 | landing_page_query.During(
163 | start_date=start_date.strftime('%Y%m%d'),
164 | end_date=today.strftime('%Y%m%d'))
165 |
166 | landing_page_query = landing_page_query.Build()
167 |
168 | report_downloader = ads_client.GetReportDownloader(version='v201809')
169 | try:
170 | landing_page_report = (
171 | report_downloader.DownloadReportAsStreamWithAwql(
172 | landing_page_query,
173 | 'CSV',
174 | skip_report_header=True,
175 | skip_report_summary=True))
176 | except Exception as e:
177 | logger.exception('Problem with retrieving landing page report')
178 | raise HTTPError(500, 'Unable to retrieve landing page report %s' % e)
179 |
180 | ads_cols = []
181 | ads_rows = []
182 | try:
183 | while True:
184 | report_line = landing_page_report.readline()
185 | if not report_line: break
186 |
187 | report_line = report_line.decode().replace('\n', '')
188 | report_row = report_line.split(',')
189 |
190 | if not ads_cols:
191 | ads_cols = report_row
192 | continue
193 |
194 | report_row = dict(zip(ads_cols, report_row))
195 | # replace row keys
196 | report_row = {REPORT_COLS[key]: val for key, val in report_row.items()}
197 | base_url = report_row['UnexpandedFinalUrlString']
198 | # removes parameters after ignore and, if the url then ends with a lone
199 | # ?, it too is removed.
200 | if '{ignore}' in base_url:
201 | base_url = base_url[0:base_url.index('{ignore}')]
202 | if base_url.endswith('?'):
203 | base_url = base_url[0:-1]
204 | report_row['BaseUrl'] = base_url
205 | report_row['CID'] = customer_id
206 | report_row['ClientName'] = customer_name
207 | # Ads reports return percentages as strings with %, so we change them
208 | # back to numbers between 0 and 1
209 | # we also need to change -- to 0 to insert values.
210 | for k, v in report_row.items():
211 | if v.endswith('%'):
212 | report_row[k] = float(v[0:-1]) / 100
213 | elif v.isdecimal():
214 | report_row[k] = float(v)
215 | elif v == ' --':
216 | report_row[k] = None
217 |
218 | ads_rows.append(report_row)
219 | except OSError as e:
220 | logger.exception('Problem reading the landing page report: %s', e)
221 | raise HTTPError(500, 'Unable to read landing page report.')
222 | finally:
223 | landing_page_report.close()
224 |
225 | if ads_rows:
226 | csv_file_name = '/tmp/ads_data.' + str(datetime.timestamp(datetime.now()))
227 | with open(csv_file_name, 'w', newline='') as csv_file:
228 | try:
229 | csv_writer = csv.DictWriter(csv_file, fieldnames=ads_cols)
230 | csv_writer.writeheader()
231 | csv_writer.writerows(ads_rows)
232 | except OSError as os_error:
233 | logger.exception(
234 | 'Problem writing the landing page report to a temp file: %s', e)
235 | raise HTTPError(500, 'Unable to write landing page report to file.')
236 | try:
237 | bq_client = bigquery.Client()
238 | bq_table = bq_client.get_table(
239 | f'{PROJECT_NAME}.agency_dashboard.ads_data')
240 | bq_job_config = bigquery.LoadJobConfig()
241 | bq_job_config.source_format = bigquery.SourceFormat.CSV
242 | bq_job_config.skip_leading_rows = 1
243 | bq_job_config.autodetect = True
244 | with open(csv_file_name, 'rb') as csv_file:
245 | try:
246 | bq_job = bq_client.load_table_from_file(
247 | csv_file, bq_table, job_config=bq_job_config)
248 | bq_job_result = bq_job.result()
249 | except Exception as ex:
250 | logger.exception('Problem loading ads data into bigquery: %s', ex)
251 | raise ex
252 | except google.cloud.exceptions.GoogleCloudError as gce:
253 | logger.exception('Problem loading ads data into bigquery: %s',
254 | gce.message)
255 | raise gce
256 |
257 | if __name__ == '__main__':
258 | app.run(host='localhost', port=8090)
259 |
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 |
2 | Apache License
3 | Version 2.0, January 2004
4 | http://www.apache.org/licenses/
5 |
6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7 |
8 | 1. Definitions.
9 |
10 | "License" shall mean the terms and conditions for use, reproduction,
11 | and distribution as defined by Sections 1 through 9 of this document.
12 |
13 | "Licensor" shall mean the copyright owner or entity authorized by
14 | the copyright owner that is granting the License.
15 |
16 | "Legal Entity" shall mean the union of the acting entity and all
17 | other entities that control, are controlled by, or are under common
18 | control with that entity. For the purposes of this definition,
19 | "control" means (i) the power, direct or indirect, to cause the
20 | direction or management of such entity, whether by contract or
21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the
22 | outstanding shares, or (iii) beneficial ownership of such entity.
23 |
24 | "You" (or "Your") shall mean an individual or Legal Entity
25 | exercising permissions granted by this License.
26 |
27 | "Source" form shall mean the preferred form for making modifications,
28 | including but not limited to software source code, documentation
29 | source, and configuration files.
30 |
31 | "Object" form shall mean any form resulting from mechanical
32 | transformation or translation of a Source form, including but
33 | not limited to compiled object code, generated documentation,
34 | and conversions to other media types.
35 |
36 | "Work" shall mean the work of authorship, whether in Source or
37 | Object form, made available under the License, as indicated by a
38 | copyright notice that is included in or attached to the work
39 | (an example is provided in the Appendix below).
40 |
41 | "Derivative Works" shall mean any work, whether in Source or Object
42 | form, that is based on (or derived from) the Work and for which the
43 | editorial revisions, annotations, elaborations, or other modifications
44 | represent, as a whole, an original work of authorship. For the purposes
45 | of this License, Derivative Works shall not include works that remain
46 | separable from, or merely link (or bind by name) to the interfaces of,
47 | the Work and Derivative Works thereof.
48 |
49 | "Contribution" shall mean any work of authorship, including
50 | the original version of the Work and any modifications or additions
51 | to that Work or Derivative Works thereof, that is intentionally
52 | submitted to Licensor for inclusion in the Work by the copyright owner
53 | or by an individual or Legal Entity authorized to submit on behalf of
54 | the copyright owner. For the purposes of this definition, "submitted"
55 | means any form of electronic, verbal, or written communication sent
56 | to the Licensor or its representatives, including but not limited to
57 | communication on electronic mailing lists, source code control systems,
58 | and issue tracking systems that are managed by, or on behalf of, the
59 | Licensor for the purpose of discussing and improving the Work, but
60 | excluding communication that is conspicuously marked or otherwise
61 | designated in writing by the copyright owner as "Not a Contribution."
62 |
63 | "Contributor" shall mean Licensor and any individual or Legal Entity
64 | on behalf of whom a Contribution has been received by Licensor and
65 | subsequently incorporated within the Work.
66 |
67 | 2. Grant of Copyright License. Subject to the terms and conditions of
68 | this License, each Contributor hereby grants to You a perpetual,
69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70 | copyright license to reproduce, prepare Derivative Works of,
71 | publicly display, publicly perform, sublicense, and distribute the
72 | Work and such Derivative Works in Source or Object form.
73 |
74 | 3. Grant of Patent License. Subject to the terms and conditions of
75 | this License, each Contributor hereby grants to You a perpetual,
76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77 | (except as stated in this section) patent license to make, have made,
78 | use, offer to sell, sell, import, and otherwise transfer the Work,
79 | where such license applies only to those patent claims licensable
80 | by such Contributor that are necessarily infringed by their
81 | Contribution(s) alone or by combination of their Contribution(s)
82 | with the Work to which such Contribution(s) was submitted. If You
83 | institute patent litigation against any entity (including a
84 | cross-claim or counterclaim in a lawsuit) alleging that the Work
85 | or a Contribution incorporated within the Work constitutes direct
86 | or contributory patent infringement, then any patent licenses
87 | granted to You under this License for that Work shall terminate
88 | as of the date such litigation is filed.
89 |
90 | 4. Redistribution. You may reproduce and distribute copies of the
91 | Work or Derivative Works thereof in any medium, with or without
92 | modifications, and in Source or Object form, provided that You
93 | meet the following conditions:
94 |
95 | (a) You must give any other recipients of the Work or
96 | Derivative Works a copy of this License; and
97 |
98 | (b) You must cause any modified files to carry prominent notices
99 | stating that You changed the files; and
100 |
101 | (c) You must retain, in the Source form of any Derivative Works
102 | that You distribute, all copyright, patent, trademark, and
103 | attribution notices from the Source form of the Work,
104 | excluding those notices that do not pertain to any part of
105 | the Derivative Works; and
106 |
107 | (d) If the Work includes a "NOTICE" text file as part of its
108 | distribution, then any Derivative Works that You distribute must
109 | include a readable copy of the attribution notices contained
110 | within such NOTICE file, excluding those notices that do not
111 | pertain to any part of the Derivative Works, in at least one
112 | of the following places: within a NOTICE text file distributed
113 | as part of the Derivative Works; within the Source form or
114 | documentation, if provided along with the Derivative Works; or,
115 | within a display generated by the Derivative Works, if and
116 | wherever such third-party notices normally appear. The contents
117 | of the NOTICE file are for informational purposes only and
118 | do not modify the License. You may add Your own attribution
119 | notices within Derivative Works that You distribute, alongside
120 | or as an addendum to the NOTICE text from the Work, provided
121 | that such additional attribution notices cannot be construed
122 | as modifying the License.
123 |
124 | You may add Your own copyright statement to Your modifications and
125 | may provide additional or different license terms and conditions
126 | for use, reproduction, or distribution of Your modifications, or
127 | for any such Derivative Works as a whole, provided Your use,
128 | reproduction, and distribution of the Work otherwise complies with
129 | the conditions stated in this License.
130 |
131 | 5. Submission of Contributions. Unless You explicitly state otherwise,
132 | any Contribution intentionally submitted for inclusion in the Work
133 | by You to the Licensor shall be under the terms and conditions of
134 | this License, without any additional terms or conditions.
135 | Notwithstanding the above, nothing herein shall supersede or modify
136 | the terms of any separate license agreement you may have executed
137 | with Licensor regarding such Contributions.
138 |
139 | 6. Trademarks. This License does not grant permission to use the trade
140 | names, trademarks, service marks, or product names of the Licensor,
141 | except as required for reasonable and customary use in describing the
142 | origin of the Work and reproducing the content of the NOTICE file.
143 |
144 | 7. Disclaimer of Warranty. Unless required by applicable law or
145 | agreed to in writing, Licensor provides the Work (and each
146 | Contributor provides its Contributions) on an "AS IS" BASIS,
147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148 | implied, including, without limitation, any warranties or conditions
149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150 | PARTICULAR PURPOSE. You are solely responsible for determining the
151 | appropriateness of using or redistributing the Work and assume any
152 | risks associated with Your exercise of permissions under this License.
153 |
154 | 8. Limitation of Liability. In no event and under no legal theory,
155 | whether in tort (including negligence), contract, or otherwise,
156 | unless required by applicable law (such as deliberate and grossly
157 | negligent acts) or agreed to in writing, shall any Contributor be
158 | liable to You for damages, including any direct, indirect, special,
159 | incidental, or consequential damages of any character arising as a
160 | result of this License or out of the use or inability to use the
161 | Work (including but not limited to damages for loss of goodwill,
162 | work stoppage, computer failure or malfunction, or any and all
163 | other commercial damages or losses), even if such Contributor
164 | has been advised of the possibility of such damages.
165 |
166 | 9. Accepting Warranty or Additional Liability. While redistributing
167 | the Work or Derivative Works thereof, You may choose to offer,
168 | and charge a fee for, acceptance of support, warranty, indemnity,
169 | or other liability obligations and/or rights consistent with this
170 | License. However, in accepting such obligations, You may act only
171 | on Your own behalf and on Your sole responsibility, not on behalf
172 | of any other Contributor, and only if You agree to indemnify,
173 | defend, and hold each Contributor harmless for any liability
174 | incurred by, or claims asserted against, such Contributor by reason
175 | of your accepting any such warranty or additional liability.
176 |
177 | END OF TERMS AND CONDITIONS
178 |
179 | APPENDIX: How to apply the Apache License to your work.
180 |
181 | To apply the Apache License to your work, attach the following
182 | boilerplate notice, with the fields enclosed by brackets "[]"
183 | replaced with your own identifying information. (Don't include
184 | the brackets!) The text should be enclosed in the appropriate
185 | comment syntax for the file format. We also recommend that a
186 | file or class name and description of purpose be included on the
187 | same "printed page" as the copyright notice for easier
188 | identification within third-party archives.
189 |
190 | Copyright [yyyy] [name of copyright owner]
191 |
192 | Licensed under the Apache License, Version 2.0 (the "License");
193 | you may not use this file except in compliance with the License.
194 | You may obtain a copy of the License at
195 |
196 | http://www.apache.org/licenses/LICENSE-2.0
197 |
198 | Unless required by applicable law or agreed to in writing, software
199 | distributed under the License is distributed on an "AS IS" BASIS,
200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201 | See the License for the specific language governing permissions and
202 | limitations under the License.
203 |
--------------------------------------------------------------------------------