39 |
40 | 5. Generate an API key and add it to `local.json`
41 | 6. Change line 37 to filter tweets on whichver terms you'd like
42 | 7. Install node modules: `npm install`
43 | 8. Run the script: `node twitter.js`
44 |
45 | ## Natural Language API + Firebase realtime Twitter dashboard demo
46 |
47 | 1. `cd` into `nl-firebase-twitter/`
48 | 2. Create a project in the [Firebase console](http://firebase.google.com/console) and install the [Firebase CLI](https://firebase.google.com/docs/cli/)
49 | 3. `cd` into the `frontend/` directory and run `firebase login` and `firebase init` to associate this with the Firebase project you just created. When prompted, don't overwrite existing files. Create a **database** and **hosting** project (no Functions).
50 | 4. In your Firebase console, click "Add Firebase to your web app". Copy the credentials to the top of the main.js file
51 | 5. `cd` into the `backend/` directory and run `npm install` to install dependencies
52 | 6. Generate a service account for your project by navigating to the "Project settings" tab in your Firebase console and then selecting "Service Accouts". Click "Generate New Private Key" and save this in your `backend/` directory as `keyfile.json`
53 | 7. Generate [Twitter Streaming API](https://dev.twitter.com/streaming/overview) credentials and copy them to `backend/local.json`
54 | 8. Navigate to the Cloud console for our project. Enabled the Natural Language API and generate an API key. Replace `YOUR-API-KEY` in `backend/local.json` with this key.
55 | 9. Replace `searchTerms` in `backend/index.js` with the search terms you'd like to filter tweets on
56 | 10. Replace `FIREBASE-PROJECT-ID` in `backend/local.json` with the id of your Firebase project
57 | 11. Set up BigQuery: in your Cloud console for the same project, create a BigQuery dataset. Then create a table in that dataset. When creating the table, click **Edit as text** and paste the following:
58 | ```
59 | id:STRING,text:STRING,user:STRING,user_time_zone:STRING,user_followers_count:INTEGER,hashtags:STRING,tokens:STRING,score:STRING,magnitude:STRING,entities:STRING
60 | ```
61 | 12. Add your BigQuery dataset and table names to `backend/local.json`.
62 | 11. Run the server: from the `backend/` directory run `node index.js`. You should see tweet data being written to your Firebase database
63 | 12. In a separate terminal process, run the frontend: from the `frontend/` directory run `firebase serve`
64 | 13. Deploy your frontend: from the `frontend/` directory run `firebase deploy`
65 |
66 |
67 | ## Multiple API demo
68 |
69 | 1. `cd` into `vision-speech-nl-translate`
70 | 2. Make sure you've set up your [GOOGLE_APPLICATION_CREDENTIALS](https://developers.google.com/identity/protocols/application-default-credentials) with a Cloud project that has the Vision, Speech, NL, and Translation APIs enabled
71 | 3. Run the script: `python textify.py`
72 | 4. Note: if you're running it with image OCR, copy an image file to your local directory
73 |
--------------------------------------------------------------------------------
/natural-language/local.json:
--------------------------------------------------------------------------------
1 | {
2 | "twitter": {
3 | "consumer_key": "TWITTER_KEY",
4 | "consumer_secret": "TWITTER_SECRET",
5 | "access_token_key": "ACCESS_TOKEN",
6 | "access_token_secret": "ACCESS_SECRET"
7 | },
8 | "nl_api_key": "your_nl_api_key",
9 | "keyfile_path": "~/path_to_your_keyfile.json",
10 | "cloud_project_id": "your_cloud_project",
11 | "bigquery_dataset": "your_bq_dataset",
12 | "bigquery_table": "your_bq_table"
13 | }
14 |
--------------------------------------------------------------------------------
/natural-language/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "natural-language",
3 | "version": "1.0.0",
4 | "description": "",
5 | "main": "twitter.js",
6 | "scripts": {
7 | "test": "echo \"Error: no test specified\" && exit 1"
8 | },
9 | "keywords": [],
10 | "author": "",
11 | "license": "ISC",
12 | "dependencies": {
13 | "@google-cloud/bigquery": "^1.0.0",
14 | "async": "^2.6.0",
15 | "request": "^2.83.0",
16 | "twitter": "^1.7.1"
17 | }
18 | }
19 |
--------------------------------------------------------------------------------
/natural-language/twitter.js:
--------------------------------------------------------------------------------
1 | // Copyright 2018 Google Inc.
2 |
3 | // Licensed under the Apache License, Version 2.0 (the "License");
4 | // you may not use this file except in compliance with the License.
5 | // You may obtain a copy of the License at
6 |
7 | // http://www.apache.org/licenses/LICENSE-2.0
8 |
9 | // Unless required by applicable law or agreed to in writing, software
10 | // distributed under the License is distributed on an "AS IS" BASIS,
11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | // See the License for the specific language governing permissions and
13 | // limitations under the License.
14 |
15 | 'use strict';
16 | const request = require('request');
17 | const Twitter = require('twitter');
18 | const async = require('async');
19 | const config = require('./local.json');
20 | const client = new Twitter(config.twitter);
21 |
22 | // Set up BigQuery
23 | // Replace this with the name of your project and the path to your keyfile
24 | const bigquery = require('@google-cloud/bigquery')({
25 | projectId: config.cloud_project_id,
26 | keyFilename: config.keyfile_path
27 | });
28 | const dataset = bigquery.dataset(config.bigquery_dataset);
29 | const table = dataset.table(config.bigquery_table);
30 |
31 | // Replace searchTerms with whatever tweets you want to stream
32 | // Details here: https://dev.twitter.com/streaming/overview/request-parameters#track
33 | const searchTerms = '#googlenext17,@googlecloud,google cloud';
34 |
35 | function callNLMethod(tweet, method) {
36 | const textUrl = `https://language.googleapis.com/v1/documents:${method}?key=${config.nl_api_key}`;
37 | let requestBody = {
38 | "document": {
39 | "type": "PLAIN_TEXT",
40 | "content": tweet.text
41 | }
42 | }
43 |
44 | let options = {
45 | url: textUrl,
46 | method: "POST",
47 | body: requestBody,
48 | json: true
49 | }
50 |
51 | return new Promise((resolve, reject) => {
52 | request(options, function(err, resp, body) {
53 | if ((!err && resp.statusCode == 200) && (body.sentences.length != 0)) {
54 | resolve(body);
55 | } else {
56 | reject(err);
57 | }
58 | });
59 | })
60 | }
61 |
62 | client.stream('statuses/filter', {track: searchTerms, language: 'en'}, function(stream) {
63 |
64 | stream.on('data', function(tweet) {
65 | if ((tweet.text != undefined) && (tweet.text.substring(0,2) != 'RT')) {
66 | async function analyzeTweet() {
67 | try {
68 | let syntaxData = await callNLMethod(tweet, 'analyzeSyntax');
69 | let sentimentData = await callNLMethod(tweet, 'analyzeSentiment');
70 |
71 | let row = {
72 | id: tweet.id_str,
73 | text: tweet.text,
74 | created_at: tweet.timestamp_ms.toString(),
75 | user_followers_count: tweet.user.followers_count,
76 | hashtags: JSON.stringify(tweet.entities.hashtags),
77 | tokens: JSON.stringify(syntaxData.tokens),
78 | score: sentimentData.documentSentiment.score,
79 | magnitude: sentimentData.documentSentiment.magnitude
80 | };
81 |
82 | table.insert(row, function(error, insertErr, apiResp) {
83 | if (error) {
84 | console.log('err', error);
85 | } else if (insertErr.length == 0) {
86 | console.log('success!');
87 | }
88 | });
89 |
90 | } catch (err) {
91 | console.log('API error: ', err);
92 | }
93 | }
94 | analyzeTweet();
95 | }
96 |
97 | });
98 |
99 | stream.on('error', function(error) {
100 | throw error;
101 | });
102 | });
103 |
--------------------------------------------------------------------------------
/nl-firebase-twitter/backend/index.js:
--------------------------------------------------------------------------------
1 | // Copyright 2017 Google Inc.
2 |
3 | // Licensed under the Apache License, Version 2.0 (the "License");
4 | // you may not use this file except in compliance with the License.
5 | // You may obtain a copy of the License at
6 |
7 | // http://www.apache.org/licenses/LICENSE-2.0
8 |
9 | // Unless required by applicable law or agreed to in writing, software
10 | // distributed under the License is distributed on an "AS IS" BASIS,
11 | // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 | // See the License for the specific language governing permissions and
13 | // limitations under the License.
14 |
15 | 'use strict';
16 |
17 | const request = require('request');
18 | const Twitter = require('twitter');
19 | const config = require('./local.json');
20 | const client = new Twitter({
21 | consumer_key: config.twitter_consumer_key,
22 | consumer_secret: config.twitter_consumer_secret,
23 | access_token_key: config.twitter_access_key,
24 | access_token_secret: config.twitter_access_secret
25 | });
26 |
27 | const gcloud = require('google-cloud')({
28 | keyFilename: 'keyfile.json',
29 | projectId: config.project_id
30 | });
31 | const bigquery = gcloud.bigquery();
32 | const dataset = bigquery.dataset(config.bigquery_dataset);
33 | const table = dataset.table(config.bigquery_table);
34 |
35 | const Filter = require('bad-words'),
36 | filter = new Filter();
37 |
38 | // Replace searchTerms with whatever tweets you want to stream
39 | // Details here: https://dev.twitter.com/streaming/overview/request-parameters#track
40 | const searchTerms = 'googleio,googledevelopers,googlecloud,firebase,machine learning,io17,googleio17';
41 |
42 | // Add a filter-level param?
43 | client.stream('statuses/filter', {track: searchTerms, language: 'en'}, function(stream) {
44 | stream.on('data', function(event) {
45 | // Exclude tweets starting with "RT"
46 | if ((event.text != undefined) && (event.text.substring(0,2) != 'RT') && (event.text === filter.clean(event.text))) {
47 | callNLApi(event);
48 | }
49 | });
50 | stream.on('error', function(error) {
51 | console.log('twitter api error: ', error);
52 | });
53 | });
54 |
55 |
56 | // INITIALIZE FIREBASE
57 | var admin = require("firebase-admin");
58 | var serviceAccount = require("./keyfile.json");
59 | admin.initializeApp({
60 | credential: admin.credential.cert(serviceAccount),
61 | databaseURL: "https://" + config.project_id + ".firebaseio.com"
62 | });
63 |
64 | const db = admin.database();
65 | const tweetRef = db.ref('latest');
66 | const hashtagRef = db.ref('hashtags');
67 |
68 | // Uses a Firebase transaction to incrememnt a counter
69 | function incrementCount(ref, child, valToIncrement) {
70 | ref.child(child).transaction(function(data) {
71 | if (data != null) {
72 | data += valToIncrement;
73 | } else {
74 | data = 1;
75 | }
76 | return data;
77 | });
78 | }
79 |
80 |
81 | tweetRef.on('value', function (snap) {
82 | if (snap.exists()) {
83 | let tweet = snap.val();
84 | let tokens = tweet['tokens'];
85 | let hashtags = tweet['hashtags'];
86 |
87 | for (let i in tokens) {
88 | let token = tokens[i];
89 | let word = token.lemma.toLowerCase();
90 |
91 | if ((acceptedWordTypes.indexOf(token.partOfSpeech.tag) != -1) && !(word.match(/[^A-Za-z0-9]/g))) {
92 | let posRef = db.ref('tokens/' + token.partOfSpeech.tag);
93 | incrementCount(posRef, word, 1);
94 | }
95 |
96 | }
97 |
98 | if (hashtags) {
99 | for (let i in hashtags) {
100 | let ht = hashtags[i];
101 | let text = ht.text.toLowerCase();
102 | let htRef = hashtagRef.child(text);
103 | incrementCount(htRef, 'totalScore', tweet.score);
104 | incrementCount(htRef, 'numMentions', 1);
105 | }
106 | }
107 | }
108 | });
109 |
110 |
111 | const acceptedWordTypes = ['ADJ']; // Add the parts of speech you'd like to graph to this array ('NOUN', 'VERB', etc.)
112 |
113 | function callNLApi(tweet) {
114 | const textUrl = "https://language.googleapis.com/v1/documents:annotateText?key=" + config.cloud_api_key;
115 | let requestBody = {
116 | "document": {
117 | "type": "PLAIN_TEXT",
118 | "content": tweet.text
119 | },
120 | "features": {
121 | "extractSyntax": true,
122 | "extractEntities": true,
123 | "extractDocumentSentiment": true
124 | }
125 | }
126 |
127 | let options = {
128 | url: textUrl,
129 | method: "POST",
130 | body: requestBody,
131 | json: true
132 | }
133 |
134 | request(options, function(err, resp, body) {
135 | if ((!err && resp.statusCode == 200) && (body.sentences.length != 0)) {
136 | let tweetForFb = {
137 | id: tweet.id_str,
138 | text: tweet.text,
139 | user: tweet.user.screen_name,
140 | user_time_zone: tweet.user.time_zone,
141 | user_followers_count: tweet.user.followers_count,
142 | hashtags: tweet.entities.hashtags,
143 | tokens: body.tokens,
144 | score: body.documentSentiment.score,
145 | magnitude: body.documentSentiment.magnitude,
146 | entities: body.entities
147 | };
148 |
149 | let bqRow = {
150 | id: tweet.id_str,
151 | text: tweet.text,
152 | user: tweet.user.screen_name,
153 | user_time_zone: tweet.user.time_zone,
154 | user_followers_count: tweet.user.followers_count,
155 | hashtags: JSON.stringify(tweet.entities.hashtags),
156 | tokens: JSON.stringify(body.tokens),
157 | score: body.documentSentiment.score,
158 | magnitude: body.documentSentiment.magnitude,
159 | entities: JSON.stringify(body.entities)
160 | }
161 |
162 | tweetRef.set(tweetForFb);
163 | table.insert(bqRow, function(error, insertErr, apiResp) {
164 | if (error) {
165 | console.log('err', error);
166 | } else if (insertErr.length == 0) {
167 | console.log('success!');
168 | }
169 | });
170 |
171 | } else {
172 | console.log('NL API error: ', err);
173 | }
174 | });
175 | }
176 |
--------------------------------------------------------------------------------
/nl-firebase-twitter/backend/local.json:
--------------------------------------------------------------------------------
1 | {
2 | "twitter_consumer_key": "",
3 | "twitter_consumer_secret": "",
4 | "twitter_access_key": "",
5 | "twitter_access_secret": "",
6 | "project_id": "",
7 | "bigquery_dataset": "",
8 | "bigquery_table": "",
9 | "cloud_api_key": ""
10 | }
--------------------------------------------------------------------------------
/nl-firebase-twitter/backend/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "twitter-nl-fb",
3 | "version": "1.0.0",
4 | "description": "",
5 | "main": "index.js",
6 | "scripts": {
7 | "test": "echo \"Error: no test specified\" && exit 1"
8 | },
9 | "keywords": [],
10 | "author": "",
11 | "license": "ISC",
12 | "dependencies": {
13 | "async": "^2.4.0",
14 | "bad-words": "^1.5.1",
15 | "firebase-admin": "^4.2.1",
16 | "google-cloud": "^0.53.0",
17 | "request": "^2.81.0",
18 | "twitter": "^1.7.0"
19 | }
20 | }
21 |
--------------------------------------------------------------------------------
/nl-firebase-twitter/frontend/database.rules.json:
--------------------------------------------------------------------------------
1 | {
2 | "rules": {
3 | ".read": true,
4 | ".write": "auth != null",
5 | "tokens": {
6 | "ADJ": {
7 | ".indexOn": ".value"
8 | }
9 | },
10 | "hashtags": {
11 | ".indexOn": "numMentions"
12 | }
13 | }
14 | }
15 |
--------------------------------------------------------------------------------
/nl-firebase-twitter/frontend/emoji-happy.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sararob/ml-talk-demos/91a3327a3b8b8e536789647151b3046c0dee5e36/nl-firebase-twitter/frontend/emoji-happy.png
--------------------------------------------------------------------------------
/nl-firebase-twitter/frontend/emoji-sad.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/sararob/ml-talk-demos/91a3327a3b8b8e536789647151b3046c0dee5e36/nl-firebase-twitter/frontend/emoji-sad.png
--------------------------------------------------------------------------------
/nl-firebase-twitter/frontend/firebase.json:
--------------------------------------------------------------------------------
1 | {
2 | "database": {
3 | "rules": "database.rules.json"
4 | },
5 | "hosting": {
6 | "public": "."
7 | }
8 | }
9 |
--------------------------------------------------------------------------------
/nl-firebase-twitter/frontend/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
39 |
43 |
44 |
40 |
42 | Latest tweet data
41 |
45 |
56 |
57 |
46 |
50 | Nouns
47 |Verbs
48 |Adjectives
49 |
51 |
52 |
53 |
54 |
55 |
58 |
68 |
69 |
59 |
61 | Latest tweet sentiment
60 |
62 | 
63 |
64 | 
65 |
66 |
67 |
63 |
64 |
65 |
66 |
70 |
79 |
71 |
72 |
73 |
74 |
75 |
76 |
77 |
78 | 
34 |
35 |
36 |
37 |