├── .gcloudignore ├── .gitignore ├── .gitmodules ├── README.md ├── app.py ├── app.yaml ├── cron.yaml ├── index.yaml ├── requirements.txt └── static └── index.html /.gcloudignore: -------------------------------------------------------------------------------- 1 | # -*- conf -*- 2 | # Duplicated in other App Engine project repos: Bridgy, Bridgy Fed, granary, 3 | # oauth-dropins, etc. If you make a change here, change them too! 4 | # 5 | # https://cloud.google.com/appengine/docs/standard/python/config/appref#Python_app_yaml_Includes 6 | 7 | *.bak 8 | *.c 9 | *.cc 10 | *.cpp 11 | *.h 12 | *.o 13 | *.pyc 14 | *.pyo 15 | *.so 16 | .git* 17 | 18 | __pycache__/ 19 | .coverage/ 20 | .git*/ 21 | browser-extension/ 22 | coverage/ 23 | debian/ 24 | doc/ 25 | docs/ 26 | example/ 27 | examples/ 28 | l/ 29 | l3/ 30 | local/ 31 | local3/ 32 | local3.7/ 33 | node_modules 34 | package-lock.json 35 | pydoc/ 36 | pydocs/ 37 | python3/ 38 | RCS/ 39 | ref/ 40 | sample/ 41 | samples/ 42 | service_account_creds.json 43 | TAGS 44 | TAGS/ 45 | test/ 46 | tests/ 47 | yarn.lock 48 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /l 2 | /local 3 | flask_secret_key 4 | micropub_access_token* 5 | twitter_access_token 6 | twitter_access_token_secret 7 | twitter_app_key 8 | twitter_app_secret 9 | twitter_scrape_headers* 10 | -------------------------------------------------------------------------------- /.gitmodules: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/snarfed/ownyourresponses/af1ad7327634a518c1e5ca8fd7fa6af837d7150c/.gitmodules -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ownyourresponses 2 | 3 | Creates posts on your web site for your likes, replies, reshares, and event RSVPs on social networks. In [IndieWeb](https://indiewebcamp.com/) terms, [PESOS](https://indiewebcamp.com/PESOS) as a service. 4 | 5 | See [PESOS for Bridgy Publish](https://snarfed.org/2015-01-22_pesos-for-bridgy-publish) for background on the motivation. 6 | 7 | Uses [Micropub](https://indiewebcamp.com/micropub). Your web site must have a 8 | Micropub endpoint. 9 | ([Here's one for WordPress](https://github.com/snarfed/wordpress-micropub), for 10 | example.) 11 | 12 | This project is placed in the public domain. You may also use it under the [CC0 license](http://creativecommons.org/publicdomain/zero/1.0/). 13 | 14 | 15 | ## Setup 16 | 17 | Setup 18 | --- 19 | 20 | 1. Clone this repo. 21 | 22 | 1. Change `MICROPUB_ENDPOINT` in `app.py` to your site's endpoint. 23 | 24 | 1. Generate a [Micropub](https://indiewebcamp.com/micropub) access token for 25 | your web site. 26 | 27 | 1. Put it in a file called `micropub_access_token` in the repo root directory. 28 | 29 | 1. Create a 30 | [Twitter app](https://apps.twitter.com/app/new) and an 31 | [Instagram app](http://instagram.com/developer/clients/manage/). (No Facebook 32 | yet, since their API doesn't expose your recent likes or comments.) 33 | 34 | 1. Put their app ids and secrets and access tokens in the repo root directory in 35 | files named `twitter_app_key`, `twitter_app_secret`, `twitter_access_token`, 36 | `instagram_client_id`, `instagram_client_secret`, and `instagram_access_token`. 37 | [Details here.](https://github.com/snarfed/oauth-dropins/blob/master/appengine_config.py) 38 | 39 | 1. Create an [App Engine](http://appengine.google.com/) app, replace 40 | `ownyourresponses` in `app.yaml` with your app id, and deploy. 41 | 42 | ...and you're done! Comment or like or retweet something, and it should 43 | automatically create a new post on your web site. 44 | 45 | 46 | ## Silo API details 47 | 48 | ### Twitter 49 | 50 | Twitter has a [streaming API](https://dev.twitter.com/docs/streaming-apis) that sends events for new favorites and tweets (including @-replies and retweets). Bridgy [has used it before](https://github.com/snarfed/bridgy/blob/master/twitter_streaming.py). [It broke when Bridgy went over 100ish Twitter users](https://github.com/snarfed/bridgy/issues/57), but it would work for just one user. Even so, [it's a bit expensive on App Engine](https://github.com/snarfed/bridgy/issues/8), so I'd probably just poll [`/statuses/user_timeline`](https://dev.twitter.com/rest/reference/get/statuses/user_timeline) and [`/favorites/list`](https://dev.twitter.com/rest/reference/get/favorites/list). 51 | 52 | ### Google+ 53 | 54 | Google+ has no way to get comments *or* +1s by user, only by post. [API docs](https://developers.google.com/+/api/latest/); [feature request](https://code.google.com/p/google-plus-platform/issues/detail?id=89); [SO answer](http://stackoverflow.com/a/19817758/186123). 55 | 56 | ### Instagram 57 | 58 | Instagram can get [likes by user](http://instagram.com/developer/endpoints/users/#get_users_feed_liked), but [not comments](http://stackoverflow.com/a/22002350/186123). 59 | 60 | ### Facebook 61 | 62 | Facebook's [Real Time Updates](https://developers.facebook.com/docs/graph-api/real-time-updates/) should work. I've already used it in [ownyourcheckin](https://github.com/snarfed/ownyourcheckin). I'd subscribe to `/user/likes` and `/user/feed`, which I _think_ should include likes and comments. I could also poll those endpoints. 63 | 64 | ...ugh, except they only tell me *that* I liked or commented on something, not *what* I liked or commented on. Here are example objects from those API endpoints: 65 | 66 | ```json 67 | { 68 | "id": "212038_10101426802642863", 69 | "from": {"id": "212038", "name": "Ryan Barrett"}, 70 | "story": "Ryan Barrett likes a post.", 71 | "story_tags": {...}, 72 | "type": "status", 73 | "created_time": "2014-12-26T17:41:20+0000", 74 | "updated_time": "2014-12-26T17:41:20+0000" 75 | } 76 | 77 | { 78 | "id": "212038_10101488100217033", 79 | "from": {"id": "212038", "name": "Ryan Barrett"}, 80 | "story": "Ryan Barrett commented on his own photo.", 81 | "story_tags": {...}, 82 | "type": "status", 83 | "created_time": "2015-02-02T16:40:44+0000", 84 | "updated_time": "2015-02-02T16:40:44+0000" 85 | } 86 | ``` 87 | 88 | I can generate links from the ids that go to the appropriate stories, e.g. https://www.facebook.com/212038/posts/10101426802642863 and https://www.facebook.com/212038/posts/10101488100217033 , but I can't get the story or comment contents via the API. :( 89 | 90 | _Update: 91 | [They fixed this in API v2.3!](https://developers.facebook.com/docs/apps/changelog#v2_3)_ 92 | > As of March 25, 2015 We now send content in Page real-time updates (RTUs). Previously, only the object's ID was in the RTU payload. Now we include content in addition to the ID including: statuses, posts, shares, photos, videos, milestones, likes and comments. 93 | -------------------------------------------------------------------------------- /app.py: -------------------------------------------------------------------------------- 1 | """OwnYourResponses: turns likes, replies, etc. into posts on your web site. 2 | 3 | Polls your social network activity and creates new posts on your web site (via 4 | Micropub) for public Facebook comments and likes, Instagram likes, and Twitter 5 | @-replies, retweets, and favorites. 6 | """ 7 | import logging 8 | import json 9 | import urllib.error, urllib.parse, urllib.request 10 | 11 | from flask import Flask 12 | from google.cloud import ndb 13 | from granary import ( 14 | as1, 15 | facebook, 16 | instagram, 17 | microformats2, 18 | source as gr_source, 19 | twitter, 20 | ) 21 | from oauth_dropins.webutil import ( 22 | appengine_info, 23 | appengine_config, 24 | flask_util, 25 | util, 26 | ) 27 | from oauth_dropins.webutil.util import json_loads 28 | 29 | # Change this to your web site's Micropub endpoint. 30 | # https://indiewebcamp.com/micropub 31 | if appengine_config.DEBUG: 32 | MICROPUB_ENDPOINT = 'http://localhost/wp-json/micropub/1.0/endpoint' 33 | MICROPUB_ACCESS_TOKEN = util.read('micropub_access_token_local') 34 | else: 35 | MICROPUB_ENDPOINT = 'https://snarfed.org/wp-json/micropub/1.0/endpoint' 36 | MICROPUB_ACCESS_TOKEN = util.read('micropub_access_token') 37 | 38 | # ActivityStreams objectTypes and verbs to create posts for. You can add or 39 | # remove types here to control what gets posted to your site. 40 | TYPES = ('like', 'comment', 'share', 'rsvp-yes', 'rsvp-no', 'rsvp-maybe') 41 | 42 | # The category to include with each response type. If you don't want categories 43 | # for any (or all) types, just remove them. 44 | CATEGORIES = { 45 | 'like': 'like', 46 | 'comment': 'reply', 47 | 'share': 'repost', 48 | 'rsvp-yes': 'rsvp', 49 | 'rsvp-no': 'rsvp', 50 | 'rsvp-maybe': 'rsvp', 51 | } 52 | 53 | FACEBOOK_ACCESS_TOKEN = util.read('facebook_access_token') 54 | INSTAGRAM_ACCESS_TOKEN = util.read('instagram_access_token') 55 | TWITTER_ACCESS_TOKEN = util.read('twitter_access_token') 56 | TWITTER_ACCESS_TOKEN_SECRET = util.read('twitter_access_token_secret') 57 | TWITTER_SCRAPE_HEADERS = json_loads(util.read('twitter_scrape_headers.schnarfed.json')) 58 | 59 | 60 | # Flask app 61 | app = Flask('ownyourresponses') 62 | app.template_folder = './templates' 63 | app.config.from_mapping( 64 | ENV='development' if appengine_info.DEBUG else 'PRODUCTION', 65 | CACHE_TYPE='SimpleCache', 66 | SECRET_KEY=util.read('flask_secret_key'), 67 | JSONIFY_PRETTYPRINT_REGULAR=True, 68 | ) 69 | app.register_error_handler(Exception, flask_util.handle_exception) 70 | 71 | app.wsgi_app = flask_util.ndb_context_middleware( 72 | app.wsgi_app, client=appengine_config.ndb_client) 73 | 74 | 75 | class Response(ndb.Model): 76 | """Key name is ActivityStreams activity id.""" 77 | activity_json = ndb.TextProperty(required=True) 78 | post_url = ndb.TextProperty() 79 | response_body = ndb.TextProperty() 80 | status = ndb.StringProperty(choices=('started', 'complete'), default='started') 81 | created = ndb.DateTimeProperty(auto_now_add=True) 82 | updated = ndb.DateTimeProperty(auto_now=True) 83 | 84 | 85 | @app.route('/cron/poll') 86 | def poll(): 87 | """Poll handler for cron job.""" 88 | # if FACEBOOK_ACCESS_TOKEN: 89 | # sources.append(facebook.Facebook(FACEBOOK_ACCESS_TOKEN)) 90 | # if INSTAGRAM_ACCESS_TOKEN: 91 | # sources.append(instagram.Instagram(INSTAGRAM_ACCESS_TOKEN)) 92 | source = twitter.Twitter(TWITTER_ACCESS_TOKEN, 93 | TWITTER_ACCESS_TOKEN_SECRET, 94 | scrape_headers=TWITTER_SCRAPE_HEADERS) 95 | 96 | activities = source.get_activities(group_id=gr_source.SELF, fetch_likes=True) 97 | resps = ndb.get_multi(ndb.Key('Response', util.trim_nulls(a['id'])) 98 | for a in activities) 99 | resps = {r.key.id(): r for r in resps if r} 100 | 101 | last_exception = None 102 | for activity in activities: 103 | obj = activity.get('object', {}) 104 | 105 | # have we already posted or started on this response? 106 | resp = resps.get(activity['id']) 107 | mf2 = microformats2.object_to_json(activity) 108 | mf2_props = microformats2.first_props(mf2.get('properties', {})) 109 | type = as1.object_type(activity) 110 | 111 | if mf2_props.get('in-reply-to'): 112 | type = 'comment' # twitter reply 113 | if type not in TYPES or (resp and resp.status == 'complete'): 114 | continue 115 | elif resp: 116 | logging.info('Retrying %s', resp) 117 | else: 118 | resp = Response.get_or_insert(activity['id'], 119 | activity_json=json.dumps(activity)) 120 | logging.info('Created new Response: %s', resp) 121 | 122 | base_id = source.base_object(activity)['id'] 123 | base = source.get_activities(activity_id=base_id)[0] 124 | # logging.info(json.dumps(base, indent=2)) 125 | 126 | # make micropub call to create post 127 | # http://indiewebcamp.com/micropub 128 | # 129 | # include access token in both header and post body for compatibility 130 | # with servers that only support one or the other (for whatever reason). 131 | headers = {'Authorization': 'Bearer ' + MICROPUB_ACCESS_TOKEN} 132 | data = { 133 | 'access_token': MICROPUB_ACCESS_TOKEN, 134 | 'h': 'entry', 135 | 'category[]': CATEGORIES.get(type), 136 | 'content[html]': render(source, activity, base), 137 | 'name': base.get('content') or base.get('object', {}).get('content'), 138 | } 139 | for key in 'in-reply-to', 'like-of', 'repost-of', 'published', 'updated': 140 | val = mf2_props.get(key) 141 | if val: 142 | data[key] = microformats2.get_string_urls([val])[0] 143 | 144 | try: 145 | result = urlopen(MICROPUB_ENDPOINT, util.trim_nulls(data), headers=headers) 146 | except urllib.error.HTTPError as exception: 147 | last_exception = exception 148 | logging.exception('%s %s', exception.reason, exception.read()) 149 | continue 150 | except urllib.error.URLError as exception: 151 | last_exception = exception 152 | logging.exception(exception.reason) 153 | continue 154 | 155 | resp.post_url = result.info().get('Location') 156 | logging.info('Created new post: %s', resp.post_url) 157 | resp.response_body = result.read() 158 | logging.info('Response body: %s', resp.response_body) 159 | 160 | resp.status = 'complete' 161 | resp.put() 162 | 163 | # uncomment for testing 164 | # return 165 | 166 | # end loop over activities 167 | return ('Failed, see logs', 500) if last_exception else 'OK' 168 | 169 | def render(source, activity, base): 170 | obj = activity.get('object') or activity 171 | content = microformats2.render_content(obj) 172 | embed = source.embed_post(base) 173 | 174 | type = as1.object_type(activity) 175 | content = activity.get('content', '') 176 | if type == 'share' and not content: 177 | content = 'retweeted this.' 178 | 179 | rendered = embed + content if type == 'comment' else content + embed 180 | 181 | mf2_class = {'like': 'u-like-of', 182 | 'share': 'u-repost-of', 183 | }.get(type, 'in-reply-to') 184 | url = (obj.get('inReplyTo') or [{}])[0].get('url') or base.get('url') 185 | rendered += """ 186 | 187 | 188 | """ % (mf2_class, url, activity.get('url')) 189 | 190 | return rendered 191 | 192 | def urlopen(url, data, headers=None): 193 | data = {key: val for key, val in data.items()} 194 | data = urllib.parse.urlencode(data).encode() 195 | 196 | logging.info('Fetching %s with headers %s, data %s', url, headers, data) 197 | if headers: 198 | url = urllib.request.Request(url, data=data, headers=headers) 199 | return urllib.request.urlopen(url, timeout=600, data=data) 200 | -------------------------------------------------------------------------------- /app.yaml: -------------------------------------------------------------------------------- 1 | # https://cloud.google.com/appengine/docs/standard/python3/config/appref 2 | 3 | # application: ownyourresponses 4 | runtime: python39 5 | 6 | # https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices 7 | # https://docs.gunicorn.org/en/latest/settings.html#timeout 8 | entrypoint: gunicorn --workers 1 --threads 3 --timeout 60 -b :$PORT app:app 9 | 10 | inbound_services: 11 | - warmup 12 | 13 | handlers: 14 | 15 | - url: / 16 | static_files: static/index.html 17 | upload: static/index.html 18 | 19 | # dynamic 20 | - url: .* 21 | script: auto 22 | secure: always 23 | # login: admin 24 | -------------------------------------------------------------------------------- /cron.yaml: -------------------------------------------------------------------------------- 1 | # timezone defaults to UTC 2 | 3 | cron: 4 | - description: polls silos for new responses 5 | url: /cron/poll 6 | schedule: every 30 minutes 7 | -------------------------------------------------------------------------------- /index.yaml: -------------------------------------------------------------------------------- 1 | indexes: 2 | # AUTOGENERATED 3 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # Note that App Engine / Cloud Build's default pip cache is overly aggressive 2 | # with GitHub dependency URLs like these. If I push a new commit to the branch 3 | # in one of these URLs, Cloud Build still uses the old cached version. 4 | # https://cloud.google.com/appengine/docs/standard/python3/specifying-dependencies 5 | # 6 | # Use this to bust that cache: gcloud -q beta app deploy --no-cache ... 7 | git+https://github.com/snarfed/granary.git#egg=granary 8 | git+https://github.com/snarfed/oauth-dropins.git#egg=oauth_dropins 9 | flask~=2.2.3 10 | gunicorn~=20.1 11 | -------------------------------------------------------------------------------- /static/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | See 5 | github.com/snarfed/ownyourresponses. 6 | 7 | 8 | --------------------------------------------------------------------------------