├── .buildpacks
├── .gitignore
├── .travis.yml
├── CHANGELOG.md
├── README.md
├── facebook_sublet_group_bot
├── TestPosts.py
├── __init__.py
├── check_and_delete.py
├── delete_test.py
├── fb_bot.py
├── fbxmpp.py
├── test.py
└── util.py
├── requirements.txt
└── runtime.txt
/.buildpacks:
--------------------------------------------------------------------------------
1 | https://github.com/heroku/heroku-buildpack-python.git
2 |
--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | *.py[cod]
2 |
3 | # C extensions
4 | *.so
5 |
6 | # Packages
7 | *.egg
8 | *.egg-info
9 | dist
10 | build
11 | eggs
12 | parts
13 | bin
14 | var
15 | sdist
16 | develop-eggs
17 | .installed.cfg
18 | lib
19 | lib64
20 | __pycache__
21 |
22 | # Installer logs
23 | pip-log.txt
24 |
25 | # Unit test / coverage reports
26 | .coverage
27 | .tox
28 | nosetests.xml
29 |
30 | # Translations
31 | *.mo
32 |
33 | # Mr Developer
34 | .mr.developer.cfg
35 | .project
36 | .pydevproject
37 |
38 | # Prop and cache files
39 | login_prop
40 | fb_subs_cache
41 | fb_subs_valid_cache
42 | ghostdriver.log
43 |
44 | # PyCharm
45 | *.xml
46 | *.iml
47 | .idea/
--------------------------------------------------------------------------------
/.travis.yml:
--------------------------------------------------------------------------------
1 | language: python
2 | python:
3 | - "2.7"
4 | # command to install dependencies
5 | install: "pip install -r requirements.txt"
6 | # command to run tests
7 | script: python facebook_sublet_group_bot/test.py
--------------------------------------------------------------------------------
/CHANGELOG.md:
--------------------------------------------------------------------------------
1 | FB Mod Bot Changelog
2 | ====================
3 |
4 | ### 02/08/14
5 | * Created changelog
6 | * Added `[PARKING]` tag checking
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | [](https://travis-ci.org/hzsweers/FB_Mod_Bot)
2 | Facebook Mod Bot
3 | ================
4 |
5 |
6 | 
7 |
8 |
9 | **[Changelog](https://github.com/hzsweers/FB_Mod_Bot/blob/master/CHANGELOG.md)**
10 |
11 | This is a facebook bot that I wrote to moderate a group I admin, essentially keeping some order and upholding some requirements per the group rules. The main purpose is to crack down on vague/uninformative posts.
12 |
13 | Anyone is welcome to use this code and/or repurpose it for their own use. All I ask is that you give me credit somewhere :)
14 |
15 | The bot is written in Python, using the [facepy](https://github.com/jgorset/facepy) API wrapper for Facebook's graph API. To make the bot functional and separate from my personal account, I created a dedicated Facebook account for my bot (literally called "Sublets Bot"), registered it as a developer, created an app, granted that same app full permissions to the account, and finally use that access token in the bot itself to authenticate with Facebook's Graph API.
16 |
17 | * *Why make a dedicated app instead of just creating an access token? Because only an app (with an app ID and secret key) can programatically request an extended access token. If I don't do that, I'd have to manually generate a new one every two hours.*
18 |
19 | The group is a sublets/roommate finding group for students at my university, where people can post sublet offerings, say they're looking for one, and also seek out other potential roommates.
20 |
21 | ### The bot has three major validation checks:
22 | * Pricing reference
23 | * There must be some sort of pricing referenced, either with `$` signs or by saying `____/month` or `____ per month`
24 | * Minimum length or craigslist link
25 | * If their post is under 200 characters, doesn't have a link to a craiglist ad, and isn't a parking offering, then they need to edit and include more details.
26 | * Proper tag
27 | * For easy searching, we require them to prepend their post with `[LOOKING]`, `[OFFERING]`, `[ROOMING]`, or `[PARKING]`.
28 | * The bot is purposefully a little fuzzy on this due to the large number of people that can't find the bracket keys on their keyboards ಠ_ಠ
29 |
30 | If any of those checks fail, it comments on the user notifying them of the problem(s) and specifies what it(they) are. Upon warning, it caches the post ID and time, giving them 24 hours to fix their post. On later runs, if a previously warned post is now valid, it removes the warning comment and the post ID from the cache. ~~I'd send a message thanking them too, but Facebook's API doesn't allow apps to send messages.~~ Yay XMPP
31 |
32 | ### TODO
33 | * ~~Prepare for Heroku hosting~~ Done!
34 | * ~~Automatically delete old posts~~ Done!
35 | * Improve documentation
36 |
37 | ### Caveats
38 | * Depending on how many posts you request at once (the `LIMIT` option in FQL queries), Facebook's API periodically just crashes and returns an internal server error. Fortunately this doesn't seem to happen unless you ask for extremely high numbers.
39 |
40 |
41 | ## License
42 |
43 | The MIT License (MIT)
44 |
45 | Copyright (c) 2014 Henri Sweers
46 |
47 | Permission is hereby granted, free of charge, to any person obtaining a copy of
48 | this software and associated documentation files (the "Software"), to deal in
49 | the Software without restriction, including without limitation the rights to
50 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
51 | the Software, and to permit persons to whom the Software is furnished to do so,
52 | subject to the following conditions:
53 |
54 | The above copyright notice and this permission notice shall be included in all
55 | copies or substantial portions of the Software.
56 |
57 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
58 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
59 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
60 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
61 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
62 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
63 |
--------------------------------------------------------------------------------
/facebook_sublet_group_bot/TestPosts.py:
--------------------------------------------------------------------------------
1 | __author__ = 'hsweers'
2 |
3 | good_posts = [
4 | """[OFFERING]
5 |
6 | Charming 1-bedroom studio apartment in Hyde Park/North Campus on UT Shuttle stop!
7 |
8 |
9 |
10 | *Quiet, downstairs corner unit next to private courtyard
11 | *UT Shuttle (IF) and Cap Metro #5 stop 20 seconds from your door
12 | *Small, 10-unit complex, renovated in 2013
13 | *Walk-in closet
14 | *Wood floors
15 | *Nonfurnished
16 | *Nonsmoking
17 | *Central AC/heat
18 | *On-site laundry
19 | *Official sublease through the property manager
20 | *Parking space close to unit
21 | *Trash, water paid
22 | *Walk to CVS, The Triangle, Daily Juice, Avenue B Grocery, tennis courts
23 |
24 | $770/mo for 12 month lease
25 | Available for Mid-August move-in
26 |
27 | *$75 application fee and $300 deposit*
28 | """,
29 |
30 | """[OFFERING]
31 | -Fall 14, Spring 15
32 | -For a shared bedroom and private bath at The Block on 221/2 and Pearl. (2 by 2 apartment)
33 | -535 per month or 636 with parking.
34 | -Message me ASAP if interested
35 | -It is for the room on the left. The other girl sharing that room is SUUPER cool!
36 | -water and electricity not included
37 | -internet/cable included
38 | -W/D in unit
39 | -large balcony
40 | -bus stop super close
41 | """,
42 |
43 | """[OFFERING]
44 |
45 | Passing my sub-lease in The Block on 28th, apartment .
46 | Due to a family emergency, I will not be attending UT, and I need someone to take over my lease as soon as possible.
47 | It is two bedroom, 2 restroom apartment in west campus, and walking distance from UT (8-15 min). The rent is $445/month plus utilities, and parking space is additional for an extra of $100/month. You will be sharing one bedroom with a girl, and there are two other girls also living in the apartment.
48 | The apartment already includes basic cable, wifi buildings, and internet.
49 | """,
50 |
51 | """[OFFERING]
52 | -FIRST MONTH'S RENT IS FREE ($899 per month becomes $824 per month!)
53 | -Subleasing my room at the Villas on Guadalupe for the Fall 2014 - Spring 2015 school year.
54 | -You get your own room, private bath, and parking spot (optional)
55 | -It is a 2 room apartment w/ loft, w/ 3 other roommates (males)
56 | -Rent is $899 per month, parking is available and is an optional additional $65.
57 | -Washer/dryer included in the unit
58 | -Unfurnished
59 | -Roommates don't smoke
60 | -Right above Torchy's Tacos and across the street from Wataburger
61 | -Apartment complex has a pool and weight room.
62 | - Move in date is Aug 16th.
63 | - Bedroom A in the layout photo.
64 | Message if interested or for more details. Thanks!
65 | """,
66 |
67 | """[LOOKING] [ROOMING]
68 |
69 | -20-year-old female, government major- Italian, exchange student from France.
70 | -Looking for a studio/ room + private bathroom if possible
71 | -For the full 2014-2015 academic year
72 | -Close to campus
73 | -Available right now if possible / max from the 17th of August
74 | -Around $750/ $800
75 | -Furnished if possible.
76 | Thanks :)
77 | """,
78 |
79 | """[OFFERING] 2BR/1BA in West Campus $865/mo. for ENTIRE UNIT (that's only $432.50 per room). Lease term is August 2014-2015. It isn't furnished, but I can leave a desk, some shelves, etc. for free if you'd like. This lease is for the entire unit, so you must have/find your own roommate. Utilities are not included, but it has free basic cable (with only one outlet, sadly). This will be an official lease sign-over through the complex. This is the cheapest place to live in West Campus with your own, private room. That's why I've lived there for the last 2 years. Location and price are amazing.
80 | Here is the Craigslist ad with lots of details and pictures:
81 | """
82 | ]
83 |
84 | bad_posts = [
85 | """[OFFERING] (UniversityEstates)
86 | -Looking to sublease my apartment
87 | -located
88 | - rent is $500 a month including ALL utilities.
89 | -Renter's insurance is required, which is $10-25 a month depending where you get it
90 | -Lease starts August 22, 2014 - July 31st, 2015
91 | -unfurnished
92 | -4x4
93 | -co-ed
94 | -Close to downtown
95 | -UT shuttle passes the front of the complex often
96 | -Includes parking
97 | - Each room has it's own lease
98 | -Official sublease
99 | -Full kitchen and laundry
100 | -Apartment complex contains pool, gym, computer lab, clubhouse.
101 | - http://www.universityestatesataustin.com/index.php
102 | For more info go ahead an message me through here! I'll be checking my other folder daily, and i'll be more than glad to help!
103 | """,
104 |
105 | """[OFFERING] 2BR/1BA in West Campus $865/mo. for ENTIRE UNIT (randomwordhere) (that's only $432.50 per room). Lease term is August 2014-2015. It isn't furnished, but I can leave a desk, some shelves, etc. for free if you'd like. This lease is for the entire unit, so you must have/find your own roommate. Utilities are not included, but it has free basic cable (with only one outlet, sadly). This will be an official lease sign-over through the complex. This is the cheapest place to live in West Campus with your own, private room. That's why I've lived there for the last 2 years. Location and price are amazing.
106 | Here is the Craigslist ad with lots of details and pictures:
107 | """,
108 |
109 | """[OFFERING][ROOMING] 2BR/1BA in West Campus $865/mo. for ENTIRE UNIT (randomwordhere) (that's only $432.50 per room). Lease term is August 2014-2015. It isn't furnished, but I can leave a desk, some shelves, etc. for free if you'd like. This lease is for the entire unit, so you must have/find your own roommate. Utilities are not included, but it has free basic cable (with only one outlet, sadly). This will be an official lease sign-over through the complex. This is the cheapest place to live in West Campus with your own, private room. That's why I've lived there for the last 2 years. Location and price are amazing.
110 | Here is the Craigslist ad with lots of details and pictures:
111 | """
112 | ]
--------------------------------------------------------------------------------
/facebook_sublet_group_bot/__init__.py:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/ZacSweers/FB_Mod_Bot/97b045db81fab91f7e0f4a46c7472ee6bcdcd7a7/facebook_sublet_group_bot/__init__.py
--------------------------------------------------------------------------------
/facebook_sublet_group_bot/check_and_delete.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # coding=utf-8
3 |
4 | import os
5 | import cPickle as pickle
6 | import datetime
7 | import facepy
8 | import getopt
9 | import time
10 | import sys
11 | import re
12 | from util import log, Color, notify_mac
13 |
14 | __author__ = 'Henri Sweers'
15 |
16 | ############## Global vars ##############
17 |
18 | # 24 hours, in seconds
19 | time_limit = 86400
20 |
21 | # Pickle cache file caching warned posts
22 | warned_db = "fb_subs_cache"
23 |
24 | # Pickle cache file caching valid posts
25 | valid_db = "fb_subs_valid_cache"
26 |
27 | # Pickle cache file for properties
28 | prop_file = "login_prop"
29 |
30 | # Boolean for key extensions
31 | extend_key = False
32 |
33 | # Dry run to disable deletions on a run
34 | dry_run = False
35 |
36 | # Boolean for checking heroku
37 | running_on_heroku = False
38 |
39 | # Tags: allowed leading characters
40 | allowed_leading_characters = ('*', '-')
41 |
42 | # Tags: allowed tags
43 | allowed_tags = {'looking', 'offering', 'parking', 'rooming'}
44 |
45 |
46 | # Junk method that I use for testing stuff periodically
47 | def test():
48 | log('Test', Color.PURPLE)
49 |
50 |
51 | # Use this method to set new vals for props, such as on your first run
52 | def set_new_props():
53 | saved_dict = load_properties()
54 |
55 | ###############################################################
56 | #### Uncomment lines below as needed to manually set stuff ####
57 | ###############################################################
58 |
59 | ###########################
60 | #### These are strings ####
61 | ###########################
62 |
63 | # saved_dict['sublets_oauth_access_token'] = "put-auth-token-here"
64 | # saved_dict['sublets_api_id'] = "put-app-id-here"
65 | # saved_dict['sublets_secret_key'] = "put-secret-key-here"
66 | # saved_dict['access_token_expiration'] = "put-access-token-expiration-here"
67 | # saved_dict['group_id'] = "put-group-id-here"
68 |
69 | ################################################################
70 | #### If you want post deletion, must be done outside of API ####
71 | ################################################################
72 |
73 | # saved_dict['FB_USER'] = "put-facebook-username-here"
74 | # saved_dict['FB_PWD'] = "put-facebook-password-here"
75 |
76 |
77 | ########################
78 | #### These are ints ####
79 | ########################
80 |
81 | # saved_dict['bot_id'] = put-bot-id-here
82 | # saved_dict['ignored_post_ids'].append()
83 | # saved_dict['ignore_source_ids'].append()
84 | # saved_dict['admin_ids'].append()
85 |
86 | #################################################################
87 | #### You can do other stuff too, the above are just examples ####
88 | #################################################################
89 |
90 | save_properties(saved_dict)
91 |
92 |
93 | # Method for initializing your prop values
94 | def init_props():
95 | test_dict = {'sublets_oauth_access_token': "put-auth-token-here",
96 | 'sublets_api_id': "put-app-id-here",
97 | 'sublets_secret_key': "put-secret-key-here",
98 | 'access_token_expiration': "put-access-token-expiration-here",
99 | 'group_id': 'put-group-id-here',
100 | 'FB_USER': "put-facebook-username-here",
101 | 'FB_PWD': "put-facebook-password-here",
102 | 'bot_id': -1,
103 | 'ignored_post_ids': [],
104 | 'ignore_source_ids': [],
105 | 'admin_ids': []}
106 | save_properties(test_dict)
107 | saved_dict = load_properties()
108 | assert test_dict == saved_dict
109 |
110 |
111 | def save_properties(data):
112 | """
113 | Save props data into either memcache (heroku only) or pickle to local file
114 | :param data: Dictionary of props data
115 | """
116 | if running_on_heroku:
117 | mc.set('props', data)
118 | else:
119 | with open(prop_file, 'w+') as login_prop_file:
120 | pickle.dump(data, login_prop_file)
121 |
122 |
123 | def load_properties():
124 | """
125 | Load properties data from either memcache (heroku only) or local pickled file
126 |
127 | :rtype : dict
128 | :return: Dictionary of all the properties
129 | """
130 | if running_on_heroku:
131 | obj = mc.get('props')
132 | if not obj:
133 | return {}
134 | else:
135 | return obj
136 | else:
137 | if os.path.isfile(prop_file):
138 | with open(prop_file, 'r+') as login_prop_file:
139 | data = pickle.load(login_prop_file)
140 | return data
141 | else:
142 | sys.exit("No prop file found")
143 |
144 |
145 | # Method for loading a cache. Either returns cached values or original data
146 | def load_cache(cachename, data):
147 | if running_on_heroku:
148 | if running_on_heroku:
149 | obj = mc.get(cachename)
150 | if not obj:
151 | return data
152 | else:
153 | return obj
154 | else:
155 | if os.path.isfile(cachename):
156 | with open(cachename, 'r+') as f:
157 |
158 | # If the file isn't at its end or empty
159 | if f.tell() != os.fstat(f.fileno()).st_size:
160 | return pickle.load(f)
161 | else:
162 | log("--No cache file found, a new one will be created", Color.BLUE)
163 | return data
164 |
165 |
166 | # Method for saving to cache
167 | def save_cache(cachename, data):
168 | if running_on_heroku:
169 | mc.set(cachename, data)
170 | else:
171 | with open(cachename, 'w+') as f:
172 | pickle.dump(data, f)
173 |
174 |
175 | # Manually update API token
176 | def update_token(token):
177 | log("Updating token", Color.BLUE)
178 | graph = facepy.GraphAPI(token)
179 | try:
180 | graph.get('me/posts')
181 | props_dict = load_properties()
182 | props_dict['sublets_oauth_access_token'] = token
183 | props_dict['access_token_expiration'] = time.time() + 7200 # 2 hours buffer
184 | save_properties(props_dict)
185 | log("Token updated, you should now extend it", Color.BLUE)
186 | except Exception as e:
187 | log("API error - " + e.message, Color.RED)
188 |
189 |
190 | # Set a property by name
191 | def update_prop(prop_name, value):
192 | if prop_name in ("sublets_oauth_access_token", "access_token_expiration"):
193 | log("Please use -u or -e to update or extend tokens", Color.RED)
194 | return
195 | log("Setting \"" + prop_name + "\" to \"" + value + "\"", Color.BLUE)
196 | props_dict = load_properties()
197 | if prop_name not in props_dict.keys():
198 | input_response = raw_input("This key doesn't exist, do you want to add it? Y/N")
199 | if input_response.lower() != "y":
200 | return
201 | props_dict[prop_name] = value
202 | save_properties(props_dict)
203 | log("Done setting \"" + prop_name + "\"", Color.BLUE)
204 |
205 |
206 | # Method for checking tag validity
207 | def get_tags_old(message_text):
208 | p = re.compile(
209 | "^(-|\*| )*([\(\[\{])((looking)|(rooming)|(offering)|(parking))([\)\]\}])(:)?(\s|$)",
210 | re.IGNORECASE)
211 |
212 | split = message_text.strip().split(" ")
213 |
214 | # Check that they're on the first line
215 | if not p.match(split[0]) and split[0] not in allowed_leading_characters:
216 | return None
217 |
218 | tags_list = [l.lower()[1:-1] for l in split for m in (p.search(l),) if m]
219 | if len(tags_list) > 0:
220 | return tags_list
221 | else:
222 | return None
223 |
224 |
225 | # Method for checking tag validity
226 | def get_tags(message_text):
227 | p = re.compile(
228 | r"^(-|\*| )*(([\(\[\{])(.+)([\)\]\}]))+(:)?(\s|$)",
229 | re.IGNORECASE)
230 |
231 | # Insert space between two consecutive tags
232 | message_text = re.sub(r"([\)\]\}])([\(\[\{])", r"\1 \2", message_text)
233 | firstline_split = message_text.split("\n")[0].strip().split()
234 |
235 | # Check that they're on the first line
236 | if not p.match(firstline_split[0]) and firstline_split[0] not in allowed_leading_characters:
237 | return None
238 |
239 | tags_list = [re.sub(r"^(-|\*| )*", "", full_tag.lower())[1:(-2 if full_tag[-1] == ":" else -1)] for full_tag in
240 | firstline_split for matched in (p.search(full_tag),) if matched]
241 |
242 | # Should really learn how to do python's builtin logging...
243 | # log('--Tags: ' + ', '.join(tags_list), Color.BLUE)
244 |
245 | if len(tags_list) > 0 and set(tags_list).issubset(allowed_tags):
246 | return tags_list
247 | else:
248 | return None
249 |
250 |
251 | # Can't have "rooming" AND "offering". These people are usually just misusing the rooming tag
252 | def validate_tags(tags):
253 | if not tags:
254 | return False
255 | elif "rooming" in tags and "offering" in tags:
256 | return False
257 | else:
258 | return True
259 |
260 |
261 | # Method for checking if pricing reference is there
262 | def check_price_validity(message_text):
263 | p = re.compile(
264 | "(\$)|((\d)+( )?((per)|(/)|(a))( )?(/)?((month)|(mon)|(mo))(\s)?)",
265 | re.IGNORECASE)
266 |
267 | if re.search(p, message_text) is not None:
268 | return True
269 | else:
270 | return False
271 |
272 |
273 | # Checking if there's a parking tag
274 | def check_for_parking_tag(message_text):
275 | p = re.compile(
276 | "^(-|\*| )*([\(\[\{])(parking)([\)\]\}])(:)?(\s|$)",
277 | re.IGNORECASE)
278 |
279 | if re.search(p, message_text):
280 | return True
281 | else:
282 | return False
283 |
284 |
285 | # Method for extending access token
286 | def extend_access_token(saved_props, token, sublets_api_id,
287 | sublets_secret_key):
288 | log("Extending access token", Color.BOLD)
289 | access_token, expires_at = facepy.get_extended_access_token(
290 | token,
291 | sublets_api_id,
292 | sublets_secret_key
293 | )
294 | new_token = access_token
295 | unixtime = time.mktime(expires_at.timetuple())
296 | print time.mktime(expires_at.timetuple())
297 | saved_props['sublets_oauth_access_token'] = new_token
298 | saved_props['access_token_expiration'] = unixtime
299 | log("Token extended", Color.BOLD)
300 |
301 |
302 | # Method for retrieving user ID's of admins in group, ignoring bot ID
303 | def retrieve_admin_ids(group_id, auth_token):
304 | # Retrieve the uids via FQL query
305 | graph = facepy.GraphAPI(auth_token)
306 | admins_query = \
307 | "SELECT uid FROM group_member WHERE gid=" + group_id + " AND" + \
308 | " administrator"
309 | admins = graph.fql(query=admins_query)
310 |
311 | # Parse out the uids from the response
312 | admins_list = [admin['uid'] for admin in admins]
313 |
314 | # Update the admin_ids in our properties
315 | saved_props = load_properties()
316 | saved_props['admin_ids'] = admins_list
317 | save_properties(saved_props)
318 |
319 | return admins_list
320 |
321 |
322 | # Delete posts older than 30 days old
323 | def delete_old_posts(graph, group_id, admin_ids):
324 | old_date = int(time.time()) - 2592000 # 30 days in seconds
325 | old_query = "SELECT post_id, message, actor_id FROM stream WHERE " + \
326 | "source_id=" + group_id + " AND created_time<" + str(old_date) + \
327 | " LIMIT 300"
328 | log("Getting posts older than:")
329 | log("\t" + datetime.datetime.fromtimestamp(old_date)
330 | .strftime('%Y-%m-%d %H:%M:%S'))
331 | posts = graph.fql(query=old_query)
332 | deleted_posts_count = 0
333 | for post in posts["data"]:
334 | post_id = post['post_id']
335 | actor_id = post['actor_id']
336 | if int(actor_id) in admin_ids:
337 | # log('\n--Ignored post: ' + post_id, Color.BLUE)
338 | continue
339 | print post_id
340 | graph.delete(post_id)
341 | deleted_posts_count += 1
342 | log("Deleted " + str(deleted_posts_count) + " old posts", Color.RED)
343 |
344 |
345 | # Main runner method
346 | def sub_group():
347 | # Load the properties
348 | saved_props = load_properties()
349 |
350 | # Access token
351 | sublets_oauth_access_token = saved_props['sublets_oauth_access_token']
352 |
353 | # Access token expiration
354 | access_token_expiration = saved_props['access_token_expiration']
355 |
356 | # API App ID
357 | sublets_api_id = saved_props['sublets_api_id']
358 |
359 | # API App secret key
360 | sublets_secret_key = saved_props['sublets_secret_key']
361 |
362 | # ID of the FB group
363 | group_id = saved_props['group_id']
364 |
365 | # IDs of admins (unused right now, might remove later)
366 | admin_ids = saved_props['admin_ids']
367 |
368 | # FQL query for the group
369 | group_query = "SELECT post_id, message, actor_id FROM stream WHERE " + \
370 | "source_id=" + group_id + " LIMIT 50"
371 |
372 | # Get current time
373 | now_time = time.time()
374 |
375 | # For logging purposes
376 | log("CURRENT CST TIMESTAMP: " + datetime.datetime.fromtimestamp(
377 | now_time - 21600).strftime('%Y-%m-%d %H:%M:%S'), Color.UNDERLINE)
378 |
379 | # Make sure the access token is still valid
380 | if access_token_expiration < now_time:
381 | sys.exit("API Token is expired")
382 |
383 | # Warn if the token's expiring soon
384 | if access_token_expiration - now_time < 604800:
385 | log("Warning - access token expires in less than a week", Color.RED)
386 | log("-- Expires on " + datetime.datetime.fromtimestamp(
387 | access_token_expiration).strftime('%Y-%m-%d %H:%M:%S'))
388 |
389 | # If you want it to automatically when it's close to exp.
390 | global extend_key
391 | extend_key = True
392 |
393 | # Extend the access token, default is ~2 months from current date
394 | if extend_key:
395 | extend_access_token(saved_props, sublets_oauth_access_token, sublets_api_id,
396 | sublets_secret_key)
397 |
398 | # Log in, try to get posts
399 | graph = facepy.GraphAPI(sublets_oauth_access_token)
400 |
401 | # Make our first request, get the group posts
402 | group_posts = graph.fql(query=group_query)
403 |
404 | # Load the pickled cache of valid posts
405 | valid_posts = []
406 | log("Checking valid cache.", Color.BOLD)
407 | valid_posts = load_cache(valid_db, valid_posts)
408 | log('--Valid cache size: ' + str(len(valid_posts)), Color.BOLD)
409 | invalid_count = 0
410 |
411 | # Loop over retrieved posts
412 | for post in group_posts["data"]:
413 |
414 | # Important data received
415 | post_message = post['message'] # Content of the post
416 | post_id = post['post_id'] # Unique ID of the post
417 |
418 | # Unique ID of the person that posted it
419 | actor_id = post['actor_id']
420 |
421 | # Ignore mods and certain posts
422 | if int(actor_id) in admin_ids:
423 | log('\n--Ignored post: ' + post_id, Color.BLUE)
424 | continue
425 |
426 | # Boolean for tracking if the post is valid
427 | valid_post = True
428 |
429 | # Log the message details
430 | # log("\n" + post_message[0:75].replace('\n', "") + "...\n--POST ID: " +
431 | # str(post_id) + "\n--ACTOR ID: " + str(actor_id))
432 |
433 | # Check for pricing
434 | if not check_price_validity(post_message):
435 | valid_post = False
436 | log('----$', Color.RED)
437 | invalid_count += 1
438 |
439 | # Check for tag validity, including tags that say rooming and offering
440 | tags = get_tags(post_message)
441 | if not validate_tags(tags):
442 | valid_post = False
443 | log('----Tag', Color.RED)
444 | invalid_count += 1
445 |
446 | # Check post length.
447 | # Allow short ones if there's a craigslist link or parking
448 | if len(post_message) < 200 \
449 | and "craigslist" not in post_message.lower() \
450 | and not check_for_parking_tag(post_message):
451 | valid_post = False
452 | log('----Length', Color.RED)
453 | invalid_count += 1
454 |
455 | # Not a valid post
456 | if not valid_post:
457 | if dry_run:
458 | log("Dry - invalid deletion", Color.RED)
459 | log("--ID: " + post_id, Color.RED)
460 | log("--Message: " + post_message, Color.RED)
461 | log("\n")
462 | else:
463 | graph.delete(post_id)
464 | else:
465 | valid_posts.append(post_id)
466 |
467 | if not dry_run:
468 | log("Deleted " + str(invalid_count) + " invalid posts", Color.RED)
469 |
470 | # # Delete posts older than 30 days
471 | delete_old_posts(graph, group_id, admin_ids)
472 |
473 | # Save the updated caches
474 | log('Saving valid cache', Color.BOLD)
475 | save_cache(valid_db, valid_posts)
476 |
477 | save_properties(saved_props)
478 |
479 | # Done
480 | notify_mac()
481 |
482 |
483 | # Main method
484 | if __name__ == "__main__":
485 |
486 | try:
487 | opts, args = getopt.getopt(sys.argv[1:], "fdpesu:n:v:g:",
488 | ["flushvalid", "dry", "printprops", "extend", "setprops", "token=", "propname=",
489 | "propvalue=", "propname="])
490 | except getopt.GetoptError:
491 | print 'check_and_delete.py -f -d -p -e -s -u -n -v -g '
492 | sys.exit(2)
493 |
494 | # Check to see if we're running on Heroku
495 | if os.environ.get('MEMCACHEDCLOUD_SERVERS', None):
496 | import bmemcached
497 |
498 | log('Running on heroku, using memcached', Color.BOLD)
499 |
500 | # Authenticate Memcached
501 | running_on_heroku = True
502 | mc = bmemcached.Client(os.environ.get('MEMCACHEDCLOUD_SERVERS').split(','),
503 | os.environ.get('MEMCACHEDCLOUD_USERNAME'), os.environ.get('MEMCACHEDCLOUD_PASSWORD'))
504 |
505 | propname = None
506 | propval = None
507 | log("Args - " + str(opts), Color.BOLD)
508 | if len(opts) != 0:
509 | for o, a in opts:
510 | if o in ("-e", "--extend"):
511 | extend_key = True
512 | elif o in ("-d", "--dry"):
513 | dry_run = True
514 | elif o in ("-s", "--setprops"):
515 | set_new_props()
516 | sys.exit()
517 | elif o in ("-u", "--update"):
518 | update_token(a)
519 | sys.exit()
520 | elif o in ("-n", "--propname"):
521 | propname = a
522 | elif o in ("-v", "--propvalue"):
523 | propval = a
524 | elif o in ("-p", "--printprops"):
525 | log("Printing props", Color.BLUE)
526 | props = load_properties()
527 | print props.keys()
528 | sys.exit()
529 | elif o in ("-g", "--getprop"):
530 | log("Getting value for " + a, Color.BLUE)
531 | props = load_properties()
532 | if a not in props.keys():
533 | sys.exit(a + " doesn't exist in props")
534 | print props[a]
535 | sys.exit()
536 | elif o in ("-f", "--flushvalid"):
537 | response = raw_input("Are you sure? Y/N")
538 | if response.lower() == 'y':
539 | log("Flushing cache for valid", Color.BLUE)
540 | save_cache(valid_db, [])
541 | log("Flushed cache", Color.BLUE)
542 | sys.exit()
543 | else:
544 | sys.exit('No valid args specified')
545 |
546 | if propname or propval:
547 | if propname and propval:
548 | log(propname + propval)
549 | update_prop(propname, propval)
550 | else:
551 | sys.exit('Must specify a prop name and value')
552 | else:
553 | sub_group()
--------------------------------------------------------------------------------
/facebook_sublet_group_bot/delete_test.py:
--------------------------------------------------------------------------------
1 | # coding=utf-8
2 | import os
3 | import pickle
4 | import sys
5 | import time
6 | import facepy
7 |
8 | __author__ = 'Henri Sweers'
9 |
10 |
11 | def load_properties():
12 | prop_file = "login_prop"
13 | if os.environ.get('MEMCACHEDCLOUD_SERVERS', None):
14 | import bmemcached
15 | mc = bmemcached.Client(os.environ.get('MEMCACHEDCLOUD_SERVERS').
16 | split(','),
17 | os.environ.get('MEMCACHEDCLOUD_USERNAME'),
18 | os.environ.get('MEMCACHEDCLOUD_PASSWORD'))
19 | obj = mc.get('props')
20 | if not obj:
21 | return {}
22 | else:
23 | return obj
24 | else:
25 | if os.path.isfile(prop_file):
26 | with open(prop_file, 'r+') as login_prop_file:
27 | data = pickle.load(login_prop_file)
28 | return data
29 | else:
30 | sys.exit("No prop file found")
31 |
32 |
33 | def test():
34 |
35 | # Check to see if we're running on Heroku, skip if we aren't
36 | if not os.environ.get('MEMCACHEDCLOUD_SERVERS', None):
37 | return True
38 |
39 | # Load the properties
40 | saved_props = load_properties()
41 |
42 | # Access token
43 | sublets_oauth_access_token = saved_props['sublets_oauth_access_token']
44 |
45 | # ID of the FB group
46 | group_id = saved_props['group_id']
47 |
48 | graph = facepy.GraphAPI(sublets_oauth_access_token)
49 |
50 | obj = graph.post(group_id + "/feed", message="test")
51 | postid = obj['id']
52 |
53 | try:
54 | graph.delete(postid)
55 | except Exception as e:
56 | print 'ERROR: ' + e.message
57 | print type(e)
58 | print 'Failed to delete with GraphAPI'
59 | return False
60 |
61 | print "Confirming deletion..."
62 | time.sleep(2)
63 | try:
64 | print graph.get(str(postid))
65 | return False
66 | except:
67 | print "Deletion confirmed ✓"
68 | return True
69 |
70 |
71 | if __name__ == "__main__":
72 | test()
73 |
--------------------------------------------------------------------------------
/facebook_sublet_group_bot/fb_bot.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python
2 | # coding=utf-8
3 |
4 | import os
5 | import cPickle as pickle
6 | import datetime
7 | import facebook
8 | import time
9 | import subprocess
10 | import sys
11 | import re
12 | from fbxmpp import SendMsgBot
13 | from util import log, Color
14 | from raven import Client
15 |
16 | __author__ = 'Henri Sweers'
17 |
18 | ############## Global vars ##############
19 |
20 | # 24 hours, in seconds
21 | time_limit = 86400
22 |
23 | # Pickle cache file caching warned posts
24 | warned_db = "fb_subs_cache"
25 |
26 | # Pickle cache file caching valid posts
27 | valid_db = "fb_subs_valid_cache"
28 |
29 | # Pickle cache file for properties
30 | prop_file = "login_prop"
31 |
32 | # Boolean for key extensions
33 | extend_key = False
34 |
35 | # Boolean for checking heroku
36 | running_on_heroku = False
37 |
38 |
39 | # Junk method that I use for testing stuff periodically
40 | def test():
41 | log('Test', Color.PURPLE)
42 |
43 |
44 | # Method for sending messages, adapted from here: http://goo.gl/oV5KtZ
45 | def send_message(recipient, message):
46 | saved_props = load_properties()
47 |
48 | # Access token
49 | access_token = saved_props['sublets_oauth_access_token']
50 |
51 | # API App ID
52 | api_key = saved_props['sublets_api_id']
53 |
54 | # User ID of the bot
55 | botid = str(saved_props['bot_id'])
56 |
57 | # The "From" Facebook ID
58 | jid = botid + '@chat.facebook.com'
59 |
60 | # The "Recipient" Facebook ID, with a hyphen for some reason
61 | to = '-' + str(recipient) + '@chat.facebook.com'
62 |
63 | xmpp = SendMsgBot(jid, to, unicode(message))
64 |
65 | xmpp.credentials['api_key'] = api_key
66 | xmpp.credentials['access_token'] = access_token
67 |
68 | if xmpp.connect(('chat.facebook.com', 5222)):
69 | xmpp.process(block=True)
70 | log('----Message sent', Color.GREEN)
71 | else:
72 | log("----Unable to connect, message sending fail", Color.RED)
73 |
74 |
75 | # Use this method to set new vals for props, such as on your first run
76 | def set_new_props():
77 | saved_dict = load_properties()
78 |
79 | ###############################################################
80 | #### Uncomment lines below as needed to manually set stuff ####
81 | ###############################################################
82 |
83 | ###########################
84 | #### These are strings ####
85 | ###########################
86 |
87 | # saved_dict['sublets_oauth_access_token'] = "put-auth-token-here"
88 | # saved_dict['sublets_api_id'] = "put-app-id-here"
89 | # saved_dict['sublets_secret_key'] = "put-secret-key-here"
90 | # saved_dict['access_token_expiration'] = "put-access-token-expiration-here"
91 | # saved_dict['group_id'] = "put-group-id-here"
92 |
93 | ################################################################
94 | #### If you want post deletion, must be done outside of API ####
95 | ################################################################
96 |
97 | # saved_dict['FB_USER'] = "put-facebook-username-here"
98 | # saved_dict['FB_PWD'] = "put-facebook-password-here"
99 |
100 |
101 | ########################
102 | #### These are ints ####
103 | ########################
104 |
105 | # saved_dict['bot_id'] = put-bot-id-here
106 | # saved_dict['ignored_post_ids'].append()
107 | # saved_dict['ignore_source_ids'].append()
108 | # saved_dict['admin_ids'].append()
109 |
110 | #################################################################
111 | #### You can do other stuff too, the above are just examples ####
112 | #################################################################
113 |
114 | save_properties(saved_dict)
115 |
116 |
117 | # Method for initializing your prop values
118 | def init_props():
119 | test_dict = {'sublets_oauth_access_token': "put-auth-token-here",
120 | 'sublets_api_id': "put-app-id-here",
121 | 'sublets_secret_key': "put-secret-key-here",
122 | 'access_token_expiration': "put-access-token-expiration-here",
123 | 'group_id': 'put-group-id-here',
124 | 'FB_USER': "put-facebook-username-here",
125 | 'FB_PWD': "put-facebook-password-here",
126 | 'bot_id': -1,
127 | 'ignored_post_ids': [],
128 | 'ignore_source_ids': [],
129 | 'admin_ids': []}
130 | save_properties(test_dict)
131 | saved_dict = load_properties()
132 | assert test_dict == saved_dict
133 |
134 |
135 | # Method for saving (with pickle) your prop values
136 | def save_properties(data):
137 | if running_on_heroku:
138 | mc.set('props', data)
139 | else:
140 | with open(prop_file, 'w+') as login_prop_file:
141 | pickle.dump(data, login_prop_file)
142 |
143 |
144 | # Method for loading (with pickle) your prop values
145 | def load_properties():
146 | if running_on_heroku:
147 | obj = mc.get('props')
148 | if not obj:
149 | return {}
150 | else:
151 | return obj
152 | else:
153 | if os.path.isfile(prop_file):
154 | with open(prop_file, 'r+') as login_prop_file:
155 | data = pickle.load(login_prop_file)
156 | return data
157 | else:
158 | sys.exit("No prop file found")
159 |
160 |
161 | # Method for loading a cache. Either returns cached values or original data
162 | def load_cache(cachename, data):
163 | if running_on_heroku:
164 | if running_on_heroku:
165 | obj = mc.get(cachename)
166 | if not obj:
167 | return data
168 | else:
169 | return obj
170 | else:
171 | if os.path.isfile(cachename):
172 | with open(cachename, 'r+') as f:
173 |
174 | # If the file isn't at its end or empty
175 | if f.tell() != os.fstat(f.fileno()).st_size:
176 | return pickle.load(f)
177 | else:
178 | log("--No cache file found, a new one will be created", Color.BLUE)
179 | return data
180 |
181 |
182 | # Method for saving to cache
183 | def save_cache(cachename, data):
184 | if running_on_heroku:
185 | mc.set(cachename, data)
186 | else:
187 | with open(cachename, 'w+') as f:
188 | pickle.dump(data, f)
189 |
190 |
191 | # Nifty method for sending notifications on my mac when it's done
192 | def notify_mac():
193 | if sys.platform == "darwin":
194 | try:
195 | subprocess.call(
196 | ["terminal-notifier", "-message", "Done", "-title", "FB_Bot",
197 | "-sound", "default"])
198 | except OSError:
199 | print "If you have terminal-notifier, this would be a notification"
200 |
201 |
202 | # Method for checking tag validity
203 | def check_tag_validity(message_text):
204 | p = re.compile(
205 | "^(-|\*| )*([\(\[\{])((looking)|(rooming)|(offering)|(parking))([\)\]\}])(:)?(\s|$)",
206 | re.IGNORECASE)
207 |
208 | if re.match(p, message_text):
209 | return True
210 | else:
211 | return False
212 |
213 |
214 | # Method for checking if pricing reference is there
215 | def check_price_validity(message_text):
216 | p = re.compile(
217 | "(\$)|((\d)+( )?((per)|(/)|(a))( )?(/)?((month)|(mon)|(mo))(\s)?)",
218 | re.IGNORECASE)
219 |
220 | if re.search(p, message_text) is not None:
221 | return True
222 | else:
223 | return False
224 |
225 |
226 | # Checking if there's a parking tag
227 | def check_for_parking_tag(message_text):
228 | p = re.compile(
229 | "^(-|\*| )*([\(\[\{])(parking)([\)\]\}])(:)?(\s|$)",
230 | re.IGNORECASE)
231 |
232 | if re.search(p, message_text):
233 | return True
234 | else:
235 | return False
236 |
237 |
238 | # Method for extending access token
239 | def extend_access_token(graph, now_time, saved_props, sublets_api_id,
240 | sublets_secret_key):
241 | log("Extending access token", Color.BOLD)
242 | result = graph.extend_access_token(sublets_api_id, sublets_secret_key)
243 | new_token = result['access_token']
244 | new_time = int(result['expires']) + now_time
245 | saved_props['sublets_oauth_access_token'] = new_token
246 | saved_props['access_token_expiration'] = new_time
247 | log("Token extended", Color.BOLD)
248 |
249 |
250 | # Method for retrieving user ID's of admins in group, ignoring bot ID
251 | def retrieve_admin_ids(group_id, bot_id, auth_token):
252 | # Retrieve the uids via FQL query
253 | graph = facebook.GraphAPI(auth_token)
254 | admins_query = \
255 | "SELECT uid FROM group_member WHERE gid=" + group_id + " AND" + \
256 | " administrator AND NOT (uid = " + str(bot_id) + ")"
257 | admins = graph.fql(query=admins_query)
258 |
259 | # Parse out the uids from the response
260 | admins_list = [admin['uid'] for admin in admins]
261 |
262 | # Update the admin_ids in our properties
263 | saved_props = load_properties()
264 | saved_props['admin_ids'] = admins_list
265 | save_properties(saved_props)
266 |
267 | return admins_list
268 |
269 |
270 | # Extracted logic for messaging admins a message
271 | def message_admins(message, auth_token, app_id, bot_id, group_id):
272 | for admin in retrieve_admin_ids(group_id, bot_id, auth_token):
273 | send_message(str(admin), message)
274 |
275 |
276 | # Delete posts older than 30 days old
277 | def delete_old_posts(graph, group_id, admins):
278 | old_date = int(time.time()) - 2592000 # 30 days in seconds
279 | old_query = "SELECT post_id, message, actor_id FROM stream WHERE " + \
280 | "source_id=" + group_id + " AND created_time<" + str(old_date) + \
281 | " LIMIT 300"
282 | log("Getting posts older than:")
283 | log("\t" + datetime.datetime.fromtimestamp(old_date)
284 | .strftime('%Y-%m-%d %H:%M:%S'))
285 | posts = graph.fql(query=old_query)
286 | log("Deleting " + str(len(posts)) + " posts", Color.RED)
287 | for post in posts:
288 | post_id = post['post_id']
289 | graph.delete_object(id=post_id)
290 |
291 | ## Old stuff messaging users their post
292 | # post_message = post['message']
293 | # post_id = post['post_id']
294 | # actor_id = post['actor_id']
295 | #
296 | # if int(actor_id) not in admins:
297 | # message = "We are deleting old posts. Your post's message is pasted" + \
298 | # " below. Feel free to repost it if you still need to.\n\n" + \
299 | # post_message
300 | #
301 | # send_message(str(actor_id), message)
302 | # log("\tDeleting " + post_id, Color.RED)
303 | # graph.delete_object(id=post_id)
304 | # time.sleep(2)
305 | # else:
306 | # log("\tSkipping admin post: " + post_id, Color.BLUE)
307 |
308 |
309 | # Main runner method
310 | def sub_group():
311 | # Load the properties
312 | saved_props = load_properties()
313 |
314 | # Access token
315 | sublets_oauth_access_token = saved_props['sublets_oauth_access_token']
316 |
317 | # Access token expiration
318 | access_token_expiration = saved_props['access_token_expiration']
319 |
320 | # API App ID
321 | sublets_api_id = saved_props['sublets_api_id']
322 |
323 | # API App secret key
324 | sublets_secret_key = saved_props['sublets_secret_key']
325 |
326 | # List of posts to ignore
327 | ignored_post_ids = saved_props['ignored_post_ids']
328 |
329 | # List of people to ignore
330 | ignore_source_ids = saved_props['ignore_source_ids']
331 |
332 | # ID of the FB group
333 | group_id = saved_props['group_id']
334 |
335 | # User ID of the bot
336 | bot_id = saved_props['bot_id']
337 |
338 | # IDs of admins (unused right now, might remove later)
339 | admin_ids = saved_props['admin_ids']
340 |
341 | # Keeping track of the posts we look at here
342 | processed_posts = []
343 |
344 | # FQL query for the group
345 | group_query = "SELECT post_id, message, actor_id FROM stream WHERE " + \
346 | "source_id=" + group_id + " LIMIT 50"
347 |
348 | # Get current time
349 | now_time = time.time()
350 |
351 | # For logging purposes
352 | log("CURRENT CST TIMESTAMP: " + datetime.datetime.fromtimestamp(
353 | now_time - 21600).strftime('%Y-%m-%d %H:%M:%S'), Color.UNDERLINE)
354 |
355 | # Make sure the access token is still valid
356 | if access_token_expiration < now_time:
357 | sys.exit("API Token is expired")
358 |
359 | # Warn if the token's expiring soon
360 | if access_token_expiration - now_time < 604800:
361 | log("Warning - access token expires in less than a week", Color.RED)
362 | log("-- Expires on " + datetime.datetime.fromtimestamp(
363 | access_token_expiration).strftime('%Y-%m-%d %H:%M:%S'))
364 |
365 | # If you want it to automatically when it's close to exp.
366 | global extend_key
367 | extend_key = True
368 |
369 | # Log in, try to get posts
370 | graph = facebook.GraphAPI(sublets_oauth_access_token)
371 |
372 | # Extend the access token, default is ~2 months from current date
373 | if extend_key:
374 | extend_access_token(graph, now_time, saved_props, sublets_api_id,
375 | sublets_secret_key)
376 |
377 | # Make our first request, get the group posts
378 | group_posts = graph.fql(query=group_query)
379 |
380 | # Load the pickled cache of previously warned posts
381 | already_warned = dict()
382 | log("Loading warned cache", Color.BOLD)
383 | already_warned = load_cache(warned_db, already_warned)
384 | log('--Loading cache size: ' + str(len(already_warned)), Color.BOLD)
385 |
386 | # Load the pickled cache of valid posts
387 | valid_posts = []
388 | log("Checking valid cache.", Color.BOLD)
389 | valid_posts = load_cache(valid_db, valid_posts)
390 | log('--Valid cache size: ' + str(len(valid_posts)), Color.BOLD)
391 |
392 | # Loop over retrieved posts
393 | for post in group_posts:
394 |
395 | # Important data received
396 | post_message = post['message'] # Content of the post
397 | post_id = post['post_id'] # Unique ID of the post
398 | processed_posts.append(post_id)
399 |
400 | # Unique ID of the person that posted it
401 | actor_id = post['actor_id']
402 |
403 | # Ignore mods and certain posts
404 | if post_id in ignored_post_ids or actor_id in ignore_source_ids or \
405 | post_id in valid_posts or int(actor_id) in admin_ids:
406 | # log('\n--Ignored post: ' + post_id, Color.BLUE)
407 | continue
408 |
409 | # Data to use
410 | post_comment = ""
411 |
412 | # Boolean for tracking if the post is valid
413 | valid_post = True
414 |
415 | # Counter for multiple items
416 | invalid_count = 0
417 |
418 | # Log the message details
419 | log("\n" + post_message[0:75].replace('\n', "") + "...\n--POST ID: " +
420 | str(post_id) + "\n--ACTOR ID: " + str(actor_id))
421 |
422 | # Check for pricing
423 | if not check_price_validity(post_message):
424 | valid_post = False
425 | invalid_count += 1
426 | post_comment += "- Please give some sort of pricing (use dollar signs!)\n"
427 | log('----$', Color.BLUE)
428 |
429 | # Check for tag validity
430 | if not check_tag_validity(post_message):
431 | valid_post = False
432 | invalid_count += 1
433 | post_comment += \
434 | "- You didn't include a proper tag\n"
435 | log('----Tag', Color.BLUE)
436 |
437 | # Check post length.
438 | # Allow short ones if there's a craigslist link or parking
439 | if len(post_message) < 200 and \
440 | "craigslist" not in post_message.lower() \
441 | and not check_for_parking_tag(post_message):
442 | valid_post = False
443 | invalid_count += 1
444 | post_comment += \
445 | "- Not enough details, please give some more info\n"
446 | log('----Length', Color.BLUE)
447 |
448 | # Not a valid post
449 | if not valid_post:
450 |
451 | if invalid_count > 1:
452 | post_comment = "Hey buddy, your post has some issues:\n" + post_comment
453 | else:
454 | post_comment = "Hey buddy, your post has an issue:\n" + post_comment
455 |
456 | # If already warned, delete if it's been more than 24 hours, ignore
457 | # if it's been less
458 | if post_id in already_warned:
459 |
460 | # Invalid, past 24 hour grace period
461 | if time_limit < now_time - already_warned[post_id]:
462 | log('--Delete: ' + post_id, Color.RED)
463 | url = "http://www.facebook.com/" + post_id
464 |
465 | # Try and delete the post with graph. If it fails, message
466 | # the admins and prompt them to delete the post
467 | try:
468 | graph.delete_object(id=post_id)
469 |
470 | log("--Confirming deletion...")
471 | try:
472 | # Give it a sec to propagate
473 | time.sleep(3)
474 | graph.get_object(id=post_id)
475 |
476 | # If it got here something went wrong
477 | message_admins(
478 | "Please make sure this is gone: " + url,
479 | sublets_oauth_access_token,
480 | sublets_api_id, bot_id, group_id)
481 | except:
482 | log("Deletion confirmed ✓", Color.GREEN)
483 | del already_warned[post_id]
484 |
485 | # Something went wrong, have the admins delete it
486 | except Exception as e:
487 | message_admins(
488 | "Delete this post: " + url,
489 | sublets_oauth_access_token,
490 | sublets_api_id, bot_id, group_id)
491 | log(e.message + " - " + str(type(e)), Color.RED)
492 |
493 | # Invalid but they still have time
494 | else:
495 | time_delta = time_limit - (now_time -
496 | already_warned[post_id])
497 | m, s = divmod(time_delta, 60)
498 | h, m = divmod(m, 60)
499 | log_message = '--Invalid, but still have '
500 | if h > 0:
501 | log_message += '%d hours and ' % h
502 | log_message += '%02d minutes' % m
503 | log(log_message, Color.RED)
504 | continue
505 |
506 | # Comment with a warning and cache the post
507 | else:
508 |
509 | # First check to make sure we haven't warned them before
510 | # by searching comments for bot comment
511 | previously_commented = False
512 | comments_query = "SELECT fromid, id, time FROM comment" + \
513 | " WHERE post_id=\"" + str(post_id) + "\""
514 | comments = graph.fql(comments_query)
515 | for comment in comments:
516 |
517 | # Found a comment from the bot
518 | if comment['fromid'] == bot_id:
519 | log('--Previously warned')
520 | log('----caching')
521 | previously_commented = True
522 | already_warned[post_id] = comment['time']
523 | break
524 |
525 | # Comment if no previous comment
526 | if not previously_commented:
527 | # Comment to post for warning
528 | post_comment += \
529 | "\nEdit your post and fix the above within 24" + \
530 | " hours, or else your post will be deleted per the" + \
531 | " group rules. Thanks!"
532 |
533 | graph.put_object(
534 | post['post_id'], "comments", message=post_comment)
535 | # Save
536 | already_warned[post_id] = now_time
537 | log('--WARNED', Color.RED)
538 |
539 | # Valid post
540 | else:
541 | log('--VALID', Color.GREEN)
542 |
543 | # Add to valid posts cache
544 | valid_posts.append(post_id)
545 | log('----caching', Color.GREEN)
546 |
547 | # Remove warning comment if it's valid now
548 | if post_id in already_warned:
549 | log('--Removing any warnings')
550 | comments_query = "SELECT fromid, id FROM comment" + \
551 | " WHERE post_id=\"" + str(post_id) + "\""
552 | comments = graph.fql(comments_query)
553 | for comment in comments:
554 | if comment['fromid'] == int(bot_id):
555 | # Delete warning comment
556 | graph.delete_object(comment['id'])
557 | log('--Warning deleted')
558 |
559 | # Message the user notifying them the comment is deleted
560 | # and thank them for fixing their post. Disabled for now
561 | # log('--Thanking user')
562 | # send_message(str(actor_id),
563 | # "Thanks for fixing your post," +
564 | # " I removed the warning comment.")
565 |
566 | # Remove post from list of warned people
567 | log('--Removing from cache')
568 | del already_warned[post_id]
569 |
570 | # Delete posts older than 30 days
571 | delete_old_posts(graph, group_id, admin_ids)
572 |
573 | # Keep our warned cache clean
574 | log('Cleaning warned posts', Color.BOLD)
575 | already_warned = dict((key, value) for (key, value) in
576 | already_warned.items() if key in processed_posts)
577 |
578 | # Save the updated caches
579 | log('Saving warned cache', Color.BOLD)
580 | save_cache(warned_db, already_warned)
581 |
582 | log('Saving valid cache', Color.BOLD)
583 | save_cache(valid_db, valid_posts)
584 |
585 | save_properties(saved_props)
586 |
587 | # Done
588 | notify_mac()
589 |
590 |
591 | # Main method
592 | if __name__ == "__main__":
593 | # Check to see if we're running on Heroku
594 | if os.environ.get('MEMCACHEDCLOUD_SERVERS', None):
595 | import bmemcached
596 |
597 | log('Running on heroku, using memcached', Color.BOLD)
598 |
599 | # Authenticate Memcached
600 | running_on_heroku = True
601 | mc = bmemcached.Client(os.environ.get('MEMCACHEDCLOUD_SERVERS').split(','),os.environ.get('MEMCACHEDCLOUD_USERNAME'),os.environ.get('MEMCACHEDCLOUD_PASSWORD'))
602 |
603 | args = sys.argv
604 | # parser = argparse.ArgumentParser()
605 | # parser.add_argument("-e", help="extend the access token on run")
606 | # parser.add_argument("-i", help="initialize properties")
607 | # parser.add_argument("-t", help="only run test method")
608 | # parser.add_argument("-p", help="set some property values")
609 | # args = parser.parse_args()
610 |
611 | # Arg parsing. I know, there's better ways to do this
612 | if len(args) > 1:
613 | if "--extend" in args:
614 | extend_key = True
615 | elif "setprops" in args:
616 | set_new_props()
617 | sys.exit()
618 | elif "init" in args:
619 | init_props()
620 | sys.exit()
621 | elif "test" in args:
622 | test()
623 | sys.exit("Done testing")
624 | else:
625 | sys.exit('No valid args specified')
626 |
627 | sub_group()
628 | # try:
629 | # sub_group()
630 | # except Exception:
631 | # if running_on_heroku:
632 | # # Use raven to capture exceptions and email
633 | # # Set your raven stuff by installing it in heroku and get the python
634 | # # setup info from the Python section
635 | # client = Client(os.environ.get('RAVEN'))
636 | # client.captureException()
637 |
--------------------------------------------------------------------------------
/facebook_sublet_group_bot/fbxmpp.py:
--------------------------------------------------------------------------------
1 | import sleekxmpp
2 | import logging
3 |
4 | logging.basicConfig(level=logging.FATAL)
5 |
6 |
7 | # Based on what I learned here:
8 | # http://goo.gl/oV5KtZ
9 | # Slightly updated since some parts didn't work
10 | class SendMsgBot(sleekxmpp.ClientXMPP):
11 |
12 | """
13 | A basic SleekXMPP bot that will log in, send a message,
14 | and then log out.
15 | """
16 |
17 | def __init__(self, jid, recipient, message):
18 |
19 | sleekxmpp.ClientXMPP.__init__(self, jid, 'ignore')
20 |
21 | # The message we wish to send, and the JID that
22 | # will receive it.
23 | self.recipient = recipient
24 | self.msg = message
25 |
26 | # The session_start event will be triggered when
27 | # the bot establishes its connection with the server
28 | # and the XML streams are ready for use. We want to
29 | # listen for this event so that we we can initialize
30 | # our roster.
31 | self.add_event_handler("session_start", self.start, threaded=True)
32 |
33 | def start(self, event):
34 |
35 | self.send_presence()
36 |
37 | self.get_roster()
38 |
39 | self.send_message(mto=self.recipient,
40 | mbody=self.msg,
41 | mtype='chat')
42 |
43 | # Using wait=True ensures that the send queue will be
44 | # emptied before ending the session.
45 | self.disconnect(wait=True)
46 |
--------------------------------------------------------------------------------
/facebook_sublet_group_bot/test.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | from util import notify_mac
4 | import unittest
5 |
6 | __author__ = 'Henri Sweers'
7 |
8 |
9 | class TestTagValidity(unittest.TestCase):
10 |
11 | def setUp(self):
12 | self.test_tags = ["Looking", "Rooming", "Offering", "Parking"]
13 | self.bad_tags = ["blah", "test", "sublease"]
14 | self.junk = """here's some other text because yeah more text to
15 | to illustrate lots more text here in the rest of the post"""
16 |
17 | def test_regular(self):
18 | from check_and_delete import get_tags
19 | for tag in self.test_tags:
20 | self.assertTrue(get_tags("(" + tag + ")"))
21 | self.assertTrue(get_tags("{" + tag + "}"))
22 | self.assertTrue(get_tags("[" + tag + "]"))
23 | self.assertTrue(get_tags("(" + tag + "]"))
24 | self.assertTrue(get_tags("{" + tag + "]"))
25 | self.assertTrue(get_tags("(" + tag + "}"))
26 | self.assertTrue(get_tags("{" + tag + ")"))
27 | self.assertFalse(get_tags(tag + ")"))
28 | self.assertFalse(get_tags("{" + tag))
29 | self.assertFalse(get_tags(tag))
30 |
31 | def test_real(self):
32 | from check_and_delete import get_tags
33 | from TestPosts import good_posts
34 | for post in good_posts:
35 | self.assertTrue(get_tags(post))
36 |
37 | def test_real_bad(self):
38 | from check_and_delete import get_tags
39 | from TestPosts import bad_posts
40 | for post in bad_posts:
41 | self.assertFalse(get_tags(post))
42 |
43 | def test_bad(self):
44 | from check_and_delete import get_tags
45 | for tag in self.bad_tags:
46 | self.assertFalse(get_tags("(" + tag + ")"))
47 | self.assertFalse(get_tags("{" + tag + "}"))
48 | self.assertFalse(get_tags("{" + tag + "}{looking}"))
49 | self.assertFalse(get_tags("[" + tag + "]"))
50 | self.assertFalse(get_tags("(" + tag + "]"))
51 | self.assertFalse(get_tags("{" + tag + "]"))
52 | self.assertFalse(get_tags("(" + tag + "}"))
53 | self.assertFalse(get_tags("{" + tag + ")"))
54 | self.assertFalse(get_tags(tag + ")"))
55 | self.assertFalse(get_tags("{" + tag))
56 | self.assertFalse(get_tags(tag))
57 |
58 | def test_misc(self):
59 | from check_and_delete import get_tags
60 | for tag in self.test_tags:
61 | self.assertTrue(get_tags("(" + tag + ") sometjunk"))
62 | self.assertFalse(get_tags("{" + tag + "}" + self.junk))
63 | self.assertFalse(get_tags("dsflkj{" + tag.lower() + ")"))
64 | self.assertFalse(get_tags("dsflkj {" + tag.lower() + ")"))
65 | self.assertTrue(get_tags("-(" + tag + ")"))
66 | self.assertTrue(get_tags("*(" + tag + ")"))
67 | self.assertTrue(get_tags("* (" + tag + ")"))
68 | self.assertTrue(get_tags(" (" + tag + ")"))
69 | self.assertTrue(get_tags("(" + tag + "):"))
70 | self.assertTrue(get_tags("(" + tag + ") :"))
71 | self.assertTrue(get_tags("(" + tag + ")(" + tag + ")"))
72 |
73 | def test_parking(self):
74 | from check_and_delete import check_for_parking_tag
75 | tag = "parking"
76 | self.assertTrue(check_for_parking_tag("(" + tag + ")"))
77 | self.assertTrue(check_for_parking_tag("{" + tag + "}"))
78 | self.assertTrue(check_for_parking_tag("[" + tag + "]"))
79 | self.assertTrue(check_for_parking_tag("(" + tag + "]"))
80 | self.assertTrue(check_for_parking_tag("{" + tag + "]"))
81 | self.assertTrue(check_for_parking_tag("(" + tag + "}"))
82 | self.assertTrue(check_for_parking_tag("{" + tag + ")"))
83 | self.assertFalse(check_for_parking_tag(tag + ")"))
84 | self.assertFalse(check_for_parking_tag("{" + tag))
85 | self.assertFalse(check_for_parking_tag(tag))
86 | self.assertTrue(check_for_parking_tag("(" + tag + ") sometjunk"))
87 | self.assertFalse(check_for_parking_tag("{" + tag + "}" + self.junk))
88 | self.assertFalse(check_for_parking_tag("dsflkj{" + tag.lower() + ")"))
89 | self.assertFalse(check_for_parking_tag("dsflkj {" + tag.lower() + ")"))
90 | self.assertTrue(check_for_parking_tag("-(" + tag + ")"))
91 | self.assertTrue(check_for_parking_tag("*(" + tag + ")"))
92 | self.assertTrue(check_for_parking_tag("* (" + tag + ")"))
93 | self.assertTrue(check_for_parking_tag(" (" + tag + ")"))
94 | self.assertTrue(check_for_parking_tag("(" + tag + "):"))
95 | self.assertTrue(check_for_parking_tag("(" + tag + ") :"))
96 |
97 | def test_rooming_and_offering(self):
98 | from check_and_delete import get_tags, validate_tags
99 | self.assertFalse(validate_tags(get_tags("[rooming][offering]")))
100 | self.assertFalse(validate_tags(get_tags("[rooming] [offering]")))
101 | self.assertFalse(validate_tags(get_tags("[offering][rooming]")))
102 | self.assertFalse(validate_tags(get_tags("[offering] [rooming]")))
103 | self.assertFalse(validate_tags(get_tags("[offering] [rooming] [parking]")))
104 | self.assertTrue(validate_tags(get_tags("[offering] [parking]")))
105 | self.assertTrue(validate_tags(get_tags("[rooming] [looking]")))
106 | self.assertTrue(validate_tags(get_tags("[rooming]")))
107 | self.assertTrue(validate_tags(get_tags("[parking]")))
108 | self.assertTrue(validate_tags(get_tags("[offering]")))
109 | self.assertTrue(validate_tags(get_tags("[looking]")))
110 |
111 |
112 | class TestPriceValidity(unittest.TestCase):
113 | def setUp(self):
114 | self.junk = """here's some other text because yeah more text to
115 | to illustrate lots more text here in the rest of the post"""
116 |
117 | def test_regular_month(self):
118 | from check_and_delete import check_price_validity
119 | self.assertTrue(check_price_validity("Blah blah $ blah"))
120 | self.assertTrue(check_price_validity("Blah blah 300 per month"))
121 | self.assertTrue(check_price_validity("Blah blah 300/month"))
122 | self.assertTrue(check_price_validity("Blah blah 300 / month"))
123 | self.assertTrue(check_price_validity("Blah blah 300 /month"))
124 | self.assertTrue(check_price_validity("Blah blah 300/ month"))
125 | self.assertTrue(check_price_validity("Blah blah 300 per/month"))
126 | self.assertTrue(check_price_validity("Blah blah 300 a month"))
127 | self.assertTrue(check_price_validity("Blah blah 300 a/month"))
128 |
129 | def test_embedded_month(self):
130 | from check_and_delete import check_price_validity
131 | self.assertTrue(check_price_validity("Blah $ blah" + self.junk))
132 | self.assertTrue(check_price_validity("Blah 300 per month" + self.junk))
133 | self.assertTrue(check_price_validity("Blah 300/month" + self.junk))
134 | self.assertTrue(check_price_validity("Blah 300 / month" + self.junk))
135 | self.assertTrue(check_price_validity("Blah 300 /month" + self.junk))
136 | self.assertTrue(check_price_validity("Blah 300/ month" + self.junk))
137 | self.assertTrue(check_price_validity("Blah 300 per/month" + self.junk))
138 | self.assertTrue(check_price_validity("Blah 300 a month" + self.junk))
139 | self.assertTrue(check_price_validity("Blah 300 a/month" + self.junk))
140 |
141 | def test_misc_month(self):
142 | from check_and_delete import check_price_validity
143 | self.assertTrue(check_price_validity("Blah$ blah" + self.junk))
144 | self.assertTrue(check_price_validity("Blah300 per month" + self.junk))
145 | self.assertTrue(check_price_validity("Blah blah 300permonth"))
146 | self.assertTrue(check_price_validity("Blah blah300/monthblah"))
147 | self.assertTrue(check_price_validity("Blah blah 300amonth"))
148 |
149 | def test_regular_mon(self):
150 | from check_and_delete import check_price_validity
151 | self.assertTrue(check_price_validity("Blah blah $ blah"))
152 | self.assertTrue(check_price_validity("Blah blah 300 per mon"))
153 | self.assertTrue(check_price_validity("Blah blah 300/mon"))
154 | self.assertTrue(check_price_validity("Blah blah 300 / mon"))
155 | self.assertTrue(check_price_validity("Blah blah 300 /mon"))
156 | self.assertTrue(check_price_validity("Blah blah 300/ mon"))
157 | self.assertTrue(check_price_validity("Blah blah 300 per/mon"))
158 | self.assertTrue(check_price_validity("Blah blah 300 a mon"))
159 | self.assertTrue(check_price_validity("Blah blah 300 a/mon"))
160 |
161 | def test_embedded_mon(self):
162 | from check_and_delete import check_price_validity
163 | self.assertTrue(check_price_validity("Blah $ blah" + self.junk))
164 | self.assertTrue(check_price_validity("Blah 300 per mon" + self.junk))
165 | self.assertTrue(check_price_validity("Blah 300/mon" + self.junk))
166 | self.assertTrue(check_price_validity("Blah 300 / mon" + self.junk))
167 | self.assertTrue(check_price_validity("Blah 300 /mon" + self.junk))
168 | self.assertTrue(check_price_validity("Blah 300/ mon" + self.junk))
169 | self.assertTrue(check_price_validity("Blah 300 per/mon" + self.junk))
170 | self.assertTrue(check_price_validity("Blah 300 a mon" + self.junk))
171 | self.assertTrue(check_price_validity("Blah 300 a/mon" + self.junk))
172 |
173 | def test_misc_mon(self):
174 | from check_and_delete import check_price_validity
175 | self.assertTrue(check_price_validity("Blah$ blah" + self.junk))
176 | self.assertTrue(check_price_validity("Blah300 per mon" + self.junk))
177 | self.assertTrue(check_price_validity("Blah blah 300permon"))
178 | self.assertTrue(check_price_validity("Blah blah300/monblah"))
179 | self.assertTrue(check_price_validity("Blah blah 300amon"))
180 |
181 | def test_regular_mo(self):
182 | from check_and_delete import check_price_validity
183 | self.assertTrue(check_price_validity("Blah blah $ blah"))
184 | self.assertTrue(check_price_validity("Blah blah 300 per mo"))
185 | self.assertTrue(check_price_validity("Blah blah 300/mo"))
186 | self.assertTrue(check_price_validity("Blah blah 300 / mo"))
187 | self.assertTrue(check_price_validity("Blah blah 300 /mo"))
188 | self.assertTrue(check_price_validity("Blah blah 300/ mo"))
189 | self.assertTrue(check_price_validity("Blah blah 300 per/mo"))
190 | self.assertTrue(check_price_validity("Blah blah 300 a mo"))
191 | self.assertTrue(check_price_validity("Blah blah 300 a/mo"))
192 |
193 | def test_embedded_mo(self):
194 | from check_and_delete import check_price_validity
195 | self.assertTrue(check_price_validity("Blah $ blah" + self.junk))
196 | self.assertTrue(check_price_validity("Blah 300 per mo" + self.junk))
197 | self.assertTrue(check_price_validity("Blah 300/mo" + self.junk))
198 | self.assertTrue(check_price_validity("Blah 300 / mo" + self.junk))
199 | self.assertTrue(check_price_validity("Blah 300 /mo" + self.junk))
200 | self.assertTrue(check_price_validity("Blah 300/ mo" + self.junk))
201 | self.assertTrue(check_price_validity("Blah 300 per/mo" + self.junk))
202 | self.assertTrue(check_price_validity("Blah 300 a mo" + self.junk))
203 | self.assertTrue(check_price_validity("Blah 300 a/mo" + self.junk))
204 |
205 | def test_misc_mo(self):
206 | from check_and_delete import check_price_validity
207 | self.assertTrue(check_price_validity("Blah$ blah" + self.junk))
208 | self.assertTrue(check_price_validity("Blah300 per mo" + self.junk))
209 | self.assertTrue(check_price_validity("Blah blah 300permo"))
210 | self.assertTrue(check_price_validity("Blah blah300/moblah"))
211 | self.assertTrue(check_price_validity("Blah blah 300amo"))
212 |
213 |
214 | class TestDeletion(unittest.TestCase):
215 |
216 | def test_regular(self):
217 | import delete_test
218 |
219 | # Suppress stdout printing
220 | _stdout = sys.stdout
221 | null = open(os.devnull, 'wb')
222 | sys.stdout = null
223 |
224 | # Run test
225 | self.assertTrue(delete_test.test())
226 |
227 | # Restore stdout printing
228 | sys.stdout = _stdout
229 |
230 |
231 | def main():
232 | print "-----------------"
233 | print "| Running tests |"
234 | print "-----------------"
235 | unittest.main()
236 | notify_mac()
237 |
238 |
239 | if __name__ == '__main__':
240 | main()
--------------------------------------------------------------------------------
/facebook_sublet_group_bot/util.py:
--------------------------------------------------------------------------------
1 | import sys
2 | import subprocess
3 |
4 |
5 | # Color class, used for colors in terminal
6 | class Color:
7 | PURPLE = '\033[95m'
8 | CYAN = '\033[96m'
9 | DARKCYAN = '\033[36m'
10 | BLUE = '\033[94m'
11 | GREEN = '\033[92m'
12 | YELLOW = '\033[93m'
13 | RED = '\033[91m'
14 | BOLD = '\033[1m'
15 | UNDERLINE = '\033[4m'
16 | END = '\033[0m'
17 |
18 |
19 | # Log method. If there's a color argument, it'll stick that in first
20 | def log(message, *colorargs):
21 | if len(colorargs) > 0:
22 | print colorargs[0] + message + Color.END
23 | else:
24 | print message
25 |
26 |
27 | # Nifty method for sending notifications on my mac when it's done
28 | def notify_mac():
29 | if sys.platform == "darwin":
30 | try:
31 | subprocess.call(
32 | ["terminal-notifier", "-message", "Tests done", "-title",
33 | "FB_Bot", "-sound", "default"])
34 | except OSError:
35 | print "If you have terminal-notifier, this would be a notification"
36 |
37 |
38 | # Reads in multi-line strings based on a certain "newline" type
39 | # Borrowed from here: http://stackoverflow.com/a/16260159/3034339
40 | def read_lines(f, newline):
41 | buf = ""
42 | while True:
43 | while newline in buf:
44 | pos = buf.index(newline)
45 | yield buf[:pos]
46 | buf = buf[pos + len(newline):]
47 | chunk = f.read(4096)
48 | if not chunk:
49 | yield buf
50 | break
51 | buf += chunk
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | sleekxmpp==1.1.11
2 | python-binary-memcached==0.21
3 | facepy==1.0.3
--------------------------------------------------------------------------------
/runtime.txt:
--------------------------------------------------------------------------------
1 | python-2.7.6
--------------------------------------------------------------------------------