├── .gitignore ├── LICENSE ├── README.md ├── comment_logger.py ├── comment_logger_srt.py ├── config.txt ├── follow_updater.py ├── irc_bot.py ├── log_all.py ├── log_selected.py └── recorded_channels.txt /.gitignore: -------------------------------------------------------------------------------- 1 | __pycache__/* 2 | comment_*/* 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 dekuNukem 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | A lightweight Twitch chat logger that logs all channels and group chats you follow on Twitch, or just a select few from a list that you provide. 2 | 3 | # Features 4 | 5 | * Log all your followed Twitch channels and group chats 6 | 7 | * Or just log from a list of channels that you specify 8 | 9 | * Automatically select chat server (regular, event, group) 10 | 11 | * Option to log chat metadata (user color, sub notification, mod events, etc.) 12 | 13 | * Configurable timestamp formats 14 | 15 | * Low CPU and memory usage 16 | 17 | * Tested on Windows 7 64-bit, OS X Yosemite, and Raspbian on Raspberry Pi 18 | 19 | # Requirements 20 | 21 | Only Python 3.2+ is needed. 22 | 23 | Download and install the latest version at https://www.python.org/downloads/ 24 | 25 | Raspbian on Raspberry Pi already has Python 3.2 so it should work out of the box. 26 | 27 | # How To Use 28 | 29 | ## 1. Set up your configuration file 30 | 31 | * Open config.txt in your favorite text editor 32 | 33 | * Add your own twitch username and oauth. You can get your oauth at http://twitchapps.com/tmi/ 34 | 35 | ![alt tag](http://i.imgur.com/467b7sb.png) 36 | 37 | * Take a look at other settings and change them if you want. 38 | 39 | ![alt tag](http://i.imgur.com/o76oDfk.png) 40 | 41 | ## 2. Start logging 42 | 43 | 44 | **Warning: Do not use Cygwin for this program in Windows. It doesn't pass keyboard interrupt correctly and will cause this program to terminate without killing its spawned loggers. Use the built-in cmd.exe instead.** 45 | 46 | 47 | ### Option 1: Log all channels and group chats you follow on Twitch 48 | 49 | Run `python3 log_all.py` in your terminal. 50 | 51 | The program will search for all your followed channels and group chats you're in and add loggers for each of them. It might take a while depending on the number of channels you follow. 52 | 53 | It will also add/remove loggers automatically as you follow/unfollow channels on Twitch. 54 | 55 | ![alt tag](http://i.imgur.com/Z3jmhEC.png) 56 | 57 | Once logging starts you should see the text files of your followed channels in comment_log folder 58 | 59 | ![alt tag](http://i.imgur.com/GLzM6nk.png) 60 | 61 | Inside of which are your intellectual and informative twitch chat logs 62 | 63 | ![alt tag](http://i.imgur.com/GGHD6O6.png) 64 | 65 | To stop the program press Control + C. 66 | 67 | ### Option 2: Only log channels from a list you provide. 68 | 69 | Open recorded_channels.txt and add channels you wish to log, one channel per line. 70 | 71 | ![alt tag](http://i.imgur.com/vzkTpgQ.png) 72 | 73 | Run `python3 log_selected.py` in your terminal. 74 | 75 | The program will verify each channel to make sure it exists and determine its type, then launch chat logger for each of them. 76 | 77 | ![alt tag](http://i.imgur.com/GVF9u7M.png) 78 | 79 | The rest is the same with Option 1 80 | 81 | # Resource Usage 82 | 83 | This program should consume minimum amount of memory and CPU while running. 84 | 85 | While logging 82 channels on Raspberry Pi 2 with 1GB memory and 900MHz ARM Cortex-A7 CPU, the program uses around 4.5MB of memory per logger and almost no CPU usage at all, with load average below 0.05 86 | 87 | The resource impact should be even less when running on a full-sized PC. 88 | 89 | ![alt tag](http://i.imgur.com/c1lN5uJ.png) 90 | 91 | I do recommend using a Raspberry Pi 2 as a logging machine and let the program run 24/7, this way you have a complete log of twitch chat, and avoid the cost of running a full-sized PC. -------------------------------------------------------------------------------- /comment_logger.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | from datetime import datetime 5 | import socket 6 | import irc_bot 7 | import configparser 8 | 9 | import pysubs2 10 | 11 | def iso8601_utc_now(): 12 | return datetime.utcnow().isoformat(sep='T') + "Z" 13 | 14 | def make_offset_str(offset_hours): 15 | offset_hours = int(offset_hours) 16 | if offset_hours == 0: 17 | return "Z" 18 | if offset_hours > 0: 19 | sign = "+" 20 | else: 21 | sign = "-" 22 | offset_str = str(abs(offset_hours)) 23 | if len(offset_str) < 2: 24 | offset_str = "0" + offset_str 25 | return sign + offset_str + ":00" 26 | 27 | def iso8601_local_now(): 28 | return datetime.now().isoformat(sep='T') + make_offset_str(utc_offset_hours) 29 | 30 | def parse_chat_server(chat_server): 31 | return chat_server.replace(' ', '').split(':') 32 | 33 | def ensure_dir(dir_path): 34 | if not os.path.exists(dir_path): 35 | print("creating directory " + dir_path) 36 | os.makedirs(dir_path) 37 | 38 | def log_add(path, content): 39 | with open(path, mode='a', encoding='utf-8') as log_file: 40 | log_file.write(content) 41 | 42 | def safe_print(content): 43 | try: 44 | print(content) 45 | except UnicodeEncodeError: 46 | print(content.encode('utf-8')) 47 | 48 | def get_timestamp(ts_format): 49 | if ts_format == 0: 50 | return str(time.time())[:15] 51 | elif ts_format == 2: 52 | return iso8601_local_now() 53 | else: 54 | return iso8601_utc_now() 55 | 56 | if(len(sys.argv) != 3): 57 | print(__file__ + ' channel server_type') 58 | sys.exit(0) 59 | 60 | current_directory = os.path.dirname(os.path.abspath(__file__)) 61 | config_path = current_directory + "/config.txt" 62 | if os.path.isfile(config_path): 63 | config = configparser.ConfigParser() 64 | config.read(config_path) 65 | username = config.get('Settings', 'username').replace(' ', '').lower() 66 | oauth = config.get('Settings', 'oauth') 67 | record_raw = config.getboolean('Settings', 'record_raw') 68 | timestamp_format = config.getint('Settings', 'timestamp_format') 69 | twitchclient_version = config.getint('Settings', 'twitchclient_version') 70 | regular_chat_server = config.get('Settings', 'regular_chat_server') 71 | group_chat_server = config.get('Settings', 'group_chat_server') 72 | event_chat_server = config.get('Settings', 'event_chat_server') 73 | else: 74 | print("config.txt not found", file=sys.stderr) 75 | sys.exit(0) 76 | 77 | ts = time.time() 78 | utc_offset_hours = int(int((datetime.fromtimestamp(ts) - datetime.utcfromtimestamp(ts)).total_seconds()) / 3600) 79 | 80 | server_dict = {'r':parse_chat_server(regular_chat_server), 'g':parse_chat_server(group_chat_server), 'e':parse_chat_server(event_chat_server)} 81 | chat_channel = sys.argv[1] 82 | chat_server = server_dict[sys.argv[2].lower()] 83 | 84 | ensure_dir(current_directory + '/comment_log') 85 | if record_raw: 86 | ensure_dir(current_directory + '/comment_log_raw') 87 | 88 | raw_log_path = current_directory + '/comment_log_raw/' + chat_channel + '.txt' 89 | log_path = current_directory + '/comment_log/' + chat_channel + '.txt' 90 | 91 | subs_log_path = current_directory + '/comment_log/' + chat_channel + '.ass' 92 | 93 | bot = irc_bot.irc_bot(username, oauth, chat_channel, chat_server[0], chat_server[1], twitchclient_version = twitchclient_version) 94 | 95 | subs = pysubs2.SSAFile() 96 | i = 0 97 | 98 | text = '' 99 | 100 | while 1: 101 | raw_msg_list = bot.get_message() 102 | if len(raw_msg_list) > 0: 103 | if len(text) > 0: 104 | end = pysubs2.time.make_time(ms=datetime.now().microsecond) 105 | subs.insert(i, pysubs2.SSAEvent(start=start, end=end, text=text.replace('\\', '\\\\'))) 106 | i = i + 1 107 | start = pysubs2.time.make_time(ms=datetime.now().microsecond) 108 | text = '' 109 | timestamp = get_timestamp(timestamp_format) 110 | for item in raw_msg_list: 111 | if record_raw: 112 | log_add(raw_log_path, timestamp + ' ' + item + '\n') 113 | username, message = irc_bot.parse_user(item) 114 | if username != '': 115 | safe_print(chat_channel + " " + username + ": " + message) 116 | log_add(log_path, timestamp + ' ' + username + ': ' + message + '\n') 117 | text += username + ": " + message + '\n' 118 | subs.save(path=subs_log_path, encoding='utf-8') 119 | 120 | 121 | -------------------------------------------------------------------------------- /comment_logger_srt.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | from datetime import datetime 5 | import socket 6 | import irc_bot 7 | import configparser 8 | 9 | from pysrt import SubRipFile 10 | from pysrt import SubRipItem 11 | from pysrt import SubRipTime 12 | 13 | def iso8601_utc_now(): 14 | return datetime.utcnow().isoformat(sep='T') + "Z" 15 | 16 | def make_offset_str(offset_hours): 17 | offset_hours = int(offset_hours) 18 | if offset_hours == 0: 19 | return "Z" 20 | if offset_hours > 0: 21 | sign = "+" 22 | else: 23 | sign = "-" 24 | offset_str = str(abs(offset_hours)) 25 | if len(offset_str) < 2: 26 | offset_str = "0" + offset_str 27 | return sign + offset_str + ":00" 28 | 29 | def iso8601_local_now(): 30 | return datetime.now().isoformat(sep='T') + make_offset_str(utc_offset_hours) 31 | 32 | def parse_chat_server(chat_server): 33 | return chat_server.replace(' ', '').split(':') 34 | 35 | def ensure_dir(dir_path): 36 | if not os.path.exists(dir_path): 37 | print("creating directory " + dir_path) 38 | os.makedirs(dir_path) 39 | 40 | def log_add(path, content): 41 | with open(path, mode='a', encoding='utf-8') as log_file: 42 | log_file.write(content) 43 | 44 | def safe_print(content): 45 | try: 46 | print(content) 47 | except UnicodeEncodeError: 48 | print(content.encode('utf-8')) 49 | 50 | def get_timestamp(ts_format): 51 | if ts_format == 0: 52 | return str(time.time())[:15] 53 | elif ts_format == 2: 54 | return iso8601_local_now() 55 | else: 56 | return iso8601_utc_now() 57 | 58 | if(len(sys.argv) != 3): 59 | print(__file__ + ' channel server_type') 60 | sys.exit(0) 61 | 62 | current_directory = os.path.dirname(os.path.abspath(__file__)) 63 | config_path = current_directory + "/config.txt" 64 | if os.path.isfile(config_path): 65 | config = configparser.ConfigParser() 66 | config.read(config_path) 67 | username = config.get('Settings', 'username').replace(' ', '').lower() 68 | oauth = config.get('Settings', 'oauth') 69 | record_raw = config.getboolean('Settings', 'record_raw') 70 | timestamp_format = config.getint('Settings', 'timestamp_format') 71 | twitchclient_version = config.getint('Settings', 'twitchclient_version') 72 | regular_chat_server = config.get('Settings', 'regular_chat_server') 73 | group_chat_server = config.get('Settings', 'group_chat_server') 74 | event_chat_server = config.get('Settings', 'event_chat_server') 75 | else: 76 | print("config.txt not found", file=sys.stderr) 77 | sys.exit(0) 78 | 79 | ts = time.time() 80 | utc_offset_hours = int(int((datetime.fromtimestamp(ts) - datetime.utcfromtimestamp(ts)).total_seconds()) / 3600) 81 | 82 | server_dict = {'r':parse_chat_server(regular_chat_server), 'g':parse_chat_server(group_chat_server), 'e':parse_chat_server(event_chat_server)} 83 | chat_channel = sys.argv[1] 84 | chat_server = server_dict[sys.argv[2].lower()] 85 | 86 | ensure_dir(current_directory + '/comment_log') 87 | if record_raw: 88 | ensure_dir(current_directory + '/comment_log_raw') 89 | 90 | raw_log_path = current_directory + '/comment_log_raw/' + chat_channel + '.txt' 91 | log_path = current_directory + '/comment_log/' + chat_channel + '.txt' 92 | 93 | srt_log_path = current_directory + '/comment_log/' + chat_channel + '.srt' 94 | 95 | bot = irc_bot.irc_bot(username, oauth, chat_channel, chat_server[0], chat_server[1], twitchclient_version = twitchclient_version) 96 | 97 | outsrt = SubRipFile() 98 | 99 | text = '' 100 | 101 | while 1: 102 | raw_msg_list = bot.get_message() 103 | if len(raw_msg_list) > 0: 104 | if len(text) > 0: 105 | end = SubRipTime.from_time(datetime.now()) 106 | item = SubRipItem(0, start, end, text) 107 | outsrt.append(item) 108 | start = SubRipTime.from_time(datetime.now()) 109 | text = '' 110 | timestamp = get_timestamp(timestamp_format) 111 | for item in raw_msg_list: 112 | if record_raw: 113 | log_add(raw_log_path, timestamp + ' ' + item + '\n') 114 | username, message = irc_bot.parse_user(item) 115 | if username != '': 116 | safe_print(chat_channel + " " + username + ": " + message) 117 | log_add(log_path, timestamp + ' ' + username + ': ' + message + '\n') 118 | text += username + ": " + message + '\n' 119 | outsrt.clean_indexes() 120 | outsrt.save(srt_log_path, encoding='utf-8') 121 | 122 | 123 | -------------------------------------------------------------------------------- /config.txt: -------------------------------------------------------------------------------- 1 | [Settings] 2 | 3 | ; Your twitch username 4 | username = gamedeff 5 | 6 | ; Your twitch oauth, grab yours at www.twitchapps.com/tmi 7 | oauth = oauth:j26lib90fnz4cf9exslhe9yp8dg6nt 8 | 9 | ; Change this to 1 to save additional chat information 10 | ; including message color, user identity(mod/sub/staff), 11 | ; sub notification, timeout/submode/clearchat/slowmode 12 | ; notifications, etc. 13 | record_raw = 1 14 | 15 | ; set it to 1 to log your own channel 16 | log_self = 0 17 | 18 | ; what kind of timestamp to use 19 | ; 0 for unix timestamp: 1427139635.0058 20 | ; 1 for human readable timestamp in UTC: 2015-03-23T19:40:35Z 21 | ; 2 for human readable timestamp in your local time: 2015-03-23T14:40:35-05:00 22 | timestamp_format = 1 23 | 24 | ; Version of TWITCHCLIENT request 25 | ; leave this alone if you don't know what it is 26 | twitchclient_version = 0 27 | 28 | ; Chat server addresses 29 | ; those are the ones that works for me 30 | ; you can also change it based on twitchstatus.com/#chat 31 | regular_chat_server = irc.twitch.tv:6667 32 | group_chat_server = 199.9.253.119:80 33 | event_chat_server = 192.16.64.143:80 -------------------------------------------------------------------------------- /follow_updater.py: -------------------------------------------------------------------------------- 1 | """ 2 | This script tracks the changes of user's followed channel and group chat 3 | and store them into cache files for other program to use 4 | """ 5 | 6 | import time 7 | import os 8 | import json 9 | import pickle 10 | import configparser 11 | import sys 12 | from urllib.request import urlopen 13 | from copy import deepcopy 14 | 15 | # create a folder if one doesn't already exist 16 | def ensure_dir(dir_path): 17 | if not os.path.exists(dir_path): 18 | print("creating directory " + dir_path) 19 | os.makedirs(dir_path) 20 | 21 | def current_time(): 22 | return int(time.time()) 23 | 24 | def is_event_chat(channel): 25 | url = "http://api.twitch.tv/api/channels/" + channel + "/chat_properties" 26 | try: 27 | info = json.loads(urlopen(url, timeout = 5).read().decode('utf-8')) 28 | return info["eventchat"] 29 | except Exception as e: 30 | print("Exception in is_event_chat: " + str(type(e)) + ": " + str(e), file=sys.stderr) 31 | return False 32 | 33 | def cache_exists(path): 34 | return os.path.isfile(path) 35 | 36 | def load_cache(path): 37 | with open(path, 'rb') as fp: 38 | return pickle.load(fp) 39 | 40 | def dump_cache(item, path): 41 | with open(path, 'wb') as fp: 42 | pickle.dump(item, fp) 43 | 44 | # save the type of channel to local cache 45 | # r for regular chat, e for event chat 46 | # g for group chat, but it's done in group_chat_lookup() below 47 | def update_channel_type_cache(channel_name): 48 | global channel_type_dict 49 | channel_type = 'r' 50 | if is_event_chat(channel_name): 51 | channel_type = 'e' 52 | print("channel " + channel_name + " is: " + channel_type) 53 | channel_type_dict[channel_name] = channel_type 54 | dump_cache(channel_type_dict, ctd_path) 55 | time.sleep(0.5) 56 | 57 | # add a new channel to the list of followed channels 58 | def followed_channels_list_add(channel_name): 59 | global followed_channels_list 60 | if channel_name not in followed_channels_list: 61 | print("new followed channel: " + channel_name) 62 | followed_channels_list.append(channel_name) 63 | dump_cache(followed_channels_list, fcl_path) 64 | 65 | # add newly followed channels to local cache 66 | # this stops as soon as all the newly followed channels 67 | # are added, to save time and load on twitch server 68 | def followed_lookup(username): 69 | # ask twitch for a list of user's followed channels, sorted by most recently followed first 70 | url = "https://api.twitch.tv/kraken/users/" + username + "/follows/channels?limit=100&sortby=created_at&direction=desc" 71 | info = json.loads(urlopen(url, timeout = 15).read().decode('utf-8')) 72 | total_followed = info["_total"] 73 | while 1: 74 | for item in info["follows"]: 75 | # going through the name of followed channels we got from twitch 76 | # from most recently followed to least recently followed 77 | channel_name = item['channel']['name'].replace(' ', '').lower() 78 | # we encounter a channel that's already in the local cache, we know we can 79 | # stop since the rest will be in cache as well 80 | if channel_name in followed_channels_list: 81 | return 82 | # if a channel is not in local cache, it's newly followed so we add it to local cache 83 | followed_channels_list_add(channel_name) 84 | update_channel_type_cache(channel_name) 85 | next_page = info["_links"]["next"] 86 | next_offset = int(next_page.split("offset=")[1].split("&")[0]) 87 | if next_offset > total_followed: 88 | break 89 | info = json.loads(urlopen(next_page, timeout = 15).read().decode('utf-8')) 90 | time.sleep(1) 91 | 92 | # but how to detect unfollowed channels? 93 | # we do a complete fresh fetch to flush out those unfollowed channels. 94 | def followed_lookup_flush(username): 95 | global followed_channels_list 96 | print("doing a fresh fetch") 97 | current_followed = [] 98 | url = "https://api.twitch.tv/kraken/users/" + username + "/follows/channels?limit=100&sortby=created_at&direction=desc" 99 | info = json.loads(urlopen(url, timeout = 15).read().decode('utf-8')) 100 | total_followed = info["_total"] 101 | while 1: 102 | for item in info["follows"]: 103 | channel_name = item['channel']['name'].replace(' ', '').lower() 104 | if channel_name not in channel_type_dict: 105 | update_channel_type_cache(channel_name) 106 | print(channel_name) 107 | current_followed.append(channel_name) 108 | next_page = info["_links"]["next"] 109 | next_offset = int(next_page.split("offset=")[1].split("&")[0]) 110 | if next_offset > total_followed: 111 | break 112 | info = json.loads(urlopen(next_page, timeout = 15).read().decode('utf-8')) 113 | time.sleep(1) 114 | del followed_channels_list 115 | followed_channels_list = current_followed 116 | dump_cache(followed_channels_list, fcl_path) 117 | 118 | # get user's group chat so they can be logged as well 119 | def group_chat_lookup(): 120 | global channel_type_dict 121 | url = "https://chatdepot.twitch.tv/room_memberships?oauth_token=" + oauth.lower().replace("oauth:", '') 122 | info = json.loads(urlopen(url, timeout = 15).read().decode('utf-8')) 123 | for item in info["memberships"]: 124 | is_confirmed = item["is_confirmed"] 125 | irc_channel = item["room"]["irc_channel"] 126 | if is_confirmed: 127 | followed_channels_list_add(irc_channel) 128 | channel_type_dict[irc_channel] = 'g' 129 | 130 | # get an update on followers and group chats 131 | def follow_update(username, flush = False): 132 | try: 133 | if flush: 134 | followed_lookup_flush(username) 135 | else: 136 | followed_lookup(username) 137 | group_chat_lookup() 138 | except Exception as e: 139 | print("Exception updating followed channels: " + str(type(e)) + ": " + str(e), file=sys.stderr) 140 | return 141 | 142 | # make sure the two caches are in sync 143 | def cache_coherence_check(): 144 | global channel_type_dict 145 | global followed_channels_list 146 | # if the type of a channel is known but it's not followed, remove it 147 | temp = deepcopy(channel_type_dict) 148 | for key in temp: 149 | if key not in followed_channels_list: 150 | print("removed " + key + " from channel type dict") 151 | channel_type_dict.pop(key) 152 | # if a followed channel doesn't have a type, add it to the cache 153 | for item in followed_channels_list: 154 | if item not in channel_type_dict: 155 | update_channel_type_cache(item) 156 | dump_cache(channel_type_dict, ctd_path) 157 | 158 | # read config file 159 | current_directory = os.path.dirname(os.path.abspath(__file__)) 160 | config_path = current_directory + "/config.txt" 161 | if os.path.isfile(config_path): 162 | config = configparser.ConfigParser() 163 | config.read(config_path) 164 | oauth = config.get('Settings', 'oauth') 165 | twitch_username = config.get('Settings', 'username').replace(' ', '').lower() 166 | else: 167 | print("config.txt not found") 168 | sys.exit(1) 169 | 170 | ensure_dir(current_directory + "/cache") 171 | one_hour = 3600 172 | followed_channels_list = [] 173 | fcl_path = current_directory + "/cache/" + twitch_username + "_followed_channels.p" 174 | channel_type_dict = {} 175 | ctd_path = current_directory + "/cache/" + twitch_username + "_channel_type.p" 176 | 177 | if cache_exists(fcl_path): 178 | followed_channels_list = load_cache(fcl_path) 179 | if cache_exists(ctd_path): 180 | channel_type_dict = load_cache(ctd_path) 181 | 182 | cache_coherence_check() 183 | last_flush = current_time() 184 | 185 | # update loop 186 | while 1: 187 | print("\nupdating on " + time.strftime("%Y/%m/%d") + ' ' + time.strftime("%H:%M:%S")) 188 | # check for unfollowed channels every 4 hours 189 | if current_time() - last_flush > 4 * one_hour: 190 | follow_update(twitch_username, flush = True) 191 | last_flush = current_time() 192 | else: 193 | # check for new followed channel every 2 minutes 194 | follow_update(twitch_username) 195 | cache_coherence_check() 196 | time.sleep(120) 197 | 198 | 199 | 200 | 201 | 202 | -------------------------------------------------------------------------------- /irc_bot.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import socket 5 | 6 | def parse_user(raw_msg): 7 | if 'PRIVMSG' in raw_msg and raw_msg.count(":") > 1: 8 | username = raw_msg.split(' ', 1)[0].split('!')[0].replace(':', '').lower() 9 | comment = raw_msg.split(' :')[-1] 10 | if username != 'jtv' and 'tmi.' not in username: 11 | return username, comment 12 | return '', '' 13 | 14 | class irc_bot(object): 15 | def __init__(self, nickname, oauth, channel, host, port, timeout = 600, twitchclient_version = 3): 16 | self.NICK = nickname 17 | self.AUTH = oauth 18 | self.CHAT_CHANNEL = channel 19 | self.HOST = host 20 | self.PORT = int(port) 21 | self.sock = socket.socket() 22 | self.timeout = timeout 23 | self.tc_version = int(twitchclient_version) 24 | self.is_connected = False 25 | 26 | def connect(self): 27 | del self.sock 28 | self.sock = socket.socket() 29 | self.sock.settimeout(5) 30 | self.sock.connect((self.HOST, self.PORT)) 31 | self.sock.send(bytes("PASS %s\r\n" % self.AUTH, "UTF-8")) 32 | self.sock.send(bytes("NICK %s\r\n" % self.NICK, "UTF-8")) 33 | self.sock.send(bytes("USER %s %s bla :%s\r\n" % (self.NICK, self.HOST, self.NICK), "UTF-8")) 34 | self.sock.send(bytes("JOIN #%s\r\n" % self.CHAT_CHANNEL, "UTF-8")); 35 | print(self.NICK + ": connected to " + self.CHAT_CHANNEL) 36 | if self.tc_version > 0: 37 | tc_message = "TWITCHCLIENT " + str(self.tc_version) + "\r\n" 38 | self.sock.send(bytes(tc_message, "UTF-8")) 39 | self.sock.settimeout(self.timeout) 40 | self.is_connected = True 41 | 42 | def update(self): 43 | if not self.is_connected: 44 | self.retry_connect() 45 | recv_buffer = [] 46 | raw_msg_list = self.sock.recv(1024).decode("UTF-8", errors = "ignore").split("\n") 47 | for item in [s for s in raw_msg_list if len(s) > 0]: 48 | item = item.replace('\r', '') 49 | if "PRIVMSG" not in item and 'tmi.twitch.tv' in item and 'PING' in item: 50 | self.sock.send(bytes("PONG tmi.twitch.tv\r\n", "UTF-8")) 51 | if "PRIVMSG" not in item and "tmi.twitch.tv" in item and "Login unsuccessful" in item: 52 | print(self.NICK + ": Login failed! check your username and oauth", file=sys.stderr) 53 | sys.exit(1) 54 | recv_buffer.insert(0, item) 55 | return recv_buffer 56 | 57 | def retry_connect(self): 58 | while 1: 59 | time.sleep(0.5) 60 | try: 61 | self.connect() 62 | except Exception as e: 63 | print("Exception while trying to connect: " + str(type(e)) + ": " + str(e), file=sys.stderr) 64 | time.sleep(2) 65 | continue 66 | break 67 | 68 | def get_message(self): 69 | while 1: 70 | try: 71 | return self.update() 72 | except Exception as e: 73 | print("Exception receiving messages: " + str(type(e)) + ": " + str(e), file=sys.stderr) 74 | self.retry_connect() 75 | 76 | def get_user_message(self): 77 | ret = [] 78 | for item in self.get_message(): 79 | username, comment = parse_user(item) 80 | if username != '': 81 | ret.append((username, comment)) 82 | return ret 83 | 84 | def send_message(self, message): 85 | try: 86 | self.sock.send(bytes("PRIVMSG #%s :%s\r\n" % (self.CHAT_CHANNEL, message), "UTF-8")) 87 | except Exception as e: 88 | print("Exception sending message: " + str(type(e)) + ": " + str(e), file=sys.stderr) 89 | self.retry_connect() -------------------------------------------------------------------------------- /log_all.py: -------------------------------------------------------------------------------- 1 | """ 2 | This script watches user's followed channels and group chats 3 | and spawns / kills comment loggers for each one 4 | """ 5 | import time 6 | import os 7 | import sys 8 | import pickle 9 | import subprocess 10 | import configparser 11 | from copy import deepcopy 12 | 13 | # kill the logger process of a channel 14 | def remove_logger(channel): 15 | global running_logger 16 | if channel in running_logger: 17 | print("killing logger for " + channel) 18 | running_logger[channel].kill() 19 | running_logger.pop(channel) 20 | 21 | # spawn a twitch chat logger for a channel 22 | def add_logger(channel): 23 | global running_logger 24 | remove_logger(channel) 25 | if channel not in running_logger: 26 | print("adding logger for " + channel) 27 | logger_path = current_directory + "/comment_logger.py" 28 | running_logger[channel] = subprocess.Popen([sys.executable,"-u", logger_path, channel, channel_type[channel]], stdout=devnull) 29 | time.sleep(0.5) 30 | 31 | # kill all running loggers then exit 32 | def stop(): 33 | print("killing child processes...", file=sys.stderr) 34 | temp = deepcopy(running_logger) 35 | for key in temp: 36 | remove_logger(key) 37 | print("killing follow_updater...") 38 | follow_updater.kill() 39 | sys.exit(0) 40 | 41 | current_directory = os.path.dirname(os.path.abspath(__file__)) 42 | config_path = current_directory + "/config.txt" 43 | if os.path.isfile(config_path): 44 | config = configparser.ConfigParser() 45 | config.read(config_path) 46 | twitch_username = config.get('Settings', 'username').replace(' ', '').lower() 47 | log_self = config.getboolean('Settings', 'log_self') 48 | else: 49 | print("config.txt not found") 50 | sys.exit(1) 51 | 52 | fcl_path = current_directory + "/cache/" + twitch_username + "_followed_channels.p" 53 | ctd_path = current_directory + "/cache/" + twitch_username + "_channel_type.p" 54 | 55 | followed_channels_prev = set() 56 | channel_type = {} 57 | running_logger = {} 58 | devnull = open(os.devnull, 'w') 59 | follow_updater = subprocess.Popen([sys.executable, "-u", current_directory + "/follow_updater.py"]) 60 | 61 | try: 62 | if log_self: 63 | channel_type[twitch_username] = 'r' 64 | add_logger(twitch_username) 65 | while 1: 66 | # grab an update of followed channels 67 | try: 68 | with open(fcl_path, 'rb') as fp: 69 | followed_list_curr = set(pickle.load(fp)) 70 | with open(ctd_path, 'rb') as fp: 71 | channel_type = pickle.load(fp) 72 | except Exception as e: 73 | print("pickle: " + str(e), file=sys.stderr) 74 | time.sleep(30) 75 | continue 76 | 77 | new_followed = followed_list_curr - followed_channels_prev 78 | unfollowed = followed_channels_prev - followed_list_curr 79 | followed_channels_prev = followed_list_curr 80 | 81 | # kill the loggers for unfollowed channels 82 | for item in unfollowed: 83 | remove_logger(item) 84 | 85 | # spawn loggers for newly followed channels 86 | for item in new_followed: 87 | add_logger(item) 88 | 89 | time.sleep(30) 90 | 91 | except KeyboardInterrupt: 92 | stop() 93 | 94 | -------------------------------------------------------------------------------- /log_selected.py: -------------------------------------------------------------------------------- 1 | import os 2 | import sys 3 | import time 4 | import json 5 | import subprocess 6 | from urllib.request import urlopen 7 | from urllib.error import HTTPError 8 | from copy import deepcopy 9 | 10 | def channel_type_check(channel): 11 | time.sleep(0.5) 12 | url = "http://api.twitch.tv/api/channels/" + channel + "/chat_properties" 13 | try: 14 | info = json.loads(urlopen(url, timeout = 5).read().decode('utf-8')) 15 | if info["eventchat"]: 16 | return "e" 17 | else: 18 | return "r" 19 | except HTTPError as e: 20 | if e.code == 404 and is_group_chat(channel): 21 | return "g" 22 | return "" 23 | 24 | def read_channels(): 25 | file_path = current_directory + "/recorded_channels.txt" 26 | if not os.path.isfile(file_path): 27 | print("no channel list file found") 28 | sys.exit(0) 29 | channel_list = [] 30 | with open(file_path) as channel_list_file: 31 | while 1: 32 | this_line = channel_list_file.readline() 33 | this_line = this_line.replace(' ', '').lower() 34 | if this_line == '': 35 | break 36 | if this_line[0] == ";": 37 | continue 38 | this_line = this_line.replace('\n', '') 39 | if len(this_line) > 0: 40 | channel_list.append(this_line) 41 | return channel_list 42 | 43 | def is_group_chat(name): 44 | return len(name) > 0 and name[0] == "_" and name.count("_") >= 2 and name[-13:].isnumeric() 45 | 46 | # kill the logger process of a channel 47 | def remove_logger(channel): 48 | global running_logger 49 | if channel in running_logger: 50 | print("killing logger for " + channel) 51 | running_logger[channel].kill() 52 | running_logger.pop(channel) 53 | 54 | # spawn a twitch chat logger for a channel 55 | def add_logger(channel, channel_type): 56 | global running_logger 57 | remove_logger(channel) 58 | if channel not in running_logger: 59 | print("spawning logger for " + channel) 60 | logger_path = current_directory + "/comment_logger.py" 61 | running_logger[channel] = subprocess.Popen([sys.executable, "-u", logger_path, channel, channel_type], stdout=devnull) 62 | time.sleep(0.5) 63 | 64 | # kill all running loggers then exit 65 | def stop(): 66 | print("killing child processes...", file=sys.stderr) 67 | temp = deepcopy(running_logger) 68 | for key in temp: 69 | remove_logger(key) 70 | sys.exit(0) 71 | 72 | current_directory = os.path.dirname(os.path.abspath(__file__)) 73 | if len(sys.argv) < 2: 74 | channel_list = read_channels() 75 | else: 76 | channel_list = sys.argv[1:] # get everything after the script name 77 | channel_type_dict = {} 78 | running_logger = {} 79 | devnull = open(os.devnull, 'w') 80 | 81 | print("\nverifying channels") 82 | for item in channel_list: 83 | print(item + " ", end = "") 84 | channel_type = channel_type_check(item) 85 | if channel_type == "": 86 | print(" doesn't exist, skipped") 87 | continue 88 | print("...ok") 89 | channel_type_dict[item] = channel_type 90 | print() 91 | 92 | try: 93 | for key in channel_type_dict: 94 | add_logger(key, channel_type_dict[key]) 95 | while 1: 96 | time.sleep(60) 97 | except KeyboardInterrupt: 98 | stop() 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | -------------------------------------------------------------------------------- /recorded_channels.txt: -------------------------------------------------------------------------------- 1 | ; add the channels you want to log 2 | ; one channel per line 3 | 4 | dansgaming 5 | zfg1 6 | nintendo 7 | hyuokjh 8 | _beefhash_1419320687531 9 | twitchplayspokemon 10 | witwix 11 | --------------------------------------------------------------------------------