├── .gitignore ├── LICENSE.md ├── README.md ├── activedays.py ├── activehours.py ├── activityovertime.py ├── examples ├── activityovertime_example.jpg ├── example_supergroup │ ├── Activity in example_supergroup.png │ ├── Most active users in example_supergroup.png │ ├── all_text.txt │ └── example_supergroup.jsonl ├── mostactiveusers_example.jpg ├── phraseovertime_example.png └── venn_example.jpg ├── getalltext.py ├── getalltextfromuser.py ├── getpinnedmessages.py ├── inactiveusers.py ├── listchatsinmemberlist.py ├── mostactiveusers.py ├── mostcommonphrases.py ├── phraseovertime.py ├── usersovertime.py ├── venn_chatlog.py └── venn_userlist.py /.gitignore: -------------------------------------------------------------------------------- 1 | json 2 | __pycache__ 3 | progress.json 4 | figures 5 | README.md.backup 6 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright (c) 2016 Tanuj Dhir 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 4 | 5 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 6 | 7 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 8 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | telegram-analysis: Analyse Telegram chat logs easily 2 | ===================================== 3 | 4 | A tool for working with the output of [telegram-history-dump](https://github.com/tvdstaaij/telegram-history-dump) 5 | 6 | - [Examples](#examples) 7 | - [Installation](#installation-linux) 8 | - [Usage Guide](#usage-guide) 9 | - [Donations](#donations) 10 | 11 | Examples 12 | --------------- 13 | `venn_userlist.py`: compare user overlap between chats 14 | ![Venn diagram example](/examples/venn_example.jpg?raw=true) 15 | `activityovertime.py`: compare activity in different chats through time 16 | ![Activity over time example chart](/examples/activityovertime_example.jpg?raw=true) 17 | `phraseovertime.py`: compare popularity of different phrases in a chat through time 18 | ![Phrase over time example chart](/examples/phraseovertime_example.png?raw=true) 19 | `mostactiveusers.py`: find who contributed the most to a chat 20 | ![Most active users example chart](/examples/mostactiveusers_example.jpg?raw=true) 21 | 22 | Installation (Linux) 23 | --------------- 24 | #### Getting data 25 | 26 | First, you need some data! To get this, you must install [telegram-cli](https://github.com/vysheng/tg) and [telegram-history-dump](https://github.com/tvdstaaij/telegram-history-dump). 27 | They both have install instructions but I'll give a short version here: 28 | 29 | - To install telegram-cli, first check your distro repos. If you're lucky, it'll be in there. If not, you'll have to compile from source. `git clone --recursive` the repo, install dependencies (listed in the readme), `./configure`, `make`, `sudo make install`. 30 | 31 | - To install telegram-history-dump, clone the repo, ensure your ruby is version 2+, and you should be set. 32 | 33 | Next, I suggest editing the telegram-history-dump config.yaml chat sections (near the top of the file) with the names of the chats you want to get chatlogs of to start with, and putting 'null' in any empty chat sections. As chatlogs can take a while to download, you might want to start with just a couple of them. It's full of explanatory comments, so this shouldn't be too difficult. The rest of the config has some sensible defaults, and it's probably not worth changing them at this point. Another thing to keep in mind when writing the config file is that if you're putting in chat names, make sure there are no commas or square brackets in them. (Also, consider using chat_ids instead of names, as names can change). 34 | 35 | Then, run `telegram-cli` with no commandline arguments and set it up with your account - just a case of putting in your phone number and an auth code. Once that's set up, run `telegram-cli --json -P 9009` and leave that terminal open. In another terminal, run the `telegram-history-dump.rb` script (it'll be in the folder where you cloned telegram-history-dump) and it'll start downloading your chatlogs. 36 | 37 | #### Installing telegram-analysis 38 | 39 | 1. Clone the repo. 40 | 2. Make sure you have Python 3 installed by running `python3`. If you don't have it, install Python 3 using your distro repos or the [official site](https://www.python.org/downloads/). 41 | 3. If you want to use any of the graphical scripts, you need matplotlib. This will probably be in your distro repos as `python-matplotlib`, but you can also install with pip or from source. If you need more guidance, check out the [official site](http://matplotlib.org/users/installing.html). 42 | 4. If you want to make venn diagrams, you need `matplotlib-venn`, which can be installed using pip. Check out the [github repo](https://github.com/konstantint/matplotlib-venn) for more information. 43 | 44 | Once you have these things, you should be able to run all the analysis scripts! 45 | 46 | Usage Guide 47 | --------------- 48 | To start with, I recommend putting your json chatlogs in a folder with the scripts, so that your /path/to/chatlog.jsonl won't be a mess of relative path shenanigans. You can do this with the dumper's config.yaml file or by copying the files (the first is better long-term in my opinion). 49 | 50 | I recommend using the -h/--help option on all the scripts rather than reading this quick run-through of the scripts, as the help text will be more detailed and correct for your version. 51 | ______ 52 | 53 | - Get all the text from a chat and print to standard output (one line per message): `./getalltext.py /path/to/chatlog.jsonl` 54 | 55 | - Get all the text in a chat by a particular user and print to standard output (one line per message): ` ./getalltextfromuser.py /path/to/chatlog.jsonl username_without_at_sign ` 56 | 57 | - Get all the text from a chat, and dump it into a text file: `./getalltext.py /path/to/chatlog.jsonl > somefile.txt` 58 | 59 | - You can combine a text-dumping with `mostcommonphrases.py` to get a list of the most commonly sent messages and their frequencies. For example, find what messages a user sends most often in a particular chat: `./getalltextfromuser.py /path/to/chatlog.jsonl username_without_at_sign | ./mostcommonphrases.py` 60 | `[['lol', 110], ['hmm', 68], ['hey', 23], etc etc]` 61 | 62 | - Get a pie chart of the most active users in a chat: `./mostactiveusers.py -f /path/to/chatlog.jsonl` 63 | 64 | - Get a graph of the usage of a particular phrase or phrases in a chat over time: `./phraseovertime.py -f /path/to/chatlog.jsonl -p "phrase1" "phrase2"` 65 | 66 | - Get a graph of the activity levels of a chat or chats over time: 67 | `./activityovertime.py -f /path/to/chatlog1.jsonl /path/to/chatlog2.jsonl` 68 | 69 | - Same as above, but instead of opening a window with the graph, save the graph as an image in a folder: `./activityovertime.py -o /output/folder/ -f /path/to/chatlog1.jsonl /path/to/chatlog2.jsonl` 70 | 71 | - The same -o or --output-folder argument can be passed to `activityovertime`, `phraseovertime`, and `mostactiveusers`. This allows, for example, scripting these so that you run them on every chat and save all the outputs to a certain directory: `for file in json/*; do ./mostactiveusers.py --output-folder figures/ --file $file` 72 | 73 | - Find a rough percentage of users in a chat who send less than 3 messages: `./inactiveusers.py /path/to/chatlog.jsonl` 74 | 75 | Note that this script outputs a number which could be taken as a member-count of a chat, but is not, because the chatlogs have no data about people leaving a chat. 76 | 77 | - Make a venn diagram showing the user overlap between two or three chats from chatlogs of those chats: `./venn_chatlog.py -f /path/to/chatlog1.jsonl /path/to/chatlog2.jsonl` 78 | 79 | Note that this script, due to a lack of data about people leaving in the chatlogs, will use a userlist of people who have *ever* been in a chat, not the actual current membership of a chat. 80 | 81 | - Make a venn diagram showing the user overlap (of current membership) between two or three chats: `./venn_userlist.py -f /path/to/memberlist.json -c "Chat Name 1" "Chat Name 2"` 82 | 83 | This script gives more accurate venn diagrams, but uses data which is not easy to get. The script to get this data might become open source in the future. 84 | 85 | - Get a list of the chats you have userlists of: `./listchatsinmemberlist.py /path/to/memberlist.json` 86 | 87 | - Graph the growth of a chat over time: `./usersovertime.py /path/to/chatlog.jsonl` 88 | 89 | Thanks to [NotAFile](https://github.com/NotAFile) for writing this one. 90 | 91 | Note that as with other scripts that use chatlogs for member counts, this one has no ability to see users leaving, so the numbers will be wrong if you use it for that. 92 | 93 | Donations 94 | ----- 95 | If you like the project and have some bitcoin lying around, you're very welcome to send some to me here: 96 | 97 | 1FaMHTtEKHg8tVbdzCuy1VkRdSVpssJu17 98 | -------------------------------------------------------------------------------- /activedays.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to plot the activity of a chat over 24 hours 4 | """ 5 | import argparse 6 | from json import loads 7 | from datetime import date,timedelta,datetime 8 | from os import path 9 | from collections import defaultdict 10 | import matplotlib.pyplot as plt 11 | from sys import maxsize 12 | 13 | def extract_info(event): 14 | text_weekday = datetime.fromtimestamp(event['date']).isoweekday() 15 | text_date = date.fromtimestamp(event['date']) 16 | text_length = len(event['text']) 17 | return text_date, text_weekday, text_length 18 | 19 | def make_ddict_in_range(json_file,start,end): 20 | """ 21 | return a defaultdict(int) of dates with activity on those dates in a date range 22 | """ 23 | events = (loads(line) for line in json_file) 24 | #generator, so whole file is not put in mem 25 | msg_infos = (extract_info(event) for event in events if 'text' in event) 26 | msg_infos = ((date,weekday,length) for (date,weekday,length) in msg_infos if date >= start and date <= end) 27 | counter = defaultdict(int) 28 | #a dict with days as keys and frequency as values 29 | day_freqs = defaultdict(int) 30 | for date_text,day_text,length in msg_infos: 31 | counter[day_text] += length 32 | day_freqs[day_text] += 1 33 | 34 | for k,v in counter.items(): 35 | counter[k] = v/day_freqs[k] 36 | #divide each day's activity by the number of times the day appeared. 37 | #this makes the bar height = average chars sent on that day 38 | #and makes the graph a more accurate representation, especially with small date ranges 39 | 40 | return counter 41 | 42 | def parse_args(): 43 | parser = argparse.ArgumentParser( 44 | description="Visualise the most active days of week in a Telegram chat") 45 | required = parser.add_argument_group('required arguments') 46 | #https://stackoverflow.com/questions/24180527/argparse-required-arguments-listed-under-optional-arguments 47 | required.add_argument( 48 | '-f', '--file', 49 | help='paths to the json file (chat log) to analyse.', 50 | required = True 51 | ) 52 | parser.add_argument( 53 | '-o', '--output-folder', 54 | help='the folder to save the activity graph image in.' 55 | 'Using this option will make the graph not display on screen.') 56 | #parser.add_argument( 57 | # '-b', '--bin-size', 58 | # help='the number of days to group together as one datapoint. ' 59 | # 'Higher number is more smooth graph, lower number is more spiky. ' 60 | # 'Default 3.', 61 | # type=int,default=3) 62 | # #and negative bin sizes are = 1 63 | parser.add_argument( 64 | '-s','--figure-size', 65 | help='the size of the figure shown or saved (X and Y size).' 66 | 'Choose an appropriate value for your screen size. Default 14 8.', 67 | nargs=2,type=int,default=[14,8] 68 | ) 69 | parser.add_argument( 70 | '-d','--date-range', 71 | help='the range of dates you want to look at data between. ' 72 | 'Must be in format YYYY-MM-DD YYYY-MM-DD with the first date ' 73 | 'the start of the range, and the second the end. Example: ' 74 | "-d '2017-11-20 2017-05-15'. Make sure you don't put a day " 75 | 'that is too high for the month eg 30th February.', 76 | default="1000-01-01 4017-01-01" 77 | #hopefully no chatlogs contain these dates :p 78 | ) 79 | 80 | return parser.parse_args() 81 | 82 | def save_figure(folder,filename): 83 | 84 | if len(filename) > 200: 85 | #file name likely to be so long as to cause issues 86 | figname = input( 87 | "This graph is going to have a very long file name. Please enter a custom name(no need to add an extension): ") 88 | else: 89 | figname = "Active days in {}".format(filename) 90 | 91 | plt.savefig("{}/{}.png".format(folder, figname)) 92 | 93 | def annotate_figure(filename,start,end): 94 | if start == date(1000,1,1) and end == date(4017,1,1): 95 | datestr = "entire chat history" 96 | plt.title("Active days in {} in {}".format(filename,datestr)) 97 | else: 98 | datestr = "between {} and {}".format(start,end) 99 | plt.title("Active days in {}, {}".format(filename,datestr)) 100 | plt.ylabel("Activity level (avg. chars sent on day)", size=14) 101 | plt.xlabel("Day of the week", size=14) 102 | plt.gca().set_xlim([1,8]) 103 | plt.xticks(([x+0.5 for x in range(8)]),['','Mon','Tue','Wed','Thu','Fri','Sat','Sun']) 104 | 105 | #if binsize > 1: 106 | # plt.ylabel("Activity level (chars per {} days)".format(binsize), size=14) 107 | #else: 108 | # plt.ylabel("Activity level (chars per day)", size=14) 109 | 110 | def get_dates(arg_dates): 111 | if " " not in arg_dates: 112 | print("You must put a space between start and end dates") 113 | exit() 114 | daterange = arg_dates.split() 115 | start_date = datetime.strptime(daterange[0], "%Y-%m-%d").date() 116 | end_date = datetime.strptime(daterange[1], "%Y-%m-%d").date() 117 | return (start_date,end_date) 118 | 119 | def main(): 120 | """ 121 | main function 122 | """ 123 | 124 | args = parse_args() 125 | 126 | filepath = args.file 127 | savefolder = args.output_folder 128 | figure_size = args.figure_size 129 | start_date,end_date = get_dates(args.date_range) 130 | 131 | filename = path.splitext(path.split(filepath)[-1])[0] 132 | 133 | plt.figure(figsize=figure_size) 134 | 135 | with open(filepath, 'r') as jsonfile: 136 | chat_counter = make_ddict_in_range( 137 | jsonfile,start_date,end_date) 138 | 139 | plt.bar(*zip(*chat_counter.items())) 140 | 141 | annotate_figure(filename,start_date,end_date) 142 | 143 | if savefolder is not None: 144 | #if there is a given folder to save the figure in, save it there 145 | save_figure(savefolder,filename) 146 | else: 147 | #if a save folder was not specified, just open a window to display graph 148 | plt.show() 149 | 150 | if __name__ == "__main__": 151 | main() 152 | -------------------------------------------------------------------------------- /activehours.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to plot the activity of a chat over 24 hours 4 | """ 5 | import argparse 6 | from json import loads 7 | from datetime import date,timedelta,datetime 8 | from os import path 9 | from collections import defaultdict 10 | import matplotlib.pyplot as plt 11 | from sys import maxsize 12 | 13 | def extract_info(event): 14 | text_time = datetime.fromtimestamp(event['date']).hour 15 | text_date = date.fromtimestamp(event['date']) 16 | text_length = len(event['text']) 17 | return text_date, text_time, text_length 18 | 19 | def make_ddict_in_range(json_file,start,end): 20 | """ 21 | return a defaultdict(int) of dates with activity on those dates in a date range 22 | """ 23 | events = (loads(line) for line in json_file) 24 | #generator, so whole file is not put in mem 25 | msg_infos = (extract_info(event) for event in events if 'text' in event) 26 | msg_infos = ((date,time,length) for (date,time,length) in msg_infos if date >= start and date <= end) 27 | counter = defaultdict(int) 28 | #a dict with hours as keys and frequency as values 29 | for date_text,time_text,length in msg_infos: 30 | counter[time_text] += length 31 | 32 | return counter 33 | 34 | def parse_args(): 35 | parser = argparse.ArgumentParser( 36 | description="Visualise the most active times of day in a Telegram chat") 37 | required = parser.add_argument_group('required arguments') 38 | #https://stackoverflow.com/questions/24180527/argparse-required-arguments-listed-under-optional-arguments 39 | required.add_argument( 40 | '-f', '--file', 41 | help='paths to the json file (chat log) to analyse.', 42 | required = True 43 | ) 44 | parser.add_argument( 45 | '-o', '--output-folder', 46 | help='the folder to save the activity graph image in.' 47 | 'Using this option will make the graph not display on screen.') 48 | #parser.add_argument( 49 | # '-b', '--bin-size', 50 | # help='the number of days to group together as one datapoint. ' 51 | # 'Higher number is more smooth graph, lower number is more spiky. ' 52 | # 'Default 3.', 53 | # type=int,default=3) 54 | # #and negative bin sizes are = 1 55 | parser.add_argument( 56 | '-s','--figure-size', 57 | help='the size of the figure shown or saved (X and Y size).' 58 | 'Choose an appropriate value for your screen size. Default 14 8.', 59 | nargs=2,type=int,default=[14,8] 60 | ) 61 | parser.add_argument( 62 | '-d','--date-range', 63 | help='the range of dates you want to look at data between. ' 64 | 'Must be in format YYYY-MM-DD YYYY-MM-DD with the first date ' 65 | 'the start of the range, and the second the end. Example: ' 66 | "-d '2017-11-20 2017-05-15'. Make sure you don't put a day " 67 | 'that is too high for the month eg 30th February.', 68 | default="1000-01-01 4017-01-01" 69 | #hopefully no chatlogs contain these dates :p 70 | ) 71 | 72 | return parser.parse_args() 73 | 74 | def save_figure(folder,filename): 75 | 76 | if len(filename) > 200: 77 | #file name likely to be so long as to cause issues 78 | figname = input( 79 | "This graph is going to have a very long file name. Please enter a custom name(no need to add an extension): ") 80 | else: 81 | figname = "Active hours in {}".format(filename) 82 | 83 | plt.savefig("{}/{}.png".format(folder, figname)) 84 | 85 | def annotate_figure(filename): 86 | plt.title("Active hours in {}".format(filename)) 87 | plt.ylabel("Activity level (chars)", size=14) 88 | plt.xlabel("Hour of the day", size=14) 89 | #sidenote: no idea what timezone lmao 90 | plt.gca().set_xlim([0,24]) 91 | plt.xticks(([x+0.5 for x in range(24)]),range(24)) 92 | 93 | #if binsize > 1: 94 | # plt.ylabel("Activity level (chars per {} days)".format(binsize), size=14) 95 | #else: 96 | # plt.ylabel("Activity level (chars per day)", size=14) 97 | 98 | def get_dates(arg_dates): 99 | if " " not in arg_dates: 100 | print("You must put a space between start and end dates") 101 | exit() 102 | daterange = arg_dates.split() 103 | start_date = datetime.strptime(daterange[0], "%Y-%m-%d").date() 104 | end_date = datetime.strptime(daterange[1], "%Y-%m-%d").date() 105 | return (start_date,end_date) 106 | 107 | def main(): 108 | """ 109 | main function 110 | """ 111 | 112 | args = parse_args() 113 | 114 | filepath = args.file 115 | savefolder = args.output_folder 116 | figure_size = args.figure_size 117 | start_date,end_date = get_dates(args.date_range) 118 | 119 | filename = path.splitext(path.split(filepath)[-1])[0] 120 | 121 | plt.figure(figsize=figure_size) 122 | 123 | with open(filepath, 'r') as jsonfile: 124 | chat_counter = make_ddict_in_range( 125 | jsonfile,start_date,end_date) 126 | 127 | plt.bar(*zip(*chat_counter.items())) 128 | 129 | annotate_figure(filename) 130 | 131 | if savefolder is not None: 132 | #if there is a given folder to save the figure in, save it there 133 | save_figure(savefolder,filename) 134 | else: 135 | #if a save folder was not specified, just open a window to display graph 136 | plt.show() 137 | 138 | if __name__ == "__main__": 139 | main() 140 | -------------------------------------------------------------------------------- /activityovertime.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to plot the activity in a chat over time 4 | """ 5 | import argparse 6 | from json import loads 7 | from datetime import date,timedelta,datetime 8 | from os import path 9 | from collections import defaultdict 10 | import matplotlib.pyplot as plt 11 | from sys import maxsize 12 | 13 | def extract_date_and_len(event): 14 | text_date = date.fromtimestamp(event['date']) 15 | text_length = len(event['text']) 16 | return text_date, text_length 17 | 18 | def make_ddict_in_range(json_file,binsize,start,end): 19 | """ 20 | return a defaultdict(int) of dates with activity on those dates in a date range 21 | """ 22 | events = (loads(line) for line in json_file) 23 | #generator, so whole file is not put in mem 24 | dates_and_lengths = (extract_date_and_len(event) for event in events if 'text' in event) 25 | dates_and_lengths = ((date,length) for (date,length) in dates_and_lengths if date >= start and date <= end) 26 | counter = defaultdict(int) 27 | #a dict with dates as keys and frequency as values 28 | if binsize > 1: 29 | #this makes binsizes ! > 1 act as 1 30 | curbin = 0 31 | for date_text,length in dates_and_lengths: 32 | if curbin == 0 or (curbin - date_text) > timedelta(days=binsize): 33 | curbin = date_text 34 | counter[curbin] += length 35 | else: 36 | for date_text,length in dates_and_lengths: 37 | counter[date_text] += length 38 | 39 | return counter 40 | 41 | def parse_args(): 42 | parser = argparse.ArgumentParser( 43 | description="Visualise and compare the activity of one or more Telegram chats over time.") 44 | required = parser.add_argument_group('required arguments') 45 | #https://stackoverflow.com/questions/24180527/argparse-required-arguments-listed-under-optional-arguments 46 | required.add_argument( 47 | '-f', '--files', 48 | help='paths to the json file(s) (chat logs) to analyse.', 49 | required = True, 50 | nargs='+' 51 | ) 52 | parser.add_argument( 53 | '-o', '--output-folder', 54 | help='the folder to save the activity graph image in.' 55 | 'Using this option will make the graph not display on screen.') 56 | parser.add_argument( 57 | '-b', '--bin-size', 58 | help='the number of days to group together as one datapoint. ' 59 | 'Higher number is more smooth graph, lower number is more spiky. ' 60 | 'Default 3.', 61 | type=int,default=3) 62 | #and negative bin sizes are = 1 63 | parser.add_argument( 64 | '-s','--figure-size', 65 | help='the size of the figure shown or saved (X and Y size).' 66 | 'Choose an appropriate value for your screen size. Default 14 8.', 67 | nargs=2,type=int,default=[14,8] 68 | ) 69 | parser.add_argument( 70 | '-d','--date-range', 71 | help='the range of dates you want to look at data between. ' 72 | 'Must be in format YYYY-MM-DD YYYY-MM-DD with the first date ' 73 | 'the start of the range, and the second the end. Example: ' 74 | "-d '2017-11-20 2017-05-15'. Make sure you don't put a day " 75 | 'that is too high for the month eg 30th February.', 76 | default="1000-01-01 4017-01-01" 77 | #hopefully no chatlogs contain these dates :p 78 | ) 79 | 80 | return parser.parse_args() 81 | 82 | def save_figure(folder,filenames): 83 | chats_string = '_'.join(filenames) 84 | 85 | if len(chats_string) > 200: 86 | #file name likely to be so long as to cause issues 87 | figname = input( 88 | "This graph is going to have a very long file name. Please enter a custom name(no need to add an extension): ") 89 | else: 90 | figname = "Activity in {}".format(chats_string) 91 | 92 | plt.savefig("{}/{}.png".format(folder, figname)) 93 | 94 | def annotate_figure(filenames,binsize): 95 | if len(filenames) > 1: 96 | plt.title("Activity in {}".format(filenames)) 97 | plt.legend(filenames, loc='best') 98 | else: 99 | plt.title("Activity in {}".format(filenames[0])) 100 | 101 | if binsize > 1: 102 | plt.ylabel("Activity level (chars per {} days)".format(binsize), size=14) 103 | else: 104 | plt.ylabel("Activity level (chars per day)", size=14) 105 | 106 | def get_dates(arg_dates): 107 | if " " not in arg_dates: 108 | print("You must put a space between start and end dates") 109 | exit() 110 | daterange = arg_dates.split() 111 | start_date = datetime.strptime(daterange[0], "%Y-%m-%d").date() 112 | end_date = datetime.strptime(daterange[1], "%Y-%m-%d").date() 113 | return (start_date,end_date) 114 | 115 | def main(): 116 | """ 117 | main function 118 | """ 119 | 120 | args = parse_args() 121 | 122 | #set up args 123 | filepaths = args.files 124 | savefolder = args.output_folder 125 | binsize = args.bin_size 126 | figure_size = args.figure_size 127 | start_date,end_date = get_dates(args.date_range) 128 | 129 | filenames = [] 130 | 131 | plt.figure(figsize=figure_size) 132 | 133 | for ind,filepath in enumerate(filepaths): 134 | with open(filepath, 'r') as jsonfile: 135 | #if args.date_range is not None: 136 | # chat_counter = make_ddict_in_date_range( 137 | # jsonfile,binsize,start_date,end_date) 138 | #else: 139 | # chat_counter = make_ddict(jsonfile,binsize) 140 | chat_counter = make_ddict_in_range( 141 | jsonfile,binsize,start_date,end_date) 142 | 143 | filenames.append(path.splitext(path.split(filepath)[-1])[0]) 144 | #make filename just the name of the file, 145 | # with no leading directories and no extension 146 | 147 | chat_activity = sorted(chat_counter.items()) 148 | #find frequency of chat events per date 149 | 150 | plt.plot(*zip(*chat_activity)) 151 | plt.grid() 152 | #because i think it looks better with the grid 153 | 154 | annotate_figure(filenames,binsize) 155 | 156 | if savefolder is not None: 157 | #if there is a given folder to save the figure in, save it there 158 | save_figure(savefolder,filenames) 159 | else: 160 | #if a save folder was not specified, just open a window to display graph 161 | plt.show() 162 | 163 | if __name__ == "__main__": 164 | main() 165 | -------------------------------------------------------------------------------- /examples/activityovertime_example.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expectocode/telegram-analysis/5f218266193e81f741ccb1512cf4bccf5fe9e2be/examples/activityovertime_example.jpg -------------------------------------------------------------------------------- /examples/example_supergroup/Activity in example_supergroup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expectocode/telegram-analysis/5f218266193e81f741ccb1512cf4bccf5fe9e2be/examples/example_supergroup/Activity in example_supergroup.png -------------------------------------------------------------------------------- /examples/example_supergroup/Most active users in example_supergroup.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expectocode/telegram-analysis/5f218266193e81f741ccb1512cf4bccf5fe9e2be/examples/example_supergroup/Most active users in example_supergroup.png -------------------------------------------------------------------------------- /examples/example_supergroup/all_text.txt: -------------------------------------------------------------------------------- 1 | @daughter_of_polonius: You are naught, you are naught: I'll mark the play. 2 | @down_with_claudius: Ay, or any show that you'll show him: be not you ashamed to show, he'll not shame to tell you what it means. 3 | @daughter_of_polonius: Will he tell us what this show meant? 4 | @down_with_claudius: We shall know by this fellow: the players cannot keep counsel; they'll tell all. 5 | @daughter_of_polonius: Belike this show imports the argument of the play. 6 | @down_with_claudius: Marry, this is miching mallecho; it means mischief. 7 | @daughter_of_polonius: What means this, my lord? 8 | @down_with_claudius: Then there's hope a great man's memory may outlive his life half a year: but, by'r lady, he must build churches, then; or else shall he suffer not thinking on, with the hobby-horse, whose epitaph is 'For, O, for, O, the hobby-horse is forgot.' 9 | @down_with_claudius: So long? Nay then, let the devil wear black, for I'll have a suit of sables. O heavens! die two months ago, and not forgotten yet? 10 | @daughter_of_polonius: Nay, 'tis twice two months, my lord. 11 | @down_with_claudius: O God, your only jig-maker. What should a man do but be merry? for, look you, how cheerfully my mother looks, and my father died within these two hours. 12 | @daughter_of_polonius: Ay, my lord. 13 | @down_with_claudius: Who, I? 14 | @daughter_of_polonius: You are merry, my lord. 15 | @down_with_claudius: Nothing. 16 | @daughter_of_polonius: What is, my lord? 17 | @down_with_claudius: That's a fair thought to lie between maids' legs. 18 | @daughter_of_polonius: I think nothing, my lord. 19 | @down_with_claudius: Do you think I meant country matters? 20 | @daughter_of_polonius: Ay, my lord. 21 | @down_with_claudius: I mean, my head upon your lap? 22 | @daughter_of_polonius: No, my lord. 23 | @down_with_claudius: Lady, shall I lie in your lap? 24 | @daughter_of_polonius: O, what a noble mind is here o'erthrown! The courtier's, soldier's, scholar's, eye, tongue, sword; The expectancy and rose of the fair state, The glass of fashion and the mould of form, The observed of all observers, quite, quite down! And I, of ladies most deject and wretched, That suck'd the honey of his music vows, Now see that noble and most sovereign reason, Like sweet bells jangled, out of tune and harsh; That unmatch'd form and feature of blown youth Blasted with ecstasy: O, woe is me, To have seen what I have seen, see what I see! 25 | @down_with_claudius: To a nunnery, go. 26 | @down_with_claudius: I have heard of your paintings too, well enough; God has given you one face, and you make yourselves another: you jig, you amble, and you lisp, and nick-name God's creatures, and make your wantonness your ignorance. Go to, I'll no more on't; it hath made me mad. I say, we will have no more marriages: those that are married already, all but one, shall live; the rest shall keep as they are. 27 | @daughter_of_polonius: O heavenly powers, restore him! 28 | @down_with_claudius: To a nunnery, go, and quickly too. Farewell. 29 | @down_with_claudius: If thou dost marry, I'll give thee this plague for thy dowry: be thou as chaste as ice, as pure as snow, thou shalt not escape calumny. Get thee to a nunnery, go: farewell. Or, if thou wilt needs marry, marry a fool; for wise men know well enough what monsters you make of them. 30 | @daughter_of_polonius: O, help him, you sweet heavens! 31 | @down_with_claudius: Let the doors be shut upon him, that he may play the fool no where but in's own house. Farewell. 32 | @daughter_of_polonius: At home, my lord. 33 | @down_with_claudius: Get thee to a nunnery: why wouldst thou be a breeder of sinners? I am myself indifferent honest; but yet I could accuse me of such things that it were better my mother had not borne me: I am very proud, revengeful, ambitious, with more offences at my beck than I have thoughts to put them in, imagination to give them shape, or time to act them in. What should such fellows as I do crawling between earth and heaven? We are arrant knaves, all; believe none of us. Go thy ways to a nunnery. Where's your father? 34 | @daughter_of_polonius: I was the more deceived. 35 | @down_with_claudius: You should not have believed me; for virtue cannot so inoculate our old stock but we shall relish of it: I loved you not. 36 | @daughter_of_polonius: Indeed, my lord, you made me believe so. 37 | @down_with_claudius: this was sometime a paradox, but now the time gives it proof. I did love you once. 38 | @down_with_claudius: Ay, truly; for the power of beauty will sooner transform honesty from what it is to a bawd than the force of honesty can translate beauty into his likeness: 39 | @daughter_of_polonius: Could beauty, my lord, have better commerce than with honesty? 40 | @down_with_claudius: That if you be honest and fair, your honesty should admit no discourse to your beauty. 41 | @daughter_of_polonius: What means your lordship? 42 | @down_with_claudius: Are you fair? 43 | @daughter_of_polonius: My lord? 44 | @down_with_claudius: Ha! ha! are you honest? 45 | @daughter_of_polonius: My honour'd lord, you know right well you did; And, with them, words of so sweet breath composed As made the things more rich: their perfume lost, Take these again; for to the noble mind Rich gifts wax poor when givers prove unkind. There, my lord. 46 | @down_with_claudius: I never gave you aught. 47 | @down_with_claudius: No, not I; 48 | @daughter_of_polonius: My lord, I have remembrances of yours, That I have longed long to re-deliver; I pray you, now receive them. 49 | @down_with_claudius: I humbly thank you; well, well, well. 50 | @daughter_of_polonius: Good my lord, How does your honour for this many a day? 51 | @down_with_claudius: The fair Ophelia! Nymph, in thy orisons Be all my sins remember'd. 52 | @down_with_claudius: And thus the native hue of resolution Is sicklied o'er with the pale cast of thought, And enterprises of great pith and moment With this regard their currents turn awry, And lose the name of action.--Soft you now! 53 | @down_with_claudius: Thus conscience does make cowards of us all; And thus the native hue of resolution Is sicklied o'er with the pale cast of thought, 54 | @down_with_claudius: The oppressor's wrong, the proud man's contumely, The pangs of despised love, the law's delay, The insolence of office and the spurns That patient merit of the unworthy takes, When he himself might his quietus make With a bare bodkin? who would fardels bear, To grunt and sweat under a weary life, But that the dread of something after death, The undiscover'd country from whose bourn No traveller returns, puzzles the will And makes us rather bear those ills we have Than fly to others that we know not of? 55 | @down_with_claudius: For in that sleep of death what dreams may come When we have shuffled off this mortal coil, Must give us pause: there's the respect That makes calamity of so long life; For who would bear the whips and scorns of time, 56 | @down_with_claudius: Devoutly to be wish'd. To die, to sleep; To sleep: perchance to dream: ay, there's the rub; 57 | @down_with_claudius: The heart-ache and the thousand natural shocks That flesh is heir to, 'tis a consummation 58 | @down_with_claudius: And by opposing end them? To die: to sleep; No more; and by a sleep to say we end 59 | @down_with_claudius: The slings and arrows of outrageous fortune, Or to take arms against a sea of troubles, 60 | @down_with_claudius: To be, or not to be: that is the question: Whether 'tis nobler in the mind to suffer 61 | -------------------------------------------------------------------------------- /examples/example_supergroup/example_supergroup.jsonl: -------------------------------------------------------------------------------- 1 | {"event":"service","id":"05000000483a6544400000000000000014581ada5066b4fc","flags":8450,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":true,"date":1496144673,"action":{"type":"chat_del_user","user":{"id":"$010000001e8b940a0000000000000000","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"}}} 2 | {"event":"message","id":"05000000483a65443f0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1496144586,"text":"You are naught, you are naught: I'll mark the play."} 3 | {"event":"message","id":"05000000483a65443e0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1496076786,"text":"Ay, or any show that you'll show him: be not you ashamed to show, he'll not shame to tell you what it means."} 4 | {"event":"message","id":"05000000483a65443d0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1496071860,"text":"Will he tell us what this show meant?"} 5 | {"event":"message","id":"05000000483a65443c0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1496071762,"text":"We shall know by this fellow: the players cannot keep counsel; they'll tell all."} 6 | {"event":"message","id":"05000000483a65443b0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1496071743,"text":"Belike this show imports the argument of the play."} 7 | {"event":"message","id":"05000000483a65443a0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1496071703,"text":"Marry, this is miching mallecho; it means mischief."} 8 | {"event":"message","id":"05000000483a6544390000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1496071364,"text":"What means this, my lord?"} 9 | {"event":"message","id":"05000000483a6544380000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495981048,"text":"Then there's hope a great man's memory may outlive his life half a year: but, by'r lady, he must build churches, then; or else shall he suffer not thinking on, with the hobby-horse, whose epitaph is 'For, O, for, O, the hobby-horse is forgot.'"} 10 | {"event":"message","id":"05000000483a6544370000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495981031,"text":"So long? Nay then, let the devil wear black, for I'll have a suit of sables. O heavens! die two months ago, and not forgotten yet?"} 11 | {"event":"message","id":"05000000483a6544360000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495980965,"text":"Nay, 'tis twice two months, my lord."} 12 | {"event":"message","id":"05000000483a6544350000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495980877,"text":"O God, your only jig-maker. What should a man do but be merry? for, look you, how cheerfully my mother looks, and my father died within these two hours."} 13 | {"event":"message","id":"05000000483a6544340000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495980852,"text":"Ay, my lord."} 14 | {"event":"message","id":"05000000483a6544330000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495980721,"text":"Who, I?"} 15 | {"event":"message","id":"05000000483a6544320000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495980712,"text":"You are merry, my lord."} 16 | {"event":"message","id":"05000000483a6544310000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495980696,"text":"Nothing."} 17 | {"event":"message","id":"05000000483a6544300000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495980655,"text":"What is, my lord?"} 18 | {"event":"message","id":"05000000483a65442f0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495980638,"text":"That's a fair thought to lie between maids' legs."} 19 | {"event":"message","id":"05000000483a65442e0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495980624,"text":"I think nothing, my lord."} 20 | {"event":"message","id":"05000000483a65442c0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495926954,"text":"Do you think I meant country matters?"} 21 | {"event":"message","id":"05000000483a65442b0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495808358,"text":"Ay, my lord."} 22 | {"event":"message","id":"05000000483a65442a0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495807705,"text":"I mean, my head upon your lap?"} 23 | {"event":"message","id":"05000000483a6544290000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495807341,"text":"No, my lord."} 24 | {"event":"message","id":"05000000483a6544280000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495807260,"text":"Lady, shall I lie in your lap?"} 25 | {"event":"message","id":"05000000483a6544270000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495806959,"text":"O, what a noble mind is here o'erthrown! The courtier's, soldier's, scholar's, eye, tongue, sword; The expectancy and rose of the fair state, The glass of fashion and the mould of form, The observed of all observers, quite, quite down! And I, of ladies most deject and wretched, That suck'd the honey of his music vows, Now see that noble and most sovereign reason, Like sweet bells jangled, out of tune and harsh; That unmatch'd form and feature of blown youth Blasted with ecstasy: O, woe is me, To have seen what I have seen, see what I see!"} 26 | {"event":"message","id":"05000000483a6544260000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495735433,"text":"To a nunnery, go."} 27 | {"event":"message","id":"05000000483a6544250000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495735433,"text":"I have heard of your paintings too, well enough; God has given you one face, and you make yourselves another: you jig, you amble, and you lisp, and nick-name God's creatures, and make your wantonness your ignorance. Go to, I'll no more on't; it hath made me mad. I say, we will have no more marriages: those that are married already, all but one, shall live; the rest shall keep as they are."} 28 | {"event":"message","id":"05000000483a6544240000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495735392,"text":"O heavenly powers, restore him!"} 29 | {"event":"message","id":"05000000483a6544230000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495735389,"text":"To a nunnery, go, and quickly too. Farewell."} 30 | {"event":"message","id":"05000000483a6544220000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495735387,"text":"If thou dost marry, I'll give thee this plague for thy dowry: be thou as chaste as ice, as pure as snow, thou shalt not escape calumny. Get thee to a nunnery, go: farewell. Or, if thou wilt needs marry, marry a fool; for wise men know well enough what monsters you make of them."} 31 | {"event":"message","id":"05000000483a6544210000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495649996,"text":"O, help him, you sweet heavens!"} 32 | {"event":"message","id":"05000000483a6544200000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495649993,"text":"Let the doors be shut upon him, that he may play the fool no where but in's own house. Farewell."} 33 | {"event":"message","id":"05000000483a65441f0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495649975,"text":"At home, my lord."} 34 | {"event":"message","id":"05000000483a65441e0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495649971,"text":"Get thee to a nunnery: why wouldst thou be a breeder of sinners? I am myself indifferent honest; but yet I could accuse me of such things that it were better my mother had not borne me: I am very proud, revengeful, ambitious, with more offences at my beck than I have thoughts to put them in, imagination to give them shape, or time to act them in. What should such fellows as I do crawling between earth and heaven? We are arrant knaves, all; believe none of us. Go thy ways to a nunnery. Where's your father?"} 35 | {"event":"message","id":"05000000483a65441d0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495562878,"text":"I was the more deceived."} 36 | {"event":"message","id":"05000000483a65441c0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495562873,"text":"You should not have believed me; for virtue cannot so inoculate our old stock but we shall relish of it: I loved you not."} 37 | {"event":"message","id":"05000000483a65441b0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495562838,"text":"Indeed, my lord, you made me believe so."} 38 | {"event":"message","id":"05000000483a65441a0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495562824,"text":"this was sometime a paradox, but now the time gives it proof. I did love you once."} 39 | {"event":"message","id":"05000000483a6544190000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495562816,"text":"Ay, truly; for the power of beauty will sooner transform honesty from what it is to a bawd than the force of honesty can translate beauty into his likeness:"} 40 | {"event":"message","id":"05000000483a6544180000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495562769,"text":"Could beauty, my lord, have better commerce than with honesty?"} 41 | {"event":"message","id":"05000000483a6544170000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495481609,"text":"That if you be honest and fair, your honesty should admit no discourse to your beauty."} 42 | {"event":"message","id":"05000000483a6544160000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495479854,"text":"What means your lordship?"} 43 | {"event":"message","id":"05000000483a6544150000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495472537,"text":"Are you fair?"} 44 | {"event":"message","id":"05000000483a6544140000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495457430,"text":"My lord?"} 45 | {"event":"message","id":"05000000483a6544130000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495456678,"text":"Ha! ha! are you honest?"} 46 | {"event":"message","id":"05000000483a6544120000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495439918,"text":"My honour'd lord, you know right well you did; And, with them, words of so sweet breath composed As made the things more rich: their perfume lost, Take these again; for to the noble mind Rich gifts wax poor when givers prove unkind. There, my lord."} 47 | {"event":"message","id":"05000000483a6544110000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495394810,"text":"I never gave you aught."} 48 | {"event":"message","id":"05000000483a6544100000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495394806,"text":"No, not I;"} 49 | {"event":"message","id":"05000000483a65440f0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495394795,"text":"My lord, I have remembrances of yours, That I have longed long to re-deliver; I pray you, now receive them."} 50 | {"event":"message","id":"05000000483a65440e0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495394702,"text":"I humbly thank you; well, well, well."} 51 | {"event":"message","id":"05000000483a65440d0000000000000014581ada5066b4fc","flags":256,"from":{"id":"$010000001e8b990a0932287e3f6c3514","peer_type":"user","peer_id":177724113,"print_name":"Ophelia","flags":196609,"first_name":"Ophelia","last_name":"","phone":"9876543210","username":"daughter_of_polonius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":false,"unread":false,"service":false,"date":1495387970,"text":"Good my lord, How does your honour for this many a day?"} 52 | {"event":"message","id":"05000000483a65440c0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293770,"text":"The fair Ophelia! Nymph, in thy orisons Be all my sins remember'd."} 53 | {"event":"message","id":"05000000483a65440b0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293764,"text":"And thus the native hue of resolution Is sicklied o'er with the pale cast of thought, And enterprises of great pith and moment With this regard their currents turn awry, And lose the name of action.--Soft you now!"} 54 | {"event":"message","id":"05000000483a65440a0000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293752,"text":"Thus conscience does make cowards of us all; And thus the native hue of resolution Is sicklied o'er with the pale cast of thought,"} 55 | {"event":"message","id":"05000000483a6544090000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293744,"text":"The oppressor's wrong, the proud man's contumely, The pangs of despised love, the law's delay, The insolence of office and the spurns That patient merit of the unworthy takes, When he himself might his quietus make With a bare bodkin? who would fardels bear, To grunt and sweat under a weary life, But that the dread of something after death, The undiscover'd country from whose bourn No traveller returns, puzzles the will And makes us rather bear those ills we have Than fly to others that we know not of?"} 56 | {"event":"message","id":"05000000483a6544080000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293720,"text":"For in that sleep of death what dreams may come When we have shuffled off this mortal coil, Must give us pause: there's the respect That makes calamity of so long life; For who would bear the whips and scorns of time,"} 57 | {"event":"message","id":"05000000483a6544070000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293712,"text":"Devoutly to be wish'd. To die, to sleep; To sleep: perchance to dream: ay, there's the rub;"} 58 | {"event":"message","id":"05000000483a6544060000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293706,"text":"The heart-ache and the thousand natural shocks That flesh is heir to, 'tis a consummation"} 59 | {"event":"message","id":"05000000483a6544050000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293700,"text":"And by opposing end them? To die: to sleep; No more; and by a sleep to say we end"} 60 | {"event":"message","id":"05000000483a6544040000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293693,"text":"The slings and arrows of outrageous fortune, Or to take arms against a sea of troubles,"} 61 | {"event":"message","id":"05000000483a6544030000000000000014581ada5066b4fc","flags":258,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":false,"date":1495293685,"text":"To be, or not to be: that is the question: Whether 'tis nobler in the mind to suffer"} 62 | {"event":"service","id":"05000000483a6544020000000000000014581ada5066b4fc","flags":8450,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":true,"date":1495293677,"action":{"type":"chat_change_photo"}} 63 | {"event":"service","id":"05000000483a6544010000000000000014581ada5066b4fc","flags":8450,"from":{"id":"$01000000ff1bd304f348271e0052ca45","peer_type":"user","peer_id":80861041,"print_name":"Hamlet","flags":720897,"first_name":"Hamlet","last_name":"","phone":"0123456789","username":"down_with_claudius"},"to":{"id":"$05000000483a634214681adb5026b7fc","peer_type":"channel","peer_id":1137794233,"print_name":"example_supergroup#1","flags":524353,"title":"example supergroup","participants_count":0,"admins_count":0,"kicked_count":0},"out":true,"unread":false,"service":true,"date":1495293612,"action":{"type":"migrated_from"}} 64 | -------------------------------------------------------------------------------- /examples/mostactiveusers_example.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expectocode/telegram-analysis/5f218266193e81f741ccb1512cf4bccf5fe9e2be/examples/mostactiveusers_example.jpg -------------------------------------------------------------------------------- /examples/phraseovertime_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expectocode/telegram-analysis/5f218266193e81f741ccb1512cf4bccf5fe9e2be/examples/phraseovertime_example.png -------------------------------------------------------------------------------- /examples/venn_example.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/expectocode/telegram-analysis/5f218266193e81f741ccb1512cf4bccf5fe9e2be/examples/venn_example.jpg -------------------------------------------------------------------------------- /getalltext.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to extract raw text from Telegram chat log 4 | """ 5 | import argparse 6 | from json import loads 7 | 8 | def main(): 9 | 10 | parser = argparse.ArgumentParser( 11 | description="Extract all raw text from a specific Telegram chat") 12 | parser.add_argument('filepath', help='the json chatlog file to analyse') 13 | parser.add_argument('-u','--usernames', help='Show usernames before messages. ' 14 | 'If someone doesn\'t have a username, the line will start with "@: ".' 15 | 'Useful when output will be read back as a chatlog.', 16 | action='store_true') 17 | parser.add_argument('-n','--no-newlines', help='Remove all newlines from messages. Useful when ' 18 | 'output will be piped into analysis expecting newline separated messages. ', 19 | action='store_true') 20 | 21 | args=parser.parse_args() 22 | filepath = args.filepath 23 | 24 | with open(filepath, 'r') as jsonfile: 25 | events = (loads(line) for line in jsonfile) 26 | for event in events: 27 | #check the event is the sort we're looking for 28 | if "from" in event and "text" in event: 29 | if args.usernames: 30 | if 'username' in event['from']: 31 | print('@' + event['from']['username'],end=': ') 32 | else: 33 | print('@',end=': ') 34 | if args.no_newlines: 35 | print(event['text'].replace('\n','')) 36 | else: 37 | print(event["text"]) 38 | 39 | if __name__ == "__main__": 40 | main() 41 | -------------------------------------------------------------------------------- /getalltextfromuser.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to extract all text sent by a particular user from a Telegram chat log 4 | """ 5 | import argparse 6 | from json import loads 7 | 8 | def main(): 9 | 10 | parser = argparse.ArgumentParser( 11 | description="Extract all raw text sent by a specific user in a specific Telegram chat") 12 | parser.add_argument( 13 | 'filepath', help='the jsonl chatlog file to analyse') 14 | parser.add_argument( 15 | 'username', help='a username of the person whose text you want (without @ sign), case insensitive') 16 | 17 | args=parser.parse_args() 18 | filepath = args.filepath 19 | username = args.username.lower() 20 | 21 | user_id = "" 22 | 23 | #first, get the ID of the user with that username. 24 | #ideally, this only runs for less than 100 messages, if its a recent username 25 | #TODO: allow user id as argument 26 | with open(filepath, 'r') as jsonfile: 27 | events = (loads(line) for line in jsonfile) 28 | for event in events: 29 | #check the event is the sort we're looking for 30 | if "from" in event: 31 | if "username" in event["from"]: 32 | if event["from"]["username"].lower() == username: 33 | user_id = event['from']['id'] 34 | break 35 | if user_id == "": 36 | print("username not found in chatlog") 37 | exit() 38 | 39 | with open(filepath, 'r') as jsonfile: 40 | events = (loads(line) for line in jsonfile) 41 | for event in events: 42 | #check the event is the sort we're looking for 43 | if "from" in event and "text" in event: 44 | if user_id == event["from"]["id"]: 45 | print(event["text"]) 46 | 47 | if __name__ == "__main__": 48 | main() 49 | -------------------------------------------------------------------------------- /getpinnedmessages.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | from json import loads 3 | import argparse 4 | 5 | parser = argparse.ArgumentParser( 6 | "Print all the pinned text messages from a Telegram chat log") 7 | parser.add_argument( 8 | 'file', 9 | help='path to the json file (chat log) to analyse') 10 | 11 | args = parser.parse_args() 12 | 13 | with open(args.file,'r') as f: 14 | jsn = [loads(line) for line in f.readlines()] 15 | 16 | pins = [x['reply_id'] for x in jsn if 17 | 'text' in x and x['text'] == 'pinned the message'] 18 | #reply_id is the ID of the message that has been pinned. 19 | pin_msgs = [x for x in jsn if x['id'] in pins if 'text' in x] 20 | #ignore pins with no text 21 | 22 | _ = [print(x['text'],'\n------------------') for x in pin_msgs] 23 | -------------------------------------------------------------------------------- /inactiveusers.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A quick hack of a program to find a rough percentage of users in a chat who have sent less than 3 messages. 4 | 5 | Warning: written at 1AM 6 | """ 7 | import argparse 8 | from json import loads 9 | from os import path 10 | from collections import defaultdict 11 | 12 | def main(): 13 | """ 14 | main function 15 | """ 16 | #cutoff for a 'non active' user 17 | minimum = 3 18 | non_active_users = 0 19 | active_users = 0 20 | parser = argparse.ArgumentParser(description="Find the number of inactive users (users who have sent less than 3 messages) in a Telegram chat") 21 | parser.add_argument('filepath', help='the jsonl chatlog file to analyse') 22 | 23 | args = parser.parse_args() 24 | filepath = args.filepath 25 | 26 | _, filename = path.split(filepath) 27 | filename, _ = path.splitext(filename) 28 | #make filename just the name of the file, with no leading directories and no extension 29 | 30 | counter = defaultdict(int) #store events from each user 31 | #names = {} #dict 32 | total_datapoints = 0 33 | 34 | with open(filepath, 'r') as jsonfile: 35 | events = (loads(line) for line in jsonfile) 36 | for event in events: 37 | if "from" in event: 38 | if "peer_id" in event["from"] and "print_name" in event["from"]: 39 | total_datapoints += 1 40 | user = event['from']['peer_id'] 41 | counter[user] += 1 42 | 43 | for person, frequency in counter.items(): 44 | if frequency < minimum: 45 | non_active_users += 1 46 | else: 47 | active_users += 1 48 | 49 | print('For this chat, there were {} users who sent less than' 50 | ' {} messages, out of a total of {}.'.format( 51 | non_active_users,minimum,non_active_users+active_users)) 52 | print("That's", round(100* non_active_users/(non_active_users + active_users),1), "%!") 53 | 54 | # print(type(*sorted(counter.items()))) 55 | # plt.pie(*zip(*sorted(counter.items()))) 56 | 57 | if __name__ == "__main__": 58 | main() 59 | -------------------------------------------------------------------------------- /listchatsinmemberlist.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to tell you which chats you have in your memberlist, sorted by title length. 4 | """ 5 | import argparse 6 | from json import loads 7 | 8 | def main(): 9 | """ 10 | main function 11 | """ 12 | parser = argparse.ArgumentParser( 13 | description="Tell you which chats you have info on in your memberlist") 14 | parser.add_argument( 15 | 'filepath', 16 | help='paths to the json userlist') 17 | 18 | args = parser.parse_args() 19 | filepath = args.filepath 20 | 21 | j = loads(open(filepath,'r').read()) 22 | 23 | sorted_j = [ x for x in sorted(list(j.items()), key=lambda a: len(a[1]['title'])) if (len(x[1]['users']) > 0) ] 24 | #this makes the JSON input into a list of tuples (chat_id, info_dict) and also removes empty chats 25 | #the importance of sorting is so that search strings are first tested on the smaller titles 26 | #eg searching 'GNU/Linux' should yield 'GNU/Linux' before 'GNU/Linux Chat' (real example) 27 | for chat in sorted_j: 28 | #because some of the memberlist things come to zero 29 | print(chat[1]['title']) 30 | 31 | if __name__ == "__main__": 32 | main() 33 | -------------------------------------------------------------------------------- /mostactiveusers.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to plot a pie chart of the most active users in a Telegram chat 4 | """ 5 | import argparse 6 | from json import loads 7 | from os import path 8 | from collections import defaultdict 9 | import matplotlib.pyplot as plt 10 | from datetime import date,datetime 11 | from operator import itemgetter 12 | 13 | def parse_args(): 14 | parser = argparse.ArgumentParser(description= 15 | "Create a pie chart showing the most active users in a Telegram chat") 16 | required = parser.add_argument_group('required arguments') 17 | required.add_argument('-f','--file', 18 | help='the jsonl chatlog file to analyse', 19 | required = True 20 | ) 21 | parser.add_argument( 22 | '-o', '--output-folder', 23 | help='the folder to save the pie chart image in.' 24 | 'Using this option will make the graph not display on screen.') 25 | parser.add_argument( 26 | '-s','--figure-size', 27 | help='the size of the figure shown or saved (X and Y size).' 28 | 'Choose an appropriate value for your screen size. Default 12 8.', 29 | nargs=2,type=int,default = [12,8] 30 | ) 31 | parser.add_argument( 32 | '-m','--minimum-percentage', 33 | help='the minimum percentage of activity a person must contribute ' 34 | 'to get their own slice of the pie chart. Default 2', 35 | type=float,default=2 36 | ) 37 | parser.add_argument( 38 | '-d','--date-range', 39 | help='the range of dates you want to look at data between. ' 40 | 'Must be in format YYYY-MM-DD YYYY-MM-DD with the first date ' 41 | 'the start of the range, and the second the end. Example: ' 42 | "-d '2017-11-20 2017-05-15'. Make sure you don't put a day " 43 | 'that is too high for the month eg 30th February.', 44 | default="1000-01-01 4017-01-01" 45 | #hopefully no chatlogs contain these dates :p 46 | ) 47 | 48 | return parser.parse_args() 49 | 50 | def get_dates(arg_dates): 51 | if " " not in arg_dates: 52 | print("You must put a space between start and end dates") 53 | exit() 54 | daterange = arg_dates.split() 55 | start_date = datetime.strptime(daterange[0], "%Y-%m-%d").date() 56 | end_date = datetime.strptime(daterange[1], "%Y-%m-%d").date() 57 | return (start_date,end_date) 58 | 59 | def extract_infos(event): 60 | text_date = date.fromtimestamp(event['date']) 61 | text_length = len(event['text']) 62 | text_userid= event['from']['peer_id'] 63 | text_printname = event['from']['print_name'] 64 | return text_date,text_length,text_userid,text_printname 65 | 66 | def make_ddict(jsonfile,start,end): 67 | """ 68 | Make a defaultdict with user IDs as keys and char count as values 69 | Return (dict of IDs -> names, total chars, defaultdict) 70 | """ 71 | names = {} #dict 72 | counter = defaultdict(int) 73 | total_datapoints = 0 74 | events = (loads(line) for line in jsonfile) 75 | messages = (extract_infos(event) for event in events if 'text' in event) 76 | messages = ((when,what,uid,who) for (when,what,uid,who) in messages if when >= start and when <= end) 77 | for (msgdate,textlength,userid,printname) in messages: 78 | total_datapoints += textlength 79 | if str(userid) not in names: 80 | #this code assumes that chatlog has most recent events first 81 | #which is default for telegram-history-dumper 82 | names[str(userid)] = printname 83 | if printname == "": 84 | names[str(userid)] = str(userid) 85 | counter[userid] += textlength 86 | 87 | return names,total_datapoints,counter 88 | 89 | def annotate_figure(filename): 90 | plt.title("Most active users in {} by chars sent".format(filename), y=1.05) 91 | plt.axis('equal') 92 | #so it plots as a circle 93 | 94 | def make_trimmed_ddict(counter,total_datapoints,names,min_percent): 95 | trimmedCounter = defaultdict(int) 96 | #find percentile to start adding people to "other" at 97 | min_chars = (min_percent/100) * total_datapoints 98 | for person, frequency in counter.items(): 99 | if frequency < min_chars: 100 | trimmedCounter["other"] += frequency 101 | else: 102 | if names[str(person)] == "other": 103 | print("Someone in this chat is called 'other'. " 104 | "They will be absorbed into the 'other' pie slice.") 105 | trimmedCounter[names[str(person)]] = frequency 106 | 107 | return trimmedCounter 108 | 109 | def main(): 110 | """ 111 | main function 112 | """ 113 | 114 | args = parse_args() 115 | filepath = args.file 116 | savefolder = args.output_folder 117 | figure_size = (args.figure_size[0],args.figure_size[1]) 118 | start_date,end_date = get_dates(args.date_range) 119 | other_percent = args.minimum_percentage 120 | #default 2 121 | #anyone who sends less than this percentage of the total is 'other' 122 | 123 | filename = path.splitext(path.split(filepath)[-1])[0] 124 | #make filename just the name of the file, with no leading directories and no extension 125 | 126 | with open(filepath, 'r') as jsonfile: 127 | names,total_datapoints,counter = make_ddict(jsonfile,start_date,end_date) 128 | 129 | trimmedCounter = make_trimmed_ddict(counter,total_datapoints,names,other_percent) 130 | 131 | sortedCounter = sorted(trimmedCounter.items(), key=itemgetter(1)) 132 | print(sortedCounter) 133 | 134 | freqList = list(zip(*sortedCounter)) 135 | plt.figure(figsize=figure_size) 136 | 137 | plt.pie(freqList[1], labels=freqList[0], startangle=135) 138 | annotate_figure(filename) 139 | # plt.set_lw(10) 140 | 141 | if savefolder is not None: 142 | #if there is a given folder to save the figure in, save it there 143 | plt.savefig("{}/Most active users in {}.png".format(savefolder, filename)) 144 | else: 145 | #if a save folder was not specified, just open a window to display graph 146 | plt.show() 147 | 148 | if __name__ == "__main__": 149 | main() 150 | -------------------------------------------------------------------------------- /mostcommonphrases.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to find the most common lines in an input. 4 | Developed to find most common messages in a chatlog 5 | """ 6 | 7 | from sys import stdin 8 | import argparse 9 | import matplotlib.pyplot as plt 10 | from collections import Counter 11 | 12 | def main(): 13 | """ 14 | main function 15 | """ 16 | 17 | parser = argparse.ArgumentParser( 18 | formatter_class=argparse.RawDescriptionHelpFormatter, 19 | description='Find the most commonly repeated lines and their frequencies from an input.', 20 | epilog="""Example usage: 21 | Get all text from a chat and find the most repeated messages: 22 | ./getalltext.py chat_name.jsonl | ./mostcommonphrases.py 23 | Find the most repeated lines in a textfile: 24 | ./mostcommonphrases.py < textfile.txt 25 | Not recommended: 26 | running this program with no arguments. This will wait for input and EOF.""" 27 | ) 28 | parser.add_argument( 29 | '-g','--graph', 30 | help='Create a bar chart showing the most common phrases', 31 | action='store_true') 32 | parser.add_argument('-o', '--output-folder', 33 | help='the folder to save the bar chart image in.' 34 | 'Using this option will make the graph not display on screen. ' 35 | 'This option has no effect if -g/--graph is not specified.') 36 | parser.add_argument('-n','--number-of-phrases', 37 | help='The number of phrases to print. Default 20.', 38 | type=int) 39 | parser.add_argument('-b','--number-of-bars', 40 | help='The number of bars to show on the bar chart (cannot be larger than number of phrases). Default 20.', 41 | type=int) 42 | parser.add_argument( 43 | '-s','--figure-size', 44 | help='the size of the figure shown or saved (X and Y size). ' 45 | 'Choose an appropriate value for your screen size. Default 14 8. ' 46 | 'This option has no effect if -g/--graph is not specified.', 47 | nargs=2,type=int 48 | ) 49 | 50 | args = parser.parse_args() 51 | 52 | number_of_phrases = args.number_of_phrases if (args.number_of_phrases is not None) else 20 53 | if args.number_of_bars is not None: 54 | #its defined, lets make sure its sane 55 | if args.number_of_bars > number_of_phrases: 56 | number_of_bars = number_of_phrases 57 | else: 58 | number_of_bars = args.number_of_bars 59 | elif args.number_of_phrases is not None: 60 | #its not defined, but if the other is let's make them equal 61 | number_of_bars = number_of_phrases 62 | else: 63 | #its not defined and neither is the other. 64 | number_of_bars = number_of_phrases 65 | if args.figure_size is not None: 66 | figure_size = (args.figure_size[0],args.figure_size[1]) 67 | else: 68 | figure_size = (14,8) 69 | 70 | sortedfreqs = Counter(map(lambda x: x.rstrip().lower(),stdin) 71 | ) .most_common(number_of_phrases) 72 | #most common phrases from a counter object which takes all the lines from stdin and strips em 73 | 74 | #delete all with freq 1 75 | sortedfreqs = [x for x in sortedfreqs if x[1] != 1] 76 | 77 | print(sortedfreqs) # output the list 78 | 79 | if args.graph: 80 | 81 | #now just deal with the top 10% of phrases 82 | sortedfreqs = sortedfreqs[:number_of_bars] 83 | #frequency_threshold = sortedfreqs[len(sortedfreqs)//5][1] 84 | #sortedfreqs = [x for x in sortedfreqs if x[1] > frequency_threshold] 85 | phrases,frequencies = list(zip(*sortedfreqs)) 86 | y_pos = range(len(phrases)) 87 | width = 0.6 88 | plt.figure(figsize=figure_size) 89 | plt.bar([x*2 for x in y_pos],frequencies,align='center',width=width) 90 | plt.ylabel('frequency') 91 | plt.title('most common phrases') 92 | plt.xticks([x*2+width/3 for x in y_pos], phrases,rotation=25,ha='right') 93 | #list comp makes the coords all shifted slightly, rotation is for readability, 94 | #ha=right ensures thaht the rotated labels have their right side under the bar they refer to 95 | plt.show() 96 | else: 97 | #show a warning message if they use a graph arg and theres no --graph 98 | pass 99 | 100 | if __name__ == "__main__": 101 | main() 102 | -------------------------------------------------------------------------------- /phraseovertime.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to plot the popularity of a phrase in a chat over time 4 | """ 5 | import argparse 6 | from json import loads 7 | from datetime import date,datetime,timedelta 8 | from os import path 9 | from collections import defaultdict 10 | import matplotlib.pyplot as plt 11 | 12 | def extract_date_and_text(event): 13 | text_date = date.fromtimestamp(event['date']) 14 | text = event['text'] 15 | return (text_date,text) 16 | 17 | def make_word_counters_in_range(jsonfile, keywords,binsize,case_sensitive,start,end): 18 | """ 19 | return a list of defaultdict(list) of phrases over time, in a specified date range. 20 | Each phrase has a dict of True/False lists for each day or bin. 21 | Date range specified with date objects. 22 | """ 23 | events = (loads(line) for line in jsonfile) 24 | #generator, so whole file is not put in mem 25 | dates_and_texts = (extract_date_and_text(event) for event in events if 'text' in event) 26 | dates_and_texts = ((date,text) for (date,text) in dates_and_texts if 27 | date > start and date < end) 28 | word_counters = [defaultdict(list) for keyword in keywords] 29 | 30 | if binsize > 1: 31 | curbin = 0 32 | for text_date,text in dates_and_texts: 33 | if curbin == 0 or (curbin - text_date) > timedelta(days=binsize): 34 | curbin = text_date 35 | for k_ind,keyword in enumerate(keywords): 36 | if case_sensitive: 37 | word_counters[k_ind][curbin].append(keyword in text) 38 | else: 39 | word_counters[k_ind][curbin].append(keyword in text.lower()) 40 | else: 41 | for text_date,text in dates_and_texts: 42 | for k_ind,keyword in enumerate(keywords): 43 | if case_sensitive: 44 | word_counters[k_ind][text_date].append(keyword in text) 45 | else: 46 | word_counters[k_ind][text_date].append(keyword in text.lower()) 47 | return word_counters 48 | 49 | def parse_args(): 50 | parser = argparse.ArgumentParser( 51 | description="Visualise and compare the usage of one or more words/phrases in a chat over time") 52 | required = parser.add_argument_group('required arguments') 53 | required.add_argument('-f','--file', 54 | help='path to the json file (chat log) to analyse', 55 | required=True) 56 | required.add_argument('-p','--phrases', 57 | help='the phrase(s) to search for', 58 | nargs='+', 59 | required = True) 60 | parser.add_argument('-c', '--case-sensitive', 61 | help='make the phrase search case sensitive', 62 | action='store_true') 63 | parser.add_argument('-o', '--output-folder', 64 | help='the folder to save the graph image in') 65 | parser.add_argument('-b', '--bin-size', 66 | help='the number of days to group together as one datapoint. Higher number is more smooth graph, lower number is more spiky. Default 3') 67 | parser.add_argument( 68 | '-s','--figure-size', 69 | help='the size of the figure shown or saved (X and Y size).' 70 | 'Choose an appropriate value for your screen size. Default 14 8.', 71 | nargs=2,type=int 72 | ) 73 | parser.add_argument( 74 | '-d','--date-range', 75 | help='the range of dates you want to look at data between. ' 76 | 'Must be in format YYYY-MM-DD YYYY-MM-DD with the first date ' 77 | 'the start of the range, and the second the end. Example: ' 78 | "-d '2017-11-20 2017-05-15'. Make sure you don't put a day " 79 | 'that is too high for the month eg 30th February.', 80 | default="1000-01-01 4017-01-01" 81 | #hopefully no chatlogs contain these dates :p 82 | 83 | ) 84 | return parser.parse_args() 85 | 86 | def draw_figure(word_counters,keywords,figure_size): 87 | frequenciesList = [] 88 | plt.figure(figsize=figure_size) 89 | for x in range(len(keywords)): 90 | #make a frequencies thing for each keyword, and plot each one onto the plot 91 | frequenciesList.append( 92 | {key: l.count(True)/len(l) * 100 for key, l in word_counters[x].items()}) 93 | #find frequency of keyword use per date 94 | plt.plot(*zip(*sorted(frequenciesList[x].items()))) 95 | 96 | plt.grid() 97 | #because i think it looks better with the grid 98 | 99 | def annotate_figure(filename,keywords,case_sensitive): 100 | if case_sensitive: 101 | #pretty self explanatory. this is added to the title. 102 | postfix=", case sensitive" 103 | else: 104 | postfix=", case insensitive" 105 | if len(keywords) > 1: 106 | plt.title( 107 | "usage of {} in {}{}".format(keywords, filename, postfix)) 108 | plt.legend(keywords) 109 | else: 110 | plt.title( 111 | 'usage of "{}" in {}{}'.format(keywords[0], filename, postfix)) 112 | #plt.legend(keywords[0]) 113 | 114 | # plt.ylabel("Percentage of messages containing \"{}\"".format(keyword), size=14) 115 | plt.ylabel("Percentage of messages containing phrase", size=14) 116 | 117 | def save_figure(folder,filename,keywords): 118 | keywords_string = '_'.join(keywords) 119 | if len(keywords_string) > 200: 120 | #file name likely to be so long as to cause issues 121 | figname = input( 122 | "This graph is going to have a very long file name. Please enter a custom name(no need to add an extension): ") 123 | else: 124 | figname = "{} in {}".format( 125 | keywords_string, filename) 126 | 127 | plt.savefig("{}/{}.png".format(folder, figname)) 128 | 129 | def get_dates(arg_dates): 130 | if " " not in arg_dates: 131 | print("You must put a space between start and end dates") 132 | exit() 133 | daterange = arg_dates.split() 134 | start_date = datetime.strptime(daterange[0], "%Y-%m-%d").date() 135 | end_date = datetime.strptime(daterange[1], "%Y-%m-%d").date() 136 | return (start_date,end_date) 137 | 138 | def main(): 139 | """ 140 | main function 141 | """ 142 | 143 | args = parse_args() 144 | filepath = args.file 145 | keywords = args.phrases 146 | savefolder = args.output_folder 147 | if args.bin_size is not None: 148 | binsize = int(args.bin_size) 149 | else: 150 | binsize = 3 151 | if args.figure_size is not None: 152 | figure_size = (args.figure_size[0],args.figure_size[1]) 153 | else: 154 | figure_size = (14,8) 155 | start_date,end_date = get_dates(args.date_range) 156 | 157 | with open(filepath, 'r') as jsonfile: 158 | word_counters = make_word_counters_in_range( 159 | jsonfile,keywords,binsize,args.case_sensitive,start_date,end_date) 160 | 161 | filename = path.splitext(path.split(filepath)[-1])[0] 162 | #make filename just the name of the file, 163 | # with no leading directories and no extension 164 | 165 | draw_figure(word_counters,keywords,figure_size) 166 | annotate_figure(filename,keywords,args.case_sensitive) 167 | 168 | if savefolder is not None: 169 | #if there is a given folder to save the figure in, save it there 170 | save_figure(savefolder,filename,keywords) 171 | else: 172 | #if a save folder was not specified, just open a window to display graph 173 | plt.show() 174 | 175 | if __name__ == "__main__": 176 | main() 177 | -------------------------------------------------------------------------------- /usersovertime.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to plot the users joining a chat over time. Note that leaving events are not noted. 4 | TODO: support multiple chats. 5 | TODO: support saving to image 6 | """ 7 | import argparse 8 | import os 9 | from json import loads 10 | from datetime import date 11 | from collections import defaultdict 12 | from pprint import pprint 13 | 14 | import matplotlib.pyplot as plt 15 | 16 | def main(): 17 | """ 18 | main function 19 | """ 20 | parser = argparse.ArgumentParser( 21 | description="Visualise the growth a chat over time - note that leavers are not recorded, so this program can only show the join rate.") 22 | parser.add_argument('path', help='the json file (chat log) to analyse') 23 | 24 | args = parser.parse_args() 25 | filepath = args.path 26 | 27 | with open(filepath, 'r') as f: 28 | events = (loads(line) for line in f) 29 | 30 | counter = defaultdict(int) 31 | for event in events: 32 | if "action" in event and (event["action"]["type"] == "chat_add_user" or event['action']['type'] == 'chat_add_user_link'): 33 | day = date.fromtimestamp(event["date"]) 34 | counter[day] += 1 35 | 36 | filename = os.path.splitext(os.path.split(filepath)[-1])[0] 37 | 38 | #frequencies = {key: l.count(True)/l.count(False) * 100 for key, l in counter.items()} 39 | users_per_day = sorted(counter.items()) 40 | 41 | u_count = 0 42 | for idx, (day, users) in enumerate(users_per_day): 43 | u_count += users 44 | users_per_day[idx] = (day, u_count) 45 | 46 | print(users_per_day) 47 | plt.plot(*zip(*users_per_day)) 48 | plt.title('members in "{}"'.format(filename)) 49 | 50 | plt.show() 51 | 52 | if __name__ == "__main__": 53 | main() 54 | -------------------------------------------------------------------------------- /venn_chatlog.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to plot the overlap of chats 4 | """ 5 | import argparse 6 | from json import loads 7 | from os import path 8 | import matplotlib.pyplot as plt 9 | from matplotlib_venn import venn2, venn3 10 | from collections import defaultdict 11 | 12 | def get_active_users(filepath): 13 | minimum = 3 14 | counter = defaultdict(int) #store events from each user 15 | #names = {} #dict 16 | active_users = set() 17 | 18 | with open(filepath, 'r') as jsonfile: 19 | events = (loads(line) for line in jsonfile) 20 | for event in events: 21 | if "from" in event: 22 | if "id" in event["from"] and "print_name" in event["from"]: 23 | user = event['from']['id'] 24 | counter[user] += 1 25 | 26 | for person, frequency in counter.items(): 27 | if frequency > minimum: 28 | active_users.add(person) 29 | 30 | return active_users 31 | 32 | def main(): 33 | """ 34 | main function 35 | """ 36 | parser = argparse.ArgumentParser( 37 | description="Visualise the overlap between 2 or 3 chats \n but note that the program is not truly accurate as it counts users who have left to be part of a chat. Also note that for 3 chats, perfect geometry may be impossible.") 38 | parser.add_argument( 39 | '-f','--files', 40 | help='paths to the json file(s) (chat logs) to analyse. Note these must be at the end of the arguments.', 41 | nargs='+', 42 | required = True) 43 | parser.add_argument('-a','--active-users', 44 | help='Only look at active users (users who have sent more than 3 messages)', 45 | action='store_true') 46 | parser.add_argument('-o', '--output-folder', 47 | help='the folder to save the graph image in') 48 | 49 | args = parser.parse_args() 50 | filepaths = args.files 51 | savefolder = args.output_folder 52 | 53 | filenames = [] 54 | users = [set() for filepath in filepaths] 55 | #create a list of users for each chat 56 | 57 | for index,filepath in enumerate(filepaths): 58 | _, temp = path.split(filepath) 59 | filenames.append(temp) 60 | filenames[filepaths.index(filepath)] , _ = path.splitext( 61 | filenames[filepaths.index(filepath)] ) 62 | 63 | print(filenames[index], "users:") 64 | 65 | with open(filepath, 'r') as jsonfile: 66 | events = (loads(line) for line in jsonfile) 67 | #generator, so whole file is not put in mem 68 | #a dict with dates as keys and frequency as values 69 | for event in events: 70 | if "action" in event and event["action"]["type"] == "chat_add_user": 71 | #print(event['action']['user']['id'], ":", event['action']['user']['print_name']) 72 | users[index].add(event['from']['id']) 73 | elif "action" in event and event['action']['type'] == 'chat_add_user_link': 74 | #print(event['from']['id'], ":", event['from']['print_name']) 75 | users[index].add(event['from']['id']) 76 | #print("index:",index) 77 | #print("len(users):",len(users)) 78 | if args.active_users: 79 | active = get_active_users(filepath) 80 | users[index] = users[index] & active 81 | 82 | print(len(users[index]),"users") 83 | 84 | if len(users) == 2: 85 | venn2([users[0], users[1]],(filenames[0], filenames[1])) 86 | elif len(users) == 3: 87 | venn3([users[0], users[1], users[2]],(filenames[0], filenames[1], filenames[2])) 88 | 89 | #print(users) 90 | 91 | if savefolder is not None: 92 | #if there is a given folder to save the figure in, save it there 93 | names_string = '_'.join(filenames) 94 | 95 | if len(names_string) > 200: 96 | #file name likely to be so long as to cause issues 97 | figname = input( 98 | "This diagram is going to have a very long file name. Please enter a custom name(no need to add an extension): ") 99 | else: 100 | figname = 'User overlap in {}'.format(names_string).replace('/','_') 101 | 102 | plt.savefig("{}/{}.png".format(savefolder, figname)) 103 | else: 104 | plt.show() 105 | 106 | if __name__ == "__main__": 107 | main() 108 | -------------------------------------------------------------------------------- /venn_userlist.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | """ 3 | A program to plot the overlap of chats 4 | """ 5 | import argparse 6 | from json import loads 7 | import matplotlib.pyplot as plt 8 | from matplotlib_venn import venn2, venn3 9 | 10 | def main(): 11 | """ 12 | main function 13 | """ 14 | parser = argparse.ArgumentParser( 15 | description="Visualise the overlap between 2 or 3 chats. Note that for 3 chats, perfect geometry may be impossible.") 16 | parser.add_argument( 17 | '-f','--file', 18 | help='paths to the json userlist', 19 | required = True) 20 | parser.add_argument( 21 | '-c','--chat_names', 22 | help="Names (or part of names) of the chats you're interested in, case insensitive", 23 | nargs='+', 24 | required = True) 25 | parser.add_argument('-o', '--output-folder', 26 | help='the folder to save the graph image in') 27 | 28 | args = parser.parse_args() 29 | filepath = args.file 30 | names = args.chat_names 31 | savefolder = args.output_folder 32 | 33 | full_names = [] 34 | userlists = [[] for name in names] 35 | 36 | j = loads(open(filepath,'r').read()) 37 | list_of_chats = [j[x] for x in j] 38 | titles = [x['title'] for x in list_of_chats] 39 | 40 | #this code works but doesn't sort j by chat title length, which is important due to the user title search thing 41 | #for index,name in enumerate(names): 42 | # found_chat = False 43 | # for chat_id in j: 44 | # if name in j[chat_id]['title'] and len(list(j[chat_id]['users'])) > 0: 45 | # #because some of the memberlist things come to zero 46 | # #print(j[chat_id]['users']) 47 | # full_names.append(j[chat_id]['title']) 48 | # userlists[index].extend( [user['id'] for user in j[chat_id]['users']] ) 49 | # found_chat = True 50 | # 51 | # if not found_chat: 52 | # print("Could not find result for", name) 53 | # exit() 54 | 55 | #magic 56 | # [x[1]['title'] for x in sorted(list(j.items()), key=lambda a: len(a[1]['title']))] 57 | 58 | sorted_j = [ x for x in sorted(list(j.items()), key=lambda a: len(a[1]['title'])) if(len(x[1]['users']) > 0) ] 59 | #this makes the JSON input into a list of tuples (chat_id, info_dict) and also removes empty chats 60 | #the importance of sorting is so that search strings are first tested on the smaller titles 61 | #eg searching 'GNU/Linux' should yield 'GNU/Linux' before 'GNU/Linux Chat' (real example) 62 | for index,name in enumerate(names): 63 | found_chat = False 64 | for chat in sorted_j: 65 | #lowercase because case sensitivity annoyed me 66 | if name.lower() in chat[1]['title'].lower() and len(chat[1]['users']) > 0 and (not found_chat): 67 | #because some of the memberlist things come to zero 68 | #print(j[chat_id]['users']) 69 | full_names.append(chat[1]['title']) 70 | userlists[index].extend( [user['id'] for user in chat[1]['users']] ) 71 | found_chat = True 72 | 73 | if not found_chat: 74 | print("Could not find result for", name) 75 | exit() 76 | 77 | if len(userlists) == 2: 78 | venn2([set(userlists[0]), set(userlists[1])],(full_names[0], full_names[1])) 79 | elif len(userlists) == 3: 80 | venn3([set(userlists[0]), set(userlists[1]), set(userlists[2])],(full_names[0], full_names[1], full_names[2])) 81 | 82 | #print(users) 83 | 84 | if savefolder is not None: 85 | #if there is a given folder to save the figure in, save it there 86 | names_string = '_'.join(full_names) 87 | 88 | if len(names_string) > 200: 89 | #file name likely to be so long as to cause issues 90 | figname = input( 91 | "This diagram is going to have a very long file name. Please enter a custom name(no need to add an extension): ") 92 | else: 93 | figname = 'User overlap in {}'.format(names_string).replace('/','_') 94 | 95 | plt.savefig("{}/{}.png".format(savefolder, figname)) 96 | else: 97 | plt.show() 98 | 99 | if __name__ == "__main__": 100 | main() 101 | --------------------------------------------------------------------------------