├── .gitignore ├── README.md ├── UserBehaviorAnalyzer.py └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore 3 | !README.md 4 | !requirements.txt 5 | !UserBehaviorAnalyzer.py 6 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # User-Behavior-Mapping-Tool 2 | 3 | Project aims to map out common user behavior on the computer. 4 | Most of the code is based on the research by kacos2000 found here: 5 | https://github.com/kacos2000/WindowsTimeline 6 | 7 | TrustedSec blog about the research behind it: 8 | https://www.trustedsec.com/blog/oh-behave-figuring-out-user-behavior/ 9 | 10 | 11 | # Installation 12 | 1. git clone the repo 13 | 2. pip3 install -r requirements.txt 14 | 15 | 16 | # Getting started 17 | To make use of this project you first need to copy out the ActivityCache.db file found on the users computer under: 18 | ``` 19 | C:\Users\%username%\AppData\Local\ConnectedDevicesPlatform\ 20 | ``` 21 | 22 | ## UserBehaviorAnalyzer.py 23 | To parse an ActivityCache.db file specify the path with the -f parameter. 24 | If you only want the main exported data (one csv) you can specify -m. 25 | Output folder is specified with the -o parameter. Folder will be created if it does not exist. 26 | If no output folder is specified the output goes in the current working directory. 27 | 28 | 29 | ``` 30 | python3 UserBehaviourAnalyzer.py -f /mnt/c/ads/ActivitiesCache.db 31 | Succesfully exported full raw database report 32 | Report gen_report_useractivity_start_and_end.csv Generated successfully 33 | Report gen_report_ApplicationLaunch_StartTime.csv Generated successfully 34 | Paths_Unique.txt Generated successfully 35 | Report gen_report_Activity_Applications.csv Generated successfully 36 | Chart gen_fig_useractivity_heatmap.jpg Generated successfully 37 | /mnt/c/gitlab/user-behavior/1. Extraction Script/UserBehaviorAnalyzer.py:565: UserWarning: FixedFormatter should only be used together with FixedLocator 38 | ax1.set_xticklabels(df1['Date'], rotation=90) 39 | Chart gen_fig_useractivity_bar.jpg Generated successfully 40 | Chart gen_fig_top10_apps_pie.jpg Generated successfully 41 | Chart gen_fig_top10_apps_bars.jpg Generated successfully 42 | ``` 43 | 44 | ## Reports 45 | 46 | ### gen_report_Activity_Applications.csv 47 | This report contains the total of time the different application has been actively used based on all the data found in the database. 48 | 49 | ### gen_report_ApplicationLaunch_StartTime.csv 50 | This reports shows the applications that are launched and parameters used (also filenames sometimes) and when it was launched. 51 | This is useful for understanding when the user starts his applications. 52 | 53 | ### gen_report_useractivity_start_and_end.csv 54 | This report groups all times for each day and finds the first entry of the day and the last. 55 | This report is useful for understanding when the user starts his day and when the last application was launched. 56 | 57 | ## Charts 58 | 59 | ### gen_fig_top10_apps_bars.jpg 60 | This shows the top 10 most used application visualized with Bars. Usage is in seconds. 61 | 62 | ### gen_fig_top10_apps_pie.jpg 63 | This shows the top 10 most used application visualized as a pie chart. Usage is in seconds. 64 | 65 | ### gen_fig_useractivity_bar.jpg 66 | This visualizes when the user is active and idle based on the first activity found per day and the last activity found per day. The y axis shows the time of day. 67 | The time is based on the timezone of the user 68 | ex 500 = 0500 (5am) 69 | ex 2000 (8pm) 70 | 71 | ### gen_fig_useractivity_heatmap.jpg 72 | This visualized the users activity sorted on days. The brighter color the more activity. The time is based on the timezone of the user 73 | 74 | ## Other 75 | 76 | ### Paths_Unique.txt 77 | This file contains unique paths the for documents/files/folders the user works towards. Perfect targets for backdoors. 78 | 79 | 80 | # Issues 81 | If you do encounter issues please create a github issue. You might need to provide the ActivitiesCache.db since it could be a case that has not been encountered. -------------------------------------------------------------------------------- /UserBehaviorAnalyzer.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # Script that parses the sqlite3 database ActivityCache.db that comes from Windows Timeline and outputs reports and charts 3 | # Author: Oddvar Moe - TrustedSec 4 | # Version: 1.2 5 | # Based on https://github.com/kacos2000/WindowsTimeline/blob/master/WinTimelineOffline.ps1 6 | import argparse 7 | import sqlite3 8 | import json 9 | import pandas as pd 10 | import base64 11 | import re 12 | import matplotlib.pyplot as plt 13 | import pandas as pd 14 | import matplotlib.pyplot as plt 15 | import seaborn as sns 16 | from collections import defaultdict 17 | import csv 18 | import os 19 | import sys 20 | from termcolor import colored 21 | 22 | if __name__ == '__main__': 23 | logo = ''' 24 | =vxr^rui))^^^< 25 | .=(()*^(v!' _LIu}ir;_` 26 | `?x|r:`'____` :~ri}Tv}yVuvr?iyv~_ 27 | `;v)! `__- .___:^:- -__.,)` `_ :Vx*!:",<='`!- `.GG~ 30 | `y! `__., _^;_' `:"__,=:*)_T=. ~<>*!,:^=!\<~:vr' `Tv` 32 | 'PV` .<=- `!` _()v*!!Lo}x*?iv;v).=_ ._.:=vVzc^!:^r<<=*~*)*_`xh). 33 | ,rx_ -___",__=,,,__:(*))*yVx*<::^ir_ ^ ._!^:}v, .:_"_">^<~:*~>*=:Mgl 34 | :M6*__~!==_:!<<.`:;^^=` `:-.=r^^~_`., *` ';=rv:="~,*Y^~<~<^~_>*!:I" 35 | !di` ____!___`;r_:: <_-:::<_"=_. ~. ~~ 'u` `__,^rv:.!,=)~vvrL~x(~:_=*xb, 36 | !T ", !```."rT=-*v_u~.,>=,_ `` ~~ x* ** `:;:-)::)::x-.=i]._::^<~^)j- 37 | l; `~ !_.)^-.\="(\).v-_` _" `_(-.w":) _= `:_:^``~!!L= ,r<,-:'_!:,)y 38 | c- =.'!_,r!^=:;.*Tu\!` __`_=!~^!,,}|!`~ ~ ``:)` i- r: ',=~=-:~:'}V 39 | `X` ~ :__^=` -Lr*mwk-`,_:~=x*=' `!*~!>_`," '.~ ` ~ -: .*~ '):`.v~~``r* TP 40 | rZ ``=!,)v^^|x?Yv*,^)``!`_-~_x<^|z:~=__.-_`-.~"` ~`` '^"` ~ ! -*<<:^;! y` 41 | II `}!~'-rxr~:_-__,` ~ !(.)- :r `. :?_.=` `= `_ _ `! :);'i"~^T 42 | `z; r!=`,`:^<:___` ` ~_!^^_- `v' -~)^:_)=:~=:___` ', `-r!,!=e: 43 | u* ^} ^-`='^^^:_____:)vxr~_`___>= .:"<^::!~__-__` '=ri**^^^*r)iT=` -:!`y~ 44 | Z~ -xr-,~=-: -::______`._-_:<:!__' `(T*r! ^y* `.`rV 46 | _6x- !!_^v'._:__` `___-' ':=,<*^^r?L*!--=~``^VI\` `-!^(xLxuGT~,=-cw` 47 | ~6` ~ _,^":~":.__` -\3y^=!<*rr^!,-_:)*:^Px: ")ivvxL}Lxvv}g#@@@d*'su 48 | .5` ~ '~ `"_:*aBs,,}xxcYxcXH5H3PGaKHMdZV!.` ,iPKx_ '!^*^` c~w@y^!*)w 49 | `Y? `:' -! ,G#@Q}I8@#0jv!. -=V##QQQB@@y `=**ZD3]V, ^YIzo._=xK_ 50 | rv_ ._- `__- T*^P#@@@Y `*xTkms35ZPi` i@gr"~G@=.xVr^)**. x6^z|I}_ 51 | '**r~` .- V" -l##- ;=_____--.'-- `W@^ )y.***)Lv, x? _V 'Z_ `<)): ^xGx :V *i' 68 | ***^, `(v- V, `V Li)~` `~r**_ !rr~=u T~ _Y~ 69 | ^i_ V V, `u`:rr^- .<**^^~:,__:~**r!` _V' _V ^], 70 | 'Y< !3 (v _\r=`-^**^:` )}?^^^.=T' c: `)T~ 71 | ''' 72 | parser = argparse.ArgumentParser(description='Tool for parsing ActivityCache.db and generating reports and charts. Will output files (csv and jpg) in the current directory.') 73 | parser.add_argument('-f','--file', default=None, type=str, help='Path to ActivityCache.db') 74 | parser.add_argument('-o','--outfolder', default=os.getcwd(), type=str, help='Output folder - Default is scripts working directory - Ex: /root/folder/test/') 75 | parser.add_argument('-m','--onlyexportmaindata', action='store_true', default=False, help='Set this to only export the main report and skip additional reports and charts generating') 76 | parser.add_argument('-v','--verbose', action='store_true', default=False, help='Enable Verbose Logging - Not implemented yet') 77 | 78 | args = parser.parse_args() 79 | if len(sys.argv)==1: 80 | print(colored("No arguments provided - Showing help\n", "red")) 81 | print(logo) 82 | parser.print_help() 83 | sys.exit(0) 84 | 85 | print(logo) 86 | file = args.file 87 | only_gen_main = args.onlyexportmaindata 88 | verbose = args.verbose 89 | outfolder = args.outfolder 90 | 91 | # Create outfolder if it does not exist 92 | isExist = os.path.exists(outfolder) 93 | if not isExist: 94 | os.makedirs(outfolder) 95 | 96 | # Connect to database and get data 97 | con = sqlite3.connect(file) 98 | con.text_factory = lambda b: b.decode(errors = 'ignore') 99 | cur = con.cursor() 100 | 101 | # Using Pandas to store the query 102 | try: 103 | query = cur.execute('''select 104 | ETag, 105 | AppId, 106 | case when AppActivityId not like '%-%-%-%-%' then AppActivityId 107 | else trim(AppActivityId,'ECB32AF3-1440-4086-94E3-5311F97F89C4\') end as 'AppActivityId', 108 | ActivityType as 'Activity_type', 109 | case ActivityStatus 110 | when 1 then 'Active' when 2 then 'Updated' when 3 then 'Deleted' when 4 then 'Ignored' 111 | end as 'ActivityStatus', 112 | Smartlookup.'group' as 'Group', 113 | MatchID, 114 | 'No' AS 'IsInUploadQueue', 115 | Priority as 'Priority', 116 | ClipboardPayload, 117 | datetime(LastModifiedTime, 'unixepoch', 'localtime')as 'LastModifiedTime', 118 | datetime(ExpirationTime, 'unixepoch', 'localtime') as 'ExpirationTime', 119 | datetime(StartTime, 'unixepoch', 'localtime') as 'StartTime', 120 | datetime(EndTime, 'unixepoch', 'localtime') as 'EndTime', 121 | case 122 | when CreatedInCloud > 0 123 | then datetime(CreatedInCloud, 'unixepoch', 'localtime') 124 | else '' 125 | end as 'CreatedInCloud', 126 | case 127 | when OriginalLastModifiedOnClient > 0 128 | then datetime(OriginalLastModifiedOnClient, 'unixepoch', 'localtime') 129 | else '' 130 | end as 'OriginalLastModifiedOnClient', 131 | Tag, 132 | PlatformDeviceId, 133 | Payload from Smartlookup 134 | order by Etag desc''') 135 | except: 136 | sys.exit('Error parsing the database') 137 | 138 | dbdata = query.fetchall() 139 | 140 | known = {'308046B0AF4A39CB': 'Mozilla Firefox 64bit', 141 | 'E7CF176E110C211B': 'Mozilla Firefox 32bit', 142 | 'DE61D971-5EBC-4F02-A3A9-6C82895E5C04': 'AddNewPrograms', 143 | '724EF170-A42D-4FEF-9F26-B60E846FBA4F': 'AdminTools', 144 | 'A520A1A4-1780-4FF6-BD18-167343C5AF16': 'AppDataLow', 145 | 'A305CE99-F527-492B-8B1A-7E76FA98D6E4': 'AppUpdates', 146 | '9E52AB10-F80D-49DF-ACB8-4330F5687855': 'CDBurning', 147 | 'DF7266AC-9274-4867-8D55-3BD661DE872D': 'ChangeRemovePrograms', 148 | 'D0384E7D-BAC3-4797-8F14-CBA229B392B5': 'CommonAdminTools', 149 | 'C1BAE2D0-10DF-4334-BEDD-7AA20B227A9D': 'CommonOEMLinks', 150 | '0139D44E-6AFE-49F2-8690-3DAFCAE6FFB8': 'CommonPrograms', 151 | 'A4115719-D62E-491D-AA7C-E74B8BE3B067': 'CommonStartMenu', 152 | '82A5EA35-D9CD-47C5-9629-E15D2F714E6E': 'CommonStartup', 153 | 'B94237E7-57AC-4347-9151-B08C6C32D1F7': 'CommonTemplates', 154 | '0AC0837C-BBF8-452A-850D-79D08E667CA7': 'Computer', 155 | '4BFEFB45-347D-4006-A5BE-AC0CB0567192': 'Conflict', 156 | '6F0CD92B-2E97-45D1-88FF-B0D186B8DEDD': 'Connections', 157 | '56784854-C6CB-462B-8169-88E350ACB882': 'Contacts', 158 | '82A74AEB-AEB4-465C-A014-D097EE346D63': 'ControlPanel', 159 | '2B0F765D-C0E9-4171-908E-08A611B84FF6': 'Cookies', 160 | 'B4BFCC3A-DB2C-424C-B029-7FE99A87C641': 'Desktop', 161 | 'FDD39AD0-238F-46AF-ADB4-6C85480369C7': 'Documents', 162 | '374DE290-123F-4565-9164-39C4925E467B': 'Downloads', 163 | '1777F761-68AD-4D8A-87BD-30B759FA33DD': 'Favorites', 164 | 'FD228CB7-AE11-4AE3-864C-16F3910AB8FE': 'Fonts', 165 | 'CAC52C1A-B53D-4EDC-92D7-6B2E8AC19434': 'Games', 166 | '054FAE61-4DD8-4787-80B6-090220C4B700': 'GameTasks', 167 | 'D9DC8A3B-B784-432E-A781-5A1130A75963': 'History', 168 | '4D9F7874-4E0C-4904-967B-40B0D20C3E4B': 'Internet', 169 | '352481E8-33BE-4251-BA85-6007CAEDCF9D': 'InternetCache', 170 | 'BFB9D5E0-C6A9-404C-B2B2-AE6DB6AF4968': 'Links', 171 | 'F1B32785-6FBA-4FCF-9D55-7B8E7F157091': 'LocalAppData', 172 | '2A00375E-224C-49DE-B8D1-440DF7EF3DDC': 'LocalizedResourcesDir', 173 | '4BD8D571-6D19-48D3-BE97-422220080E43': 'Music', 174 | 'C5ABBF53-E17F-4121-8900-86626FC2C973': 'NetHood', 175 | 'D20BEEC4-5CA8-4905-AE3B-BF251EA09B53': 'Network', 176 | '2C36C0AA-5812-4B87-BFD0-4CD0DFB19B39': 'OriginalImages', 177 | '69D2CF90-FC33-4FB7-9A0C-EBB0F0FCB43C': 'PhotoAlbums', 178 | '33E28130-4E1E-4676-835A-98395C3BC3BB': 'Pictures', 179 | 'DE92C1C7-837F-4F69-A3BB-86E631204A23': 'Playlists', 180 | '76FC4E2D-D6AD-4519-A663-37BD56068185': 'Printers', 181 | '9274BD8D-CFD1-41C3-B35E-B13F55A758F4': 'PrintHood', 182 | '5E6C858F-0E22-4760-9AFE-EA3317B67173': 'Profile', 183 | '62AB5D82-FDC1-4DC3-A9DD-070D1D495D97': 'ProgramData', 184 | '905E63B6-C1BF-494E-B29C-65B732D3D21A': 'ProgramFiles', 185 | 'F7F1ED05-9F6D-47A2-AAAE-29D317C6F066': 'ProgramFilesCommon', 186 | '6365D5A7-0F0D-45E5-87F6-0DA56B6A4F7D': 'ProgramFilesCommonX64', 187 | 'DE974D24-D9C6-4D3E-BF91-F4455120B917': 'ProgramFilesCommonX86', 188 | '6D809377-6AF0-444B-8957-A3773F02200E': 'ProgramFilesX64', 189 | '7C5A40EF-A0FB-4BFC-874A-C0F2E0B9FA8E': 'ProgramFilesX86', 190 | 'A77F5D77-2E2B-44C3-A6A2-ABA601054A51': 'Programs', 191 | 'DFDF76A2-C82A-4D63-906A-5644AC457385': 'Public', 192 | 'C4AA340D-F20F-4863-AFEF-F87EF2E6BA25': 'PublicDesktop', 193 | 'ED4824AF-DCE4-45A8-81E2-FC7965083634': 'PublicDocuments', 194 | '3D644C9B-1FB8-4F30-9B45-F670235F79C0': 'PublicDownloads', 195 | 'DEBF2536-E1A8-4C59-B6A2-414586476AEA': 'PublicGameTasks', 196 | '3214FAB5-9757-4298-BB61-92A9DEAA44FF': 'PublicMusic', 197 | 'B6EBFB86-6907-413C-9AF7-4FC2ABF07CC5': 'PublicPictures', 198 | '2400183A-6185-49FB-A2D8-4A392A602BA3': 'PublicVideos', 199 | '52A4F021-7B75-48A9-9F6B-4B87A210BC8F': 'QuickLaunch', 200 | 'AE50C081-EBD2-438A-8655-8A092E34987A': 'Recent', 201 | 'BD85E001-112E-431E-983B-7B15AC09FFF1': 'RecordedTV', 202 | 'B7534046-3ECB-4C18-BE4E-64CD4CB7D6AC': 'RecycleBin', 203 | '8AD10C31-2ADB-4296-A8F7-E4701232C972': 'ResourceDir', 204 | '3EB685DB-65F9-4CF6-A03A-E3EF65729F3D': 'RoamingAppData', 205 | 'B250C668-F57D-4EE1-A63C-290EE7D1AA1F': 'SampleMusic', 206 | 'C4900540-2379-4C75-844B-64E6FAF8716B': 'SamplePictures', 207 | '15CA69B3-30EE-49C1-ACE1-6B5EC372AFB5': 'SamplePlaylists', 208 | '859EAD94-2E85-48AD-A71A-0969CB56A6CD': 'SampleVideos', 209 | '4C5C32FF-BB9D-43B0-B5B4-2D72E54EAAA4': 'SavedGames', 210 | '7D1D3A04-DEBB-4115-95CF-2F29DA2920DA': 'SavedSearches', 211 | 'EE32E446-31CA-4ABA-814F-A5EBD2FD6D5E': 'SEARCH_CSC', 212 | '98EC0E18-2098-4D44-8644-66979315A281': 'SEARCH_MAPI', 213 | '190337D1-B8CA-4121-A639-6D472D16972A': 'SearchHome', 214 | '8983036C-27C0-404B-8F08-102D10DCFD74': 'SendTo', 215 | '7B396E54-9EC5-4300-BE0A-2482EBAE1A26': 'SidebarDefaultParts', 216 | 'A75D362E-50FC-4FB7-AC2C-A8BEAA314493': 'SidebarParts', 217 | '625B53C3-AB48-4EC1-BA1F-A1EF4146FC19': 'StartMenu', 218 | 'B97D20BB-F46A-4C97-BA10-5E3608430854': 'Startup', 219 | '43668BF8-C14E-49B2-97C9-747784D784B7': 'SyncManager', 220 | '289A9A43-BE44-4057-A41B-587A76D7E7F9': 'SyncResults', 221 | '0F214138-B1D3-4A90-BBA9-27CBC0C5389A': 'SyncSetup', 222 | '1AC14E77-02E7-4E5D-B744-2EB1AE5198B7': 'System', 223 | 'D65231B0-B2F1-4857-A4CE-A8E7C6EA7D27': 'SystemX86', 224 | 'A63293E8-664E-48DB-A079-DF759E0509F7': 'Templates', 225 | '5B3749AD-B49F-49C1-83EB-15370FBD4882': 'TreeProperties', 226 | '0762D272-C50A-4BB0-A382-697DCD729B80': 'UserProfiles', 227 | 'F3CE0F7C-4901-4ACC-8648-D5D44B04EF8F': 'UsersFiles', 228 | '18989B1D-99B5-455B-841C-AB7C74E4DDFC': 'Videos', 229 | 'F38BF404-1D43-42F2-9305-67DE0B28FC23': 'Windows'} 230 | 231 | rows = [] 232 | diclist = [] 233 | 234 | # Row[0]=ETag -- Row[1]=AppId -- Row[2]=AppActivityId -- Row[3]=Activity_type -- Row[4]=ActivityStatus -- Row[5]=Group -- Row[6]=MatchID -- Row[7]=IsInUploadQueue 235 | # Row[8]=Priority -- Row[9]=ClipboardPayload -- Row[10]=LastModifiedTime -- Row[11]=ExpirationTime -- Row[12]=StartTime -- Row[13]=EndTime -- Row[14]=CreatedInCloud 236 | # Row[15]=OriginalLastModifiedOnClient -- Row[16]=Tag -- Row[17]=PlatformDeviceId -- Row[18]=Payload 237 | 238 | for row in dbdata: 239 | #Init all vars as blanks 240 | strActivityStatus = "" 241 | strActivityType = "" 242 | strAppActivityId = "" 243 | strBackupType = "" 244 | strBackupUpdated = "" 245 | strCreationDate = "" 246 | strCreatedInCloud = "" 247 | strContent = "" 248 | strContentUrl = "" 249 | strCopiedText = "" 250 | strDescription = "" 251 | strDeviceIdentifier = "" 252 | strDeviceName = "" 253 | strDevicePlatform = "" 254 | strDeviceType = "" 255 | strDisplayName = "" 256 | strDisplayText = "" 257 | strDuration = "" 258 | strEtag = "" 259 | strEndTime = "" 260 | strExpirationTime = "" 261 | strGroup = "" 262 | strIsInUploadQueue = "" 263 | strKnownFolder = "" 264 | strLastModifiedTime = "" 265 | strMake = "" 266 | strMatchID = "" 267 | strModel = "" 268 | strName = "" 269 | strNotification = "" 270 | strObjectId = "" 271 | strOriginalLastModifiedOnClient = "" 272 | strPlatformDeviceId = "" 273 | strPriority = "" 274 | strStartTime = "" 275 | strSynched = "" 276 | strTag = "" 277 | strTimeZone = "" 278 | strType = "" 279 | strVolumeID = "" 280 | ## Fill data ## 281 | strEtag = row[0] 282 | strAppActivityId = row[2] 283 | strActivityStatus = row[4] 284 | strGroup = row[5] 285 | strMatchID = row[6] 286 | strIsInUploadQueue = row[7] 287 | strPriority = row[8] 288 | strLastModifiedTime = row[10] 289 | strExpirationTime = row[11] 290 | strStartTime = row[12] 291 | strCreatedInCloud = row[14] 292 | strOriginalLastModifiedOnClient = row[15] 293 | strTag = row[16] 294 | strPlatformDeviceId = row[17] 295 | if row[18]: 296 | if row[3] == 6: 297 | if 'type' in (json.loads(row[18])): 298 | strType = (json.loads(row[18]))['type'] 299 | if 'activeDurationSeconds' in (json.loads(row[18])): 300 | strDuration = (json.loads(row[18]))['activeDurationSeconds'] 301 | if 'userTimezone' in (json.loads(row[18])): 302 | strTimeZone = (json.loads(row[18]))['userTimezone'] 303 | if 'devicePlatform' in (json.loads(row[18])): 304 | strDevicePlatform = (json.loads(row[18]))['devicePlatform'] 305 | if row[3] == 5: 306 | if 'displayText' in (json.loads(row[18])): 307 | strDisplayText = (json.loads(row[18]))['displayText'] 308 | if 'description' in (json.loads(row[18])): 309 | strDescription = (json.loads(row[18]))['description'] 310 | if 'appDisplayName' in (json.loads(row[18])): 311 | strDisplayName = (json.loads(row[18]))['appDisplayName'] 312 | if 'contentUri' in (json.loads(row[18])): 313 | strContent = (json.loads(row[18]))['contentUri'] 314 | elif row[3] == 10: 315 | strContent = (json.loads(row[18]))['content'] # If error it might be [1]['content'] - Needs B64 decoding, but need example from live database 316 | #elseif($item.ActivityType -eq 10){[System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String(($item.Payload|ConvertFrom-Json)."1".content))} 317 | if row[3] == 2: 318 | strNotification = row[18] 319 | if (json.loads(row[1]))[0]['platform'] == "afs_crossplatform": 320 | strPlatform = (json.loads(row[1]))[1]['platform'] 321 | else: 322 | strPlatform = (json.loads(row[1]))[0]['platform'] 323 | if row[1]: 324 | if "afs_crossplatform" in (json.loads(row[1]))[0]['platform']: 325 | strSynched = "Yes" 326 | if row[3] == 10: 327 | strCopiedText = base64.b64decode((json.loads(row[9]))[0]['content']) #Convert from base64 328 | #$clipboard = if($item.ActivityType -in (10)){[System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String(($item.ClipboardPayload|ConvertFrom-Json).content))} 329 | acttypelist = [2,3,11,12,15] 330 | if row[3] in acttypelist: 331 | strAppName = (json.loads(row[1]))[0]['application'] 332 | else: 333 | if (json.loads(row[1]))[0]['platform'] == "x_exe_path": 334 | strAppName = (json.loads(row[1]))[0]['application'] 335 | elif (json.loads(row[1]))[0]['platform'] == "windows_win32": 336 | strAppName = (json.loads(row[1]))[0]['application'] 337 | elif (json.loads(row[1]))[0]['platform'] == "windows_universal": 338 | strAppName = (json.loads(row[1]))[0]['application'] 339 | elif (json.loads(row[1]))[1]['platform'] == "x_exe_path": 340 | strAppName = (json.loads(row[1]))[1]['application'] 341 | elif (json.loads(row[1]))[1]['platform'] == "windows_win32": 342 | strAppName = (json.loads(row[1]))[1]['application'] 343 | elif (json.loads(row[1]))[1]['platform'] == "windows_universal": 344 | strAppName = (json.loads(row[1]))[1]['application'] 345 | elif (json.loads(row[1]))[2]['platform'] == "x_exe_path": 346 | strAppName = (json.loads(row[1]))[2]['application'] 347 | elif (json.loads(row[1]))[2]['platform'] == "windows_win32": 348 | strAppName = (json.loads(row[1]))[2]['application'] 349 | elif (json.loads(row[1]))[2]['platform'] == "windows_universal": 350 | strAppName = (json.loads(row[1]))[2]['application'] 351 | # Replace app guid with name from the known list 352 | for k,v in known.items(): 353 | y = strAppName.split(k) 354 | strAppName = v.join(y) 355 | # Output "" if date is 1970 356 | if row[13] == '1970-01-01 01:00:00': 357 | strEndTime = "" 358 | else: 359 | strEndTime = row[13] 360 | if row[3] == 5 and strContent: #if activitytype 5 and data in payload 361 | rxuri = re.compile(r"^file://(.*?)\?") 362 | resulturi = rxuri.search(strContent) 363 | if resulturi is not None: 364 | strContentUrl = resulturi.group(0) 365 | strContentUrl = strContentUrl.rstrip("?") 366 | rxvolid = re.compile(r"VolumeId={(.*?)}") 367 | resultvolid = rxvolid.search(strContent) 368 | if resultvolid is not None: 369 | strVolumeID = resultvolid.group(1) 370 | rxobjid = re.compile(r"ObjectId={(.*?)}") 371 | resultobjit = rxobjid.search(strContent) 372 | if resultobjit is not None: 373 | strObjectId = resultobjit.group(1) 374 | rxknownfolder = re.compile(r"KnownFolderId=(.*?)\&") 375 | resultknownfolder = rxknownfolder.search(strContent) 376 | if resultknownfolder is not None: 377 | strKnownFolder = resultknownfolder.group(1) 378 | if strContentUrl and strContent: 379 | strContent = strContentUrl 380 | if row[3] == 3 and row[18]: 381 | if 'backupType' in (json.loads(row[18])): 382 | strBackupType = (json.loads(row[18]))['backupType'] 383 | if 'deviceName' in (json.loads(row[18])): 384 | strDeviceName = (json.loads(row[18]))['deviceName'] 385 | if 'deviceIdentifier' in (json.loads(row[18])): 386 | strDeviceIdentifier = (json.loads(row[18]))['deviceIdentifier'] 387 | if 'creationDate' in (json.loads(row[18])): 388 | strCreationDate = (json.loads(row[18]))['creationDate'] 389 | if 'updateDate' in (json.loads(row[18])): 390 | strBackupUpdated = (json.loads(row[18]))['updateDate'] 391 | if row[3] == 2: 392 | strActivityType = "Notification (2)" 393 | elif row[3] == 3: 394 | strActivityType = "Mobile Device Backup (3)" 395 | elif row[3] == 5: 396 | strActivityType = "Open App/File/Page (5)" 397 | elif row[3] == 6: 398 | strActivityType = "App In Use/Focus (6)" 399 | elif row[3] == 10: 400 | strActivityType = "Clipboard Text (10)" 401 | elif row[3] == 11: 402 | strActivityType = "System " + row[3] 403 | elif row[3] == 12: 404 | strActivityType = "System " + row[3] 405 | elif row[3] == 15: 406 | strActivityType = "System " + row[3] 407 | elif row[3] == 16: 408 | strActivityType = "Copy/Paste (16)" 409 | else: 410 | strActivityType = row[3] 411 | diclist.append({ 412 | 'Etag':strEtag, 413 | 'App_name':strAppName, 414 | 'DisplayName':strDisplayName, 415 | 'DisplayText':strDisplayText, 416 | 'Description':strDescription, 417 | 'AppActivityId':strAppActivityId, 418 | 'Content':strContent, 419 | 'VolumeID':strVolumeID, 420 | 'ObjectId':strObjectId, 421 | 'KnownFolder':strKnownFolder, 422 | 'Group':strGroup, 423 | 'MatchID':strMatchID, 424 | 'Tag':strTag, 425 | 'Type':strType, 426 | 'ActivityType':strActivityType, 427 | 'ActivityStatus':strActivityStatus, 428 | 'DevicePlatform':strDevicePlatform, 429 | 'Platform':strPlatform, 430 | 'IsInUploadQueue':strIsInUploadQueue, 431 | 'Synched':strSynched, 432 | 'Priority':strPriority, 433 | 'CopiedText':strCopiedText, 434 | 'Notification':strNotification, 435 | 'Duration':strDuration, 436 | #'CalculatedDuration':strCalculatedDuration, 437 | 'LastModifiedTime':strLastModifiedTime, 438 | 'ExpirationTime':strExpirationTime, 439 | 'StartTime':strStartTime, 440 | 'EndTime':strEndTime, 441 | 'CreatedInCloud':strCreatedInCloud, 442 | 'OriginalLastModifiedOnClient':strOriginalLastModifiedOnClient, 443 | 'TimeZone':strTimeZone, 444 | 'PlatformDeviceId':strPlatformDeviceId, 445 | 'DeviceType':strDeviceType, 446 | 'Name':strName, 447 | 'Make':strMake, 448 | 'Model':strModel, 449 | 'DeviceModel':strDeviceName, 450 | 'DeviceID':strDeviceIdentifier, 451 | 'BackupType':strBackupType, 452 | 'BackupCreated':strCreationDate, 453 | 'BackupUpdated':strBackupUpdated}) 454 | 455 | # Define Dataframe 456 | df = pd.DataFrame(diclist) 457 | 458 | if not len(df): 459 | sys.exit('The database is empty!') 460 | 461 | # Set datetime format and UTC 462 | df['StartTime'] = pd.to_datetime(df['StartTime'],utc=True) 463 | df['EndTime'] = pd.to_datetime(df['EndTime'],utc=True) 464 | df['LastModifiedTime'] = pd.to_datetime(df['LastModifiedTime'],utc=True) 465 | df['ExpirationTime'] = pd.to_datetime(df['ExpirationTime'],utc=True) 466 | df['CreatedInCloud'] = pd.to_datetime(df['CreatedInCloud'],utc=True) 467 | df['Duration'] = pd.to_numeric(df['Duration']) 468 | 469 | if only_gen_main: 470 | df.to_csv(os.path.join(outfolder, 'gen_report_exported_database.csv'), index = False) 471 | print("Succesfully exported full raw database report") 472 | 473 | else: 474 | df.to_csv(os.path.join(outfolder, 'gen_report_exported_database.csv'), index = False) 475 | print("Succesfully exported full raw database report") 476 | # Get TimeZone Name to set 477 | timezonedf = df['TimeZone'] 478 | nan_value = float("NaN") 479 | timezonedf.replace("", nan_value, inplace=True) 480 | timezone = timezonedf.dropna().iloc[0] 481 | 482 | # Convert to the users Timezone 483 | df['StartTime'] = df['StartTime'].dt.tz_convert(timezone) 484 | df['EndTime'] = df['EndTime'].dt.tz_convert(timezone) 485 | df['LastModifiedTime'] = df['LastModifiedTime'].dt.tz_convert(timezone) 486 | df['ExpirationTime'] = df['ExpirationTime'].dt.tz_convert(timezone) 487 | df['CreatedInCloud'] = df['CreatedInCloud'].dt.tz_convert(timezone) 488 | 489 | ##### REPORT GEN ##### 490 | # Report to show when the user is active 491 | # Export user start and end activity per day 492 | 493 | d = defaultdict(list) 494 | 495 | # Get StartTime Column from the dataframe 496 | temp = df['StartTime'].astype(str).tolist() 497 | 498 | # Get EndTime Column from the dataframe 499 | temp2 = df['EndTime'].astype(str).tolist() 500 | 501 | #Combine StartTime and EndTime into a new list 502 | temp3 = temp + temp2 503 | 504 | # Remove the NaT word from the list 505 | datelist = list(filter(lambda a: a != "NaT", temp3)) 506 | 507 | for dte in datelist: 508 | key, _ = dte.split() 509 | d[key].append(dte) 510 | 511 | # Find min and max 512 | with open(os.path.join(outfolder, 'gen_report_useractivity_start_and_end.csv'), 'w', newline='\n') as csvfile: 513 | csvwriter = csv.writer(csvfile) 514 | csvwriter.writerow(['First Time Entry', 'Last Time Entry']) 515 | for k,v in d.items(): 516 | csvwriter.writerow((min(v), max(v))) 517 | 518 | # Sort the csv and output it 519 | csvData = pd.read_csv(os.path.join(outfolder, "gen_report_useractivity_start_and_end.csv"), parse_dates=['First Time Entry','Last Time Entry']) 520 | csvData.sort_values(["First Time Entry"], 521 | axis=0, 522 | ascending=[False], 523 | inplace=True) 524 | csvData.to_csv(os.path.join(outfolder, 'gen_report_useractivity_start_and_end.csv'), index = False) 525 | print('Report gen_report_useractivity_start_and_end.csv Generated successfully') 526 | 527 | ##### REPORT GEN ##### 528 | # Report for When Applications are launched and details 529 | # Sorted on start time 530 | # StartTime, App_name, DisplayName, DisplayText, Description 531 | selection_openapp = df[ df['ActivityType'] == 'Open App/File/Page (5)'] 532 | dg_procname_start = selection_openapp.copy() 533 | dg_procname_start = dg_procname_start[['StartTime','App_name','DisplayName','DisplayText','Description']] 534 | dg_procname_start.sort_values(by=['StartTime'],inplace=True, ascending=False) 535 | dg_procname_start.to_csv(os.path.join(outfolder, 'gen_report_ApplicationLaunch_StartTime.csv'), index = False) 536 | print('Report gen_report_ApplicationLaunch_StartTime.csv Generated successfully') 537 | 538 | ##### REPORT GEN ##### 539 | # TXT file that contains all the unique paths found in the database 540 | # Description, AppActivityId, Content 541 | dg_paths = df.copy() 542 | dg_paths = dg_paths[['Description','AppActivityId','Content']] 543 | 544 | temppath1 = df['Description'].astype(str).tolist() 545 | temppath2 = df['AppActivityId'].astype(str).tolist() 546 | temppath3 = df['Content'].astype(str).tolist() 547 | temppathlist = temppath1 + temppath2 + temppath3 548 | 549 | temppathlist[:] = [x for x in temppathlist if x] # Remove empty lines 550 | mylist = set() 551 | for element in temppathlist: 552 | x = re.search(r"\w:\\\.*|(http://|https://).*|\{.*\}.*|file://.*|\\\\\\.*\\\.*", element) 553 | if x: 554 | mylist.add(x.group()) 555 | myset = set(mylist) 556 | textfile = open(os.path.join(outfolder, "Paths_Unique.txt"), "w", encoding='utf-8') 557 | for item in myset: 558 | textfile.write(str(item) + "\n") 559 | textfile.close() 560 | print('Paths_Unique.txt Generated successfully') 561 | 562 | ##### REPORT GEN ##### 563 | # Most used Applications 564 | # Lists out application and how long it has been actively in use in seconds 565 | 566 | # # Only select rows related to usage 567 | selection_activeapps = df[ df['ActivityType'] == 'App In Use/Focus (6)'] 568 | 569 | # # Copy dataframe 570 | active_apps = selection_activeapps.copy() 571 | active_apps = active_apps[['App_name','DisplayName','Duration']] 572 | 573 | # Sum the Activity column 574 | active_apps = active_apps.groupby('App_name').sum().groupby(level=[0]).sum() 575 | active_apps.sort_values(by=['Duration'],inplace=True, ascending=False) 576 | active_apps.to_csv(os.path.join(outfolder, 'gen_report_Activity_Applications.csv')) 577 | print('Report gen_report_Activity_Applications.csv Generated successfully') 578 | 579 | # ##### CHART GEN ##### 580 | # # HEATMAP For user Activity 581 | # # Chart that shows when the user starts his day and stops 582 | 583 | # # Only get app in use activites 584 | adf = df[df['ActivityType'] == "App In Use/Focus (6)"].copy() 585 | 586 | # Add day column 587 | adf['day'] = adf['StartTime'].dt.day_name() 588 | 589 | # Add hour column 590 | adf['hour'] = adf['StartTime'].dt.hour 591 | 592 | # Sort weekdays 593 | weekdays_order = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"] 594 | adf.day = pd.Categorical(adf.day,categories=weekdays_order) 595 | 596 | # Generate pivot table from data 597 | heatmap_data = pd.pivot_table(adf, values='Duration', index='day', columns='hour') 598 | 599 | # # Use Pivot table in heatmap 600 | sns.heatmap(heatmap_data, cmap="cool") 601 | plt.xlabel("Hour", size=24) 602 | plt.ylabel("Day", size=7) 603 | plt.title("User Activity (user's Timezone)", size=14) 604 | plt.tight_layout() 605 | 606 | # # Output heatmap to jpg file 607 | plt.savefig(os.path.join(outfolder, 'gen_fig_useractivity_heatmap.jpg'),dpi=300) 608 | print('Chart gen_fig_useractivity_heatmap.jpg Generated successfully') 609 | 610 | # ##### CHART GEN ##### 611 | # # Chart that shows when user is active 612 | df1 = pd.DataFrame() 613 | df1['Date'] = df['StartTime'].dt.strftime('%Y-%m-%d') 614 | 615 | df1['StartTime'] = df['StartTime'].dt.strftime('%H%M') 616 | df1['EndTime'] = df['EndTime'].dt.strftime('%H%M') 617 | df1.EndTime.fillna(df1.StartTime, inplace=True) 618 | df1['FirstActivityStart'] = df1.groupby('Date')['StartTime'].transform('min').astype(int) 619 | df1['LastActivityStart'] = df1.groupby('Date')['EndTime'].transform('max').astype(int) 620 | df1['Filler'] = 2400 621 | 622 | del df1['StartTime'] 623 | del df1['EndTime'] 624 | df1 = df1.sort_values(by='Date', ascending=True) 625 | df1.drop_duplicates(inplace=True) 626 | df1 = df1.reset_index(drop=True) 627 | fig = plt.figure() 628 | ax1 = fig.add_axes([0,0,1,1]) 629 | ax1.bar(df1['Date'],df1['FirstActivityStart'], color = '#004c6d', width = 0.35, zorder=3) 630 | ax1.bar(df1['Date'],df1['LastActivityStart'], color = '#5cceff', width = 0.35, zorder=2) 631 | ax1.bar(df1['Date'],df1['Filler'], color = '#004c6d', width = 0.35, zorder=1) 632 | 633 | ax1.legend(labels=['Idle', 'Active']) 634 | ax1.set_xticklabels(df1['Date'], rotation=90) 635 | fig.savefig(os.path.join(outfolder, 'gen_fig_useractivity_bar.jpg'), bbox_inches='tight',dpi=300) 636 | print('Chart gen_fig_useractivity_bar.jpg Generated successfully') 637 | 638 | # ##### CHART GEN ##### 639 | # # Most used Applications 640 | # # Lists out application and how long it has been actively in use in seconds 641 | top10 = active_apps.head(10) 642 | plot = top10.plot.pie(y='Duration', figsize=(5, 5), legend=False, title="Top 10 Applications", ylabel='') 643 | plt.savefig(os.path.join(outfolder, 'gen_fig_top10_apps_pie.jpg'), bbox_inches='tight',dpi=300) 644 | print('Chart gen_fig_top10_apps_pie.jpg Generated successfully') 645 | 646 | plot = top10.plot.barh(y='Duration', figsize=(5, 5), legend=False, title="Top 10 Applications", ylabel='') 647 | plt.savefig(os.path.join(outfolder, 'gen_fig_top10_apps_bars.jpg'), bbox_inches='tight',dpi=300) 648 | print('Chart gen_fig_top10_apps_bars.jpg Generated successfully') 649 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | pandas 2 | matplotlib 3 | seaborn 4 | termcolor 5 | --------------------------------------------------------------------------------