├── Readme.md ├── extract_index.py ├── real_thinking_time.py ├── real_thinking_time_points ├── web_traffic_generator.py └── web_traffic_generator_firebug.py /Readme.md: -------------------------------------------------------------------------------- 1 | Web Traffic Generator 2 | ===================== 3 | 4 | ## 1. Description of the software 5 | This tool takes as input a list of web pages and visits them by means of Selenium Web Browser (relying on Firefox). 6 | It produces as output the HARs generated by each page. 7 | HAR is the [HTTP Archive](http://www.softwareishard.com/blog/har-12-spec/), a JSON data structure containing 8 | information and performance statistics about the webpage download. 9 | 10 | For information about this Readme file and this tool please write to [martino.trevisan@polito.it](mailto:martino.trevisan@polito.it) 11 | 12 | ## 2. Dependencies 13 | This software has been tested under Ubuntu Linux and Mac OSX. You need python3 and Firefox installed. 14 | Please use Firefox <= 46.0. With >= 47.0 this software might not work. 15 | To install Firefox 46 follow [this](http://www.askmetutorials.com/2016/04/install-firefox-46-on-ubuntu-1604-1404.html) link. 16 | 17 | These packages are required: 18 | ``` 19 | python3 python3-pip 20 | ``` 21 | 22 | 23 | You must install Selenium python3 package as well as `numpy` and `scipy` 24 | ``` 25 | sudo pip3 install selenium numpy scipy 26 | ``` 27 | 28 | To run Firefox within a virtual display you need the package `xvfb` and its python package. 29 | In this way you can run this tool in a machine without a X session active (e.g., from `ssh`). 30 | Run these commands: 31 | ``` 32 | sudo apt-get install xvfb 33 | sudo pip3 install pyvirtualdisplay 34 | ``` 35 | 36 | If you can't install pip3, you can do it following 37 | [this](http://stackoverflow.com/questions/6587507/how-to-install-pip-with-python-3) link. 38 | 39 | Finally, you must download the Har Export Trigger extension for Firerfox, available [here](http://www.softwareishard.com/blog/har-export-trigger/). 40 | You must only download the *.xpi file; you don't need to install the extension in your Firefox. 41 | A fast way to retrieve it is to run the command: 42 | 43 | ``` 44 | wget https://github.com/firebug/har-export-trigger/releases/download/harexporttrigger-0.5.0-beta.10/harexporttrigger-0.5.0-beta.10.xpi 45 | ``` 46 | 47 | ## 3. Usage 48 | To run this tool, you must execute this command line: 49 | ``` 50 | usage: ./web_traffic_generator.py [-h] [-e har_export] [-r real_backoff] 51 | [-b static_backoff] [-t timeout] 52 | [-s start_page] [-v] 53 | input_file out_dir 54 | ``` 55 | 56 | Please don't execute it as an argument of the python3 interpreter. 57 | So `python3 web_traffic_generator.py` does NOT work. 58 | 59 | positional arguments: 60 | * `input_file` File where are stored the pages, one per row 61 | * `output_dir` Output directory where HAR files are saved, one for each page. 62 | 63 | optional arguments: 64 | * `-h, --help` show this help message and exit 65 | * `-e har_export, --har_export har_export` 66 | Path to Har Export extension xpi file. If not set, 67 | searches it in the code path (the same path of the web_traffic_generator.py file). 68 | * `-r real_backoff, --real_backoff real_backoff` 69 | Use real backoff distribution with maximum value 70 | seconds 71 | * `-b static_backoff, --static_backoff static_backoff` 72 | Use a static (always the same) backoff with value 73 | seconds 74 | * `-t timeout, --timeout timeout` 75 | Timeout in seconds after declaring failed a visit. 76 | Default is 30. 77 | * `-v, --virtual_display` 78 | Use a virtual display instead of the physical one. Requires `xvfb` package. 79 | 80 | ## 4. Output format 81 | This tool creates an output file where it stores the output HAR of each requested URL. 82 | For each input URL, the corresponding HAR file created in the provided directory. 83 | The name of the HAR file reflects the time when the visit was performed and requested domain (e.g., `visit_2016_06_15_13_41_22_www_google_it.har`). 84 | Some HAR could be missing, due to failed downloads (very slow pages, crashed browser ecc...) 85 | 86 | In the HAR file, at the position "log" -> "pages" -> "title", you can find a string representing the OnLoad time and 87 | the original requested URL, separated by space, for example 88 | 89 | `4.22002387046814 http://www.libero.it` 90 | 91 | The OnLoad measure is important for performance, while knowing the actual requested URL can be useful for complicated web pages. 92 | 93 | 94 | -------------------------------------------------------------------------------- /extract_index.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | import json 3 | from dateutil import parser 4 | import operator 5 | import datetime 6 | 7 | import sys 8 | 9 | input_har = json.load(open(sys.argv[1], "r")) 10 | 11 | 12 | objects=[] 13 | bytes=[] 14 | tot_bytes=0 15 | tot_obj=0 16 | 17 | start_time=parser.parse(input_har["log"]["entries"][0]["startedDateTime"]).timestamp() 18 | 19 | domain=input_har["log"]["pages"][0]["title"].split("/")[2] 20 | on_load_time=input_har["log"]["pages"][0]["pageTimings"]["onLoad"] 21 | DOM_load_time=input_har["log"]["pages"][0]["pageTimings"]["onContentLoad"] 22 | 23 | for entry in input_har["log"]["entries"]: 24 | 25 | obj_start_time=parser.parse(entry["startedDateTime"]).timestamp() 26 | obj_delta_time= sum( [ v for v in entry["timings"].values() if v != -1]) 27 | obj_time = obj_start_time - start_time + obj_delta_time/1000 28 | size = entry["response"]["bodySize"] 29 | objects.append( (obj_time, 1 )) 30 | bytes.append ( (obj_time, size)) 31 | tot_bytes+=size 32 | tot_obj+=1 33 | last_obj_time=obj_time 34 | 35 | 36 | 37 | ''' 38 | #should return 4 39 | tot_bytes=tot_obj=4 40 | bytes=( (1,1),(3,1),(4,1),(8,1) ) 41 | objects=( (1,1),(3,1),(4,1),(8,1) ) 42 | 43 | objects=sorted(objects,key=operator.itemgetter(0)) 44 | bytes=sorted(bytes,key=operator.itemgetter(0)) 45 | ''' 46 | 47 | cumul_bytes=0 48 | cumul_objects=0 49 | for i in range (len(bytes)): 50 | time, size = bytes[i] 51 | bytes[i] = (time, size/tot_bytes + cumul_bytes) 52 | objects[i] = (time, 1/tot_obj + cumul_objects) 53 | cumul_bytes=size/tot_bytes + cumul_bytes 54 | cumul_objects=1/tot_obj + cumul_objects 55 | 56 | bytes = [(t, round(1-b,5)) for t,b in bytes] 57 | objects = [(t, round(1-b,5)) for t,b in objects] 58 | 59 | 60 | prec_score=1 61 | prec_time=0 62 | byte_index=0 63 | 64 | for time, score in bytes: 65 | component=(time-prec_time)*prec_score 66 | byte_index+=component 67 | prec_score=score 68 | prec_time = time 69 | 70 | prec_score=1 71 | prec_time=0 72 | object_index=0 73 | 74 | for time, score in objects: 75 | component=(time-prec_time)*prec_score 76 | object_index+=component 77 | prec_score=score 78 | prec_time = time 79 | 80 | print(domain, byte_index, object_index, on_load_time/1000, DOM_load_time/1000 ) 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | -------------------------------------------------------------------------------- /real_thinking_time.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | import numpy as np 4 | import os 5 | from scipy.interpolate import interp1d 6 | import random 7 | 8 | input_file="real_thinking_time_points" 9 | 10 | x = [ float(row.split()[0]) for row in open (os.path.dirname(__file__)+ "/" + input_file, "r") ] 11 | y = [ float(row.split()[1]) for row in open (os.path.dirname(__file__)+ "/" + input_file, "r") ] 12 | 13 | f = interp1d(x, y) 14 | 15 | def random_thinking_time(max_value=300): 16 | 17 | time = float(f(random.random()))/1000000 18 | if time > max_value: 19 | return max_value 20 | else: 21 | return time 22 | 23 | 24 | 25 | -------------------------------------------------------------------------------- /real_thinking_time_points: -------------------------------------------------------------------------------- 1 | 0.0 0.0 2 | 0.0 1000000.0 3 | 0.000557 1013925.40756 4 | 0.000976 1028044.73209 5 | 0.001394 1042360.67398 6 | 0.001951 1056875.97118 7 | 0.002509 1071593.39982 8 | 0.003066 1086515.77465 9 | 0.003624 1101645.94963 10 | 0.004181 1116986.81847 11 | 0.005017 1132541.31515 12 | 0.005296 1148312.41454 13 | 0.005993 1164303.13292 14 | 0.006969 1180516.52857 15 | 0.007387 1196955.70236 16 | 0.008084 1213623.79834 17 | 0.008362 1230524.00436 18 | 0.00892 1247659.55263 19 | 0.010174 1265033.7204 20 | 0.010453 1282649.83053 21 | 0.01101 1300511.25217 22 | 0.011568 1318621.4014 23 | 0.011986 1336983.74182 24 | 0.012683 1355601.78533 25 | 0.01338 1374479.09268 26 | 0.014216 1393619.27422 27 | 0.014634 1413025.9906 28 | 0.015331 1432702.95341 29 | 0.016725 1452653.92595 30 | 0.017422 1472882.72391 31 | 0.017979 1493393.21612 32 | 0.019094 1514189.3253 33 | 0.019652 1535275.02878 34 | 0.020348 1556654.35927 35 | 0.021185 1578331.40565 36 | 0.0223 1600310.31374 37 | 0.023693 1622595.28708 38 | 0.02439 1645190.58775 39 | 0.026341 1668100.5372 40 | 0.026481 1691329.51703 41 | 0.027178 1714881.96987 42 | 0.027875 1738762.40022 43 | 0.029129 1762975.37529 44 | 0.029826 1787525.5259 45 | 0.030523 1812417.54737 46 | 0.031777 1837656.20039 47 | 0.032474 1863246.31193 48 | 0.033868 1889192.77621 49 | 0.03554 1915500.55557 50 | 0.036934 1942174.68149 51 | 0.038188 1969220.25548 52 | 0.040279 1996642.45011 53 | 0.042091 2024446.50998 54 | 0.044321 2052637.75271 55 | 0.045436 2081221.56999 56 | 0.046272 2110203.42857 57 | 0.048084 2139588.87134 58 | 0.049338 2169383.51839 59 | 0.051847 2199593.06803 60 | 0.053798 2230223.29797 61 | 0.056167 2261280.06634 62 | 0.058537 2292769.31287 63 | 0.060488 2324697.05999 64 | 0.062997 2357069.414 65 | 0.067038 2389892.56623 66 | 0.069268 2423172.79424 67 | 0.070523 2456916.46298 68 | 0.073728 2491130.02607 69 | 0.076237 2525820.02696 70 | 0.080418 2560993.10026 71 | 0.082369 2596655.97293 72 | 0.08446 2632815.46565 73 | 0.087247 2669478.49403 74 | 0.089338 2706652.07003 75 | 0.092265 2744343.30323 76 | 0.094355 2782559.40221 77 | 0.0977 2821307.67594 78 | 0.100627 2860595.53518 79 | 0.102718 2900430.49386 80 | 0.103693 2940820.17059 81 | 0.10662 2981772.29002 82 | 0.109686 3023294.68441 83 | 0.11122 3065395.29506 84 | 0.114146 3108082.17387 85 | 0.116655 3151363.48487 86 | 0.119164 3195247.50576 87 | 0.121533 3239742.62953 88 | 0.123902 3284857.36603 89 | 0.126551 3330600.34363 90 | 0.129617 3376980.31083 91 | 0.133101 3424006.13797 92 | 0.136725 3471686.81893 93 | 0.140348 3520031.4728 94 | 0.1423 3569049.34567 95 | 0.145087 3618749.81241 96 | 0.14899 3669142.3784 97 | 0.15122 3720236.68141 98 | 0.153031 3772042.49342 99 | 0.156934 3824569.72247 100 | 0.159164 3877828.41459 101 | 0.161672 3931828.75571 102 | 0.163902 3986581.07358 103 | 0.168084 4042095.8398 104 | 0.169756 4098383.67176 105 | 0.171986 4155455.33472 106 | 0.175331 4213321.74385 107 | 0.178676 4271993.96631 108 | 0.181463 4331483.22338 109 | 0.18439 4391800.8926 110 | 0.186899 4452958.50994 111 | 0.189547 4514967.77204 112 | 0.192474 4577840.53838 113 | 0.194983 4641588.83361 114 | 0.198885 4706224.84984 115 | 0.200697 4771760.94894 116 | 0.202787 4838209.66493 117 | 0.207526 4905583.70637 118 | 0.209895 4973895.95879 119 | 0.21338 5043159.48717 120 | 0.216307 5113387.53841 121 | 0.220627 5184593.54389 122 | 0.223275 5256791.12202 123 | 0.225645 5329994.08084 124 | 0.22885 5404216.42071 125 | 0.232892 5479472.3369 126 | 0.236376 5555776.2224 127 | 0.239721 5633142.6706 128 | 0.242787 5711586.47813 129 | 0.245993 5791122.64764 130 | 0.249477 5871766.39073 131 | 0.252962 5953533.13081 132 | 0.254774 6036438.50608 133 | 0.259094 6120498.37248 134 | 0.263415 6205728.80678 135 | 0.26676 6292146.10961 136 | 0.270523 6379766.80861 137 | 0.273868 6468607.66155 138 | 0.27777 6558685.65957 139 | 0.280836 6650018.03043 140 | 0.284042 6742622.24178 141 | 0.286411 6836516.00451 142 | 0.289617 6931717.27615 143 | 0.29338 7028244.26431 144 | 0.297561 7126115.43011 145 | 0.29993 7225349.49179 146 | 0.30216 7325965.42821 147 | 0.306202 7427982.48256 148 | 0.308293 7531420.16597 149 | 0.311916 7636298.26128 150 | 0.316376 7742636.82681 151 | 0.320279 7850456.20021 152 | 0.32223 7959777.00231 153 | 0.326272 8070620.14115 154 | 0.329477 8183006.81587 155 | 0.333659 8296958.52084 156 | 0.338537 8412497.04974 157 | 0.343554 8529644.49974 158 | 0.347178 8648423.27573 159 | 0.351638 8768856.09459 160 | 0.354983 8890965.98953 161 | 0.359024 9014776.31453 162 | 0.361672 9140310.74876 163 | 0.36669 9267593.30115 164 | 0.370314 9396648.31495 165 | 0.375192 9527500.47243 166 | 0.37784 9660174.79952 167 | 0.381185 9794696.67069 168 | 0.384808 9931091.81375 169 | 0.387735 10069386.3148 170 | 0.390801 10209606.6231 171 | 0.394843 10351779.5563 172 | 0.398328 10495932.3056 173 | 0.402091 10642092.4406 174 | 0.40669 10790287.9152 175 | 0.41101 10940547.0721 176 | 0.415052 11092898.649 177 | 0.419233 11247371.7836 178 | 0.423554 11403996.0197 179 | 0.427178 11562801.3121 180 | 0.430941 11723818.0329 181 | 0.435679 11887076.9771 182 | 0.438746 12052609.3687 183 | 0.442648 12220446.8663 184 | 0.448502 12390621.5695 185 | 0.452404 12563166.0247 186 | 0.456446 12738113.2319 187 | 0.46007 12915496.6501 188 | 0.4623 13095350.2048 189 | 0.466341 13277708.2936 190 | 0.470244 13462605.793 191 | 0.474843 13650078.0655 192 | 0.478467 13840160.9657 193 | 0.482927 14032890.8479 194 | 0.486551 14228304.5721 195 | 0.491568 14426439.5122 196 | 0.496307 14627333.562 197 | 0.500627 14831025.1434 198 | 0.505366 15037553.213 199 | 0.507735 15246957.2702 200 | 0.511498 15459277.3642 201 | 0.516237 15674554.1021 202 | 0.520836 15892828.6562 203 | 0.524739 16114142.7725 204 | 0.528502 16338538.7781 205 | 0.532125 16566059.5895 206 | 0.536725 16796748.7209 207 | 0.541603 17030650.2925 208 | 0.546202 17267809.0388 209 | 0.550105 17508270.3174 210 | 0.556376 17752080.1172 211 | 0.563345 17999285.0678 212 | 0.567666 18249932.4482 213 | 0.571847 18504070.1954 214 | 0.57561 18761746.9144 215 | 0.579233 19023011.8867 216 | 0.584948 19287915.0802 217 | 0.589129 19556507.1587 218 | 0.591777 19828839.4913 219 | 0.596794 20104964.1626 220 | 0.601115 20384933.9825 221 | 0.606829 20668802.4963 222 | 0.609617 20956623.9948 223 | 0.614355 21248453.525 224 | 0.619791 21544346.9003 225 | 0.624948 21844360.7115 226 | 0.628293 22148552.3373 227 | 0.632334 22456979.9554 228 | 0.63554 22769702.5538 229 | 0.639443 23086779.9419 230 | 0.643624 23408272.7618 231 | 0.647247 23734242.5002 232 | 0.651289 24064751.5002 233 | 0.655749 24399862.9726 234 | 0.661463 24739641.0089 235 | 0.665784 25084150.5928 236 | 0.669965 25433457.613 237 | 0.674425 25787628.8759 238 | 0.679582 26146732.118 239 | 0.683066 26510836.0191 240 | 0.687108 26880010.2154 241 | 0.69115 27254325.3128 242 | 0.696028 27633852.9005 243 | 0.700767 28018665.5646 244 | 0.704669 28408836.9018 245 | 0.709129 28804441.534 246 | 0.71331 29205555.1218 247 | 0.717213 29612254.3799 248 | 0.723066 30024617.0909 249 | 0.728502 30442722.1206 250 | 0.732265 30866649.4334 251 | 0.735749 31296480.1067 252 | 0.738955 31732296.3474 253 | 0.741603 32174181.5068 254 | 0.744808 32622220.0971 255 | 0.749408 33076497.8074 256 | 0.752613 33537101.52 257 | 0.755958 34004119.327 258 | 0.758746 34477640.5473 259 | 0.761394 34957755.7436 260 | 0.764878 35444556.7397 261 | 0.76892 35938136.638 262 | 0.772822 36438589.8376 263 | 0.776167 36946012.052 264 | 0.779791 37460500.3275 265 | 0.782857 37982153.0619 266 | 0.785784 38511070.0233 267 | 0.789686 39047352.3689 268 | 0.792613 39591102.6647 269 | 0.794983 40142424.905 270 | 0.798467 40701424.5322 271 | 0.801533 41268208.457 272 | 0.805017 41842885.079 273 | 0.80892 42425564.3072 274 | 0.81115 43016357.5811 275 | 0.814774 43615377.8921 276 | 0.818537 44222739.8051 277 | 0.821185 44838559.4802 278 | 0.823415 45462954.6953 279 | 0.827178 46096044.8683 280 | 0.829686 46737951.0799 281 | 0.832056 47388796.0972 282 | 0.834564 48048704.3966 283 | 0.837073 48717802.1879 284 | 0.838606 49396217.4388 285 | 0.841254 50084079.8985 286 | 0.842927 50781521.1233 287 | 0.845575 51488674.5014 288 | 0.847666 52205675.2785 289 | 0.849895 52932660.5836 290 | 0.853101 53669769.4554 291 | 0.855192 54417142.8687 292 | 0.857143 55174923.7613 293 | 0.859512 55943257.0617 294 | 0.861742 56722289.7164 295 | 0.864111 57512170.7184 296 | 0.865784 58313051.1353 297 | 0.867735 59125084.1383 298 | 0.869408 59948425.0319 299 | 0.870383 60783231.283 300 | 0.871359 61629662.5513 301 | 0.873728 62487880.7201 302 | 0.874425 63358049.9266 303 | 0.875819 64240336.5939 304 | 0.877352 65134909.4627 305 | 0.879582 66041939.6233 306 | 0.880139 66961600.5485 307 | 0.881951 67894068.127 308 | 0.882927 68839520.6965 309 | 0.884181 69798139.0783 310 | 0.885993 70770106.6118 311 | 0.886969 71755609.1894 312 | 0.888223 72754835.292 313 | 0.890174 73767976.0253 314 | 0.891289 74795225.1562 315 | 0.892822 75836779.15 316 | 0.893798 76892837.2076 317 | 0.895192 77963601.3041 318 | 0.896864 79049276.227 319 | 0.897979 80150069.6157 320 | 0.899233 81266192.0009 321 | 0.90007 82397856.8453 322 | 0.901185 83545280.5838 323 | 0.902439 84708682.6656 324 | 0.903275 85888285.5955 325 | 0.90439 87084314.9769 326 | 0.905645 88296999.5549 327 | 0.906063 89526571.26 328 | 0.907317 90773265.2521 329 | 0.908571 92037319.9662 330 | 0.90899 93318977.1573 331 | 0.909408 94618481.9472 332 | 0.910244 95936082.8709 333 | 0.911359 97272031.9245 334 | 0.911777 98626584.6131 335 | 0.912892 100000000.0 336 | 0.91331 101392540.756 337 | 0.914564 102804473.209 338 | 0.916237 104236067.398 339 | 0.916516 105687597.118 340 | 0.917352 107159339.982 341 | 0.918049 108651577.465 342 | 0.919024 110164594.963 343 | 0.919303 111698681.847 344 | 0.92 113254131.515 345 | 0.920697 114831241.454 346 | 0.921812 116430313.292 347 | 0.922927 118051652.857 348 | 0.923484 119695570.236 349 | 0.923624 121362379.834 350 | 0.924321 123052400.436 351 | 0.925017 124765955.263 352 | 0.925575 126503372.04 353 | 0.926132 128264983.053 354 | 0.92669 130051125.217 355 | 0.926969 131862140.139 356 | 0.928223 133698374.182 357 | 0.928362 135560178.533 358 | 0.929477 137447909.268 359 | 0.929756 139361927.422 360 | 0.930035 141302599.06 361 | 0.930592 143270295.341 362 | 0.931568 145265392.595 363 | 0.931707 147288272.391 364 | 0.932404 149339321.612 365 | 0.932544 151418932.53 366 | 0.932683 153527502.878 367 | 0.93338 155665435.927 368 | 0.934077 157833140.565 369 | 0.934495 160031031.374 370 | 0.934634 162259528.708 371 | 0.935331 164519058.775 372 | 0.93561 166810053.72 373 | 0.935889 169132951.703 374 | 0.935889 171488196.987 375 | 0.936446 173876240.022 376 | 0.936585 176297537.529 377 | 0.936864 178752552.59 378 | 0.937003 181241754.737 379 | 0.937282 183765620.039 380 | 0.937422 186324631.193 381 | 0.938118 188919277.621 382 | 0.938397 191550055.557 383 | 0.938815 194217468.149 384 | 0.939094 196922025.548 385 | 0.939791 199664245.011 386 | 0.93993 202444650.998 387 | 0.940488 205263775.271 388 | 0.941324 208122156.999 389 | 0.941603 211020342.857 390 | 0.941882 213958887.134 391 | 0.94216 216938351.839 392 | 0.942578 219959306.803 393 | 0.942997 223022329.797 394 | 0.942997 226128006.634 395 | 0.943275 229276931.287 396 | 0.943554 232469705.999 397 | 0.943554 235706941.4 398 | 0.943833 238989256.623 399 | 0.943972 242317279.424 400 | 0.943972 245691646.298 401 | 0.944111 249113002.607 402 | 0.94439 252582002.696 403 | 0.94453 256099310.026 404 | 0.944948 259665597.293 405 | 0.945366 263281546.565 406 | 0.945505 266947849.403 407 | 0.945784 270665207.003 408 | 0.945923 274434330.323 409 | 0.946341 278255940.221 410 | 0.94662 282130767.594 411 | 0.947456 286059553.518 412 | 0.947596 290043049.386 413 | 0.947735 294082017.059 414 | 0.948153 298177229.002 415 | 0.948293 302329468.441 416 | 0.948432 306539529.506 417 | 0.948571 310808217.387 418 | 0.94899 315136348.487 419 | 0.94899 319524750.576 420 | 0.949129 323974262.953 421 | 0.949408 328485736.603 422 | 0.949408 333060034.362 423 | 0.949547 337698031.083 424 | 0.949547 342400613.797 425 | 0.949686 347168681.893 426 | 0.955679 352003147.28 427 | 0.955958 356904934.568 428 | 0.956098 361874981.241 429 | 0.956516 366914237.84 430 | 0.956934 372023668.141 431 | 0.957073 377204249.342 432 | 0.957352 382456972.247 433 | 0.95777 387782841.459 434 | 0.958049 393182875.571 435 | 0.958049 398658107.358 436 | 0.958188 404209583.98 437 | 0.958188 409838367.176 438 | 0.958746 415545533.472 439 | 0.958746 421332174.385 440 | 0.958885 427199396.631 441 | 0.959024 433148322.338 442 | 0.959024 439180089.26 443 | 0.959303 445295850.994 444 | 0.959303 451496777.204 445 | 0.959582 457784053.838 446 | 0.959721 464158883.361 447 | 0.960139 470622484.984 448 | 0.960139 477176094.894 449 | 0.960279 483820966.493 450 | 0.960279 490558370.637 451 | 0.960279 497389595.879 452 | 0.960279 504315948.717 453 | 0.960418 511338753.841 454 | 0.960557 518459354.389 455 | 0.960557 525679112.202 456 | 0.960557 532999408.084 457 | 0.960697 540421642.071 458 | 0.960976 547947233.69 459 | 0.961394 555577622.24 460 | 0.961394 563314267.06 461 | 0.961394 571158647.813 462 | 0.961394 579112264.764 463 | 0.961394 587176639.073 464 | 0.961951 595353313.081 465 | 0.96223 603643850.608 466 | 0.96223 612049837.248 467 | 0.962509 620572880.678 468 | 0.962509 629214610.961 469 | 0.962787 637976680.861 470 | 0.963206 646860766.155 471 | 0.963345 655868565.957 472 | 0.963345 665001803.043 473 | 0.963345 674262224.178 474 | 0.963345 683651600.451 475 | 0.963484 693171727.616 476 | 0.963484 702824426.431 477 | 0.963484 712611543.011 478 | 0.963763 722534949.179 479 | 0.963902 732596542.822 480 | 0.963902 742798248.256 481 | 0.963902 753142016.597 482 | 0.964042 763629826.128 483 | 0.964042 774263682.681 484 | 0.964181 785045620.02 485 | 0.964181 795977700.231 486 | 0.964181 807062014.115 487 | 0.964321 818300681.587 488 | 0.964739 829695852.083 489 | 0.964739 841249704.974 490 | 0.964739 852964449.974 491 | 0.964878 864842327.573 492 | 0.965017 876885609.459 493 | 0.965017 889096598.953 494 | 0.965296 901477631.452 495 | 0.965575 914031074.876 496 | 0.965714 926759330.115 497 | 0.965993 939664831.495 498 | 0.965993 952750047.243 499 | 0.965993 966017479.952 500 | 0.965993 979469667.07 501 | 0.966272 993109181.375 502 | 0.966411 1006938631.48 503 | 0.96669 1020960662.31 504 | 0.966829 1035177955.63 505 | 0.966969 1049593230.56 506 | 0.967108 1064209244.06 507 | 0.967108 1079028791.52 508 | 0.967247 1094054707.21 509 | 0.967247 1109289864.9 510 | 0.967387 1124737178.36 511 | 0.967387 1140399601.97 512 | 0.967387 1156280131.21 513 | 0.967387 1172381803.29 514 | 0.967666 1188707697.71 515 | 0.967805 1205260936.87 516 | 0.967944 1222044686.63 517 | 0.967944 1239062156.95 518 | 0.968084 1256316602.47 519 | 0.968084 1273811323.19 520 | 0.968084 1291549665.01 521 | 0.968084 1309535020.48 522 | 0.968084 1327770829.36 523 | 0.968223 1346260579.3 524 | 0.968223 1365007806.55 525 | 0.968223 1384016096.57 526 | 0.968362 1403289084.79 527 | 0.968641 1422830457.21 528 | 0.968641 1442643951.22 529 | 0.96892 1462733356.2 530 | 0.96892 1483102514.34 531 | 0.96892 1503755321.3 532 | 0.96892 1524695727.02 533 | 0.969199 1545927736.42 534 | 0.969338 1567455410.21 535 | 0.969617 1589282865.62 536 | 0.969756 1611414277.25 537 | 0.969756 1633853877.81 538 | 0.969756 1656605958.95 539 | 0.969895 1679674872.09 540 | 0.970035 1703065029.25 541 | 0.970174 1726780903.88 542 | 0.970453 1750827031.74 543 | 0.970871 1775208011.72 544 | 0.97115 1799928506.78 545 | 0.971568 1824993244.82 546 | 0.971568 1850407019.54 547 | 0.971568 1876174691.44 548 | 0.971568 1902301188.67 549 | 0.971847 1928791508.02 550 | 0.971847 1955650715.87 551 | 0.971986 1982883949.13 552 | 0.972125 2010496416.26 553 | 0.972265 2038493398.25 554 | 0.972265 2066880249.63 555 | 0.972404 2095662399.48 556 | 0.972404 2124845352.5 557 | 0.972544 2154434690.03 558 | 0.972544 2184436071.15 559 | 0.972544 2214855233.73 560 | 0.972544 2245697995.54 561 | 0.972683 2276970255.38 562 | 0.972683 2308677994.19 563 | 0.972683 2340827276.18 564 | 0.972822 2373424250.02 565 | 0.973101 2406475150.02 566 | 0.973101 2439986297.26 567 | 0.97324 2473964100.89 568 | 0.97338 2508415059.28 569 | 0.973519 2543345761.3 570 | 0.973659 2578762887.59 571 | 0.973798 2614673211.8 572 | 0.973798 2651083601.91 573 | 0.973798 2688001021.54 574 | 0.973798 2725432531.28 575 | 0.973937 2763385290.05 576 | 0.974216 2801866556.46 577 | 0.974216 2840883690.18 578 | 0.974495 2880444153.4 579 | 0.974634 2920555512.18 580 | 0.974774 2961225437.99 581 | 0.974913 3002461709.09 582 | 0.975052 3044272212.06 583 | 0.975331 3086664943.34 584 | 0.975331 3129648010.67 585 | 0.975331 3173229634.73 586 | 0.97561 3217418150.68 587 | 0.975749 3262222009.71 588 | 0.976028 3307649780.74 589 | 0.976446 3353710152.0 590 | 0.976585 3400411932.7 591 | 0.976725 3447764054.73 592 | 0.976725 3495775574.36 593 | 0.976864 3544455673.97 594 | 0.977282 3593813663.8 595 | 0.977282 3643858983.76 596 | 0.977422 3694601205.2 597 | 0.977422 3746050032.75 598 | 0.977561 3798215306.19 599 | 0.97784 3851107002.33 600 | 0.97784 3904735236.89 601 | 0.977979 3959110266.47 602 | 0.978258 4014242490.5 603 | 0.978397 4070142453.22 604 | 0.978397 4126820845.7 605 | 0.978397 4184288507.9 606 | 0.978397 4242556430.72 607 | 0.978397 4301635758.11 608 | 0.978397 4361537789.21 609 | 0.978537 4422273980.51 610 | 0.978815 4483855948.02 611 | 0.978815 4546295469.53 612 | 0.978955 4609604486.83 613 | 0.978955 4673795107.99 614 | 0.979233 4738879609.72 615 | 0.979233 4804870439.66 616 | 0.979373 4871780218.79 617 | 0.979373 4939621743.88 618 | 0.979512 5008407989.85 619 | 0.979512 5078152112.33 620 | 0.979791 5148867450.14 621 | 0.97993 5220567527.85 622 | 0.97993 5293266058.36 623 | 0.98007 5366976945.54 624 | 0.980209 5441714286.87 625 | 0.980209 5517492376.13 626 | 0.980209 5594325706.17 627 | 0.980209 5672228971.64 628 | 0.980209 5751217071.84 629 | 0.980348 5831305113.53 630 | 0.980488 5912508413.83 631 | 0.980488 5994842503.19 632 | 0.980627 6078323128.3 633 | 0.980627 6162966255.13 634 | 0.980627 6248788072.01 635 | 0.980906 6335804992.66 636 | 0.980906 6424033659.39 637 | 0.981045 6513490946.27 638 | 0.981185 6604193962.33 639 | 0.981324 6696160054.85 640 | 0.981463 6789406812.7 641 | 0.981463 6883952069.65 642 | 0.981463 6979813907.83 643 | 0.981463 7077010661.18 644 | 0.981463 7175560918.94 645 | 0.981463 7275483529.2 646 | 0.981463 7376797602.53 647 | 0.981742 7479522515.62 648 | 0.981882 7583677915.0 649 | 0.982021 7689283720.76 650 | 0.98216 7796360130.41 651 | 0.98216 7904927622.7 652 | 0.98216 8015006961.57 653 | 0.98216 8126619200.09 654 | 0.982439 8239785684.53 655 | 0.982718 8354528058.38 656 | 0.982997 8470868266.56 657 | 0.983136 8588828559.55 658 | 0.983136 8708431497.69 659 | 0.983275 8829699955.49 660 | 0.983275 8952657126.0 661 | 0.983275 9077326525.21 662 | 0.983415 9203731996.62 663 | 0.983415 9331897715.73 664 | 0.983693 9461848194.72 665 | 0.983693 9593608287.09 666 | 0.983833 9727203192.45 667 | 0.983972 9862658461.31 668 | 0.983972 10000000000.0 669 | 0.984111 10139254075.6 670 | 0.98439 10280447320.9 671 | 0.98439 10423606739.8 672 | 0.98439 10568759711.8 673 | 0.98453 10715933998.2 674 | 0.98453 10865157746.5 675 | 0.98453 11016459496.3 676 | 0.98453 11169868184.7 677 | 0.98453 11325413151.5 678 | 0.98453 11483124145.4 679 | 0.984948 11643031329.2 680 | 0.984948 11805165285.7 681 | 0.984948 11969557023.6 682 | 0.985087 12136237983.4 683 | 0.985087 12305240043.6 684 | 0.985087 12476595526.3 685 | 0.985366 12650337204.0 686 | 0.985505 12826498305.3 687 | 0.985505 13005112521.7 688 | 0.985505 13186214013.9 689 | 0.985645 13369837418.2 690 | 0.985645 13556017853.3 691 | 0.985645 13744790926.8 692 | 0.985645 13936192742.2 693 | 0.985784 14130259906.0 694 | 0.985784 14327029534.1 695 | 0.985923 14526539259.5 696 | 0.986063 14728827239.1 697 | 0.986063 14933932161.2 698 | 0.986063 15141893253.0 699 | 0.986202 15352750287.8 700 | 0.986202 15566543592.7 701 | 0.986341 15783314056.5 702 | 0.986481 16003103137.4 703 | 0.986481 16225952870.8 704 | 0.986481 16451905877.5 705 | 0.98662 16681005372.0 706 | 0.98676 16913295170.3 707 | 0.987038 17148819698.7 708 | 0.987038 17387624002.2 709 | 0.987038 17629753752.9 710 | 0.987317 17875255259.0 711 | 0.987596 18124175473.7 712 | 0.987596 18376562003.9 713 | 0.987596 18632463119.3 714 | 0.987735 18891927762.1 715 | 0.987875 19155005555.7 716 | 0.988014 19421746814.9 717 | 0.988014 19692202554.8 718 | 0.988293 19966424501.1 719 | 0.988293 20244465099.8 720 | 0.988293 20526377527.1 721 | 0.988432 20812215699.9 722 | 0.988432 21102034285.7 723 | 0.988432 21395888713.4 724 | 0.988432 21693835183.9 725 | 0.988571 21995930680.3 726 | 0.988571 22302232979.7 727 | 0.988571 22612800663.4 728 | 0.988571 22927693128.7 729 | 0.988711 23246970599.9 730 | 0.988711 23570694140.0 731 | 0.98885 23898925662.3 732 | 0.98899 24231727942.4 733 | 0.98899 24569164629.8 734 | 0.98899 24911300260.7 735 | 0.98899 25258200269.6 736 | 0.98899 25609931002.6 737 | 0.98899 25966559729.3 738 | 0.98899 26328154656.5 739 | 0.98899 26694784940.3 740 | 0.98899 27066520700.3 741 | 0.98899 27443433032.3 742 | 0.98899 27825594022.1 743 | 0.98899 28213076759.4 744 | 0.98899 28605955351.8 745 | 0.98899 29004304938.6 746 | 0.98899 29408201705.9 747 | 0.989129 29817722900.2 748 | 0.989129 30232946844.1 749 | 0.989129 30653952950.6 750 | 0.989129 31080821738.7 751 | 0.989129 31513634848.7 752 | 0.989408 31952475057.6 753 | 0.989547 32397426295.3 754 | 0.989686 32848573660.3 755 | 0.989686 33306003436.2 756 | 0.989826 33769803108.3 757 | 0.989826 34240061379.7 758 | 0.989826 34716868189.3 759 | 0.989965 35200314728.0 760 | 0.989965 35690493456.8 761 | 0.989965 36187498124.1 762 | 0.990105 36691423784.0 763 | 0.990244 37202366814.1 764 | 0.990244 37720424934.2 765 | 0.990244 38245697224.7 766 | 0.990244 38778284145.9 767 | 0.990383 39318287557.1 768 | 0.990523 39865810735.8 769 | 0.990523 40420958398.0 770 | 0.990523 40983836717.6 771 | 0.990523 41554553347.2 772 | 0.990523 42133217438.5 773 | 0.990523 42719939663.1 774 | 0.990523 43314832233.8 775 | 0.990662 43918008926.0 776 | 0.990801 44529585099.4 777 | 0.990801 45149677720.4 778 | 0.990801 45778405383.8 779 | 0.990941 46415888336.1 780 | 0.99108 47062248498.4 781 | 0.99108 47717609489.4 782 | 0.99108 48382096649.3 783 | 0.99122 49055837063.7 784 | 0.991359 49738959587.9 785 | 0.991359 50431594871.7 786 | 0.991359 51133875384.1 787 | 0.991638 51845935438.9 788 | 0.991638 52567911220.2 789 | 0.991916 53299940808.4 790 | 0.992056 54042164207.1 791 | 0.992334 54794723369.0 792 | 0.992474 55557762224.0 793 | 0.992613 56331426706.0 794 | 0.992753 57115864781.3 795 | 0.992892 57911226476.4 796 | 0.992892 58717663907.3 797 | 0.993031 59535331308.1 798 | 0.993031 60364385060.8 799 | 0.993171 61204983724.8 800 | 0.99331 62057288067.8 801 | 0.993589 62921461096.1 802 | 0.993589 63797668086.1 803 | 0.993728 64686076615.5 804 | 0.993728 65586856595.7 805 | 0.993728 66500180304.3 806 | 0.994286 67426222417.8 807 | 0.994425 68365160045.1 808 | 0.994704 69317172761.6 809 | 0.994983 70282442643.1 810 | 0.994983 71261154301.1 811 | 0.994983 72253494917.9 812 | 0.995122 73259654282.2 813 | 0.995122 74279824825.6 814 | 0.99554 75314201659.7 815 | 0.995679 76362982612.8 816 | 0.995958 77426368268.1 817 | 0.996376 78504562002.0 818 | 0.996376 79597770023.1 819 | 0.996376 80706201411.5 820 | 0.996516 81830068158.7 821 | 0.996655 82969585208.3 822 | 0.996655 84124970497.4 823 | 0.996655 85296444997.4 824 | 0.996794 86484232757.3 825 | 0.996794 87688560945.9 826 | 0.996794 88909659895.3 827 | 0.996934 90147763145.2 828 | 0.996934 91403107487.6 829 | 0.996934 92675933011.5 830 | 0.997073 93966483149.5 831 | 0.997073 95275004724.3 832 | 0.997073 96601747995.2 833 | 0.997491 97946966707.0 834 | 0.997491 99310918137.5 835 | 0.997491 100693863148.0 836 | 0.997491 102096066231.0 837 | 0.997491 103517795563.0 838 | 0.997491 104959323056.0 839 | 0.997491 106420924406.0 840 | 0.997491 107902879152.0 841 | 0.997491 109405470721.0 842 | 0.997491 110928986490.0 843 | 0.997491 112473717836.0 844 | 0.997491 114039960197.0 845 | 0.997491 115628013121.0 846 | 0.997491 117238180329.0 847 | 0.997491 118870769771.0 848 | 0.997631 120526093687.0 849 | 0.997631 122204468663.0 850 | 0.997631 123906215695.0 851 | 0.997631 125631660247.0 852 | 0.997631 127381132319.0 853 | 0.997631 129154966501.0 854 | 0.997631 130953502048.0 855 | 0.997631 132777082936.0 856 | 0.997631 134626057930.0 857 | 0.997631 136500780655.0 858 | 0.997631 138401609657.0 859 | 0.99777 140328908479.0 860 | 0.99777 142283045721.0 861 | 0.997909 144264395122.0 862 | 0.997909 146273335620.0 863 | 0.997909 148310251434.0 864 | 0.997909 150375532130.0 865 | 0.997909 152469572702.0 866 | 0.997909 154592773642.0 867 | 0.998049 156745541021.0 868 | 0.998049 158928286562.0 869 | 0.998049 161141427725.0 870 | 0.998049 163385387781.0 871 | 0.998049 165660595895.0 872 | 0.998328 167967487209.0 873 | 0.998328 170306502925.0 874 | 0.998328 172678090388.0 875 | 0.998467 175082703174.0 876 | 0.998467 177520801172.0 877 | 0.998467 179992850678.0 878 | 0.998467 182499324482.0 879 | 0.998746 185040701954.0 880 | 0.998746 187617469144.0 881 | 0.998885 190230118867.0 882 | 0.998885 192879150802.0 883 | 0.998885 195565071587.0 884 | 0.998885 198288394913.0 885 | 0.998885 201049641626.0 886 | 0.999164 203849339825.0 887 | 0.999164 206688024963.0 888 | 0.999164 209566239948.0 889 | 0.999164 212484535250.0 890 | 0.999164 215443469003.0 891 | 0.999164 218443607115.0 892 | 0.999303 221485523373.0 893 | 0.999303 224569799554.0 894 | 0.999303 227697025538.0 895 | 0.999303 230867799419.0 896 | 0.999443 234082727618.0 897 | 0.999443 237342425002.0 898 | 0.999582 240647515002.0 899 | 0.999582 243998629726.0 900 | 0.999582 247396410089.0 901 | 0.999582 250841505928.0 902 | 0.999721 254334576130.0 903 | 0.999721 257876288759.0 904 | 0.999721 261467321180.0 905 | 0.999721 265108360191.0 906 | 0.999721 268800102154.0 907 | 0.999721 272543253128.0 908 | 0.999721 276338529005.0 909 | 0.999721 280186655646.0 910 | 0.999721 284088369018.0 911 | 0.999721 288044415340.0 912 | 0.999721 292055551218.0 913 | 0.999721 296122543799.0 914 | 0.999721 300246170909.0 915 | 0.999721 304427221206.0 916 | 0.999721 308666494334.0 917 | 0.999861 312964801067.0 918 | 0.999861 317322963473.0 919 | 0.999861 321741815068.0 920 | 0.999861 326222200971.0 921 | 0.999861 330764978074.0 922 | 0.999861 335371015200.0 923 | 0.999861 340041193270.0 924 | 0.999861 344776405473.0 925 | 0.999861 349577557436.0 926 | 0.999861 354445567397.0 927 | 0.999861 359381366380.0 928 | 0.999861 364385898376.0 929 | 0.999861 369460120520.0 930 | 0.999861 374605003275.0 931 | 0.999861 379821530619.0 932 | 0.999861 385110700233.0 933 | 0.999861 390473523689.0 934 | 0.999861 395911026647.0 935 | 0.999861 401424249050.0 936 | 0.999861 407014245322.0 937 | 0.999861 412682084570.0 938 | 0.999861 418428850790.0 939 | 0.999861 424255643072.0 940 | 0.999861 430163575811.0 941 | 0.999861 436153778921.0 942 | 0.999861 442227398051.0 943 | 0.999861 448385594802.0 944 | 0.999861 454629546953.0 945 | 0.999861 460960448683.0 946 | 0.999861 467379510799.0 947 | 0.999861 473887960972.0 948 | 0.999861 480487043966.0 949 | 0.999861 487178021879.0 950 | 0.999861 493962174388.0 951 | 0.999861 500840798985.0 952 | 1.0 507815211233.0 953 | 1.0 514886745014.0 954 | 1.0 522056752785.0 955 | 1.0 529326605836.0 956 | 1.0 536697694554.0 957 | 1.0 544171428687.0 958 | 1.0 551749237613.0 959 | 1.0 559432570617.0 960 | 1.0 567222897164.0 961 | 1.0 575121707184.0 962 | 1.0 583130511353.0 963 | 1.0 591250841383.0 964 | 1.0 599484250319.0 965 | 1.0 607832312830.0 966 | 1.0 616296625513.0 967 | 1.0 624878807201.0 968 | 1.0 633580499266.0 969 | 1.0 642403365939.0 970 | 1.0 651349094627.0 971 | 1.0 660419396233.0 972 | 1.0 669616005485.0 973 | 1.0 678940681270.0 974 | 1.0 688395206965.0 975 | 1.0 697981390783.0 976 | 1.0 707701066118.0 977 | 1.0 717556091894.0 978 | 1.0 727548352920.0 979 | 1.0 737679760253.0 980 | 1.0 747952251562.0 981 | 1.0 758367791500.0 982 | 1.0 768928372076.0 983 | 1.0 779636013041.0 984 | 1.0 790492762270.0 985 | 1.0 801500696157.0 986 | 1.0 812661920009.0 987 | 1.0 823978568453.0 988 | 1.0 835452805838.0 989 | 1.0 847086826656.0 990 | 1.0 858882855955.0 991 | 1.0 870843149769.0 992 | 1.0 882969995549.0 993 | 1.0 895265712600.0 994 | 1.0 907732652521.0 995 | 1.0 920373199662.0 996 | 1.0 933189771573.0 997 | 1.0 946184819472.0 998 | 1.0 959360828709.0 999 | 1.0 972720319245.0 1000 | 1.0 986265846131.0 1001 | -------------------------------------------------------------------------------- /web_traffic_generator.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | ''' 4 | * 5 | * Copyright (c) 2016 6 | * Politecnico di Torino. All rights reserved. 7 | * 8 | * This program is free software; you can redistribute it and/or modify 9 | * it under the terms of the GNU General Public License as published by 10 | * the Free Software Foundation; either version 2 of the License, or 11 | * (at your option) any later version. 12 | * 13 | * For bug report and other information please write to: 14 | * martino.trevisan@polito.it 15 | * 16 | * 17 | ''' 18 | 19 | 20 | #Dependancies 21 | # Install BrowserMob Proxy from https://bmp.lightbody.net/ in current directory 22 | # sudo pip3 install selenium 23 | # sudo pip3 install browsermob-proxy 24 | 25 | import json 26 | import sys 27 | import random 28 | import time 29 | import argparse 30 | import os 31 | import subprocess 32 | from selenium import webdriver 33 | from selenium.webdriver.support.ui import WebDriverWait 34 | from selenium.webdriver.common.keys import Keys 35 | from selenium.webdriver import ActionChains 36 | from urllib.parse import urlparse 37 | import concurrent.futures 38 | 39 | sys.path.append(os.path.dirname(__file__)) 40 | from real_thinking_time import random_thinking_time 41 | 42 | 43 | 44 | 45 | disp_width = 800 46 | disp_height = 600 47 | browser = "firefox" 48 | timeout = 30 49 | real_backoff = 0 50 | static_backoff = 0 51 | debug = 0 52 | out_dir="" 53 | 54 | def main(): 55 | 56 | global real_backoff 57 | global static_backoff 58 | global timeout 59 | global out_dir 60 | 61 | parser = argparse.ArgumentParser(description='Web Traffic Generator') 62 | parser.add_argument('in_file', metavar='input_file', type=str, nargs=1, 63 | help='File where are stored the pages') 64 | parser.add_argument('out_dir', metavar='out_dir', type=str, nargs=1, 65 | help='Output directory where HAR structures are saved') 66 | parser.add_argument('-e', '--har_export', metavar='har_export', type=str, nargs=1,default = [""], 67 | help='Path to Har Export extension xpi file. If not set, searches it in the code path.') 68 | parser.add_argument('-r', '--real_backoff', metavar='real_backoff', type=int, nargs=1, default = [0], 69 | help='Use real backoff distribution with maximum value seconds ') 70 | parser.add_argument('-b', '--static_backoff', metavar='static_backoff', type=int, nargs=1, default = [1], 71 | help='Use a static backoff with value seconds ') 72 | parser.add_argument('-t', '--timeout', metavar='timeout', type=int, nargs=1, default = [30], 73 | help='Timeout in seconds after declaring failed a visit. Default is 30.') 74 | parser.add_argument('-v','--virtual_display', metavar='virtual_display', default=False, action='store_const', const=True, 75 | help='Use a virtual display instead of the physical one') 76 | 77 | parser.add_argument('-s','--start_page', metavar='start_page', type=int, nargs=1, 78 | help='For internal usage, do not use') 79 | 80 | args = vars(parser.parse_args()) 81 | 82 | 83 | # Parse arguments 84 | pages_file = args['in_file'][0] 85 | pages = open(pages_file,"r").read().splitlines() 86 | out_dir = args['out_dir'][0] 87 | if not out_dir[0] == "/": 88 | out_dir=os.getcwd() + "/" + out_dir 89 | 90 | if not os.path.exists(out_dir): 91 | os.makedirs(out_dir) 92 | 93 | har_export=args["har_export"][0] 94 | if har_export == "": 95 | names = list(os.walk(os.path.dirname(__file__)))[0][2] 96 | for n in names: 97 | if "harexporttrigger" in n and ".xpi" in n: 98 | har_export = os.path.dirname(__file__) + "/" + n 99 | 100 | 101 | real_backoff = args['real_backoff'] [0] 102 | static_backoff = args['static_backoff'][0] 103 | timeout = args['timeout'][0] + real_backoff + static_backoff 104 | virtual_display = args['virtual_display'] 105 | 106 | 107 | if virtual_display: 108 | from pyvirtualdisplay import Display 109 | 110 | # Use last arguments to detect if i'm master or daemon 111 | if args["start_page"] is not None: 112 | daemon_start = args["start_page"][0] 113 | else: 114 | daemon_start = -1 115 | 116 | # If I'm the master 117 | if daemon_start == -1: 118 | 119 | print ("Using HAR export", har_export ) 120 | # Execute the slave 121 | command = " ".join(sys.argv) + " -s 0" 122 | print ("Executing:", command , file=sys.stderr) 123 | ret = subprocess.call(command, shell=True) 124 | print("Quitted slave") 125 | 126 | # Keep execting untill all pages are requested 127 | while ret != 0: 128 | # Read last page requested, and restart 129 | start = int(open("/tmp/har_state", "r").read()) 130 | print("Detected a Selenium block, restarting...", file=sys.stderr) 131 | command = " ".join(sys.argv) + " -s " + str(start) 132 | print ("Executing:", command, file=sys.stderr) 133 | ret = subprocess.call(command, shell=True) 134 | 135 | # End when all pages are requested 136 | print ("All pages requested", file=sys.stderr) 137 | time.sleep(2) 138 | sys.exit(0) 139 | 140 | else: 141 | 142 | # Start Selenium and Har Export 143 | 144 | 145 | profile = webdriver.FirefoxProfile() 146 | 147 | profile.add_extension(har_export) 148 | 149 | # Set default NetExport preferences 150 | #profile.set_preference("devtools.netmonitor.har.enableAutoExportToFile", True) 151 | #profile.set_preference("devtools.netmonitor.har.forceExport", True) 152 | profile.set_preference("extensions.netmonitor.har.autoConnect", True) 153 | profile.set_preference("devtools.netmonitor.enabled", True); 154 | profile.set_preference("extensions.netmonitor.har.contentAPIToken", "test") 155 | #profile.set_preference("devtools.netmonitor.har.forceExport", "true") 156 | profile.set_preference("devtools.netmonitor.har.defaultLogDir", out_dir) 157 | profile.set_preference("devtools.netmonitor.har.includeResponseBodies", False) 158 | #profile.set_preference("devtools.netmonitor.har.pageLoadedTimeout", "2500") 159 | 160 | # Start a virtual display if required 161 | if virtual_display: 162 | display = Display(visible=0, size=(disp_width, disp_height)) 163 | display.start() 164 | 165 | driver = webdriver.Firefox(firefox_profile=profile) 166 | time.sleep(1) 167 | 168 | # Start requesting pages from the last one 169 | pages = pages[daemon_start:] 170 | n = daemon_start 171 | 172 | # Create a thread pool 173 | executor = concurrent.futures.ThreadPoolExecutor(max_workers=1) 174 | 175 | # Start requesting pages 176 | for page in pages: 177 | if debug > 0: 178 | print("Requesting:", n , page, file=sys.stderr) 179 | 180 | # Submit the future 181 | future = executor.submit(request_url, page, driver) 182 | 183 | # Wait to complete 184 | t = 0 185 | while True: 186 | time.sleep(1) 187 | if debug > 1: 188 | print("Timeout", t, file=sys.stderr) 189 | if future.done(): 190 | break 191 | t+=1 192 | 193 | # If timeout elapses, a block happened 194 | if t >= timeout: 195 | 196 | # Try all the ways to stop everything 197 | open("/tmp/har_state", "w").write(str(n+1)) 198 | # Stop Selenium 199 | try: 200 | driver.quit() 201 | except: 202 | pass 203 | # Kill the browser 204 | try: 205 | command = "killall " + browser 206 | subprocess.call(command, shell=True) 207 | subprocess.call(command + " -KILL", shell=True) 208 | except: 209 | pass 210 | 211 | if debug > 1: 212 | print ("Quitting with errcode:", n+1,file=sys.stderr) 213 | future.cancel() 214 | print("Quitting slave", file=sys.stderr) 215 | 216 | # Suicide 217 | os.system('kill %d' % os.getpid()) 218 | 219 | n+=1 220 | 221 | # If all pages requested, exit with '0' status 222 | driver.quit() 223 | 224 | sys.exit(0) 225 | 226 | 227 | def request_url(page, driver): 228 | 229 | try: 230 | # Request the page 231 | url = page 232 | print("Requesting:", page) 233 | start_time = time.time() 234 | driver.get(url) 235 | end_time=time.time() 236 | elapsed_time = end_time - start_time 237 | 238 | domain=url.split("/")[2] 239 | driver.execute_script(get_script(domain,page,elapsed_time)) 240 | 241 | if real_backoff != 0: 242 | tm=random_thinking_time(real_backoff) 243 | else: 244 | tm=static_backoff 245 | print ("Pause:", tm) 246 | time.sleep(tm) 247 | 248 | 249 | 250 | 251 | 252 | except Exception as e: 253 | print("Exception in page loading", e) 254 | while True: 255 | pass 256 | 257 | def get_script(domain,page,elapsed_time): 258 | script='\ 259 | function triggerExport() {\ 260 | var options = {\ 261 | token: "test", \ 262 | getData: true, \ 263 | title: "' + str(elapsed_time) + " " + page +'", \ 264 | jsonp: false,\ 265 | fileName: "visit_%Y_%m_%d_%H_%M_%S_'+ domain.replace(".","_") +'"};' + \ 266 | 'HAR.triggerExport(options).then(result => {console.log(result.data);});};\ 267 | if (typeof HAR === "undefined") {\ 268 | addEventListener("har-api-ready", triggerExport, false);\ 269 | } else {\ 270 | triggerExport();\ 271 | };' 272 | 273 | return script 274 | 275 | main() 276 | -------------------------------------------------------------------------------- /web_traffic_generator_firebug.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 2 | 3 | ''' 4 | * 5 | * Copyright (c) 2016 6 | * Politecnico di Torino. All rights reserved. 7 | * 8 | * This program is free software; you can redistribute it and/or modify 9 | * it under the terms of the GNU General Public License as published by 10 | * the Free Software Foundation; either version 2 of the License, or 11 | * (at your option) any later version. 12 | * 13 | * For bug report and other information please write to: 14 | * martino.trevisan@polito.it 15 | * 16 | * 17 | * To download NetExport: 18 | * wget https://getfirebug.com/releases/netexport/netExport-0.9b7.xpi 19 | * To download FireBug: 20 | * wget https://addons.cdn.mozilla.net/user-media/addons/1843/firebug-2.0.17-fx.xpi?filehash=sha256%3A32a1eef23bdac7bb97a06b527f5c88fe94476755f76b8b1db3b3c66acddd83da 21 | ''' 22 | 23 | 24 | #Dependancies 25 | # Install BrowserMob Proxy from https://bmp.lightbody.net/ in current directory 26 | # sudo pip3 install selenium 27 | # sudo pip3 install browsermob-proxy 28 | 29 | import json 30 | import sys 31 | import random 32 | import time 33 | import argparse 34 | import os 35 | import subprocess 36 | from selenium import webdriver 37 | from selenium.webdriver.support.ui import WebDriverWait 38 | from selenium.webdriver.common.keys import Keys 39 | from selenium.webdriver import ActionChains 40 | from urllib.parse import urlparse 41 | import concurrent.futures 42 | 43 | sys.path.append(os.path.dirname(__file__)) 44 | from real_thinking_time import random_thinking_time 45 | 46 | 47 | 48 | 49 | disp_width = 800 50 | disp_height = 600 51 | browser = "firefox" 52 | timeout = 60 53 | real_backoff = 0 54 | static_backoff = 0 55 | debug = 0 56 | out_dir="" 57 | 58 | def main(): 59 | 60 | global real_backoff 61 | global static_backoff 62 | global timeout 63 | global out_dir 64 | 65 | parser = argparse.ArgumentParser(description='Web Traffic Generator') 66 | parser.add_argument('in_file', metavar='input_file', type=str, nargs=1, 67 | help='File where are stored the pages') 68 | parser.add_argument('out_dir', metavar='out_dir', type=str, nargs=1, 69 | help='Output directory where HAR structures are saved') 70 | parser.add_argument('-r', '--real_backoff', metavar='real_backoff', type=int, nargs=1, default = [0], 71 | help='Use real backoff distribution with maximum value seconds ') 72 | parser.add_argument('-b', '--static_backoff', metavar='static_backoff', type=int, nargs=1, default = [1], 73 | help='Use a static backoff with value seconds ') 74 | parser.add_argument('-t', '--timeout', metavar='timeout', type=int, nargs=1, default = [30], 75 | help='Timeout in seconds after declaring failed a visit. Default is 30.') 76 | parser.add_argument('-v','--virtual_display', metavar='virtual_display', default=False, action='store_const', const=True, 77 | help='Use a virtual display instead of the physical one') 78 | 79 | parser.add_argument('-s','--start_page', metavar='start_page', type=int, nargs=1, 80 | help='For internal usage, do not use') 81 | 82 | args = vars(parser.parse_args()) 83 | 84 | 85 | # Parse arguments 86 | pages_file = args['in_file'][0] 87 | pages = open(pages_file,"r").read().splitlines() 88 | out_dir = args['out_dir'][0] 89 | if not out_dir[0] == "/": 90 | out_dir=os.getcwd() + "/" + out_dir 91 | 92 | if not os.path.exists(out_dir): 93 | os.makedirs(out_dir) 94 | 95 | 96 | 97 | real_backoff = args['real_backoff'] [0] 98 | static_backoff = args['static_backoff'][0] 99 | timeout = args['timeout'][0] + real_backoff + static_backoff 100 | virtual_display = args['virtual_display'] 101 | 102 | 103 | if virtual_display: 104 | from pyvirtualdisplay import Display 105 | 106 | # Use last arguments to detect if i'm master or daemon 107 | if args["start_page"] is not None: 108 | daemon_start = args["start_page"][0] 109 | else: 110 | daemon_start = -1 111 | 112 | # If I'm the master 113 | if daemon_start == -1: 114 | 115 | # Execute the slave 116 | command = " ".join(sys.argv) + " -s 0" 117 | print ("Executing:", command , file=sys.stderr) 118 | ret = subprocess.call(command, shell=True) 119 | print("Quitted slave") 120 | 121 | # Keep execting untill all pages are requested 122 | while ret != 0: 123 | # Read last page requested, and restart 124 | start = int(open("/tmp/har_state", "r").read()) 125 | print("Detected a Selenium block, restarting...", file=sys.stderr) 126 | command = " ".join(sys.argv) + " -s " + str(start) 127 | print ("Executing:", command, file=sys.stderr) 128 | ret = subprocess.call(command, shell=True) 129 | 130 | # End when all pages are requested 131 | print ("All pages requested", file=sys.stderr) 132 | time.sleep(2) 133 | sys.exit(0) 134 | 135 | else: 136 | 137 | 138 | profile = webdriver.FirefoxProfile() 139 | 140 | profile.add_extension(os.path.dirname(__file__) + "/" + "netExport-0.9b7.xpi") 141 | profile.add_extension(os.path.dirname(__file__) + "/" + "firebug-2.0.17-fx.xpi") 142 | 143 | 144 | # Set default Firefox preferences 145 | profile.set_preference("app.update.enabled", False) 146 | 147 | domain = "extensions.firebug." 148 | 149 | # Set default Firebug preferences 150 | profile.set_preference(domain + "currentVersion", "1.9.2") 151 | profile.set_preference(domain + "allPagesActivation", "on") 152 | profile.set_preference(domain + "defaultPanelName", "net") 153 | profile.set_preference(domain + "net.enableSites", True) 154 | 155 | # Set default NetExport preferences 156 | profile.set_preference(domain + "netexport.alwaysEnableAutoExport", True) 157 | profile.set_preference(domain + "netexport.showPreview", False) 158 | profile.set_preference(domain + "netexport.defaultLogDir", out_dir) 159 | profile.set_preference(domain + "extensions.firebug.netexport.timeout", max(timeout-5,5) ) 160 | profile.set_preference(domain + "extensions.firebug.netexport.includeResponseBodies", False) 161 | 162 | # Start a virtual display if required 163 | if virtual_display: 164 | display = Display(visible=0, size=(disp_width, disp_height)) 165 | display.start() 166 | 167 | driver = webdriver.Firefox(firefox_profile=profile) 168 | time.sleep(1) 169 | 170 | # Start requesting pages from the last one 171 | pages = pages[daemon_start:] 172 | n = daemon_start 173 | 174 | # Create a thread pool 175 | executor = concurrent.futures.ThreadPoolExecutor(max_workers=1) 176 | 177 | # Start requesting pages 178 | for page in pages: 179 | if debug > 0: 180 | print("Requesting:", n , page, file=sys.stderr) 181 | 182 | # Submit the future 183 | future = executor.submit(request_url, page, driver) 184 | 185 | # Wait to complete 186 | t = 0 187 | while True: 188 | time.sleep(1) 189 | if debug > 1: 190 | print("Timeout", t, file=sys.stderr) 191 | if future.done(): 192 | break 193 | t+=1 194 | 195 | # If timeout elapses, a block happened 196 | if t >= timeout: 197 | 198 | # Try all the ways to stop everything 199 | open("/tmp/har_state", "w").write(str(n+1)) 200 | # Stop Selenium 201 | try: 202 | driver.quit() 203 | except: 204 | pass 205 | # Kill the browser 206 | try: 207 | command = "killall " + browser 208 | subprocess.call(command, shell=True) 209 | subprocess.call(command + " -KILL", shell=True) 210 | except: 211 | pass 212 | 213 | if debug > 1: 214 | print ("Quitting with errcode:", n+1,file=sys.stderr) 215 | future.cancel() 216 | print("Quitting slave", file=sys.stderr) 217 | 218 | # Suicide 219 | os.system('kill %d' % os.getpid()) 220 | 221 | n+=1 222 | 223 | # If all pages requested, exit with '0' status 224 | driver.get("about:blank") 225 | driver.quit() 226 | 227 | sys.exit(0) 228 | 229 | 230 | def request_url(page, driver): 231 | 232 | try: 233 | # Request the page 234 | url = page 235 | print("Requesting:", page) 236 | start_time = time.time() 237 | driver.get(url) 238 | end_time=time.time() 239 | elapsed_time = end_time - start_time 240 | 241 | #domain=url.split("/")[2] 242 | #driver.execute_script(get_script(domain,page,elapsed_time)) 243 | 244 | if real_backoff != 0: 245 | tm=random_thinking_time(real_backoff) 246 | else: 247 | tm=static_backoff 248 | print ("Pause:", tm) 249 | time.sleep(tm) 250 | 251 | 252 | 253 | 254 | 255 | except Exception as e: 256 | print("Exception in page loading", e) 257 | while True: 258 | pass 259 | 260 | def get_script(domain,page,elapsed_time): 261 | script='\ 262 | function triggerExport() {\ 263 | var options = {\ 264 | token: "test", \ 265 | getData: true, \ 266 | title: "' + str(elapsed_time) + " " + page +'", \ 267 | jsonp: false,\ 268 | fileName: "visit_%Y_%m_%d_%H_%M_%S_'+ domain.replace(".","_") +'"};' + \ 269 | 'HAR.triggerExport(options).then(result => {console.log(result.data);});};\ 270 | if (typeof HAR === "undefined") {\ 271 | addEventListener("har-api-ready", triggerExport, false);\ 272 | } else {\ 273 | triggerExport();\ 274 | };' 275 | 276 | return script 277 | 278 | main() 279 | --------------------------------------------------------------------------------