├── LICENSE ├── README.md ├── affiliates ├── 01-internetarchive-domain-to-forwarded.csv ├── 02-internetarchive-unique-urls.csv ├── 03-final-pairing-domain-to-affiliate-id.txt ├── README.md └── coinhive-redir.txt ├── censorship ├── README.md ├── blocked-eg.txt ├── blocked-tr.txt ├── parse-pcap-results.py └── test-censorship.py ├── censys ├── README.md └── censys-summary.csv ├── ooni-historical ├── README.md └── egypt-307.csv └── pcaps ├── README.md ├── adhose-trickle-riseupvpn.pcap ├── egypt-packetlogic-ttl-localization.pcap └── turkey-malware-injection.pcap /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2018, Bill Marczak 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 7 | 8 | 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 9 | 10 | 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 11 | 12 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # bad-traffic 2 | 3 | This repo contains supporting data associated with the Citizen Lab report: 4 | 5 | [Bad Traffic: Sandvine’s PacketLogic Devices Used to Deploy Government Spyware in Turkey and Syria; redirect Egyptian Users to Affiliate Ads](https://citizenlab.ca/2018/03/bad-traffic-sandvines-packetlogic-devices-deploy-government-spyware-turkey-syria) 6 | 7 | What Is It? 8 | ============ 9 | 10 | This repository contains the following directories: 11 | 12 | * [affiliates](affiliates) 13 | * An analysis of affiliate IDs found on Internet Archive WayBack Machine suspected to be linked with AdHose. 14 | 15 | * [censorship](censorship) 16 | * A look at censored content in Turkey and Egypt. 17 | 18 | * [censys](censys) 19 | * An analysis of redirections to the static.dbmads[ . ]com domain in censys data. 20 | 21 | * [ooni-historical](ooni-historical) 22 | * An extract of OONI data looking at all observed 307 redirects in Egypt. 23 | 24 | * [pcaps](pcaps) 25 | * This directory contains packet captures of the injections we observed. 26 | 27 | License 28 | ======== 29 | 30 | All code is provided under the [BSD 3-Clause License](LICENSE) 31 | 32 | All data is provided under Creative Commons 33 | Attribution-NonCommercial-ShareAlike 4.0 International and available in full 34 | [here](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode) and summarized 35 | [here](https://creativecommons.org/licenses/by-nc-sa/4.0/) 36 | 37 | -------------------------------------------------------------------------------- /affiliates/01-internetarchive-domain-to-forwarded.csv: -------------------------------------------------------------------------------- 1 | initial domain,forwarded url,snapshotname,snapshot_date 2 | nexploreads.com,http://adserver2.utextads.com/adsm.html,20161112235652.snapshot,2016-11-12 23:56:52 3 | nexploreads.com,http://go.ad2upapp.com/afu.php?id=773263,20161119102509.snapshot,2016-11-19 10:25:09 4 | nexploreads.com,http://adserver1.utextads.com/adsm.html,20161111212227.snapshot,2016-11-11 21:22:27 5 | nexploreads.com,http://go.ad2up.com/afu.php?id=773263,20161106132038.snapshot,2016-11-06 13:20:38 6 | infinitiads.com,http://adserver2.utextads.com/adsm.html,20161115051735.snapshot,2016-11-15 5:17:35 7 | infinitiads.com,http://static.dbmads.com/vs,20170515135458.snapshot,2017-05-15 13:54:58 8 | infinitiads.com,http://static.dbmads.com/staticd.html,20170510025952.snapshot,2017-05-10 2:59:52 9 | infinitiads.com,http://adserver1.utextads.com/adsm.html,20160910141118.snapshot,2016-09-10 14:11:18 10 | infinitiads.com,http://adserver2.utextads.com/adsm.html,20161027010818.snapshot,2016-10-27 1:08:18 11 | infinitiads.com,http://adserver1.utextads.com/adsm.html,20161029124321.snapshot,2016-10-29 12:43:21 12 | infinitiads.com,http://adserver1.utextads.com/adsm.html,20160926000800.snapshot,2016-09-26 0:08:00 13 | infinitiads.com,http://go.ad2upapp.com/afu.php?id=773263,20160915021409.snapshot,2016-09-15 2:14:09 14 | infinitiads.com,http://go.ad2upapp.com/afu.php?id=773263,20161111210341.snapshot,2016-11-11 21:03:41 15 | infinitiads.com,http://adserver1.utextads.com/adsm.html,20161109104018.snapshot,2016-11-09 10:40:18 16 | infinitiads.com,https://ylx-4.com/fullpage.php?section=General&pub=125652&ga=g,20170527174806.snapshot,2017-05-27 17:48:06 17 | infinitiads.com,http://adserver1.utextads.com/adsm.html,20160929040441.snapshot,2016-09-29 4:04:41 18 | infinitiads.com,http://go.ad2up.com/afu.php?id=773263,20161119033958.snapshot,2016-11-19 3:39:58 19 | infinitiads.com,http://adserver1.utextads.com/adsm.html,20161009161000.snapshot,2016-10-09 16:10:00 20 | infinitiads.com,http://adserver2.utextads.com/adsm.html,20161002022942.snapshot,2016-10-02 2:29:42 21 | infinitiads.com,http://go.ad2upapp.com/afu.php?id=773263,20161117215635.snapshot,2016-11-17 21:56:35 22 | infinitiads.com,http://adserver2.utextads.com/adsm.html,20161014155438.snapshot,2016-10-14 15:54:38 23 | infinitiads.com,http://go.ad2up.com/afu.php?id=773263,20161022095315.snapshot,2016-10-22 9:53:15 24 | infinitiads.com,http://adserver2.utextads.com/adsm.html,20161103120223.snapshot,2016-11-03 12:02:23 25 | infinitiads.com,http://adserver2.utextads.com/adsm.html,20160919041132.snapshot,2016-09-19 4:11:32 26 | infinitiads.com,http://go.ad2up.com/afu.php?id=773263,20161004191536.snapshot,2016-10-04 19:15:36 27 | infinitiads.com,http://go.ad2upapp.com/afu.php?id=773263,20161101005831.snapshot,2016-11-01 0:58:31 28 | infinitiads.com,http://static.dbmads.com/static.html,20170605083527.snapshot,2017-06-05 8:35:27 29 | infinitiads.com,http://go.ad2upapp.com/afu.php?id=773263,20160922064320.snapshot,2016-09-22 6:43:20 30 | infinitiads.com,http://go.ad2upapp.com/afu.php?id=773263,20161021041544.snapshot,2016-10-21 4:15:44 31 | static.dbmads.com,http://go.ad2upapp.com/afu.php?id=1209127,20170515091105.snapshot,2017-05-15 9:11:05 32 | static.dbmads.com,https://conceau.co/bV3ZVR0OPB3LpOvMbCmWVDJUZDDM0MyYNTjSkRyTMHDJcGzZLPTJIK2KOWTRIJxKMOjTEH=T,20180206100927.snapshot,2018-02-06 10:09:27 33 | static.dbmads.com,http://cs6hm.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20170503163144.snapshot,2017-05-03 16:31:44 34 | static.dbmads.com,http://infinitiads.com/iads.html,20170503233043.snapshot,2017-05-03 23:30:43 35 | static.dbmads.com,http://infinitiads.com/ad1/ad1.html,20170503230645.snapshot,2017-05-03 23:06:45 36 | static.dbmads.com,http://rapidyl.net/out?zoneId=1892611-1892737,20170503233128.snapshot,2017-05-03 23:31:28 37 | static.dbmads.com,http://zo.ee/4PYT,20170503233057.snapshot,2017-05-03 23:30:57 38 | static.dbmads.com,http://p166250.eclkmpbn.com/adServe/banners?tid=KF1DK&action=r,20170503182657.snapshot,2017-05-03 18:26:57 39 | static.dbmads.com,http://zo.ee/4PYT,20170320193129.snapshot,2017-03-20 19:31:29 40 | static.dbmads.com,http://nexploreads.com/iads.html,20161119053631.snapshot,2016-11-19 5:36:31 41 | static.dbmads.com,http://ceesty.com/qG3Ng7,20170531202224.snapshot,2017-05-31 20:22:24 42 | static.dbmads.com,http://p166250.eclkmpbn.com/adServe/banners?tid=KF1DK&action=r,20170131022548.snapshot,2017-01-31 2:25:48 43 | static.dbmads.com,http://adserver1.utextads.com/clk2p.html,20161213163301.snapshot,2016-12-13 16:33:01 44 | static.dbmads.com,http://infinitiads.com/iads.html,20161227040313.snapshot,2016-12-27 4:03:13 45 | static.dbmads.com,http://www.clicksgear.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20161206192147.snapshot,2016-12-06 19:21:47 46 | static.dbmads.com,http://static.dbmads.com/cstatic.html,20170506123343.snapshot,2017-05-06 12:33:43 47 | static.dbmads.com,http://adserver1.utextads.com/clk.html,20161112113632.snapshot,2016-11-12 11:36:32 48 | static.dbmads.com,http://infads-1372369412.eu-west-1.elb.amazonaws.com/stred.html,20171206090343.snapshot,2017-12-06 9:03:43 49 | static.dbmads.com,https://conceau.co/bV3ZVR0OPB3LpOvMbCmWVDJUZDDM0MyYNTjSkRyTMHDJcGzZLPTJIK2KOWTRIJxKMOjTEH=T,20180209152238.snapshot,2018-02-09 15:22:38 50 | static.dbmads.com,http://nexploreads.com/ad1/ad1.html,20170524172257.snapshot,2017-05-24 17:22:57 51 | static.dbmads.com,http://go.ad2upapp.com/afu.php?id=1209127,20170515091026.snapshot,2017-05-15 9:10:26 52 | static.dbmads.com,http://go.ad2upapp.com/afu.php?id=1209127,20170517031414.snapshot,2017-05-17 3:14:14 53 | static.dbmads.com,http://cs6hm.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20170503230640.snapshot,2017-05-03 23:06:40 54 | static.dbmads.com,http://go.ad2upapp.com/afu.php?id=1209127,20170517132907.snapshot,2017-05-17 13:29:07 55 | static.dbmads.com,http://static.dbmads.com/static.html,20170503173727.snapshot,2017-05-03 17:37:27 56 | static.dbmads.com,http://static.dbmads.com/nwp/static.html,20170527150541.snapshot,2017-05-27 15:05:41 57 | static.dbmads.com,http://go.oclasrv.com/afu.php?id=896707,20170116151527.snapshot,2017-01-16 15:15:27 58 | static.dbmads.com,http://p166250.eclkmpbn.com/adServe/banners?tid=KF1DK&action=r,20170109134150.snapshot,2017-01-09 13:41:50 59 | static.dbmads.com,http://rapidyl.net/out?zoneId=1892611-1892737,20170531202249.snapshot,2017-05-31 20:22:49 60 | static.dbmads.com,http://p166250.eclkmpbn.com/adServe/banners?tid=KF1DK&action=r,20170109203241.snapshot,2017-01-09 20:32:41 61 | static.dbmads.com,http://rapidyl.net/out?zoneId=1892611-1892737,20170503173519.snapshot,2017-05-03 17:35:19 62 | static.dbmads.com,http://www.clicksgear.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20161207093404.snapshot,2016-12-07 9:34:04 63 | static.dbmads.com,http://cpm10.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20170608220250.snapshot,2017-06-08 22:02:50 64 | static.dbmads.com,http://static.dbmads.com/cstatic.html,20170608222804.snapshot,2017-06-08 22:28:04 65 | static.dbmads.com,http://rapidyl.net/out?zoneId=1924441-1924543,20170518201758.snapshot,2017-05-18 20:17:58 66 | static.dbmads.com,http://nexploreads.com/ad1/ad1.html,20170518201832.snapshot,2017-05-18 20:18:32 67 | static.dbmads.com,http://cpm10.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20170611134733.snapshot,2017-06-11 13:47:33 68 | static.dbmads.com,http://www.hitcpm.com/watch?key=e4c634c55ad300b85c8760d9e09104cd,20171202101948.snapshot,2017-12-02 10:19:48 69 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20161021132527.snapshot,2016-10-21 13:25:27 70 | adserver1.utextads.com,http://adserver1.utextads.com/adsm.html,20161113052036.snapshot,2016-11-13 5:20:36 71 | adserver1.utextads.com,http://p166250.eclkmpbn.com/adServe/banners?tid=KF1DK&action=r,20161205154915.snapshot,2016-12-05 15:49:15 72 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20161023214108.snapshot,2016-10-23 21:41:08 73 | adserver1.utextads.com,http://www.urldelivery.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20161119022515.snapshot,2016-11-19 2:25:15 74 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=862744,20161118222401.snapshot,2016-11-18 22:24:01 75 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20160912074627.snapshot,2016-09-12 7:46:27 76 | adserver1.utextads.com,https://go.ad2up.com/afu.php?id=758873,20161106050140.snapshot,2016-11-06 5:01:40 77 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20161003172543.snapshot,2016-10-03 17:25:43 78 | adserver1.utextads.com,http://go.oclasrv.com/afu.php?id=896707,20161206191810.snapshot,2016-12-06 19:18:10 79 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20160923015745.snapshot,2016-09-23 1:57:45 80 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20161014223514.snapshot,2016-10-14 22:35:14 81 | adserver1.utextads.com,http://go.oclasrv.com/afu.php?id=896707,20161208001745.snapshot,2016-12-08 0:17:45 82 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20160928041713.snapshot,2016-09-28 4:17:13 83 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20161118041754.snapshot,2016-11-18 4:17:54 84 | adserver1.utextads.com,http://p166250.eclkmpbn.com/adServe/banners?tid=KF1DK&action=r,20161206161634.snapshot,2016-12-06 16:16:34 85 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20160930075231.snapshot,2016-09-30 7:52:31 86 | adserver1.utextads.com,http://www.urldelivery.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20161025010719.snapshot,2016-10-25 1:07:19 87 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20160924082653.snapshot,2016-09-24 8:26:53 88 | adserver1.utextads.com,https://go.ad2up.com/afu.php?id=758873,20161111144640.snapshot,2016-11-11 14:46:40 89 | adserver1.utextads.com,http://go.ad2up.com/afu.php?id=758873,20161001142252.snapshot,2016-10-01 14:22:52 90 | adserver1.utextads.com,https://go.ad2up.com/afu.php?id=758873,20161109205759.snapshot,2016-11-09 20:57:59 91 | adserver1.utextads.com,http://www.urldelivery.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,20161118230312.snapshot,2016-11-18 23:03:12 -------------------------------------------------------------------------------- /affiliates/02-internetarchive-unique-urls.csv: -------------------------------------------------------------------------------- 1 | forwarded url,has a unique id?,Id,Domain,Pairing domain to Affiliate ID 2 | http://adserver2.utextads.com/adsm.html,maybe static,,, - 3 | http://go.ad2upapp.com/afu.php?id=773263,maybe has id,773263,go.ad2upapp.com,go.ad2upapp.com - 773263 4 | http://adserver1.utextads.com/adsm.html,maybe static,,, - 5 | http://go.ad2up.com/afu.php?id=773263,maybe has id,773263,go.ad2up.com,go.ad2up.com - 773263 6 | http://static.dbmads.com/vs,maybe static,,, - 7 | http://static.dbmads.com/staticd.html,maybe static,,, - 8 | https://ylx-4.com/fullpage.php?section=General&pub=125652&ga=g,maybe has id,125652,ylx-4.com,ylx-4.com - 125652 9 | http://static.dbmads.com/static.html,maybe static,,, - 10 | http://go.ad2upapp.com/afu.php?id=1209127,maybe has id,1209127,go.ad2upapp.com,go.ad2upapp.com - 1209127 11 | https://conceau.co/bV3ZVR0OPB3LpOvMbCmWVDJUZDDM0MyYNTjSkRyTMHDJcGzZLPTJIK2KOWTRIJxKMOjTEH=T,maybe has id,bV3ZVR0OPB3LpOvMbCmWVDJUZDDM0MyYNTjSkRyTMHDJcGzZLPTJIK2KOWTRIJxKMOjTEH=T,conceau.co,conceau.co - bV3ZVR0OPB3LpOvMbCmWVDJUZDDM0MyYNTjSkRyTMHDJcGzZLPTJIK2KOWTRIJxKMOjTEH=T 12 | http://cs6hm.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,maybe has id,3e73d64a401c1e5b8b3eb33316b711e0,cs6hm.com,cs6hm.com - 3e73d64a401c1e5b8b3eb33316b711e0 13 | http://infinitiads.com/iads.html,maybe static,,, - 14 | http://infinitiads.com/ad1/ad1.html,maybe static,,, - 15 | http://rapidyl.net/out?zoneId=1892611-1892737,maybe has id,1892611-1892737,rapidyl.net,rapidyl.net - 1892611-1892737 16 | http://zo.ee/4PYT,maybe static,,, - 17 | http://p166250.eclkmpbn.com/adServe/banners?tid=KF1DK&action=r,maybe has id,KF1DK,p166250.eclkmpbn.com,p166250.eclkmpbn.com - KF1DK 18 | http://nexploreads.com/iads.html,maybe static,,, - 19 | http://ceesty.com/qG3Ng7,maybe static,,, - 20 | http://adserver1.utextads.com/clk2p.html,maybe static,,, - 21 | http://www.clicksgear.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,maybe has id,3e73d64a401c1e5b8b3eb33316b711e0,clicksgear.com,clicksgear.com - 3e73d64a401c1e5b8b3eb33316b711e0 22 | http://static.dbmads.com/cstatic.html,maybe static,,, - 23 | http://adserver1.utextads.com/clk.html,maybe static,,, - 24 | http://infads-1372369412.eu-west-1.elb.amazonaws.com/stred.html,maybe static,,, - 25 | http://nexploreads.com/ad1/ad1.html,maybe static,,, - 26 | http://static.dbmads.com/nwp/static.html,maybe static,,, - 27 | http://go.oclasrv.com/afu.php?id=896707,maybe has id,896707,go.oclasrv.com,go.oclasrv.com - 896707 28 | http://cpm10.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,maybe has id,3e73d64a401c1e5b8b3eb33316b711e0,cpm10.com,cpm10.com - 3e73d64a401c1e5b8b3eb33316b711e0 29 | http://rapidyl.net/out?zoneId=1924441-1924543,maybe has id,1924441-1924543,rapidyl.net,rapidyl.net - 1924441-1924543 30 | http://www.hitcpm.com/watch?key=e4c634c55ad300b85c8760d9e09104cd,maybe has id,e4c634c55ad300b85c8760d9e09104cd,hitcpm.com,hitcpm.com - e4c634c55ad300b85c8760d9e09104cd 31 | http://go.ad2up.com/afu.php?id=758873,maybe has id,758873,go.ad2up.com,go.ad2up.com - 758873 32 | http://www.urldelivery.com/watch?key=3e73d64a401c1e5b8b3eb33316b711e0,maybe has id,3e73d64a401c1e5b8b3eb33316b711e0,urldelivery.com,urldelivery.com - 3e73d64a401c1e5b8b3eb33316b711e0 33 | http://go.ad2up.com/afu.php?id=862744,maybe has id,862744,go.ad2up.com,go.ad2up.com - 862744 34 | https://go.ad2up.com/afu.php?id=758873,maybe has id,758873,go.ad2up.com,go.ad2up.com - 758873 -------------------------------------------------------------------------------- /affiliates/03-final-pairing-domain-to-affiliate-id.txt: -------------------------------------------------------------------------------- 1 | Pairing domain to Affiliate ID 2 | - 3 | go.ad2upapp.com - 773263 4 | go.ad2up.com - 773263 5 | ylx-4.com - 125652 6 | go.ad2upapp.com - 1209127 7 | conceau.co - bV3ZVR0OPB3LpOvMbCmWVDJUZDDM0MyYNTjSkRyTMHDJcGzZLPTJIK2KOWTRIJxKMOjTEH=T 8 | cs6hm.com - 3e73d64a401c1e5b8b3eb33316b711e0 9 | rapidyl.net - 1892611-1892737 10 | p166250.eclkmpbn.com - KF1DK 11 | clicksgear.com - 3e73d64a401c1e5b8b3eb33316b711e0 12 | go.oclasrv.com - 896707 13 | cpm10.com - 3e73d64a401c1e5b8b3eb33316b711e0 14 | rapidyl.net - 1924441-1924543 15 | hitcpm.com - e4c634c55ad300b85c8760d9e09104cd 16 | go.ad2up.com - 758873 17 | urldelivery.com - 3e73d64a401c1e5b8b3eb33316b711e0 18 | go.ad2up.com - 862744 -------------------------------------------------------------------------------- /affiliates/README.md: -------------------------------------------------------------------------------- 1 | # Affiliates 2 | 3 | This folder contains an extract of information collected from the [Internet Archive WayBack Machine](https://archive.org/web/) 4 | 5 | Archived versions of pages were downloaded using the [wayback-machine-scraper](https://github.com/sangaline/wayback-machine-scraper) 6 | written by Evan Sangaline. 7 | 8 | Contains three CSV files: 9 | 10 | ### 01-internetarchive-domain-to-forwarded.csv 11 | 12 | This is the initial extraction of the forwarded to locations of domains. 13 | 14 | Schema is as follows: 15 | 16 | * initial domain - The domain that was requested from wayback, either domain or domain plus /* 17 | * forwarded url - The URL that we were forwarded to, through JS redirection. 18 | * snapshotname - The wayback snapshot name 19 | * snapshot_date - The date of the wayback snapshot. 20 | 21 | ### 02-internetarchive-unique-urls.csv 22 | 23 | We iterated on the above data file to get a list of affiliate ids. 24 | 25 | * forwarded url - the Url we were forwarded to 26 | * has a unique id? - Boolean if we see anything that might be an affiliate id 27 | * Id - What we determine to be an affiliate id 28 | * Domain - FQDN of the forwared url. 29 | * Pairing domain to Affiliate ID - pairing of domain to id 30 | 31 | ### 03-final-pairing-domain-to-affiliate-id.txt 32 | 33 | Final list of domains mapped to affiliate ids. 34 | 35 | The other additions to the final table were items that were previously reported in the OONI report or which 36 | we manually visited. 37 | 38 | For example for the Coinhive URL: http://cnhv[ . ]co/fmwi This was because the domain http://ads[ . ]vidz4fun[ . ]com/vad1.html 39 | which we see in the OONI data forwards to this, but wayback machine did not retain this snapshot. The details of this 40 | redirect can be seen in ```coinhive-redir.txt``` -------------------------------------------------------------------------------- /affiliates/coinhive-redir.txt: -------------------------------------------------------------------------------- 1 | $ curl -v http://ads.vidz4fun.com/vad1.html 2 | 3 | 4 | * Trying 54.76.71.181... 5 | * TCP_NODELAY set 6 | * Connected to ads.vidz4fun.com (54.76.71.181) port 80 (#0) 7 | > GET /vad1.html HTTP/1.1 8 | > Host: ads.vidz4fun.com 9 | > User-Agent: curl/7.52.1 10 | > Accept: */* 11 | > 12 | < HTTP/1.1 200 OK 13 | < Date: Wed, 14 Feb 2018 21:43:26 GMT 14 | < Content-Type: text/html 15 | < Content-Length: 300 16 | < Connection: keep-alive 17 | < Server: nginx/1.4.6 (Ubuntu) 18 | < Last-Modified: Thu, 23 Nov 2017 22:47:24 GMT 19 | < ETag: "5a174ffc-12c" 20 | < Accept-Ranges: bytes 21 | < 22 | 23 | 24 | 25 | 26 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | * Curl_http_done: called premature == 0 44 | * Connection #0 to host ads.vidz4fun.com left intact 45 | -------------------------------------------------------------------------------- /censorship/README.md: -------------------------------------------------------------------------------- 1 | This directory includes all the data related to how we looked at censorship in Turkey and Egypt 2 | for this report. 3 | 4 | We tested the local list of both Turkey and Egypt available on 5 | the [Citizen Lab Test List repository](https://github.com/citizenlab/test-lists) 6 | 7 | We captured packets and ran the [test-censorship.py](test-censorship.py) script against our test lists. 8 | We then ran the collected packet capture against the [parse-pcap-results.py](parse-pcap-results.py) script to 9 | get a list of packets that matched the ipid and tcp flag signature. 10 | 11 | We then got the list of URLs that were blocked in each country. 12 | 13 | * [blocked-eg.txt](blocked-eg.txt) 14 | * [blocked-tr.txt](blocked-tr.txt) -------------------------------------------------------------------------------- /censorship/blocked-eg.txt: -------------------------------------------------------------------------------- 1 | 6april.org 2 | ahmedgamalziada.blogspot.com.eg 3 | akhbaralaalam.net 4 | akhbaralsabah.com 5 | akhbarturkiya.com 6 | alalam.ir 7 | alarab.qa 8 | alaraby.tv 9 | albedaiah.com 10 | al-gornal.com 11 | alkessa.com 12 | almanassa.com 13 | almesryoon.com 14 | al-sharq.com 15 | amlalommah.net 16 | anonymizer.ru 17 | anonymizer.secuser.com 18 | ansarportsaed.com 19 | arabi21.com 20 | arab-turkey.com 21 | astrill.com 22 | balkans.aljazeera.net 23 | blogs.aljazeera.net 24 | cairoportal.com 25 | cyberghostvpn.com 26 | dailynewsegypt.com 27 | doc.aljazeera.net 28 | elbadil.com 29 | elbehira.net 30 | elsharq.web.tv 31 | fakartany.com 32 | fastproxyservers.org 33 | freenetproject.org 34 | getfoxyproxy.org 35 | geti2p.net 36 | hamas.ps 37 | harmees.com 38 | hasamegypt.com 39 | hrw.org 40 | httptunnel.ge 41 | ida2at.com 42 | idolproxy.com 43 | ikhwanonline.com 44 | ikhwanonline.info 45 | institute.aljazeera.net 46 | kh-press.com 47 | learning.aljazeera.net 48 | liberties.aljazeera.com 49 | madaad.net 50 | madamasr.com 51 | mahnor.com 52 | marsadpress.net 53 | masralekhbaria.com 54 | masreiat.com 55 | medium.com 56 | midan.aljazeera.net 57 | motamemservice.com 58 | mubasher.aljazeera.net 59 | multiproxy.org 60 | nabdapp.com 61 | premiumproxy.net 62 | proxybonanza.com 63 | proxyhttp.net 64 | proxylistplus.com 65 | proxylisty.com 66 | proxyservers.pro 67 | psiphon.ca 68 | pzou.win 69 | qalyubiagate.com 70 | rassd.com 71 | revsoc.me 72 | sasapost.com 73 | securefor.com 74 | sockslist.net 75 | sport.aljazeera.net 76 | steadyproxy.cf 77 | steadyproxy.ga 78 | stealthy.co 79 | stopcensoring.me 80 | studies.aljazeera.net 81 | thegeekdaily.com 82 | thenewkhalij.org 83 | torproject.org 84 | trustedproxies.com 85 | tunnelbear.com 86 | tunnelguru.com 87 | tunnello.com 88 | turk.life 89 | turkpress.co 90 | unblocked.me 91 | uproxy.org 92 | vpnfacile.net 93 | wamtimes.com 94 | web.archive.org 95 | www.0proxy.space 96 | www.11proxy.pw 97 | www.2016webproxy.cf 98 | www.2016webproxy.ml 99 | www.22proxy.pw 100 | www.2fastsurfer.com 101 | www.33proxy.pw 102 | www.345proxy.ml 103 | www.456proxy.ml 104 | www.4everproxy.com 105 | www.4proxy.de 106 | www.5proxy.space 107 | www.66proxy.pw 108 | www.678proxy.ml 109 | www.789proxy.cf 110 | www.8proxy.space 111 | www.99proxy.com 112 | www.9proxy.space 113 | www.accessmeproxy.net 114 | www.accessproxy.science 115 | www.aceproxy.com 116 | www.activewebproxy.cf 117 | www.airvpn.org 118 | www.akproxy.gq 119 | www.al-akhbar.com 120 | www.alalam.ir 121 | www.alarab.qa 122 | www.alaraby.co.uk 123 | www.alborsanews.com 124 | www.aljazeera.com 125 | www.aljazeera.net 126 | www.aljisr-news.com 127 | www.allunblocked.com 128 | www.alscene.com 129 | www.al-sharq.com 130 | www.al-watan.com 131 | www.amlalommah.net 132 | www.anonymizer.com 133 | www.anonymouse.org 134 | www.anonymox.net 135 | www.anonymsurfen.com 136 | www.anonym.to 137 | www.ansarportsaed.com 138 | www.applepieproxy.xyz 139 | www.arabsolaa.com 140 | www.arabyexpressnews.com 141 | www.ardmasr.com 142 | www.asrararabiya.com 143 | www.atozproxy.com 144 | www.avpn.win 145 | www.awebproxy.com 146 | www.awmproxy.com 147 | www.bananaproxy.eu 148 | www.befreeproxy.com 149 | www.bel-ahmar.net 150 | www.bestvpn.com 151 | www.betternet.co 152 | www.bitchop.net 153 | www.blackvpn.com 154 | www.blewpass.com 155 | www.boomproxy.com 156 | www.bramka-proxy.pl 157 | www.bulgariaproxy.net 158 | www.buqd.xyz 159 | www.bvpn.win 160 | www.bypasstool.gq 161 | www.cactusvpn.com 162 | www.cairoportal.com 163 | www.coolbits.org 164 | www.cyberghostvpn.com 165 | www.dacd.win 166 | www.dailynewsegypt.com 167 | www.dailysabah.com 168 | www.dakahliaikhwan.com 169 | www.disconnect.me 170 | www.dolopo.net 171 | www.dontfilter.us 172 | www.easy-hide-ip.com 173 | www.ec-rf.org 174 | www.egyptdailynews.com 175 | www.egyptwindow.net 176 | www.egyrep.com 177 | www.elborsanews.com 178 | www.elshaab.org 179 | www.elwehda.com 180 | www.englandproxy.co.uk 181 | www.essada.net 182 | www.fastusaproxy.com 183 | www.fekrapost.com 184 | www.fiberproxy.net 185 | www.filterbypass.me 186 | www.fishproxy.com 187 | www.fj-p.com 188 | www.freeeuroproxy.com 189 | www.freeproxy.asia 190 | www.free-proxy-list.net 191 | www.freeproxylists.net 192 | www.freeproxy.ru 193 | www.freeproxyserver.co 194 | www.free-proxyserver.com 195 | www.freeproxyserver.ovh 196 | www.free-proxy.xyz 197 | www.freewebproxyserver.pw 198 | www.freshfreeproxy.ga 199 | www.freshfreeproxy.gq 200 | www.freshunblock.gq 201 | www.gardennetworks.com 202 | www.gatherproxy.com 203 | www.genmirror.com 204 | www.getfoxyproxy.org 205 | www.gharbiaonline.com 206 | www.gimmeproxy.com 207 | www.gizlenin.com 208 | www.gizlibaglanti.com 209 | www.gowebsite.link 210 | www.guardster.com 211 | www.gulf-times.com 212 | www.gwady.net 213 | www.helloproxy.science 214 | www.hidebuzz.us 215 | www.hideipvpn.com 216 | www.hideme.be 217 | www.hidemebro.com 218 | www.hidemyass.com 219 | www.hide-my-ip.com 220 | www.hidenseek.org 221 | www.hideoxy.com 222 | www.hidester.com 223 | www.hiload.in 224 | www.hola.org 225 | www.hollandproxy.eu 226 | www.homeproxy.com 227 | www.hopeproxy.com 228 | www.hotspotshield.com 229 | www.hsselite.com 230 | www.http-tunnel.com 231 | www.huffpostarabi.com 232 | www.hurriyetdailynews.com 233 | www.ibvpn.com 234 | www.idcloak.com 235 | www.idolproxy.com 236 | www.ikhwanonline.com 237 | www.ikhwanwiki.com 238 | www.inetprivacy.com 239 | www.instantunblock.com 240 | www.invisiblesurf.review 241 | www.ipjetable.net 242 | www.ipredator.se 243 | www.ipvanish.com 244 | www.ispunblock.com 245 | www.itshidden.eu 246 | www.ivacy.com 247 | www.jatoeg.org 248 | www.jmarshall.com 249 | www.justproxy.co.uk 250 | www.kingsurfproxy.com 251 | www.korabia.com 252 | www.kproxy.com 253 | www.le-vpn.com 254 | www.libertyvpn.net 255 | www.linkmetube.com 256 | www.listproxysites.com 257 | www.llowll.net 258 | www.lopana.com 259 | www.luminati.io 260 | www.maddw.com 261 | www.mainproxy.pw 262 | www.manalaa.net 263 | www.masr11.com 264 | www.masralarabia.com 265 | www.masrmix.com 266 | www.megaproxy.com 267 | www.metproxy.com 268 | www.miniprox.com 269 | www.moheet.com 270 | www.mom-rsf.org 271 | www.mullvad.net 272 | www.my-proxy.com 273 | www.myusvpn.com 274 | www.nanoproxy.de 275 | www.netherlandsproxyserver.club 276 | www.newipnow.com 277 | www.newproxy.ninja 278 | www.newsproxy.xyz 279 | www.ninjacloak.com 280 | www.ninjaweb.xyz 281 | www.noonpost.org 282 | www.nordvpn.com 283 | www.northghost.com 284 | www.notaproxy.co.uk 285 | www.nvpn.net 286 | www.odsh.win 287 | www.ooproxy.pw 288 | www.orangeproxy.net 289 | www.otlp.xyz 290 | www.ourwebproxy.ml 291 | www.overplay.net 292 | www.pandashield.com 293 | www.peacefire.org 294 | www.perfect-privacy.com 295 | www.phproxy.co 296 | www.pinkproxy.xyz 297 | www.proproxy.me 298 | www.proxfree.com 299 | www.proxhow.com 300 | www.proxify.com 301 | www.proxite.eu 302 | www.proxprox.com 303 | www.proxtube.com 304 | www.proxy4free.com 305 | www.proxy-anywhere.com 306 | www.proxyarab.com 307 | www.proxyb.com 308 | www.proxycloud.net 309 | www.proxydb.net 310 | www.proxy-deal.xyz 311 | www.proxydocker.com 312 | www.proxyeuro.pw 313 | www.proxyforme.ml 314 | www.proxyhub.in 315 | www.proxyisp.com 316 | www.proxyjapan.nu 317 | www.proxy-list.org 318 | www.proxylistpro.com 319 | www.proxyload.net 320 | www.proxy.my-addr.com 321 | www.proxynova.com 322 | www.proxyone.net 323 | www.proxyserver.com 324 | www.proxyserver.ovh 325 | www.proxy-service.de 326 | www.proxysite.club 327 | www.proxysite.com 328 | www.proxyswitcher.com 329 | www.proxyunblocker.org 330 | www.proxy.unblocksiter.com 331 | www.proxyvpn.gq 332 | www.proxyweb.com.es 333 | www.proxy.world 334 | www.proxy.yt 335 | www.proxyzan.info 336 | www.prx.im 337 | www.publicproxyservers.com 338 | www.purevpn.com 339 | www.qantara.de 340 | www.qtv.qa 341 | www.quickprox.com 342 | www.quickproxy.co.uk 343 | www.raya.com 344 | www.relakks.com 345 | www.reporter-ohne-grenzen.de 346 | www.rexoss.com 347 | www.safervpn.com 348 | www.saoudiproxy.info 349 | www.schoolproxylist.com 350 | www.school-proxy.net 351 | www.secretproxy.org 352 | www.securewebproxy.cf 353 | www.securewebproxy.gq 354 | www.seeproxy.me 355 | www.sharkiaonline.com 356 | www.shinyproxy.com 357 | www.showvision.info 358 | www.site2unblock.com 359 | www.skullproxy.com 360 | www.smarthide.com 361 | www.socks-proxy.net 362 | www.speedproxy.online 363 | www.speedproxyserver.com 364 | www.spotflux.com 365 | www.sslpro.org 366 | www.sslproxies.org 367 | www.sslsecureproxy.com 368 | www.stealthproxy.co.uk 369 | www.stupidcensorship.com 370 | www.suedeproxy.info 371 | www.sumrando.com 372 | www.supervpn.net 373 | www.surfbouncer.com 374 | www.surfeasy.com 375 | www.surfweb.gq 376 | www.swissvpn.net 377 | www.switchvpn.net 378 | www.tabproxy.com 379 | www.takesper.gq 380 | www.thebestproxy.info 381 | www.thehiddenwiki.org 382 | www.thepeninsulaqatar.com 383 | www.thevpn.guru 384 | www.tinyproxy.ga 385 | www.tiranwsanafir.com 386 | www.tomatoproxy.eu 387 | www.toolur.com 388 | www.top10bestvpn.com 389 | www.topnewproxy.gq 390 | www.topproxy.gq 391 | www.torguard.net 392 | www.torvpn.com 393 | www.totalvpn.com 394 | www.trust.zone 395 | www.tunnelbear.com 396 | www.turkeyproxy.com 397 | www.ukproxyserver.co.uk 398 | www.ultimate-anonymity.com 399 | www.ultimateproxy.net 400 | www.ultrasawt.com 401 | www.ultrasurf.us 402 | www.umayyapress.com 403 | www.unblock123.com 404 | www.unblockaccess.com 405 | www.unblockallwebsite.com 406 | www.unblockblocked.net 407 | www.unblockbook.biz 408 | www.unblockdomain.com 409 | www.unblockfreeproxy.com 410 | www.unblockmyweb.com 411 | www.unblockproxy.me 412 | www.unblockproxy.xyz 413 | www.unblock-sites.com 414 | www.unblockstreaming.com 415 | www.unblockvideos.com 416 | www.unblockweb.co 417 | www.unblockwebs.gq 418 | www.unblockwebsite.online 419 | www.unblockwebsites.us 420 | www.unblockyoutubefree.net 421 | www.unblockytproxy.com 422 | www.undofilters.com 423 | www.unhideing.link 424 | www.unotelly.com 425 | www.usproxy.nu 426 | www.us-proxy.org 427 | www.uswebproxy.com 428 | www.videounblock.com 429 | www.videounblocker.net 430 | www.vidproxy.com 431 | www.vivome.xyz 432 | www.vload.net 433 | www.vobas.com 434 | www.vpn4all.com 435 | www.vpnbook.com 436 | www.vpnbrowse.com 437 | www.vpndeluxe.com 438 | www.vpn.ht 439 | www.vpnlux.com 440 | www.vpnsecure.me 441 | www.vpntool.com 442 | www.vpnworldwide.com 443 | www.waraa-elahdath.com 444 | www.watanserb.com 445 | www.webproxy2.com 446 | www.web-proxy.co 447 | www.webproxyfree.net 448 | www.webproxy.to 449 | www.websurf.in 450 | www.windscribe.com 451 | www.wowvpn.net 452 | www.xeronet.xyz 453 | www.xitenow.com 454 | www.xitesite.com 455 | www.xroxy.com 456 | www.yellowproxy.net 457 | www.yourcheat.com 458 | www.yourprivatevpn.com 459 | www.youtubeproxy.pw 460 | www.yxorproxy.com 461 | www.zacebookpk.com 462 | www.zalmos.com 463 | www.zend2.com 464 | www.zendproxy.com 465 | www.zensur.freerk.com 466 | www.zetasurf.info 467 | www.zpn.im 468 | yanairgate.net 469 | ytunblocker.com 470 | 471 | -------------------------------------------------------------------------------- /censorship/blocked-tr.txt: -------------------------------------------------------------------------------- 1 | nos.nl 2 | wikipedia.org 3 | pkkonline.net 4 | torproject.org -------------------------------------------------------------------------------- /censorship/parse-pcap-results.py: -------------------------------------------------------------------------------- 1 | import dpkt 2 | import socket 3 | import collections 4 | 5 | PCAP_FILE="test.pcap" 6 | THIS_SRC_IP = "...." 7 | 8 | 9 | def tcp_flags(flags): 10 | ret = '' 11 | if flags & dpkt.tcp.TH_FIN: 12 | ret = ret + 'F' 13 | if flags & dpkt.tcp.TH_SYN: 14 | ret = ret + 'S' 15 | if flags & dpkt.tcp.TH_RST: 16 | ret = ret + 'R' 17 | if flags & dpkt.tcp.TH_ACK: 18 | ret = ret + 'A' 19 | if flags & dpkt.tcp.TH_URG: 20 | ret = ret + 'U' 21 | if flags & dpkt.tcp.TH_ECE: 22 | ret = ret + 'E' 23 | if flags & dpkt.tcp.TH_CWR: 24 | ret = ret + 'C' 25 | return ret 26 | 27 | pcap = dpkt.pcap.Reader(open(PCAP_FILE, "rb")) 28 | window = collections.deque([], 100) 29 | for ts, buf in pcap: 30 | eth = dpkt.ethernet.Ethernet(buf) 31 | if not isinstance(eth.data, dpkt.ip.IP): 32 | continue 33 | ip = eth.data 34 | if not isinstance(ip.data, dpkt.tcp.TCP): 35 | continue 36 | tcp = ip.data 37 | the_data = tcp.data 38 | 39 | if socket.inet_ntoa(ip.src) == THIS_SRC_IP and len(tcp.data) > 0: 40 | window.append(ip) 41 | 42 | if (ip.id == 13330 and tcp_flags(ip.data.flags) == "RA") in the_data: 43 | for wip in window: 44 | if wip.data.sport == tcp.dport: 45 | print(wip.data.data.split(b"\r\n")[1].split(b":")[1].lstrip().decode("utf-8")) 46 | -------------------------------------------------------------------------------- /censorship/test-censorship.py: -------------------------------------------------------------------------------- 1 | import socket 2 | import time 3 | from urlparse import urlparse 4 | 5 | INFRASTRUCTURE_IP="...." 6 | URL_LIST="urls-to-test" 7 | 8 | with open(URL_LIST) as f: 9 | for line in f: 10 | the_url = line.rstrip() 11 | o = urlparse(the_url) 12 | domin = o.netloc 13 | path = o.path 14 | if path == '' or path == ' ': 15 | path = "/" 16 | try: 17 | print(b"GET " + str.encode(path) + " HTTP/1.1\r\nHost: " + str.encode(domin) + b"\r\n\r\n") 18 | s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 19 | s.connect((INFRASTRUCTURE_IP, 80)) 20 | s.sendall(b"GET " + str.encode(path) + " HTTP/1.1\r\nHost: " + str.encode(domin) + b"\r\n\r\n") 21 | s.close() 22 | time.sleep(0.1) 23 | except: 24 | pass 25 | -------------------------------------------------------------------------------- /censys/README.md: -------------------------------------------------------------------------------- 1 | This directory contains a **summary** of data collected with [censys](censys.io). Censys is 2 | a platform that allows security researchers to access current and historical data on publicly 3 | available ip, and certificate data. 4 | 5 | Presented here is a summary of aggregate result counts found on Censys in the course of 6 | this research. We used Censys to find all cases where redirections were being made to the 7 | static.dbmads[ . ]com domain. 8 | 9 | To do this we ran the following query against the Censys historical ipv4 dataset 10 | through BigQuery. We used the following base query: 11 | 12 | ``` 13 | #standardSQL 14 | SELECT 15 | distinct(ip), 16 | p80.http.get.headers.location p80loc, 17 | p8080.http.get.headers.location p8080loc, 18 | p8888.http.get.headers.location p8888loc, 19 | p7547.cwmp.get.headers.location p7547loc 20 | FROM 21 | `censys-io.ipv4_public.20180104` 22 | WHERE 23 | p80.http.get.headers.location LIKE '%static.dbmads.com%' 24 | or 25 | p8080.http.get.headers.location LIKE '%static.dbmads.com%' 26 | or 27 | p7547.cwmp.get.headers.location LIKE '%static.dbmads.com%' 28 | or 29 | p8888.http.get.headers.location LIKE '%static.dbmads.com%' 30 | ; 31 | ``` 32 | 33 | We then did this against all tables and annotated the result in 34 | ```censys-summary.csv``` and noted how many rows were returned 35 | in the ```result_ct``` column. Since there are schema changes 36 | in the whole data set we noted any such changes in the ```notes``` 37 | column. 38 | 39 | 40 | 41 | -------------------------------------------------------------------------------- /censys/censys-summary.csv: -------------------------------------------------------------------------------- 1 | table_name,result_ct,notes 2 | `censys-io.ipv4_public.20150912`,0, 3 | `censys-io.ipv4_public.20151012`,0, 4 | `censys-io.ipv4_public.20151019`,0, 5 | `censys-io.ipv4_public.20151026`,0, 6 | `censys-io.ipv4_public.20151109`,0, 7 | `censys-io.ipv4_public.20151116`,0, 8 | `censys-io.ipv4_public.20151123`,0, 9 | `censys-io.ipv4_public.20151214`,0, 10 | `censys-io.ipv4_public.20151221`,0, 11 | `censys-io.ipv4_public.20151228`,0, 12 | `censys-io.ipv4_public.20160208`,0, 13 | `censys-io.ipv4_public.20160215`,0, 14 | `censys-io.ipv4_public.20160222`,0, 15 | `censys-io.ipv4_public.20160229`,0, 16 | `censys-io.ipv4_public.20160307`,0, 17 | `censys-io.ipv4_public.20160314`,0, 18 | `censys-io.ipv4_public.20160321`,0, 19 | `censys-io.ipv4_public.20160328`,0, 20 | `censys-io.ipv4_public.20160404`,0, 21 | `censys-io.ipv4_public.20160411`,0, 22 | `censys-io.ipv4_public.20160425`,0, 23 | `censys-io.ipv4_public.20160502`,0, 24 | `censys-io.ipv4_public.20160516`,0, 25 | `censys-io.ipv4_public.20160523`,0, 26 | `censys-io.ipv4_public.20160616`,0, 27 | `censys-io.ipv4_public.20160620`,0, 28 | `censys-io.ipv4_public.20160628`,0, 29 | `censys-io.ipv4_public.20160705`,0, 30 | `censys-io.ipv4_public.20160712`,0, 31 | `censys-io.ipv4_public.20160719`,0, 32 | `censys-io.ipv4_public.20160724`,0, 33 | `censys-io.ipv4_public.20160806`,0, 34 | `censys-io.ipv4_public.20160814`,0, 35 | `censys-io.ipv4_public.20160823`,0, 36 | `censys-io.ipv4_public.20160831`,0, 37 | `censys-io.ipv4_public.20160905`,0, 38 | `censys-io.ipv4_public.20160915`,0, 39 | `censys-io.ipv4_public.20160925`,0, 40 | `censys-io.ipv4_public.20161002`,0, 41 | `censys-io.ipv4_public.20161005`,0, 42 | `censys-io.ipv4_public.20161015`,0, 43 | `censys-io.ipv4_public.20161018`,0, 44 | `censys-io.ipv4_public.20161025`,0, 45 | `censys-io.ipv4_public.20161101`,0, 46 | `censys-io.ipv4_public.20161105`,0, 47 | `censys-io.ipv4_public.20161109`,0, 48 | `censys-io.ipv4_public.20161121`,0, 49 | `censys-io.ipv4_public.20161205`,0, 50 | `censys-io.ipv4_public.20161214`,0, 51 | `censys-io.ipv4_public.20161220`,0, 52 | `censys-io.ipv4_public.20161227`,0, 53 | `censys-io.ipv4_public.20170101`,0, 54 | `censys-io.ipv4_public.20170106`,0, 55 | `censys-io.ipv4_public.20170113`,0, 56 | `censys-io.ipv4_public.20170120`,0, 57 | `censys-io.ipv4_public.20170123`,0, 58 | `censys-io.ipv4_public.20170207`,0, 59 | `censys-io.ipv4_public.20170225`,0,no_more_p7545 60 | `censys-io.ipv4_public.20170228`,0, 61 | `censys-io.ipv4_public.20170317`,0, 62 | `censys-io.ipv4_public.20170321`,0, 63 | `censys-io.ipv4_public.20170328`,0, 64 | `censys-io.ipv4_public.20170404`,0, 65 | `censys-io.ipv4_public.20170411`,0, 66 | `censys-io.ipv4_public.20170418`,0, 67 | `censys-io.ipv4_public.20170425`,0, 68 | `censys-io.ipv4_public.20170502`,0, 69 | `censys-io.ipv4_public.20170509`,0, 70 | `censys-io.ipv4_public.20170516`,0, 71 | `censys-io.ipv4_public.20170523`,0, 72 | `censys-io.ipv4_public.20170608`,0, 73 | `censys-io.ipv4_public.20170629`,36, 74 | `censys-io.ipv4_public.20170704`,36, 75 | `censys-io.ipv4_public.20170715`,0, 76 | `censys-io.ipv4_public.20170718`,0, 77 | `censys-io.ipv4_public.20170725`,0, 78 | `censys-io.ipv4_public.20170801`,0, 79 | `censys-io.ipv4_public.20170808`,0, 80 | `censys-io.ipv4_public.20170815`,0, 81 | `censys-io.ipv4_public.20170905`,0, 82 | `censys-io.ipv4_public.20170912`,0, 83 | `censys-io.ipv4_public.20170919`,0, 84 | `censys-io.ipv4_public.20170920`,0, 85 | `censys-io.ipv4_public.20170921`,0, 86 | `censys-io.ipv4_public.20170922`,0, 87 | `censys-io.ipv4_public.20170923`,0, 88 | `censys-io.ipv4_public.20170924`,0, 89 | `censys-io.ipv4_public.20170925`,0, 90 | `censys-io.ipv4_public.20170926`,0, 91 | `censys-io.ipv4_public.20170927`,0, 92 | `censys-io.ipv4_public.20170928`,0, 93 | `censys-io.ipv4_public.20170929`,0, 94 | `censys-io.ipv4_public.20170930`,0, 95 | `censys-io.ipv4_public.20171001`,0, 96 | `censys-io.ipv4_public.20171002`,0, 97 | `censys-io.ipv4_public.20171003`,0, 98 | `censys-io.ipv4_public.20171004`,0, 99 | `censys-io.ipv4_public.20171005`,0, 100 | `censys-io.ipv4_public.20171006`,0, 101 | `censys-io.ipv4_public.20171007`,0, 102 | `censys-io.ipv4_public.20171008`,0, 103 | `censys-io.ipv4_public.20171009`,0, 104 | `censys-io.ipv4_public.20171010`,0, 105 | `censys-io.ipv4_public.20171011`,0, 106 | `censys-io.ipv4_public.20171012`,0, 107 | `censys-io.ipv4_public.20171013`,0, 108 | `censys-io.ipv4_public.20171014`,0, 109 | `censys-io.ipv4_public.20171015`,0, 110 | `censys-io.ipv4_public.20171016`,0, 111 | `censys-io.ipv4_public.20171017`,0, 112 | `censys-io.ipv4_public.20171018`,0, 113 | `censys-io.ipv4_public.20171019`,0, 114 | `censys-io.ipv4_public.20171020`,0, 115 | `censys-io.ipv4_public.20171021`,0, 116 | `censys-io.ipv4_public.20171022`,0, 117 | `censys-io.ipv4_public.20171023`,0, 118 | `censys-io.ipv4_public.20171024`,0, 119 | `censys-io.ipv4_public.20171025`,0, 120 | `censys-io.ipv4_public.20171026`,0, 121 | `censys-io.ipv4_public.20171027`,0, 122 | `censys-io.ipv4_public.20171028`,0, 123 | `censys-io.ipv4_public.20171029`,0, 124 | `censys-io.ipv4_public.20171030`,0, 125 | `censys-io.ipv4_public.20171031`,0, 126 | `censys-io.ipv4_public.20171101`,0, 127 | `censys-io.ipv4_public.20171102`,0, 128 | `censys-io.ipv4_public.20171103`,0, 129 | `censys-io.ipv4_public.20171107`,0, 130 | `censys-io.ipv4_public.20171109`,0,no_more_p8080_and_p8888 131 | `censys-io.ipv4_public.20171110`,0, 132 | `censys-io.ipv4_public.20171113`,0, 133 | `censys-io.ipv4_public.20171114`,0, 134 | `censys-io.ipv4_public.20171115`,0, 135 | `censys-io.ipv4_public.20171116`,0, 136 | `censys-io.ipv4_public.20171117`,0, 137 | `censys-io.ipv4_public.20171118`,0, 138 | `censys-io.ipv4_public.20171119`,0, 139 | `censys-io.ipv4_public.20171120`,0, 140 | `censys-io.ipv4_public.20171121`,0, 141 | `censys-io.ipv4_public.20171122`,0, 142 | `censys-io.ipv4_public.20171123`,0, 143 | `censys-io.ipv4_public.20171124`,0, 144 | `censys-io.ipv4_public.20171125`,0, 145 | `censys-io.ipv4_public.20171126`,0, 146 | `censys-io.ipv4_public.20171127`,0, 147 | `censys-io.ipv4_public.20171128`,0, 148 | `censys-io.ipv4_public.20171129`,0, 149 | `censys-io.ipv4_public.20171130`,0, 150 | `censys-io.ipv4_public.20171201`,0, 151 | `censys-io.ipv4_public.20171202`,0,ERR: permission_denied 152 | `censys-io.ipv4_public.20171203`,0, 153 | `censys-io.ipv4_public.20171204`,0, 154 | `censys-io.ipv4_public.20171205`,0, 155 | `censys-io.ipv4_public.20171206`,84, 156 | `censys-io.ipv4_public.20171207`,101, 157 | `censys-io.ipv4_public.20171208`,101, 158 | `censys-io.ipv4_public.20171209`,101, 159 | `censys-io.ipv4_public.20171210`,101, 160 | `censys-io.ipv4_public.20171211`,101, 161 | `censys-io.ipv4_public.20171212`,101, 162 | `censys-io.ipv4_public.20171213`,96, 163 | `censys-io.ipv4_public.20171214`,0, 164 | `censys-io.ipv4_public.20171216`,0, 165 | `censys-io.ipv4_public.20171217`,0, 166 | `censys-io.ipv4_public.20171218`,0, 167 | `censys-io.ipv4_public.20171219`,0,ERR: permission_denied 168 | `censys-io.ipv4_public.20171220`,0, 169 | `censys-io.ipv4_public.20171221`,0, 170 | `censys-io.ipv4_public.20171222`,0, 171 | `censys-io.ipv4_public.20171223`,0, 172 | `censys-io.ipv4_public.20171224`,0, 173 | `censys-io.ipv4_public.20171225`,0, 174 | `censys-io.ipv4_public.20171226`,0, 175 | `censys-io.ipv4_public.20171227`,0, 176 | `censys-io.ipv4_public.20171228`,0, 177 | `censys-io.ipv4_public.20171229`,0, 178 | `censys-io.ipv4_public.20171230`,0, 179 | `censys-io.ipv4_public.20171231`,0, 180 | `censys-io.ipv4_public.20180103`,0, 181 | `censys-io.ipv4_public.20180104`,5443, 182 | `censys-io.ipv4_public.20180105`,5443, 183 | `censys-io.ipv4_public.20180106`,5443, 184 | `censys-io.ipv4_public.20180107`,5443, 185 | `censys-io.ipv4_public.20180108`,5443, 186 | `censys-io.ipv4_public.20180109`,5443, 187 | `censys-io.ipv4_public.20180110`,5163, 188 | `censys-io.ipv4_public.20180111`,0, 189 | `censys-io.ipv4_public.20180112`,0, 190 | `censys-io.ipv4_public.20180113`,0, 191 | `censys-io.ipv4_public.20180114`,0, 192 | `censys-io.ipv4_public.20180115`,0, 193 | `censys-io.ipv4_public.20180116`,0, 194 | `censys-io.ipv4_public.20180117`,0, 195 | `censys-io.ipv4_public.20180118`,0, 196 | `censys-io.ipv4_public.20180119`,0, 197 | `censys-io.ipv4_public.20180120`,0, 198 | `censys-io.ipv4_public.20180121`,0, 199 | `censys-io.ipv4_public.20180122`,0, 200 | `censys-io.ipv4_public.20180123`,0, 201 | `censys-io.ipv4_public.20180124`,0, 202 | `censys-io.ipv4_public.20180125`,0, 203 | `censys-io.ipv4_public.20180126`,0, 204 | `censys-io.ipv4_public.20180127`,0, 205 | `censys-io.ipv4_public.20180128`,0, 206 | `censys-io.ipv4_public.20180129`,0, 207 | `censys-io.ipv4_public.20180130`,0, 208 | `censys-io.ipv4_public.20180131`,0, 209 | `censys-io.ipv4_public.20180201`,0, 210 | -------------------------------------------------------------------------------- /ooni-historical/README.md: -------------------------------------------------------------------------------- 1 | This folder contains analysis of [OONI](https://ooni.torproject.org/) data in Egypt where we see HTTP 307 responses. 2 | 3 | OONI is a project that provides free and open source software for observation of network interference and measurement. It is a project where users run software thats tests the accessibility of URLs and network services. These results are uploaded back to OONI and shared with the community. We parsed [OONI measurement files](https://measurements.ooni.torproject.org/) to find all HTTP/307 redirections within Egypt. 4 | 5 | Main output here is the ```egypt-307.csv``` which is the result of parsing OONI JSONs from Egypt from 2016-08-01 to 2018-02-01. 6 | 7 | * Columns in CSV are: 8 | * URL - The URL that was tested by an OONI user. 9 | * DATETIME - Timestamp of when the test was done. 10 | * OONI_URL - Permalink to the OONI JSON file where this was seen. 11 | * FIRST307HEADERS - The header of the first response seen. 12 | 13 | For more information about analyzing and accessing OONI data [see this post by OONI](https://ooni.torproject.org/post/mining-ooni-data/) -------------------------------------------------------------------------------- /pcaps/README.md: -------------------------------------------------------------------------------- 1 | # PCAPs 2 | 3 | This directory contains packet capture files of injections seen in the Bad Traffic Citizen Lab report. 4 | 5 | ## adhose-trickle-riseupvpn.pcap 6 | 7 | A test from [RiseUp VPN](https://riseup.net/en/vpn) against an infrastructure IP in Egypt. We sent two requests, the first with host header "copticpope.org" and the second with host header "babylon-x.com." Both requests triggered AdHose trickle mode injections. 8 | 9 | ## egypt-packetlogic-ttl-localization.pcap 10 | 11 | A test from [RiseUp VPN](https://riseup.net/en/vpn) against an infrastructure IP in Egypt. We sent a request with host header "copticpope.org", including a TTL-limited FIN/ACK packet for various TTL values. This helped us localize at which hop the DPI device involved in AdHose was seeing the FIN/ACK and tearing down its local connection state. We then sent a TTL-limited request with host header "aljazeera.net", which is a website known to be blocked in Egypt. This allowed us to verify that censorship was happening at the same hop (and likely at the same device) as AdHose. 12 | 13 | ## turkey-malware-injection.pcap 14 | 15 | A test from [DigitalOcean](https://www.digitalocean.com/) against IP addresses we observed to be targeted in four provinces in Turkey. We requested a variety of targeted files from each IP, resulting in spyware injection. -------------------------------------------------------------------------------- /pcaps/adhose-trickle-riseupvpn.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citizenlab/badtraffic/b4de770407a51fc664430ac56e74190ffb571b4a/pcaps/adhose-trickle-riseupvpn.pcap -------------------------------------------------------------------------------- /pcaps/egypt-packetlogic-ttl-localization.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citizenlab/badtraffic/b4de770407a51fc664430ac56e74190ffb571b4a/pcaps/egypt-packetlogic-ttl-localization.pcap -------------------------------------------------------------------------------- /pcaps/turkey-malware-injection.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/citizenlab/badtraffic/b4de770407a51fc664430ac56e74190ffb571b4a/pcaps/turkey-malware-injection.pcap --------------------------------------------------------------------------------