├── LICENSE ├── README.md └── csdnMove ├── Article └── CSDNBlog /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 张玉涵 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # capturedata 2 | 3 | tools to capture network data 抓包工具总结 4 | 5 | 6 | 7 | 8 | # 抓包工具 -- Charles 9 | 10 | 基于Java 跨平台： Linux , Mac OS X, Windows 11 | [官网](https://www.charlesproxy.com/) 12 | 13 | >Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet. This includes requests, responses and the HTTP headers (which contain the cookies and caching information). 14 | 15 | ![](https://www.charlesproxy.com/assets/img/sm/23/image/mac_screen_321.png?k=f28bf8eff3) 16 | 17 | ---- 18 | **以下为Linux 平台** 19 | 20 | ## 安装 21 | 22 | [下载 charles-proxy-3.11.2.tar.gz](https://www.charlesproxy.com/latest-release/download.do) 23 | 24 | ``` 25 | # 解压 26 | tar zxvf charles-proxy-3.11.2.tar.gz 27 | 28 | # 启动 charles 29 | ./charles/bin/charles 30 | ``` 31 | 32 | ## 手机抓包 33 | 34 | **前提：使手机和电脑在一个局域网内，不一定非要是一个ip段，只要是同一个路由下就可以了，比如电脑连接的有线网ip为192.168.16.12，然后手机链接的wifi ip为192.168.1.103，但是这个有线网和无线网的最终都是来自于一个外部ip，这样的话也是可以的。** 35 | 36 | 在安卓手机的 `设置 -> wlan -> 长按连接的wifi -> 修改网络 -> 高级选项 -> 代理选手动` ， 填入电脑的 ip 和 charles 监听的 端口（默认为 8888） 37 | 38 | > 查看电脑端口 ： `ifconfig` 39 | > 查看/修改 charles 监听端口： charles的 `Proxy -> Proxy Setting` 40 | 41 | 设置好之后Charles弹出确认框，点击Allow按钮即可 42 | 43 | ![图片](https://dn-coding-net-production-pp.qbox.me/ad193bbb-4d55-4feb-9e55-0634aa6dbfce.png) 44 | 45 | ![图片](https://dn-coding-net-production-pp.qbox.me/c48ee820-f6ae-4814-b505-ff4870db1f5b.png) 46 | 47 | 48 | 49 | ## 抓取 https 50 | 51 | ![图片](https://dn-coding-net-production-pp.qbox.me/027d34d1-c783-4aca-8db3-e0066bf4ca1b.png) 52 | 53 | ### 安装证书 54 | charles的 Help -> SSL Proxy -> Install charles Root 55 | 56 | ### 设置 ssl enable 57 | 58 | 在需要抓取的 https 链接上 右键 `enable ssl proxy` 59 | ![图片](https://dn-coding-net-production-pp.qbox.me/4b91cf2a-e35b-4b9b-96ac-047eafe607e6.png) 60 | 61 | 62 | 63 | 参考 [charles使用教程指南](http://drops.wooyun.org/tips/2423) 64 | 65 | 66 | 67 | 68 | # 抓包工具 -- mitmproxy 69 | 70 | 基于python 跨平台： linux, windows， OSX (Mountain Lion and later) 71 | [官网](https://mitmproxy.org/) 72 | 73 | >Mitmproxy是一个基于python的中间人代理的框架。做过渗透测试的肯定很熟悉工具burpsuite或Fiddler，这些工具能够拦截并修改http或https的数据包，对于分析数据包交互的应用来说是非常有用的。但是这些工具都是整套给我们做好了。比如如果想自己定制一套这样的工具，添加一些自己需要的功能的话，那么我想，mitmproxy将是一个比较好的选择，因为它提供了一个可供用户调用的库libmproxy（注意该库目前只支持linux系统）。 74 | 75 | ![图片](https://dn-coding-net-production-pp.qbox.me/a2d09a30-ea72-42a1-818e-0982f7227784.png) 76 | 77 | ![图片](https://dn-coding-net-production-pp.qbox.me/85e1f4e1-c914-4ac1-9984-4492aad7096f.png) 78 | 79 | ![图片](https://dn-coding-net-production-pp.qbox.me/6992699a-cb21-44b7-9114-38d1b10f0272.png) 80 | 81 | --- 82 | **以下基于Linux台（Ubuntu14.04）** 83 | 84 | ## 下载 85 | ``` 86 | $ sudo pip install mitmproxy 87 | 88 | ``` 89 | 如果下载速度慢可以下载[tar包](https://mitmproxy.org/download/mitmproxy-0.14.0.tar.gz),需要netlib依赖， ** 推荐用pip 安装 ** 90 | 91 | ## 启动 92 | 93 | ``` 94 | $ mitmproxy -b 192.168.1.29 -p 9999 95 | 96 | ``` 97 | 98 | 输入 `？` 查看帮助， `q`返回 99 | ``` 100 | A accept all intercepted flows 101 | a accept this intercepted flow 102 | b save request/response body 103 | C clear flow list or eventlog 104 | d delete flow 105 | D duplicate flow 106 | e toggle eventlog 107 | F toggle follow flow list 108 | l set limit filter pattern 109 | L load saved flows 110 | m toggle flow mark 111 | n create a new request 112 | P copy flow to clipboard 113 | r replay request 114 | U unmark all marked flows 115 | V revert changes to request 116 | w save flows 117 | W stream flows to file 118 | X kill and delete flow, even if it's mid-intercept 119 | tab tab between eventlog and flow list 120 | enter view flow 121 | | run script on this flow 122 | 123 | ``` 124 | 125 | 手机设置好代理，界面上就可以看到请求了 126 | ![图片](https://dn-coding-net-production-pp.qbox.me/c5446754-1d45-4d7d-ba48-44b6e26d3b4f.png) 127 | 128 | ## mitmproxy 查看http请求响应 129 | 130 | `C (大写)` 清除抓包结果 131 | 132 | `j` `k` 选择请求， 回车查看详细信息 133 | 134 | ![图片](https://dn-coding-net-production-pp.qbox.me/5cfa3fd1-879b-4db6-8f05-f8885d730964.png) 135 | 136 | `tab` 切换 **Request** 和 **Response** 137 | 138 | `m` Display Mode 美化信息 139 | ![图片](https://dn-coding-net-production-pp.qbox.me/70e7c4ad-f7c0-4e7a-bea6-2c4cdd36d50b.png) 140 | 上图输入 m，在输入 s，便可以以json形式展示 141 | 142 | `e` 编辑 143 | Edit request (cookies,query,path,url,header,form,raw body,method)? 144 | Edit response (cookies,code,message,header,raw body)? 145 | 146 | ### mitmproxy拦截 (Intercept) 147 | 相当于在客户端和服务器做中间人，可以修改客户端请求，修改服务器返回 148 | 149 | 输入 `i`（代表Intercept filter）即可，此时界面便会让你输入想要拦截的条件： 150 | 151 | mitmproxy的条件拦截在默认情况下是过滤抓包的URL的。也就是说当你直接输入要拦截的条件（比如输入“weibo”），那么接下来要出现抓包会将匹配的抓包整体变黄： 152 | 153 | mitmproxy条件过滤效果 154 | mitmproxy条件拦截效果 155 | 这些橘黄色的数据包都代表被拦截了，还未发送给服务器，这个时候你就可以对这些数据包进行修改，我们选择一个数据包enter进入： 156 | 157 | mitmproxy 拦截 选择数据包 158 | 与之前的类似，输入“e”，进行request编辑模式，然后输入“h”代表要编辑request的头部: 159 | 160 | mitmproxy 编辑拦截包的头部 161 | 输入enter便可对高亮的User-Agent的值进行修改，上图的weibo版本之前是5.0的，被我改成了6.0 。我们还可以对header进行添加属性，输入“a”即可，然后使用tab分别键入key和value。这里我添加了“test－test”键值对： 162 | 163 | mitmproxy 拦截header添加键值对 164 | 至此，我对拦截的request header已经修改完毕，现在要做的就是我要认可接受这个修改，然后发给服务器。所以我们输入“a”（代表“accept”）即可，等到服务器响应后，注意，mitmproxy便又了拦截服务器发过来的response（注意那个“Response intercepted”）： 165 | 166 | mitmproxy 拦截response 167 | 现在如果你想修改这个response也可以，方式同上面修改request一样。这个时候我再输入“a”，代表我接受了这个response，然后这个response便可发给客户端了: 168 | 169 | mitmproxy 拦截response之后accept 170 | 171 | 更多类型的mitmproxy拦截 172 | 173 | 同时mitmproxy还支持不同类型的条件过滤，之前在拦截字符串前面加上特定的参数比如我要拦截所有的POST request怎么办？输入：~m POST 即可（m代表method）： 174 | 175 | mitmproxy 拦截特定的request 方法 176 | 177 | 拦截所有的request： ~q 178 | 179 | 拦截特定的header： ~h 180 | 181 | 拦截特定的domain： ~d 182 | 183 | 拦截特定的响应代码（404之类的）： ~c 184 | 185 | mitmproxy官方文档。 186 | 187 | ----- 188 | 189 | 190 | # 抓包工具 -- Fiddler 191 | 192 | 基于C# windows, Linux看这里 [Mono Fiddler](http://fiddler.wikidot.com/mono) 193 | 194 | --- 195 | ## 下载 196 | 直接下载，安装即可 197 | 198 | ## 手机抓包 199 | ![](https://imququ.com/static/uploads/2013/09/Snip20130913_20.png.webp) 200 | 201 | 202 | ---- 203 | 204 | # 抓包工具 -- wireshark 205 | 206 | 跨平台：Windows，OS X ，Linux 207 | 208 | [官网](https://www.wireshark.org/) 209 | 210 | -------------------------------------------------------------------------------- /csdnMove/Article: -------------------------------------------------------------------------------- 1 | package com.hanks.csdnblog; 2 | 3 | import java.util.Date; 4 | 5 | public class Article { 6 | public String id; 7 | public String fileName; 8 | public String title; 9 | public String url; 10 | public Date publishDate; 11 | 12 | @Override 13 | public String toString() { 14 | return "Article [id=" + id + ", fileName=" + fileName + ", title=" + title + ", url=" + url + ", publishDate=" 15 | + publishDate + "]"; 16 | } 17 | 18 | 19 | } 20 | -------------------------------------------------------------------------------- /csdnMove/CSDNBlog: -------------------------------------------------------------------------------- 1 | package com.hanks; 2 | 3 | import java.io.BufferedWriter; 4 | import java.io.File; 5 | import java.io.FileWriter; 6 | import java.io.IOException; 7 | import java.text.ParseException; 8 | import java.text.SimpleDateFormat; 9 | import java.util.ArrayList; 10 | import java.util.Calendar; 11 | import java.util.Date; 12 | 13 | import org.watij.webspec.dsl.Tag; 14 | import org.watij.webspec.dsl.WebSpec; 15 | 16 | public class CSDNBlog { 17 | 18 | public static void main(String[] args) throws IOException, ParseException { 19 | 20 | String username = "xxxxxxxxxxxx"; 21 | String password = "*********"; 22 | String nickname = "hpu_zyh"; 23 | 24 | SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); 25 | SimpleDateFormat sdfShort = new SimpleDateFormat("yyyy-MM-dd"); 26 | SimpleDateFormat sdfUrl = new SimpleDateFormat("/yyyy/MM/dd/"); 27 | 28 | String loginUrl = "https://passport.csdn.net/account/login?ref=toolbar"; 29 | 30 | // 当获取文章列表的页码大于你的最大页码时，csdn会返回你全部的文章，这里设置为9999，一般不会有这么多文章的用户 31 | String articlListUrl = "http://blog.csdn.net/" + nickname + "/article/list/9999"; 32 | 33 | String editMarkdownUrl = "http://write.blog.csdn.net/mdeditor#!postId="; 34 | String editRichTextUrl = "http://write.blog.csdn.net/postedit/"; 35 | 36 | String fileDir = "C:\\Users\\hanks\\Desktop\\bbb\\"; // 存放生成文件的路径 37 | 38 | //WebSpec.silent_mode = true; //是否有图形界面 39 | WebSpec client = new WebSpec().mozilla(); 40 | client.open(loginUrl); 41 | 42 | // 登录，为了能编辑文章内容 43 | Tag usernameTag = client.findWithId("username"); 44 | usernameTag.set.value(username); 45 | Tag passwordTag = client.findWithId("password"); 46 | passwordTag.set.value(password); 47 | Tag login = client.jquery("input[class=logging]"); 48 | System.out.println(login); 49 | login.click(); 50 | 51 | // 文章列表 52 | ArrayList
articleIdList = new ArrayList<>(); 53 | client.open(articlListUrl); 54 | Tag edit_a = client.jquery("span[class=link_title]"); 55 | int lenght = edit_a.all.length(); 56 | 57 | Tag dateTag = client.jquery("span[class=link_postdate]"); 58 | 59 | System.out.println("文章个数:" + lenght); 60 | 61 | for (int i = 0; i < lenght; i++) { 62 | 63 | // 获取文章title 64 | String articleTitle = edit_a.at(i).get.innerText().trim(); 65 | System.out.println("title:" + articleTitle); 66 | 67 | // 获取文章id 68 | String url = edit_a.at(i).child.a().get("href"); 69 | int index = url.lastIndexOf('/'); 70 | String articleId = url.substring(index + 1); 71 | System.out.println(articleId); 72 | 73 | // 获取日期 74 | Date date = sdf.parse(dateTag.at(i).get.innerText() + ":00"); 75 | 76 | Article article = new Article(); 77 | article.id = articleId; 78 | article.title = articleTitle; 79 | article.publishDate = date; 80 | article.fileName = formatFilename(articleTitle); 81 | article.url = sdfUrl.format(date) + article.fileName + "/"; 82 | System.out.println(article.toString()); 83 | articleIdList.add(article); 84 | } 85 | 86 | // 每个文章解析 87 | 88 | for (int i = 85 ; i < articleIdList.size(); i++) { 89 | 90 | Article article = articleIdList.get(i); 91 | 92 | String content = ""; 93 | boolean isRichText = false; 94 | try { 95 | // markdown: http://write.blog.csdn.net/mdeditor#!postId=50102915 96 | String url = editMarkdownUrl + article.id; 97 | System.out.println(url); 98 | client.open("http://m.baidu.com"); 99 | client.open(url); 100 | System.out.println("新的链接:"+i); 101 | client.pauseUntilReady(); 102 | 103 | // 文章详情 104 | // 内容 105 | Tag contentTag = client.findWithId("wmd-input"); 106 | content = contentTag.get.innerText(); 107 | 108 | isRichText = false; 109 | 110 | if (content ==null || "null".equals(content.trim())) { 111 | // 富文本 http://write.blog.csdn.net/postedit/50102915 112 | url = editRichTextUrl + article.id; 113 | System.out.println(url); 114 | client.open(url); 115 | // client.pauseUntilReady(); 116 | 117 | // 内容 118 | contentTag = client.findWithId("editor"); 119 | content = contentTag.get.innerText(); 120 | // System.out.println(content); 121 | 122 | isRichText = true; 123 | 124 | } 125 | } catch (Exception e) { 126 | // TODO Auto-generated catch block 127 | e.printStackTrace(); 128 | } 129 | 130 | // 保存文件 131 | File file = new File(fileDir + article.fileName + (isRichText ? ".html" : ".md")); 132 | if (file.exists()) { 133 | file.delete(); 134 | } 135 | file.createNewFile(); 136 | String br = "\r\n"; 137 | 138 | StringBuilder b = new StringBuilder(); 139 | if (!isRichText) { 140 | b.append("title: " + article.title + br); 141 | b.append("date: " + sdf.format(article.publishDate) + br); 142 | b.append("---" + br + br); 143 | } else { 144 | b.append("title: " + article.title + br); 145 | b.append("date: " + sdf.format(article.publishDate) + br); 146 | b.append("---" + br + br); 147 | } 148 | 149 | b.append(content+br); 150 | 151 | writeStringToFile(file.getAbsolutePath(), b.toString()); 152 | 153 | // 舍弃修改 154 | Tag cancleEditTag = client.findWithId("btnCancel"); 155 | if (cancleEditTag.exists()) { 156 | cancleEditTag.click(); 157 | } 158 | 159 | } 160 | client.closeAll(); 161 | } 162 | 163 | // 防止文件名称出现非法字符 164 | private static String formatFilename(String title) { 165 | String regex = "[<>/\\\\|:\"\"*?\\s+.]"; 166 | String result = title.replaceAll(regex, "_"); 167 | return result; 168 | } 169 | 170 | public static void writeStringToFile(String filePath, String appendContent) { 171 | try { 172 | FileWriter fw = new FileWriter(filePath, true); 173 | BufferedWriter bw = new BufferedWriter(fw); 174 | bw.append(appendContent); 175 | bw.close(); 176 | fw.close(); 177 | } catch (Exception e) { 178 | e.printStackTrace(); 179 | } 180 | } 181 | 182 | } 183 | --------------------------------------------------------------------------------