├── .gitignore ├── Makefile ├── README.md ├── README_ja.md ├── applyer ├── agent.go ├── common.go ├── main.go ├── manager.go └── single.go ├── go.mod ├── go.sum ├── images ├── mpreader.png ├── multi_agent_mode.png ├── multi_agent_mode_with_steps.png └── single_agent_mode.png ├── mpReader └── main.go └── observer └── main.go /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | .idea 3 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | .PHONY: clean build fmt 2 | 3 | fmt: 4 | go fmt ./... 5 | clean: 6 | rm -rf ./applyer/applyer ./mpReader/mpReader ./observer/observer 7 | 8 | build: clean 9 | go build -trimpath -o ./applyer/applyer ./applyer/... 10 | go build -trimpath -o ./mpReader/mpReader ./mpReader/... 11 | go build -trimpath -o ./observer/observer ./observer/... 12 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MySQL-query-replayer (MQR) 2 | 3 | [Japanese version README(日本語)](README_ja.md) 4 | 5 | ![MQR1](https://github.com/tom--bo/mysql-query-replayer/blob/master/images/multi_agent_mode.png) 6 | 7 | MySQL Query Replayer(MQR) is the tool to reproduce clients queries by capturing TCP packets. 8 | MQR-observer extract queries and send them to Queueing MW(support only Redis now), and MQR-Applyer apply them to Target MySQL. 9 | This can extract not only network packets in real time by using libpcap but also reading `tcpdump` output files. 10 | 11 | Main goal of MQR is duplicate production queries to Target MySQL with the same QPS, same connections as the production environment. And with same order as much as possible. 12 | 13 | ## Queuing MW 14 | 15 | - Support only Redis `sorted-set` 16 | - key: `${ip}:${port}` 17 | - score: timestamp in the packets 18 | - value: Query 19 | 20 | 21 | ## Observer 22 | 23 | MQR-observer extract queries and send them to Queueing MW(support only Redis now) in real time. 24 | This affects the performance of MySQL server, so It's not recommended to use this on busy server. 25 | Instead of using this in real time, you can use `tcpdump` and use the output later. 26 | 27 | #### Queuing mode 28 | 29 | Basic mode, It extracts network packets and send them to redis server. 30 | 31 | - mP (MySQL Port) 32 | - ih (ignore host) 33 | - rh (redis host) 34 | - rP (redis Port) 35 | - rp (redis Password) 36 | 37 | 38 | #### Debug mode 39 | 40 | With `-debug` option, MQR-observer prints the extracted queries. 41 | You can also specify `-c {count}` option to limit packets count, and stop after extracting specified queries. 42 | 43 | 44 | #### Elasticsearch mode 45 | 46 | With `-eh {elasticsearch-host}` option, MQR-observer send extracted queries to Elasticsearch and analyze and visualize with `Kibana` easily. 47 | 48 | 49 | ### How to use 50 | 51 | ``` 52 | git clone https://github.com/tom--bo/mysql-query-replayer 53 | cd mysql-query-replayer/observer 54 | go build . 55 | 56 | # samples 57 | ./observer -rh 10.127.0.10 -d any -mP 3306 -ih 10.127.0.20 58 | ``` 59 | 60 | 61 | ### options 62 | 63 | | options | description | 64 | |:---:|:---| 65 | | debug | Running in debug mode, print extracted queries | 66 | | c | Limit extracting queries and stop | 67 | | f | Extract from | 68 | | d | NIC device(default `any`) used by libpcap | 69 | | s | snapshot length used by libpcap snapshotLength | 70 | | pr | promiscuous mode used by libpcap | 71 | | mh | Original MySQL host | 72 | | mP | Original MySQL port | 73 | | ih | Ignoring Host | 74 | | rh | Redis Host | 75 | | rP | Redis Port | 76 | | rp | Redis Password | 77 | | eh | Elasticsearch Host | 78 | | eP | Elasticsearch port | 79 | 80 | 81 | ## Applyer 82 | 83 | 84 | MQR-applyer poll the Queueing MW and apply them to Target MySQL. 85 | Applyer has SINGLE mode and MANAGER/AGENT mode to apply queries with same amount of connections and QPS. 86 | 87 | 88 | ### Single MODE 89 | 90 | ![MQR single mode 画像](https://github.com/tom--bo/mysql-query-replayer/blob/master/images/single_agent_mode.png) 91 | 92 | It's easy to run MQR in single mode. 93 | Let's specify Redis server and Target MySQL, that's all. 94 | 95 | ``` 96 | # sample 97 | ./applyer -mh 10.127.1.10 -mP 3306 -mu mqr_user -md mqr_db -rh 10.127.1.20 -mp mqr_passwd 98 | ``` 99 | 100 | 101 | ### Manager/Agent MODE 102 | 103 | ![MQR multi mode 画像](https://github.com/tom--bo/mysql-query-replayer/blob/master/images/multi_agent_mode.png) 104 | 105 | Manager/Agent MODE aims to reproduce the same QPS as the actual queries with same amount of connections and QPS, using multiple Applyers. 106 | One process that starts in Manager MODE, and N hosts that start in Agent MODE can realize this . 107 | 108 | Assuming that MQR-observer, Redis(queueing MW) and Target MySQL is running, the startup order is as follows 109 | 110 | 1. Start the Agent MODE processes 111 | 1. Start the Manager MODE processes 112 | 113 | (Agent MODE) 114 | ``` 115 | ./applyer -A -mh 10.127.1.10 -mP 3306 -mu mqr_user -md mqr_db -rh 10.127.1.20 -mp mqr_passwd 116 | ``` 117 | 118 | (Manager MODE) 119 | ``` 120 | ./applyer -M -agents 10.127.149.16:6060,10.127.156.69:6060,10.127.56.106:6060 -rh 10.127.159.147 121 | ``` 122 | 123 | ## mpReader 124 | 125 | This command do not use queueing MW, just read tcpdump output and replay queries. 126 | Extract queries from `tcpdump` output file and replay to Target MySQL. 127 | 128 | ### How to use 129 | 130 | ``` 131 | ./mpReader -f dump.pcap -h 10.233.76.10 -P 3306 -u tombo -d sysbench -p password 132 | ``` 133 | 134 | ### options 135 | 136 | ``` 137 | -P int 138 | mysql port (default 3306) 139 | -c int 140 | Limit processing packets count (only enable when -debug is also specified) 141 | -d string 142 | mysql database 143 | -debug 144 | debug 145 | -f string 146 | pcap file. this option invalid packet capture from devices. 147 | -h string 148 | mysql host (default "localhost") 149 | -ih string 150 | ignore mysql hosts, specify only one ip address (default "localhost") 151 | -p string 152 | mysql password 153 | -u string 154 | mysql user (default "root") 155 | ``` 156 | 157 | 158 | 159 | 160 | ## Suport environments 161 | 162 | - Centos >= 6.9 163 | - MySQL >= 5.5 (>= 5.1 as well as possible) 164 | - Theoretically, MySQL client/server protocol has compatibility from MySQL-v4.1 165 | - (golang >= 1.11) if you will develop MQR, you need not consider about go version because you can use built binary files. 166 | 167 | 168 | 169 | ## How to get packets with tcpdump 170 | 171 | Please specify the `dst port 3306` if it's possible. 172 | 173 | `tcpdump -n -nn -s 0 -i some-itfc dst port 3306 -B 4096 -c 10000 -w filename.pcap` 174 | 175 | The return packets from MySQL-server contains arbitrary data which happens to match the client packet header. 176 | If it happens to match, the parser will work unintendedly. 177 | -------------------------------------------------------------------------------- /README_ja.md: -------------------------------------------------------------------------------- 1 | # MySQL-query-replayer (MQR) 2 | 3 | ![MQR1](https://github.com/tom--bo/mysql-query-replayer/blob/master/images/multi_agent_mode.png) 4 | 5 | MQRはObserver,Queuing MW(Redis), Applyerからなる。 6 | Observerで取得したMySQLのクエリをQueuing MWへ転送し、Applyerがそれを周期的に取得して再現対象のMySQLに実行する。 7 | 再現において、MySQLのip:portの組を1コネクションとし、このコネクション数も再現する。 8 | また、コネクション単位でクエリの実行順序、QPSも可能な限り再現することを目標とする。 9 | 10 | 以降以下の用語を用いて説明する 11 | - Original MySQL: プロダクションで動いているMySQL。このMySQLのネットワークパケットを取得してReplayする 12 | - Master, Slaveのどちらでも実行可能だが、MQRはCPU, networkに高負荷をかけるので、Active Masterで実行することは推奨しない 13 | - Target MySQL: MQRが取得したクエリを再現する対象のMySQL 14 | - connection(コネクション): Original MySQLに接続しているMySQL clientのip:portを文字列とした1コネクション 15 | 16 | 17 | ## Queuing MW 18 | 19 | - redis 1台を使用 20 | - connectionをkey, パケットが到達した時刻のunix timestampをscoreとしたsorted_setでクエリを保持している 21 | - 検証用ツールなので, redisの冗長化は考慮していない 22 | 23 | 24 | ## Observer 25 | 26 | ### 概要 27 | 28 | - Original MySQL上で動作させ、libpcapを使ってネットワークパケットを取得、その中からMySQLのクエリを抽出し、Queuing MW(Reids)に送信する。 29 | - デバッグ、可視化用にファイルへの出力、Elasticsearchへの送信が可能 30 | - 1コネクションごとに1スレッド(実際にはgoroutine)を起動する 31 | - コネクションごとのQPSが高いときは1コネクションで1coreの100%に近いCPUを利用するので注意。 32 | - observerのプログラムを実行した瞬間からパケットを収集し始め、case insensitiveで`SELECT`, `SET`, `SHOW`コマンドを抽出し、モードに従った送信先に転送する。 33 | 34 | #### Queuing mode 35 | 36 | Original MySQLのネットワークパケットからコマンドを抽出し、Redisに送る通常モード。 37 | -debug, -ehオプションによる`debug mode`, `Elasticsearch mode`で**起動しない**ことでこのモードで動作する 38 | 各種オプションの設定はデフォルト値で動作するが、主に以下を設定する必要がある。 39 | - mP (MySQL Port) 40 | - ih (ignore host) 41 | - rh (redis host) 42 | - rP (redis Port) 43 | - rp (redis Password) 44 | 45 | 46 | #### debug mode 47 | 48 | `-debug` オプションを付けることでデバッグモードで動作。 49 | デバッグモードではキャプチャしたクエリをファイルに出力する。 50 | デバッグモードを利用している際に同時に`-c`オプションで数値を指定すると、その数値分のクエリを取得するとコマンドが終了される 51 | `-eh`オプションでelasticsearchへクエリを送信しているときはelasticsearchモードでのデバッグ用出力が行われる。 52 | 53 | 54 | #### Elasticsearch mode 55 | 56 | `-eh`オプションを付けてElasticsearchホストを指定することで、Elasticsearchへ取得したクエリを送信することができる。 57 | Elasticsearchへ送信するのはkibanaでの可視化を目的としているため、ElasticsearchをQueuing MWとして利用することはできない。 58 | 59 | 60 | 61 | ### 使い方 62 | 63 | ファイルのダウンロードから実行までは以下 64 | 65 | 1. 実行バイナリをdownloadして実行 66 | 1. golangの実行環境を用意して、git clone後、go run observer/main.go (https://github.com/tom--bo/mysql-query-replayer ) 67 | 1. Redisを用意 68 | 1. そのredisをオプションで指定して実行 69 | 70 | サンプル 71 | ``` 72 | ./observer -rh 10.127.0.nn -d any -mP 3306 -ih 10.127.0.mm 73 | 74 | # golangの実行環境で実行する場合↓ 75 | go run main.go -rh 10.127.0.nn -d any -mP 3306 -ih 10.127.0.mm 76 | ``` 77 | 78 | 終了時は、`-debug`, `-c `を両方指定している場合を除き、自動で停止しないので、`ctrl-c`や`kill`コマンドを利用して停止。 79 | 80 | 81 | 82 | ### options 83 | 84 | | オプション | 説明 | 85 | |:---:|:---| 86 | | debug | デバッグモードにするオプション、-ehと同時に指定しない場合、ファイルにクエリを出力する。-ehと同時に指定された場合はElasticsearchモードのdebug出力を行う | 87 | | c | count, デバッグモードと同時に指定されると分のパケットを取得した後にコマンドを終了する | 88 | | f | で指定したtcpdumpのダンプファイルからクエリを取得し動作する。あとからの検証用にパケットを保存しておく場合に有効 | 89 | | d | NICのデバイスインタフェースを指定する。すべての場合は`any`。不要なNICのパケットをフィルタリングすることで、パケットのロストを防ぐことができる。 | 90 | | s | snapshot length, tcpdumpにおけるsnapshotLengthと同じ。これで指定したbyte数分だけでのパケットをやめる。先頭1024byte分のパケットのbyte文字列を取得するなど。不要に長いパケットを刻むことでパケットのロストを防ぐことができる。 | 91 | | pr | promiscuousモードで動作。tcpdumpのpromiscuousオプションと同じ。 | 92 | | mh | MySQL Host, Original MySQLのホストを指定。 | 93 | | mP | MySQL Port, Original MySQLのポートを指定 | 94 | | ih | Ignore Host, クエリ取得から除外するclient hostを指定。mmmのエージェントとかを無視しないとコネクションが大量になる。 | 95 | | rh | Redis Host, Redisのホストを指定 | 96 | | rP | Redis Port, Redisのportを指定 | 97 | | rp | Redis Password, Redisのpasswordを指定 | 98 | | eh | Elasticsearch Host | 99 | | eP | Elasticsearch port | 100 | | ep | !無い! Elasticsearch Password | 101 | | | | 102 | 103 | 104 | ## Applyer 105 | 106 | ### 概要 107 | 108 | Queuing MW(現状ではRedisのみを想定)からObserverが抽出したコマンドを短時間のpollingで取得し、Target MySQLに実行(再現)する。 109 | 1コマンドで実行できるSingle MODEと複数台のサーバを用意してApplyerの負荷分散を行うためのManager/Agent MODEがある。 110 | Manager/Agent ModeはOriginal MySQLへのクライアント(Application server)が複数台、複数コネクションを張っている状況のクエリを再現する際、Applyer1台でQPSを再現することは不可能な環境で利用することを想定している。 111 | 112 | 1プロセス(1コマンドでの実行)でSingle MODE, Manager MODE, Agent MODEを兼任することはできない。 113 | Single MODE, Agent MODEではgoroutineによって、並列にクエリの再現を行っており、mainのgoroutineに加え(connection * 2)のgoroutineを立てており、再現できるコネクションの数は`(CPU core数 / 2) - 1` である。 114 | 115 | 116 | ### 使い方 (Single MODE) 117 | 118 | ![MQR single mode 画像](https://github.com/tom--bo/mysql-query-replayer/blob/master/images/single_agent_mode.png) 119 | 120 | 121 | `Single MODE`ではApplyerを実行するコマンドでApplyの処理が完結するため、Observerと紐付いているRedisとTarget MySQLの準備以外で必要な準備はない。 122 | 123 | Redis, Target MySQLのACLを確認し、適切なhost(ip:port), user, password等をオプションで指定する。 124 | ファイルのダウンロードから実行までは以下 125 | 126 | 127 | 1. 実行バイナリをdownloadして実行 128 | 1. golangの実行環境を用意して、git clone後、go run observer/main.go (https://github.com/tom--bo/mysql-query-replayer ) 129 | 130 | コマンドサンプル 131 | ``` 132 | ./applyer -mh 10.127.1.nn -mP 3306 -mu mqr_user -md mqr_db -rh 10.127.1.mm -mp mqr_passwd 133 | 134 | # または 135 | go run main.go -mh 10.127.1.nn -mP 3306 -mu mqr_user -md mqr_db -rh 10.127.1.mm -mp mqr_passwd 136 | ``` 137 | 138 | 139 | ### 使い方 (複数台) (Manager/Agent MODE) 140 | 141 | ![MQR multi mode 画像](https://github.com/tom--bo/mysql-query-replayer/blob/master/images/multi_agent_mode.png) 142 | 143 | Manager/Agent MODEはOriginal MySQLに接続しているclientが多数の場合に、Applyerを複数台にすることで本番と同等のQPSを再現することを目的としている。 144 | Manager MODEで起動するプロセスが1つ、Agent MODEで起動するホストがN台で動作することが可能。 145 | 146 | Redis, Target MySQLが動作していることを前提にし、起動する順序は以下 147 | 148 | 1. Agent MODEのプロセスすべてを起動 149 | 1. Manager MODEのプロセス起動 150 | 151 | リアルタイムにOriginal MySQLのコマンドを抽出する場合はManager MODEのプロセスを起動した後にObserverを起動する。 152 | 153 | ファイルのダウンロードから実行までは以下 154 | 155 | 1. 実行バイナリをdownloadして実行 156 | 1. golangの実行環境を用意して、git clone後、go run observer/main.go (https://github.com/tom--bo/mysql-query-replayer ) 157 | 1. Agent MODEのapplyerを起動 (-Aオプションを指定) 158 | 1. Manager MODEのapplyerを起動 (-Mオプションを指定) 159 | 160 | ![MQR multi mode 画像](https://github.com/tom--bo/mysql-query-replayer/blob/master/images/multi_agent_mode_with_steps.png) 161 | 162 | Agent MODEのapplyer起動, コマンドサンプル 163 | ``` 164 | ./applyer -A -mh 10.127.1.nn -mP 3306 -mu mqr_user -md mqr_db -rh 10.127.1.mm -mp mqr_passwd 165 | 166 | # または 167 | go run main.go -A -mh 10.127.1.nn -mP 3306 -mu mqr_user -md mqr_db -rh 10.127.1.mm -mp mqr_passwd 168 | ``` 169 | 170 | Manager MODEのapplyer起動, コマンドサンプル 171 | ``` 172 | ./applyer -M -agents 10.127.149.16:6060,10.127.156.69:6060,10.127.56.106:6060 -rh 10.127.159.147 173 | 174 | # または 175 | go run main.go -M -agents 10.127.149.16:6060,10.127.156.69:6060,10.127.56.106:6060 -rh 10.127.159.147 176 | ``` 177 | 178 | 179 | ### options 180 | 181 | | オプション | 説明 | 182 | |:---:|:---| 183 | | ts | time sensitive, queueにあるコマンドを即時実行するのではなく、コマンドが実行された時間との差分を考慮して、同じ間隔でコマンドを実行する。 特にobserver側でpcapのダンプファイルからコマンドを取得した場合に有効 | 184 | | M | Manager MODEで起動する。 同時に-agentオプションを指定する必要があり、これで指定されたエージェントに対してコネクションを指定すマネージャとして起動する。このモードではqueueからの取得や取得したコマンドのTarget MySQLへの適用は行わない。 | 185 | | A | Agent MODEで起動する。Manager MODEのプロセスより先に起動する必要がある。Agentとしてデフォルト6060　portでManagerからの支持を待機し、受け取ったコネクションのコマンドを適用する。 | 186 | | agents | Manager MODEで起動する際にAgent MODEで起動しているホストを指定するオプション。コマンド区切りでhostIP:portを複数指定可能。Manager MODEで起動するプロセスで指定が必須。 | 187 | | p | Agent MODEでManagerからの支持を待機するポートを指定する。デフォルト6060。Agent MODEを変更した場合はManager MODEでもportをあわせる必要がある。 | 188 | | mh | MySQL Host, Target MySQLのホストを指定。 | 189 | | mP | MySQL Port, Target MySQLのポートを指定 | 190 | | mu | MySQL user, Target MySQLのuserを指定 | 191 | | mp | MySQL Password, Target MySQLのpasswordを指定 | 192 | | md | MySQL Database, Target MySQLのdatabaseを指定, 1つしか指定できない | 193 | | rh | Redis Host, Redisのホストを指定 | 194 | | rP | Redis Port, Redisのportを指定 | 195 | | rp | Redis Password, Redisのpasswordを指定 | 196 | | ignore-limit | Agent が受け持つコネクション数の制限を外す, デフォルトは vCPU x 3 に制限されている | 197 | 198 | ## mpReader 199 | 200 | dumpファイルを直接読み込んでTarget MySQLへ実行する。 201 | 202 | ### 使い方 203 | 204 | ``` 205 | ./mpReader -f dump.pcap -h 10.233.76.xxx -P 3306 -u tombo -d sysbench -p password 206 | ``` 207 | 208 | ### options 209 | 210 | ``` 211 | -P int 212 | mysql port (default 3306) 213 | -c int 214 | Limit processing packets count (only enable when -debug is also specified) 215 | -d string 216 | mysql database 217 | -debug 218 | debug 219 | -f string 220 | pcap file. this option invalid packet capture from devices. 221 | -h string 222 | mysql host (default "localhost") 223 | -ih string 224 | ignore mysql hosts, specify only one ip address (default "localhost") 225 | -p string 226 | mysql password 227 | -u string 228 | mysql user (default "root") 229 | ``` 230 | 231 | 232 | 233 | 234 | ## Suport environments 235 | 236 | - Centos >= 6.9 237 | - MySQL >= 5.5 (>= 5.1 as well as possible) 238 | - Theoretically, MySQL client/server protocol has compatibility from MySQL-v4.1 239 | - (golang >= 1.11) if you will develop MQR, you need not consider about go version because you can use built binary files. 240 | 241 | 242 | 243 | ## How to get packets with tcpdump 244 | 245 | Please specify the `dst port 3306` if it's possible. 246 | 247 | `tcpdump -n -nn -s 0 -i some-itfc dst port 3306 -B 4096 -c 10000 -w filename.pcap` 248 | 249 | The return packets from MySQL-server contains arbitrary data which happens to match the client packet header. 250 | If it happens to match, the parser will work unintendedly. 251 | 252 | -------------------------------------------------------------------------------- /applyer/agent.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "fmt" 6 | "github.com/garyburd/redigo/redis" 7 | "github.com/labstack/echo" 8 | "net/http" 9 | "runtime" 10 | "strconv" 11 | "strings" 12 | "sync" 13 | "time" 14 | ) 15 | 16 | type agentApplyer struct { 17 | hostCnt int 18 | cpuLimit int 19 | } 20 | 21 | func (a *agentApplyer) prepare() error { 22 | cpus = runtime.NumCPU() 23 | a.cpuLimit = cpus * 3 24 | a.hostCnt = 0 25 | return nil 26 | } 27 | 28 | func (a *agentApplyer) start() { 29 | // Start server and set addHostHandler, OkHandler 30 | a.agentServer() 31 | } 32 | 33 | func (a *agentApplyer) retrieveLoop(key string, m *sync.Mutex, q *[]commandData) { 34 | for { 35 | m.Lock() 36 | ll := len(*q) 37 | m.Unlock() 38 | if ll > 10000 { 39 | // fmt.Println("more than 1000") 40 | time.Sleep(50 * time.Millisecond) 41 | continue 42 | } 43 | 44 | r := rpool.Get() 45 | queries, err := redis.Strings(r.Do("LRANGE", key, 0, 199)) 46 | r.Close() 47 | if err != nil { 48 | fmt.Println(err) 49 | } 50 | l := len(queries) 51 | if l < 1 { 52 | time.Sleep(10 * time.Millisecond) 53 | continue 54 | } 55 | 56 | r = rpool.Get() 57 | _, err = r.Do("LTRIM", key, l, -1) 58 | if err != nil { 59 | fmt.Println(err) 60 | } 61 | r.Close() 62 | 63 | tmp := []commandData{} 64 | for i := 0; i < l; i++ { 65 | // ?? need judgement of command_type 66 | val := strings.SplitN(queries[i], ";", 3) 67 | capturedTime, err := strconv.Atoi(val[1]) 68 | if err != nil { 69 | panic(err) 70 | } 71 | st := commandData{ 72 | ctype: val[0], 73 | capturedTime: capturedTime, 74 | query: val[2], 75 | } 76 | tmp = append(tmp, st) 77 | } 78 | m.Lock() 79 | *q = append(*q, tmp...) 80 | m.Unlock() 81 | } 82 | } 83 | 84 | func (a *agentApplyer) applyLoop(mq *sync.Mutex, q *[]commandData) { 85 | mysqlHost := mUser + ":" + mPassword + "@tcp(" + mHost + ":" + strconv.Itoa(mPort) + ")/" + mdb + "?loc=Local&parseTime=true" 86 | if mSocket != "" { 87 | mysqlHost = mUser + ":" + mPassword + "@unix(" + mSocket + ")/" + mdb + "?loc=Local&parseTime=true" 88 | } 89 | 90 | db, err := sql.Open("mysql", mysqlHost) 91 | if err != nil { 92 | fmt.Println("Connection to MySQL fail.") 93 | } 94 | defer db.Close() 95 | var l, ll int 96 | var queries []commandData 97 | 98 | for { 99 | mq.Lock() 100 | ll = len(*q) 101 | 102 | if ll == 0 { 103 | mq.Unlock() 104 | continue 105 | } else { 106 | queries = make([]commandData, ll) 107 | l = copy(queries, *q) 108 | *q = []commandData{} 109 | mq.Unlock() 110 | } 111 | 112 | if timeSensitive && timeDiff == 0 { 113 | n := time.Now() 114 | if err != nil { 115 | panic(err) 116 | } 117 | timeDiff = int(n.UnixNano()) - queries[0].capturedTime*1000 118 | } 119 | for i := 0; i < l; i++ { 120 | // send query to mysql 121 | if err != nil { 122 | panic(err) 123 | } 124 | 125 | now := int(time.Now().UnixNano()) 126 | sleepTime := queries[i].capturedTime*1000 + timeDiff - now 127 | if timeSensitive && sleepTime > 0 { 128 | time.Sleep(time.Duration(sleepTime) * time.Nanosecond) 129 | i -= 1 130 | continue 131 | } 132 | 133 | if queries[i].ctype == "Q" { // simple query 134 | _, err := db.Exec(queries[i].query) 135 | if err != nil { 136 | fmt.Println(err) 137 | continue 138 | } 139 | } else if queries[i].ctype == "P" { 140 | // Prepare prepared_statement 141 | } else if queries[i].ctype == "E" { 142 | // Execute prepared_statement 143 | } 144 | } 145 | } 146 | } 147 | 148 | func (a *agentApplyer) okHandler(c echo.Context) error { 149 | return c.String(http.StatusOK, "OK!") 150 | } 151 | 152 | func (a *agentApplyer) addHostHandler(c echo.Context) error { 153 | k := c.Param("key") 154 | 155 | ips := strings.Split(k, ":") 156 | if isIgnoreHosts(ips[1], ignoreHosts) { 157 | fmt.Println(ips[1] + " is specified as ignoring host") 158 | return c.String(http.StatusOK, "No") 159 | } 160 | if !ignoreConnectionLimit { 161 | if a.hostCnt >= a.cpuLimit { 162 | fmt.Println("Too many hosts, ignore " + k) 163 | return c.String(http.StatusOK, "No") 164 | } 165 | } 166 | 167 | a.hostCnt += 1 168 | q := []commandData{} 169 | m := new(sync.Mutex) 170 | 171 | hostProgress.Store(k, "0") 172 | go a.retrieveLoop(k, m, &q) 173 | go a.applyLoop(m, &q) 174 | 175 | return c.String(http.StatusOK, "Added!") 176 | } 177 | 178 | func (a *agentApplyer) agentServer() { 179 | e := echo.New() 180 | e.GET("/ok", a.okHandler) 181 | e.POST("/addHost/:key", a.addHostHandler) 182 | addr := ":" + strconv.Itoa(port) 183 | e.Logger.Fatal(e.Start(addr)) 184 | } 185 | -------------------------------------------------------------------------------- /applyer/common.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github.com/garyburd/redigo/redis" 5 | "strings" 6 | "time" 7 | ) 8 | 9 | type commandData struct { 10 | ctype string 11 | capturedTime int 12 | query string 13 | } 14 | 15 | type Applyer interface { 16 | prepare() error 17 | start() 18 | } 19 | 20 | func checkKeys(prefix string) ([]string, error) { 21 | c := rpool.Get() 22 | defer c.Close() 23 | 24 | keys, err := redis.Strings(c.Do("keys", "*")) 25 | if err != nil { 26 | return []string{}, err 27 | } 28 | 29 | ret := []string{} 30 | for _, v := range keys { 31 | if prefix == "" { 32 | if strings.HasPrefix(v, ":") { 33 | ret = append(ret, v) 34 | } 35 | } else if strings.HasPrefix(v, prefix) { 36 | ret = append(ret, v) 37 | } 38 | } 39 | 40 | return ret, nil 41 | } 42 | 43 | func newPool(addr string) *redis.Pool { 44 | f := func() (redis.Conn, error) { return redis.Dial("tcp", addr) } 45 | 46 | if rSocket != "" && rPassword != "" { 47 | f = func() (redis.Conn, error) { return redis.Dial("unix", rSocket, redis.DialPassword(rPassword)) } 48 | } else if rSocket != "" { 49 | f = func() (redis.Conn, error) { return redis.Dial("unix", rSocket) } 50 | } else if rPassword != "" { 51 | f = func() (redis.Conn, error) { return redis.Dial("tcp", addr, redis.DialPassword(rPassword)) } 52 | } 53 | 54 | return &redis.Pool{ 55 | MaxIdle: cpus*10 + 1, 56 | MaxActive: cpus*10 + 1, 57 | IdleTimeout: 2 * time.Second, 58 | Dial: f, 59 | } 60 | } 61 | 62 | func isIgnoreHosts(ip string, ignoreHosts []string) bool { 63 | for _, h := range ignoreHosts { 64 | if ip == h { 65 | return true 66 | } 67 | } 68 | return false 69 | } 70 | -------------------------------------------------------------------------------- /applyer/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "github.com/garyburd/redigo/redis" 7 | _ "github.com/go-sql-driver/mysql" 8 | "runtime" 9 | "strconv" 10 | "strings" 11 | "sync" 12 | ) 13 | 14 | var ( 15 | name string 16 | managerMode bool 17 | agentMode bool 18 | agents string 19 | port int 20 | cpus int 21 | ignoreConnectionLimit bool 22 | ignoreHostStr string 23 | ignoreHosts []string 24 | 25 | mHost string 26 | mPort int 27 | mUser string 28 | mPassword string 29 | mdb string 30 | mSocket string 31 | 32 | rHost string 33 | rPort int 34 | rPassword string 35 | rSocket string 36 | timeSensitive bool 37 | 38 | hostProgress = sync.Map{} 39 | rpool *redis.Pool 40 | timeDiff = 0 41 | ) 42 | 43 | func parseOptions() { 44 | flag.BoolVar(&timeSensitive, "ts", true, "time sensitive, understand time diff between captured_time and this command's current time") 45 | flag.BoolVar(&managerMode, "M", false, "Execute as MQR-applyer-manager") 46 | flag.BoolVar(&agentMode, "A", false, "Execute as MQR-applyer-agent") 47 | flag.StringVar(&agents, "agents", "", "specify applyer-agents like (host):(port)") 48 | flag.IntVar(&port, "p", 6060, "MQR manager/agent use this port") 49 | flag.StringVar(&name, "name", "", "process name which is used as prefix of redis key") 50 | flag.BoolVar(&ignoreConnectionLimit, "ignore-limit", false, "Ignore connection limit which is limited to vCPU*3 connections for better performance (not recommended)") 51 | 52 | // mysql 53 | flag.StringVar(&mHost, "mh", "localhost", "mysql host") 54 | flag.StringVar(&ignoreHostStr, "ih", "localhost", "ignore mysql hosts, specify only one ip address") 55 | flag.IntVar(&mPort, "mP", 3306, "mysql port") 56 | flag.StringVar(&mUser, "mu", "root", "mysql user") 57 | flag.StringVar(&mPassword, "mp", "", "mysql password") 58 | flag.StringVar(&mdb, "md", "", "mysql database") 59 | flag.StringVar(&mSocket, "mS", "", "mysql unix domain socket") 60 | 61 | // redis 62 | flag.StringVar(&rHost, "rh", "localhost", "redis host") 63 | flag.IntVar(&rPort, "rP", 6379, "redis port") 64 | flag.StringVar(&rPassword, "rp", "", "redis password") 65 | flag.StringVar(&rSocket, "rs", "", "redis unix domain socket file") 66 | 67 | flag.Parse() 68 | } 69 | 70 | func main() { 71 | parseOptions() 72 | redisHost := rHost + ":" + strconv.Itoa(rPort) 73 | cpus = runtime.NumCPU() 74 | rpool = newPool(redisHost) 75 | ignoreHosts = strings.Split(ignoreHostStr, ",") 76 | 77 | var applyer Applyer 78 | 79 | if managerMode && agentMode { 80 | fmt.Println("Can not specify both managerMode and agentMode") 81 | return 82 | } else if managerMode { // execute as manager (not both manager and agent in one process) 83 | ags := strings.Split(agents, ",") 84 | applyer = &managerApplyer{agents: ags} 85 | } else if agentMode { // as agent (wait http addHost/ endpoint called) 86 | applyer = &agentApplyer{} 87 | } else { // single mode ( 88 | applyer = &singleApplyer{} 89 | } 90 | 91 | err := applyer.prepare() 92 | if err != nil { 93 | panic(err) 94 | } 95 | 96 | applyer.start() 97 | } 98 | -------------------------------------------------------------------------------- /applyer/manager.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "io/ioutil" 7 | "net/http" 8 | "strconv" 9 | "strings" 10 | "time" 11 | ) 12 | 13 | type managerApplyer struct { 14 | agents []string 15 | tooManyConnections bool 16 | } 17 | 18 | func (a *managerApplyer) prepare() error { 19 | a.tooManyConnections = false 20 | 21 | err := a.checkMQRAgents() 22 | if err != nil { 23 | return err 24 | } 25 | return nil 26 | } 27 | 28 | func (a *managerApplyer) start() { 29 | keyMap := make(map[string]int) 30 | connCnt := 0 31 | agentCnt := len(a.agents) 32 | for { 33 | keys, err := checkKeys(name) 34 | if err != nil { 35 | fmt.Printf("%v\n", err) 36 | } 37 | for _, k := range keys { 38 | if _, ok := keyMap[k]; !ok { 39 | fmt.Println("New connection (" + k + ") is detected, " + strconv.Itoa(connCnt) + " is applied") 40 | ips := strings.Split(k, ":") 41 | keyMap[k] = connCnt 42 | if isIgnoreHosts(ips[1], ignoreHosts) { 43 | fmt.Println(ips[1] + " is specified as ignoring host") 44 | continue 45 | } 46 | go func() { 47 | err = a.addHostRequest(connCnt, k) 48 | if err != nil { 49 | fmt.Printf("%v\n", err) 50 | } 51 | }() 52 | connCnt = (connCnt + 1) % agentCnt 53 | } 54 | } 55 | time.Sleep(10 * time.Millisecond) 56 | } 57 | } 58 | 59 | func (a *managerApplyer) addHostRequest(num int, key string) error { 60 | if a.tooManyConnections { 61 | fmt.Println(a.agents[num] + " fail") 62 | return nil 63 | } 64 | agentCnt := len(agents) 65 | for i := 0; i < agentCnt; i++ { 66 | n := (num + i) % agentCnt 67 | url := "http://" + a.agents[n] + "/addHost/" + key 68 | resp, err := http.Post(url, "", nil) 69 | if err != nil { 70 | panic(err) 71 | } 72 | 73 | body, err := ioutil.ReadAll(resp.Body) 74 | if err != nil { 75 | panic(err) 76 | } 77 | 78 | if err != nil { 79 | fmt.Println(err) 80 | fmt.Println(a.agents[n] + " fail") 81 | resp.Body.Close() 82 | continue 83 | } else if resp.StatusCode < 200 || resp.StatusCode >= 300 { 84 | fmt.Println(resp.StatusCode) 85 | fmt.Println(a.agents[n] + " fail") 86 | resp.Body.Close() 87 | continue 88 | } else if string(body) == "no" { 89 | fmt.Println(a.agents[n] + " is full.") 90 | resp.Body.Close() 91 | continue 92 | } 93 | resp.Body.Close() 94 | // success 95 | return nil 96 | } 97 | 98 | a.tooManyConnections = true 99 | return nil 100 | } 101 | 102 | func (a *managerApplyer) checkMQRAgents() error { 103 | if a.agents[0] == "" { 104 | return errors.New("Failed to execute as MQR-applyer-manager: No MQR-agents are specified, use -agents option to specify MQR-agents") 105 | } 106 | for _, agent := range a.agents { 107 | url := "http://" + agent + "/ok" 108 | resp, err := http.Get(url) 109 | if err != nil { 110 | fmt.Println(err) 111 | fmt.Println(agent + "'s /ok didn't respond") 112 | return err 113 | } else if resp.StatusCode < 200 || resp.StatusCode >= 300 { 114 | fmt.Println(resp.StatusCode) 115 | return errors.New(agent + "'s /ok didn't respond") 116 | } 117 | resp.Body.Close() 118 | } 119 | return nil 120 | } 121 | -------------------------------------------------------------------------------- /applyer/single.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "fmt" 6 | "github.com/garyburd/redigo/redis" 7 | "runtime" 8 | "strconv" 9 | "strings" 10 | "sync" 11 | "time" 12 | ) 13 | 14 | type singleApplyer struct { 15 | cpuLimit int 16 | m sync.Mutex 17 | q []commandData 18 | hostProgress sync.Map 19 | } 20 | 21 | func (a *singleApplyer) prepare() error { 22 | cpus = runtime.NumCPU() 23 | a.cpuLimit = cpus * 3 24 | a.q = []commandData{} 25 | a.m = sync.Mutex{} 26 | return nil 27 | } 28 | 29 | func (a *singleApplyer) start() { 30 | keyMap := make(map[string]int) 31 | hostCnt := 0 32 | 33 | go a.retrieveLoop() 34 | go a.applyLoop() 35 | 36 | for { 37 | keys, err := checkKeys(name) 38 | if err != nil { 39 | fmt.Printf("%v\n", err) 40 | } 41 | for _, k := range keys { 42 | if _, ok := keyMap[k]; !ok { 43 | keyMap[k] = 0 44 | ips := strings.Split(k, ":") 45 | if isIgnoreHosts(ips[1], ignoreHosts) { 46 | fmt.Println(ips[1] + " is specified as ignoring host") 47 | continue 48 | } 49 | if !ignoreConnectionLimit { 50 | if hostCnt <= a.cpuLimit { 51 | a.hostProgress.Store(k, "0") 52 | hostCnt += 1 53 | } else { 54 | fmt.Println("Too many hosts, ignore " + k) 55 | } 56 | } 57 | } 58 | } 59 | time.Sleep(100 * time.Millisecond) 60 | } 61 | } 62 | 63 | func (a *singleApplyer) retrieveLoop() { 64 | pMap := map[string]string{} 65 | 66 | for { 67 | a.hostProgress.Range(func(k, v interface{}) bool { 68 | pMap[k.(string)] = v.(string) 69 | return true 70 | }) 71 | for k := range pMap { 72 | for { // wait until queue(a.q) length is less than 10000 73 | a.m.Lock() 74 | ll := len(a.q) 75 | a.m.Unlock() 76 | if ll > 10000 { 77 | // fmt.Println("more than 1000") 78 | time.Sleep(50 * time.Millisecond) 79 | continue 80 | } 81 | break 82 | } 83 | 84 | r := rpool.Get() 85 | queries, err := redis.Strings(r.Do("LRANGE", k, 0, 199)) 86 | r.Close() 87 | if err != nil { 88 | fmt.Println(err) 89 | } 90 | l := len(queries) 91 | if l < 1 { 92 | continue 93 | } 94 | r = rpool.Get() 95 | _, err = r.Do("LTRIM", k, l, -1) 96 | if err != nil { 97 | fmt.Println(err) 98 | } 99 | r.Close() 100 | 101 | tmp := []commandData{} 102 | for i := 0; i < l; i++ { 103 | // ?? need judgement of command_type 104 | val := strings.SplitN(queries[i], ";", 3) 105 | capturedTime, err := strconv.Atoi(val[1]) 106 | if err != nil { 107 | fmt.Println(err) 108 | continue 109 | } 110 | st := commandData{ 111 | ctype: val[0], 112 | capturedTime: capturedTime, 113 | query: val[2], 114 | } 115 | tmp = append(tmp, st) 116 | } 117 | a.m.Lock() 118 | a.q = append(a.q, tmp...) 119 | a.m.Unlock() 120 | } 121 | } 122 | } 123 | 124 | func (a *singleApplyer) applyLoop() { 125 | mysqlHost := mUser + ":" + mPassword + "@tcp(" + mHost + ":" + strconv.Itoa(mPort) + ")/" + mdb + "?loc=Local&parseTime=true" 126 | if mSocket != "" { 127 | mysqlHost = mUser + ":" + mPassword + "@unix(" + mSocket + ")/" + mdb + "?loc=Local&parseTime=true" 128 | } 129 | 130 | db, err := sql.Open("mysql", mysqlHost) 131 | if err != nil { 132 | fmt.Println("Connection to MySQL fail.") 133 | } 134 | defer db.Close() 135 | var l, ll int 136 | 137 | for { 138 | a.m.Lock() 139 | ll = len(a.q) 140 | a.m.Unlock() 141 | if ll == 0 { 142 | continue 143 | } 144 | a.m.Lock() 145 | queries := make([]commandData, ll) 146 | l = copy(queries, a.q) 147 | a.q = []commandData{} 148 | a.m.Unlock() 149 | if timeSensitive && timeDiff == 0 { 150 | n := time.Now() 151 | if err != nil { 152 | panic(err) 153 | } 154 | timeDiff = int(n.UnixNano()) - queries[0].capturedTime*1000 155 | } 156 | // fmt.Println("applyLoop m.Unlock()") 157 | for i := 0; i < l; i++ { 158 | // send query to mysql 159 | if err != nil { 160 | panic(err) 161 | } 162 | 163 | now := int(time.Now().UnixNano()) 164 | sleepTime := queries[i].capturedTime*1000 + timeDiff - now 165 | if timeSensitive && sleepTime > 0 { 166 | time.Sleep(time.Duration(sleepTime) * time.Nanosecond) 167 | i -= 1 168 | continue 169 | } 170 | 171 | if queries[i].ctype == "Q" { // simple query 172 | _, err := db.Exec(queries[i].query) 173 | if err != nil { 174 | fmt.Println(err) 175 | continue 176 | } 177 | } else if queries[i].ctype == "P" { 178 | // Prepare prepared_statement 179 | } else if queries[i].ctype == "E" { 180 | // Execute prepared_statement 181 | } 182 | } 183 | } 184 | } 185 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/tom--bo/mysql-query-replayer 2 | 3 | go 1.19 4 | 5 | require ( 6 | github.com/garyburd/redigo v1.6.4 7 | github.com/go-sql-driver/mysql v1.7.1 8 | github.com/google/gopacket v1.1.19 9 | github.com/labstack/echo v3.3.10+incompatible 10 | github.com/tom--bo/mysql-packet-deserializer v0.0.0-20220313094008-b66f1c4334df 11 | ) 12 | 13 | require ( 14 | github.com/labstack/gommon v0.4.0 // indirect 15 | github.com/mattn/go-colorable v0.1.11 // indirect 16 | github.com/mattn/go-isatty v0.0.14 // indirect 17 | github.com/valyala/bytebufferpool v1.0.0 // indirect 18 | github.com/valyala/fasttemplate v1.2.1 // indirect 19 | golang.org/x/crypto v0.12.0 // indirect 20 | golang.org/x/net v0.10.0 // indirect 21 | golang.org/x/sys v0.11.0 // indirect 22 | golang.org/x/text v0.12.0 // indirect 23 | ) 24 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 2 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 3 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 4 | github.com/garyburd/redigo v1.6.4 h1:LFu2R3+ZOPgSMWMOL+saa/zXRjw0ID2G8FepO53BGlg= 5 | github.com/garyburd/redigo v1.6.4/go.mod h1:rTb6epsqigu3kYKBnaF028A7Tf/Aw5s0cqA47doKKqw= 6 | github.com/go-sql-driver/mysql v1.7.1 h1:lUIinVbN1DY0xBg0eMOzmmtGoHwWBbvnWubQUrtU8EI= 7 | github.com/go-sql-driver/mysql v1.7.1/go.mod h1:OXbVy3sEdcQ2Doequ6Z5BW6fXNQTmx+9S1MCJN5yJMI= 8 | github.com/google/gopacket v1.1.19 h1:ves8RnFZPGiFnTS0uPQStjwru6uO6h+nlr9j6fL7kF8= 9 | github.com/google/gopacket v1.1.19/go.mod h1:iJ8V8n6KS+z2U1A8pUwu8bW5SyEMkXJB8Yo/Vo+TKTo= 10 | github.com/labstack/echo v3.3.10+incompatible h1:pGRcYk231ExFAyoAjAfD85kQzRJCRI8bbnE7CX5OEgg= 11 | github.com/labstack/echo v3.3.10+incompatible/go.mod h1:0INS7j/VjnFxD4E2wkz67b8cVwCLbBmJyDaka6Cmk1s= 12 | github.com/labstack/gommon v0.4.0 h1:y7cvthEAEbU0yHOf4axH8ZG2NH8knB9iNSoTO8dyIk8= 13 | github.com/labstack/gommon v0.4.0/go.mod h1:uW6kP17uPlLJsD3ijUYn3/M5bAxtlZhMI6m3MFxTMTM= 14 | github.com/mattn/go-colorable v0.1.11 h1:nQ+aFkoE2TMGc0b68U2OKSexC+eq46+XwZzWXHRmPYs= 15 | github.com/mattn/go-colorable v0.1.11/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= 16 | github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y= 17 | github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= 18 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 19 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 20 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 21 | github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY= 22 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 23 | github.com/tom--bo/mysql-packet-deserializer v0.0.0-20220313094008-b66f1c4334df h1:NJBhw3dfFR72aJII8k6/VyKnu9DHrYzwHmVP8FqULZg= 24 | github.com/tom--bo/mysql-packet-deserializer v0.0.0-20220313094008-b66f1c4334df/go.mod h1:kpWcPDnevYaCh3EykXZHfGgmgelAQsjTet5NmygrCzw= 25 | github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw= 26 | github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= 27 | github.com/valyala/fasttemplate v1.2.1 h1:TVEnxayobAdVkhQfrfes2IzOB6o+z4roRkPF52WA1u4= 28 | github.com/valyala/fasttemplate v1.2.1/go.mod h1:KHLXt3tVN2HBp8eijSv/kGJopbvo7S+qRAEEKiv+SiQ= 29 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 30 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 31 | golang.org/x/crypto v0.12.0 h1:tFM/ta59kqch6LlvYnPa0yx5a83cL2nHflFhYKvv9Yk= 32 | golang.org/x/crypto v0.12.0/go.mod h1:NF0Gs7EO5K4qLn+Ylc+fih8BSTeIjAP05siRnAh98yw= 33 | golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= 34 | golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= 35 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 36 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 37 | golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= 38 | golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= 39 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 40 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 41 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 42 | golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 43 | golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 44 | golang.org/x/sys v0.0.0-20211103235746-7861aae1554b/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 45 | golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= 46 | golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 47 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 48 | golang.org/x/text v0.12.0 h1:k+n5B8goJNdU7hSvEtMUz3d1Q6D/XW4COJSJR6fN0mc= 49 | golang.org/x/text v0.12.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE= 50 | golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= 51 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 52 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 53 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 54 | gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b h1:h8qDotaEPuJATrMmW04NCwg7v22aHH28wwpauUhK9Oo= 55 | gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 56 | -------------------------------------------------------------------------------- /images/mpreader.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tom--bo/mysql-query-replayer/a913655d1b064d4b4414d5a1f37d5454d2abe02f/images/mpreader.png -------------------------------------------------------------------------------- /images/multi_agent_mode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tom--bo/mysql-query-replayer/a913655d1b064d4b4414d5a1f37d5454d2abe02f/images/multi_agent_mode.png -------------------------------------------------------------------------------- /images/multi_agent_mode_with_steps.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tom--bo/mysql-query-replayer/a913655d1b064d4b4414d5a1f37d5454d2abe02f/images/multi_agent_mode_with_steps.png -------------------------------------------------------------------------------- /images/single_agent_mode.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tom--bo/mysql-query-replayer/a913655d1b064d4b4414d5a1f37d5454d2abe02f/images/single_agent_mode.png -------------------------------------------------------------------------------- /mpReader/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "database/sql" 5 | "errors" 6 | "flag" 7 | "fmt" 8 | _ "github.com/go-sql-driver/mysql" 9 | "github.com/google/gopacket" 10 | "github.com/google/gopacket/layers" 11 | "github.com/google/gopacket/pcap" 12 | mp "github.com/tom--bo/mysql-packet-deserializer" 13 | "io" 14 | "log" 15 | "strconv" 16 | "strings" 17 | "time" 18 | ) 19 | 20 | var ( 21 | debug bool 22 | err error 23 | read_only bool = true 24 | timelayout string = "2006-01-02 15:04:05.000000" 25 | 26 | handle *pcap.Handle 27 | packetCount int 28 | pcapfile string 29 | 30 | db *sql.DB 31 | mHost string 32 | mPort int 33 | mUser string 34 | mPassword string 35 | ignoreHostStr string 36 | ignoreHosts []string 37 | 38 | mdb string 39 | ) 40 | 41 | type MySQLPacketInfo struct { 42 | srcIP string 43 | srcPort int 44 | dstIP string 45 | dstPort int 46 | mysqlPacket []mp.IMySQLPacket 47 | capturedTime time.Time 48 | } 49 | 50 | func parseOptions() { 51 | flag.BoolVar(&debug, "debug", false, "debug") 52 | flag.StringVar(&pcapfile, "f", "", "pcap file. this option invalid packet capture from devices.") 53 | flag.IntVar(&packetCount, "c", 0, "Limit processing packets count (only enable when -debug is also specified)") 54 | 55 | // MySQL 56 | flag.StringVar(&mHost, "h", "localhost", "mysql host") 57 | flag.IntVar(&mPort, "P", 3306, "mysql port") 58 | flag.StringVar(&mUser, "u", "root", "mysql user") 59 | flag.StringVar(&mPassword, "p", "", "mysql password") 60 | flag.StringVar(&mdb, "d", "", "mysql database") 61 | flag.StringVar(&ignoreHostStr, "ih", "localhost", "ignore mysql hosts, specify only one ip address") 62 | 63 | flag.Parse() 64 | } 65 | 66 | func checkRequiredOptions() error { 67 | var err error 68 | err = nil 69 | if pcapfile == "" { 70 | fmt.Println("dumpfile is not specified!! Use -f to specify") 71 | err = errors.New("Required option is not specified") 72 | } 73 | 74 | return err 75 | } 76 | 77 | func getMySQLPacketInfo(packet gopacket.Packet) (MySQLPacketInfo, error) { 78 | applicationLayer := packet.ApplicationLayer() 79 | if applicationLayer == nil { 80 | return MySQLPacketInfo{}, errors.New("invalid packets") 81 | } 82 | 83 | frame := packet.Metadata() 84 | ipLayer := packet.Layer(layers.LayerTypeIPv4) 85 | tcpLayer := packet.Layer(layers.LayerTypeTCP) 86 | if ipLayer == nil || tcpLayer == nil { 87 | return MySQLPacketInfo{}, errors.New("Invalid_Packet") 88 | } 89 | 90 | ip, _ := ipLayer.(*layers.IPv4) 91 | tcp, _ := tcpLayer.(*layers.TCP) 92 | mcmd := mp.DeserializePacket(applicationLayer.Payload()) 93 | if len(mcmd) == 0 { 94 | return MySQLPacketInfo{}, errors.New("Not_MySQL_Packet") 95 | } 96 | return MySQLPacketInfo{ip.SrcIP.String(), int(tcp.SrcPort), ip.DstIP.String(), int(tcp.DstPort), mcmd, frame.CaptureInfo.Timestamp}, nil 97 | // return MySQLPacketInfo{"srcIP", int(tcp.SrcPort), "dstIP", int(tcp.DstPort), mcmd, frame.CaptureInfo.Timestamp}, nil 98 | } 99 | 100 | func checkReadQuery(q string) bool { 101 | q = strings.TrimSpace(q) 102 | if strings.HasPrefix(q, "select") || strings.HasPrefix(q, "SELECT") { 103 | return true 104 | } 105 | return false 106 | } 107 | 108 | func execQuery(packet gopacket.Packet) error { 109 | applicationLayer := packet.ApplicationLayer() 110 | if applicationLayer != nil { 111 | pInfo, err := getMySQLPacketInfo(packet) 112 | if err != nil { 113 | fmt.Println(err) 114 | return err 115 | } 116 | if isIgnoreHosts(pInfo.srcIP, ignoreHosts) { 117 | // fmt.Println(pInfo.srcIP + " is specified as ignoring host") 118 | return nil 119 | } 120 | 121 | if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_QUERY { 122 | cmd := pInfo.mysqlPacket[0].(mp.ComQuery) 123 | q := makeOneLine(cmd.Query) 124 | if read_only && !checkReadQuery(q) { 125 | return errors.New("not read query") 126 | } 127 | _, err := db.Exec(q) 128 | if err != nil { 129 | fmt.Println(err) 130 | return err 131 | } 132 | return nil 133 | } else { 134 | return errors.New("not COM_QUERY") 135 | } 136 | } 137 | return errors.New("something is wrong") 138 | } 139 | 140 | func makeOneLine(q string) string { 141 | q = strings.Replace(q, "\"", "'", -1) 142 | q = strings.Replace(q, "\n", " ", -1) 143 | 144 | return q 145 | } 146 | 147 | func printQuery(packet gopacket.Packet) error { 148 | applicationLayer := packet.ApplicationLayer() 149 | if applicationLayer != nil { 150 | pInfo, err := getMySQLPacketInfo(packet) 151 | if err != nil { 152 | fmt.Println(err) 153 | return err 154 | } 155 | if isIgnoreHosts(pInfo.srcIP, ignoreHosts) { 156 | // fmt.Println(pInfo.srcIP + " is specified as ignoring host") 157 | return nil 158 | } 159 | if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_QUERY { 160 | cmd := pInfo.mysqlPacket[0].(mp.ComQuery) 161 | q := makeOneLine(cmd.Query) 162 | if read_only && !checkReadQuery(q) { 163 | return errors.New("not read query") 164 | } 165 | fmt.Printf("%26s, %s \n", pInfo.capturedTime.Format(timelayout), q) 166 | if err != nil { 167 | fmt.Println(err) 168 | return err 169 | } 170 | return nil 171 | } else { 172 | return errors.New("not COM_QUERY") 173 | } 174 | } 175 | return errors.New("something is wrong") 176 | } 177 | 178 | func isIgnoreHosts(ip string, ignoreHosts []string) bool { 179 | for _, h := range ignoreHosts { 180 | if ip == h { 181 | return true 182 | } 183 | } 184 | return false 185 | } 186 | 187 | func main() { 188 | parseOptions() 189 | ignoreHosts = strings.Split(ignoreHostStr, ",") 190 | err = checkRequiredOptions() 191 | if err != nil { 192 | return 193 | } 194 | 195 | mysqlHost := mUser + ":" + mPassword + "@tcp(" + mHost + ":" + strconv.Itoa(mPort) + ")/" + mdb + "?loc=Local&parseTime=true" 196 | db, err = sql.Open("mysql", mysqlHost) 197 | if err != nil { 198 | fmt.Println("Connection to MySQL fail.") 199 | fmt.Println(err) 200 | } 201 | defer db.Close() 202 | 203 | handle, err = pcap.OpenOffline(pcapfile) 204 | if err != nil { 205 | log.Fatal(err) 206 | panic(err) 207 | } 208 | defer handle.Close() 209 | 210 | filter := "tcp and tcp[13] & 8 != 0" // tcp PSH flag is set (more smart filtering needed!) 211 | if mPort != 0 { 212 | filter += " and port " + strconv.Itoa(mPort) 213 | } 214 | 215 | err = handle.SetBPFFilter(filter) 216 | if err != nil { 217 | log.Fatal(err) 218 | } 219 | 220 | // Use the handle as a packet source to process all packets 221 | packetSource := gopacket.NewPacketSource(handle, handle.LinkType()) 222 | 223 | cnt := 0 224 | for { 225 | if packetCount != 0 && cnt >= packetCount { 226 | break 227 | } 228 | packet, err := packetSource.NextPacket() 229 | if err == io.EOF { 230 | break 231 | } else if err != nil { 232 | log.Println(err) 233 | continue 234 | } 235 | 236 | if debug { 237 | err = printQuery(packet) 238 | } else { 239 | err = execQuery(packet) 240 | 241 | } 242 | if err == nil { 243 | cnt += 1 244 | } 245 | } 246 | } 247 | -------------------------------------------------------------------------------- /observer/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "encoding/base64" 6 | "errors" 7 | "flag" 8 | "fmt" 9 | "github.com/garyburd/redigo/redis" 10 | "github.com/google/gopacket" 11 | "github.com/google/gopacket/layers" 12 | "github.com/google/gopacket/pcap" 13 | mp "github.com/tom--bo/mysql-packet-deserializer" 14 | "io" 15 | "io/ioutil" 16 | "log" 17 | "net/http" 18 | "runtime" 19 | "strconv" 20 | "strings" 21 | "time" 22 | ) 23 | 24 | var ( 25 | debug bool 26 | cpuRate int 27 | name string 28 | els bool = false 29 | timelayout string = "2006-01-02 15:04:05.000000" 30 | read_only bool = true 31 | 32 | device string 33 | snapshotLen int 34 | promiscuous bool 35 | handle *pcap.Handle 36 | packetCount int 37 | 38 | ignoreHostStr string 39 | ignoreHosts []string 40 | mPort int 41 | 42 | rHost string 43 | rPort int 44 | rPassword string 45 | pcapfile string 46 | 47 | eHost string 48 | ePort int 49 | eUser string 50 | ePasswd string 51 | 52 | rpool *redis.Pool 53 | eClient *http.Client 54 | ) 55 | 56 | type MySQLPacketInfo struct { 57 | srcIP string 58 | srcPort int 59 | dstIP string 60 | dstPort int 61 | mysqlPacket []mp.IMySQLPacket 62 | capturedTime time.Time 63 | } 64 | 65 | func parseOptions() { 66 | flag.BoolVar(&debug, "debug", false, "debug") 67 | flag.StringVar(&pcapfile, "f", "", "pcap file. this option invalid packet capture from devices.") 68 | flag.IntVar(&packetCount, "c", -1, "Limit processing packets count (only enable when -debug is also specified)") 69 | flag.StringVar(&name, "name", "", "process name which is used as prefix of redis key") 70 | flag.IntVar(&cpuRate, "cpu-rate", 2, "This is experimental option, It is NOT recommended to use this! goroutine rate for CPUs, specify the doubled rate which you want to specify") 71 | 72 | // gopacket 73 | flag.StringVar(&device, "d", "en0", "device name to capture.") 74 | flag.IntVar(&snapshotLen, "s", 1024, "snapshot length for gopacket") 75 | flag.BoolVar(&promiscuous, "pr", false, "promiscuous for gopacket") 76 | 77 | // MySQL 78 | flag.StringVar(&ignoreHostStr, "ih", "localhost", "ignore mysql hosts, specify only one ip address") 79 | flag.IntVar(&mPort, "mP", 0, "mysql port") 80 | 81 | // Redis 82 | flag.StringVar(&rHost, "rh", "localhost", "redis host") 83 | flag.IntVar(&rPort, "rP", 6379, "redis port") 84 | flag.StringVar(&rPassword, "rp", "", "redis password") 85 | 86 | // Elasticsearch 87 | flag.StringVar(&eHost, "eh", "", "Elasticsearch host") 88 | flag.IntVar(&ePort, "eP", 9200, "Elasticsearch port") 89 | flag.StringVar(&eUser, "eu", "", "Elasticsearch user (only for basic authentication)") 90 | flag.StringVar(&ePasswd, "ep", "", "Elasticsearch passwd (only for basic authentication)") 91 | 92 | flag.Parse() 93 | } 94 | 95 | func newPool(addr string, cpus int) *redis.Pool { 96 | if rPassword != "" { 97 | return &redis.Pool{ 98 | MaxIdle: cpus, 99 | MaxActive: 0, 100 | Wait: true, 101 | IdleTimeout: 10 * time.Second, 102 | Dial: func() (redis.Conn, error) { return redis.Dial("tcp", addr, redis.DialPassword(rPassword)) }, 103 | } 104 | } 105 | return &redis.Pool{ 106 | MaxIdle: cpus, 107 | MaxActive: 0, 108 | Wait: true, 109 | IdleTimeout: 10 * time.Second, 110 | Dial: func() (redis.Conn, error) { return redis.Dial("tcp", addr) }, 111 | } 112 | } 113 | 114 | func getMySQLPacketInfo(packet gopacket.Packet) (MySQLPacketInfo, error) { 115 | applicationLayer := packet.ApplicationLayer() 116 | if applicationLayer == nil { 117 | return MySQLPacketInfo{}, errors.New("invalid packets") 118 | } 119 | 120 | frame := packet.Metadata() 121 | ipLayer := packet.Layer(layers.LayerTypeIPv4) 122 | tcpLayer := packet.Layer(layers.LayerTypeTCP) 123 | if ipLayer == nil || tcpLayer == nil { 124 | return MySQLPacketInfo{}, errors.New("Invalid_Packet") 125 | } 126 | 127 | ip, _ := ipLayer.(*layers.IPv4) 128 | tcp, _ := tcpLayer.(*layers.TCP) 129 | mcmd := mp.DeserializePacket(applicationLayer.Payload()) 130 | if len(mcmd) == 0 { 131 | return MySQLPacketInfo{}, errors.New("Not_MySQL_Packet") 132 | } 133 | return MySQLPacketInfo{ip.SrcIP.String(), int(tcp.SrcPort), ip.DstIP.String(), int(tcp.DstPort), mcmd, frame.CaptureInfo.Timestamp}, nil 134 | // return MySQLPacketInfo{"srcIP", int(tcp.SrcPort), "dstIP", int(tcp.DstPort), mcmd, frame.CaptureInfo.Timestamp}, nil 135 | } 136 | 137 | func checkReadQuery(q string) bool { 138 | q = strings.TrimSpace(q) 139 | if strings.HasPrefix(q, "select") || strings.HasPrefix(q, "SELECT") { 140 | return true 141 | } 142 | return false 143 | } 144 | 145 | // when debug 146 | func writeQueriesToFile(packet gopacket.Packet, cnt int) { 147 | applicationLayer := packet.ApplicationLayer() 148 | if applicationLayer != nil { 149 | pInfo, err := getMySQLPacketInfo(packet) 150 | if err != nil { 151 | fmt.Println(err) 152 | return 153 | } 154 | 155 | fmt.Printf("%4d, %26s, %-10s: ", cnt, pInfo.capturedTime.Format(timelayout), pInfo.mysqlPacket[0].GetCommandType()) 156 | /* 157 | j, err := json.Marshal(pInfo.mysqlPacket) 158 | if err != nil { 159 | fmt.Println(err) 160 | } 161 | fmt.Println(string(j)) 162 | */ 163 | } 164 | } 165 | 166 | func isIgnoreHosts(ip string, ignoreHosts []string) bool { 167 | for _, h := range ignoreHosts { 168 | if ip == h { 169 | return true 170 | } 171 | } 172 | return false 173 | } 174 | 175 | // to redis 176 | func sendQuery(packet gopacket.Packet) { 177 | applicationLayer := packet.ApplicationLayer() 178 | if applicationLayer != nil { 179 | pInfo, err := getMySQLPacketInfo(packet) 180 | if err != nil { 181 | fmt.Println(err) 182 | return 183 | } 184 | if isIgnoreHosts(pInfo.srcIP, ignoreHosts) { 185 | return 186 | } 187 | 188 | key := name + ":" + pInfo.srcIP + ":" + strconv.Itoa(pInfo.srcPort) 189 | capturedTime := strconv.Itoa(int(pInfo.capturedTime.UnixNano() / 1000)) // Need to be shorter than eq 15 char and also can't get time in nano sec 190 | val := "" 191 | 192 | if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_QUERY { 193 | cmd := pInfo.mysqlPacket[0].(mp.ComQuery) 194 | q := makeOneLine(cmd.Query) 195 | if read_only && !checkReadQuery(q) { 196 | return 197 | } 198 | val = "Q;" + capturedTime + ";" + q 199 | } else if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_STMT_PREPARE { 200 | cmd := pInfo.mysqlPacket[0].(mp.ComSTMTPrepare) 201 | q := makeOneLine(cmd.Query) 202 | if read_only && !checkReadQuery(q) { 203 | return 204 | } 205 | val = "P;" + capturedTime + ";" + q 206 | // ?? need to add statementId in return packet 207 | } else if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_STMT_EXECUTE { 208 | cmd := pInfo.mysqlPacket[0].(mp.ComSTMTExecute) 209 | val = "E;" + capturedTime + ";" + strconv.Itoa(cmd.STMTID) + ";" + cmd.ValueOfEachParameter 210 | } else if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_STMT_FETCH { 211 | cmd := pInfo.mysqlPacket[0].(mp.ComSTMTFetch) 212 | val = "F;" + capturedTime + ";" + strconv.Itoa(cmd.STMTID) + ";" + strconv.Itoa(cmd.NumRows) 213 | } else if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_STMT_SEND_LONG_DATA { 214 | // TBD 215 | } else if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_STMT_RESET { 216 | // TBD 217 | } else if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_STMT_CLOSE { 218 | // TBD 219 | } else { 220 | return 221 | } 222 | c := rpool.Get() 223 | _, err = c.Do("RPUSH", key, val) 224 | 225 | if err != nil { 226 | panic(err) 227 | } 228 | c.Close() 229 | } 230 | } 231 | 232 | func makeOneLine(q string) string { 233 | q = strings.Replace(q, "\"", "'", -1) 234 | q = strings.Replace(q, "\n", " ", -1) 235 | 236 | return q 237 | } 238 | 239 | // to Elasticsearch 240 | func sendQueryToElasticsearch(packet gopacket.Packet) { 241 | applicationLayer := packet.ApplicationLayer() 242 | if applicationLayer != nil { 243 | pInfo, err := getMySQLPacketInfo(packet) 244 | if err != nil { 245 | fmt.Println(err) 246 | return 247 | } 248 | 249 | q := "" 250 | if pInfo.mysqlPacket[0].GetCommandType() == mp.COM_QUERY { 251 | cmd := pInfo.mysqlPacket[0].(mp.ComQuery) 252 | q = makeOneLine(cmd.Query) 253 | } else { 254 | return 255 | } 256 | if read_only && !checkReadQuery(q) { 257 | return 258 | } 259 | 260 | jsonString := fmt.Sprintf("{\"captured_time\": \"%s\", \"src_ip\":\"%s\", \"src_port\":\"%d\", \"dst_ip\":\"%s\", \"dst_port\":\"%d\", \"mysql_query\":\"%s\"}", 261 | pInfo.capturedTime.Format(timelayout), pInfo.srcIP, pInfo.srcPort, pInfo.dstIP, pInfo.dstPort, q) 262 | if debug { 263 | fmt.Println(jsonString) 264 | } 265 | 266 | var req *http.Request 267 | 268 | if eUser != "" { 269 | // ?? need to be fixed (https) 270 | req, err = http.NewRequest( 271 | "POST", 272 | "https://"+eHost+":"+strconv.Itoa(ePort)+"/mqr/query", 273 | bytes.NewBuffer([]byte(jsonString)), 274 | ) 275 | auth := eUser + ":" + ePasswd 276 | req.Header.Add("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(auth))) 277 | } else { 278 | req, err = http.NewRequest( 279 | "POST", 280 | "http://"+eHost+":"+strconv.Itoa(ePort)+"/mqr/query", 281 | bytes.NewBuffer([]byte(jsonString)), 282 | ) 283 | } 284 | if err != nil { 285 | panic(err) 286 | } 287 | req.Header.Set("Content-Type", "application/json") 288 | 289 | resp, err := eClient.Do(req) 290 | if err != nil { 291 | panic(err) 292 | } 293 | _, err = ioutil.ReadAll(resp.Body) 294 | if err != nil { 295 | panic(err) 296 | } 297 | resp.Body.Close() 298 | } 299 | } 300 | 301 | func main() { 302 | http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = 1000 303 | 304 | parseOptions() 305 | ignoreHosts = strings.Split(ignoreHostStr, ",") 306 | 307 | // redis client 308 | cpus := runtime.NumCPU() * cpuRate / 2 // Need to experimentations 309 | var err error 310 | if !debug { 311 | redisHost := rHost + ":" + strconv.Itoa(rPort) 312 | rpool = newPool(redisHost, cpus) 313 | } 314 | if eHost != "" { 315 | els = true 316 | eClient = &http.Client{} 317 | } 318 | 319 | if pcapfile != "" { 320 | // Open from pcap file 321 | handle, err = pcap.OpenOffline(pcapfile) 322 | } else { 323 | // Open device 324 | ihandler, _ := pcap.NewInactiveHandle(device) 325 | ihandler.SetBufferSize(2147483648) 326 | ihandler.SetSnapLen(snapshotLen) 327 | ihandler.SetTimeout(pcap.BlockForever) 328 | ihandler.SetPromisc(promiscuous) 329 | handle, err = ihandler.Activate() 330 | } 331 | if err != nil { 332 | log.Fatal(err) 333 | panic(err) 334 | } 335 | defer handle.Close() 336 | 337 | var filter string = "tcp and tcp[13] & 8 != 0" // tcp PSH flag is set (more smart filtering needed!) 338 | if mPort != 0 { 339 | filter += " and port " + strconv.Itoa(mPort) 340 | } 341 | 342 | err = handle.SetBPFFilter(filter) 343 | if err != nil { 344 | log.Fatal(err) 345 | } 346 | 347 | // Use the handle as a packet source to process all packets 348 | packetSource := gopacket.NewPacketSource(handle, handle.LinkType()) 349 | semaphore := make(chan bool, cpus) 350 | 351 | if els { // to Elasticsearch both debug = ON/OFF 352 | cnt := 0 353 | for { 354 | packet, err := packetSource.NextPacket() 355 | if err == io.EOF { 356 | break 357 | } else if err != nil { 358 | log.Println(err) 359 | continue 360 | } 361 | 362 | semaphore <- true 363 | go func() { 364 | defer func() { <-semaphore }() 365 | sendQueryToElasticsearch(packet) 366 | }() 367 | 368 | if packetCount != -1 { 369 | if cnt > packetCount { 370 | break 371 | } 372 | cnt++ 373 | } 374 | } 375 | } else if debug { // only debug option specified, write packet-data to file 376 | cnt := 0 377 | for { 378 | packet, err := packetSource.NextPacket() 379 | if err == io.EOF { 380 | break 381 | } else if err != nil { 382 | log.Println(err) 383 | continue 384 | } 385 | 386 | semaphore <- true 387 | go func() { 388 | defer func() { <-semaphore }() 389 | writeQueriesToFile(packet, cnt) 390 | }() 391 | 392 | if packetCount != -1 { 393 | if cnt > packetCount { 394 | break 395 | } 396 | cnt++ 397 | } 398 | } 399 | } else { // Not debug neither elasticsearch. Add redis as fast as possible 400 | for { 401 | packet, err := packetSource.NextPacket() 402 | if err == io.EOF { 403 | break 404 | } else if err != nil { 405 | log.Println(err) 406 | continue 407 | } 408 | semaphore <- true 409 | go func() { 410 | defer func() { <-semaphore }() 411 | sendQuery(packet) 412 | }() 413 | } 414 | } 415 | } 416 | --------------------------------------------------------------------------------