├── .gitignore ├── .travis.yml ├── CHANGELOG.md ├── ELASTICSEARCH.md ├── LICENSE.txt ├── Procfile ├── README.md ├── bin ├── Gofile └── gor.go ├── elasticsearch └── elasticsearch.go ├── emitter.go ├── emitter_test.go ├── input_dummy.go ├── input_file.go ├── input_raw.go ├── input_raw_test.go ├── input_tcp.go ├── input_tcp_test.go ├── limiter.go ├── limiter_test.go ├── output_dummy.go ├── output_file.go ├── output_file_test.go ├── output_http.go ├── output_http_test.go ├── output_tcp.go ├── output_tcp_test.go ├── plugins.go ├── raw_socket_listener ├── listener.go ├── tcp_message.go └── tcp_packet.go ├── settings.go ├── settings_headers.go ├── settings_methods.go ├── settings_methods_test.go ├── settings_option.go ├── test_input.go └── test_output.go /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | 3 | *.gor 4 | 5 | bin/*.gor 6 | bin/*.lock 7 | bin/a.out 8 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | go: 1.1 3 | script: sudo -E bash -c "source /etc/profile && gvm use go1.1 && export GOPATH=$HOME/gopath:$GOPATH && go test -v" 4 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | v0.7.0 - 31 Oct 2013 2 | * New modular architecture. Listener and Replay functionality merged. 3 | * Added option to equally split traffic between multiple outputs: --split-output true 4 | * Saving requests to file and replaying from it 5 | * Injecting custom headers to http requests 6 | * Advanced stats using ElasticSearch 7 | 8 | v0.3.5 - 15 Sep 2013 9 | * Significantly improved test coverage 10 | * Fixed bug with redirect replay https://github.com/buger/gor/pull/15 11 | * Added limit on listener side 12 | * Improved stability (catch and log panic, instead of exiting) 13 | * Added License file 14 | 15 | v0.3.3 - 22 Jun 2013 16 | * Using TCP instead of UDP for communication between Listener and Replay 17 | * Significantly improved performance 18 | * Fixed bugs causing locking and message dropping (concurrency issues) 19 | * Rewrote concurrency model to use more channels 20 | 21 | v0.3 - 10 Jun 2013 22 | * Use RAW_SOCKETS instead of tcpdump 23 | * Own TCP stack 24 | * All HTTP request types support 25 | * Simplified request parsing 26 | -------------------------------------------------------------------------------- /ELASTICSEARCH.md: -------------------------------------------------------------------------------- 1 | gor & elasticsearch 2 | =================== 3 | 4 | Prerequisites 5 | ------------- 6 | 7 | - elasticsearch 8 | - kibana (Get it here: http://www.elasticsearch.org/overview/kibana/) 9 | - gor 10 | 11 | 12 | elasticsearch 13 | ------------- 14 | 15 | The default elasticsearch configuration is just fine for most workloads. You won't need clustering, sharding or something like that. 16 | 17 | In this example we're installing it on our gor replay server which gives us the elasticsearch listener on _http://localhost:9200_ 18 | 19 | 20 | kibana 21 | ------ 22 | 23 | Kibana (elasticsearch analytics web-ui) is just as simple. 24 | Download it, extract it and serve it via a simple webserver. 25 | (Could be nginx or apache) 26 | 27 | You could also use a shell, ```cd``` into the kibana directory and start a little quick and dirty python webserver with: 28 | 29 | ``` 30 | python -m SimpleHTTPServer 8000 31 | ``` 32 | 33 | In this example we're also choosing the gor replay server as our kibana host. If you choose a different server you'll have to point kibana to your elasticsearch host. 34 | 35 | 36 | gor 37 | --- 38 | 39 | Start your gor replay server with elasticsearch option: 40 | 41 | ``` 42 | ./gor --input-raw :8000 --output-http http://staging.com --output-http-elasticsearch localhost:9200/gor 43 | ``` 44 | 45 | 46 | (You don't have to create the index upfront. That will be done for you automatically) 47 | 48 | 49 | Now visit your kibana url, load the predefined dashboard from the gist https://gist.github.com/gottwald/b2c875037f24719a9616 and watch the data rush in. 50 | 51 | 52 | Troubleshooting 53 | --------------- 54 | 55 | The replay process may complain about __too many open files__. 56 | That's because your typical linux shell has a small open files soft limit at 1024. 57 | You can easily raise that when you do this before starting your _gor replay_ process: 58 | 59 | ``` 60 | ulimit -n 64000 61 | ``` 62 | 63 | Please be aware, this is not a permanent setting. It's just valid for the following jobs you start from that shell. 64 | 65 | We reached the 1024 limit in our tests with a ubuntu box replaying about 9000 requests per minute. (We had very slow responses there, should be way more with fast responses) 66 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, and 10 | distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by the copyright 13 | owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all other entities 16 | that control, are controlled by, or are under common control with that entity. 17 | For the purposes of this definition, "control" means (i) the power, direct or 18 | indirect, to cause the direction or management of such entity, whether by 19 | contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the 20 | outstanding shares, or (iii) beneficial ownership of such entity. 21 | 22 | "You" (or "Your") shall mean an individual or Legal Entity exercising 23 | permissions granted by this License. 24 | 25 | "Source" form shall mean the preferred form for making modifications, including 26 | but not limited to software source code, documentation source, and configuration 27 | files. 28 | 29 | "Object" form shall mean any form resulting from mechanical transformation or 30 | translation of a Source form, including but not limited to compiled object code, 31 | generated documentation, and conversions to other media types. 32 | 33 | "Work" shall mean the work of authorship, whether in Source or Object form, made 34 | available under the License, as indicated by a copyright notice that is included 35 | in or attached to the work (an example is provided in the Appendix below). 36 | 37 | "Derivative Works" shall mean any work, whether in Source or Object form, that 38 | is based on (or derived from) the Work and for which the editorial revisions, 39 | annotations, elaborations, or other modifications represent, as a whole, an 40 | original work of authorship. For the purposes of this License, Derivative Works 41 | shall not include works that remain separable from, or merely link (or bind by 42 | name) to the interfaces of, the Work and Derivative Works thereof. 43 | 44 | "Contribution" shall mean any work of authorship, including the original version 45 | of the Work and any modifications or additions to that Work or Derivative Works 46 | thereof, that is intentionally submitted to Licensor for inclusion in the Work 47 | by the copyright owner or by an individual or Legal Entity authorized to submit 48 | on behalf of the copyright owner. For the purposes of this definition, 49 | "submitted" means any form of electronic, verbal, or written communication sent 50 | to the Licensor or its representatives, including but not limited to 51 | communication on electronic mailing lists, source code control systems, and 52 | issue tracking systems that are managed by, or on behalf of, the Licensor for 53 | the purpose of discussing and improving the Work, but excluding communication 54 | that is conspicuously marked or otherwise designated in writing by the copyright 55 | owner as "Not a Contribution." 56 | 57 | "Contributor" shall mean Licensor and any individual or Legal Entity on behalf 58 | of whom a Contribution has been received by Licensor and subsequently 59 | incorporated within the Work. 60 | 61 | 2. Grant of Copyright License. 62 | 63 | Subject to the terms and conditions of this License, each Contributor hereby 64 | grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, 65 | irrevocable copyright license to reproduce, prepare Derivative Works of, 66 | publicly display, publicly perform, sublicense, and distribute the Work and such 67 | Derivative Works in Source or Object form. 68 | 69 | 3. Grant of Patent License. 70 | 71 | Subject to the terms and conditions of this License, each Contributor hereby 72 | grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, 73 | irrevocable (except as stated in this section) patent license to make, have 74 | made, use, offer to sell, sell, import, and otherwise transfer the Work, where 75 | such license applies only to those patent claims licensable by such Contributor 76 | that are necessarily infringed by their Contribution(s) alone or by combination 77 | of their Contribution(s) with the Work to which such Contribution(s) was 78 | submitted. If You institute patent litigation against any entity (including a 79 | cross-claim or counterclaim in a lawsuit) alleging that the Work or a 80 | Contribution incorporated within the Work constitutes direct or contributory 81 | patent infringement, then any patent licenses granted to You under this License 82 | for that Work shall terminate as of the date such litigation is filed. 83 | 84 | 4. Redistribution. 85 | 86 | You may reproduce and distribute copies of the Work or Derivative Works thereof 87 | in any medium, with or without modifications, and in Source or Object form, 88 | provided that You meet the following conditions: 89 | 90 | You must give any other recipients of the Work or Derivative Works a copy of 91 | this License; and 92 | You must cause any modified files to carry prominent notices stating that You 93 | changed the files; and 94 | You must retain, in the Source form of any Derivative Works that You distribute, 95 | all copyright, patent, trademark, and attribution notices from the Source form 96 | of the Work, excluding those notices that do not pertain to any part of the 97 | Derivative Works; and 98 | If the Work includes a "NOTICE" text file as part of its distribution, then any 99 | Derivative Works that You distribute must include a readable copy of the 100 | attribution notices contained within such NOTICE file, excluding those notices 101 | that do not pertain to any part of the Derivative Works, in at least one of the 102 | following places: within a NOTICE text file distributed as part of the 103 | Derivative Works; within the Source form or documentation, if provided along 104 | with the Derivative Works; or, within a display generated by the Derivative 105 | Works, if and wherever such third-party notices normally appear. The contents of 106 | the NOTICE file are for informational purposes only and do not modify the 107 | License. You may add Your own attribution notices within Derivative Works that 108 | You distribute, alongside or as an addendum to the NOTICE text from the Work, 109 | provided that such additional attribution notices cannot be construed as 110 | modifying the License. 111 | You may add Your own copyright statement to Your modifications and may provide 112 | additional or different license terms and conditions for use, reproduction, or 113 | distribution of Your modifications, or for any such Derivative Works as a whole, 114 | provided Your use, reproduction, and distribution of the Work otherwise complies 115 | with the conditions stated in this License. 116 | 117 | 5. Submission of Contributions. 118 | 119 | Unless You explicitly state otherwise, any Contribution intentionally submitted 120 | for inclusion in the Work by You to the Licensor shall be under the terms and 121 | conditions of this License, without any additional terms or conditions. 122 | Notwithstanding the above, nothing herein shall supersede or modify the terms of 123 | any separate license agreement you may have executed with Licensor regarding 124 | such Contributions. 125 | 126 | 6. Trademarks. 127 | 128 | This License does not grant permission to use the trade names, trademarks, 129 | service marks, or product names of the Licensor, except as required for 130 | reasonable and customary use in describing the origin of the Work and 131 | reproducing the content of the NOTICE file. 132 | 133 | 7. Disclaimer of Warranty. 134 | 135 | Unless required by applicable law or agreed to in writing, Licensor provides the 136 | Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, 137 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, 138 | including, without limitation, any warranties or conditions of TITLE, 139 | NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are 140 | solely responsible for determining the appropriateness of using or 141 | redistributing the Work and assume any risks associated with Your exercise of 142 | permissions under this License. 143 | 144 | 8. Limitation of Liability. 145 | 146 | In no event and under no legal theory, whether in tort (including negligence), 147 | contract, or otherwise, unless required by applicable law (such as deliberate 148 | and grossly negligent acts) or agreed to in writing, shall any Contributor be 149 | liable to You for damages, including any direct, indirect, special, incidental, 150 | or consequential damages of any character arising as a result of this License or 151 | out of the use or inability to use the Work (including but not limited to 152 | damages for loss of goodwill, work stoppage, computer failure or malfunction, or 153 | any and all other commercial damages or losses), even if such Contributor has 154 | been advised of the possibility of such damages. 155 | 156 | 9. Accepting Warranty or Additional Liability. 157 | 158 | While redistributing the Work or Derivative Works thereof, You may choose to 159 | offer, and charge a fee for, acceptance of support, warranty, indemnity, or 160 | other liability obligations and/or rights consistent with this License. However, 161 | in accepting such obligations, You may act only on Your own behalf and on Your 162 | sole responsibility, not on behalf of any other Contributor, and only if You 163 | agree to indemnify, defend, and hold each Contributor harmless for any liability 164 | incurred by, or claims asserted against, such Contributor by reason of your 165 | accepting any such warranty or additional liability. 166 | 167 | END OF TERMS AND CONDITIONS 168 | 169 | APPENDIX: How to apply the Apache License to your work 170 | 171 | To apply the Apache License to your work, attach the following boilerplate 172 | notice, with the fields enclosed by brackets "[]" replaced with your own 173 | identifying information. (Don't include the brackets!) The text should be 174 | enclosed in the appropriate comment syntax for the file format. We also 175 | recommend that a file or class name and description of purpose be included on 176 | the same "printed page" as the copyright notice for easier identification within 177 | third-party archives. 178 | 179 | Copyright [yyyy] [name of copyright owner] 180 | 181 | Licensed under the Apache License, Version 2.0 (the "License"); 182 | you may not use this file except in compliance with the License. 183 | You may obtain a copy of the License at 184 | 185 | http://www.apache.org/licenses/LICENSE-2.0 186 | 187 | Unless required by applicable law or agreed to in writing, software 188 | distributed under the License is distributed on an "AS IS" BASIS, 189 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 190 | See the License for the specific language governing permissions and 191 | limitations under the License. 192 | -------------------------------------------------------------------------------- /Procfile: -------------------------------------------------------------------------------- 1 | web: python -m SimpleHTTPServer 8000 2 | replayed_web: python -m SimpleHTTPServer 8001 3 | listener: sudo -E go run ./bin/gor.go --input-raw :8000 --output-tcp :8002 --verbose 4 | replay: go run ./bin/gor.go --input-tcp :8002 --output-http localhost:8001 --verbose 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Stories in Ready](https://badge.waffle.io/buger/gor.png?label=ready)](https://waffle.io/buger/gor) 2 | [![Build Status](https://travis-ci.org/buger/gor.png?branch=master)](https://travis-ci.org/buger/gor) 3 | 4 | ## About 5 | 6 | Gor is a simple http traffic replication tool written in Go. 7 | Its main goal is to replay traffic from production servers to staging and dev environments. 8 | 9 | 10 | Now you can test your code on real user sessions in an automated and repeatable fashion. 11 | **No more falling down in production!** 12 | 13 | Here is basic worlkflow: The listener server catches http traffic and sends it to the replay server or saves to file.The replay server forwards traffic to a given address. 14 | 15 | 16 | ![Diagram](http://i.imgur.com/9mqj2SK.png) 17 | 18 | 19 | ## Examples 20 | 21 | ### Capture traffic from port 22 | ```bash 23 | # Run on servers where you want to catch traffic. You can run it on each `web` machine. 24 | sudo gor --input-raw :80 --output-tcp replay.local:28020 25 | 26 | # Replay server (replay.local). 27 | gor --input-tcp replay.local:28020 --output-http http://staging.com 28 | ``` 29 | 30 | ### Using 1 Gor instance for both listening and replaying 31 | It's recommended to use separate server for replaying traffic, but if you have enough CPU resources you can use single Gor instance. 32 | 33 | ``` 34 | sudo gor --input-raw :80 --output-http "http://staging.com" 35 | ``` 36 | 37 | ## Advanced use 38 | 39 | ### Rate limiting 40 | Both replay and listener support rate limiting. It can be useful if you want 41 | forward only part of production traffic and not overload your staging 42 | environment. You can specify your desired requests per second using the 43 | "|" operator after the server address: 44 | 45 | #### Limiting replay 46 | ``` 47 | # staging.server will not get more than 10 requests per second 48 | gor --input-tcp :28020 --output-http "http://staging.com|10" 49 | ``` 50 | 51 | #### Limiting listener 52 | ``` 53 | # replay server will not get more than 10 requests per second 54 | # useful for high-load environments 55 | gor --input-raw :80 --output-tcp "replay.local:28020|10" 56 | ``` 57 | 58 | ### Forward to multiple addresses 59 | 60 | You can forward traffic to multiple endpoints. Just add multiple --output-* arguments. 61 | ``` 62 | gor --input-tcp :28020 --output-http "http://staging.com" --output-http "http://dev.com" 63 | ``` 64 | 65 | #### Splitting traffic 66 | By default it will send same traffic to all outputs, but you have options to equally split it: 67 | 68 | ``` 69 | gor --input-tcp :28020 --output-http "http://staging.com" --output-http "http://dev.com" --split-output true 70 | ``` 71 | 72 | ### Saving requests to file 73 | You can save requests to file, and replay them later: 74 | ``` 75 | # write to file 76 | gor --input-raw :80 --output-file requests.gor 77 | 78 | # read from file 79 | gor --input-file requests.gor --output-http "http://staging.com" 80 | ``` 81 | 82 | **Note:** Replay will preserve the original time differences between requests. 83 | 84 | ### Injecting headers 85 | 86 | Additional headers can be injected/overwritten into requests during replay. This may be useful if the hostname that staging responds to differs from production, you need to identify requests generated by Gor, or enable feature flagged functionality in an application: 87 | 88 | ``` 89 | gor --input-raw :80 --output-http "http://staging.server" \ 90 | --output-http-header "Host: staging.server" \ 91 | --output-http-header "User-Agent: Replayed by Gor" -header " \ 92 | --output-http-header "Enable-Feature-X: true" 93 | ``` 94 | 95 | ## Filtering HTTP methods 96 | 97 | Requests not matching a specified whitelist can be filtered out. For example to strip non-nullipotent requests: 98 | 99 | ``` 100 | gor --input-raw :80 --output-http "http://staging.server" \ 101 | --output-http-method GET \ 102 | --output-http-method OPTIONS 103 | ``` 104 | 105 | ### Basic Auth 106 | 107 | If your development or staging environment is protected by Basic Authentication then those credentials can be injected in during the replay: 108 | 109 | ``` 110 | gor --input-raw :80 --output-http "http://user:pass@staging .com" 111 | ``` 112 | 113 | Note: This will overwrite any Authorization headers in the original request. 114 | 115 | ## Stats 116 | ### ElasticSearch 117 | For deep response analyze based on url, cookie, user-agent and etc. you can export response metadata to ElasticSearch. See [ELASTICSEARCH.md](ELASTICSEARCH.md) for more details. 118 | 119 | ``` 120 | gor --input-tcp :80 --output-http "http://staging.com" --output-http-elasticsearch "es_host:api_port/index_name" 121 | ``` 122 | 123 | 124 | ## Additional help 125 | 126 | Feel free to ask question directly by email or by creating github issue. 127 | 128 | ## Latest releases (including binaries) 129 | 130 | https://github.com/buger/gor/releases 131 | 132 | ## Command line reference 133 | `gor -h` output: 134 | ``` 135 | -cpuprofile="": write cpu profile to file 136 | -memprofile="": write memory profile to this file 137 | 138 | -input-dummy=[]: Used for testing outputs. Emits 'Get /' request every 1s 139 | 140 | -input-file=[]: Read requests from file: 141 | gor --input-file ./requests.gor --output-http staging.com 142 | 143 | -input-raw=[]: Capture traffic from given port (use RAW sockets and require *sudo* access): 144 | # Capture traffic from 8080 port 145 | gor --input-raw :8080 --output-http staging.com 146 | 147 | -input-tcp=[]: Used for internal communication between Gor instances. Example: 148 | # Receive requests from other Gor instances on 28020 port, and redirect output to staging 149 | gor --input-tcp :28020 --output-http staging.com 150 | 151 | -output-dummy=[]: Used for testing inputs. Just prints data coming from inputs. 152 | 153 | -output-file=[]: Write incoming requests to file: 154 | gor --input-raw :80 --output-file ./requests.gor 155 | 156 | -output-http=[]: Forwards incoming requests to given http address. 157 | # Redirect all incoming requests to staging.com address 158 | gor --input-raw :80 --output-http http://staging.com 159 | 160 | -output-http-elasticsearch="": Send request and response stats to ElasticSearch: 161 | gor --input-raw :8080 --output-http staging.com --output-http-elasticsearch 'es_host:api_port/index_name' 162 | 163 | -output-http-header=[]: Inject additional headers to http reqest: 164 | gor --input-raw :8080 --output-http staging.com --output-http-header 'User-Agent: Gor' 165 | 166 | -output-tcp=[]: Used for internal communication between Gor instances. Example: 167 | # Listen for requests on 80 port and forward them to other Gor instance on 28020 port 168 | gor --input-raw :80 --output-tcp replay.local:28020 169 | 170 | -split-output=false: By default each output gets same traffic. If set to `true` it splits traffic equally among all outputs. 171 | ``` 172 | 173 | ## Building from source 174 | 1. Setup standard Go environment http://golang.org/doc/code.html and ensure that $GOPATH environment variable properly set. 175 | 2. `go get github.com/buger/gor`. 176 | 3. `cd $GOPATH/src/github.com/buger/gor` 177 | 4. `go build ./bin/gor.go` to get binary, or `go run ./bin/gor.go` to build and run (useful for development) 178 | 179 | ## FAQ 180 | 181 | ### What OS are supported? 182 | For now only Linux based. *BSD (including MacOS is not supported yet, check https://github.com/buger/gor/issues/22 for details) 183 | 184 | ### Why does the `--input-raw` requires sudo or root access? 185 | Listener works by sniffing traffic from a given port. It's accessible 186 | only by using sudo or root access. 187 | 188 | ### I'm getting 'too many open files' error 189 | Typical linux shell has a small open files soft limit at 1024. You can easily raise that when you do this before starting your gor replay process: 190 | 191 | ulimit -n 64000 192 | 193 | More about ulimit: http://blog.thecodingmachine.com/content/solving-too-many-open-files-exception-red5-or-any-other-application 194 | 195 | ## Tuning 196 | 197 | To achieve the top most performance you should tune the source server system limits: 198 | 199 | net.ipv4.tcp_max_tw_buckets = 65536 200 | net.ipv4.tcp_tw_recycle = 1 201 | net.ipv4.tcp_tw_reuse = 0 202 | net.ipv4.tcp_max_syn_backlog = 131072 203 | net.ipv4.tcp_syn_retries = 3 204 | net.ipv4.tcp_synack_retries = 3 205 | net.ipv4.tcp_retries1 = 3 206 | net.ipv4.tcp_retries2 = 8 207 | net.ipv4.tcp_rmem = 16384 174760 349520 208 | net.ipv4.tcp_wmem = 16384 131072 262144 209 | net.ipv4.tcp_mem = 262144 524288 1048576 210 | net.ipv4.tcp_max_orphans = 65536 211 | net.ipv4.tcp_fin_timeout = 10 212 | net.ipv4.tcp_low_latency = 1 213 | net.ipv4.tcp_syncookies = 0 214 | 215 | 216 | ## Contributing 217 | 218 | 1. Fork it 219 | 2. Create your feature branch (git checkout -b my-new-feature) 220 | 3. Commit your changes (git commit -am 'Added some feature') 221 | 4. Push to the branch (git push origin my-new-feature) 222 | 5. Create new Pull Request 223 | 224 | ## Companies using Gor 225 | 226 | * http://granify.com 227 | * [GOV.UK](https://www.gov.uk) ([Government Digital Service](http://digital.cabinetoffice.gov.uk/)) 228 | * To add your company drop me a line to github.com/buger or leonsbox@gmail.com 229 | -------------------------------------------------------------------------------- /bin/Gofile: -------------------------------------------------------------------------------- 1 | { 2 | "env" : [ 3 | { 4 | "name" : "development", 5 | "packages": [ 6 | { "name" : "github.com/buger/gor", "branch" : "master" }, 7 | { "name" : "github.com/mattbaird/elastigo", "branch" : "master" } 8 | ] 9 | } 10 | ] 11 | } 12 | -------------------------------------------------------------------------------- /bin/gor.go: -------------------------------------------------------------------------------- 1 | // Gor is simple http traffic replication tool written in Go. Its main goal to replay traffic from production servers to staging and dev environments. 2 | // Now you can test your code on real user sessions in an automated and repeatable fashion. 3 | package main 4 | 5 | import ( 6 | "flag" 7 | "fmt" 8 | "log" 9 | "os" 10 | "runtime/debug" 11 | "runtime/pprof" 12 | "time" 13 | 14 | "github.com/buger/gor" 15 | ) 16 | 17 | var ( 18 | mode string 19 | cpuprofile = flag.String("cpuprofile", "", "write cpu profile to file") 20 | memprofile = flag.String("memprofile", "", "write memory profile to this file") 21 | ) 22 | 23 | func main() { 24 | // Don't exit on panic 25 | defer func() { 26 | if r := recover(); r != nil { 27 | if _, ok := r.(error); !ok { 28 | fmt.Printf("PANIC: pkg: %v %s \n", r, debug.Stack()) 29 | } 30 | } 31 | }() 32 | 33 | fmt.Println("Version:", gor.VERSION) 34 | 35 | flag.Parse() 36 | gor.InitPlugins() 37 | 38 | if len(gor.Plugins.Inputs) == 0 || len(gor.Plugins.Outputs) == 0 { 39 | log.Fatal("Required at least 1 input and 1 output") 40 | } 41 | 42 | if *memprofile != "" { 43 | profileMEM(*memprofile) 44 | } 45 | 46 | if *cpuprofile != "" { 47 | profileCPU(*cpuprofile) 48 | } 49 | 50 | gor.Start(nil) 51 | } 52 | 53 | func profileCPU(cpuprofile string) { 54 | if cpuprofile != "" { 55 | f, err := os.Create(cpuprofile) 56 | if err != nil { 57 | log.Fatal(err) 58 | } 59 | pprof.StartCPUProfile(f) 60 | 61 | time.AfterFunc(60*time.Second, func() { 62 | pprof.StopCPUProfile() 63 | f.Close() 64 | log.Println("Stop profiling after 60 seconds") 65 | }) 66 | } 67 | } 68 | 69 | func profileMEM(memprofile string) { 70 | if memprofile != "" { 71 | f, err := os.Create(memprofile) 72 | if err != nil { 73 | log.Fatal(err) 74 | } 75 | time.AfterFunc(60*time.Second, func() { 76 | pprof.WriteHeapProfile(f) 77 | f.Close() 78 | }) 79 | } 80 | } 81 | -------------------------------------------------------------------------------- /elasticsearch/elasticsearch.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "encoding/json" 5 | "github.com/mattbaird/elastigo/api" 6 | "github.com/mattbaird/elastigo/core" 7 | "log" 8 | "net/http" 9 | "regexp" 10 | "time" 11 | ) 12 | 13 | type ESUriErorr struct{} 14 | 15 | func (e *ESUriErorr) Error() string { 16 | return "Wrong ElasticSearch URL format. Expected to be: host:port/index_name" 17 | } 18 | 19 | type ESPlugin struct { 20 | Active bool 21 | ApiPort string 22 | Host string 23 | Index string 24 | indexor *core.BulkIndexer 25 | done chan bool 26 | } 27 | 28 | type ESRequestResponse struct { 29 | ReqUrl string `json:"Req_URL"` 30 | ReqMethod string `json:"Req_Method"` 31 | ReqUserAgent string `json:"Req_User-Agent"` 32 | ReqAcceptLanguage string `json:"Req_Accept-Language,omitempty"` 33 | ReqAccept string `json:"Req_Accept,omitempty"` 34 | ReqAcceptEncoding string `json:"Req_Accept-Encoding,omitempty"` 35 | ReqIfModifiedSince string `json:"Req_If-Modified-Since,omitempty"` 36 | ReqConnection string `json:"Req_Connection,omitempty"` 37 | ReqCookies []*http.Cookie `json:"Req_Cookies,omitempty"` 38 | RespStatus string `json:"Resp_Status"` 39 | RespStatusCode int `json:"Resp_Status-Code"` 40 | RespProto string `json:"Resp_Proto,omitempty"` 41 | RespContentLength int64 `json:"Resp_Content-Length,omitempty"` 42 | RespContentType string `json:"Resp_Content-Type,omitempty"` 43 | RespTransferEncoding []string `json:"Resp_Transfer-Encoding,omitempty"` 44 | RespContentEncoding string `json:"Resp_Content-Encoding,omitempty"` 45 | RespExpires string `json:"Resp_Expires,omitempty"` 46 | RespCacheControl string `json:"Resp_Cache-Control,omitempty"` 47 | RespVary string `json:"Resp_Vary,omitempty"` 48 | RespSetCookie string `json:"Resp_Set-Cookie,omitempty"` 49 | Rtt int64 `json:"RTT"` 50 | Timestamp time.Time 51 | } 52 | 53 | // Parse ElasticSearch URI 54 | // 55 | // Proper format is: host:port/index_name 56 | func parseURI(URI string) (err error, host string, port string, index string) { 57 | rURI := regexp.MustCompile("(.+):([0-9]+)/(.+)") 58 | match := rURI.FindAllStringSubmatch(URI, -1) 59 | 60 | if len(match) == 0 { 61 | err = new(ESUriErorr) 62 | } else { 63 | host = match[0][1] 64 | port = match[0][2] 65 | index = match[0][3] 66 | } 67 | 68 | return 69 | } 70 | 71 | func (p *ESPlugin) Init(URI string) { 72 | var err error 73 | 74 | err, p.Host, p.ApiPort, p.Index = parseURI(URI) 75 | 76 | if err != nil { 77 | log.Fatal("Can't initialize ElasticSearch plugin.", err) 78 | } 79 | 80 | api.Domain = p.Host 81 | api.Port = p.ApiPort 82 | 83 | p.indexor = core.NewBulkIndexerErrors(50, 60) 84 | p.done = make(chan bool) 85 | p.indexor.Run(p.done) 86 | 87 | // Only start the ErrorHandler goroutine when in verbose mode 88 | // no need to burn ressources otherwise 89 | // go p.ErrorHandler() 90 | 91 | log.Println("Initialized Elasticsearch Plugin") 92 | return 93 | } 94 | 95 | func (p *ESPlugin) IndexerShutdown() { 96 | p.done <- true 97 | return 98 | } 99 | 100 | func (p *ESPlugin) ErrorHandler() { 101 | for { 102 | errBuf := <-p.indexor.ErrorChannel 103 | log.Println(errBuf.Err) 104 | } 105 | } 106 | 107 | func (p *ESPlugin) RttDurationToMs(d time.Duration) int64 { 108 | sec := d / time.Second 109 | nsec := d % time.Second 110 | fl := float64(sec) + float64(nsec)*1e-6 111 | return int64(fl) 112 | } 113 | 114 | func (p *ESPlugin) ResponseAnalyze(req *http.Request, resp *http.Response, start, stop time.Time) { 115 | if resp == nil { 116 | // nil http response - skipped elasticsearch export for this request 117 | return 118 | } 119 | t := time.Now() 120 | rtt := p.RttDurationToMs(stop.Sub(start)) 121 | 122 | esResp := ESRequestResponse{ 123 | ReqUrl: req.URL.String(), 124 | ReqMethod: req.Method, 125 | ReqUserAgent: req.UserAgent(), 126 | ReqAcceptLanguage: req.Header.Get("Accept-Language"), 127 | ReqAccept: req.Header.Get("Accept"), 128 | ReqAcceptEncoding: req.Header.Get("Accept-Encoding"), 129 | ReqIfModifiedSince: req.Header.Get("If-Modified-Since"), 130 | ReqConnection: req.Header.Get("Connection"), 131 | ReqCookies: req.Cookies(), 132 | RespStatus: resp.Status, 133 | RespStatusCode: resp.StatusCode, 134 | RespProto: resp.Proto, 135 | RespContentLength: resp.ContentLength, 136 | RespContentType: resp.Header.Get("Content-Type"), 137 | RespTransferEncoding: resp.TransferEncoding, 138 | RespContentEncoding: resp.Header.Get("Content-Encoding"), 139 | RespExpires: resp.Header.Get("Expires"), 140 | RespCacheControl: resp.Header.Get("Cache-Control"), 141 | RespVary: resp.Header.Get("Vary"), 142 | RespSetCookie: resp.Header.Get("Set-Cookie"), 143 | Rtt: rtt, 144 | Timestamp: t, 145 | } 146 | j, err := json.Marshal(&esResp) 147 | if err != nil { 148 | log.Println(err) 149 | } else { 150 | p.indexor.Index(p.Index, "RequestResponse", "", "", &t, j) 151 | } 152 | return 153 | } 154 | -------------------------------------------------------------------------------- /emitter.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | ) 6 | 7 | func Start(stop chan int) { 8 | for _, in := range Plugins.Inputs { 9 | go CopyMulty(in, Plugins.Outputs...) 10 | } 11 | 12 | select { 13 | case <-stop: 14 | return 15 | } 16 | } 17 | 18 | // Copy from 1 reader to multiple writers 19 | func CopyMulty(src io.Reader, writers ...io.Writer) (err error) { 20 | buf := make([]byte, 32*1024) 21 | wIndex := 0 22 | 23 | for { 24 | nr, er := src.Read(buf) 25 | if nr > 0 { 26 | Debug("Sending", src, ": ", string(buf[0:nr])) 27 | 28 | if Settings.splitOutput { 29 | // Simple round robin 30 | writers[wIndex].Write(buf[0:nr]) 31 | 32 | wIndex++ 33 | 34 | if wIndex >= len(writers) { 35 | wIndex = 0 36 | } 37 | } else { 38 | for _, dst := range writers { 39 | dst.Write(buf[0:nr]) 40 | } 41 | } 42 | 43 | } 44 | if er == io.EOF { 45 | break 46 | } 47 | if er != nil { 48 | err = er 49 | break 50 | } 51 | } 52 | return err 53 | } 54 | -------------------------------------------------------------------------------- /emitter_test.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | "sync" 6 | "sync/atomic" 7 | "testing" 8 | ) 9 | 10 | func TestEmitter(t *testing.T) { 11 | wg := new(sync.WaitGroup) 12 | quit := make(chan int) 13 | 14 | input := NewTestInput() 15 | output := NewTestOutput(func(data []byte) { 16 | wg.Done() 17 | }) 18 | 19 | Plugins.Inputs = []io.Reader{input} 20 | Plugins.Outputs = []io.Writer{output} 21 | 22 | go Start(quit) 23 | 24 | for i := 0; i < 1000; i++ { 25 | wg.Add(1) 26 | input.EmitGET() 27 | } 28 | 29 | wg.Wait() 30 | 31 | close(quit) 32 | } 33 | 34 | func TestEmitterRoundRobin(t *testing.T) { 35 | wg := new(sync.WaitGroup) 36 | quit := make(chan int) 37 | 38 | input := NewTestInput() 39 | 40 | var counter1, counter2 int32 41 | 42 | output1 := NewTestOutput(func(data []byte) { 43 | atomic.AddInt32(&counter1, 1) 44 | wg.Done() 45 | }) 46 | 47 | output2 := NewTestOutput(func(data []byte) { 48 | atomic.AddInt32(&counter2, 1) 49 | wg.Done() 50 | }) 51 | 52 | Plugins.Inputs = []io.Reader{input} 53 | Plugins.Outputs = []io.Writer{output1, output2} 54 | 55 | Settings.splitOutput = true 56 | 57 | go Start(quit) 58 | 59 | for i := 0; i < 1000; i++ { 60 | wg.Add(1) 61 | input.EmitGET() 62 | } 63 | 64 | wg.Wait() 65 | 66 | close(quit) 67 | 68 | if counter1 == 0 || counter2 == 0 { 69 | t.Errorf("Round robin should split traffic equally: %d vs %d", counter1, counter2) 70 | } 71 | 72 | Settings.splitOutput = false 73 | } 74 | -------------------------------------------------------------------------------- /input_dummy.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "time" 5 | ) 6 | 7 | type DummyInput struct { 8 | data chan []byte 9 | } 10 | 11 | func NewDummyInput(options string) (di *DummyInput) { 12 | di = new(DummyInput) 13 | di.data = make(chan []byte) 14 | 15 | go di.emit() 16 | 17 | return 18 | } 19 | 20 | func (i *DummyInput) Read(data []byte) (int, error) { 21 | buf := <-i.data 22 | copy(data, buf) 23 | 24 | return len(buf), nil 25 | } 26 | 27 | func (i *DummyInput) emit() { 28 | ticker := time.NewTicker(time.Second) 29 | 30 | for { 31 | select { 32 | case <-ticker.C: 33 | i.data <- []byte("GET / HTTP/1.1\r\n\r\n") 34 | } 35 | } 36 | } 37 | 38 | func (i *DummyInput) String() string { 39 | return "Dummy Input" 40 | } 41 | -------------------------------------------------------------------------------- /input_file.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "encoding/gob" 5 | "log" 6 | "os" 7 | "time" 8 | ) 9 | 10 | type FileInput struct { 11 | data chan []byte 12 | path string 13 | decoder *gob.Decoder 14 | } 15 | 16 | func NewFileInput(path string) (i *FileInput) { 17 | i = new(FileInput) 18 | i.data = make(chan []byte) 19 | i.path = path 20 | i.Init(path) 21 | 22 | go i.emit() 23 | 24 | return 25 | } 26 | 27 | func (i *FileInput) Init(path string) { 28 | file, err := os.Open(path) 29 | 30 | if err != nil { 31 | log.Fatal(i, "Cannot open file %q. Error: %s", path, err) 32 | } 33 | 34 | i.decoder = gob.NewDecoder(file) 35 | } 36 | 37 | func (i *FileInput) Read(data []byte) (int, error) { 38 | buf := <-i.data 39 | copy(data, buf) 40 | 41 | return len(buf), nil 42 | } 43 | 44 | func (i *FileInput) String() string { 45 | return "File input: " + i.path 46 | } 47 | 48 | func (i *FileInput) emit() { 49 | var lastTime int64 50 | 51 | for { 52 | raw := new(RawRequest) 53 | err := i.decoder.Decode(raw) 54 | 55 | if err != nil { 56 | return 57 | } 58 | 59 | if lastTime != 0 { 60 | time.Sleep(time.Duration(raw.Timestamp - lastTime)) 61 | lastTime = raw.Timestamp 62 | } 63 | 64 | i.data <- raw.Request 65 | } 66 | } 67 | -------------------------------------------------------------------------------- /input_raw.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | raw "github.com/buger/gor/raw_socket_listener" 5 | "log" 6 | "net" 7 | ) 8 | 9 | type RAWInput struct { 10 | data chan []byte 11 | address string 12 | } 13 | 14 | func NewRAWInput(address string) (i *RAWInput) { 15 | i = new(RAWInput) 16 | i.data = make(chan []byte) 17 | i.address = address 18 | 19 | go i.listen(address) 20 | 21 | return 22 | } 23 | 24 | func (i *RAWInput) Read(data []byte) (int, error) { 25 | buf := <-i.data 26 | copy(data, buf) 27 | 28 | return len(buf), nil 29 | } 30 | 31 | func (i *RAWInput) listen(address string) { 32 | host, port, err := net.SplitHostPort(address) 33 | 34 | if err != nil { 35 | log.Fatal("input-raw: error while parsing address", err) 36 | } 37 | 38 | listener := raw.NewListener(host, port) 39 | 40 | for { 41 | // Receiving TCPMessage object 42 | m := listener.Receive() 43 | 44 | i.data <- m.Bytes() 45 | } 46 | } 47 | 48 | func (i *RAWInput) String() string { 49 | return "RAW Socket input: " + i.address 50 | } 51 | -------------------------------------------------------------------------------- /input_raw_test.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | "net/http" 6 | "sync" 7 | "testing" 8 | ) 9 | 10 | func TestRAWInput(t *testing.T) { 11 | startHTTP := func(addr string, cb func(*http.Request)) { 12 | handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { 13 | cb(r) 14 | }) 15 | 16 | go http.ListenAndServe(addr, handler) 17 | } 18 | 19 | wg := new(sync.WaitGroup) 20 | quit := make(chan int) 21 | 22 | input := NewRAWInput("127.0.0.1:50004") 23 | output := NewTestOutput(func(data []byte) { 24 | wg.Done() 25 | }) 26 | 27 | startHTTP("127.0.0.1:50004", func(req *http.Request) {}) 28 | 29 | Plugins.Inputs = []io.Reader{input} 30 | Plugins.Outputs = []io.Writer{output} 31 | 32 | go Start(quit) 33 | 34 | wg.Add(100) 35 | for i := 0; i < 100; i++ { 36 | res, _ := http.Get("http://127.0.0.1:50004") 37 | res.Body.Close() 38 | } 39 | 40 | wg.Wait() 41 | 42 | close(quit) 43 | } 44 | -------------------------------------------------------------------------------- /input_tcp.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | "log" 6 | "net" 7 | ) 8 | 9 | // Can be tested using nc tool: 10 | // echo "asdad" | nc 127.0.0.1 27017 11 | // 12 | type TCPInput struct { 13 | data chan []byte 14 | address string 15 | } 16 | 17 | func NewTCPInput(address string) (i *TCPInput) { 18 | i = new(TCPInput) 19 | i.data = make(chan []byte) 20 | i.address = address 21 | 22 | i.listen(address) 23 | 24 | return 25 | } 26 | 27 | func (i *TCPInput) Read(data []byte) (int, error) { 28 | buf := <-i.data 29 | copy(data, buf) 30 | 31 | return len(buf), nil 32 | } 33 | 34 | func (i *TCPInput) listen(address string) { 35 | listener, err := net.Listen("tcp", address) 36 | 37 | if err != nil { 38 | log.Fatal("Can't start:", err) 39 | } 40 | 41 | go func() { 42 | for { 43 | conn, err := listener.Accept() 44 | 45 | if err != nil { 46 | log.Println("Error while Accept()", err) 47 | continue 48 | } 49 | 50 | go i.handleConnection(conn) 51 | } 52 | }() 53 | } 54 | 55 | func (i *TCPInput) handleConnection(conn net.Conn) { 56 | defer conn.Close() 57 | 58 | var read = true 59 | var response []byte 60 | var buf []byte 61 | 62 | buf = make([]byte, 4094) 63 | 64 | for read { 65 | n, err := conn.Read(buf) 66 | 67 | switch err { 68 | case io.EOF: 69 | read = false 70 | case nil: 71 | response = append(response, buf[:n]...) 72 | if n < 4096 { 73 | read = false 74 | } 75 | default: 76 | read = false 77 | } 78 | } 79 | 80 | i.data <- response 81 | } 82 | 83 | func (i *TCPInput) String() string { 84 | return "TCP input: " + i.address 85 | } 86 | -------------------------------------------------------------------------------- /input_tcp_test.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | "log" 6 | "net" 7 | "sync" 8 | "testing" 9 | ) 10 | 11 | func TestTCPInput(t *testing.T) { 12 | wg := new(sync.WaitGroup) 13 | quit := make(chan int) 14 | 15 | input := NewTCPInput("127.0.0.1:50001") 16 | output := NewTestOutput(func(data []byte) { 17 | wg.Done() 18 | }) 19 | 20 | Plugins.Inputs = []io.Reader{input} 21 | Plugins.Outputs = []io.Writer{output} 22 | 23 | go Start(quit) 24 | 25 | for i := 0; i < 100; i++ { 26 | wg.Add(1) 27 | sendTCP("127.0.0.1:50001", []byte("GET / HTTP/1.1\r\n\r\n")) 28 | } 29 | 30 | wg.Wait() 31 | 32 | close(quit) 33 | } 34 | 35 | func sendTCP(addr string, data []byte) { 36 | tcpAddr, err := net.ResolveTCPAddr("tcp", addr) 37 | if err != nil { 38 | log.Fatal(err) 39 | } 40 | 41 | conn, err := net.DialTCP("tcp", nil, tcpAddr) 42 | if err != nil { 43 | log.Fatal(err) 44 | } 45 | 46 | conn.Write(data) 47 | } 48 | -------------------------------------------------------------------------------- /limiter.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "fmt" 5 | "io" 6 | "time" 7 | ) 8 | 9 | type Limiter struct { 10 | writer io.Writer 11 | limit int 12 | 13 | currentRPS int 14 | currentTime int64 15 | } 16 | 17 | func NewLimiter(writer io.Writer, limit int) (l *Limiter) { 18 | l = new(Limiter) 19 | l.limit = limit 20 | l.writer = writer 21 | l.currentTime = time.Now().UnixNano() 22 | 23 | return 24 | } 25 | 26 | func (l *Limiter) Write(data []byte) (n int, err error) { 27 | if (time.Now().UnixNano() - l.currentTime) > time.Second.Nanoseconds() { 28 | l.currentTime = time.Now().UnixNano() 29 | l.currentRPS = 0 30 | } 31 | 32 | if l.currentRPS >= l.limit { 33 | return 0, nil 34 | } 35 | 36 | n, err = l.writer.Write(data) 37 | 38 | l.currentRPS++ 39 | 40 | return 41 | } 42 | 43 | func (l *Limiter) String() string { 44 | return fmt.Sprintf("Limiting %s to: %d", l.writer, l.limit) 45 | } 46 | -------------------------------------------------------------------------------- /limiter_test.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | "sync" 6 | "testing" 7 | ) 8 | 9 | func TestLimiter(t *testing.T) { 10 | wg := new(sync.WaitGroup) 11 | quit := make(chan int) 12 | 13 | input := NewTestInput() 14 | output := NewLimiter(NewTestOutput(func(data []byte) { 15 | wg.Done() 16 | }), 10) 17 | wg.Add(10) 18 | 19 | Plugins.Inputs = []io.Reader{input} 20 | Plugins.Outputs = []io.Writer{output} 21 | 22 | go Start(quit) 23 | 24 | for i := 0; i < 100; i++ { 25 | input.EmitGET() 26 | } 27 | 28 | wg.Wait() 29 | 30 | close(quit) 31 | } 32 | -------------------------------------------------------------------------------- /output_dummy.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "fmt" 5 | ) 6 | 7 | type DummyOutput struct { 8 | } 9 | 10 | func NewDummyOutput(options string) (di *DummyOutput) { 11 | di = new(DummyOutput) 12 | 13 | return 14 | } 15 | 16 | func (i *DummyOutput) Write(data []byte) (int, error) { 17 | fmt.Println("Writing message: ", data) 18 | 19 | return len(data), nil 20 | } 21 | 22 | func (i *DummyOutput) String() string { 23 | return "Dummy Output" 24 | } 25 | -------------------------------------------------------------------------------- /output_file.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "encoding/gob" 5 | "log" 6 | "os" 7 | "time" 8 | ) 9 | 10 | type RawRequest struct { 11 | Timestamp int64 12 | Request []byte 13 | } 14 | 15 | type FileOutput struct { 16 | path string 17 | encoder *gob.Encoder 18 | file *os.File 19 | } 20 | 21 | func NewFileOutput(path string) (o *FileOutput) { 22 | o = new(FileOutput) 23 | o.path = path 24 | o.Init(path) 25 | 26 | return 27 | } 28 | 29 | func (o *FileOutput) Init(path string) { 30 | var err error 31 | 32 | o.file, err = os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0660) 33 | 34 | if err != nil { 35 | log.Fatal(o, "Cannot open file %q. Error: %s", path, err) 36 | } 37 | 38 | o.encoder = gob.NewEncoder(o.file) 39 | } 40 | 41 | func (o *FileOutput) Write(data []byte) (n int, err error) { 42 | raw := RawRequest{time.Now().UnixNano(), data} 43 | 44 | o.encoder.Encode(raw) 45 | 46 | return len(data), nil 47 | } 48 | 49 | func (o *FileOutput) String() string { 50 | return "File output: " + o.path 51 | } 52 | -------------------------------------------------------------------------------- /output_file_test.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | "sync" 6 | "testing" 7 | ) 8 | 9 | func TestFileOutput(t *testing.T) { 10 | wg := new(sync.WaitGroup) 11 | quit := make(chan int) 12 | 13 | input := NewTestInput() 14 | output := NewFileOutput("/tmp/test_requests.gor") 15 | 16 | Plugins.Inputs = []io.Reader{input} 17 | Plugins.Outputs = []io.Writer{output} 18 | 19 | go Start(quit) 20 | 21 | for i := 0; i < 100; i++ { 22 | wg.Add(2) 23 | input.EmitGET() 24 | input.EmitPOST() 25 | } 26 | close(quit) 27 | 28 | quit = make(chan int) 29 | 30 | input2 := NewFileInput("/tmp/test_requests.gor") 31 | output2 := NewTestOutput(func(data []byte) { 32 | wg.Done() 33 | }) 34 | 35 | Plugins.Inputs = []io.Reader{input2} 36 | Plugins.Outputs = []io.Writer{output2} 37 | 38 | go Start(quit) 39 | 40 | wg.Wait() 41 | close(quit) 42 | } 43 | -------------------------------------------------------------------------------- /output_http.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "bufio" 5 | "bytes" 6 | es "github.com/buger/gor/elasticsearch" 7 | "io" 8 | "log" 9 | "net/http" 10 | "net/url" 11 | "strconv" 12 | "strings" 13 | "time" 14 | ) 15 | 16 | type RedirectNotAllowed struct{} 17 | 18 | func (e *RedirectNotAllowed) Error() string { 19 | return "Redirects not allowed" 20 | } 21 | 22 | // customCheckRedirect disables redirects https://github.com/buger/gor/pull/15 23 | func customCheckRedirect(req *http.Request, via []*http.Request) error { 24 | if len(via) >= 0 { 25 | return new(RedirectNotAllowed) 26 | } 27 | return nil 28 | } 29 | 30 | // ParseRequest in []byte returns a http request or an error 31 | func ParseRequest(data []byte) (request *http.Request, err error) { 32 | buf := bytes.NewBuffer(data) 33 | reader := bufio.NewReader(buf) 34 | 35 | request, err = http.ReadRequest(reader) 36 | 37 | return 38 | } 39 | 40 | type HTTPOutput struct { 41 | address string 42 | limit int 43 | 44 | headers HTTPHeaders 45 | methods HTTPMethods 46 | 47 | elasticSearch *es.ESPlugin 48 | } 49 | 50 | func NewHTTPOutput(options string, headers HTTPHeaders, methods HTTPMethods, elasticSearchAddr string) io.Writer { 51 | o := new(HTTPOutput) 52 | 53 | optionsArr := strings.Split(options, "|") 54 | address := optionsArr[0] 55 | 56 | if !strings.HasPrefix(address, "http") { 57 | address = "http://" + address 58 | } 59 | 60 | o.address = address 61 | o.headers = headers 62 | o.methods = methods 63 | 64 | if elasticSearchAddr != "" { 65 | o.elasticSearch = new(es.ESPlugin) 66 | o.elasticSearch.Init(elasticSearchAddr) 67 | } 68 | 69 | if len(optionsArr) > 1 { 70 | o.limit, _ = strconv.Atoi(optionsArr[1]) 71 | } 72 | 73 | if o.limit > 0 { 74 | return NewLimiter(o, o.limit) 75 | } else { 76 | return o 77 | } 78 | } 79 | 80 | func (o *HTTPOutput) Write(data []byte) (n int, err error) { 81 | buf := make([]byte, len(data)) 82 | copy(buf, data) 83 | 84 | go o.sendRequest(buf) 85 | 86 | return len(data), nil 87 | } 88 | 89 | func (o *HTTPOutput) sendRequest(data []byte) { 90 | request, err := ParseRequest(data) 91 | 92 | if err != nil { 93 | log.Println("Can not parse request", string(data), err) 94 | return 95 | } 96 | 97 | if len(o.methods) > 0 && !o.methods.Contains(request.Method) { 98 | return 99 | } 100 | 101 | client := &http.Client{ 102 | CheckRedirect: customCheckRedirect, 103 | } 104 | 105 | // Change HOST of original request 106 | URL := o.address + request.URL.Path + "?" + request.URL.RawQuery 107 | 108 | request.RequestURI = "" 109 | request.URL, _ = url.ParseRequestURI(URL) 110 | 111 | for _, header := range o.headers { 112 | request.Header.Set(header.Name, header.Value) 113 | } 114 | 115 | start := time.Now() 116 | resp, err := client.Do(request) 117 | stop := time.Now() 118 | 119 | // We should not count Redirect as errors 120 | if _, ok := err.(*RedirectNotAllowed); ok { 121 | err = nil 122 | } 123 | 124 | if err == nil { 125 | defer resp.Body.Close() 126 | } else { 127 | log.Println("Request error:", err) 128 | } 129 | 130 | if o.elasticSearch != nil { 131 | o.elasticSearch.ResponseAnalyze(request, resp, start, stop) 132 | } 133 | } 134 | 135 | func (o *HTTPOutput) String() string { 136 | return "HTTP output: " + o.address 137 | } 138 | -------------------------------------------------------------------------------- /output_http_test.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | "net/http" 6 | "sync" 7 | "testing" 8 | ) 9 | 10 | func TestHTTPOutput(t *testing.T) { 11 | startHTTP := func(addr string, cb func(*http.Request)) { 12 | handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { 13 | cb(r) 14 | }) 15 | 16 | go http.ListenAndServe(addr, handler) 17 | } 18 | 19 | wg := new(sync.WaitGroup) 20 | quit := make(chan int) 21 | 22 | input := NewTestInput() 23 | 24 | headers := HTTPHeaders{HTTPHeader{"User-Agent", "Gor"}} 25 | methods := HTTPMethods{"GET", "PUT", "POST"} 26 | output := NewHTTPOutput("127.0.0.1:50003", headers, methods, "") 27 | 28 | startHTTP("127.0.0.1:50003", func(req *http.Request) { 29 | if req.Header.Get("User-Agent") != "Gor" { 30 | t.Error("Wrong header") 31 | } 32 | 33 | if req.Method == "OPTIONS" { 34 | t.Error("Wrong method") 35 | } 36 | 37 | wg.Done() 38 | }) 39 | 40 | Plugins.Inputs = []io.Reader{input} 41 | Plugins.Outputs = []io.Writer{output} 42 | 43 | go Start(quit) 44 | 45 | for i := 0; i < 100; i++ { 46 | wg.Add(2) 47 | input.EmitPOST() 48 | input.EmitOPTIONS() 49 | input.EmitGET() 50 | } 51 | 52 | wg.Wait() 53 | 54 | close(quit) 55 | } 56 | -------------------------------------------------------------------------------- /output_tcp.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "fmt" 5 | "io" 6 | "log" 7 | "net" 8 | "strconv" 9 | "strings" 10 | ) 11 | 12 | type TCPOutput struct { 13 | address string 14 | limit int 15 | } 16 | 17 | func NewTCPOutput(options string) io.Writer { 18 | o := new(TCPOutput) 19 | 20 | optionsArr := strings.Split(options, "|") 21 | o.address = optionsArr[0] 22 | 23 | if len(optionsArr) > 1 { 24 | o.limit, _ = strconv.Atoi(optionsArr[1]) 25 | } 26 | 27 | if o.limit > 0 { 28 | return NewLimiter(o, o.limit) 29 | } else { 30 | return o 31 | } 32 | } 33 | 34 | func (o *TCPOutput) Write(data []byte) (n int, err error) { 35 | conn, err := o.connect(o.address) 36 | defer conn.Close() 37 | 38 | if err != nil { 39 | n, err = conn.Write(data) 40 | } 41 | 42 | return 43 | } 44 | 45 | func (o *TCPOutput) connect(address string) (conn net.Conn, err error) { 46 | conn, err = net.Dial("tcp", address) 47 | 48 | if err != nil { 49 | log.Println("Connection error ", err, o.address) 50 | } 51 | 52 | return 53 | } 54 | 55 | func (o *TCPOutput) String() string { 56 | return fmt.Sprintf("TCP output %s, limit: %d", o.address, o.limit) 57 | } 58 | -------------------------------------------------------------------------------- /output_tcp_test.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | "log" 6 | "net" 7 | "sync" 8 | "testing" 9 | ) 10 | 11 | func TestTCPOutput(t *testing.T) { 12 | wg := new(sync.WaitGroup) 13 | quit := make(chan int) 14 | 15 | input := NewTestInput() 16 | output := NewTCPOutput(":50002") 17 | 18 | startTCP(":50002", func(data []byte) { 19 | wg.Done() 20 | }) 21 | 22 | Plugins.Inputs = []io.Reader{input} 23 | Plugins.Outputs = []io.Writer{output} 24 | 25 | go Start(quit) 26 | 27 | for i := 0; i < 100; i++ { 28 | wg.Add(1) 29 | input.EmitGET() 30 | } 31 | 32 | wg.Wait() 33 | 34 | close(quit) 35 | } 36 | 37 | func startTCP(addr string, cb func([]byte)) { 38 | listener, err := net.Listen("tcp", addr) 39 | 40 | if err != nil { 41 | log.Fatal("Can't start:", err) 42 | } 43 | 44 | go func() { 45 | for { 46 | conn, _ := listener.Accept() 47 | 48 | var read = true 49 | var response []byte 50 | var buf []byte 51 | 52 | buf = make([]byte, 4094) 53 | 54 | for read { 55 | n, err := conn.Read(buf) 56 | 57 | switch err { 58 | case io.EOF: 59 | read = false 60 | case nil: 61 | response = append(response, buf[:n]...) 62 | if n < 4096 { 63 | read = false 64 | } 65 | default: 66 | read = false 67 | } 68 | } 69 | 70 | cb(response) 71 | 72 | conn.Close() 73 | } 74 | }() 75 | } 76 | -------------------------------------------------------------------------------- /plugins.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "io" 5 | ) 6 | 7 | type InOutPlugins struct { 8 | Inputs []io.Reader 9 | Outputs []io.Writer 10 | } 11 | 12 | var Plugins *InOutPlugins = new(InOutPlugins) 13 | 14 | func InitPlugins() { 15 | for _, options := range Settings.inputDummy { 16 | Plugins.Inputs = append(Plugins.Inputs, NewDummyInput(options)) 17 | } 18 | 19 | for _, options := range Settings.outputDummy { 20 | Plugins.Outputs = append(Plugins.Outputs, NewDummyOutput(options)) 21 | } 22 | 23 | for _, options := range Settings.inputRAW { 24 | Plugins.Inputs = append(Plugins.Inputs, NewRAWInput(options)) 25 | } 26 | 27 | for _, options := range Settings.inputTCP { 28 | Plugins.Inputs = append(Plugins.Inputs, NewTCPInput(options)) 29 | } 30 | 31 | for _, options := range Settings.outputTCP { 32 | Plugins.Outputs = append(Plugins.Outputs, NewTCPOutput(options)) 33 | } 34 | 35 | for _, options := range Settings.inputFile { 36 | Plugins.Inputs = append(Plugins.Inputs, NewFileInput(options)) 37 | } 38 | 39 | for _, options := range Settings.outputFile { 40 | Plugins.Outputs = append(Plugins.Outputs, NewFileOutput(options)) 41 | } 42 | 43 | for _, options := range Settings.outputHTTP { 44 | Plugins.Outputs = append(Plugins.Outputs, NewHTTPOutput(options, Settings.outputHTTPHeaders, Settings.outputHTTPMethods, Settings.outputHTTPElasticSearch)) 45 | } 46 | } 47 | -------------------------------------------------------------------------------- /raw_socket_listener/listener.go: -------------------------------------------------------------------------------- 1 | package raw_socket 2 | 3 | import ( 4 | "encoding/binary" 5 | "log" 6 | "net" 7 | "strconv" 8 | ) 9 | 10 | // Capture traffic from socket using RAW_SOCKET's 11 | // http://en.wikipedia.org/wiki/Raw_socket 12 | // 13 | // RAW_SOCKET allow you listen for traffic on any port (e.g. sniffing) because they operate on IP level. 14 | // Ports is TCP feature, same as flow control, reliable transmission and etc. 15 | // Since we can't use default TCP libraries RAWTCPLitener implements own TCP layer 16 | // TCP packets is parsed using tcp_packet.go, and flow control is managed by tcp_message.go 17 | type Listener struct { 18 | messages map[uint32]*TCPMessage // buffer of TCPMessages waiting to be send 19 | 20 | c_packets chan *TCPPacket 21 | c_messages chan *TCPMessage // Messages ready to be send to client 22 | 23 | c_del_message chan *TCPMessage // Used for notifications about completed or expired messages 24 | 25 | addr string // IP to listen 26 | port int // Port to listen 27 | } 28 | 29 | // RAWTCPListen creates a listener to capture traffic from RAW_SOCKET 30 | func NewListener(addr string, port string) (rawListener *Listener) { 31 | rawListener = &Listener{} 32 | 33 | rawListener.c_packets = make(chan *TCPPacket, 100) 34 | rawListener.c_messages = make(chan *TCPMessage, 100) 35 | rawListener.c_del_message = make(chan *TCPMessage, 100) 36 | rawListener.messages = make(map[uint32]*TCPMessage) 37 | 38 | rawListener.addr = addr 39 | rawListener.port, _ = strconv.Atoi(port) 40 | 41 | go rawListener.listen() 42 | go rawListener.readRAWSocket() 43 | 44 | return 45 | } 46 | 47 | func (t *Listener) listen() { 48 | for { 49 | select { 50 | // If message ready for deletion it means that its also complete or expired by timeout 51 | case message := <-t.c_del_message: 52 | t.c_messages <- message 53 | delete(t.messages, message.Ack) 54 | 55 | // We need to use channels to process each packet to avoid data races 56 | case packet := <-t.c_packets: 57 | t.processTCPPacket(packet) 58 | } 59 | } 60 | } 61 | 62 | func (t *Listener) readRAWSocket() { 63 | conn, e := net.ListenPacket("ip4:tcp", t.addr) 64 | defer conn.Close() 65 | 66 | if e != nil { 67 | log.Fatal(e) 68 | } 69 | 70 | buf := make([]byte, 4096*2) 71 | 72 | for { 73 | // Note: ReadFrom receive messages without IP header 74 | n, _, err := conn.ReadFrom(buf) 75 | 76 | if err != nil { 77 | log.Println("Error:", err) 78 | continue 79 | } 80 | 81 | if n > 0 { 82 | t.parsePacket(buf[:n]) 83 | } 84 | } 85 | } 86 | 87 | func (t *Listener) parsePacket(buf []byte) { 88 | if t.isIncomingDataPacket(buf) { 89 | new_buf := make([]byte, len(buf)) 90 | copy(new_buf, buf) 91 | 92 | t.c_packets <- ParseTCPPacket(new_buf) 93 | } 94 | } 95 | 96 | func (t *Listener) isIncomingDataPacket(buf []byte) bool { 97 | // To avoid full packet parsing every time, we manually parsing values needed for packet filtering 98 | // http://en.wikipedia.org/wiki/Transmission_Control_Protocol 99 | dest_port := binary.BigEndian.Uint16(buf[2:4]) 100 | 101 | // Because RAW_SOCKET can't be bound to port, we have to control it by ourself 102 | if int(dest_port) == t.port { 103 | // Get the 'data offset' (size of the TCP header in 32-bit words) 104 | dataOffset := (buf[12] & 0xF0) >> 4 105 | 106 | // We need only packets with data inside 107 | // Check that the buffer is larger than the size of the TCP header 108 | if len(buf) > int(dataOffset*4) { 109 | // We should create new buffer because go slices is pointers. So buffer data shoud be immutable. 110 | return true 111 | } 112 | } 113 | 114 | return false 115 | } 116 | 117 | // Trying to add packet to existing message or creating new message 118 | // 119 | // For TCP message unique id is Acknowledgment number (see tcp_packet.go) 120 | func (t *Listener) processTCPPacket(packet *TCPPacket) { 121 | var message *TCPMessage 122 | 123 | message, ok := t.messages[packet.Ack] 124 | 125 | if !ok { 126 | // We sending c_del_message channel, so message object can communicate with Listener and notify it if message completed 127 | message = NewTCPMessage(packet.Ack, t.c_del_message) 128 | t.messages[packet.Ack] = message 129 | } 130 | 131 | // Adding packet to message 132 | message.c_packets <- packet 133 | } 134 | 135 | // Receive TCP messages from the listener channel 136 | func (t *Listener) Receive() *TCPMessage { 137 | return <-t.c_messages 138 | } 139 | -------------------------------------------------------------------------------- /raw_socket_listener/tcp_message.go: -------------------------------------------------------------------------------- 1 | package raw_socket 2 | 3 | import ( 4 | "log" 5 | "sort" 6 | "time" 7 | ) 8 | 9 | const MSG_EXPIRE = 200 * time.Millisecond 10 | 11 | // TCPMessage ensure that all TCP packets for given request is received, and processed in right sequence 12 | // Its needed because all TCP message can be fragmented or re-transmitted 13 | // 14 | // Each TCP Packet have 2 ids: acknowledgment - message_id, and sequence - packet_id 15 | // Message can be compiled from unique packets with same message_id which sorted by sequence 16 | // Message is received if we didn't receive any packets for 200ms 17 | type TCPMessage struct { 18 | Ack uint32 // Message ID 19 | packets []*TCPPacket 20 | 21 | timer *time.Timer // Used for expire check 22 | 23 | c_packets chan *TCPPacket 24 | 25 | c_del_message chan *TCPMessage 26 | } 27 | 28 | // NewTCPMessage pointer created from a Acknowledgment number and a channel of messages readuy to be deleted 29 | func NewTCPMessage(Ack uint32, c_del chan *TCPMessage) (msg *TCPMessage) { 30 | msg = &TCPMessage{Ack: Ack} 31 | 32 | msg.c_packets = make(chan *TCPPacket) 33 | msg.c_del_message = c_del // used for notifying that message completed or expired 34 | 35 | // Every time we receive packet we reset this timer 36 | msg.timer = time.AfterFunc(MSG_EXPIRE, msg.Timeout) 37 | 38 | go msg.listen() 39 | 40 | return 41 | } 42 | 43 | func (t *TCPMessage) listen() { 44 | for { 45 | select { 46 | case packet, more := <-t.c_packets: 47 | if more { 48 | t.AddPacket(packet) 49 | } else { 50 | // Stop loop if channel closed 51 | return 52 | } 53 | } 54 | } 55 | } 56 | 57 | // Timeout notifies message to stop listening, close channel and message ready to be sent 58 | func (t *TCPMessage) Timeout() { 59 | close(t.c_packets) // Notify to stop listen loop and close channel 60 | t.c_del_message <- t // Notify RAWListener that message is ready to be send to replay server 61 | } 62 | 63 | // Bytes sorts packets in right orders and return message content 64 | func (t *TCPMessage) Bytes() (output []byte) { 65 | mk := make([]int, len(t.packets)) 66 | 67 | i := 0 68 | for k, _ := range t.packets { 69 | mk[i] = k 70 | i++ 71 | } 72 | 73 | sort.Ints(mk) 74 | 75 | for _, k := range mk { 76 | output = append(output, t.packets[k].Data...) 77 | } 78 | 79 | return 80 | } 81 | 82 | // AddPacket to the message and ensure packet uniqueness 83 | // TCP allows that packet can be re-send multiple times 84 | func (t *TCPMessage) AddPacket(packet *TCPPacket) { 85 | packetFound := false 86 | 87 | for _, pkt := range t.packets { 88 | if packet.Seq == pkt.Seq { 89 | packetFound = true 90 | break 91 | } 92 | } 93 | 94 | if packetFound { 95 | log.Println("Received packet with same sequence") 96 | } else { 97 | t.packets = append(t.packets, packet) 98 | } 99 | 100 | // Reset message timeout timer 101 | t.timer.Reset(MSG_EXPIRE) 102 | } 103 | -------------------------------------------------------------------------------- /raw_socket_listener/tcp_packet.go: -------------------------------------------------------------------------------- 1 | package raw_socket 2 | 3 | import ( 4 | "encoding/binary" 5 | "strconv" 6 | "strings" 7 | ) 8 | 9 | // TCP Flags 10 | const ( 11 | TCP_FIN = 1 << iota 12 | TCP_SYN 13 | TCP_RST 14 | TCP_PSH 15 | TCP_ACK 16 | TCP_URG 17 | TCP_ECE 18 | TCP_CWR 19 | TCP_NS 20 | ) 21 | 22 | // Simple TCP packet parser 23 | // 24 | // Packet structure: http://en.wikipedia.org/wiki/Transmission_Control_Protocol 25 | type TCPPacket struct { 26 | SrcPort uint16 27 | DestPort uint16 28 | Seq uint32 29 | Ack uint32 30 | DataOffset uint8 31 | Flags uint16 32 | Window uint16 33 | Checksum uint16 34 | Urgent uint16 35 | 36 | Data []byte 37 | } 38 | 39 | func ParseTCPPacket(b []byte) (p *TCPPacket) { 40 | p = &TCPPacket{Data: b} 41 | p.ParseBasic() 42 | 43 | return p 44 | } 45 | 46 | // Parse TCP Packet, inspired by: https://github.com/miekg/pcap/blob/master/packet.go 47 | func (t *TCPPacket) Parse() { 48 | t.ParseBasic() 49 | t.SrcPort = binary.BigEndian.Uint16(t.Data[0:2]) 50 | t.DestPort = binary.BigEndian.Uint16(t.Data[2:4]) 51 | t.Flags = binary.BigEndian.Uint16(t.Data[12:14]) & 0x1FF 52 | t.Window = binary.BigEndian.Uint16(t.Data[14:16]) 53 | t.Checksum = binary.BigEndian.Uint16(t.Data[16:18]) 54 | t.Urgent = binary.BigEndian.Uint16(t.Data[18:20]) 55 | } 56 | 57 | // ParseBasic set of fields 58 | func (t *TCPPacket) ParseBasic() { 59 | t.Seq = binary.BigEndian.Uint32(t.Data[4:8]) 60 | t.Ack = binary.BigEndian.Uint32(t.Data[8:12]) 61 | t.DataOffset = (t.Data[12] & 0xF0) >> 4 62 | 63 | t.Data = t.Data[t.DataOffset*4:] 64 | } 65 | 66 | // String output for a TCP Packet 67 | func (t *TCPPacket) String() string { 68 | return strings.Join([]string{ 69 | "Source port: " + strconv.Itoa(int(t.SrcPort)), 70 | "Dest port:" + strconv.Itoa(int(t.DestPort)), 71 | "Sequence:" + strconv.Itoa(int(t.Seq)), 72 | "Acknowledgment:" + strconv.Itoa(int(t.Ack)), 73 | "Header len:" + strconv.Itoa(int(t.DataOffset)), 74 | 75 | "Flag ns:" + strconv.FormatBool(t.Flags&TCP_NS != 0), 76 | "Flag crw:" + strconv.FormatBool(t.Flags&TCP_CWR != 0), 77 | "Flag ece:" + strconv.FormatBool(t.Flags&TCP_ECE != 0), 78 | "Flag urg:" + strconv.FormatBool(t.Flags&TCP_URG != 0), 79 | "Flag ack:" + strconv.FormatBool(t.Flags&TCP_ACK != 0), 80 | "Flag psh:" + strconv.FormatBool(t.Flags&TCP_PSH != 0), 81 | "Flag rst:" + strconv.FormatBool(t.Flags&TCP_RST != 0), 82 | "Flag syn:" + strconv.FormatBool(t.Flags&TCP_SYN != 0), 83 | "Flag fin:" + strconv.FormatBool(t.Flags&TCP_FIN != 0), 84 | 85 | "Window size:" + strconv.Itoa(int(t.Window)), 86 | "Checksum:" + strconv.Itoa(int(t.Checksum)), 87 | 88 | "Data:" + string(t.Data), 89 | }, "\n") 90 | } 91 | -------------------------------------------------------------------------------- /settings.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "log" 7 | "os" 8 | ) 9 | 10 | const ( 11 | VERSION = "0.7.0" 12 | ) 13 | 14 | type AppSettings struct { 15 | verbose bool 16 | 17 | splitOutput bool 18 | 19 | inputDummy MultiOption 20 | outputDummy MultiOption 21 | 22 | inputTCP MultiOption 23 | outputTCP MultiOption 24 | 25 | inputFile MultiOption 26 | outputFile MultiOption 27 | 28 | inputRAW MultiOption 29 | 30 | outputHTTP MultiOption 31 | outputHTTPHeaders HTTPHeaders 32 | outputHTTPMethods HTTPMethods 33 | outputHTTPElasticSearch string 34 | } 35 | 36 | var Settings AppSettings = AppSettings{} 37 | 38 | func usage() { 39 | fmt.Printf("Gor is a simple http traffic replication tool written in Go. Its main goal is to replay traffic from production servers to staging and dev environments.\nProject page: https://github.com/buger/gor\nAuthor: leonsbox@gmail.com\nCurrent Version: %s\n\n", VERSION) 40 | flag.PrintDefaults() 41 | os.Exit(2) 42 | } 43 | 44 | func init() { 45 | flag.Usage = usage 46 | 47 | flag.BoolVar(&Settings.verbose, "verbose", false, "Turn on verbose/debug output") 48 | 49 | flag.BoolVar(&Settings.splitOutput, "split-output", false, "By default each output gets same traffic. If set to `true` it splits traffic equally among all outputs.") 50 | 51 | flag.Var(&Settings.inputDummy, "input-dummy", "Used for testing outputs. Emits 'Get /' request every 1s") 52 | flag.Var(&Settings.outputDummy, "output-dummy", "Used for testing inputs. Just prints data coming from inputs.") 53 | 54 | flag.Var(&Settings.inputTCP, "input-tcp", "Used for internal communication between Gor instances. Example: \n\t# Receive requests from other Gor instances on 28020 port, and redirect output to staging\n\tgor --input-tcp :28020 --output-http staging.com") 55 | flag.Var(&Settings.outputTCP, "output-tcp", "Used for internal communication between Gor instances. Example: \n\t# Listen for requests on 80 port and forward them to other Gor instance on 28020 port\n\tgor --input-raw :80 --output-tcp replay.local:28020") 56 | 57 | flag.Var(&Settings.inputFile, "input-file", "Read requests from file: \n\tgor --input-file ./requests.gor --output-http staging.com") 58 | flag.Var(&Settings.outputFile, "output-file", "Write incoming requests to file: \n\tgor --input-raw :80 --output-file ./requests.gor") 59 | 60 | flag.Var(&Settings.inputRAW, "input-raw", "Capture traffic from given port (use RAW sockets and require *sudo* access):\n\t# Capture traffic from 8080 port\n\tgor --input-raw :8080 --output-http staging.com") 61 | 62 | flag.Var(&Settings.outputHTTP, "output-http", "Forwards incoming requests to given http address.\n\t# Redirect all incoming requests to staging.com address \n\tgor --input-raw :80 --output-http http://staging.com") 63 | flag.Var(&Settings.outputHTTPHeaders, "output-http-header", "Inject additional headers to http reqest:\n\tgor --input-raw :8080 --output-http staging.com --output-http-header 'User-Agent: Gor'") 64 | flag.Var(&Settings.outputHTTPMethods, "output-http-method", "Whitelist of HTTP methods to replay. Anything else will be dropped:\n\tgor --input-raw :8080 --output-http staging.com --output-http-method GET --output-http-method OPTIONS") 65 | flag.StringVar(&Settings.outputHTTPElasticSearch, "output-http-elasticsearch", "", "Send request and response stats to ElasticSearch:\n\tgor --input-raw :8080 --output-http staging.com --output-http-elasticsearch 'es_host:api_port/index_name'") 66 | } 67 | 68 | func Debug(args ...interface{}) { 69 | if Settings.verbose { 70 | log.Println(args...) 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /settings_headers.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "strings" 7 | ) 8 | 9 | type HTTPHeaders []HTTPHeader 10 | type HTTPHeader struct { 11 | Name string 12 | Value string 13 | } 14 | 15 | func (h *HTTPHeaders) String() string { 16 | return fmt.Sprint(*h) 17 | } 18 | 19 | func (h *HTTPHeaders) Set(value string) error { 20 | v := strings.SplitN(value, ":", 2) 21 | if len(v) != 2 { 22 | return errors.New("Expected `Key: Value`") 23 | } 24 | 25 | header := HTTPHeader{ 26 | strings.TrimSpace(v[0]), 27 | strings.TrimSpace(v[1]), 28 | } 29 | 30 | *h = append(*h, header) 31 | return nil 32 | } 33 | -------------------------------------------------------------------------------- /settings_methods.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "fmt" 5 | "strings" 6 | ) 7 | 8 | type HTTPMethods []string 9 | 10 | func (h *HTTPMethods) String() string { 11 | return fmt.Sprint(*h) 12 | } 13 | 14 | func (h *HTTPMethods) Set(value string) error { 15 | *h = append(*h, strings.ToUpper(value)) 16 | return nil 17 | } 18 | 19 | func (h *HTTPMethods) Contains(value string) bool { 20 | for _, method := range *h { 21 | if value == method { 22 | return true 23 | } 24 | } 25 | return false 26 | } 27 | -------------------------------------------------------------------------------- /settings_methods_test.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | func TestHTTPMethods(t *testing.T) { 8 | methods := HTTPMethods{} 9 | 10 | methods.Set("lower") 11 | methods.Set("UPPER") 12 | 13 | if !methods.Contains("LOWER") { 14 | t.Error("Does not contain LOWER") 15 | } 16 | 17 | if !methods.Contains("UPPER") { 18 | t.Error("Does not contain UPPER") 19 | } 20 | 21 | if methods.Contains("ABSENT") { 22 | t.Error("Does contain ABSENT") 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /settings_option.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | import ( 4 | "fmt" 5 | ) 6 | 7 | type MultiOption []string 8 | 9 | func (h *MultiOption) String() string { 10 | return fmt.Sprint(*h) 11 | } 12 | 13 | func (h *MultiOption) Set(value string) error { 14 | *h = append(*h, value) 15 | return nil 16 | } 17 | -------------------------------------------------------------------------------- /test_input.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | type TestInput struct { 4 | data chan []byte 5 | } 6 | 7 | func NewTestInput() (i *TestInput) { 8 | i = new(TestInput) 9 | i.data = make(chan []byte) 10 | 11 | return 12 | } 13 | 14 | func (i *TestInput) Read(data []byte) (int, error) { 15 | buf := <-i.data 16 | copy(data, buf) 17 | 18 | return len(buf), nil 19 | } 20 | 21 | func (i *TestInput) EmitGET() { 22 | i.data <- []byte("GET / HTTP/1.1\r\n\r\n") 23 | } 24 | 25 | func (i *TestInput) EmitPOST() { 26 | i.data <- []byte("POST /pub/WWW/ HTTP/1.1\nHost: www.w3.org\r\n\r\na=1&b=2\r\n\r\n") 27 | } 28 | 29 | func (i *TestInput) EmitOPTIONS() { 30 | i.data <- []byte("OPTIONS / HTTP/1.1\nHost: www.w3.org\r\n\r\n") 31 | } 32 | 33 | func (i *TestInput) String() string { 34 | return "Test Input" 35 | } 36 | -------------------------------------------------------------------------------- /test_output.go: -------------------------------------------------------------------------------- 1 | package gor 2 | 3 | type writeCallback func(data []byte) 4 | 5 | type TestOutput struct { 6 | cb writeCallback 7 | } 8 | 9 | func NewTestOutput(cb writeCallback) (i *TestOutput) { 10 | i = new(TestOutput) 11 | i.cb = cb 12 | 13 | return 14 | } 15 | 16 | func (i *TestOutput) Write(data []byte) (int, error) { 17 | i.cb(data) 18 | 19 | return len(data), nil 20 | } 21 | 22 | func (i *TestOutput) String() string { 23 | return "Test Input" 24 | } 25 | --------------------------------------------------------------------------------