└── README.md /README.md: -------------------------------------------------------------------------------- 1 | httpipe is a format that allows communicating and storing collections 2 | of HTTP requests and responses. It is line based and uses JSON, so that 3 | it works well with UNIX tools and is easy to implement with common 4 | libraries. 5 | 6 | # Basics 7 | A httpipe communication is a unidirectional stream of JSON objects, 8 | one per line. Each JSON object contains the information about one 9 | HTTP request and/or response. Here is an example for a stream of two 10 | requests: 11 | 12 | ``` 13 | {"host":"test.net","port":80,"tls":false,"req":"GET /foo HTTP/1.1\r\nHost: test.net:80\r\n\r\n"} 14 | {"host":"test.net","port":80,"tls":false,"req":"GET /bar HTTP/1.1\r\nHost: test.net:80\r\n\r\n"} 15 | ``` 16 | 17 | Here is a single line containing a request and response: 18 | 19 | ``` 20 | {"host":"test.net","port":443,"tls":true,"req":"GET /src HTTP/1.1\r\nHost: test.net:443\r\n\r\n","resp":"HTTP/1.1 200 OK\r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: 59\r\n\r\n\u003cpre\u003e\n\u003ca href=\"/\"\u003eNothing to see here, go back.\u003c/a\u003e\n\u003c/pre\u003e\n"} 21 | ``` 22 | 23 | # JSON fields 24 | No field in the JSON is required by this specification. However, 25 | different tools will require different fields. It is recommended to 26 | include as many fields as possible when generating data and require only 27 | the fields that are absolutely necessary when consuming data. 28 | 29 | | Field | Type | Description | 30 | | ----- | ---- | ----------- | 31 | | `host` | string | The host that a request should be sent to or a response was received from. | 32 | | `port` | int | The port that is used with `host`. Recommended defaults are 443 if `tls` is `true` and 80 if it is `false`. | 33 | | `tls` | bool | Whether TLS (HTTPS) is used or not (HTTP). Recommended default is `true`. | 34 | | `req` | string | The full HTTP request, including HTTP method, header fields and, if applicable, payload. | 35 | | `reqat` | string | The timestamp when the request was sent, in UTC, in the form `yyyy-mm-ddThh:mm:ssZ`. | 36 | | `ping` | int | The delay between sending the last byte of the request and receiving the first byte of the response in milliseconds. | 37 | | `resp` | string | The full HTTP response, including HTTP status line, response header fields and payload (if applicable). | 38 | | `err` | string | A textual description of an error that occurred during a request. | 39 | | `errno` | int | An error number indicating the type of error that occurred during a request. | 40 | 41 | ## Error numbers 42 | The `errno` value indicates problems that lead to missing HTTP 43 | responses. Each error number has two digits, where the first digit 44 | specifies the category of the error and the second digit further 45 | characterizes the error. 46 | 47 | ### 1x: DNS errors 48 | 1x errors occur when a hostname could not be resolved. They will never 49 | occur if IPs are used instead of hostnames. The following error numbers 50 | are defined: 51 | 52 | - 10: Hostname could not be resolved. 53 | - 11: Hostname resolution timed out. 54 | 55 | ### 2x: TLS errors 56 | 2x errors occur when there are problems with TLS. They will only occur 57 | with HTTPS requests and never with HTTP. The following error numbers are 58 | defined: 59 | 60 | - 20: Server uses invalid certificate. 61 | - 21: Server uses untrusted certificate authority. 62 | 63 | ### 3x: Connection errors 64 | 3x errors occur when the server did not respond to a request. The 65 | following error numbers are defined: 66 | 67 | - 30: Server refused connection. 68 | - 31: Connection timed out. 69 | 70 | # Tools 71 | Here are some tools—some written with httpipe in mind, some not—that can 72 | work with httpipe streams: 73 | - [pfuzz](https://github.com/codesoap/pfuzz): Create requests 74 | - [preq](https://github.com/codesoap/preq): Fetch responses for requests 75 | - [drip](https://github.com/codesoap/drip): Can be used for rate limiting with preq 76 | - [jq](https://jqlang.github.io/jq/): A powerful JSON tool for manipulations, filtering and pretty printing 77 | - [hpstat](https://github.com/codesoap/hpstat): Filter responses by HTTP status codes 78 | - [hp2url](https://github.com/codesoap/hp2url): Print URLs for httpipe input 79 | 80 | ## Examples 81 | Here are some examples of the mentioned tools and some more UNIX tools 82 | in action: 83 | 84 | ```bash 85 | # Make a request to Google: 86 | pfuzz -u 'https://google.com' | preq 87 | 88 | # Just print the status line of the response: 89 | pfuzz -u 'https://google.com' | preq | jq -r .resp | head -n1 90 | 91 | # Try some different subdomains, but wait 500ms between requests: 92 | echo "www\nmail\nfoo" | drip 500ms | pfuzz -w - -u 'https://FUZZ.test.com' | preq 93 | 94 | # Generate requests for some random 10 paths: 95 | sort --random-sort /usr/share/dict/words | head -n 10 | pfuzz -w - -u 'https://test.net/FUZZ' 96 | 97 | # Store results in a file for later analysis: 98 | pfuzz -w /usr/share/dict/words -u 'https://FUZZ.test.net' | drip | preq > results 99 | 100 | # Also show the progress (as the request count): 101 | pfuzz -w /usr/share/dict/words -u 'https://FUZZ.test.net' | drip | preq | pv -l > results 102 | 103 | # Show the response of the first result: 104 | head -n1 results | jq -r .resp | less 105 | 106 | # Show all 5xx responses: 107 | cat results | hpstat 500:599 | jq . 108 | 109 | # Filter out lines for a specific host; note that jq generates valid 110 | # httpipe output with the -c flag: 111 | jq -c 'select(.host == "mail.test.net")' results 112 | 113 | # Execute the requests from the file 'reqs', but on a different port: 114 | jq -c '.port = 8080' reqs | drip | preq 115 | ``` 116 | --------------------------------------------------------------------------------