├── .gitignore ├── README.md ├── LICENSE └── checkprovider /.gitignore: -------------------------------------------------------------------------------- 1 | /blib/ 2 | /.build/ 3 | _build/ 4 | cover_db/ 5 | inc/ 6 | Build 7 | !Build/ 8 | Build.bat 9 | .last_cover_stats 10 | /Makefile 11 | /Makefile.old 12 | /MANIFEST.bak 13 | /META.yml 14 | /META.json 15 | /MYMETA.* 16 | nytprof.out 17 | /pm_to_blib 18 | *.o 19 | *.bs 20 | /_eumm/ 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # checkprovider 2 | checkprovider - Generate timing information for HTTP requests. 3 | 4 | # Description 5 | checkprovider returns a breakdown of timing information for requests to 6 | a backend By default, the script runs 10 7 | queries at a rate of one per second, and outputs timing information for each 8 | request to stdout. 9 | 10 | The timing results are presented in tabulated form, with the following column 11 | headings: (all times are in milliseconds) 12 | 13 | ## Timestamp 14 | 15 | The timestamp of when the request started 16 | 17 | ## DNS 18 | 19 | The time taken to perform the DNS lookup for the server. 20 | 21 | ## Socket 22 | 23 | The time required to create a socket for this request. 24 | 25 | ## Connect 26 | 27 | The time taken to connect to the TCP endpoint. 28 | 29 | ## TLS 30 | 31 | The time taken to complete the TLS handshake. 32 | 33 | ## Write 34 | 35 | The time spent writing the HTTP request to the socket. 36 | 37 | ## 1st Byte 38 | 39 | This is the time spent waiting for the first byte of the response. 40 | 41 | ## Read 42 | 43 | The total time spent reading the response from the HTTP server. 44 | 45 | ## Close 46 | 47 | The time to close the socket after reading the full response. 48 | 49 | ## Total 50 | 51 | The total time elapsed during the request. 52 | 53 | ## Bytes 54 | 55 | The number of bytes read in the response (including response headers). 56 | 57 | ## Status 58 | 59 | This column shows either `OK` or `NOT OK` to indicate whether the request 60 | was successful (i.e. returned an HTTP 200 response). If any other response 61 | was obtained, or the initial connection was unsuccessful then all fields from 62 | `Connect` through to `Close` will be empty. 63 | 64 | checkprovider will die with an error if the initial DNS lookup or socket creation fail. 65 | 66 | ## Term 67 | 68 | This column shows the query term used for the request. 69 | 70 | ## Example 71 | 72 | Here is an example of the output: 73 | 74 | 
 75 | $ checkprovider https://www.yahoo.com/
 76 |                                                       Request Times (ms)
 77 |        Timestamp             DNS   Socket  Connect      TLS    Write  1stByte     Read    Close    Total   Bytes Status
 78 | 2016-01-30 06:54:53 UTC     1.02     0.10     1.54    19.38     0.10    67.87   227.48     0.32   318.14  432019 OK (200)
 79 | 2016-01-30 06:54:54 UTC     1.15     0.13     1.31    18.40     0.10    70.84   403.08     0.24   495.38  439123 OK (200)
 80 | 2016-01-30 06:54:55 UTC     1.07     0.10     1.47    18.15     0.09    66.33   234.07     0.22   321.65  437772 OK (200)
 81 | 2016-01-30 06:54:56 UTC     0.98     0.09     1.39    19.93     0.09    65.90   258.00     0.34   346.88  433950 OK (200)
 82 | 2016-01-30 06:54:57 UTC     0.99     0.10     1.32    17.78     0.09   104.22   268.77     0.26   393.73  433018 OK (200)
 83 | 2016-01-30 06:54:58 UTC     1.05     0.10     1.42    18.88     0.06    78.30   279.99     0.22   380.18  433947 OK (200)
 84 | 2016-01-30 06:54:59 UTC     1.01     0.09     1.42    17.44     0.10    63.76   241.61     0.30   325.90  441893 OK (200)
 85 | 2016-01-30 06:55:00 UTC     1.27     0.12     1.38    19.11     0.07    74.45   279.24     0.51   376.29  439629 OK (200)
 86 | 2016-01-30 06:55:01 UTC     8.97     0.00     1.69    12.00     0.00    85.00   213.00     6.00   326.96  439342 OK (200)
 87 | 2016-01-30 06:55:02 UTC     1.00     0.00     2.00    23.01     0.00    68.99   264.00     1.00   360.00  440608 OK (200)
 88 |
89 | 90 | # Dependencies 91 | 92 | The current version is written in Perl and depends on the following non-core modules: 93 | * IO::Socket::SSL 94 | 95 | The following core modules are also required: 96 | * File::Basename 97 | * Getopt::Long 98 | * Pod::Usage 99 | * POSIX 100 | * Socket 101 | * Time::HiRes 102 | 103 | The folloring modules are optional: 104 | * URI::Encode (Note: only used for `-T` option) 105 | 106 | # Installation 107 | 108 | For now this code is not packaged (see also TODO). To use you install the dependencies and then fetch this repo. For example on redhat-based systems: 109 | 110 | $ sudo yum install perl-IO-Socket-SSL perl-URI-Encode 111 | $ git clone -o upstream git@github.com:yahoo/checkprovider.git 112 | $ export PATH=$PATH:`pwd`/checkprovider 113 | 114 | # Options 115 | 116 | The following command line options are supported by checkprovider: 117 | 118 |
119 |
--no-blank
120 |
Suppress the blank line at the end of the list of timing results.
121 | 122 |
-d|--debug
123 |
Print some debug information to stdout.
124 | 125 |
-F feed|--feed feed
126 |
Set the "feed" part of the url, eg -f /forecastxml
127 | 128 |
-f frequency|--frequency frequency
129 |
Set the frequency of the requests in seconds. This is the inverse of the query rate, and takes precendence over any rate specified with `--rate`.
130 | 131 |
-h|--no-header
132 |
Suppress the display of header at the top of the list of timing results.
133 | 134 |
-H header|--header header
135 |
Specify an additional HTTP request header. Note: this option can be specified multiple times to provide multiple request headers.
136 | 137 |
-?|--help
138 |
Display full help information.
139 | 140 |
-p period|--period period
141 |
This specifies the period, in whole seconds, over which the requests are to be made. The default is 10 seconds.
142 | 143 |
-r rate|--rate rate
144 |
This specifies the integer rate of requests per second that should be made by checkprovider. The rate and period determine the number of requests sent to the HTTP server. The default rate is 1 request per second.
145 | 146 |
-Q|--query string
147 |
The query part of the URL, eg. `-q "q=london"`.
148 | 149 |
-q|--quiet
150 |
Suppress the blank line at the end of the results as well as the header. This has the same effect as using `--no-header` and `--no-blank`.
151 | 152 |
-T|--termsfile filename
153 |
A file containing a list of terms to be used in place of the `[TERM]` string in the URL. The format of the file is one term per line.
154 | 155 |
-v|--version
156 |
Display the version information then quit.
157 | 158 |
-w host|--host host
159 |
Specify the HTTP Host header.
160 | 161 |
-U useragent|--useragent useragent
162 |
Specify the User-Agent HTTP header.
163 |
164 | 165 | # TODO 166 | 167 | This is a work in progress and we welcome pull requests. Here are a few things that still need work: 168 | 169 | * Packaging (rpm/deb/other?) 170 | * Protocol support (HTTP/1.0 and HTTP/2) 171 | * pretty graph generation 172 | 173 | # License 174 | 175 | Code licensed under the Apache License Version 2.0. See LICENSE file for terms. 176 | 177 | # Contributors 178 | 179 | * Richard Misenheimer 180 | * Ian Brayshaw 181 | * Leong-Kui Lim 182 | * Peter Ellehauge 183 | * Scott Beardsley 184 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | -------------------------------------------------------------------------------- /checkprovider: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl 2 | # 3 | # Copyright 2016, Yahoo Inc. 4 | # Copyrights licensed under the Apache 2.0 License. 5 | # See the accompanying LICENSE file for terms. 6 | # 7 | use strict; 8 | use warnings; 9 | 10 | use POSIX qw( :sys_wait_h strftime BUFSIZ ); 11 | use Socket; 12 | use IO::Socket::SSL; 13 | use Pod::Usage; 14 | use Time::HiRes qw( time sleep ); 15 | use Getopt::Long; 16 | use File::Basename; 17 | 18 | use constant VERSION => '1.1'; 19 | 20 | # 21 | # configuration 22 | # 23 | 24 | # what's the script name? 25 | use constant NAME => basename $0; 26 | 27 | # define the configuration for this script 28 | # - what are the details of the test request? 29 | use constant PORT => 80; 30 | use constant TLSPORT => 443; 31 | 32 | # - what is the rate (per second) of requests, and how long should 33 | # the test run for? 34 | use constant RATE => 1; # 1 request per second 35 | use constant PERIOD => 10; # 10 second test duration 36 | 37 | my %UAs = ( 38 | 'default' => NAME . "/" . VERSION, 39 | 'smartphone' => 40 | 'Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25', 41 | 'tablet' => 42 | 'Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10', 43 | 'desktop' => 44 | 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36', 45 | 'ie6' => 46 | 'Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)', 47 | 'ie7' => 'Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)', 48 | 'ie8' => 49 | 'Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB7.4; InfoPath.2; SV1; .NET CLR 3.3.69573; WOW64; en-US)', 50 | 'ie9' => 'Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US)', 51 | 'ie10' => 52 | 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 7.0; InfoPath.3; .NET CLR 3.1.40767; Trident/6.0; en-IN)', 53 | 'ie11' => 54 | 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; AS; rv:11.0) like Gecko', 55 | 'chrome' => 56 | 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36', 57 | 'firefox' => 58 | 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1', 59 | 'ios6' => 60 | 'Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25', 61 | 'ios7' => 62 | 'Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/7.0 Mobile/10A5376e Safari/8536.25', 63 | 'ios8' => 64 | 'Mozilla/5.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/8.0 Mobile/10A5376e Safari/8536.25', 65 | 'android' => 66 | 'Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19' 67 | ); 68 | 69 | my $UA = "default"; 70 | 71 | # - define the EOL sequence (CRLF) for HTTP 72 | use constant EOL => "\015\012"; 73 | 74 | # - what's the format for displaying time? 75 | use constant TIMEFMT => "%Y-%m-%d %T %Z"; 76 | 77 | # 78 | # subroutine prototypes 79 | # 80 | 81 | sub time_request ($); # time a single request 82 | sub show_header (); # display the output header 83 | sub show_response ($); # display the statistics of a single request 84 | 85 | # 86 | # process the command line options 87 | # 88 | my $version = undef; # display the script version? 89 | my $help = undef; # display the script help? 90 | my $no_header = undef; # suppress the header? 91 | my $no_blank = undef; # suppress the trailing blank line? 92 | my $output_format = "def"; # 93 | my $quiet = undef; # suppress the header and the trailing blank line? 94 | my $rate = RATE; # the rate of requests 95 | my $period = PERIOD; # the period of the requests 96 | my $feed = "/"; # the feed 97 | my $query = ""; # the query part 98 | my $termsfile = undef; # the query term list in a file 99 | my $port = PORT; # the port number for the above hostname 100 | my $frequency = undef; # the frequency of the queries 101 | my $debug = 0; # no debugging 102 | my $url = undef; # 103 | my $inputurl = undef; # 104 | my $usernotice = undef; 105 | my $httphost = undef; # the HTTP header Host: 106 | my $hostip = undef; # ip address 107 | my @headers = (); # additional request headers 108 | my @terms = (); # list of terms to replace in the query string 109 | my $position = 0; # used to keep track of which term is being used 110 | 111 | pod2usage( { -verbose => 2 } ) if ( $#ARGV < 0 ); 112 | 113 | Getopt::Long::Configure('bundling'); 114 | GetOptions( 115 | 'version|v' => \$version, 116 | 'no-header|h' => \$no_header, 117 | 'no-blank|b' => \$no_blank, 118 | 'format|O=s' => \$output_format, 119 | 'quiet|q' => \$quiet, 120 | 'rate|r=f' => \$rate, 121 | 'frequency|f=f' => \$frequency, 122 | 'period|p=i' => \$period, 123 | 'feed|F=s' => \$feed, 124 | 'query|Q=s' => \$query, 125 | 'termsfile|T=s' => \$termsfile, 126 | 'host|w=s' => \$httphost, 127 | 'wip=s' => \$hostip, 128 | 'port=s' => \$port, 129 | 'debug=i' => \$debug, 130 | 'd+' => \$debug, 131 | 'url|u=s' => \$inputurl, 132 | 'useragent|U=s' => \$UA, 133 | 'help|?' => \$help, 134 | 'header|H=s' => \@headers 135 | ) or pod2usage; 136 | 137 | if ( $#ARGV >= 0 ) { 138 | $inputurl = $ARGV[0]; 139 | } 140 | 141 | if ( defined $inputurl ) { 142 | $url = $inputurl; 143 | $url =~ m/^(http[s])*:\/\//; # match http[s]:// 144 | if ( $1 && $1 eq 'https' ) { 145 | $port = TLSPORT; 146 | } 147 | $url =~ s/^http[s]*:\/\///; # strip http[s]:// 148 | if ( $url =~ /([^\/]*)(\/[^\?]*)\?(.*)/ ) { 149 | $hostip = $1; 150 | $feed = $2; 151 | $query = $3; 152 | } 153 | elsif ( $url =~ /([^\/]*)(\/[^\?]*)/ ) { 154 | $hostip = $1; 155 | $feed = $2; 156 | } 157 | else { 158 | $hostip = $url; 159 | } 160 | } 161 | 162 | # read query terms from a file store in @terms array 163 | if ( defined $termsfile ) { 164 | if ( -f $termsfile ) { 165 | open my $fh, '<:encoding(UTF-8)', $termsfile or die "cannot open file: $termsfile"; 166 | chomp(@terms = <$fh>); 167 | close $fh; 168 | } else { 169 | print "FATAL: cannot find file: $termsfile\n"; 170 | exit(1); 171 | } 172 | } 173 | 174 | if ( $hostip =~ m/([^:]*):(\d*)/ ) { 175 | $hostip = $1; 176 | $port = $2; 177 | } 178 | 179 | # should we display the help information? 180 | pod2usage( { -verbose => 2 } ) if ($help); 181 | 182 | # should we report the script version? 183 | print NAME . ' v' . VERSION . "\n" 184 | and exit 0 185 | if ($version); 186 | 187 | # 188 | # run the timing tests 189 | # 190 | 191 | # should we show the header? 192 | show_header unless ( $no_header || $quiet ); 193 | 194 | # how many tests should we run? 195 | my $total = 196 | ( !$period ) ? 0 197 | : ( defined $frequency ) ? $period / $frequency 198 | : $period * $rate; 199 | for ( my $test = 0 ; !$period || $test < $total ; $total && $test++ ) { 200 | 201 | # fork a child process 202 | my $pid = fork; 203 | 204 | # if we are the parent process then we should commence a non-blocking wait 205 | # for child processes 206 | if ($pid) { 207 | 208 | # advance the term list position for the next child 209 | if ( ($position + 1) > $#terms ) { 210 | $position = 0; 211 | } else { 212 | $position = $position + 1; 213 | } 214 | 215 | # sleep to set the required request rate 216 | ( defined $frequency ) and sleep $frequency 217 | or sleep 1. / $rate; 218 | 219 | # don't actually need to do anything other than catch the CHLD signals 220 | 1 while ( ( my $child = waitpid( -1 => WNOHANG ) ) > 0 ); 221 | 222 | # otherwise, if we are the child process, then perform the test 223 | } 224 | elsif ( defined $pid ) { 225 | show_response 226 | time_request $hostip; # run a request and display the results 227 | exit 0; # that's all folks 228 | 229 | # else, there's a problem, so give up 230 | } 231 | else { 232 | die NAME . ": could not fork(): $!\n"; 233 | } 234 | } 235 | 236 | # wait for child processes to terminate 237 | 1 while ( ( my $child = waitpid( -1 => 0 ) ) > 0 ); 238 | 239 | # give a blank line after this set of results (if required) 240 | print "\n" unless ( $no_blank || $quiet ); 241 | 242 | # 243 | # that's all folks 244 | # 245 | 246 | exit 0; 247 | 248 | # subroutines only beyond this point 249 | ############################################################################ 250 | 251 | # time_request( ) 252 | # 253 | # Return a hash of component times for a single request of the host. 254 | sub time_request($) { 255 | my $hostip = shift; # what IP should we query? 256 | my %stats = (); # the collected timing statistics 257 | my $tls = 0; # secure or insecure? 258 | 259 | local $!; 260 | 261 | # perform initialisation before we start timing 262 | my $bytes; 263 | undef $bytes; # bytes read 264 | my $error; 265 | undef $error; # has an error occurred? 266 | my $socket; 267 | undef $socket; # socket handle 268 | my $content; 269 | undef $content; # the response content 270 | my @request; 271 | undef @request; # the lines of the request 272 | if ( $query eq "" ) { 273 | push @request, sprintf( "GET %s HTTP/1.1", $feed ); 274 | } 275 | else { 276 | # replace query term and save off term in stats hash 277 | if ( defined $termsfile && $query =~ m/(\[TERM\])/ ) { 278 | my $term = $terms[$position]; 279 | # try to encode the term if URI::Encode is available 280 | if ( eval "use URI::Encode; 1" ) { 281 | $term = (URI::Encode->new( { encode_reserved => 1 } ))->encode($term); 282 | } 283 | $query =~ s/\[TERM\]/$term/g; 284 | $stats{term} = $term; 285 | } 286 | push @request, sprintf( "GET %s?%s HTTP/1.1", $feed, $query ); 287 | } 288 | push @request, 289 | sprintf( "Host: %s", 290 | defined($httphost) 291 | ? $httphost 292 | : ( $port ne PORT ? "$hostip:$port" : $hostip ) ); 293 | $tls = ( $port eq TLSPORT ); 294 | if ( $UAs{$UA} ) { 295 | push @request, sprintf("User-Agent: $UAs{$UA}"); 296 | } 297 | else { 298 | push @request, sprintf("User-Agent: $UA"); 299 | } 300 | push @request, sprintf("Connection: close"); 301 | push @request, @headers; 302 | my $request = join EOL, @request, '', ''; 303 | if ($debug) { 304 | print "$request"; 305 | print "doing TLS? $tls\n"; 306 | } 307 | my $proto = getprotobyname('tcp'); 308 | 309 | # set up a routine for handling the termination of time_request() 310 | my $done = sub { 311 | 312 | # set the total response time (if it's not already set) 313 | $stats{total} ||= time - $stats{time}; 314 | 315 | # set the error string (if there is one) 316 | $stats{error} ||= $_[0] || ''; 317 | $stats{ip} ||= $_[1] || ''; 318 | 319 | # return the response statistics 320 | return \%stats; 321 | }; # $done 322 | 323 | # record the time the test was run 324 | my $start = $stats{time} = time; 325 | 326 | # perform DNS lookup 327 | my $ip; 328 | 329 | $ip = inet_aton($hostip) 330 | or return $done->( "DNS lookup failed for " . $hostip ); 331 | my $addr = sockaddr_in( $port, $ip ); 332 | $stats{dns} = time - $start; 333 | $ip = inet_ntoa($ip); 334 | 335 | # create the socket 336 | $start = time; 337 | socket( $socket, PF_INET, SOCK_STREAM, $proto ) 338 | or return $done->( "socket creation failed: $!", $ip ); 339 | $stats{socket} = time - $start; 340 | 341 | # TCP connect 342 | $start = time; 343 | connect( $socket => $addr ) 344 | or return $done->( "connection failed: " . $!, $ip ); 345 | $stats{connect} = time - $start; 346 | 347 | # turn on unbuffered I/O for this socket 348 | select( ( select($socket), $| = 1 )[0] ); 349 | 350 | # upgrade to TLS 351 | if ($tls) { 352 | $start = time; 353 | IO::Socket::SSL->start_SSL( 354 | $socket, 355 | { 356 | SSL_hostname => $httphost, 357 | SSL_ca_path => '/etc/ssl/certs', 358 | SSL_verify_mode => SSL_VERIFY_NONE 359 | } 360 | ) 361 | or return $done->( "tls handshake failed: " . $! . $SSL_ERROR, $ip ); 362 | $stats{tls} = time - $start; 363 | } 364 | else { 365 | $stats{tls} = 0; 366 | } 367 | 368 | # send the request and close the socket for writing 369 | $start = time; 370 | ( syswrite( $socket, $request ) == length $request ) 371 | or return $done->( "write failed: $!", $ip ); 372 | 373 | #shutdown $socket => 1; 374 | $stats{write} = time - $start; 375 | 376 | # now we want to time how long it takes to get the first byte 377 | $start = time; 378 | ( defined( sysread $socket, $content, 1 ) ) 379 | or return $done->( "read failed: $!", $ip ); 380 | $stats{firstbyte} = time - $start; 381 | 382 | # read the rest of the response 383 | $content .= $_ while ( sysread $socket, $_, BUFSIZ ); 384 | $stats{read} = time - ( $start + $stats{firstbyte} ); 385 | 386 | if ( $debug > 1 ) { 387 | print STDERR "$content\n"; 388 | } 389 | 390 | # close the socket 391 | $start = time; 392 | close $socket; 393 | $stats{close} = time - $start; 394 | $stats{total} = time - $stats{time}; 395 | 396 | # store the response length 397 | $stats{bytes} = ( defined $content ) ? length $content : 0; 398 | 399 | $stats{content} = $content; 400 | 401 | # was this response successful? 402 | $stats{status} = ( $content =~ m#HTTP/\S+\s+(\d+)\s+#o )[0] if ($content); 403 | 404 | # return the reference to the statistics array 405 | return $done->( "", $ip ); 406 | } # time_request() 407 | 408 | # show_header() 409 | # 410 | # Display the timing information header. 411 | sub show_header() { 412 | 413 | # determine the length of the time display 414 | my $length = length strftime( TIMEFMT, localtime ); 415 | 416 | # how much indentation should there be for the time? 417 | # - we want it centred above each timestamp 418 | my $time = 'Timestamp'; 419 | my $indent = int( ( $length - length $time ) / 2 ); 420 | $time = sprintf( "%s%-*s", ' ' x $indent, $length - $indent, $time ); 421 | 422 | # generate the times header 423 | my $times = join( ' ', 424 | ' DNS', 425 | ' Socket', 426 | ' Connect', 427 | ' TLS', 428 | ' Write ', 429 | '1stByte', 430 | ' Read ', 431 | ' Close ', 432 | ' Total ' ); 433 | 434 | # generate a header centred over the component time headers 435 | my $units = "Request Times (ms)"; 436 | $indent = int( ( length($times) - length($units) ) / 2 ); 437 | $units = sprintf( "%s%s", ' ' x ( $length + $indent ), $units ); 438 | 439 | # generate the complete header 440 | my $termsheader = ( defined $termsfile ) ? " [Term]" : ""; 441 | print "$units\n$time $times Bytes Status$termsheader\n"; 442 | } # show_header() 443 | 444 | # show_response($) 445 | # 446 | # Display the timing information. 447 | sub show_response($) { 448 | my $stats = shift; 449 | 450 | # generate a string representation of the time of the request 451 | my $time = strftime( TIMEFMT, localtime( int $stats->{time} ) ); 452 | 453 | # generate the output for this set of timings 454 | # - time 455 | # - DNS lookup 456 | # - socket creation 457 | # - connect 458 | # - TLS Handshake 459 | # - request write 460 | # - 1st byte of response 461 | # - response total 462 | # - close 463 | # - total time 464 | # - bytes 465 | # - status 466 | # - term 467 | # if the connect failed, then all other values will be undefined 468 | $stats->{$_} = 469 | ( defined $stats->{$_} ) 470 | ? sprintf( "%8.2f", $stats->{$_} * 1000. ) 471 | : ' ' x 8 foreach ( 472 | qw( dns socket connect tls write firstbyte 473 | read close total ) 474 | ); 475 | 476 | # convert the number of bytes into a well presented number 477 | $stats->{bytes} = 478 | ( defined $stats->{bytes} ) 479 | ? sprintf( "%6d", $stats->{bytes} ) 480 | : ' ' x 6; 481 | 482 | # what status should we display? 483 | # - if we got a 200 response code, then simply display OK, otherwise, if 484 | # we got something other than OK, show NOT OK and include the response 485 | # code in the error message (provided the error message is not already 486 | # defined) 487 | if ( defined $stats->{status} ) { 488 | if ( $stats->{status} == 200 ) { 489 | $stats->{error} = 'OK'; 490 | } 491 | else { 492 | $stats->{error} = 'NOT OK'; 493 | } 494 | } 495 | else { 496 | $stats->{error} = 'NOT OK'; 497 | } 498 | 499 | # generate the output for this response 500 | if ( $output_format eq "def" ) { 501 | print join( 502 | " ", $time, 503 | map { $stats->{$_} } 504 | qw( dns socket connect tls write firstbyte read 505 | close total ) 506 | ); 507 | print " "; 508 | print join( " ", map { $stats->{$_} } qw( bytes error ) ); 509 | print " (" . $stats->{status} . ")"; 510 | if ( defined $stats->{term} ) { 511 | print " [" . $stats->{term} . "]"; 512 | } 513 | } 514 | else { 515 | print strftime( "%s ", localtime( int $stats->{time} ) ); 516 | print join( " ", map { $stats->{$_} } ( split /,/, $output_format ) ); 517 | } 518 | print "\n"; 519 | } # show_response() 520 | 521 | __END__ 522 | =head1 NAME 523 | 524 | B - Generate timing information for HTTP requests. 525 | 526 | =head1 SYNOPSIS 527 | 528 | % ./checkprovider { url | options } 529 | % ./checkprovider --help 530 | 531 | =head1 DESCRIPTION 532 | 533 | B returns a breakdown of timing information for requests to 534 | a backend By default, the script runs 10 535 | queries at a rate of one per second, and outputs timing information for each 536 | request to C. 537 | 538 | The timing results are presented in tabulated form, with the following column 539 | headings: (all times are in milliseconds) 540 | 541 | =over 4 542 | 543 | =item C 544 | 545 | This is the timestamp of when the request starts. 546 | 547 | 548 | =item C 549 | 550 | The time taken to perform the DNS lookup for the server. 551 | 552 | 553 | =item C 554 | 555 | The time required to create a socket for this request. 556 | 557 | 558 | =item C 559 | 560 | The time taken to connect to the TCP endpoint. 561 | 562 | 563 | =item C 564 | 565 | The time taken to complete the TLS handshake. 566 | 567 | 568 | =item C 569 | 570 | The time spent writing the HTTP request to the socket. 571 | 572 | 573 | =item C<1st Byte> 574 | 575 | This is the time spent waiting for the first byte of the response. 576 | 577 | 578 | =item C 579 | 580 | The total time spent reading the response from the HTTP server. 581 | 582 | 583 | =item C 584 | 585 | The time to close the socket after reading the full response. 586 | 587 | 588 | =item C 589 | 590 | The total time elapsed during the request. 591 | 592 | 593 | =item C 594 | 595 | The number of bytes read in the response (including response headers). 596 | 597 | 598 | =item C 599 | 600 | This column shows either C or C to indicate whether the request 601 | was successful (i.e. returned an HTTP 200 response). If any other response 602 | was obtained, or the initial connection was unsuccessful then all fields from 603 | C through to C will be empty. 604 | 605 | =back 606 | 607 | B will die with an error if the initial DNS lookup or socket 608 | creation fail. 609 | 610 | 611 | =head1 OPTIONS 612 | 613 | The following command line options are supported by B: 614 | 615 | =over 4 616 | 617 | =item --no-blank 618 | 619 | Suppress the blank line at the end of the list of timing results. 620 | 621 | 622 | =item -d 623 | 624 | =item --debug 625 | 626 | Print some debug information to stdout. 627 | 628 | 629 | =item -F I 630 | 631 | =item --feed I 632 | 633 | Set the "feed" part of the url, eg -f /forecastxml 634 | 635 | 636 | =item -f I 637 | 638 | =item --frequency I 639 | 640 | Set the frequency of the requests in seconds. This is the inverse of the 641 | query I, and takes precendence over any rate specified with C<--rate>. 642 | 643 | 644 | =item -h 645 | 646 | =item --no-header 647 | 648 | Suppress the display of header at the top of the list of timing results. 649 | 650 | 651 | =item -? 652 | 653 | =item --help 654 | 655 | Display this full help information 656 | 657 | 658 | =item -p I 659 | 660 | =item --period I 661 | 662 | This specifies the period, in whole seconds, over which the requests are to be 663 | made. The default is 10 seconds. 664 | 665 | 666 | =item -r I 667 | 668 | =item --rate I 669 | 670 | This specifies the integer rate of requests per second that should be made by 671 | B. The I and I determine the number of requests 672 | sent to the HTTP server. The default rate is 1 request per second. 673 | 674 | 675 | =item -Q 676 | 677 | =item --query 678 | 679 | The query part of the URL, eg. -q "q=london". 680 | 681 | 682 | =item -q 683 | 684 | =item --quiet 685 | 686 | Suppress the blank line at the end of the results as well as the header. This 687 | has the same effect as using C<--no-header> and C<--no-blank>. 688 | 689 | 690 | =item -v 691 | 692 | =item --version 693 | 694 | Display the version information then quit. 695 | 696 | 697 | =item -w I 698 | 699 | =item --host I 700 | 701 | Specify the HTTP Host header. 702 | 703 | 704 | =item -U I 705 | 706 | =item -useragent I 707 | 708 | Specify the User-Agent HTTP header. 709 | 710 | 711 | =back 712 | 713 | =head1 EXAMPLES 714 | 715 | =over 4 716 | 717 | 718 | =item checkprovider "https://example.org/" 719 | 720 | Sends the request to example.org over port 443 (since the URI scheme is https). 721 | 722 | 723 | =item checkprovider -H "Accept-Encoding: gzip, deflate" "https://example.org/" 724 | 725 | Sends the request to example.org over port 443 with the "Accept-Encoding: gzip, deflate" request header. 726 | 727 | 728 | =item checkprovider -r 20 -p 5 "https://example.org/search?q=beer" 729 | 730 | Sends the request to example.org over port 443 sending at a rate of 20 qps for a duration of 5 seconds. 731 | 732 | 733 | =item checkprovider -T termlist "https://example.org/search?p=[TERM]" 734 | 735 | Iterate through the lines in file termlist and replace [TERM] with the line before sending the request. 736 | 737 | 738 | =back 739 | 740 | =head1 ACKNOWLEDGEMENTS 741 | 742 | Derived from goto-metric.pl by Richard Misenheimer. 743 | 744 | 745 | =head1 COPYRIGHT 746 | 747 | Copyright 2016, Yahoo Inc. 748 | Copyrights licensed under the Apache 2.0 License. 749 | See the accompanying LICENSE file for terms. 750 | 751 | =head1 CONTRIBUTORS 752 | 753 | =over 754 | 755 | =item Richard Misenheimer 756 | 757 | =item Ian Brayshaw 758 | 759 | =item Leong-Kui Lim 760 | 761 | =item Peter Ellehauge 762 | 763 | =item Scott Beardsley 764 | 765 | 766 | =back 767 | 768 | =cut 769 | --------------------------------------------------------------------------------