├── FetchData ├── FetchEvent ├── FetchMetadata ├── FetchSyn ├── README.md ├── _config.yml └── docs ├── fetchsyn.md ├── other-centers.md └── tutorial.md /FetchData: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl 2 | # 3 | # FetchData 4 | # 5 | # Find the most current version at http://service.iris.edu/clients/ 6 | # 7 | # Fetch data and related metadata from web services. The default web 8 | # service are from the IRIS DMC, other FDSN web services may be 9 | # specified by setting the following environment variables: 10 | # 11 | # SERVICEBASE = the base URI of the service(s) to use (http://service.iris.edu/) 12 | # TIMESERIESWS = complete URI of service (http://service.iris.edu/fdsnws/dataselect/1) 13 | # METADATAWS = complete URI of service (http://service.iris.edu/fdsnws/station/1) 14 | # SACPZWS = complete URI of service (http://service.iris.edu/irisws/sacpz/1) 15 | # RESPWS = complete URI of service (http://service.iris.edu/irisws/resp/1) 16 | # FEDCATWS = complete URI of service (http://service.iris.edu/irisws/fedcatalog/1) 17 | # 18 | # This program is primarily written to select and fetch waveform data 19 | # but can also fetch metadata and response information if those 20 | # services exist at the specified data center. The fdsnws-dataselect 21 | # service is a minimum requirement for use of this script. The 22 | # fdsnws-station service is required if metadata is to be retrieved or 23 | # if geographic selection options are used. 24 | # 25 | # Dependencies: This script should run without problems on Perl 26 | # release 5.10 or newer, older versions of Perl might require the 27 | # installation of the following modules (and their dependencies): 28 | # Bundle::LWP (libwww-perl) 29 | # 30 | ## Data selection 31 | # 32 | # Data is generally selected by specifying network, station, location, 33 | # channel, quality, start time and end time. The name parameters may 34 | # contain wildcard characters. All input options are optional but 35 | # waveform requests should include a time window. Data may be 36 | # selected one of three ways: 37 | # 38 | # 1) Command line arguments: -N, -S, -L, -C, -Q, -s, -e 39 | # 40 | # 2) A BREQ_FAST formatted file, http://ds.iris.edu/manuals/breq_fast.htm 41 | # 42 | # 3) A selection file containing a list of: 43 | # Net Sta Loc Chan Start End 44 | # 45 | # Example selection file contents: 46 | # II BFO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 47 | # IU ANMO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 48 | # IU COLA 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 49 | # 50 | # For the command line arguments and the selection file the network, 51 | # station location and channel fields may contain the common * and ? 52 | # wildcards, meaning zero-to-many and a single character respectively. 53 | # These fields may also be comma-separated lists, for example, the 54 | # network may be specified as II,IU,TA to select three networks. 55 | # 56 | ## Data output 57 | # 58 | # miniSEED: If the -o option is used to specify an output file 59 | # waveform data will be requested based on the selection and all 60 | # written to the single file. 61 | # 62 | # metadata: If the -m option is used to specifiy a metadata file a 63 | # line will be written to the file for each channel epoch and will 64 | # contain: 65 | # "net|sta|loc|chan|lat|lon|elev|depth|azimuth|dip|instrument|scale|scalefreq|scaleunits|samplerate|start|end" 66 | # 67 | # This metadata file can be used directly with mseed2sac or tracedsp 68 | # to create SAC files including basic metadata. 69 | # 70 | # SAC P&Zs: If the -sd option is given SAC Poles and Zeros will be 71 | # fetched and a file for each channel will be written to the specified 72 | # directory with the name 'SACPZ.Net.Sta.Loc.Chan'. If this option is 73 | # used while fetching waveform data, only channels which returned 74 | # waveforms will be requested. 75 | # 76 | # RESP: If the -rd option is given SEED RESP (as used by evalresp) 77 | # will be fetched and a file for each channel will be written to the 78 | #specified directory with the name 'RESP.Net.Sta.Loc.Chan'. If this 79 | # option is used while fetching waveform data, only channels which 80 | # returned waveforms will be requested. 81 | # 82 | # 83 | # ## Change history ## 84 | # 85 | # 2013.042: 86 | # - Rename to FetchData (from FetchBulkData), truncate change log. 87 | # - Use the LWP::UserAgent method env_proxy() to check for and use connection 88 | # proxy information from environment variables (e.g. http_proxy). 89 | # - Add checking of environment variables that will override the web 90 | # service base path (i.e. host name). 91 | # - Change to allow data requests without metadata fetching. 92 | # 93 | # 2013.067: 94 | # - Changed metadata parsing to understand FDSN StationXML schema. 95 | # - Create override service URLs for ws-sacpz and ws-resp until they 96 | # are migrated to service.iris.edu. 97 | # 98 | # 2013.074: 99 | # - Add work around for bug in Perl's Digest Authorization headers 100 | # that conflicts with pedantic behavior of Apache Tomcat, eventually 101 | # Tomcat will be more lenient and this work around will be removed. 102 | # 103 | # 2013.077: 104 | # - Convert metadata output line to be bar (|) separated instead of 105 | # comma separated and leave dip in SEED convention. 106 | # - Do not translate commas to semicolons in instrument name in metadata. 107 | # 108 | # 2013.086: 109 | # - Remove code to filter Authorization headers, Apache Tomcat has been fixed 110 | # to accept Digest Authentication credentials as submitted by libwww/LWP. 111 | # 112 | # 2013.118: 113 | # - Fix parsing of start and end times from metadata that are used when no 114 | # start and/or end is specified by the caller. 115 | # 116 | # 2013.150: 117 | # - Allow dash characters in breqfast formatted requests for the network 118 | # fields to support virtual networks that use dashes. 119 | # 120 | # 2013.186: 121 | # - Change service URL override command line options to match 122 | # environment variables. 123 | # 124 | # 2013.197: 125 | # - Fix parsing of element values of "0". 126 | # 127 | # 2013.198: 128 | # - Add test for minimum version of LWP (libwww) module of 5.806. 129 | # 130 | # 2013.212: 131 | # - Fetch metadata for request by default, this allows grouping of time series 132 | # requests and ultimately more efficient recovery in the case of connection 133 | # breaks. Also added an option of --nometadata or -nm to suppress the 134 | # fetching of metadata when it is not strictly needed. 135 | # - Remove lingering overrides to deprecated service locations. 136 | # 137 | # 2014.056: 138 | # - Allow gzip'ed HTTP encoding for metadata, SACPZ and RESP requests if 139 | # support exists on the local system. 140 | # - Add the -noretry option, when used the script will exit on time series 141 | # request timeouts/errors with no retries. 142 | # 143 | # 2014.084: 144 | # - Add -q option to make the script quiet except for errors. 145 | # - Exit value will be 1 if any service requests failed. 146 | # 147 | # 2014.107: 148 | # - Instantiate new UserAgent client for each group when fetching time series 149 | # instead of reusing the same client object. This is to make sure no 150 | # authentication details are shared between requests. 151 | # 152 | # 2014.129: 153 | # - Convert metadata fetching to use POST capability of fdsnws-station. 154 | # This allows making a single metadata request when the request is a list of 155 | # many selections (selection list file or BREQ_FAST), instead of generating a 156 | # request for each selection line. More efficient. 157 | # - Code simplification: separately manage request list for secondary metadata 158 | # such as SACPZ or RESP, track and request a range from earliest to latest 159 | # metadata epochs for each channel. This fixes a bug where only the last epoch 160 | # is represented in a metadata file when the request crosses many epochs. 161 | # 162 | # 2014.134: 163 | # - Optimize the metadata and request window matching by using compiled regexes. 164 | # 165 | # 2014.135: 166 | # - Fix matching requests and metadata for open time windows. 167 | # - Optimize request and metadata matching with a nested hash of compiled regexes. 168 | # - Avoid selecting too much secondary metadata (SACPZ and RESP) by shrinking 169 | # the request window by one second on each end. 170 | # 171 | # 2014.136: 172 | # - Fetch metadata for virtual networks separately from all other metadata in 173 | # order to properly match data requests with metadata. 174 | # - Properly match lists in network, station, location and channel fields. 175 | # 176 | # 2014.142: 177 | # - Fetch metadata using extents for each unique NSLC group, this can be a much 178 | # smaller (and faster query) for requests with a large number of repeated NSLCs. 179 | # 180 | # 2014.168: 181 | # - Explicitly match metadata epochs to time series requests to avoid 182 | # matching substrings, e.g. MONP matching MONP2. Only new code effected. 183 | # - Accept empty location strings in metadata as "--" identifiers. 184 | # - Follow redirects for POST method in addition to default GET and HEAD. 185 | # - A small bit of special output for 429 (Too Many Requests) results to 186 | # help the user understand what is going on if a server were to return this. 187 | # - Fix handling of metadata with no end times (open interval). 188 | # 189 | # 2014.253: 190 | # - Add detection of stream truncation by checking for "#STREAMERROR" 191 | # at the end of the content buffer. As there is no way to commicate 192 | # an error in HTTP after the transfer has started (and a full byte 193 | # count is not known) the DMC's servers will include an error message 194 | # in the stream when an error occurs. This should occur rarely. 195 | # - Optimize downloading, in particular for time series, by a) avoiding 196 | # checks for gzip-encoded HTTP streams when not needed and b) avoiding 197 | # copying of the data buffer. 198 | # 199 | # 2014.322: 200 | # - Add -F federation option, this will cause the request to be sent 201 | # to a federator catalog service. The response of the catalog service 202 | # is parsed and requests are sent to each identified data center. 203 | # - Restructure internal flow for multiple data center handling. 204 | # - Add FederateRequest() to handle federation catalog servicing. 205 | # 206 | # 2014.323: 207 | # - Add -O and -M options to write all federated output to the same files. 208 | # By default a data center prefix is added to the output from each DC. 209 | # - Only include a quality specification in time series requests if 210 | # supplied by the user. 211 | # 212 | # 2014.325: 213 | # - Add error message and more graceful failure when service interfaces 214 | # have not been identified for requested data. 215 | # 216 | # 2014.342: 217 | # - Federator: Add parsing of values for SACPZSERVICE and RESPSERVICE in 218 | # addition to the already parsed STATIONSERVICE and DATASELECTSERVICE. 219 | # - Federator: Gracefully skip unrecognized SERVICE declarations and 220 | # key=value parameters. 221 | # - Fix creation of SACPZ and RESP directories. 222 | # - Include output file names in diagnostic output. 223 | # - Add data center identifier, when present, to header of metadata files. 224 | # 225 | # 2014.351: 226 | # - Avoid undefined reference by checking for metadata before trying to access it. 227 | # 228 | # 2015.014: 229 | # - Change validation of channel codes in breq_fast parsing to accept 230 | # values less than 3 characters, this will allow single '*' wildcards. 231 | # - Add support for matching metadata using exclusions as supported by 232 | # the DMC's fdsnws-station service. 233 | # 234 | # 2015.135: 235 | # - Fix SAC PZ and RESP output directory designation and creation when 236 | # Federation is being performed. Data center specific directories are 237 | # now created in the directory specified for the output. 238 | # 239 | # 2015.246: 240 | # - Restore capability to write output miniSEED to stdout by specifying 241 | # the output file as a single dash. 242 | # - Support non-persisent, session cookies for HTTP requests. 243 | # - On authentication errors, retry the request a single time. 244 | # 245 | # 2015.341: 246 | # - Change retry count for failed time series requests from 60 to 10. 247 | # This mitigates long delays somewhat when data centers do not fail 248 | # for a very long time. 249 | # 250 | # 2016.007: 251 | # - Trim trailing slashes from service endpoint URLs as a convenience. 252 | # 253 | # 2016.062: 254 | # - Do not retry on 413 response and print server response on all non-auth errors. 255 | # - Trim trailing slash from SERVICEBASE values. 256 | # 257 | # 2016.089: 258 | # - Optimize the matching of metadata to requests by avoiding Tie'd hashes and 259 | # combining regular expressions for fewer match executions. 260 | # 261 | # 2016.260: 262 | # - When writing separate output files per data center, append data center ID 263 | # to the file name portion of the specified path instead of just the beginning. 264 | # - Only do 2 retries for time series requests when Federating. 265 | # - Add warnings when using minimum segment length and longest segment only 266 | # options when Federating, many data centers do no support these leading to 267 | # sometimes cryptic errors. 268 | # 269 | # 2016.299: 270 | # - Fix appending of data center ID to output file names when 271 | # directories are not specified. 272 | # 273 | # 2017.017: 274 | # - Add a minimum request interval and use it to throttle web service request 275 | # loops to avoid generating requests too fast. At 50 milliseconds the throttling 276 | # will not engage for the vast majority of users and uses. 277 | # 278 | # 2017.164: 279 | # - Set $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0 to allow usage with 280 | # HTTPS endpoints without peforming certificate verification. 281 | # - Allow -e end time specification to be an offset relative to the 282 | # start time in the pattern #.#[SMHD], e.g. 30m, 1h, 2D, etc. 283 | # - Allow -F option to accept a list of data centers that is passed 284 | # to the Federator to limit the response to specific centers. 285 | # Centers: http://service.iris.edu/irisws/fedcatalog/1/datacenters 286 | # - Add more details to help message if -h is specified more than once. 287 | # Relegate lesser used options to the extended help and add more details 288 | # of federation usage and specification of alternate WS endpoints. 289 | # 290 | # 2018.337: 291 | # - Add -X option to download metadata as StationXML at response level. 292 | # 293 | # 2020.314: 294 | # - Add explicit test for no requests after discovery and skip data center if needed. 295 | # 296 | # 2025.126 297 | # - Use usleep() instead of nanosleep() to support Strawberry Perl on Windows 298 | # - Add ":raw" to output file handles for binary writing under Windows 299 | # Changes suggested by menke at ldeo. 300 | # 301 | # Author: Chad Trabant, EarthScope Data Services 302 | 303 | use strict; 304 | use File::Basename; 305 | use File::Spec; 306 | use Getopt::Long; 307 | use LWP 5.806; # Require minimum version 308 | use LWP::UserAgent; 309 | use HTTP::Status qw(status_message); 310 | use HTTP::Date; 311 | use Time::HiRes qw(usleep); 312 | 313 | my $version = "2025.126"; 314 | 315 | my $scriptname = basename($0); 316 | 317 | # Default web service base 318 | my $servicebase = 'http://service.iris.edu'; 319 | 320 | # Check for environment variable overrides for servicebase 321 | $servicebase = $ENV{'SERVICEBASE'} if ( exists $ENV{'SERVICEBASE'} ); 322 | $servicebase =~ s/\/$//; # Trim trailing slash 323 | 324 | # Web service for time series data 325 | my $timeseriesservice = "$servicebase/fdsnws/dataselect/1"; 326 | 327 | # Check for environment variable override for timeseriesservice 328 | $timeseriesservice = $ENV{'TIMESERIESWS'} if ( exists $ENV{'TIMESERIESWS'} ); 329 | 330 | # Default web service for metadata 331 | my $metadataservice = "$servicebase/fdsnws/station/1"; 332 | 333 | # Check for environment variable override for metadataservice 334 | $metadataservice = $ENV{'METADATAWS'} if ( exists $ENV{'METADATAWS'} ); 335 | 336 | # Web service for SAC P&Z 337 | my $sacpzservice = "$servicebase/irisws/sacpz/1"; 338 | 339 | # Check for environment variable override for sacpzservice 340 | $sacpzservice = $ENV{'SACPZWS'} if ( exists $ENV{'SACPZWS'} ); 341 | 342 | # Web service for RESP 343 | my $respservice = "$servicebase/irisws/resp/1"; 344 | 345 | # Check for environment variable override for respservice 346 | $respservice = $ENV{'RESPWS'} if ( exists $ENV{'RESPWS'} ); 347 | 348 | # Web service for federation catalog 349 | my $fedcatservice = "$servicebase/irisws/fedcatalog/1"; 350 | 351 | # Check for environment variable override for fedcatservice 352 | $fedcatservice = $ENV{'FEDCATWS'} if ( exists $ENV{'FEDCATWS'} ); 353 | 354 | # HTTP UserAgent reported to web services 355 | my $useragent = "$scriptname/$version Perl/$] " . new LWP::UserAgent->_agent; 356 | 357 | # Waveform data request group size in terms of station-days 358 | my $groupstadays = 30; 359 | 360 | # A minimum time interval (seconds) for web service requests 361 | # This is used to avoid sending requests too quickly 362 | my $minimumrequestinterval = 0.05; 363 | 364 | # Allow encrypted connections without checking for valid certificate matching 365 | # the expected hostname. 366 | $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; 367 | 368 | my $usage = undef; 369 | my $verbose = 0; 370 | my $nobsprint = undef; 371 | 372 | my $net = undef; 373 | my $sta = undef; 374 | my $loc = undef; 375 | my $chan = undef; 376 | my $qual = undef; 377 | my $starttime = undef; 378 | my $endtime = undef; 379 | my @latrange = (); # (minlat:maxlat) 380 | my @lonrange = (); # (minlon:maxlon) 381 | my @degrange = (); # (lat:lon:maxradius[:minradius]) 382 | my $selectfile = undef; 383 | my $bfastfile = undef; 384 | my $mslopt = undef; 385 | my $lsoopt = undef; 386 | my $appname = undef; 387 | my $auth = undef; 388 | my $outfile = undef; 389 | my $outfileapp = undef; 390 | my $sacpzdir = undef; 391 | my $respdir = undef; 392 | my $metafile = undef; 393 | my $metafileapp= undef; 394 | my $nometadata = undef; 395 | my $sxmlfile = undef; 396 | my $noretry = undef; 397 | my $retries = 10; 398 | my $federate = undef; 399 | my $exitvalue = 0; 400 | 401 | my $inflater = undef; 402 | 403 | # If Compress::Raw::Zlib is available configure inflater for RFC 1952 (gzip) 404 | if ( eval("use Compress::Raw::Zlib; 1") ) { 405 | use Compress::Raw::Zlib; 406 | $inflater = new Compress::Raw::Zlib::Inflate( -WindowBits => WANT_GZIP, 407 | -ConsumeInput => 0 ); 408 | } 409 | 410 | # Parse command line arguments 411 | Getopt::Long::Configure ("bundling_override"); 412 | my $getoptsret = GetOptions( 413 | 'help|usage|h+' => \$usage, 414 | 'verbose|v+' => \$verbose, 415 | 'quiet|q' => sub { $verbose = -1; }, 416 | 'nobs' => \$nobsprint, 417 | 'nometadata|nm' => \$nometadata, 418 | 'noretry|nr' => \$noretry, 419 | 'federate|F:s' => \$federate, 420 | 'net|N=s' => \$net, 421 | 'sta|S=s' => \$sta, 422 | 'loc|L=s' => \$loc, 423 | 'chan|C=s' => \$chan, 424 | 'qual|Q=s' => \$qual, 425 | 'starttime|s=s' => \$starttime, 426 | 'endtime|e=s' => \$endtime, 427 | 'lat=s' => \@latrange, 428 | 'lon=s' => \@lonrange, 429 | 'radius=s' => \@degrange, 430 | 'selectfile|l=s' => \$selectfile, 431 | 'bfastfile|b=s' => \$bfastfile, 432 | 'msl=s' => \$mslopt, 433 | 'lso' => \$lsoopt, 434 | 'appname|A=s' => \$appname, 435 | 'auth|a=s' => \$auth, 436 | 'outfile|o=s' => \$outfile, 437 | 'outfileapp|O=s' => \$outfileapp, 438 | 'sacpzdir|sd=s' => \$sacpzdir, 439 | 'respdir|rd=s' => \$respdir, 440 | 'metafile|m=s' => \$metafile, 441 | 'metafileapp|M=s' => \$metafileapp, 442 | 'sxmlfile|X=s' => \$sxmlfile, 443 | 'timeseriesws=s' => \$timeseriesservice, 444 | 'metadataws=s' => \$metadataservice, 445 | 'sacpzws=s' => \$sacpzservice, 446 | 'respws=s' => \$respservice, 447 | ); 448 | 449 | my $required = (defined $net || defined $sta || 450 | defined $loc || defined $chan || 451 | scalar @latrange || scalar @lonrange || scalar @degrange || 452 | defined $starttime || defined $endtime || 453 | defined $selectfile || defined $bfastfile ); 454 | 455 | if ( ! $getoptsret || $usage || ! $required ) { 456 | print "$scriptname: collect time series and related metadata (version $version)\n"; 457 | print "http://service.iris.edu/clients/\n\n"; 458 | print "Usage: $scriptname [options]\n\n"; 459 | print " Options:\n"; 460 | print " -v Increase verbosity, may be specified multiple times\n"; 461 | print " -h Print this help message, if multiple print more help\n"; 462 | print " -q Be quiet, do not print anything but errors\n"; 463 | print " -N,--net Network code, list and wildcards (* and ?) accepted\n"; 464 | print " -S,--sta Station code, list and wildcards (* and ?) accepted\n"; 465 | print " -L,--loc Location ID, list and wildcards (* and ?) accepted\n"; 466 | print " -C,--chan Channel codes, list and wildcards (* and ?) accepted\n"; 467 | print " -Q,--qual Quality indicator, by default no quality is specified\n"; 468 | print " -s starttime Specify start time (YYYY-MM-DD,HH:MM:SS.ssssss)\n"; 469 | print " -e endtime Specify end time (YYYY-MM-DD,HH:MM:SS.ssssss or #[SMHD])\n"; 470 | print " --lat min:max Specify a minimum and/or maximum latitude range\n"; 471 | print " --lon min:max Specify a minimum and/or maximum longitude range\n"; 472 | print " --radius lat:lon:maxradius[:minradius]\n"; 473 | print " Specify circular region with optional minimum radius\n"; 474 | print " -l listfile Read list of selections from file\n"; 475 | print " -b bfastfile Read list of selections from BREQ_FAST file\n"; 476 | print " -a user:pass User and password for access to restricted data\n"; 477 | print "\n"; 478 | print " -F [DC1[,DC2]] Federate the request to multiple data centers if needed\n"; 479 | print " Federation may be limited to an optional list of DCs\n"; 480 | print " Output files are prefixed by data center identifiers\n"; 481 | print "\n"; 482 | print " -o outfile Fetch time series data and write to output file\n"; 483 | print " -sd sacpzdir Fetch SAC P&Zs and write files to sacpzdir\n"; 484 | print " -rd respdir Fetch RESP and write files to respdir\n"; 485 | print " -m metafile Write basic metadata to specified file\n"; 486 | print " -X SXMLfile Write response-level StationXML to specified file\n"; 487 | print "\n"; 488 | 489 | if ( $usage >= 2 ) { 490 | print " More options and help:\n"; 491 | print " -nm Do not request metadata unless output file requested\n"; 492 | print " -nr No retry, exit immediately on time series request errors\n"; 493 | print " -msl length Limit returned data to a minimum segment length\n"; 494 | print " -lso Limit returned data to the longest segment only\n"; 495 | print " -A appname Application/version string for identification\n"; 496 | print " -O outfile Write all timeseries to a single file, useful with -F\n"; 497 | print " -M metafile Write all metadata to a single file, useful with -F\n"; 498 | print "\n"; 499 | print "== Specifying data centers for federation\n"; 500 | print " List of data center identifiers:\n"; 501 | print " http://service.iris.edu/irisws/fedcatalog/1/datacenters\n"; 502 | print " To avoid a center, negate it by adding a 'not' prefix, e.g. 'notIRISDMC'\n"; 503 | print "\n"; 504 | print "== Specifying alternate web service endpoints (when not Federating)\n"; 505 | print " Alternate service endpoints may be specified using the following options:\n"; 506 | print " -timeseriesws URL (e.g. http://service.iris.edu/fdsnws/dataselect/1)\n"; 507 | print " -metadataws URL (e.g. http://service.iris.edu/fdsnws/station/1)\n"; 508 | print " -respws URL (e.g. http://service.iris.edu/irisws/resp/1)\n"; 509 | print " -sacpzws URL (e.g. http://service.iris.edu/irisws/sacpz/1/)\n"; 510 | print "\n"; 511 | } 512 | 513 | exit 1; 514 | } 515 | 516 | # Truncate any existing appending output file and assign to outfile 517 | if ( $outfileapp ) { 518 | die "Cannot specify both -o and -O\n" if ( $outfile ); 519 | 520 | if ( -f "$outfileapp" ) { 521 | truncate ($outfileapp, 0) || die "Cannot truncate existing file $outfileapp\n"; 522 | } 523 | 524 | $outfile = $outfileapp; 525 | } 526 | 527 | # Truncate any existing appending metadata file and assign to metafile 528 | if ( $metafileapp ) { 529 | die "Cannot specify both -m and -M\n" if ( $metafile ); 530 | 531 | if ( -f "$metafileapp" ) { 532 | truncate ($metafileapp, 0) || die "Cannot truncate existing file $metafileapp\n"; 533 | } 534 | 535 | $metafile = $metafileapp; 536 | } 537 | 538 | if ( ! $outfile && ! $metafile && ! $sxmlfile && ! $sacpzdir && ! $respdir ) { 539 | die "No output options specified, try -h for usage information\n"; 540 | } 541 | 542 | # Print script name and local time string 543 | if ( $verbose >= 1 ) { 544 | my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time); 545 | printf STDERR "$scriptname ($version) at %4d-%02d-%02d %02d:%02d:%02d\n", $year+1900, $mon+1, $mday, $hour, $min, $sec; 546 | } 547 | 548 | # Check for existence of output directories 549 | if ( $sacpzdir && ! -d "$sacpzdir" ) { 550 | die "Cannot find SAC P&Zs output directory: $sacpzdir\n"; 551 | } 552 | if ( $respdir && ! -d "$respdir" ) { 553 | die "Cannot find RESP output directory: $respdir\n"; 554 | } 555 | 556 | # Check for time window if requesting time series data 557 | if ( $outfile && ( ! defined $selectfile && ! defined $bfastfile && 558 | ( ! defined $starttime || ! defined $endtime ) ) ) { 559 | die "Cannot request time series data without start and end times\n"; 560 | } 561 | 562 | # Normalize time strings given on the command line 563 | if ( $starttime ) { 564 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $starttime); 565 | $starttime = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 566 | $starttime .= ".$subsec" if ( $subsec ); 567 | } 568 | 569 | if ( $endtime ) { 570 | # Check for and parse time in duration pattern: #.#[SMHD] 571 | if ( $endtime =~ /^[\d\.]+[SsMmHhDd]?$/ ) { 572 | if ( $starttime ) { 573 | my ($offset, $timeunit) = $endtime =~ /^([\d\.]+)([SsMmHhDd]?)$/i; 574 | $timeunit = 'S' if (! $timeunit); 575 | 576 | # Convert offset value to seconds if specified as days, hours or minutes 577 | if ($timeunit =~ /[Dd]/) { 578 | $offset *= 86400; 579 | } 580 | elsif ($timeunit =~ /[Hh]/) { 581 | $offset *= 3600; 582 | } 583 | elsif ($timeunit =~ /[Mm]/) { 584 | $offset *= 60; 585 | } 586 | 587 | # Calculate end time from start + offset and generate string 588 | my $rstartepoch = str2time ($starttime, "UTC"); 589 | if ( defined $rstartepoch ) { 590 | $endtime = &mktimestring($rstartepoch + $offset, 1); 591 | } 592 | else { 593 | die "Unable to parse start time: '$starttime'\n" 594 | } 595 | } 596 | else { 597 | die "Cannot specify end time as duration without specifying start time\n"; 598 | } 599 | } 600 | else { 601 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $endtime); 602 | $endtime = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 603 | $endtime .= ".$subsec" if ( $subsec ); 604 | } 605 | } 606 | 607 | # Validate and prepare lat, lon and radius input 608 | if ( scalar @latrange ) { 609 | @latrange = split (/:/, $latrange[0]); 610 | 611 | if ( defined $latrange[0] && ($latrange[0] < -90.0 || $latrange[0] > 90.0) ) { 612 | die "Minimum latitude out of range: $latrange[0]\n"; 613 | } 614 | if ( defined $latrange[1] && ($latrange[1] < -90.0 || $latrange[1] > 90.0) ) { 615 | die "Maximum latitude out of range: $latrange[1]\n"; 616 | } 617 | } 618 | if ( scalar @lonrange ) { 619 | @lonrange = split (/\:/, $lonrange[0]); 620 | 621 | if ( defined $lonrange[0] && ($lonrange[0] < -180.0 || $lonrange[0] > 180.0) ) { 622 | die "Minimum longitude out of range: $lonrange[0]\n"; 623 | } 624 | if ( defined $lonrange[1] && ($lonrange[1] < -180.0 || $lonrange[1] > 180.0) ) { 625 | die "Maximum longitude out of range: $lonrange[1]\n"; 626 | } 627 | } 628 | if ( scalar @degrange ) { 629 | @degrange = split (/\:/, $degrange[0]); 630 | 631 | if ( scalar @degrange < 3 || scalar @degrange > 4 ) { 632 | die "Unrecognized radius specification: @degrange\n"; 633 | } 634 | if ( defined $degrange[0] && ($degrange[0] < -90.0 || $degrange[0] > 90.0) ) { 635 | die "Radius latitude out of range: $degrange[0]\n"; 636 | } 637 | if ( defined $degrange[1] && ($degrange[1] < -180.0 || $degrange[1] > 180.0) ) { 638 | die "Radius longitude out of range: $degrange[1]\n"; 639 | } 640 | } 641 | 642 | # An array to hold data selections 643 | my @selections = (); 644 | 645 | # Add command line selection to list 646 | if ( defined $net || defined $sta || defined $loc || defined $chan || 647 | defined $starttime || defined $endtime ) { 648 | push (@selections,"$net|$sta|$loc|$chan|$starttime|$endtime"); 649 | } 650 | 651 | # Read selection list file 652 | if ( $selectfile ) { 653 | print STDERR "Reading data selection from list file '$selectfile'\n"; 654 | &ReadSelectFile ($selectfile); 655 | } 656 | 657 | # Read BREQ_FAST file 658 | if ( $bfastfile ) { 659 | print STDERR "Reading data selection from BREQ_FAST file '$bfastfile'\n"; 660 | &ReadBFastFile ($bfastfile); 661 | } 662 | 663 | # Report complete data selections 664 | if ( $verbose > 2 ) { 665 | print STDERR "== Data selections ==\n"; 666 | foreach my $select ( @selections ) { 667 | print STDERR " $select\n"; 668 | } 669 | print STDERR "Latitude range: $latrange[0] : $latrange[1]\n" if ( scalar @latrange ); 670 | print STDERR "Longitude range: $lonrange[0] : $lonrange[1]\n" if ( scalar @lonrange ); 671 | print STDERR "Radius range: $degrange[0] : $degrange[1] : $degrange[2] : $degrange[3]\n" if ( scalar @degrange ); 672 | } 673 | 674 | # A mega hash for data center details, requests and some results 675 | # 676 | # datacenter{DATACENTER}{website} = URL 677 | # datacenter{DATACENTER}{timeseriesws} = URL 678 | # datacenter{DATACENTER}{metadataws} = URL 679 | # datacenter{DATACENTER}{sacpzws} = URL 680 | # datacenter{DATACENTER}{respws} = URL 681 | # 682 | # datacenter{DATACENTER}{selection} = ref to ARRAY of selections 683 | # datacenter{DATACENTER}{request} = ref to HASH of requests 684 | # datacenter{DATACENTER}{metarequest} = ref to HASH of metadata requests (time extents) 685 | # datacenter{DATACENTER}{metadata} = ref to ARRAY of metadata 686 | my %datacenter = (); 687 | 688 | # A buffer for metadata service responses 689 | my $metadataresponse; 690 | 691 | # Track bytes downloaded in callback handlers 692 | my $datasize = 0; 693 | 694 | # Fetch metadata from the station web service by default unless the nometadata option 695 | # is specified or if metadata output file has been requested or if geographic range 696 | # selection is requested. 697 | $nometadata = undef if ( $metafile || $sxmlfile || $sacpzdir || $respdir 698 | || scalar @latrange || scalar @lonrange || scalar @degrange ); 699 | 700 | # Resolve federated requests 701 | if ( defined $federate ) { 702 | # Translate negation specified as 'not' with the '-' needed by the service 703 | $federate =~ s/not/\-/g; 704 | 705 | # Set number of time series request retries to 2 706 | $retries = 2; 707 | 708 | # Print warnings for options not commonly supported 709 | if ( $mslopt ) { 710 | print STDERR "WARNING: Minimum segment length (-msl) option is not broadly supported by data centers\n" 711 | } 712 | if ( $lsoopt ) { 713 | print STDERR "WARNING: Longest segment only (-lso) option is not broadly supported by data centers\n" 714 | } 715 | 716 | &FederateRequest( $fedcatservice, \@selections ); 717 | 718 | if ( $verbose >= 1 ) { 719 | printf STDERR "Federation catalog results from %d data center(s):\n", scalar keys %datacenter; 720 | foreach my $dckey ( sort keys %datacenter ) { 721 | printf STDERR "Data center: $dckey, %d selections\n", scalar @{$datacenter{$dckey}{selection}}; 722 | print STDERR " MetadataWS: $datacenter{$dckey}{metadataws}\n" if ( $datacenter{$dckey}{metadataws} ); 723 | print STDERR " TimeSeriesWS: $datacenter{$dckey}{timeseriesws}\n" if ( $datacenter{$dckey}{timeseriesws} ); 724 | print STDERR " SACPZWS: $datacenter{$dckey}{sacpzws}\n" if ( $datacenter{$dckey}{sacpzws} ); 725 | print STDERR " RESPWS: $datacenter{$dckey}{respws}\n" if ( $datacenter{$dckey}{respws} ); 726 | } 727 | } 728 | } 729 | # Otherwise set up default (empty) data center 730 | else { 731 | # Trim trailing slashes from service endpoints 732 | $timeseriesservice =~ s/\/$//; 733 | $metadataservice =~ s/\/$//; 734 | $sacpzservice =~ s/\/$//; 735 | $respservice =~ s/\/$//; 736 | 737 | # Add default/environmental entries to datacenter hash 738 | $datacenter{""}{timeseriesws} = $timeseriesservice; 739 | $datacenter{""}{metadataws} = $metadataservice; 740 | $datacenter{""}{sacpzws} = $sacpzservice; 741 | $datacenter{""}{respws} = $respservice; 742 | 743 | # User selections used directly 744 | $datacenter{""}{selection} = \@selections; 745 | } 746 | 747 | # Process each data center 748 | foreach my $dckey ( sort keys %datacenter ) { 749 | 750 | if ( $dckey ) { 751 | printf STDERR "Fetching data from $dckey (%s)\n", $datacenter{$dckey}{website}; 752 | } 753 | 754 | # Fetch metadata unless requested not to 755 | if ( ! defined $nometadata ) { 756 | if ( ! exists $datacenter{$dckey}{metadataws} ) { 757 | print STDERR "Cannot fetch metadata, no fdsnws-station service available for data center $dckey\n"; 758 | } 759 | else { 760 | &FetchMetaData( $dckey ); 761 | } 762 | } 763 | # Build request hash directly from selections if not fetching metadata and not already populated 764 | elsif ( ! exists $datacenter{$dckey}{request} ) { 765 | foreach my $selection ( @{$datacenter{$dckey}{selection}} ) { 766 | my ($snet,$ssta,$sloc,$schan,$sstart,$send) = split (/\|/,$selection); 767 | 768 | # Subsitute non-specified fields with wildcards 769 | $snet = "*" if ( ! $snet ); 770 | $ssta = "*" if ( ! $ssta ); 771 | $sloc = "*" if ( ! $sloc ); 772 | $schan = "*" if ( ! $schan ); 773 | 774 | $datacenter{$dckey}{request}->{"$snet|$ssta|$sloc|$schan|$sstart|$send"} = "$sstart|$send"; 775 | } 776 | } 777 | 778 | # Report complete data request and metadata request 779 | if ( $verbose > 2 ) { 780 | printf STDERR "== Request list (%d) ==\n", scalar keys %{$datacenter{$dckey}{request}}; 781 | foreach my $req ( sort keys %{$datacenter{$dckey}{request}} ) { 782 | print STDERR " $req (metadata: $datacenter{$dckey}{request}->{$req})\n"; 783 | } 784 | print STDERR "== End of request list ==\n"; 785 | 786 | printf STDERR "== Metadata request list (%d) ==\n", scalar keys %{$datacenter{$dckey}{metarequest}}; 787 | foreach my $req ( sort keys %{$datacenter{$dckey}{metarequest}} ) { 788 | print STDERR " $req (metadata: $datacenter{$dckey}{metarequest}->{$req})\n"; 789 | } 790 | print STDERR "== End of metadata request list ==\n"; 791 | } 792 | 793 | # Done with this data center if no requests 794 | if (scalar keys %{$datacenter{$dckey}{request}} <= 0 && 795 | scalar keys %{$datacenter{$dckey}{metarequest}} <= 0) { 796 | if ( $verbose ) { 797 | print STDERR "No requests for data center $dckey\n"; 798 | } 799 | 800 | next; 801 | } 802 | 803 | # Fetch time series data if output file specified 804 | if ( $outfile ) { 805 | if ( ! exists $datacenter{$dckey}{timeseriesws} ) { 806 | print STDERR "Cannot fetch time series, no fdsnws-dataselect service available for data center $dckey\n"; 807 | } 808 | else { 809 | # Determine output file mode (overwrite or append) and add data center prefix if needed 810 | my $outfilemode = ( defined $outfileapp ) ? ">>:raw" : ">:raw"; 811 | my $outfilename = $outfile; 812 | if ( ! defined $outfileapp && $dckey ) { 813 | # Add data center identifier, $dckey, to the beginning of the file name 814 | my ($volume,$directories,$file) = File::Spec->splitpath ($outfilename); 815 | 816 | if ( $directories ) { 817 | $outfilename = File::Spec->catfile ($directories, "$dckey-$file"); 818 | } 819 | else { 820 | $outfilename = "$dckey-$file"; 821 | } 822 | } 823 | 824 | &FetchTimeSeriesData( $dckey, $outfilename, $outfilemode ) if ( $outfile ); 825 | } 826 | } 827 | 828 | # Collect SAC P&Zs if output directory specified 829 | if ( $sacpzdir ) { 830 | if ( ! exists $datacenter{$dckey}{sacpzws} ) { 831 | print STDERR "Cannot fetch SAC PZs, no SACPZ service available for data center $dckey\n"; 832 | } 833 | else { 834 | my $dcsacpzdir = ( $dckey ) ? File::Spec->catdir ($sacpzdir,$dckey) : $sacpzdir; 835 | 836 | if ( ! -d "$dcsacpzdir" ) { 837 | mkdir ($dcsacpzdir, 0755) || die "Cannot create directory $dcsacpzdir: $!\n"; 838 | } 839 | 840 | &FetchSACPZ( $dckey, $dcsacpzdir ); 841 | } 842 | } 843 | 844 | # Collect RESP if output directory specified 845 | if ( $respdir ) { 846 | if ( ! exists $datacenter{$dckey}{respws} ) { 847 | print STDERR "Cannot fetch RESP, no RESP service available for data center $dckey\n"; 848 | } 849 | else { 850 | my $dcrespdir = ( $dckey ) ? File::Spec->catdir ($respdir,$dckey) : $respdir; 851 | 852 | if ( ! -d "$dcrespdir" ) { 853 | mkdir ($dcrespdir, 0755) || die "Cannot create directory $dcrespdir: $!\n"; 854 | } 855 | 856 | &FetchRESP( $dckey, $dcrespdir ); 857 | } 858 | } 859 | 860 | # Collect StationXML 861 | if ( $sxmlfile ) { 862 | if ( ! exists $datacenter{$dckey}{metadataws} ) { 863 | print STDERR "Cannot fetch StationXML, no metadata service available for data center $dckey\n"; 864 | } 865 | else { 866 | my $dcsxmlfile = $sxmlfile; 867 | 868 | if ( $dckey ) { 869 | # Add data center identifier, $dckey, to the beginning of the file name 870 | my ($volume,$directories,$file) = File::Spec->splitpath ($sxmlfile); 871 | 872 | if ( $directories ) { 873 | $dcsxmlfile = File::Spec->catfile ($directories, "$dckey-$file"); 874 | } 875 | else { 876 | $dcsxmlfile = "$dckey-$file"; 877 | } 878 | } 879 | 880 | &FetchStationXML( $dckey, $dcsxmlfile ); 881 | } 882 | } 883 | 884 | # Write metadata to file 885 | if ( $metafile && exists $datacenter{$dckey}{metadata} ) { 886 | if ( scalar @{$datacenter{$dckey}{metadata}} <= 0 ) { 887 | printf STDERR "No metdata available\n"; 888 | } 889 | else { 890 | # Open metadata file, appending if requested, adding data center prefix if needed 891 | my $mode = ( defined $metafileapp ) ? ">>:raw" : ">:raw"; 892 | my $metafilename = $metafile; 893 | if ( ! defined $metafileapp && $dckey ) { 894 | # Add data center identifier, $dckey, to the beginning of the file name 895 | my ($volume,$directories,$file) = File::Spec->splitpath ($metafilename); 896 | 897 | if ( $directories ) { 898 | $metafilename = File::Spec->catfile ($directories, "$dckey-$file"); 899 | } 900 | else { 901 | $metafilename = "$dckey-$file"; 902 | } 903 | } 904 | 905 | open (META, $mode, $metafilename) || die "Cannot open metadata file '$metafilename': $!\n"; 906 | 907 | printf STDERR "Writing metadata (%d channel epochs) to file: %s\n", 908 | scalar @{$datacenter{$dckey}{metadata}}, $metafilename if ( $verbose >= 0 ); 909 | 910 | # Print data center identifier 911 | printf META "#$dckey: %s\n", $datacenter{$dckey}{website} if ( $dckey ); 912 | 913 | # Print header line 914 | print META "#net|sta|loc|chan|lat|lon|elev|depth|azimuth|dip|instrument|scale|scalefreq|scaleunits|samplerate|start|end\n"; 915 | 916 | foreach my $channel ( sort @{$datacenter{$dckey}{metadata}} ) { 917 | my ($net,$sta,$loc,$chan,$start,$end,$lat,$lon,$elev,$depth,$azimuth,$dip,$instrument,$samplerate,$sens,$sensfreq,$sensunit) = 918 | split (/\|/, $channel); 919 | 920 | $sensfreq = sprintf ("%0g", $sensfreq); 921 | $samplerate = sprintf ("%0g", $samplerate); 922 | 923 | print META "$net|$sta|$loc|$chan|$lat|$lon|$elev|$depth|$azimuth|$dip|$instrument|$sens|$sensfreq|$sensunit|$samplerate|$start|$end\n"; 924 | } 925 | 926 | close META; 927 | } 928 | } 929 | } # Done looping through data centers 930 | 931 | my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time); 932 | printf (STDERR "DONE at %4d-%02d-%02d %02d:%02d:%02d\n", 933 | $year+1900, $mon+1, $mday, $hour, $min, $sec) if ( $verbose >= 0 ); 934 | 935 | exit $exitvalue; 936 | ## End of main 937 | 938 | 939 | ###################################################################### 940 | # ReadSelectFile: 941 | # 942 | # Read selection list file and add entries to the @selections array. 943 | # 944 | # Selection lines are expected to be in the following form: 945 | # 946 | # "Net Sta Loc Chan Start End" 947 | # 948 | # The Net, Sta, Loc and Channel fields are required and can be 949 | # specified as wildcards. 950 | ###################################################################### 951 | sub ReadSelectFile { 952 | my $selectfile = shift; 953 | 954 | open (SF, "<$selectfile") || die "Cannot open '$selectfile': $!\n"; 955 | 956 | foreach my $line ( ) { 957 | chomp $line; 958 | next if ( $line =~ /^\#/ ); # Skip comment lines 959 | 960 | my ($net,$sta,$loc,$chan,$start,$end) = split (' ', $line); 961 | 962 | next if ( ! defined $chan ); 963 | 964 | # Normalize time strings 965 | if ( $start ) { 966 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $start); 967 | $start = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 968 | $start .= ".$subsec" if ( $subsec ); 969 | } 970 | 971 | if ( $end ) { 972 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $end); 973 | $end = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 974 | $end .= ".$subsec" if ( $subsec ); 975 | } 976 | 977 | # Add selection to global list 978 | push (@selections,"$net|$sta|$loc|$chan|$start|$end"); 979 | } 980 | 981 | close SF; 982 | } # End of ReadSelectFile() 983 | 984 | 985 | ###################################################################### 986 | # ReadBFastFile: 987 | # 988 | # Read BREQ_FAST file and add entries to the @selections array. 989 | # 990 | ###################################################################### 991 | sub ReadBFastFile { 992 | my $bfastfile = shift; 993 | 994 | open (BF, "<$bfastfile") || die "Cannot open '$bfastfile': $!\n"; 995 | 996 | my $linecount = 0; 997 | BFLINE: foreach my $line ( ) { 998 | chomp $line; 999 | $linecount++; 1000 | next if ( ! $line ); # Skip empty lines 1001 | 1002 | # Capture .QUALTIY header 1003 | if ( $line =~ /^\.QUALITY .*$/ ) { 1004 | ($qual) = $line =~ /^\.QUALITY ([DRQMBE])/; 1005 | next; 1006 | } 1007 | 1008 | next if ( $line =~ /^\./ ); # Skip other header lines 1009 | 1010 | my ($sta,$net,$syear,$smon,$sday,$shour,$smin,$ssec,$eyear,$emon,$eday,$ehour,$emin,$esec,$count,@chans) = split (' ', $line); 1011 | 1012 | # Simple validation of BREQ FAST fields 1013 | if ( $sta !~ /^[A-Za-z0-9*?]{1,5}$/ ) { 1014 | print "Unrecognized station code: '$sta', skipping line $linecount\n" if ( $verbose >= 1 ); 1015 | next; 1016 | } 1017 | if ( $net !~ /^[-_A-Za-z0-9*?]+$/ ) { 1018 | print "Unrecognized network code: '$net', skipping line $linecount\n" if ( $verbose >= 1 ); 1019 | next; 1020 | } 1021 | if ( $syear !~ /^\d\d\d\d$/ ) { 1022 | print "Unrecognized start year: '$syear', skipping line $linecount\n" if ( $verbose >= 1 ); 1023 | next; 1024 | } 1025 | if ( $smon !~ /^\d{1,2}$/ ) { 1026 | print "Unrecognized start month: '$smon', skipping line $linecount\n" if ( $verbose >= 1 ); 1027 | next; 1028 | } 1029 | if ( $sday !~ /^\d{1,2}$/ ) { 1030 | print "Unrecognized start day: '$sday', skipping line $linecount\n" if ( $verbose >= 1 ); 1031 | next; 1032 | } 1033 | if ( $shour !~ /^\d{1,2}$/ ) { 1034 | print "Unrecognized start hour: '$shour', skipping line $linecount\n" if ( $verbose >= 1 ); 1035 | next; 1036 | } 1037 | if ( $smin !~ /^\d{1,2}$/ ) { 1038 | print "Unrecognized start min: '$smin', skipping line $linecount\n" if ( $verbose >= 1 ); 1039 | next; 1040 | } 1041 | if ( $ssec !~ /^\d{1,2}\.?\d{0,6}?$/ ) { 1042 | print "Unrecognized start seconds: '$ssec', skipping line $linecount\n" if ( $verbose >= 1 ); 1043 | next; 1044 | } 1045 | if ( $eyear !~ /^\d\d\d\d$/ ) { 1046 | print "Unrecognized end year: '$eyear', skipping line $linecount\n" if ( $verbose >= 1 ); 1047 | next; 1048 | } 1049 | if ( $emon !~ /^\d{1,2}$/ ) { 1050 | print "Unrecognized end month: '$emon', skipping line $linecount\n" if ( $verbose >= 1 ); 1051 | next; 1052 | } 1053 | if ( $eday !~ /^\d{1,2}$/ ) { 1054 | print "Unrecognized end day: '$eday', skipping line $linecount\n" if ( $verbose >= 1 ); 1055 | next; 1056 | } 1057 | if ( $ehour !~ /^\d{1,2}$/ ) { 1058 | print "Unrecognized end hour: '$ehour', skipping line $linecount\n" if ( $verbose >= 1 ); 1059 | next; 1060 | } 1061 | if ( $emin !~ /^\d{1,2}$/ ) { 1062 | print "Unrecognized end min: '$emin', skipping line $linecount\n" if ( $verbose >= 1 ); 1063 | next; 1064 | } 1065 | if ( $esec !~ /^\d{1,2}\.?\d{0,6}?$/ ) { 1066 | print "Unrecognized end seconds: '$esec', skipping line $linecount\n" if ( $verbose >= 1 ); 1067 | next; 1068 | } 1069 | if ( $count !~ /^\d+$/ || $count <= 0 ) { 1070 | print "Invalid channel count field: '$count', skipping line $linecount\n" if ( $verbose >= 1 ); 1071 | next; 1072 | } 1073 | if ( scalar @chans <= 0 ) { 1074 | print "No channels specified, skipping line $linecount\n" if ( $verbose >= 1 ); 1075 | next; 1076 | } 1077 | 1078 | # Extract location ID if present, i.e. if channel count is one less than present 1079 | my $loc = undef; 1080 | $loc = pop @chans if ( scalar @chans == ($count+1) ); 1081 | 1082 | if ( $loc && $loc !~ /^[A-Za-z0-9*?\-]{1,2}$/ ) { 1083 | print "Unrecognized location ID: '$loc', skipping line $linecount\n" if ( $verbose >= 1 ); 1084 | next; 1085 | } 1086 | 1087 | foreach my $chan ( @chans ) { 1088 | if ( $chan !~ /^[A-Za-z0-9*?]{1,3}$/ ) { 1089 | print "Unrecognized channel codes: '$chan', skipping line $linecount\n" if ( $verbose >= 1 ); 1090 | next BFLINE; 1091 | } 1092 | } 1093 | 1094 | if ( scalar @chans != $count ) { 1095 | printf "Channel count field ($count) does not match number of channels specified (%d), skipping line $linecount\n", 1096 | scalar @chans if ( $verbose >= 1 ); 1097 | next; 1098 | } 1099 | 1100 | # Normalize time strings 1101 | my ($ssec,$ssub) = split (/\./, $ssec); 1102 | my $start = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $syear, $smon, $sday, $shour, $smin, $ssec); 1103 | $start .= ".$ssub" if ( $ssub ); 1104 | my ($esec,$esub) = split (/\./, $esec); 1105 | my $end = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $eyear, $emon, $eday, $ehour, $emin, $esec); 1106 | $end .= ".$esub" if ( $esub ); 1107 | 1108 | # Add selection to global list for each channel 1109 | foreach my $chan ( @chans ) { 1110 | push (@selections,"$net|$sta|$loc|$chan|$start|$end"); 1111 | } 1112 | } 1113 | 1114 | close BF; 1115 | } # End of ReadBFastFile() 1116 | 1117 | 1118 | ###################################################################### 1119 | # FederateRequest: 1120 | # 1121 | # Submit selections to federation catalog service and parse response 1122 | # into per-data center requests. 1123 | # 1124 | # This will populate the following %datacenter components: 1125 | # %datacenter{DATACENTER}{website} 1126 | # %datacenter{DATACENTER}{stationws} 1127 | # %datacenter{DATACENTER}{dataselectws} 1128 | # %datacenter{DATACENTER}{sacpzws} 1129 | # %datacenter{DATACENTER}{respws} 1130 | # %datacenter{DATACENTER}{selection} 1131 | # 1132 | # Return count of unique data centers identified in response on 1133 | # success and undef on error. 1134 | ###################################################################### 1135 | sub FederateRequest { 1136 | my $fedcatws = shift; 1137 | my $selectionref = shift; 1138 | 1139 | my $datacentercount = 0; 1140 | 1141 | # Create HTTP user agent 1142 | my $ua = RequestAgent->new(); 1143 | $ua->env_proxy; 1144 | 1145 | # Create web service URI 1146 | my $uri = $fedcatws . "/query"; 1147 | 1148 | # Create POST data selection 1149 | my $postdata = ""; 1150 | 1151 | if ( length($federate) > 0 ) { 1152 | $postdata .= "datacenter=$federate\n"; 1153 | } 1154 | if ( scalar @latrange ) { 1155 | $postdata .= "minlatitude=$latrange[0]\n" if ( defined $latrange[0] ); 1156 | $postdata .= "maxlatitude=$latrange[1]\n" if ( defined $latrange[1] ); 1157 | } 1158 | if ( scalar @lonrange ) { 1159 | $postdata .= "minlongitude=$lonrange[0]\n" if ( defined $lonrange[0] ); 1160 | $postdata .= "maxlongitude=$lonrange[1]\n" if ( defined $lonrange[1] ); 1161 | } 1162 | if ( scalar @degrange ) { 1163 | $postdata .= "latitude=$degrange[0]\n" if ( defined $degrange[0] ); 1164 | $postdata .= "longitude=$degrange[1]\n" if ( defined $degrange[1] ); 1165 | $postdata .= "maxradius=$degrange[2]\n" if ( defined $degrange[2] ); 1166 | $postdata .= "minradius=$degrange[3]\n" if ( defined $degrange[3] ); 1167 | } 1168 | 1169 | # Translate selections to POST body repeat lines and build selection hash for matching 1170 | foreach my $selection ( @{$selectionref} ) { 1171 | my ($snet,$ssta,$sloc,$schan,$sstart,$send) = split (/\|/,$selection); 1172 | 1173 | # Subsitute non-specified fields with wildcards 1174 | $snet = "*" if ( ! $snet ); 1175 | $ssta = "*" if ( ! $ssta ); 1176 | $sloc = "*" if ( ! $sloc ); 1177 | $schan = "*" if ( ! $schan ); 1178 | my $pstart = ( $sstart ) ? $sstart : "*"; 1179 | my $pend = ( $send ) ? $send : "*"; 1180 | 1181 | $postdata .= "$snet $ssta $sloc $schan $pstart $pend\n"; 1182 | } 1183 | 1184 | my $ftime = Time::HiRes::time; 1185 | 1186 | print STDERR "Federator catalog URI: '$uri'\n" if ( $verbose >= 2 ); 1187 | print STDERR "Federator catalog (POST):\n$postdata" if ( $verbose > 1 ); 1188 | 1189 | print STDERR "Fetching federator catalog results :: " if ( $verbose >= 1 ); 1190 | 1191 | $datasize = 0; 1192 | $metadataresponse = ""; 1193 | 1194 | # Fetch metadata from web service using callback routine 1195 | my $response = ( $inflater ) ? 1196 | $ua->post($uri, 'Accept-Encoding' => 'gzip', Content => $postdata, ':content_cb' => \&MDCallBack ) : 1197 | $ua->post($uri, Content => $postdata, ':content_cb' => \&MDCallBack ); 1198 | 1199 | $inflater->inflateReset if ( $inflater ); 1200 | 1201 | if ( $response->code == 204 ) { 1202 | print (STDERR "No federator catalog results available\n") if ( $verbose >= 1 ); 1203 | return; 1204 | } 1205 | elsif ( ! $response->is_success() ) { 1206 | print (STDERR "Error fetching federator catalog result: " 1207 | . $response->code . " :: " . status_message($response->code) . "\n"); 1208 | print STDERR "------\n" . $response->decoded_content . "\n------\n"; 1209 | print STDERR " URI: '$uri'\n" if ( $verbose >= 2 ); 1210 | 1211 | $exitvalue = 1; 1212 | } 1213 | else { 1214 | printf (STDERR "%s\n", ($nobsprint)?sizestring($datasize):"") if ( $verbose >= 1 ); 1215 | } 1216 | 1217 | printf STDERR "Federator response code: %d\n", $response->code if ( $verbose >= 1 ); 1218 | if ( $verbose >= 2 ) { 1219 | print STDERR "Federator response:\n"; 1220 | print STDERR "-----------\n$metadataresponse\n-----------\n"; 1221 | } 1222 | 1223 | my $duration = Time::HiRes::time - $ftime; 1224 | my $rate = $datasize/(($duration)?$duration:0.000001); 1225 | printf (STDERR "Received %s from federator catalog in %.1f seconds (%s/s)\n", 1226 | sizestring($datasize), $duration, sizestring($rate)) if ( $verbose >= 0 ); 1227 | 1228 | # Parse response from catalog service 1229 | # Selections: "net|sta|loc|chan|starttime|endtime" 1230 | my $dckey = undef; 1231 | foreach my $line ( split (/[\n\r]+/, $metadataresponse) ) { 1232 | chomp $line; 1233 | 1234 | next if ( $line =~ /^#.*/ ); # Skip comment lines beginning with '#' 1235 | 1236 | # Reset data center parsing on empty line 1237 | if ( ! $line ) { 1238 | $dckey = undef; 1239 | next; 1240 | } 1241 | 1242 | my ($key,$website) = $line =~ /^DATACENTER\=([^,]+)\,(.*)$/; 1243 | my ($stationws) = $line =~ /^STATIONSERVICE\=(.+)$/; 1244 | my ($dataselectws) = $line =~ /^DATASELECTSERVICE\=(.+)$/; 1245 | my ($sacpzws) = $line =~ /^SACPZSERVICE\=(.+)$/; 1246 | my ($respws) = $line =~ /^RESPSERVICE\=(.+)$/; 1247 | 1248 | if ( $key ) { 1249 | $dckey = $key; 1250 | $datacenter{$dckey}{website} = $website; 1251 | $datacentercount++; 1252 | } 1253 | elsif ( $stationws ) { 1254 | if ( $dckey ) { 1255 | $stationws =~ s/\/$//; # Trim trailing slash 1256 | $datacenter{$dckey}{metadataws} = $stationws; 1257 | } 1258 | else { 1259 | print STDERR "Federation catalog service returned STATIONSERVICE without DATACENTER declared\n"; 1260 | return undef; 1261 | } 1262 | } 1263 | elsif ( $dataselectws ) { 1264 | if ( $dckey ) { 1265 | $dataselectws =~ s/\/$//; # Trim trailing slash 1266 | $datacenter{$dckey}{timeseriesws} = $dataselectws; 1267 | } 1268 | else { 1269 | print STDERR "Federation catalog service returned DATASELECTSERVICE without DATACENTER declared\n"; 1270 | return undef; 1271 | } 1272 | } 1273 | elsif ( $sacpzws ) { 1274 | if ( $dckey ) { 1275 | $sacpzws =~ s/\/$//; # Trim trailing slash 1276 | $datacenter{$dckey}{sacpzws} = $sacpzws; 1277 | } 1278 | else { 1279 | print STDERR "Federation catalog service returned SACPZSERVICE without DATACENTER declared\n"; 1280 | return undef; 1281 | } 1282 | } 1283 | elsif ( $respws ) { 1284 | if ( $dckey ) { 1285 | $respws =~ s/\/$//; # Trim trailing slash 1286 | $datacenter{$dckey}{respws} = $respws; 1287 | } 1288 | else { 1289 | print STDERR "Federation catalog service returned RESPSERVICE without DATACENTER declared\n"; 1290 | return undef; 1291 | } 1292 | } 1293 | # Ignore any other service declarations 1294 | elsif ( $line =~ /^.*SERVICE\=.+$/ ) 1295 | { 1296 | print STDERR "Unused service declaration: $line\n" if ( $verbose >= 2 ); 1297 | } 1298 | # Ignore key=value parameters 1299 | elsif ( $line =~ /^[\-\.\w]+\=[\-\.\w]+$/ ) 1300 | { 1301 | print STDERR "Unused key=value: $line\n" if ( $verbose >= 2 ); 1302 | } 1303 | # All other lines should be selection lines 1304 | else { 1305 | my ($net,$sta,$loc,$chan,$start,$end) = split (/\s+/, $line); 1306 | 1307 | if ( ! defined $end ) { 1308 | print STDERR "Federation catalog service returned unrecognized selection line:\n'$line'\n"; 1309 | return undef; 1310 | } 1311 | 1312 | # Add to data center selection list 1313 | if ( $dckey ) { 1314 | push ( @{$datacenter{$dckey}{selection}}, "$net|$sta|$loc|$chan|$start|$end"); 1315 | } 1316 | else { 1317 | print STDERR "Federation catalog service returned selecion line without DATACENTER declared\n"; 1318 | return undef; 1319 | } 1320 | } 1321 | } # Done parsing federator catalog response 1322 | 1323 | return $datacentercount; 1324 | } # End of FederateRequest 1325 | 1326 | 1327 | ###################################################################### 1328 | # FetchTimeSeriesData: 1329 | # 1330 | # Collect time series data for each entry in the %request hash. All 1331 | # returned data is written to the global output file (outfile). 1332 | # 1333 | # The request list is separatated into groups where the group size is 1334 | # defined in terms of station-days. If the request for a group fails 1335 | # it will be retried, after too many failures we give up. 1336 | # 1337 | ###################################################################### 1338 | sub FetchTimeSeriesData { 1339 | my $dckey = shift; 1340 | my $outfilename = shift; 1341 | my $outfilemode = shift; 1342 | 1343 | # Open output file with specified name and mode 1344 | if ( $outfilename ne "-" ) { 1345 | open (OUT, $outfilemode, $outfilename) || die "Cannot open output file '$outfilename': $!\n"; 1346 | } 1347 | else { 1348 | open (OUT, ">&:raw", \*STDOUT) || die "Cannot open STDOUT: $!\n"; 1349 | } 1350 | 1351 | my $count = 0; 1352 | 1353 | # Determine request data groups to avoid single large requests, 1354 | # this is done for two reasons: 1355 | # 1) To facilitate re-starting of requests after broken connections 1356 | # wihout needing re-submit the entire request 1357 | # 2) Avoid service timeouts 1358 | my @grouprequest = (); 1359 | 1360 | my $groupdays = 0; 1361 | my $groupidx = 0; 1362 | my $groupsta = undef; 1363 | foreach my $req ( sort keys %{$datacenter{$dckey}{request}} ) { 1364 | my ($wnet,$wsta,$wloc,$wchan,$wstart,$wend) = split (/\|/, $req); 1365 | $count++; 1366 | 1367 | # Determine day coverage for this request 1368 | my $rstartepoch = str2time ($wstart, "UTC"); 1369 | my $rendepoch = str2time ($wend, "UTC"); 1370 | my $reqdays = int ((($rendepoch - $rstartepoch) / 86400.0) + 0.5); 1371 | $reqdays = 1 if ( $reqdays < 1 ); 1372 | 1373 | $groupsta = $wsta if ( ! defined $groupsta ); 1374 | 1375 | # Assume first request for a station represents all channels in terms of days 1376 | if ( $wsta ne $groupsta ) { 1377 | $groupdays += $reqdays; 1378 | $groupsta = $wsta; 1379 | } 1380 | 1381 | # If beyond groupstadays move to the next group 1382 | if ( $groupdays >= $groupstadays ) { 1383 | $groupdays = 0; 1384 | $groupidx++; 1385 | } 1386 | 1387 | # Add request to current group 1388 | push (@{$grouprequest[$groupidx]}, "$wnet $wsta $wloc $wchan $wstart $wend"); 1389 | } 1390 | 1391 | if ( ! $count ) { 1392 | print STDERR "No data selections to request\n"; 1393 | return; 1394 | } 1395 | 1396 | print STDERR "Fetching time series data ($count selections)\n" if ( $verbose >= 1 ); 1397 | my $ftime = Time::HiRes::time; 1398 | my $totalbytes = 0; 1399 | my $requestfinish = undef; 1400 | 1401 | # Request each data group 1402 | my $groupnum = 1; 1403 | my $groupcnt = scalar @grouprequest; 1404 | my $fetchcnt = 1; 1405 | my $outoffset = 0; 1406 | foreach my $groupref ( @grouprequest ) { 1407 | REDOGROUP: 1408 | # Create web service URI 1409 | my $query = ( $auth ) ? "queryauth" : "query"; 1410 | my $uri = $datacenter{$dckey}{timeseriesws} . "/$query"; 1411 | 1412 | # Create POST data selection: specify options followed by selections 1413 | my $postdata = ""; 1414 | $postdata .= "quality=$qual\n" if ( defined $qual ); 1415 | $postdata .= "minimumlength=$mslopt\n" if ( defined $mslopt ); 1416 | $postdata .= "longestonly=true\n" if ( defined $lsoopt ); 1417 | 1418 | foreach my $req ( @{$groupref} ) { 1419 | $postdata .= "$req\n"; 1420 | } 1421 | 1422 | # Impose the minimum request interval, potentially throttling loop 1423 | if (defined $requestfinish && (Time::HiRes::time - $requestfinish) < $minimumrequestinterval) { 1424 | my $remaining = $minimumrequestinterval - (Time::HiRes::time - $requestfinish); 1425 | usleep ($remaining * 1E6) if ( $remaining > 0 ); 1426 | } 1427 | 1428 | print STDERR "Time series URI: '$uri'\n" if ( $verbose > 1 ); 1429 | print STDERR "Data selection (POST):\n$postdata" if ( $verbose > 1 ); 1430 | 1431 | print STDERR "Downloading time series data (group $groupnum of $groupcnt) :: " if ( $verbose >= 1 ); 1432 | 1433 | $datasize = 0; 1434 | 1435 | # Create HTTP user agent 1436 | my $ua = RequestAgent->new(); 1437 | $ua->env_proxy; 1438 | 1439 | # Fetch time series data from web service using callback routine 1440 | my $response = $ua->post($uri, Content => $postdata, ':content_cb' => \&DLCallBack_NoGZIP ); 1441 | 1442 | $requestfinish = Time::HiRes::time; 1443 | 1444 | if ( $response->code == 204 ) { 1445 | print (STDERR "No data available\n") if ( $verbose >= 1 ); 1446 | } 1447 | elsif ( $response->code == 401 ) { 1448 | # If this is the first authentication failure try one more time 1449 | if ( $fetchcnt == 1 ) { 1450 | $fetchcnt++; 1451 | goto REDOGROUP; 1452 | } 1453 | 1454 | print (STDERR "AUTHORIZATION FAILED, username and password not recognized\n"); 1455 | last; 1456 | } 1457 | elsif ( ! $response->is_success() ) { 1458 | print (STDERR "Error fetching time series data: " 1459 | . $response->code . " :: " . $response->message . "\n"); 1460 | 1461 | if ( $response->code == 429 ) { 1462 | print STDERR "Usage has exceeded data center limit, try making fewer concurrent requests\n\n"; 1463 | } 1464 | 1465 | print STDERR "------\n" . $response->decoded_content . "\n------\n"; 1466 | print STDERR " URI: '$uri'\n" if ( $verbose > 1 ); 1467 | 1468 | # Exit immediately if we are not retrying 1469 | exit 1 if ( $noretry ); 1470 | 1471 | # Exit on fatal 400 (Bad Request) and 413 (Request Entity Too Large) 1472 | exit 1 if ( $response->code == 400 || $response->code == 413 ); 1473 | 1474 | # For real output files rewind position to the end of the last group data 1475 | seek (OUT, $outoffset, 0) if ( $outfile ne "-" ); 1476 | 1477 | # Retry in 10 seconds or give up if already tried 10 times. 1478 | if ( $fetchcnt < $retries ) { 1479 | print STDERR "Retrying request in 10 seconds ($fetchcnt of $retries retries)\n"; 1480 | sleep 10; 1481 | $fetchcnt++; 1482 | goto REDOGROUP; 1483 | } 1484 | else { 1485 | print STDERR "Too many retries, giving up.\n"; 1486 | last; 1487 | } 1488 | } 1489 | else { 1490 | printf (STDERR "%s\n", ($nobsprint)?sizestring($datasize):"") if ( $verbose >= 1 ); 1491 | } 1492 | 1493 | # Get ready for next group 1494 | $fetchcnt = 1; 1495 | $groupnum++; 1496 | $outoffset = tell (OUT); 1497 | $totalbytes += $datasize; 1498 | } 1499 | 1500 | close OUT; 1501 | 1502 | my $duration = Time::HiRes::time - $ftime; 1503 | my $rate = $totalbytes/(($duration)?$duration:0.000001); 1504 | printf (STDERR "Received %s of time series data in %.1f seconds (%s/s) - written to %s\n", 1505 | sizestring($totalbytes), $duration, sizestring($rate), $outfilename) if ( $verbose >= 0 ); 1506 | 1507 | # Remove empty file 1508 | unlink $outfile if ( -z $outfile ); 1509 | } # End of FetchTimeSeriesData 1510 | 1511 | 1512 | ###################################################################### 1513 | # FetchSACPZ: 1514 | # 1515 | # Fetch SAC Poles and Zeros for each entry in the %metarequest hash 1516 | # with a defined value. The result for each channel is written to a 1517 | # separate file in the specified directory. 1518 | # 1519 | ###################################################################### 1520 | sub FetchSACPZ { 1521 | my $dckey = shift; 1522 | my $dcdir = shift; 1523 | 1524 | # Create HTTP user agent 1525 | my $ua = RequestAgent->new(); 1526 | $ua->env_proxy; 1527 | 1528 | my $count = 0; 1529 | my $total = 0; 1530 | foreach my $req ( keys %{$datacenter{$dckey}{metarequest}} ) 1531 | { $total++ if ( defined $datacenter{$dckey}{metarequest}->{$req} ); } 1532 | 1533 | print STDERR "Fetching SAC Poles and Zeros, writing to '$dcdir'\n" if ( $verbose >= 1 ); 1534 | my $ftime = Time::HiRes::time; 1535 | my $totalbytes = 0; 1536 | my $requestfinish = undef; 1537 | 1538 | foreach my $req ( sort keys %{$datacenter{$dckey}{metarequest}} ) { 1539 | # Skip entries with values not defined, perhaps no data was fetched 1540 | next if ( ! defined $datacenter{$dckey}{metarequest}->{$req} ); 1541 | 1542 | my ($rnet,$rsta,$rloc,$rchan) = split (/\|/, $req); 1543 | my ($mstart,$mend) = split (/\|/, $datacenter{$dckey}{metarequest}->{$req}); 1544 | 1545 | # Create time strings for request, shrink window by one second to avoid 1546 | # matching too many metadata ranges by avoiding the boundary. 1547 | my $rstart = &mktimestring ($mstart + 1); 1548 | my $rend = &mktimestring ($mend - 1); 1549 | $count++; 1550 | 1551 | # Generate output file name and open 1552 | my $sacpzfile = "$dcdir/SACPZ.$rnet.$rsta.$rloc.$rchan"; 1553 | if ( ! open (OUT, ">:raw", $sacpzfile) ) { 1554 | print STDERR "Cannot open output file '$sacpzfile': $!\n"; 1555 | next; 1556 | } 1557 | 1558 | # Create web service URI 1559 | my $uri = $datacenter{$dckey}{sacpzws} . "/query?net=$rnet&sta=$rsta&loc=$rloc&cha=$rchan"; 1560 | $uri .= "&starttime=$rstart" if ( $rstart ); 1561 | $uri .= "&endtime=$rend" if ( $rend ); 1562 | 1563 | # Impose the minimum request interval, potentially throttling loop 1564 | if (defined $requestfinish && (Time::HiRes::time - $requestfinish) < $minimumrequestinterval) { 1565 | my $remaining = $minimumrequestinterval - (Time::HiRes::time - $requestfinish); 1566 | usleep ($remaining * 1E6) if ( $remaining > 0 ); 1567 | } 1568 | 1569 | print STDERR "SAC-PZ URI: '$uri'\n" if ( $verbose > 1 ); 1570 | 1571 | print STDERR "Downloading $sacpzfile ($count/$total) :: " if ( $verbose >= 1 ); 1572 | 1573 | $datasize = 0; 1574 | 1575 | # Fetch data from web service using callback routine 1576 | my $response = ( $inflater ) ? 1577 | $ua->get($uri, 'Accept-Encoding' => 'gzip', ':content_cb' => \&DLCallBack_GZIP ) : 1578 | $ua->get($uri, ':content_cb' => \&DLCallBack_NoGZIP ); 1579 | 1580 | $requestfinish = Time::HiRes::time; 1581 | 1582 | $inflater->inflateReset if ( $inflater ); 1583 | 1584 | if ( $response->code == 404 || $response->code == 204 ) { 1585 | print (STDERR "No data available\n") if ( $verbose >= 1 ); 1586 | } 1587 | elsif ( ! $response->is_success() ) { 1588 | print (STDERR "Error fetching SAC PZ data: " 1589 | . $response->code . " :: " . status_message($response->code) . "\n"); 1590 | print STDERR "------\n" . $response->decoded_content . "\n------\n"; 1591 | print STDERR " URI: '$uri'\n" if ( $verbose > 1 ); 1592 | 1593 | $exitvalue = 1; 1594 | } 1595 | else { 1596 | printf (STDERR "%s\n", ($nobsprint)?sizestring($datasize):"") if ( $verbose >= 1 ); 1597 | } 1598 | 1599 | # Add data bytes to global total 1600 | $totalbytes += $datasize; 1601 | 1602 | close OUT; 1603 | 1604 | # Remove file if no data was fetched 1605 | unlink $sacpzfile if ( $datasize == 0 ); 1606 | } 1607 | 1608 | my $duration = Time::HiRes::time - $ftime; 1609 | my $rate = $totalbytes/(($duration)?$duration:0.000001); 1610 | printf (STDERR "Received %s of SAC P&Zs in %.1f seconds (%s/s)\n", 1611 | sizestring($totalbytes), $duration, sizestring($rate)) if ( $verbose >= 0 ); 1612 | 1613 | } # End of FetchSACPZ 1614 | 1615 | 1616 | ###################################################################### 1617 | # FetchRESP: 1618 | # 1619 | # Fetch SEED RESP for each entry in the %metarequest hash with a value 1620 | # of 1. The result for each channel is written to a separate file in 1621 | # the specified directory. 1622 | # 1623 | ###################################################################### 1624 | sub FetchRESP { 1625 | my $dckey = shift; 1626 | my $dcdir = shift; 1627 | 1628 | # Create HTTP user agent 1629 | my $ua = RequestAgent->new(); 1630 | $ua->env_proxy; 1631 | 1632 | my $count = 0; 1633 | my $total = 0; 1634 | foreach my $req ( keys %{$datacenter{$dckey}{metarequest}} ) 1635 | { $total++ if ( defined $datacenter{$dckey}{metarequest}->{$req} ); } 1636 | 1637 | print STDERR "Fetching RESP, writing to '$dcdir'\n" if ( $verbose >= 1 ); 1638 | my $ftime = Time::HiRes::time; 1639 | my $totalbytes = 0; 1640 | my $requestfinish = undef; 1641 | 1642 | foreach my $req ( sort keys %{$datacenter{$dckey}{metarequest}} ) { 1643 | # Skip entries with values not defined, perhaps no data was fetched 1644 | next if ( ! defined $datacenter{$dckey}{metarequest}->{$req} ); 1645 | 1646 | my ($rnet,$rsta,$rloc,$rchan) = split (/\|/, $req); 1647 | my ($mstart,$mend) = split (/\|/, $datacenter{$dckey}{metarequest}->{$req}); 1648 | 1649 | # Create time strings for request, shrink window by one second to avoid 1650 | # matching too many metadata ranges by avoiding the boundary. 1651 | my $rstart = &mktimestring ($mstart + 1); 1652 | my $rend = &mktimestring ($mend - 1); 1653 | $count++; 1654 | 1655 | # Translate metadata location ID from "--" to blank 1656 | my $ploc = ( $rloc eq "--" ) ? "" : $rloc; 1657 | 1658 | # Generate output file name and open 1659 | my $respfile = "$dcdir/RESP.$rnet.$rsta.$ploc.$rchan"; 1660 | if ( ! open (OUT, ">:raw", $respfile) ) { 1661 | print STDERR "Cannot open output file '$respfile': $!\n"; 1662 | next; 1663 | } 1664 | 1665 | # Create web service URI 1666 | my $uri = $datacenter{$dckey}{respws} . "/query?net=$rnet&sta=$rsta&loc=$rloc&cha=$rchan"; 1667 | $uri .= "&starttime=$rstart" if ( $rstart ); 1668 | $uri .= "&endtime=$rend" if ( $rend ); 1669 | 1670 | # Impose the minimum request interval, potentially throttling loop 1671 | if (defined $requestfinish && (Time::HiRes::time - $requestfinish) < $minimumrequestinterval) { 1672 | my $remaining = $minimumrequestinterval - (Time::HiRes::time - $requestfinish); 1673 | usleep ($remaining * 1E6) if ( $remaining > 0 ); 1674 | } 1675 | 1676 | print STDERR "RESP URI: '$uri'\n" if ( $verbose > 1 ); 1677 | 1678 | print STDERR "Downloading $respfile ($count/$total) :: " if ( $verbose >= 1 ); 1679 | 1680 | $datasize = 0; 1681 | 1682 | # Fetch data from web service using callback routine 1683 | my $response = ( $inflater ) ? 1684 | $ua->get($uri, 'Accept-Encoding' => 'gzip', ':content_cb' => \&DLCallBack_GZIP ) : 1685 | $ua->get($uri, ':content_cb' => \&DLCallBack_NoGZIP ); 1686 | 1687 | $requestfinish = Time::HiRes::time; 1688 | 1689 | $inflater->inflateReset if ( $inflater ); 1690 | 1691 | if ( $response->code == 404 || $response->code == 204 ) { 1692 | print (STDERR "No data available\n") if ( $verbose >= 1 ); 1693 | } 1694 | elsif ( ! $response->is_success() ) { 1695 | print (STDERR "Error fetching RESP data: " 1696 | . $response->code . " :: " . status_message($response->code) . "\n"); 1697 | print STDERR "------\n" . $response->decoded_content . "\n------\n"; 1698 | print STDERR " URI: '$uri'\n" if ( $verbose > 1 ); 1699 | 1700 | $exitvalue = 1; 1701 | } 1702 | else { 1703 | printf (STDERR "%s\n", ($nobsprint)?sizestring($datasize):"") if ( $verbose >= 1 ); 1704 | } 1705 | 1706 | # Add data bytes to global total 1707 | $totalbytes += $datasize; 1708 | 1709 | close OUT; 1710 | 1711 | # Remove file if no data was fetched 1712 | unlink $respfile if ( $datasize == 0 ); 1713 | } 1714 | 1715 | my $duration = Time::HiRes::time - $ftime; 1716 | my $rate = $totalbytes/(($duration)?$duration:0.000001); 1717 | printf (STDERR "Received %s of RESP in %.1f seconds (%s/s)\n", 1718 | sizestring($totalbytes), $duration, sizestring($rate)) if ( $verbose >= 0 ); 1719 | 1720 | } # End of FetchRESP 1721 | 1722 | 1723 | ###################################################################### 1724 | # FetchStationXML: 1725 | # 1726 | # Fetch StationXML for each entry in the %metarequest hash with a value 1727 | # of 1. The result for each channel is written to the specified file. 1728 | # 1729 | ###################################################################### 1730 | sub FetchStationXML { 1731 | my $dckey = shift; 1732 | my $dcsxmlfile = shift; 1733 | 1734 | # Create HTTP user agent 1735 | my $ua = RequestAgent->new(); 1736 | $ua->env_proxy; 1737 | 1738 | # Create web service URI 1739 | my $uri = $datacenter{$dckey}{metadataws} . "/query"; 1740 | 1741 | # Create POST data selection: specify options followed by selections 1742 | my $postdata = "level=response\n"; 1743 | $postdata .= "format=xml\n"; 1744 | 1745 | foreach my $req ( sort keys %{$datacenter{$dckey}{metarequest}} ) { 1746 | # Skip entries with values not defined, perhaps no data was fetched 1747 | next if ( ! defined $datacenter{$dckey}{metarequest}->{$req} ); 1748 | 1749 | my ($rnet,$rsta,$rloc,$rchan) = split (/\|/, $req); 1750 | my ($mstart,$mend) = split (/\|/, $datacenter{$dckey}{metarequest}->{$req}); 1751 | 1752 | # Create time strings for request, shrink window by one second to avoid 1753 | # matching too many metadata ranges by avoiding the boundary. 1754 | my $rstart = &mktimestring ($mstart + 1); 1755 | my $rend = &mktimestring ($mend - 1); 1756 | 1757 | $postdata .= "$rnet $rsta $rloc $rchan $rstart $rend\n"; 1758 | } 1759 | 1760 | if ( ! open (OUT, ">:raw", $dcsxmlfile) ) { 1761 | print STDERR "Cannot open output file '$dcsxmlfile': $!\n"; 1762 | return; 1763 | } 1764 | 1765 | print STDERR "StationXML URI: '$uri'\n" if ( $verbose >= 2 ); 1766 | print STDERR "StationXML selection (POST):\n$postdata" if ( $verbose > 1 ); 1767 | 1768 | print STDERR "Fetching StationXML ($dcsxmlfile) :: " if ( $verbose >= 1 ); 1769 | my $ftime = Time::HiRes::time; 1770 | 1771 | $datasize = 0; 1772 | 1773 | # Fetch StationXML from web service using callback routine 1774 | my $response = ( $inflater ) ? 1775 | $ua->post($uri, 'Accept-Encoding' => 'gzip', Content => $postdata, ':content_cb' => \&DLCallBack_GZIP ) : 1776 | $ua->post($uri, Content => $postdata, ':content_cb' => \&DLCallBack_NoGZIP ); 1777 | 1778 | $inflater->inflateReset if ( $inflater ); 1779 | 1780 | if ( $response->code == 404 || $response->code == 204 ) { 1781 | print (STDERR "No StationXML available\n") if ( $verbose >= 1 ); 1782 | } 1783 | elsif ( ! $response->is_success() ) { 1784 | print (STDERR "Error fetching StationXML: " 1785 | . $response->code . " :: " . status_message($response->code) . "\n"); 1786 | print STDERR "------\n" . $response->decoded_content . "\n------\n"; 1787 | print STDERR " URI: '$uri'\n" if ( $verbose >= 2 ); 1788 | 1789 | $exitvalue = 1; 1790 | } 1791 | else { 1792 | printf (STDERR "%s\n", ($nobsprint)?sizestring($datasize):"") if ( $verbose >= 1 ); 1793 | } 1794 | 1795 | my $duration = Time::HiRes::time - $ftime; 1796 | my $rate = $datasize/(($duration)?$duration:0.000001); 1797 | printf (STDERR "Received %s of StationXML in %.1f seconds (%s/s)\n", 1798 | sizestring($datasize), $duration, sizestring($rate)) if ( $verbose >= 0 ); 1799 | 1800 | close OUT; 1801 | 1802 | # Remove file if no data was fetched 1803 | unlink $dcsxmlfile if ( $datasize == 0 ); 1804 | 1805 | } # End of FetchStationXML 1806 | 1807 | ###################################################################### 1808 | # DLCallBack_NoGZIP: 1809 | # 1810 | # A call back for LWP downloading that passes the data directly to the 1811 | # low-level worker function with no detection of gzip encoding. 1812 | ###################################################################### 1813 | sub DLCallBack_NoGZIP { 1814 | # @_ :: 0=data, 1=response object, 2=protocol object 1815 | 1816 | &DLCallBack_Worker ($_[0], 0); 1817 | } 1818 | 1819 | 1820 | ###################################################################### 1821 | # DLCallBack_GZIP: 1822 | # 1823 | # A call back for LWP downloading that checks for gzip encoding and 1824 | # sets the inflation flag for the low-level worker function. 1825 | ###################################################################### 1826 | sub DLCallBack_GZIP { 1827 | # @_ :: 0=data, 1=response object, 2=protocol object 1828 | 1829 | my $inflateflag = ( $_[1]->content_encoding() =~ /gzip/ ); 1830 | 1831 | &DLCallBack_Worker ($_[0], $inflateflag); 1832 | } 1833 | 1834 | 1835 | ###################################################################### 1836 | # DLCallBack_Worker: 1837 | # 1838 | # A worker function used by the LWP call back routines. 1839 | # 1840 | # Write received data to output file, tally up the received data size 1841 | # and print and updated (overwriting) byte count string. 1842 | ###################################################################### 1843 | sub DLCallBack_Worker { 1844 | # @_ :: 0=data, 1=inflate flag 1845 | 1846 | $datasize += length($_[0]); 1847 | 1848 | if ( $_[1] ) { 1849 | my $datablock = ""; 1850 | $inflater->inflate($_[0], $datablock); 1851 | print OUT $datablock; 1852 | } 1853 | else { 1854 | print OUT $_[0]; 1855 | } 1856 | 1857 | if ( $verbose >= 1 && ! $nobsprint ) { 1858 | printf (STDERR "%-10.10s\b\b\b\b\b\b\b\b\b\b", sizestring($datasize)); 1859 | } 1860 | 1861 | # Detect stream truncation by checking for a trailing "#STREAMERROR" 1862 | if ( $_[0] =~ /#STREAMERROR$/ ) { 1863 | print STDERR "\nERROR: Stream truncated, download likely incomplete\n"; 1864 | 1865 | exit 1; 1866 | } 1867 | } 1868 | 1869 | 1870 | ###################################################################### 1871 | # FetchMetaData: 1872 | # 1873 | # Collect metadata and expand wildcards for selected data set. 1874 | # 1875 | # Resulting metadata is placed in the global @metadata array with each 1876 | # entry taking the following form: 1877 | # "net|sta|loc|chan|start|end|lat|lon|elev|depth|azimuth|dip|instrument|samplerate|sensitivity|sensfreq|sensunits" 1878 | # 1879 | # In addition, an entry for the unique NSLCQ time-window is added to 1880 | # the %request hash, used later to request data. The value of the 1881 | # request hash entries is maintained to be the range of Channel epochs 1882 | # that match the time selection. 1883 | # 1884 | # As an exception to fetching all metadata at once, any selection 1885 | # specified with a virtual network (starting with [_.~]) is fetched 1886 | # individually. This is needed to properly match the returned metadata 1887 | # (that does not contain virtual network codes) to the time range of 1888 | # the request. 1889 | # 1890 | ###################################################################### 1891 | sub FetchMetaData { 1892 | my $dckey = shift; 1893 | 1894 | my $mtime = Time::HiRes::time; 1895 | 1896 | # Split selections into lists for virtual networks and regular networks 1897 | # Requests including virtual networks are identified by searching 1898 | # for [_.~] in the first (network) field. 1899 | my @vnetselections = grep { (split(/\|/))[0] =~ /[\_\.\~]/ } @{$datacenter{$dckey}{selection}}; 1900 | my @netselections = grep { (split(/\|/))[0] !~ /[\_\.\~]/ } @{$datacenter{$dckey}{selection}}; 1901 | 1902 | my $totalepochs = 0; 1903 | 1904 | # Fetch metadata for virtual network requests individually 1905 | foreach my $selection ( @vnetselections ) { 1906 | $totalepochs += &FetchMetaDataHelper ($dckey, [$selection]); 1907 | } 1908 | 1909 | # Process all regular networks as a group 1910 | if ( scalar @netselections ) { 1911 | $totalepochs += &FetchMetaDataHelper ($dckey, \@netselections); 1912 | } 1913 | 1914 | my $duration = Time::HiRes::time - $mtime; 1915 | printf (STDERR "Processed metadata for $totalepochs channel epochs in %.1f seconds\n", 1916 | $duration) if ( $verbose >= 0 ); 1917 | 1918 | } # End of FetchMetaData 1919 | 1920 | 1921 | ###################################################################### 1922 | # FetchMetaDataHelper: 1923 | # 1924 | # Construct, issue and process request for metadata on behalf of 1925 | # FetchMetaData(). This will populate the global @metadata array and 1926 | # %request hash. 1927 | # 1928 | # Returns the total number of metadata epochs processed. 1929 | ###################################################################### 1930 | sub FetchMetaDataHelper { 1931 | my $dckey = shift; 1932 | my $selectionref = shift; 1933 | 1934 | # Create HTTP user agent 1935 | my $ua = RequestAgent->new(); 1936 | $ua->env_proxy; 1937 | 1938 | # Create web service URI 1939 | my $uri = $datacenter{$dckey}{metadataws} . "/query"; 1940 | 1941 | # Create POST data selection: specify options followed by selections 1942 | my $postdata = "level=channel\n"; 1943 | $postdata .= "format=text\n"; 1944 | 1945 | if ( scalar @latrange ) { 1946 | $postdata .= "minlatitude=$latrange[0]\n" if ( defined $latrange[0] ); 1947 | $postdata .= "maxlatitude=$latrange[1]\n" if ( defined $latrange[1] ); 1948 | } 1949 | if ( scalar @lonrange ) { 1950 | $postdata .= "minlongitude=$lonrange[0]\n" if ( defined $lonrange[0] ); 1951 | $postdata .= "maxlongitude=$lonrange[1]\n" if ( defined $lonrange[1] ); 1952 | } 1953 | if ( scalar @degrange ) { 1954 | $postdata .= "latitude=$degrange[0]\n" if ( defined $degrange[0] ); 1955 | $postdata .= "longitude=$degrange[1]\n" if ( defined $degrange[1] ); 1956 | $postdata .= "maxradius=$degrange[2]\n" if ( defined $degrange[2] ); 1957 | $postdata .= "minradius=$degrange[3]\n" if ( defined $degrange[3] ); 1958 | } 1959 | 1960 | # A nested hash used to match data requests with metadata 1961 | my %selectmatch = (); 1962 | 1963 | # A nested hash of time extents for each NSLC request 1964 | my %selectextent = (); 1965 | 1966 | # Translate selections to POST body repeat lines and build selection hash for matching 1967 | foreach my $selection ( @{$selectionref} ) { 1968 | my ($snet,$ssta,$sloc,$schan,$sstart,$send) = split (/\|/,$selection); 1969 | 1970 | # Substitute non-specified fields with wildcards 1971 | $snet = "*" if ( ! $snet ); 1972 | $ssta = "*" if ( ! $ssta ); 1973 | $sloc = "*" if ( ! $sloc ); 1974 | $schan = "*" if ( ! $schan ); 1975 | my $pstart = ( $sstart ) ? $sstart : "*"; 1976 | my $pend = ( $send ) ? $send : "*"; 1977 | 1978 | my $kstart = ( $sstart ) ? str2time ($sstart, "UTC") : undef; 1979 | my $kend = ( $send ) ? str2time ($send, "UTC") : undef; 1980 | 1981 | # Track time extents for each NSLC 1982 | if ( ! exists $selectextent{$snet}{$ssta}{$sloc}{$schan} ) { 1983 | $selectextent{$snet}{$ssta}{$sloc}{$schan} = [$pstart,$pend,$kstart,$kend]; 1984 | } 1985 | else { 1986 | my ($estart,$eend,$ekstart,$ekend) = @{$selectextent{$snet}{$ssta}{$sloc}{$schan}}; 1987 | 1988 | if ( ! defined $kstart || $kstart < $ekstart ) { 1989 | $ekstart = $kstart; 1990 | $estart = $pstart; 1991 | } 1992 | if ( ! defined $kend || $kend > $ekend ) { 1993 | $ekend = $kend; 1994 | $eend = $pend; 1995 | } 1996 | 1997 | $selectextent{$snet}{$ssta}{$sloc}{$schan} = [$estart,$eend,$ekstart,$ekend]; 1998 | } 1999 | 2000 | # Use '*' for virtual network matching, search for [_.~] to identify vnets 2001 | # The caller of this subroutine should guarantee only a single request 2002 | $snet = "*" if ( $snet =~ /[\_\.\~]/ ); 2003 | 2004 | # Nested hash keys: convert simple globbing wildcards (and list) to regex 2005 | # Translations: '*' => '.*' and '?' => '.' and ',' => '|' 2006 | $snet =~ s/\*/\.\*/g; $snet =~ s/\?/\./g; $snet =~ s/\,/\|/g; 2007 | $ssta =~ s/\*/\.\*/g; $ssta =~ s/\?/\./g; $ssta =~ s/\,/\|/g; 2008 | $sloc =~ s/\*/\.\*/g; $sloc =~ s/\?/\./g; $sloc =~ s/\,/\|/g; 2009 | $schan =~ s/\*/\.\*/g; $schan =~ s/\?/\./g; $schan =~ s/\,/\|/g; 2010 | 2011 | # Separate selections and exclusions. The below voodoo: 2012 | # split on | separator, select entries starting with/without - exclusion, 2013 | # wrap with ^$ terminators and rebuild expression. Finally, remove the - characters. 2014 | my $snetselect = join ('|', map { qq/\^$_\$/ } grep (!/^-/, split (/\|/,$snet))); 2015 | my $snetexclude = join ('|', map { qq/\^$_\$/ } grep (/^-/, split (/\|/,$snet))); 2016 | $snetexclude =~ s/\-//g; 2017 | 2018 | my $sstaselect = join ('|', map { qq/\^$_\$/ } grep (!/^-/, split (/\|/,$ssta))); 2019 | my $sstaexclude = join ('|', map { qq/\^$_\$/ } grep (/^-/, split (/\|/,$ssta))); 2020 | $sstaexclude =~ s/\-//g; 2021 | 2022 | my $slocselect = join ('|', map { qq/\^$_\$/ } grep (!/^-/, split (/\|/,$sloc))); 2023 | my $slocexclude = join ('|', map { qq/\^$_\$/ } grep (/^-/, split (/\|/,$sloc))); 2024 | $slocexclude =~ s/\-//g; 2025 | 2026 | my $schanselect = join ('|', map { qq/\^$_\$/ } grep (!/^-/, split (/\|/,$schan))); 2027 | my $schanexclude = join ('|', map { qq/\^$_\$/ } grep (/^-/, split (/\|/,$schan))); 2028 | $schanexclude =~ s/\-//g; 2029 | 2030 | # Compile regular expressions for faster comparison. 2031 | # Selections and exclusions are combined into a single expression as needed. 2032 | if ( $snetexclude && $snetselect ) { $snet = qr/(?!(${snetexclude}))(?=(${snetselect}))/ } 2033 | elsif ( $snetexclude ) { $snet = qr/(?!(${snetexclude}))/ } 2034 | else { $snet = qr/(${snetselect})/ } 2035 | 2036 | if ( $sstaexclude && $sstaselect ) { $ssta = qr/(?!(${sstaexclude}))(?=(${sstaselect}))/ } 2037 | elsif ( $sstaexclude ) { $ssta = qr/(?!(${sstaexclude}))/ } 2038 | else { $ssta = qr/(${sstaselect})/ } 2039 | 2040 | if ( $slocexclude && $slocselect ) { $sloc = qr/(?!(${slocexclude}))(?=(${slocselect}))/ } 2041 | elsif ( $slocexclude ) { $sloc = qr/(?!(${slocexclude}))/ } 2042 | else { $sloc = qr/(${slocselect})/ } 2043 | 2044 | if ( $schanexclude && $schanselect ) { $schan = qr/(?!(${schanexclude}))(?=(${schanselect}))/ } 2045 | elsif ( $schanexclude ) { $schan = qr/(?!(${schanexclude}))/ } 2046 | else { $schan = qr/(${schanselect})/ } 2047 | 2048 | # Add entry to selection matching hash, using hashes of regexes as keys 2049 | $selectmatch{$snet}{$ssta}{$sloc}{$schan}{"$kstart|$kend"} = "$sstart|$send"; 2050 | } 2051 | 2052 | # Build sorted POST selection from time extents for channels 2053 | foreach my $net ( sort keys %selectextent ) { 2054 | foreach my $sta ( sort keys %{$selectextent{$net}} ) { 2055 | foreach my $loc ( sort keys %{$selectextent{$net}{$sta}} ) { 2056 | foreach my $chan ( sort keys %{$selectextent{$net}{$sta}{$loc}} ) { 2057 | my ($start,$end) = @{$selectextent{$net}{$sta}{$loc}{$chan}}; 2058 | 2059 | $postdata .= "$net $sta $loc $chan $start $end\n"; 2060 | } 2061 | } 2062 | } 2063 | } 2064 | 2065 | my $ftime = Time::HiRes::time; 2066 | 2067 | print STDERR "Metadata URI: '$uri'\n" if ( $verbose >= 2 ); 2068 | print STDERR "Metadata selection (POST):\n$postdata" if ( $verbose > 1 ); 2069 | 2070 | print STDERR "Fetching metadata :: " if ( $verbose >= 1 ); 2071 | 2072 | $datasize = 0; 2073 | $metadataresponse = ""; 2074 | 2075 | # Fetch metadata from web service using callback routine 2076 | my $response = ( $inflater ) ? 2077 | $ua->post($uri, 'Accept-Encoding' => 'gzip', Content => $postdata, ':content_cb' => \&MDCallBack ) : 2078 | $ua->post($uri, Content => $postdata, ':content_cb' => \&MDCallBack ); 2079 | 2080 | $inflater->inflateReset if ( $inflater ); 2081 | 2082 | if ( $response->code == 204 ) { 2083 | print (STDERR "No metadata available\n") if ( $verbose >= 1 ); 2084 | return; 2085 | } 2086 | elsif ( ! $response->is_success() ) { 2087 | print (STDERR "Error fetching metadata: " 2088 | . $response->code . " :: " . status_message($response->code) . "\n"); 2089 | print STDERR "------\n" . $response->decoded_content . "\n------\n"; 2090 | print STDERR " URI: '$uri'\n" if ( $verbose >= 2 ); 2091 | 2092 | $exitvalue = 1; 2093 | } 2094 | else { 2095 | printf (STDERR "%s\n", ($nobsprint)?sizestring($datasize):"") if ( $verbose >= 1 ); 2096 | } 2097 | 2098 | my $duration = Time::HiRes::time - $ftime; 2099 | my $rate = $datasize/(($duration)?$duration:0.000001); 2100 | printf (STDERR "Received %s of metadata in %.1f seconds (%s/s)\n", 2101 | sizestring($datasize), $duration, sizestring($rate)) if ( $verbose >= 0 ); 2102 | 2103 | # Return if no metadata received 2104 | return if ( length $metadataresponse <= 0 ); 2105 | 2106 | my $totalepochs = 0; 2107 | my $ptime = Time::HiRes::time; 2108 | 2109 | print STDERR "Parsing metadata and generating requests... " if ( $verbose >= 1 ); 2110 | 2111 | # Matching request entries to metadata. 2112 | # 2113 | # This is a many-to-many relationship due to the wildcards, a single 2114 | # request entry can match multiple metadata entries and a single metadata 2115 | # entry can match multiple request entries. 2116 | # 2117 | # Loop through each metadata return line and do the following: 2118 | # 1) insert into metadata storage list 2119 | # 2) search for matching user selection entries and 2120 | # a) create request hash entries for each selection that matches using the 2121 | # original request time window. 2122 | # b) track the largest combined metadata time window for each channel 2123 | # for requesting secondary metadata (SACPZ and RESP). 2124 | 2125 | foreach my $line ( split (/[\n\r]+/, $metadataresponse) ) { 2126 | chomp $line; 2127 | next if ( $line =~ /^#.*/ ); # Skip comment lines beginning with '#' 2128 | 2129 | my ($mnet,$msta,$mloc,$mchan,$mlat,$mlon,$melev,$mdepth,$mazimuth,$mdip,$minstrument, 2130 | $mscale,$mscalefreq,$mscaleunits,$msamplerate,$mstart,$mend) = split(/\|/, $line); 2131 | 2132 | # Translate metadata location ID to "--" if it is spaces or empty 2133 | $mloc = "--" if ( $mloc eq " " || $mloc eq "" ); 2134 | 2135 | # Cleanup start and end strings, with truncation to 2038-01-01T00:00:00 for older Perls 2136 | my ($y,$mo,$d,$h,$m,$s) = $mstart =~ /^(\d{4,4})[-\/,:](\d{1,2})[-\/,:](\d{1,2})[-\/,:T](\d{1,2})[-\/,:](\d{1,2})[-\/,:](\d{1,2}).*/; 2137 | my $cstart = ( $y >= 2038 ) ? "2038-01-01T00:00:00" : sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $y,$mo,$d,$h,$m,$s); 2138 | my ($y,$mo,$d,$h,$m,$s) = $mend =~ /^(\d{4,4})[-\/,:](\d{1,2})[-\/,:](\d{1,2})[-\/,:T](\d{1,2})[-\/,:](\d{1,2})[-\/,:](\d{1,2}).*/; 2139 | my $cend = ( $y >= 2038 || ! $mend ) ? "2038-01-01T00:00:00" : sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $y,$mo,$d,$h,$m,$s); 2140 | 2141 | # Determine epoch times for metadata start and end for matching below 2142 | my $cstart = str2time ($cstart, "UTC"); 2143 | my $cend = str2time ($cend, "UTC"); 2144 | 2145 | # Track if metadata has been added to @metadata, only add once if matching a request 2146 | my $metadatalisted = 0; 2147 | 2148 | # Match metadata to selections (requests) using nested hash of (compiled) regexes 2149 | foreach my $knet ( keys %selectmatch ) { 2150 | next if ( $mnet !~ $knet ); 2151 | 2152 | foreach my $ksta ( keys %{$selectmatch{$knet}} ) { 2153 | next if ( $msta !~ $ksta ); 2154 | 2155 | foreach my $kloc ( keys %{$selectmatch{$knet}{$ksta}} ) { 2156 | next if ( $mloc !~ $kloc ); 2157 | 2158 | foreach my $kchan ( keys %{$selectmatch{$knet}{$ksta}{$kloc}} ) { 2159 | next if ( $mchan !~ $kchan ); 2160 | 2161 | # Check for time overlap (intersection) with metadata 2162 | foreach my $timekey ( keys %{$selectmatch{$knet}{$ksta}{$kloc}{$kchan}} ) { 2163 | my ($kstart,$kend) = split (/\|/, $timekey); 2164 | 2165 | # If time ranges intersect add to request hash, account for unspecified/undef time entries 2166 | if ( ($cstart <= $kend || ! $kend) && ($cend >= $kstart || ! $kstart) ) { 2167 | my ($sstart,$send) = split (/\|/, $selectmatch{$knet}{$ksta}{$kloc}{$kchan}{$timekey}); 2168 | 2169 | # Push channel epoch metadata into storage list 2170 | if ( ! $metadatalisted ) { 2171 | push (@{$datacenter{$dckey}{metadata}}, 2172 | "$mnet|$msta|$mloc|$mchan|$mstart|$mend|$mlat|$mlon|$melev|$mdepth|$mazimuth|$mdip|$minstrument|$msamplerate|$mscale|$mscalefreq|$mscaleunits"); 2173 | $metadatalisted = 1; 2174 | $totalepochs++; 2175 | } 2176 | 2177 | # Add entry to request hash with value of matching metadata 2178 | $datacenter{$dckey}{request}->{"$mnet|$msta|$mloc|$mchan|$sstart|$send"} = "$mstart|$mend"; 2179 | 2180 | # Track widest metadata range for NSLC for SACPZ and RESP requests 2181 | if ( ! exists $datacenter{$dckey}{metarequest}->{"$mnet|$msta|$mloc|$mchan"} ) { 2182 | $datacenter{$dckey}{metarequest}->{"$mnet|$msta|$mloc|$mchan"} = "$cstart|$cend"; 2183 | } 2184 | else { 2185 | my ($vstart,$vend) = split (/\|/, $datacenter{$dckey}{metarequest}->{"$mnet|$msta|$mloc|$mchan"}); 2186 | 2187 | $vstart = $cstart if ( $cstart < $vstart ); 2188 | $vend = $cend if ( $cend > $vend ); 2189 | 2190 | $datacenter{$dckey}{metarequest}->{"$mnet|$msta|$mloc|$mchan"} = "$vstart|$vend"; 2191 | } 2192 | } 2193 | } 2194 | } 2195 | } 2196 | } 2197 | } 2198 | } # End of looping through metadata results 2199 | 2200 | printf STDERR "Done (%.1f seconds)\n", Time::HiRes::time - $ptime if ( $verbose >= 1 ); 2201 | 2202 | return $totalepochs; 2203 | } # End of FetchMetaDataHelper() 2204 | 2205 | 2206 | ###################################################################### 2207 | # MDCallBack: 2208 | # 2209 | # A call back for LWP downloading of metadata. 2210 | # 2211 | # Add received data to metadataresponse string, tally up the received 2212 | # data size and print and updated (overwriting) byte count string. 2213 | ###################################################################### 2214 | sub MDCallBack { 2215 | my ($data, $response, $protocol) = @_; 2216 | $datasize += length($data); 2217 | 2218 | if ( $response->content_encoding() =~ /gzip/ ) { 2219 | my $datablock = ""; 2220 | $inflater->inflate($data, $datablock); 2221 | $metadataresponse .= $datablock; 2222 | } 2223 | else { 2224 | $metadataresponse .= $data; 2225 | } 2226 | 2227 | if ( $verbose >= 1 && ! $nobsprint ) { 2228 | printf (STDERR "%-10.10s\b\b\b\b\b\b\b\b\b\b", sizestring($datasize)); 2229 | } 2230 | } 2231 | 2232 | 2233 | ###################################################################### 2234 | # mktimestring (time, [withSubSeconds, [withZSuffix]]): 2235 | # 2236 | # Return a time string in YYYY-MM-DDTHH:MM:SS[.ssssss][Z] format 2237 | # for the specified time value. 2238 | # 2239 | # If 'withSubSeconds' is true include microsecond resolution. 2240 | # If 'withZSuffix' is true inlude a 'Z' TZ designator suffix. 2241 | ###################################################################### 2242 | sub mktimestring { 2243 | my $time = shift; 2244 | my $withSubSeconds = shift; 2245 | my $withZSuffix = shift; 2246 | 2247 | return undef if ( ! $time ); 2248 | 2249 | my ($sec,$min,$hour,$mday,$mon,$year) = gmtime ($time); 2250 | 2251 | $year += 1900; 2252 | $mon += 1; 2253 | 2254 | my $timestr = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", 2255 | $year,$mon,$mday,$hour,$min,$sec); 2256 | 2257 | # Add subseconds at microsecond resolution if requested 2258 | if ( $withSubSeconds ) { 2259 | my $usec = int (($time - int $time) * 1000000.0 + 0.5); 2260 | $timestr .= sprintf (".%06d", $usec); 2261 | } 2262 | 2263 | # Add 'Z' for denoting UTC time zone if requested 2264 | if ( $withZSuffix ) { 2265 | $timestr .= "Z"; 2266 | } 2267 | 2268 | return $timestr; 2269 | } 2270 | 2271 | 2272 | ###################################################################### 2273 | # sizestring (bytes): 2274 | # 2275 | # Return a clean size string for a given byte count. 2276 | ###################################################################### 2277 | sub sizestring { # sizestring (bytes) 2278 | my $bytes = shift; 2279 | 2280 | if ( $bytes < 1000 ) { 2281 | return sprintf "%d Bytes", $bytes; 2282 | } 2283 | elsif ( ($bytes / 1024) < 1000 ) { 2284 | return sprintf "%.1f KB", $bytes / 1024; 2285 | } 2286 | elsif ( ($bytes / 1024 / 1024) < 1000 ) { 2287 | return sprintf "%.1f MB", $bytes / 1024 / 1024; 2288 | } 2289 | elsif ( ($bytes / 1024 / 1024 / 1024) < 1000 ) { 2290 | return sprintf "%.1f GB", $bytes / 1024 / 1024 / 1024; 2291 | } 2292 | elsif ( ($bytes / 1024 / 1024 / 1024 / 1024) < 1000 ) { 2293 | return sprintf "%.1f TB", $bytes / 1024 / 1024 / 1024 / 1024; 2294 | } 2295 | else { 2296 | return ""; 2297 | } 2298 | } # End of sizestring() 2299 | 2300 | 2301 | ###################################################################### 2302 | # 2303 | # Package RequestAgent: a superclass for LWP::UserAgent with override 2304 | # of LWP::UserAgent methods to set default user agent and handle 2305 | # authentication credentials. 2306 | # 2307 | ###################################################################### 2308 | BEGIN { 2309 | use LWP; 2310 | package RequestAgent; 2311 | our @ISA = qw(LWP::UserAgent); 2312 | 2313 | sub new 2314 | { 2315 | my $self = LWP::UserAgent::new(@_); 2316 | 2317 | # Set up UserAgent 2318 | my $fulluseragent = $useragent; 2319 | $fulluseragent .= " ($appname)" if ( $appname ); 2320 | $self->agent($fulluseragent); 2321 | 2322 | # Follow redirects on POST method in addition to GET and HEAD 2323 | push @{ $self->requests_redirectable }, 'POST'; 2324 | 2325 | # Support non-persisent, session cookies 2326 | $self->cookie_jar ( {} ); 2327 | 2328 | $self; 2329 | } 2330 | 2331 | sub get_basic_credentials 2332 | { 2333 | my ($self, $realm, $uri) = @_; 2334 | 2335 | if ( defined $auth ) { 2336 | return split(':', $auth, 2); 2337 | } 2338 | elsif (-t) { 2339 | my $netloc = $uri->host_port; 2340 | print "\n"; 2341 | print "Enter username for $realm at $netloc: "; 2342 | my $user = ; 2343 | chomp($user); 2344 | return (undef, undef) unless length $user; 2345 | print "Password: "; 2346 | system("stty -echo"); 2347 | my $password = ; 2348 | system("stty echo"); 2349 | print "\n"; # because we disabled echo 2350 | chomp($password); 2351 | return ($user, $password); 2352 | } 2353 | else { 2354 | return (undef, undef) 2355 | } 2356 | } 2357 | } # End of LWP::UserAgent override 2358 | -------------------------------------------------------------------------------- /FetchEvent: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl 2 | # 3 | # FetchEvent 4 | # 5 | # Find the most current version at https://earthscope.github.io/fetch-scripts/ 6 | # 7 | # Fetch event parameters from web services. The default web service 8 | # is from the International Seismological Centre (ISC), other FDSN web 9 | # service implementations may be specified by setting the following 10 | # environment variables: 11 | # 12 | # SERVICEBASE = the base URI of the service(s) to use (https://www.isc.ac.uk/) 13 | # EVENTWS = complete URI of service (https://www.isc.ac.uk/fdsnws/event/1) 14 | # 15 | # Dependencies: This script should run without problems on Perl 16 | # release 5.10 or newer, older versions of Perl might require the 17 | # installation of the following modules (and their dependencies): 18 | # XML::SAX 19 | # Bundle::LWP (libwww-perl) 20 | # 21 | # Installation of the XML::SAX::ExpatXS module can significantly 22 | # speed up the parsing of results returned as XML. 23 | # 24 | # The returned XML event information is parsed and printed to the 25 | # console or a file in the following ways: 26 | # 27 | # The default "parsable" output includes fields separated by vertical 28 | # bars (|) and follows this pattern: 29 | # 30 | # EventID|OriginTime|Latitude|Longitude|Depth/km|Author|Catalog|Contributor,ContribEventID|MagType,MagValue,MagAuthor|MagType,MagValue,MagAuthor|...|EventLocationName 31 | # OriginTime|Latitude|Longitude|Depth/km|Author|Catalog|Contributor,ContribEventID 32 | # 33 | # The event ID and the primary origin details are all contained on a single 34 | # line along with all magnitude estimates associated with the event. 35 | # Secondary origins, if requested, are listed below and indented to line up 36 | # with the equivalient fields of the primary origin. 37 | # 38 | # The "alternate" output (the -altform option) follows this pattern: 39 | # EVENT eventid= 40 | # ORIGIN ,,,,,, originid= 41 | # ORIGIN ,,,,,, originid= 42 | # MAGNITUDE ,, magnitudeid= 43 | # MAGNITUDE ,, magnitudeid= 44 | # 45 | # An EVENT can be followed by multiple ORIGIN lines if the --allorigins 46 | # option is specified and the primary origin is listed first. An 47 | # EVENT may also be followed by one or more MAGNITUDE lines. 48 | # 49 | # ## Change history ## 50 | # 51 | # 2011.161: 52 | # - Initial version. 53 | # 54 | # 2011.270: 55 | # - Allow start and end times to be indenpendently specified. 56 | # - Reorganize lat, lon, depth and mag range options to compact form that 57 | # that matches other Fetch scripts. 58 | # - Add --radius option for selecting a circular region. 59 | # - Add --magtype option. 60 | # 61 | # 2011.272: 62 | # - Reorganize default output for easy reading and hopefully easy parsing. 63 | # - Add the --altform option (unadvertised). 64 | # 65 | # 2011.273: 66 | # - Fix magnitude selection. 67 | # 68 | # 2011.277: 69 | # - Remove debugging code. 70 | # 71 | # 2011.337: 72 | # - Print error message text from service on errors. 73 | # - Add undocumented -nobs option to prevent printing backspace characters in status. 74 | # - Remove "query" portion from base URLs, moving them to the service calls. 75 | # - Include local time in output of DONE line. 76 | # 77 | # 2012.045: 78 | # - Print "No data available" when return code is 204 in addition to 404. 79 | # - Support for start/end times specified up to microsecond resolution. 80 | # 81 | # 2012.206: 82 | # - Parse primary magnitude ID and sort it to front of the mag list 83 | # - Add -allorigins and -allmags to match new service parameters 84 | # - Deprecate secondary option: remove from usage message but leave in code 85 | # - Lower-case some previously CamelCase parameter names for consistency 86 | # 87 | # 2012.221 88 | # - Add limit argument that takes limit:offset (undocumented, for now) 89 | # 90 | # 2013.077 91 | # - Use the LWP::UserAgent method env_proxy() to check for and use connection 92 | # proxy information from environment variables (e.g. http_proxy). 93 | # - Add checking of environment variables that will override the web 94 | # service base path (i.e. host name). 95 | # - Change updated after to take full resolution date and time. 96 | # 97 | # 2013.119 98 | # - Fix parsing of event, origin & magnitude IDs to make them case insensitive, 99 | # recent changes in the service changed the case and broke the script. 100 | # - The event service now returns depths in meters, convert these meters to 101 | # kilometers to match the previous output and common convention. 102 | # - Fix positioning of number fields that have no decimal point and adjust widths 103 | # of fields for expected sizes. 104 | # 105 | # 2013.123 106 | # - Attempt to parse event, origin and magnitude IDs from patterns other than 107 | # the one used by the IRIS services. 108 | # - Check for no origins for an event and print appropriate message on output. 109 | # 110 | # 2013.154 111 | # - Change (unadvertised) option -eventurl to -eventws to match env. variable. 112 | # 113 | # 2013.162 114 | # - Parse and store complete publicID for event, origin and magnitude IDs. 115 | # Add a function to attempt to parse shorter IDs from full public IDs for 116 | # printing. This makes the script compatible for services that do not follow 117 | # the IRIS publicID convention. 118 | # 119 | # 2013.176 120 | # - Fixed comparisons involving the full public IDs. This had hindered the 121 | # functionality of -allorigins and -allmags 122 | # --celso 123 | # 124 | # 2013.197 125 | # - Fix parsing of element values of "0". 126 | # 127 | # 2013.198 128 | # - Add test for minimum version of LWP (libwww) module of 5.806. 129 | # 130 | # 2013.316 131 | # - Allow latitude, longitude, depth, magnitude and geographic ranges 132 | # to be specified with the value of 0. Same treatment for the undocumented 133 | # 'limit' option. 134 | # 135 | # 2014.056 136 | # - Allow gzip'ed HTTP encoding for event data requests if support 137 | # exists on the local system. 138 | # 139 | # 2014.232 140 | # - Add "compressed from " to diagnostic message when the server returns 141 | # gzip'ed content to differentiate the size transmitted versus the result. 142 | # 143 | # 2014.290 144 | # - Change help message to refer to 'NEIC PDE' intead of just PDE. 145 | # 146 | # 2014.310 147 | # - Add event ID parsing to include 'evid' token, needed for the ISC service. 148 | # 149 | # 2014.340 150 | # - Fix parsing/conversion of depth value if data is separated by XML parser. 151 | # 152 | # 2024.113 153 | # - Set PERL_LWP_SSL_VERIFY_HOSTNAME to 0 to allow encrypted connections without 154 | # checking for valid certificate matching the expected hostname. 155 | # - Change the default event source to the ISC fdsnws-event service. 156 | # - Add support for the USGS FDSN fdsnws-event service. 157 | # - Allow -e end time specification to be an offset relative to the 158 | # start time in the pattern #.#[SMHD], e.g. 30m, 1h, 2D, etc. 159 | # 160 | # Author: Chad Trabant, EarthScope Data Services 161 | 162 | use strict; 163 | use File::Basename; 164 | use Getopt::Long; 165 | use LWP 5.806; # Require minimum version 166 | use LWP::UserAgent; 167 | use HTTP::Status qw(status_message); 168 | use HTTP::Date; 169 | use Time::HiRes; 170 | 171 | my $version = "2024.113"; 172 | 173 | my $scriptname = basename($0); 174 | 175 | # Default web service base 176 | my $servicebase = 'https://www.isc.ac.uk'; 177 | 178 | # Check for environment variable overrides for servicebase 179 | $servicebase = $ENV{'SERVICEBASE'} if ( exists $ENV{'SERVICEBASE'} ); 180 | 181 | # Web service for events 182 | my $eventservice = "$servicebase/fdsnws/event/1"; 183 | 184 | # Check for environment variable override for timeseriesservice 185 | $eventservice = $ENV{'EVENTWS'} if ( exists $ENV{'EVENTWS'} ); 186 | 187 | my $usgs_eventservice = "https://earthquake.usgs.gov/fdsnws/event/1"; 188 | 189 | # HTTP UserAgent reported to web services 190 | my $useragent = "$scriptname/$version Perl/$] " . new LWP::UserAgent->_agent; 191 | 192 | # Allow encrypted connections without checking for valid certificate matching 193 | # the expected hostname. 194 | $ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0; 195 | 196 | my $usage = undef; 197 | my $verbose = undef; 198 | my $nobsprint = undef; 199 | 200 | my $starttime = undef; 201 | my $endtime = undef; 202 | my @latrange = (); # (minlat:maxlat) 203 | my @lonrange = (); # (minlon:maxlon) 204 | my @degrange = (); # (lat:lon:maxradius[:minradius]) 205 | my @deprange = (); # (mindepth:maxdepth) 206 | my @magrange = (); # (minmag:maxmag) 207 | my $magtype = undef; 208 | my $catalog = undef; 209 | my $contributor = undef; 210 | my $updatedafter = undef; 211 | my $altform = undef; 212 | my @limitrange = (); # (limit:offset) 213 | my $allorigins = undef; 214 | my $allmags = undef; 215 | my $orderbymag = undef; 216 | 217 | my $eventid = undef; 218 | my $originid = undef; 219 | 220 | my $appname = undef; 221 | my $auth = undef; 222 | my $outfile = undef; 223 | my $xmlfile = undef; 224 | 225 | my $eventxml = undef; 226 | 227 | # Event identification tokens for extracting minimal ID from publicID 228 | my @eventidtokens = ("eventid","evid"); 229 | 230 | my $inflater = undef; 231 | 232 | # If Compress::Raw::Zlib is available configure inflater for RFC 1952 (gzip) 233 | if ( eval("use Compress::Raw::Zlib; 1") ) { 234 | use Compress::Raw::Zlib; 235 | $inflater = new Compress::Raw::Zlib::Inflate( -WindowBits => WANT_GZIP, 236 | -ConsumeInput => 0 ); 237 | } 238 | 239 | # Parse command line arguments 240 | Getopt::Long::Configure ("bundling_override"); 241 | my $getoptsret = GetOptions ( 'help|usage|h' => \$usage, 242 | 'verbose|v+' => \$verbose, 243 | 'nobs' => \$nobsprint, 244 | 'starttime|s=s' => \$starttime, 245 | 'endtime|e=s' => \$endtime, 246 | 'lat=s' => \@latrange, 247 | 'lon=s' => \@lonrange, 248 | 'radius=s' => \@degrange, 249 | 'depth=s' => \@deprange, 250 | 'mag=s' => \@magrange, 251 | 'magtype=s' => \$magtype, 252 | 'catalog|cat=s' => \$catalog, 253 | 'contributor|con=s' => \$contributor, 254 | 'updatedafter|ua=s' => \$updatedafter, 255 | 'altform' => \$altform, 256 | 'limit=s' => \@limitrange, 257 | 'allorigins' => \$allorigins, 258 | 'allmags' => \$allmags, 259 | 'orderbymag' => \$orderbymag, 260 | 261 | 'eventid|evid=s' => \$eventid, 262 | 'originid|orid=s' => \$originid, 263 | 264 | 'appname|A=s' => \$appname, 265 | 'auth|a=s' => \$auth, 266 | 'outfile|o=s' => \$outfile, 267 | 'xmlfile|X=s' => \$xmlfile, 268 | 'eventws=s' => \$eventservice, 269 | 'usgs|U' => sub { $eventservice = $usgs_eventservice; }, 270 | ); 271 | 272 | my $required = ( defined $starttime || defined $endtime || 273 | scalar @latrange || scalar @lonrange || scalar @degrange || 274 | scalar @deprange || scalar @magrange || 275 | defined $catalog || defined $contributor || 276 | defined $eventid || defined $originid ); 277 | 278 | if ( ! $getoptsret || $usage || ! $required ) { 279 | print "$scriptname: collect event information ($version)\n"; 280 | print "https://earthscope.github.io/fetch-scripts/\n\n"; 281 | print "Usage: $scriptname [options]\n\n"; 282 | print " Options:\n"; 283 | print " -v More verbosity, may be specified multiple times (-vv, -vvv)\n"; 284 | print " -usgs or -U Use the USGS FDSN event service instead of the default ISC service\n"; 285 | print "\n"; 286 | print " -s starttime Limit to origins after time (YYYY-MM-DD,HH:MM:SS.sss)\n"; 287 | print " -e endtime Limit to origins before time (YYYY-MM-DD,HH:MM:SS.sss or or #[SMHD])\n"; 288 | print " --lat min:max Specify a minimum and/or maximum latitude range\n"; 289 | print " --lon min:max Specify a minimum and/or maximum longitude range\n"; 290 | print " --radius lat:lon:maxradius[:minradius]\n"; 291 | print " Specify circular region with optional minimum radius\n"; 292 | print " --depth min:max Specify a minimum and/or maximum depth in kilometers\n"; 293 | print " --mag min:max Specify a minimum and/or maximum magnitude\n"; 294 | print " --magtype type Specify a magnitude type for magnitude range limits\n"; 295 | print " --cat name Limit to origins from specific catalog (e.g. ISC, 'NEIC PDE', GCMT)\n"; 296 | print " --con name Limit to origins from specific contributor (e.g. ISC, NEIC)\n"; 297 | print " --ua date Limit to origins updated after date (YYYY-MM-DD,HH:MM:SS)\n"; 298 | print "\n"; 299 | # print " --altform Print output in an alternate form, default is more parsable\n"; 300 | # print " --limit max:off Limit to max events, optionally starting at offset\n"; 301 | print " --allorigins Return all origins, default is only primary origin per event\n"; 302 | print " --allmags Return all magnitudes, default is only primary magnitude per event\n"; 303 | print " --orderbymag Order results by magnitude instead of time\n"; 304 | print "\n"; 305 | print " --evid id Select a specific event by DMC event ID\n"; 306 | print " --orid id Select a specific event by DMC origin ID\n"; 307 | print "\n"; 308 | print " -X xmlfile Write raw returned XML to xmlfile\n"; 309 | print " -A appname Application/version string for identification\n"; 310 | print "\n"; 311 | print " -o outfile Write event information to specified file, default: console\n"; 312 | print "\n"; 313 | exit 1; 314 | } 315 | 316 | # Print script name and local time string 317 | if ( $verbose ) { 318 | my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time); 319 | printf STDERR "$scriptname ($version) at %4d-%02d-%02d %02d:%02d:%02d\n", $year+1900, $mon+1, $mday, $hour, $min, $sec; 320 | } 321 | 322 | # Normalize time strings 323 | if ( $starttime ) { 324 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $starttime); 325 | $starttime = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 326 | $starttime .= ".$subsec" if ( $subsec ); 327 | } 328 | 329 | if ( $endtime ) 330 | { 331 | # Check for and parse time in duration pattern: #.#[SMHD] 332 | if ( $endtime =~ /^[\d\.]+[SsMmHhDd]?$/ ) { 333 | if ( $starttime ) { 334 | my ($offset, $timeunit) = $endtime =~ /^([\d\.]+)([SsMmHhDd]?)$/i; 335 | $timeunit = 'S' if (! $timeunit); 336 | 337 | # Convert offset value to seconds if specified as days, hours or minutes 338 | if ($timeunit =~ /[Dd]/) { 339 | $offset *= 86400; 340 | } 341 | elsif ($timeunit =~ /[Hh]/) { 342 | $offset *= 3600; 343 | } 344 | elsif ($timeunit =~ /[Mm]/) { 345 | $offset *= 60; 346 | } 347 | 348 | # Calculate end time from start + offset and generate string 349 | my $rstartepoch = str2time ($starttime, "UTC"); 350 | if ( defined $rstartepoch ) { 351 | my ($sec,$min,$hour,$mday,$mon,$year) = gmtime($rstartepoch + $offset); 352 | $endtime = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year+1900, $mon+1, $mday, $hour, $min, $sec); 353 | } 354 | else { 355 | die "Unable to parse start time: '$starttime'\n" 356 | } 357 | } 358 | else { 359 | die "Cannot specify end time as duration without specifying start time\n"; 360 | } 361 | } 362 | else { 363 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $endtime); 364 | $endtime = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 365 | $endtime .= ".$subsec" if ( $subsec ); 366 | } 367 | } 368 | 369 | if ( $updatedafter ) { 370 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $updatedafter); 371 | $updatedafter = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 372 | $updatedafter .= ".$subsec" if ( $subsec ); 373 | } 374 | 375 | # Validate and prepare lat, lon and radius input 376 | if ( scalar @latrange ) { 377 | @latrange = split (/:/, $latrange[0]); 378 | 379 | if ( defined $latrange[0] && ($latrange[0] < -90.0 || $latrange[0] > 90.0) ) { 380 | die "Minimum latitude out of range: $latrange[0]\n"; 381 | } 382 | if ( defined $latrange[1] && ($latrange[1] < -90.0 || $latrange[1] > 90.0) ) { 383 | die "Maximum latitude out of range: $latrange[1]\n"; 384 | } 385 | } 386 | if ( scalar @lonrange ) { 387 | @lonrange = split (/\:/, $lonrange[0]); 388 | 389 | if ( defined $lonrange[0] && ($lonrange[0] < -180.0 || $lonrange[0] > 180.0) ) { 390 | die "Minimum longitude out of range: $lonrange[0]\n"; 391 | } 392 | if ( defined $lonrange[1] && ($lonrange[1] < -180.0 || $lonrange[1] > 180.0) ) { 393 | die "Maximum longitude out of range: $lonrange[1]\n"; 394 | } 395 | } 396 | if ( scalar @degrange ) { 397 | @degrange = split (/\:/, $degrange[0]); 398 | 399 | if ( scalar @degrange < 3 || scalar @degrange > 4 ) { 400 | die "Unrecognized radius specification: @degrange\n"; 401 | } 402 | if ( defined $degrange[0] && ($degrange[0] < -90.0 || $degrange[0] > 90.0) ) { 403 | die "Radius latitude out of range: $degrange[0]\n"; 404 | } 405 | if ( defined $degrange[1] && ($degrange[1] < -180.0 || $degrange[1] > 180.0) ) { 406 | die "Radius longitude out of range: $degrange[1]\n"; 407 | } 408 | } 409 | if ( scalar @deprange ) { 410 | @deprange = split (/\:/, $deprange[0]); 411 | 412 | if ( defined $deprange[0] && ($deprange[0] < -7000 || $deprange[0] > 7000) ) { 413 | die "Minimum depth out of range: $deprange[0]\n"; 414 | } 415 | if ( defined $deprange[1] && ($deprange[1] < -7000 || $deprange[1] > 7000) ) { 416 | die "Maximum depth out of range: $deprange[1]\n"; 417 | } 418 | } 419 | if ( scalar @magrange ) { 420 | @magrange = split (/\:/, $magrange[0]); 421 | 422 | if ( defined $magrange[0] && ($magrange[0] < -50 || $magrange[0] > 15) ) { 423 | die "Minimum magnitude out of range: $magrange[0]\n"; 424 | } 425 | if ( defined $magrange[1] && ($magrange[1] < -50 || $magrange[1] > 15) ) { 426 | die "Maximum magnitude out of range: $magrange[1]\n"; 427 | } 428 | } 429 | if ( scalar @limitrange ) { 430 | @limitrange = split (/\:/, $limitrange[0]); 431 | 432 | if ( defined $limitrange[0] && ($limitrange[0] < 1) ) { 433 | die "Event limit out of range: $limitrange[0]\n"; 434 | } 435 | if ( defined $limitrange[1] && ($limitrange[1] < 0) ) { 436 | die "Event count offset out of range: $limitrange[1]\n"; 437 | } 438 | } 439 | 440 | # Report data selection criteria 441 | if ( $verbose > 2 ) { 442 | print STDERR "Latitude range: $latrange[0] : $latrange[1]\n" if ( scalar @latrange ); 443 | print STDERR "Longitude range: $lonrange[0] : $lonrange[1]\n" if ( scalar @lonrange ); 444 | print STDERR "Radius range: $degrange[0] : $degrange[1] : $degrange[2] : $degrange[3]\n" if ( scalar @degrange ); 445 | print STDERR "Depth range: $deprange[0] : $deprange[1]\n" if ( scalar @deprange ); 446 | print STDERR "Magnitude range: $magrange[0] : $magrange[1]\n" if ( scalar @magrange ); 447 | print STDERR "Magnitude range: $limitrange[0] : $limitrange[1]\n" if ( scalar @limitrange ); 448 | } 449 | 450 | # Containers to hold event, origin and magnitude details 451 | my @eventids = (); 452 | my %eventname = (); 453 | my %eventprime = (); 454 | my %eventprimemag = (); 455 | my %origins = (); 456 | my %mags = (); 457 | 458 | my $datasize; 459 | 460 | # Fetch events from the web service 461 | &FetchEvents(); 462 | 463 | my $totalevents = 0; 464 | my $totalorigins = 0; 465 | my $totalmags = 0; 466 | 467 | # Write to either specified output file or stdout 468 | my $eventfile = ( $outfile ) ? $outfile : "-"; 469 | 470 | if ( scalar @eventids <= 0 ) { 471 | printf STDERR "No events selected\n", scalar @eventids; 472 | } 473 | 474 | open (EVENT, ">$eventfile") || die "Cannot open event file '$eventfile': $!\n"; 475 | 476 | # Write event information to stdout/console or a file if specified 477 | if ( ! defined $altform ) { 478 | # Parsable lines, one event line with primary origin, fields delimited with '|': 479 | # EventID|OriginTime|Latitude|Longitude|Depth/km|Author|Catalog|Contributor,ContribEventID|MagType,MagValue,MagAuthor|MagType,MagValue,MagAuthor|...|EventLocationName 480 | # OriginTime|Latitude|Longitude|Depth/km|Author|Catalog|Contributor,ContribEventID 481 | 482 | foreach my $eventid ( @eventids ) { 483 | $totalevents++; 484 | 485 | # Print event ID 486 | my $printeventid = &ExtractID($eventid, @eventidtokens); 487 | printf EVENT "%-8s", $printeventid; 488 | 489 | # Generate list of origin IDs with primary origin first and others sorted on ascending time 490 | my @originids = (exists $origins{$eventid}{$eventprime{$eventid}}) ? ($eventprime{$eventid}) : (); 491 | foreach my $originid ( sort { ${$origins{$eventid}{$b}}[0] cmp ${$origins{$eventid}{$a}}[0] } keys %{$origins{$eventid}} ) { 492 | push (@originids, $originid) if ( $originid ne $eventprime{$eventid} ); 493 | } 494 | $totalorigins += scalar @originids; 495 | 496 | if ( scalar @originids <= 0 ) { 497 | print EVENT " -- NO ORIGINS\n"; 498 | next; 499 | } 500 | 501 | # Print first/primary origin 502 | my ($time,$lat,$lon,$depth,$author,$catalog,$contributor,$contributoreventid) = @{$origins{$eventid}{$originids[0]}}; 503 | printf (EVENT "|%-24s|%-8s|%-9s|%-6s|%s|%s|%s%s", 504 | $time,&decimalposition($lat,4),&decimalposition($lon,5),&decimalposition($depth,5), 505 | $author,$catalog,$contributor,($contributoreventid)?",$contributoreventid":""); 506 | 507 | # Generate list of magnitude IDs with primary magnitude first and others sorted on ascending value 508 | my @magids = (exists $mags{$eventid}{$eventprimemag{$eventid}}) ? ($eventprimemag{$eventid}) : (); 509 | foreach my $magid ( sort { ${$mags{$eventid}{$b}}[1] cmp ${$mags{$eventid}{$a}}[1] } keys %{$mags{$eventid}} ) { 510 | push (@magids, $magid) if ( $magid ne $eventprimemag{$eventid} ); 511 | } 512 | $totalmags += scalar @magids; 513 | 514 | # Print magnitudes 515 | foreach my $magid ( @magids ) { 516 | my $magstr = join (',', @{$mags{$eventid}{$magid}}); 517 | $magstr =~ s/,*$//; # Cut trailing commas 518 | print EVENT "|$magstr"; 519 | } 520 | 521 | # Print event name and newline 522 | printf EVENT "|$eventname{$eventid}\n"; 523 | 524 | # Print secondary origin lines 525 | foreach my $originid ( @originids[1..$#originids] ) { 526 | my ($time,$lat,$lon,$depth,$author,$catalog,$contributor,$contributoreventid) = @{$origins{$eventid}{$originid}}; 527 | printf (EVENT " %-24s|%-8s|%-9s|%-6s|%s|%s|%s%s\n", 528 | $time,&decimalposition($lat,4),&decimalposition($lon,5),&decimalposition($depth,5), 529 | $author,$catalog,$contributor,($contributoreventid)?",$contributoreventid":""); 530 | } 531 | } 532 | } 533 | else { 534 | # Alternate lines, primary origin is first: 535 | # EVENT eventid= 536 | # ORIGIN ,,,,,, originid= 537 | # ORIGIN ,,,,,, originid= 538 | # MAGNITUDE ,, magnitudeid= 539 | # MAGNITUDE ,, magnitudeid= 540 | 541 | foreach my $eventid ( @eventids ) { 542 | $totalevents++; 543 | 544 | # Print event ID 545 | my $printeventid = &ExtractID($eventid, @eventidtokens); 546 | printf EVENT "EVENT $eventname{$eventid} eventid=$printeventid\n"; 547 | 548 | # Generate list of origin IDs with primary origin firstand others sorted on ascending time 549 | my @originids = (exists $origins{$eventid}{$eventprime{$eventid}}) ? ($eventprime{$eventid}) : (); 550 | foreach my $originid ( sort { ${$origins{$eventid}{$b}}[0] cmp ${$origins{$eventid}{$a}}[0] } keys %{$origins{$eventid}} ) { 551 | push (@originids, $originid) if ( $originid ne $eventprime{$eventid} ); 552 | } 553 | 554 | # Generate list of magnitude IDs with primary magnitude first and others sorted on ascending value 555 | my @magids = (exists $mags{$eventid}{$eventprimemag{$eventid}}) ? ($eventprimemag{$eventid}) : (); 556 | foreach my $magid ( sort { ${$mags{$eventid}{$b}}[1] cmp ${$mags{$eventid}{$a}}[1] } keys %{$mags{$eventid}} ) { 557 | push (@magids, $magid) if ( $magid ne $eventprimemag{$eventid} ); 558 | } 559 | 560 | # Print origin lines 561 | foreach my $originid ( @originids ) { 562 | $totalorigins++; 563 | my ($time,$lat,$lon,$depth,$author,$catalog,$contributor,$contributoreventid) = @{$origins{$eventid}{$originid}}; 564 | 565 | printf EVENT " ORIGIN %s,%s,%s,%s,%s,%s,%s originid=$originid\n", 566 | $time,$lat,$lon,$depth,$author,$catalog,$contributor; 567 | } 568 | 569 | # Print magnitude lines 570 | foreach my $magid ( @magids ) { 571 | $totalmags++; 572 | print EVENT " MAGNITUDE " . join (',', @{$mags{$eventid}{$magid}}) . " magnitudeid=$magid\n"; 573 | } 574 | } 575 | } 576 | 577 | close EVENT; 578 | 579 | if ( $verbose && scalar @eventids > 0 ) { 580 | printf STDERR "Selected %d events (%d origins, %d magnitudes)\n", 581 | $totalevents, $totalorigins, $totalmags; 582 | } 583 | 584 | my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time); 585 | printf STDERR "DONE at %4d-%02d-%02d %02d:%02d:%02d\n", $year+1900, $mon+1, $mday, $hour, $min, $sec; 586 | ## End of main 587 | 588 | 589 | ###################################################################### 590 | # decimalposition: 591 | # 592 | # Return a string left-padded with spaces such that the decimal is at 593 | # the specified position in the string. 594 | ###################################################################### 595 | sub decimalposition { # decimalposition (value, position) 596 | my ($value, $position) = @_; 597 | 598 | my $idx = index ($value, '.'); 599 | my $len = length $value; 600 | 601 | $idx = $len if ( $idx < 0 ); 602 | 603 | return sprintf "%*s", ($position - $idx - 1 + $len), $value; 604 | 605 | } # End of decimalposition() 606 | 607 | 608 | ###################################################################### 609 | # ExtractID: 610 | # 611 | # Attempt to extract an ID from a full publicID value. The smaller ID 612 | # value is generally better for printing and is normally usable as the 613 | # value of the eventid service parameter. 614 | # 615 | # An NEIC example for : 616 | # quakeml://comcat.cr.usgs.gov/fdsnws/event/1/query?eventid=pde20100115001825990_25&format=quakeml 617 | # 618 | # A SeisComP3 example for : 619 | # smi:scs/0.6/gfz2013hykj 620 | # 621 | # An ISC example for : 622 | # smi:ISC/evid=15925507 623 | # 624 | ###################################################################### 625 | sub ExtractID { # ExtractID (publicid, token1[, token2, token3, ..]) 626 | my $publicid = shift; 627 | my @tokens = @_; 628 | 629 | my $id = undef; 630 | my $token = undef; 631 | 632 | foreach my $tok ( @tokens ) { 633 | if ( $publicid =~ /.*${tok}\=[0-9a-zA-Z]+/i ) { # Check for token=ID 634 | $token = $tok; 635 | last; 636 | } 637 | } 638 | if ( $token ) { 639 | ($id) = $publicid =~ /.*${token}\=([0-9a-zA-Z]+)/i; 640 | } 641 | elsif ( $publicid =~ /.*\/[^\/:]+$/ ) { # Check for ID following last slash 642 | ($id) = $publicid =~ /.*\/([^\/:]+)$/; 643 | } 644 | else { # Fall back to simply using the full public ID 645 | $id = $publicid; 646 | } 647 | 648 | return $id; 649 | } # End of ExtractID() 650 | 651 | 652 | ###################################################################### 653 | # FetchEvents: 654 | # 655 | # Collect event information for selected data set. 656 | # 657 | # Resulting information is placed in the following global structures: 658 | # @eventids - an array containing the returned event IDs in order 659 | # %eventname 660 | # %eventprime 661 | # %eventprimemag 662 | # %origins 663 | # %mags 664 | # 665 | ###################################################################### 666 | sub FetchEvents { 667 | 668 | # Create HTTP user agent 669 | my $ua = RequestAgent->new(); 670 | $ua->env_proxy; 671 | 672 | # Create web service URI 673 | my $uri = "${eventservice}/query?"; 674 | $uri .= "orderby="; $uri .= ($orderbymag) ? "magnitude" : "time"; 675 | $uri .= "&starttime=$starttime" if ( $starttime ); 676 | $uri .= "&endtime=$endtime" if ( $endtime ); 677 | if ( scalar @latrange ) { 678 | $uri .= "&minlat=$latrange[0]" if ( defined $latrange[0] ); 679 | $uri .= "&maxlat=$latrange[1]" if ( defined $latrange[1] ); 680 | } 681 | if ( scalar @lonrange ) { 682 | $uri .= "&minlon=$lonrange[0]" if ( defined $lonrange[0] ); 683 | $uri .= "&maxlon=$lonrange[1]" if ( defined $lonrange[1] ); 684 | } 685 | if ( scalar @degrange ) { 686 | $uri .= "&lat=$degrange[0]" if ( defined $degrange[0] ); 687 | $uri .= "&lon=$degrange[1]" if ( defined $degrange[1] ); 688 | $uri .= "&maxradius=$degrange[2]" if ( defined $degrange[2] ); 689 | $uri .= "&minradius=$degrange[3]" if ( defined $degrange[3] ); 690 | } 691 | if ( scalar @deprange ) { 692 | $uri .= "&mindepth=$deprange[0]" if ( defined $deprange[0] ); 693 | $uri .= "&maxdepth=$deprange[1]" if ( defined $deprange[1] ); 694 | } 695 | if ( scalar @magrange ) { 696 | $uri .= "&minmag=$magrange[0]" if ( defined $magrange[0] ); 697 | $uri .= "&maxmag=$magrange[1]" if ( defined $magrange[1] ); 698 | } 699 | $uri .= "&magtype=$magtype" if ( $magtype ); 700 | $uri .= "&catalog=$catalog" if ( $catalog ); 701 | $uri .= "&contributor=$contributor" if ( $contributor ); 702 | $uri .= "&updatedafter=$updatedafter" if ( $updatedafter ); 703 | if ( scalar @limitrange ) { 704 | $uri .= "&limit=$limitrange[0]" if ( defined $limitrange[0] ); 705 | $uri .= "&offset=$limitrange[1]" if ( defined $limitrange[1] ); 706 | } 707 | $uri .= "&includeallorigins=true" if ( $allorigins ); 708 | $uri .= "&includeallmagnitudes=true" if ( $allmags ); 709 | 710 | $uri .= "&eventid=$eventid" if ( $eventid ); 711 | $uri .= "&originid=$originid" if ( $originid ); 712 | 713 | my $ftime = Time::HiRes::time; 714 | 715 | print STDERR "Event URI: '$uri'\n" if ( $verbose > 1 ); 716 | 717 | print STDERR "Fetching event information :: " if ( $verbose ); 718 | 719 | $datasize = 0; 720 | $eventxml = ""; 721 | 722 | # Fetch event information from web service using callback routine 723 | my $response = ( $inflater ) ? 724 | $ua->get($uri, 'Accept-Encoding' => 'gzip', ':content_cb' => \&EDCallBack ) : 725 | $ua->get($uri, ':content_cb' => \&EDCallBack ); 726 | 727 | $inflater->inflateReset if ( $inflater ); 728 | 729 | if ( $response->code == 404 || $response->code == 204 ) { 730 | print (STDERR "No data available\n") if ( $verbose ); 731 | return; 732 | } 733 | elsif ( ! $response->is_success() ) { 734 | print (STDERR "Error fetching data: " 735 | . $response->code . " :: " . status_message($response->code) . "\n"); 736 | print STDERR "------\n" . $response->decoded_content . "\n------\n"; 737 | print STDERR " URI: '$uri'\n" if ( $verbose > 1 ); 738 | } 739 | else { 740 | printf (STDERR "%s\n", ($nobsprint)?sizestring($datasize):"") if ( $verbose ); 741 | } 742 | 743 | my $duration = Time::HiRes::time - $ftime; 744 | my $rate = $datasize/(($duration)?$duration:0.000001); 745 | printf (STDERR "Received %s%s of event information in %.1f seconds (%s/s)\n", 746 | sizestring($datasize), 747 | ( $response->content_encoding() =~ /gzip/ ) ? sprintf (" (compressed from %s)", sizestring(length $eventxml)) : "", 748 | $duration, sizestring($rate)); 749 | 750 | # Return if no metadata received 751 | return if ( length $eventxml <= 0 ); 752 | 753 | # Create stream oriented XML parser instance 754 | use XML::SAX; 755 | my $parser = new XML::SAX::ParserFactory->parser( Handler => ESHandler->new ); 756 | 757 | my $totalevents = 0; 758 | my $totalorigins = 0; 759 | 760 | my $ptime = Time::HiRes::time; 761 | 762 | print STDERR "Parsing XML event data... " if ( $verbose ); 763 | 764 | # Open file to store event XML 765 | my $eventxmlfile = ( $xmlfile ) ? $xmlfile : "eventdata-$$.xml"; 766 | if ( open (EXML, ">$eventxmlfile") ) { 767 | # Write XML and close file 768 | print EXML $eventxml; 769 | close EXML; 770 | 771 | # Parse XML from file 772 | $parser->parse_file ($eventxmlfile); 773 | 774 | # Remove temporary XML file 775 | if ( ! defined $xmlfile ) { 776 | if ( ! unlink $eventxmlfile ) { 777 | print STDERR "Cannot remove temporary XML file: $!\n"; 778 | } 779 | } 780 | } 781 | # Otherwise parse the XML in memory 782 | else { 783 | printf STDERR " in memory (possibly slow), " if ( $verbose ); 784 | 785 | # Parse XML event data from string 786 | $parser->parse_string ($eventxml); 787 | } 788 | 789 | printf STDERR "Done (%.1f seconds)\n", Time::HiRes::time - $ptime if ( $verbose ); 790 | 791 | my $duration = Time::HiRes::time - $ftime; 792 | my $rate = $datasize/(($duration)?$duration:0.000001); 793 | printf (STDERR "Processed event information for $totalevents events, $totalorigins origins in %.1f seconds (%s/s)\n", 794 | $duration, sizestring($rate)); 795 | 796 | ## End of this routine, below is the XML parsing handler used above 797 | 798 | ## Beginning of SAX ESHandler, event-based streaming XML parsing 799 | package ESHandler; 800 | use base qw(XML::SAX::Base); 801 | use HTTP::Date; 802 | 803 | my $inevent = 0; 804 | my $inpreferredOriginID = 0; 805 | my $inpreferredMagnitudeID = 0; 806 | my $indescription = 0; 807 | my $intext = 0; 808 | my $invalue = 0; 809 | 810 | my $inorigin = 0; 811 | my $intime = 0; 812 | my $inlat = 0; 813 | my $inlon = 0; 814 | my $indepth = 0; 815 | my $increationInfo = 0; 816 | my $inauthor = 0; 817 | 818 | my $inmagnitude = 0; 819 | my $inmag = 0; 820 | my $intype = 0; 821 | 822 | my $name = undef; 823 | 824 | my ($name,$eventid,$originid,$porigin,$pmagnitude,$time,$lat,$lon,$depth,$author,$catalog,$contributor,$contributoreventid) = (undef) x 13; 825 | my ($magnitudeid,$mag,$magtype,$magauthor) = (undef) x 4; 826 | 827 | sub start_element { 828 | my ($self,$element) = @_; 829 | 830 | if ( $element->{Name} eq "event" ) { 831 | ($name,$eventid,$originid,$porigin,$pmagnitude,$time,$lat,$lon,$depth,$author,$catalog,$contributor,$contributoreventid) = (undef) x 13; 832 | ($magnitudeid,$mag,$magtype,$magauthor) = (undef) x 4; 833 | 834 | $eventid = $element->{Attributes}->{'{}publicID'}->{Value}; 835 | 836 | $inevent = 1; 837 | } 838 | 839 | if ( $inevent ) { 840 | if ( $element->{Name} eq "preferredOriginID" ) { $inpreferredOriginID = 1; } 841 | elsif ( $element->{Name} eq "preferredMagnitudeID" ) { $inpreferredMagnitudeID = 1; } 842 | elsif ( $element->{Name} eq "description" ) { $indescription = 1; } 843 | 844 | if ( $indescription && $element->{Name} eq "text" ) { $intext = 1; } 845 | 846 | if ( $element->{Name} eq "origin" ) { 847 | $originid = $element->{Attributes}->{'{}publicID'}->{Value}; 848 | 849 | # Search for catalog and contributor from any namespace 850 | foreach my $key (keys %{$element->{Attributes}}) { 851 | $catalog = $element->{Attributes}->{$key}->{Value} if ( $key =~ /^\{.*\}catalog$/ ); 852 | $contributor = $element->{Attributes}->{$key}->{Value} if ( $key =~ /^\{.*\}contributor$/ ); 853 | $contributoreventid = $element->{Attributes}->{$key}->{Value} if ( $key =~ /^\{.*\}contributorEventId$/ ); 854 | } 855 | 856 | $inorigin = 1; 857 | } 858 | 859 | if ( $element->{Name} eq "magnitude" ) { 860 | $magnitudeid = $element->{Attributes}->{'{}publicID'}->{Value}; 861 | 862 | $inmagnitude = 1; 863 | } 864 | } 865 | 866 | if ( $inorigin ) { 867 | if ( $element->{Name} eq "time" ) { $intime = 1; } 868 | elsif ( $element->{Name} eq "latitude" ) { $inlat = 1; } 869 | elsif ( $element->{Name} eq "longitude" ) { $inlon = 1; } 870 | elsif ( $element->{Name} eq "depth" ) { $indepth = 1; } 871 | elsif ( $element->{Name} eq "creationInfo" ) { $increationInfo = 1; } 872 | 873 | if ( ($intime || $inlat || $inlon || $indepth || $increationInfo) && $element->{Name} eq "value" ) { $invalue = 1; } 874 | if ( $increationInfo && $element->{Name} eq "author" ) { $inauthor = 1; } 875 | } 876 | 877 | if ( $inmagnitude ) { 878 | if ( $element->{Name} eq "mag" ) { $inmag = 1; } 879 | elsif ( $element->{Name} eq "type" ) { $intype = 1; } 880 | elsif ( $element->{Name} eq "creationInfo" ) { $increationInfo = 1; } 881 | 882 | if ( $inmag && $element->{Name} eq "value" ) { $invalue = 1; } 883 | if ( $increationInfo && $element->{Name} eq "author" ) { $inauthor = 1; } 884 | } 885 | } 886 | 887 | sub end_element { 888 | my ($self,$element) = @_; 889 | 890 | if ( $element->{Name} eq "event" ) { 891 | push (@eventids, $eventid); 892 | 893 | $eventname{$eventid} = $name; 894 | $eventprime{$eventid} = $porigin; 895 | $eventprimemag{$eventid} = $pmagnitude; 896 | $totalevents++; 897 | 898 | ($name,$eventid,$porigin,$pmagnitude) = (undef) x 4; 899 | $inevent = 0; 900 | } 901 | 902 | if ( $inevent ) { 903 | if ( $element->{Name} eq "preferredOriginID" ) { 904 | $inpreferredOriginID = 0; 905 | } 906 | elsif ( $element->{Name} eq "preferredMagnitudeID" ) { 907 | $inpreferredMagnitudeID = 0; 908 | } 909 | elsif ( $element->{Name} eq "description" ) { $indescription = 0; } 910 | 911 | if ( $indescription && $element->{Name} eq "text" ) { $intext = 0; } 912 | 913 | if ( $element->{Name} eq "origin" ) { 914 | # Convert ISO exchange time to something slightly more readable 915 | $time =~ s/\-/\//g; 916 | $time =~ s/T/ /; 917 | 918 | # Convert QuakeML depth in meters to kilometers 919 | $depth = $depth / 1000.0; 920 | 921 | @{$origins{$eventid}{$originid}} = ($time,$lat,$lon,$depth,$author,$catalog,$contributor,$contributoreventid); 922 | $totalorigins++; 923 | 924 | ($originid,$time,$lat,$lon,$depth,$author,$catalog,$contributor,$contributoreventid) = (undef) x 9; 925 | $inorigin = 0; 926 | } 927 | 928 | if ( $element->{Name} eq "magnitude" ) { 929 | @{$mags{$eventid}{$magnitudeid}} = ($magtype,$mag,$magauthor); 930 | 931 | ($magnitudeid,$mag,$magtype,$magauthor) = (undef) x 4; 932 | $inmagnitude = 0; 933 | } 934 | } 935 | 936 | if ( $inorigin ) { 937 | if ( $element->{Name} eq "time" ) { $intime = 0; } 938 | elsif ( $element->{Name} eq "latitude" ) { $inlat = 0; } 939 | elsif ( $element->{Name} eq "longitude" ) { $inlon = 0; } 940 | elsif ( $element->{Name} eq "depth" ) { $indepth = 0; } 941 | elsif ( $element->{Name} eq "creationInfo" ) { $increationInfo = 0; } 942 | 943 | if ( ($intime || $inlat || $inlon || $indepth || $increationInfo) && $element->{Name} eq "value" ) { $invalue = 0; } 944 | if ( $increationInfo && $element->{Name} eq "author" ) { $inauthor = 0; } 945 | } 946 | 947 | if ( $inmagnitude ) { 948 | if ( $element->{Name} eq "mag" ) { $inmag = 0; } 949 | elsif ( $element->{Name} eq "type" ) { $intype = 0; } 950 | elsif ( $element->{Name} eq "creationInfo" ) { $increationInfo = 0; } 951 | 952 | if ( $inmag && $element->{Name} eq "value" ) { $invalue = 0; } 953 | if ( $increationInfo && $element->{Name} eq "author" ) { $inauthor = 0; } 954 | } 955 | } 956 | 957 | sub characters { 958 | my ($self,$element) = @_; 959 | 960 | if ( defined $element->{Data} && $inevent ) { 961 | 962 | if ( $inpreferredOriginID ) { $porigin .= $element->{Data}; } 963 | elsif ( $inpreferredMagnitudeID ) { $pmagnitude .= $element->{Data}; } 964 | elsif ( $indescription && $intext ) { $name .= $element->{Data}; } 965 | 966 | elsif ( $inorigin ) { 967 | if ( $intime && $invalue ) { $time .= $element->{Data}; } 968 | elsif ($inlat && $invalue ) { $lat .= $element->{Data}; } 969 | elsif ($inlon && $invalue ) { $lon .= $element->{Data}; } 970 | elsif ($indepth && $invalue ) { $depth .= $element->{Data}; } 971 | elsif ($increationInfo && $inauthor ) { $author .= $element->{Data}; } 972 | } 973 | 974 | elsif ( $inmagnitude ) { 975 | if ( $inmag && $invalue ) { $mag .= $element->{Data}; } 976 | elsif ( $intype ) { $magtype .= $element->{Data}; } 977 | elsif ( $increationInfo && $inauthor ) { $magauthor .= $element->{Data}; } 978 | } 979 | } 980 | } # End of SAX ESHandler 981 | } # End of FetchMetaData() 982 | 983 | 984 | ###################################################################### 985 | # EDCallBack: 986 | # 987 | # A call back for LWP downloading. 988 | # 989 | # Add received data to eventxml string, tally up the received data 990 | # size and print and updated (overwriting) byte count string. 991 | ###################################################################### 992 | sub EDCallBack { 993 | my ($data, $response, $protocol) = @_; 994 | $datasize += length($data); 995 | 996 | if ( $response->content_encoding() =~ /gzip/ ) { 997 | my $datablock = ""; 998 | $inflater->inflate($data, $datablock); 999 | $eventxml .= $datablock; 1000 | } 1001 | else { 1002 | $eventxml .= $data; 1003 | } 1004 | 1005 | if ( $verbose && ! $nobsprint ) { 1006 | printf (STDERR "%-10.10s\b\b\b\b\b\b\b\b\b\b", sizestring($datasize)); 1007 | } 1008 | } 1009 | 1010 | 1011 | ###################################################################### 1012 | # sizestring (bytes): 1013 | # 1014 | # Return a clean size string for a given byte count. 1015 | ###################################################################### 1016 | sub sizestring { # sizestring (bytes) 1017 | my $bytes = shift; 1018 | 1019 | if ( $bytes < 1000 ) { 1020 | return sprintf "%d Bytes", $bytes; 1021 | } 1022 | elsif ( ($bytes / 1024) < 1000 ) { 1023 | return sprintf "%.1f KB", $bytes / 1024; 1024 | } 1025 | elsif ( ($bytes / 1024 / 1024) < 1000 ) { 1026 | return sprintf "%.1f MB", $bytes / 1024 / 1024; 1027 | } 1028 | elsif ( ($bytes / 1024 / 1024 / 1024) < 1000 ) { 1029 | return sprintf "%.1f GB", $bytes / 1024 / 1024 / 1024; 1030 | } 1031 | elsif ( ($bytes / 1024 / 1024 / 1024 / 1024) < 1000 ) { 1032 | return sprintf "%.1f TB", $bytes / 1024 / 1024 / 1024 / 1024; 1033 | } 1034 | else { 1035 | return ""; 1036 | } 1037 | } # End of sizestring() 1038 | 1039 | 1040 | ###################################################################### 1041 | # 1042 | # Package RequestAgent: a superclass for LWP::UserAgent with override 1043 | # of LWP::UserAgent methods to set default user agent and handle 1044 | # authentication credentials. 1045 | # 1046 | ###################################################################### 1047 | BEGIN { 1048 | use LWP; 1049 | package RequestAgent; 1050 | our @ISA = qw(LWP::UserAgent); 1051 | 1052 | sub new 1053 | { 1054 | my $self = LWP::UserAgent::new(@_); 1055 | my $fulluseragent = $useragent; 1056 | $fulluseragent .= " ($appname)" if ( $appname ); 1057 | $self->agent($fulluseragent); 1058 | $self; 1059 | } 1060 | 1061 | sub get_basic_credentials 1062 | { 1063 | my ($self, $realm, $uri) = @_; 1064 | 1065 | if ( defined $auth ) { 1066 | return split(':', $auth, 2); 1067 | } 1068 | elsif (-t) { 1069 | my $netloc = $uri->host_port; 1070 | print "\n"; 1071 | print "Enter username for $realm at $netloc: "; 1072 | my $user = ; 1073 | chomp($user); 1074 | return (undef, undef) unless length $user; 1075 | print "Password: "; 1076 | system("stty -echo"); 1077 | my $password = ; 1078 | system("stty echo"); 1079 | print "\n"; # because we disabled echo 1080 | chomp($password); 1081 | return ($user, $password); 1082 | } 1083 | else { 1084 | return (undef, undef) 1085 | } 1086 | } 1087 | } # End of LWP::UserAgent override 1088 | -------------------------------------------------------------------------------- /FetchMetadata: -------------------------------------------------------------------------------- 1 | #!/usr/bin/perl 2 | # 3 | # FetchMetadata 4 | # 5 | # Find the most current version at http://service.iris.edu/clients/ 6 | # 7 | # Fetch metadata from a web service. The default web service is from 8 | # the IRIS DMC, other FDSN web services may be specified by setting 9 | # the following environment variables: 10 | # 11 | # SERVICEBASE = the base URI of the service(s) to use (http://service.iris.edu/) 12 | # METADATAWS = complete URI of service (http://service.iris.edu/fdsnws/station/1) 13 | # 14 | # Dependencies: This script should run without problems on Perl 15 | # release 5.10 or newer, older versions of Perl might require the 16 | # installation of the following modules (and their dependencies): 17 | # XML::SAX 18 | # Bundle::LWP (libwww-perl) 19 | # 20 | # Installation of the XML::SAX::ExpatXS module can significantly 21 | # speed up the parsing of metadata results returned as XML. 22 | # 23 | ## Metadata selection 24 | # 25 | # Metadata is generally selected by specifying network, station, 26 | # location, channel, start time and end time. The name parameters may 27 | # contain wildcard characters. All input options are optional. 28 | # Selections may be specified one of three ways: 29 | # 30 | # 1) Command line arguments: -N, -S, -L, -C, -Q, -s, -e 31 | # 32 | # 2) A BREQ_FAST formatted file, http://www.iris.edu/manuals/breq_fast.htm 33 | # 34 | # 3) A selection file containing a list of: 35 | # Net Sta Loc Chan Start End 36 | # 37 | # Example selection file contents: 38 | # II BFO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 39 | # IU ANMO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 40 | # IU COLA 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 41 | # 42 | # For the command line arguments and the selection file the network, 43 | # station location and channel fields may contain the common * and ? 44 | # wildcards, meaning zero-to-many and a single character respectively. 45 | # These fields may also be comma-separated lists, for example, the 46 | # network may be specified as II,IU,TA to select three networks. 47 | # 48 | ## Data output 49 | # 50 | # A line will be written to the file for each channel epoch and will 51 | # contain: 52 | # "net|sta|loc|chan|lat|lon|elev|depth|azimuth|dip|instrument|scale|scalefreq|scaleunits|samplerate|start|end" 53 | # 54 | # This metadata file can be used directly with mseed2sac or tracedsp 55 | # to create SAC files including basic metadata. 56 | # 57 | # 58 | # ## Change history ## 59 | # 60 | # 2010.217: 61 | # - Initial version, split from FetchBulkData 2010.217 62 | # 63 | # 2010.273 64 | # - Add -A option to specify an application name/version, this will be 65 | # added to the UserAgent string. 66 | # 67 | # 2010.342 68 | # - Convert metadata fetching to use the production ws-station service 69 | # instead of the deprecated service previously used. 70 | # - Rearrange and add fields to the metadata file output lines. 71 | # 72 | # 2011.020 73 | # - Rework metadata parser for changed content (InstrumentSensitivity). 74 | # - Add diagnostics and more graceful failure to metadata retrieval. 75 | # 76 | # 2011.046 77 | # - Add -ua option to limit results to data updated after a date. 78 | # - Save received XML metadata to a temporary file, the PurePerl SAX 79 | # XML parser is much faster when working on files. 80 | # - Allow comma-separated lists for selection by changing the delimiter 81 | # for selection entries. 82 | # 83 | # 2011.089 84 | # - Fix error message to report unrecognized channels in BREQ_FAST files. 85 | # 86 | # 2011.164 87 | # - Add Perl and LWP version to UserAgent and wsclient URL to usage message. 88 | # - Print script name and local time when verbose. 89 | # - Add example of selection file and more comments on lists and wildcarding. 90 | # - Add -msurl option to override metadata service query URL. 91 | # - Add -sta option to request and parse station level information only 92 | # 93 | # 2011.172 94 | # - Remove newline and carriage return characters and trailing spaces from 95 | # site and instrument strings. 96 | # 97 | # 2011.337 98 | # - Print error message text from service on errors. 99 | # - Add undocumented -nobs option to prevent printing backspace characters in status. 100 | # - Remove "query" portion from base URLs, moving them to the service calls. 101 | # - Include local time in output of DONE line. 102 | # 103 | # 2012.045 104 | # - Print "No data available" when return code is 204 in addition to 404. 105 | # - Allow start/end times specified with microsecond resolution, subseconds are currently ignored. 106 | # - Allow -- location IDs in BREQ_FAST formatted selections. 107 | # 108 | # 2012.061 109 | # - Add -resp option to request response level information, no extra parsing 110 | # or printing of response level details is included, useful for saving the XML. 111 | # - Add index numbers to XML output file if more than one request is submitted 112 | # to the server. Using a selection list or BREQ_FAST file for input can generate 113 | # multiple requests, one request per line in fact. 114 | # 115 | # 2012.068: 116 | # - Use new starttime and endtime parameters for ws-station instead of 117 | # deprecated timewindow. These options can now be specified independently. 118 | # 119 | # 2012.100: 120 | # - Parse site name in a new location of StationXML, site is now placed in the 121 | # Site->Name element, was previously in the Site->Country element. 122 | # 123 | # 2012.102: 124 | # - Correctly parse epoch times from channel epoch by skipping the new channel 125 | # comment dates, this corrects the operating times in the metadata files. 126 | # 127 | # 2012.234: 128 | # - Remove sorting, print data in order returned by the service. 129 | # - Use vertical bars as field separators instead of commas, do not translate commas. 130 | # - Do not convert dip to SAC convention, leave it in SEED convention. 131 | # 132 | # 2013.077: 133 | # - Changed metadata parsing to understand FDSN StationXML schema. 134 | # - Use the LWP::UserAgent method env_proxy() to check for and use connection 135 | # proxy information from environment variables (e.g. http_proxy). 136 | # - Add checking of environment variables that will override the web 137 | # service base path (i.e. host name). 138 | # 139 | # 2013.186 140 | # - Change service URL override command line options to match 141 | # environment variables. 142 | # 143 | # 2013.197 144 | # - Fix parsing of element values of "0". 145 | # 146 | # 2013.198 147 | # - Add test for minimum version of LWP (libwww) module of 5.806. 148 | # 149 | # 2013.254 150 | # - Allow validation of (virtual) network codes to contain a dash 151 | # when parsing a BREQFAST file. 152 | # 153 | # 2014.056 154 | # - Allow gzip'ed HTTP encoding for metadata requests if support 155 | # exists on the local system. 156 | # 157 | # 2014.202 158 | # - Exit with non-zero value on web service errors. 159 | # 160 | # 2014.316 161 | # - Add -mts/-matchtimeseries option to turn submit the matchtimeseries 162 | # parameter to the metadata service. 163 | # 164 | # 2015.007 165 | # - Allow 1-3 character channel codes in breq_fast parsing, this allows 166 | # single and double character strings with wildcards. 167 | # 168 | # Author: Chad Trabant, IRIS Data Management Center 169 | 170 | use strict; 171 | use File::Basename; 172 | use Getopt::Long; 173 | use LWP 5.806; # Require minimum version 174 | use LWP::UserAgent; 175 | use HTTP::Status qw(status_message); 176 | use HTTP::Date; 177 | use Time::HiRes; 178 | 179 | my $version = "2015.007"; 180 | 181 | my $scriptname = basename($0); 182 | 183 | # Default web service base 184 | my $servicebase = 'http://service.iris.edu'; 185 | 186 | # Check for environment variable overrides for servicebase 187 | $servicebase = $ENV{'SERVICEBASE'} if ( exists $ENV{'SERVICEBASE'} ); 188 | 189 | # Default web service for metadata 190 | my $metadataservice = "$servicebase/fdsnws/station/1"; 191 | 192 | # Check for environment variable override for metadataservice 193 | $metadataservice = $ENV{'METADATAWS'} if ( exists $ENV{'METADATAWS'} ); 194 | 195 | # HTTP UserAgent reported to web services 196 | my $useragent = "$scriptname/$version Perl/$] " . new LWP::UserAgent->_agent; 197 | 198 | my $usage = undef; 199 | my $verbose = undef; 200 | my $nobsprint = undef; 201 | 202 | my $net = undef; 203 | my $sta = undef; 204 | my $loc = undef; 205 | my $chan = undef; 206 | my $starttime = undef; 207 | my $endtime = undef; 208 | my $updatedafter = undef; 209 | my $matchtimeseries = undef; 210 | my $selectfile = undef; 211 | my $bfastfile = undef; 212 | my $stalevel = undef; 213 | my $resplevel = undef; 214 | my $appname = undef; 215 | my $auth = undef; 216 | my $outfile = undef; 217 | my $xmlfile = undef; 218 | my $exitvalue = 0; 219 | 220 | my $inflater = undef; 221 | 222 | # If Compress::Raw::Zlib is available configure inflater for RFC 1952 (gzip) 223 | if ( eval("use Compress::Raw::Zlib; 1") ) { 224 | use Compress::Raw::Zlib; 225 | $inflater = new Compress::Raw::Zlib::Inflate( -WindowBits => WANT_GZIP, 226 | -ConsumeInput => 0 ); 227 | } 228 | 229 | # Parse command line arguments 230 | Getopt::Long::Configure ("bundling_override"); 231 | my $getoptsret = GetOptions ( 'help|usage|h' => \$usage, 232 | 'verbose|v+' => \$verbose, 233 | 'nobs' => \$nobsprint, 234 | 'net|N=s' => \$net, 235 | 'sta|S=s' => \$sta, 236 | 'loc|L=s' => \$loc, 237 | 'chan|C=s' => \$chan, 238 | 'starttime|s=s' => \$starttime, 239 | 'endtime|e=s' => \$endtime, 240 | 'updatedafter|ua=s' => \$updatedafter, 241 | 'matchtimeseries|mts' => \$matchtimeseries, 242 | 'selectfile|l=s' => \$selectfile, 243 | 'bfastfile|b=s' => \$bfastfile, 244 | 'stalevel|sta' => \$stalevel, 245 | 'resplevel|resp' => \$resplevel, 246 | 'appname|A=s' => \$appname, 247 | 'auth|a=s' => \$auth, 248 | 'outfile|o=s' => \$outfile, 249 | 'xmlfile|X=s' => \$xmlfile, 250 | 'metadataws=s' => \$metadataservice, 251 | ); 252 | 253 | my $required = ( defined $net || defined $sta || 254 | defined $loc || defined $chan || 255 | defined $starttime || defined $endtime || 256 | defined $selectfile || defined $bfastfile ); 257 | 258 | if ( ! $getoptsret || $usage || ! $required ) { 259 | print "$scriptname: collect channel metadata ($version)\n"; 260 | print "http://service.iris.edu/clients/\n\n"; 261 | print "Usage: $scriptname [options]\n\n"; 262 | print " Options:\n"; 263 | print " -v Increase verbosity, may be specified multiple times\n"; 264 | print " -N,--net Network code, list and wildcards (* and ?) accepted\n"; 265 | print " -S,--sta Station code, list and wildcards (* and ?) accepted\n"; 266 | print " -L,--loc Location ID, list and wildcards (* and ?) accepted\n"; 267 | print " -C,--chan Channel codes, list and wildcards (* and ?) accepted\n"; 268 | print " -s starttime Specify start time (YYYY-MM-DD,HH:MM:SS)\n"; 269 | print " -e endtime Specify end time (YYYY-MM-DD,HH:MM:SS)\n"; 270 | print " -ua date Limit to metadata updated after date (YYYY-MM-DD,HH:MM:SS)\n"; 271 | print " -mts Limit to metadata where criteria match time series data\n"; 272 | print " -X xmlfile Write raw returned FDSN StationXML to xmlfile\n"; 273 | print " -l listfile Read list of selections from file\n"; 274 | print " -b bfastfile Read list of selections from BREQ_FAST file\n"; 275 | print " -sta Print station level information, default is channel\n"; 276 | print " -resp Request response level information, no details printed\n"; 277 | print " -A appname Application/version string for identification\n"; 278 | print "\n"; 279 | print " -o outfile Write basic metadata to specified file instead of printing\n"; 280 | print "\n"; 281 | exit 1; 282 | } 283 | 284 | # Print script name and local time string 285 | if ( $verbose ) { 286 | my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time); 287 | printf STDERR "$scriptname ($version) at %4d-%02d-%02d %02d:%02d:%02d\n", $year+1900, $mon+1, $mday, $hour, $min, $sec; 288 | } 289 | 290 | # Normalize time strings 291 | if ( $starttime ) { 292 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $starttime); 293 | $starttime = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 294 | $starttime .= ".$subsec" if ( $subsec ); 295 | } 296 | 297 | if ( $endtime ) { 298 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $endtime); 299 | $endtime = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 300 | $endtime .= ".$subsec" if ( $subsec ); 301 | } 302 | 303 | if ( $updatedafter ) { 304 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $updatedafter); 305 | $updatedafter = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 306 | $updatedafter .= ".$subsec" if ( $subsec ); 307 | } 308 | 309 | # An array to hold data selections 310 | my @selections = (); 311 | 312 | # Add command line selection to list 313 | if ( defined $net || defined $sta || defined $loc || defined $chan || 314 | defined $starttime || defined $endtime ) { 315 | push (@selections,"$net|$sta|$loc|$chan|$starttime|$endtime"); 316 | } 317 | 318 | # Read selection list file 319 | if ( $selectfile ) { 320 | print STDERR "Reading data selection from list file '$selectfile'\n"; 321 | &ReadSelectFile ($selectfile); 322 | } 323 | 324 | # Read BREQ_FAST file 325 | if ( $bfastfile ) { 326 | print STDERR "Reading data selection from BREQ_FAST file '$bfastfile'\n"; 327 | &ReadBFastFile ($bfastfile); 328 | } 329 | 330 | # Report complete data selections 331 | if ( $verbose > 2 ) { 332 | print STDERR "== Data selections ==\n"; 333 | foreach my $select ( @selections ) { 334 | print STDERR " $select\n"; 335 | } 336 | } 337 | 338 | # An array to hold channel list and metadata 339 | my @metadata = (); 340 | my $metadataxml; 341 | my $datasize; 342 | 343 | # Fetch metadata from the station web service 344 | my $selectidx = 0; 345 | foreach my $selection ( @selections ) { 346 | my ($snet,$ssta,$sloc,$schan,$sstart,$send) = split (/\|/,$selection); 347 | &FetchMetaData($snet,$ssta,$sloc,$schan,$sstart,$send); 348 | $selectidx++; 349 | } 350 | 351 | # Write to either specified output file or stdout 352 | my $metafile = ( $outfile ) ? $outfile : "-"; 353 | 354 | # Write metadata to file 355 | if ( $metafile ) { 356 | if ( scalar @metadata <= 0 ) { 357 | printf STDERR "No metdata available\n", scalar @metadata; 358 | } 359 | elsif ( ! defined $stalevel ) { 360 | printf STDERR "Writing metadata (%d channel epochs) file\n", scalar @metadata if ( $verbose ); 361 | 362 | open (META, ">$metafile") || die "Cannot open metadata file '$metafile': $!\n"; 363 | 364 | # Print header line 365 | print META "#net|sta|loc|chan|lat|lon|elev|depth|azimuth|dip|instrument|scale|scalefreq|scaleunits|samplerate|start|end\n"; 366 | 367 | foreach my $channel ( @metadata ) { 368 | my ($net,$sta,$loc,$chan,$start,$end,$lat,$lon,$elev,$depth,$azimuth,$dip,$instrument,$samplerate,$sens,$sensfreq,$sensunit) = 369 | split (/\|/, $channel); 370 | 371 | $sensfreq = sprintf ("%0g", $sensfreq); 372 | $samplerate = sprintf ("%0g", $samplerate); 373 | 374 | print META "$net|$sta|$loc|$chan|$lat|$lon|$elev|$depth|$azimuth|$dip|$instrument|$sens|$sensfreq|$sensunit|$samplerate|$start|$end\n"; 375 | } 376 | 377 | close META; 378 | } 379 | else { 380 | printf STDERR "Writing metadata (%d station epochs) file\n", scalar @metadata if ( $verbose ); 381 | 382 | open (META, ">$metafile") || die "Cannot open metadata file '$metafile': $!\n"; 383 | 384 | # Print header line 385 | print META "#net|sta|lat|lon|elev|site|start|end\n"; 386 | 387 | foreach my $station ( @metadata ) { 388 | my ($net,$sta,$start,$end,$lat,$lon,$elev,$site) = split (/\|/, $station); 389 | 390 | print META "$net|$sta|$lat|$lon|$elev|$site|$start|$end\n"; 391 | } 392 | 393 | close META; 394 | } 395 | } 396 | 397 | my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time); 398 | printf STDERR "DONE at %4d-%02d-%02d %02d:%02d:%02d\n", $year+1900, $mon+1, $mday, $hour, $min, $sec; 399 | 400 | exit $exitvalue; 401 | ## End of main 402 | 403 | 404 | ###################################################################### 405 | # ReadSelectFile: 406 | # 407 | # Read selection list file and add entries to the @selections array. 408 | # 409 | # Selection lines are expected to be in the following form: 410 | # 411 | # "Net Sta Loc Chan Start End" 412 | # 413 | # The Net, Sta, Loc and Channel fields are required and can be 414 | # specified as wildcards. 415 | ###################################################################### 416 | sub ReadSelectFile { 417 | my $selectfile = shift; 418 | 419 | open (SF, "<$selectfile") || die "Cannot open '$selectfile': $!\n"; 420 | 421 | foreach my $line ( ) { 422 | chomp $line; 423 | next if ( $line =~ /^\#/ ); # Skip comment lines 424 | 425 | my ($net,$sta,$loc,$chan,$start,$end) = split (' ', $line); 426 | 427 | next if ( ! defined $chan ); 428 | 429 | # Normalize time strings 430 | if ( $start ) { 431 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $start); 432 | $start = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 433 | $start .= ".$subsec" if ( $subsec ); 434 | } 435 | 436 | if ( $end ) { 437 | my ($year,$month,$mday,$hour,$min,$sec,$subsec) = split (/[-:,.\s\/T]/, $end); 438 | $end = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $year, $month, $mday, $hour, $min, $sec); 439 | $end .= ".$subsec" if ( $subsec ); 440 | } 441 | 442 | # Add selection to global list 443 | push (@selections,"$net|$sta|$loc|$chan|$start|$end"); 444 | } 445 | 446 | close SF; 447 | } # End of ReadSelectFile() 448 | 449 | 450 | ###################################################################### 451 | # ReadBFastFile: 452 | # 453 | # Read BREQ_FAST file and add entries to the @selections array. 454 | # 455 | ###################################################################### 456 | sub ReadBFastFile { 457 | my $bfastfile = shift; 458 | 459 | open (BF, "<$bfastfile") || die "Cannot open '$bfastfile': $!\n"; 460 | 461 | my $linecount = 0; 462 | BFLINE: foreach my $line ( ) { 463 | chomp $line; 464 | $linecount++; 465 | next if ( ! $line ); # Skip empty lines 466 | 467 | next if ( $line =~ /^\./ ); # Skip other header lines 468 | 469 | my ($sta,$net,$syear,$smon,$sday,$shour,$smin,$ssec,$eyear,$emon,$eday,$ehour,$emin,$esec,$count,@chans) = split (' ', $line); 470 | 471 | # Simple validation of BREQ FAST fields 472 | if ( $sta !~ /^[A-Za-z0-9*?]{1,5}$/ ) { 473 | print "Unrecognized station code: '$sta', skipping line $linecount\n" if ( $verbose ); 474 | next; 475 | } 476 | if ( $net !~ /^[-_A-Za-z0-9*?]+$/ ) { 477 | print "Unrecognized network code: '$net', skipping line $linecount\n" if ( $verbose ); 478 | next; 479 | } 480 | if ( $syear !~ /^\d\d\d\d$/ ) { 481 | print "Unrecognized start year: '$syear', skipping line $linecount\n" if ( $verbose ); 482 | next; 483 | } 484 | if ( $smon !~ /^\d{1,2}$/ ) { 485 | print "Unrecognized start month: '$smon', skipping line $linecount\n" if ( $verbose ); 486 | next; 487 | } 488 | if ( $sday !~ /^\d{1,2}$/ ) { 489 | print "Unrecognized start day: '$sday', skipping line $linecount\n" if ( $verbose ); 490 | next; 491 | } 492 | if ( $shour !~ /^\d{1,2}$/ ) { 493 | print "Unrecognized start hour: '$shour', skipping line $linecount\n" if ( $verbose ); 494 | next; 495 | } 496 | if ( $smin !~ /^\d{1,2}$/ ) { 497 | print "Unrecognized start min: '$smin', skipping line $linecount\n" if ( $verbose ); 498 | next; 499 | } 500 | if ( $ssec !~ /^\d{1,2}\.?\d{0,6}?$/ ) { 501 | print "Unrecognized start seconds: '$ssec', skipping line $linecount\n" if ( $verbose ); 502 | next; 503 | } 504 | if ( $eyear !~ /^\d\d\d\d$/ ) { 505 | print "Unrecognized end year: '$eyear', skipping line $linecount\n" if ( $verbose ); 506 | next; 507 | } 508 | if ( $emon !~ /^\d{1,2}$/ ) { 509 | print "Unrecognized end month: '$emon', skipping line $linecount\n" if ( $verbose ); 510 | next; 511 | } 512 | if ( $eday !~ /^\d{1,2}$/ ) { 513 | print "Unrecognized end day: '$eday', skipping line $linecount\n" if ( $verbose ); 514 | next; 515 | } 516 | if ( $ehour !~ /^\d{1,2}$/ ) { 517 | print "Unrecognized end hour: '$ehour', skipping line $linecount\n" if ( $verbose ); 518 | next; 519 | } 520 | if ( $emin !~ /^\d{1,2}$/ ) { 521 | print "Unrecognized end min: '$emin', skipping line $linecount\n" if ( $verbose ); 522 | next; 523 | } 524 | if ( $esec !~ /^\d{1,2}\.?\d{0,6}?$/ ) { 525 | print "Unrecognized end seconds: '$esec', skipping line $linecount\n" if ( $verbose ); 526 | next; 527 | } 528 | if ( $count !~ /^\d+$/ || $count <= 0 ) { 529 | print "Invalid channel count field: '$count', skipping line $linecount\n" if ( $verbose ); 530 | next; 531 | } 532 | if ( scalar @chans <= 0 ) { 533 | print "No channels specified, skipping line $linecount\n" if ( $verbose ); 534 | next; 535 | } 536 | 537 | # Extract location ID if present, i.e. if channel count is one less than present 538 | my $loc = undef; 539 | $loc = pop @chans if ( scalar @chans == ($count+1) ); 540 | 541 | if ( $loc && $loc !~ /^[A-Za-z0-9*?\-]{1,2}$/ ) { 542 | print "Unrecognized location ID: '$loc', skipping line $linecount\n" if ( $verbose ); 543 | next; 544 | } 545 | 546 | foreach my $chan ( @chans ) { 547 | if ( $chan !~ /^[A-Za-z0-9*?]{1,3}$/ ) { 548 | print "Unrecognized channel codes: '$chan', skipping line $linecount\n" if ( $verbose ); 549 | next BFLINE; 550 | } 551 | } 552 | 553 | if ( scalar @chans != $count ) { 554 | printf "Channel count field ($count) does not match number of channels specified (%d), skipping line $linecount\n", 555 | scalar @chans if ( $verbose ); 556 | next; 557 | } 558 | 559 | # Normalize time strings 560 | my ($ssec,$ssub) = split (/\./, $ssec); 561 | my $start = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $syear, $smon, $sday, $shour, $smin, $ssec); 562 | $start .= ".$ssub" if ( $ssub ); 563 | my ($esec,$esub) = split (/\./, $esec); 564 | my $end = sprintf ("%04d-%02d-%02dT%02d:%02d:%02d", $eyear, $emon, $eday, $ehour, $emin, $esec); 565 | $end .= ".$esub" if ( $esub ); 566 | 567 | # Add selection to global list for each channel 568 | foreach my $chan ( @chans ) { 569 | push (@selections,"$net|$sta|$loc|$chan|$start|$end"); 570 | } 571 | } 572 | 573 | close BF; 574 | } # End of ReadBFastFile() 575 | 576 | 577 | ###################################################################### 578 | # FetchMetaData: 579 | # 580 | # Collect metadata and expand wildcards for selected data set. 581 | # 582 | # Resulting metadata is placed in the global @metadata array with each 583 | # entry taking the following form: 584 | # "net|sta|loc|chan|start|end|lat|lon|elev|depth|azimuth|dip|instrument|samplerate|sensitivity|sensfreq|sensunits" 585 | # 586 | ###################################################################### 587 | sub FetchMetaData { 588 | my ($rnet,$rsta,$rloc,$rchan,$rstart,$rend) = @_; 589 | 590 | # Create HTTP user agent 591 | my $ua = RequestAgent->new(); 592 | $ua->env_proxy; 593 | 594 | my $level = "channel"; 595 | $level = "station" if ( $stalevel ); 596 | $level = "response" if ( $resplevel ); 597 | 598 | # Create web service URI 599 | my $uri = "${metadataservice}/query?level=$level"; 600 | $uri .= "&network=$rnet" if ( $rnet ); 601 | $uri .= "&station=$rsta" if ( $rsta ); 602 | $uri .= "&location=$rloc" if ( $rloc ); 603 | $uri .= "&channel=$rchan" if ( $rchan ); 604 | $uri .= "&starttime=$rstart" if ( $rstart ); 605 | $uri .= "&endtime=$rend" if ( $rend ); 606 | $uri .= "&updatedafter=$updatedafter" if ( $updatedafter ); 607 | $uri .= "&matchtimeseries=TRUE" if ( $matchtimeseries ); 608 | 609 | my $ftime = Time::HiRes::time; 610 | 611 | print STDERR "Metadata URI: '$uri'\n" if ( $verbose > 1 ); 612 | 613 | print STDERR "Fetching metadata :: " if ( $verbose ); 614 | 615 | $datasize = 0; 616 | $metadataxml = ""; 617 | 618 | # Fetch metadata from web service using callback routine 619 | my $response = ( $inflater ) ? 620 | $ua->get($uri, 'Accept-Encoding' => 'gzip', ':content_cb' => \&MDCallBack ) : 621 | $ua->get($uri, ':content_cb' => \&MDCallBack ); 622 | 623 | $inflater->inflateReset if ( $inflater ); 624 | 625 | if ( $response->code == 204 ) { 626 | print (STDERR "No data available\n") if ( $verbose ); 627 | return; 628 | } 629 | elsif ( ! $response->is_success() ) { 630 | print (STDERR "Error fetching data: " 631 | . $response->code . " :: " . status_message($response->code) . "\n"); 632 | print STDERR "------\n" . $response->decoded_content . "\n------\n"; 633 | print STDERR " URI: '$uri'\n" if ( $verbose > 1 ); 634 | 635 | $exitvalue = 1; 636 | } 637 | else { 638 | printf (STDERR "%s\n", ($nobsprint)?sizestring($datasize):"") if ( $verbose ); 639 | } 640 | 641 | my $duration = Time::HiRes::time - $ftime; 642 | my $rate = $datasize/(($duration)?$duration:0.000001); 643 | printf (STDERR "Received %s of metadata in %.1f seconds (%s/s)\n", 644 | sizestring($datasize), $duration, sizestring($rate)); 645 | 646 | # Return if no metadata received 647 | return if ( length $metadataxml <= 0 ); 648 | 649 | # Create stream oriented XML parser instance 650 | use XML::SAX; 651 | my $parser = undef; 652 | if ( ! defined $stalevel ) { 653 | $parser = new XML::SAX::ParserFactory->parser( Handler => MDSHandlerChannel->new ); 654 | } 655 | else { 656 | $parser = new XML::SAX::ParserFactory->parser( Handler => MDSHandlerStation->new ); 657 | } 658 | 659 | my $totalepochs = 0; 660 | 661 | my $ptime = Time::HiRes::time; 662 | 663 | print STDERR "Parsing XML metadata... " if ( $verbose ); 664 | 665 | # Open file to store metadata XML 666 | my $xmlfileidx = ( $selectidx ) ? "$xmlfile.$selectidx" : $xmlfile; # Make separate files for each request 667 | my $metadataxmlfile = ( $xmlfile ) ? $xmlfileidx : "metadata-$$.xml"; 668 | if ( open (MXML, ">$metadataxmlfile") ) { 669 | # Write XML and close file 670 | print MXML $metadataxml; 671 | close MXML; 672 | 673 | # Parse XML metadata from file 674 | $parser->parse_file ($metadataxmlfile); 675 | 676 | # Remove temporary XML metadata file 677 | if ( ! defined $xmlfile ) { 678 | if ( ! unlink $metadataxmlfile ) { 679 | print STDERR "Cannot remove temporary XML metadata file: $!\n"; 680 | } 681 | } 682 | } 683 | # Otherwise parse the XML in memory 684 | else { 685 | printf STDERR " in memory (possibly slow), " if ( $verbose ); 686 | 687 | # Parse XML metadata from string 688 | $parser->parse_string ($metadataxml); 689 | } 690 | 691 | printf STDERR "Done (%.1f seconds)\n", Time::HiRes::time - $ptime if ( $verbose ); 692 | 693 | my $duration = Time::HiRes::time - $ftime; 694 | my $rate = $datasize/(($duration)?$duration:0.000001); 695 | printf (STDERR "Processed metadata for $totalepochs %s epochs in %.1f seconds (%s/s)\n", 696 | ($stalevel)?"station":"channel",$duration, sizestring($rate)); 697 | 698 | ## End of this routine, below is the XML parsing handlers used above 699 | 700 | ## Beginning of SAX MDSHandlerChannel, event-based streaming XML parsing for Channel level FDSN StationXML 701 | package MDSHandlerChannel; 702 | use base qw(XML::SAX::Base); 703 | use HTTP::Date; 704 | 705 | my $inchannel = 0; 706 | my $inlat = 0; 707 | my $inlon = 0; 708 | my $inelevation = 0; 709 | my $indepth = 0; 710 | my $inazimuth = 0; 711 | my $indip = 0; 712 | my $insensor = 0; 713 | my $insensortype = 0; 714 | my $insamplerate = 0; 715 | 716 | my $ininstsens = 0; 717 | my $insensvalue = 0; 718 | my $insensfreq = 0; 719 | my $ininputunits = 0; 720 | my $inunitname = 0; 721 | 722 | my ($net,$sta,$loc,$chan,$start,$end,$lat,$lon,$elev,$depth,$azimuth,$dip,$instrument,$samplerate,$sens,$sensfreq,$sensunit) = (undef) x 17; 723 | 724 | sub start_element { 725 | my ($self,$element) = @_; 726 | 727 | if ( $element->{Name} eq "Network" ) { 728 | $net = $element->{Attributes}->{'{}code'}->{Value}; 729 | } 730 | 731 | elsif ( $element->{Name} eq "Station" ) { 732 | ($sta,$loc,$chan,$start,$end,$lat,$lon,$elev,$depth,$azimuth,$dip,$instrument,$samplerate,$sens,$sensfreq,$sensunit) = (undef) x 16; 733 | 734 | $sta = $element->{Attributes}->{'{}code'}->{Value}; 735 | } 736 | 737 | elsif ( $element->{Name} eq "Channel" ) { 738 | $loc = $element->{Attributes}->{'{}locationCode'}->{Value}; 739 | $chan = $element->{Attributes}->{'{}code'}->{Value}; 740 | $start = $element->{Attributes}->{'{}startDate'}->{Value}; 741 | $end = $element->{Attributes}->{'{}endDate'}->{Value}; 742 | $inchannel = 1; 743 | } 744 | 745 | if ( $inchannel ) { 746 | if ( $element->{Name} eq "Latitude" ) { $inlat = 1; } 747 | elsif ( $element->{Name} eq "Longitude" ) { $inlon = 1; } 748 | elsif ( $element->{Name} eq "Elevation" ) { $inelevation = 1; } 749 | elsif ( $element->{Name} eq "Depth" ) { $indepth = 1; } 750 | elsif ( $element->{Name} eq "Azimuth" ) { $inazimuth = 1; } 751 | elsif ( $element->{Name} eq "Dip" ) { $indip = 1; } 752 | elsif ( $element->{Name} eq "SampleRate" ) { $insamplerate = 1; } 753 | elsif ( $element->{Name} eq "Sensor" ) { $insensor = 1; } 754 | elsif ( $element->{Name} eq "InstrumentSensitivity" ) { $ininstsens = 1; } 755 | } 756 | 757 | if ( $insensor ) { 758 | if ( $element->{Name} eq "Type" ) { $insensortype = 1; } 759 | } 760 | 761 | if ( $ininstsens ) { 762 | if ( $element->{Name} eq "Value" ) { $insensvalue = 1; } 763 | elsif ( $element->{Name} eq "Frequency" ) { $insensfreq = 1; } 764 | elsif ( $element->{Name} eq "InputUnits" ) { $ininputunits = 1; } 765 | } 766 | 767 | if ( $ininputunits ) { 768 | if ( $element->{Name} eq "Name" ) { $inunitname = 1; } 769 | } 770 | } 771 | 772 | sub end_element { 773 | my ($self,$element) = @_; 774 | 775 | if ( $element->{Name} eq "Network" ) { 776 | $net = 0; 777 | } 778 | 779 | elsif ( $element->{Name} eq "Station" ) { 780 | $sta = 0; 781 | } 782 | 783 | elsif ( $element->{Name} eq "Channel" ) { 784 | # Track epoch count 785 | $totalepochs++; 786 | 787 | # Translate metadata location ID to "--" if it's spaces 788 | my $dloc = ( $loc eq " " ) ? "--" : $loc; 789 | 790 | # Remove newlines, returns & trailing spaces in metadata instrument name 791 | $instrument =~ s/[\n\r]//g; 792 | $instrument =~ s/\s*$//g; 793 | 794 | # Cleanup start and end strings 795 | ($start) = $start =~ /^(\d{4,4}[-\/,:]\d{1,2}[-\/,:]\d{1,2}[-\/,:T]\d{1,2}[-\/,:]\d{1,2}[-\/,:]\d{1,2}).*/; 796 | ($end) = $end =~ /^(\d{4,4}[-\/,:]\d{1,2}[-\/,:]\d{1,2}[-\/,:T]\d{1,2}[-\/,:]\d{1,2}[-\/,:]\d{1,2}).*/; 797 | 798 | # Push channel epoch metadata into storage array 799 | push (@metadata, "$net|$sta|$dloc|$chan|$start|$end|$lat|$lon|$elev|$depth|$azimuth|$dip|$instrument|$samplerate|$sens|$sensfreq|$sensunit"); 800 | 801 | # Reset Epoch level fields 802 | ($loc,$chan,$start,$end,$lat,$lon,$elev,$depth,$azimuth,$dip,$instrument,$samplerate,$sens,$sensfreq,$sensunit) = (undef) x 15; 803 | $inchannel = 0; 804 | } 805 | 806 | if ( $inchannel ) { 807 | if ( $element->{Name} eq "Latitude" ) { $inlat = 0; } 808 | elsif ( $element->{Name} eq "Longitude" ) { $inlon = 0; } 809 | elsif ( $element->{Name} eq "Elevation" ) { $inelevation = 0; } 810 | elsif ( $element->{Name} eq "Depth" ) { $indepth = 0; } 811 | elsif ( $element->{Name} eq "Azimuth" ) { $inazimuth = 0; } 812 | elsif ( $element->{Name} eq "Dip" ) { $indip = 0; } 813 | elsif ( $element->{Name} eq "SampleRate" ) { $insamplerate = 0; } 814 | elsif ( $element->{Name} eq "Sensor" ) { $insensor = 0; } 815 | elsif ( $element->{Name} eq "InstrumentSensitivity" ) { $ininstsens = 0; } 816 | } 817 | 818 | if ( $insensor ) { 819 | if ( $element->{Name} eq "Type" ) { $insensortype = 0; } 820 | } 821 | 822 | if ( $ininstsens ) { 823 | if ( $element->{Name} eq "Value" ) { $insensvalue = 0; } 824 | elsif ( $element->{Name} eq "Frequency" ) { $insensfreq = 0; } 825 | elsif ( $element->{Name} eq "InputUnits" ) { $ininputunits = 0; } 826 | } 827 | 828 | if ( $ininputunits ) { 829 | if ( $element->{Name} eq "Name" ) { $inunitname = 0; } 830 | } 831 | } 832 | 833 | sub characters { 834 | my ($self,$element) = @_; 835 | 836 | if ( defined $element->{Data} ) { 837 | if ( $inlat ) { $lat .= $element->{Data}; } 838 | elsif ( $inlon ) { $lon .= $element->{Data}; } 839 | elsif ( $inelevation ) { $elev .= $element->{Data}; } 840 | elsif ( $indepth ) { $depth .= $element->{Data}; } 841 | elsif ( $inazimuth ) { $azimuth .= $element->{Data}; } 842 | elsif ( $indip ) { $dip .= $element->{Data}; } 843 | elsif ( $insamplerate ) { $samplerate .= $element->{Data}; } 844 | 845 | elsif ( $insensortype ) { $instrument .= $element->{Data}; } 846 | 847 | elsif ( $insensvalue ) { $sens .= $element->{Data}; } 848 | elsif ( $insensfreq ) { $sensfreq .= $element->{Data}; } 849 | elsif ( $inunitname ) { $sensunit .= $element->{Data}; } 850 | } 851 | } # End of SAX MDSHandlerChannel 852 | 853 | ## Beginning of SAX MDSHandlerStation, event-based streaming XML parsing for Station level FDSN StationXML 854 | package MDSHandlerStation; 855 | use base qw(XML::SAX::Base); 856 | use HTTP::Date; 857 | 858 | my $instation = 0; 859 | my $inlat = 0; 860 | my $inlon = 0; 861 | my $inelevation = 0; 862 | my $insite = 0; 863 | my $inname = 0; 864 | 865 | my ($net,$sta,$start,$end,$lat,$lon,$elev,$site) = (undef) x 8; 866 | 867 | sub start_element { 868 | my ($self,$element) = @_; 869 | 870 | if ( $element->{Name} eq "Network" ) { 871 | $net = $element->{Attributes}->{'{}code'}->{Value}; 872 | } 873 | 874 | elsif ( $element->{Name} eq "Station" ) { 875 | ($sta,$start,$end,$lat,$lon,$elev,$site) = (undef) x 7; 876 | 877 | $sta = $element->{Attributes}->{'{}code'}->{Value}; 878 | $start = $element->{Attributes}->{'{}startDate'}->{Value}; 879 | $end = $element->{Attributes}->{'{}endDate'}->{Value}; 880 | $instation = 1; 881 | } 882 | 883 | if ( $instation ) { 884 | if ( $element->{Name} eq "Latitude" ) { $inlat = 1; } 885 | elsif ( $element->{Name} eq "Longitude" ) { $inlon = 1; } 886 | elsif ( $element->{Name} eq "Elevation" ) { $inelevation = 1; } 887 | elsif ( $element->{Name} eq "Site" ) { $insite = 1; } 888 | } 889 | 890 | if ( $insite ) { 891 | if ( $element->{Name} eq "Name" ) { $inname = 1; } 892 | } 893 | } 894 | 895 | sub end_element { 896 | my ($self,$element) = @_; 897 | 898 | if ( $element->{Name} eq "Network" ) { 899 | $net = 0; 900 | } 901 | 902 | elsif ( $element->{Name} eq "Station" ) { 903 | # Track epoch count 904 | $totalepochs++; 905 | 906 | # Remove newlines, returns and trailing spaces in site name 907 | $site =~ s/[\n\r]//g; 908 | $site =~ s/\s*$//g; 909 | 910 | # Cleanup start and end strings 911 | ($start) = $start =~ /^(\d{4,4}[-\/,:]\d{1,2}[-\/,:]\d{1,2}[-\/,:T]\d{1,2}[-\/,:]\d{1,2}[-\/,:]\d{1,2}).*/; 912 | ($end) = $end =~ /^(\d{4,4}[-\/,:]\d{1,2}[-\/,:]\d{1,2}[-\/,:T]\d{1,2}[-\/,:]\d{1,2}[-\/,:]\d{1,2}).*/; 913 | 914 | # Push channel epoch metadata into storage array 915 | push (@metadata, "$net|$sta|$start|$end|$lat|$lon|$elev|$site"); 916 | 917 | # Reset StationEpoch level fields 918 | ($sta,$start,$end,$lat,$lon,$elev,$site) = (undef) x 7; 919 | $instation = 0; 920 | } 921 | 922 | if ( $instation ) { 923 | if ( $element->{Name} eq "Latitude" ) { $inlat = 0; } 924 | elsif ( $element->{Name} eq "Longitude" ) { $inlon = 0; } 925 | elsif ( $element->{Name} eq "Elevation" ) { $inelevation = 0; } 926 | elsif ( $element->{Name} eq "Site" ) { $insite = 0; } 927 | } 928 | 929 | if ( $insite ) { 930 | if ( $element->{Name} eq "Name" ) { $inname = 0; } 931 | } 932 | } 933 | 934 | sub characters { 935 | my ($self,$element) = @_; 936 | 937 | if ( defined $element->{Data} ) { 938 | if ( $inlat ) { $lat .= $element->{Data}; } 939 | elsif ( $inlon ) { $lon .= $element->{Data}; } 940 | elsif ( $inelevation ) { $elev .= $element->{Data}; } 941 | 942 | elsif ( $inname ) { $site .= $element->{Data}; } 943 | } 944 | } # End of SAX MDSHandlerStation 945 | } # End of FetchMetaData() 946 | 947 | 948 | ###################################################################### 949 | # MDCallBack: 950 | # 951 | # A call back for LWP downloading of metadata. 952 | # 953 | # Add received data to metadataxml string, tally up the received data 954 | # size and print and updated (overwriting) byte count string. 955 | ###################################################################### 956 | sub MDCallBack { 957 | my ($data, $response, $protocol) = @_; 958 | $datasize += length($data); 959 | 960 | if ( $response->content_encoding() =~ /gzip/ ) { 961 | my $datablock = ""; 962 | $inflater->inflate($data, $datablock); 963 | $metadataxml .= $datablock; 964 | } 965 | else { 966 | $metadataxml .= $data; 967 | } 968 | 969 | if ( $verbose && ! $nobsprint ) { 970 | printf (STDERR "%-10.10s\b\b\b\b\b\b\b\b\b\b", sizestring($datasize)); 971 | } 972 | } 973 | 974 | 975 | ###################################################################### 976 | # sizestring (bytes): 977 | # 978 | # Return a clean size string for a given byte count. 979 | ###################################################################### 980 | sub sizestring { # sizestring (bytes) 981 | my $bytes = shift; 982 | 983 | if ( $bytes < 1000 ) { 984 | return sprintf "%d Bytes", $bytes; 985 | } 986 | elsif ( ($bytes / 1024) < 1000 ) { 987 | return sprintf "%.1f KB", $bytes / 1024; 988 | } 989 | elsif ( ($bytes / 1024 / 1024) < 1000 ) { 990 | return sprintf "%.1f MB", $bytes / 1024 / 1024; 991 | } 992 | elsif ( ($bytes / 1024 / 1024 / 1024) < 1000 ) { 993 | return sprintf "%.1f GB", $bytes / 1024 / 1024 / 1024; 994 | } 995 | elsif ( ($bytes / 1024 / 1024 / 1024 / 1024) < 1000 ) { 996 | return sprintf "%.1f TB", $bytes / 1024 / 1024 / 1024 / 1024; 997 | } 998 | else { 999 | return ""; 1000 | } 1001 | } # End of sizestring() 1002 | 1003 | 1004 | ###################################################################### 1005 | # 1006 | # Package RequestAgent: a superclass for LWP::UserAgent with override 1007 | # of LWP::UserAgent methods to set default user agent and handle 1008 | # authentication credentials. 1009 | # 1010 | ###################################################################### 1011 | BEGIN { 1012 | use LWP; 1013 | package RequestAgent; 1014 | our @ISA = qw(LWP::UserAgent); 1015 | 1016 | sub new 1017 | { 1018 | my $self = LWP::UserAgent::new(@_); 1019 | my $fulluseragent = $useragent; 1020 | $fulluseragent .= " ($appname)" if ( $appname ); 1021 | $self->agent($fulluseragent); 1022 | $self; 1023 | } 1024 | 1025 | sub get_basic_credentials 1026 | { 1027 | my ($self, $realm, $uri) = @_; 1028 | 1029 | if ( defined $auth ) { 1030 | return split(':', $auth, 2); 1031 | } 1032 | elsif (-t) { 1033 | my $netloc = $uri->host_port; 1034 | print "\n"; 1035 | print "Enter username for $realm at $netloc: "; 1036 | my $user = ; 1037 | chomp($user); 1038 | return (undef, undef) unless length $user; 1039 | print "Password: "; 1040 | system("stty -echo"); 1041 | my $password = ; 1042 | system("stty echo"); 1043 | print "\n"; # because we disabled echo 1044 | chomp($password); 1045 | return ($user, $password); 1046 | } 1047 | else { 1048 | return (undef, undef) 1049 | } 1050 | } 1051 | } # End of LWP::UserAgent override 1052 | -------------------------------------------------------------------------------- /FetchSyn: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # 3 | # FetchSyn 4 | # 5 | # For help type: FetchSyn --u 6 | # 7 | # FetchSyn is a python command line client to download synthetic 8 | # seismograms from the IRIS Synthetics Engine (Syngine). 9 | # Find the most current version at http://service.iris.edu/clients/ 10 | # 11 | # SYNGINE OVERVIEW: http://ds.iris.edu/ds/products/syngine 12 | # SYNGINE MODELS: http://ds.iris.edu/ds/products/syngine/#models 13 | # SYNGINE PARAMETERS: http://service.iris.edu/irisws/syngine/1 14 | # 15 | # Python Dependencies: 16 | # *must have the requests package 17 | # *currently written for python2 and python3 18 | # 19 | # FetchSyn usage: A source and receiver is required, everything else has a default. 20 | # 21 | # RECEIVER: one of the following is required 22 | # A) rlat Receiver longitude 23 | # rlon Receiver latitude 24 | # B) sta,S Station name. use * to wildcard. 25 | # net,N Network name 26 | # Notes: *if only a net is given, then all sta will be included 27 | # *virtual networks are valid networks, e.g. _GSN 28 | # C) recfile Textfile with a list of receivers as (NET STA) or 29 | # (RLAT RLON [netcode stacode loccode]) 30 | # Valid examples in a recfile: 31 | # IU ANMO 32 | # II * 33 | # -89 -179 34 | # -89 -178 S0002 35 | # -89 -177 N2 S0003 L2 36 | # -89 -176 netcode=N2 stacode=S0004 loccode=L3 37 | # 38 | # SOURCES: one of the following is required 39 | # A) evid Event id [catalog]:[eventid] 40 | # B) slat source latitude 41 | # slon source longitude 42 | # sdepm source depth in meters 43 | # b1) mt source moment tensor in NM: mrr,mtt,mpp,mrt,mrp,mtp 44 | # b2) dc source double couple: strike,dip,rake [,scalar moment in NM] 45 | # b3) force source force in NM: Fr,Ft,Fp 46 | # 47 | # ORIGIN/START/END TIME (optional) 48 | # o Origin Time. Must be absolute (YYYY-MM-DD,HH:MM:SS) or (YYYY,MM,DD,HH,MM,SS) 49 | # s Start Time. Either 1) absolute time (YYYY-MM-DD,HH:MM:SS) 50 | # 2) phase-relative offset e.g. P+10 or ScS-100 51 | # 3) an offset from origin in seconds 52 | # e End Time. Either 1) absolute time (YYYY-MM-DD,HH:MM:SS) 53 | # 2) phase-relative offset e.g. P+10 or ScS-100 54 | # 3) an offset (duration) from start time in seconds 55 | # 56 | # 57 | # OUTPUT: 58 | # Output files are either in miniSEED (mseed) or a zip archive of SAC 59 | # files (saczip). SAC files will have the headers populated. 60 | # 61 | # QUICK EXAMPLES 62 | # FetchSyn -recfile rec.txt -slat 38.3 -slon 142.3 -sdepm 24400 -dc 203,10,88,5.3e22 63 | # FetchSyn -N IU -S ANMO,KIP -evid GCMT:M201103110546A 64 | # FetchSyn -N IU -S ANMO,KIP -evid GCMT:M201103110546A -sdepm 35000 65 | # FetchSyn -N _GSN -evid GCMT:M201103110546A -model iasp91_2s -dt 0.05 -label TOHOKU 66 | # -format mseed -C ZRT -units velocity -s P-10 -e 300 67 | # 68 | # ## Change history ## 69 | # 70 | # 2015.343: 71 | # -Initial release 72 | # 73 | # 2015:345: 74 | # -Removed modelname validation, since server already does that. 75 | # -added .strip() to slat & slon to remove whitespace when negative values added 76 | # -updated error messaging 77 | # 78 | # 2015.352: 79 | # -added support for the doublecouple, momenttensors & force sources. 80 | # -added .strip() to rlat,rlon,dc,mt, and force to remove whitespace when negative 81 | # values added 82 | # -modified argparse flags to accept single dash arguments 83 | # 84 | # 2016.007 85 | # -added user-agent info sent with each request as well as user-added agent info. 86 | # 87 | # Author: Alex Hutko, IRIS Data Management Center 88 | 89 | from __future__ import print_function 90 | 91 | FSversion = '2016.007' 92 | 93 | import requests 94 | import datetime 95 | import re 96 | import sys 97 | import time 98 | import argparse 99 | import csv 100 | try: 101 | import StringIO 102 | pythonversion = 2 103 | except ImportError: 104 | from io import BytesIO 105 | pythonversion = 3 106 | 107 | ######################## 108 | # Used to check if an input number is an actual number and within bounds 109 | def check_number(s,min,max): 110 | try: 111 | float(s) 112 | s=float(s) 113 | if ( s >= min and s <= max ): 114 | ierror = 0 115 | else: 116 | ierror = 1 117 | except: 118 | ierror = 2 119 | return ierror 120 | ######################## 121 | 122 | CurrentTime0 = time.time() 123 | 124 | for i, arg in enumerate(sys.argv): 125 | if (arg[0] == '-') and arg[1].isdigit(): sys.argv[i] = ' ' + arg 126 | 127 | parser = argparse.ArgumentParser() 128 | 129 | parser.add_argument('-A','--A', action='store', dest='agent',help='UserAgent info') 130 | 131 | parser.add_argument('-recfile','--recfile','--ReceiverFile', action='store', dest='ReceiverFile',help='receiver file name with NET STATION') 132 | parser.add_argument('-u','--u', action='store_true', dest='helpmenu',help='extended HELP MENU') 133 | parser.add_argument('-outfile','--outfile', action='store', dest='outfile',help='output file label' ) 134 | 135 | parser.add_argument('-model','--model', action='store', dest='model',help='model from list: http://ds.iris.edu/ds/products/syngine' ) 136 | parser.add_argument('-format','--format', action='store', dest='formatstring',help='miniseed or saczip') 137 | parser.add_argument('-label','--label', action='store', dest='label',help='label') 138 | parser.add_argument('-C','--C','--components', action='store', dest='components',help='components: any combination of ZNERT') 139 | parser.add_argument('-units','--units', action='store', dest='units',help='displacement, velocity, or acceleration (or d,v,a)') 140 | parser.add_argument('-scale','--scale', action='store', dest='scale',help='amplitude scaling factor') 141 | parser.add_argument('-dt','--dt', action='store', dest='dt',help='sampling interval in sec') 142 | parser.add_argument('-kernelwidth','--kernelwidth', action='store', dest='kernelwidth',help='width of sinc resampling kernel') 143 | 144 | parser.add_argument('-origin','-origintime','--origin','--origintime', action='store', dest='originstring',help='end time YYYY-MM-DDTHH:MM:SS') 145 | parser.add_argument('-s','-start','-starttime','--s','--start','--starttime', action='store', dest='startstring',help='source time YYYY-MM-DDTHH:MM:SS or PHASE+-offset') 146 | parser.add_argument('-e','-end','-endtime','--e','--end','-endtime', action='store', dest='endstring',help='end time YYYY-MM-DDTHH:MM:SS or PHASE+-offset or offset_from_start') 147 | 148 | parser.add_argument('-N','-network','--N','--network', action='store', dest='network',help='network') 149 | parser.add_argument('-S','-station','--S','--station', action='store', dest='station',help='station') 150 | parser.add_argument('-rlat','--rlat', action='store', dest='rlat',help='reciver lat') 151 | parser.add_argument('-rlon','--rlon', action='store', dest='rlon',help='receiver lon') 152 | parser.add_argument('-Ncode','-networkcode','--Ncode','--networkcode', action='store', dest='netcode',help='networkcode') 153 | parser.add_argument('-Scode','-stationcode','--Scode','--stationcode', action='store', dest='stacode',help='stationcode') 154 | parser.add_argument('-Lcode','-locationcode','--Lcode','--locationcode', action='store', dest='loccode',help='locationcode') 155 | 156 | parser.add_argument('-evid','-eventid','--evid','--eventid', action='store', dest='evid',help='event id') 157 | parser.add_argument('-slat','--slat', action='store', dest='slat',help='source lat') 158 | parser.add_argument('-slon','--slon', action='store', dest='slon',help='source lon') 159 | parser.add_argument('-sdepm','--sdepm', action='store', dest='sdepm',help='source depth in meters') 160 | 161 | parser.add_argument('-mt','--mt','--sourcemomenttensor', action='store', dest='momentstring',help='moment: mrr,mtt,mpp,mrt,mrp,mtp in N-m (1 J = 1 N-m = 1e7 dyne-cm). Comma delimited, no spaces or brackets.') 162 | parser.add_argument('-dc','--dc','--sourcedoublecouple', action='store', dest='doublecouplestring',help='strike,dip,rake[,ScalarMoment_in_NM]') 163 | parser.add_argument('-force','--force','-forces','--forces','--sourceforce', action='store', dest='forcestring',help='force source in NM: Fr,Ft,Fp') 164 | 165 | results = parser.parse_args() 166 | helpmenu = results.helpmenu 167 | agent = results.agent 168 | ReceiverFile = results.ReceiverFile 169 | evid = results.evid 170 | slat = results.slat 171 | slon = results.slon 172 | sdepm = results.sdepm 173 | rlat = results.rlat 174 | rlon = results.rlon 175 | networkinput = results.network 176 | stationinput = results.station 177 | netcode = results.netcode 178 | stacode = results.stacode 179 | loccode = results.loccode 180 | network = results.network 181 | station = results.station 182 | momentstring = results.momentstring 183 | doublecouplestring = results.doublecouplestring 184 | forcestring = results.forcestring 185 | startstring = results.startstring 186 | originstring = results.originstring 187 | endstring = results.endstring 188 | components = results.components 189 | units = results.units 190 | scale = results.scale 191 | dt = results.dt 192 | kernelwidth = results.kernelwidth 193 | modelstring = results.model 194 | formatstring = results.formatstring 195 | outfile = results.outfile 196 | label = results.label 197 | 198 | #---------- help menu 199 | 200 | if ( helpmenu is True ): 201 | print ('') 202 | print ('FetchSyn: collect Syngine synthetic seismograms (version 2015.300)') 203 | print ('http://service.iris.edu/clients') 204 | print ('') 205 | print ('Usage: FetchSyn [options]') 206 | print ('') 207 | print (' EXAMPLES') 208 | print (' FetchSyn -recfile rec.txt -slat 38.3 -slon 142.3 -sdepm 24400 -dc 203,10,88,5.3e22') 209 | print (' FetchSyn -N IU -S ANMO,KIP -evid GCMT:M201103110546A') 210 | print (' FetchSyn -N _GSN -evid GCMT:M201103110546A -model iasp91_2s -dt 0.05 -label TOHOKU') 211 | print (' -format mseed -C ZRT -units velocity -s P-10 -e 300') 212 | print (' ') 213 | print ('Options: ') 214 | print ('------------- At least one receiver required -----') 215 | print ('-recfile Text file with list of receivers (NET STA) or ') 216 | print (' (RLAT RLON [netcode stacode loccode])') 217 | print ('-N, -net Network name. Virtual nets valid e.g. : _GSN') 218 | print ('-S, -sta Station name, or comma separated names') 219 | print ('-rlat Receiver latitude') 220 | print ('-rlon Receiver longitude') 221 | print ('-Ncode,-networkcode Network name when rlat/rlon used') 222 | print ('-Scode,-stationcode Station name when rlat/rlon used') 223 | print ('-Lcode,-locationcode Location code for synthetic') 224 | print ('------------- A source is required ---------------') 225 | print ('-evid Event id [catalog]:[eventid] e.g. GCMT:C201002270634A' ) 226 | print ('-slat Source latitude') 227 | print ('-slon Source longitude ') 228 | print ('-sdepm Source depth in meters ') 229 | print ('-mt Source moment tensor in NM (mrr,mtt,mpp,mrt,mrp,mtp)') 230 | print ('-dc Source double couple (strike,dip,rake [,scalar moment])') 231 | print ('-force Source force in NM (Fr,Ft,Fp)') 232 | print ('------------- Optional --------------------------') 233 | print ('-model Model name http://ds.iris.edu/ds/products/syngine/#models') 234 | print ('-format Either miniseed or saczip for output') 235 | print ('-label Apply a label to be included in file names') 236 | print ('-C, -components Orientation, any combination of ZNERT') 237 | print ('-units Either (D)displacement, (V)velocity or (A)acceleration') 238 | print ('-dt Sampling interval in seconds') 239 | print ('-kernelwidth Width of the sinc kernel used for resampling') 240 | print ('-o, -origintime Source origin time (YYYY-MM-DD,HH:MM:SS)') 241 | print ('-s, -starttime Trace start time (YYYY-MM-DD,HH:MM:SS) or phase-+offset e.g. P+10') 242 | print (' or offset from origin time in seconds') 243 | print ('-e, -endtime Trace end time (YYYY-MM-DD,HH:MM:SS) or phase-+offset or duration_in_s') 244 | print ('-A Add a user-agent string for identification ') 245 | print (' ') 246 | 247 | #---------- User-agent info 248 | 249 | agentstatement = 'FetchSyn/' + FSversion + ' Python/' + sys.version.split(' ')[0] + ' Requests/' + requests.__version__ 250 | if ( agent is not None ): 251 | agentstatement = agentstatement + ' (' + str(agent) + ')' 252 | user_agent = {'User-Agent': str(agentstatement) } 253 | 254 | #---------- Make sure there is a source and a receiver 255 | 256 | if ( rlat is None and rlon is None and network is None and station is None and ReceiverFile is None ): 257 | print ('Error: a receiver is required ') 258 | exit() 259 | if ( slat is None and slon is None and evid is None ): 260 | print ('Error: a source is required ') 261 | exit() 262 | 263 | #---------- validate format 264 | 265 | if ( formatstring is not None ): 266 | format_list = ['mseed','miniseed','ms','sac','SAC','zip','saczip'] 267 | if ( formatstring not in format_list) : 268 | print ('Error: invalid format type:',formatstring,' Must be miniseed or saczip ') 269 | exit() 270 | if ( formatstring[0] == 'm' ): 271 | formatstring = 'miniseed' 272 | else: 273 | formatstring = 'saczip' 274 | 275 | #---------- validate components 276 | 277 | if ( components is not None ): 278 | component_list = ['Z','N','E','R','T'] 279 | while re.search(r'([A-Z])(.*)\1', components): 280 | components = re.sub(r'([A-Z])(.*)\1', r'\1\2', components) 281 | for i in range(len(components)): 282 | if ( components[i] not in component_list) : 283 | print ('Error: invalid components:',components,' Must be any combination of ZNERT ') 284 | exit() 285 | 286 | #---------- validate units 287 | 288 | if ( units is not None ): 289 | unit_list = ['displacement','velocity','acceleration','d','v','a','D','V','A'] 290 | if ( units not in unit_list) : 291 | print ('Error: invalid unit type:',unit,' Must be displacement,velocity,acceleration,d,v,or a ') 292 | exit() 293 | if ( units == 'd' or units == 'D' ): 294 | units = 'displacement' 295 | if ( units == 'v' or units == 'V' ): 296 | units = 'velocity' 297 | if ( units == 'a' or units == 'A' ): 298 | units = 'acceleration' 299 | 300 | #---------- validate scale 301 | 302 | if ( scale is not None ): 303 | scalecheck = check_number(scale,-1e26,1e26) 304 | if ( scalecheck != 0 ): 305 | print ('Error: bad value for scale:',scale,' Must be between -1e26 and 1e26') 306 | exit() 307 | 308 | #--------- validate dt 309 | 310 | if ( dt is not None ): 311 | dtcheck = check_number(dt,0.001,5) 312 | if ( dtcheck != 0 ): 313 | print ('Error: bad value for dt:',dt,' Must be between 0.001 and 5') 314 | exit() 315 | 316 | #---------- validate kernelwidth 317 | 318 | if ( kernelwidth is not None ): 319 | kernelwidthcheck = check_number(kernelwidth,1,1000) 320 | if ( kernelwidthcheck != 0 ): 321 | if ( isinstance(kernelwidth,int) == False ): 322 | print ('Error: bad value for kernelwidth:',kernelwidth,' Must be positive integer') 323 | exit() 324 | kernelwidth = str(int(float(kernelwidth))) 325 | 326 | #---------- validate source location: slat,slon,sdepm 327 | 328 | if ( slat is not None ): 329 | slatcheck = check_number(slat,-90,90) 330 | if ( slatcheck != 0 ): 331 | print ('Error: bad value for slat',slat,' Must be between -90 and 90.') 332 | exit() 333 | 334 | if ( slon is not None ): 335 | sloncheck = check_number(slon,-180,360) 336 | if ( sloncheck != 0 ): 337 | print ('Error: bad value for slon',slon,' Must be between -180 and 360.') 338 | exit() 339 | if ( float(slon) > 180 ): 340 | slonF = float(slon) - 360 341 | slon = str(slonF) 342 | 343 | if ( sdepm is not None ): 344 | sdepmcheck = check_number(sdepm,0,700000) 345 | if ( sdepmcheck != 0 ): 346 | print ('Error: bad value for sdepm',sdepm,' Must be between 0 and 700000.') 347 | exit() 348 | 349 | if ( slat is not None and slon is not None and sdepm is None ): 350 | print ('Error: sourcedepthinmeters (sdepm) is required when using slat & slon ') 351 | exit() 352 | 353 | #---------- take care of the origntime if given 354 | 355 | if ( originstring is not None ): 356 | origin = originstring.replace(':',',') 357 | origin = origin.replace('T',',') 358 | origin = origin.replace('-',',') 359 | origin = origin.replace('/',',') 360 | origin = origin.split(',') 361 | if ( len(origin) != 6): 362 | print ('Error: invalid origintime format. Use: 1900,1,31,23,59,59 or 1900-12-31T23:59:59') 363 | exit() 364 | else: 365 | origin = origin[0] + "-" + origin[1] + "-" + origin[2] + "T" + origin[3] + ":" + origin[4] + ":" + origin[5] 366 | 367 | #---------- take care of the starttime if given 368 | 369 | if ( startstring is not None ): 370 | start = startstring.replace(':',',') 371 | start = start.replace('T',',') 372 | start = start.replace('/',',') 373 | start = start.replace('-',',') 374 | start = start.replace('+',',') 375 | start = start.split(',') 376 | if ( len(start) == 6 ): 377 | start = start[0] + "-" + start[1] + "-" + start[2] + "T" + start[3] + ":" + start[4] + ":" + start[5] 378 | elif ( len(start) == 2 ): 379 | startcheck = check_number(start[1],0,18000) 380 | if ( startcheck !=0 ): 381 | print ('Error: invalid starttime:',startstring,' Valid: 1900,01,31,23,59,0 or 1900-01-31T23:59:0 or P-20 or ScS+200 or 100') 382 | exit() 383 | else: 384 | start=startstring 385 | start = start.replace('+','%2B') 386 | elif ( len(start) == 1): 387 | startcheck = check_number(start[0],0,18000) 388 | if ( startcheck !=0 ): 389 | print ('Error: invalid starttime:',startstring,' Valid: 1900,01,31,23,59,0 or 1900-01-31T23:59:0 or P-20 or ScS+200 or 100') 390 | exit() 391 | else: 392 | start=startstring 393 | else: 394 | print ('Error: invalid starttime:',startstring,' Valid: 1900,01,31,23,59,0 or 1900-01-31T23:59:0 or P-20 or ScS+200 or 100') 395 | exit() 396 | 397 | #---------- take care of the endtime if given 398 | 399 | if ( endstring is not None ): 400 | end = endstring.replace(':',',') 401 | end = end.replace('T',',') 402 | end = end.replace('/',',') 403 | end = end.replace('-',',') 404 | end = end.replace('+',',') 405 | end = end.split(',') 406 | if ( len(end) == 6 ): 407 | end = end[0] + "-" + end[1] + "-" + end[2] + "T" + end[3] + ":" + end[4] + ":" + end[5] 408 | elif ( len(end) == 2 ): 409 | endcheck = check_number(end[1],0,18000) 410 | if ( endcheck !=0 ): 411 | print ('Error: invalid endtime:',endstring,' Valid: 1900,01,31,23,59,0 or 1900-01-31T23:59:0 or P-20 or ScS+200 or 100') 412 | exit() 413 | else: 414 | end=endstring 415 | end = end.replace('+','%2B') 416 | elif ( len(end) == 1 ): 417 | endcheck = check_number(end[0],0,18000) 418 | if ( endcheck !=0 ): 419 | print ('Error: invalid endtime:',endstring,' Valid: 1900,01,31,23,59,0 or 1900-01-31T23:59:0 or P-20 or ScS+200 or 100') 420 | exit() 421 | else: 422 | end=endstring 423 | else: 424 | print ('Error: invalid endtime:',endstring,' Valid: 1900,01,31,23,59,0 or 1900-01-31T23:59:0 or P-20 or ScS+200 or 100') 425 | exit() 426 | 427 | #---------- validate sourcemomenttensor 428 | 429 | if ( momentstring is not None ): 430 | moment = momentstring.split(',') 431 | if ( len(moment) != 6 ): 432 | print ('Error: sourcemomenttensor must have 6 comma separated numbers') 433 | exit() 434 | else: 435 | for i in range(0,len(moment)): 436 | momentcheck = check_number(moment[i],-1e26,1e26) 437 | if ( momentcheck != 0 ): 438 | print ('Error: bad value for moment component ',i+1,':',moment[i],' Must be between -1e26 and 1e26 (Nm)') 439 | exit() 440 | 441 | #---------- validate sourcedoublecouple 442 | 443 | if ( doublecouplestring is not None ): 444 | doublecouple = doublecouplestring.split(',') 445 | if ( len(doublecouple) == 3 or len(doublecouple) == 4 ): 446 | strikecheck = check_number(doublecouple[0],-180,360) 447 | if ( strikecheck != 0 ): 448 | print ('Error: bad value for strike:',doublecouple[0],' Must be between -180 and 360') 449 | exit() 450 | dipcheck = check_number(doublecouple[1],0,90) 451 | if ( dipcheck != 0 ): 452 | print ('Error: bad value for dip:',doublecouple[1],' Must be between 0 and 90') 453 | exit() 454 | rakecheck = check_number(doublecouple[2],-180,360) 455 | if ( rakecheck != 0 ): 456 | print ('Error: bad value for rake:',doublecouple[2],' Must be between -180 and 180') 457 | exit() 458 | if ( len(doublecouple) == 4 ): 459 | Mocheck = check_number(doublecouple[3],0,1e26) 460 | if ( Mocheck != 0 ): 461 | print ('Error: bad value for scalar moment :',doublecouple[3],' Must be between 0 and 1e26 (Nm)') 462 | exit() 463 | else: 464 | print ('Error: sourcedoublecouple must have 3 or 4 comma separated numbers') 465 | exit() 466 | 467 | #---------- validate sourceforce 468 | 469 | if ( forcestring is not None ): 470 | force = forcestring.split(',') 471 | if ( len(force) != 3 ): 472 | print ('Error: sourceforce must have 3 comma separated numbers') 473 | exit() 474 | for i in range(0,len(force)): 475 | forcecheck = check_number(force[i],-1e26,1e26) 476 | if ( forcecheck != 0 ): 477 | print ('Error: bad value for force component ',i+1,':',force[i],' Must be between -1e26 and 1e26 (Nm)') 478 | exit() 479 | 480 | #---------- validate rlat & rlon 481 | 482 | if ( rlat is not None ): 483 | rlatcheck = check_number(rlat,-90,90) 484 | if ( rlatcheck != 0 ): 485 | print ('Error: bad value for rlat',rlat,' Must be between -90 and 90.') 486 | exit() 487 | 488 | if ( rlon is not None ): 489 | rloncheck = check_number(rlon,-180,360) 490 | if ( rloncheck != 0 ): 491 | print ('Error: bad value for rlon',rlon,' Must be between -180 and 360.') 492 | exit() 493 | if ( float(rlon) > 180 ): 494 | rlonF = float(rlon) - 360 495 | rlon = str(rlonF) 496 | 497 | if ( stacode is None ): 498 | stacode = 'S0001' 499 | 500 | #---------- validate the network(s) 501 | 502 | if ( network is not None ): 503 | network = network.split(',') 504 | Nnetworks = len(network) 505 | if ( Nnetworks == 1 ): 506 | netstring = 'http://service.iris.edu/fdsnws/station/1/query?level=network&format=text&network=' + network[0] 507 | r=requests.get(netstring,headers=user_agent) 508 | if ( r.status_code != 200 ): 509 | print ('Error: invalid network: ', network) 510 | exit() 511 | else: 512 | for i in range(0,Nnetworks): 513 | r=requests.get('http://service.iris.edu/fdsnws/station/1/query?level=network&format=text&network=' + network[i],headers=user_agent) 514 | if ( r.status_code != 200 ): 515 | print ('Error: invalid network: ',network[i]) 516 | exit() 517 | 518 | #---------- validate the station(s) 519 | 520 | if ( station is not None ): 521 | station = station.split(',') 522 | Nstations = len(station) 523 | if ( Nstations == 1 ): 524 | stastring = 'http://service.iris.edu/fdsnws/station/1/query?level=station&format=text&network=' + network[0] + '&station=' + station[0] 525 | r=requests.get(stastring,headers=user_agent) 526 | if ( r.status_code != 200 ): 527 | print ('Error: invalid net station: ',network[0], station) 528 | exit() 529 | else: 530 | for i in range(0,Nstations): 531 | r=requests.get('http://service.iris.edu/fdsnws/station/1/query?level=station&format=text&network=' + network[0] + '&station=' + station[i],headers=user_agent) 532 | if ( r.status_code != 200 ): 533 | print ('Error: invalid net station: ',network[0], station[i]) 534 | exit() 535 | if ( station[0] == "*" ): 536 | station = None; 537 | 538 | #---------- read in list of stations 539 | 540 | if ( ReceiverFile is not None ): 541 | f = open(ReceiverFile,'r') 542 | i = 0 543 | ListReceivers = [] 544 | for line in f: 545 | i=i+1 546 | fields = line.split() 547 | if ( len(fields) < 2 ): 548 | print ('Error: invalid ReceiverFile line: ',line,' Must have at least (NET STAT) or (rlat rlon)') 549 | exit() 550 | else: 551 | LR1 = fields[0] 552 | LR2 = fields[1] 553 | if any( c in "ABCDEFGHIJKLMNOPQRSTUVWXYZ" for c in LR1 ): 554 | stastring = 'http://service.iris.edu/fdsnws/station/1/query?level=station&format=text&network=' + LR1 + '&station=' + LR2 555 | r=requests.get(stastring,headers=user_agent) 556 | if ( r.status_code != 200 ): 557 | print ('Error: invalid net station: ',LR1, LR2,' on line ',i,' of ',ReceiverFile) 558 | exit() 559 | ListReceivers.append(line) 560 | else: 561 | LR1check = check_number(LR1,-90,90) 562 | if ( LR1check != 0 ): 563 | print ('Error: bad value for rlat:',LR1,' on line ', i,' of ',ReceiverFile, ' Must be between -90 and 90.') 564 | LR2check = check_number(LR2,-180,360) 565 | if ( LR2check != 0 ): 566 | print ('Error: bad value for rlon:',LR2,' on line ', i,' of ',ReceiverFile, ' Must be between -180 and 360.') 567 | LR_Lon = float(LR2) 568 | if ( LR_Lon > 180 ): 569 | LR_Lon = LR_Lon - 360 570 | Nfields = len(fields) 571 | NewLine = LR1 + " " + str(LR_Lon).strip() 572 | if any ( c in "code" for c in line ): 573 | if ( Nfields == 3 ): 574 | NewLine = NewLine + " " + fields[2] 575 | if ( Nfields == 4 ): 576 | NewLine = NewLine + " " + fields[2] + " " + fields[3] 577 | if ( Nfields == 5 ): 578 | NewLine = NewLine + " " + fields[2] + " " + fields[3] + " " + fields[4] 579 | else: 580 | if ( Nfields == 3 ): 581 | NewLine = NewLine + " stacode=" + fields[2] 582 | if ( Nfields == 4 ): 583 | NewLine = NewLine + " netcode=" + fields[2] + " stacode=" + fields[3] 584 | if ( Nfields == 5 ): 585 | NewLine = NewLine + " netcode=" + fields[2] + " stacode=" + fields[3] + " loccode=" + fields[4] 586 | if ( Nfields == 2 ): 587 | NewLine = NewLine + " stacode=S" + str(i).zfill(4) 588 | ListReceivers.append(NewLine) 589 | NReceivers = i 590 | 591 | #------------- Build the POST file for the webservice query ----------- 592 | #-------- Two versions here since python 2 vs 3 handle StrinIO differently ----- 593 | 594 | url = "" 595 | 596 | if ( modelstring is not None): 597 | url = url + "\nmodel=" + modelstring 598 | 599 | if ( formatstring is not None ): 600 | url = url + "\nformat=" + formatstring 601 | 602 | if ( label is not None ): 603 | url = url + "\nlabel=" + label 604 | 605 | if ( components is not None ): 606 | url = url + "\ncomponents=" + components 607 | 608 | if ( units is not None ): 609 | url = url + "\nunits=" + units 610 | 611 | if ( scale is not None ): 612 | url = url + "\nscale=" + scale 613 | 614 | if ( dt is not None ): 615 | url = url + "\ndt=" + dt 616 | 617 | if ( kernelwidth is not None ): 618 | url = url + "\nkernelwidth=" + kernelwidth 619 | 620 | if ( originstring is not None ): 621 | url = url + "\norigintime=" + origin 622 | 623 | if ( startstring is not None ): 624 | url = url + "\nstarttime=" + start 625 | 626 | if ( endstring is not None ): 627 | url = url + "\nendtime=" + end 628 | 629 | if ( evid is not None ): 630 | url = url + "\neventid=" + evid 631 | 632 | if ( slat is not None ): 633 | url = url + "\nsourcelatitude=" + slat.strip() 634 | 635 | if ( slon is not None ): 636 | url = url + "\nsourcelongitude=" + slon.strip() 637 | 638 | if ( sdepm is not None ): 639 | url = url + "\nsourcedepthinmeters=" + sdepm 640 | 641 | if ( momentstring is not None ): 642 | url = url + "\nsourcemomenttensor=" + momentstring.strip() 643 | 644 | if ( doublecouplestring is not None ): 645 | url = url + "\nsourcedoublecouple=" + doublecouplestring.strip() 646 | 647 | if ( forcestring is not None ): 648 | url = url + "\nsourceforce=" + forcestring.strip() 649 | 650 | if ( network is not None ): 651 | if ( station is not None ): 652 | for i in range(0,Nstations): 653 | url = url + "\n" + network[0] + " " + station[i] 654 | else: 655 | for i in range(0,Nnetworks): 656 | url = url + "\n" + network[i] + " *" 657 | 658 | if ( rlat is not None and rlon is not None ): 659 | url = url + "\n" + rlat.strip() + " " + rlon.strip() 660 | if ( netcode is not None ): 661 | url = url + " netcode=" + netcode + " " 662 | if ( stacode is not None ): 663 | url = url + " stacode=" + stacode + " " 664 | if ( loccode is not None ): 665 | url = url + " loccode=" + loccode 666 | 667 | if ( ReceiverFile is not None ): 668 | for i in range(0,NReceivers): 669 | url = url + "\n" + ListReceivers[i] 670 | 671 | print ('-------- your POST file ---------') 672 | print (url) 673 | print ('---------------------------------') 674 | 675 | if ( pythonversion == 2 ): 676 | inputstring = StringIO.StringIO(url) 677 | elif ( pythonversion == 3 ): 678 | url = bytes(url,encoding="ascii") 679 | inputstring = BytesIO(url) 680 | else: 681 | print ('StringIO is incompatible with this version of python. Try v2.X or v3') 682 | exit() 683 | 684 | #------------- Build the output file name ------------ 685 | 686 | if ( outfile is not None ): 687 | outputfile = outfile + ".zip" 688 | if ( formatstring == 'miniseed' ): 689 | outputfile = outfile + ".mseed" 690 | elif ( label is not None ): 691 | outputfile = label + ".zip" 692 | if ( formatstring == 'miniseed' ): 693 | outputfile = label + ".mseed" 694 | else: 695 | outputfile = 'Synthetics.zip' 696 | if ( formatstring == 'miniseed' ): 697 | outputfile = 'Synthetics.mseed' 698 | 699 | #------------- Make the request -------------- 700 | 701 | with open(outputfile, 'wb') as output: 702 | r = requests.post(url="http://service.iris.edu/irisws/syngine/1/query", data=inputstring,headers=user_agent) 703 | if ( r.status_code == 200 ): 704 | print ('status = 200. Successful request. Writing output to: ',outputfile) 705 | output.write(r.content) 706 | if ( formatstring is not 'miniseed' ): 707 | print ('Logfile: Synthetics.log') 708 | else: 709 | print ('Request was not served. ',r.status_code,r.content) 710 | 711 | now = datetime.datetime.utcnow() 712 | print (now.strftime("%Y-%m-%d %H:%M:%S"), 'UTC') 713 | 714 | CurrentTime1 = time.time() 715 | print ('Request took: ',CurrentTime1-CurrentTime0,'seconds') 716 | 717 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # fetch-scripts - command line access to seismic data 2 | 3 | Convenient access to seismological data via FDSN and other web services. 4 | 5 | The scripts are written in Perl and Python and should run on most systems. 6 | 7 | ## The scripts 8 | 9 | **[FetchData](FetchData)** (Perl) - Fetch time series and optionally, related metadata, matching SAC Poles and Zeros and matching SEED RESP files. Time series data are returned in miniSEED format, metadata is saved as a simple ASCII list. 10 | 11 | **[FetchEvent](FetchEvent)** (Perl) - Fetch event parameters and print simple text summary. Works with any fdsnws-event service. 12 | 13 | **[FetchSyn](FetchSyn)** (Python) - Fetch synthetics seismograms from the [EarthScope Syngine](https://service.iris.edu/irisws/syngine/1/) service. See [FetchSyn usage](docs/fetchsyn.md). 14 | 15 | ## Quick start 16 | 17 | 1. Download the latest version of desired scripts at the links above 18 | 1. Make the scripts executable (e.g 'chmod +x FetchData') if needed 19 | 1. Run script to print usage message 20 | 21 | ## Documentation and examples 22 | 23 | Further description and example usage can be found [in the tutorial](docs/tutorial.md) 24 | 25 | ## Using the scripts with other data centers 26 | 27 | See documentation on [using the scripts with other data centers](docs/other-centers.md) 28 | 29 | ## Perl requirements 30 | 31 | The Perl scripts should run with any relatively recent Perl interpreter. In some cases, particularly for older Perl releases, you might need to install some required modules, namely **XML::SAX** and **Bundle::LWP** (libwww-perl). The procedure to install these modules is system dependent, for most Unix/Linux OSes they can be installed using the package management system. In addition to Perl only network access to the EarthScope web services is needed. 32 | 33 | For large data requests installing the **XML::SAX::ExpatXS** Perl module will significantly increase the parsing speed of XML metadata used by all scripts. 34 | 35 | ## Running the Perl scripts on Windows 36 | 37 | Most Windows computers do not have Perl by default and one must be installed. Any distribution of Perl should work as long as the required modules are included. The [Strawberry Perl](http://strawberryperl.com/) distribution for Windows is known to work and is recommended. 38 | 39 | Once installed, the Fetch scripts may be run from a command prompt (e.g. Windows PowerShell) by typing **perl** followed by the name of the script. For example: 40 | 41 | ```Console 42 | PS C:\Users\username\Downloads> perl .\FetchData 43 | ``` 44 | 45 | **Note**: the Fetch scripts are command-line oriented with no GUI, double-clicking them in Explorer using does not do anything useful. 46 | 47 | ## Copyright and License 48 | 49 | The fetch-scripts are an [EarthScope](https://www.earthscope.org/) product. 50 | 51 | Licensed under the Apache License, Version 2.0 (the "License"); 52 | you may not use this file except in compliance with the License. 53 | You may obtain a copy of the License at 54 | 55 | [http://www.apache.org/licenses/LICENSE-2.0](http://www.apache.org/licenses/LICENSE-2.0) 56 | 57 | Unless required by applicable law or agreed to in writing, software 58 | distributed under the License is distributed on an "AS IS" BASIS, 59 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 60 | See the License for the specific language governing permissions and 61 | limitations under the License. 62 | 63 | Copyright (c) 2018 EarthScope Consortium 64 | 65 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | name: EarthScope 2 | title: fetch-scripts - command line access to seismic data 3 | markdown: kramdown 4 | permalink: pretty 5 | remote_theme: EarthScope/theme_ghpages 6 | github_corner_link: https://github.com/EarthScope/fetch-scripts 7 | -------------------------------------------------------------------------------- /docs/fetchsyn.md: -------------------------------------------------------------------------------- 1 | ## Python dependencies 2 | 3 | The requests library is needed. Currently `FetchSyn` is compatible with python 2 & 3. 4 | 5 | ## Executing the scripts 6 | 7 | After downloading the scripts they can be executed from the command line either by explicitly invoking python or by making the scripts executable and installing the path to python in the first line. To explicitly invoke python to execute a script type the following: 8 | 9 | ```Console 10 | $ python FetchSyn 11 | ``` 12 | 13 | ## Help menu to list parameters 14 | 15 | To access the help menu, which lists all input parameters: 16 | 17 | ```Console 18 | $ FetchSyn -u 19 | ``` 20 | 21 | ## Examples 22 | 23 | To generate a synthetic for IU.ANMO for the Tohoku earthquake using it's GCMT origin and moment tensor: 24 | 25 | ```Console 26 | $ FetchSyn -N IU -S ANMO -evid GCMT:M201103110546A 27 | ``` 28 | 29 | Same request, but add station KIP and make the source depth 35km and return the vertical (Z), radial (R) and transverse (T) components rather than the default ZNE components: 30 | 31 | ```Console 32 | $ FetchSyn -N IU -S ANMO,KIP -C ZRT -evid GCMT:M201103110546A -sdepm 35000 33 | ``` 34 | 35 | 36 | Same request, but: 37 | * include all stations in the virtual network _GSN 38 | * use a different [model](http://ds.iris.edu/ds/products/syngine/#models) 39 | * change the sample rate to 0.05 sps 40 | * make the output miniseed rather than saczip 41 | * return velocity not the default (displacement) 42 | * cut the traces to 10 seconds before P 43 | * make traces 300 seconds in duration 44 | * add user-agent identification (helps DMC & helps you if there are problems) 45 | 46 | ```Console 47 | $ FetchSyn -N _GSN -model prem_a_20s -dt 0.05 -format mseed -units velocity -start P-10 -end 300 -A "Joe Seismologist" -C ZRT -evid GCMT:M201103110546A -sdepm 35000 48 | ``` 49 | 50 | ### Manually input source and use a file with a list of receivers 51 | 52 | Example using a text file with receiver names. Each line of the receiver file is an example of valid formatting. The networkcode, stationcode, locationcode in the later lines is optional. 53 | 54 | rec.txt: 55 | ```Console 56 | IU ANMO 57 | II * 58 | -89 -179 59 | -89 -178 S0002 60 | -89 -177 N2 S0003 L2 61 | ``` 62 | 63 | Manually input the origin using a double couple source (strike,dip,rake[,optional_scalar_moment_in_NM]) 64 | 65 | ```Console 66 | $ FetchSyn -recfile rec.txt -origin 2011,03,11,05,46,24 -slat 38.3 -slon 142.3 -sdepm 24400 -dc 203,10,88,5.3e22 -C Z 67 | ``` 68 | 69 | -------------------------------------------------------------------------------- /docs/other-centers.md: -------------------------------------------------------------------------------- 1 | ## Overview 2 | 3 | The Fetch scripts may be used to access information at other data centers as long as the service interfaces are compatible with the versions offered by EarthScope. Many of the services are [standardized by the FDSN](https://fdsn.org/webservices/) and offered by other data centers. By default, the scripts are configured to fetch information from EarthScope. Alternate locations of service interfaces may be specified. 4 | 5 | ## Specifying alternate service locations using environment variables 6 | 7 | Default service locations can be overridden by setting the following environment variables: 8 | 9 | **SERVICEBASE** = the base URI of the service(s) to use (http://service.iris.edu/) 10 | 11 | **TIMESERIESWS** = complete URI of service (http://service.iris.edu/fdsnws/dataselect/1) 12 | 13 | **METADATAWS** = complete URI of service (http://service.iris.edu/fdsnws/station/1) 14 | 15 | **EVENTWS** = complete URI of service (http://service.iris.edu/fdsnws/event/1) 16 | 17 | **SACPZWS** = complete URI of service (http://service.iris.edu/irisws/sacpz/1) 18 | 19 | **RESPWS** = complete URI of service (http://service.iris.edu/irisws/resp/1) 20 | 21 | The *SERVICEBASE* variable will be used for all services if set. The other, service-specific variables, are explicit paths to each service. The service-specific variables are useful if the path after the hostname is different than used by the IRIS DMC, or if users wish to use services at different data centers (e.g. events from one center and time series from another). 22 | 23 | ## Specifying alternate service locations using command line options 24 | 25 | Default service locations can be overridden by specifying the following command line options: 26 | 27 | `-timeseriesws` 28 | 29 | `-metadataws` 30 | 31 | `-eventws` 32 | 33 | `-sacpzws` 34 | 35 | `-respws` 36 | 37 | Obviously, each script only accepts the argument(s) appropriate for its operation. 38 | 39 | ## Wrapper scripts for other accessing other data centers 40 | 41 | Simple shell wrapper scripts can be created to redirect access to alternate locations. For example: 42 | 43 | **FetchData-IRISDMC**: 44 | ```bash 45 | #!/bin/bash 46 | 47 | # Set servce base path, change to your own service host 48 | SERVICEBASE="http://service.iris.edu" 49 | 50 | # Set all service specific locations using the service base 51 | TIMESERIESWS="${SERVICEBASE}/fdsnws/dataselect/1" 52 | METADATAWS="${SERVICEBASE}/fdsnws/station/1" 53 | EVENTWS="${SERVICEBASE}/fdsnws/event/1" 54 | SACPZWS="${SERVICEBASE}/irisws/sacpz/1" 55 | RESPWS="${SERVICEBASE}/irisws/resp/1" 56 | 57 | export SERVICEBASE TIMESERIESWS METADATAWS EVENTWS SACPZWS RESPWS 58 | 59 | exec FetchData "$@" 60 | ``` 61 | 62 | The above script can be customized to redirect all traffic to an alternate data center by replacing the URLs with another location. 63 | 64 | 65 | -------------------------------------------------------------------------------- /docs/tutorial.md: -------------------------------------------------------------------------------- 1 | ## Executing the scripts 2 | 3 | After downloading the scripts they can be executed from the command line either by explicitly invoking Perl or by making the scripts executable and letting the OS run Perl for you. To explicitly invoke Perl to execute a script type the following: 4 | 5 | ```Console 6 | $ perl FetchData 7 | ``` 8 | 9 | To make a script executable on the Unix-like operating system use the `chmod` command, after which the script can be run like any other program from the command line: 10 | 11 | ```Console 12 | $ chmod +x FetchData 13 | $ ./FetchData 14 | ``` 15 | 16 | ## Requesting miniSEED and simple metadata using `FetchData` 17 | 18 | To request the first hour of the year 2011 for BHZ channels from GSN stations, execute the following command: 19 | 20 | ```Console 21 | $ FetchData -N _GSN -C BHZ -s 2011-01-01T00:00:00 -e 2011-01-01T01:00:00 -o GSN.mseed -m GSN.metadata 22 | ``` 23 | 24 | Note that the network specification used is a virtual network code; regular network codes may also be specified. The start and end times are specified in an ISO defined format, with a capital 'T' separating the date from the time. Commas are also permissible to separate the date and time components. 25 | 26 | The received miniSEED will be saved to the `GSN.mseed` file and simple ASCII metadata will be saved to the `GSN.metadata` file. These two files can be used to convert the data to SAC format with the [mseed2sac](https://github.com/EarthScope/mseed2sac): 27 | 28 | ```Console 29 | $ mseed2sac GSN.mseed -m GSN.metadata 30 | ``` 31 | 32 | ## Requesting event information for magnitude 6+ events within 10 degrees of location 33 | 34 | To request magnitude 6+ events within 20 degrees of the main shock of the Tohoku-Oki, Japan Earthquake on or after March 11th 2011, execute the following command: 35 | 36 | ```Console 37 | $ FetchEvent -s 2011-03-11 --radius 38.2:142.3:20 --mag 6 38 | ``` 39 | 40 | h2. Requesting data from multiple data centers (federating) 41 | 42 | If the `-F` option is used with FetchData (version 2015.014 and later) it will first submit the request to the IRIS Federator to determine which data centers have data matching the criteria, then it will make a request to each data center to retrieve the miniSEED. 43 | 44 | For example, the request 1+ hour of global LHZ channels for the 1995 Mw 8.0 Chile earthquake a command like this may be suitable: 45 | 46 | ```Console 47 | $ FetchData -F -C LHZ -s 1995-7-30T05:11:23 -e 1995-7-30T06:30:00 -o event.mseed -m event.meta 48 | ``` 49 | 50 | *Note*: not all FDSN data centers support the direct retrieval of metadata in SEED RESP and SAC PZ formats, consequently requesting such information when federating the request will usually fail with some data centers. Simple metadata, as saved by the *-m* option, should always work as it only relies on a compliant fdsnws-station service. 51 | 52 | ## Wildcards and lists 53 | 54 | The network, station, location and channel selections may include both wildcards and lists. The supported wildcards are *, meaning zero to many characters, and ?, meaning a single character. Example wildcard usage: 'BH*' and 'L??'. A list is specified as simple comma-separated values, for example: 'BHE,BHN,BHZ'. Illustrating both of these features using command line selections (the same applies to the selection list file): 55 | 56 | ```Console 57 | $ FetchData -N IU -S ANMO,COLA -C 'BH*,LH*' -s 2011-01-01T00:00:00 -e 2011-01-01T01:00:00 -o ANMO-COLA.mseed 58 | ``` 59 | 60 | Note that the wildcard characters usually need to be quoted to avoid them being interpreted by the shell. 61 | 62 | ## Selecting data by region 63 | 64 | The FetchData script allows data selection by geographic region in two ways: 1) specifying minimum and maximum latitude and/or longitude and 2) specifying a point and a maximum radius in degrees including the optional specification of a minimum radius to define a circular range from a point. For example, to request 10 minutes of miniSEED data for all BHZ channels available within 10 degrees of the M5 2010 Ottowa, Canada earthquake at latitude 45.88 and longitude -75.48 try the follow: 65 | 66 | ```Console 67 | $ FetchData -radius 45.88:-75.48:10 -m OttowaM5.meta -o OttowaM5.mseed -C BHZ -s 2010-06-23T17:41 -e 2010-06-23T17:51 68 | ``` 69 | 70 | ## Large data selections 71 | 72 | To specify many selections or combinations not possible using the command line options the Fetch scripts can read a file containing either a simple ASCII selection list or a BREQ_FAST formatted request. 73 | 74 | For a file containing a selection list use the -l option: 75 | 76 | ```Console 77 | $ FetchData -l myselection.txt -o mydata.mseed 78 | ``` 79 | 80 | The expected selection list format is documented [below](#Selection-List-Format). 81 | 82 | For a file containing a legacy BREQ_FAST-formatted request use the -b option: 83 | 84 | ```Console 85 | $ FetchData -b mybreqfast.txt -o mydata.mseed 86 | ``` 87 | 88 | ## Reducing data segmentation 89 | 90 | Specific to FetchData are two options for limiting the level of segmented data returned: 91 | 92 | * `-msl _length_` :: Specify minimum segment length in seconds, no segments shorter than length will be returned 93 | * `-lso` :: Return only the longest segment from each distinct channel 94 | 95 | Warning: These options limit the data returned, using them incorrectly or when data is very gappy can result in no returned data. These options do not fill gaps. 96 | 97 | ## Requesting SEED RESP or SAC Poles and Zeros files with miniSEED data 98 | 99 | The `FetchData` script can collect the SEED RESP or SAC Poles and Zeros matching the requested data at the same time it collects the time series data. An individual RESP and/or SACPZ file is created for each channel requested. You must specify an existing directory to which to write the files. 100 | 101 | To collect SEED RESP with data use the -rd option (using a dot for the current directory): 102 | 103 | ```Console 104 | $ FetchData -S COLA -C BHZ -s 2011-01-01T00:00:00 -e 2011-01-01T01:00:00 -o COLA.mseed -rd . 105 | ``` 106 | 107 | To collect SAC Poles and Zeros with data use the `-sd` option (using a dot for the current directory): 108 | 109 | ```Console 110 | $ FetchData -S COLA -C BHZ -s 2011-01-01T00:00:00 -e 2011-01-01T01:00:00 -o COLA.mseed -sd . 111 | ``` 112 | 113 | ## Selecting a blank (space-space) SEED location ID 114 | 115 | Many SEED location IDs are blank. Since these location IDs are stored as a 2-character field in the SEED format a blank location ID means the field is actually 2 ASCII space characters. To specifically select blank location IDs space characters may be used. Because spaces can sometimes be troublesome to provide exactly in scripts, in selection files, etc. the web services will also accept IDs specified as '--' (two dashes) which will be mapped to spaces. 116 | 117 | ## Selection List Format 118 | 119 | All of the Fetch scripts which request information about time series channels will accept a selection list to define the request. 120 | 121 | A selection list is simply an ASCII file where each line specifies a complete data selection containing the following space-separated values: 122 | 123 | ``` 124 | Network Station Location Channel StartTime EndTime 125 | ``` 126 | 127 | For example, a selection list for a small time window for BHZ channels from the selected GSN stations would be: 128 | 129 | ``` 130 | II AAK 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 131 | II ABKT 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 132 | II ABPO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 133 | II ALE 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 134 | II ARU 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 135 | II ASCN 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 136 | II BFO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 137 | II BORG 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 138 | IU ADK 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 139 | IU AFI 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 140 | IU ANMO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 141 | IU ANTO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 142 | IU BBSR 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 143 | IU BILL 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 144 | IU CASY 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 145 | IU CCM 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 146 | IU CHTO 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 147 | IU COLA 00 BHZ 2011-01-01T00:00:00 2011-01-01T01:00:00 148 | ``` 149 | 150 | With this selection list saved to a file name `myselection.txt` it can be used, for example, like this: 151 | 152 | ```Console 153 | $ FetchData -l myselection.txt -o mydata.mseed 154 | ``` 155 | 156 | --------------------------------------------------------------------------------