├── .gitignore ├── LICENSE ├── README.md ├── example ├── README.txt ├── data │ ├── GROUND_RADAR │ │ └── cfrad.20150106_130000.000_to_20150106_130700.000_CPOL_PPI_level1a.nc │ └── SPACE_RADAR │ │ └── 2A-VL-128E9S133E14S.GPM.Ku.V6-20160118.20150106-S125313-E125447.004868.V04A.HDF5 └── example_config.ini ├── msgr ├── __init__.py ├── __version__.py ├── core.py ├── cross_validation.py ├── io │ ├── __init__.py │ ├── read_gpm.py │ ├── read_radar.py │ ├── read_trmm.py │ └── save_data.py └── utils │ ├── __init__.py │ ├── misc.py │ └── reflectivity_conversion.py ├── requirements.txt ├── scripts ├── find_good_case.py ├── generate_config_matchvol.py └── matchvol.py └── setup.py /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode/settings.json 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2017 Valentin Louf 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # MSGR - Matching Satellite and Ground Radar 2 | 3 | ## Disclaimer 4 | 5 | This dataset is supported by a funding from the U.S. Department of Energy as part of the Atmospheric Radiation Measurement (ARM) Climate Research Facility, an Office of Science user facility. 6 | 7 | If you use this dataset to prepare a publication, please consider offering me (Valentin Louf) co-authorship and add the following line in the acknowledgments: 8 | 9 | > This work has been supported by the U.S. Department of Energy Atmospheric Systems Research Program through the grant DE-SC0014063. 10 | 11 | ## References 12 | 13 | - Louf, V., A. Protat, R. A. Warren, S. M. Collis, D. B. Wolff, S. Raunyiar, C. Jakob, and W. A. Petersen, 2018: An integrated approach to weather radar calibration and monitoring using ground clutter and satellite comparisons. J. Atmos. Ocean. Technol., JTECH-D-18-0007.1, doi:10.1175/JTECH-D-18-0007.1. [http://journals.ametsoc.org/doi/10.1175/JTECH-D-18-0007.1] 14 | 15 | - Warren, R. A., A. Protat, S. T. Siems, H. A. Ramsay, V. Louf, M. J. Manton, and T. A. Kane, 2018: Calibrating Ground-Based Radars against TRMM and GPM. J. Atmos. Ocean. Technol., 35, 323–346, doi:10.1175/JTECH-D-17-0128.1. 16 | 17 | 18 | ## Installation 19 | 20 | Just type `python setup.py install`. The full list of required package is in requirements.txt, just type `pip install -r requirements.txt`. 21 | 22 | ## Required modules 23 | 24 | You will need: 25 | >- Python ARM Radar Toolkit [(Py-ART) ][1] 26 | >- [SciPy][2] 27 | >- [NumPy][2] 28 | >- Python Data Analysis Library [(pandas)][3] 29 | >- [Pyproj][4] 30 | >- h5py 31 | >- netCDF4 32 | 33 | This code has been tested and conceived on Python 3.5 and Python 3.6. It will *NOT* work with python 2.X versions. 34 | 35 | ## Usage 36 | 37 | `matchvol.py` need a configuration file to run, an example can be generated by typing `generate_config_matchvol.py --example`. Just pass the configuration file like that: `matchvol.py -c config.ini`. 38 | 39 | You should only modify the config.ini file to match your own configuration and then run python matchvol.py in a terminal. In the config.ini file you can choose: 40 | * Number of CPU for multiprocessing. 41 | * Start and end date of processing in YYYYMMDD format. 42 | * Input path for ground radar files, satellite files. 43 | * Output path for saved data. 44 | * The radar general information (name and ID, used for naming the output saving file), latitude, longitude, altitude, and beamwidth. 45 | * Min and max range (in m) of ground radar. 46 | * Different thresholds for comparison like: 47 | - Threshold on minimum satellite and ground radar reflectivity. 48 | - Minimum number of pair for comparison. 49 | - Minimum number of satellite profiles for comparison. 50 | - Maximum time difference between radar and satellite, in seconds. 51 | * Choose between the use of dBZ or natural units for the statistical calculations. 52 | * Declare that satellite is GPM (false for TRMM). 53 | * Ground radar is C-Band (false for S-Band). 54 | * Writing results in output directory. 55 | * Correct ground radar attenuation using pyart. 56 | 57 | ## Satellite data 58 | 59 | For GPM you need the 2AKu product. For TRMM you need the 2APR product. 60 | 61 | The website where you can download TRMM and GPM data is https://storm.pps.eosdis.nasa.gov/storm/data/Service.jsp?serviceName=RestrictedOrder. You need to register before you can order data. One you've entered your pre-registered email address you can enter the details of your order. You want the following options: 62 | * Under 'Order type' select 'Standalone order' 63 | * Under 'Coincidence' select 'None or Satellite-Ground Validation Site' 64 | * Under 'Options' select 'Subset Geographic Area' then 'Subset Geographically' (leave 'Include only swaths with...' blank), and also select 'Parameter Subsetting' 65 | * Under 'Product Type' select '2AKu' under 'Algorithm' and check the box that comes up below. Note that this is for GPM. For TRMM you have to select 2APR, but if you're only working with GPM you don't need to worry about that. 66 | * Under 'Temporal Criteria' set the range of dates you want data for. 67 | * Under 'Special Area Of Interest' specify the limits of your domain (it should encompass the 150km range ring of your radar(s)). Give it a 'Location Alias'. 68 | * Under 'Parameter Subset' choose the following: From 'scanStatus' select 'dataQuality'. From 'PRE' select 'landSurfaceType', 'binClutterFreeBottom' and 'flagPrecip'. From 'CSF' select 'flagBB', 'heightBB', 'widthBB', 'qualityBB', 'typePrecip' and 'qualityTypePrecip'. From 'SLV' select 'zFactorCorrected'. From 'NS' select 'Latitude', 'Longitude'. From 'ScanTime' select 'Year', 'Month', 'DayOfMonth', 'Hour', 'Minute', 'Second' 69 | * Provide an identifier, then select 'No' for 'Do you want to generate Read and Write routines for this subset', and set 'HDF' as the 'Output Data Format'. 70 | * Under 'Search Results' select all the files by clicking the top-most check box. 71 | Hitting submit should then get you what you want. 72 | 73 | ## Radar data 74 | 75 | Because this codes uses pyart to read radar data, the only limitations are pyart's limitations. 76 | 77 | In the case that the radar's data you are using do not use the "traditional" naming convention, especially for reflectivity (names are 'reflectivity', 'DBZ', 'DBZ_F'), you can add your own by modifying the read_radar.py file in ./MSGR/io/ and change (better add) one of the lines like this: `refl_slice = radar.fields['reflectivity']['data'][sweep_slice] # Reflectivity` 78 | 79 | [1]: https://github.com/ARM-DOE/pyart 80 | [2]: http://www.scipy.org/ 81 | [3]: http://pandas.pydata.org/ 82 | [4]: http://jswhit.github.io/pyproj/ 83 | -------------------------------------------------------------------------------- /example/README.txt: -------------------------------------------------------------------------------- 1 | Once MSGR is installed, type: 2 | 3 | matchvol.py -c example_config.ini 4 | 5 | In the example directory to test. -------------------------------------------------------------------------------- /example/data/GROUND_RADAR/cfrad.20150106_130000.000_to_20150106_130700.000_CPOL_PPI_level1a.nc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vlouf/matchproj-python/78c2fd72c7655cfd38ec478d4c9b31fca971ff7f/example/data/GROUND_RADAR/cfrad.20150106_130000.000_to_20150106_130700.000_CPOL_PPI_level1a.nc -------------------------------------------------------------------------------- /example/data/SPACE_RADAR/2A-VL-128E9S133E14S.GPM.Ku.V6-20160118.20150106-S125313-E125447.004868.V04A.HDF5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vlouf/matchproj-python/78c2fd72c7655cfd38ec478d4c9b31fca971ff7f/example/data/SPACE_RADAR/2A-VL-128E9S133E14S.GPM.Ku.V6-20160118.20150106-S125313-E125447.004868.V04A.HDF5 -------------------------------------------------------------------------------- /example/example_config.ini: -------------------------------------------------------------------------------- 1 | [general] 2 | # Number of CPU for multiprocessing 3 | # start and end date in YYYYMMDD format 4 | ncpu = 1 5 | start_date = 20150101 6 | end_date = 20150331 7 | 8 | [path] 9 | # Path to ground radar data 10 | # Path to space radar data 11 | # Path to output directory 12 | ground_radar = ./data/GROUND_RADAR 13 | satellite = ./data/SPACE_RADAR 14 | output = ./data/OUTPUT 15 | log = ./data/LOG 16 | 17 | [radar] 18 | # radar_name is the radar name (str) 19 | # radar_id is the radar id (needed for saving output data) (str) 20 | # rmin is the minimum radar range we start looking for data 21 | # rmax is the maximum radar range 22 | # lat/lon are the latitudes, longitudes of the radar 23 | # altitude is the altitude of the radar 24 | # offset is the reflectivity offset in dB you want to apply to radar data. 25 | # Units in meters and degrees 26 | radar_name = CPOL 27 | radar_id = IDR59 28 | rmin = 15000 29 | rmax = 150000 30 | longitude = 131.04530334 31 | latitude = -12.24880028 32 | altitude = 42 33 | beamwidth = 1.0 34 | offset = 0 35 | 36 | [thresholds] 37 | # Threshold on satellite reflectivity 38 | # Minimum number of pair 39 | # Minimum number of satellite profiles 40 | # Maximum time diffenrece between radar and satellite, in seconds 41 | # Threshold on ground radar reflectivity 42 | min_sat_reflec = 17 43 | min_pair = 10 44 | min_profiles = 10 45 | max_time_delta = 600 46 | min_gr_reflec = 17 47 | 48 | [switch] 49 | # Case insenstive, can be yes/no, y/n, true/false, 1/0 50 | # Using dBZ or natural units for the statistical calculations 51 | # Satellite is GPM (false for TRMM) 52 | # Ground radar is C-Band (false for S-Band) 53 | # Writing results in output directory 54 | # Correct ground radar attenuation using pyart 55 | dbz = False 56 | gpm = True 57 | cband = True 58 | write = True 59 | correct_gr_attenuation = True 60 | -------------------------------------------------------------------------------- /msgr/__init__.py: -------------------------------------------------------------------------------- 1 | from .cross_validation import match_volumes 2 | -------------------------------------------------------------------------------- /msgr/__version__.py: -------------------------------------------------------------------------------- 1 | __version__ = '2.1' 2 | -------------------------------------------------------------------------------- /msgr/core.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | import configparser 3 | 4 | import numpy as np 5 | 6 | from numpy import sqrt, cos, sin, tan, pi 7 | 8 | from .io.read_gpm import read_gpm 9 | from .io.read_trmm import read_trmm 10 | from .utils import reflectivity_conversion 11 | 12 | 13 | class Radar: 14 | def __init__(self, config_file, gr_offset=0): 15 | self.offset = gr_offset 16 | 17 | # Read radar config file. 18 | config = self._read_configfile(config_file) 19 | GR_param = config['radar'] 20 | 21 | self._check_keys(GR_param) # raise error if wrong key. 22 | 23 | # Get radar reflectivity band 24 | self.radar_band = GR_param.get("band") 25 | 26 | # Radar name 27 | self.name = GR_param.get('radar_name') 28 | self.id = GR_param.get('radar_id') 29 | 30 | # Radar lat/lon 31 | self.longitude = GR_param.getfloat('longitude') 32 | self.latitude = GR_param.getfloat('latitude') 33 | 34 | # Range 35 | self.rmin = GR_param.getfloat('rmin') 36 | self.rmax = GR_param.getfloat('rmax') 37 | 38 | # x/y min/max 39 | self.xmin = -1 * self.rmax 40 | self.xmax = self.rmax 41 | self.ymin = -1 * self.rmax 42 | self.ymax = self.rmax 43 | 44 | # others infos. 45 | self.altitude = GR_param.getfloat('altitude') 46 | self.beamwidth = GR_param.getfloat('beamwidth') 47 | self.min_refl_thrld = config['thresholds'].getfloat('min_gr_reflec') 48 | self.l_cband = config['switch'].getboolean('cband') 49 | 50 | # Compute Earth gaussian radius. 51 | self.gaussian_radius = self._radar_gaussian_curvature() 52 | 53 | self.fields = dict() 54 | 55 | def _read_configfile(self, config_file): 56 | config = configparser.ConfigParser() 57 | config.read(config_file) 58 | return config 59 | 60 | def _check_keys(self, GR_param): 61 | keytab = ['radar_name', 'rmin', 'rmax', 'radar_id', 'longitude', 'latitude', 'altitude', 'beamwidth', 'offset'] 62 | for mykey in keytab: 63 | try: 64 | GR_param[mykey] 65 | except KeyError: 66 | raise KeyError("Problem with configuration file, key: '{}' is missing and/or invalid.".format(mykey)) 67 | return None 68 | 69 | def _radar_gaussian_curvature(self): 70 | ''' 71 | Determine the Earth's Gaussian radius of curvature at the radar 72 | https://en.wikipedia.org/wiki/Earth_radius#Radii_of_curvature 73 | ''' 74 | lat0 = self.latitude 75 | 76 | # Major and minor radii of the Ellipsoid 77 | a = 6378137.0 # Earth radius in meters 78 | e2 = 0.0066943800 79 | b = a * sqrt(1 - e2) 80 | 81 | tmp = (a * cos(pi / 180 * lat0))**2 + (b * sin(pi / 180 * lat0))**2 # Denominator 82 | an = (a**2) / sqrt(tmp) # Radius of curvature in the prime vertical (east–west direction) 83 | am = (a * b)**2 / tmp**1.5 # Radius of curvature in the north–south meridian 84 | ag = sqrt(an * am) # Earth's Gaussian radius of curvature 85 | ae = (4 / 3.) * ag 86 | 87 | return ae 88 | 89 | def set_fields(self, mydict): 90 | """ 91 | Populate field dictionnary 92 | """ 93 | for k, v in mydict.items(): 94 | self.fields[k] = v 95 | 96 | def get_cartesian_coordinates(self): 97 | rg = self.fields['range'] 98 | ag = self.fields['azang'] 99 | eg = self.fields['elev_3d'] 100 | # Determine the Cartesian coordinates of the ground radar's pixels 101 | zg = sqrt(rg**2 + (self.gaussian_radius + self.altitude)**2 + 102 | 2 * rg * (self.gaussian_radius + self.altitude) * sin(pi / 180 * eg)) - self.gaussian_radius 103 | sg = self.gaussian_radius * np.arcsin(rg * cos(pi / 180 * eg) / (self.gaussian_radius + zg)) 104 | xg = sg * cos(pi / 180 * (90 - ag)) 105 | yg = sg * sin(pi / 180 * (90 - ag)) 106 | 107 | return xg, yg, zg 108 | 109 | 110 | class Satellite: 111 | def __init__(self, config_file, sat_file_1, sat_file_2A25_trmm=None): 112 | config = self._read_configfile(config_file) 113 | thresholds = config['thresholds'] 114 | try: 115 | sat_offset = config['satellite'].getfloat("sat_offset") 116 | except KeyError: 117 | sat_offset = None 118 | pass 119 | 120 | self.l_gpm = config['switch'].getboolean('gpm') 121 | self.TRMM_NEW_VERSION = False 122 | if not self.l_gpm and sat_file_2A25_trmm is None: 123 | self.TRMM_NEW_VERSION = True 124 | # raise ValueError("Configuration file says that the satellite is TRMM but no TRMM 2A25 files given.") 125 | 126 | self.min_prof_nb = thresholds.getint('min_profiles') # minprof 127 | self.max_time_delta = thresholds.getfloat('max_time_delta') # maxdt 128 | self.min_refl_thrld = thresholds.getfloat('min_sat_reflec') # minrefp 129 | self.min_pair_nb = thresholds.getint('min_pair') # minpair 130 | 131 | # Orbit parameters 132 | if self.l_gpm: 133 | self.altitude = 407000. # orbital height of GPM (zt) 134 | self.dr = 125. # gate spacing of GPM (drt) 135 | satdata = read_gpm(sat_file_1, sat_offset) 136 | elif self.TRMM_NEW_VERSION: 137 | self.altitude = 402500. # orbital height of TRMM (post boost) 138 | self.dr = 250. # gate spacing of TRMM 139 | satdata = read_gpm(sat_file_1, sat_offset) 140 | else: 141 | self.altitude = 402500. # orbital height of TRMM (post boost) 142 | self.dr = 250. # gate spacing of TRMM 143 | satdata = read_trmm(sat_file_1, sat_file_2A25_trmm, sat_offset) 144 | 145 | self.beamwidth = 0.71 146 | 147 | if satdata is None: 148 | raise ValueError("Incorrect satellite data file.") 149 | 150 | self.__dict__.update(**satdata) 151 | 152 | # Determine the direction of the scans 153 | self.alpha = np.abs(-17.04 + np.arange(self.nray) * 0.71) 154 | 155 | def _read_configfile(self, config_file): 156 | config = configparser.ConfigParser() 157 | config.read(config_file) 158 | return config 159 | -------------------------------------------------------------------------------- /msgr/cross_validation.py: -------------------------------------------------------------------------------- 1 | # Python standard library 2 | import logging 3 | import datetime 4 | import warnings 5 | import itertools 6 | 7 | # Other libraries 8 | import pyproj 9 | import numpy as np 10 | from numpy import sqrt, cos, sin, tan, pi, exp 11 | 12 | # Custom modules 13 | from .core import Radar, Satellite 14 | from .io.read_radar import read_radar 15 | from .utils import reflectivity_conversion 16 | from .utils.misc import * 17 | 18 | 19 | def _matching(satellite, cpol, nprof, reflectivity_satellite, 20 | refp_ss, refp_sh, xp, yp, zp, rt, ep, alpha, zbb, l_dbz=True): 21 | """ 22 | The volume matching is done here 23 | 24 | Parameters: 25 | =========== 26 | satellite: Object 27 | Satellite object defined in core.py 28 | cpol: Object 29 | Ground radar object defined in core.py 30 | nprof: int 31 | Number of precipitating satellite rays in domain 32 | reflectivity_satellite: ndarray 33 | Satellite reflectivity. 34 | refp_ss: ndarray 35 | Satellite stratiform reflectivity converted to S/C band. 36 | refp_sh: ndarray 37 | Satellite convective reflectivity converted to S/C band. 38 | xp: ndarray 39 | x-cartesian coordinates of satellite data corrected from parallax with 40 | respect to the ground radar. 41 | yp: ndarray 42 | y-cartesian coordinates of satellite data corrected from parallax with 43 | respect to the ground radar. 44 | zp: ndarray 45 | z-cartesian coordinates of satellite data corrected from parallax with 46 | respect to the ground radar. 47 | rt: ndarray 48 | Approximate volume of each satellite bin 49 | ep: ndarray 50 | elev_pr_grref 51 | alpha: ndarray 52 | Angle 53 | zbb: float 54 | Bright band altitude. 55 | l_dbz: bool 56 | Are the statistics over reflectivity done in natural units or in dBZ. 57 | 58 | Returns: 59 | ======== 60 | """ 61 | zt = satellite.altitude 62 | bwt = satellite.beamwidth 63 | drt = satellite.dr 64 | 65 | xg, yg, zg = cpol.get_cartesian_coordinates() 66 | rg = cpol.fields['range'] 67 | reflectivity_ground_radar = cpol.fields['reflec'] 68 | elang = cpol.fields['elang'] 69 | ntilt = cpol.fields['ntilt'] 70 | dr = cpol.fields['dr'] 71 | # Convert ground radar reflectivity to Ku-band 72 | refg_ku = reflectivity_conversion.convert_to_Ku(reflectivity_ground_radar, zg, zbb, cpol.radar_band) 73 | 74 | bwr = cpol.beamwidth 75 | earth_gaussian_radius = cpol.gaussian_radius 76 | rmax = cpol.rmax 77 | 78 | try: 79 | reflectivity_satellite = reflectivity_satellite.filled(np.NaN) 80 | except AttributeError: 81 | pass 82 | try: 83 | reflectivity_ground_radar = reflectivity_ground_radar.filled(np.NaN) 84 | except AttributeError: 85 | pass 86 | 87 | # Create arrays to store comparison variables 88 | '''Coordinates''' 89 | x = np.zeros((nprof, ntilt)) # x coordinate of sample 90 | y = np.zeros((nprof, ntilt)) # y coordinate of sample 91 | z = np.zeros((nprof, ntilt)) # z coordinate of sample 92 | dz = np.zeros((nprof, ntilt)) # depth of sample 93 | ds = np.zeros((nprof, ntilt)) # width of sample 94 | r = np.zeros((nprof, ntilt)) # range of sample from ground radar 95 | 96 | '''Reflectivities''' 97 | ref1 = np.zeros((nprof, ntilt)) + np.NaN # PR reflectivity 98 | ref2 = np.zeros((nprof, ntilt)) + np.NaN # GR reflectivity 99 | ref3 = np.zeros((nprof, ntilt)) + np.NaN # PR reflec S-band, snow 100 | ref4 = np.zeros((nprof, ntilt)) + np.NaN # PR reflec S-band, hail 101 | ref5 = np.zeros((nprof, ntilt)) + np.NaN # GR reflectivity Ku-band 102 | iref1 = np.zeros((nprof, ntilt)) + np.NaN # path-integrated PR reflec 103 | iref2 = np.zeros((nprof, ntilt)) + np.NaN # path-integrated GR reflec 104 | stdv1 = np.zeros((nprof, ntilt)) + np.NaN # STD of PR reflectivity 105 | stdv2 = np.zeros((nprof, ntilt)) + np.NaN # STD of GR reflectivity 106 | 107 | '''Number of bins in sample''' 108 | ntot1 = np.zeros((nprof, ntilt), dtype=int) # Total nb of PR bin in sample 109 | # Nb of rejected PR bin in sample 110 | nrej1 = np.zeros((nprof, ntilt), dtype=int) 111 | ntot2 = np.zeros((nprof, ntilt), dtype=int) # Total nb of GR bin in sample 112 | # Nb of rejected GR bin in sample 113 | nrej2 = np.zeros((nprof, ntilt), dtype=int) 114 | # Total volume of PR bins in sample 115 | vol1 = np.zeros((nprof, ntilt)) + np.NaN 116 | # Total volume of GR bins in sample 117 | vol2 = np.zeros((nprof, ntilt)) + np.NaN 118 | 119 | # Compute the volume of each radar (sat/ground) bin 120 | volp = 1.e-9 * np.pi * drt * (rt * np.pi / 180 * bwt / 2.)**2 121 | volg = 1e-9 * np.pi * dr * (rg * np.pi / 180 * bwr / 2)**2 122 | 123 | # Compute the path-integrated reflectivities at every points 124 | nat_refp = 10**(reflectivity_satellite / 10.0) # In natural units 125 | nat_refg = 10**(reflectivity_ground_radar / 10.0) 126 | irefp = np.fliplr(np.nancumsum(np.fliplr(nat_refp), axis=1)) 127 | irefg = np.nancumsum(nat_refg, axis=0) 128 | irefp = drt * (irefp - nat_refp / 2) 129 | irefg = dr * (irefg - nat_refg / 2) 130 | irefp = 10 * np.log10(irefp) 131 | irefg = 10 * np.log10(irefg) 132 | 133 | # Convert to linear units 134 | if not l_dbz: 135 | reflectivity_satellite = 10**(reflectivity_satellite / 10.0) 136 | reflectivity_ground_radar = 10**(reflectivity_ground_radar / 10.0) 137 | refp_ss = 10**(refp_ss / 10.0) 138 | refp_sh = 10**(refp_sh / 10.0) 139 | refg_ku = 10**(refg_ku / 10.0) 140 | 141 | irefp = 10**(irefp / 10.0) 142 | irefg = 10**(irefg / 10.0) 143 | 144 | # Loop over the TRMM/GPM profiles and Loop over the GR elevation scan 145 | for ii, jj in itertools.product(range(nprof), range(ntilt)): 146 | # Identify those PR bins which fall within the GR sweep 147 | ip = np.where((ep[ii, :] >= elang[jj] - bwr / 2) & (ep[ii, :] <= elang[jj] + bwr / 2)) 148 | 149 | # Store the number of bins 150 | ntot1[ii, jj] = len(ip) 151 | if len(ip) == 0: 152 | continue 153 | 154 | x[ii, jj] = np.mean(xp[ii, ip]) 155 | y[ii, jj] = np.mean(yp[ii, ip]) 156 | z[ii, jj] = np.mean(zp[ii, ip]) 157 | 158 | # Compute the thickness of the layer 159 | nip = len(ip) 160 | dz[ii, jj] = nip * drt * cos(pi / 180 * alpha[ii, 0]) 161 | 162 | # Compute the PR averaging volume 163 | vol1[ii, jj] = np.sum(volp[ii, ip]) 164 | 165 | # Note the mean TRMM beam diameter 166 | ds[ii, jj] = pi / 180 * bwt * \ 167 | np.mean((zt - zp[ii, ip]) / cos(pi / 180 * alpha[ii, ip])) 168 | 169 | # Note the radar range 170 | s = sqrt(x[ii, jj]**2 + y[ii, jj]**2) 171 | r[ii, jj] = (earth_gaussian_radius + z[ii, jj]) * \ 172 | sin(s / earth_gaussian_radius) / cos(pi / 180 * elang[jj]) 173 | 174 | # Check that sample is within radar range 175 | if r[ii, jj] + ds[ii, jj] / 2 > rmax: 176 | continue 177 | 178 | # Extract the relevant PR data 179 | refp1 = reflectivity_satellite[ii, ip].flatten() 180 | refp2 = refp_ss[ii, ip].flatten() 181 | refp3 = refp_sh[ii, ip].flatten() 182 | irefp1 = irefp[ii, ip].flatten() 183 | 184 | # Average over those bins that exceed the reflectivity 185 | # threshold (linear average) 186 | 187 | try: 188 | ref1[ii, jj] = np.nanmean(refp1) 189 | except ValueError: 190 | pass 191 | try: 192 | ref3[ii, jj] = np.nanmean(refp2) 193 | except ValueError: 194 | pass 195 | try: 196 | ref4[ii, jj] = np.nanmean(refp3) 197 | except ValueError: 198 | pass 199 | try: 200 | iref1[ii, jj] = np.nanmean(irefp1) 201 | except ValueError: 202 | pass 203 | 204 | try: 205 | if not l_dbz: 206 | stdv1[ii, jj] = np.nanstd(10 * np.log10(refp1)) 207 | else: 208 | stdv1[ii, jj] = np.nanstd(refp1) 209 | except ValueError: 210 | pass 211 | 212 | # Note the number of rejected bins 213 | nrej1[ii, jj] = int(np.sum(np.isnan(refp1))) 214 | if ~np.isnan(stdv1[ii, jj]) and nip - nrej1[ii, jj] > 1: 215 | continue 216 | 217 | # Compute the horizontal distance to all the GR bins 218 | d = sqrt((xg[:, :, jj] - x[ii, jj])**2 + (yg[:, :, jj] - y[ii, jj])**2) 219 | 220 | # Find all GR bins within the SR beam 221 | igx, igy = np.where(d <= ds[ii, jj] / 2) 222 | 223 | # Store the number of bins 224 | ntot2[ii, jj] = len(igx) 225 | if len(igx) == 0: 226 | continue 227 | 228 | # Extract the relevant GR data 229 | refg1 = reflectivity_ground_radar[:, :, jj][igx, igy].flatten() 230 | refg2 = refg_ku[:, :, jj][igx, igy].flatten() 231 | volg1 = volg[:, :, jj][igx, igy].flatten() 232 | irefg1 = irefg[:, :, jj][igx, igy].flatten() 233 | 234 | # Comupte the GR averaging volume 235 | vol2[ii, jj] = np.sum(volg1) 236 | 237 | # Average over those bins that exceed the reflectivity 238 | # threshold (exponential distance and volume weighting) 239 | w = volg1 * exp(-1 * (d[igx, igy] / (ds[ii, jj] / 2.))**2) 240 | w = w * refg1 / refg2 241 | 242 | ref2[ii, jj] = np.nansum(w * refg1) / np.nansum(w[~np.isnan(refg1)]) 243 | 244 | # if ref2[ii, jj] < minrefp: 245 | # ref2[ii, jj] = np.NaN 246 | 247 | ref5[ii, jj] = np.nansum(w * refg2) / np.nansum(w[~np.isnan(refg2)]) 248 | iref2[ii, jj] = np.nansum(w * irefg1) / np.nansum(w[~np.isnan(irefg1)]) 249 | 250 | if not l_dbz: 251 | stdv2[ii, jj] = np.nanstd(10 * np.log10(refg1)) 252 | else: 253 | stdv2[ii, jj] = np.nanstd(refg1) 254 | 255 | # Note the number of rejected bins 256 | nrej2[ii, jj] = int(np.sum(np.isnan(refg1))) 257 | # END FOR (satellite profiles, radar elevation) 258 | 259 | # Convert back to dBZ 260 | iref1 = 10 * np.log10(iref1) 261 | iref2 = 10 * np.log10(iref2) 262 | 263 | if not l_dbz: 264 | ref1 = 10 * np.log10(ref1) 265 | ref2 = 10 * np.log10(ref2) 266 | ref3 = 10 * np.log10(ref3) 267 | ref4 = 10 * np.log10(ref4) 268 | ref5 = 10 * np.log10(ref5) 269 | 270 | # Correct std 271 | stdv1[np.isnan(stdv1)] = 0 272 | stdv2[np.isnan(stdv2)] = 0 273 | 274 | ref2[ref2 < cpol.min_refl_thrld] = np.NaN 275 | 276 | # Extract comparison pairs 277 | ipairx, ipairy = np.where((~np.isnan(ref1)) & (~np.isnan(ref2))) 278 | if len(ipairx) < satellite.min_pair_nb: 279 | print_red('Insufficient comparison pairs.') 280 | return None 281 | 282 | # Save structure 283 | match_vol = dict() 284 | match_vol['zbb'] = zbb # Average bright band height 285 | match_vol['x'] = x[ipairx, ipairy] # x coordinate of sample 286 | match_vol['y'] = y[ipairx, ipairy] # y coordinate of sample 287 | match_vol['z'] = z[ipairx, ipairy] # z coordinate of sample 288 | match_vol['dz'] = dz[ipairx, ipairy] # depth of sample 289 | match_vol['ds'] = ds[ipairx, ipairy] # width of sample 290 | match_vol['r'] = r[ipairx, ipairy] # range of sample from GR 291 | match_vol['el'] = cpol.fields['elang'][ipairy] # Elevation angle 292 | 293 | match_vol['ref1'] = ref1[ipairx, ipairy] # PR reflectivity (Ku-band) 294 | match_vol['ref2'] = ref2[ipairx, ipairy] # GR reflectivity (S/C-band) 295 | match_vol['ref3'] = ref3[ipairx, ipairy] # PR reflectivity (S-band stratiform) 296 | match_vol['ref4'] = ref4[ipairx, ipairy] # PR reflectivity (S-band convective) 297 | match_vol['ref5'] = ref5[ipairx, ipairy] # GR reflectivity (Ku-band) 298 | match_vol['iref1'] = iref1[ipairx, ipairy] # path-integrated PR reflectivity 299 | match_vol['iref2'] = iref2[ipairx, ipairy] # path-integrated GR reflectivity 300 | match_vol['ntot1'] = ntot1[ipairx, ipairy] # total number of PR bins in sample 301 | match_vol['nrej1'] = nrej1[ipairx, ipairy] # number of rejected PR bins in sample 302 | match_vol['ntot2'] = ntot2[ipairx, ipairy] # total number of GR bins in sample 303 | match_vol['nrej2'] = nrej2[ipairx, ipairy] # number of rejected GR bins in sample 304 | 305 | match_vol['stdv1'] = stdv1[ipairx, ipairy] # std. dev. of PR reflectivity in sample 306 | match_vol['stdv2'] = stdv2[ipairx, ipairy] # std. dev. of GR reflectivity in sample 307 | match_vol['vol1'] = vol1[ipairx, ipairy] # total volume of PR bins in sample 308 | match_vol['vol2'] = vol2[ipairx, ipairy] # total volume of GR bins in sample 309 | 310 | return match_vol, ipairx 311 | 312 | 313 | def correct_parallax(xc, yc, xp, yp, alpha, the_range): 314 | """ 315 | Correct for parallax to get x, y, z coordinates. 316 | 317 | alpha dim is nprof x 1 and now we want nprof x nbin 318 | xc, yc, xp, yp dimensions are nprof x 1 319 | """ 320 | 321 | nprof, nbin = the_range.shape 322 | alpha_corr = np.zeros((nprof, nbin)) 323 | xc0 = np.zeros((nprof, nbin)) 324 | yc0 = np.zeros((nprof, nbin)) 325 | xp0 = np.zeros((nprof, nbin)) 326 | yp0 = np.zeros((nprof, nbin)) 327 | for idx in range(0, nbin): 328 | alpha_corr[:, idx] = alpha[:] 329 | xc0[:, idx] = xc[:] 330 | yc0[:, idx] = yc[:] 331 | xp0[:, idx] = xp[:] 332 | yp0[:, idx] = yp[:] 333 | 334 | alpha = alpha_corr 335 | zp = the_range * cos(pi / 180. * alpha_corr) 336 | ds = the_range * sin(pi / 180. * alpha_corr) 337 | ang = np.arctan2(yp0 - yc0, xp0 - xc0) 338 | dx = ds * cos(ang) 339 | dy = ds * sin(ang) 340 | xp = xp0 + dx 341 | yp = yp0 + dy 342 | 343 | return xp, yp, zp, ds, alpha 344 | 345 | 346 | def match_volumes(configuration_file, radfile, sat_file_1, sat_file_2A25_trmm=None, dtime_sat=None, 347 | radar_band="C", l_dbz=True, l_atten=True, gr_offset=0, log_path='.'): 348 | ''' 349 | MATCHPROJ_FUN 350 | 351 | Parameters 352 | ========== 353 | configuration_file: str 354 | Configuration file. 355 | radfile: st 356 | Ground radar file corresponding to satellite pass. 357 | sat_file_1: str. 358 | GPM file or TRMM 2A23 file. 359 | sat_file_2A25_trmm: str 360 | TRMM 2A25 files (None for GPM). 361 | dtime_sat: str 362 | Date of current processing. 363 | radar_band: str 364 | Possible values are 'S', 'C', or 'X' 365 | l_dbz, l_atten: bool 366 | Switches for use of linear reflectivity, GPM or TRMM, and attenuation 367 | correction 368 | 369 | Returns 370 | ======= 371 | match_vol: dict 372 | A dictionnary structure containing the comparable reflectivities. 373 | ''' 374 | if radar_band != "C" and radar_band != "S" and radar_band != "X": 375 | print_red(f"Ground radar frequency band unknown. You said {radar_band}. " + 376 | "The supported values are 'S', 'C', and 'X'. Doing nothing.") 377 | return None 378 | 379 | logging.basicConfig(filename=log_path + "log_matchvol_{}.log".format(dtime_sat.strftime("%Y%m%d")), level=logging.DEBUG) 380 | # Spawning Radar and Satellite 381 | cpol = Radar(configuration_file, gr_offset=gr_offset) 382 | satellite = Satellite(configuration_file, sat_file_1, sat_file_2A25_trmm) 383 | 384 | # Projecting on a WGS84 grid. 385 | pyproj_config = "+proj=tmerc +lon_0=%f +lat_0=%f +ellps=WGS84" % (cpol.longitude, cpol.latitude) 386 | smap = pyproj.Proj(pyproj_config) 387 | 388 | # Convert to Cartesian coordinates 389 | satellite_proj_cart = smap(satellite.lon, satellite.lat) 390 | xproj_sat = satellite_proj_cart[0] 391 | yproj_sat = satellite_proj_cart[1] 392 | 393 | # Identify profiles withing the domnain 394 | ioverx, iovery = np.where((xproj_sat >= cpol.xmin) & (xproj_sat <= cpol.xmax) & 395 | (yproj_sat >= cpol.ymin) & (yproj_sat <= cpol.ymax)) 396 | if len(ioverx) == 0: 397 | print_red("Insufficient satellite rays in domain for " + dtime_sat.strftime("%d %b %Y")) 398 | logging.error("Insufficient satellite rays in domain for " + dtime_sat.strftime("%d %b %Y")) 399 | return None 400 | 401 | # Note the first and last scan indices 402 | # i1x, i1y = np.min(ioverx), np.min(iovery) 403 | # i2x, i2y = np.max(ioverx), np.max(iovery) 404 | 405 | # Determine the datetime of the closest approach of TRMM to the GR 406 | xclose_sat = xproj_sat[:, 24] # Grid center 407 | yclose_sat = yproj_sat[:, 24] 408 | # iclose = np.argmin(sqrt(xclose_sat**2 + yclose_sat**2)) 409 | 410 | # Compute the distance of every ray to the radar 411 | dist_to_gr_rays = sqrt(xproj_sat**2 + yproj_sat**2) 412 | 413 | # Identify precipitating profiles within the radaar range limits 414 | iscan, iray = np.where((dist_to_gr_rays >= cpol.rmin) & (dist_to_gr_rays <= cpol.rmax) & (satellite.pflag == 2)) 415 | nprof = len(iscan) 416 | if nprof < satellite.min_prof_nb: 417 | print_red('Insufficient precipitating satellite rays in domain %i.' % (nprof)) 418 | logging.error('Insufficient precipitating satellite rays in domain %i.' % (nprof)) 419 | return None 420 | 421 | # Extract data for these rays 422 | xproj_sat = xproj_sat[iscan, iray] 423 | yproj_sat = yproj_sat[iscan, iray] 424 | xclose_sat = xclose_sat[iscan] 425 | yclose_sat = yclose_sat[iscan] 426 | ptype = satellite.ptype[iscan, iray] 427 | zbb = satellite.zbb[iscan, iray] 428 | bbwidth = satellite.bbwidth[iscan, iray] 429 | sfc = satellite.sfc[iscan, iray] 430 | quality = satellite.quality[iscan, iray] 431 | alpha = satellite.alpha[iray] 432 | 433 | tmp = np.zeros((nprof, satellite.nbin), dtype=float) 434 | for k in range(0, satellite.nbin): 435 | tmp[:, k] = (satellite.refl[:, :, k])[iscan, iray] 436 | dbz_sat = tmp 437 | 438 | # we want a shape of (nprof, satellite.nbin) 439 | range_sat_2d = np.zeros((nprof, satellite.nbin)) 440 | for idx in range(0, nprof): 441 | range_sat_2d[idx, :] = satellite.dr * np.arange(satellite.nbin) 442 | 443 | # Correct coordinates from parallax errors. 444 | rslt = correct_parallax(xclose_sat, yclose_sat, xproj_sat, yproj_sat, alpha, range_sat_2d) 445 | xproj_sat_pxcorr, yproj_sat_pxcorr, z_sat_pxcorr, ds_pxcorr, alpha_pxcorr = rslt 446 | 447 | if len(ds_pxcorr) == 0: 448 | return None 449 | if np.min(ds_pxcorr) < 0: 450 | return None 451 | 452 | # Compute the (approximate) volume of each PR bin 453 | rt = satellite.altitude / cos(pi / 180 * alpha_pxcorr) - range_sat_2d 454 | 455 | # Compute the ground-radar coordinates of the PR pixels 456 | gamma = sqrt(xproj_sat_pxcorr**2 + yproj_sat_pxcorr**2) / cpol.gaussian_radius 457 | elev_pr_grref = 180 / pi * np.arctan((cos(gamma) - (cpol.gaussian_radius + cpol.altitude) / (cpol.gaussian_radius + z_sat_pxcorr)) / sin(gamma)) 458 | # range_pr_grref = (cpol.gaussian_radius + z_sat_pxcorr) * sin(gamma) / cos(pi / 180 * elev_pr_grref) # Not used 459 | # azi_pr_grref = 90 - 180 / pi * np.arctan2(yproj_sat_pxcorr, xproj_sat_pxcorr) # Not used 460 | 461 | # Set all values less than satellite.min_refl_thrld as missing 462 | dbz_sat = np.ma.masked_where(dbz_sat < satellite.min_refl_thrld, dbz_sat) 463 | 464 | # Determine the median brightband height 465 | ibb = np.where((zbb > 0) & (bbwidth > 0) & (quality == 1))[0] 466 | nbb = len(ibb) 467 | if nbb >= satellite.min_prof_nb: 468 | zbb = np.median(zbb[ibb]) 469 | bbwidth = np.median(bbwidth[ibb]) 470 | refp_ss, refp_sh = reflectivity_conversion.convert_sat_refl_to_gr_band(dbz_sat, 471 | z_sat_pxcorr, 472 | zbb, 473 | bbwidth, 474 | radar_band=radar_band) 475 | else: 476 | refp_ss, refp_sh = reflectivity_conversion.convert_gpmrefl_grband_dfr(dbz_sat, 477 | radar_band=radar_band) 478 | 479 | print_green("Satellite side OK.") 480 | print_yellow("Reading {}.".format(radfile)) 481 | radar = read_radar(radfile, offset=cpol.offset) 482 | if radar is None: 483 | print_red("Could not read the ground radar file. Doing nothing.") 484 | return None 485 | dtime_radar = radar['time'] 486 | time_difference = np.abs(dtime_sat - dtime_radar) 487 | 488 | cpol.set_fields(radar) 489 | print_yellow("Ground radar data loaded.") 490 | 491 | # The Call. 492 | print_magenta("Starting volume matching.") 493 | match_vol, ipairx = _matching(satellite, cpol, nprof, dbz_sat, refp_ss, refp_sh, xproj_sat_pxcorr, 494 | yproj_sat_pxcorr, z_sat_pxcorr, rt, elev_pr_grref, alpha_pxcorr, zbb, l_dbz) 495 | if match_vol is None: 496 | return None 497 | 498 | print_magenta("Volume matching done.") 499 | 500 | match_vol['date'] = dtime_sat 501 | match_vol['bbwidth'] = bbwidth 502 | match_vol['dt'] = time_difference.seconds 503 | 504 | match_vol['sfc'] = sfc[ipairx] 505 | match_vol['ptype'] = ptype[ipairx] 506 | match_vol['iray'] = iray[ipairx] 507 | match_vol['iscan'] = iscan[ipairx] 508 | 509 | return match_vol 510 | -------------------------------------------------------------------------------- /msgr/io/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vlouf/matchproj-python/78c2fd72c7655cfd38ec478d4c9b31fca971ff7f/msgr/io/__init__.py -------------------------------------------------------------------------------- /msgr/io/read_gpm.py: -------------------------------------------------------------------------------- 1 | import datetime 2 | 3 | import h5py 4 | import numpy as np 5 | import itertools 6 | 7 | def read_date_from_GPM(infile, radar_lat, radar_lon): 8 | """ 9 | Extract datetime from TRMM HDF files. 10 | 11 | Parameters: 12 | =========== 13 | infile: str 14 | Satellite data filename. 15 | radar_lat: float 16 | Latitude of ground radar 17 | radar_lon: float 18 | Longitude of ground radar 19 | 20 | Returns: 21 | ======== 22 | gpm_date: datetime 23 | Datetime of satellite data at ground radar position. 24 | min_dist: float 25 | Minimal distance between satellite swath and ground radar, i.e. 26 | is satellite swath are in ground radar domain? 27 | """ 28 | with h5py.File(infile, 'r') as file_id: 29 | obj_id = file_id['NS'] 30 | # Read GPM lat/lon 31 | latitude = obj_id['Latitude'].value 32 | longitude = obj_id['Longitude'].value 33 | # Read time data 34 | mem_id = obj_id['ScanTime'] 35 | year = mem_id['Year'].value 36 | month = mem_id['Month'].value 37 | day = mem_id['DayOfMonth'].value 38 | hour = mem_id['Hour'].value 39 | minute = mem_id['Minute'].value 40 | second = mem_id['Second'].value 41 | 42 | # Using distance, find min to radar 43 | dist = np.sqrt((latitude - radar_lat)**2 + (longitude - radar_lon)**2) 44 | dist_atrack = np.amin(dist, axis=1) # Min distance along track axis 45 | radar_center = np.argmin(dist_atrack) 46 | min_dist = np.amin(dist_atrack) 47 | gpm_date = datetime.datetime(year[radar_center], month[radar_center], day[radar_center], 48 | hour[radar_center], minute[radar_center], second[radar_center]) 49 | return gpm_date, min_dist 50 | 51 | 52 | def read_gpm(infile, sat_offset=None): 53 | """ 54 | READ_GPM 55 | Read HDF5 GPM file with these parameters: 56 | - dataQuality 57 | - landSurfaceType 58 | - flagPrecip 59 | - flagBB 60 | - heightBB 61 | - widthBB 62 | - qualityBB 63 | - typePrecip 64 | - qualityTypePrecip 65 | - zFactorCorrected 66 | 67 | It will reverse direction along the beam for reflectivity so that the first 68 | value along the z-axis of the reflectivity array corresponds to the ground 69 | value and not the top of the atm. 70 | 71 | Returns a dictionnary containing the data. 72 | """ 73 | 74 | with h5py.File(infile, 'r') as file_id: 75 | obj_id = file_id['NS'] 76 | 77 | lat = obj_id['Latitude'].value 78 | lon = obj_id['Longitude'].value 79 | 80 | # Read time data 81 | mem_id = obj_id['ScanTime'] 82 | year = mem_id['Year'].value 83 | month = mem_id['Month'].value 84 | day = mem_id['DayOfMonth'].value 85 | hour = mem_id['Hour'].value 86 | minute = mem_id['Minute'].value 87 | second = mem_id['Second'].value 88 | 89 | # Read in the surface type and precipitation flag 90 | mem_id = obj_id['PRE'] 91 | sfc = mem_id['landSurfaceType'].value 92 | pflag = mem_id['flagPrecip'].value 93 | try: 94 | cfb = mem_id['binClutterFreeBottom'].value 95 | except KeyError: 96 | pass 97 | 98 | # Read in the brightband and precipitation type data 99 | mem_id = obj_id['CSF'] 100 | zbb = mem_id['heightBB'].value 101 | qbb = mem_id['qualityBB'].value 102 | qtype = mem_id['qualityTypePrecip'].value 103 | ptype = mem_id['typePrecip'].value 104 | bbwidth = mem_id['widthBB'].value 105 | 106 | # Read in the data quality 107 | mem_id = obj_id['scanStatus'] 108 | quality = mem_id['dataQuality'].value 109 | 110 | # Read in the reflectivity data 111 | mem_id = obj_id['SLV'] 112 | # Removing the 1.3 dB offset from GPM. 113 | if sat_offset is not None: 114 | print(f"{sat_offset}dB offset applied to GPM reflectivity.") 115 | refl = mem_id['zFactorCorrected'].value + sat_offset 116 | else: 117 | refl = mem_id['zFactorCorrected'].value 118 | 119 | # Determine the dimensions 120 | if refl.ndim != 3: 121 | print("Invalid number of dimensions") 122 | return None 123 | 124 | nscan, nray, nbin = refl.shape 125 | 126 | # remove clutter free base 127 | if 'cfb' in locals(): 128 | for ii, jj in itertools.product(range(nscan), range(nray)): 129 | cfb_idx = cfb[ii, jj] - 1 #convert to zero-base index for python 130 | if not cfb_idx == -9999: 131 | refl[:, :, (cfb_idx + 1):] = -9999.9 #set values below the clutter free base to the default fill value for reflectivity (idx of 1 is top of profile) 132 | 133 | # Reverse direction along the beam 134 | refl = refl[:, :, ::-1] 135 | 136 | #print(refl[1,1,:]) 137 | # Change pflag=1 to pflag=2 to be consistent with 'Rain certain' in TRMM 138 | pflag[pflag == 1] = 2 139 | 140 | # Simplify the precipitation types 141 | ptype = ptype / 10000000.0 142 | 143 | # Simplify the surface types 144 | imissx, imissy = np.where(sfc == -9999) 145 | nmiss = len(imissx) 146 | sfc = sfc / 100 + 1 147 | if nmiss > 0: 148 | sfc[imissx, imissy] = 0 149 | 150 | # Set a quality indicator for the BB and precip type data 151 | quality = np.zeros((nscan, nray)) 152 | quality[((qbb == 0) | (qbb == 1)) & (qtype == 1)] = 1 153 | quality[(qbb > 1) | (qtype > 1)] = 2 154 | 155 | # Store data in a dict 156 | data_dict = dict() 157 | data_dict = {'nscan': nscan, 'nray': nray, 'nbin': nbin, 'year': year, 'month': month, 158 | 'day': day, 'hour': hour, 'minute': minute, 'second': second, 'lon': lon, 159 | 'lat': lat, 'pflag': pflag, 'ptype': ptype, 'zbb': zbb, 'bbwidth': bbwidth, 160 | 'sfc': sfc, 'quality': quality, 'refl': refl} 161 | 162 | return data_dict -------------------------------------------------------------------------------- /msgr/io/read_radar.py: -------------------------------------------------------------------------------- 1 | import traceback 2 | 3 | # Other libraries. 4 | import pyart 5 | import cftime 6 | import numpy as np 7 | 8 | from scipy.interpolate import griddata 9 | 10 | 11 | def _transform_grid(r, azi, dbz): 12 | # Sort reflectivity field such as dbz[0, :] corresponds to azimuth 0 degree. 13 | pos = np.argsort(azi) 14 | azi = azi[pos] 15 | dbz = dbz[pos, :] 16 | if len(azi) == 360: 17 | return dbz 18 | 19 | # The azimuth length is not equal to 360, we'll have to interpolate it. 20 | yazi = np.arange(0, 360) 21 | XR, YA = np.meshgrid(r, yazi) 22 | Xi, Yi = np.meshgrid(r, azi) 23 | vq = griddata((Xi.flatten(), Yi.flatten()), dbz.flatten(), 24 | (XR.flatten(), YA.flatten())).reshape(YA.shape) 25 | 26 | return vq 27 | 28 | 29 | def transform_reflectivity(radar, refl_name): 30 | """ 31 | Transform reflectivity from a 2D array of dimensions (time, range) to (range, azimuth, elevation) 32 | 33 | Parameters: 34 | =========== 35 | radar: Py-ART obj 36 | refl_name: str 37 | Name of the reflectivity field. 38 | 39 | Returns: 40 | ======== 41 | reflectivity_3D: ndarray 60)] = np.NaN # Remove extreme values. 127 | if offset is not None: 128 | reflectivity += offset 129 | 130 | r = radar.range['data'] 131 | azimuth = np.arange(0, 360) 132 | elevation = np.zeros(ntilt) 133 | for idx in range(ntilt): 134 | sl = radar.get_slice(idx) 135 | elevation[idx] = radar.elevation['data'][sl].mean() 136 | 137 | # Make 3D matrices for coordinates shape (r, azi, elev) 138 | rg2d = np.repeat(r[:, np.newaxis], nbeam, axis=1) 139 | rg3d = np.repeat(rg2d[:, :, np.newaxis], ntilt, axis=2) 140 | # Azimuth 141 | az2d = np.repeat(azimuth[:, np.newaxis], ntilt, axis=1) 142 | az3d = np.repeat(az2d[np.newaxis, :, :], ngate, axis=0) 143 | # Elevation 144 | el2d = np.repeat(elevation[np.newaxis, :], nbeam, axis=0) 145 | el3d = np.repeat(el2d[np.newaxis, :, :], ngate, axis=0) 146 | 147 | data_dict = dict() 148 | data_dict['ngate'] = ngate 149 | data_dict['nbeam'] = nbeam 150 | data_dict['ntilt'] = ntilt 151 | data_dict['range'] = rg3d 152 | data_dict['azang'] = az3d 153 | data_dict['elev_3d'] = el3d 154 | data_dict['elang'] = elevation 155 | data_dict['dr'] = r[1] - r[0] 156 | data_dict['reflec'] = reflectivity 157 | data_dict['time'] = dtime_radar 158 | 159 | return data_dict 160 | -------------------------------------------------------------------------------- /msgr/io/read_trmm.py: -------------------------------------------------------------------------------- 1 | # Python Standard Library 2 | import datetime 3 | 4 | # Other libraries. 5 | import netCDF4 6 | import numpy as np 7 | # from pyhdf.SD import SD, SDC 8 | 9 | def read_date_from_TRMM(hdf_file1, radar_lat, radar_lon): 10 | """ 11 | Extract datetime from TRMM HDF files. 12 | 13 | Parameters: 14 | =========== 15 | hdf_file1: str 16 | File name for TRMM satellite file. 17 | radar_lat: float 18 | Latitude of ground radar 19 | radar_lon: float 20 | Longitude of ground radar 21 | 22 | Returns: 23 | ======== 24 | trmm_date: datetime 25 | Datetime of satellite data at ground radar position. 26 | min_dist: float 27 | Minimal distance between satellite swath and ground radar, i.e. 28 | is satellite swath are in ground radar domain? 29 | """ 30 | with netCDF4.Dataset(hdf_file1, 'r') as ncid: 31 | year = ncid['Year'][:] 32 | month = ncid['Month'][:] 33 | day = ncid['DayOfMonth'][:] 34 | hour = ncid['Hour'][:] 35 | minute = ncid['Minute'][:] 36 | second = ncid['Second'][:] 37 | latitude = ncid['Latitude'][:] 38 | longitude = ncid['Longitude'][:] 39 | 40 | # Using distance, find min to radar 41 | dist = np.sqrt((latitude - radar_lat)**2 + (longitude - radar_lon)**2) 42 | dist_atrack = np.amin(dist, axis=1) # Min distance along track axis 43 | radar_center = np.argmin(dist_atrack) 44 | min_dist = np.amin(dist_atrack) 45 | trmm_date = datetime.datetime(year[radar_center], month[radar_center], day[radar_center], 46 | hour[radar_center], minute[radar_center], second[radar_center]) 47 | return trmm_date, min_dist 48 | 49 | 50 | def read_trmm(hdf_file1, hdf_file2, sat_offset=None): 51 | ''' 52 | Reads TRMM 2A23 and 2A25 data files. 53 | 54 | Parameters: 55 | =========== 56 | hdf_file1: str 57 | File name for TRMM 2A23 data. 58 | hdf_file2: str 59 | File name for TRMM 2A25 data. 60 | 61 | Returns: 62 | ======== 63 | data_dict: dict 64 | Dictionnary containing all the needed data from the 2A23 and 2A25 files. 65 | ''' 66 | with netCDF4.Dataset(hdf_file1, 'r') as ncid: 67 | year = ncid['Year'][:] 68 | month = ncid['Month'][:] 69 | day = ncid['DayOfMonth'][:] 70 | hour = ncid['Hour'][:] 71 | minute = ncid['Minute'][:] 72 | second = ncid['Second'][:] 73 | Latitude = ncid['Latitude'][:] 74 | Longitude = ncid['Longitude'][:] 75 | bbwidth = ncid['BBwidth'][:] 76 | HBB = ncid['HBB'][:] 77 | dataQuality = ncid['dataQuality'][:] 78 | rainFlag = ncid['rainFlag'][:] 79 | rainType = ncid['rainType'][:] 80 | status = ncid['status'][:] 81 | 82 | if dataQuality.max() != 0: 83 | return None 84 | 85 | with netCDF4.Dataset(hdf_file2, 'r') as ncid: 86 | # Latitude25 = ncid['Latitude'][:] 87 | # Longitude25 = ncid['Longitude'][:] 88 | correctZFactor = ncid['correctZFactor'][:] 89 | nscan, nray, nbin = correctZFactor.shape 90 | 91 | reflectivity = correctZFactor / 100.0 92 | 93 | # Reverse direction along the beam 94 | if sat_offset is not None: 95 | print(f"{sat_offset}dB offset applied to TRMM reflectivity.") 96 | reflectivity = reflectivity[:, :, ::-1] + sat_offset 97 | else: 98 | reflectivity = reflectivity[:, :, ::-1] 99 | 100 | rainFlag[(rainFlag >= 10) & (rainFlag < 20)] = 1 101 | rainFlag[rainFlag >= 20] = 2 102 | 103 | ptype = rainType.copy() 104 | ptype[rainType >= 300] = 3 105 | ptype[(rainType >= 200) & (rainType < 300)] = 2 106 | ptype[(rainType >= 100) & (rainType < 200)] = 1 107 | ptype[rainType == -88] = 0 108 | ptype[rainType == -99] = 1 109 | 110 | # Extract the surface Type 111 | sfc = np.zeros(status.shape, dtype=int) - 1 112 | sfc[status == 168] = 0 113 | sfc[status % 10 == 0] = 1 114 | sfc[(status - 1) % 10 == 0] = 2 115 | sfc[(status - 2) % 10 == 0] = 3 116 | sfc[(status - 3) % 10 == 0] = 4 117 | sfc[(status - 4) % 10 == 0] = 5 118 | sfc[(status - 9) % 10 == 0] = 9 119 | 120 | # Extract 2A23 quality 121 | quality = np.zeros(status.shape, dtype=int) - 1 122 | quality[status == 168] = 0 123 | quality[status < 50] = 1 124 | quality[status >= 50] = 2 125 | 126 | data_dict = dict() 127 | data_dict = {'nscan': nscan, 'nray': nray, 'nbin': nbin, 'year': year, 128 | 'month': month, 'day': day, 'hour': hour, 'minute': minute, 129 | 'second': second, 'lon': Longitude, 'lat': Latitude, 130 | 'pflag': rainFlag, 'ptype': ptype, 'zbb': HBB, 'bbwidth': bbwidth, 131 | 'sfc': sfc, 'quality': quality, 'refl': reflectivity} 132 | 133 | return data_dict 134 | -------------------------------------------------------------------------------- /msgr/io/save_data.py: -------------------------------------------------------------------------------- 1 | import netCDF4 2 | 3 | 4 | def _get_metadata(): 5 | metadat = dict() 6 | metadat['iscan'] = {'long_name': 'PR scan index', 'units': None} 7 | metadat['iray'] = {'long_name': 'PR ray index', 'units': None} 8 | metadat['itilt'] = {'long_name': 'GR tilt index', 'units': None} 9 | metadat['x'] = {'long_name': 'W-E location w.r.t. radar', 'units': 'm'} 10 | metadat['y'] = {'long_name': 'S-N location w.r.t. radar', 'units': 'm'} 11 | metadat['z'] = {'long_name': 'Height above MSL', 'units': 'm'} 12 | metadat['r'] = {'long_name': 'Range from GR', 'units': 'm'} 13 | metadat['el'] = {'long_name': 'Elevation angle', 'units': 'deg'} 14 | metadat['dz'] = {'long_name': 'Depth of averaging volume', 'units': 'm'} 15 | metadat['ds'] = {'long_name': 'Diameter of averaging volume', 'units': 'm'} 16 | metadat['dt'] = {'long_name': 'Time difference between GR and PR samples', 'units': 's'} 17 | metadat['ntot1'] = {'long_name': 'Number of PR points in averaging volume', 'units': None} 18 | metadat['ntot2'] = {'long_name': 'Number of GR points in averaging volume', 'units': None} 19 | metadat['nrej1'] = {'long_name': 'Number of rejected PR points in averaging volume', 'units': None} 20 | metadat['nrej2'] = {'long_name': 'Number of rejected GR points in averaging volume', 'units': None} 21 | metadat['sfc'] = {'long_name': 'Surface type (1=Ocean, 2=Land, 3=Coast, 4=Lake, 5=Unknown)', 'units': None} 22 | metadat['ptype'] = {'long_name': 'Precipitation type (1=Strat, 2=Conv, 3=Other)', 'units': None} 23 | metadat['ref1'] = {'long_name': 'PR reflectivity', 'units': 'dBZ'} 24 | metadat['ref2'] = {'long_name': 'GR reflectivity', 'units': 'dBZ'} 25 | metadat['ref3'] = {'long_name': 'PR reflectivity (S-band, Snow)', 'units': 'dBZ'} 26 | metadat['ref4'] = {'long_name': 'PR reflectivity (S-band, Hail)', 'units': 'dBZ'} 27 | metadat['ref5'] = {'long_name': 'GR reflectivity (Ku-band)', 'units': 'dBZ'} 28 | metadat['stdv1'] = {'long_name': 'Standard deviation of PR reflectivity', 'units': 'dB'} 29 | metadat['stdv2'] = {'long_name': 'Standard deviation of GR reflectivity', 'units': 'dB'} 30 | metadat['iref1'] = {'long_name': 'Path-integrated PR reflectivity', 'units': 'dB'} 31 | metadat['iref2'] = {'long_name': 'Path-integrated GR reflectivity', 'units': 'dB'} 32 | metadat['zbb'] = {'long_name': 'Average bright band height', 'units': 'm'} 33 | metadat['bbwidth'] = {'long_name': 'Average bright band width', 'units': 'm'} 34 | metadat['nbb'] = {'long_name': 'Number of profiles with a bright band', 'units': None} 35 | metadat['vol1'] = {'long_name': 'PR averaging volume', 'units': 'km^3'} 36 | metadat['vol2'] = {'long_name': 'GR averaging volume', 'units': 'km^3'} 37 | 38 | return metadat 39 | 40 | 41 | def save_data(outfilename, data, date, offset1=None, offset2=None, nb_pass=0): 42 | """ 43 | SAVE_DATA 44 | Dumps data in a python's pickle file 45 | Will try to populate the metadata based on the key name. 46 | 47 | Parameters: 48 | =========== 49 | outfilename: str 50 | Output file name. 51 | data: dict 52 | Dictionnary of data to save. 53 | do_hdf: bool 54 | Save as a HDF file. 55 | """ 56 | metadat = _get_metadata() 57 | 58 | xdim = len(data['ref1']) 59 | tiltdim = len(data['el']) 60 | profdim = len(data['sfc']) 61 | 62 | with netCDF4.Dataset(outfilename, "w", format="NETCDF4") as rootgrp: 63 | # Create dimension 64 | rootgrp.createDimension("x", xdim) 65 | rootgrp.createDimension('tilt', tiltdim) 66 | rootgrp.createDimension('profile', profdim) 67 | rootgrp.createDimension('time', 1) 68 | time = rootgrp.createVariable("time", "f8", ('time')) 69 | time.units = "seconds since 1970-01-01T00:00:00Z" 70 | time[:] = netCDF4.date2num(date, "seconds since 1970-01-01T00:00:00Z") 71 | 72 | if nb_pass == 0: 73 | ncoff = rootgrp.createVariable("offset1", "f8", ("time")) 74 | ncoff[:] = offset1 75 | ncoff.setncattr_string("description", "Difference reflectivity Satellite - Ground Radar. PASS 1") 76 | else: 77 | ncoff = rootgrp.createVariable("offset1", "f8", ("time")) 78 | ncoff[:] = offset1 79 | ncoff.setncattr_string("description", "Difference reflectivity Satellite - Ground Radar. PASS 1") 80 | 81 | ncoff = rootgrp.createVariable("offset2", "f8", ("time")) 82 | ncoff[:] = offset2 83 | ncoff.setncattr_string("description", "Difference reflectivity Satellite - Ground Radar. PASS 2") 84 | 85 | ncoff = rootgrp.createVariable("offset_total", "f8", ("time")) 86 | ncoff[:] = offset2 + offset1 87 | ncoff.setncattr_string("description", "Difference reflectivity Satellite - Ground Radar. TOTAL") 88 | 89 | for k, v in data.items(): 90 | if k in ['date', 'dt']: 91 | # rootgrp.setncattr(k, v) 92 | continue 93 | 94 | if k == "el": 95 | ncelev = rootgrp.createVariable('elevation', 'f8', ("tilt"), zlib=True) 96 | ncelev[:] = v 97 | ncelev.setncattr_string("long_name", metadat[k]['long_name']) 98 | if metadat[k]['units'] is not None: 99 | ncelev.units = metadat[k]['units'] 100 | continue 101 | 102 | if k in ['sfc', 'ptype', 'iray', 'iscan']: 103 | ncprof = rootgrp.createVariable(k, 'f8', ("profile"), zlib=True) 104 | ncprof[:] = v 105 | ncprof.setncattr_string("long_name", metadat[k]['long_name']) 106 | if metadat[k]['units'] is not None: 107 | ncprof.units = metadat[k]['units'] 108 | continue 109 | 110 | ncmoment = rootgrp.createVariable(k, 'f8', ("x",), zlib=True) 111 | ncmoment[:] = v 112 | try: 113 | if metadat[k]['units'] is not None: 114 | ncmoment.units = metadat[k]['units'] 115 | ncmoment.setncattr_string("long_name", metadat[k]['long_name']) 116 | except KeyError: 117 | pass 118 | 119 | return None 120 | -------------------------------------------------------------------------------- /msgr/utils/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/vlouf/matchproj-python/78c2fd72c7655cfd38ec478d4c9b31fca971ff7f/msgr/utils/__init__.py -------------------------------------------------------------------------------- /msgr/utils/misc.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module regroups a bunch of usefull functions: 3 | - find_file_with_string (return the element of a list containing a specific string) 4 | - nancumsum (numpy's cumsum with support of NaN values) 5 | - get_files (walks into directories and returns all the files with supported extension) 6 | - get_time_from_filename (uses regular expression to catch the date in a filename) 7 | - get_closest_date (get the closest date in a list) 8 | - get_filename_from_date (takes a list of files and a date and returns the file closest to this date) 9 | """ 10 | 11 | import re 12 | import os 13 | import copy 14 | import crayons # Color terminal 15 | import datetime 16 | import numpy as np 17 | from dateutil import parser 18 | 19 | import cftime 20 | import pandas as pd 21 | import pyart 22 | import xarray as xr 23 | 24 | 25 | def find_file_with_string(flist, orb): 26 | """ 27 | FIND_FILE_WITH_STRING 28 | Return the element of a list flist containing the value of orb 29 | 30 | Parameters 31 | ========== 32 | flist: list[str] 33 | List of file list 34 | orb: str 35 | String we are looking for in the list 36 | 37 | Returns 38 | ======= 39 | Element of a list flist containing the value of orb 40 | """ 41 | return [fd for fd in flist if orb in fd][0] 42 | 43 | 44 | def nancumsum(a, ax=0): 45 | ''' 46 | NANCUMSUM 47 | Cumsum in numpy does not ignore the NaN values, this one does 48 | Note that nancumsum will be implemented in numpy v1.12 49 | ''' 50 | 51 | tmp = copy.deepcopy(a) 52 | tmp[np.isnan(tmp)] = 0 53 | rslt = np.cumsum(tmp, axis=ax) 54 | rslt[np.isnan(a)] = np.NaN 55 | 56 | return rslt 57 | 58 | 59 | def get_files(inpath, date=None): 60 | ''' 61 | GET_FILES 62 | Returns a list of with the supported extension (netcdf) in the given 63 | path. Will recursively search in subdirectories too. If provided a date 64 | (string or datetime object) it will only returns the files whose 65 | filename matches. 66 | ''' 67 | 68 | supported_extension = ['.nc', '.NC', '.cdf', '.hdf5', '.h5', '.HDF5', 69 | '.H5', '.lassen', '.PPI', '.UF', ".gz", ".GZ", ".zip"] 70 | flist = [] 71 | 72 | # Check date type 73 | if type(date) == datetime.datetime: 74 | date = date.strftime("%Y%m%d") 75 | 76 | for dirpath, dirnames, filenames in os.walk(inpath): 77 | for filenames_slice in filenames: 78 | 79 | # If no date provided, nothing new under the sun 80 | if date is None: 81 | pass # pretends there was no if statement 82 | elif date in filenames_slice: 83 | pass # pretends there was no if statement 84 | else: 85 | continue 86 | 87 | file_extension = os.path.splitext(str(filenames_slice))[1] 88 | # Get extension 89 | 90 | if np.any(np.in1d(supported_extension, file_extension)): 91 | # Check if file extension is in the list of supported ones 92 | the_path = os.path.join(dirpath, filenames_slice) 93 | elif '.RAW' in filenames_slice: 94 | the_path = os.path.join(dirpath, filenames_slice) 95 | else: # If not test next file. 96 | continue 97 | 98 | # File does have the supported extension, we keep it for returning 99 | # list 100 | flist.append(the_path) 101 | 102 | to_return = flist 103 | 104 | return sorted(to_return) # Type: List[str, ...] 105 | 106 | 107 | def get_time_from_filename(filename, date): 108 | ''' 109 | GET_TIME_FROM_FILENAME 110 | Capture the time string inside the filename and returns it. 111 | 112 | Parameters: 113 | =========== 114 | filename: str 115 | String to parse for date. 116 | date: str 117 | Date (format YYYYMMDD) to look for in files. 118 | 119 | Returns: 120 | ======== 121 | date_time: datetime 122 | Datetime corresponding to given filename. 123 | ''' 124 | # Looking for date followed by underscore (or not) and 6 (or 4) consecutives 125 | # number (i.e. the time) 126 | # There is maybe an optionnal character (like _) between date and time 127 | 128 | try: 129 | dtstr = re.findall(date + '.[0-9]{6}', filename) 130 | dtime = datetime.datetime.strptime(dtstr[0], '%Y%m%d.%H%M%S') 131 | return dtime 132 | except Exception: 133 | pass 134 | 135 | if filename[-2:] == "nc" or filename[-2:] == "NC": 136 | ds = xr.open_dataset(filename) 137 | # I wish it was simpler in python but it's not. 138 | dates = pd.DatetimeIndex([ds.time.values[0]]) 139 | return dates.to_pydatetime()[0] 140 | 141 | if filename[-2:] == "gz" or '.RAW' in filename: 142 | # SIGMET file date convention. 143 | radar = pyart.io.read(filename) 144 | dtime = cftime.num2pydate(radar.time['data'][0], radar.time['units']) 145 | return dtime 146 | else: 147 | strlist = re.findall(date + ".?[0-9]{6}", filename) 148 | if len(strlist) == 0: 149 | strlist = re.findall(date + ".?[0-9]{4}", filename) 150 | 151 | try: 152 | date_time = parser.parse(strlist[0], fuzzy=True) 153 | except ValueError: 154 | date_time = datetime.datetime.strptime(strlist[0], "%Y%m%d") 155 | except IndexError: 156 | date_time = None 157 | 158 | return date_time # Type: str 159 | 160 | 161 | def get_closest_date(list_date, base_time): 162 | ''' 163 | GET_CLOSEST_DATE 164 | from: http://stackoverflow.com/a/17249470/846892 165 | ''' 166 | 167 | b_d = base_time 168 | 169 | def func(x): 170 | dd = x 171 | delta = dd - b_d if dd > b_d else datetime.timedelta.max 172 | return delta 173 | 174 | return min(list_date, key=func) # Type: datetime 175 | 176 | 177 | def get_filename_from_date(file_list, the_date): 178 | ''' 179 | GET_FILENAME_FROM_DATE 180 | Looks for a file in a list of file with the exact corresponding date and 181 | returns it. 182 | ''' 183 | 184 | # There is maybe an optionnal character(underscore) between date and time 185 | rt_str = the_date.strftime("%y%m%d.?%H%M") 186 | for the_file in file_list: 187 | try: 188 | re.findall(rt_str, the_file)[0] # If does not exist it raises an error 189 | to_return = the_file 190 | break # We found what we are looking for, exiting the loop 191 | except IndexError: 192 | continue 193 | 194 | return to_return # Type: str 195 | 196 | 197 | def print_with_time(txt): 198 | ''' 199 | PRINT_WITH_TIME 200 | ''' 201 | pfix = "[" + str(datetime.datetime.now().isoformat()) + "]\t" 202 | print(crayons.blue(pfix, bold=True) + txt) 203 | return None 204 | 205 | 206 | # To print in color in the terminal. Pretty much self-explanatory. 207 | def print_red(txt, bold=False): 208 | print_with_time(crayons.red(txt, bold)) 209 | return None 210 | 211 | 212 | def print_green(txt, bold=False): 213 | print_with_time(crayons.green(txt, bold)) 214 | return None 215 | 216 | 217 | def print_yellow(txt, bold=False): 218 | print_with_time(crayons.yellow(txt, bold)) 219 | return None 220 | 221 | 222 | def print_blue(txt, bold=False): 223 | print(crayons.blue(txt, bold)) 224 | return None 225 | 226 | 227 | def print_magenta(txt, bold=False): 228 | print(crayons.magenta(txt, bold)) 229 | return None 230 | -------------------------------------------------------------------------------- /msgr/utils/reflectivity_conversion.py: -------------------------------------------------------------------------------- 1 | import copy 2 | import numpy as np 3 | 4 | 5 | def convert_sat_refl_to_gr_band(refp, zp, zbb, bbwidth, radar_band='S'): 6 | """ 7 | Convert the satellite reflectivity to S, C, or X-band using the Cao et al. 8 | (2013) method. 9 | 10 | Parameters 11 | ========== 12 | refp: 13 | Satellite reflectivity field. 14 | zp: 15 | Altitude. 16 | zbb: 17 | Bright band height. 18 | bbwidth: 19 | Bright band width. 20 | radar_band: str 21 | Possible values are 'S', 'C', or 'X' 22 | 23 | Return 24 | ====== 25 | refp_ss: 26 | Stratiform reflectivity conversion from Ku-band to S-band 27 | refp_sh: 28 | Convective reflectivity conversion from Ku-band to S-band 29 | """ 30 | # Set coefficients for conversion from Ku-band to S-band 31 | # Rain 90% 80% 70% 60% 50% 40% 30% 20% 10% Snow 32 | as0=[ 4.78e-2, 4.12e-2, 8.12e-2, 1.59e-1, 2.87e-1, 4.93e-1, 8.16e-1, 1.31e+0, 2.01e+0, 2.82e+0, 1.74e-1] 33 | as1=[ 1.23e-2, 3.66e-3, 2.00e-3, 9.42e-4, 5.29e-4, 5.96e-4, 1.22e-3, 2.11e-3, 3.34e-3, 5.33e-3, 1.35e-2] 34 | as2=[-3.50e-4, 1.17e-3, 1.04e-3, 8.16e-4, 6.59e-4, 5.85e-4, 6.13e-4, 7.01e-4, 8.24e-4, 1.01e-3,-1.38e-3] 35 | as3=[-3.30e-5,-8.08e-5,-6.44e-5,-4.97e-5,-4.15e-5,-3.89e-5,-4.15e-5,-4.58e-5,-5.06e-5,-5.78e-5, 4.74e-5] 36 | as4=[ 4.27e-7, 9.25e-7, 7.41e-7, 6.13e-7, 5.80e-7, 6.16e-7, 7.12e-7, 8.22e-7, 9.39e-7, 1.10e-6, 0] 37 | # Rain 90% 80% 70% 60% 50% 40% 30% 20% 10% Hail 38 | ah0=[ 4.78e-2, 1.80e-1, 1.95e-1, 1.88e-1, 2.36e-1, 2.70e-1, 2.98e-1, 2.85e-1, 1.75e-1, 4.30e-2, 8.80e-2] 39 | ah1=[ 1.23e-2,-3.73e-2,-3.83e-2,-3.29e-2,-3.46e-2,-2.94e-2,-2.10e-2,-9.96e-3,-8.05e-3,-8.27e-3, 5.39e-2] 40 | ah2=[-3.50e-4, 4.08e-3, 4.14e-3, 3.75e-3, 3.71e-3, 3.22e-3, 2.44e-3, 1.45e-3, 1.21e-3, 1.66e-3,-2.99e-4] 41 | ah3=[-3.30e-5,-1.59e-4,-1.54e-4,-1.39e-4,-1.30e-4,-1.12e-4,-8.56e-5,-5.33e-5,-4.66e-5,-7.19e-5, 1.90e-5] 42 | ah4=[ 4.27e-7, 1.59e-6, 1.51e-6, 1.37e-6, 1.29e-6, 1.15e-6, 9.40e-7, 6.71e-7, 6.33e-7, 9.52e-7, 0] 43 | 44 | refp_ss = np.zeros(refp.shape) + np.NaN # snow 45 | refp_sh = np.zeros(refp.shape) + np.NaN # hail 46 | zmlt = zbb + bbwidth / 2. # APPROXIMATION! 47 | zmlb = zbb - bbwidth / 2. # APPROXIMATION! 48 | ratio = (zp - zmlb) / (zmlt - zmlb) 49 | 50 | iax, iay = np.where(ratio >= 1) 51 | # above melting layer 52 | if len(iax) > 0: 53 | dfrs = as0[10] + as1[10] * refp[iax, iay] + as2[10] * refp[iax, iay]**2 + as3[10] * refp[iax, iay]**3 + as4[10] * refp[iax, iay]**4 54 | dfrh = ah0[10] + ah1[10] * refp[iax, iay] + ah2[10] * refp[iax, iay]**2 + ah3[10] * refp[iax, iay]**3 + ah4[10] * refp[iax, iay]**4 55 | refp_ss[iax, iay] = refp[iax, iay] + dfrs 56 | refp_sh[iax, iay] = refp[iax, iay] + dfrh 57 | 58 | ibx, iby = np.where(ratio <= 0) 59 | if len(ibx) > 0: # below the melting layer 60 | dfrs = as0[0] + as1[0] * refp[ibx, iby] + as2[0] * refp[ibx, iby]**2 + as3[0] * refp[ibx, iby]**3 + as4[0] * refp[ibx, iby]**4 61 | dfrh = ah0[0] + ah1[0] * refp[ibx, iby] + ah2[0] * refp[ibx, iby]**2 + ah3[0] * refp[ibx, iby]**3 + ah4[0] * refp[ibx, iby]**4 62 | refp_ss[ibx, iby] = refp[ibx, iby] + dfrs 63 | refp_sh[ibx, iby] = refp[ibx, iby] + dfrh 64 | 65 | imx, imy = np.where((ratio > 0) & (ratio < 1)) 66 | if len(imx) > 0: # within the melting layer 67 | ind = np.round(ratio[imx, imy]).astype(int)[0] 68 | dfrs = as0[ind] + as1[ind] * refp[imx, imy] + as2[ind] * refp[imx, imy]**2 + as3[ind] * refp[imx, imy]**3 + as4[ind] * refp[imx, imy]**4 69 | dfrh = ah0[ind] + ah1[ind] * refp[imx, imy] + ah2[ind] * refp[imx, imy]**2 + ah3[ind] * refp[imx, imy]**3 + ah4[ind] * refp[imx, imy]**4 70 | refp_ss[imx, imy] = refp[imx, imy] + dfrs 71 | refp_sh[imx, imy] = refp[imx, imy] + dfrh 72 | 73 | # Jackson Tan's fix for C-band 74 | if radar_band == 'C': 75 | deltas = 5.3 / 10.0 * (refp_ss - refp) 76 | refp_ss = refp + deltas 77 | deltah = 5.3 / 10.0 * (refp_sh - refp) 78 | refp_sh = refp + deltah 79 | elif radar_band == 'X': 80 | deltas = 3.2 / 10.0 * (refp_ss - refp) 81 | refp_ss = refp + deltas 82 | deltah = 3.2 / 10.0 * (refp_sh - refp) 83 | refp_sh = refp + deltah 84 | 85 | return refp_ss, refp_sh 86 | 87 | 88 | def convert_gpmrefl_grband_dfr(refp, radar_band=None): 89 | ''' 90 | Convert GPM reflectivity to ground radar band using the DFR relationship 91 | found in Louf et al. (2019) paper. 92 | NOTE: It's a continuous relationship, so here there is no difference here 93 | between conv/strat. 94 | Parameters 95 | ========== 96 | refp: 97 | Satellite reflectivity field. 98 | radar_band: str 99 | Possible values are 'S', 'C', or 'X' 100 | Return 101 | ====== 102 | refl_strat: 103 | Stratiform reflectivity conversion from Ku-band to S-band 104 | refl_conv: 105 | Convective reflectivity conversion from Ku-band to S-band 106 | ''' 107 | if radar_band == 'S': 108 | cof = np.array([ 2.01236803e-07, -6.50694273e-06, 1.10885533e-03, -6.47985914e-02, -7.46518423e-02]) 109 | dfr = np.poly1d(cof) 110 | elif radar_band == 'C': 111 | cof = np.array([ 1.21547932e-06, -1.23266138e-04, 6.38562875e-03, -1.52248868e-01, 5.33556919e-01]) 112 | dfr = np.poly1d(cof) 113 | elif radar_band == 'X': 114 | # Use of C band DFR relationship multiply by ratio 115 | cof = np.array([ 1.21547932e-06, -1.23266138e-04, 6.38562875e-03, -1.52248868e-01, 5.33556919e-01]) 116 | dfr = 3.2 / 5.5 * np.poly1d(cof) 117 | else: 118 | raise ValueError(f'Radar reflectivity band ({radar_band}) not supported.') 119 | 120 | # It's a continuous relationship, so here there is no difference between conv/strat. 121 | refl_strat = refp + dfr(refp) 122 | refl_conv = refl_strat 123 | 124 | return refl_strat, refl_conv 125 | 126 | 127 | def convert_to_Ku(refg, zg, zbb, radar_band='S'): 128 | ''' 129 | From Liao and Meneghini (2009) 130 | 131 | Parameters 132 | ========== 133 | refg: 134 | Ground radar reflectivity field. 135 | zg: 136 | Altitude. 137 | zbb: 138 | Bright band height. 139 | bbwidth: 140 | Bright band width. 141 | radar_band: str 142 | Possible values are 'S', 'C', or 'X' 143 | 144 | Returns 145 | ======= 146 | refg_ku: 147 | Ground radar reflectivity field converted to Ku-band. 148 | ''' 149 | 150 | refg_ku = np.zeros(refg.shape) + np.NaN 151 | iax, iay, iaz = np.where(zg >= zbb) 152 | 153 | # Above bright band 154 | if len(iax) > 0: 155 | refg_ku[iax, iay, iaz] = 0.185074 + 1.01378 * refg[iax, iay, iaz] - 0.00189212 * refg[iax, iay, iaz]**2 156 | 157 | # Below bright band 158 | ibx, iby, ibz = np.where(zg < zbb) 159 | if len(ibx) > 0: 160 | refg_ku[ibx, iby, ibz] = -1.50393 + 1.07274 * refg[ibx, iby, ibz] + 0.000165393 * refg[ibx, iby, ibz]**2 161 | 162 | # Jackson Tan's fix for C-band 163 | if radar_band == 'C': 164 | delta = (refg_ku - refg) * 5.3 / 10.0 165 | refg_ku = refg + delta 166 | elif radar_band == 'X': 167 | delta = (refg_ku - refg) * 3.2 / 10.0 168 | refg_ku = refg + delta 169 | 170 | return refg_ku 171 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | arm-pyart>=1.8.0 2 | crayons>=0.1.2 3 | h5py>=2.6.0 4 | netCDF4>=1.2.7 5 | numpy>=1.11.1 6 | pandas>=0.19.2 7 | pyproj>=1.9.5.1 8 | scipy>=0.18 9 | -------------------------------------------------------------------------------- /scripts/find_good_case.py: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env python 2 | """ 3 | MSGR Matching Satellite and Ground Radar 4 | ======================================== 5 | 6 | @author: Valentin Louf 7 | @date: 2016-12-06 (creation) 2017-10-05 (current version) 8 | @email: valentin.louf@bom.gov.au 9 | @company: Monash University/Bureau of Meteorology 10 | """ 11 | # Standard library import 12 | import os 13 | import re 14 | import glob 15 | import time 16 | import argparse 17 | import datetime 18 | import warnings 19 | import traceback 20 | import configparser 21 | 22 | from multiprocessing import Pool 23 | 24 | # Others lib. 25 | import numpy as np 26 | import pandas as pd 27 | 28 | # Custom lib. 29 | from msgr import cross_validation 30 | from msgr.utils.misc import * # bunch of useful functions 31 | from msgr.io.save_data import save_data 32 | 33 | 34 | def get_orbit_number(infile): 35 | """ 36 | Look for an orbit number in the given filename. 37 | 38 | Parameters: 39 | =========== 40 | infile: str 41 | Input file. 42 | 43 | Returns: 44 | ======== 45 | orbit: str 46 | Supposed orbit number 47 | """ 48 | orbit = re.findall("[0-9]{6}", infile)[-1] # Get orbit number 49 | return orbit 50 | 51 | 52 | def get_satfile_list(satdir, date, l_gpm): 53 | """ 54 | Get a list of satellite files. 55 | 56 | Parameters: 57 | =========== 58 | satdir: str 59 | Path to satellite data directory. 60 | date: str 61 | Date, format YYYYMMDD 62 | l_gpm: bool 63 | Is this GPM or TRMM? 64 | 65 | Returns: 66 | ======== 67 | satfiles: str 68 | List of GPM files or TRMM 2A23 files. 69 | satfiles2: str 70 | List of TRMM 2A23 files (None for GPM). 71 | """ 72 | # Looking for satellite data files. 73 | if l_gpm: 74 | satfiles = glob.glob(satdir + '/*' + date + '*.HDF5') 75 | satfiles2 = None 76 | else: 77 | satfiles = glob.glob(satdir + '/*2A23*' + date + '*.HDF') 78 | satfiles2 = glob.glob(satdir + '/*2A25*' + date + '*.HDF') 79 | 80 | # Checking if found any satellite data file. 81 | if len(satfiles) == 0: 82 | satfiles, satfiles2 = None, None 83 | 84 | return satfiles, satfiles2 85 | 86 | 87 | def production_line_manager(configuration_file, the_date, outdir, satdir, rid, 88 | gr_offset, l_cband=True, l_dbz=True, l_gpm=True, 89 | l_atten=True, l_write=True): 90 | """ 91 | Here we locate the satellite files, call the comparison function 92 | match_volumes, and send the results for saving. The real deal is the 93 | match_volumes from msgr.core.msgr module. 94 | 95 | Parameters 96 | ========== 97 | the_date: datetime 98 | The day for comparison. 99 | parameters_dict: dict 100 | Dictionnary containing all parameters from the configuration file. 101 | """ 102 | print("") 103 | date = the_date.strftime("%Y%m%d") 104 | 105 | # Looking for satellites. 106 | satfiles, satfiles2 = get_satfile_list(satdir, date, l_gpm) 107 | if satfiles is None: 108 | print_red("No satellite swaths for %s." % (date)) 109 | return None 110 | 111 | if len(satfiles) > 5: 112 | print_red("There are more than 5 files for {}. Something probably wrong in the files name. Skipping this date.".format(date)) 113 | return None 114 | 115 | # Looping over satellite file 116 | for one_sat_file in satfiles: 117 | # Start chrono 118 | tick = time.time() 119 | 120 | # Get orbit number 121 | orbit = get_orbit_number(one_sat_file) 122 | print_with_time("Orbit #{} -- {}.".format(orbit, date)) 123 | 124 | if not l_gpm: 125 | try: 126 | # Trying to find corresponding 2A25 TRMM file based on the orbit 127 | fd_25 = find_file_with_string(satfiles2, orbit) 128 | except IndexError: 129 | print_red("Found no matching 2A25 file for TRMM.") 130 | continue 131 | else: 132 | fd_25 = None 133 | 134 | # Calling processing function for TRMM 135 | match_vol = cross_validation.match_volumes(configuration_file, None, one_sat_file, 136 | sat_file_2A25_trmm=fd_25, dtime=the_date, l_cband=l_cband, 137 | l_dbz=l_dbz, l_gpm=l_gpm, l_atten=l_atten) 138 | 139 | return None 140 | 141 | 142 | def multiproc_manager(kwargs): 143 | """ 144 | Buffer function that handles Exceptions while running the multiprocessing. 145 | All the arguments are identical to the 146 | """ 147 | try: 148 | production_line_manager(*kwargs) 149 | except Exception: 150 | traceback.print_exc() 151 | pass 152 | 153 | return None 154 | 155 | 156 | def main(): 157 | """ 158 | Reading general informations from configuration file like ncpu, start_date, 159 | end_date, all the switches. Loop over dates, spawn multiprocessing, and call the 160 | production_line_manager. 161 | """ 162 | print_with_time("Loading configuration file.") 163 | # Reading configuration file 164 | config = configparser.ConfigParser() 165 | config.read(CONFIG_FILE) 166 | 167 | # General info. 168 | general = config['general'] 169 | ncpu = general.getint('ncpu') 170 | date1 = general.get('start_date') 171 | date2 = general.get('end_date') 172 | 173 | # Data path 174 | path = config['path'] 175 | raddir = path.get('ground_radar') 176 | satdir = path.get('satellite') 177 | outdir = path.get('output') 178 | 179 | # Check if dirs exist. 180 | if not os.path.isdir(raddir): 181 | print_red("Wrong radar directory in configuration file.") 182 | return None 183 | if not os.path.isdir(satdir): 184 | print_red("Wrong satellite directory in configuration file.") 185 | return None 186 | if not os.path.isdir(outdir): 187 | print_red("Wrong output directory in configuration file.") 188 | return None 189 | 190 | # Switch for writing out volume-matched data 191 | switch = config['switch'] 192 | l_write = switch.getboolean('write') 193 | l_cband = switch.getboolean('cband') # Switch for C-band GR 194 | l_dbz = switch.getboolean('dbz') # Switch for averaging in dBZ 195 | l_gpm = switch.getboolean('gpm') # Switch for GPM PR data 196 | l_atten = switch.getboolean('correct_gr_attenuation') 197 | 198 | # General info about the ground radar (ID and OFFSET to apply.) 199 | GR_param = config['radar'] 200 | rid = GR_param.get('radar_id') 201 | try: 202 | gr_offset = GR_param.getfloat('offset') 203 | except KeyError: 204 | gr_offset = 0 205 | 206 | start_date = datetime.datetime.strptime(date1, '%Y%m%d') 207 | end_date = datetime.datetime.strptime(date2, '%Y%m%d') 208 | 209 | print_yellow("Generating ground radar file list.") 210 | total_radar_file_list = get_files(raddir) 211 | print_yellow("Found {} supported radar files in {}.".format(len(total_radar_file_list), raddir)) 212 | 213 | date_list = pd.date_range(start_date, end_date) 214 | args_list = [None] * len(date_list) 215 | for cnt, onedate in enumerate(date_list): 216 | mydate = onedate.strftime("%Y%m%d") 217 | # radar_file_list = [f for f in total_radar_file_list if mydate in f] 218 | 219 | # Argument list for multiprocessing. 220 | args_list[cnt] = (CONFIG_FILE, onedate, outdir, satdir, rid, gr_offset, 221 | l_cband, l_dbz, l_gpm, l_atten, l_write) 222 | 223 | # Start multiprocessing. 224 | with Pool(ncpu) as pool: 225 | pool.map(multiproc_manager, args_list) 226 | 227 | return None 228 | 229 | 230 | if __name__ == '__main__': 231 | """ 232 | Global variables definition. 233 | 234 | Parameters: 235 | =========== 236 | CONFIG_FILE: str 237 | Configuration file (.ini) 238 | """ 239 | parser_description = "Start MSGR - volume Matching Satellite and Ground Radar." 240 | parser = argparse.ArgumentParser(description=parser_description) 241 | 242 | parser.add_argument('-c', '--config', type=str, dest='config_file', help='Path to configuration file.', default=None, required=True) 243 | 244 | args = parser.parse_args() 245 | CONFIG_FILE = args.config_file 246 | 247 | warnings.simplefilter("ignore") 248 | main() 249 | -------------------------------------------------------------------------------- /scripts/generate_config_matchvol.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import argparse 3 | 4 | 5 | def get_gnrl_path(season): 6 | '''GET_GNRL_PATH''' 7 | '''season: season of treatment''' 8 | '''return the general path for data directory on RAIJIN''' 9 | 10 | custom_path = '/g/data2/rr5/vhl548/season_' + season + '/' 11 | custom_path_bis_1213 = '/g/data2/rr5/vhl548/season_1213_cfradial/' 12 | uf_path = '/g/data2/rr5/arm/data/cpol_UF/' + season + '/ppi/' 13 | lassen_path = '/g/data2/rr5/arm/data/cpol_lassen/cpol_' + season + '/PPI/' 14 | 15 | the_path = {"9899": {'path': lassen_path, 'subdir': False}, 16 | "9900": {'path': lassen_path, 'subdir': False}, 17 | "0102": {'path': lassen_path, 'subdir': False}, 18 | "0203": {'path': lassen_path, 'subdir': False}, 19 | "0304": {'path': uf_path, 'subdir': False}, 20 | "0405": {'path': uf_path, 'subdir': False}, 21 | "0506": {'path': uf_path, 'subdir': True}, 22 | "0607": {'path': uf_path, 'subdir': True}, 23 | "0910": {'path': custom_path, 'subdir': True}, 24 | "1011": {'path': custom_path, 'subdir': True}, 25 | "1112": {'path': custom_path, 'subdir': True}, 26 | "1213": {'path': custom_path_bis_1213, 'subdir': True}, 27 | "1314": {'path': custom_path, 'subdir': True}, 28 | "1415": {'path': custom_path, 'subdir': True}, 29 | "1516": {'path': custom_path, 'subdir': False} 30 | } 31 | 32 | return the_path[season] # type: Dict[str, bool] 33 | 34 | 35 | def get_date(season_str): 36 | ''' 37 | GET_YEAR 38 | Does not work for season BEFORE 1998 or AFTER 2020 (just add dates) 39 | 40 | Parameter 41 | ========== 42 | season_str: str 43 | string of season, e.g. "1213" for 2012/2013 44 | 45 | Returns 46 | ======= 47 | season_start: str 48 | Beginning of the season 49 | season_end: str 50 | End of the season 51 | ''' 52 | 53 | the_years = {"9899": ['19981206', '19990507'], 54 | "9900": ['19991104', '20000403'], 55 | "0102": ['20011026', '20020407'], 56 | "0203": ['20021029', '20030714'], 57 | "0304": ['20031020', '20040504'], 58 | "0405": ['20041201', '20050228'], 59 | "0506": ['20051110', '20060430'], 60 | "0607": ['20061012', '20070419'], 61 | "0910": ['20091124', '20100430'], 62 | "1011": ['20101105', '20110329'], 63 | "1112": ['20111120', '20120420'], 64 | "1213": ['20121109', '20130506'], 65 | "1314": ['20131011', '20140504'], 66 | "1415": ['20141203', '20150602'], 67 | "1516": ['20151006', '20160215'] 68 | } 69 | 70 | return the_years[season_str] # type: List[int, int] 71 | 72 | 73 | def write_example(): 74 | 75 | ex = """[general] 76 | # Number of CPU for multiprocessing 77 | # start and end date in YYYYMMDD format 78 | ncpu = 8 79 | start_date = 20121109 80 | end_date = 20130506 81 | 82 | [path] 83 | ground_radar = /path/to/ground/radar/data/ 84 | satellite = /path/to/satellite/radar/data/ 85 | output = /path/to/ouptut/directory/ 86 | log = /path/to/log/directory/ 87 | 88 | [radar] 89 | # rmin is the minimum radar range (in meter) we start looking for data. 90 | # must be at least 15000m. rmax is the radar maximum range (in m). 91 | # offset is the offset, in dB, by which we want to correct the reflectivity. 92 | # Units in meters and degrees 93 | radar_name = MYSUPERRADAR 94 | radar_id = MYSUPERRADAR_ID1 95 | band = C 96 | rmin = 15000 97 | rmax = 150000 98 | longitude = 12.8 99 | latitude = 48.0 100 | altitude = 42 101 | beamwidth = 1.0 102 | offset = 0 103 | sat_offset = 0 104 | 105 | [thresholds] 106 | # Threshold on satellite reflectivity 107 | # Minimum number of pair 108 | # Minimum number of satellite profiles 109 | # Maximum time diffenrece between radar and satellite, in seconds 110 | # Threshold on ground radar reflectivity 111 | min_sat_reflec = 17 112 | min_pair = 10 113 | min_profiles = 10 114 | max_time_delta = 300 115 | min_gr_reflec = 17 116 | 117 | [switch] 118 | # Case insenstive, can be yes/no, y/n, true/false, 1/0 119 | # Using dBZ or natural units for the statistical calculations 120 | # Satellite is GPM (false for TRMM) 121 | # Ground radar is C-Band (false for S-Band) 122 | # Writing results in output directory 123 | # Correct ground radar attenuation using pyart 124 | dbz = False 125 | gpm = False 126 | write = True 127 | correct_gr_attenuation = False 128 | """ 129 | 130 | with open('matchvol_configuration.ini', 'w') as fid: 131 | fid.write(ex) 132 | 133 | print("Example configuration file written in: matchvol_configuration.ini") 134 | print("Once you have modified the configuration file, type: ") 135 | print("matchvol.py -c matchvol_configuration.ini") 136 | return None 137 | 138 | 139 | def main(): 140 | 141 | st_date, end_date = get_date(season) 142 | gr_path = get_gnrl_path(season)['path'] 143 | 144 | pfix = '''[general] 145 | # Number of CPU for multiprocessing 146 | # start and end date in YYYYMMDD format 147 | ncpu = %i 148 | start_date = %s 149 | end_date = %s 150 | 151 | [path] 152 | ground_radar = %s 153 | satellite = /g/data2/rr5/vhl548/TRMM 154 | output = /home/548/vhl548/data/ 155 | log = /home/548/vhl548/log/ 156 | 157 | ''' % (ncpu, st_date, end_date, gr_path) 158 | 159 | sfix = '''[radar] 160 | # rmin is the minimum radar range we start looking for data 161 | # Units in meters and degrees 162 | radar_name = CPOL 163 | radar_id = IDR59 164 | band = C 165 | rmin = 15000 166 | rmax = 150000 167 | longitude = 131.04530334 168 | latitude = -12.24880028 169 | altitude = 42 170 | beamwidth = 1.0 171 | offset = 0 172 | 173 | [satellite] 174 | # Satellite reflectivity offset. 175 | sat_offset = 0 176 | 177 | [thresholds] 178 | # Threshold on satellite reflectivity 179 | # Minimum number of pair 180 | # Minimum number of satellite profiles 181 | # Maximum time diffenrece between radar and satellite, in seconds 182 | # Threshold on ground radar reflectivity 183 | min_sat_reflec = 17 184 | min_pair = 10 185 | min_profiles = 10 186 | max_time_delta = 300 187 | min_gr_reflec = 17 188 | 189 | [switch] 190 | # Case insenstive, can be yes/no, y/n, true/false, 1/0 191 | # Using dBZ or natural units for the statistical calculations 192 | # Satellite is GPM (false for TRMM) 193 | # Ground radar is C-Band (false for S-Band) 194 | # Writing results in output directory 195 | # Correct ground radar attenuation using pyart 196 | dbz = False 197 | gpm = False 198 | write = True 199 | correct_gr_attenuation = True 200 | intermediary = False 201 | ''' 202 | 203 | with open(output_fname, 'w') as fid: 204 | fid.write(pfix) 205 | fid.write(sfix) 206 | 207 | return None 208 | 209 | 210 | if __name__ == '__main__': 211 | welcome_msg = "Automatic creation of the configuration file for MSGR for CPOL data on NCI RAIJIN (only)." 212 | 213 | parser = argparse.ArgumentParser(description=welcome_msg) 214 | parser.add_argument('-j', '--cpu', dest='ncpu', default=16, type=int, help='Number of process') 215 | parser.add_argument('-o', '--output', dest='output', default=None, type=str, help='Name of the output file.') 216 | parser.add_argument('-s', '--season', dest='season', default=None, type=str, help='Season to parse.') 217 | 218 | feature_parser = parser.add_mutually_exclusive_group(required=False) 219 | feature_parser.add_argument('--example', dest='example', action='store_true', help='Create example file.') 220 | feature_parser.add_argument('--no-example', dest='example', action='store_false', help="Don't create example file.") 221 | parser.set_defaults(example=False) 222 | 223 | args = parser.parse_args() 224 | ncpu = args.ncpu 225 | season = args.season 226 | example = args.example 227 | output_fname = args.output 228 | 229 | if output_fname is None: 230 | output_fname = "config.ini" 231 | 232 | if ".ini" not in output_fname: 233 | output_fname += ".ini" 234 | 235 | # No season provided and this is not an example. 236 | if season is None: 237 | if not example: 238 | print("You need to provide an argument, type `generate_config_matchvol -h` for help.") 239 | sys.exit() 240 | 241 | if example: 242 | write_example() 243 | else: 244 | main() 245 | 246 | # End of __main__ 247 | -------------------------------------------------------------------------------- /scripts/matchvol.py: -------------------------------------------------------------------------------- 1 | """ 2 | MSGR Matching Satellite and Ground Radar 3 | ======================================== 4 | 5 | @author: Valentin Louf 6 | @date: 2016-12-06 (creation) 2018-04-18 (current version) 7 | @email: valentin.louf@bom.gov.au 8 | @company: Monash University/Bureau of Meteorology 9 | """ 10 | # Standard library import 11 | import os 12 | import glob 13 | import time 14 | import argparse 15 | import datetime 16 | import warnings 17 | import traceback 18 | import configparser 19 | import zipfile 20 | import tempfile 21 | 22 | 23 | from multiprocessing import Pool 24 | 25 | # Others lib. 26 | import numpy as np 27 | 28 | # Custom lib 29 | from msgr import cross_validation 30 | from msgr.io.read_gpm import read_date_from_GPM 31 | from msgr.io.read_trmm import read_date_from_TRMM 32 | from msgr.io.save_data import save_data 33 | from msgr.utils.misc import * 34 | 35 | 36 | def compute_offset(matchvol_data): 37 | z = matchvol_data['z'] 38 | ref1 = matchvol_data['ref1'] 39 | ref5 = matchvol_data['ref5'] 40 | stdv1 = matchvol_data['stdv1'] 41 | stdv2 = matchvol_data['stdv2'] 42 | 43 | pos = (z > 4e3) | (ref1 >= 36) | (stdv1 > 4) | (stdv2 > 4) | (ref5 >= 36) | (ref1 == 0) | (ref5 < 21) 44 | ref1[pos] = np.NaN 45 | 46 | dref_ku = (ref5 - ref1) 47 | dref_ku = dref_ku[~np.isnan(dref_ku)] 48 | 49 | if len(dref_ku) <= 1: 50 | return np.NaN 51 | else: 52 | offset = - np.median(dref_ku) # !!! Note the MINUS sign !!! 53 | return offset 54 | 55 | 56 | def get_orbit_number(infile): 57 | """ 58 | Look for an orbit number in the given filename. 59 | 60 | Parameters: 61 | =========== 62 | infile: str 63 | Input file. 64 | 65 | Returns: 66 | ======== 67 | orbit: str 68 | Supposed orbit number 69 | """ 70 | orbit = re.findall("[0-9]{6}", infile)[-1] # Get orbit number 71 | return orbit 72 | 73 | 74 | def get_satfile_list(satdir, date, l_gpm): 75 | """ 76 | Get a list of satellite files. 77 | 78 | Parameters: 79 | =========== 80 | satdir: str 81 | Path to satellite data directory. 82 | date: str 83 | Date, format YYYYMMDD 84 | l_gpm: bool 85 | Is this GPM or TRMM? 86 | 87 | Returns: 88 | ======== 89 | satfiles: str 90 | List of GPM files or TRMM 2A23 files. 91 | satfiles2: str 92 | List of TRMM 2A23 files (None for GPM). 93 | """ 94 | # Looking for satellite data files. 95 | satfiles = glob.glob(os.path.join(satdir, '**', f'*{date}*.HDF5'), recursive=True) 96 | satfiles2 = None 97 | if len(satfiles) == 0: 98 | # Old version of TRMM products, HDF format. 99 | satfiles = sorted(glob.glob(os.path.join(satdir, f'*2A23*{date}*.HDF'))) 100 | satfiles2 = sorted(glob.glob(os.path.join(satdir, f'*2A25*{date}*.HDF'))) 101 | 102 | # Checking if found any satellite data file. 103 | if len(satfiles) == 0: 104 | satfiles, satfiles2 = None, None 105 | 106 | return satfiles, satfiles2 107 | 108 | 109 | def check_directory(radar_dir, satellite_dir, output_dir, logdir): 110 | # Check if dirs exist. 111 | if not os.path.isdir(radar_dir): 112 | raise FileNotFoundError('Ground radar data directory not found.') 113 | if not os.path.isdir(satellite_dir): 114 | raise FileNotFoundError('Satellite data directory not found.') 115 | 116 | try: 117 | os.mkdir(output_dir) 118 | except FileExistsError: 119 | pass 120 | 121 | try: 122 | os.mkdir(logdir) 123 | except FileExistsError: 124 | pass 125 | 126 | return None 127 | 128 | 129 | def multiprocessing_driver(CONFIG_FILE, ground_radar_file, one_sat_file, sat_file_2A25_trmm, 130 | satellite_dtime, radar_band, l_dbz, l_atten, gr_offset, 131 | l_write, rid, orbit, outdir, logdir): 132 | """ 133 | Buffer function that handles Exceptions while running the multiprocessing. 134 | Automatically runs the comparison 2 times. First with the raw radar data, 135 | then it computes the offset between ground radars and satellites and runs 136 | a second time with that offset. 137 | """ 138 | datestr = satellite_dtime.strftime('%Y%m%d') 139 | 140 | for pass_number in range(2): 141 | try: 142 | # Calling processing function for TRMM 143 | tick = time.time() 144 | match_vol = cross_validation.match_volumes(CONFIG_FILE, ground_radar_file, one_sat_file, sat_file_2A25_trmm, 145 | satellite_dtime, radar_band, l_dbz, l_atten, gr_offset, logdir) 146 | except Exception: 147 | traceback.print_exc() 148 | return None 149 | 150 | print_with_time("Comparison done in %.2fs." % (time.time() - tick)) 151 | if match_vol is None: 152 | print_red('The comparison returned nothing.') 153 | return None 154 | 155 | delta_zh = compute_offset(match_vol) 156 | # Saving data 157 | if l_write: 158 | # Output file name. 159 | outfilename = "RID_{}_ORBIT_{}_DATE_{}_PASS_{}.nc".format(rid, orbit, datestr, pass_number + 1) 160 | outfilename = os.path.join(outdir, outfilename) 161 | print_green("Saving data to {}.".format(outfilename), bold=True) 162 | if pass_number == 0: 163 | save_data(outfilename, match_vol, satellite_dtime, offset1=gr_offset, nb_pass=pass_number) 164 | else: 165 | save_data(outfilename, match_vol, satellite_dtime, offset1=gr_offset, 166 | offset2=delta_zh, nb_pass=pass_number) 167 | 168 | gr_offset = delta_zh 169 | 170 | if np.abs(gr_offset) < 1: 171 | print_green(f"No significant difference between ground radar and satellite" + 172 | f" found for {datestr}. Not doing anymore pass.") 173 | break 174 | elif np.isnan(gr_offset): 175 | print_red(f"Invalid offset found. Stopping comparison for {datestr}.") 176 | return None 177 | elif pass_number == 0: 178 | print_magenta(f"The difference between the ground radar data and the" + 179 | f" satellite data for {datestr} is of {gr_offset:0.2} dB.") 180 | 181 | return None 182 | 183 | 184 | def main(): 185 | print_with_time("Loading configuration file.") 186 | # Reading configuration file 187 | config = configparser.ConfigParser() 188 | config.read(CONFIG_FILE) 189 | 190 | # General info. 191 | general = config['general'] 192 | ncpu = general.getint('ncpu') 193 | date1 = general.get('start_date') 194 | date2 = general.get('end_date') 195 | 196 | # Data path 197 | path = config['path'] 198 | radar_dir = path.get('ground_radar') 199 | satellite_dir = path.get('satellite') 200 | outdir = path.get('output') 201 | logdir = path.get('log') 202 | 203 | # Thresholds 204 | thresholds = config['thresholds'] 205 | max_time_delta = thresholds.getfloat('max_time_delta') # in second 206 | 207 | # General info about the ground radar (ID and OFFSET to apply.) 208 | # Radar location too. 209 | GR_param = config['radar'] 210 | try: 211 | radar_band = GR_param.get("band") 212 | except KeyError: 213 | raise KeyError("Please use 'band' in [radar] section of the configuration file instead of cband in switches." + 214 | " The possible values for band are S, C, or X.") 215 | rid = GR_param.get('radar_id') 216 | radar_lat = GR_param.getfloat('latitude') 217 | radar_lon = GR_param.getfloat('longitude') 218 | rmax = GR_param.getfloat('rmax') 219 | try: 220 | gr_offset = GR_param.getfloat('offset') 221 | except KeyError: 222 | gr_offset = 0 223 | 224 | if radar_band == "C": 225 | print_yellow("You say that the ground radar is C-band") 226 | elif radar_band == "S": 227 | print_yellow("You say that the ground radar is S-band") 228 | elif radar_band == "X": 229 | print_yellow("You say that the ground radar is X-band") 230 | print_yellow("Reflectivity conversion to X-band not yet supported.") 231 | else: 232 | print_red(f"Ground radar frequency band unknown. You said {radar_band}. " + 233 | "The supported values are 'S', 'C', and 'X'. Doing nothing.") 234 | return None 235 | 236 | # Switches. 237 | switch = config['switch'] 238 | l_write = switch.getboolean('write') # switch to save data. 239 | l_dbz = switch.getboolean('dbz') # Switch for averaging in dBZ 240 | l_gpm = switch.getboolean('gpm') # Switch for GPM PR data 241 | l_atten = switch.getboolean('correct_gr_attenuation') 242 | 243 | # Finish reading configuration file. 244 | check_directory(radar_dir, satellite_dir, outdir, logdir) 245 | 246 | start_date = datetime.datetime.strptime(date1, '%Y%m%d') 247 | end_date = datetime.datetime.strptime(date2, '%Y%m%d') 248 | nbdays = (end_date - start_date).days 249 | date_list = [start_date + datetime.timedelta(days=d) for d in range(nbdays)] 250 | 251 | print_yellow("Generating ground radar file list.") 252 | total_radar_file_list = get_files(radar_dir) 253 | print_yellow(f"Found {len(total_radar_file_list)} supported radar files in {radar_dir}.") 254 | 255 | print_green("Building database.") 256 | args_list = [] 257 | for date in date_list: 258 | datestr = date.strftime('%Y%m%d') 259 | 260 | # Extracting radar file list for this date from the total radar file list. 261 | 262 | radar_file_list = [f for f in total_radar_file_list if datestr in f] 263 | 264 | if len(radar_file_list) == 0: 265 | print_yellow(f"No ground radar file found for this date {datestr}") 266 | continue 267 | # else: 268 | # print_green(f"Found {len(radar_file_list)} radar files for date {datestr}") 269 | 270 | #Extract file listing from zip if required (rq0 specific) 271 | if '.zip' in radar_file_list[0]: 272 | #declare zip ffn 273 | zip_ffn = radar_file_list[0] 274 | #create temp_path for use later when extracting volumes from the zip 275 | unzip_temp_path = tempfile.mkdtemp() 276 | #extract name list 277 | zip = zipfile.ZipFile(zip_ffn) 278 | radar_file_list = zip.namelist() 279 | zip.close() 280 | else: 281 | zip_ffn = None 282 | 283 | # Looking for satellite data corresponding to this date. 284 | satfiles, satfiles2 = get_satfile_list(satellite_dir, datestr, l_gpm) 285 | if satfiles is None: 286 | print_red(f"No satellite data for {datestr}.") 287 | continue 288 | 289 | # Obtaining the satellite file(s) and reading its exact date and time. 290 | for cnt, one_sat_file in enumerate(satfiles): 291 | if satfiles2 is not None: 292 | # Old version of TRMM 293 | sat_file_2A25_trmm = satfiles2[cnt] 294 | satellite_dtime, satellite_dist = read_date_from_TRMM(one_sat_file, radar_lat, radar_lon) 295 | else: 296 | # GPM or new version of TRMM 297 | sat_file_2A25_trmm = None 298 | satellite_dtime, satellite_dist = read_date_from_GPM(one_sat_file, radar_lat, radar_lon) 299 | 300 | # check satellite dist 301 | if satellite_dist > rmax: 302 | print_red("Ground radar position not inside satellite swath.") 303 | continue 304 | 305 | orbit = get_orbit_number(one_sat_file) 306 | 307 | # Get the datetime for each radar files 308 | radar_dtime = [get_time_from_filename(radfile, datestr) for radfile in radar_file_list] 309 | radar_dtime = list(filter(None, radar_dtime)) # Removing None values 310 | 311 | closest_dtime_rad = get_closest_date(radar_dtime, satellite_dtime) 312 | time_difference = np.abs(satellite_dtime - closest_dtime_rad) 313 | if time_difference.seconds > max_time_delta: 314 | print_red(f"Found {len(radar_file_list)} ground radar files for date {datestr}. " + 315 | f'Smallest time difference is {time_difference.seconds}s while the' + 316 | f' maximum time difference allowed is {max_time_delta}s.', bold=True) 317 | continue 318 | 319 | # Radar file corresponding to the nearest scan time 320 | ground_radar_file = get_filename_from_date(radar_file_list, closest_dtime_rad) 321 | 322 | # if processing radar data from rq0, ground_radar_file will need to be extracted from the zip 323 | if zip_ffn: 324 | #extract to temp dir 325 | zip = zipfile.ZipFile(zip_ffn) 326 | zip.extract(ground_radar_file, path=unzip_temp_path) 327 | zip.close() 328 | #generate path to temp path 329 | ground_radar_file = '/'.join([unzip_temp_path, ground_radar_file]) 330 | 331 | # Argument list for multiprocessing. 332 | args_list.append((CONFIG_FILE, ground_radar_file, one_sat_file, 333 | sat_file_2A25_trmm, satellite_dtime, radar_band, 334 | l_dbz, l_atten, gr_offset, l_write, rid, orbit, outdir, logdir)) 335 | 336 | if len(args_list) == 0: 337 | print_red("Nothing to do. Is the configuration file correct?") 338 | return None 339 | 340 | print_green('Database built. Start processing.') 341 | # Start multiprocessing. 342 | with Pool(ncpu) as pool: 343 | pool.starmap(multiprocessing_driver, args_list) 344 | 345 | #remove zip temp path 346 | if zip_ffn: 347 | os.system('rm -rf ' + unzip_temp_path) 348 | 349 | print_green('MSGR processing complete.') 350 | 351 | return None 352 | 353 | 354 | if __name__ == '__main__': 355 | """ 356 | Global variables definition. 357 | 358 | Parameters: 359 | =========== 360 | CONFIG_FILE: str 361 | Configuration file (.ini) 362 | """ 363 | parser_description = "Start MSGR - volume Matching Satellite and Ground Radar." 364 | parser = argparse.ArgumentParser(description=parser_description) 365 | 366 | parser.add_argument('-c', '--config', type=str, dest='config_file', 367 | help='Configuration file.', default=None, required=True) 368 | 369 | args = parser.parse_args() 370 | CONFIG_FILE = args.config_file 371 | 372 | warnings.simplefilter("ignore") 373 | main() 374 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | 4 | # Note: To use the 'upload' functionality of this file, you must: 5 | # $ pip install twine 6 | 7 | import io 8 | import os 9 | import sys 10 | from shutil import rmtree 11 | 12 | from setuptools import find_packages, setup, Command 13 | 14 | # Package meta-data. 15 | NAME = 'msgr' 16 | DESCRIPTION = 'Match Satellite and Ground Radar (MSGR). Cross-validation of reflectivity data from ground radar with TRMM and/or GPM' 17 | URL = 'https://gitlab.bom.gov.au/vlouf/Matchproj-python' 18 | EMAIL = 'valentin.louf@bom.gov.au' 19 | AUTHOR = 'Valentin Louf' 20 | 21 | # What packages are required for this module to be executed? 22 | REQUIRED = [ 23 | "arm_pyart", "numpy", "crayons", "netCDF4", "h5py", "pyproj", 'xarray', 24 | ] 25 | 26 | # The rest you shouldn't have to touch too much :) 27 | # ------------------------------------------------ 28 | # Except, perhaps the License and Trove Classifiers! 29 | # If you do change the License, remember to change the Trove Classifier for that! 30 | 31 | here = os.path.abspath(os.path.dirname(__file__)) 32 | 33 | # Import the README and use it as the long-description. 34 | # Note: this will only work if 'README.rst' is present in your MANIFEST.in file! 35 | with io.open(os.path.join(here, 'README.md'), encoding='utf-8') as f: 36 | long_description = '\n' + f.read() 37 | 38 | # Load the package's __version__.py module as a dictionary. 39 | about = {} 40 | with open(os.path.join(here, NAME, '__version__.py')) as f: 41 | exec(f.read(), about) 42 | 43 | 44 | class PublishCommand(Command): 45 | """Support setup.py publish.""" 46 | 47 | description = 'Build and publish the package.' 48 | user_options = [] 49 | 50 | @staticmethod 51 | def status(s): 52 | """Prints things in bold.""" 53 | print('\033[1m{0}\033[0m'.format(s)) 54 | 55 | def initialize_options(self): 56 | pass 57 | 58 | def finalize_options(self): 59 | pass 60 | 61 | def run(self): 62 | try: 63 | self.status('Removing previous builds…') 64 | rmtree(os.path.join(here, 'dist')) 65 | except FileNotFoundError: 66 | pass 67 | 68 | self.status('Building Source and Wheel (universal) distribution…') 69 | os.system('{0} setup.py sdist bdist_wheel --universal'.format(sys.executable)) 70 | 71 | self.status('Uploading the package to PyPi via Twine…') 72 | os.system('twine upload dist/*') 73 | 74 | sys.exit() 75 | 76 | 77 | # Where the magic happens: 78 | setup( 79 | name=NAME, 80 | version=about['__version__'], 81 | description=DESCRIPTION, 82 | long_description=long_description, 83 | author=AUTHOR, 84 | author_email=EMAIL, 85 | url=URL, 86 | packages=find_packages(exclude=('tests', 'example')), 87 | # If your package is a single module, use this instead of 'packages': 88 | # py_modules=['mypackage'], 89 | 90 | # entry_points={ 91 | # 'console_scripts': ['mycli=mymodule:cli'], 92 | # }, 93 | install_requires=REQUIRED, 94 | include_package_data=True, 95 | license='ISC', 96 | classifiers=[ 97 | # Trove classifiers 98 | 'License :: OSI Approved :: MIT License', 99 | 'Programming Language :: Python', 100 | 'Programming Language :: Python :: 3.5', 101 | 'Programming Language :: Python :: 3.6', 102 | 'Programming Language :: Python :: 3.7', 103 | ], 104 | # $ setup.py publish support. 105 | cmdclass={ 106 | 'publish': PublishCommand, 107 | }, 108 | ) 109 | --------------------------------------------------------------------------------