├── README.md ├── ligo ├── GW150914_4_template.hdf5 ├── GW150914_ASDs.png ├── GW150914_H1_SNR.png ├── GW150914_H1_matchfreq.png ├── GW150914_H1_matchtime.png ├── GW150914_H1_shifted.wav ├── GW150914_H1_spectrogram.png ├── GW150914_H1_spectrogram_whitened.png ├── GW150914_H1_whitenbp.wav ├── GW150914_L1_SNR.png ├── GW150914_L1_matchfreq.png ├── GW150914_L1_matchtime.png ├── GW150914_L1_shifted.wav ├── GW150914_L1_spectrogram.png ├── GW150914_L1_spectrogram_whitened.png ├── GW150914_L1_whitenbp.wav ├── GW150914_strain.png ├── GW150914_template.png ├── GW150914_template_shifted.wav ├── GW150914_template_whiten.wav ├── GW151226_4_template.hdf5 ├── H-H1_LOSC_4_V1-1126259446-32.hdf5 ├── H-H1_LOSC_4_V1-1128678884-32.hdf5 ├── H-H1_LOSC_4_V1-1135136334-32.hdf5 ├── L-L1_LOSC_4_V1-1126259446-32.hdf5 ├── L-L1_LOSC_4_V1-1128678884-32.hdf5 ├── L-L1_LOSC_4_V1-1135136334-32.hdf5 ├── LVT151012_4_template.hdf5 ├── O1_events.json ├── gwave_pyconjp2016.ipynb ├── pyconfig1.png ├── pyconfig2.png ├── pyconfig3.png ├── pyconfig4.png ├── readligo.py └── readligo.pyc ├── meteology └── ulmo_pyconjp2016.ipynb ├── pyconjp2016.pdf └── sesimology ├── obspy_pyconjp2016.ipynb ├── stations └── GR.NRDL.xml └── waveforms └── GR.NRDL..HHZ__20110311T054232Z__20110311T064732Z.mseed /README.md: -------------------------------------------------------------------------------- 1 | 2 | # PyConJP2016 3 | 4 | ## 内容 5 | 6 | クラウドサービス等で安価になった計算資源，大量にあるオープンなデータ，そして多様なオープンソースのツール群．これらを利用すれば，勤務外や週末に計算機上で科学的な解析・分析や推測や予測を自分の手で行うことができる時代になっています． 7 | 8 | 本トークでは他の言語に較べて鬼のように豊かなエコシステムをもつ，Pythonの科学計算スタックを屈指し，プライベートプロジェクトとしてサイエンスをすることについて，環境の構築方法，スタックの紹介，そして事例を提示します．計算環境を整え，基礎知識を蓄え，少しのやる気さえあれば，みんながサイエンスのプロジェクトを行うことができます． 9 | 10 | 普段の職業は「みんなちがって」も「みんな」が計算機上でサイエンスが可能になっている「いい」時代にいるのです．みんなちがってみんないい．Pythonの科学計算スタックはそれを可能にすることを本トークで実感することができたら幸いです． 11 | 12 | ## スライド 13 | - 週末サイエンティストのススメ: http://www.slideshare.net/yutakashino/pyconjp2016 14 | 15 | ## 事例：Jupyterノートブック 16 | - 事例1.気象データ可視化: https://github.com/bakfoo/pyconjp2016/blob/master/meteology/ulmo_pyconjp2016.ipynb 17 | - 事例2.地震波解析: https://github.com/bakfoo/pyconjp2016/blob/master/sesimology/obspy_pyconjp2016.ipynb 18 | - 事例3.重力波解析: https://github.com/bakfoo/pyconjp2016/blob/master/ligo/gwave_pyconjp2016.ipynb 19 | 20 | 21 | ## More information 22 | 23 | - Official site: http://www.bakfoo.com/ 24 | - Twitter (Japanese): https://twitter.com/yutakashino 25 | 26 | ## License 27 | 28 | MIT License 29 | -------------------------------------------------------------------------------- /ligo/GW150914_4_template.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_4_template.hdf5 -------------------------------------------------------------------------------- /ligo/GW150914_ASDs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_ASDs.png -------------------------------------------------------------------------------- /ligo/GW150914_H1_SNR.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_H1_SNR.png -------------------------------------------------------------------------------- /ligo/GW150914_H1_matchfreq.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_H1_matchfreq.png -------------------------------------------------------------------------------- /ligo/GW150914_H1_matchtime.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_H1_matchtime.png -------------------------------------------------------------------------------- /ligo/GW150914_H1_shifted.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_H1_shifted.wav -------------------------------------------------------------------------------- /ligo/GW150914_H1_spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_H1_spectrogram.png -------------------------------------------------------------------------------- /ligo/GW150914_H1_spectrogram_whitened.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_H1_spectrogram_whitened.png -------------------------------------------------------------------------------- /ligo/GW150914_H1_whitenbp.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_H1_whitenbp.wav -------------------------------------------------------------------------------- /ligo/GW150914_L1_SNR.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_L1_SNR.png -------------------------------------------------------------------------------- /ligo/GW150914_L1_matchfreq.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_L1_matchfreq.png -------------------------------------------------------------------------------- /ligo/GW150914_L1_matchtime.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_L1_matchtime.png -------------------------------------------------------------------------------- /ligo/GW150914_L1_shifted.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_L1_shifted.wav -------------------------------------------------------------------------------- /ligo/GW150914_L1_spectrogram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_L1_spectrogram.png -------------------------------------------------------------------------------- /ligo/GW150914_L1_spectrogram_whitened.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_L1_spectrogram_whitened.png -------------------------------------------------------------------------------- /ligo/GW150914_L1_whitenbp.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_L1_whitenbp.wav -------------------------------------------------------------------------------- /ligo/GW150914_strain.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_strain.png -------------------------------------------------------------------------------- /ligo/GW150914_template.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_template.png -------------------------------------------------------------------------------- /ligo/GW150914_template_shifted.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_template_shifted.wav -------------------------------------------------------------------------------- /ligo/GW150914_template_whiten.wav: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW150914_template_whiten.wav -------------------------------------------------------------------------------- /ligo/GW151226_4_template.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/GW151226_4_template.hdf5 -------------------------------------------------------------------------------- /ligo/H-H1_LOSC_4_V1-1126259446-32.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/H-H1_LOSC_4_V1-1126259446-32.hdf5 -------------------------------------------------------------------------------- /ligo/H-H1_LOSC_4_V1-1128678884-32.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/H-H1_LOSC_4_V1-1128678884-32.hdf5 -------------------------------------------------------------------------------- /ligo/H-H1_LOSC_4_V1-1135136334-32.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/H-H1_LOSC_4_V1-1135136334-32.hdf5 -------------------------------------------------------------------------------- /ligo/L-L1_LOSC_4_V1-1126259446-32.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/L-L1_LOSC_4_V1-1126259446-32.hdf5 -------------------------------------------------------------------------------- /ligo/L-L1_LOSC_4_V1-1128678884-32.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/L-L1_LOSC_4_V1-1128678884-32.hdf5 -------------------------------------------------------------------------------- /ligo/L-L1_LOSC_4_V1-1135136334-32.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/L-L1_LOSC_4_V1-1135136334-32.hdf5 -------------------------------------------------------------------------------- /ligo/LVT151012_4_template.hdf5: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/LVT151012_4_template.hdf5 -------------------------------------------------------------------------------- /ligo/O1_events.json: -------------------------------------------------------------------------------- 1 | { 2 | "GW150914":{ 3 | "name":"GW150914", 4 | "URL" : "https://losc.ligo.org/s/events/GW150914/", 5 | "fn_H1" : "H-H1_LOSC_4_V1-1126259446-32.hdf5", 6 | "fn_L1" : "L-L1_LOSC_4_V1-1126259446-32.hdf5", 7 | "fn_template" : "GW150914_4_template.hdf5", 8 | "fs" : 4096, 9 | "tevent" : 1126259462.44, 10 | "utcevent" : "2015-09-14T09:50:45.44", 11 | "m1" : 41.743, 12 | "m2" : 29.237, 13 | "a1" : 0.355, 14 | "a2" : -0.769, 15 | "approx" : "lalsim.SEOBNRv2", 16 | "fband" : [43.0,300.0], 17 | "f_min" : 10.0 18 | }, 19 | "LVT151012":{ 20 | "name":"LVT151012", 21 | "URL" : "https://losc.ligo.org/s/events/LVT151012/", 22 | "fn_H1" : "H-H1_LOSC_4_V1-1128678884-32.hdf5", 23 | "fn_L1" : "L-L1_LOSC_4_V1-1128678884-32.hdf5", 24 | "fn_template" : "LVT151012_4_template.hdf5", 25 | "fs" : 4096, 26 | "tevent" : 1128678900.44, 27 | "utcevent" : "2015-10-12T09:54:43.44", 28 | "m1" : 44.111, 29 | "m2" : 11.205, 30 | "a1" : 0.447, 31 | "a2" : -0.434, 32 | "approx" : "lalsim.SEOBNRv2", 33 | "fband" : [43.0,400.0], 34 | "f_min" : 10.0 35 | }, 36 | "GW151226":{ 37 | "name":"GW151226", 38 | "URL" : "https://losc.ligo.org/s/events/GW151226/", 39 | "fn_H1" : "H-H1_LOSC_4_V1-1135136334-32.hdf5", 40 | "fn_L1" : "L-L1_LOSC_4_V1-1135136334-32.hdf5", 41 | "fn_template" : "GW151226_4_template.hdf5", 42 | "fs" : 4096, 43 | "tevent" : 1135136350.65, 44 | "utcevent" : "2015-12-26T03:38:53.65", 45 | "m1" : 19.6427, 46 | "m2" : 6.7054, 47 | "a1" : 0.3998, 48 | "a2" : -0.0396, 49 | "approx" : "lalsim.SEOBNRv2", 50 | "fband" : [43.0,800.0], 51 | "f_min" : 10.0 52 | } 53 | } 54 | -------------------------------------------------------------------------------- /ligo/pyconfig1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/pyconfig1.png -------------------------------------------------------------------------------- /ligo/pyconfig2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/pyconfig2.png -------------------------------------------------------------------------------- /ligo/pyconfig3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/pyconfig3.png -------------------------------------------------------------------------------- /ligo/pyconfig4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/pyconfig4.png -------------------------------------------------------------------------------- /ligo/readligo.py: -------------------------------------------------------------------------------- 1 | """ 2 | readligo.py 3 | Version 0.2 4 | April 21, 2016 5 | Jonah Kanner, Roy Williams, and Alan Weinstein 6 | 7 | Updates in this version: 8 | * Should now work with both Python 2 and Python 3 9 | 10 | This module provides tools for reading LIGO data 11 | files. Data along with supporting documentation 12 | can be downloaded from the losc web site: 13 | https://losc.ligo.org 14 | 15 | Some possible use cases are shown below. 16 | 17 | Example #0: 18 | To load all data from a single file: 19 | strain, time, dq = rl.loaddata('ligo_data/H-H1_LOSC_4_V1-842653696-4096.hdf5', 'H1') 20 | 21 | Example #1: 22 | segList = getsegs(842657792, 842658792, 'H1') 23 | for (start, stop) in segList: 24 | strain, meta, dq = getstrain(start, stop, 'H1') 25 | # -- Analysis code here 26 | ... 27 | 28 | This default configuration assumes that the needed LIGO data 29 | files are available in the current working directory or a 30 | subdirectory. LIGO data between the input GPS times is loaded 31 | into STRAIN. META is a dictionary of gps start, gps stop, and the 32 | sample time. DQ is a dictionary of data quality flags. 33 | 34 | Example #2 35 | segList = SegmentList('H1_segs.txt') 36 | 37 | In Example 2, 'H1_segs.txt' is a segment list downloaded from the 38 | LOSC web site using the Timeline application. This may be used in the same 39 | manner as segList in example 1. 40 | 41 | Example #3 42 | filelist = FileList(directory='/home/ligodata') 43 | segList = getsegs(842657792, 842658792, 'H1', filelist=filelist) 44 | for start, stop in segList: 45 | strain, meta, dq = getstrain(start, stop, 'H1', filelist=filelist) 46 | # -- Analysis code here 47 | 48 | In this example, the first command searches the indicated directory and 49 | sub-directories for LIGO data files. This list of data files is then 50 | used to construct a segment list and load the requested data. 51 | 52 | -- SEGMENT LISTS -- 53 | 54 | Segment lists may be downloaded from the LOSC web site 55 | using the Timeline Query Form or constructed directly 56 | from the data files. 57 | 58 | Read in a segment list downloaded from the Timeline 59 | application on the LOSC web site with SegmentList: 60 | >> seglist = SegmentList('H1_segs.txt') 61 | OR 62 | Construct a segment list directly from the LIGO 63 | data files with getsegs(): 64 | >> seglist = getsegs(842657792, 842658792, 'H1', flag='DATA', filelist=None) 65 | 66 | """ 67 | 68 | import numpy as np 69 | import os 70 | import fnmatch 71 | 72 | def read_frame(filename, ifo, readstrain=True): 73 | """ 74 | Helper function to read frame files 75 | """ 76 | try: 77 | import Fr 78 | except: 79 | from pylal import Fr 80 | 81 | if ifo is None: 82 | raise TypeError("""To read GWF data, ifo must be 'H1', 'H2', or 'L1'. 83 | def loaddata(filename, ifo=None):""") 84 | 85 | #-- Read strain channel 86 | strain_name = ifo + ':LOSC-STRAIN' 87 | if readstrain: 88 | sd = Fr.frgetvect(filename, strain_name) 89 | strain = sd[0] 90 | gpsStart = sd[1] 91 | ts = sd[3][0] 92 | else: 93 | ts = 1 94 | strain = 0 95 | 96 | #-- Read DQ channel 97 | dq_name = ifo + ':LOSC-DQMASK' 98 | qd = Fr.frgetvect(filename, dq_name) 99 | gpsStart = qd[1] 100 | qmask = np.array(qd[0]) 101 | dq_ts = qd[3][0] 102 | shortnameList_wbit = qd[5].split() 103 | shortnameList = [name.split(':')[1] for name in shortnameList_wbit] 104 | 105 | #-- Read Injection channel 106 | inj_name = ifo + ':LOSC-INJMASK' 107 | injdata = Fr.frgetvect(filename, inj_name) 108 | injmask = injdata[0] 109 | injnamelist_bit = injdata[5].split() 110 | injnamelist = [name.split(':')[1] for name in injnamelist_bit] 111 | 112 | return strain, gpsStart, ts, qmask, shortnameList, injmask, injnamelist 113 | 114 | def read_hdf5(filename, readstrain=True): 115 | """ 116 | Helper function to read HDF5 files 117 | """ 118 | import h5py 119 | dataFile = h5py.File(filename, 'r') 120 | 121 | #-- Read the strain 122 | if readstrain: 123 | strain = dataFile['strain']['Strain'][...] 124 | else: 125 | strain = 0 126 | 127 | ts = dataFile['strain']['Strain'].attrs['Xspacing'] 128 | 129 | #-- Read the DQ information 130 | dqInfo = dataFile['quality']['simple'] 131 | qmask = dqInfo['DQmask'][...] 132 | shortnameArray = dqInfo['DQShortnames'].value 133 | shortnameList = list(shortnameArray) 134 | 135 | # -- Read the INJ information 136 | injInfo = dataFile['quality/injections'] 137 | injmask = injInfo['Injmask'][...] 138 | injnameArray = injInfo['InjShortnames'].value 139 | injnameList = list(injnameArray) 140 | 141 | #-- Read the meta data 142 | meta = dataFile['meta'] 143 | gpsStart = meta['GPSstart'].value 144 | 145 | dataFile.close() 146 | return strain, gpsStart, ts, qmask, shortnameList, injmask, injnameList 147 | 148 | def loaddata(filename, ifo=None, tvec=True, readstrain=True): 149 | """ 150 | The input filename should be a LOSC .hdf5 file or a LOSC .gwf 151 | file. The file type will be determined from the extenstion. 152 | The detector should be H1, H2, or L1. 153 | 154 | The return value is: 155 | STRAIN, TIME, CHANNEL_DICT 156 | 157 | STRAIN is a vector of strain values 158 | TIME is a vector of time values to match the STRAIN vector 159 | unless the flag tvec=False. In that case, TIME is a 160 | dictionary of meta values. 161 | CHANNEL_DICT is a dictionary of data quality channels 162 | """ 163 | 164 | # -- Check for zero length file 165 | if os.stat(filename).st_size == 0: 166 | return None, None, None 167 | 168 | file_ext = os.path.splitext(filename)[1] 169 | if (file_ext.upper() == '.GWF'): 170 | strain, gpsStart, ts, qmask, shortnameList, injmask, injnameList = read_frame(filename, ifo, readstrain) 171 | else: 172 | strain, gpsStart, ts, qmask, shortnameList, injmask, injnameList = read_hdf5(filename, readstrain) 173 | 174 | #-- Create the time vector 175 | gpsEnd = gpsStart + len(qmask) 176 | if tvec: 177 | time = np.arange(gpsStart, gpsEnd, ts) 178 | else: 179 | meta = {} 180 | meta['start'] = gpsStart 181 | meta['stop'] = gpsEnd 182 | meta['dt'] = ts 183 | 184 | #-- Create 1 Hz DQ channel for each DQ and INJ channel 185 | channel_dict = {} #-- 1 Hz, mask 186 | slice_dict = {} #-- sampling freq. of stain, a list of slices 187 | final_one_hz = np.zeros(qmask.shape, dtype='int32') 188 | for flag in shortnameList: 189 | bit = shortnameList.index(flag) 190 | # Special check for python 3 191 | if isinstance(flag, bytes): flag = flag.decode("utf-8") 192 | 193 | channel_dict[flag] = (qmask >> bit) & 1 194 | 195 | for flag in injnameList: 196 | bit = injnameList.index(flag) 197 | # Special check for python 3 198 | if isinstance(flag, bytes): flag = flag.decode("utf-8") 199 | 200 | channel_dict[flag] = (injmask >> bit) & 1 201 | 202 | #-- Calculate the DEFAULT channel 203 | try: 204 | channel_dict['DEFAULT'] = ( channel_dict['DATA'] ) 205 | except: 206 | print("Warning: Failed to calculate DEFAULT data quality channel") 207 | 208 | if tvec: 209 | return strain, time, channel_dict 210 | else: 211 | return strain, meta, channel_dict 212 | 213 | 214 | def dq2segs(channel, gps_start): 215 | """ 216 | This function takes a DQ CHANNEL (as returned by loaddata or getstrain) and 217 | the GPS_START time of the channel and returns a segment 218 | list. The DQ Channel is assumed to be a 1 Hz channel. 219 | 220 | Returns of a list of segment GPS start and stop times. 221 | """ 222 | #-- Check if the user input a dictionary 223 | if type(channel) == dict: 224 | try: 225 | channel = channel['DEFAULT'] 226 | except: 227 | print("ERROR: Could not find DEFAULT channel in dictionary") 228 | raise 229 | 230 | #-- Create the segment list 231 | segments = dq_channel_to_seglist(channel, fs=1) 232 | t0 = gps_start 233 | segList = [(int(seg.start+t0), int(seg.stop+t0)) for seg in segments] 234 | return SegmentList(segList) 235 | 236 | def dq_channel_to_seglist(channel, fs=4096): 237 | """ 238 | WARNING: 239 | This function is designed to work the output of the low level function 240 | LOADDATA, not the output from the main data loading function GETSTRAIN. 241 | 242 | Takes a data quality 1 Hz channel, as returned by 243 | loaddata, and returns a segment list. The segment 244 | list is really a list of slices for the strain 245 | associated strain vector. 246 | 247 | If CHANNEL is a dictionary instead of a single channel, 248 | an attempt is made to return a segment list for the DEFAULT 249 | channel. 250 | 251 | Returns a list of slices which can be used directly with the 252 | strain and time outputs of LOADDATA. 253 | """ 254 | #-- Check if the user input a dictionary 255 | if type(channel) == dict: 256 | try: 257 | channel = channel['DEFAULT'] 258 | except: 259 | print("ERROR: Could not find DEFAULT channel in dictionary") 260 | raise 261 | 262 | # -- Create the segment list 263 | condition = channel > 0 264 | boundaries = np.where(np.diff(condition) == True)[0] 265 | # -- Need to +1 due to how np.diff works 266 | boundaries = boundaries + 1 267 | # if the array "begins" True, we need to complete the first segment 268 | if condition[0]: 269 | boundaries = np.append(0,boundaries) 270 | # if the array "ends" True, we need to complete the last segment 271 | if condition[-1]: 272 | boundaries = np.append(boundaries,len(condition)) 273 | 274 | # -- group the segment boundaries two by two 275 | segments = boundaries.reshape((len(boundaries)/2,2)) 276 | # -- Account for sampling frequency and return a slice 277 | segment_list = [slice(start*fs, stop*fs) for (start,stop) in segments] 278 | 279 | return segment_list 280 | 281 | class FileList(): 282 | """ 283 | Class for lists of LIGO data files. 284 | 285 | When a FileList instance is created, DIRECTORY will 286 | be searched for LIGO data files. Sub-directories 287 | will be searched as well. By default, the current 288 | working directory is searched. 289 | """ 290 | def __init__(self, directory=None, cache=None): 291 | 292 | # -- Set default directory 293 | if directory is None: 294 | if os.path.isdir('/archive/losc/strain-gwf'): 295 | directory='/archive/losc/strain-gwf' 296 | else: 297 | directory='.' 298 | 299 | print("Using data directory {0} ...".format(directory)) 300 | self.directory = directory 301 | self.cache = cache 302 | if cache is None: 303 | self.list = self.searchdir(directory) 304 | else: 305 | self.readcache() 306 | 307 | def searchdir(self, directory='.'): 308 | frameList = [] 309 | hdfList = [] 310 | for root, dirnames, filenames in os.walk(directory): 311 | for filename in fnmatch.filter(filenames, '*.gwf'): 312 | frameList.append(os.path.join(root, filename)) 313 | for filename in fnmatch.filter(filenames, '*.hdf5'): 314 | hdfList.append(os.path.join(root, filename)) 315 | return frameList + hdfList 316 | 317 | def writecache(self, cacheName): 318 | outfile = open(cacheName, 'w') 319 | for file in self.list: 320 | outfile.write(file + '\n') 321 | outfile.close() 322 | 323 | def readcache(self): 324 | infile = open(self.cache, 'r') 325 | self.list = infile.read().split() 326 | infile.close() 327 | 328 | def findfile(self, gps, ifo): 329 | start_gps = gps - (gps % 4096) 330 | filenamelist = fnmatch.filter(self.list, '*' + '-' + ifo + '*' + '-' + str(start_gps) + '-' + '*') 331 | if len(filenamelist) == 0: 332 | print("WARNING! No file found for GPS {0} and IFO {1}".format(gps, ifo)) 333 | return None 334 | else: 335 | return filenamelist[0] 336 | 337 | def getstrain(start, stop, ifo, filelist=None): 338 | """ 339 | START should be the starting gps time of the data to be loaded. 340 | STOP should be the end gps time of the data to be loaded. 341 | IFO should be 'H1', 'H2', or 'L1'. 342 | FILELIST is an optional argument that is a FileList() instance. 343 | 344 | The return value is (strain, meta, dq) 345 | 346 | STRAIN: The data as a strain time series 347 | META: A dictionary of meta data, especially the start time, stop time, 348 | and sample time 349 | DQ: A dictionary of the data quality flags 350 | """ 351 | 352 | if filelist is None: 353 | filelist = FileList() 354 | 355 | # -- Check if this is a science segment 356 | segList = getsegs(start, stop, ifo, flag='DATA', filelist=filelist) 357 | sl = segList.seglist 358 | if (sl[0][0] == start) and (sl[0][1] == stop): 359 | pass 360 | else: 361 | raise TypeError("""Error in getstrain. 362 | Requested times include times where the data file was not found 363 | or instrument not in SCIENCE mode. 364 | Use readligo.getsegs() to construct a segment list. 365 | The science mode segment list for the requested time range is: 366 | {0}""".format(segList)) 367 | 368 | # -- Construct list of expected file start times 369 | first = start - (start % 4096) 370 | gpsList = np.arange(first, stop, 4096) 371 | 372 | m_strain = np.array([]) 373 | m_dq = None 374 | # -- Loop over needed files 375 | for time in gpsList: 376 | filename = filelist.findfile(time, ifo) 377 | print("Loading {0}".format(filename)) 378 | 379 | #-- Read in data 380 | strain, meta, dq = loaddata(filename, ifo, tvec=False) 381 | if len(m_strain) == 0: 382 | m_start = meta['start'] 383 | dt = meta['dt'] 384 | m_stop = meta['stop'] 385 | m_strain = np.append(m_strain, strain) 386 | if m_dq is None: 387 | m_dq = dq 388 | else: 389 | for key in dq.keys(): 390 | m_dq[key] = np.append(m_dq[key], dq[key]) 391 | 392 | # -- Trim the data 393 | lndx = np.abs(start - m_start)*(1.0/dt) 394 | rndx = np.abs(stop - m_start)*(1.0/dt) 395 | 396 | m_strain = m_strain[lndx:rndx] 397 | for key in m_dq.keys(): 398 | m_dq[key] = m_dq[key][lndx*dt:rndx*dt] 399 | 400 | meta['start'] = start 401 | meta['stop'] = stop 402 | meta['dt'] = dt 403 | 404 | return m_strain, meta, m_dq 405 | 406 | class SegmentList(): 407 | def __init__(self, filename, numcolumns=3): 408 | 409 | if type(filename) is str: 410 | if numcolumns == 4: 411 | number, start, stop, duration = np.loadtxt(filename, dtype='int',unpack=True) 412 | elif numcolumns == 2: 413 | start, stop = np.loadtxt(filename, dtype='int',unpack=True) 414 | elif numcolumns == 3: 415 | start, stop, duration = np.loadtxt(filename, dtype='int',unpack=True) 416 | self.seglist = zip(start, stop) 417 | elif type(filename) is list: 418 | self.seglist = filename 419 | else: 420 | raise TypeError("SegmentList() expects the name of a segmentlist file from the LOSC website Timeline") 421 | 422 | def __repr__(self): 423 | return 'SegmentList( {0} )'.format(self.seglist) 424 | def __iter__(self): 425 | return iter(self.seglist) 426 | def __getitem__(self, key): 427 | return self.seglist[key] 428 | 429 | def getsegs(start, stop, ifo, flag='DATA', filelist=None): 430 | """ 431 | Method for constructing a segment list from 432 | LOSC data files. By default, the method uses 433 | files in the current working directory to 434 | construct a segment list. 435 | 436 | If a FileList is passed in the flag FILELIST, 437 | then those files will be searched for segments 438 | passing the DQ flag passed as the FLAG argument. 439 | """ 440 | 441 | if filelist is None: 442 | filelist = FileList() 443 | 444 | # -- Construct list of expected file start times 445 | first = start - (start % 4096) 446 | gpsList = np.arange(first, stop, 4096) 447 | m_dq = None 448 | 449 | # -- Initialize segment list 450 | segList = [] 451 | 452 | # -- Loop over needed files 453 | for time in gpsList: 454 | filename = filelist.findfile(time, ifo) 455 | 456 | #-- Read in data 457 | if filename is None: 458 | print("WARNING! No file found with GPS start time {0}".format(time)) 459 | print("Segment list may contain errors due to missing files.") 460 | continue 461 | else: 462 | try: 463 | strain, meta, dq = loaddata(filename, ifo, tvec=False, readstrain=False) 464 | except: 465 | print("WARNING! Failed to load file {0}".format(filename)) 466 | print("Segment list may contain errors due to corrupt files.") 467 | continue 468 | 469 | if dq is None: 470 | print("Warning! Found zero length file {0}".format(filename)) 471 | print("Segment list may contain errors.") 472 | continue 473 | 474 | #-- Add segments to list on a file-by-file basis 475 | chan = dq[flag] 476 | indxlist = dq_channel_to_seglist(chan, fs=1.0) 477 | i_start = meta['start'] 478 | i_seglist = [(indx.start+i_start, indx.stop+i_start) for indx in indxlist] 479 | i_seglist = [(int(begin), int(end)) for begin, end in i_seglist] 480 | segList = segList + i_seglist 481 | 482 | # -- Sort segments 483 | segList.sort() 484 | 485 | # -- Merge overlapping segments 486 | for i in range(0, len(segList)-1): 487 | seg1 = segList[i] 488 | seg2 = segList[i+1] 489 | 490 | if seg1[1] == seg2[0]: 491 | segList[i] = None 492 | segList[i+1] = (seg1[0], seg2[1]) 493 | # -- Remove placeholder segments 494 | segList = [seg for seg in segList if seg is not None] 495 | 496 | # -- Trim segment list to fit within requested start/stop times 497 | for seg in segList: 498 | idx = segList.index(seg) 499 | if (seg[1] < start): 500 | segList[idx] = None 501 | elif (seg[0] > stop): 502 | segList[idx] = None 503 | elif (seg[0] < start) and (seg[1] > stop): 504 | segList[idx] = (start, stop) 505 | elif (seg[0] < start): 506 | segList[idx] = (start, seg[1]) 507 | elif (seg[1] > stop): 508 | segList[idx] = (seg[0], stop) 509 | # -- Remove placeholder segments 510 | segList = [seg for seg in segList if seg is not None] 511 | 512 | return SegmentList(segList) 513 | 514 | 515 | -------------------------------------------------------------------------------- /ligo/readligo.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/ligo/readligo.pyc -------------------------------------------------------------------------------- /pyconjp2016.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/pyconjp2016.pdf -------------------------------------------------------------------------------- /sesimology/stations/GR.NRDL.xml: -------------------------------------------------------------------------------- 1 | SeisComP3BGR2016-09-20T12:07:44German Regional Seismic Network, BGR Hannover52.4943210.10728-355GRSN Station NiedersachsenLower_Saxony2003-06-20T00:00:0052.4943210.10728-35500-901001001Steim20.01STS225188917001m/sCOUNTSM/SVLAPLACE (RADIANS/SECOND)11-0.03674290.036754-0.0367429-0.036754000025188917001VCOUNTSDIGITAL10010001052.4943210.10728-3550001001001Steim20.01STS225188917001m/sCOUNTSM/SVLAPLACE (RADIANS/SECOND)11-0.03674290.036754-0.0367429-0.036754000025188917001VCOUNTSDIGITAL10010001052.4943210.10728-35509001001001Steim20.01STS225188917001m/sCOUNTSM/SVLAPLACE (RADIANS/SECOND)11-0.03674290.036754-0.0367429-0.036754000025188917001VCOUNTSDIGITAL100100010 2 | -------------------------------------------------------------------------------- /sesimology/waveforms/GR.NRDL..HHZ__20110311T054232Z__20110311T064732Z.mseed: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/bakfoo/pyconjp2016/c508381b71454634595c85f10b6c8cb3141ec6f3/sesimology/waveforms/GR.NRDL..HHZ__20110311T054232Z__20110311T064732Z.mseed --------------------------------------------------------------------------------