├── README.md
├── bulk.sh
├── demo.ipynb
├── example_list.txt
├── graph.xml
├── prep.py
├── prepro.sh
├── productdef.yaml
├── radar average.ipynb
└── subset.sh
/README.md:
--------------------------------------------------------------------------------
1 | Pre-processing of Sentinel 1
2 | ============================
3 |
4 | This project aims to connect [Sentinel 1](https://sentinel.esa.int/web/sentinel/missions/sentinel-1)
5 | with the [Open Data Cube](https://github.com/opendatacube/datacube-core).
6 |
7 | It takes Synthetic Aperture Radar data (specifically GRD scenes from the Sentinel 1 platform) and prepares it for ingestion into an opendatacube instance (such as Digital Earth Australia), using the Sentinel Toolbox (SNAP) software.
8 |
9 | *NOTE: in-progress, use at own risk.*
10 |
11 | Processing steps
12 | ----------------
13 | Prerequisites:
14 | - Sentinel 1 GRD (ground-range, detected amplitude) scenes.
15 | - Precise orbital ephemeris metadata. (Possibly also calibration?)
16 | - A digital model for the elevations of the scattering surface (DSM/DEM).
17 | - gpt (graph processing tool) from the Sentinel Toolbox software.
18 | - Access to a configured ODC instance.
19 |
20 | Stages:
21 |
22 | 1. Update metadata (i.e. orbit vectors)
23 | 2. Trim border noise (an artifact of the S1 GRD products)
24 | 3. Calibrate (radiometric, outputting beta-nought)
25 | 4. Flatten (radiometric terrain correction)
26 | 5. Range-Doppler (geometric terrain correction)
27 | 6. Format for the AGDC (e.g. export metadata, tile and index)
28 |
29 | Implementation
30 | --------------
31 | Initially will use auto-downloaded auxilliary data (ephemeris, DEM). Later, intend to use GDAL tools to subset a DSM, or test efficiency of chunked file-formats for the DSM raster.
32 |
33 | Steps 1-4 will be combined in a gpt xml.
34 |
35 | Step 5 will be a gpt command-line instruction. (Some operators chain together inefficiently, at least in previous gpt versions.)
36 |
37 | Step 6 will be a python prep script.
38 |
39 | The overall orchestration will initially be a shell script. (Other options would be a Makefile or a python cluster scheduling script.)
40 |
41 | A jupyter notebook will demonstrate the result (using opendatacube API).
42 |
43 | Known flaws
44 | -----------
45 |
46 | - Ocean is masked out. (This is due to the nodata value used for the DEM by the terrain correction steps.)
47 | - Border noise is not entirely eliminated (some perimeter pixels).
48 | - Could conceive a more efficient unified radiometric/geometric terrain-correction operator (to reduce file IO concerning DSM)?
49 | - Further comparison with GAMMA software output is necessary.
50 | - Signal intensity units unspecified.
51 | - Currently autodownloading ancilliary data (e.g. using 3s SRTM, which is suboptimal).
52 | - Output format is ENVI raster (approximately 10x larger than input zip) rather than cloud optimised GeoTIFF.
53 |
54 | Instructions
55 | ------------
56 |
57 | **Process imagery**
58 |
59 | 1. Ensure the graph processing tool is available (run "ln -s ../snap6/bin/gpt gpt" after installing SNAP)
60 | 2. Batch process some scenes (run "./bulk.sh example_list.txt" after confirming example input)
61 |
62 | (Takes 10-15min/scene, using 4 cores and 10-15GB memory, on VDI@NCI.)
63 |
64 | **Insert into Open Data Cube**
65 |
66 | 3. Ensure the environment has been prepared (run "datacube system check")
67 | 4. Define the products (run "datacube product add productdef.yaml")
68 | 5. For each newly preprocessed scene, run a preparation script (e.g. "python prep.py output1.dim") to generate metadata (yaml) in an appropriate format for datacube indexing.
69 | 6. For each of those prepared scenes, index into the datacube (e.g. "datacube dataset add output*.yaml --auto-match")
70 | 7. Verify the data using the datacube API (e.g. a python notebook).
71 |
72 |
73 |
--------------------------------------------------------------------------------
/bulk.sh:
--------------------------------------------------------------------------------
1 | #!/bin/env bash
2 | #
3 | # Preprocess many Sentinel 1 scenes using gpt
4 | #
5 |
6 | # For testing, try a random subset of the full input list:
7 | # $ sort -R full.list | head > subset.list
8 |
9 | # For complete run:
10 | # $ /usr/bin/time ./bulk.sh full.list | tee master.log
11 |
12 |
13 | if [ "$#" != "1" ]; then # validate argument count
14 | echo Usage: . bulk.sh input_list.txt
15 | else
16 | echo "Processing files listed in $1"
17 |
18 |
19 | date
20 |
21 | while read -r line; do
22 | short=$(echo $line | sed s:^.\*/:: | sed s:\.zip$::)
23 |
24 | echo $short
25 |
26 | . prepro.sh $line $short.dim > $short.log 2>&1
27 |
28 | [ -e $short.data ] && echo DONE || echo ABORT # check whether final output produced (not whether ok)
29 | done < $1
30 |
31 | date
32 | fi
33 |
--------------------------------------------------------------------------------
/demo.ipynb:
--------------------------------------------------------------------------------
1 | {
2 | "cells": [
3 | {
4 | "cell_type": "code",
5 | "execution_count": 1,
6 | "metadata": {
7 | "collapsed": true
8 | },
9 | "outputs": [],
10 | "source": [
11 | "import datacube"
12 | ]
13 | },
14 | {
15 | "cell_type": "code",
16 | "execution_count": 2,
17 | "metadata": {
18 | "collapsed": true
19 | },
20 | "outputs": [],
21 | "source": [
22 | "dc = datacube.Datacube()"
23 | ]
24 | },
25 | {
26 | "cell_type": "code",
27 | "execution_count": 31,
28 | "metadata": {
29 | "collapsed": false
30 | },
31 | "outputs": [
32 | {
33 | "data": {
34 | "text/html": [
35 | "\n",
36 | "
\n",
37 | " \n",
38 | " \n",
39 | " | \n",
40 | " name | \n",
41 | " description | \n",
42 | " instrument | \n",
43 | " format | \n",
44 | " product_type | \n",
45 | " platform | \n",
46 | " crs | \n",
47 | " resolution | \n",
48 | " tile_size | \n",
49 | " spatial_dimensions | \n",
50 | "
\n",
51 | " \n",
52 | " | id | \n",
53 | " | \n",
54 | " | \n",
55 | " | \n",
56 | " | \n",
57 | " | \n",
58 | " | \n",
59 | " | \n",
60 | " | \n",
61 | " | \n",
62 | " | \n",
63 | "
\n",
64 | " \n",
65 | " \n",
66 | " \n",
67 | " | 3 | \n",
68 | " s1a_gamma0_scene | \n",
69 | " Sentinel-1A SAR Gamma0 | \n",
70 | " SAR | \n",
71 | " ENVI | \n",
72 | " gamma0 | \n",
73 | " SENTINEL_1A | \n",
74 | " NaN | \n",
75 | " NaN | \n",
76 | " NaN | \n",
77 | " NaN | \n",
78 | "
\n",
79 | " \n",
80 | " | 4 | \n",
81 | " s1b_gamma0_scene | \n",
82 | " Sentinel-1B SAR Gamma0 | \n",
83 | " SAR | \n",
84 | " ENVI | \n",
85 | " gamma0 | \n",
86 | " SENTINEL_1B | \n",
87 | " NaN | \n",
88 | " NaN | \n",
89 | " NaN | \n",
90 | " NaN | \n",
91 | "
\n",
92 | " \n",
93 | "
\n",
94 | "
\n",
128 | "
\n",
129 | " \n",
130 | " \n",
131 | " | \n",
132 | " | \n",
133 | " dtype | \n",
134 | " name | \n",
135 | " nodata | \n",
136 | " units | \n",
137 | "
\n",
138 | " \n",
139 | " | product | \n",
140 | " measurement | \n",
141 | " | \n",
142 | " | \n",
143 | " | \n",
144 | " | \n",
145 | "
\n",
146 | " \n",
147 | " \n",
148 | " \n",
149 | " | s1a_gamma0_scene | \n",
150 | " vh | \n",
151 | " float32 | \n",
152 | " vh | \n",
153 | " 0 | \n",
154 | " 1 | \n",
155 | "
\n",
156 | " \n",
157 | " | vv | \n",
158 | " float32 | \n",
159 | " vv | \n",
160 | " 0 | \n",
161 | " 1 | \n",
162 | "
\n",
163 | " \n",
164 | " | s1b_gamma0_scene | \n",
165 | " vh | \n",
166 | " float32 | \n",
167 | " vh | \n",
168 | " 0 | \n",
169 | " 1 | \n",
170 | "
\n",
171 | " \n",
172 | " | vv | \n",
173 | " float32 | \n",
174 | " vv | \n",
175 | " 0 | \n",
176 | " 1 | \n",
177 | "
\n",
178 | " \n",
179 | "
\n",
180 | "