45 |
46 |
Abstract
47 |
48 | Due to the nature of their pathways, NASA Terra and NASA Aqua satellites capture imagery containing “swath gaps'' which are areas of no data. Swath gaps can overlap the region of interest (ROI) completely, often rendering the entire imagery unusable by Machine Learning (ML) models. This problem is further exacerbated when the ROI rarely occurs (e.g. a hurricane) and, on occurrence, is partially overlapped with a swath gap. With annotated data as supervision, a model can learn to differentiate between the area of focus and the swath gap. However, annotation is expensive and currently the vast majority of existing data is unannotated. Hence, we propose an augmentation technique that considerably removes the existence of swath gaps in order to allow CNNs to focus on the ROI, and thus successfully use data with swath gaps for training. We experiment on the UC Merced Land Use Dataset, where we add swath gaps through empty polygons (up to 20% areas) and then apply augmentation techniques to fill the swath gaps. We compare the model trained with our augmentation techniques on the swath gap-filled data with the model trained on the original swath gap-less data and note highly augmented performance. Additionally, we perform a qualitative analysis using activation maps that visualizes the effectiveness of our trained network in not paying attention to the swath gaps. We also evaluate our results with a human baseline and show that, in certain cases, the filled swath gaps look so realistic that even a human evaluator did not distinguish between original satellite images and swath gap-filled images. Since this method is aimed at unlabeled data, it is widely generalizable and impactful for large scale unannotated datasets from various space data domains.
49 |
50 |
51 |
129 |
130 |
What are Swath Gaps?
131 | Swath gaps are empty or no data regions that occur in MODIS imagery. They exist because the MODIS satellites have a swath bandwidth of only 2330 km wide, causing consecutive orbits at the equator to miss coverage. Due to this missing coverage, these regions of missing data are present primarily as nine spindle shapes around the equator; however, they also occur at the North and South Poles during seasons of minimal sunlight. This uncollected data can be visualized as black spindles or swath gaps by NASA's Earth Observing System Data and Information System in the
NASA Worldview.
132 |
Why are Swath Gaps important?
133 | Through data exploration experiments and input from the NASA IMPACT team, we noted that similarity search experiments that find the most similar image, supposedly based on regions of interests (ROIs) such as hurricanes or beaches, instead return images with similarly placed swath gaps. These search engines focus on swath gaps, rather than concentrating on the ROI. Thus, we determine that when notable swath gaps are present in satellite imagery, specifically used for training Earth-sciences machine learning (ML) models, these areas of missing data render the entire image unusable by unsupervised training models. Further, ML pattern recognition algorithms begin recognizing the image's swath gap as its main feature, rather than the features of its primary ROI. This is an issue because, given the nature of satellite imagery, events of interest or ROIs are already quite sporadic in regards to rare events such as tornadoes, wildfires, and volcanic eruptions, and thus every piece of data is valuable. With already limited data, the occurrence of a swath gap overlapping the ROI further reduces the available data.
134 | We address the problem that swath gaps create when present in unlabeled image datasets that are used to train unsupervised machine learning models. The following is an example of swath gaps present due to NASA Terra and NASA Aqua satellite pathways.
135 |
136 |
137 |
138 |
139 |
140 |
141 |
142 |
143 |
144 |
145 |
146 |
147 |
Our Filling Methods
148 | We resolve the problem that swath gaps pose by experimenting with three different methods: Random RGB (filling the swath gap with randomly selected RGB pixels), Pixel RGB (filling the swath gap with randomly selected RGB pixels from the image), and Neighbor RGB (filling the swath gap with randomly selected RGB pixels from a dynamic radius of the given pixel to be filled).
149 |
150 |
151 |
152 |
153 |
154 |
155 |
156 |
157 |
158 |
159 |
160 |
161 |
Evaluation Activation Maps
162 | One way in which we assess our fill methods is by analyzing the effects they have on activation maps as seen below. We further evaluated our fill methods by using an autoencoder to perform similarity searches - given an input image, we ask the autoencoder to return the four most similar images from the dataset. Separate autoencoder models with each augmentation policy are trained. Testing was then conducted on the three filling methods by querying images filled with our methods into the autoencoder. These results greatly improved as methodology increased from method one (Random RGB) to two (Pixel RGB) to three (Neighbor RGB), with successively more images categorized correctly per fill method. Repetition of this experiment indicated that filling method three, Neighbor RGB, was the most efficient, with consistently three or four out of the four "most similar" images being categorized correctly.
163 |
164 |
165 |
166 |
167 |
168 |
169 |
170 |
171 |
172 |
173 |
36 |
37 |
38 | 
39 |
40 | 
8 |
9 | The command-line tool is intended to obscure regions of null or missing data from machine learning pattern-recognition algorithms. However, this package can be used to fill in an image's missing data or a given RGB value in the image for any purpose.
10 |
11 | More information on our work with this package can be found [on Arxiv](https://arxiv.org/abs/2106.07113), or [](https://colab.research.google.com/drive/1GYyXAA_EbUveTeDgwy-wMHj1LXIdXdcy?usp=sharing)
12 |
13 | ## Usage
14 |
15 | #### Installation
16 |
17 | This package can be installed by typing the following into your command line:
18 |
19 | `pip install git+https://github.com/spaceml-org/Missing-Pixel-Filler.git#egg=missing_pixel_filler`
20 |
21 | #### Functions
22 |
23 | *Filling Method 1*
24 |
25 | `fill_swath_with_random_rgb(img)` - Selects pixel values from a random Gaussian distribution to fill missing pixel values.
26 |
27 | *Filling Method 2*
28 |
29 | `fill_swath_with_random_pixel_from_image_new(img)` - Fills swath gap with random pixel from non-missing portion of data.
30 |
31 | |-- `get_random_pixel_from_image(x_arr, y_arr)` - Selects other pixel values from non-missing portions of the image.
32 |
33 | *Filling Method 3*
34 |
35 | `fill_swath_with_neighboring_pixel(img)` - Fills "dynamic" system to fill swath, with nearest pixels having higher probability of selection.
36 |
37 | |-- `get_neighboring_pixel()` - Selects pixel pixel values from non-missing portions of the image to fill missing pixel values with probability inversely proportional to distance.
38 |
39 | #### Arguments
40 |
41 | The function arguments are as follows:
42 |
43 | `img` - numpy array of input image in utf8 form.
44 |
45 | `color ` - Optional parameter - color of missing data to fill in (0 - black, 256 - white). Default is black.
46 |
47 | `current_window_size` - for adaptive nearest neighbors method (method 3), radius for selecting neighbor pixels. Default is 10 pixels.
48 |
49 | ## Example
50 |
51 | Below are some examples with missing data regions filled by our python function (filling method 3). The function call is as follows:
52 |
53 | `fill_swath_with_neighboring_pixel(img)`.
54 |
55 | Image results:
56 |
57 | **Pre-fill:**
58 | 
59 |
60 | **Post-fill:**
61 | 
62 |
63 | ## FAQs
64 |
65 | #### What type of images can be used with the python package?
66 |
67 | Our package works best if less than 25% of the image data is missing. This package was originally developed to fill missing data from NASA WorldView, in which an image containing 10% to 25% of the total image is still considered for training data.
68 |
69 | #### How does missing-pixel-filler recognize "missing data"?
70 |
71 | By default, our code recognizes "missing data" as [0,0,0] RGB, i.e. black, pixels. However, the user has the option to change what RGB values are recognized as "missing data" using the `color` parameter to the functions.
72 |
73 | #### How has missing-pixel-filler been used?
74 |
75 | Our code was created as a part of research done under SpaceML and alongside NASA's Impact Team. Our goal was to reduce the effects of swath gaps found in NASA Terra and Aqua satellite images in unsupervised machine learning, as missing data may pose a notable pattern that incorrectly attracts the attention of unsupervised models. More information on our work can be found [here](https://drive.google.com/file/d/18LSWDsXX9PdDLoYNuzKGLzKUZEuGzAo_/view?usp=sharing).
76 |
77 | ## Citation
78 |
79 | If `missing-pixel-filler` is useful in your research, please consider citing
80 |
81 | ```
82 | @article{caochen2020swathgaps,
83 | title={Reducing Effects of Swath Gaps in Unsupervised Machine Learning},
84 | author={Chen, Sarah and Cao, Esther and Koul, Anirudh and Ganju, Siddha and Praveen, Satyarth and Kasam, Meher Anand},
85 | journal={Committee on Space Research Machine Learning for Space Sciences Workshop},
86 | year={2021}
87 | }
88 | ```
89 |
--------------------------------------------------------------------------------
/docs/css:
--------------------------------------------------------------------------------
1 | /* cyrillic-ext */
2 | @font-face {
3 | font-family: 'Roboto';
4 | font-style: normal;
5 | font-weight: 300;
6 | src: local('Roboto Light'), local('Roboto-Light'), url(http://fonts.gstatic.com/s/roboto/v15/0eC6fl06luXEYWpBSJvXCIX0hVgzZQUfRDuZrPvH3D8.woff2) format('woff2');
7 | unicode-range: U+0460-052F, U+20B4, U+2DE0-2DFF, U+A640-A69F;
8 | }
9 | /* cyrillic */
10 | @font-face {
11 | font-family: 'Roboto';
12 | font-style: normal;
13 | font-weight: 300;
14 | src: local('Roboto Light'), local('Roboto-Light'), url(http://fonts.gstatic.com/s/roboto/v15/Fl4y0QdOxyyTHEGMXX8kcYX0hVgzZQUfRDuZrPvH3D8.woff2) format('woff2');
15 | unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;
16 | }
17 | /* greek-ext */
18 | @font-face {
19 | font-family: 'Roboto';
20 | font-style: normal;
21 | font-weight: 300;
22 | src: local('Roboto Light'), local('Roboto-Light'), url(http://fonts.gstatic.com/s/roboto/v15/-L14Jk06m6pUHB-5mXQQnYX0hVgzZQUfRDuZrPvH3D8.woff2) format('woff2');
23 | unicode-range: U+1F00-1FFF;
24 | }
25 | /* greek */
26 | @font-face {
27 | font-family: 'Roboto';
28 | font-style: normal;
29 | font-weight: 300;
30 | src: local('Roboto Light'), local('Roboto-Light'), url(http://fonts.gstatic.com/s/roboto/v15/I3S1wsgSg9YCurV6PUkTOYX0hVgzZQUfRDuZrPvH3D8.woff2) format('woff2');
31 | unicode-range: U+0370-03FF;
32 | }
33 | /* vietnamese */
34 | @font-face {
35 | font-family: 'Roboto';
36 | font-style: normal;
37 | font-weight: 300;
38 | src: local('Roboto Light'), local('Roboto-Light'), url(http://fonts.gstatic.com/s/roboto/v15/NYDWBdD4gIq26G5XYbHsFIX0hVgzZQUfRDuZrPvH3D8.woff2) format('woff2');
39 | unicode-range: U+0102-0103, U+1EA0-1EF9, U+20AB;
40 | }
41 | /* latin-ext */
42 | @font-face {
43 | font-family: 'Roboto';
44 | font-style: normal;
45 | font-weight: 300;
46 | src: local('Roboto Light'), local('Roboto-Light'), url(http://fonts.gstatic.com/s/roboto/v15/Pru33qjShpZSmG3z6VYwnYX0hVgzZQUfRDuZrPvH3D8.woff2) format('woff2');
47 | unicode-range: U+0100-024F, U+1E00-1EFF, U+20A0-20AB, U+20AD-20CF, U+2C60-2C7F, U+A720-A7FF;
48 | }
49 | /* latin */
50 | @font-face {
51 | font-family: 'Roboto';
52 | font-style: normal;
53 | font-weight: 300;
54 | src: local('Roboto Light'), local('Roboto-Light'), url(http://fonts.gstatic.com/s/roboto/v15/Hgo13k-tfSpn0qi1SFdUfZBw1xU1rKptJj_0jans920.woff2) format('woff2');
55 | unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2212, U+2215;
56 | }
57 | /* cyrillic-ext */
58 | @font-face {
59 | font-family: 'Roboto';
60 | font-style: normal;
61 | font-weight: 400;
62 | src: local('Roboto'), local('Roboto-Regular'), url(http://fonts.gstatic.com/s/roboto/v15/sTdaA6j0Psb920Vjv-mrzH-_kf6ByYO6CLYdB4HQE-Y.woff2) format('woff2');
63 | unicode-range: U+0460-052F, U+20B4, U+2DE0-2DFF, U+A640-A69F;
64 | }
65 | /* cyrillic */
66 | @font-face {
67 | font-family: 'Roboto';
68 | font-style: normal;
69 | font-weight: 400;
70 | src: local('Roboto'), local('Roboto-Regular'), url(http://fonts.gstatic.com/s/roboto/v15/uYECMKoHcO9x1wdmbyHIm3-_kf6ByYO6CLYdB4HQE-Y.woff2) format('woff2');
71 | unicode-range: U+0400-045F, U+0490-0491, U+04B0-04B1, U+2116;
72 | }
73 | /* greek-ext */
74 | @font-face {
75 | font-family: 'Roboto';
76 | font-style: normal;
77 | font-weight: 400;
78 | src: local('Roboto'), local('Roboto-Regular'), url(http://fonts.gstatic.com/s/roboto/v15/tnj4SB6DNbdaQnsM8CFqBX-_kf6ByYO6CLYdB4HQE-Y.woff2) format('woff2');
79 | unicode-range: U+1F00-1FFF;
80 | }
81 | /* greek */
82 | @font-face {
83 | font-family: 'Roboto';
84 | font-style: normal;
85 | font-weight: 400;
86 | src: local('Roboto'), local('Roboto-Regular'), url(http://fonts.gstatic.com/s/roboto/v15/_VYFx-s824kXq_Ul2BHqYH-_kf6ByYO6CLYdB4HQE-Y.woff2) format('woff2');
87 | unicode-range: U+0370-03FF;
88 | }
89 | /* vietnamese */
90 | @font-face {
91 | font-family: 'Roboto';
92 | font-style: normal;
93 | font-weight: 400;
94 | src: local('Roboto'), local('Roboto-Regular'), url(http://fonts.gstatic.com/s/roboto/v15/NJ4vxlgWwWbEsv18dAhqnn-_kf6ByYO6CLYdB4HQE-Y.woff2) format('woff2');
95 | unicode-range: U+0102-0103, U+1EA0-1EF9, U+20AB;
96 | }
97 | /* latin-ext */
98 | @font-face {
99 | font-family: 'Roboto';
100 | font-style: normal;
101 | font-weight: 400;
102 | src: local('Roboto'), local('Roboto-Regular'), url(http://fonts.gstatic.com/s/roboto/v15/Ks_cVxiCiwUWVsFWFA3Bjn-_kf6ByYO6CLYdB4HQE-Y.woff2) format('woff2');
103 | unicode-range: U+0100-024F, U+1E00-1EFF, U+20A0-20AB, U+20AD-20CF, U+2C60-2C7F, U+A720-A7FF;
104 | }
105 | /* latin */
106 | @font-face {
107 | font-family: 'Roboto';
108 | font-style: normal;
109 | font-weight: 400;
110 | src: local('Roboto'), local('Roboto-Regular'), url(http://fonts.gstatic.com/s/roboto/v15/oMMgfZMQthOryQo9n22dcuvvDin1pK8aKteLpeZ5c0A.woff2) format('woff2');
111 | unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2212, U+2215;
112 | }
113 |
--------------------------------------------------------------------------------
/docs/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
11 |
12 | 
56 |
57 | 
