├── .gitignore ├── LICENSE ├── Presentation.pdf ├── README.md ├── Vagrantfile ├── docker-compose.yml ├── indexed_storage └── S2_MSI_L2A │ ├── sen2cor_ard.yaml │ └── sen2cor_prepare.py └── notebooks ├── 1. Sentinel 2 - Product definition.ipynb ├── 2. Sentinel 2 - Image Indexing.ipynb ├── 3. Sentinel 2 - Normalized Vegetation Index (NDVI).ipynb └── img ├── ndvi_example.png └── xarray.png /.gitignore: -------------------------------------------------------------------------------- 1 | .vagrant 2 | .ipynb_checkpoints 3 | *.zip 4 | *.SAFE 5 | *.tif -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Aurelio Vivas 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Presentation.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DonAurelio/geopython-2021/0e92162e9b58008b82f3d2ce00263e9c6e48069e/Presentation.pdf -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Introduction to the Open Data Cube: GeoPython 2021 2 | 3 | Welcome to this repo, here you will find all the material related to the talk. 4 | 5 | 1. Presetation 6 | 2. References 7 | 3. Deploy your own Data Cube 8 | 9 | ## Presenation 10 | 11 | Feel free to take at look at the `Presentation.pdf` file. 12 | 13 | ## References 14 | 15 | We cover from 1 to 5 during the talk. Reference 6 have more introductory notebooks from the DEA datacube. Finally, there are amazing Geoscience-BigData related projects that may be interesting to explore, references 7 and 8 cover one of these projects. 16 | 17 | 1. [What is the Open Data Cube?](https://medium.com/opendatacube/what-is-open-data-cube-805af60820d7) 18 | 2. [Imaging the Past](https://landsat.gsfc.nasa.gov/article/imaging-past) 19 | 3. [Open Data Cube Web Page](https://www.opendatacube.org/) 20 | 4. [Open Data Cube Manual](https://datacube-core.readthedocs.io/en/latest/) 21 | 5. [Cube in a Box](https://www.opendatacube.org/ciab) 22 | 6. [Digital Earth Australia notebooks and tools repository](https://github.com/GeoscienceAustralia/dea-notebooks) 23 | 7. [Pangeo: A community platform for Big Data geoscience](https://pangeo.io/) 24 | 8. [Pangeo History](https://medium.com/pangeo/pangeo-2-0-2bedf099582d) 25 | 26 | ## Deploy your own Data Cube 27 | 28 | Here I will show you how you can deploy your own instance of the open datacube using the configuration files I have prepared for you. So let's do it, I promise it won't be painful. 29 | 30 | This configuration file were made to fulfil the purpose of this talk, then if you are considering to continue the exploration of the open data cube use the [Cube in a Box](https://www.opendatacube.org/ciab) alternative. 31 | 32 | ### Requirements 33 | 34 | 1. Computer with at least 8 GB of RAM memory. 35 | 2. Windows or Linux OS 36 | 37 | ### Preliminary configuration 38 | 39 | 1. Download and install [Virtualbox 6.1.18](https://download.virtualbox.org/virtualbox/6.1.18/VirtualBox-6.1.18-142142-Win.exe) 40 | 2. Download and install [Vagrant 2.2.15](https://releases.hashicorp.com/vagrant/2.2.15/vagrant_2.2.15_x86_64.msi) 41 | 3. Reboot your computer 42 | 43 | ### Deployment of a Virtual Machine with Vagrant 44 | 45 | Vangrant is a tool that automates the deployment of virtual machines. All the details and configuration of the virtual machine is already set for me in the `Vagrantfile`, you can take a look on this. On this file I configured an ubuntu based virtual machine, this machine will use two host ports the `8080` for Jupyter Notebooks and `8081` for the open data cube explorer. Make sure this ports are not used by other application on your computer. 46 | 47 | We use a virtual machine in order to easy the deployment of containers and avoid other tecnical details that can take place during the deployment, since this virtual machine is already prepared to do that. However, if you are familiar with containers you can follow up to the 4 step and go to the next section to deploy the open data cube containers directly on your system. 48 | 49 | 1. Download [geopython-2021-main.zip](https://github.com/DonAurelio/geopython-2021/archive/refs/heads/main.zip) repository. 50 | 2. Extract the ZIP file. 51 | 3. Get into the `geopython-2021-main` folder, make sure you can see the `Vagrantfile` on this folder. 52 | 4. Being located in the directory `geopython-2021-main`, open a PowerShell console (in Windows) or the Terminal (in Linux). 53 | 5. Use the following command in order to start a Virtual Machine (VM) automatically 54 | 55 | ```bash 56 | vagrant up --provision 57 | ``` 58 | 59 | The `vagrant up --provision` command will perform the following steps: (1) look for the `Vagrantfile` that is located in your current directory, (2) it will read the specifications set on this file in order configure the Virtual Machine for you, (3) Download the ubuntu VM image from the Vagrant Cloud, and (4) run the ubuntu VM on your computer. The `--provision` flag will will tell vagrant to install `docker` and `docker-compose` as is specified in the `Vagrantfile`. Docker and docker-compose are the tools requiered to deploy the open data cube containers. 60 | 61 | **NOTE:** The `--provision` flag is used only the first time you deploy the virtual machine. 62 | 63 | #### Vagrant Usefull Commands 64 | 65 | * *stop* the virtual machine use `vagrant halt` 66 | * *start* the virtual machine use `vagrant up` 67 | * *enter* the virtual machine console use `vagrant ssh` 68 | * *exit* the virtual machine `exit` 69 | * *destroy* the virtual machine use `vagrant destroy`, if you do this, the next time you need to use the virtual machine, use the `vagrant up --provision` command. 70 | 71 | ### Deployment of Open Data Cube Containers 72 | 73 | Assuming you did step 5 of the previous section, use the `vagrant ssh` to get into the virtual machine you created in the previuos section. Then, use the following commands to deploy the open data cube containers. 74 | 75 | Get into the `/vagrant` directory 76 | 77 | ```bash 78 | vagrant@vagrant:~$ cd /vagrant 79 | ``` 80 | 81 | Check if the `docker-compose.yml` is present in the current directory. 82 | 83 | ```bash 84 | vagrant@vagrant:/vagrant$ ls -l 85 | 86 | total 1780 87 | -rwxrwxrwx 1 vagrant vagrant 1268 Apr 14 23:59 docker-compose.yml 88 | ... 89 | ``` 90 | 91 | Start postgis and jupyter containers 92 | 93 | ```bash 94 | vagrant@vagrant:~$ sudo docker-compose up -d 95 | ``` 96 | Check that all containers are working properly 97 | 98 | ```bash 99 | vagrant@vagrant:~$ sudo docker-compose ps 100 | ``` 101 | 102 | Initialize the data cube database 103 | 104 | ```bash 105 | vagrant@vagrant:~$ sudo docker-compose exec jupyter datacube system init 106 | 107 | Initialising database... 108 | Created. 109 | Checking indexes/views. 110 | Done. 111 | ``` 112 | 113 | Open the browser at [http://localhost:8080](http://localhost:8080) to display the JupyterLab interface. Then, 114 | follow the instructions given in the notebooks. Enjoy your journey !! 115 | 116 | 1. Sentinel 2 - Product definition.ipynb 117 | 2. Sentinel 2 - Image Indexing.ipynb 118 | 3. Sentinel 2 - Normalized Vegetation Index (NDVI).ipynb -------------------------------------------------------------------------------- /Vagrantfile: -------------------------------------------------------------------------------- 1 | # -*- mode: ruby -*- 2 | # vi: set ft=ruby : 3 | 4 | # All Vagrant configuration is done below. The "2" in Vagrant.configure 5 | # configures the configuration version (we support older styles for 6 | # backwards compatibility). Please don't change it unless you know what 7 | # you're doing. 8 | Vagrant.configure("2") do |config| 9 | # The most common configuration options are documented and commented below. 10 | # For a complete reference, please see the online documentation at 11 | # https://docs.vagrantup.com. 12 | 13 | # Every Vagrant development environment requires a box. You can search for 14 | # boxes at https://vagrantcloud.com/search. 15 | config.vm.box = "hashicorp/bionic64" 16 | 17 | # Disable automatic box update checking. If you disable this, then 18 | # boxes will only be checked for updates when the user runs 19 | # `vagrant box outdated`. This is not recommended. 20 | # config.vm.box_check_update = false 21 | 22 | # Create a forwarded port mapping which allows access to a specific port 23 | # within the machine from a port on the host machine. In the example below, 24 | # accessing "localhost:8080" will access port 80 on the guest machine. 25 | # NOTE: This will enable public access to the opened port 26 | config.vm.network "forwarded_port", guest: 8081, host: 8081 27 | 28 | # Create a forwarded port mapping which allows access to a specific port 29 | # within the machine from a port on the host machine and only allow access 30 | # via 127.0.0.1 to disable public access 31 | # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1" 32 | 33 | # Create a private network, which allows host-only access to the machine 34 | # using a specific IP. 35 | # config.vm.network "private_network", ip: "192.168.33.10" 36 | 37 | # Create a public network, which generally matched to bridged network. 38 | # Bridged networks make the machine appear as another physical device on 39 | # your network. 40 | # config.vm.network "public_network" 41 | 42 | # Share an additional folder to the guest VM. The first argument is 43 | # the path on the host to the actual folder. The second argument is 44 | # the path on the guest to mount the folder. And the optional third 45 | # argument is a set of non-required options. 46 | # config.vm.synced_folder "../data", "/vagrant_data" 47 | 48 | # Provider-specific configuration so you can fine-tune various 49 | # backing providers for Vagrant. These expose provider-specific options. 50 | # Example for VirtualBox: 51 | # 52 | config.vm.provider "virtualbox" do |vb| 53 | # Display the VirtualBox GUI when booting the machine 54 | # vb.gui = true 55 | 56 | # Customize the amount of memory on the VM: 57 | vb.memory = "4096" 58 | end 59 | # 60 | # View the documentation for the provider you are using for more 61 | # information on available options. 62 | 63 | # Enable provisioning with a shell script. Additional provisioners such as 64 | # Ansible, Chef, Docker, Puppet and Salt are also available. Please see the 65 | # documentation for more information about their specific syntax and use. 66 | config.vm.provision "shell", inline: <<-SHELL 67 | sudo apt-get update 68 | sudo apt-get install -y unzip 69 | sudo apt-get install -y docker.io 70 | sudo usermod -aG docker $USER 71 | sudo curl -s -L "https://github.com/docker/compose/releases/download/1.29.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose 72 | sudo chmod +x /usr/local/bin/docker-compose 73 | SHELL 74 | 75 | end 76 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | 4 | postgis: 5 | image: postgis/postgis:10-2.5 6 | environment: 7 | - POSTGRES_DB=datacube 8 | - POSTGRES_USER=datacube 9 | - POSTGRES_PASSWORD=datacube 10 | volumes: 11 | - postgis-db-volume:/var/lib/postgresql/data 12 | restart: always 13 | 14 | jupyter: 15 | image: donaurelio/datacube:1.8.2-ubuntu-20.04 16 | environment: 17 | - PATH=$PATH:/home/datacube/.local/bin 18 | # open data cube specific variables 19 | - DB_DATABASE=datacube 20 | - DB_HOSTNAME=postgis 21 | - DB_USERNAME=datacube 22 | - DB_PASSWORD=datacube 23 | volumes: 24 | - /vagrant:/vagrant 25 | command: bash -c "cd /vagrant && jupyter-lab --no-browser --LabApp.token='' --port=8081 --ip=0.0.0.0 --allow-root" 26 | ports: 27 | - 8081:8081 28 | restart: always 29 | 30 | volumes: 31 | postgis-db-volume: 32 | -------------------------------------------------------------------------------- /indexed_storage/S2_MSI_L2A/sen2cor_ard.yaml: -------------------------------------------------------------------------------- 1 | # This file describes Sentinel 2 Data, processed through sen2cor 2 | 3 | name: s2_sen2cor_ard_granule_EO3 4 | description: Sentinel-L2A MSI ARD - NBAR NBART and Pixel Quality 5 | metadata_type: eo3 6 | 7 | metadata: 8 | product: 9 | name: s2_sen2cor_ard_granule_EO3 10 | 11 | measurements: 12 | - name: 'B01_60m' 13 | aliases: [band_1, coastal_aerosol] 14 | dtype: uint16 15 | nodata: 0 16 | units: 'reflectance' 17 | - name: 'B02_10m' 18 | aliases: [band_2, blue] 19 | dtype: uint16 20 | nodata: 0 21 | units: 'reflectance' 22 | - name: 'B03_10m' 23 | aliases: [band_3, green] 24 | dtype: uint16 25 | nodata: 0 26 | units: 'reflectance' 27 | - name: 'B04_10m' 28 | aliases: [band_4, red] 29 | dtype: uint16 30 | nodata: 0 31 | units: 'reflectance' 32 | - name: 'B05_20m' 33 | aliases: [band_5, veg5] 34 | dtype: uint16 35 | nodata: 0 36 | units: 'reflectance' 37 | - name: 'B06_20m' 38 | aliases: [band_6, veg6] 39 | dtype: uint16 40 | nodata: 0 41 | units: 'reflectance' 42 | - name: 'B07_20m' 43 | aliases: [band_7, veg7] 44 | dtype: uint16 45 | nodata: 0 46 | units: 'reflectance' 47 | - name: 'B08_10m' 48 | aliases: [band_8, nir] 49 | dtype: uint16 50 | nodata: 0 51 | units: 'reflectance' 52 | - name: 'B8A_20m' 53 | aliases: [band_8a, narrow_nir] 54 | dtype: uint16 55 | nodata: 0 56 | units: 'reflectance' 57 | - name: 'B09_60m' 58 | aliases: [band_9, water_vapour] 59 | dtype: uint16 60 | nodata: 0 61 | units: 'reflectance' 62 | - name: 'B11_20m' 63 | aliases: [band_11, swir1] 64 | dtype: uint16 65 | nodata: 0 66 | units: 'reflectance' 67 | - name: 'B12_20m' 68 | aliases: [band_12, swir2] 69 | dtype: uint16 70 | nodata: 0 71 | units: 'reflectance' 72 | - name: 'SCL_20m' 73 | aliases: [scl] 74 | dtype: uint8 75 | nodata: 0 76 | units: '1' 77 | flags_definition: 78 | scl: 79 | bits: [1,2,3,4,5,6,7,8,9,10,11] 80 | description: Scene classification 81 | values: 82 | 1: Saturated or defective pixel 83 | 2: Dark features / Shadows 84 | 3: Cloud shadows 85 | 4: Vegetation 86 | 5: Not vegetated 87 | 6: Water 88 | 7: Unclassified 89 | 8: Cloud medium probability 90 | 9: cloud high probability 91 | 10: Thin cirrus 92 | 11: Snow or ice 93 | -------------------------------------------------------------------------------- /indexed_storage/S2_MSI_L2A/sen2cor_prepare.py: -------------------------------------------------------------------------------- 1 | # coding=utf-8 2 | """ 3 | Ingest data from the command-line. 4 | """ 5 | from __future__ import absolute_import 6 | 7 | import logging 8 | import os 9 | import uuid 10 | from pathlib import Path 11 | from xml.etree import ElementTree 12 | 13 | import click 14 | import rasterio.features 15 | import shapely.affinity 16 | import shapely.geometry 17 | import shapely.ops 18 | import yaml 19 | from osgeo import osr 20 | from rasterio.errors import RasterioIOError 21 | 22 | 23 | # image boundary imports 24 | 25 | 26 | # IMAGE BOUNDARY CODE 27 | 28 | def safe_valid_region(images, mask_value=None): 29 | try: 30 | return valid_region(images, mask_value) 31 | except (OSError, RasterioIOError): 32 | return None 33 | 34 | 35 | def valid_region(images, mask_value=None): 36 | mask = None 37 | for fname in images: 38 | # ensure formats match 39 | with rasterio.open(str(fname), 'r') as ds: 40 | transform = ds.affine 41 | 42 | img = ds.read(1) 43 | 44 | if mask_value is not None: 45 | new_mask = img & mask_value == mask_value 46 | else: 47 | # TODO update when sen2cor format write finalised new_mask = img != ds.nodata 48 | new_mask = img != 0 49 | if mask is None: 50 | mask = new_mask 51 | else: 52 | mask |= new_mask 53 | 54 | shapes = rasterio.features.shapes(mask.astype('uint8'), mask=mask) 55 | shape = shapely.ops.unary_union([shapely.geometry.shape(shape) for shape, val in shapes if val == 1]) 56 | type(shapes) 57 | 58 | geom = shape.convex_hull 59 | 60 | # buffer by 1 pixel 61 | geom = geom.buffer(1, join_style=3, cap_style=3) 62 | 63 | # simplify with 1 pixel radius 64 | geom = geom.simplify(1) 65 | 66 | # intersect with image bounding box 67 | geom = geom.intersection(shapely.geometry.box(0, 0, mask.shape[1], mask.shape[0])) 68 | 69 | # transform from pixel space into CRS space 70 | geom = shapely.affinity.affine_transform(geom, (transform.a, transform.b, transform.d, 71 | transform.e, transform.xoff, transform.yoff)) 72 | 73 | output = shapely.geometry.mapping(geom) 74 | 75 | return geom 76 | 77 | 78 | def _to_lists(x): 79 | """ 80 | Returns lists of lists when given tuples of tuples 81 | """ 82 | if isinstance(x, tuple): 83 | return [_to_lists(el) for el in x] 84 | 85 | return x 86 | 87 | def get_size(root, res): 88 | nrows = int(root.findall('./*/Tile_Geocoding/Size[@resolution="'+str(res)+'"]/NROWS')[0].text) 89 | ncols = int(root.findall('./*/Tile_Geocoding/Size[@resolution="'+str(res)+'"]/NCOLS')[0].text) 90 | return { 91 | 'nrows': nrows, 92 | 'ncols': ncols, 93 | } 94 | 95 | def safe_get_grids(image): 96 | try: 97 | return get_grids(image) 98 | except (OSError, RasterioIOError): 99 | return None 100 | 101 | def get_grids(image): 102 | src = rasterio.open(str(image), 'r') 103 | shape = src.shape 104 | transform = src.transform 105 | return { 106 | 'shape': shape, 107 | 'transform': [x for x in transform] 108 | } 109 | 110 | 111 | def get_geo_ref_points(root): 112 | nrows = int(root.findall('./*/Tile_Geocoding/Size[@resolution="10"]/NROWS')[0].text) 113 | ncols = int(root.findall('./*/Tile_Geocoding/Size[@resolution="10"]/NCOLS')[0].text) 114 | 115 | ulx = int(root.findall('./*/Tile_Geocoding/Geoposition[@resolution="10"]/ULX')[0].text) 116 | uly = int(root.findall('./*/Tile_Geocoding/Geoposition[@resolution="10"]/ULY')[0].text) 117 | 118 | xdim = int(root.findall('./*/Tile_Geocoding/Geoposition[@resolution="10"]/XDIM')[0].text) 119 | ydim = int(root.findall('./*/Tile_Geocoding/Geoposition[@resolution="10"]/YDIM')[0].text) 120 | 121 | return { 122 | 'ul': {'x': ulx, 'y': uly}, 123 | 'ur': {'x': ulx + ncols * abs(xdim), 'y': uly}, 124 | 'll': {'x': ulx, 'y': uly - nrows * abs(ydim)}, 125 | 'lr': {'x': ulx + ncols * abs(xdim), 'y': uly - nrows * abs(ydim)}, 126 | } 127 | 128 | 129 | def get_coords(geo_ref_points, spatial_ref): 130 | t = osr.CoordinateTransformation(spatial_ref, spatial_ref.CloneGeogCS()) 131 | 132 | def transform(p): 133 | lon, lat, z = t.TransformPoint(p['x'], p['y']) 134 | return {'lon': lon, 'lat': lat} 135 | 136 | return {key: transform(p) for key, p in geo_ref_points.items()} 137 | 138 | 139 | def prepare_dataset(path): 140 | root = ElementTree.parse(str(path)).getroot() 141 | level = root.findall('./*/Product_Info/PROCESSING_LEVEL')[0].text 142 | product_type = root.findall('./*/Product_Info/PRODUCT_TYPE')[0].text 143 | ct_time = root.findall('./*/Product_Info/GENERATION_TIME')[0].text 144 | print(level, product_type, ct_time) 145 | # granuleslist = [(granule.get('granuleIdentifier'), [imid.text for imid in granule.findall('IMAGE_FILE')]) for 146 | # granule in 147 | # root.findall('./*/Product_Info/Product_Organisation/Granule_List/Granules')] 148 | # Assume multiple granules 149 | single_granule_archive = False 150 | granules = {granule.get('granuleIdentifier'): [imid.text for imid in granule.findall('IMAGE_ID')] 151 | for granule in root.findall('./*/Product_Info/Product_Organisation/Granule_List/Granules')} 152 | if not granules: 153 | single_granule_archive = True 154 | granules = {granule.get('granuleIdentifier'): [imid.text for imid in granule.findall('IMAGE_FILE')] 155 | for granule in root.findall('./*/Product_Info/Product_Organisation/Granule_List/Granule')} 156 | if not [] in granules.values(): 157 | single_granule_archive = True 158 | else: 159 | granules = {granule.get('granuleIdentifier'): [imid.text for imid in granule.findall('IMAGE_ID')] 160 | for granule in root.findall('./*/Product_Info/Product_Organisation/Granule_List/Granule')} 161 | single_granule_archive = False 162 | 163 | # current = 0 164 | # list = [] 165 | # granules = {} 166 | # for i in granuleslist: 167 | # granules[i[0]] = {} 168 | # for key in granules.keys(): 169 | 170 | # granulecontent = [] 171 | # for j in granuleslist: 172 | 173 | # if key in j: 174 | # granulecontent = granulecontent + j[1] 175 | 176 | # granules[key] = granulecontent 177 | grouped_images = [] 178 | documents = [] 179 | for granule_id, images in granules.items(): 180 | images_ten_list = [] 181 | images_twenty_list = [] 182 | images_sixty_list = [] 183 | images_classification = [] 184 | # gran_path = str(path.parent.joinpath('GRANULE', granule_id, granule_id[:-7].replace('MSI', 'MTD') + '.xml')) 185 | img_data_path = str(path.parent.joinpath('GRANULE', granule_id, 'IMG_DATA')) 186 | 187 | gran_path = str(path.parent.joinpath('GRANULE', granule_id, granule_id[:-7].replace('MSI', 'MTD') + '.xml')) 188 | if not Path(gran_path).exists(): 189 | gran_path = str(path.parent.joinpath(images[0])) 190 | gran_path = str(Path(gran_path).parents[2].joinpath('MTD_TL.xml')) 191 | root = ElementTree.parse(gran_path).getroot() 192 | 193 | if not Path(img_data_path).exists(): 194 | # img_data_path = str(Path(gran_path).parents[0].joinpath('IMG_DATA')) 195 | img_data_path = str(Path(path).parent) 196 | 197 | if single_granule_archive is False: 198 | img_data_path = img_data_path + str(Path('GRANULE').joinpath(granule_id, 'IMG_DATA')) 199 | 200 | root = ElementTree.parse(gran_path).getroot() 201 | sensing_time = root.findall('./*/SENSING_TIME')[0].text 202 | img_data_path = str(path.parent.joinpath('GRANULE', granule_id, 'IMG_DATA')) 203 | img_data_path_r10 = str(path.parent.joinpath('GRANULE', granule_id, 'IMG_DATA', 'R10m')) 204 | img_data_path_r20 = str(path.parent.joinpath('GRANULE', granule_id, 'IMG_DATA', 'R20m')) 205 | img_data_path_r60 = str(path.parent.joinpath('GRANULE', granule_id, 'IMG_DATA', 'R60m')) 206 | 207 | for image in images: 208 | # print('IMAGE',image) 209 | # print('img_data_path', img_data_path) 210 | # image = str(Path(image).name) 211 | # print('IMAGE',image) 212 | classification_list = ['SCL'] 213 | ten_list = ['B02_10m', 'B03_10m', 'B04_10m', 'B08_10m'] 214 | twenty_list = ['B05_20m', 'B06_20m', 'B07_20m', 'B11_20m', 'B12_20m', 'B8A_20m', 215 | 'B02_20m', 'B03_20m', 'B04_20m'] 216 | sixty_list = ['B01_60m', 'B02_60m', 'B03_60m', 'B04_60m', 'B8A_60m', 'B09_60m', 217 | 'B05_60m', 'B06_60m', 'B07_60m', 'B11_60m', 'B12_60m'] 218 | 219 | for item in classification_list: 220 | if item in image: 221 | # TODO include 60m classification 222 | if '20m' in image: 223 | images_classification.append(os.path.join(str(path.parent), image + ".jp2")) 224 | 225 | for item in ten_list: 226 | if item in image: 227 | images_ten_list.append(os.path.join(str(path.parent), image + ".jp2")) 228 | grouped_images.append(os.path.join(str(path.parent), image + ".jp2")) 229 | for item in twenty_list: 230 | if item in image: 231 | images_twenty_list.append(os.path.join(str(path.parent), image + ".jp2")) 232 | grouped_images.append(os.path.join(str(path.parent), image + ".jp2")) 233 | for item in sixty_list: 234 | if item in image: 235 | images_sixty_list.append(os.path.join(str(path.parent), image + ".jp2")) 236 | grouped_images.append(os.path.join(str(path.parent), image + ".jp2")) 237 | 238 | station = root.findall('./*/Archiving_Info/ARCHIVING_CENTRE')[0].text 239 | 240 | cs_code = root.findall('./*/Tile_Geocoding/HORIZONTAL_CS_CODE')[0].text 241 | spatial_ref = osr.SpatialReference() 242 | 243 | pvi_fileName = str(root.findall('./*/PVI_FILENAME')).split("/")[-1] 244 | images_naming = pvi_fileName[0: -7] 245 | 246 | 247 | spatial_ref.SetFromUserInput(cs_code) 248 | 249 | spectral_dict = {image[-11:-4]: {'path': str(Path(image)), 'layer': 1, } for image in grouped_images} 250 | scl_dict = {'SCL_20m': {'path': str(Path(classification)), 'layer': 1, } for classification in 251 | images_classification} 252 | spectral_dict.update(scl_dict) 253 | 254 | geo_ref_points = get_geo_ref_points(root) 255 | 256 | documents.append({ 257 | 'id': str(uuid.uuid4()), 258 | '$schema': 'https://schemas.opendatacube.org/dataset', 259 | 'product': {'name': 's2_sen2cor_ard_granule_EO3'}, 260 | 'crs': cs_code, 261 | 'grids': {'default': safe_get_grids(images_twenty_list[0]), '10m_res': safe_get_grids(images_ten_list[0]), '60m_res': safe_get_grids(images_sixty_list[0])}, 262 | 'measurements': { 'B01_60m': {'grid': '60m_res', 'path': spectral_dict['B01_60m']['path']}, 263 | 'B02_10m': {'grid': '10m_res', 'path': spectral_dict['B02_10m']['path']}, 264 | 'B03_10m': {'grid': '10m_res', 'path': spectral_dict['B03_10m']['path']}, 265 | 'B04_10m': {'grid': '10m_res', 'path': spectral_dict['B04_10m']['path']}, 266 | 'B05_20m': {'path': spectral_dict['B05_20m']['path']}, 267 | 'B06_20m': {'path': spectral_dict['B06_20m']['path']}, 268 | 'B07_20m': {'path': spectral_dict['B07_20m']['path']}, 269 | 'B08_10m': {'grid': '10m_res', 'path': spectral_dict['B08_10m']['path']}, 270 | 'B8A_20m': {'path': spectral_dict['B8A_20m']['path']}, 271 | 'B09_60m': {'grid': '60m_res', 'path': spectral_dict['B09_60m']['path']}, 272 | 'B11_20m': {'path': spectral_dict['B11_20m']['path']}, 273 | 'B12_20m': {'path': spectral_dict['B12_20m']['path']}, 274 | 'SCL_20m': {'path': spectral_dict['SCL_20m']['path']} 275 | }, 276 | 'properties': {'eo:platform': 'Sentinel-2A', 277 | 'eo:instrument': 'MSI', 278 | 'eo:product_type': product_type, 279 | 'datetime': ct_time, 280 | 'odc:file_format': 'JPEG2000', 281 | 'dea:dataset_maturity': 'final', 282 | 'odc:product_family': 'ard' 283 | 284 | }, 285 | }) 286 | return documents 287 | 288 | 289 | @click.command( 290 | help="Prepare Sentinel 2 L2 sen2cor dataset SR and SC for ingestion into the Data Cube. " 291 | "eg. python sen2cor_prepare.py .SAFE --output .yaml") 292 | @click.argument('datasets', 293 | type=click.Path(exists=True, readable=True, writable=False), 294 | nargs=-1) 295 | @click.option('--output', help="Write datasets into this directory", 296 | type=click.Path(exists=False, writable=True, dir_okay=True)) 297 | def main(datasets, output): 298 | logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s', level=logging.INFO) 299 | 300 | for dataset in datasets: 301 | 302 | path = Path(dataset).absolute() 303 | if path.is_dir(): 304 | # path = Path(path.joinpath(path.stem.replace('PRD_MSIL2A', 'MTD_SAFL2A') + '.xml')) 305 | for file in os.listdir(path): 306 | if file.endswith(".xml"): 307 | if file.startswith("MTD"): 308 | path = Path(os.path.join(path, file)) 309 | if path.suffix != '.xml': 310 | raise RuntimeError('want xml') 311 | 312 | logging.info("Processing %s", path) 313 | 314 | documents = prepare_dataset(path) 315 | 316 | output_path = Path(output) 317 | if 'xml' in str(path): 318 | yaml_path = output_path.joinpath(path.parent.name + '.yaml') 319 | else: 320 | yaml_path = output_path.joinpath(path.name + '.yaml') 321 | 322 | if documents: 323 | logging.info("Writing %s dataset(s) into %s", len(documents), yaml_path) 324 | with open(yaml_path, 'w') as stream: 325 | yaml.safe_dump_all(documents, stream, sort_keys=False) 326 | else: 327 | logging.info("No datasets discovered. Bye!") 328 | 329 | 330 | if __name__ == "__main__": 331 | main() 332 | -------------------------------------------------------------------------------- /notebooks/1. Sentinel 2 - Product definition.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Sentinel 2 - Product definition\n", 8 | "\n", 9 | "The datacube needs to know about the properties of the scenes produced for every satellite we are interested in. As every satellite takes different **measurements** (a.k.a spectral **bands**), the datacube needs to care about organizing data of the same satellite together. For this reason, for every satellite a **Product** must be defined in the datacube. To define a product we require a Product Description File.\n", 10 | "\n", 11 | "* **Product Description File:** tells the datacube which data contains every scene, i.e., which bands (measurements) can be found. This metadata will be used by the data cube to create a bunch of database tables to hold relevant information about the scenes that facilitates subsequent queries." 12 | ] 13 | }, 14 | { 15 | "cell_type": "markdown", 16 | "metadata": {}, 17 | "source": [ 18 | "### Product Definition\n", 19 | "\n", 20 | "Check if there are already products defined in the datacube. Of course you have not defined produts yet so the next command will not display information." 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": 1, 26 | "metadata": {}, 27 | "outputs": [], 28 | "source": [ 29 | "!datacube product list" 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "metadata": {}, 35 | "source": [ 36 | "Use the **product definition file** located at `/vagrant/indexed_storage/S2_MSI_L2A/sen2cor_ard.yaml` to define the Sentinel 2 product." 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 15, 42 | "metadata": {}, 43 | "outputs": [ 44 | { 45 | "name": "stdout", 46 | "output_type": "stream", 47 | "text": [ 48 | "Adding \"s2_sen2cor_ard_granule_EO3\" (this might take a while) DONE\n" 49 | ] 50 | } 51 | ], 52 | "source": [ 53 | "!datacube product add /vagrant/indexed_storage/S2_MSI_L2A/sen2cor_ard.yaml" 54 | ] 55 | }, 56 | { 57 | "cell_type": "markdown", 58 | "metadata": {}, 59 | "source": [ 60 | "List the **products** created on the datacube." 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 16, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "\u001b[32ms2_sen2cor_ard_granule_EO3\u001b[0m Sentinel-L2A MSI ARD - NBAR NBART and Pixel Quality\n" 73 | ] 74 | } 75 | ], 76 | "source": [ 77 | "!datacube product list" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "Remmeber the `s2_sen2cor_ard_granule_EO3` since this is the identifier of the Sentinel 2 product.This identifier is defined in the product definition file `/vagrant/indexed_storage/S2_MSI_L2A/sen2cor_ard.yaml`." 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "metadata": {}, 90 | "source": [ 91 | "### References\n", 92 | "\n", 93 | "1. [Open Data Cube Product](https://datacube-core.readthedocs.io/en/latest/architecture/data_model.html#product)" 94 | ] 95 | } 96 | ], 97 | "metadata": { 98 | "kernelspec": { 99 | "display_name": "Python 3", 100 | "language": "python", 101 | "name": "python3" 102 | }, 103 | "language_info": { 104 | "codemirror_mode": { 105 | "name": "ipython", 106 | "version": 3 107 | }, 108 | "file_extension": ".py", 109 | "mimetype": "text/x-python", 110 | "name": "python", 111 | "nbconvert_exporter": "python", 112 | "pygments_lexer": "ipython3", 113 | "version": "3.8.5" 114 | } 115 | }, 116 | "nbformat": 4, 117 | "nbformat_minor": 4 118 | } 119 | -------------------------------------------------------------------------------- /notebooks/2. Sentinel 2 - Image Indexing.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Sentinel 2 - Image Indexing\n", 8 | "\n", 9 | "**NOTE:** You can download the Sentinel 2 Images used on this notebook for [here](https://drive.google.com/drive/folders/1eNWlKCcia3FsnhFA7k61m3pnkFp3OiF6?usp=sharing). Then place the zip files into the `/indexed_storage/S2_MSI_L2A/` directory.\n", 10 | "\n", 11 | "Image indexing consists of telling the data cube where we have images in our file system. On this section we will index Sentinel 2 images.\n", 12 | "\n", 13 | "Images are indexed into the data cube in order that we can access access images' data through the data cube Python API. An image must be prepared to be indexed in the data cube, so we require a preparation script called **Metadata Generation Script** in the open datacube jargon.\n", 14 | "\n", 15 | "* **Metadata Generation Script (.py):** This script is used to generate a metadata file for a given scene. This metadata is used by the datacube to create indexes that easy scene data retrieval." 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "To keep our scenes organized, we will place the downloaded Sentinel 2 images on the following directory `/vagrant/indexed_storage/S2_MSI_L2A/`." 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "Verify that the scene you downloaded earlier is in the `/vagrant/indexed_storage/S2_MSI_L2A/` directory" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": 1, 35 | "metadata": {}, 36 | "outputs": [ 37 | { 38 | "name": "stdout", 39 | "output_type": "stream", 40 | "text": [ 41 | "S2A_MSIL2A_20210130T151701_N0214_R125_T18NYM_20210130T174015.SAFE\n", 42 | "S2A_MSIL2A_20210130T151701_N0214_R125_T18NYM_20210130T174015.zip\n", 43 | "S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.zip\n", 44 | "sen2cor_ard.yaml\n", 45 | "sen2cor_prepare.py\n" 46 | ] 47 | } 48 | ], 49 | "source": [ 50 | "!ls /vagrant/indexed_storage/S2_MSI_L2A" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "Select one of the .zip files you have downloaded and set `image_file_name` with the image identifier as shown in the example below." 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 2, 63 | "metadata": {}, 64 | "outputs": [], 65 | "source": [ 66 | "# Set image_file_name with the image identifier (do not use the .zip extension)\n", 67 | "image_file_name = 'S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459'" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "metadata": {}, 73 | "source": [ 74 | "Unzip the selected image in the `/vagrant/indexed_storage/S2_MSI_L2A/` directory" 75 | ] 76 | }, 77 | { 78 | "cell_type": "code", 79 | "execution_count": 3, 80 | "metadata": {}, 81 | "outputs": [ 82 | { 83 | "name": "stdout", 84 | "output_type": "stream", 85 | "text": [ 86 | "Archive: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.zip\n", 87 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/\n", 88 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/INSPIRE.xml \n", 89 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/rep_info/\n", 90 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/rep_info/S2_PDI_Level-2A_Tile_Metadata.xsd \n", 91 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/rep_info/S2_PDI_Level-2A_Datastrip_Metadata.xsd \n", 92 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/rep_info/S2_User_Product_Level-2A_Metadata.xsd \n", 93 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/HTML/\n", 94 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/HTML/star_bg.jpg \n", 95 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/HTML/banner_1.png \n", 96 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/HTML/banner_2.png \n", 97 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/HTML/UserProduct_index.html \n", 98 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/HTML/UserProduct_index.xsl \n", 99 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/HTML/banner_3.png \n", 100 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/MTD_MSIL2A.xml \n", 101 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/\n", 102 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/\n", 103 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/MTD_TL.xml \n", 104 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/\n", 105 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B05.gml \n", 106 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B12.gml \n", 107 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/SENSOR_QUALITY.xml \n", 108 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B11.gml \n", 109 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B06.gml \n", 110 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B8A.gml \n", 111 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/T18NYM_20210115T151709_PVI.jp2 \n", 112 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B8A.gml \n", 113 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B04.gml \n", 114 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B05.gml \n", 115 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B04.gml \n", 116 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B07.gml \n", 117 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B09.gml \n", 118 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B11.gml \n", 119 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B03.gml \n", 120 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SNWPRB_60m.jp2 \n", 121 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B08.gml \n", 122 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B12.gml \n", 123 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B12.gml \n", 124 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B03.gml \n", 125 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B07.gml \n", 126 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B02.gml \n", 127 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B09.gml \n", 128 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B08.gml \n", 129 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B8A.gml \n", 130 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B04.gml \n", 131 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B09.gml \n", 132 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B03.gml \n", 133 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B08.gml \n", 134 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_CLDPRB_60m.jp2 \n", 135 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B02.gml \n", 136 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B05.gml \n", 137 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_CLOUDS_B00.gml \n", 138 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B09.gml \n", 139 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B10.gml \n", 140 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B06.gml \n", 141 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B02.gml \n", 142 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B08.gml \n", 143 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B03.gml \n", 144 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B03.gml \n", 145 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B02.gml \n", 146 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B09.gml \n", 147 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_CLDPRB_20m.jp2 \n", 148 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B06.gml \n", 149 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B08.gml \n", 150 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B01.gml \n", 151 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B10.gml \n", 152 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/GENERAL_QUALITY.xml \n", 153 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B8A.gml \n", 154 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B05.gml \n", 155 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B11.gml \n", 156 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B10.gml \n", 157 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B01.gml \n", 158 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B10.gml \n", 159 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B07.gml \n", 160 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B05.gml \n", 161 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SNWPRB_20m.jp2 \n", 162 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B12.gml \n", 163 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B06.gml \n", 164 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B11.gml \n", 165 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B07.gml \n", 166 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B04.gml \n", 167 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B07.gml \n", 168 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B8A.gml \n", 169 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_NODATA_B06.gml \n", 170 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B01.gml \n", 171 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B10.gml \n", 172 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B12.gml \n", 173 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/RADIOMETRIC_QUALITY.xml \n", 174 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B02.gml \n", 175 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_TECQUA_B11.gml \n", 176 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/GEOMETRIC_QUALITY.xml \n", 177 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_SATURA_B01.gml \n", 178 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DETFOO_B01.gml \n", 179 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/FORMAT_CORRECTNESS.xml \n", 180 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/QI_DATA/MSK_DEFECT_B04.gml \n", 181 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/AUX_DATA/\n", 182 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/AUX_DATA/AUX_ECMWFT \n", 183 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/\n", 184 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/\n", 185 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_TCI_60m.jp2 \n", 186 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B11_60m.jp2 \n", 187 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B06_60m.jp2 \n", 188 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B02_60m.jp2 \n", 189 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_AOT_60m.jp2 \n", 190 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B8A_60m.jp2 \n", 191 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_WVP_60m.jp2 \n", 192 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B09_60m.jp2 \n", 193 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B01_60m.jp2 \n", 194 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B05_60m.jp2 \n", 195 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_SCL_60m.jp2 \n", 196 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B04_60m.jp2 \n", 197 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B07_60m.jp2 \n", 198 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B12_60m.jp2 \n", 199 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R60m/T18NYM_20210115T151709_B03_60m.jp2 \n", 200 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/\n", 201 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B03_20m.jp2 \n", 202 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B12_20m.jp2 \n", 203 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B02_20m.jp2 \n", 204 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_WVP_20m.jp2 \n", 205 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_SCL_20m.jp2 \n", 206 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B8A_20m.jp2 \n", 207 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B04_20m.jp2 \n", 208 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B06_20m.jp2 \n", 209 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B05_20m.jp2 \n", 210 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B07_20m.jp2 \n", 211 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_AOT_20m.jp2 \n", 212 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_B11_20m.jp2 \n", 213 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R20m/T18NYM_20210115T151709_TCI_20m.jp2 \n", 214 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R10m/\n", 215 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R10m/T18NYM_20210115T151709_B04_10m.jp2 \n", 216 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R10m/T18NYM_20210115T151709_B02_10m.jp2 \n", 217 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R10m/T18NYM_20210115T151709_WVP_10m.jp2 \n", 218 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R10m/T18NYM_20210115T151709_AOT_10m.jp2 \n", 219 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R10m/T18NYM_20210115T151709_B03_10m.jp2 \n", 220 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R10m/T18NYM_20210115T151709_B08_10m.jp2 \n", 221 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/GRANULE/L2A_T18NYM_A020171_20210115T151703/IMG_DATA/R10m/T18NYM_20210115T151709_TCI_10m.jp2 \n", 222 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/AUX_DATA/\n", 223 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/\n", 224 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/DS_VGS2_20210115T174459_S20210115T151703/\n", 225 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/DS_VGS2_20210115T174459_S20210115T151703/MTD_DS.xml \n", 226 | " creating: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/DS_VGS2_20210115T174459_S20210115T151703/QI_DATA/\n", 227 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/DS_VGS2_20210115T174459_S20210115T151703/QI_DATA/SENSOR_QUALITY.xml \n", 228 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/DS_VGS2_20210115T174459_S20210115T151703/QI_DATA/GENERAL_QUALITY.xml \n", 229 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/DS_VGS2_20210115T174459_S20210115T151703/QI_DATA/RADIOMETRIC_QUALITY.xml \n", 230 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/DS_VGS2_20210115T174459_S20210115T151703/QI_DATA/GEOMETRIC_QUALITY.xml \n", 231 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/DATASTRIP/DS_VGS2_20210115T174459_S20210115T151703/QI_DATA/FORMAT_CORRECTNESS.xml \n", 232 | " extracting: /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/manifest.safe \n" 233 | ] 234 | } 235 | ], 236 | "source": [ 237 | "image_zip_path = f'/vagrant/indexed_storage/S2_MSI_L2A/{image_file_name}.zip'\n", 238 | "image_dst_path = '/vagrant/indexed_storage/S2_MSI_L2A/'\n", 239 | "\n", 240 | "!unzip $image_zip_path -d $image_dst_path" 241 | ] 242 | }, 243 | { 244 | "cell_type": "markdown", 245 | "metadata": {}, 246 | "source": [ 247 | "Generate the metadata for the selected image. Use the `/vagrant/indexed_storage/S2_MSI_L2A/sen2cor_prepare.py` metadata preparation script. " 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": 4, 253 | "metadata": {}, 254 | "outputs": [ 255 | { 256 | "name": "stdout", 257 | "output_type": "stream", 258 | "text": [ 259 | "2021-04-15 15:05:24,537 INFO Processing /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/MTD_MSIL2A.xml\n", 260 | "Level-2A S2MSI2A 2021-01-15T17:44:59.000000Z\n", 261 | "2021-04-15 15:05:24,622 INFO Writing 1 dataset(s) into /vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE.yaml\n" 262 | ] 263 | } 264 | ], 265 | "source": [ 266 | "metadata_prepare_script_file_path = '/vagrant/indexed_storage/S2_MSI_L2A/sen2cor_prepare.py'\n", 267 | "unziped_image_path = f'/vagrant/indexed_storage/S2_MSI_L2A/{image_file_name}.SAFE'\n", 268 | "metadata_dst_file_path = f'/vagrant/indexed_storage/S2_MSI_L2A/{image_file_name}.SAFE'\n", 269 | "\n", 270 | "!python3 $metadata_prepare_script_file_path $unziped_image_path --output $metadata_dst_file_path" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "metadata": {}, 276 | "source": [ 277 | "Check if the metadata file **.SAFE.yaml** was generated into the image directory." 278 | ] 279 | }, 280 | { 281 | "cell_type": "code", 282 | "execution_count": 5, 283 | "metadata": {}, 284 | "outputs": [ 285 | { 286 | "name": "stdout", 287 | "output_type": "stream", 288 | "text": [ 289 | "total 176\n", 290 | "drwxrwxrwx 1 1000 1000 0 Jan 15 20:42 AUX_DATA\n", 291 | "drwxrwxrwx 1 1000 1000 0 Jan 15 20:42 DATASTRIP\n", 292 | "drwxrwxrwx 1 1000 1000 0 Jan 15 20:42 GRANULE\n", 293 | "drwxrwxrwx 1 1000 1000 4096 Jan 15 20:42 HTML\n", 294 | "-rwxrwxrwx 1 1000 1000 18601 Jan 15 20:42 INSPIRE.xml\n", 295 | "-rwxrwxrwx 1 1000 1000 52630 Jan 15 20:42 MTD_MSIL2A.xml\n", 296 | "-rwxrwxrwx 1 1000 1000 3776 Apr 15 15:05 S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE.yaml\n", 297 | "-rwxrwxrwx 1 1000 1000 91676 Jan 15 20:42 manifest.safe\n", 298 | "drwxrwxrwx 1 1000 1000 4096 Jan 15 20:42 rep_info\n" 299 | ] 300 | } 301 | ], 302 | "source": [ 303 | "!ls -l $unziped_image_path" 304 | ] 305 | }, 306 | { 307 | "cell_type": "markdown", 308 | "metadata": {}, 309 | "source": [ 310 | "Now our dataset is ready for indexation. Use the following command to index your image (also know as dataset) into the datacube." 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": 6, 316 | "metadata": {}, 317 | "outputs": [], 318 | "source": [ 319 | "image_metadata_file_path = f'/vagrant/indexed_storage/S2_MSI_L2A/{image_file_name}.SAFE/{image_file_name}.SAFE.yaml'\n", 320 | "\n", 321 | "!datacube dataset add $image_metadata_file_path" 322 | ] 323 | }, 324 | { 325 | "cell_type": "markdown", 326 | "metadata": {}, 327 | "source": [ 328 | "Check the new data have been indexed." 329 | ] 330 | }, 331 | { 332 | "cell_type": "code", 333 | "execution_count": 7, 334 | "metadata": {}, 335 | "outputs": [ 336 | { 337 | "name": "stdout", 338 | "output_type": "stream", 339 | "text": [ 340 | "id: a78a5bff-1bc2-4b76-abc5-6d87001ad02d\n", 341 | "product: s2_sen2cor_ard_granule_EO3\n", 342 | "status: active\n", 343 | "locations:\n", 344 | "- file:///vagrant/indexed_storage/S2_MSI_L2A/S2A_MSIL2A_20210130T151701_N0214_R125_T18NYM_20210130T174015.SAFE/S2A_MSIL2A_20210130T151701_N0214_R125_T18NYM_20210130T174015.SAFE.yaml\n", 345 | "fields:\n", 346 | " creation_time: null\n", 347 | " dataset_maturity: final\n", 348 | " format: JPEG2000\n", 349 | " instrument: MSI\n", 350 | " label: null\n", 351 | " lat: {begin: 5.3333943012481395, end: 6.329895366800824}\n", 352 | " lon: {begin: -73.19553295981784, end: -72.20042267994468}\n", 353 | " platform: Sentinel-2A\n", 354 | " product_family: ard\n", 355 | " region_code: null\n", 356 | " time: {begin: '2021-01-30T17:40:15+00:00', end: '2021-01-30T17:40:15+00:00'}\n", 357 | "---\n", 358 | "id: 9e81c0f0-c8fa-48b2-83b1-8a9a2f6ae041\n", 359 | "product: s2_sen2cor_ard_granule_EO3\n", 360 | "status: active\n", 361 | "locations:\n", 362 | "- file:///vagrant/indexed_storage/S2_MSI_L2A/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE/S2B_MSIL2A_20210115T151709_N0214_R125_T18NYM_20210115T174459.SAFE.yaml\n", 363 | "fields:\n", 364 | " creation_time: null\n", 365 | " dataset_maturity: final\n", 366 | " format: JPEG2000\n", 367 | " instrument: MSI\n", 368 | " label: null\n", 369 | " lat: {begin: 5.3333943012481395, end: 6.329895366800824}\n", 370 | " lon: {begin: -73.19553295981784, end: -72.20042267994468}\n", 371 | " platform: Sentinel-2A\n", 372 | " product_family: ard\n", 373 | " region_code: null\n", 374 | " time: {begin: '2021-01-15T17:44:59+00:00', end: '2021-01-15T17:44:59+00:00'}\n" 375 | ] 376 | } 377 | ], 378 | "source": [ 379 | "!datacube dataset search" 380 | ] 381 | }, 382 | { 383 | "cell_type": "markdown", 384 | "metadata": {}, 385 | "source": [ 386 | "### References\n", 387 | "\n", 388 | "1. [Indexing Data](https://datacube-core.readthedocs.io/en/latest/ops/indexing.html#indexing-data)" 389 | ] 390 | } 391 | ], 392 | "metadata": { 393 | "kernelspec": { 394 | "display_name": "Python 3", 395 | "language": "python", 396 | "name": "python3" 397 | }, 398 | "language_info": { 399 | "codemirror_mode": { 400 | "name": "ipython", 401 | "version": 3 402 | }, 403 | "file_extension": ".py", 404 | "mimetype": "text/x-python", 405 | "name": "python", 406 | "nbconvert_exporter": "python", 407 | "pygments_lexer": "ipython3", 408 | "version": "3.8.5" 409 | } 410 | }, 411 | "nbformat": 4, 412 | "nbformat_minor": 4 413 | } 414 | -------------------------------------------------------------------------------- /notebooks/img/ndvi_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DonAurelio/geopython-2021/0e92162e9b58008b82f3d2ce00263e9c6e48069e/notebooks/img/ndvi_example.png -------------------------------------------------------------------------------- /notebooks/img/xarray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DonAurelio/geopython-2021/0e92162e9b58008b82f3d2ce00263e9c6e48069e/notebooks/img/xarray.png --------------------------------------------------------------------------------