├── .gitattributes ├── README.md ├── environment.yml ├── grid ├── .ipynb_checkpoints │ └── analysisGrid-checkpoint.ipynb ├── ISEA3H_res_table.png ├── analysisGrid.ipynb └── gridsGlobal │ ├── ISEA3H_4.dbf │ ├── ISEA3H_4.prj │ ├── ISEA3H_4.shp │ └── ISEA3H_4.shx ├── images ├── dcLogo.png ├── dcLogo.tfw ├── dcLogo.tif ├── dcLogo.tif.aux.xml ├── dcLogo.tif.ovr ├── dcLogoV2.tfw ├── dcLogoV2.tif ├── dcLogoV2.tif.aux.xml ├── dcLogoV2.tif.ovr ├── dcLogoV2.tiff ├── globalGridExample.jpg ├── imageCollectionStack.jpg ├── prompt.png ├── stAndrewsExample.jpg └── waterOccurrenceIntertidal.jpg ├── intertidalGrid ├── .ipynb_checkpoints │ └── intertidalGrid-checkpoint.ipynb ├── Asset_manager_table_upload.png ├── __pycache__ │ └── cxrIntertidalGrid.cpython-38.pyc ├── assetUpload.jpg ├── cxrIntertidalGrid.py ├── drawingAoI.jpg └── intertidalGrid.ipynb └── waterOccurrence ├── .ipynb_checkpoints └── waterOccurrence-checkpoint.ipynb ├── __pycache__ └── cxrWaterOccurrence.cpython-38.pyc ├── cxrWaterOccurrence.py └── waterOccurrence.ipynb /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![DOI](https://zenodo.org/badge/DOI/10.1016/j.rsase.2021.100499.svg)](https://doi.org/10.1016/j.rsase.2021.100499) 2 | 3 | # 1.0 Coast X-Ray 4 | 5 | The intertidal zone is a dynamic environment, which results in the relatively frequent change of high and line water marks. It is important to be able to monitor these changes, as intertidal stability or instability can have a considerable influence on the rate and extent of coastal erosion and flooding, which are both expected to worsen with climate change. 6 | 7 | Bespoke tidal surveys to capture the full extent of the intertidal is often expensive and logistically difficult, especially in areas with large tidal ranges. However, an approach developed within the Scottish [Dynamic Coast](http://www.DynamicCoast.com) project using Sentinel 2 data offers insight into the intertidal zone, quickly and easily. 8 | 9 | Using a time-series of Sentinel 2 images for an area of interest, with each image capturing a slighlty different tidal position. We can then identify the water in each image (using the Normalised Difference Water Index (NDWI)). It is then possible to create an image that represents the frequency of water occurrence across the intertidal zone. This can be used to inform assessments of intertidal geomorphology, as shown in the image below: 10 | 11 |

12 | 13 |

14 | 15 | Areas that are always water (i.e. areas below Low Water) will always be covered by water and have a water occurrence frequency of 100% (i.e. in all of the images in the image collection, water was always identified in that location). Areas that are sometimes covered by water will represent the intertidal zone. An example of the Coast X-Ray output for St. Andrews, Fife, Scotland, is shown below: 16 | 17 |

18 | 19 |

20 | 21 | A fully interactive map based version can be see [here](https://jamesmfitton.users.earthengine.app/view/coastxray). 22 | 23 | This GitHub respository supports the [journal publication](https://doi.org/10.1016/j.rsase.2021.100499), and includes the code and instructions to create a Coast X-Ray water occurrence output for your area of interest. 24 | 25 | ## 1.1 The Code 26 | 27 | Coast X-Ray utilises [Google Earth Engine](https://earthengine.google.com/) (GEE) to produce the outputs. This is a very powerful tool and if you are a new user of GEE, it is highly recommended that you take a look at the [introduction material](https://developers.google.com/earth-engine/) and to become familiar with the [GEE code editor](https://code.earthengine.google.com/) and GEE terminology before proceeding with the Coast X-Ray code. 28 | 29 | GEE uses JavaScript within the online code editor, however, added functionality (mainly associated with the exporting of assets) can be achieved by using Python via the [Python API](https://developers.google.com/earth-engine/python_install). Coast X-Ray uses the Python API, however, there is also some code which runs within [R](https://www.r-project.org/). 30 | 31 | ## 1.2 The Code Explained 32 | 33 | It can be sometimes difficult, especially for new users, to understand what the code does and for what purpose. Below is a succinct explanation of the processes that are carried out by the code. Note, that for the code to function correctly these steps need to be performed in sequence. 34 | 35 | **Step 1. Generate a grid [grid/analysisGrid.ipynb](grid/analysisGrid.ipynb)** 36 | 37 | In order to make processing within GEE more efficient areas of interest are broken up into small areas using a grid. Coast X-Ray uses a Discrete Global Grid (DDG) to split the globe into small areas. Coast X-Ray makes use of the ISEA3H: Icosahedral Snyder Equal Area Aperture 3 Hexagonal Grid which can be accessed via the R package [dggridR](https://cran.r-project.org/web/packages/dggridR/vignettes/dggridR.html). A hexagonal grid was chosen as this ensures that (almost) all cells are of an equal area, regardless where you are on the globe. The ISEA3H grid can be created at different resolutions (i.e. cell sizes), with an example of given below of the 5th resolution: 38 | 39 |

40 | 41 |

42 | 43 | The code within *globalGrid/notebooks/globalGrid.ipynb* generates a global grid of hexagons, at the users desired resolution, which is then saved locally. The code runs in a Jupyter Notebook, but note that it is within an 'R' kernel. A grid resolution of 12 is recommended as a good balance between size and efficiency. 44 | 45 | **Step 2 - Generate an intertidal grid [intertidalGrid/intertidalGrid.ipynb](intertidalGrid/intertidalGrid.ipynb)** 46 | 47 | The global grid created in Step 1 is large and needs to be filtered down to an area of interest. However, the code within *grid/intertidalGrid.ipynb* not only filters the grid cells to the area of interest, but also to the cells that contain intertidal areas. This means that a whole country can be used as an area of interest, however only the intertidal/coastal cells will be selected for use in the subsequent analysis. 48 | 49 | To identify the intertidal/coastal cells three supporting datasets are used: data on the intertidal area [Murray et al. 2019](https://www.nature.com/articles/s41586-018-0805-8?WT.feed_name=subjects_biological-sciences), bathymetry data [GEBCO](https://www.gebco.net/), and the [OpenStreetMap](https://wiki.openstreetmap.org/wiki/Coastline) coastline. 50 | 51 | **Step 3 - Generate water occurrence analysis [waterOccurrence/waterOccurrence.ipynb](waterOccurrence/waterOccurrence.ipynb)** 52 | 53 | The intertidal grid that was generated in the Step 2 will now be used to produce a water occurrence analysis. Each of the cells within the intertidal grid are processed in turn by the script in *waterOccurrence/waterOccurrence.ipynb*, producing a separate image for each cell. The images are added to GEE within an image collection, which can support filtering of the images and mosaicking. 54 | 55 | For each cell within the intertidal grid, the water occurrence image is produced by: 56 | - clipping the Sentinel 2 image collection to the grid cell 57 | - filtering out images that do not cover all of the grid cell area (due to cloud cover or being at the edge of an image) 58 | - calculating the Normalised Difference Water Index (NDWI) for each image 59 | - identifying the water in each image using a fixed threshold approach 60 | - amalgamating the time series of images to produce a single image by calculating the water occurrence percentage 61 | - exporting the image to an image collection on GEE 62 | - exporting image metadata to a feature dataset 63 | 64 | **Further Analysis** 65 | 66 | Using the image metadata output from step 3, if you have available to you via a tidal model or a tide gauge, a tide stage can be assigned to each image in a grid cell using the data/time of image aquisition. 67 | 68 | Coast X-Ray was developed using the [NOC POLPRED] (https://noc-innovations.co.uk/software/offshore) model API under license. Therefore, code to allocate a tide stage to each image and the further processing is not provided here. However, the freely accessible [FES2014](https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html) tidal model has been used by others and a version of Coast X-Ray is in development. 69 | 70 | ## 3.0 Installation 71 | 72 | To use the code within Coast X-Ray the simplest approach is to create an environment in which to run the code. The environment mimics the Python environment that the code was developed upon, therefore allowing the code to run without errors. This environment is created and managed using [Anaconda](https://www.anaconda.com). The following instructions will guide you through the steps to install Anaconda and use Coast X-Ray. 73 | 74 | GEE also offer a [guide](https://developers.google.com/earth-engine/python_install-conda) on installing GEE in an Anaconda environment and information on the GEE [Python API](https://developers.google.com/earth-engine/python_install). 75 | 76 | **Step 1: Download Coast X-Ray:** Clone or download the Coast X-Ray repository from GitHub using the button 'Clone or Download' above and save it somewhere locally on your computer, e.g. C:\CoastXRay. 77 | 78 | **Step 2. Anaconda:** Install [Anaconda](https://www.anaconda.com/download/). Open the Anaconda prompt (PC) or on a Mac or Linux system open a terminal window. Use the `cd` command to change the directory, and navigate to go the folder where you have downloaded the Coast X-Ray repository (see [here](https://www.digitalcitizen.life/command-prompt-how-use-basic-commands) for information on how to use the command prompt). 79 | 80 | **Step 3: Create a new environment:** Create a new environment named `coastxray` with all the required packages: 81 | 82 | ``` 83 | conda env create -f environment.yml -n coastxray 84 | ``` 85 | 86 | This environment should ensure that all the required packages are installed. This environment is called `coastxray`. The next step is to active the environment in the command prompt with: 87 | 88 | ``` 89 | conda activate coastxray 90 | ``` 91 | 92 | The terminal command line prompt should now have changed from (base) to (coastxray). 93 | 94 | ### 3.2 Activate Google Earth Engine Python API 95 | 96 | To use GEE you need to create an account. Go to https://earthengine.google.com and sign up. Once you have an account, in the Anaconda prompt run: 97 | 98 | ``` 99 | earthengine authenticate 100 | ``` 101 | 102 | This will open a web browser and ask you to login into your GEE account. An authorisation code will be given, which you copy and paste back in the Anaconda prompt. 103 | 104 | This completes the installation. 105 | 106 | 107 | ### 3.3 How to use the scripts 108 | 109 | The scripts are designed to run in sequence analysisGrid -> intertidalGrid -> waterOccurrence. They are also designed to run in [Jupyter Notebooks](https://jupyter.org/). This is a "web-based interactive development environment" and allows for comments and descriptions to be added to the code to support/aid understanding. 110 | 111 | Ensure that you have navigated to the CoastXRay directory within the Anaconda prompt, e.g. C:\CoastXRay, and you have activated the Coast X-Ray environment, then type: 112 | 113 | ``` 114 | jupyter notebook 115 | ``` 116 | Activating the environment and starting jupyter notebooks should look like this in your Anaconda Prompt: 117 | 118 | 119 |

120 | 121 |

122 | 123 | This will then open a web browser, and you should be able to see the files within the C:\CoastXRay directory. The files that end with .ipynb contain the scripts. For more information on how to use Jupyter Notebooks, see this [video](https://www.youtube.com/watch?v=HW29067qVWk). 124 | 125 | You should use the scripts in the following order: 126 | 1. grid/analysisGrid.ipynb 127 | 2. intertidalGrid/intertidalGrid.ipynb 128 | 3. waterOccurrence/waterOccurrence.ipynb 129 | 130 | 131 | ## 4.0 Support 132 | 133 | If you are having a problem, please create a new [Issue](https://github.com/jamesfitton/CoastXRay/issues). This keeps all problems and solutions in the same place and acts as a resource for other users to address the same/similar problems. 134 | 135 | 136 | ## 5.0 Acknowledgements 137 | 138 | Coast X-Ray was developed by [Dr. James M. Fitton](https://twitter.com/j_m_fitton), [Dr. Alistair Rennie](https://twitter.com/RennieAlistair), Dr. Jim Hansom, [Freya Muir](https://twitter.com/fme_muir), and [Dr. Martin Hurst](https://twitter.com/martindhurst) as part of the Scottish [Dynamic Coast](www.DynamicCoast.com) project. 139 | 140 | The authors would like to thank [Gennadii Donchyts](https://twitter.com/gena_d) and [Rodrigo E. Principe](https://github.com/fitoprincipe) for developing various elements of open-access code that was included within Coast X-Ray code, and [Kilian Vos](https://twitter.com/VosKilian) for open-access to CoastSat whose [GitHub](https://github.com/kvos/CoastSat) was a template for the development of the Coast X-Ray GitHub. 141 | 142 |

143 | 144 |

145 | 146 | -------------------------------------------------------------------------------- /environment.yml: -------------------------------------------------------------------------------- 1 | name: coastxray 2 | channels: 3 | - defaults 4 | - conda-forge 5 | dependencies: 6 | - _r-mutex=1.0.1=anacondar_1 7 | - aiohttp=3.8.1=py38h2bbff1b_1 8 | - aiosignal=1.2.0=pyhd3eb1b0_0 9 | - async-timeout=4.0.1=pyhd3eb1b0_0 10 | - attrs=19.3.0=py_0 11 | - backcall=0.1.0=py_0 12 | - bleach=3.1.5=pyh9f0ad1d_0 13 | - brotlipy=0.7.0=py38h1e8a9f7_1000 14 | - bzip2=1.0.8=h8ffe710_4 15 | - ca-certificates=2022.07.19=haa95532_0 16 | - cachetools=4.1.1=py_0 17 | - certifi=2022.6.15=pyhd8ed1ab_1 18 | - cffi=1.14.0=py38ha419a9e_0 19 | - chardet=3.0.4=py38h32f6830_1006 20 | - charset-normalizer=2.0.4=pyhd3eb1b0_0 21 | - colorama=0.4.3=py_0 22 | - conda=4.12.0=py38haa244fe_0 23 | - conda-package-handling=1.7.3=py38h31c79cd_1 24 | - cryptography=2.9.2=py38hba49e27_0 25 | - decorator=4.4.2=py_0 26 | - defusedxml=0.6.0=py_0 27 | - earthengine-api=0.1.321=pyhd8ed1ab_0 28 | - entrypoints=0.3=py38h32f6830_1001 29 | - freetype=2.10.4=h546665d_1 30 | - fribidi=1.0.10=h8d14728_0 31 | - frozenlist=1.2.0=py38h2bbff1b_0 32 | - future=0.18.2=py38h32f6830_1 33 | - glib=2.69.1=h5dc1a3c_1 34 | - google-api-core=1.25.1=pyhd3eb1b0_0 35 | - google-api-python-client=1.12.8=pyhd3deb0d_0 36 | - google-auth=1.33.0=pyhd3eb1b0_0 37 | - google-auth-httplib2=0.0.3=py_3 38 | - google-cloud-core=1.3.0=py_0 39 | - google-cloud-storage=1.28.1=pyh9f0ad1d_0 40 | - google-resumable-media=0.5.1=pyh9f0ad1d_0 41 | - googleapis-common-protos=1.51.0=py38hc8b782b_2 42 | - graphite2=1.3.14=hd77b12b_1 43 | - harfbuzz=4.3.0=hda2c7e1_0 44 | - httplib2=0.18.1=pyh9f0ad1d_0 45 | - httplib2shim=0.0.3=pyh9f0ad1d_0 46 | - icu=58.2=ha925a31_3 47 | - idna=2.10=pyh9f0ad1d_0 48 | - importlib-metadata=1.6.1=py38h32f6830_0 49 | - importlib_metadata=1.6.1=0 50 | - intel-openmp=2020.0=166 51 | - ipykernel=5.3.0=py38h5ca1d4c_0 52 | - ipython=7.15.0=py38h32f6830_0 53 | - ipython_genutils=0.2.0=py_1 54 | - jbig=2.1=h8d14728_2003 55 | - jedi=0.17.0=py38h32f6830_0 56 | - jinja2=2.11.2=pyh9f0ad1d_0 57 | - jpeg=9e=h8ffe710_2 58 | - jsonschema=3.2.0=py38h32f6830_1 59 | - jupyter_client=6.1.3=py_0 60 | - jupyter_core=4.6.3=py38h32f6830_1 61 | - lerc=2.2.1=h0e60522_0 62 | - libarchive=3.5.2=hb45042f_1 63 | - libblas=3.8.0=15_mkl 64 | - libcurl=7.82.0=h86230a5_0 65 | - libdeflate=1.7=h8ffe710_5 66 | - libffi=3.4.2=h8ffe710_5 67 | - libgit2=1.3.0=h8648793_1 68 | - libiconv=1.16=he774522_0 69 | - liblapack=3.8.0=15_mkl 70 | - libmamba=0.19.1=h44daa3b_0 71 | - libmambapy=0.19.1=py38h2bfd5b9_0 72 | - libpng=1.6.37=h1d00b33_2 73 | - libprotobuf=3.12.3=h7bd577a_0 74 | - libsodium=1.0.18=h8d14728_1 75 | - libsolv=0.7.19=h7755175_5 76 | - libssh2=1.10.0=h680486a_2 77 | - libtiff=4.3.0=h0c97f57_1 78 | - libxml2=2.9.12=h0ad7f3c_0 79 | - lz4-c=1.9.3=h8ffe710_1 80 | - lzo=2.10=he774522_1000 81 | - m2w64-bwidget=1.9.10=2 82 | - m2w64-bzip2=1.0.6=6 83 | - m2w64-expat=2.1.1=2 84 | - m2w64-fftw=3.3.4=6 85 | - m2w64-flac=1.3.1=3 86 | - m2w64-gcc-libgfortran=5.3.0=6 87 | - m2w64-gcc-libs=5.3.0=7 88 | - m2w64-gcc-libs-core=5.3.0=7 89 | - m2w64-gettext=0.19.7=2 90 | - m2w64-gmp=6.1.0=2 91 | - m2w64-gsl=2.1=2 92 | - m2w64-icu=58.2=heb44b8b_2 93 | - m2w64-libiconv=1.14=6 94 | - m2w64-libjpeg-turbo=1.4.2=3 95 | - m2w64-libogg=1.3.2=3 96 | - m2w64-libpng=1.6.21=2 97 | - m2w64-libsndfile=1.0.26=2 98 | - m2w64-libtiff=4.0.6=2 99 | - m2w64-libvorbis=1.3.5=2 100 | - m2w64-libwinpthread-git=5.0.0.4634.697f757=2 101 | - m2w64-libxml2=2.9.3=4 102 | - m2w64-mpfr=3.1.4=4 103 | - m2w64-pcre2=10.34=0 104 | - m2w64-speex=1.2rc2=3 105 | - m2w64-speexdsp=1.2rc3=3 106 | - m2w64-tcl=8.6.5=3 107 | - m2w64-tk=8.6.5=3 108 | - m2w64-tktable=2.10=5 109 | - m2w64-wineditline=2.101=5 110 | - m2w64-xz=5.2.2=2 111 | - m2w64-zlib=1.2.8=10 112 | - mamba=0.19.1=py38hecfeebb_0 113 | - markupsafe=1.1.1=py38h9de7a3e_1 114 | - menuinst=1.4.19=py38haa244fe_0 115 | - mistune=0.8.4=py38h9de7a3e_1001 116 | - mkl=2020.0=166 117 | - msys2-conda-epoch=20160418=1 118 | - multidict=5.1.0=py38h2bbff1b_2 119 | - nbconvert=5.6.1=py38h32f6830_1 120 | - nbformat=5.0.6=py_0 121 | - notebook=6.0.3=py38h32f6830_0 122 | - openssl=1.1.1q=h8ffe710_0 123 | - packaging=20.4=pyh9f0ad1d_0 124 | - pandoc=2.9.2.1=0 125 | - pandocfilters=1.4.2=py_1 126 | - parso=0.7.0=pyh9f0ad1d_0 127 | - pcre=8.45=h0e60522_0 128 | - pickleshare=0.7.5=py38h32f6830_1001 129 | - pip=20.1.1=py_1 130 | - prometheus_client=0.8.0=pyh9f0ad1d_0 131 | - prompt-toolkit=3.0.5=py_0 132 | - protobuf=3.12.3=py38h6a66eb0_0 133 | - pyasn1=0.4.8=py_0 134 | - pyasn1-modules=0.2.7=py_0 135 | - pybind11-abi=4=hd8ed1ab_3 136 | - pycosat=0.6.3=py38h294d835_1009 137 | - pycparser=2.20=pyh9f0ad1d_2 138 | - pygments=2.6.1=py_0 139 | - pyopenssl=20.0.1=pyhd3eb1b0_1 140 | - pyparsing=2.4.7=pyh9f0ad1d_0 141 | - pyrsistent=0.16.0=py38h9de7a3e_0 142 | - pysocks=1.7.1=py38h32f6830_1 143 | - python=3.8.3=cpython_h5fd99cc_0 144 | - python-dateutil=2.8.1=py_0 145 | - python_abi=3.8=1_cp38 146 | - pytz=2020.1=pyh9f0ad1d_0 147 | - pywin32=227=py38hfa6e2cd_0 148 | - pywinpty=0.5.7=py38_0 149 | - pyzmq=23.2.1=py38h09162b1_0 150 | - r-askpass=1.1=r41h6d2157b_2 151 | - r-assertthat=0.2.1=r41hc72bb7e_2 152 | - r-backports=1.4.1=r41h6d2157b_0 153 | - r-base=4.1.3=hddad469_1 154 | - r-base64enc=0.1_3=r41h6d2157b_1004 155 | - r-bit=4.0.4=r41h6d2157b_0 156 | - r-bit64=4.0.5=r41h6d2157b_0 157 | - r-blob=1.2.3=r41hc72bb7e_0 158 | - r-boot=1.3_28=r41hc72bb7e_0 159 | - r-brew=1.0_7=r41hc72bb7e_0 160 | - r-brio=1.1.3=r41h6d2157b_0 161 | - r-broom=1.0.1=r41hc72bb7e_0 162 | - r-bslib=0.4.0=r41hc72bb7e_0 163 | - r-cachem=1.0.6=r41h6d2157b_0 164 | - r-callr=3.7.2=r41hc72bb7e_0 165 | - r-caret=6.0_93=r41h6d2157b_0 166 | - r-cellranger=1.1.0=r41hc72bb7e_1004 167 | - r-class=7.3_20=r41h6d2157b_0 168 | - r-cli=3.3.0=r41ha856d6a_0 169 | - r-clipr=0.8.0=r41hc72bb7e_0 170 | - r-cluster=2.1.3=r41he816bda_0 171 | - r-codetools=0.2_18=r41hc72bb7e_0 172 | - r-colorspace=2.0_3=r41h6d2157b_0 173 | - r-commonmark=1.8.0=r41h6d2157b_0 174 | - r-cpp11=0.4.2=r41hc72bb7e_0 175 | - r-crayon=1.5.1=r41hc72bb7e_0 176 | - r-credentials=1.3.2=r41hc72bb7e_0 177 | - r-crul=1.2.0=r41h785f33e_0 178 | - r-curl=4.3.2=r41h6d2157b_0 179 | - r-data.table=1.14.2=r41h6d2157b_0 180 | - r-dbi=1.1.3=r41hc72bb7e_0 181 | - r-dbplyr=2.2.1=r41hc72bb7e_0 182 | - r-desc=1.4.1=r41hc72bb7e_0 183 | - r-devtools=2.4.4=r41hc72bb7e_0 184 | - r-dggridr=2.0.4=r41ha856d6a_0 185 | - r-diffobj=0.3.5=r41h6d2157b_0 186 | - r-digest=0.6.29=r41ha856d6a_0 187 | - r-downlit=0.4.2=r41hc72bb7e_0 188 | - r-dplyr=1.0.10=r41ha856d6a_0 189 | - r-dtplyr=1.2.2=r41hc72bb7e_0 190 | - r-e1071=1.7_11=r41ha856d6a_0 191 | - r-ellipsis=0.3.2=r41h6d2157b_0 192 | - r-essentials=4.1=r41hd8ed1ab_2002 193 | - r-evaluate=0.16=r41hc72bb7e_0 194 | - r-fansi=1.0.3=r41h6d2157b_0 195 | - r-farver=2.1.1=r41ha856d6a_0 196 | - r-fastmap=1.1.0=r41ha856d6a_0 197 | - r-fontawesome=0.3.0=r41hc72bb7e_0 198 | - r-forcats=0.5.2=r41hc72bb7e_0 199 | - r-foreach=1.5.2=r41hc72bb7e_0 200 | - r-foreign=0.8_81=r41h6d2157b_0 201 | - r-formatr=1.12=r41hc72bb7e_0 202 | - r-fs=1.5.2=r41ha856d6a_1 203 | - r-future=1.27.0=r41hc72bb7e_0 204 | - r-future.apply=1.9.0=r41hc72bb7e_0 205 | - r-gargle=1.2.0=r41hc72bb7e_0 206 | - r-generics=0.1.3=r41hc72bb7e_0 207 | - r-gert=1.5.0=r41hd1255ac_0 208 | - r-ggplot2=3.3.6=r41hc72bb7e_0 209 | - r-gh=1.3.0=r41hc72bb7e_0 210 | - r-gistr=0.9.0=r41hc72bb7e_0 211 | - r-gitcreds=0.1.1=r41hc72bb7e_0 212 | - r-glmnet=4.1_2=r41he816bda_0 213 | - r-globals=0.16.1=r41hc72bb7e_0 214 | - r-glue=1.6.2=r41h6d2157b_0 215 | - r-googledrive=2.0.0=r41hc72bb7e_0 216 | - r-googlesheets4=1.0.1=r41h785f33e_0 217 | - r-gower=1.0.0=r41h6d2157b_0 218 | - r-gtable=0.3.1=r41hc72bb7e_0 219 | - r-hardhat=1.2.0=r41hc72bb7e_0 220 | - r-haven=2.5.0=r41ha856d6a_0 221 | - r-hexbin=1.28.2=r41he816bda_0 222 | - r-highr=0.9=r41hc72bb7e_0 223 | - r-hms=1.1.2=r41hc72bb7e_0 224 | - r-htmltools=0.5.3=r41ha856d6a_0 225 | - r-htmlwidgets=1.5.4=r41hc72bb7e_0 226 | - r-httpcode=0.3.0=r41h57928b3_1 227 | - r-httpuv=1.6.5=r41ha856d6a_0 228 | - r-httr=1.4.4=r41hc72bb7e_0 229 | - r-ids=1.0.1=r41hc72bb7e_1 230 | - r-ini=0.3.1=r41hc72bb7e_1003 231 | - r-ipred=0.9_13=r41h6d2157b_0 232 | - r-irdisplay=1.1=r41hd8ed1ab_0 233 | - r-irkernel=1.3=r41hc72bb7e_0 234 | - r-isoband=0.2.5=r41ha856d6a_0 235 | - r-iterators=1.0.14=r41hc72bb7e_0 236 | - r-jquerylib=0.1.4=r41hc72bb7e_0 237 | - r-jsonlite=1.8.0=r41h6d2157b_0 238 | - r-kernsmooth=2.23_20=r41h090b817_0 239 | - r-knitr=1.40=r41hc72bb7e_0 240 | - r-labeling=0.4.2=r41hc72bb7e_1 241 | - r-later=1.2.0=r41ha856d6a_0 242 | - r-lattice=0.20_45=r41h6d2157b_0 243 | - r-lava=1.6.10=r41hc72bb7e_0 244 | - r-lazyeval=0.2.2=r41h6d2157b_2 245 | - r-lifecycle=1.0.1=r41hc72bb7e_0 246 | - r-listenv=0.8.0=r41hc72bb7e_1 247 | - r-lobstr=1.1.2=r41ha856d6a_0 248 | - r-lubridate=1.8.0=r41ha856d6a_0 249 | - r-magrittr=2.0.3=r41h6d2157b_0 250 | - r-maps=3.4.0=r41h6d2157b_0 251 | - r-markdown=1.1=r41h6d2157b_1 252 | - r-mass=7.3_58.1=r41h6d2157b_0 253 | - r-matrix=1.4_1=r41hd2e18ed_0 254 | - r-memoise=2.0.1=r41hc72bb7e_0 255 | - r-mgcv=1.8_40=r41hd2e18ed_0 256 | - r-mime=0.12=r41h6d2157b_0 257 | - r-miniui=0.1.1.1=r41hc72bb7e_1002 258 | - r-modelmetrics=1.2.2.2=r41ha856d6a_1 259 | - r-modelr=0.1.9=r41hc72bb7e_0 260 | - r-munsell=0.5.0=r41hc72bb7e_1004 261 | - r-nlme=3.1_159=r41he816bda_0 262 | - r-nnet=7.3_17=r41h6d2157b_0 263 | - r-numderiv=2016.8_1.1=r41hc72bb7e_3 264 | - r-openssl=2.0.2=r41hc08181c_0 265 | - r-parallelly=1.32.1=r41hc72bb7e_0 266 | - r-pbdzmq=0.3_7=r41hf3fe701_0 267 | - r-pillar=1.8.1=r41hc72bb7e_0 268 | - r-pkgbuild=1.3.1=r41hc72bb7e_0 269 | - r-pkgconfig=2.0.3=r41hc72bb7e_1 270 | - r-pkgdown=2.0.6=r41hc72bb7e_0 271 | - r-pkgload=1.3.0=r41hc72bb7e_0 272 | - r-plyr=1.8.7=r41ha856d6a_0 273 | - r-praise=1.0.0=r41hc72bb7e_1005 274 | - r-prettyunits=1.1.1=r41hc72bb7e_1 275 | - r-proc=1.18.0=r41ha856d6a_0 276 | - r-processx=3.7.0=r41h6d2157b_0 277 | - r-prodlim=2019.11.13=r41ha856d6a_1 278 | - r-profvis=0.3.7=r41h6d2157b_0 279 | - r-progress=1.2.2=r41hc72bb7e_2 280 | - r-progressr=0.11.0=r41hc72bb7e_0 281 | - r-promises=1.2.0.1=r41ha856d6a_0 282 | - r-proxy=0.4_27=r41h6d2157b_0 283 | - r-pryr=0.1.5=r41ha856d6a_0 284 | - r-ps=1.7.1=r41h6d2157b_0 285 | - r-purrr=0.3.4=r41h6d2157b_1 286 | - r-quantmod=0.4.20=r41hc72bb7e_0 287 | - r-r6=2.5.1=r41hc72bb7e_0 288 | - r-ragg=1.2.2=r41h9af28ed_0 289 | - r-randomforest=4.7_1.1=r41he816bda_0 290 | - r-rappdirs=0.3.3=r41h6d2157b_0 291 | - r-rbokeh=0.5.2=r41hc72bb7e_1 292 | - r-rcmdcheck=1.4.0=r41h785f33e_0 293 | - r-rcolorbrewer=1.1_3=r41h785f33e_0 294 | - r-rcpp=1.0.9=r41ha856d6a_1 295 | - r-readr=2.1.2=r41ha856d6a_0 296 | - r-readxl=1.4.1=r41h09f486c_0 297 | - r-recipes=1.0.1=r41hc72bb7e_0 298 | - r-recommended=4.1=r41hd8ed1ab_1004 299 | - r-rematch=1.0.1=r41hc72bb7e_1004 300 | - r-rematch2=2.1.2=r41hc72bb7e_1 301 | - r-remotes=2.4.2=r41hc72bb7e_0 302 | - r-repr=1.1.4=r41h785f33e_0 303 | - r-reprex=2.0.2=r41hc72bb7e_0 304 | - r-reshape2=1.4.4=r41ha856d6a_1 305 | - r-rgdal=1.5_29=r41h57928b3_1 306 | - r-rlang=1.0.5=r41ha856d6a_0 307 | - r-rmarkdown=2.16=r41hc72bb7e_0 308 | - r-roxygen2=7.2.1=r41ha856d6a_0 309 | - r-rpart=4.1.16=r41h6d2157b_0 310 | - r-rprojroot=2.0.3=r41hc72bb7e_0 311 | - r-rstudioapi=0.14=r41hc72bb7e_0 312 | - r-rversions=2.1.2=r41hc72bb7e_0 313 | - r-rvest=1.0.3=r41hc72bb7e_0 314 | - r-sass=0.4.2=r41ha856d6a_0 315 | - r-scales=1.2.1=r41hc72bb7e_0 316 | - r-selectr=0.4_2=r41hc72bb7e_1 317 | - r-sessioninfo=1.2.2=r41hc72bb7e_0 318 | - r-shape=1.4.6=r41ha770c72_0 319 | - r-shiny=1.7.2=r41h785f33e_0 320 | - r-sourcetools=0.1.7=r41ha856d6a_1002 321 | - r-sp=1.5_0=r41h6d2157b_0 322 | - r-spatial=7.3_15=r41h6d2157b_0 323 | - r-squarem=2021.1=r41hc72bb7e_0 324 | - r-stringi=1.7.8=r41ha856d6a_0 325 | - r-stringr=1.4.1=r41hc72bb7e_0 326 | - r-survival=3.4_0=r41h6d2157b_0 327 | - r-sys=3.4=r41h6d2157b_0 328 | - r-systemfonts=1.0.4=r41h93c4b31_0 329 | - r-testthat=3.1.4=r41ha856d6a_0 330 | - r-textshaping=0.3.6=r41haa3e5ab_1 331 | - r-tibble=3.1.8=r41h6d2157b_0 332 | - r-tidyr=1.2.0=r41ha856d6a_0 333 | - r-tidyselect=1.1.2=r41h6addd8b_0 334 | - r-tidyverse=1.3.2=r41hc72bb7e_0 335 | - r-timedate=4021.104=r41hc72bb7e_0 336 | - r-tinytex=0.41=r41hc72bb7e_0 337 | - r-triebeard=0.3.0=r41ha856d6a_1004 338 | - r-ttr=0.24.3=r41h6d2157b_0 339 | - r-tzdb=0.3.0=r41ha856d6a_0 340 | - r-urlchecker=1.0.1=r41hc72bb7e_0 341 | - r-urltools=1.7.3=r41ha856d6a_2 342 | - r-usethis=2.1.6=r41hc72bb7e_0 343 | - r-utf8=1.2.2=r41h6d2157b_0 344 | - r-uuid=1.1_0=r41h6d2157b_0 345 | - r-vctrs=0.4.1=r41ha856d6a_0 346 | - r-viridislite=0.4.1=r41hc72bb7e_0 347 | - r-vroom=1.5.7=r41ha856d6a_0 348 | - r-waldo=0.4.0=r41hc72bb7e_0 349 | - r-whisker=0.4=r41hc72bb7e_1 350 | - r-withr=2.5.0=r41hc72bb7e_0 351 | - r-xfun=0.32=r41ha856d6a_0 352 | - r-xml2=1.3.3=r41ha856d6a_1 353 | - r-xopen=1.0.0=r41hc72bb7e_1003 354 | - r-xtable=1.8_4=r41hc72bb7e_3 355 | - r-xts=0.12.1=r41h6d2157b_0 356 | - r-yaml=2.3.5=r41h6d2157b_0 357 | - r-zeallot=0.1.0=r41hc72bb7e_1003 358 | - r-zip=2.2.0=r41h6d2157b_0 359 | - r-zoo=1.8_10=r41h6d2157b_0 360 | - reproc=14.2.3=h8ffe710_0 361 | - reproc-cpp=14.2.3=h0e60522_0 362 | - requests=2.24.0=pyh9f0ad1d_0 363 | - rsa=4.6=pyh9f0ad1d_0 364 | - ruamel_yaml=0.15.80=py38h294d835_1006 365 | - send2trash=1.5.0=py_0 366 | - setuptools=47.1.1=py38h32f6830_0 367 | - simplejson=3.17.0=py38h9de7a3e_1 368 | - six=1.15.0=pyh9f0ad1d_0 369 | - sqlite=3.30.1=hfa6e2cd_0 370 | - terminado=0.8.3=py38h32f6830_1 371 | - testpath=0.4.4=py_0 372 | - tornado=6.0.4=py38hfa6e2cd_0 373 | - tqdm=4.64.0=pyhd8ed1ab_0 374 | - traitlets=4.3.3=py38h32f6830_1 375 | - typing-extensions=4.3.0=py38haa95532_0 376 | - typing_extensions=4.3.0=py38haa95532_0 377 | - ucrt=10.0.20348.0=h57928b3_0 378 | - uritemplate=3.0.1=py_0 379 | - urllib3=1.25.9=py_0 380 | - vc=14.1=h869be7e_1 381 | - vs2015_runtime=14.29.30139=h890b9b1_7 382 | - wcwidth=0.2.4=pyh9f0ad1d_0 383 | - webencodings=0.5.1=py_1 384 | - wheel=0.34.2=py_1 385 | - win_inet_pton=1.1.0=py38_0 386 | - wincertstore=0.2=py38_1003 387 | - winpty=0.4.3=4 388 | - xz=5.2.5=h62dcd97_1 389 | - yaml=0.2.5=h8ffe710_2 390 | - yaml-cpp=0.6.3=ha925a31_4 391 | - yarl=1.8.1=py38h2bbff1b_0 392 | - zeromq=4.3.4=h0e60522_1 393 | - zipp=3.1.0=py_0 394 | - zlib=1.2.11=h2fa13f4_1006 395 | - zstd=1.5.0=h6255e5f_0 396 | prefix: C:\Users\james\anaconda3\envs\cxr01 397 | -------------------------------------------------------------------------------- /grid/.ipynb_checkpoints/analysisGrid-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 1.0 A hexagonal analysis grid" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "A global grid is required to split the analysis of the satellite imagery into smaller areas within which the analysis is conducted. Coast X-Ray makes use of the ISEA3H: Icosahedral Snyder Equal Area Aperture 3 Hexagonal Grid which can be accessed via the R package [dggridR](https://cran.r-project.org/web/packages/dggridR). A hexagonal grid was chosen as this ensures that (almost) all cells are of an equal area, regardless where you are on the globe.\n", 15 | "\n", 16 | "The ISEA3H grid is available at different resolutions, with resolution '12' (cells have an area of approximately 96 km²) being the most appropriate for Coast X-Ray analysis. The other resolutions are as follows:" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "![ISEA3H_res_table.png](./ISEA3H_res_table.png)\n", 24 | "\n" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "## 1.2 Install R Packages" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "The dggridR package needs to be installed, along with the dplyr, rgdal, ggplot2, and maps. If these are not installed already uncomment the code below and run the code cell." 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "## required packages\n", 48 | "# libraries\n", 49 | "# Load R libraries\n", 50 | " if(!require(\"pacman\"))\n", 51 | " install.packages(\"pacman\")\n", 52 | " library(\"pacman\")\n", 53 | "\n", 54 | "p_load(\"sp\", \"dplyr\", \"ggplot2\", \"dggridR\", \"rgeos\")\n", 55 | "\n", 56 | "print(\"Loaded Packages:\")\n", 57 | "p_loaded()\n" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "We can test everything is working is working correctly by creating a grid and mapping the output. Below we create a grid with the resolution of 5, which covers the globe in 2,432 cells. If everything is working, you should see a global map showing green hexagons." 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": { 71 | "scrolled": false 72 | }, 73 | "outputs": [], 74 | "source": [ 75 | "# Resolution of grid\n", 76 | "resolution <- 5\n", 77 | "\n", 78 | "# Contruct the grid\n", 79 | "dggs <- dgconstruct(res=resolution)\n", 80 | "global <- dgearthgrid(dggs, frame=TRUE)\n", 81 | "\n", 82 | "# Get spatial data for the countries\n", 83 | "countries <- map_data(\"world\")\n", 84 | "\n", 85 | "# Create a plot of the grid and world\n", 86 | "p<- ggplot() + \n", 87 | " geom_polygon(data=countries, aes(x=long, y=lat, group=group), fill=NA, color=\"black\") +\n", 88 | " geom_polygon(data=global, aes(x=long, y=lat, group=group), fill=\"green\", alpha=0.4) +\n", 89 | " geom_path (data=global, aes(x=long, y=lat, group=group), alpha=0.4, color=\"white\")\n", 90 | "p" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "## 1.3 Export a global grid " 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "If the above map is created successfully, you can export a shapefile grid to a local directory using the code below. This creates a global grid, we will remove the unwanted cells from the global grid for your area of interest in the next step." 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "#you may need to increase memory size that R can access when exporting the higher (10 and up) resolution levels\n", 114 | "memory.limit(32000)\n", 115 | "\n", 116 | "#set the resolution of the grid (see the ISEA3H information at the top if this doc)\n", 117 | "resolution <- 12\n", 118 | "\n", 119 | "#create the global grid\n", 120 | "dggs <- dgconstruct(res=resolution)\n", 121 | "global <- dgearthgrid(dggs, frame=FALSE)\n", 122 | "\n", 123 | "#set the export path and file name\n", 124 | "exportPath <- paste0(\"./gridsGlobal/ISEA3H_\", resolution, \".shp\")\n", 125 | "\n", 126 | "#prepare the grid for output\n", 127 | "global.cell <- data.frame(cell=getSpPPolygonsIDSlots(global),row.names=getSpPPolygonsIDSlots(global))\n", 128 | "global <- SpatialPolygonsDataFrame(global, global.cell)\n", 129 | "\n", 130 | "for(i in 1:length(global@polygons)) {\n", 131 | " if(max(global@polygons[[i]]@Polygons[[1]]@coords[,1]) - \n", 132 | " min(global@polygons[[i]]@Polygons[[1]]@coords[,1]) > 270) {\n", 133 | " global@polygons[[i]]@Polygons[[1]]@coords[,1] <- (global@polygons[[i]]@Polygons[[1]]@coords[,1] +360) %% 360\n", 134 | " }\n", 135 | "}\n", 136 | "\n", 137 | "#write the shapefile to local\n", 138 | "#it takes about three days, to export the 13 grid, an hour for the 12 grid, 10 mins for the 11, and much quicker for the smaller resolutions (computer dependent)\n", 139 | "writeOGR(global, exportPath,\"\", \"ESRI Shapefile\")" 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": {}, 145 | "source": [ 146 | "**END**" 147 | ] 148 | } 149 | ], 150 | "metadata": { 151 | "kernelspec": { 152 | "display_name": "R", 153 | "language": "R", 154 | "name": "ir" 155 | }, 156 | "language_info": { 157 | "codemirror_mode": "r", 158 | "file_extension": ".r", 159 | "mimetype": "text/x-r-source", 160 | "name": "R", 161 | "pygments_lexer": "r", 162 | "version": "4.1.3" 163 | } 164 | }, 165 | "nbformat": 4, 166 | "nbformat_minor": 2 167 | } 168 | -------------------------------------------------------------------------------- /grid/ISEA3H_res_table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/grid/ISEA3H_res_table.png -------------------------------------------------------------------------------- /grid/analysisGrid.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 1.0 A hexagonal analysis grid" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "A global grid is required to split the analysis of the satellite imagery into smaller areas within which the analysis is conducted. Coast X-Ray makes use of the ISEA3H: Icosahedral Snyder Equal Area Aperture 3 Hexagonal Grid which can be accessed via the R package [dggridR](https://cran.r-project.org/web/packages/dggridR). A hexagonal grid was chosen as this ensures that (almost) all cells are of an equal area, regardless where you are on the globe.\n", 15 | "\n", 16 | "The ISEA3H grid is available at different resolutions, with resolution '12' (cells have an area of approximately 96 km²) being the most appropriate for Coast X-Ray analysis. The other resolutions are as follows:" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "![ISEA3H_res_table.png](./ISEA3H_res_table.png)\n", 24 | "\n" 25 | ] 26 | }, 27 | { 28 | "cell_type": "markdown", 29 | "metadata": {}, 30 | "source": [ 31 | "## 1.2 Install R Packages" 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "The dggridR package needs to be installed, along with the dplyr, rgdal, ggplot2, and maps. If these are not installed already uncomment the code below and run the code cell." 39 | ] 40 | }, 41 | { 42 | "cell_type": "code", 43 | "execution_count": null, 44 | "metadata": {}, 45 | "outputs": [], 46 | "source": [ 47 | "## required packages\n", 48 | "# libraries\n", 49 | "# Load R libraries\n", 50 | " if(!require(\"pacman\"))\n", 51 | " install.packages(\"pacman\")\n", 52 | " library(\"pacman\")\n", 53 | "\n", 54 | "p_load(\"sp\", \"dplyr\", \"ggplot2\", \"dggridR\", \"rgeos\")\n", 55 | "\n", 56 | "print(\"Loaded Packages:\")\n", 57 | "p_loaded()\n" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "metadata": {}, 63 | "source": [ 64 | "We can test everything is working is working correctly by creating a grid and mapping the output. Below we create a grid with the resolution of 5, which covers the globe in 2,432 cells. If everything is working, you should see a global map showing green hexagons." 65 | ] 66 | }, 67 | { 68 | "cell_type": "code", 69 | "execution_count": null, 70 | "metadata": { 71 | "scrolled": false 72 | }, 73 | "outputs": [], 74 | "source": [ 75 | "# Resolution of grid\n", 76 | "resolution <- 5\n", 77 | "\n", 78 | "# Contruct the grid\n", 79 | "dggs <- dgconstruct(res=resolution)\n", 80 | "global <- dgearthgrid(dggs, frame=TRUE)\n", 81 | "\n", 82 | "# Get spatial data for the countries\n", 83 | "countries <- map_data(\"world\")\n", 84 | "\n", 85 | "# Create a plot of the grid and world\n", 86 | "p<- ggplot() + \n", 87 | " geom_polygon(data=countries, aes(x=long, y=lat, group=group), fill=NA, color=\"black\") +\n", 88 | " geom_polygon(data=global, aes(x=long, y=lat, group=group), fill=\"green\", alpha=0.4) +\n", 89 | " geom_path (data=global, aes(x=long, y=lat, group=group), alpha=0.4, color=\"white\")\n", 90 | "p" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "## 1.3 Export a global grid " 98 | ] 99 | }, 100 | { 101 | "cell_type": "markdown", 102 | "metadata": {}, 103 | "source": [ 104 | "If the above map is created successfully, you can export a shapefile grid to a local directory using the code below. This creates a global grid, we will remove the unwanted cells from the global grid for your area of interest in the next step." 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "#you may need to increase memory size that R can access when exporting the higher (10 and up) resolution levels\n", 114 | "memory.limit(32000)\n", 115 | "\n", 116 | "#set the resolution of the grid (see the ISEA3H information at the top if this doc)\n", 117 | "resolution <- 12\n", 118 | "\n", 119 | "#create the global grid\n", 120 | "dggs <- dgconstruct(res=resolution)\n", 121 | "global <- dgearthgrid(dggs, frame=FALSE)\n", 122 | "\n", 123 | "#set the export path and file name\n", 124 | "exportPath <- paste0(\"./gridsGlobal/ISEA3H_\", resolution, \".shp\")\n", 125 | "\n", 126 | "#prepare the grid for output\n", 127 | "global.cell <- data.frame(cell=getSpPPolygonsIDSlots(global),row.names=getSpPPolygonsIDSlots(global))\n", 128 | "global <- SpatialPolygonsDataFrame(global, global.cell)\n", 129 | "\n", 130 | "for(i in 1:length(global@polygons)) {\n", 131 | " if(max(global@polygons[[i]]@Polygons[[1]]@coords[,1]) - \n", 132 | " min(global@polygons[[i]]@Polygons[[1]]@coords[,1]) > 270) {\n", 133 | " global@polygons[[i]]@Polygons[[1]]@coords[,1] <- (global@polygons[[i]]@Polygons[[1]]@coords[,1] +360) %% 360\n", 134 | " }\n", 135 | "}\n", 136 | "\n", 137 | "#write the shapefile to local\n", 138 | "#it takes about three days, to export the 13 grid, an hour for the 12 grid, 10 mins for the 11, and much quicker for the smaller resolutions (computer dependent)\n", 139 | "writeOGR(global, exportPath,\"\", \"ESRI Shapefile\")" 140 | ] 141 | }, 142 | { 143 | "cell_type": "markdown", 144 | "metadata": {}, 145 | "source": [ 146 | "**END**" 147 | ] 148 | } 149 | ], 150 | "metadata": { 151 | "kernelspec": { 152 | "display_name": "R", 153 | "language": "R", 154 | "name": "ir" 155 | }, 156 | "language_info": { 157 | "codemirror_mode": "r", 158 | "file_extension": ".r", 159 | "mimetype": "text/x-r-source", 160 | "name": "R", 161 | "pygments_lexer": "r", 162 | "version": "4.1.3" 163 | } 164 | }, 165 | "nbformat": 4, 166 | "nbformat_minor": 2 167 | } 168 | -------------------------------------------------------------------------------- /grid/gridsGlobal/ISEA3H_4.dbf: -------------------------------------------------------------------------------- 1 | z ,AQWcellCP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 -------------------------------------------------------------------------------- /grid/gridsGlobal/ISEA3H_4.prj: -------------------------------------------------------------------------------- 1 | GEOGCS["GCS_unknown",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137.0,298.257223563]],PRIMEM["Greenwich",0.0],UNIT["Degree",0.0174532925199433]] -------------------------------------------------------------------------------- /grid/gridsGlobal/ISEA3H_4.shp: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/grid/gridsGlobal/ISEA3H_4.shp -------------------------------------------------------------------------------- /grid/gridsGlobal/ISEA3H_4.shx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/grid/gridsGlobal/ISEA3H_4.shx -------------------------------------------------------------------------------- /images/dcLogo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/dcLogo.png -------------------------------------------------------------------------------- /images/dcLogo.tfw: -------------------------------------------------------------------------------- 1 | 0.0104166667 2 | 0.0000000000 3 | 0.0000000000 4 | -0.0104166667 5 | -0.4947916667 6 | 0.4947916667 7 | -------------------------------------------------------------------------------- /images/dcLogo.tif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/dcLogo.tif -------------------------------------------------------------------------------- /images/dcLogo.tif.aux.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | Generic 4 | 5 | 6 | 7 | 8 | -0.5 9 | 255.5 10 | 256 11 | 1 12 | 0 13 | 3323|128|125|121|158|80|83|78|60|90|99|84|78|64|47|53|79|123|60|63|92|97|105|71|108|109|89|98|108|74|99|101|113|95|72|93|97|74|91|92|97|57|90|69|93|129|88|63|59|85|90|87|64|92|89|68|87|88|65|83|63|85|86|66|86|81|57|107|80|85|63|58|88|72|67|59|80|66|77|83|70|48|78|69|56|71|61|68|72|43|65|65|59|62|63|46|67|66|54|46|38|65|60|59|43|54|52|43|37|44|42|50|40|52|47|26|57|28|29|31|40|36|35|25|34|30|8|13|1|13|2|12|1|11|1|10|10|1|7|2|7|0|16|5|1|11|1|8|13|0|11|8|1|10|4|11|13|5|8|8|1|6|9|5|13|13|5|13|10|4|15|9|11|2|10|12|16|2|11|43|5|13|2|12|28|22|13|13|18|4|11|7|13|15|12|14|12|3|11|8|16|33|8|17|36|12|17|20|17|12|14|23|13|26|18|21|22|25|19|22|64|44|29|22|32|13|31|30|19|24|32|30|30|67|47|64|43|40|45|95|61|103|64|133|117|141|150|234|34991|31|6|10|15|13|17|7035 14 | 15 | 16 | 17 | ATHEMATIC 18 | 7698.149350395086,6136.341944207499,4223.575591958596 19 | 20 | 255 21 | 202.42602539062 22 | 0 23 | 1 24 | 1 25 | 87.739098185445 26 | 27 | 28 | 29 | 30 | 31 | -0.5 32 | 255.5 33 | 256 34 | 1 35 | 0 36 | 3316|88|63|75|108|38|57|46|30|33|48|20|23|25|21|24|40|67|15|16|11|10|12|7|15|19|22|6|14|8|6|9|53|9|10|5|4|10|1|4|11|2|6|5|3|36|1|0|4|1|9|1|2|5|0|9|0|0|6|3|3|1|0|10|3|4|2|0|26|2|7|0|0|9|0|9|0|9|0|1|8|1|7|0|8|0|5|1|1|6|3|9|24|18|14|86|56|70|46|43|109|77|85|46|55|95|87|139|127|123|131|122|162|150|99|117|142|156|102|172|121|89|162|122|121|112|151|123|127|118|115|147|89|149|114|113|138|148|86|94|129|118|120|108|122|82|115|100|107|108|101|105|92|93|98|110|73|69|103|76|99|78|45|68|75|71|76|62|40|53|76|35|60|24|13|14|10|10|12|41|5|4|12|7|5|28|20|18|17|4|4|14|12|14|14|8|15|13|9|7|15|28|9|10|32|10|16|24|19|19|19|16|22|17|13|21|21|24|24|20|27|318|555|615|641|522|464|563|522|543|464|397|275|221|208|153|116|57|54|72|68|116|60|64|160|82|129|275|370|34841|1|2|1|3|6|1060 37 | 38 | 39 | 40 | ATHEMATIC 41 | 6136.341944207499,5324.52586469427,4421.968028347 42 | 43 | 255 44 | 209.86765834263 45 | 0 46 | 1 47 | 1 48 | 72.969348802729 49 | 50 | 51 | 52 | 53 | 54 | -0.5 55 | 255.5 56 | 256 57 | 1 58 | 0 59 | 3318|92|63|76|108|38|58|53|35|36|50|22|24|26|21|24|41|67|15|19|18|14|15|8|17|20|24|7|17|9|6|12|53|11|11|6|5|11|4|8|14|7|10|6|7|37|3|1|4|2|11|2|4|8|4|12|3|6|7|7|6|3|3|11|4|5|5|1|27|3|9|4|6|12|6|14|4|10|2|5|9|4|9|2|12|4|108|80|146|118|144|86|140|88|176|78|147|74|148|108|142|74|176|46|173|72|138|68|170|77|136|73|138|71|164|50|181|63|137|90|136|64|139|94|128|65|132|61|130|88|117|81|109|51|99|52|84|66|74|42|67|52|73|29|54|24|70|17|50|31|36|29|36|22|38|15|36|19|22|15|24|9|18|15|5|11|12|0|10|8|2|13|8|5|1|9|19|6|1|9|36|4|5|40|81|167|127|202|147|194|287|283|341|219|330|325|275|251|287|293|256|265|291|275|242|258|274|242|180|263|206|187|180|196|141|150|134|121|86|67|16|8|50|41|15|17|10|21|16|9|15|19|29|28|26|52|48|54|51|54|112|117|94|68|87|184|153|184|233|455|34689|2|1|3|6|1060 60 | 61 | 62 | 63 | ATHEMATIC 64 | 4223.575591958596,4421.968028347,5584.745889226824 65 | 66 | 255 67 | 207.20077078683 68 | 0 69 | 1 70 | 1 71 | 74.731157419291 72 | 73 | 74 | 75 | -------------------------------------------------------------------------------- /images/dcLogo.tif.ovr: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/dcLogo.tif.ovr -------------------------------------------------------------------------------- /images/dcLogoV2.tfw: -------------------------------------------------------------------------------- 1 | 0.0104166667 2 | 0.0000000000 3 | 0.0000000000 4 | -0.0104166667 5 | -0.4947916667 6 | 0.4947916667 7 | -------------------------------------------------------------------------------- /images/dcLogoV2.tif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/dcLogoV2.tif -------------------------------------------------------------------------------- /images/dcLogoV2.tif.aux.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | Generic 4 | 5 | 6 | 7 | 8 | -0.5 9 | 255.5 10 | 256 11 | 1 12 | 0 13 | 2304|235|220|239|146|168|138|123|100|110|105|104|82|88|79|73|75|106|54|58|82|65|80|77|73|96|106|142|95|124|133|93|129|72|83|84|46|108|95|98|170|64|99|78|109|111|84|82|75|137|86|96|57|100|64|73|94|46|97|94|81|101|43|75|55|72|79|101|68|85|82|101|86|74|70|45|47|53|61|83|92|69|48|53|123|80|100|54|92|45|35|50|51|69|58|65|49|89|60|52|31|66|84|37|51|56|24|36|33|41|45|44|50|64|33|34|35|37|40|36|25|31|28|26|20|18|29|22|23|18|15|18|16|12|14|13|16|11|13|6|14|11|11|12|11|16|14|14|17|7|12|10|9|8|9|10|12|12|7|15|9|6|12|16|7|18|6|6|7|5|12|10|11|15|10|10|13|35|16|22|10|24|21|26|17|18|14|12|16|16|11|18|17|12|14|14|10|11|16|21|50|12|26|16|27|26|25|24|35|29|30|29|38|37|35|58|35|38|24|60|42|66|51|48|69|78|76|77|99|111|92|120|154|143|186|217|237|283|325|405|478|581|784|952|742|1214|1644|1079|19185|2117|1337|1005|1026|1049|2679|4793 14 | 15 | 16 | 17 | ATHEMATIC 18 | 7432.736716549449,6001.410424243891,4171.321697158875 19 | 20 | 255 21 | 202.18626185826 22 | 0 23 | 1 24 | 1 25 | 86.213320992463 26 | 27 | 28 | 29 | 30 | 31 | -0.5 32 | 255.5 33 | 256 34 | 1 35 | 0 36 | 1868|409|192|249|187|129|146|114|101|85|82|90|67|52|48|47|36|85|31|17|16|26|24|20|21|18|13|19|11|8|12|17|18|11|8|8|12|8|10|7|10|3|4|3|4|8|28|5|5|5|8|6|2|1|2|2|5|2|3|4|4|3|4|6|6|7|5|6|17|5|6|8|5|5|8|9|4|12|2|9|13|8|18|7|52|15|20|26|34|22|22|36|30|30|44|32|35|34|43|45|57|55|41|43|52|77|66|117|124|213|127|172|127|129|106|94|127|114|117|222|120|103|167|102|148|112|150|152|189|105|107|82|86|128|151|134|91|130|118|110|86|125|132|125|105|148|99|86|96|52|121|79|111|104|105|84|73|116|117|92|77|78|69|44|49|65|43|49|40|42|29|36|47|33|30|16|24|15|33|21|10|8|22|7|15|16|10|14|9|18|16|7|11|9|6|10|13|7|8|14|27|28|5|13|18|14|18|15|18|22|20|22|19|31|19|32|36|38|94|68|91|142|381|719|741|491|534|421|494|482|610|335|330|318|261|216|218|229|260|224|316|334|432|554|554|734|879|1283|1752|19704|2570|1606|1256|827|740|2829 37 | 38 | 39 | 40 | ATHEMATIC 41 | 6001.410424243891,5255.328783308296,4360.966372885188 42 | 43 | 255 44 | 209.9413016183 45 | 0 46 | 1 47 | 1 48 | 72.493646503044 49 | 50 | 51 | 52 | 53 | 54 | -0.5 55 | 255.5 56 | 256 57 | 1 58 | 0 59 | 266|64|1676|366|180|200|178|134|141|113|104|90|81|83|64|55|46|87|38|40|32|16|18|29|29|26|17|20|14|21|14|10|12|21|18|13|8|11|13|8|9|12|8|3|5|4|8|10|31|8|8|9|7|8|6|5|5|3|7|5|3|7|8|5|8|11|8|9|5|7|15|8|11|11|9|3|13|10|11|8|8|16|10|12|45|20|46|49|146|143|76|76|108|40|208|65|122|61|74|142|141|91|185|156|53|100|65|167|206|149|35|104|97|56|129|66|168|231|197|135|51|50|176|53|191|52|89|71|84|156|117|54|75|65|57|85|48|69|53|55|53|28|38|51|47|43|63|62|82|52|40|28|49|50|30|37|45|36|47|34|43|32|31|30|40|39|48|33|38|31|37|32|29|30|40|38|47|41|60|55|83|71|75|68|85|76|135|145|133|172|187|288|255|265|240|261|448|304|267|198|294|203|316|248|184|216|214|248|233|279|181|210|178|133|146|82|89|71|71|73|65|53|45|69|84|51|32|70|56|55|63|94|123|101|92|128|125|182|224|200|303|260|493|423|675|758|567|1060|781|1550|2077|17657|1060|1419|1176|5723 60 | 61 | 62 | 63 | ATHEMATIC 64 | 4171.321697158875,4360.966372885188,5431.566795140606 65 | 66 | 255 67 | 207.40781947545 68 | 0 69 | 1 70 | 1 71 | 73.699164141397 72 | 73 | 74 | 75 | -------------------------------------------------------------------------------- /images/dcLogoV2.tif.ovr: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/dcLogoV2.tif.ovr -------------------------------------------------------------------------------- /images/dcLogoV2.tiff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/dcLogoV2.tiff -------------------------------------------------------------------------------- /images/globalGridExample.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/globalGridExample.jpg -------------------------------------------------------------------------------- /images/imageCollectionStack.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/imageCollectionStack.jpg -------------------------------------------------------------------------------- /images/prompt.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/prompt.png -------------------------------------------------------------------------------- /images/stAndrewsExample.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/stAndrewsExample.jpg -------------------------------------------------------------------------------- /images/waterOccurrenceIntertidal.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/images/waterOccurrenceIntertidal.jpg -------------------------------------------------------------------------------- /intertidalGrid/.ipynb_checkpoints/intertidalGrid-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 2.0 Creating an Intertidal Grid" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "In the previous step, we created a hexagonal grid. We can filter this grid down to only the cells that cover the coast/intertidal areas of the Area of Interest (AoI). This is done so we do not waste time/resources analysing cells that we have no interest in.\n", 15 | "\n", 16 | "We will use Google Earth Engine (GEE) via the python API. Therefore the python API should already be **imported** and **authenticated** for the following code to work. " 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## 2.1 Python Setup" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "Python needs to be setup with the modules that are going to be used.\n" 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "# Standard library imports\n", 40 | "import datetime\n", 41 | "import time\n", 42 | "import os\n", 43 | "\n", 44 | "# Import Earth Engine\n", 45 | "import ee\n", 46 | "\n", 47 | "# Import Coast X-Ray Module\n", 48 | "import cxrIntertidalGrid as cxr\n", 49 | "cxr.ee = ee" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": {}, 55 | "source": [ 56 | "Then, Google Earth Engine needs to be initialised (GEE needs to be authenticated before this step: see https://developers.google.com/earth-engine/python_install-conda). Check to see if GEE is successfully intialised by runninging the code below." 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": {}, 63 | "outputs": [], 64 | "source": [ 65 | "# Initalise GEE\n", 66 | "try:\n", 67 | " ee.Initialize()\n", 68 | " print('The Earth Engine package initialised successfully!')\n", 69 | "except ee.EEException as e:\n", 70 | " print('The Earth Engine package failed to initialise!')\n", 71 | "except:\n", 72 | " print(\"Unexpected error:\", sys.exc_info()[0])\n", 73 | " raise" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "## 2.2 GEE " 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "Coast X-Ray will need to be able to upload and export data/assets to your GEE account. It is therefore necessary to create a folder structure that will support this. " 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "# Your GEE username\n", 97 | "user = \"username\" # replace with your GEE username\n", 98 | "\n", 99 | "# the folder path you wish to import the global grid in to on GEE, e.g. Grid.\n", 100 | "gridPath = \"Grid\"\n", 101 | "\n", 102 | "# the name of the final intertidal grid output, e.g. ISEA3H_12_Scotland\n", 103 | "intertidalGridName = 'ISEA3H_12_Scotland'\n" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "The grid and export folders are then created on your GEE account:" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": {}, 117 | "outputs": [], 118 | "source": [ 119 | "# NB If you have created this folder before in a previous run of this script this step is not necessary, however \n", 120 | "# it does not create problems if it is rerun\n", 121 | "\n", 122 | "# create the export folder\n", 123 | "os.system(\"earthengine create folder users/\" + user + \"/\" + gridPath)" 124 | ] 125 | }, 126 | { 127 | "cell_type": "markdown", 128 | "metadata": {}, 129 | "source": [ 130 | "## 2.3 Upload the Global Grid to GEE" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "metadata": {}, 136 | "source": [ 137 | "In order to be able to use the grid that we created previously, we need to upload it to GEE. There are a number of ways to upload data to GEE, however the simplest is to upload data via the GEE Code Editor interface https://code.earthengine.google.com." 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "### 2.3.1 Code Editor Upload \n", 145 | "\n", 146 | "Go to the code editor, and click on the assets tab in the top-left, then 'New'. You'll be then see the following options:\n", 147 | "\n", 148 | "

\n", 149 | "\n", 150 | "To upload a shapefile, select 'Shape files', which will then give you the following menu. Click on Select, and then navigate to the folder where you have created the grid shapefiles and select the appropriate files: the shp, zip, dbf, prj, shx, cpg, fix, qix, sbn or shp.xml.\n", 151 | "\n", 152 | "

\n", 153 | "\n", 154 | "In the asset name, add the folder path which was created above and an appropriate file name e.g. Grid/ISEA3H_12.\n", 155 | "\n", 156 | "Click OK, and a task should appear in the 'Tasks' tab in the top-right. You can monitor the progress of the upload here. **Once it has finished uploading (the task will turn blue) you can proceed with the rest of the script**.\n", 157 | "\n", 158 | "For more information on importing assets go to https://developers.google.com/earth-engine/importing." 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "## 2.4 Area of Interest" 166 | ] 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "metadata": {}, 171 | "source": [ 172 | "The AoI for the analysis needs to be defined. This is done using by creating an ee.Geometry.Polygon. The easiest way to do this is with the Code Editor. Go to https://code.earthengine.google.com, then draw a shape on the map using the tools in the top-left of the map (1 in the image below).\n", 173 | "\n", 174 | "When you have finished drawing the shape it will appear in the imports area of the code editor. Click on the small blue code button to show the generated code (2 in the image below). Copy and paste the code and replace the geometry below, removing the color information (the text between the /* */).\n", 175 | "\n", 176 | "

\n", 177 | "\n", 178 | "You can find more information about drawing geometries at https://developers.google.com/earth-engine/playground#geometry-tools.\n", 179 | "\n", 180 | "*A shapefile which covers your AoI can be uploaded to GEE as described above in 2.3. This can then be used instead of the ee.Geometry.Polygon below. e.g. aoi = ee.FeatureCollection('users/xxx/AreasOfInterest/Scotland')*" 181 | ] 182 | }, 183 | { 184 | "cell_type": "code", 185 | "execution_count": null, 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "# Scotland AOI Example\n", 190 | "aoi = ee.Geometry.Polygon(\n", 191 | " [[[-0.9840287169561179, 61.13564198277186],\n", 192 | " [-3.071431060706118, 59.847949925299126],\n", 193 | " [-8.125141998206118, 58.20873815824193],\n", 194 | " [-8.752613334602048, 57.84454583892709],\n", 195 | " [-8.235005279456118, 56.53869912008898],\n", 196 | " [-6.125630279456118, 55.30769017836354],\n", 197 | " [-5.936565679354999, 55.238738807677024],\n", 198 | " [-5.356587310706118, 54.82960313608375],\n", 199 | " [-5.005024810706118, 54.448120789563],\n", 200 | " [-4.417335104159065, 54.5353182238508],\n", 201 | " [-3.994282623206118, 54.46089457496747],\n", 202 | " [-3.329609771643618, 54.47366437484056],\n", 203 | " [-1.115864654456118, 55.33269460948912],\n", 204 | " [0.46616659554388207, 61.093185357531226]]]);\n", 205 | "\n" 206 | ] 207 | }, 208 | { 209 | "cell_type": "markdown", 210 | "metadata": {}, 211 | "source": [ 212 | "### 2.3.2. Importing the Grid\n", 213 | "\n", 214 | "Now that the grid has been uploaded to GEE as an asset, it is now possible to access it and use this grid within our code as a 'Feature Collection'." 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": null, 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "# Global hexagonal grid at 12 resolution\n", 224 | "\n", 225 | "# Direct gridPath to the global grid within your GEE (an example is given)\n", 226 | "globalGridPath = 'users/username/Grid/ISEA3H_12' # Replace username with your GEE username\n", 227 | "\n", 228 | "grid = ee.FeatureCollection(globalGridPath)" 229 | ] 230 | }, 231 | { 232 | "cell_type": "markdown", 233 | "metadata": {}, 234 | "source": [ 235 | "The global grid is then filtered so that only cells that are within the bounds of the AoI polygon are retained. This limits the number of cells that are then used in the next stage of filtering." 236 | ] 237 | }, 238 | { 239 | "cell_type": "code", 240 | "execution_count": null, 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [ 244 | "# Filter the grid to the area of interest\n", 245 | "cxr.aoiGrid = grid.filterBounds(aoi)\n", 246 | "\n", 247 | "print(cxr.aoiGrid.size().getInfo(), 'Grid Cells')" 248 | ] 249 | }, 250 | { 251 | "cell_type": "markdown", 252 | "metadata": {}, 253 | "source": [ 254 | "## 2.5 Intertidal Grid" 255 | ] 256 | }, 257 | { 258 | "cell_type": "markdown", 259 | "metadata": {}, 260 | "source": [ 261 | "We want to further remove cells that cover the inland and far offshore areas of the grid. This is done by using three supporting datasets: data on the intertidal area [Murray et al. 2019](https://www.nature.com/articles/s41586-018-0805-8?WT.feed_name=subjects_biological-sciences), bathymetry data [GEBCO](https://www.gebco.net/), and the [OpenStreetMap](https://wiki.openstreetmap.org/wiki/Coastline) coastline.\n", 262 | "\n", 263 | "These datasets are used to create a mask dataset which represents areas that are at, or near, the coast. This mask can then be used to filter the AoI grid further." 264 | ] 265 | }, 266 | { 267 | "cell_type": "code", 268 | "execution_count": null, 269 | "metadata": {}, 270 | "outputs": [], 271 | "source": [ 272 | "# Intertidal data - helps to identify areas are intertidal - Murray et al. 2019\n", 273 | "intertidal = ee.ImageCollection(\"UQ/murray/Intertidal/v1_1/global_intertidal\") \\\n", 274 | " .filterBounds(cxr.aoiGrid)\\\n", 275 | " .filterMetadata('system:index', 'equals', '2014-2016')\n", 276 | "\n", 277 | "# Bathymetry data to help get rid of the incorrect deep water classifications in the Murray intertidal data\n", 278 | "bathymetryGebco = ee.Image('users/gena/GEBCO_2014_2D')\n", 279 | "\n", 280 | "# Gets the areas shallower than -10 m depth\n", 281 | "shallowWaterMask = bathymetryGebco.resample('bicubic').gt(-10)\n", 282 | "\n", 283 | "# Smooths the shallow water data\n", 284 | "shallowWaterMask = cxr.focalMax(shallowWaterMask, 10)\n", 285 | "\n", 286 | "# Creates a mask of itself\n", 287 | "shallowWaterMask = shallowWaterMask.mask(shallowWaterMask)\n", 288 | "\n", 289 | "# OSM coastline - used to find the approximate positon of the coast\n", 290 | "# this can be replaced with your own OSM coastline asset e.g. https://osmcode.org/osmcoastline/\n", 291 | "coastline = ee.FeatureCollection(\"users/jamesmfitton/OSM/Coastal/coastlines\").filterBounds(cxr.aoiGrid) " 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "metadata": {}, 298 | "outputs": [], 299 | "source": [ 300 | "### Generate the coastal mask ###\n", 301 | "\n", 302 | "# Create an empty image into which to paint the coastline\n", 303 | "empty = ee.Image().byte();\n", 304 | "\n", 305 | "# Turn the coastline into an image and make the band name match the intertidal band name\n", 306 | "coastlineImage = empty.paint(coastline,1,10).rename(['classification']); \n", 307 | "\n", 308 | "# merge the intertidal and coastline image data\n", 309 | "intertidal = intertidal.merge(coastlineImage) \n", 310 | "intertidalGrid = intertidal.map(cxr.clip)\n", 311 | "\n", 312 | "# buffer the intertidal estimate by 1 km to allow for errors\n", 313 | "intertidalBuffer = ee.Image(1) \\\n", 314 | " .cumulativeCost(\n", 315 | " **{'source': intertidalGrid.mosaic(), 'maxDistance': 1000}).lt(1000)\n", 316 | "intertidalBuffer = intertidalBuffer.updateMask(intertidalBuffer.eq(1));\n", 317 | "\n", 318 | "# remove the areas of deep water that are sometimes incorrectly classified as intertidal by the Murray et al. intertidal data\n", 319 | "intertidalBuffer = intertidalBuffer.updateMask(shallowWaterMask.eq(1)) " 320 | ] 321 | }, 322 | { 323 | "cell_type": "code", 324 | "execution_count": null, 325 | "metadata": {}, 326 | "outputs": [], 327 | "source": [ 328 | "### Filter out non-coastal cells ###\n", 329 | "\n", 330 | "# finds which cells of the grid are within the intertidal buffer\n", 331 | "intertidalGrid = intertidalBuffer.reduceRegions(**{\n", 332 | " 'collection': cxr.aoiGrid,\n", 333 | " 'reducer': ee.Reducer.min(),\n", 334 | " 'scale': 30,\n", 335 | " 'tileScale': 1\n", 336 | "}).filterMetadata('min', 'greater_than', 0);\n", 337 | "\n", 338 | "intertidalGrid = intertidalGrid.map(cxr.stringToNumber)\n", 339 | "\n", 340 | "\n", 341 | "print(intertidalGrid.size().getInfo(), 'Intertidal Grid Cells')" 342 | ] 343 | }, 344 | { 345 | "cell_type": "markdown", 346 | "metadata": {}, 347 | "source": [ 348 | "## 2.7 Export " 349 | ] 350 | }, 351 | { 352 | "cell_type": "markdown", 353 | "metadata": {}, 354 | "source": [ 355 | "The intertidal grid is then exported to GEE as an asset, where it can used in subsequent scripts.\n", 356 | "\n", 357 | "The intertidal grid name and the folder it will reside in are set in Section 2.2." 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": null, 363 | "metadata": {}, 364 | "outputs": [], 365 | "source": [ 366 | "#The intertidal grid is then uploaded to GEE as an asset.\n", 367 | "task = ee.batch.Export.table.toAsset(intertidalGrid, \"Coast XRay AoI Intertidal Grid\", \"users/\" \n", 368 | " + user + \"/\" + gridPath + \"/\" + intertidalGridName)\n", 369 | "task.start()\n", 370 | "\n" 371 | ] 372 | }, 373 | { 374 | "cell_type": "markdown", 375 | "metadata": {}, 376 | "source": [ 377 | "**END**" 378 | ] 379 | } 380 | ], 381 | "metadata": { 382 | "kernelspec": { 383 | "display_name": "Python 3", 384 | "language": "python", 385 | "name": "python3" 386 | }, 387 | "language_info": { 388 | "codemirror_mode": { 389 | "name": "ipython", 390 | "version": 3 391 | }, 392 | "file_extension": ".py", 393 | "mimetype": "text/x-python", 394 | "name": "python", 395 | "nbconvert_exporter": "python", 396 | "pygments_lexer": "ipython3", 397 | "version": "3.8.3" 398 | } 399 | }, 400 | "nbformat": 4, 401 | "nbformat_minor": 2 402 | } 403 | -------------------------------------------------------------------------------- /waterOccurrence/.ipynb_checkpoints/waterOccurrence-checkpoint.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 3.0 Water Occurrence" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "The intertidal grid that was generated in the previous step will now be used to produce a water occurrence analysis. The water occurrence functions will run for each individual cell, producing a seperate image for each cell." 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "## 3.1 Python Setup" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "We need to import the appropriate modules into python:" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "# Standard library imports\n", 38 | "import datetime\n", 39 | "import time\n", 40 | "import os\n", 41 | "\n", 42 | "# Import Earth Engine\n", 43 | "import ee\n", 44 | "\n", 45 | "# Import Coast X-Ray Module\n", 46 | "import cxrWaterOccurrence as cxr\n", 47 | "cxr.ee = ee\n" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "Then, Google Earth Engine needs to be initialised (GEE needs to be authenticated before this step)." 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "# Initalise GEE\n", 64 | "try:\n", 65 | " ee.Initialize()\n", 66 | " print('The Earth Engine package initialized successfully!')\n", 67 | "except ee.EEException as e:\n", 68 | " print('The Earth Engine package failed to initialize!')\n", 69 | "except:\n", 70 | " print(\"Unexpected error:\", sys.exc_info()[0])\n", 71 | " raise" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "## 3.2 User inputs" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": {}, 84 | "source": [ 85 | "The requires information from the user with regards GEE account inforomation, location of the intertidal grid (and its resolution), a AoI name, the start and end dates of the analysis, and also the location that the image collection should be placed on GEE." 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": {}, 92 | "outputs": [], 93 | "source": [ 94 | "### USER INPUT STARTS ###\n", 95 | "\n", 96 | "# inputs\n", 97 | "#your GEE username\n", 98 | "user = \"username\" # replace with username\n", 99 | "\n", 100 | "# the path to the intertidal grid on GEE\n", 101 | "intertidalGridPath = \"GlobalGrid/AoI/ISEA3H_12_UKIre\"\n", 102 | "\n", 103 | "# the resolution of the grid\n", 104 | "cxr.gridResolution = 12\n", 105 | "\n", 106 | "# name of area of interest\n", 107 | "aoiName = 'Area'\n", 108 | "\n", 109 | "# start date for the analysis\n", 110 | "cxr.startDate = '2015-09-01'\n", 111 | "\n", 112 | "# end date for the analysis (automatically set as today's date)\n", 113 | "cxr.endDate = str(datetime.date.today())\n", 114 | "# cxr.endDate = '2019-09-30'\n", 115 | " \n", 116 | "# outputs\n", 117 | "# the folder path you wish to export the water occurrence image collection to\n", 118 | "exportFolderPath = \"CoastXRayPython/Outputs\"\n", 119 | "\n", 120 | "# the folder path you wish to export the water occurrence feature collection to\n", 121 | "exportFolderPathGridCellSummary = \"CoastXRayPython/GridCellSummary\"\n", 122 | "\n", 123 | "# the folder path you wish to export the temporary water occurrence feature collection to\n", 124 | "exportFolderPathGridCellSummaryTemp = \"CoastXRayPython/GridCellSummaryTemp\"\n", 125 | "\n", 126 | "### USER INPUT ENDS ###\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "## 3.3 GEE Collection Directory\n", 134 | "\n", 135 | "An empty image collection needs to be created on GEE in an appropriate folder that will recieve the output images:" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "metadata": {}, 142 | "outputs": [], 143 | "source": [ 144 | "# Creates the folders and image collection on GEE to recieve the outputs\n", 145 | "\n", 146 | "# Create the path\n", 147 | "exportCollection = aoiName + '_' + cxr.startDate + '_' + cxr.endDate\n", 148 | "\n", 149 | "# Create the export collection\n", 150 | "os.system(\"earthengine create collection users/\" + user + \"/\" + exportFolderPath + \"/\" + exportCollection + \" -p\")\n", 151 | "os.system(\"earthengine create folder users/\" + user + \"/\" + exportFolderPathGridCellSummary + \"/\" + exportCollection + \" -p\")\n", 152 | "os.system(\"earthengine create folder users/\" + user + \"/\" + exportFolderPathGridCellSummaryTemp + \"/\" + exportCollection + \" -p\")\n" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "## 3.4 Intertidal Grid and Sentinel 2 Image Collection" 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": {}, 165 | "source": [ 166 | "The intertidal grid needs to be accessed and then stored as a Feature Collection:" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": {}, 173 | "outputs": [], 174 | "source": [ 175 | "# Construct the path to the intertidal grid\n", 176 | "intertidalGrid = \"users/\" + user + \"/\" + intertidalGridPath\n", 177 | "intertidalGrid = ee.FeatureCollection(intertidalGrid)\n", 178 | "intertidalGridSize = intertidalGrid.size().getInfo()\n", 179 | "\n", 180 | "print('The grid consists of ' + str(intertidalGridSize) + ' intertidal grid cells.')\n" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": {}, 186 | "source": [ 187 | "The Sentinel 2 image collection is created then filtered firstly by cloud cover (images with cloud cover less then 90 are retained), then by time (using the start and end date set in 3.1), then by location (using the intertidal grid):" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "metadata": {}, 194 | "outputs": [], 195 | "source": [ 196 | "# Create the inital Sentinel 2-1C collection\n", 197 | "gridBounds = ee.Feature(intertidalGrid.union().first()).bounds().geometry()\n", 198 | "\n", 199 | "collection = ee.ImageCollection(\"COPERNICUS/S2\")\\\n", 200 | " .filterMetadata('CLOUD_COVERAGE_ASSESSMENT', 'not_greater_than', 90)\\\n", 201 | " .filterDate(cxr.startDate,cxr.endDate)\\\n", 202 | " .filterBounds(gridBounds)\n", 203 | "\n", 204 | "print('There are ' + str(collection.size().getInfo()) + ' images in the collection.')" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "## 3.5 Cloud Masking" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "Imagery is collected from the Sentinel 2 in all weather conditions. This means that images can be collected where the Earth's surface is partly, or even completely, obscured by clouds. Furthemore, cloud shadows can also create inaccuracies within the analysis, producing erroneous results. \n", 219 | "\n", 220 | "To limit these issues, firstly, images with extensive cloud cover are excluded from the image collection (see above). Secondly, clouds within the remaining Sentinel 2 images are removed (otherwise known as cloud masking). This is done using the Quality Assurrance band (QA60) in the image. The clouds are identified and then masked out of the image, and stored within the collectionCloudMasked image collection.\n", 221 | "\n", 222 | "*Note that there are a number of ways to mask the cloud in an image - the approach used here is probably the simplest and could be improved in future versions*" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": null, 228 | "metadata": {}, 229 | "outputs": [], 230 | "source": [ 231 | "collectionCloudMasked = ee.ImageCollection(collection).map(cxr.cloudMask) " 232 | ] 233 | }, 234 | { 235 | "cell_type": "markdown", 236 | "metadata": {}, 237 | "source": [ 238 | "## 3.6 Water Occurrence Analysis" 239 | ] 240 | }, 241 | { 242 | "cell_type": "markdown", 243 | "metadata": {}, 244 | "source": [ 245 | "The script below uses a for loop, which means that the script runs on each of the grid cells within the intertidal grid sequentially. On each grid cell the following steps are conducted:\n" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [ 254 | "#Convert the feature collection to a list to allow looping:\n", 255 | "intertidalGridList = intertidalGrid.toList(intertidalGridSize)\n", 256 | "\n", 257 | "#an empty list to receive the GEE task ids\n", 258 | "taskIDList = []\n", 259 | "\n", 260 | "#an empty list to recieve the features of each cell\n", 261 | "gridFeaturesList = []\n", 262 | "\n", 263 | "#get a list of number to allow the looping through the feature collection\n", 264 | "for i in range(intertidalGridSize):\n", 265 | "# for i in range(1): #use this if you just want to test it on a single grid cell\n", 266 | " #get the grid cell feature\n", 267 | " cxr.gridCellFeature = ee.Feature(intertidalGridList.get(i))\n", 268 | " \n", 269 | " #get the geometry of the grid cell\n", 270 | " cxr.gridCell = ee.Feature(intertidalGridList.get(i)).buffer(20).geometry()\n", 271 | " \n", 272 | "#STEP 1: Filter the collection to the boundary of the feature\n", 273 | " \n", 274 | " #SENTINEL2-1C\n", 275 | " cxr.gridCellCollection = collectionCloudMasked.filterBounds(cxr.gridCell)\n", 276 | " \n", 277 | "#STEP 2: Clip the images in the collection to the extent of the grid cell\n", 278 | " #SENTINEL2-1C\n", 279 | " cxr.gridCellCollection = cxr.gridCellCollection.map(cxr.gridCellCollectionClip)\n", 280 | "\n", 281 | "#STEP 3: Remove the duplicate images from the collection\n", 282 | "\n", 283 | " #SENTINEL2-1C\n", 284 | " cxr.gridCellCollection = cxr.gridCellCollection.map(cxr.removeDuplicates)\n", 285 | " cxr.gridCellCollection = ee.ImageCollection(cxr.gridCellCollection).distinct('dateId')\n", 286 | "\n", 287 | "#STEP 4: Mosaic the images that occur on the same day/time \n", 288 | "\n", 289 | " #SENTINEL2-1C\n", 290 | " cxr.gridCellCollection = cxr.mosaicSameDay(ee.ImageCollection(cxr.gridCellCollection))\n", 291 | " \n", 292 | "#STEP 5: Calculate the area of the image within the grid cell and set it as a property of the image\n", 293 | "\n", 294 | " #SENTINEL2-1C \n", 295 | " cxr.gridCellCollection = cxr.gridCellCollection.map(cxr.setImageArea)\n", 296 | "\n", 297 | "#STEP 6: Filter out the images that do not sufficiently cover the grid cell\n", 298 | "\n", 299 | " #the cloud cover filter value (%) to start \n", 300 | " startFilter = 99 #up to 99.5?\n", 301 | " \n", 302 | " #the interval to reduce the cloud cover filter by each iteration\n", 303 | " interval = 0.5\n", 304 | " \n", 305 | " #number of filters to test\n", 306 | " iterations = 5\n", 307 | " \n", 308 | " #the minimum mumber of images required to make the water occurrence output - default = 30\n", 309 | " minImages = 30\n", 310 | " \n", 311 | " #the default cloud cover filter value (%) if a better one cannot be found\n", 312 | " defaultFilter = 90\n", 313 | " \n", 314 | " #find the cloud cover value\n", 315 | " cxr.cellCloudCoverFilterValue = cxr.setFilterDecending(cxr.gridCellCollection, startFilter, interval, iterations, minImages, defaultFilter)\n", 316 | " \n", 317 | "#STEP 7: Filter the collection based on the cloud cover value\n", 318 | " cxr.gridCellCollection = cxr.gridCellCollection.filterMetadata('gridCellCoverage', 'greater_than', cxr.cellCloudCoverFilterValue)\n", 319 | "\n", 320 | "#STEP 8: Convert the images in the collection into NDWI images and identify the water in each image\n", 321 | " \n", 322 | " #calulate the NDWI for each image\n", 323 | " gridCellNDWICollection = ee.ImageCollection(cxr.gridCellCollection).map(cxr.NDWI)\n", 324 | " \n", 325 | " #set the threshold for water extraction from the NDWI image\n", 326 | " cxr.ndwiThreshold = 0.2\n", 327 | "\n", 328 | " gridCellNDWIWaterCollection = gridCellNDWICollection.map(cxr.ndwiWater)\n", 329 | " \n", 330 | "#STEP 9: Calculate the water occurrence of the collection\n", 331 | "\n", 332 | " #count the numer of water occurrences at each pixel\n", 333 | " waterReduceSum = gridCellNDWIWaterCollection.reduce(ee.Reducer.sum()).int16().rename('waterOccurrenceCount')\n", 334 | " \n", 335 | " #remove pixels that only have 1 for water occurence - QA check - review this\n", 336 | " waterReduceSum = waterReduceSum.where(waterReduceSum.eq(1), 0)\n", 337 | "\n", 338 | " #calculates the water occurence % \n", 339 | " cxr.gridCellNDWICollectionSize = gridCellNDWICollection.size()\n", 340 | " waterPercentage = waterReduceSum.divide(cxr.gridCellNDWICollectionSize).multiply(100).rename('waterOccurrencePercentage').addBands(waterReduceSum)\n", 341 | "\n", 342 | " #adds the image collection size as a band\n", 343 | " mask = waterPercentage.select('waterOccurrenceCount').mask()\n", 344 | " \n", 345 | " #add a band within the water occurrence image that holds the number of images used to calculate the water occurrence\n", 346 | " bandCollectionLength = ee.Image.constant(cxr.gridCellNDWICollectionSize).uint16().rename('numberOfImagesAnalysed').updateMask(mask)\n", 347 | " gridCellWaterOccurrenceOutput = waterPercentage.addBands(bandCollectionLength)\n", 348 | " \n", 349 | "#STEP 10: Create a median NDWI band and add the band to the water occurrence image\n", 350 | " \n", 351 | " #calculate the median NDWI\n", 352 | " ndwiMedian = gridCellNDWICollection.median().rename('ndwiMedian')\n", 353 | " \n", 354 | " #add ndwiMedian as a band to the water occurrence image\n", 355 | " gridCellWaterOccurrenceOutput = gridCellWaterOccurrenceOutput.addBands(ndwiMedian)\n", 356 | " \n", 357 | "# #STEP 11: Create a feature collection, with each feature containing information about the images used\n", 358 | " \n", 359 | " #get the grid cell number and convert it to string\n", 360 | " cxr.cellNumber = cxr.gridCellFeature.get('cell').getInfo() \n", 361 | " cxr.cellNumberStr = str(cxr.cellNumber)\n", 362 | " \n", 363 | " #get the date of the earliest image \n", 364 | " cxr.earliestImage = ee.Date(cxr.gridCellCollection.sort('system:time_start', True).first().get('dateTime')).format(\"YYYY-MM-dd\")\n", 365 | " \n", 366 | " #get the date of the earliest image \n", 367 | " cxr.latestImage = ee.Date(cxr.gridCellCollection.sort('system:time_start', False).first().get('dateTime')).format(\"YYYY-MM-dd\")\n", 368 | " \n", 369 | " #add the image metadata to the feature properties\n", 370 | " gridCellWaterOccurrenceOutput = gridCellWaterOccurrenceOutput\\\n", 371 | " .set('numberOfImagesAnalysed', ee.Number(cxr.gridCellNDWICollectionSize))\\\n", 372 | " .set('analysisStart', ee.Date(cxr.startDate).format(\"YYYY-MM-dd\"))\\\n", 373 | " .set('analysisEnd', ee.Date(cxr.endDate).format(\"YYYY-MM-dd\"))\\\n", 374 | " .set('earliestImageAnalysed', cxr.earliestImage)\\\n", 375 | " .set('latestImageAnalysed', cxr.latestImage)\\\n", 376 | " .set('cellCloudCoverThreshold', ee.Number(cxr.cellCloudCoverFilterValue))\\\n", 377 | " .set('cell', ee.Number(cxr.cellNumber))\\\n", 378 | " .set('cellResolution', ee.Number(cxr.gridResolution))\\\n", 379 | " \n", 380 | "#STEP 12 - Add the grid cell feature to a collection of the other grid cell features \n", 381 | "\n", 382 | " #get the list of dates\n", 383 | " dateList = cxr.gridCellCollection.aggregate_array('dateTime')\n", 384 | "\n", 385 | " #append the cell geometry and properties to the main list\n", 386 | " gridCellFeatures = ee.List(dateList).map(cxr.featureProperties)\n", 387 | " \n", 388 | "#STEP 13: Export images\n", 389 | "\n", 390 | " #EXPORT\n", 391 | " #generate the export file name\n", 392 | " exportFileName = str(cxr.gridResolution) + \"_\" + cxr.cellNumberStr + \"_\" + aoiName + \"_\" + cxr.startDate + \"_\" + cxr.endDate\n", 393 | " exportPath = \"users/\" + user + \"/\" + exportFolderPath + \"/\" + exportCollection + \"/\" + exportFileName \n", 394 | " exportPathGrid = \"users/\" + user + \"/\" + exportFolderPathGridCellSummaryTemp + \"/\" + exportCollection + \"/\" + exportFileName + '_GCS'\n", 395 | " \n", 396 | " #create the export task and start it \n", 397 | " task = ee.batch.Export.image.toAsset(**{\n", 398 | " 'image': gridCellWaterOccurrenceOutput,\n", 399 | " 'description': \"CoastXRay: \" + cxr.cellNumberStr, \n", 400 | " 'assetId': exportPath,\n", 401 | " 'region': cxr.gridCell.getInfo()['coordinates'],\n", 402 | " 'scale': 10\n", 403 | " })\n", 404 | " task.start()\n", 405 | " print('Started:' + exportFileName)\n", 406 | " \n", 407 | " #get the task ID and append it to the list\n", 408 | " taskID=task.status()['id']\n", 409 | " taskIDList.append(taskID)\n", 410 | "\n", 411 | " task = ee.batch.Export.table.toAsset(**{\n", 412 | " 'collection': ee.FeatureCollection(gridCellFeatures).sort('dateMilli'), \n", 413 | " 'description': \"CoastXRay CGS: \" + cxr.cellNumberStr,\n", 414 | " 'assetId': exportPathGrid\n", 415 | " })\n", 416 | "\n", 417 | " task.start()\n", 418 | " print('Started:' + exportFileName + '_GCS')" 419 | ] 420 | }, 421 | { 422 | "cell_type": "markdown", 423 | "metadata": {}, 424 | "source": [ 425 | "## 3.7 Merge Grid Cell Summary Features\n", 426 | "\n", 427 | "Each of the grid cells have a feature collection produced which summaries the images used. To make this easier to manage it is best to merge the all the feature collections in to one master dataset. \n", 428 | "\n", 429 | "**This should be run after all the cells have been processed on GEE.**\n" 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": null, 435 | "metadata": {}, 436 | "outputs": [], 437 | "source": [ 438 | "# merge all the grid cell summary features into one feature collection\n", 439 | "\n", 440 | "# get a list of the features in the temporary directory\n", 441 | "assetList = ee.data.getList({'id': exportFolderPathGridCellSummaryTemp})\n", 442 | "\n", 443 | "# count the size and create a list range \n", 444 | "assetListSize = ee.List(assetList).size().getInfo()\n", 445 | "listRange = list(range(0, assetListSize-1))\n", 446 | "\n", 447 | "# create an empty feature collection\n", 448 | "fc = ee.FeatureCollection([])\n", 449 | "\n", 450 | "# get all the features and merge them into one feature collection\n", 451 | "for i in listRange:\n", 452 | " fc = fc.merge(ee.FeatureCollection(assetList[i]['id'])) \n", 453 | "\n", 454 | "# export the feature collection\n", 455 | "task = ee.batch.Export.table.toAsset(**{\n", 456 | " 'collection': fc, \n", 457 | " 'description': \"Grid Cell Summary Feature Collection\",\n", 458 | " 'assetId': \"users/\" + user + \"/\" + exportFolderPathGridCellSummary + \"/\" + exportCollection\n", 459 | " })\n", 460 | "\n", 461 | "task.start()" 462 | ] 463 | }, 464 | { 465 | "cell_type": "markdown", 466 | "metadata": {}, 467 | "source": [ 468 | "**END**" 469 | ] 470 | } 471 | ], 472 | "metadata": { 473 | "kernelspec": { 474 | "display_name": "Python 3", 475 | "language": "python", 476 | "name": "python3" 477 | }, 478 | "language_info": { 479 | "codemirror_mode": { 480 | "name": "ipython", 481 | "version": 3 482 | }, 483 | "file_extension": ".py", 484 | "mimetype": "text/x-python", 485 | "name": "python", 486 | "nbconvert_exporter": "python", 487 | "pygments_lexer": "ipython3", 488 | "version": "3.8.3" 489 | } 490 | }, 491 | "nbformat": 4, 492 | "nbformat_minor": 2 493 | } 494 | -------------------------------------------------------------------------------- /waterOccurrence/__pycache__/cxrWaterOccurrence.cpython-38.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jamesfitton/cxr/27c317c9e4257ed687389674c1732ac30bee74fe/waterOccurrence/__pycache__/cxrWaterOccurrence.cpython-38.pyc -------------------------------------------------------------------------------- /waterOccurrence/cxrWaterOccurrence.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # coding: utf-8 3 | # In[ ]: 4 | def cloudMask(image): 5 | """ Masks clouds within the image based on QA bands 6 | :param image: an image 7 | :type: image: ee.Image 8 | :return: Input image masked to cloud free areas 9 | :rtype: ee.Image 10 | """ 11 | #select the QA band and the B4 band 12 | qa = image.select('QA60').int16() 13 | qab4 = image.select("B4") 14 | 15 | cloudBitMask = 1024 #qa.bitwiseAnd(Math.pow(2, 10)) or 1 << 10 16 | cirrusBitMask = 2048 #aq.bitwiseAnd(Math.pow(2, 11)) or 1 << 11 17 | 18 | #Both flags should be set to zero, indicating clear conditions. //b1 1500 add? 19 | mask = qa.bitwiseAnd(cloudBitMask).eq(0).And(qa.bitwiseAnd(cirrusBitMask).eq(0)).And(qab4.lt(2500)) 20 | 21 | return image.updateMask(mask) 22 | 23 | def gridCellCollectionClip(image): 24 | return image.clip(gridCell).select(['B2', 'B3', 'B4', 'B8', 'B11']).rename('blue', 'green', 'red', 'nir', 'swir1') 25 | 26 | def removeDuplicates(image): 27 | """ Adds an ID to the image properties based on date and tile. 28 | :param image: an image 29 | :type: image: ee.Image 30 | :return: Input image with dateID property 31 | :rtype: ee.Image 32 | """ 33 | #get the image date 34 | date = ee.String(image.get('system:index')).slice(0,8) 35 | #get the image tile 36 | tile = ee.String(image.get('MGRS_TILE')) 37 | #create the id 38 | dateId = date.cat('_').cat(tile) 39 | 40 | return ee.Image(image).set('dateId', dateId) 41 | 42 | def replace(image, to_replace, to_add): 43 | """ Replace one band of the image with a provided band from https://github.com/gee-community/gee_tools 44 | :param to_replace: name of the band to replace. If the image hasn't got 45 | that band, it will be added to the image. 46 | :type to_replace: str 47 | :param to_add: Image (one band) containing the band to add. If an Image 48 | with more than one band is provided, it uses the first band. 49 | :type to_add: ee.Image 50 | :return: Same Image provided with the band replaced 51 | :rtype: ee.Image 52 | """ 53 | band = to_add.select([0]) 54 | bands = image.bandNames() 55 | resto = bands.remove(to_replace) 56 | img_resto = image.select(resto) 57 | img_final = img_resto.addBands(band) 58 | return img_final 59 | 60 | def mosaicSameDay(collection): 61 | """ Return a collection where images from the same day are mosaicked from https://github.com/gee-community/gee_tools 62 | :param reducer: the reducer to use for merging images from the same day. 63 | Defaults to 'first' 64 | :type reducer: ee.Reducer 65 | :return: a new image collection with 1 image per day. The only property 66 | kept is `system:time_start` 67 | :rtype: ee.ImageCollection 68 | """ 69 | def make_date_list(img, l): 70 | l = ee.List(l) 71 | img = ee.Image(img) 72 | date = img.date() 73 | # make clean date 74 | day = date.get('day') 75 | month = date.get('month') 76 | year = date.get('year') 77 | clean_date = ee.Date.fromYMD(year, month, day) 78 | condition = l.contains(clean_date) 79 | 80 | return ee.Algorithms.If(condition, l, l.add(clean_date)) 81 | 82 | col_list = collection.toList(collection.size()) 83 | date_list = ee.List(col_list.iterate(make_date_list, ee.List([]))) 84 | 85 | first_img = ee.Image(collection.first()) 86 | bands = first_img.bandNames() 87 | 88 | def make_col(date): 89 | date = ee.Date(date) 90 | filtered = collection.filterDate(date, date.advance(1, 'day')) 91 | 92 | #mean azimuth of images 93 | meanAzimuth = filtered.aggregate_array('MEAN_SOLAR_AZIMUTH_ANGLE') 94 | meanAzimuth = ee.List(meanAzimuth).reduce(ee.Reducer.mean()) 95 | 96 | #mean zenith 97 | meanZenith = filtered.aggregate_array('MEAN_SOLAR_ZENITH_ANGLE') 98 | meanZenith = ee.List(meanZenith).reduce(ee.Reducer.mean()) 99 | 100 | #mean cloud cover 101 | meanCloud = filtered.aggregate_array('CLOUD_COVERAGE_ASSESSMENT') 102 | meanCloud = ee.List(meanCloud).reduce(ee.Reducer.mean()) 103 | 104 | #number of images 105 | numberImages = filtered.size() 106 | 107 | mosaic = filtered.mosaic() 108 | mosaic = mosaic.set('system:time_start', date.millis(), 109 | 'system:footprint', mergeGeometries(filtered)) 110 | 111 | mosaic = mosaic.rename(bands) 112 | def reproject(bname, mos): 113 | mos = ee.Image(mos) 114 | mos_bnames = mos.bandNames() 115 | bname = ee.String(bname) 116 | proj = first_img.select(bname).projection() 117 | 118 | newmos = ee.Image(ee.Algorithms.If( 119 | mos_bnames.contains(bname), 120 | replace(mos, bname, mos.select(bname).setDefaultProjection(proj)), 121 | mos)) 122 | 123 | return newmos 124 | 125 | mosaic = ee.Image(bands.iterate(reproject, mosaic)) 126 | return mosaic .set('dateTime', filtered.first().get('system:time_start')) .set('MEAN_SOLAR_ZENITH_ANGLE', meanZenith) .set('MEAN_SOLAR_AZIMUTH_ANGLE', meanAzimuth) .set('CLOUD_COVERAGE_ASSESSMENT', meanCloud) .set('mosaicImageCount', numberImages) 127 | 128 | new_col = ee.ImageCollection.fromImages(date_list.map(make_col)) 129 | return new_col 130 | 131 | def setImageArea(image): 132 | """ Calculates the area of the image covered by the image 133 | :param image: an image 134 | :type: image: ee.Image 135 | :return: Input image with gridCellCoverage property added 136 | :rtype: ee.Image 137 | """ 138 | #select a 10m band from the image 139 | blueBand = image.select('blue') 140 | # imageArea = blueBand.multiply(ee.Image.pixelArea()) 141 | imageArea = blueBand.reduceRegion(**{ 142 | 'reducer': ee.Reducer.count(), 143 | 'scale': 10, 144 | 'geometry': gridCell})\ 145 | .get('blue') 146 | 147 | imageArea = ee.Number(imageArea).multiply(100) #100 m2 cell area (10*10) 148 | 149 | 150 | gridCellArea = gridCell.area(1) 151 | 152 | return image.set('gridCellCoverage', ee.Number(imageArea).divide(gridCellArea).multiply(100))#.set('gridCellArea', gridCellArea).set('imageArea', imageArea) 153 | 154 | #FUNCTION: identifies the cloud cover threshold to use in order to return a minimum number of images 155 | def setFilterDecending(collection, startFilter, interval, iterations, minImages, defaultFilter): 156 | """ Identifies the cloud cover threshold to use in order to return a minimum number of images 157 | :param collection: an image collection 158 | :type: collection: ee.ImageCollection 159 | :param startFilter: the intitial cloud cover filter value 160 | :type: startFilter: ee.Float 161 | :param interval: the steps/interval that the filter should be reduced by 162 | :type: interval: ee.Float 163 | :param iterations: the number of filters to test 164 | :type: iterations: ee.Int 165 | :param minImages: the minimum number of images required to produce the water occurrence output 166 | :type: minImages: ee.Int 167 | :param defaultFilter: the default filter to use if no suitable filter is identified 168 | :type: defaultFilter: ee.Float 169 | :return: Cloud cover filter value used to filter out images with inadequate cell coverage 170 | :rtype: ee.Float 171 | """ 172 | #creates a list to map through and craetes the filter values to test 173 | list = ee.List.sequence(0, iterations, ee.Number(interval).toFloat()) 174 | 175 | def minImageTest(i): 176 | #creates the test filter value 177 | listIndex = list.indexOf(i) 178 | #decending filter value, hence subtract 179 | filterValue = ee.Number(startFilter).subtract(listIndex.multiply(interval)) 180 | 181 | #filters the collection using the test value 182 | collectionFiltered = collection.filterMetadata('gridCellCoverage', 'greater_than', filterValue) 183 | 184 | #tests whether the collection is larger than the minimum number of images 185 | minImagesTest = ee.Algorithms.IsEqual(collectionFiltered.size().gte(minImages), 0) 186 | 187 | #if the filter value produces the minimum number if images the filter value is returned 188 | #condition, trueCase, falseCase) 189 | return ee.Algorithms.If(minImagesTest, None, filterValue) 190 | 191 | bestFilterList = list.map(minImageTest) 192 | 193 | #bestFilterList is a list of the filter values that meet the minimum number of images value 194 | 195 | #removes nulls from the list and sorts it in ascending order 196 | bestFilter = bestFilterList.removeAll([None]).sort().reverse() #need to reverse the list if using decending filter values 197 | 198 | #test whether the list is empty 199 | allNulls = ee.Algorithms.IsEqual(bestFilter.size(), 0) 200 | 201 | #condition, trueCase, falseCase 202 | return ee.Algorithms.If(allNulls, defaultFilter, bestFilter.get(0)) 203 | 204 | def NDWI(image): 205 | """ Calculates the NDWI of an image 206 | :param image: an image 207 | :type: image: ee.Image 208 | :return: An image with a single ndwi band 209 | :rtype: ee.Image 210 | """ 211 | return image.normalizedDifference(['green', 'nir']).select(['nd'],['ndwi']) 212 | 213 | def ndwiWater(image): 214 | """ Identifies water in an image based on the ndwi and a threshold value 215 | :param image: an image 216 | :type: image: ee.Image 217 | :return: An image where water is identified as 1 and non-water as 0 218 | :rtype: ee.Image 219 | """ 220 | # find the water using a fixed NDWI threshold 221 | NDWIWater = image.gt(ndwiThreshold) 222 | 223 | #clean up the image #look into dilation in morphology module 224 | connectedPixelSize = 512 #this can cause an internal error if set too high (1024 created errors for some cells) 225 | def waterClean(image, size): 226 | connectedPixels = image.int().connectedPixelCount(size) 227 | pixelCountWhere = image.where(connectedPixels.lt(size), 0) 228 | return pixelCountWhere 229 | 230 | imageCleaned = waterClean(NDWIWater, connectedPixelSize) 231 | 232 | time = image.get('system:time_start') 233 | 234 | return imageCleaned.set('system:time_start', time) 235 | 236 | def featureProperties(date): 237 | """ Allocates image properties to the grid cell feature properties 238 | :param date: image date 239 | :type: date: ee.Date 240 | :return: A feature of the grid cell with a range of properties 241 | :rtype: ee.Feature 242 | """ 243 | image = gridCellCollection.filterMetadata('dateTime', 'equals', date).first() 244 | imageId = image.get('system:index') 245 | meanAzimuth = image.get('MEAN_SOLAR_AZIMUTH_ANGLE') 246 | meanZenith = image.get('MEAN_SOLAR_ZENITH_ANGLE') 247 | cloudCoverage = image.get('CLOUD_COVERAGE_ASSESSMENT') 248 | mosaicImageCount = image.get('mosaicImageCount') 249 | 250 | 251 | return gridCellFeature.set('date', ee.Date(date).format("YYYY-MM-dd HH:mm")).set('dateMilli', date).set('imageID', imageId).set('imageMeanAzimuth', meanAzimuth).set('imageMeanZenith', meanZenith).set('imageCloudCover', cloudCoverage).set('numberOfImagesAnalysed', ee.Number(gridCellNDWICollectionSize)).set('analysisStart', ee.Date(startDate).format("YYYY-MM-dd")).set('analysisEnd', ee.Date(endDate).format("YYYY-MM-dd")).set('earliestImageAnalysed', earliestImage).set('latestImageAnalysed', latestImage).set('cellCloudCoverThreshold', ee.Number(cellCloudCoverFilterValue)).set('cell', ee.Number(cellNumber)).set('cellResolution', ee.Number(gridResolution)).set('mosaicImageCount', mosaicImageCount) 252 | 253 | def mergeGeometries(collection): 254 | """ Merge the geometries of many images. Return ee.Geometry """ 255 | imlist = collection.toList(collection.size()) 256 | 257 | first = ee.Image(imlist.get(0)) 258 | rest = imlist.slice(1) 259 | 260 | def wrap(img, ini): 261 | ini = ee.Geometry(ini) 262 | img = ee.Image(img) 263 | geom = img.geometry() 264 | union = geom.union(ini) 265 | return union.dissolve() 266 | 267 | return ee.Geometry(rest.iterate(wrap, first.geometry())) 268 | -------------------------------------------------------------------------------- /waterOccurrence/waterOccurrence.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 3.0 Water Occurrence" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "The intertidal grid that was generated in the previous step will now be used to produce a water occurrence analysis. The water occurrence functions will run for each individual cell, producing a seperate image for each cell." 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "## 3.1 Python Setup" 22 | ] 23 | }, 24 | { 25 | "cell_type": "markdown", 26 | "metadata": {}, 27 | "source": [ 28 | "We need to import the appropriate modules into python:" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "# Standard library imports\n", 38 | "import datetime\n", 39 | "import time\n", 40 | "import os\n", 41 | "\n", 42 | "# Import Earth Engine\n", 43 | "import ee\n", 44 | "\n", 45 | "# Import Coast X-Ray Module\n", 46 | "import cxrWaterOccurrence as cxr\n", 47 | "cxr.ee = ee\n" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "Then, Google Earth Engine needs to be initialised (GEE needs to be authenticated before this step)." 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "# Initalise GEE\n", 64 | "try:\n", 65 | " ee.Initialize()\n", 66 | " print('The Earth Engine package initialized successfully!')\n", 67 | "except ee.EEException as e:\n", 68 | " print('The Earth Engine package failed to initialize!')\n", 69 | "except:\n", 70 | " print(\"Unexpected error:\", sys.exc_info()[0])\n", 71 | " raise" 72 | ] 73 | }, 74 | { 75 | "cell_type": "markdown", 76 | "metadata": {}, 77 | "source": [ 78 | "## 3.2 User inputs" 79 | ] 80 | }, 81 | { 82 | "cell_type": "markdown", 83 | "metadata": {}, 84 | "source": [ 85 | "The requires information from the user with regards GEE account inforomation, location of the intertidal grid (and its resolution), a AoI name, the start and end dates of the analysis, and also the location that the image collection should be placed on GEE." 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": {}, 92 | "outputs": [], 93 | "source": [ 94 | "### USER INPUT STARTS ###\n", 95 | "\n", 96 | "# inputs\n", 97 | "#your GEE username\n", 98 | "user = \"username\" # replace with username\n", 99 | "\n", 100 | "# the path to the intertidal grid on GEE\n", 101 | "intertidalGridPath = \"GlobalGrid/AoI/ISEA3H_12_UKIre\"\n", 102 | "\n", 103 | "# the resolution of the grid\n", 104 | "cxr.gridResolution = 12\n", 105 | "\n", 106 | "# name of area of interest\n", 107 | "aoiName = 'Area'\n", 108 | "\n", 109 | "# start date for the analysis\n", 110 | "cxr.startDate = '2015-09-01'\n", 111 | "\n", 112 | "# end date for the analysis (automatically set as today's date)\n", 113 | "cxr.endDate = str(datetime.date.today())\n", 114 | "# cxr.endDate = '2019-09-30'\n", 115 | " \n", 116 | "# outputs\n", 117 | "# the folder path you wish to export the water occurrence image collection to\n", 118 | "exportFolderPath = \"CoastXRayPython/Outputs\"\n", 119 | "\n", 120 | "# the folder path you wish to export the water occurrence feature collection to\n", 121 | "exportFolderPathGridCellSummary = \"CoastXRayPython/GridCellSummary\"\n", 122 | "\n", 123 | "# the folder path you wish to export the temporary water occurrence feature collection to\n", 124 | "exportFolderPathGridCellSummaryTemp = \"CoastXRayPython/GridCellSummaryTemp\"\n", 125 | "\n", 126 | "### USER INPUT ENDS ###\n" 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "## 3.3 GEE Collection Directory\n", 134 | "\n", 135 | "An empty image collection needs to be created on GEE in an appropriate folder that will recieve the output images:" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "metadata": {}, 142 | "outputs": [], 143 | "source": [ 144 | "# Creates the folders and image collection on GEE to recieve the outputs\n", 145 | "\n", 146 | "# Create the path\n", 147 | "exportCollection = aoiName + '_' + cxr.startDate + '_' + cxr.endDate\n", 148 | "\n", 149 | "# Create the export collection\n", 150 | "os.system(\"earthengine create collection users/\" + user + \"/\" + exportFolderPath + \"/\" + exportCollection + \" -p\")\n", 151 | "os.system(\"earthengine create folder users/\" + user + \"/\" + exportFolderPathGridCellSummary + \"/\" + exportCollection + \" -p\")\n", 152 | "os.system(\"earthengine create folder users/\" + user + \"/\" + exportFolderPathGridCellSummaryTemp + \"/\" + exportCollection + \" -p\")\n" 153 | ] 154 | }, 155 | { 156 | "cell_type": "markdown", 157 | "metadata": {}, 158 | "source": [ 159 | "## 3.4 Intertidal Grid and Sentinel 2 Image Collection" 160 | ] 161 | }, 162 | { 163 | "cell_type": "markdown", 164 | "metadata": {}, 165 | "source": [ 166 | "The intertidal grid needs to be accessed and then stored as a Feature Collection:" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": {}, 173 | "outputs": [], 174 | "source": [ 175 | "# Construct the path to the intertidal grid\n", 176 | "intertidalGrid = \"users/\" + user + \"/\" + intertidalGridPath\n", 177 | "intertidalGrid = ee.FeatureCollection(intertidalGrid)\n", 178 | "intertidalGridSize = intertidalGrid.size().getInfo()\n", 179 | "\n", 180 | "print('The grid consists of ' + str(intertidalGridSize) + ' intertidal grid cells.')\n" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": {}, 186 | "source": [ 187 | "The Sentinel 2 image collection is created then filtered firstly by cloud cover (images with cloud cover less then 90 are retained), then by time (using the start and end date set in 3.1), then by location (using the intertidal grid):" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "metadata": {}, 194 | "outputs": [], 195 | "source": [ 196 | "# Create the inital Sentinel 2-1C collection\n", 197 | "gridBounds = ee.Feature(intertidalGrid.union().first()).bounds().geometry()\n", 198 | "\n", 199 | "collection = ee.ImageCollection(\"COPERNICUS/S2\")\\\n", 200 | " .filterMetadata('CLOUD_COVERAGE_ASSESSMENT', 'not_greater_than', 90)\\\n", 201 | " .filterDate(cxr.startDate,cxr.endDate)\\\n", 202 | " .filterBounds(gridBounds)\n", 203 | "\n", 204 | "print('There are ' + str(collection.size().getInfo()) + ' images in the collection.')" 205 | ] 206 | }, 207 | { 208 | "cell_type": "markdown", 209 | "metadata": {}, 210 | "source": [ 211 | "## 3.5 Cloud Masking" 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "Imagery is collected from the Sentinel 2 in all weather conditions. This means that images can be collected where the Earth's surface is partly, or even completely, obscured by clouds. Furthemore, cloud shadows can also create inaccuracies within the analysis, producing erroneous results. \n", 219 | "\n", 220 | "To limit these issues, firstly, images with extensive cloud cover are excluded from the image collection (see above). Secondly, clouds within the remaining Sentinel 2 images are removed (otherwise known as cloud masking). This is done using the Quality Assurrance band (QA60) in the image. The clouds are identified and then masked out of the image, and stored within the collectionCloudMasked image collection.\n", 221 | "\n", 222 | "*Note that there are a number of ways to mask the cloud in an image - the approach used here is probably the simplest and could be improved in future versions*" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": null, 228 | "metadata": {}, 229 | "outputs": [], 230 | "source": [ 231 | "collectionCloudMasked = ee.ImageCollection(collection).map(cxr.cloudMask) " 232 | ] 233 | }, 234 | { 235 | "cell_type": "markdown", 236 | "metadata": {}, 237 | "source": [ 238 | "## 3.6 Water Occurrence Analysis" 239 | ] 240 | }, 241 | { 242 | "cell_type": "markdown", 243 | "metadata": {}, 244 | "source": [ 245 | "The script below uses a for loop, which means that the script runs on each of the grid cells within the intertidal grid sequentially. On each grid cell the following steps are conducted:\n" 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [ 254 | "#Convert the feature collection to a list to allow looping:\n", 255 | "intertidalGridList = intertidalGrid.toList(intertidalGridSize)\n", 256 | "\n", 257 | "#an empty list to receive the GEE task ids\n", 258 | "taskIDList = []\n", 259 | "\n", 260 | "#an empty list to recieve the features of each cell\n", 261 | "gridFeaturesList = []\n", 262 | "\n", 263 | "#get a list of number to allow the looping through the feature collection\n", 264 | "for i in range(intertidalGridSize):\n", 265 | "# for i in range(1): #use this if you just want to test it on a single grid cell\n", 266 | " #get the grid cell feature\n", 267 | " cxr.gridCellFeature = ee.Feature(intertidalGridList.get(i))\n", 268 | " \n", 269 | " #get the geometry of the grid cell\n", 270 | " cxr.gridCell = ee.Feature(intertidalGridList.get(i)).buffer(20).geometry()\n", 271 | " \n", 272 | "#STEP 1: Filter the collection to the boundary of the feature\n", 273 | " \n", 274 | " #SENTINEL2-1C\n", 275 | " cxr.gridCellCollection = collectionCloudMasked.filterBounds(cxr.gridCell)\n", 276 | " \n", 277 | "#STEP 2: Clip the images in the collection to the extent of the grid cell\n", 278 | " #SENTINEL2-1C\n", 279 | " cxr.gridCellCollection = cxr.gridCellCollection.map(cxr.gridCellCollectionClip)\n", 280 | "\n", 281 | "#STEP 3: Remove the duplicate images from the collection\n", 282 | "\n", 283 | " #SENTINEL2-1C\n", 284 | " cxr.gridCellCollection = cxr.gridCellCollection.map(cxr.removeDuplicates)\n", 285 | " cxr.gridCellCollection = ee.ImageCollection(cxr.gridCellCollection).distinct('dateId')\n", 286 | "\n", 287 | "#STEP 4: Mosaic the images that occur on the same day/time \n", 288 | "\n", 289 | " #SENTINEL2-1C\n", 290 | " cxr.gridCellCollection = cxr.mosaicSameDay(ee.ImageCollection(cxr.gridCellCollection))\n", 291 | " \n", 292 | "#STEP 5: Calculate the area of the image within the grid cell and set it as a property of the image\n", 293 | "\n", 294 | " #SENTINEL2-1C \n", 295 | " cxr.gridCellCollection = cxr.gridCellCollection.map(cxr.setImageArea)\n", 296 | "\n", 297 | "#STEP 6: Filter out the images that do not sufficiently cover the grid cell\n", 298 | "\n", 299 | " #the cloud cover filter value (%) to start \n", 300 | " startFilter = 99 #up to 99.5?\n", 301 | " \n", 302 | " #the interval to reduce the cloud cover filter by each iteration\n", 303 | " interval = 0.5\n", 304 | " \n", 305 | " #number of filters to test\n", 306 | " iterations = 5\n", 307 | " \n", 308 | " #the minimum mumber of images required to make the water occurrence output - default = 30\n", 309 | " minImages = 30\n", 310 | " \n", 311 | " #the default cloud cover filter value (%) if a better one cannot be found\n", 312 | " defaultFilter = 90\n", 313 | " \n", 314 | " #find the cloud cover value\n", 315 | " cxr.cellCloudCoverFilterValue = cxr.setFilterDecending(cxr.gridCellCollection, startFilter, interval, iterations, minImages, defaultFilter)\n", 316 | " \n", 317 | "#STEP 7: Filter the collection based on the cloud cover value\n", 318 | " cxr.gridCellCollection = cxr.gridCellCollection.filterMetadata('gridCellCoverage', 'greater_than', cxr.cellCloudCoverFilterValue)\n", 319 | "\n", 320 | "#STEP 8: Convert the images in the collection into NDWI images and identify the water in each image\n", 321 | " \n", 322 | " #calulate the NDWI for each image\n", 323 | " gridCellNDWICollection = ee.ImageCollection(cxr.gridCellCollection).map(cxr.NDWI)\n", 324 | " \n", 325 | " #set the threshold for water extraction from the NDWI image\n", 326 | " cxr.ndwiThreshold = 0.2\n", 327 | "\n", 328 | " gridCellNDWIWaterCollection = gridCellNDWICollection.map(cxr.ndwiWater)\n", 329 | " \n", 330 | "#STEP 9: Calculate the water occurrence of the collection\n", 331 | "\n", 332 | " #count the numer of water occurrences at each pixel\n", 333 | " waterReduceSum = gridCellNDWIWaterCollection.reduce(ee.Reducer.sum()).int16().rename('waterOccurrenceCount')\n", 334 | " \n", 335 | " #remove pixels that only have 1 for water occurence - QA check - review this\n", 336 | " waterReduceSum = waterReduceSum.where(waterReduceSum.eq(1), 0)\n", 337 | "\n", 338 | " #calculates the water occurence % \n", 339 | " cxr.gridCellNDWICollectionSize = gridCellNDWICollection.size()\n", 340 | " waterPercentage = waterReduceSum.divide(cxr.gridCellNDWICollectionSize).multiply(100).rename('waterOccurrencePercentage').addBands(waterReduceSum)\n", 341 | "\n", 342 | " #adds the image collection size as a band\n", 343 | " mask = waterPercentage.select('waterOccurrenceCount').mask()\n", 344 | " \n", 345 | " #add a band within the water occurrence image that holds the number of images used to calculate the water occurrence\n", 346 | " bandCollectionLength = ee.Image.constant(cxr.gridCellNDWICollectionSize).uint16().rename('numberOfImagesAnalysed').updateMask(mask)\n", 347 | " gridCellWaterOccurrenceOutput = waterPercentage.addBands(bandCollectionLength)\n", 348 | " \n", 349 | "#STEP 10: Create a median NDWI band and add the band to the water occurrence image\n", 350 | " \n", 351 | " #calculate the median NDWI\n", 352 | " ndwiMedian = gridCellNDWICollection.median().rename('ndwiMedian')\n", 353 | " \n", 354 | " #add ndwiMedian as a band to the water occurrence image\n", 355 | " gridCellWaterOccurrenceOutput = gridCellWaterOccurrenceOutput.addBands(ndwiMedian)\n", 356 | " \n", 357 | "# #STEP 11: Create a feature collection, with each feature containing information about the images used\n", 358 | " \n", 359 | " #get the grid cell number and convert it to string\n", 360 | " cxr.cellNumber = cxr.gridCellFeature.get('cell').getInfo() \n", 361 | " cxr.cellNumberStr = str(cxr.cellNumber)\n", 362 | " \n", 363 | " #get the date of the earliest image \n", 364 | " cxr.earliestImage = ee.Date(cxr.gridCellCollection.sort('system:time_start', True).first().get('dateTime')).format(\"YYYY-MM-dd\")\n", 365 | " \n", 366 | " #get the date of the earliest image \n", 367 | " cxr.latestImage = ee.Date(cxr.gridCellCollection.sort('system:time_start', False).first().get('dateTime')).format(\"YYYY-MM-dd\")\n", 368 | " \n", 369 | " #add the image metadata to the feature properties\n", 370 | " gridCellWaterOccurrenceOutput = gridCellWaterOccurrenceOutput\\\n", 371 | " .set('numberOfImagesAnalysed', ee.Number(cxr.gridCellNDWICollectionSize))\\\n", 372 | " .set('analysisStart', ee.Date(cxr.startDate).format(\"YYYY-MM-dd\"))\\\n", 373 | " .set('analysisEnd', ee.Date(cxr.endDate).format(\"YYYY-MM-dd\"))\\\n", 374 | " .set('earliestImageAnalysed', cxr.earliestImage)\\\n", 375 | " .set('latestImageAnalysed', cxr.latestImage)\\\n", 376 | " .set('cellCloudCoverThreshold', ee.Number(cxr.cellCloudCoverFilterValue))\\\n", 377 | " .set('cell', ee.Number(cxr.cellNumber))\\\n", 378 | " .set('cellResolution', ee.Number(cxr.gridResolution))\\\n", 379 | " \n", 380 | "#STEP 12 - Add the grid cell feature to a collection of the other grid cell features \n", 381 | "\n", 382 | " #get the list of dates\n", 383 | " dateList = cxr.gridCellCollection.aggregate_array('dateTime')\n", 384 | "\n", 385 | " #append the cell geometry and properties to the main list\n", 386 | " gridCellFeatures = ee.List(dateList).map(cxr.featureProperties)\n", 387 | " \n", 388 | "#STEP 13: Export images\n", 389 | "\n", 390 | " #EXPORT\n", 391 | " #generate the export file name\n", 392 | " exportFileName = str(cxr.gridResolution) + \"_\" + cxr.cellNumberStr + \"_\" + aoiName + \"_\" + cxr.startDate + \"_\" + cxr.endDate\n", 393 | " exportPath = \"users/\" + user + \"/\" + exportFolderPath + \"/\" + exportCollection + \"/\" + exportFileName \n", 394 | " exportPathGrid = \"users/\" + user + \"/\" + exportFolderPathGridCellSummaryTemp + \"/\" + exportCollection + \"/\" + exportFileName + '_GCS'\n", 395 | " \n", 396 | " #create the export task and start it \n", 397 | " task = ee.batch.Export.image.toAsset(**{\n", 398 | " 'image': gridCellWaterOccurrenceOutput,\n", 399 | " 'description': \"CoastXRay: \" + cxr.cellNumberStr, \n", 400 | " 'assetId': exportPath,\n", 401 | " 'region': cxr.gridCell.getInfo()['coordinates'],\n", 402 | " 'scale': 10\n", 403 | " })\n", 404 | " task.start()\n", 405 | " print('Started:' + exportFileName)\n", 406 | " \n", 407 | " #get the task ID and append it to the list\n", 408 | " taskID=task.status()['id']\n", 409 | " taskIDList.append(taskID)\n", 410 | "\n", 411 | " task = ee.batch.Export.table.toAsset(**{\n", 412 | " 'collection': ee.FeatureCollection(gridCellFeatures).sort('dateMilli'), \n", 413 | " 'description': \"CoastXRay CGS: \" + cxr.cellNumberStr,\n", 414 | " 'assetId': exportPathGrid\n", 415 | " })\n", 416 | "\n", 417 | " task.start()\n", 418 | " print('Started:' + exportFileName + '_GCS')" 419 | ] 420 | }, 421 | { 422 | "cell_type": "markdown", 423 | "metadata": {}, 424 | "source": [ 425 | "## 3.7 Merge Grid Cell Summary Features\n", 426 | "\n", 427 | "Each of the grid cells have a feature collection produced which summaries the images used. To make this easier to manage it is best to merge the all the feature collections in to one master dataset. \n", 428 | "\n", 429 | "**This should be run after all the cells have been processed on GEE.**\n" 430 | ] 431 | }, 432 | { 433 | "cell_type": "code", 434 | "execution_count": null, 435 | "metadata": {}, 436 | "outputs": [], 437 | "source": [ 438 | "# merge all the grid cell summary features into one feature collection\n", 439 | "\n", 440 | "# get a list of the features in the temporary directory\n", 441 | "assetList = ee.data.getList({'id': exportFolderPathGridCellSummaryTemp})\n", 442 | "\n", 443 | "# count the size and create a list range \n", 444 | "assetListSize = ee.List(assetList).size().getInfo()\n", 445 | "listRange = list(range(0, assetListSize-1))\n", 446 | "\n", 447 | "# create an empty feature collection\n", 448 | "fc = ee.FeatureCollection([])\n", 449 | "\n", 450 | "# get all the features and merge them into one feature collection\n", 451 | "for i in listRange:\n", 452 | " fc = fc.merge(ee.FeatureCollection(assetList[i]['id'])) \n", 453 | "\n", 454 | "# export the feature collection\n", 455 | "task = ee.batch.Export.table.toAsset(**{\n", 456 | " 'collection': fc, \n", 457 | " 'description': \"Grid Cell Summary Feature Collection\",\n", 458 | " 'assetId': \"users/\" + user + \"/\" + exportFolderPathGridCellSummary + \"/\" + exportCollection\n", 459 | " })\n", 460 | "\n", 461 | "task.start()" 462 | ] 463 | }, 464 | { 465 | "cell_type": "markdown", 466 | "metadata": {}, 467 | "source": [ 468 | "**END**" 469 | ] 470 | } 471 | ], 472 | "metadata": { 473 | "kernelspec": { 474 | "display_name": "Python 3", 475 | "language": "python", 476 | "name": "python3" 477 | }, 478 | "language_info": { 479 | "codemirror_mode": { 480 | "name": "ipython", 481 | "version": 3 482 | }, 483 | "file_extension": ".py", 484 | "mimetype": "text/x-python", 485 | "name": "python", 486 | "nbconvert_exporter": "python", 487 | "pygments_lexer": "ipython3", 488 | "version": "3.8.3" 489 | } 490 | }, 491 | "nbformat": 4, 492 | "nbformat_minor": 2 493 | } 494 | --------------------------------------------------------------------------------