├── .gitignore ├── 2025_satm.qmd ├── CHANGE_LOG.md ├── CITATION.cff ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── LICENSE.md ├── README.md ├── _quarto.yml ├── about.qmd ├── data ├── City_of_Boulder_City_Limits.zip ├── hawaii.geojson ├── landcover.geojson └── tower_data.csv ├── guides ├── appeears_ecostress.md └── earthdata_search_ecostress.md ├── img ├── AppEEARS_Explore.png ├── AppEEARS_downloadSample.png ├── AppEEARS_point_area.png ├── AppEEARS_signIn.png ├── AppEEARS_viewSample.png ├── cloud_access_documentation.png ├── ecostress_iss.png ├── eds_cloud_access_info_ecostress.png ├── eds_cloud_product_search_ECOSTRESS.png ├── eds_collection_search_ECOSTRESS.png ├── eds_ecov002_l2t_lste_http_download.png ├── eds_ecov002_l2t_lste_multi_granule_download.png ├── eds_ecov002_l2t_lste_multi_granule_download_https_links_src.png ├── eds_ecov002_l2t_lste_multi_granule_download_s3_links_src.png ├── eds_ecov002_l2t_lste_multi_granule_selection.png ├── eds_ecov002_l2t_lste_s3_access.png └── lp_jpl_vitals_light.png ├── index.qmd ├── python ├── how-tos │ ├── how_to_direct_access_http_ecostress_cog.ipynb │ └── how_to_direct_access_s3_ecostress_cog.ipynb ├── scripts │ ├── ecostress_swath2grid │ │ ├── ECOSTRESS_swath2grid.py │ │ └── swath2grid.md │ └── extract_geolocation_flag │ │ ├── ECOSTRESS_geolocation.py │ │ └── geolocation.md └── tutorials │ ├── ECOSTRESS_Tutorial.ipynb │ ├── Work_with_ECOSTRESS_L2T_LSTE_data.ipynb │ └── Working_with_ECOSTRESS_Tiled_data.ipynb ├── setup.md └── styles.css /.gitignore: -------------------------------------------------------------------------------- 1 | *.h5 2 | *.tif 3 | _MACOSX* 4 | __MACOSX/ 5 | tutorial_aj_notes/ 6 | tutorial_aj_notes.zip 7 | .ipynb_checkpoints/ 8 | img/ 9 | /.quarto/ 10 | /_site/* 11 | /_freeze/ 12 | 13 | -------------------------------------------------------------------------------- /2025_satm.qmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: 2025 ECOTRESS Science and Application Team Meeting 3 | jupyter: 4 | jupytext: 5 | text_representation: 6 | extension: .qmd 7 | format_name: quarto 8 | format_version: '1.0' 9 | jupytext_version: 1.17.3 10 | kernelspec: 11 | display_name: Python 3 (ipykernel) 12 | language: python 13 | name: python3 14 | --- 15 | 16 | 17 | Welcome to 2025 ECOSTRESS Science and Applications Team Meeting workshop inspired by the [NASA Space Apps Challenge](https://www.spaceappschallenge.org/)! 18 | In this hands-on workshop, you will learn different ways to find, access, and explore ECOSTRESS data. You will also have two hours to create the most compelling ECOSTRESS image or application. The winning submission will be featured on the ECOSTRESS website. 19 | 20 | This session is hosted by NASA [Land Processes Distributed Active Archive Center (LP DAAC)](https://lpdaac.usgs.gov/) and [NASA Jet Propulsion Laboratory (JPL)](https://www.jpl.nasa.gov/) with support from the NASA [Openscapes](https://nasa-openscapes.github.io/) project. 21 | Hands-on exercises will be executed from a [Jupyter Hub on the Openscapes 2i2c cloud instance](https://workshop.openscapes.2i2c.cloud/). 22 | 23 | ## Learning Objectives 24 | - LP DAAC ECOSTRESS data updates 25 | - Ways to find and access ECOSTRESS data 26 | - How to explore ECOSTRESS tiled data using Earthdata Search 27 | - How to explore ECOSTRESS tiled data programmatically 28 | - How to evaluate quality of ECOSTESS data 29 | 30 | ## Slides 31 | 32 | 33 | 34 | ## Knowledge & Career Level: 35 | Beginner, intermediate 36 | 37 | ## Contact Info: 38 | Email: LPDAAC@usgs.gov 39 | Voice: +1-866-573-3222 40 | Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹ 41 | Website: -------------------------------------------------------------------------------- /CHANGE_LOG.md: -------------------------------------------------------------------------------- 1 | # Change Log 2 | 3 | All notable changes to this project will be documented in this file. 4 | The format is based on [Keep a Changelog](http://keepachangelog.com/) 5 | and this project adheres to [Semantic Versioning](http://semver.org/). 6 | _________________________________________________________________________ 7 | ## 2025-09-12 8 | 9 | > - Added python/tutorials/Work_with_ECOSTRESS_Tiled_data.ipynb 10 | > - Created a webbook 11 | > - Added information for ECOSTESS SATM 2025 12 | 13 | ## 2025-03-28 14 | 15 | > - Added How to direct access notebooks 16 | > - Added Working with ECOSTRESS L2T LSTE Data Tutorial 17 | 18 | ## 2023-09-27 19 | 20 | > - Added the extract_geolocation script 21 | 22 | ## 2023-09-27 23 | 24 | > - Added Existing ECOSTRESS resources 25 | > - Added Updated README -------------------------------------------------------------------------------- /CITATION.cff: -------------------------------------------------------------------------------- 1 | cff-version: 1.2.0 2 | title: ECOSTRESS Data Resources 3 | message: >- 4 | If you use this collection of resources in your research 5 | we would appreciate an appropriate citation. 6 | type: software 7 | authors: 8 | - given-names: Mahsa 9 | family-names: Jami 10 | affiliation: >- 11 | KBR, Inc., Contractor to the U.S. Geological Survey, 12 | Earth Resources Observation and Science Center, 13 | National Aeronautics and Space Administration Land 14 | Processes Distributed Active Archive Center 15 | orcid: 'https://orcid.org/0000-0002-3594-3004' 16 | - given-names: Erik A 17 | family-names: Bolch 18 | affiliation: >- 19 | KBR, Inc., Contractor to the U.S. Geological Survey, 20 | Earth Resources Observation and Science Center, 21 | National Aeronautics and Space Administration Land 22 | Processes Distributed Active Archive Center 23 | orcid: 'https://orcid.org/0000-0003-2470-4048' 24 | - given-names: Cole K 25 | family-names: Krehbiel 26 | affiliation: >- 27 | National Aeronautics and Space Administration Land 28 | Processes Distributed Active Archive Center, 29 | U.S. Geological Survey, Earth Resources 30 | Observation and Science Center 31 | orcid: 'https://orcid.org/0000-0003-2645-3449' 32 | - given-names: Aaron M 33 | family-names: Friesz 34 | affiliation: >- 35 | KBR, Inc., Contractor to the U.S. Geological Survey, 36 | Earth Resources Observation and Science Center, 37 | National Aeronautics and Space Administration Land 38 | Processes Distributed Active Archive Center 39 | orcid: 'https://orcid.org/0000-0003-4096-3824' 40 | repository-code: 'https://github.com/nasa/EMIT-Data-Resources' 41 | abstract: >- 42 | This repository contains guides and Jupyter Notebooks to 43 | help users access and work with data from the Ecosystem 44 | Spaceborne Thermal Radiometer Experiment on Space Station 45 | (ECOSTRESS) mission. 46 | license: Apache-2.0 47 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Code of Conduct 2 | 3 | ## 1. Our Commitment 4 | 5 | We are dedicated to fostering a respectful environment for everyone contributing to this project. We expect all participants to treat each other with respect, professionalism, and kindness. 6 | 7 | ## 2. Expected Behavior 8 | 9 | - Be respectful and considerate of others. 10 | - Engage in constructive discussions and offer helpful feedback. 11 | - Gracefully accept constructive criticism. 12 | 13 | ## 3. Unacceptable Behavior 14 | 15 | The following behaviors will not be tolerated: 16 | 17 | - Harassment, discrimination, or intimidation of any kind. 18 | - Offensive, abusive, or derogatory language and actions. 19 | - Personal attacks or insults. 20 | - Trolling or disruptive conduct. 21 | - Sharing inappropriate content. 22 | 23 | ## 4. Reporting Violations 24 | If you experience or witness any behavior that violates this Code of Conduct, please report it by contacting the project maintainers. All reports will be reviewed confidentially. 25 | 26 | ## 5. Enforcement 27 | Violations of this Code of Conduct may result in actions such as warnings, temporary bans, or permanent exclusion from participation at the discretion of the maintainers. 28 | 29 | ## Contact Info 30 | Email: 31 | Voice: +1-866-573-3222 32 | Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹ 33 | Website: 34 | Date last modified: 01-22-2025 35 | 36 | ¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I. 37 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to Our GitHub 2 | 3 | This page is your one-stop shop for uncovering how to contribute to our Github! 4 | 5 | ## We Want Your Help! 6 | 7 | No, really, we do! Please come and participate in our community and lets do science together! Depending on your level of interaction with the Land Processes Data Active Archive Center (LP DAAC) and the LP DAAC GitHub, visitors to the site can be described as: 8 | - A **community member**: anyone in the open science community who visits a LP DAAC site, utilizes LP DAAC online tools, or attends a LP DAAC event. 9 | - A **participant**: anyone who posts a comment or poses a question in the *GitHub Discussion Space*, reports a site bug or requests a new resource in *GitHub Issues*, or attends a LP DAAC event and utilizes any virtual chat features during that event. 10 | - A **contributor**: anyone who forks this GitHub repository and submits pull requests to make additions or changes to the posted content. 11 | 12 | Everyone reading this page is a community member, and we hope everyone will post comments and join discussions as a participant. Contributors are welcome, particularly to help find and point to other open science resources. 13 | 14 | ## Ways to Contribute to the GitHub 15 | There are two main ways to contribute to the LP DAAC GitHub. 16 | - **Suggest a change, addition, or deletion to what is already on the GitHub using [Issues](https://github.com/nasa/Transform-to-Open-Science/issues).** Issues can be about any LP DAAC plans, timelines, and content. 17 | - Before submitting a new [issue](https://github.com/nasa/LPDAAC-Data-Resources/issues), check to make sure [a similar issue isn't already open](https://github.com/nasa/LPDAAC-Data-Resources/issues). If one is, contribute to that issue thread with your feedback. 18 | - When submitting a bug report, please try to provide as much detail as possible, i.e. a screenshot or [gist](https://gist.github.com/) that demonstrates the problem, the technology you are using, and any relevant links. 19 | - Issues labeled :sparkles:[`help wanted`](https://github.com/nasa/LPDAAC-Data-Resources/labels/help%20wanted):sparkles: make it easy for you to find ways you can contribute today. 20 | - **Become a contributor!** [Fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) the repository and [make commits](https://docs.github.com/en/get-started/quickstart/contributing-to-projects#making-and-pushing-changes) to add resources and additional materials. Here are some ways you can contribute: 21 | - by reporting bugs 22 | - by suggesting new features 23 | - by translating content to a new language 24 | - by writing or editing documentation 25 | - by writing specifications 26 | - by writing code and documentation (**no pull request is too small**: fix typos, add code comments, clean up inconsistent whitespace) 27 | - by closing [issues](https://github.com/nasa/LPDAAC-Data-Resources/issues) 28 | 29 | In the spirit of open source software, everyone is encouraged to help improve this project! 30 | 31 | ## New to GitHub? Start here! 32 | You can [sign up for GitHub here](https://github.com/)! The NASA Transform to Open Science Team has made a short video demonstrating how to make an easy pull request [here](https://youtu.be/PHoScPeMWHI). 33 | 34 | For a more in-depth start, we suggest *[Getting Started with Git and GitHub: The Complete Beginner’s Guide](https://towardsdatascience.com/getting-started-with-git-and-github-6fcd0f2d4ac6)* and *[The Beginners Guide to Git and GitHub](https://www.freecodecamp.org/news/the-beginners-guide-to-git-github/)*. We've summarized some of the most important points below. 35 | 36 | ### Making a Change 37 | *This section is attributed to [NumFOCUS](https://github.com/numfocus/getting-started-with-open-source/blob/master/CONTRIBUTING.md) and [Adrienne Friend](https://github.com/adriennefriend/imposter-syndrome-disclaimer).* 38 | 39 | Once you've identified something you'd like to help with you're ready to make a change to the project repository! 40 | 41 | 1. First, describe what you're planning to do as a comment to the issue, (and this might mean making a new issue). 42 | 43 | [This blog](https://www.igvita.com/2011/12/19/dont-push-your-pull-requests/) is a nice explanation of why putting this work in up front is so useful to everyone involved. 44 | 45 | 2. Fork this repository to your profile. 46 | 47 | You can now do whatever you want with this copy of the project. You won't mess up anyone else's work so you're super safe. 48 | 49 | Make sure to [keep your fork up to date]( https://github.com/KirstieJane/STEMMRoleModels/wiki/Syncing-your-fork-to-the-original-repository-via-the-browser) with the master repository. 50 | 51 | 3. Make the changes you've discussed. 52 | 53 | Try to keep the changes focused rather than changing lots of things at once. If you feel tempted to branch out then please *literally* branch out: create separate branches for different updates to make the next step much easier! 54 | 55 | 4. Submit a pull request. 56 | 57 | A member of the executive team will review your changes, have a bit of discussion and hopefully merge them in! 58 | 59 | N.B. you don't have to be ready to merge to make a pull request! We encourage you to submit a pull request as early as you want to. They help us to keep track of progress and help you to get earlier feedback. 60 | 61 | ## Development Model 62 | 63 | For accepting new contributions, TOPS uses the [forking workflow](https://guides.github.com/activities/forking/). As the first step of your contribution, you'll want to fork this repository, make a local clone of it, add your contribution, and then create a pull request back to the LP DAAC repository. 64 | 65 | All documentation should be written using Markdown and Github Markdown-supported HTML. 66 | 67 | ## Attribution 68 | These contributing guidelines are adapted from the NASA Transform to Open Science github, available at https://github.com/nasa/Transform-to-Open-Science/blob/main/CONTRIBUTING.md. 69 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ECOSTRESS-Data-Resources 2 | 3 | Welcome! This repository provides guides, short how-tos, and tutorials to help users access and work with data from the Ecosystem Spaceborne Thermal Radiometer Experiment on Space Station ([ECOSTRESS](https://ecostress.jpl.nasa.gov/)) mission distributed by the Land Processes Distributed Active Archive Center ([LP DAAC](https://lpdaac.usgs.gov/)). In the interest of open science this repository has been made public but is still under active development. All jupyter notebooks and scripts should be functional, however, changes or additions may be made. Contributions from all parties are welcome. 4 | 5 | --- 6 | 7 | ## ECOSTRESS Background 8 | 9 | The ECOsystem Spaceborne Thermal Radiometer Experiment on Space Station ([ECOSTRESS](https://lpdaac.usgs.gov/data/get-started-data/collection-overview/#ECOSTRESS_anchor)) is aboard the International Space Station (ISS) and measures the temperature of plants to better understand how much water plants need and how they respond to stress. ECOSTRESS was launched to the ISS on June 29, 2018. It has a viewing swath width of around 384 km and views the surface of the Earth from 53.6° N latitude to 53.6° S latitude with variable revisit times, dependent on the orbit of the ISS. 10 | 11 | ECOSTRESS addresses three overarching science questions: How is the terrestrial biosphere responding to changes in water availability? How do changes in diurnal vegetation water stress impact the global carbon cycle? Can agricultural vulnerability be reduced through advanced monitoring of agricultural water consumptive use and improved drought estimation? ECOSTRESS uses a multispectral thermal infrared radiometer to measure the surface temperature. The radiometer obtains detailed images of the Earth’s surface at ~70 m spatial resolution that can provide information on the temperature of an individual farmer’s field. Learn more on the [ECOSTRESS website](https://ecostress.jpl.nasa.gov/). 12 | 13 | ECOSTRESS Data Products are distributed by the [LP DAAC](https://lpdaac.usgs.gov/). Learn more about ECOSTRESS data products from [ECOSTRESS Product Pages](https://lpdaac.usgs.gov/product_search/?query=ECOSTRESS&status=Operational&view=cards&sort=title&page=1&per_page=30) and search for and download ECOSTRESS data products using [NASA EarthData Search](https://search.earthdata.nasa.gov/search?q=ECOSTRESS) or programmatically using NASA's [Common Metadata Repository](https://cmr.earthdata.nasa.gov/search)(CMR). 14 | 15 | --- 16 | 17 | ## Prerequisites/Setup Instructions 18 | 19 | Instructions for setting up a compatible environment for working with ECOSTRESS data is linked below. 20 | - [`Python` set up instructions](https://github.com/nasa/LPDAAC-Data-Resources/tree/main/setup/setup_instructions_python.md) 21 | 22 | --- 23 | ## Getting Started 24 | 25 | Clone or download the [ECOSTRESS-Data-Resources repository](https://github.com/nasa/ECOSTRESS-Data-Resources). 26 | 27 | - [Download](https://github.com/nasa/ECOSTRESS-Data-Resources/archive/refs/heads/main.zip) 28 | - To clone the repository, type `git clone https://github.com/nasa/ECOSTRESS-Data-Resources.git` in the command line. 29 | 30 | --- 31 | ## Repository Contents 32 | 33 | Content in this repository includes Python tutorials, how-tos, scripts, defined modules that will be called from the Python resources, and setup instructions. The supporting files for use cases are stored in `Data` folder. 34 | > Resources stored in this repository are listed below: 35 | 36 | | Repository Contents | Type | Summary | 37 | |----|-----|----| 38 | | **[earthdata_search_ecostress.md](/guides/earthdata_search_ecostress.md)** | Markdown Guide | Demonstrates how to work with Earthdata Search to access ECOSTRESS data| 39 | | **[appeears_ecostress.md](/guides/appeears_ecostress.md)** | Markdown Guide | Demonstrates how to work with AppEEARS to access and transform ECOSTRESS data| 40 | | **[how_to_direct_access_http_ecostress_cog.ipynb](/python/how-tos/how_to_direct_access_http_ecostress_cog.ipynb)** | Jupyter Notebook | Demonstrates how to stream ECOSTRESS COG data from the Earthdata Cloud using HTTP links| 41 | | **[how_to_direct_access_s3_ecostress_cog.ipynb](/python/how-tos/how_to_direct_access_s3_ecostress_cog.ipynb)** | Jupyter Notebook | Demonstrates how to directly access ECOSTRESS COG data from the Earthdata Cloud using S3 links| 42 | | **[ECOSTRESS_Tutorial.ipynb](/python/tutorials/ECOSTRESS_Tutorial.ipynb)** | Jupyter Notebook | Demonstrates how to work with the ECOSTRESS Evapotranspiration PT-JPL Daily L3| 43 | | **[ECOSTRESS_swath2grid.py](/python/scripts/ecostress_swath2grid)** | Command line executable | Demonstrates how to convert ECOSTRESS swath data products into projected GeoTIFFs| 44 | | **[ECOSTRESS_geolocation.py](/python/scripts/extract_geolocation_flag)** | Command line executable | Demonstrates how to extract `GeolocationAccuracyQA` flag for ECOSTRESS version 2 data| 45 | 46 | --- 47 | 48 | ## Helpful Links 49 | 50 | + [ECOSTRESS Collection Overview](https://lpdaac.usgs.gov/data/get-started-data/collection-overview/missions/ecostress-overview/) 51 | + [NASA JPL ECOSTRESS Tutorials](https://ecostress.jpl.nasa.gov/tutorials) 52 | + [2023 ECOSTRESS Science and Applications Team Meeting Workshop Recording](https://ecostress.jpl.nasa.gov/downloads/science_team_meetings/2023/workshopvid.mp4) 53 | + [ECOSTRESS Website](https://ecostress.jpl.nasa.gov/) 54 | + [AppEEARS Website](https://appeears.earthdatacloud.nasa.gov/) 55 | + [LP DAAC Website](https://lpdaac.usgs.gov/) 56 | + [LP DAAC GitHub](https://github.com/nasa/LPDAAC-Data-Resources) 57 | 58 | --- 59 | 60 | ## Contact Info: 61 | 62 | Email: LPDAAC@usgs.gov 63 | Voice: +1-866-573-3222 64 | Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹ 65 | Website: 66 | Date last modified: 04-15-2025 67 | 68 | ¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I. 69 | -------------------------------------------------------------------------------- /_quarto.yml: -------------------------------------------------------------------------------- 1 | project: 2 | type: website 3 | output-dir: _site 4 | 5 | execute: 6 | freeze: true # never re-execute computational content during project render, needed for .qmd and .rmd since quarto does not render .ipynb by default 7 | 8 | website: 9 | page-navigation: true 10 | page-footer: 11 | center: "ECOSTRESS Data Resources" 12 | title: "ECOSTRESS-Data-Resources" 13 | site-url: https://nasa.github.io/ECOSTRESS-Data-Resources/ 14 | repo-url: https://github.com/nasa/ECOSTRESS-Data-Resources/ 15 | sidebar: 16 | logo: img/lp_jpl_vitals_light.png 17 | title: ECOSTRESS Data Resources 18 | pinned: true 19 | align: center 20 | tools: 21 | - icon: globe 22 | href: https://nasa.github.io/ECOSTRESS-Data-Resources/ 23 | text: "ECOSTRESS Data Resources Website" 24 | - icon: github 25 | href: https://github.com/nasa/ECOSTRESS-Data-Resources/ 26 | text: "ECOSTRESS Data Resources Repository" 27 | style: "floating" 28 | search: true 29 | collapse-level: 1 30 | contents: 31 | - section: "Overview" 32 | contents: 33 | - text: "Welcome" 34 | href: index.qmd 35 | - section: "Workshops" 36 | contents: 37 | - text: "2025 ECOSTRESS Science and Application Team Meeting" 38 | href: 2025_satm.qmd 39 | - text: "Setup Instructions" 40 | href: setup.md 41 | - section: "Guides" 42 | contents: 43 | - text: "Search and Access ECOSTRESS Data using Earthdata Search" 44 | href: guides/earthdata_search_ecostress.md 45 | - text: "Access and Subset ECOSTRESS Data using AppEEARS" 46 | href: guides/appeears_ecostress.md 47 | - section: "Python Notebooks" 48 | contents: 49 | - text: "How to Directly Access ECOSTRESS Data (HTTP)" 50 | href: python/how-tos/how_to_direct_access_http_ecostress_cog.ipynb 51 | - text: "How to Directly Access ECOSTRESS Data (S3)" 52 | href: python/how-tos/how_to_direct_access_s3_ecostress_cog.ipynb 53 | - text: "Explore the ECOSTRESS Tiled Data" 54 | href: python/tutorials/Working_with_ECOSTRESS_Tiled_data.ipynb 55 | - section: "Scripts" 56 | contents: 57 | - text: "ECOSTRESS Swath to Grid Conversion Script" 58 | href: python/scripts/ecostress_swath2grid/swath2grid.md 59 | - text: "Extract ECOSTRESS Geolocation Accuracy QA Flag" 60 | href: python/scripts/extract_geolocation_flag/geolocation.md 61 | - section: "Contributing" 62 | contents: 63 | - href: CONTRIBUTING.md 64 | - href: CODE_OF_CONDUCT.md 65 | 66 | format: 67 | html: 68 | theme: 69 | - cosmo 70 | - brand 71 | css: styles.css 72 | toc: true 73 | 74 | 75 | 76 | -------------------------------------------------------------------------------- /about.qmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: "About" 3 | --- 4 | 5 | About this site 6 | -------------------------------------------------------------------------------- /data/City_of_Boulder_City_Limits.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/data/City_of_Boulder_City_Limits.zip -------------------------------------------------------------------------------- /data/hawaii.geojson: -------------------------------------------------------------------------------- 1 | {"type":"FeatureCollection","features":[{"type":"Feature","properties":{},"geometry":{"coordinates":[[[-156.08325817024362,19.761648519502998],[-155.94536162626957,19.055265378036196],[-155.68912658483336,18.884416413037513],[-154.7956649879457,19.49868320811794],[-155.15701319757088,19.977862511765977],[-155.88944801467946,20.289321000519536],[-156.08325817024362,19.761648519502998]]],"type":"Polygon"}}]} -------------------------------------------------------------------------------- /data/tower_data.csv: -------------------------------------------------------------------------------- 1 | Time,LE 2 | , 3 | 8/5/2018 9:30,18.0277515 4 | , 5 | 8/5/2018 10:00,68.49596979 6 | , 7 | 8/5/2018 10:30,89.05003071 8 | , 9 | 8/5/2018 11:00,154.5641991 10 | , 11 | 8/5/2018 11:30,118.6850899 12 | , 13 | 8/5/2018 12:00,107.0718696 14 | , 15 | 8/5/2018 12:30,189.5512486 16 | , 17 | 8/5/2018 13:00,129.9851118 18 | , 19 | 8/5/2018 13:30, 20 | , 21 | 8/5/2018 14:00,139.6517454 22 | , 23 | 8/5/2018 14:30,155.4161125 24 | , 25 | 8/5/2018 15:00,124.5119356 26 | , 27 | 8/5/2018 15:30,106.666146 28 | , 29 | 8/5/2018 16:00,176.3211116 30 | , 31 | 8/5/2018 16:30,203.3630579 32 | , 33 | 8/5/2018 17:00,166.7149037 34 | , 35 | 8/5/2018 17:30,124.2360275 36 | , 37 | 8/5/2018 18:00,96.23245004 38 | , 39 | 8/5/2018 18:30,133.1213652 40 | , 41 | 8/5/2018 19:00,44.66026575 42 | , 43 | 8/5/2018 19:30,11.61805179 44 | , 45 | 8/5/2018 20:00,44.20149188 46 | -------------------------------------------------------------------------------- /guides/appeears_ecostress.md: -------------------------------------------------------------------------------- 1 | 2 | # Application for Extracting and Exploring Analysis Ready Samples (AρρEEARS) 3 | 4 | 5 | The Application for Extracting and Exploring Analysis Ready Samples ([AppEEARS](https://appeears.earthdatacloud.nasa.gov/)) offers a simple and efficient way to access and transform geospatial data from a variety of federal data archives in an easy-to-use web application interface. AppEEARS enables users to subset geospatial data spatially, temporally, and by band/layer for point and area samples. AppEEARS returns not only the requested data, but also the associated quality values, and offers interactive visualizations with summary statistics in the web interface. This tutorial shows how to use AppEEARS to access ECOSTRESS version 2 data hosted in the cloud. 6 | 7 | ### Step 1. Sign in 8 | 9 | Sign in using your Earthdata login credentials. If you do not have an Earthdata account, Create an Earthdata Login account at https://urs.earthdata.nasa.gov. 10 | ![*Figure caption: AppEEARS Sign In*](../img/AppEEARS_signIn.png) 11 | 12 | ### Step 2. Extract the Sample 13 | 14 | Select the Point or Area sample using Extract dropdown. You will be directed to Extract Area or Point Sample page. 15 | 1. Enter your sample name. 16 | 2. Upload your **area of interest** or draw a polygon on the leaflet map for area sample. For point sample, provide the CSV file including the latitude and longitude coordinates. You can also use the map to manually select your locations or type them directly. 17 | 3. Select your **time period of interest**. 18 | 4. Add **datasets** you are interested in to your Selected Layers. You can choose from various [data collections available in AppEEARS](https://appeears.earthdatacloud.nasa.gov/products). You can click on the (i) icon for the dataset to see more details. 19 | In this example we are interested in the [ECOSTRESS LSTE](https://doi.org/10.5067/ECOSTRESS/ECO_L2_LSTE.002) which is managed by the LP DAAC and made available from the NASA Earthdata Cloud archive hosted in the AWS cloud. 20 | 21 | 5. For area sample, you can select your **output file format**. You also have an option to **reproject** all your layers to another coordinate reference system. 22 | 6. Now you can **submit**. 23 | 24 | ![*Figure caption: Extract area and point sample for ECOSTRESS data available in AWS cloud in AppEEARS*](../img/AppEEARS_point_area.png) 25 | 26 | Once your request is complete, you can **View** and **Download** your results from the Explore Requets page. 27 | 28 | ![*Figure caption: Refine search*](../img/AppEEARS_Explore.png) 29 | 30 | ### Step 3. Explore the outputs 31 | From the Explore Requests page, click the View icon in order to view and interact with your results. This will take you to the View Area Sample page. 32 | ![*Figure caption: View Sample Results*](../img/AppEEARS_viewSample.png) 33 | 34 | 35 | ### Step 4. Download the outputs 36 | 37 | Finally navigate to Download Sample page by clicking the Download icon on the Explore Requests page or from View Sample page to download your results. Besides your actual outputs, you will have access to supporting files including a text file with URLs to source data, a JSON file you can use to recreate the same sample, decoded quality information, and CSV file with the layer statistics. 38 | 39 | ![*Figure caption: Download Sample Results*](../img/AppEEARS_downloadSample.png) 40 | 41 | 42 | Check out AppEEARS [help documentation](https://appeears.earthdatacloud.nasa.gov/help) for more details. If you wish to access AppEEARS programatically check out [AppEEARS API documenation](https://appeears.earthdatacloud.nasa.gov/api/). 43 | -------------------------------------------------------------------------------- /guides/earthdata_search_ecostress.md: -------------------------------------------------------------------------------- 1 | # Earthdata Search 2 | 3 | This tutorial guides you through how to use [Earthdata Search](https://search.earthdata.nasa.gov/) for NASA Earth observations search and discovery, and how to connect the search output (e.g. download or access links) to a programmatic workflow (locally or from within the cloud). 4 | 5 | ### Step 1. Go to Earthdata Search and Login 6 | 7 | Go to Earthdata Search and use your Earthdata login credentials to log in. If you do not have an Earthdata account, please see the [Workshop Prerequisites](https://nasa-openscapes.github.io/2022-Fall-ECOSTRESS-Cloud-Workshop/prerequisites/) for guidance. 8 | 9 | ### Step 2. Search for dataset of interest 10 | 11 | Use the search box in the upper left to type key words. In this example we are interested in the [ECOSTRESS LSTE](https://search.earthdata.nasa.gov/search/granules?p=C2076090826-LPCLOUD&pg[0][v]=f&pg[0][gsk]=-start_date&q=ECOSTRESS&ff=Available%20from%20AWS%20Cloud&tl=1666964694.567!3!!&lat=20.601562500000004&long=-71.3671875) which is managed by the LP DAAC and made available from the NASA Earthdata Cloud archive hosted in AWS cloud. 12 | 13 | Type **ECOSTRESS** in the search bar, then click on the **"Available from AWS Cloud"** filter option on the left. 14 | 15 | ![*Figure caption: Search for ECOSTRESS data available in AWS cloud in Earthdata Search portal*](../img/eds_collection_search_ECOSTRESS.png) 16 | 17 | Let's refine our search further. Now, let's search for `ECOSTRESS ECO_L2T_LSTE` in the search box. A single Earthdata Search Collection is returned. 18 | 19 | Click on the (i) icon for the dataset to read more details, including the **dataset shortname** (helpful for programmatic workflows) just below the dataset name; here `ECO_L2T_LSTE`. 20 | 21 | ![*Figure caption: Refine search*](../img/eds_cloud_product_search_ECOSTRESS.png) 22 | 23 | ### Step 3. Explore the dataset details, including Cloud Access information 24 | 25 | Once we click the (i), scrolling down the info page for the dataset we will see Cloud Access information, such as: 26 | 27 | - whether the dataset is available in the cloud 28 | - the cloud **Region** (all NASA Earthdata Cloud data is/will be in `us-west-2` region) 29 | - the S3 storage **bucket** and **object prefix** where this data is located 30 | - link that generates **AWS S3 Credentials** for in-cloud data access (we will cover this in the Direct Data Access Tutorials) 31 | - link to **documentation** describing the In-region Direct S3 Access to Buckets. *Note*: these will be unique depending on the DAAC where the data is archived. (We will show examples of direct in-region access in Tutorial 3.) 32 | 33 | ![*Figure caption: Cloud access info in EDS*](../img/eds_cloud_access_info_ecostress.png) 34 | 35 | ![*Figure caption: Documentation describing the In-region Direct S3 Access to Buckets*](../img/cloud_access_documentation.png) 36 | 37 | **Note**: Clicking on "For Developers" to expand will provide programmatic endpoints such as those for the CMR API, and more. 38 | 39 | For now, let's say we are interested in getting download link(s) or access link(s) for specific data files (granules) within this collection. 40 | 41 | At the top of the dataset info section, click on *Search Results*, which will take us back to the list of datasets matching our search parameters. Clicking on the dataset (`ECOSTRESS ECO_L2T_LSTE`) displaying a list of files (granules) that are part of the dataset (collection). 42 | 43 | ### Step 4a. Download or data access for a single granule 44 | 45 | To download files for a granule click the download arrow on the card (or list row) 46 | 47 | ![*Figure caption: Download granules*](../img/eds_ecov002_l2t_lste_http_download.png) 48 | 49 | You can also get the S3 information (e.g., AWS region, bucket, temperary credentials for S3 access, and file names) by selecting the **AWS S3 Access** tab. 50 | 51 | ![*Figure caption: S3 access for granules*](../img/eds_ecov002_l2t_lste_s3_access.png) 52 | 53 | #### Step 4b. Download or data access for multiple granule 54 | 55 | To download multiple granules, click on the green **+** symbol to add files to our project. Click on the green button towards the bottom that says "Download". This will take us to another page with options to customize our download or access link(s). 56 | 57 | ![*Figure caption: Select granules and click download*](../img/eds_ecov002_l2t_lste_multi_granule_selection.png) 58 | 59 | On the next page click the **Direct Download** option and click the green **Download Data** on the bottom left side of the page. 60 | 61 | ![*Figure caption: Direct download multiple granules*](../img/eds_ecov002_l2t_lste_multi_granule_download.png) 62 | 63 | We're now taken to the final page for instructions to download and links for data access in the cloud. You should see three tabs: `Download Files`, `AWS S3 Access`, `Download Script`: 64 | 65 | ![*Figure caption: Download to local*](../img/eds_ecov002_l2t_lste_multi_granule_download_https_links_src.png) 66 | 67 | ![*Figure caption: Direct S3 access*](../img/eds_ecov002_l2t_lste_multi_granule_download_s3_links_src.png) 68 | 69 | The *Download Files* tab provides the `https://` links for downloading the files locally 70 | 71 | The *AWS S3 Access* tab provides the `S3://` links, which is what we would use to access the data directly in-region (us-west-2) within the AWS cloud. 72 | -------------------------------------------------------------------------------- /img/AppEEARS_Explore.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/AppEEARS_Explore.png -------------------------------------------------------------------------------- /img/AppEEARS_downloadSample.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/AppEEARS_downloadSample.png -------------------------------------------------------------------------------- /img/AppEEARS_point_area.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/AppEEARS_point_area.png -------------------------------------------------------------------------------- /img/AppEEARS_signIn.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/AppEEARS_signIn.png -------------------------------------------------------------------------------- /img/AppEEARS_viewSample.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/AppEEARS_viewSample.png -------------------------------------------------------------------------------- /img/cloud_access_documentation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/cloud_access_documentation.png -------------------------------------------------------------------------------- /img/ecostress_iss.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/ecostress_iss.png -------------------------------------------------------------------------------- /img/eds_cloud_access_info_ecostress.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_cloud_access_info_ecostress.png -------------------------------------------------------------------------------- /img/eds_cloud_product_search_ECOSTRESS.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_cloud_product_search_ECOSTRESS.png -------------------------------------------------------------------------------- /img/eds_collection_search_ECOSTRESS.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_collection_search_ECOSTRESS.png -------------------------------------------------------------------------------- /img/eds_ecov002_l2t_lste_http_download.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_ecov002_l2t_lste_http_download.png -------------------------------------------------------------------------------- /img/eds_ecov002_l2t_lste_multi_granule_download.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_ecov002_l2t_lste_multi_granule_download.png -------------------------------------------------------------------------------- /img/eds_ecov002_l2t_lste_multi_granule_download_https_links_src.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_ecov002_l2t_lste_multi_granule_download_https_links_src.png -------------------------------------------------------------------------------- /img/eds_ecov002_l2t_lste_multi_granule_download_s3_links_src.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_ecov002_l2t_lste_multi_granule_download_s3_links_src.png -------------------------------------------------------------------------------- /img/eds_ecov002_l2t_lste_multi_granule_selection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_ecov002_l2t_lste_multi_granule_selection.png -------------------------------------------------------------------------------- /img/eds_ecov002_l2t_lste_s3_access.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/eds_ecov002_l2t_lste_s3_access.png -------------------------------------------------------------------------------- /img/lp_jpl_vitals_light.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/nasa/ECOSTRESS-Data-Resources/20dfefbfb793d924aa6b222ddc4c9464e66ff4a6/img/lp_jpl_vitals_light.png -------------------------------------------------------------------------------- /index.qmd: -------------------------------------------------------------------------------- 1 | --- 2 | title: ECOSTRESS Data Resources 3 | jupyter: 4 | jupytext: 5 | text_representation: 6 | extension: .qmd 7 | format_name: quarto 8 | format_version: '1.0' 9 | jupytext_version: 1.17.2 10 | kernelspec: 11 | display_name: Python 3 (ipykernel) 12 | language: python 13 | name: python3 14 | --- 15 | 16 |

17 | 18 |

19 | 20 | **Welcome!** This repository provides guides, short how-tos, and tutorials to help users access and work with data from the Ecosystem Spaceborne Thermal Radiometer Experiment on Space Station ([ECOSTRESS](https://ecostress.jpl.nasa.gov/)) mission distributed by the Land Processes Distributed Active Archive Center ([LP DAAC](https://www.earthdata.nasa.gov/centers/lp-daac)). In the interest of open science this repository has been made public but is still under active development. All jupyter notebooks and scripts should be functional, however, changes or additions may be made. Contributions from all parties are welcome. 21 | 22 | 23 | ## ECOSTRESS Background 24 | 25 | The ECOsystem Spaceborne Thermal Radiometer Experiment on Space Station ([ECOSTRESS](https://www.earthdata.nasa.gov/data/instruments/ecostress)) is aboard the International Space Station (ISS) and measures the temperature of plants to better understand how much water plants need and how they respond to stress. ECOSTRESS was launched to the ISS on June 29, 2018. It has a viewing swath width of around 384 km and views the surface of the Earth from 53.6° N latitude to 53.6° S latitude with variable revisit times, dependent on the orbit of the ISS. 26 | 27 | ECOSTRESS addresses three overarching science questions: How is the terrestrial biosphere responding to changes in water availability? How do changes in diurnal vegetation water stress impact the global carbon cycle? Can agricultural vulnerability be reduced through advanced monitoring of agricultural water consumptive use and improved drought estimation? ECOSTRESS uses a multispectral thermal infrared radiometer to measure the surface temperature. The radiometer obtains detailed images of the Earth’s surface at ~70 m spatial resolution that can provide information on the temperature of an individual farmer’s field. Learn more on the [ECOSTRESS website](https://ecostress.jpl.nasa.gov/). 28 | 29 | 30 | ## Contact Info: 31 | 32 | Email: LPDAAC@usgs.gov 33 | Voice: +1-866-573-3222 34 | Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹ 35 | Website: -------------------------------------------------------------------------------- /python/how-tos/how_to_direct_access_http_ecostress_cog.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "persistent-death", 6 | "metadata": {}, 7 | "source": [ 8 | "# How to: Directly Access ECOSTRESS Data (HTTP)\n", 9 | "\n", 10 | "**Summary**\n", 11 | "\n", 12 | "In this notebook, we will access data for the [ECOSTRESS Tiled Land Surface Temperature and Emissivity Instantaneous L2 Global 70 m V002](https://doi.org/10.5067/ECOSTRESS/ECO_L2T_LSTE.002) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band. We will access a single COG file, Land Surface Temperature (LST), directly loading it into memory, leveraging the cloud-optimized format, rather than downloading the file. To accomplish this we will create a `requests_https_session` using the `earthaccess` Python library, which will handle passing our Earthdata Login credentials to the NASA Earthdata system, and the `rasterio` and `rioxarray` Python libraries to load the data into memory so we can easily work with it. \n", 13 | "\n", 14 | "**Background**\n", 15 | "\n", 16 | "The [ECOSTRESS mission](https://ecostress.jpl.nasa.gov/) is answering these questions by accurately measuring the temperature of plants. Plants regulate their temperature by releasing water through tiny pores on their leaves called stomata. If they have sufficient water they can maintain their temperature, but if there is insufficient water, their temperatures rise and this temperature rise can be measured with ECOSTRESS. The images acquired by ECOSTRESS are the most detailed temperature images of the surface ever acquired from space and can be used to measure the temperature of an individual farm field. These temperature images, along with auxiliary inputs, are used to produce one of the primary science outputs of ECOSTRESS: evapotranspiration, an indicator of plant health vby measuring the evaporation and transpiration of water through a plant.\n", 17 | "\n", 18 | "**Learning Objectives**\n", 19 | "\n", 20 | "- How to authenticate to the NASA Earthdata system using the `earthaccess` Python library and a netrc file\n", 21 | "- How to access and load a Cloud Optimized GeoTIFF (COG) file using `rasterio` and `rioxarray`\n", 22 | "- How to visualize the data using `hvplot`\n", 23 | "\n", 24 | "**Requirements**\n", 25 | "\n", 26 | "- NASA [Earthdata Login](https://urs.earthdata.nasa.gov/) account. If you do not have an Earthdata Account, you can create one [here](https://urs.earthdata.nasa.gov/users/new). \n", 27 | "- A compatible Python Environment - See **setup_instructions.md** in the `/setup/` folder \n", 28 | "\n", 29 | "**Outline**\n", 30 | "1. Setup\n", 31 | "2. Load file directly to memory\n", 32 | "3. Visualize the data" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "id": "usual-disabled", 38 | "metadata": {}, 39 | "source": [ 40 | "## 1. Setup\n", 41 | "\n", 42 | "Import the required libraries." 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": null, 48 | "id": "boolean-horse", 49 | "metadata": {}, 50 | "outputs": [], 51 | "source": [ 52 | "import os\n", 53 | "import rasterio as rio\n", 54 | "import rioxarray as rxr\n", 55 | "import hvplot.xarray\n", 56 | "import earthaccess" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "id": "3d6cc59c", 62 | "metadata": {}, 63 | "source": [ 64 | "### Authentication\n", 65 | "\n", 66 | "Log into Earthdata using the `Auth` and `login` functions from the `earthaccess` library. The `persist=True` argument will create a local `.netrc` file if it doesn't exist, or add your login info to an existing `.netrc` file. If no Earthdata Login credentials are found in the `.netrc` you'll be prompted for them." 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "id": "8383f944", 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "auth = earthaccess.login(persist = True)" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "id": "6046e8f8", 82 | "metadata": {}, 83 | "source": [ 84 | "### Context Manager\n", 85 | "\n", 86 | "For this exercise, we are going to open up a context manager for the notebook using the `rasterio.env` module to store the required GDAL configurations we need to access the data from Earthdata Cloud. The context manager sends the authentication information when connecting to a file and can also customize how the file is handled locally. Geospatial data access Python packages like `rasterio` and `rioxarray` depend on GDAL, leveraging GDAL’s “Virtual File Systems” to read remote files. GDAL has a lot of environment variables that control its behavior. Changing these settings can mean the difference between being able to access a file or not. They can also have an impact on the performance. Please see the [GDAL config options documentation](https://gdal.org/en/stable/user/configoptions.html) for more details and all available options.\n", 87 | "\n", 88 | "While the context manager is open (rio_env.__enter__()) we will be able to run the open or get data commands that would typically be executed within a \"with\" statement. Entering the context manager for the entirety of the notebook allows us to more freely interact with the data. We’ll close the context manager (rio_env.__exit__()) at the end of the notebook." 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "id": "29d3a563", 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "# Set up and enter context manager\n", 99 | "rio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n", 100 | " GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n", 101 | " GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'),\n", 102 | " GDAL_HTTP_MAX_RETRY=10,\n", 103 | " GDAL_HTTP_RETRY_DELAY=0.5)\n", 104 | "rio_env.__enter__()" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "id": "83f08677", 110 | "metadata": {}, 111 | "source": [ 112 | "Above, `GDAL_HTTP_COOKIEFILE` and `GDAL_HTTP_COOKIEJAR` tell GDAL to use a cookie for authentication and where to find that cookie. `GDAL_DISABLE_READDIR_ON_OPEN` tells gdal not to look for any auxiliary or sidecar files in the directory, which can slow down access. `GDAL_HTTP_MAX_RETRY` and `GDAL_HTTP_RETRY_DELAY` tell GDAL to retry the connection a number of times and how long to wait before retrying. These are nice options to add in the case that a connection fails temporarily, and will allow the workflow to continue without re-running.\n", 113 | "\n", 114 | "In this example, we use cookies to pass authentication information via the context manager; however, this can also be accomplished by sending an Earthdata Login token, when working with versions of **gdal > 3.7.0.** See below for a commented out example." 115 | ] 116 | }, 117 | { 118 | "cell_type": "code", 119 | "execution_count": null, 120 | "id": "8d586415", 121 | "metadata": {}, 122 | "outputs": [], 123 | "source": [ 124 | "# rio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n", 125 | "# GDAL_HTTP_AUTH='BEARER',\n", 126 | "# GDAL_HTTP_BEARER=auth.token['access_token'],\n", 127 | "# GDAL_HTTP_MAX_RETRY=10,\n", 128 | "# GDAL_HTTP_RETRY_DELAY=0.5)\n", 129 | "# rio_env.__enter__()" 130 | ] 131 | }, 132 | { 133 | "cell_type": "markdown", 134 | "id": "expired-crowd", 135 | "metadata": {}, 136 | "source": [ 137 | "## 2. Load the File Directly into Memory\n", 138 | "In this example we're interested in the ECOSTRESS data collection from NASA's LP DAAC in Earthdata Cloud. Below we specify the URL to the data asset in Earthdata Cloud. This URL can be found via [Earthdata Search](https://search.earthdata.nasa.gov/search) or programmatically through `earthaccess`, the CMR API or CMR-STAC API. There are programmatic examples in the [Python tutorials for ECOSTRESS](../tutorials/), and an [earthdata search example](../../guides/earthdata_search_ecostress.md) available as well." 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": null, 144 | "id": "gorgeous-relief", 145 | "metadata": {}, 146 | "outputs": [], 147 | "source": [ 148 | "https_url = 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ECO_L2T_LSTE.002/ECOv002_L2T_LSTE_24479_001_11SKU_20221030T092522_0710_01/ECOv002_L2T_LSTE_24479_001_11SKU_20221030T092522_0710_01_LST.tif'" 149 | ] 150 | }, 151 | { 152 | "cell_type": "markdown", 153 | "id": "fabulous-cornell", 154 | "metadata": {}, 155 | "source": [ 156 | "Read in the ECOSTRESS LST URL into our workspace using `rioxarray`. This utilizes the context manager that we have entered. Optionally we can use the `mask_and_scale` argument to mask and apply the scale and offset values for the data. " 157 | ] 158 | }, 159 | { 160 | "cell_type": "code", 161 | "execution_count": null, 162 | "id": "blocked-municipality", 163 | "metadata": {}, 164 | "outputs": [], 165 | "source": [ 166 | "# Open data with rioxarray\n", 167 | "da = rxr.open_rasterio(https_url, mask_and_scale=True)\n", 168 | "da" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "id": "african-acting", 174 | "metadata": {}, 175 | "source": [ 176 | "The file is read into Python as an `xarray` `dataarray` with a **band**, **x**, and **y** dimension. In this example the **band** dimension is meaningless, so we'll use the `squeeze()` function to remove **band** as a dimension. " 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "id": "vocal-center", 183 | "metadata": {}, 184 | "outputs": [], 185 | "source": [ 186 | "da_lst = da.squeeze('band', drop=True)\n", 187 | "da_lst" 188 | ] 189 | }, 190 | { 191 | "cell_type": "markdown", 192 | "id": "fdafb279-acbe-473a-aa34-200da33901ce", 193 | "metadata": {}, 194 | "source": [ 195 | "## 3. Visualize the Data\n", 196 | "\n", 197 | "Plot the `dataarray`, representing the ECOSTRESS band, using `hvplot`. Since ECOSTRESS tiles are in UTM projections, to visualize this with a basemap tile, we'll need to reproject to EPSG:4326 for the visual. This can be accomplished using the `rio.reproject()` function." 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "id": "a623c4b1", 204 | "metadata": {}, 205 | "outputs": [], 206 | "source": [ 207 | "da_lst_reproj = da_lst.rio.reproject(\"EPSG:4326\")" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": null, 213 | "id": "outdoor-handling", 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "da_lst_reproj.hvplot.image(x = 'x',\n", 218 | " y = 'y',\n", 219 | " crs = 'EPSG:4326',\n", 220 | " cmap='jet',\n", 221 | " tiles='EsriImagery',\n", 222 | " title = f'{https_url.split(\"/\")[-1]}',\n", 223 | " frame_width=500)" 224 | ] 225 | }, 226 | { 227 | "cell_type": "markdown", 228 | "id": "86efc013-97ad-4af1-9e20-dea661567d04", 229 | "metadata": {}, 230 | "source": [ 231 | "Exit the context manager." 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "id": "fitting-liechtenstein", 238 | "metadata": {}, 239 | "outputs": [], 240 | "source": [ 241 | "rio_env.__exit__()" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "id": "25b31b8b", 247 | "metadata": {}, 248 | "source": [ 249 | "## Contact Info: \n", 250 | "\n", 251 | "Email: LPDAAC@usgs.gov \n", 252 | "Voice: +1-866-573-3222 \n", 253 | "Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹ \n", 254 | "Website: \n", 255 | "\n", 256 | "¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I. " 257 | ] 258 | } 259 | ], 260 | "metadata": { 261 | "kernelspec": { 262 | "display_name": "Python 3 (ipykernel)", 263 | "language": "python", 264 | "name": "python3" 265 | }, 266 | "language_info": { 267 | "codemirror_mode": { 268 | "name": "ipython", 269 | "version": 3 270 | }, 271 | "file_extension": ".py", 272 | "mimetype": "text/x-python", 273 | "name": "python", 274 | "nbconvert_exporter": "python", 275 | "pygments_lexer": "ipython3", 276 | "version": "3.11.13" 277 | } 278 | }, 279 | "nbformat": 4, 280 | "nbformat_minor": 5 281 | } 282 | -------------------------------------------------------------------------------- /python/how-tos/how_to_direct_access_s3_ecostress_cog.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "prompt-junction", 6 | "metadata": {}, 7 | "source": [ 8 | "# How to: Directly Access ECOSTRESS Data (S3)\n", 9 | "\n", 10 | "\n", 11 | "NASA Earthdata Cloud data is stored in an Amazon Web Services (AWS) Simple Storage Service (S3) bucket. Data located within these buckets can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region. In this notebook, we will access data for the [ECOSTRESS Tiled Land Surface Temperature and Emissivity Instantaneous L2 Global 70 m V002](https://doi.org/10.5067/ECOSTRESS/ECO_L2T_LSTE.002) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band. We will access a single COG file, Land Surface Temperature (LST), directly loading it into memory from inside the AWS cloud (us-west-2 region, specifically). To accomplish this we will retrieve temporary AWS credentials and load the data directly to memory as an `xarray` `dataarray`. This approach leverages native protocols for efficient access to the data directly in its S3 bucket.\n", 12 | "\n", 13 | "**Background**\n", 14 | "\n", 15 | "The [ECOSTRESS mission](https://ecostress.jpl.nasa.gov/) is answering these questions by accurately measuring the temperature of plants. Plants regulate their temperature by releasing water through tiny pores on their leaves called stomata. If they have sufficient water they can maintain their temperature, but if there is insufficient water, their temperatures rise and this temperature rise can be measured with ECOSTRESS. The images acquired by ECOSTRESS are the most detailed temperature images of the surface ever acquired from space and can be used to measure the temperature of an individual farmers field. These temperature images, along with auxiliary inputs, are used to produce one of the primary science outputs of ECOSTRESS: evapotranspiration, an indicator of plant health via the measure of evaporation and transpiration of water through a plant.\n", 16 | "\n", 17 | "**Learning Objectives**\n", 18 | "\n", 19 | "- How to authenticate to the NASA Earthdata system using the `earthaccess` Python library and a netrc file\n", 20 | "- How to access and load a Cloud Optimized GeoTIFF (COG) file using `rasterio` and `rioxarray`\n", 21 | "- How to visualize the data using `hvplot`\n", 22 | "\n", 23 | "**Requirements**\n", 24 | "\n", 25 | "- **A cloud computing environment located within the AWS us-west2 region.**\n", 26 | "- NASA [Earthdata Login](https://urs.earthdata.nasa.gov/) account. If you do not have an Earthdata Account, you can create one [here](https://urs.earthdata.nasa.gov/users/new). \n", 27 | "- A compatible Python Environment - See **setup_instructions.md** in the `/setup/` folder \n", 28 | "\n", 29 | "**Outline**\n", 30 | "1. Setup\n", 31 | "2. Load file directly to memory\n", 32 | "3. Visualize the data" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "id": "bizarre-antarctica", 38 | "metadata": {}, 39 | "source": [ 40 | "## 1. Setup\n", 41 | "Import the required libraries." 42 | ] 43 | }, 44 | { 45 | "cell_type": "code", 46 | "execution_count": null, 47 | "id": "descending-shooting", 48 | "metadata": {}, 49 | "outputs": [], 50 | "source": [ 51 | "import os\n", 52 | "import boto3\n", 53 | "import rasterio as rio\n", 54 | "from rasterio.session import AWSSession\n", 55 | "import rioxarray as rxr\n", 56 | "import hvplot.xarray\n", 57 | "import earthaccess" 58 | ] 59 | }, 60 | { 61 | "cell_type": "markdown", 62 | "id": "annoying-wrist", 63 | "metadata": {}, 64 | "source": [ 65 | "### Authentication and Temporary AWS Credentials\n", 66 | "\n", 67 | "Log into Earthdata using the `login` functions from the `earthaccess` library. The `persist=True` argument will create a local `.netrc` file if it doesn't exist, or add your login info to an existing `.netrc` file. If no Earthdata Login credentials are found in the `.netrc` you'll be prompted for them. After signing into Earthdata, these credentials can be used to get temporary AWS credentials so we can interact with S3 objects from applicable Earthdata Cloud buckets. Each NASA data center has different AWS credentials endpoints. The `earthaccess` library can be used to retrieve credentials by just providing a data center name (e.g. 'podaac','gesdisc','lpdaac','ornldaac','ghrcdaac'). In this case, ECOSTRESS data is archived by the LP DAAC (\"lpdaac\"), so we'll want those temporary credentials." 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "id": "a926e7c8", 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "# Log into Earthdata\n", 78 | "auth = earthaccess.login(persist = True)" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "id": "613b8ff0-8f1e-49d5-817e-20b762fd95ec", 85 | "metadata": {}, 86 | "outputs": [], 87 | "source": [ 88 | "# Retrieve Temporary Credentials for LP DAAC Data\n", 89 | "temp_creds_req = earthaccess.get_s3_credentials(daac='lpdaac')" 90 | ] 91 | }, 92 | { 93 | "cell_type": "markdown", 94 | "id": "e370da8f-884e-4a8d-8072-e6896745488f", 95 | "metadata": {}, 96 | "source": [ 97 | "Alternatively, we can manually define S3 credential endpoints and make a request using the `requests` python library to retrive them. This process will use the login info from the `.netrc` file. Uncomment and use below if you prefer." 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "id": "selected-execution", 104 | "metadata": {}, 105 | "outputs": [], 106 | "source": [ 107 | "# import requests\n", 108 | "# s3_cred_endpoint = {\n", 109 | "# 'podaac':'https://archive.podaac.earthdata.nasa.gov/s3credentials',\n", 110 | "# 'gesdisc': 'https://data.gesdisc.earthdata.nasa.gov/s3credentials',\n", 111 | "# 'lpdaac':'https://data.lpdaac.earthdatacloud.nasa.gov/s3credentials',\n", 112 | "# 'ornldaac': 'https://data.ornldaac.earthdata.nasa.gov/s3credentials',\n", 113 | "# 'ghrcdaac': 'https://data.ghrc.earthdata.nasa.gov/s3credentials'\n", 114 | "# }\n", 115 | "# def get_temp_creds(provider):\n", 116 | "# return requests.get(s3_cred_endpoint[provider]).json()\n", 117 | "# temp_creds_req = get_temp_creds('lpdaac')" 118 | ] 119 | }, 120 | { 121 | "cell_type": "markdown", 122 | "id": "36bd4773-c2d8-4999-9142-d7d2b3cd6491", 123 | "metadata": {}, 124 | "source": [ 125 | "Create a `boto3` Session object using your temporary credentials. This Session is used to pass credentials and configuration to AWS so we can interact with S3 objects from applicable buckets." 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "id": "defined-navigator", 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "session = boto3.Session(aws_access_key_id=temp_creds_req['accessKeyId'], \n", 136 | " aws_secret_access_key=temp_creds_req['secretAccessKey'],\n", 137 | " aws_session_token=temp_creds_req['sessionToken'],\n", 138 | " region_name='us-west-2')" 139 | ] 140 | }, 141 | { 142 | "cell_type": "markdown", 143 | "id": "indoor-integration", 144 | "metadata": {}, 145 | "source": [ 146 | "### Context Manager\n", 147 | "\n", 148 | "For this exercise, we are going to open up a context manager for the notebook using the `rasterio.env` module to store the required GDAL configurations we need to access the data from Earthdata Cloud. The context manager sends the authentication information when connecting to a file and can also customize how the file is handled locally. The GDAL environment variables must be configured to access COGs in Earthdata Cloud. Geospatial data access Python packages like rasterio and rioxarray depend on GDAL, leveraging GDAL’s “Virtual File Systems” to read remote files. GDAL has a lot of environment variables that control its behavior. Changing these settings can mean the difference between being able to access a file or not. They can also have an impact on the performance. Please see the [GDAL config options documentation](https://gdal.org/en/stable/user/configoptions.html) for more details and all available options.\n", 149 | "\n", 150 | "While the context manager is open (rio_env.__enter__()) we will be able to run commands that open or get data that would typically be executed within a \"with\" statement. Entering the context manager for the entirety of the notebook allows us to more freely interact with the data. We’ll close the context manager (rio_env.__exit__()) at the end of the notebook." 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "id": "sustainable-tournament", 157 | "metadata": {}, 158 | "outputs": [], 159 | "source": [ 160 | "rio_env = rio.Env(AWSSession(session),\n", 161 | " GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n", 162 | " GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n", 163 | " GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'),\n", 164 | " GDAL_HTTP_MAX_RETRY=10,\n", 165 | " GDAL_HTTP_RETRY_DELAY=0.5)\n", 166 | "rio_env.__enter__()" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "id": "5181f2ad-db0e-4499-b4b0-9778821b13ac", 172 | "metadata": {}, 173 | "source": [ 174 | "Above, `GDAL_HTTP_COOKIEFILE` and `GDAL_HTTP_COOKIEJAR` tell GDAL to use a cookie for authentication and where to find that cookie. `GDAL_DISABLE_READDIR_ON_OPEN` tells gdal not to look for any auxiliary or sidecar files in the directory, which can slow down access. `GDAL_HTTP_MAX_RETRY` and `GDAL_HTTP_RETRY_DELAY` tell GDAL to retry the connection a number of times and how long to wait before retrying. These are nice options to add in the case that a connection fails temporarily, and will allow the workflow to continue without re-running." 175 | ] 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "id": "0ae86f5f-354c-4ede-b3c6-622fb4f737e0", 180 | "metadata": {}, 181 | "source": [ 182 | "## 2. Load the File Directly into Memory (S3)\n", 183 | "\n", 184 | "In this example we're interested in the ECOSTRESS data collection from NASA's LP DAAC in Earthdata Cloud. Below we specify the URL to the data asset in Earthdata Cloud. This URL can be found via [Earthdata Search](https://search.earthdata.nasa.gov/search) or programmatically through `earthaccess`, the CMR API or CMR-STAC API. There are programmatic examples in the [Python tutorials for ECOSTRESS](../tutorials/), and an [earthdata search example](../../guides/earthdata_search_ecostress.md) available as well." 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": null, 190 | "id": "cultural-breach", 191 | "metadata": {}, 192 | "outputs": [], 193 | "source": [ 194 | "s3_url_lst = 's3://lp-prod-protected/ECO_L2T_LSTE.002/ECOv002_L2T_LSTE_24479_001_11SKU_20221030T092522_0710_01/ECOv002_L2T_LSTE_24479_001_11SKU_20221030T092522_0710_01_LST.tif'" 195 | ] 196 | }, 197 | { 198 | "cell_type": "markdown", 199 | "id": "4bacb14f-fce2-42c2-ac2a-c6c20c4c5152", 200 | "metadata": {}, 201 | "source": [ 202 | "Read in the ECOSTRESS LSt URL into our workspace using `rioxarray`. This utilizes the context manager that we have entered. Optionally we can use the `mask_and_scale` argument to mask and apply the scale and offset values for the data. " 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": null, 208 | "id": "tutorial-extent", 209 | "metadata": {}, 210 | "outputs": [], 211 | "source": [ 212 | "da = rxr.open_rasterio(s3_url_lst)\n", 213 | "da" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "id": "78752272-d939-4e20-8ef7-cc6950a4baf1", 219 | "metadata": {}, 220 | "source": [ 221 | "The file is read into Python as an `xarray` `dataarray` with a **band**, **x**, and **y** dimension. In this example the **band** dimension is meaningless, so we'll use the `squeeze()` function to remove **band** as a dimension. " 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": null, 227 | "id": "laughing-carolina", 228 | "metadata": {}, 229 | "outputs": [], 230 | "source": [ 231 | "da_lst = da.squeeze('band', drop=True)\n", 232 | "da_lst" 233 | ] 234 | }, 235 | { 236 | "cell_type": "markdown", 237 | "id": "2e7f9202-b96b-4782-9b0f-128aba681cf7", 238 | "metadata": {}, 239 | "source": [ 240 | "## 3. Visualize the Data\n", 241 | "\n", 242 | "Plot the `dataarray`, representing the ECOSTRESS band, using `hvplot`. Since ECOSTRESS tiles are in UTM projections, to visualize this with a basemap tile, we'll need to reproject to EPSG:4326 for the visual. This can be accomplished using the `rio.reproject()` function." 243 | ] 244 | }, 245 | { 246 | "cell_type": "code", 247 | "execution_count": null, 248 | "id": "f18d9323-4b5a-45fe-b502-77189cd55528", 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "da_lst_reproj = da_lst.rio.reproject(\"EPSG:4326\")" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": null, 258 | "id": "85711ef3-9755-4802-a791-6a89c7706137", 259 | "metadata": {}, 260 | "outputs": [], 261 | "source": [ 262 | "da_lst_reproj.hvplot.image(x = 'x',\n", 263 | " y = 'y',\n", 264 | " crs = 'EPSG:4326',\n", 265 | " cmap='jet',\n", 266 | " tiles='EsriImagery',\n", 267 | " title = f'{s3_url_lst.split(\"/\")[-1]}',\n", 268 | " frame_width=500)" 269 | ] 270 | }, 271 | { 272 | "cell_type": "markdown", 273 | "id": "8c875b38-64bc-4841-ab96-8f17478e4872", 274 | "metadata": {}, 275 | "source": [ 276 | "Exit the context manager." 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": null, 282 | "id": "gorgeous-christianity", 283 | "metadata": {}, 284 | "outputs": [], 285 | "source": [ 286 | "rio_env.__exit__()" 287 | ] 288 | }, 289 | { 290 | "cell_type": "markdown", 291 | "id": "3642731f-3839-4f96-8f52-aa7896e4b703", 292 | "metadata": {}, 293 | "source": [ 294 | "## Contact Info: \n", 295 | "\n", 296 | "Email: LPDAAC@usgs.gov \n", 297 | "Voice: +1-866-573-3222 \n", 298 | "Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹ \n", 299 | "Website: \n", 300 | "\n", 301 | "¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I. " 302 | ] 303 | } 304 | ], 305 | "metadata": { 306 | "kernelspec": { 307 | "display_name": "Python 3 (ipykernel)", 308 | "language": "python", 309 | "name": "python3" 310 | }, 311 | "language_info": { 312 | "codemirror_mode": { 313 | "name": "ipython", 314 | "version": 3 315 | }, 316 | "file_extension": ".py", 317 | "mimetype": "text/x-python", 318 | "name": "python", 319 | "nbconvert_exporter": "python", 320 | "pygments_lexer": "ipython3", 321 | "version": "3.11.13" 322 | } 323 | }, 324 | "nbformat": 4, 325 | "nbformat_minor": 5 326 | } 327 | -------------------------------------------------------------------------------- /python/scripts/ecostress_swath2grid/ECOSTRESS_swath2grid.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | --------------------------------------------------------------------------------------------------- 4 | ECOSTRESS Swath to Grid Conversion Script with Brightness Temperature Conversion Option 5 | Author: LP DAAC 6 | Last Updated: 11/21/2022 7 | See README for additional information: 8 | https://git.earthdata.nasa.gov/projects/LPDUR/repos/ecostress_swath2grid/browse 9 | --------------------------------------------------------------------------------------------------- 10 | """ 11 | # Load necessary packages into Python 12 | import h5py 13 | import os 14 | import pyproj 15 | import sys 16 | import argparse 17 | import math 18 | import numpy as np 19 | import warnings 20 | #from pyproj import CRS 21 | warnings.simplefilter(action='ignore', category=UserWarning) 22 | from pyresample import geometry as geom 23 | from pyresample import kd_tree as kdt 24 | from osgeo import gdal, gdal_array, gdalconst, osr 25 | warnings.simplefilter(action='ignore', category=FutureWarning) 26 | 27 | # --------------------------COMMAND LINE ARGUMENTS AND ERROR HANDLING---------------------------- # 28 | # Set up argument and error handling 29 | parser = argparse.ArgumentParser(description='Performs ECOSTRESS Swath to Grid Conversion') 30 | parser.add_argument('--proj', required=True, choices=['UTM', 'GEO'], help='Projection desired for output GeoTIFFs') 31 | parser.add_argument('--dir', required=True, help='Local directory containing ECOSTRESS Swath files to be processed') 32 | parser.add_argument('--sds', required=False, help='Specific science datasets (SDS) to extract from ECOSTRESS Granules \ 33 | (see README for a list of available SDS)') 34 | parser.add_argument('--utmzone', required=False, help='UTM zone (EPSG Code) desired for all outputs--only required if needed to \ 35 | override default UTM zone which is assigned based on the center location for each ECOSTRESS granule') 36 | parser.add_argument('--bt', required=False, help='Optional argument to convert radiance to brightness temperature \ 37 | for the L1B products', action='store_true') 38 | args = parser.parse_args() 39 | 40 | # --------------------------------SET ARGUMENTS TO VARIABLES------------------------------------- # 41 | # Format and set input/working directory from user-defined arg 42 | if args.dir[-1] != '/' and args.dir[-1] != '\\': 43 | inDir = args.dir.strip("'").strip('"') + os.sep 44 | else: 45 | inDir = args.dir 46 | 47 | scriptDir = os.getcwd() + os.sep 48 | # Find input directory 49 | try: 50 | os.chdir(inDir) 51 | except FileNotFoundError: 52 | print('error: input directory (--dir) provided does not exist or was not found') 53 | sys.exit(2) 54 | 55 | crsIN = args.proj # Options include 'UTM' or 'GEO' 56 | 57 | # -------------------------------------SET UP WORKSPACE------------------------------------------ # 58 | # Create and set output directory 59 | outDir = os.path.normpath((os.path.split(inDir)[0] + os.sep + 'output')) + os.sep 60 | if not os.path.exists(outDir): 61 | os.makedirs(outDir) 62 | 63 | # Create lists of ECOSTRESS HDF-EOS5 files (geo, data) in the directory 64 | geoList = [f for f in os.listdir() if f.startswith('ECO') and f.endswith('.h5') and 'GEO' in f] 65 | ecoList = [f for f in os.listdir() if f.startswith('ECO') and f.endswith('.h5')] 66 | 67 | # Check to verify if any ECOSTRESS files are in the directory provided 68 | if len(ecoList) == 0: 69 | print(f'No ECOSTRESS files found in {inDir}') 70 | sys.exit(2) 71 | # -------------------------------------DEFINE FUNCTIONS------------------------------------------ # 72 | # Write function to determine which UTM zone to use: 73 | def utmLookup(lat, lon): 74 | utm = str((math.floor((lon + 180) / 6) % 60) + 1) 75 | if len(utm) == 1: 76 | utm = '0' + utm 77 | if lat >= 0: 78 | epsg_code = '326' + utm 79 | else: 80 | epsg_code = '327' + utm 81 | return epsg_code 82 | 83 | 84 | # If brightness temperature is requested, define a function to convert radiance to BT 85 | if args.bt: 86 | def btConversion(s, ecoSD): 87 | try: 88 | # Look up table containing brightness temperature conversions coefficients 89 | lutFile = h5py.File('EcostressBrightnessTemperatureV01.h5', "r") 90 | except: 91 | try: 92 | lutFile = h5py.File(f'{scriptDir}EcostressBrightnessTemperatureV01.h5', "r") 93 | except: 94 | print('Brightness Temp LUT not found, download the table at https://git.earthdata.nasa.gov/projects/LPDUR/repos/ecostress_swath2grid/browse and make sure it is in the same directory as the ECOSTRESS files to be processed or the ECOSTRESS_swath2grid.py script.') 95 | sys.exit(2) 96 | # Open the lookup table for the desired radiance band 97 | lut = lutFile[f"/lut/radiance_{s[-1]}"][()].astype(float) 98 | lutFile.close() 99 | 100 | # Take the input radiance array, and map to bt conversion coefficient depending on radiance band and value 101 | ecoSDflat = ecoSD.flatten() 102 | ecoSDflat = np.where(np.logical_and(0 <= ecoSDflat, ecoSDflat <= 60), ecoSDflat, 0) # Filter fill values 103 | indexLUT = np.int32(ecoSDflat / 0.001) 104 | 105 | # Interpolate the LUT values for each radiance based on two nearest LUT values 106 | bt = np.empty(ecoSDflat.shape, dtype=ecoSDflat.dtype) 107 | radiance_x0 = indexLUT * 0.001 108 | radiance_x1 = radiance_x0 + 0.001 109 | factor0 = (radiance_x1 - ecoSDflat) / 0.001 110 | factor1 = (ecoSDflat - radiance_x0) / 0.001 111 | bt[:] = (factor0 * lut[indexLUT]) + (factor1 * lut[indexLUT + 1]) 112 | 113 | # Set out of range radiance and fill values back to fill value 114 | bt = np.where((ecoSDflat != 0), bt, fv) 115 | bt = bt.reshape(ecoSD.shape) 116 | return bt 117 | 118 | if args.utmzone is not None and args.utmzone[0:3] != '326' and args.utmzone[0:3] != '327': 119 | parser.error('--utmzone requires the EPSG code (http://spatialreference.org/ref/epsg/) for a UTM zone, and only WGS84 datum is supported') 120 | # ------------------------------------IMPORT ECOSTRESS FILE-------------------------------------- # 121 | # Batch process all files in the input directory 122 | for i, e in enumerate(ecoList): 123 | i += 1 124 | print('Processing: {} ({} of {})'.format(e, str(i), str(len(ecoList)))) 125 | f = h5py.File(e, "r") # Read in ECOSTRESS HDF5-EOS data file 126 | ecoName = e.split('.h5')[0] # Keep original filename 127 | eco_objs = [] 128 | f.visit(eco_objs.append) # Retrieve list of datasets 129 | 130 | # Search for relevant SDS inside data file 131 | ecoSDS = [str(o) for o in eco_objs if isinstance(f[o], h5py.Dataset)] 132 | 133 | # Added functionality for dataset subsetting (--sds argument) 134 | if args.sds is not None: 135 | sds = args.sds.split(',') 136 | ecoSDS = [e for e in ecoSDS if e.endswith(tuple(sds))] 137 | if ecoSDS == []: 138 | print('No matching SDS layers found for {}'.format(e)) 139 | continue 140 | # ---------------------------------CONVERT SWATH DATA TO GRID------------------------------------ # 141 | # ALEXI products already gridded, bypass below 142 | if 'ALEXI_USDA' in e: 143 | cols, rows, dims = 3000, 3000, (3000, 3000) 144 | ecoSDS = [s for s in ecoSDS if f[s].shape == dims] # Omit NA layers/objs 145 | if ecoSDS == []: 146 | print('No matching SDS layers found for {}'.format(e)) 147 | continue 148 | else: 149 | # ---------------------------------IMPORT GEOLOCATION FILE--------------------------------------- # 150 | geo = [g for g in geoList if e[-37:-10] in g] # Match GEO filename--updated to exclude build ID 151 | if len(geo) != 0 or 'L1B_MAP' in e: # Proceed if GEO/MAP file 152 | if 'L1B_MAP' in e: 153 | g = f # Map file contains lat/lon 154 | else: 155 | g = h5py.File(geo[0], "r") # Read in GEO file 156 | geo_objs = [] 157 | g.visit(geo_objs.append) 158 | 159 | # Search for relevant SDS inside data file 160 | latSD = [str(o) for o in geo_objs if isinstance(g[o], h5py.Dataset) and '/latitude' in o] 161 | lonSD = [str(o) for o in geo_objs if isinstance(g[o], h5py.Dataset) and '/longitude' in o] 162 | lat = g[latSD[0]][()].astype(float) # Open Lat array 163 | lon = g[lonSD[0]][()].astype(float) # Open Lon array 164 | dims = lat.shape 165 | ecoSDS = [s for s in ecoSDS if f[s].shape == dims] # Omit NA layers/objs 166 | if ecoSDS == []: 167 | print('No matching SDS layers found for {}'.format(e)) 168 | continue 169 | # --------------------------------SWATH TO GEOREFERENCED ARRAYS---------------------------------- # 170 | swathDef = geom.SwathDefinition(lons=lon, lats=lat) 171 | midLat, midLon = np.mean(lat), np.mean(lon) 172 | 173 | if crsIN == 'UTM': 174 | if args.utmzone is None: 175 | epsg = utmLookup(midLat, midLon) # Determine UTM zone that center of scene is in 176 | else: 177 | epsg = args.utmzone 178 | epsgConvert = pyproj.Proj("+init=EPSG:{}".format(epsg)) 179 | proj, pName = 'utm', 'Universal Transverse Mercator' 180 | projDict = {'proj': proj, 'zone': epsg[-2:], 'ellps': 'WGS84', 'datum': 'WGS84', 'units': 'm'} 181 | if epsg[2] == '7': 182 | projDict['south'] = 'True' # Add for s. hemisphere UTM zones 183 | llLon, llLat = epsgConvert(np.min(lon), np.min(lat), inverse=False) 184 | urLon, urLat = epsgConvert(np.max(lon), np.max(lat), inverse=False) 185 | areaExtent = (llLon, llLat, urLon, urLat) 186 | ps = 70 # 70 is pixel size (meters) 187 | 188 | if crsIN == 'GEO': 189 | # Use info from aeqd bbox to calculate output cols/rows/pixel size 190 | epsgConvert = pyproj.Proj("+proj=aeqd +lat_0={} +lon_0={}".format(midLat, midLon)) 191 | llLon, llLat = epsgConvert(np.min(lon), np.min(lat), inverse=False) 192 | urLon, urLat = epsgConvert(np.max(lon), np.max(lat), inverse=False) 193 | areaExtent = (llLon, llLat, urLon, urLat) 194 | cols = int(round((areaExtent[2] - areaExtent[0])/70)) # 70 m pixel size 195 | rows = int(round((areaExtent[3] - areaExtent[1])/70)) 196 | '''Use no. rows and columns generated above from the aeqd projection 197 | to set a representative number of rows and columns, which will then be translated 198 | to degrees below, then take the smaller of the two pixel dims to determine output size''' 199 | epsg, proj, pName = '4326', 'longlat', 'Geographic' 200 | llLon, llLat, urLon, urLat = np.min(lon), np.min(lat), np.max(lon), np.max(lat) 201 | areaExtent = (llLon, llLat, urLon, urLat) 202 | projDict = pyproj.CRS("epsg:4326") 203 | areaDef = geom.AreaDefinition(epsg, pName, proj, projDict, cols, rows, areaExtent) 204 | ps = np.min([areaDef.pixel_size_x, areaDef.pixel_size_y]) # Square pixels 205 | 206 | cols = int(round((areaExtent[2] - areaExtent[0])/ps)) # Calculate the output cols 207 | rows = int(round((areaExtent[3] - areaExtent[1])/ps)) # Calculate the output rows 208 | areaDef = geom.AreaDefinition(epsg, pName, proj, projDict, cols, rows, areaExtent) 209 | index, outdex, indexArr, distArr = kdt.get_neighbour_info(swathDef, areaDef, 210, neighbours=1) 210 | else: 211 | print('ECO1BGEO File not found for {}'.format(e)) 212 | continue 213 | 214 | # ------------------LOOP THROUGH SDS CONVERT SWATH2GRID AND APPLY GEOREFERENCING----------------- # 215 | for s in ecoSDS: 216 | ecoSD = f[s][()] # Create array and read dimensions 217 | 218 | # Scale factor and add offset attribute names updated in build 6, accounted for below: 219 | scaleName = [a for a in f[s].attrs if 'scale' in a.lower()] # '_Scale' or 'scale_factor' 220 | addoffName = [a for a in f[s].attrs if 'offset' in a.lower()] 221 | 222 | # Read SDS Attributes if available 223 | try: 224 | fv = int(f[s].attrs['_FillValue']) 225 | except KeyError: 226 | fv = None 227 | except ValueError: 228 | if f[s].attrs['_FillValue'] == b'n/a': 229 | fv = None 230 | elif type(f[s].attrs['_FillValue'][0]) == np.float32: 231 | fv = np.nan 232 | else: 233 | fv = f[s].attrs['_FillValue'][0] 234 | try: 235 | sf = f[s].attrs[scaleName[0]][0] 236 | except: 237 | sf = 1 238 | try: 239 | add_off = f[s].attrs[addoffName[0]][0] 240 | except: 241 | add_off = 0 242 | 243 | # Convert to brightness temperature if bt argument is set, product is L1B, and SDS is Radiance 244 | if args.bt and 'RAD' in e: 245 | if 'radiance' in s: 246 | ecoSD = btConversion(s, ecoSD) 247 | 248 | if 'ALEXI_USDA' in e: # USDA Contains proj info in metadata 249 | if 'ET' in e: 250 | metaName = 'L3_ET_ALEXI Metadata' 251 | else: 252 | metaName = 'L4_ESI_ALEXI Metadata' 253 | gt = f[f"{metaName}/Geotransform"][()] 254 | proj = f['{}/OGC_Well_Known_Text'.format(metaName)][()].decode('UTF-8') 255 | sdGEO = ecoSD 256 | else: 257 | try: 258 | # Perform kdtree resampling (swath 2 grid conversion) 259 | sdGEO = kdt.get_sample_from_neighbour_info('nn', areaDef.shape, ecoSD, index, outdex, indexArr, fill_value=fv) 260 | ps = np.min([areaDef.pixel_size_x, areaDef.pixel_size_y]) 261 | gt = [areaDef.area_extent[0], ps, 0, areaDef.area_extent[3], 0, -ps] 262 | except ValueError: 263 | continue 264 | 265 | # Apply Scale Factor and Add Offset 266 | sdGEO = sdGEO * sf + add_off 267 | 268 | # Set fill value 269 | if fv is not None: 270 | sdGEO[sdGEO == fv * sf + add_off] = fv 271 | 272 | # -------------------------------------EXPORT GEOTIFFS------------------------------------------- # 273 | # For USDA, export to UTM, then convert to GEO 274 | if 'ALEXI_USDA' in e and crsIN == 'GEO': 275 | tempName = '{}{}_{}_{}.tif'.format(outDir, ecoName, s.rsplit('/')[-1], 'TEMP') 276 | outName = tempName 277 | elif args.bt and 'RAD' in e and 'radiance' in s: 278 | outName = '{}{}_{}_{}.tif'.format(outDir, ecoName, s.rsplit('/')[-1], crsIN).replace('radiance', 'brightnesstemperature') 279 | else: 280 | outName = '{}{}_{}_{}.tif'.format(outDir, ecoName, s.rsplit('/')[-1], crsIN) 281 | 282 | # Get driver, specify dimensions, define and set output geotransform 283 | height, width = sdGEO.shape # Define geotiff dimensions 284 | driv = gdal.GetDriverByName('GTiff') 285 | dataType = gdal_array.NumericTypeCodeToGDALTypeCode(sdGEO.dtype) 286 | d = driv.Create(outName, width, height, 1, dataType) 287 | d.SetGeoTransform(gt) 288 | 289 | # Create and set output projection, write output array data 290 | if 'ALEXI_USDA' in e: 291 | d.SetProjection(proj) 292 | else: 293 | # Define target SRS 294 | srs = osr.SpatialReference() 295 | srs.ImportFromEPSG(int(epsg)) 296 | d.SetProjection(srs.ExportToWkt()) 297 | band = d.GetRasterBand(1) 298 | band.WriteArray(sdGEO) 299 | 300 | # Define fill value if it exists, if not, set to mask fill value 301 | if fv is not None and fv != 'NaN': 302 | band.SetNoDataValue(fv) 303 | else: 304 | try: 305 | band.SetNoDataValue(int(sdGEO.fill_value)) 306 | except AttributeError: 307 | pass 308 | except TypeError: 309 | pass 310 | 311 | band.FlushCache() 312 | d, band = None, None 313 | 314 | if 'ALEXI_USDA' in e and crsIN == 'GEO': 315 | # Define target SRS 316 | srs = osr.SpatialReference() 317 | srs.ImportFromEPSG(int('4326')) 318 | srs = srs.ExportToWkt() 319 | 320 | # Open temp file, get default vals for target dims & geotransform 321 | dd = gdal.Open(tempName, gdalconst.GA_ReadOnly) 322 | vrt = gdal.AutoCreateWarpedVRT(dd, None, srs, gdal.GRA_NearestNeighbour, 0.125) 323 | 324 | # Create the final warped raster 325 | outName = '{}{}_{}_{}.tif'.format(outDir, ecoName, s.rsplit('/')[-1], crsIN) 326 | d = driv.CreateCopy(outName, vrt) 327 | dd, d, vrt = None, None, None 328 | os.remove(tempName) 329 | -------------------------------------------------------------------------------- /python/scripts/ecostress_swath2grid/swath2grid.md: -------------------------------------------------------------------------------- 1 | # ECOSTRESS Swath to Grid Conversion Script 2 | 3 | ## Objective 4 | 5 | The ECOSTRESS_swath2grid.py script converts ECOSTRESS swath data products, stored in Hierarchical Data Format version 5 (HDF5, .h5) into projected GeoTIFFs. When executing this script, a user will submit a desired output projection and input directory containing ECOSTRESS swath data products as command line arguments. The script begins by opening any of the ECOSTRESS products listed below that are contained in the input directory. Next, it uses the latitude and longitude arrays from the ECO1BGEO product (except for L3/L4 ALEXI_USDA and ECO1BMAPRAD products) to resample the swath dataset to a grid using nearest neighbor resampling (`Pyresample/kdtree`). Note that you will need to download the ECO1BGEO files that correspond to your higher level product files. From there, the script defines the coordinate reference system (CRS) input by the user (options include UTM Zones and Geographic (EPSG:4326)). There is an optional argument to override the default UTM zone selected by the script (see below) if needed. Ultimately, the script exports the gridded array as a GeoTIFF (`GDAL`). By default, the script will loop through and perform the aforementioned steps for each science dataset (SDS) in the HDF5 file. There is an optional argument that allows you to select a subset of SDS layers within a given product (see details below). There is another optional argument that allows users to convert L1B radiance to brightness temperature. The resulting GeoTIFF files can be imported with spatial reference into GIS and Remote Sensing software programs. The script also will batch process all ECOSTRESS swath files contained in the input directory provided. For ECOSTRESS products that include a scale factor in the metadata, the output will be scaled, and for products that include a fill value in the file metadata, this will be carried over into the GeoTIFF outputs. For layers that do not contain a fill value in the file metadata, the fill value will be defined as the highest possible value for the given datatype of an SDS. 6 | 7 | ## Available Products 8 | 9 | ### Collection 1 10 | 11 | 1. ECO1BGEO 12 | 2. ECO1BMAPRAD (lat/lon arrays contained within, ECO1BGEO not needed; radiance can be converted to brightness temperature) 13 | 3. ECO1BRAD (radiance can be converted to brightness temperature) 14 | 4. ECO2CLD 15 | 5. ECO2LSTE 16 | 6. ECO3ETALEXIU (30 m, in UTM Projection, ECO1BGEO not needed) 17 | 7. ECO3ETPTJPL 18 | 8. ECO3ANCQA 19 | 9. ECO4ESIPTJPL 20 | 10. ECO4ESIALEXIU (30 m, in UTM Projection, ECO1BGEO not needed) 21 | 11. ECO4WUE 22 | 23 | ### Collection 2 24 | 25 | 1. ECO_L1B_GEO 26 | 2. ECO_L1B_ATT 27 | 3. ECO_L1B_RAD 28 | 4. ECO_L2_LSTE 29 | 5. ECO_L2_CLOUD 30 | 31 | > Note that you will need to separately download the ECO1BGEO files that correspond to the files you have downloaded for products 2-4, 6-8, and 10 above. 32 | 33 | ## Prerequisites 34 | 35 | > Disclaimer: Script has been tested on Windows and MacOS using the specifications identified below. 36 | 37 | + **Python version 3.8** 38 | + `GDAL` 39 | + `h5py` 40 | + `pyproj` 41 | + `pyresample` 42 | + `numpy` 43 | + `pandas` 44 | + `scipy` 45 | 46 | + A [NASA Earthdata Login](https://urs.earthdata.nasa.gov/) account is required to download the data used in this tutorial. You can create an account at the link provided. 47 | 48 | ## Procedures 49 | 50 | ### Getting Started 51 | 52 | 1. Download ECOSTRESS higher level products and corresponding ECO1BGEO files (ordered separately) from the [LP DAAC Data Pool](https://e4ftl01.cr.usgs.gov/ECOSTRESS/) or [Earthdata Search Client](https://search.earthdata.nasa.gov/search?fi=ECOSTRESS) to a local directory (see above for applicable products) 53 | 2. Copy/clone/download [ECOSTRESS_swath2grid.py](https://git.earthdata.nasa.gov/projects/LPDUR/repos/ecostress_swath2grid/browse/ECOSTRESS_swath2grid.py) from LP DAAC Data User Resources Repository 54 | 55 | ### Python Environment Setup 56 | 57 | 1. It is recommended to use [Conda](https://conda.io/docs/), an environment manager to set up a compatible Python environment. Download Conda for your OS here: . Once you have Conda installed, Follow the instructions below to successfully setup a Python environment on MacOS or Windows. 58 | 2. Using your preferred command line interface (command prompt, terminal, cmder, etc.) type the following to successfully create a compatible python environment: 59 | 60 | ```text 61 | > conda create -n ecostress -c conda-forge --yes python=3.8 h5py pyproj pyresample gdal pandas scipy 62 | > conda activate ecostress 63 | ``` 64 | 65 | > NOTE: If you are having trouble activating your environment, or loading specific packages once you have activated your environment? Try entering `conda update conda` or `conda update --all` 66 | 67 | [Additional information](https://conda.io/docs/user-guide/tasks/manage-environments.html) on setting up and managing Conda environments. 68 | 69 | If you prefer to not install Conda, the same setup and dependencies can be achieved by using another package manager such as pip. 70 | 71 | Still having trouble getting a compatible Python environment set up? Contact [LP DAAC User Services](https://lpdaac.usgs.gov/lpdaac-contact-us/). 72 | 73 | ## Script Execution 74 | 75 | Once you have set up your MacOS/Windows environment and it has been activated, run the script with the following in your Command Prompt/terminal window: 76 | 77 | ```text 78 | > python ECOSTRESS_swath2grid.py --proj --dir 79 | ``` 80 | 81 | Where: 82 | 83 | + GEO = Geographic lat/lon, EPSG code 4326 84 | + UTM = Universal Transverse Mercator Zones (north/south) with WGS84 datum 85 | 86 | > NOTE: `--proj` argument is case sensitive 87 | 88 | **Example** 89 | 90 | ```text 91 | > python ECOSTRESS_swath2grid.py --proj GEO --dir C:\Users\ECOSTRESS\ 92 | ``` 93 | 94 | If UTM is selected, the script will calculate the UTM zone by using the location of the center of each ECOSTRESS granule. If you prefer to set the UTM zone manually, you can do so by adding the optional argument `--utmzone `. This optional argument will override the default functionality for users who desire all ECOSTRESS granules to be in a common UTM projection, regardless of the center location of the granule. 95 | 96 | **Example** 97 | 98 | ```text 99 | > python ECOSTRESS_swath2grid.py --proj UTM --dir --utmzone 100 | ``` 101 | 102 | > NOTE: You can look up EPSG codes for UTM zones at: , note that only WGS84 datum is supported, and thus EPSG codes for UTM north zones will begin with `326` and utm south zones with `327` 103 | 104 | The default functionality is to export each science dataset (SDS) layer contained in an ECOSTRESS product as a GeoTIFF. If you prefer to only export one or more layers, you can do so by adding the optional argument `--sds ` (comma separated with no spaces, see below for specific SDS layer names by product--note that the input is case sensitive) 105 | 106 | **Example** 107 | 108 | ```text 109 | > python ECOSTRESS_swath2grid.py --proj GEO --dir C:\Users\ECOSTRESS\ --sds LST,QC,Emis1 110 | ``` 111 | 112 | > NOTE: SDS layer names for each product are below in section [Subsetting Layers](#subsetting-layers). 113 | 114 | The default functionality for L1B radiance is to export each layer contained as radiance (W/m^2/sr/um). If you prefer to export the L1B datasets as brightness temperature (K), you can do so by adding the optional argument `--bt` (see section [Radiance to Brightness Temperature Conversion](#radiance-to-brightness-temperature-conversion) for additional information) 115 | 116 | **Example** 117 | 118 | ```text 119 | > python ECOSTRESS_swath2grid.py --proj GEO --dir C:\Users\ECOSTRESS\ --bt` 120 | ``` 121 | 122 | ## Radiance to Brightness Temperature Conversion 123 | 124 | To use the `--bt` optional command line argument to convert L1B radiance science datasets (bands 1-5) from radiance (W/m^2/sr/um) to brightness temperature (K), you will need to add the `--bt` argument to your script execution command. 125 | Example: 126 | 127 | ```text 128 | > python ECOSTRESS_swath2grid.py --proj GEO --dir C:\Users\ECOSTRESS\ --bt 129 | ``` 130 | 131 | > **Note:** You will first need to download the `EcostressBrightnessTemperatureV01.h5` Lookup Table (LUT) and save it to the directory containing your `ECOSTRESS_swath2grid.py` script, or the directory containing the ECOSTRESS L1B files to be processed. 132 | 133 | ## Subsetting Layers 134 | 135 | To use the `--sds` optional command line argument in order to select a subset of science datasets from an ECOSTRESS granule, you will need to submit 1 or more SDS layers names into the `--sds` argument exactly as they appear in the list below. **The SDS layers must be comma separated, with no spaces between SDS!** 136 | 137 | + Example for a single layer: `--sds LST` 138 | + Example for multiple layers: `--sds ETcanopy,ETdaily,ETinst` 139 | 140 | ## Available ECOSTRESS Layers 141 | 142 | ### Collection 1 143 | 144 | 1. **ECO1BGEO** 145 | + height 146 | + land_fraction 147 | + latitude 148 | + longitude 149 | + solar_azimuth 150 | + solar_zenith 151 | + view_azimuth 152 | + view_zenith 153 | 154 | 2. **ECO1BMAPRAD** 155 | + data_quality_1 156 | + data_quality_2 157 | + data_quality_3 158 | + data_quality_4 159 | + data_quality_5 160 | + height 161 | + latitude 162 | + longitude 163 | + radiance_1 (can be exported to brightnesstemperature_1) 164 | + radiance_2 (can be exported to brightnesstemperature_2) 165 | + radiance_3 (can be exported to brightnesstemperature_3) 166 | + radiance_4 (can be exported to brightnesstemperature_4) 167 | + radiance_5 (can be exported to brightnesstemperature_5) 168 | + solar_azimuth 169 | + solar_zenith 170 | + swir_dn 171 | + view_azimuth 172 | + view_zenith 173 | 174 | 3. **ECO1BRAD** 175 | + data_quality_1 176 | + data_quality_2 177 | + data_quality_3 178 | + data_quality_4 179 | + data_quality_5 180 | + radiance_1 (can be exported to brightnesstemperature_1) 181 | + radiance_2 (can be exported to brightnesstemperature_2) 182 | + radiance_3 (can be exported to brightnesstemperature_3) 183 | + radiance_4 (can be exported to brightnesstemperature_4) 184 | + radiance_5 (can be exported to brightnesstemperature_5) 185 | + swir_dn 186 | 187 | 4. **ECO2CLD** 188 | + CloudMask 189 | 190 | 5. **ECO2LSTE** 191 | + Emis1 192 | + Emis1_err 193 | + Emis2 194 | + Emis2_err 195 | + Emis3 196 | + Emis3_err 197 | + Emis4 198 | + Emis4_err 199 | + Emis5 200 | + Emis5_err 201 | + EmisWB 202 | + LST 203 | + LST_err 204 | + PWV 205 | + QC 206 | 207 | 6. **ECO3ETALEXIU** 208 | + ETdaily 209 | + ETdailyUncertainty 210 | + QualityFlag 211 | 212 | 7. **ECO3ETPTJPL** 213 | + ETcanopy 214 | + ETdaily 215 | + ETinst 216 | + ETinstUncertainty 217 | + ETinterception 218 | + ETsoil 219 | 220 | 8. **ECO3ANCQA** 221 | + ECOSTRESS_L2_QC 222 | + Landsat8_QC 223 | + MCD12Q1_QC 224 | + MCD43A3_QC 225 | + MOD04_QC 226 | + MOD06_1km_QC 227 | + MOD06_5km_QC 228 | + MOD07_QC 229 | + MOD13Q1_QC 230 | + MOD17A2H_QC 231 | + MOD44W_QC 232 | 233 | 9. **ECO4ESIALEXIU** 234 | + ESIdaily 235 | + ESIdailyUncertainty 236 | + QualityFlag 237 | 238 | 10. **ECO4ESIPTJPL** 239 | + ESIavg 240 | + PET 241 | 242 | 11. **ECO4WUE** 243 | + WUEavg 244 | 245 | ### Collection 2 246 | 247 | 1. **ECO_L1B_GEO** 248 | + height 249 | + land_fraction 250 | + latitude 251 | + line_start_time_j2000 252 | + longitude 253 | + solar_azimuth 254 | + solar_zenith 255 | + view_azimuth 256 | + view_zenith 257 | 258 | 2. **ECO_L1B_ATT** 259 | + Attitude quaternion 260 | + Attitude time_j2000 261 | + Ephemeris eci_position 262 | + Ephemeris eci_velocity 263 | + Ephemeris time_j2000 264 | + Uncorrected Attitude quaternion 265 | + Uncorrected Attitude time_j2000 266 | + Uncorrected Ephemeris eci_position 267 | + Uncorrected Ephemeris eci_velocity 268 | + Uncorrected Ephemeris time_j2000 269 | 270 | 3. **ECO_L1B_RAD** 271 | + radiance_1 272 | + radiance_2 273 | + radiance_3 274 | + radiance_4 275 | + radiance_5 276 | + data_quality_1 277 | + data_quality_2 278 | + data_quality_3 279 | + data_quality_4 280 | + data_quality_5 281 | + swir_dn 282 | + EncoderValue 283 | + line_start_time_j2000 284 | 285 | 4. **ECO_L2_LSTE** 286 | + LST 287 | + QC 288 | + Emis1 289 | + Emis2 290 | + Emis3 291 | + Emis4 292 | + Emis5 293 | + LST_err 294 | + Emis1_err 295 | + Emis2_err 296 | + Emis3_err 297 | + Emis4_err 298 | + Emis5_err 299 | + EmisWB 300 | + PWV 301 | + cloud_mask 302 | + water_mask 303 | 304 | 5. **ECO_L2_CLOUD** 305 | + Cloud_confidence 306 | + Cloud_final 307 | 308 | 309 | ## Contact Information 310 | ### Author: LP DAAC 1 311 | 312 | **Contact:** LPDAAC@usgs.gov 313 | **Voice:** +1-866-573-3222 314 | **Organization:** Land Processes Distributed Active Archive Center (LP DAAC) 315 | **Website:** 316 | **Date last modified:** 11-21-2022 317 | 318 | 1LP DAAC Work performed under NASA contract NNG14HH33I. 319 | -------------------------------------------------------------------------------- /python/scripts/extract_geolocation_flag/ECOSTRESS_geolocation.py: -------------------------------------------------------------------------------- 1 | """ 2 | =============================================================================== 3 | This script contains a workflow to extract the Geolocation Accuracy QA flag 4 | from ECOSTRESS L1B_GEO file level metadata. 5 | 6 | ------------------------------------------------------------------------------- 7 | Authors: Mahsa Jami, Erik Bolch 8 | Contact: lpdaac@usgs.gov 9 | Last Updated: 2025-02-03 10 | =============================================================================== 11 | """ 12 | # Import necessary packages 13 | import earthaccess 14 | import pandas as pd 15 | import xarray as xr 16 | import argparse 17 | import os 18 | from colorama import Fore, Back, Style 19 | import multiprocessing 20 | import warnings 21 | from tqdm import tqdm # Import tqdm for the progress bar 22 | 23 | 24 | # Suppress the specific UserWarning 25 | warnings.filterwarnings("ignore", message="Creating scratch directories is taking a surprisingly long time.") 26 | 27 | # ----------------------------------USER-DEFINED VARIABLES--------------------------------------- # 28 | def get_link(id): 29 | """This function extract the associated ECO_L1B_GEO URL using date and time information.""" 30 | attempts = 0 31 | url = '' 32 | while attempts < 3: 33 | try: 34 | response = earthaccess.search_data(short_name="ECO_L1B_GEO",version='002',granule_name = f'*{id}*') 35 | # print(response) 36 | # get the L1B_GEO access URL 37 | url = [granule.data_links(access="external") for granule in response] 38 | break 39 | except: 40 | attempts += 1 41 | return(url) 42 | 43 | def file_geolocation(url): 44 | """This function extract the GeolocationAccuracyQA flag from file metadata using xarray.""" 45 | fs = earthaccess.get_fsspec_https_session() 46 | fp = fs.open(url) 47 | l1geo = xr.open_dataset(fp, engine='h5netcdf', group='L1GEOMetadata') 48 | try: 49 | label = l1geo['GeolocationAccuracyQA'].data.item() 50 | except: 51 | label = 'missing' 52 | return(label) 53 | 54 | def dmrpp_geolocation(url): 55 | """This function extract the GeolocationAccuracyQA flag from .dmrpp metadata file.""" 56 | try: 57 | response = requests.get(url) 58 | # print(response.raise_for_status()) 59 | # Decode using ISO-8859-1 60 | xml_content = response.content.decode("ISO-8859-1") 61 | # Parse XML 62 | root = ET.fromstring(xml_content) 63 | 64 | for group in root.iter(): 65 | if "Group" in group.tag: 66 | group_name = group.attrib.get("name", "Unnamed Group") 67 | if group_name == "L1GEOMetadata": 68 | for elem in group.iter(): 69 | if "String" in elem.tag: 70 | string = elem.attrib.get("name", "Unnamed Attribute") 71 | if "GeolocationAccuracyQA" in string and "GeolocationAccuracyQAExplanation" not in string: 72 | for e in elem.iter(): 73 | encoded_value = e.text.strip() # Remove extra spaces/newlines 74 | label = base64.b64decode(encoded_value).decode("utf-8", errors="ignore") 75 | except: 76 | label = 'failed' 77 | return(label) 78 | 79 | 80 | if __name__ == "__main__": 81 | multiprocessing.freeze_support() 82 | 83 | # Set up argument 84 | parser = argparse.ArgumentParser(description='Extract the gelocation accuracy QA flag') 85 | parser.add_argument('-dir', '--directory', required=True, help='Specify directory to save files to') 86 | parser.add_argument('-f', '--files', required=True, help='A single granule URL, or the location of csv or textfile containing granule URLs') 87 | 88 | args = parser.parse_args() 89 | 90 | saveDir = args.directory # Set local directory to save file to 91 | files = args.files # Define granules that their geolocation flags are needed 92 | 93 | # ---------------------------------SET UP WORKSPACE---------------------------------------------- # 94 | 95 | # Create a list of granule IDs based on input type of files above 96 | if files.endswith('.txt') or files.endswith('.csv'): 97 | with open(files, 'r') as f: 98 | fileList = f.read().splitlines() # If input is text/csv file with file URLs 99 | 100 | out_name = f"{files.rsplit('.',1)[0]}_geolocation_flag.csv" 101 | elif isinstance(files, str): 102 | fileList = [files] # If input is a single file 103 | out_name = f"{files.rsplit('/',1)[1].rsplit('.')[0]}_geolocation_flag.csv" 104 | 105 | if not fileList: 106 | print(Fore.RED + "Error: No granules provided in the input file.") 107 | exit(1) 108 | 109 | # Check if the directory exists 110 | if not os.path.isdir(saveDir): 111 | os.makedirs(saveDir) 112 | 113 | out_dir = os.path.join(saveDir, out_name) 114 | 115 | # --------------------------------AUTHENTICATION CONFIGURATION----------------------------------- # 116 | # AUthenticate using earthaccess.login function. 117 | 118 | auth = earthaccess.login(persist = True) 119 | print(Fore.GREEN + 'Please note: If you just entered your Earthdata Login credentials, your username and password are now stored in a .netrc file located in your home directory on this system.') 120 | print(Style.RESET_ALL) 121 | 122 | # ------------------------------------------------------------------------------- # 123 | def get_geolocation_label(granID_db, f): 124 | # print(f'Working on {f}') 125 | 126 | id = [f for f in f.rsplit('/', 1)[1].rsplit('_') if len(f)==15 and '.' not in f][0] 127 | # print(id) 128 | if id in list(granID_db['id']): 129 | pr = granID_db[granID_db['id'] == id].iloc[0] 130 | row = pd.DataFrame({'granule': [f],'L1B_GEO': pr['L1B_GEO'], 'id': [id], 131 | 'GeolocationAccuracyQA': pr['GeolocationAccuracyQA'], 132 | 'note': pr['note']}, index=[0]) 133 | else: 134 | if 'ECOv002_' in f: 135 | url = get_link(id) 136 | if url: 137 | h5_link = [l for l in url[0] if l.endswith('.h5')][0] 138 | # print(h5_link) 139 | dmrpp_link = f'{h5_link}.dmrpp' 140 | label = dmrpp_geolocation(dmrpp_link) 141 | if label == 'failed': 142 | label = file_geolocation(h5_link) 143 | # print(label) 144 | row = pd.DataFrame({'granule': [f],'L1B_GEO': [h5_link.rsplit('/')[-1]], 'id': [id], 145 | 'GeolocationAccuracyQA': [label], 'note': '' }) 146 | else: 147 | # print(f'Failed to access ECO_L1B_GEO granule.') 148 | row = pd.DataFrame({'granule': [],'L1B_GEO': [], 'id': [], 'GeolocationAccuracyQA': [], 149 | 'note': 'Failed to access ECO_L1B_GEO granule' }) 150 | else: 151 | print(f'{f} is not associated to the ECOSTRESS version 2 collections.') 152 | row = pd.DataFrame() 153 | return(row) 154 | 155 | 156 | 157 | # -----------------------------------------GET THE LIST OF GRANULE ID(S)-------------------------------------- # 158 | # Initialize an empty DataFrame for granule data 159 | granID_db = pd.DataFrame(columns=['granule','L1B_GEO','id' ,'GeolocationAccuracyQA', 'note']) 160 | 161 | # Run the script 162 | 163 | # Add a progress bar to the loop 164 | for f in tqdm(fileList, desc="Processing files", unit="file"): 165 | if f: 166 | row_db = get_geolocation_label(granID_db, f) 167 | granID_db = pd.concat([granID_db, row_db], ignore_index=True) 168 | 169 | # -----------------------------------------SAVE THE DATABASE AS AN OUTPUT-------------------------------------- # 170 | # Remove 'id' column if it exists 171 | if 'id' in granID_db.columns: 172 | del granID_db['id'] 173 | 174 | # Check if the DataFrame is empty 175 | if granID_db.empty: 176 | print(Fore.RED + "Please note that there is no geolocation accuracy QA flag for your input data. This flag is not available for ECOSTRESS version 1 data collections.") 177 | else: 178 | # Save the DataFrame to a CSV file 179 | granID_db.to_csv(out_dir, index=False) 180 | print(Fore.RED + f'The output CSV file is stored at: {out_dir}') 181 | print(Style.RESET_ALL) -------------------------------------------------------------------------------- /python/scripts/extract_geolocation_flag/geolocation.md: -------------------------------------------------------------------------------- 1 | 2 | # Extract ECOSTRESS Geolocation Accuracy QA Flag 3 | 4 | ## Objective 5 | 6 | NASA's Land Processes Distributed Active Archive Center (LP DAAC) is responsible for archiving and distributing ECOSTRESS data products through the LP DAAC Cumulus cloud archive. ECOSTRESS Version 2 data is available in swath, gridded, and tiled formats, provided in both Cloud Optimized GeoTIFF (COG) and HDF5 file formats. ECOSTRESS Swath Geolocation data ([ECO_L1B_GEO v002](https://doi.org/10.5067/ECOSTRESS/ECO_L1B_GEO.002)) includes essential metadata fields, such as the `Geolocation Accuracy QA` flag, which are not carried over to higher-level data products. `Geolocation Accuracy QA` flag provides an indication of geolocation error. The gelocation accuracy for corrected data could be better than 50 meters but the data was processed without correcting the geolocation could have up to 7 km geolocation error. Users of ECOSTRESS data should evaluate the geolocation accuracy and whether larger errors can be tolerated for their applications. Otherwise, data flagged as “Suspect” or “No Match” may not be suitable for you. Additional details are provided in Section 4.2 of the [User Guide](https://lpdaac.usgs.gov/documents/1491/ECO1B_User_Guide_V2.pdf). 7 | 8 | > `Geolocation Accuracy QA` could have four different flags: 9 | 10 | >- **Best**: Image matching was performed for this scene. Good geolocation accuracy. 11 | >- **Good**: Image matching was performed in a nearby scene. Good geolocation accuracy. 12 | >- **Suspect**: Image matching occurred somewhere in the orbit. Expect increased error. 13 | >- **No Match**: No matches in orbit. Expect large geolocation errors. 14 | 15 | This Python-based command-line script enables users to retrieve the `Geolocation Accuracy QA` flags for the input data. Input data can be provided as: 16 | 1. A list of download URLs or granule filenames stored in a text or CSV file. 17 | 2. A download URL or granule filename as a string for a single file. 18 | 19 | The script automates the retrieval process, allowing users to efficiently determine the geolocation accuracy of their ECOSTRESS data products. Retrieved geolocation flags will be stored as a .csv file. 20 | 21 | ## Products 22 | 23 | This script works with all the ECOSTRESS version 2 data products. 24 | 25 | ## Prerequisites/Setup Instructions   26 | 27 | ### Environment Setup 28 | 29 | If you do not have an Environment Manager installed, we recommend [mamba](https://mamba.readthedocs.io/en/latest/) to manage Python packages. Details instruction on how to install an environment manager can be found [here](https://github.com/nasa/LPDAAC-Data-Resources/blob/main/setup/setup_instructions_python.md). Once you have it installed, type the following in your preferred command line interface (command prompt, terminal, cmder, etc.) to create a compatible environment. 30 | 31 | ``` 32 | mamba create -n ecostress_geo python=3.12 earthaccess xarray h5netcdf 33 | ``` 34 | **This script is tested using `earthaccess` version 0.13.** 35 | 36 | **If you are using conda, replace the "mamba" with "conda" and be patient.** 37 | 38 | Instruction for setting up a compatible environment for all LP DAAC Python resources is available at: 39 | 40 | ### NASA Earthdata Login: 41 | 42 | **You will need a NASA Earthdata Login account to access LP DAAC data (and use this script).** To create a NASA Earthdata Login account, go to the [Earthdata Login website](https://urs.earthdata.nasa.gov) and click the “Register” button, which is next to the green “Log In” button under the Password entry box. Fill in the required boxes (indicated with a red asterisk), then click on the “Register for Earthdata Login” green button at the bottom of the page. An email including instructions for activating your profile will be sent to you. Activate your profile to complete the registration process. 43 | 44 | ### **.netrc file** 45 | 46 | The netrc file is needed to download NASA Earthdata science data from a scripting environment like Python. There are multiple methods to [create a .netrc file for authntication](https://github.com/nasa/LPDAAC-Data-Resources/tree/main/guides/create_netrc_file.md). Here, the `earthaccess` package is used to automatically create a `.netrc` file using your Earthdata login credentials. If you do not have a netrc file configured in your home directory, the script will prompt you for input on your NASA Earthdata Login Username and Password. Enter your username and password and hit enter to continue downloading your data. **Please note that your Earthdata Login info, your username, and password, will be stored in a .netrc file located in the Home directory on this system you are using.** You will get the same message when you run the script as a reminder. If you do not trust the machine you are using, make sure to delete the created netrc file.   47 | 48 | ## Procedures: 49 | 50 | ### Getting Started: 51 | 52 | #### 1. Save the download URL for your data from  [NASA Earthdata Search](https://search.earthdata.nasa.gov/) for a single file. For multiple files, download the text file containing URLs to files.   53 | 54 | #### 2. Access `ECOSTRESS_geolocation.py` from `ECOSTRESS-Data-Resources` Github Repository.   55 | 56 | You can download the raw file for the script from 57 |     58 | Additionally, you can download all contents of this repository as a [zip file](https://github.com/nasa/ECOSTRESS-Data-Resources/archive/refs/heads/main.zip). You can also clone the repository by typing the following into command line. 59 | 60 | ```bash 61 | git clone https://github.com/nasa/ECOSTRESS-Data-Resources.git 62 | ``` 63 | 64 | Afterwards, navigate to `python/scripts/extract_geolocation_flag/ECOSTRESS_geolocation.py`.   65 | 66 | ## Script Execution 67 | 68 | Activate your MacOS/Windows environment, run the script with the following in your Command Prompt/terminal window: 69 | 70 | ```cmd 71 | python ECOSTRESS_geolocation.py -dir -f 72 | ``` 73 | 74 | **Example 1, extracting the geolocation flag for a list of ECOSTESS files:** 75 | 76 | ```cmd 77 | python C:\User\Downloads\ECOSTRESS_geolocation.py -dir C:\User\downloads -f C:\User\downloads\ECOSTRESS-granule-list.txt 78 | ``` 79 | 80 | **Example 2, extracting the geolocation flags for a single ECOSTRESS file:** 81 | 82 | ```cmd 83 | python C:\User\Downloads\DAACDataDownload.py  -dir C:\User\downloads -f https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/ECO_L4T_ESI.002/ECOv002_L4T_ESI_36662_013_07LFK_20241223T220254_0713_01/ECOv002_L4T_ESI_36662_013_07LFK_20241223T220254_0713_01_water.tif 84 | ``` 85 | 86 | 87 | ## Contact Information 88 | ### Author: LP DAAC 1 89 | 90 | **Contact:** LPDAAC@usgs.gov 91 | **Voice:** +1-866-573-3222 92 | **Organization:** Land Processes Distributed Active Archive Center (LP DAAC) 93 | **Website:** 94 | **Date last modified:** 02-03-2025 95 | 96 | 1LP DAAC Work performed under NASA contract NNG14HH33I. 97 | -------------------------------------------------------------------------------- /python/tutorials/ECOSTRESS_Tutorial.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Working with ECOSTRESS Evapotranspiration Data\n", 8 | "### This tutorial demonstrates how to work with the ECOSTRESS Evapotranspiration PT-JPL Daily L3 Global 70m Version 1 ([ECO3ETPTJPL.001](https://doi.org/10.5067/ECOSTRESS/ECO3ETPTJPL.001)) data product in Python.\n", 9 | "The Land Processes Distributed Active Archive Center (LP DAAC) distributes the Ecosystem Spaceborne Thermal Radiometer Experiment on Space Station ([ECOSTRESS](https://ecostress.jpl.nasa.gov/)) data products. The ECOSTRESS mission is tasked with measuring the temperature of plants to better understand how much water plants need and how they respond to stress. ECOSTRESS products are archived and distributed in the HDF5 file format as swath-based products.\n", 10 | "\n", 11 | "In this tutorial, you will use Python to perform a swath to grid conversion to project the swath data on to a grid with a defined coordinate reference system (CRS), compare ECOSTRESS data with ground-based AmeriFlux flux tower observations, and export science dataset (SDS) layers as GeoTIFF files that can be loaded into a GIS and/or Remote Sensing software program. \n", 12 | "***\n", 13 | "### Example: Converting a swath ECO3ETPTJPL.001 HDF5 file into a GeoTIFF with a defined CRS and comparing ECOSTRESS Evapotranspiration (ET) with ground-based ET observations from an AmeriFlux flux tower location in California. \n", 14 | "#### Data Used in the Example: \n", 15 | "- Data Product: ECOSTRESS Evapotranspiration PT-JPL Daily L3 Global 70m Version 1 ([ECO3ETPTJPL.001](https://doi.org/10.5067/ECOSTRESS/ECO3ETPTJPL.001)) \n", 16 | " - Science Dataset (SDS) layers: \n", 17 | " - ETinst \n", 18 | " - ETinstUncertainty \n", 19 | "- Data Product: ECOSTRESS Geolocation Daily L1B Global 70m Version 1 ([ECO1BGEO.001](https://doi.org/10.5067/ECOSTRESS/ECO1BGEO.001)) \n", 20 | " - Science Dataset (SDS) layers: \n", 21 | " - Latitude \n", 22 | " - Longitude \n", 23 | "- Data Product: AmeriFlux Ground Observations for [Flux Tower US-CZ3](https://ameriflux.lbl.gov/sites/siteinfo/US-CZ3): Sierra Critical Zone, Sierra Transect, Sierran Mixed Conifer, P301\n", 24 | " - Variables: \n", 25 | " - Latent Heat (W/m$^{2}$) \n", 26 | "*** \n", 27 | "# Topics Covered:\n", 28 | "1. **Getting Started** \n", 29 | " 1a. Import Packages \n", 30 | " 1b. Set Up the Working Environment \n", 31 | " 1c. Retrieve Files \n", 32 | "2. **Importing and Interpreting Data** \n", 33 | " 2a. Open an ECOSTRESS HDF5 File and Read File Metadata \n", 34 | " 2b. Subset SDS Layers \n", 35 | "3. **Performing Swath2grid Conversion** \n", 36 | " 3a. Import Geolocation File \n", 37 | " 3b. Define Projection and Output Grid \n", 38 | " 3c. Read SDS Metadata \n", 39 | " 3d. Perform K-D Tree Resampling \n", 40 | " 3e. Basic Image Processing \n", 41 | "4. **Exporting Results** \n", 42 | " 4a. Set Up a Dictionary \n", 43 | " 4b. Define CRS and Export as GeoTIFFs \n", 44 | "5. **Combining ECOSTRESS and AmeriFlux Tower Data** \n", 45 | " 5a. Loading Tables with Pandas \n", 46 | " 5b. Locate ECOSTRESS Pixel from Lat/Lon Coordinates \n", 47 | "6. **Visualizing Data** \n", 48 | " 6a. Create Colormap \n", 49 | " 6b. Plot ET Data \n", 50 | " 6c. Exporting an Image \n", 51 | "7. **Comparing Observations** \n", 52 | " 7a. Calculate Distribution of ECOSTRESS Data \n", 53 | " 7b. Visualize Ground Observations \n", 54 | " 7c. Combine ECOSTRESS and Ground Observations \n", 55 | "\n", 56 | "***\n", 57 | "# Before Starting this Tutorial: \n", 58 | "#### If you are simply looking to batch process/perform the swath2grid conversion for ECOSTRESS files, be sure to check out the [ECOSTRESS Swath to Grid Conversion Script](https://git.earthdata.nasa.gov/projects/LPDUR/repos/ecostress_swath2grid/browse).\n", 59 | "\n", 60 | "NOTE: This tutorial was developed specifically for the ECOSTRESS Evapotranspiration PT-JPL Level 3, Version 1 HDF5 files and will need to be adapted to work with other ECOSTRESS products. \n", 61 | "## Dependencies: \n", 62 | "*Disclaimer: This tutorial has been tested on Windows and MacOS using the specifications identified below.* \n", 63 | "+ #### Python Version 3.8 \n", 64 | " + `h5py` \n", 65 | " + `pyproj` \n", 66 | " + `matplotlib` \n", 67 | " + `pandas` \n", 68 | " + `pyresample` \n", 69 | " + `pandas` \n", 70 | " + `scipy` \n", 71 | " + `gdal` \n", 72 | " + `jupyter notebook`\n", 73 | "\n", 74 | "- A [NASA Earthdata Login](https://urs.earthdata.nasa.gov/) account is required to download the data used in this tutorial. You can create an account at the link provided. \n", 75 | "- The [Geospatial Data Abstraction Library](http://www.gdal.org/) (GDAL) is required.\n", 76 | "\n", 77 | "---\n", 78 | "# Procedures: \n", 79 | "## Getting Started: \n", 80 | "#### 1. This tutorial uses data from ECOSTRESS Version 1, including an ECO3ETPTJPL.001 (and accompanying ECO1BGEO.001) observation from August 05, 2018. You can download the files directly from the [LP DAAC Data Pool](https://e4ftl01.cr.usgs.gov/ECOSTRESS/) at:\n", 81 | " - [ECOSTRESS_L1B_GEO_00468_007_20180805T220314_0601_03.h5](https://e4ftl01.cr.usgs.gov/ECOSTRESS/ECO1BGEO.001/2018.08.05/ECOSTRESS_L1B_GEO_00468_007_20180805T220314_0601_03.h5) \n", 82 | " - [ECOSTRESS_L3_ET_PT-JPL_00468_007_20180805T220314_0601_04.h5](https://e4ftl01.cr.usgs.gov/ECOSTRESS/ECO3ETPTJPL.001/2018.08.05/ECOSTRESS_L3_ET_PT-JPL_00468_007_20180805T220314_0601_04.h5) \n", 83 | "**A [NASA Earthdata Login](https://urs.earthdata.nasa.gov/) account is required to download the data used in this tutorial. You can create an account at the link provided.** \n", 84 | "\n", 85 | " - Ancillary Files Needed: \n", 86 | " - The AmeriFlux Latent Heat data used in Section 4 can be downloaded via a [csv file](https://git.earthdata.nasa.gov/projects/LPDUR/repos/tutorial-ecostress/raw/tower_data.csv?at=refs%2Fheads%2Fmain). \n", 87 | "\n", 88 | "The `tower_data.csv` file will need to be downloaded into the same directory as the tutorial in order to execute the tutorial. \n", 89 | "#### 2.\tCopy/clone/download the [ECOSTRESS Tutorial repo](https://git.earthdata.nasa.gov/rest/api/latest/projects/LPDUR/repos/tutorial-ecostress/archive?format=zip), or the desired tutorial from the LP DAAC Data User Resources Repository: \n", 90 | " - [Working with ECOSTRESS Evapotranspiration Data in Python Jupyter Notebook](https://git.earthdata.nasa.gov/projects/LPDUR/repos/tutorial-ecostress/browse/ECOSTRESS_Tutorial.ipynb) \n", 91 | "
\n", 92 | "NOTE: This tutorial was developed specifically for the ECOSTRESS Evapotranspiration PT-JPL Level 3, Version 1 HDF5 files and will need to be adapted to work with other ECOSTRESS products.
" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "metadata": {}, 98 | "source": [ 99 | "***\n", 100 | "

1. Getting Started

\n", 101 | "\n", 102 | "***\n", 103 | "## 1a. Import Packages \n", 104 | "#### Import the python packages required to complete this tutorial." 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "metadata": {}, 111 | "outputs": [], 112 | "source": [ 113 | "# Import packages\n", 114 | "import warnings\n", 115 | "warnings.filterwarnings('ignore')\n", 116 | "\n", 117 | "import h5py\n", 118 | "import os\n", 119 | "from os.path import join\n", 120 | "import pyproj\n", 121 | "import numpy as np\n", 122 | "import pandas as pd\n", 123 | "import datetime\n", 124 | "from dateutil import parser\n", 125 | "import matplotlib.pyplot as plt\n", 126 | "from matplotlib.colors import LinearSegmentedColormap\n", 127 | "from pyresample import geometry as geom\n", 128 | "from pyresample import kd_tree as kdt\n", 129 | "from osgeo import gdal, gdal_array, gdalconst, osr\n", 130 | "\n", 131 | "# Set plots inside of the Jupyter Notebook\n", 132 | "%matplotlib inline" 133 | ] 134 | }, 135 | { 136 | "cell_type": "markdown", 137 | "metadata": {}, 138 | "source": [ 139 | "***\n", 140 | "## 1b. Set Up the Working Environment\n", 141 | "#### The input directory is defined as the current working directory. Note that you will need to have the jupyter notebook and example data (.h5 and .csv) stored in this directory in order to execute the tutorial successfully." 142 | ] 143 | }, 144 | { 145 | "cell_type": "code", 146 | "execution_count": null, 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "# Current working directory will be set as the input directory\n", 151 | "inDir = \"../data\" #os.getcwd() + os.sep \n", 152 | "print(\"input directory:\\n{}\".format(inDir))\n", 153 | "\n", 154 | "# Set output directory\n", 155 | "outDir = os.path.normpath(os.path.split(inDir)[0] + os.sep + 'output') + os.sep \n", 156 | "print(\"output directory:\\n{}\".format(outDir))\n", 157 | "\n", 158 | "# Create output directory\n", 159 | "if not os.path.exists(outDir): \n", 160 | " os.makedirs(outDir)" 161 | ] 162 | }, 163 | { 164 | "cell_type": "markdown", 165 | "metadata": {}, 166 | "source": [ 167 | "#### Make sure that the ECOSTRESS .h5 data files, and Ameriflux ET data file (.csv) are located in the input directory printed above." 168 | ] 169 | }, 170 | { 171 | "cell_type": "markdown", 172 | "metadata": {}, 173 | "source": [ 174 | "***\n", 175 | "## 1c. Retrieve Files\n", 176 | "#### Make sure that the ECO1BGEO and ECO3ETPTJPL .h5 files listed in the directions have been downloaded to the `inDir` defined above to follow along in the tutorial." 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "metadata": {}, 183 | "outputs": [], 184 | "source": [ 185 | "os.chdir(inDir)" 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "metadata": {}, 192 | "outputs": [], 193 | "source": [ 194 | "# List directory contents and create lists of ECOSTRESS HDF5 files (GEO, ET)\n", 195 | "geoList = [file for file in os.listdir() if file.endswith('.h5') and 'GEO' in file]\n", 196 | "print(\"geolocation:\\n{}\".format(\"\\n\".join(geoList)))\n", 197 | "ecoList = [file for file in os.listdir() if file.endswith('.h5') and 'GEO' not in file]\n", 198 | "print(\"products:\\n{}\".format(\"\\n\".join(ecoList)))" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "#### The standard format for ECOSTRESS filenames is as follows:\n", 206 | "> **ECOSTRESS_L3_ET_PT-JPL**: Product Type \n", 207 | "**00468**: Orbit number; starting at start of mission, ascending equatorial crossing \n", 208 | "**007**: Scene ID; starting at first scene of first orbit \n", 209 | "**20180805T220314**: Date and time of data start: YYYYMMDDThhmmss \n", 210 | "**0601**: Build ID of software that generated product, Major+Minor (2+2 digits) \n", 211 | "**04**: Product version number (2 digits) " 212 | ] 213 | }, 214 | { 215 | "cell_type": "markdown", 216 | "metadata": {}, 217 | "source": [ 218 | "***\n", 219 | "

2. Importing and Interpreting Data

\n", 220 | "\n", 221 | "***\n", 222 | "## 2a. Open an ECOSTRESS HDF5 File\n", 223 | "#### Read in an ECOSTRESS HDF5 file using the `h5py` package." 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "metadata": { 230 | "scrolled": true 231 | }, 232 | "outputs": [], 233 | "source": [ 234 | "f = h5py.File(ecoList[0]) # Read in ECOSTRESS HDF5 file\n", 235 | "ecoName = ecoList[0].split('.h5')[0] # Keep original filename\n", 236 | "print(ecoName)" 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "metadata": {}, 242 | "source": [ 243 | "***\n", 244 | "## 2b. Subset SDS Layers and read SDS Metadata\n", 245 | "#### Identify and generate a list of all the SDS layers in the HDF5 file." 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "metadata": {}, 252 | "outputs": [], 253 | "source": [ 254 | "# Create a list of all SDS inside of the .h5 file\n", 255 | "eco_objs = []\n", 256 | "f.visit(eco_objs.append)\n", 257 | "ecoSDS = [str(obj) for obj in eco_objs if isinstance(f[obj], h5py.Dataset)] \n", 258 | "for dataset in ecoSDS[0:10]: \n", 259 | " print(dataset)" 260 | ] 261 | }, 262 | { 263 | "cell_type": "markdown", 264 | "metadata": {}, 265 | "source": [ 266 | "#### Below, subset the SDS list to the two layers needed for comparison with the ground-based AmeriFlux data, `ETinst` and `ETinstUncertainty`." 267 | ] 268 | }, 269 | { 270 | "cell_type": "code", 271 | "execution_count": null, 272 | "metadata": {}, 273 | "outputs": [], 274 | "source": [ 275 | "# Subset list to ETinst and ETinstUncertainty\n", 276 | "sds = ['ETinst', 'ETinstUncertainty']\n", 277 | "ecoSDS = [dataset for dataset in ecoSDS if dataset.endswith(tuple(sds))]\n", 278 | "for dataset in ecoSDS:\n", 279 | " print(dataset.split('/')[-1])" 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": {}, 285 | "source": [ 286 | "***\n", 287 | "

3. Performing Swath2grid Conversion

\n", 288 | "\n", 289 | "#### Resample the native ECOSTRESS swath data to a grid with defined coordinate reference system (CRS). \n", 290 | "***\n", 291 | "## 3a. Import Geolocation File\n", 292 | "#### The `latitude` and `longitude` arrays from the ECO1BGEO product for the same ECOSTRESS orbit/scene ID are needed to perform the swath2grid conversion on the ECO3ETPT-JPL file. " 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": null, 298 | "metadata": {}, 299 | "outputs": [], 300 | "source": [ 301 | "# Find the matching ECO1BGEO file from the file list\n", 302 | "geo = [geoFile for geoFile in geoList if ecoList[0].split('ECOSTRESS_L3_ET_PT-JPL_')[-1].split('T')[0] in geoFile]\n", 303 | "print(geo[0])\n", 304 | "print(ecoList[0])" 305 | ] 306 | }, 307 | { 308 | "cell_type": "markdown", 309 | "metadata": {}, 310 | "source": [ 311 | "#### Read in the ECO1BGEO file, search for the `latitude` and `longitude` SDS, and import into Python as arrays." 312 | ] 313 | }, 314 | { 315 | "cell_type": "code", 316 | "execution_count": null, 317 | "metadata": {}, 318 | "outputs": [], 319 | "source": [ 320 | "# Open Geo File\n", 321 | "g = h5py.File(geo[0])\n", 322 | "geo_objs = []\n", 323 | "g.visit(geo_objs.append)\n", 324 | "\n", 325 | "# Search for lat/lon SDS inside data file\n", 326 | "latSD = [str(obj) for obj in geo_objs if isinstance(g[obj], h5py.Dataset) and '/latitude' in obj]\n", 327 | "lonSD = [str(obj) for obj in geo_objs if isinstance(g[obj], h5py.Dataset) and '/longitude' in obj]\n", 328 | "\n", 329 | "# Open SDS as arrays\n", 330 | "lat = g[latSD[0]][()].astype(float)\n", 331 | "lon = g[lonSD[0]][()].astype(float)\n", 332 | "\n", 333 | "# Read the array dimensions\n", 334 | "dims = lat.shape\n", 335 | "print(dims)" 336 | ] 337 | }, 338 | { 339 | "cell_type": "markdown", 340 | "metadata": {}, 341 | "source": [ 342 | "***\n", 343 | "## 3b. Define Projection and Output Grid\n", 344 | "#### The `latitude` and `longitude` arrays from the ECO1BGEO product for the same ECOSTRESS orbit/scene ID are needed to perform the swath2grid conversion on the ECO3ETPT-JPL file. \n", 345 | "#### The following sections use the [pyresample package](https://pyresample.readthedocs.io/en/stable/) to resample the ECOSTRESS swath dataset to a grid using nearest neighbor method. This process begins by defining the swath dimensions using the lat/lon arrays below." 346 | ] 347 | }, 348 | { 349 | "cell_type": "code", 350 | "execution_count": null, 351 | "metadata": {}, 352 | "outputs": [], 353 | "source": [ 354 | "# Set swath definition from lat/lon arrays\n", 355 | "swathDef = geom.SwathDefinition(lons=lon, lats=lat)\n", 356 | "swathDef.corners" 357 | ] 358 | }, 359 | { 360 | "cell_type": "markdown", 361 | "metadata": {}, 362 | "source": [ 363 | "#### Define the coordinates in the middle of the swath, which are used to calculate an estimate of the output rows/columns for the gridded output. " 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": null, 369 | "metadata": {}, 370 | "outputs": [], 371 | "source": [ 372 | "# Define the lat/lon for the middle of the swath\n", 373 | "mid = [int(lat.shape[1] / 2) - 1, int(lat.shape[0] / 2) - 1]\n", 374 | "midLat, midLon = lat[mid[0]][mid[1]], lon[mid[0]][mid[1]]\n", 375 | "midLat, midLon" 376 | ] 377 | }, 378 | { 379 | "cell_type": "markdown", 380 | "metadata": {}, 381 | "source": [ 382 | "#### Below, `pyproj.Proj` is used to perform a cartographic transformation by defining an Azimuthal Equidistant projection centered on the midpoint of the swath. Once the projection is defined, convert the lower left and upper right corners of the lat/lon arrays to a location (in meters) in the new projection. Lastly, measure the distance between the corners and divide by 70 (meters), the nominal pixel size that we are aiming for. Azimuthal Equidistant projection was chosen here based on the following characteristics of this projection:\n", 383 | "- Units in meters (necessary for defining 70 m pixels) \n", 384 | "- Distances between all points are proportionally correct from center point\n", 385 | "- Azimuth (direction) are correct from the center point" 386 | ] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "execution_count": null, 391 | "metadata": {}, 392 | "outputs": [], 393 | "source": [ 394 | "# Define AEQD projection centered at swath center\n", 395 | "epsgConvert = pyproj.Proj(\"+proj=aeqd +lat_0={} +lon_0={}\".format(midLat, midLon))\n", 396 | "\n", 397 | "# Use info from AEQD projection bbox to calculate output cols/rows/pixel size\n", 398 | "llLon, llLat = epsgConvert(np.min(lon), np.min(lat), inverse=False)\n", 399 | "urLon, urLat = epsgConvert(np.max(lon), np.max(lat), inverse=False)\n", 400 | "areaExtent = (llLon, llLat, urLon, urLat)\n", 401 | "cols = int(round((areaExtent[2] - areaExtent[0]) / 70)) # 70 m pixel size\n", 402 | "rows = int(round((areaExtent[3] - areaExtent[1]) / 70))" 403 | ] 404 | }, 405 | { 406 | "cell_type": "markdown", 407 | "metadata": {}, 408 | "source": [ 409 | "#### Use number of rows and columns generated above from the AEQD projection to set a representative number of rows and columns in the Geographic area definition, which will then be translated to degrees below, then take the smaller of the two pixel dims to determine output size and ensure square pixels." 410 | ] 411 | }, 412 | { 413 | "cell_type": "code", 414 | "execution_count": null, 415 | "metadata": {}, 416 | "outputs": [], 417 | "source": [ 418 | "# Define Geographic projection\n", 419 | "epsg, proj, pName = '4326', 'longlat', 'Geographic'\n", 420 | "\n", 421 | "# Define bounding box of swath\n", 422 | "llLon, llLat, urLon, urLat = np.min(lon), np.min(lat), np.max(lon), np.max(lat)\n", 423 | "areaExtent = (llLon, llLat, urLon, urLat)\n", 424 | "\n", 425 | "# Create area definition with estimated number of columns and rows\n", 426 | "projDict = pyproj.CRS(\"epsg:4326\")\n", 427 | "areaDef = geom.AreaDefinition(epsg, pName, proj, projDict, cols, rows, areaExtent)" 428 | ] 429 | }, 430 | { 431 | "cell_type": "markdown", 432 | "metadata": {}, 433 | "source": [ 434 | "#### Below, square the pixels by setting the pixel size to the smaller of the x any y values output by the `AreaDefinition`, then use the pixel size to recalculate the number of output cols/rows." 435 | ] 436 | }, 437 | { 438 | "cell_type": "code", 439 | "execution_count": null, 440 | "metadata": {}, 441 | "outputs": [], 442 | "source": [ 443 | "# Square pixels and calculate output cols/rows\n", 444 | "ps = np.min([areaDef.pixel_size_x, areaDef.pixel_size_y])\n", 445 | "cols = int(round((areaExtent[2] - areaExtent[0]) / ps))\n", 446 | "rows = int(round((areaExtent[3] - areaExtent[1]) / ps))\n", 447 | "\n", 448 | "# Set up a new Geographic area definition with the refined cols/rows\n", 449 | "areaDef = geom.AreaDefinition(epsg, pName, proj, projDict, cols, rows, areaExtent)" 450 | ] 451 | }, 452 | { 453 | "cell_type": "markdown", 454 | "metadata": {}, 455 | "source": [ 456 | "#### Below, use `pyresample kd_tree`'s [get_neighbour_info](https://pyresample.readthedocs.io/en/latest/swath.html#resampling-from-neighbour-info) to create arrays with information on the nearest neighbor to each grid point. \n", 457 | "#### This is the most computationally heavy task in the swath2grid conversion and using `get_neighbour_info` speeds up the process if you plan to resample multiple SDS within an ECOSTRESS product (compute once instead of for every SDS). " 458 | ] 459 | }, 460 | { 461 | "cell_type": "code", 462 | "execution_count": null, 463 | "metadata": {}, 464 | "outputs": [], 465 | "source": [ 466 | "# Get arrays with information about the nearest neighbor to each grid point \n", 467 | "index, outdex, indexArr, distArr = kdt.get_neighbour_info(swathDef, areaDef, 210, neighbours=1)" 468 | ] 469 | }, 470 | { 471 | "cell_type": "markdown", 472 | "metadata": {}, 473 | "source": [ 474 | "#### Above, the function is comparing the swath and area definitions to locate the nearest neighbor (neighbours=1). 210 is the `radius_of_influence`, or the radius used to search for the nearest neighboring pixel in the swath (in meters). " 475 | ] 476 | }, 477 | { 478 | "cell_type": "markdown", 479 | "metadata": {}, 480 | "source": [ 481 | "***\n", 482 | "## 3c. Read SDS Metadata\n", 483 | "#### List the attributes for the `ETinst` layer, which can then be used to define the fill value and scale factor. " 484 | ] 485 | }, 486 | { 487 | "cell_type": "code", 488 | "execution_count": null, 489 | "metadata": {}, 490 | "outputs": [], 491 | "source": [ 492 | "# Read in ETinst and print out SDS attributes\n", 493 | "s = ecoSDS[0]\n", 494 | "ecoSD = f[s][()]\n", 495 | "for attr in f[s].attrs:\n", 496 | " if type(f[s].attrs[attr]) == np.ndarray:\n", 497 | " print(f'{attr} = {f[s].attrs[attr][0]}')\n", 498 | " else:\n", 499 | " print(f'{attr} = {f[s].attrs[attr].decode(\"utf-8\")}')" 500 | ] 501 | }, 502 | { 503 | "cell_type": "markdown", 504 | "metadata": {}, 505 | "source": [ 506 | "#### Extract the scale factor, add offset and fill value from the SDS metadata." 507 | ] 508 | }, 509 | { 510 | "cell_type": "code", 511 | "execution_count": null, 512 | "metadata": {}, 513 | "outputs": [], 514 | "source": [ 515 | "f[s].attrs['_FillValue'][0]" 516 | ] 517 | }, 518 | { 519 | "cell_type": "code", 520 | "execution_count": null, 521 | "metadata": {}, 522 | "outputs": [], 523 | "source": [ 524 | "# Read SDS attributes and define fill value, add offset, and scale factor if available\n", 525 | "try:\n", 526 | " fv = int(f[s].attrs['_FillValue'])\n", 527 | "except KeyError:\n", 528 | " fv = None\n", 529 | "except ValueError:\n", 530 | " fv = f[s].attrs['_FillValue'][0]\n", 531 | "try:\n", 532 | " sf = f[s].attrs['_Scale'][0]\n", 533 | "except:\n", 534 | " sf = 1\n", 535 | "try:\n", 536 | " add_off = f[s].attrs['_Offset'][0]\n", 537 | "except:\n", 538 | " add_off = 0\n", 539 | "try:\n", 540 | " units = f[s].attrs['units'].decode(\"utf-8\")\n", 541 | "except:\n", 542 | " units = 'none'" 543 | ] 544 | }, 545 | { 546 | "cell_type": "markdown", 547 | "metadata": {}, 548 | "source": [ 549 | "***\n", 550 | "## 3d. Perform K-D Tree Resampling\n", 551 | "#### Remember that the resampling has been split into two steps. In section 3b. arrays containing the nearest neighbor to each grid point were created. The second step is to use those arrays to retrieve a resampled result. " 552 | ] 553 | }, 554 | { 555 | "cell_type": "code", 556 | "execution_count": null, 557 | "metadata": {}, 558 | "outputs": [], 559 | "source": [ 560 | "# Perform K-D Tree nearest neighbor resampling (swath 2 grid conversion)\n", 561 | "ETgeo = kdt.get_sample_from_neighbour_info('nn', areaDef.shape, ecoSD, index, outdex, indexArr, fill_value=None)" 562 | ] 563 | }, 564 | { 565 | "cell_type": "markdown", 566 | "metadata": {}, 567 | "source": [ 568 | "#### Above, resample the swath `ecoSD` array using nearest neighbor (already calculated in section 3b. and defined above as the `index`, `outdex`, and `indexArr`), and also set the fill value that was defined in section 3c. \n", 569 | "#### Below, define the geotransform for the output (upper left x, horizontal pixel size, rotation, upper left y, rotation, vertical pixel size)." 570 | ] 571 | }, 572 | { 573 | "cell_type": "code", 574 | "execution_count": null, 575 | "metadata": {}, 576 | "outputs": [], 577 | "source": [ 578 | "# Define the geotransform \n", 579 | "gt = [areaDef.area_extent[0], ps, 0, areaDef.area_extent[3], 0, -ps]\n", 580 | "gt" 581 | ] 582 | }, 583 | { 584 | "cell_type": "markdown", 585 | "metadata": {}, 586 | "source": [ 587 | "***\n", 588 | "## 3e. Basic Image Processing\n", 589 | "#### Apply the scale factor and add offset and set the fill value defined in the previous section on the resampled data." 590 | ] 591 | }, 592 | { 593 | "cell_type": "code", 594 | "execution_count": null, 595 | "metadata": {}, 596 | "outputs": [], 597 | "source": [ 598 | "ETgeo = ETgeo * sf + add_off # Apply Scale Factor and Add Offset\n", 599 | "ETgeo[ETgeo == fv * sf + add_off] = fv # Set Fill Value" 600 | ] 601 | }, 602 | { 603 | "cell_type": "markdown", 604 | "metadata": {}, 605 | "source": [ 606 | "#### Rerun steps 3c - 3e for `ETinstUncertainty`." 607 | ] 608 | }, 609 | { 610 | "cell_type": "code", 611 | "execution_count": null, 612 | "metadata": {}, 613 | "outputs": [], 614 | "source": [ 615 | "s = ecoSDS[1]\n", 616 | "ecoSD = f[s][()]\n", 617 | "try:\n", 618 | " fv = int(f[s].attrs['_FillValue'])\n", 619 | "except KeyError:\n", 620 | " fv = None\n", 621 | "except ValueError:\n", 622 | " fv = f[s].attrs['_FillValue'][0]\n", 623 | "try:\n", 624 | " sf = f[s].attrs['_Scale'][0]\n", 625 | "except:\n", 626 | " sf = 1\n", 627 | "try:\n", 628 | " add_off = f[s].attrs['_Offset'][0]\n", 629 | "except:\n", 630 | " add_off = 0\n", 631 | "UNgeo = kdt.get_sample_from_neighbour_info('nn', areaDef.shape, ecoSD, index, outdex, indexArr, fill_value=None)\n", 632 | "UNgeo = UNgeo * sf + add_off\n", 633 | "UNgeo[UNgeo == fv * sf + add_off] = fv" 634 | ] 635 | }, 636 | { 637 | "cell_type": "markdown", 638 | "metadata": {}, 639 | "source": [ 640 | "***\n", 641 | "

4. Exporting Results

\n", 642 | "\n", 643 | "***\n", 644 | "## 4a. Set Up a Dictionary\n", 645 | "#### In this section, create a dictionary containing each of the arrays that will be exported as GeoTIFFs." 646 | ] 647 | }, 648 | { 649 | "cell_type": "code", 650 | "execution_count": null, 651 | "metadata": {}, 652 | "outputs": [], 653 | "source": [ 654 | "# Set up dictionary of arrays to export\n", 655 | "outFiles = {'ETinst': ETgeo, 'ETinstUncertainty': UNgeo}" 656 | ] 657 | }, 658 | { 659 | "cell_type": "markdown", 660 | "metadata": {}, 661 | "source": [ 662 | "***\n", 663 | "## 4b. Define CRS and Export as GeoTIFFs\n", 664 | "#### Now that the data have been imported and resampled into a gridded raster array, export the results as GeoTIFFs using a `for` loop in this section." 665 | ] 666 | }, 667 | { 668 | "cell_type": "code", 669 | "execution_count": null, 670 | "metadata": {}, 671 | "outputs": [], 672 | "source": [ 673 | "fv = np.nan" 674 | ] 675 | }, 676 | { 677 | "cell_type": "code", 678 | "execution_count": null, 679 | "metadata": { 680 | "scrolled": true 681 | }, 682 | "outputs": [], 683 | "source": [ 684 | "# Loop through each item in dictionary created above\n", 685 | "for file in outFiles:\n", 686 | " \n", 687 | " # Set up output name\n", 688 | " outName = join(outDir, '{}_{}.tif'.format(ecoName, file))\n", 689 | " print(\"output file:\\n{}\\n\".format(outName))\n", 690 | " \n", 691 | " # Get driver, specify dimensions, define and set output geotransform\n", 692 | " height, width = outFiles[file].shape\n", 693 | " driv = gdal.GetDriverByName('GTiff')\n", 694 | " dataType = gdal_array.NumericTypeCodeToGDALTypeCode(outFiles[file].dtype)\n", 695 | " d = driv.Create(outName, width, height, 1, dataType)\n", 696 | " d.SetGeoTransform(gt)\n", 697 | " \n", 698 | " # Create and set output projection, write output array data\n", 699 | " # Define target SRS\n", 700 | " srs = osr.SpatialReference()\n", 701 | " srs.ImportFromEPSG(int(epsg))\n", 702 | " d.SetProjection(srs.ExportToWkt())\n", 703 | " srs.ExportToWkt()\n", 704 | " \n", 705 | " # Write array to band\n", 706 | " band = d.GetRasterBand(1)\n", 707 | " band.WriteArray(outFiles[file])\n", 708 | " \n", 709 | " # Define fill value if it exists, if not, set to mask fill value\n", 710 | " if fv is not None and fv != 'NaN':\n", 711 | " band.SetNoDataValue(fv)\n", 712 | " else:\n", 713 | " try:\n", 714 | " band.SetNoDataValue(outFiles[file].fill_value)\n", 715 | " except:\n", 716 | " pass\n", 717 | " band.FlushCache()\n", 718 | " d, band = None, None " 719 | ] 720 | }, 721 | { 722 | "cell_type": "markdown", 723 | "metadata": {}, 724 | "source": [ 725 | "***\n", 726 | "

5. Combining ECOSTRESS and AmeriFlux Tower Data

\n", 727 | "\n", 728 | "***\n", 729 | "## 5a. Loading Tables with Pandas\n", 730 | "#### In this section, begin by highlighting how to open a csv file using the `pandas` package. \n", 731 | "> The AmeriFlux tower data was provided by Mike Goulden for the AmeriFlux US-CZ3 tower. The csv includes half-hourly observations of Latent Heat (W/m$^{2}$) for the same day as the ECOSTRESS observation. " 732 | ] 733 | }, 734 | { 735 | "cell_type": "code", 736 | "execution_count": null, 737 | "metadata": {}, 738 | "outputs": [], 739 | "source": [ 740 | "# Import csv file with AmeriFlux data and drop NaNs\n", 741 | "towerData = pd.read_csv('tower_data.csv')\n", 742 | "towerData = towerData.dropna()" 743 | ] 744 | }, 745 | { 746 | "cell_type": "markdown", 747 | "metadata": {}, 748 | "source": [ 749 | "#### Next, use the `parser` package and a lambda function to go through each time stamp and reformat to date and time objects." 750 | ] 751 | }, 752 | { 753 | "cell_type": "code", 754 | "execution_count": null, 755 | "metadata": {}, 756 | "outputs": [], 757 | "source": [ 758 | "# Define a lambda function to use the parser packgage to convert each time stamp to a datetime object\n", 759 | "towerData[\"Date/Time\"] = towerData[\"Time\"].apply(lambda x: parser.parse(x))\n", 760 | "towerData[\"Time\"] = towerData[\"Date/Time\"].apply(lambda x: datetime.time(x.hour, x.minute))\n", 761 | "towerData = towerData[[\"Date/Time\", \"Time\", \"LE\"]]\n", 762 | "towerData" 763 | ] 764 | }, 765 | { 766 | "cell_type": "markdown", 767 | "metadata": {}, 768 | "source": [ 769 | "***\n", 770 | "## 5b. Locate ECOSTRESS Pixel from Lat/Lon Coordinates\n", 771 | "\n", 772 | "#### Calculate the gridded pixel nearest to the tower location." 773 | ] 774 | }, 775 | { 776 | "cell_type": "code", 777 | "execution_count": null, 778 | "metadata": {}, 779 | "outputs": [], 780 | "source": [ 781 | "towerLat, towerLon = 37.0674, -119.1951 # AmeriFlux US-CZ3 tower location\n", 782 | "\n", 783 | "# Calculate tower lat/lon distance from upper left corner, then divide by pixel size to find x,y pixel location\n", 784 | "Tcol = int(round((towerLon - gt[0]) / gt[1]))\n", 785 | "Trow = int(round((towerLat - gt[3]) / gt[5]))\n", 786 | "\n", 787 | "# Print ET at the tower location\n", 788 | "ETgeo[Trow, Tcol] " 789 | ] 790 | }, 791 | { 792 | "cell_type": "markdown", 793 | "metadata": {}, 794 | "source": [ 795 | "***\n", 796 | "

6. Visualizing Data

\n", 797 | "\n", 798 | "***\n", 799 | "## 6a. Create a Colormap\n", 800 | "#### Before plotting the ET data, set up an Evapotranspiration color map using `LinearSegmentedColormap` from the `matplotlib` package." 801 | ] 802 | }, 803 | { 804 | "cell_type": "code", 805 | "execution_count": null, 806 | "metadata": {}, 807 | "outputs": [], 808 | "source": [ 809 | "# Create a colormap for the ET data\n", 810 | "ETcolors = [\"#f6e8c3\", \"#d8b365\", \"#99974a\", \"#53792d\", \"#6bdfd2\", \"#1839c5\"]\n", 811 | "ETcmap = LinearSegmentedColormap.from_list(\"ET\", ETcolors)" 812 | ] 813 | }, 814 | { 815 | "cell_type": "markdown", 816 | "metadata": {}, 817 | "source": [ 818 | "***\n", 819 | "## 6b. Calculate Local Overpass Time\n", 820 | "#### ECOSTRESS observation times are reported in Universal Time Coordinated (UTC). Below, grab the observation time from the filename and convert to local time using the longitude location of the tower." 821 | ] 822 | }, 823 | { 824 | "cell_type": "code", 825 | "execution_count": null, 826 | "metadata": {}, 827 | "outputs": [], 828 | "source": [ 829 | "# Grab UTC time of observation from file name\n", 830 | "ecoTime = ecoName.split('_')[-3]\n", 831 | "ecoTime" 832 | ] 833 | }, 834 | { 835 | "cell_type": "markdown", 836 | "metadata": {}, 837 | "source": [ 838 | "#### Next, convert UTC observation time to local overpass time. " 839 | ] 840 | }, 841 | { 842 | "cell_type": "code", 843 | "execution_count": null, 844 | "metadata": {}, 845 | "outputs": [], 846 | "source": [ 847 | "observationTime = parser.parse(ecoTime)\n", 848 | "solarOverpass = observationTime + datetime.timedelta(hours=(np.radians(towerLon) / np.pi * 12))\n", 849 | "overpass = datetime.time(solarOverpass.hour, solarOverpass.minute)\n", 850 | "date = observationTime.strftime('%Y-%m-%d')" 851 | ] 852 | }, 853 | { 854 | "cell_type": "markdown", 855 | "metadata": {}, 856 | "source": [ 857 | "***\n", 858 | "## 6c. Plot ET Data\n", 859 | "#### In this section, begin by highlighting the functionality of the `matplotlib` plotting package. First, make a plot of the entire gridded ET output. Next, zoom in on the tower location and add some additional parameters to the plot. Finally, export the completed plot to an image file." 860 | ] 861 | }, 862 | { 863 | "cell_type": "code", 864 | "execution_count": null, 865 | "metadata": {}, 866 | "outputs": [], 867 | "source": [ 868 | "title = 'ECO3ETPTJPL Evapotranspiration'\n", 869 | "SDSname = ecoSDS[0].split(\"/\")[-1]\n", 870 | "fig = plt.figure(figsize=(9.7,7.6)) # Set the figure size (x,y)\n", 871 | "fig.suptitle(f'{title} ({ecoSDS[0].split(\"/\")[-1]})\\n{date} at {overpass}', fontsize=22) # Add title for the plots\n", 872 | "plt.axis('off') # Remove axes from plot\n", 873 | "im = plt.imshow(ETgeo, cmap=ETcmap); # Plot array using colormap\n", 874 | "plt.scatter(Tcol, Trow, color=\"black\", marker='x') # Plot tower location\n", 875 | "\n", 876 | "# Add a colormap legend\n", 877 | "plt.colorbar(im, orientation='horizontal', fraction=0.05, pad=0.004, label=f'ET ({units})', shrink=0.6).outline.set_visible(True)" 878 | ] 879 | }, 880 | { 881 | "cell_type": "markdown", 882 | "metadata": {}, 883 | "source": [ 884 | "***\n", 885 | "## 6d. Exporting an Image\n", 886 | "#### Zoom in to get a closer look at the region surrounding the AmeriFlux tower by creating a subset." 887 | ] 888 | }, 889 | { 890 | "cell_type": "code", 891 | "execution_count": null, 892 | "metadata": {}, 893 | "outputs": [], 894 | "source": [ 895 | "# Set a Radius and calculate subset region from flux tower location (row, col)\n", 896 | "radius = 1700\n", 897 | "ETsubset = ETgeo[(Trow - radius):(Trow + radius + 1), (Tcol - radius):(Tcol + radius + 1)]" 898 | ] 899 | }, 900 | { 901 | "cell_type": "markdown", 902 | "metadata": {}, 903 | "source": [ 904 | "#### Make another plot, this time zoomed in to the tower location. Export the plot as a .png file. " 905 | ] 906 | }, 907 | { 908 | "cell_type": "code", 909 | "execution_count": null, 910 | "metadata": {}, 911 | "outputs": [], 912 | "source": [ 913 | "fig = plt.figure(figsize=(14,12)) # Set the figure size (x,y)\n", 914 | "fig.suptitle(f'{title} ({SDSname})\\n{date} at {overpass}', fontsize=26) # Add title for the plots\n", 915 | "plt.axis('off') # Remove axes from plot\n", 916 | "im = plt.imshow(ETsubset, cmap=ETcmap); # Plot array using colormap\n", 917 | "plt.scatter(ETsubset.shape[0]/2, ETsubset.shape[1]/2, color=\"black\", marker='x') # Tower is in middle of subset\n", 918 | "\n", 919 | "# Add a colormap legend\n", 920 | "plt.colorbar(im, orientation='horizontal', fraction=0.05, pad=0.004, label=f'ET ({units})', shrink=0.6).outline.set_visible(True)\n", 921 | "\n", 922 | "# Set up file name and export to png file\n", 923 | "figure_filename = join(outDir, \"{}_{}.png\".format(ecoName, SDSname))\n", 924 | "print(\"figure filename: {}\".format(figure_filename))\n", 925 | "fig.savefig(figure_filename, dpi=300)\n", 926 | "plt.show()" 927 | ] 928 | }, 929 | { 930 | "cell_type": "markdown", 931 | "metadata": {}, 932 | "source": [ 933 | "***\n", 934 | "

7. Comparing Observations

\n", 935 | "\n", 936 | "***\n", 937 | "## 7a. Calculate Distribution of ECOSTRESS Data\n", 938 | "#### First, collect a 3x3 grid centered on the flux tower pixel as a subset to calculate statistics on. " 939 | ] 940 | }, 941 | { 942 | "cell_type": "code", 943 | "execution_count": null, 944 | "metadata": {}, 945 | "outputs": [], 946 | "source": [ 947 | "# Subset data to 3x3 grid surrounding flux tower for both layers\n", 948 | "ETfootprint = ETgeo[(Trow - 1):(Trow + 2), (Tcol - 1):(Tcol + 2)] \n", 949 | "UNfootprint = UNgeo[(Trow - 1):(Trow + 2), (Tcol - 1):(Tcol + 2)] \n", 950 | "print(ETfootprint)" 951 | ] 952 | }, 953 | { 954 | "cell_type": "markdown", 955 | "metadata": {}, 956 | "source": [ 957 | "#### In case the 3x3 grid contains missing values, use `np.nanmedian` to ignoring missing values and calculate the measure of central tendency." 958 | ] 959 | }, 960 | { 961 | "cell_type": "code", 962 | "execution_count": null, 963 | "metadata": {}, 964 | "outputs": [], 965 | "source": [ 966 | "ETmedian = np.nanmedian(ETfootprint)\n", 967 | "UNmedian = np.nanmedian(UNfootprint)\n", 968 | "print(f\"Median ET: {ETmedian:0.3f} \\nUncertainty: {UNmedian:0.3f}\")" 969 | ] 970 | }, 971 | { 972 | "cell_type": "markdown", 973 | "metadata": {}, 974 | "source": [ 975 | "#### Next, generate a probability density function for the 3x3 grid of ET values." 976 | ] 977 | }, 978 | { 979 | "cell_type": "code", 980 | "execution_count": null, 981 | "metadata": {}, 982 | "outputs": [], 983 | "source": [ 984 | "pd.DataFrame({title: ETfootprint.flatten()}).plot.kde() # Pandas Kernel Density Estimate\n", 985 | "plt.title(f'Probability Density of {SDSname} Surrounding US-CZ3\\n{date} at {overpass}'); # Title\n", 986 | "plt.xlabel(f'{SDSname} ({units})'); # X-axis label" 987 | ] 988 | }, 989 | { 990 | "cell_type": "markdown", 991 | "metadata": {}, 992 | "source": [ 993 | "***\n", 994 | "## 7b. Visualize Ground Observations \n", 995 | "\n", 996 | "#### Next, examine the series of eddy covariance observations from the AmeriFlux US-CZ3 dataset." 997 | ] 998 | }, 999 | { 1000 | "cell_type": "code", 1001 | "execution_count": null, 1002 | "metadata": {}, 1003 | "outputs": [], 1004 | "source": [ 1005 | "fig = plt.figure(figsize=(12,7)) # Set fig size (x,y)\n", 1006 | "ax = fig.add_subplot(111) # Create a subplot\n", 1007 | "ax.plot(towerData['Date/Time'], towerData.LE, 'k', lw=2.5, color='black') # Plot as a black line\n", 1008 | "ax.set_title(f'Time Series of Eddy Covariance Data at Site US-CZ3', fontsize=20, fontweight='bold'); # Set Title\n", 1009 | "ax.set_ylabel('Latent Heat (W/m^2)', fontsize=18); # Y-axis label\n", 1010 | "ax.set_xlabel(f'Time of Day ({date})', fontsize=18); # X-axis label\n", 1011 | "ax.set_xticks(towerData['Date/Time']); # Set the x ticks\n", 1012 | "ax.set_xticklabels(towerData['Time'], rotation=45,fontsize=12); # Set x tick labels" 1013 | ] 1014 | }, 1015 | { 1016 | "cell_type": "markdown", 1017 | "metadata": {}, 1018 | "source": [ 1019 | "#### Above, we can see the daily range in Latent Heat as captured by the eddy covariance observations on the flux tower." 1020 | ] 1021 | }, 1022 | { 1023 | "cell_type": "markdown", 1024 | "metadata": {}, 1025 | "source": [ 1026 | "***\n", 1027 | "## 7c. Combine ECOSTRESS and Ground Observations \n", 1028 | "\n", 1029 | "#### Finally, compare the ECOSTRESS Evapotranspiration and uncertainty with the time series of observations from the flux tower." 1030 | ] 1031 | }, 1032 | { 1033 | "cell_type": "code", 1034 | "execution_count": null, 1035 | "metadata": {}, 1036 | "outputs": [], 1037 | "source": [ 1038 | "# Set the figure size, create a subplot\n", 1039 | "fig = plt.figure(1, figsize=(12, 7))\n", 1040 | "ax = fig.add_subplot(111)\n", 1041 | "\n", 1042 | "# Plot the flux tower observatisons followed by the ecostress median ET and median uncertainty\n", 1043 | "ax.plot(towerData['Date/Time'], towerData.LE, 'k', lw=2.5, color='black')\n", 1044 | "ax.plot(solarOverpass, ETmedian, 'bo', ms=10, color='darkgreen')\n", 1045 | "ax.errorbar(solarOverpass, ETmedian, yerr=UNmedian, lw=2.0, c='lightgreen', fmt='o', capsize=3, capthick=2)\n", 1046 | "\n", 1047 | "# Set x/y axes and labels\n", 1048 | "ax.set_xlabel(f'Time of Day ({date})', fontsize=18);\n", 1049 | "ax.set_xticks(towerData['Date/Time']);\n", 1050 | "ax.set_xticklabels(towerData.Time, rotation=45,fontsize=12);\n", 1051 | "ax.set_ylabel(\"Latent Heat Flux (Wm2)\", fontsize=16, fontweight='bold')\n", 1052 | "\n", 1053 | "# Add a title and export figure as png file\n", 1054 | "ax.set_title(f\"Time Series of Eddy Covariance Data at Site US-CZ3\\n vs. {title} ({SDSname})\", fontsize=22)\n", 1055 | "figure_filename = join(outDir, \"{}_{}_vs_fluxtower.png\".format(ecoName, SDSname))\n", 1056 | "print(\"figure filename: {}\".format(figure_filename))\n", 1057 | "fig.savefig(figure_filename, bbox_inches='tight')" 1058 | ] 1059 | }, 1060 | { 1061 | "cell_type": "markdown", 1062 | "metadata": {}, 1063 | "source": [ 1064 | "***\n", 1065 | "## Citations\n", 1066 | "- Hook, S., Fisher, J. (2019). ECOSTRESS Evapotranspiration PT-JPL Daily L3 Global 70 m V001 [Data set]. NASA EOSDIS Land Processes DAAC. Accessed 2021-03-11 from https://doi.org/10.5067/ECOSTRESS/ECO3ETPTJPL.001. \n", 1067 | "\n", 1068 | "\n", 1069 | "- Goulden, M., (2018). AmeriFlux US-CZ3 Sierra Critical Zone, Sierra Transect, Sierran Mixed Conifer, P301, [doi:10.17190/AMF/1419512](http://dx.doi.org/10.17190/AMF/1419512). \n", 1070 | "\n", 1071 | "\n", 1072 | "- Krehbiel, C., and Halverson, G.H., (2019). Working with ECOSTRESS Evapotranspiration Data [Jupyter Notebook]. Retrieved from https://git.earthdata.nasa.gov/projects/LPDUR/repos/tutorial-ecostress/browse" 1073 | ] 1074 | }, 1075 | { 1076 | "cell_type": "markdown", 1077 | "metadata": {}, 1078 | "source": [ 1079 | "## Contact Info: \n", 1080 | "Material written by Cole Krehbiel1 and Gregory Halverson2 \n", 1081 | "Email: LPDAAC@usgs.gov \n", 1082 | "Voice: +1-866-573-3222 \n", 1083 | "Organization: Land Processes Distributed Active Archive Center (LP DAAC)<\n", 1084 | "Website: \n", 1085 | "\n", 1086 | "\n", 1087 | "1Innovate! Inc., contractor to the U.S. Geological Survey, Earth Resources Observation and Science (EROS) Center, Sioux Falls, South Dakota, 57198-001, USA. Work performed under USGS contract G15PD00467 for LP DAAC3. \n", 1088 | "2Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, USA. \n", 1089 | "\n", 1090 | "3LP DAAC Work performed under NASA contract NNG14HH33I. \n", 1091 | "\n" 1092 | ] 1093 | } 1094 | ], 1095 | "metadata": { 1096 | "kernelspec": { 1097 | "display_name": "Python 3 (ipykernel)", 1098 | "language": "python", 1099 | "name": "python3" 1100 | }, 1101 | "language_info": { 1102 | "codemirror_mode": { 1103 | "name": "ipython", 1104 | "version": 3 1105 | }, 1106 | "file_extension": ".py", 1107 | "mimetype": "text/x-python", 1108 | "name": "python", 1109 | "nbconvert_exporter": "python", 1110 | "pygments_lexer": "ipython3", 1111 | "version": "3.11.13" 1112 | } 1113 | }, 1114 | "nbformat": 4, 1115 | "nbformat_minor": 4 1116 | } 1117 | -------------------------------------------------------------------------------- /python/tutorials/Work_with_ECOSTRESS_L2T_LSTE_data.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "tags": [] 7 | }, 8 | "source": [ 9 | "# Working with ECOSTRESS L2T LSTE Data \n", 10 | "\n", 11 | "**Summary** \n", 12 | "\n", 13 | "This notebook will show how to access [ECOsystem Spaceborne Thermal Radiometer Experiment on Space Station (ECOSTRESS)](https://ecostress.jpl.nasa.gov/) data programmatically using the [`earthaccess`](https://github.com/nsidc/earthaccess) python library to authenticate, search, download, and stream (directly access) data. It shows how to work with ECOSTRESS Tiled Land Surface Temperature and Emissivity ([`ECOSTRESS_L2T_LSTE`](https://doi.org/10.5067/ECOSTRESS/ECO_L2T_LSTE.002)) product hosted in the Earthdata Cloud and managed by the Land Processes Distributed Active Archive Center ([LP DAAC](https://lpdaac.usgs.gov/)). \n", 14 | "\n", 15 | "**Background** \n", 16 | "\n", 17 | "The [ECOSTRESS mission](https://ecostress.jpl.nasa.gov/) is answering these questions by accurately measuring the temperature of plants. Plants regulate their temperature by releasing water through tiny pores on their leaves called stomata. If they have sufficient water they can maintain their temperature, but if there is insufficient water, their temperatures rise and this temperature rise can be measured with ECOSTRESS. The images acquired by ECOSTRESS are the most detailed temperature images of the surface ever acquired from space and can be used to measure the temperature of an individual farm field. These temperature images, along with auxiliary inputs, are used to produce one of the primary science outputs of ECOSTRESS: evapotranspiration, an indicator of plant health via the measure of evaporation and transpiration of water through a plant.\n", 18 | "\n", 19 | "More details about ECOSTRESS and its associated products can be found on the [ECOSTRESS website](https://ecostress.jpl.nasa.gov/) and [ECOSTRESS product pages](https://lpdaac.usgs.gov/product_search/?collections=ECOSTRESS&status=Operational&view=cards&sort=title) hosted by the Land Processes Distributed Active Archive Center (LP DAAC).\n", 20 | "\n", 21 | "**Learning Objectives** \n", 22 | "\n", 23 | "- How to search ECOSTRESS data using `earthaccess`\n", 24 | "- How to stream or download ECOSTRESS data\n", 25 | "- How to clip ECOSTRESS data to a Region of Interest (ROI)\n", 26 | "- How to quality filter ECOSTRESS data\n", 27 | "- How to export the processed ECOSTRESS data\n", 28 | "\n", 29 | "**Requirements** \n", 30 | "\n", 31 | "- NASA [Earthdata Login](https://urs.earthdata.nasa.gov/) account. If you do not have an Earthdata Account, you can create one [here](https://urs.earthdata.nasa.gov/users/new). \n", 32 | "- A compatible Python Environment - See **setup_instructions.md** in the `/setup/` folder \n", 33 | "\n", 34 | "**Tutorial Outline**\n", 35 | "\n", 36 | "1. Setup\n", 37 | "2. Searching for ECOSTRESS L2T LSTE Data\n", 38 | "3. Accessing ECOSTRESS L2T LSTE Data\n", 39 | "4. Clipping ECOSTRESS L2T LSTE Data to a Region of Interest (ROI)\n", 40 | "5. Quality Filtering ECOSTRESS L2T LSTE Data\n", 41 | "6. Exporting Processed ECOSTRESS L2T LSTE Data" 42 | ] 43 | }, 44 | { 45 | "cell_type": "markdown", 46 | "metadata": {}, 47 | "source": [ 48 | "## 1. Setup \n", 49 | "\n", 50 | "Import the required libraries." 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": null, 56 | "metadata": { 57 | "tags": [] 58 | }, 59 | "outputs": [], 60 | "source": [ 61 | "# Import Packages\n", 62 | "import warnings\n", 63 | "# Some cells may generate warnings that we can ignore. Comment below lines to see.\n", 64 | "warnings.filterwarnings('ignore')\n", 65 | "\n", 66 | "import os, sys, shutil\n", 67 | "import earthaccess\n", 68 | "import numpy as np\n", 69 | "from osgeo import gdal\n", 70 | "import rasterio as rio\n", 71 | "import rioxarray as rxr\n", 72 | "import xarray as xr\n", 73 | "import hvplot.xarray\n", 74 | "import hvplot.pandas\n", 75 | "import geopandas as gp\n", 76 | "from zipfile import ZipFile " 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "### Authentication\n", 84 | "\n", 85 | "Log into Earthdata using the `Auth` and `login` functions from the `earthaccess` library. The `persist=True` argument will create a local `.netrc` file if it doesn't exist, or add your login info to an existing `.netrc` file. If no Earthdata Login credentials are found in the `.netrc` you'll be prompted for them.\n" 86 | ] 87 | }, 88 | { 89 | "cell_type": "code", 90 | "execution_count": null, 91 | "metadata": {}, 92 | "outputs": [], 93 | "source": [ 94 | "auth = earthaccess.login(persist = True)\n", 95 | "# are we authenticated?\n", 96 | "print(auth.authenticated)" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "## 2. Searching for ECOSTRESS L2T LSTE Data\n", 104 | "\n", 105 | "In this example, we will use the cloud-hosted `ECOSTRESS_L2T_LSTE` product but the searching process can be used with other EMIT or ECOSTRESS products, other collections, or different data providers, as well as across multiple catalogs with some modification. The Land Surface Temperature and Emissivity values from ECOSTRESS Level 2 Tiled Land Surface Temperature (ECO_L2T_LSTE) are derived from five thermal infrared (TIR) bands using a physics-based Temperature and Emissivity Separation (TES) algorithm. This tiled data product uses a modified version of the Military Grid Reference System (MGRS) which divides Universal Transverse Mercator (UTM) zones into square tiles that are 109.8 km by 109.8 km with a 70 meter (m) spatial resolution. \n", 106 | "\n", 107 | "### Define Your Query Parameters\n", 108 | "\n", 109 | "We can search for granules using attributes such as collection short name, collection ID, acquisition time, and spatial footprint.\n", 110 | "\n", 111 | "##### Spatial region of interest (ROI)\n", 112 | "\n", 113 | "For this example, our spatial region of interest (ROI) will be Boulder city boundary. when we are working with multi-feature ROI, we can use a bounding box with larger spatial extent including all the features. To do this, we will first open a geojson file containing our region of interest (ROI) then simplify it to a bounding box by getting the bounds and putting them into a tuple. We will use the `total_bounds` property to get the bounding box of our ROI, and add that to a python tuple, which is the expected data type for the bounding_box parameter `earthaccess` `search_data`.\n", 114 | "\n", 115 | "\n", 116 | "**Note:** If our features are spread out spatially in the ROI, we can search for data available for each feature separately to avoid accessing a large volume of data we do not need." 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": {}, 122 | "source": [ 123 | "Our ROI is stored as a .zip file. First we need to extract all the members of the zip into a specific location. " 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": null, 129 | "metadata": {}, 130 | "outputs": [], 131 | "source": [ 132 | "with ZipFile('../../data/City_of_Boulder_City_Limits.zip', 'r') as zObject: \n", 133 | " zObject.extractall( path=\"../../data\")" 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "metadata": {}, 139 | "source": [ 140 | "Below, the polygon is opened using `geopandas` library and Coordinate Reference System (crs) is printed. Our polygon is in geographic coordinate reference system (`EPSG:3857`)." 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": {}, 147 | "outputs": [], 148 | "source": [ 149 | "polygon = gp.read_file('../../data/City_of_Boulder_City_Limits.shp')\n", 150 | "polygon.crs" 151 | ] 152 | }, 153 | { 154 | "cell_type": "markdown", 155 | "metadata": {}, 156 | "source": [ 157 | "Since this data is in **EPSG:3857**, we need to reproject to **EPSG:4326** to get latitude and longitude values for our search. We can use the `to_crs` method from `geopandas` to reproject our polygon." 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": null, 163 | "metadata": {}, 164 | "outputs": [], 165 | "source": [ 166 | "polygon_4326 = polygon.to_crs(\"EPSG:4326\")" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": {}, 173 | "outputs": [], 174 | "source": [ 175 | "polygon_4326.hvplot(tiles='ESRI', color='#d95f02',alpha=0.6, crs='EPSG:4326', frame_height=405, frame_width=720, fontscale=2) " 176 | ] 177 | }, 178 | { 179 | "cell_type": "code", 180 | "execution_count": null, 181 | "metadata": {}, 182 | "outputs": [], 183 | "source": [ 184 | "bbox = tuple(list(polygon_4326.total_bounds))\n", 185 | "bbox" 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "metadata": {}, 191 | "source": [ 192 | "Below, the parameters including `provider`, `short_name`, `version`, `bounding_box`, `temporal`, and `count` are used for our query. " 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": null, 198 | "metadata": {}, 199 | "outputs": [], 200 | "source": [ 201 | "results = earthaccess.search_data(\n", 202 | " provider='LPCLOUD',\n", 203 | " short_name='ECO_L2T_LSTE',\n", 204 | " version='002',\n", 205 | " bounding_box=bbox,\n", 206 | " temporal=('2023-07-01','2023-08-01'),\n", 207 | " count=100\n", 208 | ")" 209 | ] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "metadata": {}, 214 | "source": [ 215 | "Next, get the downloadable links for LSTE and quality layers using `data_links()` method from `earthaccess`. " 216 | ] 217 | }, 218 | { 219 | "cell_type": "code", 220 | "execution_count": null, 221 | "metadata": {}, 222 | "outputs": [], 223 | "source": [ 224 | "lst_links = [l for dl in results for l in dl.data_links() if 'LST.tif' in l]\n", 225 | "lst_links" 226 | ] 227 | }, 228 | { 229 | "cell_type": "code", 230 | "execution_count": null, 231 | "metadata": {}, 232 | "outputs": [], 233 | "source": [ 234 | "qc_links = [l for dl in results for l in dl.data_links() if 'QC.tif' in l]\n", 235 | "qc_links" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "Let's take a look at the ECOSTRESS tiled data file name: \n", 243 | "\n", 244 | " Filename: **ECOv002_L2T_LSTE_28527_009_13TDE_20230718T081442_0710_01_LST.tif** \n", 245 | "\n", 246 | " ECO : Sensor \n", 247 | " v002 : Product Version \n", 248 | " L2T : Processing Level and Type (T = Tile) \n", 249 | " LSTE : Geophysical Parameter \n", 250 | " 28527 : Orbit Number \n", 251 | " 009 : Scene ID \n", 252 | " 13TDE : Military Grid Reference System (MGRS) Tile ID \n", 253 | " 20230718 : Date of Acquisition (YYYYMMDD) \n", 254 | " T081442 : Time of Acquisition (HHMMSS) (in UTC) \n", 255 | " 0710 : Build ID of software that generated product, Major+Minor (2+2 digits) \n", 256 | " 01 : Product Iteration Number \n", 257 | " LST : Layer/band Name (each layer is a separate file) \n", 258 | " .tif : Data Format for Tile \n", 259 | "\n", 260 | "\n", 261 | "Looking at Military Grid Reference System (MGRS) Tile ID of the outputs, they all are all in UTM Zone 13. " 262 | ] 263 | }, 264 | { 265 | "cell_type": "markdown", 266 | "metadata": {}, 267 | "source": [ 268 | "## 3. Accessing ECOSTRESS L2T Land Surface Temperature and Emissivity Data\n", 269 | "\n", 270 | "ECOSTRESS data is stored in NASA's Earthdata Cloud and can be accessed in different ways.\n", 271 | "\n", 272 | "- *Downloaded* - This has been available since the existance of the NASA DAAC. Users can use the data link(s) to download the data files to their local working environment. This can be done whether the user is working from a non-cloud or cloud environment.\n", 273 | "- *Streamed* - Streaming is on-the-fly random reading of remote files, i.e. files not saved locally. The data accessed, however, must be able to be held in the workspaces' memory. This can be done whether the user is working from a non-cloud or cloud environment.\n", 274 | "- *Accessed in-place (i.e., direct s3 access)* - This is only available for working environment deployed in AWS us-west-2.\n", 275 | "\n", 276 | "In this example, we will show how to stream the data. For that, the gdal configuration is set and the one of our LSTE files is read into the workspace using `open_rasterio` from the `rioxarray` library. Since the file consists of only 1 layer, we can `squeeze` it, removing the `band` dimension." 277 | ] 278 | }, 279 | { 280 | "cell_type": "code", 281 | "execution_count": null, 282 | "metadata": {}, 283 | "outputs": [], 284 | "source": [ 285 | "rio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR',\n", 286 | " GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n", 287 | " GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'),\n", 288 | " GDAL_HTTP_MAX_RETRY=10,\n", 289 | " GDAL_HTTP_RETRY_DELAY=0.5)\n", 290 | "rio_env.__enter__()" 291 | ] 292 | }, 293 | { 294 | "cell_type": "code", 295 | "execution_count": null, 296 | "metadata": { 297 | "tags": [] 298 | }, 299 | "outputs": [], 300 | "source": [ 301 | "eco_lst_ds = rxr.open_rasterio(lst_links[14]).squeeze('band', drop=True)\n", 302 | "eco_lst_ds" 303 | ] 304 | }, 305 | { 306 | "cell_type": "markdown", 307 | "metadata": {}, 308 | "source": [ 309 | "As mentioned, the ECOSTRESS product we are using here is tiled and the CRS is dependent on UTM zone. For this tile, we can look at the `spatial_ref` variable through the interactive object above to see details such as the well-known-text (WKT) representation of the CRS and other attributes. \n", 310 | "We are using `hvplot` for visualization here. For detailed information on available open-source Python tools and libraries for data visualization see .\n", 311 | "\n", 312 | "Now let's plot the data using `hvplot`. `reproject` function is applied only for the visualization. Make sure to specify the CRS argument within the `hvplot.image` function so the ESRI imagery RBG background tile aligns with our scene. \n", 313 | "\n" 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "metadata": {}, 320 | "outputs": [], 321 | "source": [ 322 | "size_opts = dict(frame_height=405, frame_width=720, fontscale=2)\n", 323 | "\n", 324 | "eco_lst_ds.rio.reproject('EPSG:4326').hvplot.image(x='x', y='y', **size_opts, \n", 325 | " cmap='inferno', tiles='ESRI', xlabel='Longitude', \n", 326 | " ylabel='Latitude', title='ECOSTRESS LST (K)', \n", 327 | " crs='EPSG:4326')*polygon_4326.hvplot(color='Green',alpha=0.5, \n", 328 | " crs='EPSG:4326', rasterize=True)" 329 | ] 330 | }, 331 | { 332 | "cell_type": "markdown", 333 | "metadata": {}, 334 | "source": [ 335 | "## 4. Cropping ECOSTRESS Data \n", 336 | "\n", 337 | "`clip` function from `rasterio` is used to mask data outside of our region of interest. Before clipping, we need to reproject our ROI to the projection of our dataset which is UTM zone 13N. " 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": null, 343 | "metadata": {}, 344 | "outputs": [], 345 | "source": [ 346 | "polygon" 347 | ] 348 | }, 349 | { 350 | "cell_type": "code", 351 | "execution_count": null, 352 | "metadata": {}, 353 | "outputs": [], 354 | "source": [ 355 | "eco_lst_ds.rio.crs" 356 | ] 357 | }, 358 | { 359 | "cell_type": "code", 360 | "execution_count": null, 361 | "metadata": {}, 362 | "outputs": [], 363 | "source": [ 364 | "polygon_reproj = polygon.to_crs(eco_lst_ds.rio.crs)" 365 | ] 366 | }, 367 | { 368 | "cell_type": "code", 369 | "execution_count": null, 370 | "metadata": {}, 371 | "outputs": [], 372 | "source": [ 373 | "polygon_reproj.crs" 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": null, 379 | "metadata": { 380 | "tags": [] 381 | }, 382 | "outputs": [], 383 | "source": [ 384 | "eco_lst_roi = eco_lst_ds.rio.clip(polygon_reproj.geometry.values, polygon_reproj.crs, all_touched=True)" 385 | ] 386 | }, 387 | { 388 | "cell_type": "code", 389 | "execution_count": null, 390 | "metadata": { 391 | "tags": [] 392 | }, 393 | "outputs": [], 394 | "source": [ 395 | "eco_lst_roi.hvplot.image(\n", 396 | " geo=True,cmap='inferno',**size_opts, tiles='ESRI',alpha=0.8, \n", 397 | " title='Cropped ECOSTRESS LST (K)', xlabel='Longitude',ylabel='Latitude', \n", 398 | " crs='EPSG:32613', rasterize=True)" 399 | ] 400 | }, 401 | { 402 | "cell_type": "markdown", 403 | "metadata": {}, 404 | "source": [ 405 | "Next, we will repeate the same process for the associated quality layer." 406 | ] 407 | }, 408 | { 409 | "cell_type": "code", 410 | "execution_count": null, 411 | "metadata": {}, 412 | "outputs": [], 413 | "source": [ 414 | "eco_qc_ds = rxr.open_rasterio(qc_links[14]).squeeze('band', drop=True)\n", 415 | "\n", 416 | "eco_qc_roi = eco_qc_ds.rio.clip(polygon_reproj.geometry.values, polygon_reproj.crs, all_touched=True) # assign a different value to fill value \n", 417 | "\n", 418 | "eco_qc_roi.hvplot.image(\n", 419 | " geo=True, cmap='inferno',**size_opts, tiles='ESRI',alpha=0.8, \n", 420 | " title='Cropped ECOSTRESS LST (K)', xlabel='Longitude',ylabel='Latitude', \n", 421 | " crs='EPSG:32613', rasterize=True)" 422 | ] 423 | }, 424 | { 425 | "cell_type": "markdown", 426 | "metadata": {}, 427 | "source": [ 428 | "## 5. Quality Filtering\n", 429 | "\n", 430 | "The quality values are 16 digits bit values with bits 0 and 1 being the mandatory QA flag that will be interpreted as:\n", 431 | "\n", 432 | " 00 = Pixel produced, best quality\n", 433 | " 01 = Pixel produced, nominal quality. Either one or more of the following conditions are met: \n", 434 | "\n", 435 | " 1. emissivity in both bands 4 and 5 < 0.95, i.e. possible cloud contamination \n", 436 | " 2. low transmissivity due to high water vapor loading (<0.4), check PWV values and error estimates \n", 437 | " 3. Pixel falls on missing scan line in bands 1&5, and filled using spatial neural net. Check error estimates \n", 438 | " Recommend more detailed analysis of other QC information \n", 439 | " 10 = Pixel produced, but cloud detected \n", 440 | " 11 = Pixel not produced due to missing/bad data, user should check Data quality flag bits \n", 441 | "\n", 442 | "The detailed quality information is provided in Table 3-5 in ECOSTRESS [Product Specification Document](https://lpdaac.usgs.gov/documents/380/ECO2_PSD_V1.pdf). \n" 443 | ] 444 | }, 445 | { 446 | "cell_type": "markdown", 447 | "metadata": {}, 448 | "source": [ 449 | "Below, the unique quality values are extracted from the clipped data, and only the values showing good quality are kept. " 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "execution_count": null, 455 | "metadata": {}, 456 | "outputs": [], 457 | "source": [ 458 | "quality_vals = np.unique(eco_qc_roi.values).tolist()\n", 459 | "good_q = [q for q in quality_vals if np.binary_repr(q, width=16)[-2:] == '00']\n", 460 | "good_q\n" 461 | ] 462 | }, 463 | { 464 | "cell_type": "markdown", 465 | "metadata": {}, 466 | "source": [ 467 | "`.where` method is used to filter the quality and keep only the LSTE values generated with the best quality. " 468 | ] 469 | }, 470 | { 471 | "cell_type": "code", 472 | "execution_count": null, 473 | "metadata": {}, 474 | "outputs": [], 475 | "source": [ 476 | "eco_lst_roi_mask = eco_lst_roi.where(eco_qc_roi.isin(good_q))\n", 477 | "eco_lst_roi_mask" 478 | ] 479 | }, 480 | { 481 | "cell_type": "code", 482 | "execution_count": null, 483 | "metadata": {}, 484 | "outputs": [], 485 | "source": [ 486 | "eco_lst_roi_mask.hvplot.image(\n", 487 | " geo=True,cmap='inferno',**size_opts, tiles='ESRI',alpha=0.9, \n", 488 | " title='Quality Masked ECOSTRESS LST (K)', xlabel='Longitude',ylabel='Latitude', \n", 489 | " crs='EPSG:32613', rasterize=True)" 490 | ] 491 | }, 492 | { 493 | "cell_type": "markdown", 494 | "metadata": {}, 495 | "source": [ 496 | "## 6. Writing Outputs \n", 497 | "\n", 498 | "We now have a ECOSTRESS scene that is clipped to our ROI with only good quality values. Finally, we can save this file locally. " 499 | ] 500 | }, 501 | { 502 | "cell_type": "code", 503 | "execution_count": null, 504 | "metadata": {}, 505 | "outputs": [], 506 | "source": [ 507 | "out_name = f\"../../data/{lst_links[14].split('/')[-1].split('.tif')[0]}_clipped.tif\"\n", 508 | "\n", 509 | "eco_lst_roi_mask.rio.to_raster(raster_path=out_name, driver='COG')" 510 | ] 511 | }, 512 | { 513 | "cell_type": "markdown", 514 | "metadata": {}, 515 | "source": [ 516 | "## Contact Info: \n", 517 | "\n", 518 | "Email: LPDAAC@usgs.gov \n", 519 | "Voice: +1-866-573-3222 \n", 520 | "Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹ \n", 521 | "Website: \n", 522 | "\n", 523 | "¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I. " 524 | ] 525 | } 526 | ], 527 | "metadata": { 528 | "kernelspec": { 529 | "display_name": "Python 3 (ipykernel)", 530 | "language": "python", 531 | "name": "python3" 532 | }, 533 | "language_info": { 534 | "codemirror_mode": { 535 | "name": "ipython", 536 | "version": 3 537 | }, 538 | "file_extension": ".py", 539 | "mimetype": "text/x-python", 540 | "name": "python", 541 | "nbconvert_exporter": "python", 542 | "pygments_lexer": "ipython3", 543 | "version": "3.11.13" 544 | } 545 | }, 546 | "nbformat": 4, 547 | "nbformat_minor": 4 548 | } 549 | -------------------------------------------------------------------------------- /setup.md: -------------------------------------------------------------------------------- 1 | ## Set Up Instruction 2 | 3 | **If you are attending the workshop:** 4 | - Go to the [Openscapes 2i2c JupyterHub](https://workshop.openscapes.2i2c.cloud/) and Log in using your email as the username and the shared password. Select **Python** and choose a session size of **14.8 GB RAM / up to 3.75 CPUs**. Most notebooks will run with this option unless otherwise noted. 5 | 6 | - Review the [prerequisites and workshop set up instructions](https://nasa.github.io/VITALS/setup/workshop_setup.html) for details. 7 | 8 | **If you are running the notebooks on your local machine:** 9 | - Follow the [instructions for setting up Python environmnet](https://github.com/nasa/LPDAAC-Data-Resources/blob/main/setup/setup_instructions_python.md) locally. 10 | 11 | 12 | ## Cloning the Repository 13 | 14 | To work with the notebooks or modules, clone the repository to your desired directory: 15 | 16 | ```cmd 17 | git clone https://github.com/nasa/ECOSTRESS-Data-Resources.git 18 | ``` 19 | 20 | If you plan to edit or contribute to the `ECOSTRESS-Data-Resources` repository, we recommend following a fork and pull workflow: first fork the repository, then clone your fork to your local machine, make changes, push changes to your fork, then make a pull request back to the main repository. An example can be found in the [CONTRIBUTING.md](../CONTRIBUTING.md) file. -------------------------------------------------------------------------------- /styles.css: -------------------------------------------------------------------------------- 1 | /* css styles */ 2 | --------------------------------------------------------------------------------