├── scripts ├── requirements.txt └── process.py ├── Makefile ├── .gitignore ├── LICENSE ├── data └── glaciers.csv ├── .github └── workflows │ └── actions.yml ├── archive └── glaciers_fig-1.csv ├── datapackage.json └── README.md /scripts/requirements.txt: -------------------------------------------------------------------------------- 1 | beautifulsoup4==4.12.3 2 | requests==2.32.3 3 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: 2 | python scripts/process.py 3 | 4 | clean: 5 | rm data/* archive/* 6 | 7 | .PHONY: clean 8 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | venv/ 13 | build/ 14 | develop-eggs/ 15 | dist/ 16 | downloads/ 17 | eggs/ 18 | .eggs/ 19 | lib/ 20 | lib64/ 21 | parts/ 22 | sdist/ 23 | var/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | .DS_Store 38 | 39 | # Unit test / coverage reports 40 | htmlcov/ 41 | .tox/ 42 | .coverage 43 | .coverage.* 44 | .cache 45 | nosetests.xml 46 | coverage.xml 47 | *,cover 48 | .hypothesis/ 49 | 50 | # Translations 51 | *.mo 52 | *.pot 53 | 54 | # Django stuff: 55 | *.log 56 | 57 | # Sphinx documentation 58 | docs/_build/ 59 | 60 | # PyBuilder 61 | target/ 62 | 63 | *~ 64 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | This is free and unencumbered software released into the public domain. 2 | 3 | Anyone is free to copy, modify, publish, use, compile, sell, or 4 | distribute this software, either in source code form or as a compiled 5 | binary, for any purpose, commercial or non-commercial, and by any 6 | means. 7 | 8 | In jurisdictions that recognize copyright laws, the author or authors 9 | of this software dedicate any and all copyright interest in the 10 | software to the public domain. We make this dedication for the benefit 11 | of the public at large and to the detriment of our heirs and 12 | successors. We intend this dedication to be an overt act of 13 | relinquishment in perpetuity of all present and future rights to this 14 | software under copyright law. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 20 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 21 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 22 | OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | For more information, please refer to 25 | -------------------------------------------------------------------------------- /data/glaciers.csv: -------------------------------------------------------------------------------- 1 | Year,Mean cumulative mass balance,Number of observations 2 | 1956,0, 3 | 1957,-0.094,12 4 | 1958,-0.963,12 5 | 1959,-1.431,13 6 | 1960,-2.008,15 7 | 1961,-2.445,16 8 | 1962,-2.648,20 9 | 1963,-3,23 10 | 1964,-2.682,23 11 | 1965,-2.524,25 12 | 1966,-2.75,28 13 | 1967,-2.869,30 14 | 1968,-2.939,33 15 | 1969,-3.428,34 16 | 1970,-3.715,35 17 | 1971,-3.946,35 18 | 1972,-4.225,35 19 | 1973,-4.402,35 20 | 1974,-4.589,35 21 | 1975,-4.815,36 22 | 1976,-4.998,38 23 | 1977,-5.255,39 24 | 1978,-5.443,39 25 | 1979,-5.86,39 26 | 1980,-5.983,40 27 | 1981,-6.174,39 28 | 1982,-6.662,41 29 | 1983,-6.535,42 30 | 1984,-6.794,45 31 | 1985,-7.102,45 32 | 1986,-7.583,47 33 | 1987,-7.487,46 34 | 1988,-7.561,47 35 | 1989,-7.789,51 36 | 1990,-8.274,54 37 | 1991,-8.777,55 38 | 1992,-8.893,60 39 | 1993,-9.026,61 40 | 1994,-9.557,60 41 | 1995,-10.016,59 42 | 1996,-10.489,59 43 | 1997,-11.13,59 44 | 1998,-11.852,59 45 | 1999,-12.551,57 46 | 2000,-12.91,57 47 | 2001,-13.181,57 48 | 2002,-13.609,57 49 | 2003,-14.134,57 50 | 2004,-14.866,55 51 | 2005,-15.683,58 52 | 2006,-16.398,58 53 | 2007,-16.938,57 54 | 2008,-17.313,58 55 | 2009,-17.765,58 56 | 2010,-18.614,58 57 | 2011,-19.351,59 58 | 2012,-20.076,60 59 | 2013,-20.788,56 60 | 2014,-21.489,56 61 | 2015,-22.293,60 62 | 2016,-23.276,60 63 | 2017,-23.928,60 64 | 2018,-24.865,60 65 | 2019,-25.859,61 66 | 2020,-26.742,61 67 | 2021,-27.419,61 68 | 2022,-28.509,60 69 | 2023,-29.738,53 70 | -------------------------------------------------------------------------------- /.github/workflows/actions.yml: -------------------------------------------------------------------------------- 1 | name: Automated Workflow for Glacier dataset 2 | 3 | on: 4 | # Schedule to run on the 1st day of each month 5 | schedule: 6 | - cron: '0 0 1 * *' 7 | 8 | # Run on push to master branch 9 | push: 10 | branches: 11 | - main 12 | 13 | # Run on pull requests targeting the master branch 14 | pull_request: 15 | branches: 16 | - main 17 | 18 | # Allows manual triggering of the workflow 19 | workflow_dispatch: 20 | 21 | jobs: 22 | build: 23 | runs-on: ubuntu-latest 24 | 25 | # Conditional job trigger based on the branch being present 26 | if: github.ref == 'refs/heads/main' 27 | 28 | steps: 29 | - name: Check out repository 30 | uses: actions/checkout@v3 31 | 32 | - name: Set up Python 3.12 33 | uses: actions/setup-python@v4 34 | with: 35 | python-version: '3.12' 36 | 37 | - name: Install Python dependencies 38 | run: | 39 | python -m venv venv 40 | source venv/bin/activate 41 | pip install --upgrade pip 42 | pip install -r scripts/requirements.txt 43 | 44 | - name: Run Makefile 45 | run: | 46 | source venv/bin/activate 47 | make 48 | 49 | - name: Push and Commit 50 | env: 51 | CI_COMMIT_NAME: "Automated commit" 52 | CI_COMMIT_EMAIL: "actions@users.noreply.github.com" 53 | CI_COMMIT_MESSAGE: "Automated commit" 54 | run: | 55 | git config --global user.email "${{env.CI_COMMIT_EMAIL}}" 56 | git config --global user.name "${{env.CI_COMMIT_NAME}}" 57 | git diff --quiet && echo "No changes to commit" || (git add data/world-cities.csv && git commit -m "${{env.CI_COMMIT_MESSAGE}}" && git push) -------------------------------------------------------------------------------- /archive/glaciers_fig-1.csv: -------------------------------------------------------------------------------- 1 | "Figure 1. Average Cumulative Mass Balance of ""Reference"" Glaciers Worldwide, 1956-2023",, 2 | Source: EPA's Climate Change Indicators in the United States: www.epa.gov/climate-indicators,, 3 | "Data source: WGMS, 2024",, 4 | Web update: June 2024,, 5 | Units: m of water equivalent; count,, 6 | ,, 7 | Year,Mean cumulative mass balance,Number of observations 8 | 1956,0, 9 | 1957,-0.094,12 10 | 1958,-0.963,12 11 | 1959,-1.431,13 12 | 1960,-2.008,15 13 | 1961,-2.445,16 14 | 1962,-2.648,20 15 | 1963,-3,23 16 | 1964,-2.682,23 17 | 1965,-2.524,25 18 | 1966,-2.75,28 19 | 1967,-2.869,30 20 | 1968,-2.939,33 21 | 1969,-3.428,34 22 | 1970,-3.715,35 23 | 1971,-3.946,35 24 | 1972,-4.225,35 25 | 1973,-4.402,35 26 | 1974,-4.589,35 27 | 1975,-4.815,36 28 | 1976,-4.998,38 29 | 1977,-5.255,39 30 | 1978,-5.443,39 31 | 1979,-5.86,39 32 | 1980,-5.983,40 33 | 1981,-6.174,39 34 | 1982,-6.662,41 35 | 1983,-6.535,42 36 | 1984,-6.794,45 37 | 1985,-7.102,45 38 | 1986,-7.583,47 39 | 1987,-7.487,46 40 | 1988,-7.561,47 41 | 1989,-7.789,51 42 | 1990,-8.274,54 43 | 1991,-8.777,55 44 | 1992,-8.893,60 45 | 1993,-9.026,61 46 | 1994,-9.557,60 47 | 1995,-10.016,59 48 | 1996,-10.489,59 49 | 1997,-11.13,59 50 | 1998,-11.852,59 51 | 1999,-12.551,57 52 | 2000,-12.91,57 53 | 2001,-13.181,57 54 | 2002,-13.609,57 55 | 2003,-14.134,57 56 | 2004,-14.866,55 57 | 2005,-15.683,58 58 | 2006,-16.398,58 59 | 2007,-16.938,57 60 | 2008,-17.313,58 61 | 2009,-17.765,58 62 | 2010,-18.614,58 63 | 2011,-19.351,59 64 | 2012,-20.076,60 65 | 2013,-20.788,56 66 | 2014,-21.489,56 67 | 2015,-22.293,60 68 | 2016,-23.276,60 69 | 2017,-23.928,60 70 | 2018,-24.865,60 71 | 2019,-25.859,61 72 | 2020,-26.742,61 73 | 2021,-27.419,61 74 | 2022,-28.509,60 75 | 2023,-29.738,53 76 | -------------------------------------------------------------------------------- /scripts/process.py: -------------------------------------------------------------------------------- 1 | import re 2 | import csv 3 | import requests 4 | 5 | from bs4 import BeautifulSoup 6 | 7 | source = 'https://www.epa.gov/climate-indicators/climate-change-indicators-glaciers/' 8 | archive = 'archive/glaciers_fig-1.csv' 9 | data = 'data/glaciers.csv' 10 | 11 | def get_glaciers_data(): 12 | response = requests.get(source) 13 | html = response.content 14 | soup = BeautifulSoup(html, 'html.parser') 15 | a = soup.find_all('a', href=True) 16 | glacier_url = '' 17 | for elem in a: 18 | if 'glacier' in elem['href'] and '.csv' in elem['href']: 19 | glacier_url = elem['href'] 20 | break 21 | print('Glacier URL:', glacier_url) 22 | if glacier_url == '': 23 | print('No glacier data found') 24 | return 25 | return glacier_url 26 | 27 | def execute(): 28 | # Step 1: Get Glacier URL data 29 | print('Getting glacier data...') 30 | glacier_url = get_glaciers_data() 31 | print('Glacier data found at:', glacier_url) 32 | # Step 2: Read the data from the URL and produce the CSV files 33 | print('Processing data...') 34 | with requests.Session() as s: 35 | download = s.get('https://www.epa.gov' + glacier_url) 36 | decoded_content = download.content.decode('utf-8') 37 | cr = csv.reader(decoded_content.splitlines(), delimiter=',') 38 | my_list = list(cr) 39 | updated_list = my_list[7:] 40 | header = ['Year', 'Mean cumulative mass balance', 'Number of observations'] 41 | with open(data, 'w', newline='') as f, open(archive, 'w', newline='') as f2: 42 | writer = csv.writer(f) 43 | writer.writerow(header) 44 | writer.writerows(updated_list) 45 | 46 | writer2 = csv.writer(f2) 47 | writer2.writerows(my_list) 48 | print('Data processed successfully') 49 | if __name__ == '__main__': 50 | execute() 51 | -------------------------------------------------------------------------------- /datapackage.json: -------------------------------------------------------------------------------- 1 | { 2 | "description": "This is cumulative change in mass balance of a set of reference glaciers worldwide beginning in 1945. The values represents the average of all the glaciers that were measured. Negative values indicate a net loss of ice and snow compared with the base year of 1945. For consistency, measurements are in meters of water equivalent, which represent changes in the average thickness of a glacier.", 3 | "homepage": "http://www3.epa.gov/climatechange/science/indicators/snow-ice/glaciers.html", 4 | "licenses": [ 5 | { 6 | "name": "ODC-PDDL-1.0", 7 | "path": "http://opendatacommons.org/licenses/pddl/", 8 | "title": "Open Data Commons Public Domain Dedication and License v1.0" 9 | } 10 | ], 11 | "name": "glacier-mass-balance", 12 | "resources": [ 13 | { 14 | "format": "csv", 15 | "mediatype": "text/csv", 16 | "name": "glaciers", 17 | "path": "data/glaciers.csv", 18 | "schema": { 19 | "fields": [ 20 | { 21 | "description": "Year of measurement", 22 | "name": "Year", 23 | "type": "year" 24 | }, 25 | { 26 | "description": "Average mass of measured glacier", 27 | "name": "Mean cumulative mass balance", 28 | "type": "number" 29 | }, 30 | { 31 | "description": "Number of glaciers observed", 32 | "name": "Number of observations", 33 | "type": "number" 34 | } 35 | ] 36 | } 37 | } 38 | ], 39 | "sources": [ 40 | { 41 | "name": "Average cumulative mass balance of reference Glaciers worldwide", 42 | "path": "http://www3.epa.gov/climatechange/images/indicator_downloads/glaciers_fig-1.csv", 43 | "title": "Average cumulative mass balance of reference Glaciers worldwide" 44 | } 45 | ], 46 | "title": "Average cumulative mass balance of reference Glaciers worldwide", 47 | "views": [ 48 | { 49 | "name": "graph", 50 | "title": "Average mass of measured glacier", 51 | "resourceName": "glaciers", 52 | "specType": "simple", 53 | "spec": { 54 | "type": "lines-and-points", 55 | "group": "Year", 56 | "series": [ 57 | "Mean cumulative mass balance" 58 | ] 59 | } 60 | } 61 | ], 62 | "collection": "climate-change" 63 | } -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | badge 2 | 3 | Average cumulative mass balance of "reference" Glaciers worldwide from 1945-2014 sourced from [US EPA][datahome] and the [World Glacier Monitoring Service (WGMS)][wgms]. 4 | This is cumulative change in mass balance of a set of "reference" glaciers worldwide beginning in 1945. The values represents the average of all the glaciers that were measured. Negative values indicate a net loss of ice and snow compared with the base year of 1945. For consistency, measurements are in meters of water equivalent, which represent changes in the average thickness of a glacier. 5 | 6 | [datahome]: http://www3.epa.gov/climatechange/science/indicators/snow-ice/glaciers.html 7 | [wgms]: http://wgms.ch/data_databaseversions/ 8 | 9 | ## Data 10 | 11 | ### Sources 12 | 13 | - Climate Change Indicators in the United States: https://www.epa.gov/climate-indicators/climate-change-indicators-glaciers 14 | - World Glacier Monitoring Service (WGMS): http://wgms.ch/downloads/wgms_2013_gmbb12.pdf 15 | 16 | ### Related publications: 17 | 18 | WGMS (2015): Global Glacier Change Bulletin No. 1 (2012-2013). Zemp, M., Gärtner-Roer, I., Nussbaumer, S.U., Hüsler, F., Machguth, H., Mölg, N., Paul, F., and Hoelzle, M. (eds.), ICSU(WDS)/IUGG(IACS)/UNEP/UNESCO/WMO, World Glacier Monitoring Service, Zurich, Switzerland, 230 pp. Based on database version: doi: 10.5904/wgms-fog-2015-11. 19 | WGMS (2013): Glacier Mass Balance Bulletin No. 12 (2010-2011). Zemp, M., Nussbaumer, S.U., Naegeli, K., Gärtner-Roer, I., Paul, F., Hoelzle, M. and Haeberli, W. (eds.), ICSU (WDS) / IUGG (IACS) / UNEP / UNESCO / WMO, World Glacier Monitoring Service, Zurich, Switzerland: 106 pp., publication based on database version:doi:10.5904/wgms-fog-2013-11. 20 | 21 | WGMS (2012): Fluctuations of Glaciers 2005-2010 (Vol. X): Zemp, M., Frey, H., Gärtner-Roer, I., Nussbaumer, S.U., Hoelzle, M., Paul, F. & W. Haeberli (eds.), ICSU (WDS)/ IUGG (IACS)/ UNEP/ UNESCO/ WMO, World Glacier Monitoring Service, Zurich, Switzerland. Based on database version doi: 10.5904/wgms-fog-2012-11. 22 | 23 | WGMS (World Glacier Monitoring Service). 2015 update to data originally published in: WGMS. 2013. Glacier mass balance bulletin no. 12 (2010–2011). Zemp, M., S.U. Nussbaumer, K. Naegeli, I. Gärtner-Roer, F. Paul, M. Hoelzle, and W. Haeberli (eds.). ICSU (WDS)/IUGG (IACS)/UNEP/UNESCO/WMO. Zurich, Switzerland: World Glacier Monitoring Service. http://wgms.ch/downloads/wgms_2013_gmbb12.pdf. WGMS World Glacier Monitoring Service, Zurich, Switzerland 24 | 25 | ## License 26 | 27 | ### Data 28 | 29 | EPA is Federal Government so public domain we would assume. 30 | 31 | WGMS make their data available as "Open access for scientific and educational purposes under requirement of correct citation": 32 | 33 | > WGMS (2015): Fluctuations of Glaciers Database. World Glacier Monitoring Service, Zurich, Switzerland. DOI:10.5904/wgms-fog-2015-11. 34 | 35 | ### Additional work 36 | 37 | > All the additional work made to build this Data Package is made available under the Public Domain Dedication and License v1.0 whose full text can be found at: http://www.opendatacommons.org/licenses/pddl/1.0/ 38 | 39 | ## Citations 40 | 41 | > WGMS (World Glacier Monitoring Service). 2015 update to data originally published in: WGMS. 2013. Glacier mass balance bulletin no. 12 (2010–2011). Zemp, M., S.U. Nussbaumer, K. Naegeli, I. Gärtner-Roer, F. Paul, M. Hoelzle, and W. Haeberli (eds.). ICSU (WDS)/IUGG (IACS)/UNEP/UNESCO/WMO. Zurich, Switzerland: World Glacier Monitoring Service. http://wgms.ch/downloads/wgms_2013_gmbb12.pdf. 42 | 43 | ## Useful info 44 | 45 | There does seem to be more recent data because there are graphs with more recent e.g. [graphs like this on Climate.gov](https://www.climate.gov/news-features/understanding-climate/2012-state-climate-glaciers), or [PDF data like this from World Glacier monitoring service](http://wgms.ch/products_fog/) and this [PDF from WGMS](http://wgms.ch/downloads/wgms_2012_fogX.pdf). 46 | 47 | * Data from World Glacier Monitoring Service (WGMS): http://wgms.ch/data_databaseversions/ 48 | Here is zip file with datasets: http://wgms.ch/downloads/DOI-WGMS-FoG-2015-11.zip 49 | Internationally collected, standardized dataset on changes in glaciers (length, area, volume, mass), based on in-situ and remotely sensed observations, as well as on reconstructions. 50 | * [GLIMS Glacier database](http://www.glims.org). From the site: "GLIMS (Global Land Ice Measurements from Space) is a project designed to monitor the world's glaciers primarily using data from optical satellite instruments, such as ASTER (Advanced Spaceborne Thermal Emission and reflection Radiometer)." Data can be found at http://glims.colorado.edu/glacierdata/ which links to http://www.glims.org/download/. Data is large tgz files - probably containing geodata on individual glacier outlines. Stats on data contents at http://glims.colorado.edu/glacierdata/db_summary_stats.php - these show coverage from 1870 to 2011 and thousands of glaciers worldwide. 51 | * Relationship to WGMS World Glacier Inventory: "The GLIMS database is designed to be a logical extension of the World Glacier Inventory (WGI) of the World Glacier Monitoring Service (WGMS) and stores the full complement of the WGMS-defined glacier characteristics." [source](https://nsidc.org/glims/) 52 | * The US EPA also have this [Graph: Average cumulative mass balance of "reference" Glaciers worldwide from 1945-2014](http://www3.epa.gov/climatechange/science/indicators/snow-ice/glaciers.html). You can get the CSV Data for the graph here: http://www3.epa.gov/climatechange/images/indicator_downloads/glaciers_fig-1.csv. According to the footnote for the graph the original source is: 53 | 54 | > WGMS (World Glacier Monitoring Service). 2015 update to data originally published in: WGMS. 2013. Glacier mass balance bulletin no. 12 (2010–2011). Zemp, M., S.U. Nussbaumer, K. Naegeli, I. Gärtner-Roer, F. Paul, M. Hoelzle, and W. Haeberli (eds.). ICSU (WDS)/IUGG (IACS)/UNEP/UNESCO/WMO. Zurich, Switzerland: World Glacier Monitoring Service. http://wgms.ch/downloads/wgms_2013_gmbb12.pdf. 55 | * **NOAA** Glacier Mass Balance and Regime Measurements and Analysis, 1945-2003, Version 1 56 | [Data can be found here](http://nsidc.org/data/g10002) but I'm not sure if this link will lead you directly, it requires registration, but here is the link for datasets: ftp://sidads.colorado.edu/pub/DATASETS/NOAA/G10002/Supplement2005/ 57 | --------------------------------------------------------------------------------