├── code └── .gitkeep ├── AUTHORS ├── episodes ├── fig │ ├── .gitkeep │ ├── 23_SCI.png │ ├── 22_scope.png │ ├── 19_lifespan.png │ ├── 05_power_usage.png │ ├── 09_marginal_CI.png │ ├── 10_marginal_CI.png │ ├── 11_marginal_CI.png │ ├── 12_marginal_CI.png │ ├── 13_carbon_aware.png │ ├── 18_amortization.png │ ├── 30_hourly_match.png │ ├── github-io-pages.png │ ├── 07_variability_CI.png │ ├── 08_variability_CI.png │ ├── 16_demand_shaping.png │ ├── 28_daily_vs_hourly.png │ ├── github-main-branch.png │ ├── 01_carbon_efficiency.png │ ├── 03_energy_efficiency.png │ ├── 14_spatial_shifting.png │ ├── 15_temporal_shifting.png │ ├── 17_embodioed_carbon.png │ ├── 24_carbon_reduction.png │ ├── 26-27_100_renewable.png │ ├── 29_daily_consumption.png │ ├── 31_carbon_awareness.png │ ├── 04_high-carbon_sources.png │ ├── 25_climate_commitments.png │ ├── github-gh-pages-branch.png │ ├── 06_energy_proportionality.png │ ├── 20_increasing_utilization.png │ ├── 21_private_vs_public_cloud.png │ ├── containers-cookie-cutter.png │ ├── 02_monitoring_climate_change.png │ ├── 06_energy proportionality_updated.png │ ├── 06_energy_proportionality_updated.png │ ├── GSF_Principles_Patterns_Practices.png │ └── GSF_Principles_Patterns_Practices_v2.png ├── lunch-break.md ├── morning-break.md ├── afternoon-break.md ├── next-steps.md ├── introduction.md ├── carbon-efficiency.md ├── renewable-energy.md ├── hardware-efficiency.md └── carbon-awareness.md ├── .github ├── workflows │ ├── sandpaper-version.txt │ ├── pr-close-signal.yaml │ ├── pr-post-remove-branch.yaml │ ├── pr-preflight.yaml │ ├── sandpaper-main.yaml │ ├── workbench-beta-phase.yml │ ├── update-workflows.yaml │ ├── pr-receive.yaml │ ├── update-cache.yaml │ ├── pr-comment.yaml │ └── README.md ├── PULL_REQUEST_TEMPLATE.md └── ISSUE_TEMPLATE.md ├── CITATION ├── bin ├── boilerplate │ ├── CITATION │ ├── AUTHORS │ ├── setup.md │ ├── _extras │ │ ├── discuss.md │ │ ├── guide.md │ │ ├── about.md │ │ └── figures.md │ ├── reference.md │ ├── _episodes │ │ └── 01-introduction.md │ ├── index.md │ ├── .travis.yml │ ├── README.md │ ├── _config.yml │ └── CONTRIBUTING.md ├── knit_lessons.sh ├── markdown_ast.rb ├── test_lesson_check.py ├── lesson_initialize.py ├── generate_md_episodes.R ├── chunk-options.R ├── repo_check.py ├── util.py └── workshop_check.py ├── Gemfile ├── learners ├── discuss.md ├── reference.md └── setup.md ├── .gitpod.yml ├── instructors ├── about.md ├── L1-Introduction.pptx ├── L2-carbon-efficiency.pptx ├── L3-energy-efficiency.pptx ├── L4-carbon-awareness.pptx ├── L7-measurement-hpcci.pptx ├── L8-reducing-emissions.pptx ├── L5-hardware-efficiency.pptx ├── L6-measurement-estimating-emissions.pptx └── instructor-notes.md ├── site └── README.md ├── aio.md ├── links.md ├── .gitpod.Dockerfile ├── .editorconfig ├── CODE_OF_CONDUCT.md ├── .gitignore ├── profiles └── learner-profiles.md ├── config.yaml ├── README.md ├── LICENSE.md ├── index.md ├── Makefile └── CONTRIBUTING.md /code/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /AUTHORS: -------------------------------------------------------------------------------- 1 | Andy Turner 2 | -------------------------------------------------------------------------------- /episodes/fig/.gitkeep: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.github/workflows/sandpaper-version.txt: -------------------------------------------------------------------------------- 1 | 0.17.1 2 | -------------------------------------------------------------------------------- /CITATION: -------------------------------------------------------------------------------- 1 | A. Turner, "Green software use on HPC" 2 | -------------------------------------------------------------------------------- /bin/boilerplate/CITATION: -------------------------------------------------------------------------------- 1 | FIXME: describe how to cite this lesson. -------------------------------------------------------------------------------- /bin/boilerplate/AUTHORS: -------------------------------------------------------------------------------- 1 | FIXME: list authors' names and email addresses. -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | gem 'github-pages', group: :jekyll_plugins 3 | -------------------------------------------------------------------------------- /learners/discuss.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Discussion 3 | --- 4 | 5 | FIXME 6 | 7 | 8 | 9 | 10 | -------------------------------------------------------------------------------- /.gitpod.yml: -------------------------------------------------------------------------------- 1 | image: 2 | file: .gitpod.Dockerfile 3 | 4 | tasks: 5 | - init: make && bundle install 6 | -------------------------------------------------------------------------------- /bin/boilerplate/setup.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Setup 3 | --- 4 | FIXME 5 | 6 | 7 | {% include links.md %} 8 | -------------------------------------------------------------------------------- /instructors/about.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: About 3 | --- 4 | 5 | {% include carpentries.html %} 6 | 7 | 8 | 9 | -------------------------------------------------------------------------------- /site/README.md: -------------------------------------------------------------------------------- 1 | This directory contains rendered lesson materials. Please do not edit files 2 | here. 3 | -------------------------------------------------------------------------------- /episodes/fig/23_SCI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/23_SCI.png -------------------------------------------------------------------------------- /bin/boilerplate/_extras/discuss.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Discussion 3 | --- 4 | FIXME 5 | 6 | {% include links.md %} 7 | -------------------------------------------------------------------------------- /episodes/fig/22_scope.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/22_scope.png -------------------------------------------------------------------------------- /bin/boilerplate/_extras/guide.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Instructor Notes" 3 | --- 4 | FIXME 5 | 6 | {% include links.md %} 7 | -------------------------------------------------------------------------------- /episodes/fig/19_lifespan.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/19_lifespan.png -------------------------------------------------------------------------------- /aio.md: -------------------------------------------------------------------------------- 1 | --- 2 | permalink: /aio/index.html 3 | --- 4 | 5 | {% include base_path.html %} 6 | 7 | {% include aio-script.md %} 8 | -------------------------------------------------------------------------------- /bin/boilerplate/_extras/about.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: About 3 | --- 4 | {% include carpentries.html %} 5 | {% include links.md %} 6 | -------------------------------------------------------------------------------- /episodes/fig/05_power_usage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/05_power_usage.png -------------------------------------------------------------------------------- /episodes/fig/09_marginal_CI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/09_marginal_CI.png -------------------------------------------------------------------------------- /episodes/fig/10_marginal_CI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/10_marginal_CI.png -------------------------------------------------------------------------------- /episodes/fig/11_marginal_CI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/11_marginal_CI.png -------------------------------------------------------------------------------- /episodes/fig/12_marginal_CI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/12_marginal_CI.png -------------------------------------------------------------------------------- /episodes/fig/13_carbon_aware.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/13_carbon_aware.png -------------------------------------------------------------------------------- /episodes/fig/18_amortization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/18_amortization.png -------------------------------------------------------------------------------- /episodes/fig/30_hourly_match.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/30_hourly_match.png -------------------------------------------------------------------------------- /episodes/fig/github-io-pages.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/github-io-pages.png -------------------------------------------------------------------------------- /instructors/L1-Introduction.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/instructors/L1-Introduction.pptx -------------------------------------------------------------------------------- /episodes/fig/07_variability_CI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/07_variability_CI.png -------------------------------------------------------------------------------- /episodes/fig/08_variability_CI.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/08_variability_CI.png -------------------------------------------------------------------------------- /episodes/fig/16_demand_shaping.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/16_demand_shaping.png -------------------------------------------------------------------------------- /episodes/fig/28_daily_vs_hourly.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/28_daily_vs_hourly.png -------------------------------------------------------------------------------- /episodes/fig/github-main-branch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/github-main-branch.png -------------------------------------------------------------------------------- /episodes/lunch-break.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Lunch Break 3 | teaching: 45 4 | exercises: 0 5 | --- 6 | 7 | Lunch break 8 | 9 | 10 | -------------------------------------------------------------------------------- /episodes/fig/01_carbon_efficiency.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/01_carbon_efficiency.png -------------------------------------------------------------------------------- /episodes/fig/03_energy_efficiency.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/03_energy_efficiency.png -------------------------------------------------------------------------------- /episodes/fig/14_spatial_shifting.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/14_spatial_shifting.png -------------------------------------------------------------------------------- /episodes/fig/15_temporal_shifting.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/15_temporal_shifting.png -------------------------------------------------------------------------------- /episodes/fig/17_embodioed_carbon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/17_embodioed_carbon.png -------------------------------------------------------------------------------- /episodes/fig/24_carbon_reduction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/24_carbon_reduction.png -------------------------------------------------------------------------------- /episodes/fig/26-27_100_renewable.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/26-27_100_renewable.png -------------------------------------------------------------------------------- /episodes/fig/29_daily_consumption.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/29_daily_consumption.png -------------------------------------------------------------------------------- /episodes/fig/31_carbon_awareness.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/31_carbon_awareness.png -------------------------------------------------------------------------------- /instructors/L2-carbon-efficiency.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/instructors/L2-carbon-efficiency.pptx -------------------------------------------------------------------------------- /instructors/L3-energy-efficiency.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/instructors/L3-energy-efficiency.pptx -------------------------------------------------------------------------------- /instructors/L4-carbon-awareness.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/instructors/L4-carbon-awareness.pptx -------------------------------------------------------------------------------- /instructors/L7-measurement-hpcci.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/instructors/L7-measurement-hpcci.pptx -------------------------------------------------------------------------------- /instructors/L8-reducing-emissions.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/instructors/L8-reducing-emissions.pptx -------------------------------------------------------------------------------- /bin/boilerplate/reference.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: reference 3 | --- 4 | 5 | ## Glossary 6 | 7 | FIXME 8 | 9 | {% include links.md %} 10 | -------------------------------------------------------------------------------- /episodes/fig/04_high-carbon_sources.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/04_high-carbon_sources.png -------------------------------------------------------------------------------- /episodes/fig/25_climate_commitments.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/25_climate_commitments.png -------------------------------------------------------------------------------- /episodes/fig/github-gh-pages-branch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/github-gh-pages-branch.png -------------------------------------------------------------------------------- /episodes/morning-break.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Morning Break 3 | teaching: 15 4 | exercises: 0 5 | --- 6 | 7 | Refreshments break 8 | 9 | 10 | -------------------------------------------------------------------------------- /instructors/L5-hardware-efficiency.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/instructors/L5-hardware-efficiency.pptx -------------------------------------------------------------------------------- /episodes/afternoon-break.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Afternoon Break 3 | teaching: 15 4 | exercises: 0 5 | --- 6 | 7 | Refreshments break 8 | 9 | 10 | -------------------------------------------------------------------------------- /episodes/fig/06_energy_proportionality.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/06_energy_proportionality.png -------------------------------------------------------------------------------- /episodes/fig/20_increasing_utilization.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/20_increasing_utilization.png -------------------------------------------------------------------------------- /episodes/fig/21_private_vs_public_cloud.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/21_private_vs_public_cloud.png -------------------------------------------------------------------------------- /episodes/fig/containers-cookie-cutter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/containers-cookie-cutter.png -------------------------------------------------------------------------------- /episodes/fig/02_monitoring_climate_change.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/02_monitoring_climate_change.png -------------------------------------------------------------------------------- /episodes/fig/06_energy proportionality_updated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/06_energy proportionality_updated.png -------------------------------------------------------------------------------- /episodes/fig/06_energy_proportionality_updated.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/06_energy_proportionality_updated.png -------------------------------------------------------------------------------- /episodes/fig/GSF_Principles_Patterns_Practices.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/GSF_Principles_Patterns_Practices.png -------------------------------------------------------------------------------- /episodes/fig/GSF_Principles_Patterns_Practices_v2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/episodes/fig/GSF_Principles_Patterns_Practices_v2.png -------------------------------------------------------------------------------- /instructors/L6-measurement-estimating-emissions.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/EPCCed/2025-10-22_GreenHPC_Bham/main/instructors/L6-measurement-estimating-emissions.pptx -------------------------------------------------------------------------------- /links.md: -------------------------------------------------------------------------------- 1 | [Green software on HPC repository]: https://github.com/EPCCed/green-software-archer2 2 | [open a lesson issue]: https://github.com/EPCCed/green-software-archer2/issues/new -------------------------------------------------------------------------------- /.gitpod.Dockerfile: -------------------------------------------------------------------------------- 1 | FROM gitpod/workspace-full 2 | 3 | # Install custom tools, runtimes, etc. 4 | # For example "bastet", a command-line tetris clone: 5 | # RUN brew install bastet 6 | # 7 | # More information: https://www.gitpod.io/docs/config-docker/ 8 | -------------------------------------------------------------------------------- /bin/knit_lessons.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Only try running R to translate files if there are some files present. 4 | # The Makefile passes in the names of files. 5 | 6 | if [ $# -eq 2 ] ; then 7 | Rscript -e "source('bin/generate_md_episodes.R')" "$@" 8 | fi 9 | -------------------------------------------------------------------------------- /bin/markdown_ast.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | # Use Kramdown parser to produce AST for Markdown document. 4 | 5 | require "kramdown" 6 | require "json" 7 | 8 | markdown = STDIN.read() 9 | doc = Kramdown::Document.new(markdown) 10 | tree = doc.to_hash_a_s_t 11 | puts JSON.pretty_generate(tree) 12 | -------------------------------------------------------------------------------- /bin/boilerplate/_episodes/01-introduction.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Introduction" 3 | teaching: 0 4 | exercises: 0 5 | questions: 6 | - "Key question (FIXME)" 7 | objectives: 8 | - "First learning objective. (FIXME)" 9 | keypoints: 10 | - "First key point. Brief Answer to questions. (FIXME)" 11 | --- 12 | FIXME 13 | 14 | {% include links.md %} 15 | 16 | -------------------------------------------------------------------------------- /bin/boilerplate/index.md: -------------------------------------------------------------------------------- 1 | --- 2 | layout: lesson 3 | root: . # Is the only page that doesn't follow the pattern /:path/index.html 4 | permalink: index.html # Is the only page that doesn't follow the pattern /:path/index.html 5 | --- 6 | FIXME: home page introduction 7 | 8 | 9 | 10 | > ## Prerequisites 11 | > 12 | > FIXME 13 | {: .prereq} 14 | 15 | {% include links.md %} 16 | -------------------------------------------------------------------------------- /.editorconfig: -------------------------------------------------------------------------------- 1 | root = true 2 | 3 | [*] 4 | charset = utf-8 5 | insert_final_newline = true 6 | trim_trailing_whitespace = true 7 | 8 | [*.md] 9 | indent_size = 2 10 | indent_style = space 11 | max_line_length = 100 # Please keep this in sync with bin/lesson_check.py! 12 | 13 | [*.r] 14 | max_line_length = 80 15 | 16 | [*.py] 17 | indent_size = 4 18 | indent_style = space 19 | max_line_length = 79 20 | 21 | [*.sh] 22 | end_of_line = lf 23 | 24 | [Makefile] 25 | indent_style = tab 26 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Contributor Code of Conduct" 3 | --- 4 | 5 | As contributors and maintainers of this project, 6 | we pledge to follow the [The Carpentries Code of Conduct][coc]. 7 | 8 | Instances of abusive, harassing, or otherwise unacceptable behavior 9 | may be reported by following our [reporting guidelines][coc-reporting]. 10 | 11 | 12 | [coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html 13 | [coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html 14 | -------------------------------------------------------------------------------- /.github/workflows/pr-close-signal.yaml: -------------------------------------------------------------------------------- 1 | name: "Bot: Send Close Pull Request Signal" 2 | 3 | on: 4 | pull_request: 5 | types: 6 | [closed] 7 | 8 | jobs: 9 | send-close-signal: 10 | name: "Send closing signal" 11 | runs-on: ubuntu-22.04 12 | if: ${{ github.event.action == 'closed' }} 13 | steps: 14 | - name: "Create PRtifact" 15 | run: | 16 | mkdir -p ./pr 17 | printf ${{ github.event.number }} > ./pr/NUM 18 | - name: Upload Diff 19 | uses: actions/upload-artifact@v4 20 | with: 21 | name: pr 22 | path: ./pr 23 | -------------------------------------------------------------------------------- /bin/test_lesson_check.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import unittest 4 | 5 | import lesson_check 6 | import util 7 | 8 | 9 | class TestFileList(unittest.TestCase): 10 | def setUp(self): 11 | self.reporter = util.Reporter() # TODO: refactor reporter class. 12 | 13 | def test_file_list_has_expected_entries(self): 14 | # For first pass, simply assume that all required files are present 15 | all_filenames = [filename.replace('%', '') 16 | for filename in lesson_check.REQUIRED_FILES] 17 | 18 | lesson_check.check_fileset('', self.reporter, all_filenames) 19 | self.assertEqual(len(self.reporter.messages), 0) 20 | 21 | 22 | if __name__ == "__main__": 23 | unittest.main() 24 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | Please delete this line and the text below before submitting your contribution. 2 | 3 | --- 4 | 5 | Thanks for contributing! If this contribution is for instructor training, please send an email to checkout@carpentries.org with a link to this contribution so we can record your progress. You’ve completed your contribution step for instructor checkout just by submitting this contribution. 6 | 7 | Please keep in mind that lesson maintainers are volunteers and it may be some time before they can respond to your contribution. Although not all contributions can be incorporated into the lesson materials, we appreciate your time and effort to improve the curriculum. If you have any questions about the lesson maintenance process or would like to volunteer your time as a contribution reviewer, please contact The Carpentries Team at team@carpentries.org. 8 | 9 | --- 10 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | Please delete this line and the text below before submitting your contribution. 2 | 3 | --- 4 | 5 | Thanks for contributing! If this contribution is for instructor training, please send an email to checkout@carpentries.org with a link to this contribution so we can record your progress. You’ve completed your contribution step for instructor checkout just by submitting this contribution. 6 | 7 | If this issue is about a specific episode within a lesson, please provide its link or filename. 8 | 9 | Please keep in mind that lesson maintainers are volunteers and it may be some time before they can respond to your contribution. Although not all contributions can be incorporated into the lesson materials, we appreciate your time and effort to improve the curriculum. If you have any questions about the lesson maintenance process or would like to volunteer your time as a contribution reviewer, please contact The Carpentries Team at team@carpentries.org. 10 | 11 | --- 12 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # sandpaper files 2 | episodes/*html 3 | site/* 4 | !site/README.md 5 | 6 | # History files 7 | .Rhistory 8 | .Rapp.history 9 | # Session Data files 10 | .RData 11 | # User-specific files 12 | .Ruserdata 13 | # Example code in package build process 14 | *-Ex.R 15 | # Output files from R CMD build 16 | /*.tar.gz 17 | # Output files from R CMD check 18 | /*.Rcheck/ 19 | # RStudio files 20 | .Rproj.user/ 21 | # produced vignettes 22 | vignettes/*.html 23 | vignettes/*.pdf 24 | # OAuth2 token, see https://github.com/hadley/httr/releases/tag/v0.3 25 | .httr-oauth 26 | # knitr and R markdown default cache directories 27 | *_cache/ 28 | /cache/ 29 | # Temporary files created by R markdown 30 | *.utf8.md 31 | *.knit.md 32 | # R Environment Variables 33 | .Renviron 34 | # pkgdown site 35 | docs/ 36 | # translation temp files 37 | po/*~ 38 | # renv detritus 39 | renv/sandbox/ 40 | *.pyc 41 | *~ 42 | .DS_Store 43 | .ipynb_checkpoints 44 | .sass-cache 45 | __pycache__ 46 | _site 47 | .Rproj.user 48 | /Gemfile.lock 49 | vendor 50 | .bundle 51 | -------------------------------------------------------------------------------- /.github/workflows/pr-post-remove-branch.yaml: -------------------------------------------------------------------------------- 1 | name: "Bot: Remove Temporary PR Branch" 2 | 3 | on: 4 | workflow_run: 5 | workflows: ["Bot: Send Close Pull Request Signal"] 6 | types: 7 | - completed 8 | 9 | jobs: 10 | delete: 11 | name: "Delete branch from Pull Request" 12 | runs-on: ubuntu-22.04 13 | if: > 14 | github.event.workflow_run.event == 'pull_request' && 15 | github.event.workflow_run.conclusion == 'success' 16 | permissions: 17 | contents: write 18 | steps: 19 | - name: 'Download artifact' 20 | uses: carpentries/actions/download-workflow-artifact@main 21 | with: 22 | run: ${{ github.event.workflow_run.id }} 23 | name: pr 24 | - name: "Get PR Number" 25 | id: get-pr 26 | run: | 27 | unzip pr.zip 28 | echo "NUM=$(<./NUM)" >> $GITHUB_OUTPUT 29 | - name: 'Remove branch' 30 | uses: carpentries/actions/remove-branch@main 31 | with: 32 | pr: ${{ steps.get-pr.outputs.NUM }} 33 | -------------------------------------------------------------------------------- /bin/boilerplate/.travis.yml: -------------------------------------------------------------------------------- 1 | dist: xenial # Ubuntu 16.04 (required for python 3.7) 2 | language: python 3 | python: 3.7 4 | branches: 5 | only: 6 | - gh-pages 7 | - /.*/ 8 | before_install: 9 | - sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9 10 | - echo "deb https://cran.rstudio.com/bin/linux/ubuntu trusty/" | sudo tee -a /etc/apt/sources.list 11 | - sudo apt-get update -y 12 | - sudo apt-get install -y r-base 13 | - sudo Rscript -e "install.packages('knitr', repos = 'https://', dependencies = TRUE)" 14 | - sudo Rscript -e "install.packages('stringr', repos = 'https://cran.rstudio.com', dependencies = TRUE)" 15 | - sudo Rscript -e "install.packages('checkpoint', repos = 'https://cran.rstudio.com', dependencies = TRUE)" 16 | - sudo Rscript -e "install.packages('ggplot2', repos = 'https://cran.rstudio.com', dependencies = TRUE)" 17 | - rvm default 18 | - gem install json kramdown jekyll 19 | install: 20 | - pip install pyyaml 21 | script: 22 | - make lesson-check-all 23 | - make --always-make site 24 | -------------------------------------------------------------------------------- /bin/lesson_initialize.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | """Initialize a newly-created repository.""" 4 | 5 | 6 | import sys 7 | import os 8 | import shutil 9 | 10 | BOILERPLATE = ( 11 | '.travis.yml', 12 | 'AUTHORS', 13 | 'CITATION', 14 | 'CONTRIBUTING.md', 15 | 'README.md', 16 | '_config.yml', 17 | '_episodes/01-introduction.md', 18 | '_extras/about.md', 19 | '_extras/discuss.md', 20 | '_extras/figures.md', 21 | '_extras/guide.md', 22 | 'index.md', 23 | 'reference.md', 24 | 'setup.md', 25 | ) 26 | 27 | 28 | def main(): 29 | """Check for collisions, then create.""" 30 | 31 | # Check. 32 | errors = False 33 | for path in BOILERPLATE: 34 | if os.path.exists(path): 35 | print('Warning: {0} already exists.'.format(path), file=sys.stderr) 36 | errors = True 37 | if errors: 38 | print('**Exiting without creating files.**', file=sys.stderr) 39 | sys.exit(1) 40 | 41 | # Create. 42 | for path in BOILERPLATE: 43 | shutil.copyfile( 44 | "bin/boilerplate/{}".format(path), 45 | path 46 | ) 47 | 48 | 49 | if __name__ == '__main__': 50 | main() 51 | -------------------------------------------------------------------------------- /.github/workflows/pr-preflight.yaml: -------------------------------------------------------------------------------- 1 | name: "Pull Request Preflight Check" 2 | 3 | on: 4 | pull_request_target: 5 | branches: 6 | ["main"] 7 | types: 8 | ["opened", "synchronize", "reopened"] 9 | 10 | jobs: 11 | test-pr: 12 | name: "Test if pull request is valid" 13 | if: ${{ github.event.action != 'closed' }} 14 | runs-on: ubuntu-22.04 15 | outputs: 16 | is_valid: ${{ steps.check-pr.outputs.VALID }} 17 | permissions: 18 | pull-requests: write 19 | steps: 20 | - name: "Get Invalid Hashes File" 21 | id: hash 22 | run: | 23 | echo "json<> $GITHUB_OUTPUT 26 | - name: "Check PR" 27 | id: check-pr 28 | uses: carpentries/actions/check-valid-pr@main 29 | with: 30 | pr: ${{ github.event.number }} 31 | invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} 32 | fail_on_error: true 33 | - name: "Comment result of validation" 34 | id: comment-diff 35 | if: ${{ always() }} 36 | uses: carpentries/actions/comment-diff@main 37 | with: 38 | pr: ${{ github.event.number }} 39 | body: ${{ steps.check-pr.outputs.MSG }} 40 | -------------------------------------------------------------------------------- /bin/boilerplate/README.md: -------------------------------------------------------------------------------- 1 | # FIXME Lesson title 2 | 3 | [![Create a Slack Account with us](https://img.shields.io/badge/Create_Slack_Account-The_Carpentries-071159.svg)](https://swc-slack-invite.herokuapp.com/) 4 | 5 | This repository generates the corresponding lesson website from [The Carpentries](https://carpentries.org/) repertoire of lessons. 6 | 7 | ## Contributing 8 | 9 | We welcome all contributions to improve the lesson! Maintainers will do their best to help you if you have any 10 | questions, concerns, or experience any difficulties along the way. 11 | 12 | We'd like to ask you to familiarize yourself with our [Contribution Guide](CONTRIBUTING.md) and have a look at 13 | the [more detailed guidelines][lesson-example] on proper formatting, ways to render the lesson locally, and even 14 | how to write new episodes. 15 | 16 | Please see the current list of [issues][FIXME] for ideas for contributing to this 17 | repository. For making your contribution, we use the GitHub flow, which is 18 | nicely explained in the chapter [Contributing to a Project](http://git-scm.com/book/en/v2/GitHub-Contributing-to-a-Project) in Pro Git 19 | by Scott Chacon. 20 | Look for the tag ![good_first_issue](https://img.shields.io/badge/-good%20first%20issue-gold.svg). This indicates that the mantainers will welcome a pull request fixing this issue. 21 | 22 | 23 | ## Maintainer(s) 24 | 25 | Current maintainers of this lesson are 26 | 27 | * FIXME 28 | * FIXME 29 | * FIXME 30 | 31 | 32 | ## Authors 33 | 34 | A list of contributors to the lesson can be found in [AUTHORS](AUTHORS) 35 | 36 | ## Citation 37 | 38 | To cite this lesson, please consult with [CITATION](CITATION) 39 | 40 | [lesson-example]: https://carpentries.github.io/lesson-example 41 | -------------------------------------------------------------------------------- /episodes/next-steps.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Next Steps 3 | teaching: 10 4 | exercises: 0 5 | --- 6 | 7 | ::::::::::::::::::::::::::::::::::::::: objectives 8 | 9 | - Provide links to other useful resources around reducing and eliminating emissions 10 | 11 | :::::::::::::::::::::::::::::::::::::::::::::::::: 12 | 13 | :::::::::::::::::::::::::::::::::::::::: questions 14 | 15 | - What other resources are available to help me be more environmentally sustainable? 16 | 17 | :::::::::::::::::::::::::::::::::::::::::::::::::: 18 | 19 | We have now reached the end of the material for this workshop. In this section we provide links to useful resources around carbon emissions that may help you on your journey towards reducing and eliminating emissions from your work and beyond. 20 | 21 | ## Tools and emissions quantification 22 | 23 | - [Green Algorithms](https://www.green-algorithms.org/) 24 | - [How Bad are Bananas?](https://howbadarebananas.com/) - Book, Mike Berners-Lee 25 | - [Data Carbon Scorecard](https://digitaldecarb.org/data-carbon-scorecard/) 26 | - [Cloud Carbon Footprint](https://www.cloudcarbonfootprint.org/) 27 | 28 | ## Communities and groups 29 | 30 | - [Green SIG - Society of Research Software Engineering](https://socrse.github.io/green-sig/) 31 | - [NetDRIVE](https://eng.ox.ac.uk/netdrive/) 32 | 33 | ## Sustainable research 34 | 35 | - [Digital Humanities Climate Coalition Toolkit](https://sas-dhrh.github.io/dhcc-toolkit/) 36 | - [GreenDiSC](https://www.software.ac.uk/GreenDiSC) 37 | 38 | ## Other training 39 | 40 | - [Green Software Practitioner](https://learn.greensoftware.foundation/) 41 | - [Carbon Literacy Toolkits](https://carbonliteracy.com/sectors/) 42 | 43 | 44 | :::::::::::::::::::::::::::::::::::::::: keypoints 45 | 46 | - There are a lot of useful resources for understanding and reducing emissions. 47 | 48 | :::::::::::::::::::::::::::::::::::::::::::::::::: -------------------------------------------------------------------------------- /profiles/learner-profiles.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Learner Profiles 3 | --- 4 | 5 | In this section we provide some details of example learner profiles and 6 | suggest some possible different pathways or technical focuses to consider 7 | when teaching or planning a lesson based around this Docker material. As such, 8 | the information in this section is not designed to define fixed approaches and 9 | structures for teaching this material. It is instead aimed to provide ideas 10 | and inspiration and to encourage you to think about your audience when 11 | preparing to teach this material. The information here is based on both 12 | discussions about the intended audiences for this material and on direct 13 | experiences of instructors who have taught it at workshops following different 14 | technical pathways. 15 | 16 | ## Learner profiles 17 | 18 | We begin by providing some example learner profiles to highlight the potential 19 | target audience and the types of different research and technical backgrounds 20 | that you may find among learners engaging with this material. With these 21 | profiles, we aim to encourage you to think about the learners attending your 22 | workshop(s) and which episodes it may be most useful to teach. 23 | 24 | TODO: add learner profiles 25 | 26 | 27 | Considering things from a higher level, we also highlight three core groups of 28 | learners, based on job roles, who you may find attending lessons covering this 29 | material. While recognising that there are likely to be many learners who 30 | don't fit into one of the following groups, or who span more than one of them, 31 | we hope that highlighting these groups helps to provide an example of the 32 | different types of skills and expertise that learners engaging with this 33 | material may have: 34 | 35 | - **Researchers:** TODO 36 | 37 | - **RSEs:** TODO 38 | 39 | - **Systems professionals:** TODO 40 | 41 | 42 | 43 | 44 | -------------------------------------------------------------------------------- /.github/workflows/sandpaper-main.yaml: -------------------------------------------------------------------------------- 1 | name: "01 Build and Deploy Site" 2 | 3 | on: 4 | push: 5 | branches: 6 | - main 7 | - master 8 | schedule: 9 | - cron: '0 0 * * 2' 10 | workflow_dispatch: 11 | inputs: 12 | name: 13 | description: 'Who triggered this build?' 14 | required: true 15 | default: 'Maintainer (via GitHub)' 16 | reset: 17 | description: 'Reset cached markdown files' 18 | required: false 19 | default: false 20 | type: boolean 21 | jobs: 22 | full-build: 23 | name: "Build Full Site" 24 | 25 | # 2024-10-01: ubuntu-latest is now 24.04 and R is not installed by default in the runner image 26 | # pin to 22.04 for now 27 | runs-on: ubuntu-22.04 28 | permissions: 29 | checks: write 30 | contents: write 31 | pages: write 32 | env: 33 | GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} 34 | RENV_PATHS_ROOT: ~/.local/share/renv/ 35 | steps: 36 | 37 | - name: "Checkout Lesson" 38 | uses: actions/checkout@v4 39 | 40 | - name: "Set up R" 41 | uses: r-lib/actions/setup-r@v2 42 | with: 43 | use-public-rspm: true 44 | install-r: false 45 | 46 | - name: "Set up Pandoc" 47 | uses: r-lib/actions/setup-pandoc@v2 48 | 49 | - name: "Setup Lesson Engine" 50 | uses: carpentries/actions/setup-sandpaper@main 51 | with: 52 | cache-version: ${{ secrets.CACHE_VERSION }} 53 | 54 | - name: "Setup Package Cache" 55 | uses: carpentries/actions/setup-lesson-deps@main 56 | with: 57 | cache-version: ${{ secrets.CACHE_VERSION }} 58 | 59 | - name: "Deploy Site" 60 | run: | 61 | reset <- "${{ github.event.inputs.reset }}" == "true" 62 | sandpaper::package_cache_trigger(TRUE) 63 | sandpaper:::ci_deploy(reset = reset) 64 | shell: Rscript {0} 65 | -------------------------------------------------------------------------------- /bin/generate_md_episodes.R: -------------------------------------------------------------------------------- 1 | generate_md_episodes <- function() { 2 | 3 | library("methods") 4 | 5 | if (!require("remotes", quietly = TRUE)) { 6 | install.packages("remotes", repos = c(CRAN = "https://cloud.r-project.org/")) 7 | } 8 | 9 | if (!require("requirements", quietly = TRUE)) { 10 | remotes::install_github("hadley/requirements") 11 | } 12 | 13 | required_pkgs <- unique(c( 14 | ## Packages for episodes 15 | requirements:::req_dir("_episodes_rmd"), 16 | ## Pacakges for tools 17 | requirements:::req_dir("bin") 18 | )) 19 | 20 | missing_pkgs <- setdiff(required_pkgs, rownames(installed.packages())) 21 | 22 | if (length(missing_pkgs)) { 23 | message("Installing missing required packages: ", 24 | paste(missing_pkgs, collapse=", ")) 25 | install.packages(missing_pkgs) 26 | } 27 | 28 | if (require("knitr") && packageVersion("knitr") < '1.9.19') 29 | stop("knitr must be version 1.9.20 or higher") 30 | 31 | ## get the Rmd file to process from the command line, and generate the path for their respective outputs 32 | args <- commandArgs(trailingOnly = TRUE) 33 | if (!identical(length(args), 2L)) { 34 | stop("input and output file must be passed to the script") 35 | } 36 | 37 | src_rmd <- args[1] 38 | dest_md <- args[2] 39 | 40 | ## knit the Rmd into markdown 41 | knitr::knit(src_rmd, output = dest_md) 42 | 43 | # Read the generated md files and add comments advising not to edit them 44 | vapply(dest_md, function(y) { 45 | con <- file(y) 46 | mdfile <- readLines(con) 47 | if (mdfile[1] != "---") 48 | stop("Input file does not have a valid header") 49 | mdfile <- append(mdfile, "# Please do not edit this file directly; it is auto generated.", after = 1) 50 | mdfile <- append(mdfile, paste("# Instead, please edit", 51 | basename(y), "in _episodes_rmd/"), after = 2) 52 | writeLines(mdfile, con) 53 | close(con) 54 | return(paste("Warning added to YAML header of", y)) 55 | }, 56 | character(1)) 57 | } 58 | 59 | generate_md_episodes() 60 | -------------------------------------------------------------------------------- /.github/workflows/workbench-beta-phase.yml: -------------------------------------------------------------------------------- 1 | name: "Deploy to AWS" 2 | 3 | on: 4 | workflow_run: 5 | workflows: ["01 Build and Deploy Site"] 6 | types: 7 | - completed 8 | workflow_dispatch: 9 | 10 | jobs: 11 | preflight: 12 | name: "Preflight Check" 13 | runs-on: ubuntu-latest 14 | outputs: 15 | ok: ${{ steps.check.outputs.ok }} 16 | folder: ${{ steps.check.outputs.folder }} 17 | steps: 18 | - id: check 19 | run: | 20 | if [[ -z "${{ secrets.DISTRIBUTION }}" || -z "${{ secrets.AWS_ACCESS_KEY_ID }}" || -z "${{ secrets.AWS_SECRET_ACCESS_KEY }}" ]]; then 21 | echo ":information_source: No site configured" >> $GITHUB_STEP_SUMMARY 22 | echo "" >> $GITHUB_STEP_SUMMARY 23 | echo 'To deploy the preview on AWS, you need the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `DISTRIBUTION` secrets set up' >> $GITHUB_STEP_SUMMARY 24 | else 25 | echo "::set-output name=folder::"$(sed -E 's^.+/(.+)^\1^' <<< ${{ github.repository }}) 26 | echo "::set-output name=ok::true" 27 | fi 28 | 29 | full-build: 30 | name: "Deploy to AWS" 31 | needs: [preflight] 32 | if: ${{ needs.preflight.outputs.ok }} 33 | runs-on: ubuntu-latest 34 | steps: 35 | 36 | - name: "Checkout site folder" 37 | uses: actions/checkout@v3 38 | with: 39 | ref: 'gh-pages' 40 | path: 'source' 41 | 42 | - name: "Deploy to Bucket" 43 | uses: jakejarvis/s3-sync-action@v0.5.1 44 | with: 45 | args: --acl public-read --follow-symlinks --delete --exclude '.git/*' 46 | env: 47 | AWS_S3_BUCKET: preview.carpentries.org 48 | AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} 49 | AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} 50 | SOURCE_DIR: 'source' 51 | DEST_DIR: ${{ needs.preflight.outputs.folder }} 52 | 53 | - name: "Invalidate CloudFront" 54 | uses: chetan/invalidate-cloudfront-action@master 55 | env: 56 | PATHS: /* 57 | AWS_REGION: 'us-east-1' 58 | DISTRIBUTION: ${{ secrets.DISTRIBUTION }} 59 | AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} 60 | AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} 61 | -------------------------------------------------------------------------------- /bin/boilerplate/_extras/figures.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Figures 3 | --- 4 | 5 | {% include base_path.html %} 6 | 7 | 61 | 62 | {% for episode in site.episodes %} 63 |
64 | {% endfor %} 65 | 66 | {% include links.md %} 67 | -------------------------------------------------------------------------------- /.github/workflows/update-workflows.yaml: -------------------------------------------------------------------------------- 1 | name: "02 Maintain: Update Workflow Files" 2 | 3 | on: 4 | workflow_dispatch: 5 | inputs: 6 | name: 7 | description: 'Who triggered this build (enter github username to tag yourself)?' 8 | required: true 9 | default: 'weekly run' 10 | clean: 11 | description: 'Workflow files/file extensions to clean (no wildcards, enter "" for none)' 12 | required: false 13 | default: '.yaml' 14 | schedule: 15 | # Run every Tuesday 16 | - cron: '0 0 * * 2' 17 | 18 | jobs: 19 | check_token: 20 | name: "Check SANDPAPER_WORKFLOW token" 21 | runs-on: ubuntu-22.04 22 | outputs: 23 | workflow: ${{ steps.validate.outputs.wf }} 24 | repo: ${{ steps.validate.outputs.repo }} 25 | steps: 26 | - name: "validate token" 27 | id: validate 28 | uses: carpentries/actions/check-valid-credentials@main 29 | with: 30 | token: ${{ secrets.SANDPAPER_WORKFLOW }} 31 | 32 | update_workflow: 33 | name: "Update Workflow" 34 | runs-on: ubuntu-22.04 35 | needs: check_token 36 | if: ${{ needs.check_token.outputs.workflow == 'true' }} 37 | steps: 38 | - name: "Checkout Repository" 39 | uses: actions/checkout@v4 40 | 41 | - name: Update Workflows 42 | id: update 43 | uses: carpentries/actions/update-workflows@main 44 | with: 45 | clean: ${{ github.event.inputs.clean }} 46 | 47 | - name: Create Pull Request 48 | id: cpr 49 | if: "${{ steps.update.outputs.new }}" 50 | uses: carpentries/create-pull-request@main 51 | with: 52 | token: ${{ secrets.SANDPAPER_WORKFLOW }} 53 | delete-branch: true 54 | branch: "update/workflows" 55 | commit-message: "[actions] update sandpaper workflow to version ${{ steps.update.outputs.new }}" 56 | title: "Update Workflows to Version ${{ steps.update.outputs.new }}" 57 | body: | 58 | :robot: This is an automated build 59 | 60 | Update Workflows from sandpaper version ${{ steps.update.outputs.old }} -> ${{ steps.update.outputs.new }} 61 | 62 | - Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }} 63 | 64 | [1]: https://github.com/carpentries/create-pull-request/tree/main 65 | labels: "type: template and tools" 66 | draft: false 67 | -------------------------------------------------------------------------------- /config.yaml: -------------------------------------------------------------------------------- 1 | #------------------------------------------------------------ 2 | # Values for this lesson. 3 | #------------------------------------------------------------ 4 | 5 | # Which carpentry is this (swc, dc, lc, or cp)? 6 | # swc: Software Carpentry 7 | # dc: Data Carpentry 8 | # lc: Library Carpentry 9 | # cp: Carpentries (to use for instructor training for instance) 10 | # incubator: The Carpentries Incubator 11 | carpentry: 'incubator' 12 | 13 | # Overall title for pages. 14 | title: 'Green software use on HPC' 15 | 16 | # Date the lesson was created (YYYY-MM-DD, this is empty by default) 17 | created: 18 | 19 | # Comma-separated list of keywords for the lesson 20 | keywords: 'software, data, lesson, environment' 21 | 22 | # Life cycle stage of the lesson 23 | # possible values: pre-alpha, alpha, beta, stable 24 | life_cycle: 'stable' 25 | 26 | # License of the lesson materials (recommended CC-BY 4.0) 27 | license: 'CC-BY 4.0' 28 | 29 | # Link to the source repository for this lesson 30 | source: 'https://github.com/EPCCed/2025-10-22_GreenHPC_Bham' 31 | 32 | # Default branch of your lesson 33 | branch: 'main' 34 | 35 | # Who to contact if there are any issues 36 | contact: 'a.turner@epcc.ed.ac.uk' 37 | 38 | # Navigation ------------------------------------------------ 39 | # 40 | # Use the following menu items to specify the order of 41 | # individual pages in each dropdown section. Leave blank to 42 | # include all pages in the folder. 43 | # 44 | # Example ------------- 45 | # 46 | # episodes: 47 | # - introduction.md 48 | # - first-steps.md 49 | # 50 | # learners: 51 | # - setup.md 52 | # 53 | # instructors: 54 | # - instructor-notes.md 55 | # 56 | # profiles: 57 | # - one-learner.md 58 | # - another-learner.md 59 | 60 | # Order of episodes in your lesson 61 | episodes: 62 | - introduction.md 63 | - carbon-efficiency.md 64 | - energy-efficiency.md 65 | - morning-break.md 66 | - carbon-awareness.md 67 | - hardware-efficiency.md 68 | - lunch-break.md 69 | - measurement.md 70 | - afternoon-break.md 71 | - reducing-emissions.md 72 | - next-steps.md 73 | # - renewable-energy.md 74 | 75 | # Information for Learners 76 | learners: 77 | 78 | # Information for Instructors 79 | instructors: 80 | 81 | # Learner Profiles 82 | profiles: 83 | 84 | # Customisation --------------------------------------------- 85 | # 86 | # This space below is where custom yaml items (e.g. pinning 87 | # sandpaper and varnish versions) should live 88 | 89 | 90 | analytics: carpentries 91 | lang: en 92 | workbench-beta: yes 93 | -------------------------------------------------------------------------------- /bin/chunk-options.R: -------------------------------------------------------------------------------- 1 | # These settings control the behavior of all chunks in the novice R materials. 2 | # For example, to generate the lessons with all the output hidden, simply change 3 | # `results` from "markup" to "hide". 4 | # For more information on available chunk options, see 5 | # http://yihui.name/knitr/options#chunk_options 6 | 7 | library("knitr") 8 | 9 | fix_fig_path <- function(pth) file.path("..", pth) 10 | 11 | 12 | ## We set the path for the figures globally below, so if we want to 13 | ## customize it for individual episodes, we can append a prefix to the 14 | ## global path. For instance, if we call knitr_fig_path("01-") in the 15 | ## first episode of the lesson, it will generate the figures in 16 | ## `fig/rmd-01-` 17 | knitr_fig_path <- function(prefix) { 18 | new_path <- paste0(opts_chunk$get("fig.path"), 19 | prefix) 20 | opts_chunk$set(fig.path = new_path) 21 | } 22 | 23 | ## We use the rmd- prefix for the figures generated by the lessons so 24 | ## they can be easily identified and deleted by `make clean-rmd`. The 25 | ## working directory when the lessons are generated is the root so the 26 | ## figures need to be saved in fig/, but when the site is generated, 27 | ## the episodes will be one level down. We fix the path using the 28 | ## `fig.process` option. 29 | 30 | opts_chunk$set(tidy = FALSE, results = "markup", comment = NA, 31 | fig.align = "center", fig.path = "fig/rmd-", 32 | fig.process = fix_fig_path, 33 | fig.width = 8.5, fig.height = 8.5, 34 | fig.retina = 2) 35 | 36 | # The hooks below add html tags to the code chunks and their output so that they 37 | # are properly formatted when the site is built. 38 | 39 | hook_in <- function(x, options) { 40 | lg <- tolower(options$engine) 41 | style <- paste0(".language-", lg) 42 | 43 | stringr::str_c("\n\n~~~\n", 44 | paste0(x, collapse="\n"), 45 | "\n~~~\n{: ", style, "}\n\n") 46 | } 47 | 48 | hook_out <- function(x, options) { 49 | x <- gsub("\n$", "", x) 50 | stringr::str_c("\n\n~~~\n", 51 | paste0(x, collapse="\n"), 52 | "\n~~~\n{: .output}\n\n") 53 | } 54 | 55 | hook_error <- function(x, options) { 56 | x <- gsub("\n$", "", x) 57 | stringr::str_c("\n\n~~~\n", 58 | paste0(x, collapse="\n"), 59 | "\n~~~\n{: .error}\n\n") 60 | } 61 | 62 | knit_hooks$set(source = hook_in, output = hook_out, warning = hook_error, 63 | error = hook_error, message = hook_out) 64 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | > **ATTENTION** This is an experimental test of [The Carpentries Workbench](https://carpentries.github.io/workbench) lesson infrastructure. 2 | > It was automatically converted from the source lesson via [the lesson transition script](https://github.com/carpentries/lesson-transition/). 3 | > 4 | > If anything seems off, please contact Zhian Kamvar [zkamvar@carpentries.org](mailto:zkamvar@carpentries.org) 5 | 6 | [![Gitpod ready-to-code](https://img.shields.io/badge/Gitpod-ready--to--code-blue?logo=gitpod)](https://gitpod.io/#https://github.com/carpentries-incubator/docker-introduction) 7 | [![The Carpentries Lab Review Status](https://badges.carpentries-lab.org/15_status.svg)](https://github.com/carpentries-lab/reviews/issues/15) 8 | 9 | # Green software use on HPC 10 | 11 | [![Create a Slack Account with us](https://img.shields.io/badge/Create_Slack_Account-The_Carpentries-071159.svg)](https://swc-slack-invite.herokuapp.com/) 12 | 13 | This lesson introduces environmental sustainability principles in the context of high performance computing (HPC) systems. Understanding the scale of emissions from different sources is critical to being able to make changes to work in a more environmentally sustainable way. This lesson will give you the ability to understand emissions arising from use of HPC system and how to quantify them. We will use practical examples and real data from an existing HPC facility to illustrate the concepts. 14 | 15 | We use the [UK National Supercomputing Service, ARCHER2](https://www.archer2.ac.uk) as an example throughout this lesson but the principles and learning should be applicable to any HPC system. 16 | 17 | > This workshop is based on and builds on the material in the [Green Software Practitioner](https://learn.greensoftware.foundation/) course developed by the [Green Software Foundation](https://greensoftware.foundation/). 18 | 19 | Target audience 20 | This lesson is aimed at all stakeholders in HPC use (e.g. researchers, RSEs, funders). No knowledge of environmental sustainability principles is assumed. 21 | 22 | Prerequisites 23 | There are no prerequisites for this lesson. 24 | 25 | 26 | ## Contributing 27 | 28 | We welcome all contributions to improve the lesson! Maintainers will do their best to help you if you have any 29 | questions, concerns, or experience any difficulties along the way. 30 | 31 | We'd like to ask you to familiarize yourself with our [Contribution Guide](CONTRIBUTING.md) and have a look at 32 | the [more detailed guidelines][lesson-example] on proper formatting, ways to render the lesson locally, and even 33 | how to write new episodes. 34 | 35 | Please see the current list of [issues](https://github.com/carpentries-incubator/docker-introduction/issues) for ideas for contributing to this 36 | repository. For making your contribution, we use the GitHub flow, which is 37 | nicely explained in the chapter [Contributing to a Project](https://git-scm.com/book/en/v2/GitHub-Contributing-to-a-Project) in Pro Git 38 | by Scott Chacon. 39 | Look for the tag ![good\_first\_issue](https://img.shields.io/badge/-good%20first%20issue-gold.svg). This indicates that the mantainers will welcome a pull request fixing this issue. 40 | 41 | ## Maintainer(s) 42 | 43 | Current maintainers of this lesson are 44 | 45 | - [Andy Turner](https://github.com/aturner-epcc) 46 | 47 | ## Authors 48 | 49 | A list of contributors to the lesson can be found in [AUTHORS](AUTHORS) 50 | 51 | ## Citation 52 | 53 | To cite this lesson, please consult with [CITATION](CITATION) 54 | 55 | [lesson-example]: https://carpentries.github.io/lesson-example 56 | 57 | 58 | 59 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Licenses" 3 | --- 4 | 5 | ## Instructional Material 6 | 7 | All Carpentries (Software Carpentry, Data Carpentry, and Library Carpentry) 8 | instructional material is made available under the [Creative Commons 9 | Attribution license][cc-by-human]. The following is a human-readable summary of 10 | (and not a substitute for) the [full legal text of the CC BY 4.0 11 | license][cc-by-legal]. 12 | 13 | You are free: 14 | 15 | - to **Share**---copy and redistribute the material in any medium or format 16 | - to **Adapt**---remix, transform, and build upon the material 17 | 18 | for any purpose, even commercially. 19 | 20 | The licensor cannot revoke these freedoms as long as you follow the license 21 | terms. 22 | 23 | Under the following terms: 24 | 25 | - **Attribution**---You must give appropriate credit (mentioning that your work 26 | is derived from work that is Copyright (c) The Carpentries and, where 27 | practical, linking to ), provide a [link to the 28 | license][cc-by-human], and indicate if changes were made. You may do so in 29 | any reasonable manner, but not in any way that suggests the licensor endorses 30 | you or your use. 31 | 32 | - **No additional restrictions**---You may not apply legal terms or 33 | technological measures that legally restrict others from doing anything the 34 | license permits. With the understanding that: 35 | 36 | Notices: 37 | 38 | * You do not have to comply with the license for elements of the material in 39 | the public domain or where your use is permitted by an applicable exception 40 | or limitation. 41 | * No warranties are given. The license may not give you all of the permissions 42 | necessary for your intended use. For example, other rights such as publicity, 43 | privacy, or moral rights may limit how you use the material. 44 | 45 | ## Software 46 | 47 | Except where otherwise noted, the example programs and other software provided 48 | by The Carpentries are made available under the [OSI][osi]-approved [MIT 49 | license][mit-license]. 50 | 51 | Permission is hereby granted, free of charge, to any person obtaining a copy of 52 | this software and associated documentation files (the "Software"), to deal in 53 | the Software without restriction, including without limitation the rights to 54 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 55 | of the Software, and to permit persons to whom the Software is furnished to do 56 | so, subject to the following conditions: 57 | 58 | The above copyright notice and this permission notice shall be included in all 59 | copies or substantial portions of the Software. 60 | 61 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 62 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 63 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 64 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 65 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 66 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 67 | SOFTWARE. 68 | 69 | ## Trademark 70 | 71 | "The Carpentries", "Software Carpentry", "Data Carpentry", and "Library 72 | Carpentry" and their respective logos are registered trademarks of [Community 73 | Initiatives][ci]. 74 | 75 | [cc-by-human]: https://creativecommons.org/licenses/by/4.0/ 76 | [cc-by-legal]: https://creativecommons.org/licenses/by/4.0/legalcode 77 | [mit-license]: https://opensource.org/licenses/mit-license.html 78 | [ci]: https://communityin.org/ 79 | [osi]: https://opensource.org 80 | -------------------------------------------------------------------------------- /index.md: -------------------------------------------------------------------------------- 1 | --- 2 | permalink: index.html 3 | site: sandpaper::sandpaper_site 4 | --- 5 | 6 | This lesson introduces environmental sustainability principles in the context of 7 | high performance computing (HPC) systems. Understanding the scale of emissions from 8 | different sources is critical to being able to make changes to work in a more environmentally 9 | sustainable way. This lesson will give you the ability to understand emissions arising from 10 | use of HPC systems and how to quantify them. We will use practical examples and real data from 11 | an existing HPC facility to illustrate the concepts. 12 | 13 | We use the [UK National Supercomputing Service, ARCHER2](https://www.archer2.ac.uk) as an 14 | example throughout this lesson but the principles and learning should be applicable to 15 | any HPC system. 16 | 17 | After completing this lesson you should: 18 | 19 | - Be able to understand the concept of carbon efficiency and how it relates to energy efficiency - 20 | including cases where energy efficiency can be at odds with carbon efficiency 21 | - Know about how HPC systems potentially reduce emissions 22 | - Understand carbon intensity of electricity generation and the implications for carbon aware use 23 | of HPC 24 | - Appreciate the embodied emissions associated with HPC hardware and how they impact carbon aware 25 | use of HPC 26 | - Be aware of the frameworks used to measure and report on carbon emissions and how the terms used 27 | in these frameworks map onto HPC 28 | - Gain practical advice on how you can measure and improve the carbon efficiency of your use of 29 | HPC 30 | 31 | ## Location and timings 32 | 33 | - Date: 22 October 2025 34 | - Time: 10:00-15:30 BST 35 | - Location: G08 Elm House, University of Birmingham, UK ([location](https://www.archer2.ac.uk/training/locations/birmingham.html)) 36 | - Instructor: Andy Turner, EPCC 37 | 38 | ## Schedule 39 | 40 | | Time | Episode | 41 | |-----:|---------| 42 | | 10:00 | Introduction | 43 | | 10:15 | Carbon efficiency | 44 | | 10:35 | Energy efficiency | 45 | | 11:05 | Morning break | 46 | | 11:25 | Carbon awareness | 47 | | 11:55 | Hardware efficiency | 48 | | 12:15 | Lunch | 49 | | 13:00 | Measurement | 50 | | 14:00 | Afternoon break | 51 | | 14:20 | Reducing emissions | 52 | | 15:00 | Next steps | 53 | | 15:30 | End | 54 | 55 | ## Registration 56 | 57 | - [Registration on ARCHER2 website](https://www.archer2.ac.uk/training/courses/251022-green-computing/) 58 | 59 | ## Catering 60 | 61 | No refreshments or lunch are provided. Local organisors will be on hand to suggest locations. 62 | 63 | 64 | :::::::::::::::::::::::::::::::::::::: callout 65 | 66 | This workshop is based on and builds on the material in the [Green Software Practitioner](https://learn.greensoftware.foundation/) 67 | course developed by the [Green Software Foundation](https://greensoftware.foundation/). 68 | 69 | :::::::::::::::::::::::::::::::::::::::::::::::::: 70 | 71 | 72 | ::::::::::::::::::::::::::::::::::::::::: callout 73 | 74 | ## Target audience 75 | 76 | Anyone involved in using or delivering HPC resources - a user, a developer of software for use on HPC, an HPC service operator, or those involved in procuring HPC systems. By attending this course, you will be able to better understand the effect of HPC systems on carbon emissions, where it fits with emissions from other activities, and what concrete actions can be taken to reduce emissions and the size of the impact from these reductions. No knowledge of environmental sustainability principles is assumed. 77 | 78 | 79 | :::::::::::::::::::::::::::::::::::::::::::::::::: 80 | 81 | :::::::::::::::::::::::::::::::::::::::::: prereq 82 | 83 | ## Prerequisites 84 | 85 | Attendees should familiarise themselves with the key concepts below before attending the workshop. 86 | 87 | :::::::::::::::::::::::::::::::::::::::::::::::::: 88 | 89 | 90 | 92 | -------------------------------------------------------------------------------- /bin/boilerplate/_config.yml: -------------------------------------------------------------------------------- 1 | #------------------------------------------------------------ 2 | # Values for this lesson. 3 | #------------------------------------------------------------ 4 | 5 | # Which carpentry is this ("swc", "dc", "lc", or "cp")? 6 | # swc: Software Carpentry 7 | # dc: Data Carpentry 8 | # lc: Library Carpentry 9 | # cp: Carpentries (to use for instructor traning for instance) 10 | carpentry: "swc" 11 | 12 | # Overall title for pages. 13 | title: "Lesson Title" 14 | 15 | # Life cycle stage of the lesson 16 | # possible values: "pre-alpha", "alpha", "beta", "stable" 17 | life_cycle: "pre-alpha" 18 | 19 | #------------------------------------------------------------ 20 | # Generic settings (should not need to change). 21 | #------------------------------------------------------------ 22 | 23 | # What kind of thing is this ("workshop" or "lesson")? 24 | kind: "lesson" 25 | 26 | # Magic to make URLs resolve both locally and on GitHub. 27 | # See https://help.github.com/articles/repository-metadata-on-github-pages/. 28 | # Please don't change it: / is correct. 29 | repository: / 30 | 31 | # Email address, no mailto: 32 | email: "team@carpentries.org" 33 | 34 | # Sites. 35 | amy_site: "https://amy.software-carpentry.org/workshops" 36 | carpentries_github: "https://github.com/carpentries" 37 | carpentries_pages: "https://carpentries.github.io" 38 | carpentries_site: "https://carpentries.org/" 39 | dc_site: "http://datacarpentry.org" 40 | example_repo: "https://github.com/carpentries/lesson-example" 41 | example_site: "https://carpentries.github.io/lesson-example" 42 | lc_site: "https://librarycarpentry.org/" 43 | swc_github: "https://github.com/swcarpentry" 44 | swc_pages: "https://swcarpentry.github.io" 45 | swc_site: "https://software-carpentry.org" 46 | template_repo: "https://github.com/carpentries/styles" 47 | training_site: "https://carpentries.github.io/instructor-training" 48 | workshop_repo: "https://github.com/carpentries/workshop-template" 49 | workshop_site: "https://carpentries.github.io/workshop-template" 50 | cc_by_human: "https://creativecommons.org/licenses/by/4.0/" 51 | 52 | # Surveys. 53 | swc_pre_survey: "https://www.surveymonkey.com/r/swc_pre_workshop_v1?workshop_id=" 54 | swc_post_survey: "https://www.surveymonkey.com/r/swc_post_workshop_v1?workshop_id=" 55 | training_post_survey: "https://www.surveymonkey.com/r/post-instructor-training" 56 | dc_pre_survey: "https://www.surveymonkey.com/r/dcpreworkshopassessment?workshop_id=" 57 | dc_post_survey: "https://www.surveymonkey.com/r/dcpostworkshopassessment?workshop_id=" 58 | lc_pre_survey: "https://www.surveymonkey.com/r/lcpreworkshopsurvey?workshop_id=" 59 | lc_post_survey: "https://www.surveymonkey.com/r/lcpostworkshopsurvey?workshop_id=" 60 | instructor_pre_survey: "https://www.surveymonkey.com/r/instructor_training_pre_survey?workshop_id=" 61 | instructor_post_survey: "https://www.surveymonkey.com/r/instructor_training_post_survey?workshop_id=" 62 | 63 | 64 | # Start time in minutes (0 to be clock-independent, 540 to show a start at 09:00 am). 65 | start_time: 0 66 | 67 | # Specify that things in the episodes collection should be output. 68 | collections: 69 | episodes: 70 | output: true 71 | permalink: /:path/index.html 72 | extras: 73 | output: true 74 | permalink: /:path/index.html 75 | 76 | # Set the default layout for things in the episodes collection. 77 | defaults: 78 | - values: 79 | root: . 80 | layout: page 81 | - scope: 82 | path: "" 83 | type: episodes 84 | values: 85 | root: .. 86 | layout: episode 87 | - scope: 88 | path: "" 89 | type: extras 90 | values: 91 | root: .. 92 | layout: page 93 | 94 | # Files and directories that are not to be copied. 95 | exclude: 96 | - Makefile 97 | - bin/ 98 | - .Rproj.user/ 99 | 100 | # Turn on built-in syntax highlighting. 101 | highlighter: rouge 102 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | ## ======================================== 2 | ## Commands for both workshop and lesson websites. 3 | 4 | # Settings 5 | MAKEFILES=Makefile $(wildcard *.mk) 6 | JEKYLL=bundle exec jekyll 7 | JEKYLL_VERSION=3.8.5 8 | PARSER=bin/markdown_ast.rb 9 | DST=_site 10 | 11 | # Controls 12 | .PHONY : commands clean files 13 | .NOTPARALLEL: 14 | all : commands 15 | 16 | ## commands : show all commands. 17 | commands : 18 | @grep -h -E '^##' ${MAKEFILES} | sed -e 's/## //g' 19 | 20 | ## docker-serve : use docker to build the site 21 | docker-serve : 22 | docker run --rm -it -v ${PWD}:/srv/jekyll -p 127.0.0.1:4000:4000 jekyll/jekyll:${JEKYLL_VERSION} make serve 23 | 24 | ## serve : run a local server. 25 | serve : lesson-md 26 | ${JEKYLL} serve 27 | 28 | ## site : build files but do not run a server. 29 | site : lesson-md 30 | ${JEKYLL} build 31 | 32 | # repo-check : check repository settings. 33 | repo-check : 34 | @bin/repo_check.py -s . 35 | 36 | ## clean : clean up junk files. 37 | clean : 38 | @rm -rf ${DST} 39 | @rm -rf .sass-cache 40 | @rm -rf bin/__pycache__ 41 | @find . -name .DS_Store -exec rm {} \; 42 | @find . -name '*~' -exec rm {} \; 43 | @find . -name '*.pyc' -exec rm {} \; 44 | 45 | ## clean-rmd : clean intermediate R files (that need to be committed to the repo). 46 | clean-rmd : 47 | @rm -rf ${RMD_DST} 48 | @rm -rf fig/rmd-* 49 | 50 | ## ---------------------------------------- 51 | ## Commands specific to workshop websites. 52 | 53 | .PHONY : workshop-check 54 | 55 | ## workshop-check : check workshop homepage. 56 | workshop-check : 57 | @bin/workshop_check.py . 58 | 59 | ## ---------------------------------------- 60 | ## Commands specific to lesson websites. 61 | 62 | .PHONY : lesson-check lesson-md lesson-files lesson-fixme 63 | 64 | # RMarkdown files 65 | RMD_SRC = $(wildcard _episodes_rmd/??-*.Rmd) 66 | RMD_DST = $(patsubst _episodes_rmd/%.Rmd,_episodes/%.md,$(RMD_SRC)) 67 | 68 | # Lesson source files in the order they appear in the navigation menu. 69 | MARKDOWN_SRC = \ 70 | index.md \ 71 | CODE_OF_CONDUCT.md \ 72 | setup.md \ 73 | $(sort $(wildcard _episodes/*.md)) \ 74 | reference.md \ 75 | $(sort $(wildcard _extras/*.md)) \ 76 | LICENSE.md 77 | 78 | # Generated lesson files in the order they appear in the navigation menu. 79 | HTML_DST = \ 80 | ${DST}/index.html \ 81 | ${DST}/conduct/index.html \ 82 | ${DST}/setup/index.html \ 83 | $(patsubst _episodes/%.md,${DST}/%/index.html,$(sort $(wildcard _episodes/*.md))) \ 84 | ${DST}/reference/index.html \ 85 | $(patsubst _extras/%.md,${DST}/%/index.html,$(sort $(wildcard _extras/*.md))) \ 86 | ${DST}/license/index.html 87 | 88 | ## lesson-md : convert Rmarkdown files to markdown 89 | lesson-md : ${RMD_DST} 90 | 91 | _episodes/%.md: _episodes_rmd/%.Rmd 92 | @bin/knit_lessons.sh $< $@ 93 | 94 | ## lesson-check : validate lesson Markdown. 95 | lesson-check : lesson-fixme 96 | @bin/lesson_check.py -s . -p ${PARSER} -r _includes/links.md 97 | 98 | ## lesson-check-all : validate lesson Markdown, checking line lengths and trailing whitespace. 99 | lesson-check-all : 100 | @bin/lesson_check.py -s . -p ${PARSER} -r _includes/links.md -l -w --permissive 101 | 102 | ## unittest : run unit tests on checking tools. 103 | unittest : 104 | @bin/test_lesson_check.py 105 | 106 | ## lesson-files : show expected names of generated files for debugging. 107 | lesson-files : 108 | @echo 'RMD_SRC:' ${RMD_SRC} 109 | @echo 'RMD_DST:' ${RMD_DST} 110 | @echo 'MARKDOWN_SRC:' ${MARKDOWN_SRC} 111 | @echo 'HTML_DST:' ${HTML_DST} 112 | 113 | ## lesson-fixme : show FIXME markers embedded in source files. 114 | lesson-fixme : 115 | @fgrep -i -n FIXME ${MARKDOWN_SRC} || true 116 | 117 | #------------------------------------------------------------------------------- 118 | # Include extra commands if available. 119 | #------------------------------------------------------------------------------- 120 | 121 | -include commands.mk 122 | -------------------------------------------------------------------------------- /.github/workflows/pr-receive.yaml: -------------------------------------------------------------------------------- 1 | name: "Receive Pull Request" 2 | 3 | on: 4 | pull_request: 5 | types: 6 | [opened, synchronize, reopened] 7 | 8 | concurrency: 9 | group: ${{ github.ref }} 10 | cancel-in-progress: true 11 | 12 | jobs: 13 | test-pr: 14 | name: "Record PR number" 15 | if: ${{ github.event.action != 'closed' }} 16 | runs-on: ubuntu-22.04 17 | outputs: 18 | is_valid: ${{ steps.check-pr.outputs.VALID }} 19 | steps: 20 | - name: "Record PR number" 21 | id: record 22 | if: ${{ always() }} 23 | run: | 24 | echo ${{ github.event.number }} > ${{ github.workspace }}/NR # 2022-03-02: artifact name fixed to be NR 25 | - name: "Upload PR number" 26 | id: upload 27 | if: ${{ always() }} 28 | uses: actions/upload-artifact@v4 29 | with: 30 | name: pr 31 | path: ${{ github.workspace }}/NR 32 | - name: "Get Invalid Hashes File" 33 | id: hash 34 | run: | 35 | echo "json<> $GITHUB_OUTPUT 38 | - name: "echo output" 39 | run: | 40 | echo "${{ steps.hash.outputs.json }}" 41 | - name: "Check PR" 42 | id: check-pr 43 | uses: carpentries/actions/check-valid-pr@main 44 | with: 45 | pr: ${{ github.event.number }} 46 | invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} 47 | 48 | build-md-source: 49 | name: "Build markdown source files if valid" 50 | needs: test-pr 51 | runs-on: ubuntu-22.04 52 | if: ${{ needs.test-pr.outputs.is_valid == 'true' }} 53 | env: 54 | GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} 55 | RENV_PATHS_ROOT: ~/.local/share/renv/ 56 | CHIVE: ${{ github.workspace }}/site/chive 57 | PR: ${{ github.workspace }}/site/pr 58 | MD: ${{ github.workspace }}/site/built 59 | steps: 60 | - name: "Check Out Main Branch" 61 | uses: actions/checkout@v4 62 | 63 | - name: "Check Out Staging Branch" 64 | uses: actions/checkout@v4 65 | with: 66 | ref: md-outputs 67 | path: ${{ env.MD }} 68 | 69 | - name: "Set up R" 70 | uses: r-lib/actions/setup-r@v2 71 | with: 72 | use-public-rspm: true 73 | install-r: false 74 | 75 | - name: "Set up Pandoc" 76 | uses: r-lib/actions/setup-pandoc@v2 77 | 78 | - name: "Setup Lesson Engine" 79 | uses: carpentries/actions/setup-sandpaper@main 80 | with: 81 | cache-version: ${{ secrets.CACHE_VERSION }} 82 | 83 | - name: "Setup Package Cache" 84 | uses: carpentries/actions/setup-lesson-deps@main 85 | with: 86 | cache-version: ${{ secrets.CACHE_VERSION }} 87 | 88 | - name: "Validate and Build Markdown" 89 | id: build-site 90 | run: | 91 | sandpaper::package_cache_trigger(TRUE) 92 | sandpaper::validate_lesson(path = '${{ github.workspace }}') 93 | sandpaper:::build_markdown(path = '${{ github.workspace }}', quiet = FALSE) 94 | shell: Rscript {0} 95 | 96 | - name: "Generate Artifacts" 97 | id: generate-artifacts 98 | run: | 99 | sandpaper:::ci_bundle_pr_artifacts( 100 | repo = '${{ github.repository }}', 101 | pr_number = '${{ github.event.number }}', 102 | path_md = '${{ env.MD }}', 103 | path_pr = '${{ env.PR }}', 104 | path_archive = '${{ env.CHIVE }}', 105 | branch = 'md-outputs' 106 | ) 107 | shell: Rscript {0} 108 | 109 | - name: "Upload PR" 110 | uses: actions/upload-artifact@v4 111 | with: 112 | name: pr 113 | path: ${{ env.PR }} 114 | overwrite: true 115 | 116 | - name: "Upload Diff" 117 | uses: actions/upload-artifact@v4 118 | with: 119 | name: diff 120 | path: ${{ env.CHIVE }} 121 | retention-days: 1 122 | 123 | - name: "Upload Build" 124 | uses: actions/upload-artifact@v4 125 | with: 126 | name: built 127 | path: ${{ env.MD }} 128 | retention-days: 1 129 | 130 | - name: "Teardown" 131 | run: sandpaper::reset_site() 132 | shell: Rscript {0} 133 | -------------------------------------------------------------------------------- /.github/workflows/update-cache.yaml: -------------------------------------------------------------------------------- 1 | name: "03 Maintain: Update Package Cache" 2 | 3 | on: 4 | workflow_dispatch: 5 | inputs: 6 | name: 7 | description: 'Who triggered this build (enter github username to tag yourself)?' 8 | required: true 9 | default: 'monthly run' 10 | schedule: 11 | # Run every tuesday 12 | - cron: '0 0 * * 2' 13 | 14 | jobs: 15 | preflight: 16 | name: "Preflight Check" 17 | runs-on: ubuntu-22.04 18 | outputs: 19 | ok: ${{ steps.check.outputs.ok }} 20 | steps: 21 | - id: check 22 | run: | 23 | if [[ ${{ github.event_name }} == 'workflow_dispatch' ]]; then 24 | echo "ok=true" >> $GITHUB_OUTPUT 25 | echo "Running on request" 26 | # using single brackets here to avoid 08 being interpreted as octal 27 | # https://github.com/carpentries/sandpaper/issues/250 28 | elif [ `date +%d` -le 7 ]; then 29 | # If the Tuesday lands in the first week of the month, run it 30 | echo "ok=true" >> $GITHUB_OUTPUT 31 | echo "Running on schedule" 32 | else 33 | echo "ok=false" >> $GITHUB_OUTPUT 34 | echo "Not Running Today" 35 | fi 36 | 37 | check_renv: 38 | name: "Check if We Need {renv}" 39 | runs-on: ubuntu-22.04 40 | needs: preflight 41 | if: ${{ needs.preflight.outputs.ok == 'true'}} 42 | outputs: 43 | needed: ${{ steps.renv.outputs.exists }} 44 | steps: 45 | - name: "Checkout Lesson" 46 | uses: actions/checkout@v4 47 | - id: renv 48 | run: | 49 | if [[ -d renv ]]; then 50 | echo "exists=true" >> $GITHUB_OUTPUT 51 | fi 52 | 53 | check_token: 54 | name: "Check SANDPAPER_WORKFLOW token" 55 | runs-on: ubuntu-22.04 56 | needs: check_renv 57 | if: ${{ needs.check_renv.outputs.needed == 'true' }} 58 | outputs: 59 | workflow: ${{ steps.validate.outputs.wf }} 60 | repo: ${{ steps.validate.outputs.repo }} 61 | steps: 62 | - name: "validate token" 63 | id: validate 64 | uses: carpentries/actions/check-valid-credentials@main 65 | with: 66 | token: ${{ secrets.SANDPAPER_WORKFLOW }} 67 | 68 | update_cache: 69 | name: "Update Package Cache" 70 | needs: check_token 71 | if: ${{ needs.check_token.outputs.repo== 'true' }} 72 | runs-on: ubuntu-22.04 73 | env: 74 | GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} 75 | RENV_PATHS_ROOT: ~/.local/share/renv/ 76 | steps: 77 | 78 | - name: "Checkout Lesson" 79 | uses: actions/checkout@v4 80 | 81 | - name: "Set up R" 82 | uses: r-lib/actions/setup-r@v2 83 | with: 84 | use-public-rspm: true 85 | install-r: false 86 | 87 | - name: "Update {renv} deps and determine if a PR is needed" 88 | id: update 89 | uses: carpentries/actions/update-lockfile@main 90 | with: 91 | cache-version: ${{ secrets.CACHE_VERSION }} 92 | 93 | - name: Create Pull Request 94 | id: cpr 95 | if: ${{ steps.update.outputs.n > 0 }} 96 | uses: carpentries/create-pull-request@main 97 | with: 98 | token: ${{ secrets.SANDPAPER_WORKFLOW }} 99 | delete-branch: true 100 | branch: "update/packages" 101 | commit-message: "[actions] update ${{ steps.update.outputs.n }} packages" 102 | title: "Update ${{ steps.update.outputs.n }} packages" 103 | body: | 104 | :robot: This is an automated build 105 | 106 | This will update ${{ steps.update.outputs.n }} packages in your lesson with the following versions: 107 | 108 | ``` 109 | ${{ steps.update.outputs.report }} 110 | ``` 111 | 112 | :stopwatch: In a few minutes, a comment will appear that will show you how the output has changed based on these updates. 113 | 114 | If you want to inspect these changes locally, you can use the following code to check out a new branch: 115 | 116 | ```bash 117 | git fetch origin update/packages 118 | git checkout update/packages 119 | ``` 120 | 121 | - Auto-generated by [create-pull-request][1] on ${{ steps.update.outputs.date }} 122 | 123 | [1]: https://github.com/carpentries/create-pull-request/tree/main 124 | labels: "type: package cache" 125 | draft: false 126 | -------------------------------------------------------------------------------- /instructors/instructor-notes.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Instructor Notes 3 | --- 4 | 5 | ## Before Teaching This Lesson 6 | 7 | *You are strongly advised to run through the lesson content prior to teaching 8 | the lesson to ensure that everything works as expected.* 9 | 10 | If you experience any issues, please [open an issue][open a lesson issue] in the lesson 11 | repository describing the problem and platform(s) affected. The lesson maintainers will 12 | aim to resolve the issue as soon as possible but we also welcome the opening 13 | of pull requests (linked to issues) that resolve anything that doesn't work as 14 | expected with the lesson content. 15 | 16 | ## Miscellaneous Tips 17 | 18 | - **Timing**: This lesson fits well in a single day, running from 1000-1530 with 2 hours of 19 | breaks. There is plenty of leeway in the material for questions from attendees. 20 | - **Exercises**: Some of the exercises at the start of the material work best in-person where 21 | people can get into small groups (max. 4 people) to discuss. 22 | - **Presentation slides**: A set of Powerpoint presentation slides are provided as part of the 23 | materials. We have found that using these rather than the Carpentries template material leads 24 | to a better experience for learners. 25 | - **Reflection Exercise**: At the beginning and end of the workshop, give participants time to 26 | reflect on what they want to get out of the workshop (at the beginning) and what they 27 | can apply to their work (at the end). Using the shared notes doc is a great way to 28 | do this and a good way to make sure that you've addressed specific concerns or goals 29 | of the participants. 30 | 31 | ## Presentation slides 32 | 33 | - [Introduction (pptx)](L1-Introduction.pptx) 34 | - [Carbon Efficiency (pptx)](L2-carbon-efficiency.pptx) 35 | - [Energy Efficiency (pptx)](L3-energy-efficiency.pptx) 36 | - [Carbon Awareness (pptx)](L4-carbon-awareness.pptx) 37 | - [Hardware Efficiency (pptx)](L5-hardware-efficiency.pptx) 38 | - [Measurement 1: Estimating Emissions (pptx)](L6-measurement-estimating-emissions.pptx) 39 | - [Measurement 2: HPC-CI (pptx)](L7-measurement-hpcci.pptx) 40 | - [Reducing Emissions (pptx)](L8-reducing-emissions.pptx) 41 | 42 | ## Learner Profiles and Pathways 43 | 44 | In this section we provide some details of example learner profiles to consider 45 | when teaching or planning a lesson based around this material. As such, 46 | the information in this section is not designed to define fixed approaches and 47 | structures for teaching this material. It is instead aimed to provide ideas 48 | and inspiration and to encourage you to think about your audience when 49 | preparing to teach this material. The information here is based on both 50 | discussions about the intended audiences for this material and on direct 51 | experiences of instructors who have taught it at workshops following different 52 | technical pathways. 53 | 54 | ### Learner profiles 55 | 56 | We begin by providing some example learner profiles to highlight the potential 57 | target audience and the types of different research and technical backgrounds 58 | that you may find among learners engaging with this material. With these 59 | profiles, we aim to encourage you to think about the learners attending your 60 | workshop(s) and which episodes it may be most useful to teach. 61 | 62 | Considering things from a higher level, we also highlight three core groups of 63 | learners, based on job roles, who you may find attending lessons covering this 64 | material. While recognising that there are likely to be many learners who 65 | don't fit into one of the following groups, or who span more than one of them, 66 | we hope that highlighting these groups helps to provide an example of the 67 | different types of skills and expertise that learners engaging with this 68 | material may have: 69 | 70 | - **Researchers:** 71 | 72 | Jane is a chemistry researcher who uses computational modelling in their 73 | projects as well as conducting experiments in a laboratory. They would like 74 | to know how to measure their emissions from HPC system use to compare to 75 | emissions from the lab work. They also want to know how they can potentially 76 | improve their emissions efficiency on HPC systems and estimate emissions for 77 | future funding applications. 78 | 79 | - **RSEs:** 80 | 81 | Jian is a research software engineer working on a mature research software 82 | codebase that is widely used on HPC systems worldwide. He wants to be able 83 | to assess the emissions associted with the software use on different HPC 84 | architectures and in different locations to understand how these different 85 | aspects affect emissions reduction from using the software. He also wants to 86 | understand if he should focus on energy efficiency or performance efficiency 87 | to reduce emissions from using the software on HPC systems. 88 | 89 | - **Systems procurement teams:** 90 | 91 | Maria works for a funding body and is involved in procuring new HPC systems 92 | for use by academic researchers. She is keen to maximise the emissions 93 | efficiency of new HPC systems that she is involved in procuring. She wants 94 | to improve her understanding of carbon emissions arising from HPC systems and 95 | learn how HPC systems can be setup and run to reduce emissions. 96 | 97 | 98 | 99 | 100 | 101 | 102 | -------------------------------------------------------------------------------- /bin/repo_check.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | """ 4 | Check repository settings. 5 | """ 6 | 7 | 8 | import sys 9 | import os 10 | from subprocess import Popen, PIPE 11 | import re 12 | from argparse import ArgumentParser 13 | 14 | from util import Reporter, require 15 | 16 | # Import this way to produce a more useful error message. 17 | try: 18 | import requests 19 | except ImportError: 20 | print('Unable to import requests module: please install requests', file=sys.stderr) 21 | sys.exit(1) 22 | 23 | 24 | # Pattern to match Git command-line output for remotes => (user name, project name). 25 | P_GIT_REMOTE = re.compile(r'upstream\s+[^:]+:([^/]+)/([^.]+)\.git\s+\(fetch\)') 26 | 27 | # Repository URL format string. 28 | F_REPO_URL = 'https://github.com/{0}/{1}/' 29 | 30 | # Pattern to match repository URLs => (user name, project name) 31 | P_REPO_URL = re.compile(r'https?://github\.com/([^.]+)/([^/]+)/?') 32 | 33 | # API URL format string. 34 | F_API_URL = 'https://api.github.com/repos/{0}/{1}/labels' 35 | 36 | # Expected labels and colors. 37 | EXPECTED = { 38 | 'help wanted': 'dcecc7', 39 | 'status:in progress': '9bcc65', 40 | 'status:changes requested': '679f38', 41 | 'status:wait': 'fff2df', 42 | 'status:refer to cac': 'ffdfb2', 43 | 'status:need more info': 'ee6c00', 44 | 'status:blocked': 'e55100', 45 | 'status:out of scope': 'eeeeee', 46 | 'status:duplicate': 'bdbdbd', 47 | 'type:typo text': 'f8bad0', 48 | 'type:bug': 'eb3f79', 49 | 'type:formatting': 'ac1357', 50 | 'type:template and tools': '7985cb', 51 | 'type:instructor guide': '00887a', 52 | 'type:discussion': 'b2e5fc', 53 | 'type:enhancement': '7fdeea', 54 | 'type:clarification': '00acc0', 55 | 'type:teaching example': 'ced8dc', 56 | 'good first issue': 'ffeb3a', 57 | 'high priority': 'd22e2e' 58 | } 59 | 60 | 61 | def main(): 62 | """ 63 | Main driver. 64 | """ 65 | 66 | args = parse_args() 67 | reporter = Reporter() 68 | repo_url = get_repo_url(args.repo_url) 69 | check_labels(reporter, repo_url) 70 | reporter.report() 71 | 72 | 73 | def parse_args(): 74 | """ 75 | Parse command-line arguments. 76 | """ 77 | 78 | parser = ArgumentParser(description="""Check repository settings.""") 79 | parser.add_argument('-r', '--repo', 80 | default=None, 81 | dest='repo_url', 82 | help='repository URL') 83 | parser.add_argument('-s', '--source', 84 | default=os.curdir, 85 | dest='source_dir', 86 | help='source directory') 87 | 88 | args, extras = parser.parse_known_args() 89 | require(not extras, 90 | 'Unexpected trailing command-line arguments "{0}"'.format(extras)) 91 | 92 | return args 93 | 94 | 95 | def get_repo_url(repo_url): 96 | """ 97 | Figure out which repository to query. 98 | """ 99 | 100 | # Explicitly specified. 101 | if repo_url is not None: 102 | return repo_url 103 | 104 | # Guess. 105 | cmd = 'git remote -v' 106 | p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, 107 | close_fds=True, universal_newlines=True) 108 | stdout_data, stderr_data = p.communicate() 109 | stdout_data = stdout_data.split('\n') 110 | matches = [P_GIT_REMOTE.match(line) for line in stdout_data] 111 | matches = [m for m in matches if m is not None] 112 | require(len(matches) == 1, 113 | 'Unexpected output from git remote command: "{0}"'.format(matches)) 114 | 115 | username = matches[0].group(1) 116 | require( 117 | username, 'empty username in git remote output {0}'.format(matches[0])) 118 | 119 | project_name = matches[0].group(2) 120 | require( 121 | username, 'empty project name in git remote output {0}'.format(matches[0])) 122 | 123 | url = F_REPO_URL.format(username, project_name) 124 | return url 125 | 126 | 127 | def check_labels(reporter, repo_url): 128 | """ 129 | Check labels in repository. 130 | """ 131 | 132 | actual = get_labels(repo_url) 133 | extra = set(actual.keys()) - set(EXPECTED.keys()) 134 | 135 | reporter.check(not extra, 136 | None, 137 | 'Extra label(s) in repository {0}: {1}', 138 | repo_url, ', '.join(sorted(extra))) 139 | 140 | missing = set(EXPECTED.keys()) - set(actual.keys()) 141 | reporter.check(not missing, 142 | None, 143 | 'Missing label(s) in repository {0}: {1}', 144 | repo_url, ', '.join(sorted(missing))) 145 | 146 | overlap = set(EXPECTED.keys()).intersection(set(actual.keys())) 147 | for name in sorted(overlap): 148 | reporter.check(EXPECTED[name].lower() == actual[name].lower(), 149 | None, 150 | 'Color mis-match for label {0} in {1}: expected {2}, found {3}', 151 | name, repo_url, EXPECTED[name], actual[name]) 152 | 153 | 154 | def get_labels(repo_url): 155 | """ 156 | Get actual labels from repository. 157 | """ 158 | 159 | m = P_REPO_URL.match(repo_url) 160 | require( 161 | m, 'repository URL {0} does not match expected pattern'.format(repo_url)) 162 | 163 | username = m.group(1) 164 | require(username, 'empty username in repository URL {0}'.format(repo_url)) 165 | 166 | project_name = m.group(2) 167 | require( 168 | username, 'empty project name in repository URL {0}'.format(repo_url)) 169 | 170 | url = F_API_URL.format(username, project_name) 171 | r = requests.get(url) 172 | require(r.status_code == 200, 173 | 'Request for {0} failed with {1}'.format(url, r.status_code)) 174 | 175 | result = {} 176 | for entry in r.json(): 177 | result[entry['name']] = entry['color'] 178 | return result 179 | 180 | 181 | if __name__ == '__main__': 182 | main() 183 | -------------------------------------------------------------------------------- /bin/util.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import os 3 | import json 4 | from subprocess import Popen, PIPE 5 | 6 | # Import this way to produce a more useful error message. 7 | try: 8 | import yaml 9 | except ImportError: 10 | print('Unable to import YAML module: please install PyYAML', file=sys.stderr) 11 | sys.exit(1) 12 | 13 | 14 | # Things an image file's name can end with. 15 | IMAGE_FILE_SUFFIX = { 16 | '.gif', 17 | '.jpg', 18 | '.png', 19 | '.svg' 20 | } 21 | 22 | # Files that shouldn't be present. 23 | UNWANTED_FILES = [ 24 | '.nojekyll' 25 | ] 26 | 27 | # Marker to show that an expected value hasn't been provided. 28 | # (Can't use 'None' because that might be a legitimate value.) 29 | REPORTER_NOT_SET = [] 30 | 31 | 32 | class Reporter: 33 | """Collect and report errors.""" 34 | 35 | def __init__(self): 36 | """Constructor.""" 37 | self.messages = [] 38 | 39 | def check_field(self, filename, name, values, key, expected=REPORTER_NOT_SET): 40 | """Check that a dictionary has an expected value.""" 41 | 42 | if key not in values: 43 | self.add(filename, '{0} does not contain {1}', name, key) 44 | elif expected is REPORTER_NOT_SET: 45 | pass 46 | elif type(expected) in (tuple, set, list): 47 | if values[key] not in expected: 48 | self.add( 49 | filename, '{0} {1} value {2} is not in {3}', name, key, values[key], expected) 50 | elif values[key] != expected: 51 | self.add(filename, '{0} {1} is {2} not {3}', 52 | name, key, values[key], expected) 53 | 54 | def check(self, condition, location, fmt, *args): 55 | """Append error if condition not met.""" 56 | 57 | if not condition: 58 | self.add(location, fmt, *args) 59 | 60 | def add(self, location, fmt, *args): 61 | """Append error unilaterally.""" 62 | 63 | self.messages.append((location, fmt.format(*args))) 64 | 65 | @staticmethod 66 | def pretty(item): 67 | location, message = item 68 | if isinstance(location, type(None)): 69 | return message 70 | elif isinstance(location, str): 71 | return location + ': ' + message 72 | elif isinstance(location, tuple): 73 | return '{0}:{1}: '.format(*location) + message 74 | 75 | print('Unknown item "{0}"'.format(item), file=sys.stderr) 76 | return NotImplemented 77 | 78 | @staticmethod 79 | def key(item): 80 | location, message = item 81 | if isinstance(location, type(None)): 82 | return ('', -1, message) 83 | elif isinstance(location, str): 84 | return (location, -1, message) 85 | elif isinstance(location, tuple): 86 | return (location[0], location[1], message) 87 | 88 | print('Unknown item "{0}"'.format(item), file=sys.stderr) 89 | return NotImplemented 90 | 91 | def report(self, stream=sys.stdout): 92 | """Report all messages in order.""" 93 | 94 | if not self.messages: 95 | return 96 | 97 | for m in sorted(self.messages, key=self.key): 98 | print(self.pretty(m), file=stream) 99 | 100 | 101 | def read_markdown(parser, path): 102 | """ 103 | Get YAML and AST for Markdown file, returning 104 | {'metadata':yaml, 'metadata_len':N, 'text':text, 'lines':[(i, line, len)], 'doc':doc}. 105 | """ 106 | 107 | # Split and extract YAML (if present). 108 | with open(path, 'r') as reader: 109 | body = reader.read() 110 | metadata_raw, metadata_yaml, body = split_metadata(path, body) 111 | 112 | # Split into lines. 113 | metadata_len = 0 if metadata_raw is None else metadata_raw.count('\n') 114 | lines = [(metadata_len+i+1, line, len(line)) 115 | for (i, line) in enumerate(body.split('\n'))] 116 | 117 | # Parse Markdown. 118 | cmd = 'ruby {0}'.format(parser) 119 | p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, 120 | close_fds=True, universal_newlines=True) 121 | stdout_data, stderr_data = p.communicate(body) 122 | doc = json.loads(stdout_data) 123 | 124 | return { 125 | 'metadata': metadata_yaml, 126 | 'metadata_len': metadata_len, 127 | 'text': body, 128 | 'lines': lines, 129 | 'doc': doc 130 | } 131 | 132 | 133 | def split_metadata(path, text): 134 | """ 135 | Get raw (text) metadata, metadata as YAML, and rest of body. 136 | If no metadata, return (None, None, body). 137 | """ 138 | 139 | metadata_raw = None 140 | metadata_yaml = None 141 | 142 | pieces = text.split('---', 2) 143 | if len(pieces) == 3: 144 | metadata_raw = pieces[1] 145 | text = pieces[2] 146 | try: 147 | metadata_yaml = yaml.load(metadata_raw, Loader=yaml.FullLoader) 148 | except yaml.YAMLError as e: 149 | print('Unable to parse YAML header in {0}:\n{1}'.format( 150 | path, e), file=sys.stderr) 151 | sys.exit(1) 152 | 153 | return metadata_raw, metadata_yaml, text 154 | 155 | 156 | def load_yaml(filename): 157 | """ 158 | Wrapper around YAML loading so that 'import yaml' is only needed 159 | in one file. 160 | """ 161 | 162 | try: 163 | with open(filename, 'r') as reader: 164 | return yaml.load(reader, Loader=yaml.FullLoader) 165 | except (yaml.YAMLError, IOError) as e: 166 | print('Unable to load YAML file {0}:\n{1}'.format( 167 | filename, e), file=sys.stderr) 168 | sys.exit(1) 169 | 170 | 171 | def check_unwanted_files(dir_path, reporter): 172 | """ 173 | Check that unwanted files are not present. 174 | """ 175 | 176 | for filename in UNWANTED_FILES: 177 | path = os.path.join(dir_path, filename) 178 | reporter.check(not os.path.exists(path), 179 | path, 180 | "Unwanted file found") 181 | 182 | 183 | def require(condition, message): 184 | """Fail if condition not met.""" 185 | 186 | if not condition: 187 | print(message, file=sys.stderr) 188 | sys.exit(1) 189 | -------------------------------------------------------------------------------- /episodes/introduction.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Introduction 3 | teaching: 10 4 | exercises: 5 5 | --- 6 | 7 | ::::::::::::::::::::::::::::::::::::::: objectives 8 | 9 | - Define green HPC system use. 10 | - Appreciate emissions from HPC in the wider context of other research activities. 11 | 12 | :::::::::::::::::::::::::::::::::::::::::::::::::: 13 | 14 | :::::::::::::::::::::::::::::::::::::::: questions 15 | 16 | - What is green HPC system use and how can we approach it? 17 | - What research activities produce carbon emissions? 18 | 19 | :::::::::::::::::::::::::::::::::::::::::::::::::: 20 | 21 | ## What is green HPC system use? 22 | 23 | Green HPC system use is carbon-efficient HPC system use, meaning it emits the least carbon possible. Only three activities reduce the carbon emissions associated with HPC system use: energy efficiency, carbon awareness, and hardware efficiency. 24 | 25 | This workshop will explain all of these concepts, how to apply them to use cases on high-performance computing (HPC) systems and how to measure them, as well as some of the international guidelines and organisations that guide and monitor this space. 26 | 27 | ![Diagram illustrating three activities that can reduce carbon emissions from software use](./fig/01_carbon_efficiency.png "image_tooltip") 28 | 29 | 30 | ## How to understand green HPC system use 31 | 32 | In this workshop, we cover the covers 6 key areas of green use of HPC systems (based on the principles of green software engineering): 33 | 34 | 1. **Carbon Efficiency**: Emit the least amount of carbon possible. 35 | 2. **Energy Efficiency**: Use the least amount of energy possible. 36 | 3. **Carbon Awareness**: Do more when the electricity is cleaner and do less when the electricity is dirtier. 37 | 4. **Hardware Efficiency**: Use the least amount of embodied carbon possible. 38 | 5. **Measurement**: What you cannot measure, you cannot improve. 39 | 6. **Reducing Emissions**: Understand mechanisms of carbon reduction. 40 | 41 | Each of these episodes will introduce some new concepts and explain in detail why they are important in terms of the climate, and how you can apply them on HPC systems. We will use the [ARCHER2 UK National Supercomputing Serivce](https://www.archer2.ac.uk) to provide concrete examples of how the principles can be applied. 42 | 43 | ## Other work activities 44 | 45 | Use of HPC will be only one part of your activities associated with your work and may not even be the largest source of carbon emissions (though this does not mean you should not work to make your use of HPC more carbon efficient!). As well as understanding carbon emissions from HPC, which this workshop will help you do, you also need to assess other activities you undertake as part of your work and try to estimate emissions associated with them. 46 | 47 | ::::::::::::::::::::::::::::::::::::::: challenge 48 | 49 | ## Exercise: What does your work look like? 50 | 51 | Write down the activities that you do as part of your work or research that could be a source of carbon emissions other than from HPC systems. Once you have them written down, can you rank them in order of what you think the largest source of emissions will be to the smallest? 52 | 53 | ::::::::::::::: solution 54 | 55 | ## Solution 56 | 57 | You may have come up with activities from the following list: 58 | 59 | - Travel to/from place of work 60 | - Travel to/from and accommodation at conferences, training and meetings 61 | - Use of other computing resources (e.g. laptops/workstations, cloud resources and the internet) 62 | - Data storage 63 | - Laboratory work - chemicals, instruments 64 | - Printed materials - books, journal articles 65 | 66 | Ranking them in order is more difficult and will differ from person to person. Typically, any sort of flying to conferences or meetings will be the most carbon intensive activity you can undertake through your work. Whether it is the largest contribution depends on how often you fly. If you work in a laboratory, that may well be the next highest source of emissions though if you commute a long way and use a petrol/diesel car that may also be one of the highest sources of emissions. 67 | 68 | ::::::::::::::::::::::::: 69 | 70 | :::::::::::::::::::::::::::::::::::::::::::::::::: 71 | 72 | In fact, **all** activities that we undertake as part of our work (and personal life) produce carbon emissions. While, in an ideal world, we would be able to reduce emissions from all our activities we have finite time and effort so need to prioritise based on which reduction strategies will have the largest impact. This means that the first step is to estimate the emissions from different activities we are involved in as part of our work, even if the estimate is very rough. The links below can help to calculate your emissions: 73 | 74 | - [Travel emissions calculator (Deloitte UK)](https://www.deloitte.com/uk/en/issues/climate/content/travel-emissions-calculator.html) 75 | - [Emissions from laptops (Circular Computing)](https://circularcomputing.com/news/carbon-footprint-laptop/) 76 | - [Transparent reporting of research-related greenhouse gas emissions through the scientific CO2nduct initiative](https://www.nature.com/articles/s42005-022-00930-2) 77 | - [Quantifying the carbon footprint of clinical trials: guidance development and case studies](https://pmc.ncbi.nlm.nih.gov/articles/PMC10823997/) 78 | - [Emissions measurement and reporting approaches for the public sector (UK Government)](https://assets.publishing.service.gov.uk/media/66fa76de30536cb927482953/emissions-measurement-and-reporting-approaches-for-the-public-sector-report.pdf) 79 | - [Carbon footprint and mitigation strategies of three chemistry laboratories](https://pubs.rsc.org/en/content/articlehtml/2024/gc/d3gc03668e) 80 | 81 | 82 | 84 | 85 | :::::::::::::::::::::::::::::::::::::::: keypoints 86 | 87 | - This course covers green HPC system use, in particular: 88 | - **Carbon Efficiency**: Emit the least amount of carbon possible. 89 | - **Energy Efficiency**: Use the least amount of energy possible. 90 | - **Carbon Awareness**: Do more when the electricity is cleaner and do less when the electricity is dirtier. 91 | - **Hardware Efficiency**: Use the least amount of embodied carbon possible. 92 | - **Measurement**: What you cannot measure, you cannot improve. 93 | - **Reducing Emissions**: Understand mechanisms of carbon reduction. 94 | 95 | :::::::::::::::::::::::::::::::::::::::::::::::::: 96 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | ## Contributing 2 | 3 | [The Carpentries][cp-site] ([Software Carpentry][swc-site], [Data 4 | Carpentry][dc-site], and [Library Carpentry][lc-site]) are open source 5 | projects, and we welcome contributions of all kinds: new lessons, fixes to 6 | existing material, bug reports, and reviews of proposed changes are all 7 | welcome. 8 | 9 | ### Contributor Agreement 10 | 11 | By contributing, you agree that we may redistribute your work under [our 12 | license](LICENSE.md). In exchange, we will address your issues and/or assess 13 | your change proposal as promptly as we can, and help you become a member of our 14 | community. Everyone involved in [The Carpentries][cp-site] agrees to abide by 15 | our [code of conduct](CODE_OF_CONDUCT.md). 16 | 17 | ### How to Contribute 18 | 19 | The easiest way to get started is to file an issue to tell us about a spelling 20 | mistake, some awkward wording, or a factual error. This is a good way to 21 | introduce yourself and to meet some of our community members. 22 | 23 | 1. If you do not have a [GitHub][github] account, you can [send us comments by 24 | email][contact]. However, we will be able to respond more quickly if you use 25 | one of the other methods described below. 26 | 27 | 2. If you have a [GitHub][github] account, or are willing to [create 28 | one][github-join], but do not know how to use Git, you can report problems 29 | or suggest improvements by [creating an issue][repo-issues]. This allows us 30 | to assign the item to someone and to respond to it in a threaded discussion. 31 | 32 | 3. If you are comfortable with Git, and would like to add or change material, 33 | you can submit a pull request (PR). Instructions for doing this are 34 | [included below](#using-github). For inspiration about changes that need to 35 | be made, check out the [list of open issues][issues] across the Carpentries. 36 | 37 | Note: if you want to build the website locally, please refer to [The Workbench 38 | documentation][template-doc]. 39 | 40 | ### Where to Contribute 41 | 42 | 1. If you wish to change this lesson, add issues and pull requests here. 43 | 2. If you wish to change the template used for workshop websites, please refer 44 | to [The Workbench documentation][template-doc]. 45 | 46 | 47 | ### What to Contribute 48 | 49 | There are many ways to contribute, from writing new exercises and improving 50 | existing ones to updating or filling in the documentation and submitting [bug 51 | reports][issues] about things that do not work, are not clear, or are missing. 52 | If you are looking for ideas, please see [the list of issues for this 53 | repository][repo-issues], or the issues for [Data Carpentry][dc-issues], 54 | [Library Carpentry][lc-issues], and [Software Carpentry][swc-issues] projects. 55 | 56 | Comments on issues and reviews of pull requests are just as welcome: we are 57 | smarter together than we are on our own. **Reviews from novices and newcomers 58 | are particularly valuable**: it's easy for people who have been using these 59 | lessons for a while to forget how impenetrable some of this material can be, so 60 | fresh eyes are always welcome. 61 | 62 | ### What *Not* to Contribute 63 | 64 | Our lessons already contain more material than we can cover in a typical 65 | workshop, so we are usually *not* looking for more concepts or tools to add to 66 | them. As a rule, if you want to introduce a new idea, you must (a) estimate how 67 | long it will take to teach and (b) explain what you would take out to make room 68 | for it. The first encourages contributors to be honest about requirements; the 69 | second, to think hard about priorities. 70 | 71 | We are also not looking for exercises or other material that only run on one 72 | platform. Our workshops typically contain a mixture of Windows, macOS, and 73 | Linux users; in order to be usable, our lessons must run equally well on all 74 | three. 75 | 76 | ### Using GitHub 77 | 78 | If you choose to contribute via GitHub, you may want to look at [How to 79 | Contribute to an Open Source Project on GitHub][how-contribute]. In brief, we 80 | use [GitHub flow][github-flow] to manage changes: 81 | 82 | 1. Create a new branch in your desktop copy of this repository for each 83 | significant change. 84 | 2. Commit the change in that branch. 85 | 3. Push that branch to your fork of this repository on GitHub. 86 | 4. Submit a pull request from that branch to the [upstream repository][repo]. 87 | 5. If you receive feedback, make changes on your desktop and push to your 88 | branch on GitHub: the pull request will update automatically. 89 | 90 | NB: The published copy of the lesson is usually in the `main` branch. 91 | 92 | Each lesson has a team of maintainers who review issues and pull requests or 93 | encourage others to do so. The maintainers are community volunteers, and have 94 | final say over what gets merged into the lesson. 95 | 96 | ### Other Resources 97 | 98 | The Carpentries is a global organisation with volunteers and learners all over 99 | the world. We share values of inclusivity and a passion for sharing knowledge, 100 | teaching and learning. There are several ways to connect with The Carpentries 101 | community listed at including via social 102 | media, slack, newsletters, and email lists. You can also [reach us by 103 | email][contact]. 104 | 105 | [repo]: https://example.com/FIXME 106 | [repo-issues]: https://example.com/FIXME/issues 107 | [contact]: mailto:team@carpentries.org 108 | [cp-site]: https://carpentries.org/ 109 | [dc-issues]: https://github.com/issues?q=user%3Adatacarpentry 110 | [dc-lessons]: https://datacarpentry.org/lessons/ 111 | [dc-site]: https://datacarpentry.org/ 112 | [discuss-list]: https://carpentries.topicbox.com/groups/discuss 113 | [github]: https://github.com 114 | [github-flow]: https://guides.github.com/introduction/flow/ 115 | [github-join]: https://github.com/join 116 | [how-contribute]: https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github 117 | [issues]: https://carpentries.org/help-wanted-issues/ 118 | [lc-issues]: https://github.com/issues?q=user%3ALibraryCarpentry 119 | [swc-issues]: https://github.com/issues?q=user%3Aswcarpentry 120 | [swc-lessons]: https://software-carpentry.org/lessons/ 121 | [swc-site]: https://software-carpentry.org/ 122 | [lc-site]: https://librarycarpentry.org/ 123 | [template-doc]: https://carpentries.github.io/workbench/ 124 | -------------------------------------------------------------------------------- /.github/workflows/pr-comment.yaml: -------------------------------------------------------------------------------- 1 | name: "Bot: Comment on the Pull Request" 2 | 3 | # read-write repo token 4 | # access to secrets 5 | on: 6 | workflow_run: 7 | workflows: ["Receive Pull Request"] 8 | types: 9 | - completed 10 | 11 | concurrency: 12 | group: pr-${{ github.event.workflow_run.pull_requests[0].number }} 13 | cancel-in-progress: true 14 | 15 | 16 | jobs: 17 | # Pull requests are valid if: 18 | # - they match the sha of the workflow run head commit 19 | # - they are open 20 | # - no .github files were committed 21 | test-pr: 22 | name: "Test if pull request is valid" 23 | runs-on: ubuntu-22.04 24 | if: > 25 | github.event.workflow_run.event == 'pull_request' && 26 | github.event.workflow_run.conclusion == 'success' 27 | outputs: 28 | is_valid: ${{ steps.check-pr.outputs.VALID }} 29 | payload: ${{ steps.check-pr.outputs.payload }} 30 | number: ${{ steps.get-pr.outputs.NUM }} 31 | msg: ${{ steps.check-pr.outputs.MSG }} 32 | steps: 33 | - name: 'Download PR artifact' 34 | id: dl 35 | uses: carpentries/actions/download-workflow-artifact@main 36 | with: 37 | run: ${{ github.event.workflow_run.id }} 38 | name: 'pr' 39 | 40 | - name: "Get PR Number" 41 | if: ${{ steps.dl.outputs.success == 'true' }} 42 | id: get-pr 43 | run: | 44 | unzip pr.zip 45 | echo "NUM=$(<./NR)" >> $GITHUB_OUTPUT 46 | 47 | - name: "Fail if PR number was not present" 48 | id: bad-pr 49 | if: ${{ steps.dl.outputs.success != 'true' }} 50 | run: | 51 | echo '::error::A pull request number was not recorded. The pull request that triggered this workflow is likely malicious.' 52 | exit 1 53 | - name: "Get Invalid Hashes File" 54 | id: hash 55 | run: | 56 | echo "json<> $GITHUB_OUTPUT 59 | - name: "Check PR" 60 | id: check-pr 61 | if: ${{ steps.dl.outputs.success == 'true' }} 62 | uses: carpentries/actions/check-valid-pr@main 63 | with: 64 | pr: ${{ steps.get-pr.outputs.NUM }} 65 | sha: ${{ github.event.workflow_run.head_sha }} 66 | headroom: 3 # if it's within the last three commits, we can keep going, because it's likely rapid-fire 67 | invalid: ${{ fromJSON(steps.hash.outputs.json)[github.repository] }} 68 | fail_on_error: true 69 | 70 | # Create an orphan branch on this repository with two commits 71 | # - the current HEAD of the md-outputs branch 72 | # - the output from running the current HEAD of the pull request through 73 | # the md generator 74 | create-branch: 75 | name: "Create Git Branch" 76 | needs: test-pr 77 | runs-on: ubuntu-22.04 78 | if: ${{ needs.test-pr.outputs.is_valid == 'true' }} 79 | env: 80 | NR: ${{ needs.test-pr.outputs.number }} 81 | permissions: 82 | contents: write 83 | steps: 84 | - name: 'Checkout md outputs' 85 | uses: actions/checkout@v4 86 | with: 87 | ref: md-outputs 88 | path: built 89 | fetch-depth: 1 90 | 91 | - name: 'Download built markdown' 92 | id: dl 93 | uses: carpentries/actions/download-workflow-artifact@main 94 | with: 95 | run: ${{ github.event.workflow_run.id }} 96 | name: 'built' 97 | 98 | - if: ${{ steps.dl.outputs.success == 'true' }} 99 | run: unzip built.zip 100 | 101 | - name: "Create orphan and push" 102 | if: ${{ steps.dl.outputs.success == 'true' }} 103 | run: | 104 | cd built/ 105 | git config --local user.email "actions@github.com" 106 | git config --local user.name "GitHub Actions" 107 | CURR_HEAD=$(git rev-parse HEAD) 108 | git checkout --orphan md-outputs-PR-${NR} 109 | git add -A 110 | git commit -m "source commit: ${CURR_HEAD}" 111 | ls -A | grep -v '^.git$' | xargs -I _ rm -r '_' 112 | cd .. 113 | unzip -o -d built built.zip 114 | cd built 115 | git add -A 116 | git commit --allow-empty -m "differences for PR #${NR}" 117 | git push -u --force --set-upstream origin md-outputs-PR-${NR} 118 | 119 | # Comment on the Pull Request with a link to the branch and the diff 120 | comment-pr: 121 | name: "Comment on Pull Request" 122 | needs: [test-pr, create-branch] 123 | runs-on: ubuntu-22.04 124 | if: ${{ needs.test-pr.outputs.is_valid == 'true' }} 125 | env: 126 | NR: ${{ needs.test-pr.outputs.number }} 127 | permissions: 128 | pull-requests: write 129 | steps: 130 | - name: 'Download comment artifact' 131 | id: dl 132 | uses: carpentries/actions/download-workflow-artifact@main 133 | with: 134 | run: ${{ github.event.workflow_run.id }} 135 | name: 'diff' 136 | 137 | - if: ${{ steps.dl.outputs.success == 'true' }} 138 | run: unzip ${{ github.workspace }}/diff.zip 139 | 140 | - name: "Comment on PR" 141 | id: comment-diff 142 | if: ${{ steps.dl.outputs.success == 'true' }} 143 | uses: carpentries/actions/comment-diff@main 144 | with: 145 | pr: ${{ env.NR }} 146 | path: ${{ github.workspace }}/diff.md 147 | 148 | # Comment if the PR is open and matches the SHA, but the workflow files have 149 | # changed 150 | comment-changed-workflow: 151 | name: "Comment if workflow files have changed" 152 | needs: test-pr 153 | runs-on: ubuntu-22.04 154 | if: ${{ always() && needs.test-pr.outputs.is_valid == 'false' }} 155 | env: 156 | NR: ${{ github.event.workflow_run.pull_requests[0].number }} 157 | body: ${{ needs.test-pr.outputs.msg }} 158 | permissions: 159 | pull-requests: write 160 | steps: 161 | - name: 'Check for spoofing' 162 | id: dl 163 | uses: carpentries/actions/download-workflow-artifact@main 164 | with: 165 | run: ${{ github.event.workflow_run.id }} 166 | name: 'built' 167 | 168 | - name: 'Alert if spoofed' 169 | id: spoof 170 | if: ${{ steps.dl.outputs.success == 'true' }} 171 | run: | 172 | echo 'body<> $GITHUB_ENV 173 | echo '' >> $GITHUB_ENV 174 | echo '## :x: DANGER :x:' >> $GITHUB_ENV 175 | echo 'This pull request has modified workflows that created output. Close this now.' >> $GITHUB_ENV 176 | echo '' >> $GITHUB_ENV 177 | echo 'EOF' >> $GITHUB_ENV 178 | 179 | - name: "Comment on PR" 180 | id: comment-diff 181 | uses: carpentries/actions/comment-diff@main 182 | with: 183 | pr: ${{ env.NR }} 184 | body: ${{ env.body }} 185 | -------------------------------------------------------------------------------- /bin/boilerplate/CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | [The Carpentries][c-site] ([Software Carpentry][swc-site], [Data Carpentry][dc-site], and [Library Carpentry][lc-site]) are open source projects, 4 | and we welcome contributions of all kinds: 5 | new lessons, 6 | fixes to existing material, 7 | bug reports, 8 | and reviews of proposed changes are all welcome. 9 | 10 | ## Contributor Agreement 11 | 12 | By contributing, 13 | you agree that we may redistribute your work under [our license](LICENSE.md). 14 | In exchange, 15 | we will address your issues and/or assess your change proposal as promptly as we can, 16 | and help you become a member of our community. 17 | Everyone involved in [The Carpentries][c-site] 18 | agrees to abide by our [code of conduct](CODE_OF_CONDUCT.md). 19 | 20 | ## How to Contribute 21 | 22 | The easiest way to get started is to file an issue 23 | to tell us about a spelling mistake, 24 | some awkward wording, 25 | or a factual error. 26 | This is a good way to introduce yourself 27 | and to meet some of our community members. 28 | 29 | 1. If you do not have a [GitHub][github] account, 30 | you can [send us comments by email][email]. 31 | However, 32 | we will be able to respond more quickly if you use one of the other methods described below. 33 | 34 | 2. If you have a [GitHub][github] account, 35 | or are willing to [create one][github-join], 36 | but do not know how to use Git, 37 | you can report problems or suggest improvements by [creating an issue][issues]. 38 | This allows us to assign the item to someone 39 | and to respond to it in a threaded discussion. 40 | 41 | 3. If you are comfortable with Git, 42 | and would like to add or change material, 43 | you can submit a pull request (PR). 44 | Instructions for doing this are [included below](#using-github). 45 | 46 | ## Where to Contribute 47 | 48 | 1. If you wish to change this lesson, 49 | please work in , 50 | which can be viewed at . 51 | 52 | 2. If you wish to change the example lesson, 53 | please work in , 54 | which documents the format of our lessons 55 | and can be viewed at . 56 | 57 | 3. If you wish to change the template used for workshop websites, 58 | please work in . 59 | The home page of that repository explains how to set up workshop websites, 60 | while the extra pages in 61 | provide more background on our design choices. 62 | 63 | 4. If you wish to change CSS style files, tools, 64 | or HTML boilerplate for lessons or workshops stored in `_includes` or `_layouts`, 65 | please work in . 66 | 67 | ## What to Contribute 68 | 69 | There are many ways to contribute, 70 | from writing new exercises and improving existing ones 71 | to updating or filling in the documentation 72 | and submitting [bug reports][issues] 73 | about things that don't work, aren't clear, or are missing. 74 | If you are looking for ideas, please see the 'Issues' tab for 75 | a list of issues associated with this repository, 76 | or you may also look at the issues for [Data Carpentry][dc-issues], 77 | [Software Carpentry][swc-issues], and [Library Carpentry][lc-issues] projects. 78 | 79 | Comments on issues and reviews of pull requests are just as welcome: 80 | we are smarter together than we are on our own. 81 | Reviews from novices and newcomers are particularly valuable: 82 | it's easy for people who have been using these lessons for a while 83 | to forget how impenetrable some of this material can be, 84 | so fresh eyes are always welcome. 85 | 86 | ## What *Not* to Contribute 87 | 88 | Our lessons already contain more material than we can cover in a typical workshop, 89 | so we are usually *not* looking for more concepts or tools to add to them. 90 | As a rule, 91 | if you want to introduce a new idea, 92 | you must (a) estimate how long it will take to teach 93 | and (b) explain what you would take out to make room for it. 94 | The first encourages contributors to be honest about requirements; 95 | the second, to think hard about priorities. 96 | 97 | We are also not looking for exercises or other material that only run on one platform. 98 | Our workshops typically contain a mixture of Windows, macOS, and Linux users; 99 | in order to be usable, 100 | our lessons must run equally well on all three. 101 | 102 | ## Using GitHub 103 | 104 | If you choose to contribute via GitHub, you may want to look at 105 | [How to Contribute to an Open Source Project on GitHub][how-contribute]. 106 | To manage changes, we follow [GitHub flow][github-flow]. 107 | Each lesson has two maintainers who review issues and pull requests or encourage others to do so. 108 | The maintainers are community volunteers and have final say over what gets merged into the lesson. 109 | To use the web interface for contributing to a lesson: 110 | 111 | 1. Fork the originating repository to your GitHub profile. 112 | 2. Within your version of the forked repository, move to the `gh-pages` branch and 113 | create a new branch for each significant change being made. 114 | 3. Navigate to the file(s) you wish to change within the new branches and make revisions as required. 115 | 4. Commit all changed files within the appropriate branches. 116 | 5. Create individual pull requests from each of your changed branches 117 | to the `gh-pages` branch within the originating repository. 118 | 6. If you receive feedback, make changes using your issue-specific branches of the forked 119 | repository and the pull requests will update automatically. 120 | 7. Repeat as needed until all feedback has been addressed. 121 | 122 | When starting work, please make sure your clone of the originating `gh-pages` branch is up-to-date 123 | before creating your own revision-specific branch(es) from there. 124 | Additionally, please only work from your newly-created branch(es) and *not* 125 | your clone of the originating `gh-pages` branch. 126 | Lastly, published copies of all the lessons are available in the `gh-pages` branch of the originating 127 | repository for reference while revising. 128 | 129 | ## Other Resources 130 | 131 | General discussion of [Software Carpentry][swc-site] and [Data Carpentry][dc-site] 132 | happens on the [discussion mailing list][discuss-list], 133 | which everyone is welcome to join. 134 | You can also [reach us by email][email]. 135 | 136 | [email]: mailto:admin@software-carpentry.org 137 | [dc-issues]: https://github.com/issues?q=user%3Adatacarpentry 138 | [dc-lessons]: http://datacarpentry.org/lessons/ 139 | [dc-site]: http://datacarpentry.org/ 140 | [discuss-list]: http://lists.software-carpentry.org/listinfo/discuss 141 | [github]: https://github.com 142 | [github-flow]: https://guides.github.com/introduction/flow/ 143 | [github-join]: https://github.com/join 144 | [how-contribute]: https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github 145 | [issues]: https://guides.github.com/features/issues/ 146 | [swc-issues]: https://github.com/issues?q=user%3Aswcarpentry 147 | [swc-lessons]: https://software-carpentry.org/lessons/ 148 | [swc-site]: https://software-carpentry.org/ 149 | [c-site]: https://carpentries.org/ 150 | [lc-site]: https://librarycarpentry.org/ 151 | [lc-issues]: https://github.com/issues?q=user%3Alibrarycarpentry 152 | -------------------------------------------------------------------------------- /episodes/carbon-efficiency.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Carbon Efficiency 3 | teaching: 5 4 | exercises: 15 5 | --- 6 | 7 | ::::::::::::::::::::::::::::::::::::::: objectives 8 | 9 | - Understand what CO2e means. 10 | - Appreciate positive emissions impacts from HPC systems. 11 | 12 | :::::::::::::::::::::::::::::::::::::::::::::::::: 13 | 14 | :::::::::::::::::::::::::::::::::::::::: questions 15 | 16 | - What does the term carbon equivalence (CO2e) mean? 17 | - Can we also have positive emissions impacts from HPC systems? 18 | 19 | :::::::::::::::::::::::::::::::::::::::::::::::::: 20 | 21 | ## Carbon and CO2e 22 | 23 | Carbon is often used as a broad term to refer to the impact of all types of emissions and activities on global warming. CO2e (sometimes: CO2eq/CO2-eq), which stands for *carbon equivalence*, is a measurement term used to measure this impact. For example, 1 ton of methane has the same warming effect as about [84 tons of CO2 over 20 years](https://energy.ec.europa.eu/topics/oil-gas-and-coal/methane-emissions_en), so we normalise it to 84 tons CO2e. We may shorten even further to just carbon, which is a term often used to refer to all greenhouse gases (GHGs). 24 | 25 | ## Carbon efficiency in HPC system use 26 | 27 | In the HPC space, the part we can play in the climate solution is **using HPC systems in a carbon efficient way**. Being carbon efficient is about making sure your use of HPC systems emits the least carbon possible for the work you are doing. Or, for providers of HPC systems, it means procuring and operating the system in such a way as to minimise the carbon emissions. Ideally, you would also be able to quantify the **positive** carbon impacts from the work (or HPC system itself) to understand the overall net impact on carbon emissions - though doing this is typically quite challenging. 28 | 29 | ::::::::::::::::::::::::::::::::::::::: challenge 30 | 31 | ## Exercise: Carbon emissions from HPC systems 32 | 33 | What do you think the sources of carbon emissions may be from HPC system use? Write down 2-3 ideas of where you think emissions may come from. 34 | 35 | ::::::::::::::: solution 36 | 37 | ## Solution 38 | 39 | You may have come up with sources such as: 40 | 41 | - Emissions from electricity generation to power the HPC system 42 | - Embodied emissions from manufacturing the HPC system components such as processors, memory and storage 43 | - Embodied emissions from constructing the data centre and other infrastructure to house the HPC system 44 | - Emissions associated with the people working to procure, run, and support the HPC system 45 | 46 | ::::::::::::::::::::::::: 47 | 48 | :::::::::::::::::::::::::::::::::::::::::::::::::: 49 | 50 | 51 | ::::::::::::::::::::::::::::::::::::::: challenge 52 | 53 | ## Exercise: Positive emissions impacts 54 | 55 | Write down the outcomes or activities that you do as part of your research or work that could be producing positive carbon emission impacts. Once you have them written down, can you rank them in order of what you think the largest impact on emissions will be to the smallest? Can you think of any way in which you might go about quantifying this positive impact so it could be included in a carbon audit of your work? 56 | 57 | ::::::::::::::: solution 58 | 59 | ## Solution 60 | 61 | You may have come up with outcomes such as: 62 | 63 | - The research or work you are engaged in is developing new innovations and technologies to reduce or eliminate emissions 64 | - The research or work you are engaged in is providing data that is critical to efforts to quantify climate change and/or net zero strategies 65 | - You are involved in training, education and/or activities raising awareness of carbon emissions and helping others to reduce their emissions 66 | - You are developing resources and tools that help others to quantify and reduce their emissions 67 | 68 | Quantifying positive emissions impacts are often even more difficult than quantifying the emissions we produce through our work but are critical to understanding how we are helping to meet climate commitments (which we will discuss later in this workshop). 69 | 70 | For example, attending this workshop may empower you to go away and take some action to reduce the emissions associated with your work. Maybe each person who takes action following this workshop reduces their overall emissions by 2% this year but only 50% of people who come on the workshop manage to have this impact. This means that overall, we would see a 1% reduction in emissions for this year per person attending the workshop. If we assume that the work emissions per person per year are around 6,000 kgCO2e then this workshop would be estimated to save 60 kgCO2e per person attending the workshop. 71 | 72 | ::::::::::::::::::::::::: 73 | 74 | :::::::::::::::::::::::::::::::::::::::::::::::::: 75 | 76 | ### Positive emissions impacts 77 | 78 | As well as reducing emissions from our use of HPC systems there are typically other sources of positive carbon emissions impact associated with HPC. 79 | 80 | The main source of reduced emissions from HPC system use is in the research that leads to new technology, policies and approaches to reducing emissions. Some examples include: 81 | 82 | - HPC services run the climate models that are used to provide evidence for setting emissions reductions policies and targets across the world. 83 | Research and modelling on HPC services leads to development of improved zero emission energy generation by, for example, modelling new wind turbine and wind farm designs. 84 | - Modelling to support the development of new energy storage technologies such as improved batteries. 85 | 86 | The emissions reductions from such activities are extremely difficult to quantify for a number of reasons so, at the moment, these are not factored in to the emissions estimates for HPC systems and research more broadly but you should be aware of them and think about how they might apply to your research in particular. 87 | 88 | As well as the research activities on HPC systems leading to reductions in emissions, there are other activities that HPC services can potentially take to have positive emissions impacts, for example: 89 | 90 | - Using the waste heat generated by large scale HPC services as a heat source for homes, businesses or farming. For example, the LUMI HPC system in Finland is connected to the district heat network for the local city of Kaajani and helps heat homes and businesses. 91 | - Incorporating environmental and biodiversity improvements into the service. For example, for the ACF data centre (which is in a rural location) that hosts ARCHER2, EPCC have been working to improve the site's biodiversity and improve habitats. 92 | - Responsible carbon offset schemes could also potentially be used to reduce emissions if they are undertaken as part of providing an HPC service. 93 | 94 | :::::::::::::::::::::::::::::::::::::::: keypoints 95 | 96 | - CO2e is a measurement term used to measure the impact of all greenhouse gases. 97 | - Some activities we undertake on HPC systems can potentially have positive carbon impacts but these can be very difficult to quantify. 98 | - Everything we do emits carbon into the atmosphere, and our goal is to emit the least amount of carbon possible. This constitutes the first principle of green HPC system use: **carbon efficiency, emitting the least amount of carbon possible per unit of work**. 99 | 100 | :::::::::::::::::::::::::::::::::::::::::::::::::: 101 | 102 | 104 | 105 | 106 | -------------------------------------------------------------------------------- /episodes/renewable-energy.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Renewable Energy 3 | teaching: 30 4 | exercises: 0 5 | --- 6 | 7 | ::::::::::::::::::::::::::::::::::::::: objectives 8 | 9 | - Gain a more detailed understanding of how renewable energy feeds into reducing emissions. 10 | - Understand the limits of Renewable Energy Certificates. 11 | - Learn about different renewable energy matching approaches. 12 | 13 | :::::::::::::::::::::::::::::::::::::::::::::::::: 14 | 15 | :::::::::::::::::::::::::::::::::::::::: questions 16 | 17 | - What does 100% renewable energy actually mean? 18 | - How do we claim 100% renewable energy on a power grid with mixed energy sources? 19 | - What impact do different renewable energy matching strategies make? 20 | 21 | :::::::::::::::::::::::::::::::::::::::::::::::::: 22 | 23 | 24 | ## 100% Renewable 25 | 26 | When organisations set a target of 100% renewable power, they might distinguish between being **matched by** vs. **powered by** renewable energy sources. 27 | 28 | **Powered by** means you are directly powered by a renewable power source, say a hydro dam. In that scenario, the energy the device receives only originates from that source, so you can confidently say that you are 100% powered by renewables. 29 | 30 | For most people, we live on an interconnected grid, with many producers pumping electricity in and many consumers taking electricity out. This means the electrons coming into your device are a mixture of all the electrons going into the grid. For example, suppose the grid only has 5% of wind supply. You are getting 5% of wind-generated electrons and 95% fossil fuel-generated electrons. 31 | 32 | You cannot track individual electrons. Once the electrons from a wind farm are on a grid, they all mix with the electrons from a fossil fuel plant. So there is no way for a consumer to insist the electrons that it uses only come from renewable sources. 33 | 34 | ### Renewable Energy Certificates (REC) 35 | 36 | To solve this problem, a renewable plant sells two things. The first is its electricity, which it sells into a grid. The second is a REC, a [Renewable Energy Certificates](https://www.epa.gov/green-power-markets/renewable-energy-certificates-recs). 1 REC equals 1 kWh of energy. 37 | 38 | If you want to be 100% matched by renewable energy and are on the grid, the solution is to buy enough RECs to cover the amount of electricity you consume. For instance, if you consume 100 kWh of electricity every day, then to be 100% matched by renewables, you buy 100 RECs. 39 | 40 | When organisations set 100% renewable targets purchasing RECs on the market is the solution they often employ to meet their commitments. 41 | 42 | ### PPAs 43 | 44 | You might also hear the term PPA used alongside RECs. A PPA is a [Power Purchase Agreement](https://ppp.worldbank.org/public-private-partnership/sector/energy/energy-power-agreements/power-purchase-agreements), which is another way to purchase RECs. If you estimate you need 500 MWh of electricity per year for a particular data center, you might sign a PPA to purchase 500 MWh per year from a renewable plant. You would then get all the RECs associated with this power plant. 45 | 46 | PPAs are typically very long-term contracts. A renewable plant can find financing with one of these agreements since it already has had a buyer for its electricity for many years. 47 | 48 | PPAs encourage something called **additionality**. Purchasing a PPA drives the creation of new renewable plants. PPAs are a solution that gets us towards a future where everyone has access to 100% renewable energy. 49 | 50 | ## 24/7 Hourly Matching 51 | 52 | When it comes to 100% renewable claims, the critical question is, what is the granularity of matching? Do you sum up and net off yearly, monthly, weekly, daily, or hourly? That question is essential because to truly transition to renewable energy, we need 100% of the power to come from low-carbon energy sources like renewables 100% of the time. This fine granular matching is often called _[24/7 hourly matching](https://www.epa.gov/green-power-markets/247-hourly-matching-electricity)_. 53 | 54 | 24/7 hourly matching is one of the many strategies we need to employ to help accelerate the transition to a 100% renewable-powered grid. For example, [Google](https://sustainability.google/progress/energy/) and [Microsoft](https://blogs.microsoft.com/blog/2021/07/14/made-to-measure-sustainability-commitment-progress-and-updates/) have both committed to 24/7 hourly matching by 2030. 55 | 56 | ### Daily vs hourly matching 57 | 58 | Imagine an organisation has a demand curve like this, each blue square represents 1kWh: 59 | 60 | ![Demand curve for electricity use](./fig/29_daily_consumption.png "Demand curve for electricity use") 61 | 62 | They have purchased RECs from a wind farm that generated electricity with a curve, so each green square represents 1 REC. Matching by day means the organization consumed 18 kWh and bought 18 RECs. As a result, they netted off to zero. So they can say they are **100% matched by renewable energy daily.** 63 | 64 | However, if we looked at it in hourly buckets (each square here is 2 hrs in length), then it seems a bit different: 65 | 66 | ![Diagram illustrating impact of hourly matching](./fig/30_hourly_match.png "Diagram illustrating impact of hourly matching") 67 | 68 | The total amount of energy consumed is still 18kWh. However, there are only a few hours in the day where we are 100% matched by renewable energy for that hour. So for some hours, we have way more renewable energy than we need. Conversely, we have way less renewable energy than we require for most hours. 69 | 70 | In the above example, they are **100% matched by renewable energy on an hourly basis for only 6 hrs of the day**. 71 | 72 | ### Carbon-free energy 73 | 74 | The number we use to describe how successful we are at 24/7 hourly matching is the carbon-free energy percentage. 75 | 76 | Carbon-free energy is defined as [the average percentage of carbon-free energy consumed in a particular location on an hourly basis](https://cloud.google.com/sustainability/region-carbon#understanding). 77 | 78 | So for the previous example, if measured using daily matching, we are 100% matched with renewable energy. However, we are only 33.1% matched if measured using hourly matching. **The CFE percentage is, therefore, 33.1%**. 79 | 80 | ### Carbon Awareness as part of a 24/7 Hourly Matching Strategy 81 | 82 | Carbon aware computing involves responding to electrical carbon intensity signals and changing the **behavior** of software, so it emits less carbon. Carbon awareness also helps an organisation meet their 24/7 hourly matching target and increase its CFE percentage. 83 | 84 | One example of a behavior change is shifting compute to a time when more renewable energy is available. For example, delaying the start of a power intensive simulation (or even delaying charging of a laptop) to when the carbon intensity of electricity is lower, and the supply of renewable energy is higher. 85 | 86 | ![Temporal shifting along with matching strategy](./fig/31_carbon_awareness.png "Temporal shifting along with matching strategy") 87 | 88 | 89 | :::::::::::::::::::::::::::::::::::::::: keypoints 90 | 91 | - "Powered by" 100% renewable energy means that renewable energy sources are directly connected to the HPC system. 92 | - "Matched by" 100% renewable energy means that the electricity for the HPC system is purchased from the grid along with renewable energy certificates to the same capacity as the usage. 93 | - When electricity is "matched by" 100% renewable energy the matching interval has a large impact on how it drives transition to green energy. The term "carbon free energy" is used to quantify how well we are matched on a 24/7 basis. 94 | - Large power purchase agreements (PPA) can be designed to incentivise the transition to green energy. 95 | 96 | :::::::::::::::::::::::::::::::::::::::::::::::::: 97 | -------------------------------------------------------------------------------- /learners/reference.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: 'Glossary' 3 | --- 4 | 5 | ## Acronyms 6 | 7 | 8 | 9 | 10 | 12 | 14 | 16 | 17 | 18 | 20 | 22 | 24 | 25 | 26 | 28 | 30 | 32 | 33 | 34 | 36 | 38 | 40 | 41 | 42 | 44 | 46 | 48 | 49 | 50 | 52 | 54 | 56 | 57 | 58 | 60 | 62 | 64 | 65 | 66 | 68 | 70 | 72 | 73 | 74 | 76 | 78 | 80 | 81 | 82 | 84 | 86 | 88 | 89 | 90 | 92 | 94 | 96 | 97 | 98 | 100 | 102 | 104 | 105 | 106 | 108 | 110 | 112 | 113 | 114 | 116 | 118 | 120 | 121 | 122 | 124 | 126 | 128 | 129 | 130 | 132 | 134 | 136 | 137 | 138 | 140 | 142 | 144 | 145 | 146 | 148 | 150 | 152 | 153 | 154 | 156 | 158 | 160 | 161 | 162 | 164 | 166 | 168 | 169 | 170 | 172 | 174 | 176 | 177 | 178 | 180 | 182 | 184 | 185 | 186 | 188 | 190 | 192 | 193 | 194 | 196 | 198 | 200 | 201 | 202 | 204 | 206 | 208 | 209 |
Term 11 | Acronym 13 | Definition 15 |
Greenhouse gases 19 | GHGs 21 | Greenhouse gases are a group of gases that trap heat from solar radiation in the Earth's atmosphere. These gases act as a blanket, increasing the temperature on the surface of the Earth. 23 |
Carbon dioxide 27 | CO2 29 | One of the most common greenhouse gases. 31 |
Methane 35 | CH4 37 | A common gaseous hydrocarbon, which has a warming effect 84 times that of CO2 over 20 years and 28 times that of CO2 over 100 years. 39 |
Sulfur hexafluoride 43 | SF6 45 | A man-made gas used as an electrical insulator that has a warming effect 23,500 times that of CO2. 47 |
Carbon dioxide equivalent 51 | CO2eq / CO2-eq / CO2e 53 | Carbon is used as a common form of measurement for all greenhouse gases. This unit of measurement indicates the potential impact of non-CO2 gases on global warming in carbon terms. 55 |
Kilowatt hours 59 | kWh 61 | The standard unit of measurement for energy consumption. 63 |
Power usage effectiveness 67 | PUE 69 | The metric used to measure data center energy efficiency. 71 |
Global warming potential 75 | GWP 77 | The potential impact of greenhouse gases on global warming. Measured in terms of CO2e. 79 |
World Meteorological Organization 83 | WMO 85 | A specialized agency of the United Nations whose mandate covers weather, climate and water resources. 87 |
Million metric tonnes of carbon dioxide equivalent 91 | MMTCDE 93 | Measurement term for CO2eq 95 |
Nationally Determined Contribution 99 | NDC 101 | The means by which members of the Paris Climate Agreement are expected to update their progress. 103 |
Conference of the Parties 107 | COP 109 | An annual event involving all parties in the United Nations Framework Convention on Climate Change. 111 |
United Nations Framework Convention on Climate Change 115 | UNFCCC 117 | A group created to achieve the stabilization of greenhouse gas concentrations in the atmosphere at a level that would prevent dangerous interference with the climate system. 119 |
Intergovernmental Panel on Climate Change 123 | IPCC 125 | The objective of the IPCC is to provide governments at all levels with scientific information that they can use to develop climate policies. 127 |
Science Based Targets initiative 131 | SBTi 133 | A body that defines and promotes best practice in science-based target setting. For example, creating the standards for net zero 135 |
Carbon-free energy 139 | CFE 141 | This term is usually used to talk about the percentage of renewable energy used as a proportion of the total energy used 143 |
Power Purchase Agreement 147 | PPA 149 | A contract you sign with a power plant to purchase RECs 151 |
Renewable Energy Credit 155 | REC 157 | Renewable energy credits (also known as renewable energy certificates) represent the energy generated by renewable energy sources 159 |
Paris Climate Agreement 163 | PCA 165 | An international treaty agreed in 2015 by 196 parties and the UN to reduce the Earth's temperature increase 167 |
joules 171 | J 173 | Energy is measured in joules (J). 175 |
grams of carbon per kilowatt hour 179 | gCO2eq/kWh 181 | The standard unit of carbon intensity is gCO2eq/kWh, or grams of carbon per kilowatt hour. 183 |
Voluntary Carbon Market 187 | VCM 189 | A decentralized market where private actors voluntarily buy and sell carbon credits that represent certified removals or reductions of greenhouse gases (GHGs) in the atmosphere. 191 |
Verified Carbon Standard 195 | VCS 197 | A standard for certifying carbon emissions reductions. 199 |
Software Carbon Intensity 203 | SCI 205 | A standard which gives an actionable approach to software designers, developers and operations to measure the carbon impacts of their systems. 207 |
210 | 211 | ## Useful Terms 212 | 213 | * [Carbon Intensity](./carbon-awareness/#carbon-intensity) - Measures the amount of greenhouse gases emitted per unit of electricity produced. 214 | * [Demand Shaping](./carbon-awareness/#demand-shaping) - The strategy of shaping our computation to match the existing clean electricity supply. 215 | * [Greenhouse Gas protocol](https://ghgprotocol.org) - The most widely used and internationally recognized greenhouse gas accounting standard. 216 | * [Value chain emissions](https://www.cisl.cam.ac.uk/education/graduate-study/pgcerts/value-chain-defs) - These are scope 3 emissions according to the GHG protocol, and the most significant source of emissions. They encompass the full range of activities needed to create a product or service, from conception to distribution. 217 | * [Energy proportionality](./energy-efficiency/#energy-proportionality) - Measures the relationship between power consumed by a computer and the rate at which useful work is done (its utilization). 218 | * [Static power draw](./energy-efficiency/#static-power-draw) - This refers to how much electricity is drawn when a device is in an idle state. 219 | * [Embodied Carbon](./hardware-efficiency/#embodied-carbon) (also known as "embedded carbon") - The amount of carbon pollution emitted during the creation and disposal of a device. 220 | -------------------------------------------------------------------------------- /learners/setup.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Setup 3 | --- 4 | 5 | Before attending the workshop you should spend a short amount of time familiarising yourself with the 6 | "Key concepts" described below. 7 | 8 | The other section on Green Software Engineering is provided for interest (particularly if you are a software developer or are designing AI models). Reading this section is not required to participate in the course. 9 | 10 | ## Key concepts 11 | 12 | ### Global warming vs climate change 13 | 14 | [Global warming](https://climate.nasa.gov/global-warming-vs-climate-change/) is the long-term heating of Earth's climate system observed since the pre-industrial period (between 1850 and 1900) due to human activities, primarily fossil fuel burning. [Climate change](https://climate.nasa.gov/global-warming-vs-climate-change/) is long-term shifts in temperatures and weather patterns. These shifts may be natural, but since the 1800s, human activities have been the main driver of climate change. 15 | 16 | ### Climate vs weather 17 | 18 | Weather refers to the conditions of the atmosphere in a short period of time. Climate refers to the conditions of the atmosphere over long periods of time. Any changes to the long-term condition of the atmosphere will also cause changes to the short-term conditions. Some examples of measurable changes to weather conditions due to climate change are: 19 | 20 | - Changes to the water cycle, including rainfall 21 | - Melting of ice 22 | - Heating of the land, air, and ocean 23 | - Changes in ocean currents, acidity, and salinity 24 | 25 | These changes can lead to flooding - both in coastal areas and due to increased rainfall - drought, wildfires and more frequent extreme weather conditions. 26 | 27 | ### Greenhouse gases and the greenhouse effect 28 | 29 | Greenhouse gases are a group of gases that trap heat from solar radiation in the Earth's atmosphere. These gases act as a blanket, increasing the temperature on the surface of the Earth. This is a natural phenomenon which has been accelerated due to man-made carbon emissions. Now the global climate is changing at a faster rate than that at which animals and plants can adapt. 30 | 31 | Greenhouse gases and the greenhouse effect are crucial to all life on Earth and often come from natural sources like animals, volcanoes, and other geological activity. The greenhouse effect allows the Earth to maintain a higher temperature than it would without them by capturing more heat from solar radiation. Like many other natural processes of the Earth, the greenhouse effect is a balance that can be upset by multiple factors. 32 | 33 | ## Monitoring climate change 34 | 35 | As a result of the effects of climate change and an ever-increasing number of destructive weather events, efforts have been made by the global community to address these issues and take steps to control and limit global warming in order to mitigate and reverse the effects of climate change. 36 | 37 | The [**Paris Climate Agreement**](https://unfccc.int/process-and-meetings/the-paris-agreement/the-paris-agreement) is an international treaty agreed in 2015 by 196 parties and the UN to reduce the Earth's temperature increase. The agreement is to keep the rise in global mean temperature to 2°C compared to pre-industrial levels, with a preferable lower limit of 1.5°C. The agreement is reviewed every five years and mobilizes finance to developing nations to mitigate the impacts of climate change and prepare for and adapt to the environmental effects caused by climate change. In addition, each party is expected to update its progress through a Nationally Determined Contribution (NDC). As of October 2024, [195 parties](https://www.un.org/en/climatechange/paris-agreement), including the UK, have joined the agreement. 38 | 39 | The [**United Nations Framework Convention on Climate Change (UNFCCC)**](https://unfccc.int/process-and-meetings/the-convention/what-is-the-united-nations-framework-convention-on-climate-change) is a group created to achieve the stabilization of greenhouse gas concentrations in the atmosphere at a level that would prevent dangerous interference with the climate system. 40 | 41 | The [**COP (Conference of the Parties)**](https://unfccc.int/process/bodies/supreme-bodies/conference-of-the-parties-cop) is an annual event involving all parties in the United Nations Framework Convention on Climate Change. At the conference, each party member's progress on tackling global warming, as agreed as part of the Paris Climate Agreement, is reviewed and assessed. The COP is also a chance for parties to come together and make decisions that will reduce the effects of global warming. Common topics include strategies to reduce carbon, financing low carbon strategies and preservation of natural habitats. 42 | 43 | The [**IPCC (Intergovernmental Panel on Climate Change)**](https://www.ipcc.ch/about/), created by the UN in 1988, aims to provide governments at all levels with scientific information that they can use to develop climate policies. IPCC reports are also a key input into international climate change negotiations. The IPCC is an organization of governments that are members of the United Nations or the World Meteorological Organization (WMO). The IPCC currently has 195 members. 44 | 45 | We will always emit carbon through our activities, but being carbon efficient means minimising the amount of carbon emitted per unit of work. We aim to ensure that for each gram of carbon we emit into the atmosphere, we extract the most value possible. 46 | 47 | ![Picture of user minimising carbon emissions](../episodes/fig/02_monitoring_climate_change.png "image_tooltip") 48 | 49 | ## History of green software engineering 50 | 51 | In 2019 the original eight principles of green software engineering were released. The 2022 update of the principles took on feedback received over the years, merging some principles and adding a new one regarding understanding climate commitments. The principles as they are written are aimed primarily at software engineers but they are more broadly applicable to anyone making use of digital infrastructure. In this lesson we specifically look at how they can be applied to use of HPC resources. The principles are: 52 | 53 | 1. **Carbon Efficiency**: Emit the least amount of carbon possible. 54 | 2. **Energy Efficiency**: Use the least amount of energy possible. 55 | 3. **Carbon Awareness**: Do more when the electricity is cleaner and do less when the electricity is dirtier. 56 | 4. **Hardware Efficiency**: Use the least amount of embodied carbon possible. 57 | 5. **Measurement**: What you cannot measure, you cannot improve. 58 | 6. **Climate Commitments:** Understand the exact mechanism of carbon reduction. 59 | 60 | These principles were originally described by the [Green Software Foundation](https://greensoftware.foundation/). 61 | 62 | ## Principles, Patterns, and Practices. 63 | 64 | Above, we described the **principles** of green software, a core set of competencies needed to define, build and run green software. To support software engineers in this area, The Green Software Foundation has developed a catalogue of *green software patterns*. While not specific to use of HPC and aimed at software developers rather than users, they provide interesting information on how particular software types and their use can be made more environmentally sustainable. 65 | 66 | A green software [**pattern**](https://patterns.greensoftware.foundation/) is a specific example of how to apply one or more principles in a real-world example. Whereas principles describe the theory that underpins green software, patterns are the practical advice software practitioners can use in their software applications today. Patterns are vendor-neutral. 67 | 68 | A green software **practice** is a pattern applied to a specific vendor's product and informs practitioners about how to use that product in a more sustainable way. 69 | 70 | Practices should refer to patterns that should refer to principles. 71 | 72 | The green software foundation publishes a [catalog of vendor-neutral green software patterns](https://patterns.greensoftware.foundation/) across various categories. 73 | 74 | ![Diagram illustrating green software principles, patterns, and practices](../episodes/fig/GSF_Principles_Patterns_Practices_v2.png "Green Software Principles, Patterns, and Practices") 75 | 76 | 78 | -------------------------------------------------------------------------------- /episodes/hardware-efficiency.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Hardware Efficiency 3 | teaching: 20 4 | exercises: 0 5 | --- 6 | 7 | ::::::::::::::::::::::::::::::::::::::: objectives 8 | 9 | - Understand what is meant by "embodied carbon" in the context of HPC systems. 10 | - Learn how embodied carbon efficiency can be bettered by extending the lifespan of HPC systems and improving performance of applications on HPC systems. 11 | 12 | :::::::::::::::::::::::::::::::::::::::::::::::::: 13 | 14 | :::::::::::::::::::::::::::::::::::::::: questions 15 | 16 | - What is embodied carbon on HPC systems? 17 | - How can embodied carbon efficiency be improved on HPC systems? 18 | - What can I do to improve the embodied carbon efficiency during my use of HPC systems? 19 | 20 | :::::::::::::::::::::::::::::::::::::::::::::::::: 21 | 22 | ## Introduction 23 | 24 | The hardware that makes up the HPC systems you are using is an important element to consider when looking to be a greener user of HPC. Indeed, for HPC systems located where carbon intensity of electricity generation is low, the hardware may be the dominant component of emissions associated with HPC system use. 25 | 26 | You will see how embodied carbon is a hidden cost when it comes to hardware and the different measures you can take to reduce the impact that the creation, destruction, and running of this hardware involves. More specifically, extending hardware lifetime and improving the efficiency of your hardware use are the main ways of reducing this impact. 27 | 28 | ## Key concepts 29 | 30 | ### Embodied carbon 31 | 32 | The manufacturing of the device you are using to participate in this workshop produced carbon and disposing of it at the end of its life may release more. Embodied (or embedded) carbon is the amount of carbon emissions due to the creation and disposal of a device. 33 | 34 | When calculating the total carbon emissions for HPC services, both the carbon emissions associated with running the system as well as the embodied carbon of the system must be accounted for. 35 | 36 | ![Bar chart illustrating embodied carbon from various end-user devices](./fig/17_embodioed_carbon.png "Bar chart illustrating embodied carbon from various end-user devices") 37 | 38 | Embodied carbon varies significantly between different hardware. For some devices, the carbon emitted during manufacturing is much higher than that emitted during usage, as illustrated by a [study](https://www.ifi.uzh.ch/dam/jcr:fa4e956e-7a53-4038-98a5-00e09e2f4303/Study_Digitalization_Climate_Protection_Summary_Oct2017.pdf) from University of Zurich. As a result, the embodied carbon cost can sometimes be much higher than the carbon cost of the electricity powering it. 39 | 40 | By thinking in terms of embodied carbon, any device, even one not consuming electricity, is responsible for the release of carbon over its lifetime. 41 | 42 | ### Amortisation 43 | 44 | A way to account for embodied carbon is to amortise the carbon over the expected lifespan of a device. For example, suppose it took 4000 kgCO2e to build an HPC system, and we expect it to last four years. Amortisation means that we can say the HPC system emits 1000 kgCO2e/year in embodied carbon alone. 45 | 46 | ![Diagram illustrating amortisation](./fig/18_amortization.png "Diagram illustrating amortisation") 47 | 48 | Rather than years, we typically amortise the embodied emissions over the total amount of resource available on the HPC system over its lifetime. For example, if the resource use of an HPC system is measured in GPUh then we would amortise the embodied emissions over the total GPUh available on the service over its whole lifetime (to give an emissions rate in kgCO2e/GPUh). 49 | 50 | ## How to improve hardware efficiency 51 | 52 | By taking into account the embodied carbon, it is clear that by the time we come to install an HPC system, it has already emitted a large amount of carbon. Computers also have a limited lifespan, which means they are eventually unable to handle modern workloads, or they suffer failures and need to be replaced. In these terms, hardware consumption is a proxy for carbon, and since our goal is to be carbon efficient, we must also be hardware efficient. 53 | 54 | There are two main approaches to improving hardware carbon efficiency: 55 | 56 | - **Extending the lifespan** of the hardware - which reduces the carbon emission rate per unit of resource due to amortisation. 57 | - **Increasing the utilisation and performance** of the hardware - getting more useful work out of the hardware per unit of resource. 58 | 59 | ### Extending the lifespan of hardware 60 | 61 | In the example we saw previously, if we can add just one more year to the lifespan of our HPC system, then the amortised carbon emissions rate drops from 1000 kgCO2e/year to 800 kgCO2e/year. 62 | 63 | ![Diagram illustrating amortisation](./fig/19_lifespan.png "Diagram illustrating amortisation") 64 | 65 | HPC systems have historically had lifetimes of around 5-7 years at which point they are replaced by newer systems that provide improved performance and functionality and that are typically more energy efficient. However, extending the lifetime of HPC systems may lead to improved carbon efficiency through the amortisation of embodied carbon, and using older HPC systems for our research may actually lead to a more carbon efficient use of HPC. 66 | 67 | ### Increasing utilisation and performance 68 | 69 | As well as increasing the lifespan to improve the embodied carbon efficiency, we can also work to make sure we get the most out of the HPC hardware during its lifetime. 70 | 71 | At a service/system level this often corresponds to maximising the usage of the service - it's better to have 100% utilisation than 20% utilisation because of the cost of embodied carbon (and also because of the fact that even idle components in HPC systems consume some electricity). 72 | 73 | For individual users and groups, improving the carbon efficiency with respect to embodied carbon typically corresponds to increasing the performance of their use of the HPC system so they get more output per unit of time. Of course, increasing the performance of your use may lead to higher power draw and higher electricity use which increases the emissions from the use of electricity. This means that you need to know what the balance between embodied emissions and emissions from electricity use are for the HPC system you are using to make useful choices about improving your carbon efficiency. There are three potential scenarios to maximise carbon efficiency: 74 | 75 | - **Embodied carbon dominates -** run as high performance as possible irrespective of electricity use 76 | - **Embodied carbon and carbon from electricity use are evenly balanced -** you need to find a balance of performance and energy efficiency 77 | - **Carbon from electricity use dominates -** run in as energy efficient a manner as possible 78 | 79 | We will look at a specific example of this balance in a later episode of this workshop. 80 | 81 | :::::::::::::::::::::::::::::::::::::::: callout 82 | 83 | ## Carbon efficiency can be complex - flexibility is key 84 | 85 | Understanding the most carbon efficient way to make use of HPC systems (and the most carbon efficient way to operate them, including choosing their lifetime) can be complex as it depends on many factors. These can include the embodied carbon of the hardware, the lifetime of the system, the carbon intensity of the electricity supply and how that changes over the service lifetime, and the efficiency of the workload you are running on the type of hardware provided by the system. However, one key aspect of being able to use HPC in a carbon efficient way is for your workflow to have the flexibility to run on different system types efficiently. 86 | 87 | :::::::::::::::::::::::::::::::::::::::::::::::::: 88 | 89 | :::::::::::::::::::::::::::::::::::::::: keypoints 90 | 91 | - Embodied carbon of an HPC system is the amount of carbon pollution emitted during the creation and disposal of the HPC system. 92 | - When calculating your total carbon pollution, you must consider both that which is emitted when running on the HPC system as well as the embodied carbon associated with its creation and disposal. 93 | - Extending the lifetime of an HPC system has the effect of amortising the carbon emitted so that its embodied CO2e/year is reduced. 94 | - Increasing utilisation and performance also improve the embodied carbon efficiency from HPC system use. 95 | 96 | :::::::::::::::::::::::::::::::::::::::::::::::::: 97 | 98 | 99 | -------------------------------------------------------------------------------- /.github/workflows/README.md: -------------------------------------------------------------------------------- 1 | # Carpentries Workflows 2 | 3 | This directory contains workflows to be used for Lessons using the {sandpaper} 4 | lesson infrastructure. Two of these workflows require R (`sandpaper-main.yaml` 5 | and `pr-receive.yaml`) and the rest are bots to handle pull request management. 6 | 7 | These workflows will likely change as {sandpaper} evolves, so it is important to 8 | keep them up-to-date. To do this in your lesson you can do the following in your 9 | R console: 10 | 11 | ```r 12 | # Install/Update sandpaper 13 | options(repos = c(carpentries = "https://carpentries.r-universe.dev/", 14 | CRAN = "https://cloud.r-project.org")) 15 | install.packages("sandpaper") 16 | 17 | # update the workflows in your lesson 18 | library("sandpaper") 19 | update_github_workflows() 20 | ``` 21 | 22 | Inside this folder, you will find a file called `sandpaper-version.txt`, which 23 | will contain a version number for sandpaper. This will be used in the future to 24 | alert you if a workflow update is needed. 25 | 26 | What follows are the descriptions of the workflow files: 27 | 28 | ## Deployment 29 | 30 | ### 01 Build and Deploy (sandpaper-main.yaml) 31 | 32 | This is the main driver that will only act on the main branch of the repository. 33 | This workflow does the following: 34 | 35 | 1. checks out the lesson 36 | 2. provisions the following resources 37 | - R 38 | - pandoc 39 | - lesson infrastructure (stored in a cache) 40 | - lesson dependencies if needed (stored in a cache) 41 | 3. builds the lesson via `sandpaper:::ci_deploy()` 42 | 43 | #### Caching 44 | 45 | This workflow has two caches; one cache is for the lesson infrastructure and 46 | the other is for the lesson dependencies if the lesson contains rendered 47 | content. These caches are invalidated by new versions of the infrastructure and 48 | the `renv.lock` file, respectively. If there is a problem with the cache, 49 | manual invaliation is necessary. You will need maintain access to the repository 50 | and you can either go to the actions tab and [click on the caches button to find 51 | and invalidate the failing cache](https://github.blog/changelog/2022-10-20-manage-caches-in-your-actions-workflows-from-web-interface/) 52 | or by setting the `CACHE_VERSION` secret to the current date (which will 53 | invalidate all of the caches). 54 | 55 | ## Updates 56 | 57 | ### Setup Information 58 | 59 | These workflows run on a schedule and at the maintainer's request. Because they 60 | create pull requests that update workflows/require the downstream actions to run, 61 | they need a special repository/organization secret token called 62 | `SANDPAPER_WORKFLOW` and it must have the `public_repo` and `workflow` scope. 63 | 64 | This can be an individual user token, OR it can be a trusted bot account. If you 65 | have a repository in one of the official Carpentries accounts, then you do not 66 | need to worry about this token being present because the Carpentries Core Team 67 | will take care of supplying this token. 68 | 69 | If you want to use your personal account: you can go to 70 | 71 | to create a token. Once you have created your token, you should copy it to your 72 | clipboard and then go to your repository's settings > secrets > actions and 73 | create or edit the `SANDPAPER_WORKFLOW` secret, pasting in the generated token. 74 | 75 | If you do not specify your token correctly, the runs will not fail and they will 76 | give you instructions to provide the token for your repository. 77 | 78 | ### 02 Maintain: Update Workflow Files (update-workflow.yaml) 79 | 80 | The {sandpaper} repository was designed to do as much as possible to separate 81 | the tools from the content. For local builds, this is absolutely true, but 82 | there is a minor issue when it comes to workflow files: they must live inside 83 | the repository. 84 | 85 | This workflow ensures that the workflow files are up-to-date. The way it work is 86 | to download the update-workflows.sh script from GitHub and run it. The script 87 | will do the following: 88 | 89 | 1. check the recorded version of sandpaper against the current version on github 90 | 2. update the files if there is a difference in versions 91 | 92 | After the files are updated, if there are any changes, they are pushed to a 93 | branch called `update/workflows` and a pull request is created. Maintainers are 94 | encouraged to review the changes and accept the pull request if the outputs 95 | are okay. 96 | 97 | This update is run weekly or on demand. 98 | 99 | ### 03 Maintain: Update Package Cache (update-cache.yaml) 100 | 101 | For lessons that have generated content, we use {renv} to ensure that the output 102 | is stable. This is controlled by a single lockfile which documents the packages 103 | needed for the lesson and the version numbers. This workflow is skipped in 104 | lessons that do not have generated content. 105 | 106 | Because the lessons need to remain current with the package ecosystem, it's a 107 | good idea to make sure these packages can be updated periodically. The 108 | update cache workflow will do this by checking for updates, applying them in a 109 | branch called `updates/packages` and creating a pull request with _only the 110 | lockfile changed_. 111 | 112 | From here, the markdown documents will be rebuilt and you can inspect what has 113 | changed based on how the packages have updated. 114 | 115 | ## Pull Request and Review Management 116 | 117 | Because our lessons execute code, pull requests are a secruity risk for any 118 | lesson and thus have security measures associted with them. **Do not merge any 119 | pull requests that do not pass checks and do not have bots commented on them.** 120 | 121 | This series of workflows all go together and are described in the following 122 | diagram and the below sections: 123 | 124 | ![Graph representation of a pull request](https://carpentries.github.io/sandpaper/articles/img/pr-flow.dot.svg) 125 | 126 | ### Pre Flight Pull Request Validation (pr-preflight.yaml) 127 | 128 | This workflow runs every time a pull request is created and its purpose is to 129 | validate that the pull request is okay to run. This means the following things: 130 | 131 | 1. The pull request does not contain modified workflow files 132 | 2. If the pull request contains modified workflow files, it does not contain 133 | modified content files (such as a situation where @carpentries-bot will 134 | make an automated pull request) 135 | 3. The pull request does not contain an invalid commit hash (e.g. from a fork 136 | that was made before a lesson was transitioned from styles to use the 137 | workbench). 138 | 139 | Once the checks are finished, a comment is issued to the pull request, which 140 | will allow maintainers to determine if it is safe to run the 141 | "Receive Pull Request" workflow from new contributors. 142 | 143 | ### Receive Pull Request (pr-receive.yaml) 144 | 145 | **Note of caution:** This workflow runs arbitrary code by anyone who creates a 146 | pull request. GitHub has safeguarded the token used in this workflow to have no 147 | priviledges in the repository, but we have taken precautions to protect against 148 | spoofing. 149 | 150 | This workflow is triggered with every push to a pull request. If this workflow 151 | is already running and a new push is sent to the pull request, the workflow 152 | running from the previous push will be cancelled and a new workflow run will be 153 | started. 154 | 155 | The first step of this workflow is to check if it is valid (e.g. that no 156 | workflow files have been modified). If there are workflow files that have been 157 | modified, a comment is made that indicates that the workflow is not run. If 158 | both a workflow file and lesson content is modified, an error will occurr. 159 | 160 | The second step (if valid) is to build the generated content from the pull 161 | request. This builds the content and uploads three artifacts: 162 | 163 | 1. The pull request number (pr) 164 | 2. A summary of changes after the rendering process (diff) 165 | 3. The rendered files (build) 166 | 167 | Because this workflow builds generated content, it follows the same general 168 | process as the `sandpaper-main` workflow with the same caching mechanisms. 169 | 170 | The artifacts produced are used by the next workflow. 171 | 172 | ### Comment on Pull Request (pr-comment.yaml) 173 | 174 | This workflow is triggered if the `pr-receive.yaml` workflow is successful. 175 | The steps in this workflow are: 176 | 177 | 1. Test if the workflow is valid and comment the validity of the workflow to the 178 | pull request. 179 | 2. If it is valid: create an orphan branch with two commits: the current state 180 | of the repository and the proposed changes. 181 | 3. If it is valid: update the pull request comment with the summary of changes 182 | 183 | Importantly: if the pull request is invalid, the branch is not created so any 184 | malicious code is not published. 185 | 186 | From here, the maintainer can request changes from the author and eventually 187 | either merge or reject the PR. When this happens, if the PR was valid, the 188 | preview branch needs to be deleted. 189 | 190 | ### Send Close PR Signal (pr-close-signal.yaml) 191 | 192 | Triggered any time a pull request is closed. This emits an artifact that is the 193 | pull request number for the next action 194 | 195 | ### Remove Pull Request Branch (pr-post-remove-branch.yaml) 196 | 197 | Tiggered by `pr-close-signal.yaml`. This removes the temporary branch associated with 198 | the pull request (if it was created). 199 | -------------------------------------------------------------------------------- /bin/workshop_check.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | '''Check that a workshop's index.html metadata is valid. See the 4 | docstrings on the checking functions for a summary of the checks. 5 | ''' 6 | 7 | 8 | import sys 9 | import os 10 | import re 11 | from datetime import date 12 | from util import Reporter, split_metadata, load_yaml, check_unwanted_files 13 | 14 | # Metadata field patterns. 15 | EMAIL_PATTERN = r'[^@]+@[^@]+\.[^@]+' 16 | HUMANTIME_PATTERN = r'((0?[1-9]|1[0-2]):[0-5]\d(am|pm)(-|to)(0?[1-9]|1[0-2]):[0-5]\d(am|pm))|((0?\d|1\d|2[0-3]):[0-5]\d(-|to)(0?\d|1\d|2[0-3]):[0-5]\d)' 17 | EVENTBRITE_PATTERN = r'\d{9,10}' 18 | URL_PATTERN = r'https?://.+' 19 | 20 | # Defaults. 21 | CARPENTRIES = ("dc", "swc", "lc", "cp") 22 | DEFAULT_CONTACT_EMAIL = 'admin@software-carpentry.org' 23 | 24 | USAGE = 'Usage: "workshop_check.py path/to/root/directory"' 25 | 26 | # Country and language codes. Note that codes mean different things: 'ar' 27 | # is 'Arabic' as a language but 'Argentina' as a country. 28 | 29 | ISO_COUNTRY = [ 30 | 'ad', 'ae', 'af', 'ag', 'ai', 'al', 'am', 'an', 'ao', 'aq', 'ar', 'as', 31 | 'at', 'au', 'aw', 'ax', 'az', 'ba', 'bb', 'bd', 'be', 'bf', 'bg', 'bh', 32 | 'bi', 'bj', 'bm', 'bn', 'bo', 'br', 'bs', 'bt', 'bv', 'bw', 'by', 'bz', 33 | 'ca', 'cc', 'cd', 'cf', 'cg', 'ch', 'ci', 'ck', 'cl', 'cm', 'cn', 'co', 34 | 'cr', 'cu', 'cv', 'cx', 'cy', 'cz', 'de', 'dj', 'dk', 'dm', 'do', 'dz', 35 | 'ec', 'ee', 'eg', 'eh', 'er', 'es', 'et', 'eu', 'fi', 'fj', 'fk', 'fm', 36 | 'fo', 'fr', 'ga', 'gb', 'gd', 'ge', 'gf', 'gg', 'gh', 'gi', 'gl', 'gm', 37 | 'gn', 'gp', 'gq', 'gr', 'gs', 'gt', 'gu', 'gw', 'gy', 'hk', 'hm', 'hn', 38 | 'hr', 'ht', 'hu', 'id', 'ie', 'il', 'im', 'in', 'io', 'iq', 'ir', 'is', 39 | 'it', 'je', 'jm', 'jo', 'jp', 'ke', 'kg', 'kh', 'ki', 'km', 'kn', 'kp', 40 | 'kr', 'kw', 'ky', 'kz', 'la', 'lb', 'lc', 'li', 'lk', 'lr', 'ls', 'lt', 41 | 'lu', 'lv', 'ly', 'ma', 'mc', 'md', 'me', 'mg', 'mh', 'mk', 'ml', 'mm', 42 | 'mn', 'mo', 'mp', 'mq', 'mr', 'ms', 'mt', 'mu', 'mv', 'mw', 'mx', 'my', 43 | 'mz', 'na', 'nc', 'ne', 'nf', 'ng', 'ni', 'nl', 'no', 'np', 'nr', 'nu', 44 | 'nz', 'om', 'pa', 'pe', 'pf', 'pg', 'ph', 'pk', 'pl', 'pm', 'pn', 'pr', 45 | 'ps', 'pt', 'pw', 'py', 'qa', 're', 'ro', 'rs', 'ru', 'rw', 'sa', 'sb', 46 | 'sc', 'sd', 'se', 'sg', 'sh', 'si', 'sj', 'sk', 'sl', 'sm', 'sn', 'so', 47 | 'sr', 'st', 'sv', 'sy', 'sz', 'tc', 'td', 'tf', 'tg', 'th', 'tj', 'tk', 48 | 'tl', 'tm', 'tn', 'to', 'tr', 'tt', 'tv', 'tw', 'tz', 'ua', 'ug', 'um', 49 | 'us', 'uy', 'uz', 'va', 'vc', 've', 'vg', 'vi', 'vn', 'vu', 'wf', 'ws', 50 | 'ye', 'yt', 'za', 'zm', 'zw' 51 | ] 52 | 53 | ISO_LANGUAGE = [ 54 | 'aa', 'ab', 'ae', 'af', 'ak', 'am', 'an', 'ar', 'as', 'av', 'ay', 'az', 55 | 'ba', 'be', 'bg', 'bh', 'bi', 'bm', 'bn', 'bo', 'br', 'bs', 'ca', 'ce', 56 | 'ch', 'co', 'cr', 'cs', 'cu', 'cv', 'cy', 'da', 'de', 'dv', 'dz', 'ee', 57 | 'el', 'en', 'eo', 'es', 'et', 'eu', 'fa', 'ff', 'fi', 'fj', 'fo', 'fr', 58 | 'fy', 'ga', 'gd', 'gl', 'gn', 'gu', 'gv', 'ha', 'he', 'hi', 'ho', 'hr', 59 | 'ht', 'hu', 'hy', 'hz', 'ia', 'id', 'ie', 'ig', 'ii', 'ik', 'io', 'is', 60 | 'it', 'iu', 'ja', 'jv', 'ka', 'kg', 'ki', 'kj', 'kk', 'kl', 'km', 'kn', 61 | 'ko', 'kr', 'ks', 'ku', 'kv', 'kw', 'ky', 'la', 'lb', 'lg', 'li', 'ln', 62 | 'lo', 'lt', 'lu', 'lv', 'mg', 'mh', 'mi', 'mk', 'ml', 'mn', 'mr', 'ms', 63 | 'mt', 'my', 'na', 'nb', 'nd', 'ne', 'ng', 'nl', 'nn', 'no', 'nr', 'nv', 64 | 'ny', 'oc', 'oj', 'om', 'or', 'os', 'pa', 'pi', 'pl', 'ps', 'pt', 'qu', 65 | 'rm', 'rn', 'ro', 'ru', 'rw', 'sa', 'sc', 'sd', 'se', 'sg', 'si', 'sk', 66 | 'sl', 'sm', 'sn', 'so', 'sq', 'sr', 'ss', 'st', 'su', 'sv', 'sw', 'ta', 67 | 'te', 'tg', 'th', 'ti', 'tk', 'tl', 'tn', 'to', 'tr', 'ts', 'tt', 'tw', 68 | 'ty', 'ug', 'uk', 'ur', 'uz', 've', 'vi', 'vo', 'wa', 'wo', 'xh', 'yi', 69 | 'yo', 'za', 'zh', 'zu' 70 | ] 71 | 72 | 73 | def look_for_fixme(func): 74 | """Decorator to fail test if text argument starts with "FIXME".""" 75 | 76 | def inner(arg): 77 | if (arg is not None) and \ 78 | isinstance(arg, str) and \ 79 | arg.lstrip().startswith('FIXME'): 80 | return False 81 | return func(arg) 82 | return inner 83 | 84 | 85 | @look_for_fixme 86 | def check_layout(layout): 87 | '''"layout" in YAML header must be "workshop".''' 88 | 89 | return layout == 'workshop' 90 | 91 | 92 | @look_for_fixme 93 | def check_carpentry(layout): 94 | '''"carpentry" in YAML header must be "dc", "swc", "lc", or "cp".''' 95 | 96 | return layout in CARPENTRIES 97 | 98 | 99 | @look_for_fixme 100 | def check_country(country): 101 | '''"country" must be a lowercase ISO-3166 two-letter code.''' 102 | 103 | return country in ISO_COUNTRY 104 | 105 | 106 | @look_for_fixme 107 | def check_language(language): 108 | '''"language" must be a lowercase ISO-639 two-letter code.''' 109 | 110 | return language in ISO_LANGUAGE 111 | 112 | 113 | @look_for_fixme 114 | def check_humandate(date): 115 | """ 116 | 'humandate' must be a human-readable date with a 3-letter month 117 | and 4-digit year. Examples include 'Feb 18-20, 2025' and 'Feb 18 118 | and 20, 2025'. It may be in languages other than English, but the 119 | month name should be kept short to aid formatting of the main 120 | Carpentries web site. 121 | """ 122 | 123 | if ',' not in date: 124 | return False 125 | 126 | month_dates, year = date.split(',') 127 | 128 | # The first three characters of month_dates are not empty 129 | month = month_dates[:3] 130 | if any(char == ' ' for char in month): 131 | return False 132 | 133 | # But the fourth character is empty ("February" is illegal) 134 | if month_dates[3] != ' ': 135 | return False 136 | 137 | # year contains *only* numbers 138 | try: 139 | int(year) 140 | except: 141 | return False 142 | 143 | return True 144 | 145 | 146 | @look_for_fixme 147 | def check_humantime(time): 148 | """ 149 | 'humantime' is a human-readable start and end time for the 150 | workshop, such as '09:00 - 16:00'. 151 | """ 152 | 153 | return bool(re.match(HUMANTIME_PATTERN, time.replace(' ', ''))) 154 | 155 | 156 | def check_date(this_date): 157 | """ 158 | 'startdate' and 'enddate' are machine-readable start and end dates 159 | for the workshop, and must be in YYYY-MM-DD format, e.g., 160 | '2015-07-01'. 161 | """ 162 | 163 | # YAML automatically loads valid dates as datetime.date. 164 | return isinstance(this_date, date) 165 | 166 | 167 | @look_for_fixme 168 | def check_latitude_longitude(latlng): 169 | """ 170 | 'latlng' must be a valid latitude and longitude represented as two 171 | floating-point numbers separated by a comma. 172 | """ 173 | 174 | try: 175 | lat, lng = latlng.split(',') 176 | lat = float(lat) 177 | lng = float(lng) 178 | return (-90.0 <= lat <= 90.0) and (-180.0 <= lng <= 180.0) 179 | except ValueError: 180 | return False 181 | 182 | 183 | def check_instructors(instructors): 184 | """ 185 | 'instructor' must be a non-empty comma-separated list of quoted 186 | names, e.g. ['First name', 'Second name', ...']. Do not use 'TBD' 187 | or other placeholders. 188 | """ 189 | 190 | # YAML automatically loads list-like strings as lists. 191 | return isinstance(instructors, list) and len(instructors) > 0 192 | 193 | 194 | def check_helpers(helpers): 195 | """ 196 | 'helper' must be a comma-separated list of quoted names, 197 | e.g. ['First name', 'Second name', ...']. The list may be empty. 198 | Do not use 'TBD' or other placeholders. 199 | """ 200 | 201 | # YAML automatically loads list-like strings as lists. 202 | return isinstance(helpers, list) and len(helpers) >= 0 203 | 204 | 205 | @look_for_fixme 206 | def check_emails(emails): 207 | """ 208 | 'emails' must be a comma-separated list of valid email addresses. 209 | The list may be empty. A valid email address consists of characters, 210 | an '@', and more characters. It should not contain the default contact 211 | """ 212 | 213 | # YAML automatically loads list-like strings as lists. 214 | if (isinstance(emails, list) and len(emails) >= 0): 215 | for email in emails: 216 | if ((not bool(re.match(EMAIL_PATTERN, email))) or (email == DEFAULT_CONTACT_EMAIL)): 217 | return False 218 | else: 219 | return False 220 | 221 | return True 222 | 223 | 224 | def check_eventbrite(eventbrite): 225 | """ 226 | 'eventbrite' (the Eventbrite registration key) must be 9 or more 227 | digits. It may appear as an integer or as a string. 228 | """ 229 | 230 | if isinstance(eventbrite, int): 231 | return True 232 | else: 233 | return bool(re.match(EVENTBRITE_PATTERN, eventbrite)) 234 | 235 | 236 | @look_for_fixme 237 | def check_collaborative_notes(collaborative_notes): 238 | """ 239 | 'collaborative_notes' must be a valid URL. 240 | """ 241 | 242 | return bool(re.match(URL_PATTERN, collaborative_notes)) 243 | 244 | 245 | @look_for_fixme 246 | def check_pass(value): 247 | """ 248 | This test always passes (it is used for 'checking' things like the 249 | workshop address, for which no sensible validation is feasible). 250 | """ 251 | 252 | return True 253 | 254 | 255 | HANDLERS = { 256 | 'layout': (True, check_layout, 'layout isn\'t "workshop"'), 257 | 258 | 'carpentry': (True, check_carpentry, 'carpentry isn\'t in ' + 259 | ', '.join(CARPENTRIES)), 260 | 261 | 'country': (True, check_country, 262 | 'country invalid: must use lowercase two-letter ISO code ' + 263 | 'from ' + ', '.join(ISO_COUNTRY)), 264 | 265 | 'language': (False, check_language, 266 | 'language invalid: must use lowercase two-letter ISO code' + 267 | ' from ' + ', '.join(ISO_LANGUAGE)), 268 | 269 | 'humandate': (True, check_humandate, 270 | 'humandate invalid. Please use three-letter months like ' + 271 | '"Jan" and four-letter years like "2025"'), 272 | 273 | 'humantime': (True, check_humantime, 274 | 'humantime doesn\'t include numbers'), 275 | 276 | 'startdate': (True, check_date, 277 | 'startdate invalid. Must be of format year-month-day, ' + 278 | 'i.e., 2014-01-31'), 279 | 280 | 'enddate': (False, check_date, 281 | 'enddate invalid. Must be of format year-month-day, i.e.,' + 282 | ' 2014-01-31'), 283 | 284 | 'latlng': (True, check_latitude_longitude, 285 | 'latlng invalid. Check that it is two floating point ' + 286 | 'numbers, separated by a comma'), 287 | 288 | 'instructor': (True, check_instructors, 289 | 'instructor list isn\'t a valid list of format ' + 290 | '["First instructor", "Second instructor",..]'), 291 | 292 | 'helper': (True, check_helpers, 293 | 'helper list isn\'t a valid list of format ' + 294 | '["First helper", "Second helper",..]'), 295 | 296 | 'email': (True, check_emails, 297 | 'contact email list isn\'t a valid list of format ' + 298 | '["me@example.org", "you@example.org",..] or contains incorrectly formatted email addresses or ' + 299 | '"{0}".'.format(DEFAULT_CONTACT_EMAIL)), 300 | 301 | 'eventbrite': (False, check_eventbrite, 'Eventbrite key appears invalid'), 302 | 303 | 'collaborative_notes': (False, check_collaborative_notes, 'Collaborative Notes URL appears invalid'), 304 | 305 | 'venue': (False, check_pass, 'venue name not specified'), 306 | 307 | 'address': (False, check_pass, 'address not specified') 308 | } 309 | 310 | # REQUIRED is all required categories. 311 | REQUIRED = {k for k in HANDLERS if HANDLERS[k][0]} 312 | 313 | # OPTIONAL is all optional categories. 314 | OPTIONAL = {k for k in HANDLERS if not HANDLERS[k][0]} 315 | 316 | 317 | def check_blank_lines(reporter, raw): 318 | """ 319 | Blank lines are not allowed in category headers. 320 | """ 321 | 322 | lines = [(i, x) for (i, x) in enumerate( 323 | raw.strip().split('\n')) if not x.strip()] 324 | reporter.check(not lines, 325 | None, 326 | 'Blank line(s) in header: {0}', 327 | ', '.join(["{0}: {1}".format(i, x.rstrip()) for (i, x) in lines])) 328 | 329 | 330 | def check_categories(reporter, left, right, msg): 331 | """ 332 | Report differences (if any) between two sets of categories. 333 | """ 334 | 335 | diff = left - right 336 | reporter.check(len(diff) == 0, 337 | None, 338 | '{0}: offending entries {1}', 339 | msg, sorted(list(diff))) 340 | 341 | 342 | def check_file(reporter, path, data): 343 | """ 344 | Get header from file, call all other functions, and check file for 345 | validity. 346 | """ 347 | 348 | # Get metadata as text and as YAML. 349 | raw, header, body = split_metadata(path, data) 350 | 351 | # Do we have any blank lines in the header? 352 | check_blank_lines(reporter, raw) 353 | 354 | # Look through all header entries. If the category is in the input 355 | # file and is either required or we have actual data (as opposed to 356 | # a commented-out entry), we check it. If it *isn't* in the header 357 | # but is required, report an error. 358 | for category in HANDLERS: 359 | required, handler, message = HANDLERS[category] 360 | if category in header: 361 | if required or header[category]: 362 | reporter.check(handler(header[category]), 363 | None, 364 | '{0}\n actual value "{1}"', 365 | message, header[category]) 366 | elif required: 367 | reporter.add(None, 368 | 'Missing mandatory key "{0}"', 369 | category) 370 | 371 | # Check whether we have missing or too many categories 372 | seen_categories = set(header.keys()) 373 | check_categories(reporter, REQUIRED, seen_categories, 374 | 'Missing categories') 375 | check_categories(reporter, seen_categories, REQUIRED.union(OPTIONAL), 376 | 'Superfluous categories') 377 | 378 | 379 | def check_config(reporter, filename): 380 | """ 381 | Check YAML configuration file. 382 | """ 383 | 384 | config = load_yaml(filename) 385 | 386 | kind = config.get('kind', None) 387 | reporter.check(kind == 'workshop', 388 | filename, 389 | 'Missing or unknown kind of event: {0}', 390 | kind) 391 | 392 | carpentry = config.get('carpentry', None) 393 | reporter.check(carpentry in ('swc', 'dc', 'lc', 'cp'), 394 | filename, 395 | 'Missing or unknown carpentry: {0}', 396 | carpentry) 397 | 398 | 399 | def main(): 400 | '''Run as the main program.''' 401 | 402 | if len(sys.argv) != 2: 403 | print(USAGE, file=sys.stderr) 404 | sys.exit(1) 405 | 406 | root_dir = sys.argv[1] 407 | index_file = os.path.join(root_dir, 'index.html') 408 | config_file = os.path.join(root_dir, '_config.yml') 409 | 410 | reporter = Reporter() 411 | check_config(reporter, config_file) 412 | check_unwanted_files(root_dir, reporter) 413 | with open(index_file) as reader: 414 | data = reader.read() 415 | check_file(reporter, index_file, data) 416 | reporter.report() 417 | 418 | 419 | if __name__ == '__main__': 420 | main() 421 | -------------------------------------------------------------------------------- /episodes/carbon-awareness.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Carbon Awareness 3 | teaching: 20 4 | exercises: 15 5 | --- 6 | 7 | ::::::::::::::::::::::::::::::::::::::: objectives 8 | 9 | - Understand the concept of *carbon intensity* in electricity generation. 10 | - Appreciate the level of variation in carbon intensity across the UK and the world. 11 | - Understand techniques for greener use of HPC: demand shifting and demand shaping. 12 | 13 | :::::::::::::::::::::::::::::::::::::::::::::::::: 14 | 15 | :::::::::::::::::::::::::::::::::::::::: questions 16 | 17 | - How does electricity generation affect GHG emissions? 18 | - What is *carbon intensity* of electricity generation and how does it vary geographically and temporally? 19 | - What techniques can I use to make my use of HPC greener and influence transition to low-carbon electricity generation? 20 | 21 | :::::::::::::::::::::::::::::::::::::::::::::::::: 22 | 23 | ## Introduction 24 | 25 | Not all electricity is produced in the same way. In different locations and times, electricity is generated using a variety of sources with varying carbon emissions. Some sources (such as wind, solar, or hydroelectric) are clean, renewable sources that emit little carbon. On the other hand, fossil fuel sources emit carbon at varying degrees to produce electricity. For example, both gas and coal emit more carbon than renewable sources, but gas-burning power plants emit less carbon than coal-burning power plants. 26 | 27 | Carbon awareness is the idea of doing more when more energy comes from low carbon sources and doing less when more energy comes from high carbon sources. 28 | 29 | ## Key concepts 30 | 31 | ### Carbon intensity 32 | 33 | *Carbon Intensity (CI)* measures how much carbon (CO2e) is emitted per kilowatt-hour (kWh) of electricity consumed. The standard unit of carbon intensity is gCO2e/kWh, or grams of carbon per kilowatt hour. 34 | 35 | If your computer is plugged directly into a wind farm, its electricity would have a carbon intensity of 0 gCO2e/kWh since a wind farm emits no carbon to produce that electricity. However, most people cannot plug directly into wind farms; instead, they plug into power grids supplied with electricity from various sources. 36 | 37 | :::::::::::::::::::::::::::::::::::::::: callout 38 | 39 | ## Embodied carbon of renewable sources 40 | 41 | In reality, the carbon intensity of renewable sources still have some GHG emissions associated with them from the emissions used to build, operate and decommission them. As we will see in a later episode, these emissions are usually amortised across the lifetime of the facility (in this case, the lifetime of the generation facility). By convention, these embodied emissions are not usually included in carbon intensity values for generated energy as they are complex to calculate and the emissions saved by replacing non-renewable sources with renewable sources have vastly outweighed the embodied emissions of renewable sources. These emissions sources will become a more important component of energy generation as electricity grids continue to decarbonise. 42 | 43 | :::::::::::::::::::::::::::::::::::::::::::::::::: 44 | 45 | Once on a grid, you can not control which sources supply the electricity you are using; you simply get a mix of everything. So, your carbon intensity will be a mix of all the current power sources in a grid, both the lower- and the higher-carbon sources. 46 | 47 | ### Variability of carbon intensity 48 | 49 | Carbon intensity varies by location since some regions have an energy mix containing more clean energy sources than others. This table shows the average carbon intensity for UK regions over 2024 (ref: [electricityinfo.org regional carbon intensity history](https://electricityinfo.org/region-archive/)): 50 | 51 | | Type | Regions | Carbon Intensity (gCO2e/kWh) | 52 | |--|------|------:| 53 | | Low | N. Scotland, S. Scotland, N.E. England, N.W. England | 22 - 48 | 54 | | Low Medium | N. Wales | 77 | 55 | | Medium | E. England, London, W. Midlands, S.E. England, Yorkshire | 108 - 135 | 56 | | High Medium | S. England, E. Midlands | 186 - 203 | 57 | | High | S.W. England, S. Wales, | 242 - 255 | 58 | 59 | Similarly, this table compares values across the world in 2023 (ref: [Our World In Data](https://ourworldindata.org/grapher/carbon-intensity-electricity?tab=table)): 60 | 61 | | Country | Carbon Intensity (gCO2e/kWh) | 62 | |--------|------:| 63 | | France | 56 | 64 | | UK | 238 | 65 | | USA | 369 | 66 | | Germany | 381 | 67 | | Russia | 441 | 68 | | Japan | 485 | 69 | | Australia | 549 | 70 | 71 | Carbon intensity also changes over time due to the inherent variability of renewable energy caused by the unpredictability of weather conditions. For example, when it is cloudy or the wind is not blowing, carbon intensity increases since more of the electricity in your mix comes from sources that emit carbon. 72 | 73 | ![Diagram illustrating change in carbon intensity over time due to changes in atmospheric conditions](./fig/08_variability_CI.png "Diagram illustrating change in carbon intensity over time due to changes in atmospheric conditions") 74 | 75 | ::::::::::::::::::::::::::::::::::::::: challenge 76 | 77 | ## Exercise: Carbon emissions from HPC systems 78 | 79 | One estimate of the power draw of the ARCHER2 HPC system is 3.1 MW (including overheads from the power/cooling plant). Mean carbon intensities from different UK regions in 2024 give low emissions regions as ~30 gCO2e/kWh, medium emissions regions as ~120 gCO2e/kWh and high emissions regions as ~250 gCO2e/kWh. What would the carbon emissions be from electricity use in 1 year of ARCHER2 operations in the three different emissions regimes? 80 | 81 | ::::::::::::::: solution 82 | 83 | ## Solution 84 | 85 | First we need to estimate the amount of energy consumed by ARCHER2 in kWh for 1 year from the power draw estimate, this is given by: 86 | 87 | ``` 88 | 3,100 kW x 365 days x 24 hours = 27,156,000 kWh 89 | ``` 90 | 91 | Now, we can multiply this energy use by the carbon intensity values to get the estimated emissions from a year of ARCHER2 operation in the three different locations. e.g. for the low emissions location: 92 | 93 | ``` 94 | 27,156,000 kWh x 0.030 kgCO2e/kWh = 815,000 kgCO2e 95 | ``` 96 | 97 | - Low CI (30 gCO2e/kWh) = 815,000 kgCO2e 98 | - Medium CI (120 gCO2e/kWh) = 3,260,720.0 kgCO2e 99 | - High CI (250 gCO2e/kWh) = 6,790,000.0 kgCO2e 100 | 101 | Hosting ARCHER2 in a high CI region instead of a low CI region would lead to an additional 5,980,000 kgCO2e emissions per year. 102 | 103 | ::::::::::::::::::::::::: 104 | 105 | ::::::::::::::::::::::::::::::::::::::: 106 | 107 | ### Dispatchability & curtailment 108 | 109 | Electricity demand varies during the day and supply always needs to be able to meet that demand. A brownout (a dip in the voltage level of the power line) occurs if a utility does not produce enough electricity to meet demand. Conversely, if a utility produces more electricity than is required, then to stop infrastructure burning out, breakers trip and we have blackouts. 110 | 111 | There needs to be a balance between the demand and supply of electricity at all times and the responsibility for this usually falls to the utility provider. 112 | 113 | In the case of fossil fuels such as coal, it is easier to control the power produced for this supply; this is called **dispatchability**. However, in the case of renewable power sources such as wind farms, the power produced cannot easily be controlled (we cannot control how much the wind blows). If the power source produces more electricity than is needed, that electricity is thrown away; this is called **curtailment**. 114 | 115 | ### Marginal carbon intensity 116 | 117 | If you suddenly need to access more power - for example, you need to turn on a light - that energy comes from the marginal power plant. The marginal power plant is dispatchable, which means marginal power plants are often powered by fossil fuels. 118 | 119 | Marginal carbon intensity is the carbon intensity of the power plant that would have to be employed to meet any new demand. 120 | 121 | Fossil-fueled power plants rarely scale down to 0. They have a minimum functioning threshold, and some do not scale; they are considered a consistent, always-on baseload. Because of this, we sometimes have the scenario where we curtail (throw away) renewable energy while still consuming energy from fossil fuel power plants. 122 | 123 | ![Diagram illustrating marginal carbon intensity and curtailment](./fig/09_marginal_CI.png "Diagram illustrating marginal carbon intensity and curtailment") 124 | 125 | In these situations, the marginal carbon intensity will be 0 gCO2e/kWh since we know that any new demand will match the renewable energy we are curtailing. 126 | 127 | ### Energy markets 128 | 129 | The exact market model varies around the world but broadly follows the same model. 130 | 131 | When the demand for electricity goes down, utilities need to **reduce** the supply to balance supply and demand. They can do this in one of two ways: 132 | 133 | 1. **Buy less energy from fossil fuel plants.** 134 | 135 | ![Diagram illustrating reduction in demand from fossil fuel plants](./fig/10_marginal_CI.png "Diagram illustrating reduction in demand from fossil fuel plants") 136 | 137 | Energy from fossil fuel plants is usually the most expensive so this is the preferred method. This directly translates to burning fewer fossil fuels. 138 | 139 | 2. **Buy less energy from renewable sources**. 140 | 141 | Renewable sources are the cheapest, so they prefer not to do this. If a renewable source does not manage to sell all of its electricity, it has to throw the rest away. 142 | 143 | Reducing the amount of electricity consumed by your use of HPC can help decrease the carbon intensity of the energy required as the first thing to be scaled back are fossil fuels. 144 | 145 | When the demand for electricity goes up, utilities need to **increase** the supply to balance supply and demand. They can do this in one of two ways: 146 | 147 | 1. **Buy more energy from renewable sources that are currently being curtailed** 148 | 149 | ![Diagram illustrating increase in demand from renewable sources](./fig/11_marginal_CI.png "Diagram illustrating increase in demand from renewable sources") 150 | 151 | If you are curtailing, it means you have excess energy you could dispatch. Renewable energy is already the cheapest, so curtailed renewable energy will be the cheapest dispatchable energy source. Renewable plants will then sell the energy they would have had to curtail. 152 | 153 | 2. **Buy more energy from fossil fuel plants**. 154 | 155 | ![Diagram illustrating increase in demand from non-renewable sources](./fig/12_marginal_CI.png "Diagram illustrating increase in demand from non-renewable sources") 156 | 157 | Fossil fuels are inherently dispatchable; they can quickly increase energy production by burning more. However, coal costs money, so this is the least preferred solution. 158 | 159 | Energy markets are some of the most complex markets in the world so the above explanation is a simplification. But what is important to understand is that our goal is to increase investment into lower carbon energy sources, like renewables, and decrease investment into higher carbon sources, like coal. The best way to ensure money flows in the right direction is to make sure you use electricity with the least carbon intensity. 160 | 161 | ## How to be more carbon aware 162 | 163 | ::: tip 164 | 165 | Using electricity when the carbon intensity is low is the best way to ensure investment flows towards low-carbon emitting plants and away from high-carbon emitting plants. 166 | 167 | ::: 168 | 169 | ![Image of transition from high-carbon to low-carbon energy sources](./fig/13_carbon_aware.png "Image of transition from high-carbon to low-carbon energy sources") 170 | 171 | There is a global transformation happening right now. All around the world, electricity grids are changing from primarily burning fossil fuels to sourcing energy from lower carbon sources like wind and solar. This is one of our best hopes for meeting our global reduction targets. As green users of HPC systems, let's see some of the ways we can help accelerate that transition. 172 | 173 | The primary driver for the transition is economic rather than any sustainability target. Renewables are winning because they are cheaper and getting even more affordable over time. So, to help accelerate the transition, we need to make renewable plants more profitable and fossil fuel plants less profitable. The best way to do that is to use more electricity when it's coming from lower-carbon sources like renewables and less electricity when it's coming from higher-carbon sources. 174 | 175 | Carbon intensity is lower when more energy comes from lower-carbon sources and higher when it comes from higher-carbon sources. 176 | 177 | ### Demand shifting 178 | 179 | Being carbon aware means responding to shifts in carbon intensity by increasing or decreasing your demand. If your work allows you to be flexible with when and where you run workloads, you can shift accordingly - consuming electricity when the carbon intensity is lower and pausing production when it is higher. For example, running a simulation or model at a different time or in a different region with much lower carbon intensity. 180 | 181 | [Studies](https://ieeexplore.ieee.org/document/6128960) show these actions can result in 45% to 99% carbon reductions depending on the number of renewables powering the grid. 182 | 183 | Demand shifting can be further broken down into spatial and temporal shifting. 184 | 185 | #### Spatial shifting 186 | 187 | Spatial shifting means moving your computation to another physical location where the current carbon intensity is lower. It might be a region that naturally has lower carbon sources of energy. For example, using an HPC facility sited in a location that is currently windy and so has a high proportion of wind power. 188 | 189 | ![Diagram illustrating spatial demand shifting](./fig/14_spatial_shifting.png "Diagram illustrating spatial demand shifting") 190 | 191 | This requires you to have the ability to run on different HPC resources in different locations - for example, you could chose between running on a local HPC resource or a national HPC resource located elsewhere. 192 | 193 | #### Temporal shifting 194 | 195 | If you cannot shift your computation spatially to another region, another option you have is to shift to another time. Perhaps later in the day or night when it's sunnier or windier and, therefore, the carbon intensity is lower. This is called temporal demand shifting. We can predict future carbon intensity reasonably well through advances in weather forecasting. 196 | 197 | ![Diagram illustrating spatial demand shifting](./fig/15_temporal_shifting.png "Diagram illustrating spatial demand shifting") 198 | 199 | An example of a service that can predict carbon intensity to enable temporal demand shifting is the [carbonintensity.org.uk website and API](https://carbonintensity.org.uk). 200 | 201 | Some of the large technology companies have recognised the importance of carbon awareness and are using advanced modeling techniques to implement demand shifting. 202 | 203 | - **Google Carbon Aware Data Centers** - Google launched a project to [make some of the cloud workloads carbon aware](https://blog.google/outreach-initiatives/sustainability/carbon-aware-computing-location/). They created models to predict tomorrow's carbon intensity and workload. They then shaped large-scale workloads so more would happen when and where the carbon intensity is lowest, but in such a way that they could still handle the expected load. 204 | - **Microsoft Carbon Aware Windows** - [Microsoft announced a project to make Windows 11 more sustainable](https://www.techradar.com/news/windows-11-is-getting-an-eco-friendly-update-but-could-microsoft-do-more). Initially, this means running Windows updates when the carbon intensity is lower. 205 | 206 | ::: tip 207 | 208 | To be able to take advantage of temporal shifting on HPC systems typically requires support for such approaches in the configuration of the job scheduling system and or policies of the service you are using. While this is not widespread at the moment, more and more services are looking at ways to enable this. We will think about and mention concrete actions and approaches to this below. 209 | 210 | ::: 211 | 212 | ### Demand shaping 213 | 214 | Demand shifting (as described above) is the strategy of moving computation to regions or times when the carbon intensity is lowest. Demand shaping is a similar strategy. However, instead of moving demand to a different region or time, we shape our computation to match the existing supply. 215 | 216 | ![Diagram illustrating demand shaping](./fig/16_demand_shaping.png "Diagram illustrating demand shaping") 217 | 218 | - If carbon intensity is low, increase the demand; do more (or faster) calculations. 219 | - If carbon intensity is high, decrease demand; do fewer (or slower) calculations. 220 | 221 | Demand shaping for carbon-aware HPC use is all about the supply of carbon. When the carbon cost of running your simulation or model becomes high, shape the demand to match the supply of carbon. This can happen automatically, or the user can make a choice. 222 | 223 | Eco mode is an example of demand shaping. Eco modes are found in everyday appliances like cars or washing machines. When activated, some amount of performance is sacrificed in order to consume fewer resources (gas or electricity). Because there is this trade-off with performance, eco modes are always presented to a user as a choice. 224 | 225 | Software or HPC services can also have eco modes that can - either automatically or with user consent - make decisions to reduce carbon emissions. 226 | 227 | One example of this is video conferencing software that adjusts streaming quality automatically. Rather than streaming at the highest quality possible at all times, it reduces the video quality to prioritise audio when the bandwidth is low. 228 | 229 | Another example is TCP/IP: the transfer speed increases in response to how much data is broadcast over the wire. 230 | 231 | A third example is progressive enhancement with the web. The web experience improves depending on the resources and bandwidth available on the end user's device. 232 | 233 | Demand shaping is related to a broader concept in sustainability, which is to reduce consumption. We can achieve a lot by becoming more efficient with resources, but we also need to consume less at some point. 234 | 235 | As green users of HPC systems, we could consider cancelling or reducing the power intensity of our workflow when the carbon intensity is high instead of demand shifting - reducing the energy demands of our work. 236 | 237 | ::::::::::::::::::::::::::::::::::::::: challenge 238 | 239 | ## Exercise: Demand shaping and temporal shifting in HPC 240 | 241 | A typical HPC system has a wide variety of jobs to schedule and different HPC systems have different scheduling and charging policies for jobs that run on them. Write down some ideas on how you could potentially modify the scheduling and charging of jobs to enable demand shaping and/or temporal shifting on an HPC system? 242 | 243 | ::::::::::::::: solution 244 | 245 | ## Solution 246 | 247 | - Scheduling based on power intensity 248 | - Manual: users place jobs into different queues based on predicted power intensity; charging discounts for users who use this facility 249 | - Automatic: system detects/predicts power intensity of jobs and schedules accordingly 250 | 251 | - Credit based system: 252 | - Earning and spending periods: during earning period, users start with a number of tokens and subtract based on efficiency (to stop more tokens for more use). During spend period, users with most efficiency from earning period get priority on system 253 | - This type of approach [has been piloted on the Fugaku HPC system in Japan](https://doi.ieeecomputersociety.org/10.1109/SC41406.2024.00030) 254 | 255 | 256 | - Power capping approaches: 257 | - System power cap fluctuates with grid carbon intensity 258 | - Need a tool to distribute power cap amongst jobs 259 | - For example [HPE's PowerSched tool](https://cug.org/proceedings/cug2023_proceedings/includes/files/pap113s2-file1.pdf) 260 | 261 | ::::::::::::::::::::::::: 262 | 263 | ::::::::::::::::::::::::::::::::::::::: 264 | 265 | :::::::::::::::::::::::::::::::::::::::: keypoints 266 | 267 | - Carbon awareness means understanding that the energy you consume does not always have the same impact in terms of carbon intensity. 268 | - Carbon intensity varies depending on the time and place it is consumed. 269 | - The nature of fossil fuels and renewable energy sources means that consuming energy when carbon intensity is low increases the demand for renewable energy sources and increases the percentage of renewable energy in the supply. 270 | - Demand shifting means moving your energy consumption to different locations or times of days where the carbon intensity is lower. 271 | - Demand shaping means adapting your energy consumption around carbon intensity variability in order to consume more in periods of low intensity and less in periods of high intensity. 272 | 273 | :::::::::::::::::::::::::::::::::::::::::::::::::: 274 | 275 | 276 | --------------------------------------------------------------------------------