├── .gitignore ├── .prettierrc.toml ├── jupyter-tunnel ├── jlab-casper.sh └── jlab-cheyenne.sh ├── scheduled-workflows ├── README.md ├── tasks.py └── glade-catalog-update.py ├── environments ├── catalog-builder.yml ├── postBuild ├── playground-postBuild ├── pangeo-env.yml └── playground-environment.yml ├── setup.cfg ├── pyproject.toml ├── LICENSE ├── .pre-commit-config.yaml └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | .vscode/settings.json 2 | nohup.out 3 | *.out 4 | -------------------------------------------------------------------------------- /.prettierrc.toml: -------------------------------------------------------------------------------- 1 | tabWidth = 2 2 | semi = false 3 | singleQuote = true 4 | -------------------------------------------------------------------------------- /jupyter-tunnel/jlab-casper.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "ssh -N -L 8877:`hostname`:8877 $USER@`hostname`.ucar.edu" 4 | jupyter lab --no-browser --ip=`hostname` --port=8877 5 | -------------------------------------------------------------------------------- /jupyter-tunnel/jlab-cheyenne.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "ssh -N -L 8877:`hostname`:8877 $USER@cheyenne.ucar.edu" 4 | jupyter lab --no-browser --ip=`hostname` --port=8877 5 | -------------------------------------------------------------------------------- /scheduled-workflows/README.md: -------------------------------------------------------------------------------- 1 | # Scheduled Workflows 2 | 3 | ## GLADE CMIP6 Catalog Update 4 | 5 | ```bash 6 | conda activate esm-catalog-builder 7 | invoke update 8 | ``` 9 | -------------------------------------------------------------------------------- /environments/catalog-builder.yml: -------------------------------------------------------------------------------- 1 | name: esm-catalog-builder 2 | channels: 3 | - conda-forge 4 | dependencies: 5 | - python 6 | - intake 7 | - pandas 8 | - dask 9 | - requests 10 | - click 11 | - pip 12 | - invoke 13 | - pip: 14 | - schedule 15 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [flake8] 2 | ignore = 3 | max-line-length = 100 4 | max-complexity = 18 5 | select = B,C,E,F,W,T4,B9 6 | 7 | [isort] 8 | known_first_party= 9 | known_third_party=invoke 10 | multi_line_output=3 11 | include_trailing_comma=True 12 | force_grid_wrap=0 13 | combine_as_imports=True 14 | line_length=100 15 | skip= 16 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.black] 2 | line-length = 100 3 | target-version = ['py38'] 4 | skip-string-normalization = true 5 | 6 | [tool.nbqa.config] 7 | isort = "setup.cfg" 8 | black = "pyproject.toml" 9 | 10 | [tool.nbqa.mutate] 11 | isort = 1 12 | black = 1 13 | pyupgrade = 1 14 | 15 | [tool.nbqa.addopts] 16 | pyupgrade = ["--py36-plus"] 17 | -------------------------------------------------------------------------------- /environments/postBuild: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | jupyter serverextension enable --py nbserverproxy --sys-prefix 3 | jupyter labextension install @jupyter-widgets/jupyterlab-manager \ 4 | @pyviz/jupyterlab_pyviz \ 5 | jupyter-leaflet \ 6 | dask-labextension \ 7 | @jupyterlab/toc 8 | 9 | #EOF 10 | -------------------------------------------------------------------------------- /environments/playground-postBuild: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | jupyter labextension install \ 4 | @jupyter-widgets/jupyterlab-manager \ 5 | dask-labextension jupyterlab-nvdashboard \ 6 | @jupyter-widgets/jupyterlab-sidecar bqplot \ 7 | jupyter-threejs jupyter-leaflet ipysheet \ 8 | ipytree ipycanvas jupyter-matplotlib jupyter-vuetify ipyvolume \ 9 | @jupyter-voila/jupyterlab-preview @pyviz/jupyterlab_pyviz \ 10 | @krassowski/jupyterlab-lsp @mamba-org/gator-lab 11 | 12 | 13 | #EOF 14 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Anderson Banihirwe 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /environments/pangeo-env.yml: -------------------------------------------------------------------------------- 1 | name: pangeo 2 | channels: 3 | - pyviz 4 | - conda-forge 5 | - intake 6 | - default 7 | dependencies: 8 | - autopep8 9 | - black 10 | - bokeh 11 | - datashader 12 | - dask 13 | - dask-jobqueue 14 | - dask-ml 15 | - distributed 16 | - flake8 17 | - gcsfs 18 | - geoviews 19 | - graphviz 20 | - holoviews 21 | - hvplot 22 | - intake 23 | - iris 24 | - iris-sample-data 25 | - importnb 26 | - intake-xarray 27 | - ipywidgets 28 | - jupyter 29 | - jupyterlab 30 | - jupyter_contrib_nbextensions 31 | - jupyter_nbextensions_configurator 32 | - lz4 33 | - matplotlib 34 | - metakernel 35 | - mpi4py 36 | - nb_conda_kernels 37 | - netcdf4 38 | - numpy 39 | - nbserverproxy 40 | - nodejs 41 | - numcodecs 42 | - pandas 43 | - python=3.6 44 | - python-blosc 45 | - pytest 46 | - scikit-image 47 | - scikit-learn 48 | - scipy 49 | - tensorflow 50 | - unyt 51 | - xarray 52 | - xonsh 53 | - xo 54 | - yt 55 | - zarr 56 | - twine 57 | - wheel 58 | - sphinx 59 | - numpydoc 60 | - sphinx_rtd_theme 61 | - pip: 62 | - bash_kernel 63 | - version-information 64 | - xonsh_kernel 65 | # - kubernetes 66 | # - git+https://github.com/dask/dask-kubernetes.git 67 | # - git+https://github.com/xgcm/xgcm.git 68 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | repos: 2 | - repo: https://github.com/pre-commit/pre-commit-hooks 3 | rev: v3.3.0 4 | hooks: 5 | - id: trailing-whitespace 6 | - id: end-of-file-fixer 7 | - id: check-docstring-first 8 | - id: check-json 9 | - id: check-yaml 10 | - id: double-quote-string-fixer 11 | 12 | - repo: https://github.com/ambv/black 13 | rev: 20.8b1 14 | hooks: 15 | - id: black 16 | 17 | - repo: https://github.com/keewis/blackdoc 18 | rev: v0.3 19 | hooks: 20 | - id: blackdoc 21 | 22 | - repo: https://gitlab.com/pycqa/flake8 23 | rev: 3.8.4 24 | hooks: 25 | - id: flake8 26 | 27 | - repo: https://github.com/asottile/seed-isort-config 28 | rev: v2.2.0 29 | hooks: 30 | - id: seed-isort-config 31 | - repo: https://github.com/pre-commit/mirrors-isort 32 | rev: v5.6.4 33 | hooks: 34 | - id: isort 35 | 36 | - repo: https://github.com/prettier/pre-commit 37 | rev: 57f39166b5a5a504d6808b87ab98d41ebf095b46 38 | hooks: 39 | - id: prettier 40 | 41 | - repo: https://github.com/nbQA-dev/nbQA 42 | rev: 0.5.1 43 | hooks: 44 | - id: nbqa-black 45 | additional_dependencies: [black==20.8b1] 46 | - id: nbqa-pyupgrade 47 | additional_dependencies: [pyupgrade==2.7.3] 48 | - id: nbqa-isort 49 | additional_dependencies: [isort==5.6.4] 50 | -------------------------------------------------------------------------------- /environments/playground-environment.yml: -------------------------------------------------------------------------------- 1 | name: playground 2 | channels: 3 | - nvidia 4 | - conda-forge 5 | - pytorch 6 | - fastai 7 | dependencies: 8 | # - libnetcdf=4.7.4=*_105 9 | - arviz 10 | - binutils-meta 11 | - bokeh>=2.2.3 12 | - bqplot 13 | - c-compiler 14 | - cartopy 15 | - cmip6_preprocessing 16 | - cmocean 17 | - compilers 18 | - cudatoolkit=10.2 19 | - cudnn 20 | - cupy>=8.1.0 21 | - cxx-compiler 22 | - cython 23 | - dask-jobqueue 24 | - dask-labextension 25 | - dask>=2.30 26 | - datashader 27 | - distributed>=2.30 28 | - fastai 29 | - fortran-compiler 30 | - gcsfs 31 | - geemap 32 | - geoviews 33 | - graphviz 34 | - hvplot 35 | - intake 36 | - intake-esm>=2020.11.4 37 | - intake-xarray 38 | - ipycanvas 39 | - ipyevents 40 | - ipyleaflet>=0.12.6 41 | - ipympl 42 | - ipysheet 43 | - ipytree 44 | - ipyvolume>=0.6.0a6 45 | - ipyvuetify 46 | - ipywidgets 47 | - jax 48 | - jaxlib 49 | - jupyter-lsp 50 | - jupyterlab 51 | - jupyterlab-nvdashboard 52 | - kaggle 53 | - libopencv 54 | - line_profiler 55 | - mamba_gator 56 | - matplotlib 57 | - mkl 58 | - mkl-service 59 | - nc-time-axis 60 | - nodejs 61 | - numba>=0.51.2 62 | - numpyro 63 | - opencv 64 | - panel 65 | - pip 66 | - proplot 67 | - py-opencv 68 | - pyjanitor 69 | - pymc3>=3.9 70 | - pysnooper 71 | - python-graphviz 72 | - python-language-server 73 | - python=3.8 74 | - pythreejs 75 | - pytorch>=1.7.0 76 | - pyvista 77 | - pywwt 78 | - redis 79 | - redis-py 80 | - regionmask 81 | - s3fs 82 | - scikit-learn 83 | - scipy 84 | - seaborn 85 | - sidecar 86 | - spacy 87 | - spatialpandas 88 | - tensorboard 89 | - theano 90 | - torchvision 91 | - tqdm 92 | - voila-vuetify 93 | - voila>=0.1 94 | - vtk 95 | - watermark 96 | - xarray>=0.16 97 | - xesmf>=0.4 98 | - xrft 99 | - zarr>=2.5 100 | # - theano-pymc 101 | - pip: 102 | - handcalcs 103 | - scalene 104 | - sktime 105 | -------------------------------------------------------------------------------- /scheduled-workflows/tasks.py: -------------------------------------------------------------------------------- 1 | import json 2 | import os 3 | from contextlib import contextmanager 4 | from datetime import datetime 5 | 6 | from invoke import task 7 | 8 | last_updated = datetime.now().utcnow().strftime('%Y-%m-%dT%H:%M:%SZ') 9 | main_repo = '/glade/collections/cmip/catalog/intake-esm-datastore' 10 | 11 | 12 | @contextmanager 13 | def chdir(path): 14 | """Change working directory to `path` and restore it again 15 | This context manager is useful if `path` stops existing during your 16 | operations. 17 | """ 18 | old_dir = os.getcwd() 19 | os.chdir(path) 20 | try: 21 | yield 22 | finally: 23 | os.chdir(old_dir) 24 | 25 | 26 | @task 27 | def build(c): 28 | with chdir(f'{main_repo}/builders'): 29 | 30 | catalog_path = '../catalogs/glade-cmip6.csv.gz' 31 | command = f"""git pull origin master && source activate esm-catalog-builder && 32 | python cmip.py --root-path /glade/collections/cmip/CMIP6 \ 33 | --pick-latest-version \ 34 | --cmip-version 6 \ 35 | --csv-filepath {catalog_path} \ 36 | --depth 4""" 37 | c.run(command) 38 | 39 | command = f'git diff --text {catalog_path} 2>/dev/null | wc -l' 40 | output = c.run(command).stdout 41 | if int(output) > 0: 42 | path = '../catalogs/glade-cmip6.json' 43 | with open(path, 'r') as f: 44 | data = json.load(f) 45 | data['last_updated'] = last_updated 46 | 47 | with open(path, 'w') as f: 48 | json.dump(data, f) 49 | 50 | 51 | @task(build) 52 | def update(c): 53 | with chdir(main_repo): 54 | commit_message = f'Scheduled CMIP6 catalog update: {last_updated}' 55 | cmd = 'source activate base && git add catalogs && pre-commit run' 56 | try: 57 | c.run(cmd) 58 | except Exception: 59 | pass 60 | 61 | cmd = f"git add catalogs && git commit -m '{commit_message}' && git push" 62 | c.run(cmd) 63 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Cheatsheet 2 | 3 | - [Cheatsheet](#cheatsheet) 4 | - [1. Glade Quota: `gladequota`](#1-glade-quota-gladequota) 5 | - [2. Using environment modules](#2-using-environment-modules) 6 | - [3. Interacting with the job schedulers](#3-interacting-with-the-job-schedulers) 7 | - [3.1 PBS on Cheyenne](#31-pbs-on-cheyenne) 8 | - [3.2 Slurm on DAV](#32-slurm-on-dav) 9 | - [4. Accessing Geyser, and Caldera using the Slurm scheduler](#4-accessing-geyser-and-caldera-using-the-slurm-scheduler) 10 | - [4.1. Basic commands for managing jobs when using Slurm](#41-basic-commands-for-managing-jobs-when-using-slurm) 11 | - [4.2. Starting interactive jobs](#42-starting-interactive-jobs) 12 | 13 | ## 1. Glade Quota: `gladequota` 14 | 15 | ``` 16 | abanihi@cheyenne1: ~ $ gladequota 17 | Current GLADE space usage: abanihi 18 | 19 | Space Used Quota % Full # Files 20 | ---------------------------------- ----------- ----------- --------- ----------- 21 | /glade/scratch/abanihi 0.00 TB 10.00 TB 0.00 % 4 22 | /glade/work/abanihi 0.64 GB 1024.00 GB 0.06 % 14098 23 | /glade/scratch_old/abanihi 0.00 TB 10.00 TB 0.00 % 741 24 | /glade/p_old/work/abanihi 0.00 GB 512.00 GB 0.00 % 1 25 | /glade/u/home/abanihi 1.54 GB 50.00 GB 3.08 % 19414 26 | 27 | /glade/scratch - 38.1% used (5712 TB used out of 15000 TB total) 28 | ``` 29 | 30 | ## 2. Using environment modules 31 | 32 | - `module add/remove ` 33 | - `module avail`: show all currently-loadable modules 34 | - `module list` : show loaded modules 35 | - `module purge` : remove all loaded modules 36 | - `module save/restore ` : create/load a saved set of software 37 | - `module spider ` : search for a particular module 38 | 39 | ## 3. Interacting with the job schedulers 40 | 41 | ### 3.1 PBS on Cheyenne 42 | 43 | - `qsub