├── .gitignore ├── 01_getting_started ├── 00_Setup.md ├── 01_QuickStart.ipynb ├── 02_CoreConcepts.md ├── 03_DataOrganization.ipynb ├── 04_RedshiftCatalogs.ipynb ├── 05_Spectra.ipynb ├── 06_PublicDatabaseAccess.ipynb ├── EDR │ ├── DESI_petal_healpix_rosette.ipynb │ ├── EDR_AnalyzeZcat_colab.ipynb │ ├── EDR_analyze_Zcat.ipynb │ ├── README.md │ ├── figures_from_EDR_paper.md │ ├── installing_EDR_kernels_at_NERSC.md │ ├── intro_to_DESI_EDR_files.ipynb │ └── intro_to_DESI_EDR_files_colab.ipynb └── README.md ├── 02_digging_deeper ├── README.md ├── RedrockOutputs.ipynb └── ResolutionMatrix.ipynb ├── DataQA ├── README.md ├── sframe-r6-00104282.png ├── sframe-r6-00104284.png ├── sframe-r6-00104285.png ├── sframe-r6-00104286.png ├── sframe-r6-beforeafter.png ├── sframe-r7-00104284.png ├── sframe-r8-00104284.png ├── sframe-sky-r6-00104284.png ├── skyline-6563-xy.png ├── skymodel-r6-00104284.png ├── skymodel-r6-beforeafter.png ├── skyresiduals-focalplane.png ├── tile-qa-7951-thru20211014-PR1452.png ├── tile-qa-7951-thru20211014.png └── tile7951-night20211014-fiber3000.png ├── LICENSE.rst ├── README.md ├── archive ├── GFA_targets.ipynb ├── Intro_to_DESI_spectra.ipynb ├── README.rst ├── dc17a-truth.ipynb ├── input.yaml ├── quickgen.ipynb ├── simulating-desi-spectra.ipynb └── survey-simulations.ipynb ├── database ├── How_to_use_SPARCL_at_NERSC.ipynb ├── README.md └── spectroscopic-production-database.ipynb ├── fiberassign ├── FiberAssign.ipynb ├── FiberAssignAlgorithms_Part1.ipynb ├── FiberAssignAlgorithms_Part2.ipynb ├── FiberAssignDECaLS.ipynb ├── FiberAssignMocks.ipynb └── restrict_reach.ipynb ├── meetings ├── Dec2020 │ ├── 3pcf │ │ ├── 3pt_ckd_hiell_cleaner_for_DESI_tutorial.py │ │ ├── DESI_3PCF_Tutorial_Slides.pdf │ │ └── sample_feb27_unity_weights_rescale400_first500.dat │ ├── LSS.ipynb │ ├── ML_TensorFlow │ │ ├── Images │ │ │ ├── SDSS_QSO_spec.png │ │ │ ├── ann.png │ │ │ ├── ann_layers.jpg │ │ │ ├── architecture.png │ │ │ ├── boxes.png │ │ │ ├── computational_graph_hpc.png │ │ │ ├── computational_graph_simple.png │ │ │ ├── general_logistic_unit.png │ │ │ ├── optimization_gd.png │ │ │ └── target.png │ │ ├── ML_TensorFlow.ipynb │ │ └── setup.txt │ ├── README.md │ ├── galaxies │ │ ├── coadd-66003-20200315-1-00055654.few.fits │ │ ├── csp.png │ │ ├── csp_simple.png │ │ ├── fitting_redshifts.ipynb │ │ ├── fitting_sps.ipynb │ │ ├── sfh.png │ │ ├── ssp.png │ │ └── zh.png │ └── quickquasars │ │ ├── README.md │ │ ├── quickquasars.ipynb │ │ └── scripts │ │ └── quickquasars_script.sh ├── Dec2021 │ ├── Everest_Tutorial_Dec2021.ipynb │ ├── README.md │ ├── SpectroperfectionismExample.ipynb │ └── mpi4py │ │ ├── .gitignore │ │ ├── README.md │ │ ├── Screen Shot 2016-04-09 at 6.54.01 PM.png │ │ ├── bcast.py │ │ ├── ex1_helloworld.py │ │ ├── ex2_p2p.py │ │ ├── ex3_bcast.py │ │ ├── ex4_scatter.py │ │ ├── ex5_gather.py │ │ ├── ex6_simple_pi.py │ │ ├── ex7_mpi_pi.py │ │ ├── gather.py │ │ ├── helloworld.c │ │ ├── helloworld.py │ │ ├── p2p.py │ │ ├── pi_mpi_gather.py │ │ ├── pi_mpi_p2p.py │ │ ├── pi_simple.py │ │ ├── scatter.py │ │ └── scatter_gather.py ├── Dec2022 │ ├── README.md │ └── using-perlmutter │ │ ├── README.md │ │ ├── compute-pi-examples │ │ ├── library.py │ │ ├── pi-mpi.py │ │ ├── pi-multiprocessing.py │ │ ├── pi-numba.py │ │ ├── pi-numpy.py │ │ ├── pi-serial.c │ │ ├── pi-serial.py │ │ ├── pi-threads.py │ │ ├── sleep-serial.py │ │ └── sleep-threads.py │ │ └── data-task-examples │ │ ├── data-cupy-mpi4py.py │ │ ├── data-cupy.py │ │ ├── data-mpi4py.py │ │ ├── data-multiprocessing.py │ │ ├── data-serial.py │ │ └── scan-num-threads.sh ├── Dec2023 │ └── NERSC │ │ ├── README.md │ │ ├── coaddfiles.txt │ │ ├── exercise1.py │ │ ├── exercise2.py │ │ └── solutions │ │ ├── exercise1_gnuparallel.txt │ │ ├── exercise1_mpi.py │ │ ├── exercise1_multiproc_files.py │ │ └── exercise1_multiproc_spectra.py ├── Dec2024 │ └── DataIntro │ │ ├── 00_GettingStarted.md │ │ ├── 01_CoreConcepts.ipynb │ │ ├── 02_DataOrganization.ipynb │ │ ├── 03_RedshiftCatalogs.ipynb │ │ ├── 04_Spectra.ipynb │ │ └── 05_NERSC.md ├── June2021 │ ├── Intro_to_a_DESI_Release.ipynb │ ├── README.md │ ├── git-intro.ipynb │ ├── images │ │ └── threecommits.png │ └── nersc_computing │ │ ├── computing_at_nersc.ipynb │ │ ├── example_batch.batch │ │ ├── kernel.json │ │ ├── load_myenv.sh │ │ ├── myenv.sh │ │ └── test_script.py └── June2022 │ └── README.md ├── mpl └── desi.mplstyle ├── onskydata ├── Fujilupe_AnalyzeZcat.ipynb ├── Intro_to_DESI_SV_spectra.ipynb ├── spectro_completeness.ipynb └── spectro_nz.ipynb ├── py └── desitutorials.py ├── redrock ├── RedrockBOSSDemo.ipynb ├── RedrockNthBestFit.ipynb ├── RedrockOutputs.ipynb ├── RedrockPlotSpec.md └── rrplotspec.png ├── simulations └── README.md └── testing ├── auto_test.html ├── readme.txt └── test_papermill.sh /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | env/ 12 | build/ 13 | develop-eggs/ 14 | dist/ 15 | downloads/ 16 | eggs/ 17 | .eggs/ 18 | lib/ 19 | lib64/ 20 | parts/ 21 | sdist/ 22 | var/ 23 | wheels/ 24 | *.egg-info/ 25 | .installed.cfg 26 | *.egg 27 | 28 | # PyInstaller 29 | # Usually these files are written by a python script from a template 30 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 31 | *.manifest 32 | *.spec 33 | 34 | # Installer logs 35 | pip-log.txt 36 | pip-delete-this-directory.txt 37 | 38 | # Unit test / coverage reports 39 | htmlcov/ 40 | .tox/ 41 | .coverage 42 | .coverage.* 43 | .cache 44 | nosetests.xml 45 | coverage.xml 46 | *.cover 47 | .hypothesis/ 48 | 49 | # Translations 50 | *.mo 51 | *.pot 52 | 53 | # Django stuff: 54 | *.log 55 | local_settings.py 56 | 57 | # Flask stuff: 58 | instance/ 59 | .webassets-cache 60 | 61 | # Scrapy stuff: 62 | .scrapy 63 | 64 | # Sphinx documentation 65 | docs/_build/ 66 | 67 | # PyBuilder 68 | target/ 69 | 70 | # Jupyter Notebook 71 | .ipynb_checkpoints 72 | 73 | # pyenv 74 | .python-version 75 | 76 | # celery beat schedule file 77 | celerybeat-schedule 78 | 79 | # SageMath parsed files 80 | *.sage.py 81 | 82 | # dotenv 83 | .env 84 | 85 | # virtualenv 86 | .venv 87 | venv/ 88 | ENV/ 89 | 90 | # Spyder project settings 91 | .spyderproject 92 | .spyproject 93 | 94 | # Rope project settings 95 | .ropeproject 96 | 97 | # mkdocs documentation 98 | /site 99 | 100 | # mypy 101 | .mypy_cache/ 102 | 103 | # SQLite database produced by redshift-database tutorial. 104 | minitest-*.db 105 | 106 | # Generated scripts created by notebooks 107 | select_targets.py 108 | -------------------------------------------------------------------------------- /01_getting_started/00_Setup.md: -------------------------------------------------------------------------------- 1 | # Setup to run these tutorials 2 | 3 | ## Running the tutorials on a local machine 4 | 5 | To run these tutorials on a local machine, you will need a Python environment 6 | with a set of 3rd party libraries, plus some DESI code and example data. 7 | 8 | 1. Install [miniforge](https://conda-forge.org/download/) or 9 | [anaconda](https://www.anaconda.com/download) 10 | 11 | 2. Create a DESI environment with the required packages 12 | 13 | ``` 14 | conda create --name desi 'numpy<2' scipy 'astropy<7' ipython jupyter \ 15 | matplotlib numba pytest fitsio h5py healpy requests 16 | conda activate desi 17 | ``` 18 | 19 | 3. Install the DESI-specific packages 20 | 21 | ``` 22 | pip install git+https://github.com/desihub/desiutil@3.5.0 23 | pip install git+https://github.com/desihub/desitarget@2.9.0 24 | pip install git+https://github.com/desihub/desispec@0.69.0 25 | pip install git+https://github.com/desihub/desimodel@0.19.3 26 | install_desimodel_data 27 | pip install git+https://github.com/desihub/redrock@0.20.4 28 | install_redrock_templates 29 | ``` 30 | 31 | **NOTE**: To get the absolute latest versions, you can replace those tag X.Y.Z numbers 32 | with "main". If you previously installed main and want to update it again, 33 | use `pip install --force-reinstall ...`. 34 | 35 | 4. Download example DESI data 36 | 37 | ``` 38 | curl https://raw.githubusercontent.com/desihub/desida/refs/tags/1.0.0/bin/desi_get_dr_subset > desi_get_dr_subset 39 | python desi_get_dr_subset 40 | ``` 41 | 42 | **NOTE**: if you don't have curl installed, you can also use wget or download `desi_get_dr_subset` using a web browser. 43 | 44 | **NOTE**: even a "tiny" subset of DR1 is ~40 GB and can take an hour or more to download 45 | even with a fast internet connection. Once you have gotten past the 46 | "Downloading files for healpix 23040..." step you can proceed with the tutorials 47 | while the remaining tile-based data finish downloading 48 | (needed for [03_DataOrganization.ipynb](03_DataOrganization.ipynb) and [05_Spectra.ipynb](05_Spectra.ipynb) 49 | but not the others) 50 | 51 | **NOTE**: Most of these tutorials require downloading example files. 52 | The [06_PublicDatabaseAccess.ipynb](06_PublicDatabaseAccess.ipynb) tutorial demonstrates 53 | an alternative method of accessing the data through public database queries and 54 | downloading spectra on-the-fly using 55 | [SPARCL](https://astrosparcl.datalab.noirlab.edu), without needing any local data. 56 | 57 | 5. Set `$DESI_ROOT` to the path where you just downloaded the data 58 | 59 | * bash: `export DESI_ROOT=$PWD/tiny_dr1` 60 | * tcsh: `setenv DESI_ROOT $PWD/tiny_dr1` 61 | 62 | 6. Get the tutorials and start Jupyter 63 | 64 | ``` 65 | git clone https://github.com/desihub/tutorials 66 | cd tutorials 67 | jupyter lab . 68 | ``` 69 | 70 | **Safari Users**: JupyterLab 4.3.5 and Safari 16.4 have a bug rendering newly added cells. 71 | This can be fixed under Settings -> Settings Editor -> Notebook -> Windowing mode 72 | (near bottom) -> set to "defer" 73 | 74 | 7. Proceed with the [01_getting_started/01_QuickStart.ipynb](01_QuickStart.ipynb) tutorial in the left panel. 75 | 76 | 77 | ## Running the tutorials at NERSC 78 | 79 | If you have access to [NERSC](https://nersc.gov), e.g. through DESI, DES, LSST-DESC, CMB-S4, etc., 80 | you can run the tutorials at https://juypter.nersc.gov and have access to the full public DESI data 81 | without having to copy or download anything. 82 | To do this, you will need to install a "Jupyter kernel" that tells it where to find the pre-installed 83 | DESI code and data. 84 | 85 | 1. Log into perlmutter.nersc.gov and install Jupyter kernel with DESI packages configured for DR1: 86 | 87 | ``` 88 | ssh perlmutter.nersc.gov 89 | /global/common/software/desi/install_jupyter_kernel --software-version 25.3 --data-release dr1 90 | ``` 91 | 92 | 2. Get a copy of these tutorials 93 | ``` 94 | git clone https://github.com/desihub/tutorials 95 | ``` 96 | 97 | 3. Login to https://jupyter.nersc.gov, start a "Perlmutter Login Node" server, 98 | and navigate to the location where you cloned the tutorials. 99 | 100 | 4. Start with [01_getting_started/01_QuickStart.ipynb](01_QuickStart.ipynb). 101 | For each tutorial, Kernel -> Change Kernel... (or click on the kernel name 102 | in the upper right) and change the kernel to the "DESI 25.3 DR1" kernel 103 | that you installed in step 1. 104 | 105 | **NOTE**: If you get an `ModuleNotFoundError` or `ImportError` when running these 106 | tutorials, you probably forgot to switch to a DESI kernel first. 107 | In that case, switch kernels and restart the notebook from the beginning. 108 | 109 | **NOTE**: at NERSC you do *not* need to get a copy of the data since that is already there. 110 | The Jupyter kernel will set `$DESI_ROOT` to the DR1 location, and the tutorials will find 111 | it from there. 112 | 113 | 114 | -------------------------------------------------------------------------------- /01_getting_started/02_CoreConcepts.md: -------------------------------------------------------------------------------- 1 | # Core Concepts 2 | 3 | This tutorial briefly introduces a few key concepts in DESI. 4 | Understanding these will help you understand the other tutorials more easily. 5 | 6 | If you are viewing this file from within a Jupyter server, 7 | right click and select "Show Markdown Preview" to see a formatted version. 8 | 9 | ## SURVEY and PROGRAM 10 | 11 | DESI observations are organized by 12 | * **SURVEY** = Phases of DESI observations with a common goal, e.g. "main" or "sv1" 13 | * **PROGRAM** = Subsets of SURVEYs split by observing conditions, e.g. "dark" or "bright" 14 | 15 | **Why this matters**: Data processing groups data by SURVEY and PROGRAM on disk, 16 | and does not combine data across SURVEY and PROGRAM even if it is the same 17 | object on the sky. This keeps the different scientific goals of DESI 18 | independent of each other. When analyzing DESI data, you need to know what 19 | SURVEY and PROGRAM you care about. 20 | 21 | Primary **SURVEY**s and **PROGRAMS** in DESI are 22 | 23 | | SURVEY | Purpose | 24 | | :---- | :--- | 25 | | Survey Validation 1 (sv1) | Tune cuts for target selection; extra high S/N data | 26 | | Survey Validation 3 (sv3) | Many overlapping observations to get all targets on a given patch of sky ("highly complete") | 27 | | Main (main) | The core cosmology program of DESI | 28 | 29 | | PROGRAM | Purpose | 30 | | :---- | :--- | 31 | | dark | Best observing conditions for faintest targets: ELG, LRG, QSO | 32 | | bright | Moon up / poor seeing / poor transparency: BGS, MWS | 33 | | backup | Very bad conditions: bright stars | 34 | 35 | ## Tiles and Healpix 36 | 37 | A DESI "tile" is a specific pointing of the telescope and fiber positioners to 38 | observe a specific set of objects on the sky. Tiles are associated with a 39 | SURVEY and PROGRAM. Tiles are observed with one more more exposures on 40 | one more more nights until they have achieved a consistent signal-to-noise (S/N) goal 41 | set by the SURVEY+PROGRAM. Since a single tile cannot observe all the targets on 42 | a given patch of sky, the DESI tiles overlap so that if a given target is not observed 43 | on one tile, it gets another chance on a future overlapping tile. 44 | 45 | Some targets are observed on multiple tiles to get more S/N than they would get on 46 | a single tile, e.g. Lyman-alpha QSOs at z>2. In this case we want to coadd data 47 | across tiles. Some science studies also want all spectra in a single patch of sky 48 | and it would be a pain to have to look up and read N>>1 seperate tile files just to 49 | get those spectra. For these reasons, data processing also groups spectra by 50 | "healpix", which is a diamond-shaped tesselation of the sky. All spectra in a given (healpix, survey, program) are grouped together into files and coadded. 51 | 52 | **Why this matters**: If you want the highest S/N data combined across all observations, you want to use the healpix data. If you need to track performance vs. time or are working with 53 | custom observations on a special tile, you want the tile data. 54 | 55 | **Digging Deeper**: The [DESI_petal_healpix_rosette](EDR/DESI_petal_healpix_rosette.ipynb) tutorial explores these ideas in more detail including reading and plotting targets grouped by tiles vs. healpix. 56 | 57 | 58 | ## Petals, Spectrographs, Cameras, and Fibers 59 | 60 | The DESI focal plane is divided into 10 separate "petals". 61 | Each petal has 500 fibers which map to a single "spectrograph". 62 | The petal number [0-9] is the same as the spectrograph number [0-9] 63 | and in practice these are used interchangeably. 64 | The `10*500=5000` DESI fibers are mapped to the spectrographs such that 65 | ``` 66 | PETAL = SPECTROGRAPH = FIBER//500 67 | ``` 68 | 69 | Each spectrograph has 3 "cameras" which split the light by blue (b), red (r), and near-infrared (z) bands. These cameras are named by the band+spectrograph, e.g. "b0", "r1", "z9". 70 | 71 | 72 | **Caveat**: if you get involved in hardware operations, there is a different numbering 73 | scheme for the hardware spectrographs `smN` developed while they were being manufactured, 74 | before they were plugged in to the petals. Most people do not need to know about this 75 | distinction. 76 | 77 | ## Spectra and Catalogs 78 | 79 | The core DESI data are *spectra*, i.e. flux vs. wavelength. 80 | When we measure quantities from spectra like the redshift or the flux in emission lines, 81 | these measurements can be grouped in to tables in *catalogs*. Many analyses can be 82 | performed on catalogs generated by others without ever needing to read the much larger spectra files. 83 | 84 | ## Data Releases and Mountain-themed Spectroscopic Productions 85 | 86 | DESI data processing runs are named after mountains, alphabetically increasing with time. 87 | A given mountain or "spectroscopic production run" (specprod) represents a self-consistent 88 | processing of the data with a set of code tags. 89 | 90 | Public data releases, such as DR1, contain one or more spectroscopic productions, Valued Added Catalogs contributed 91 | by the DESI science collaboration using those productions, and supporting material such as the raw data. 92 | 93 | Typically a data release will have one primary production. In the case of DR1, that is "iron", located at NERSC in 94 | 95 | ``` 96 | /global/cfs/cdirs/desi/public/dr1/spectro/redux/iron 97 | ``` 98 | 99 | It is good practice for all of your scripts and notebooks to set the production directory once at the very top instead of hardcoding e.g. "dr1" and "iron" many places. This makes it easier to switch from one release/production to a newer one in the future. It's even better to reference this to an environment variable `$DESI_ROOT` (=`/global/cfs/cdirs/desi/public/dr1` at NERSC) so that you can copy a subset of the data to your laptop or home institution and still have the same scripts work without a bunch of NERSC-specific hardcoded paths. 100 | 101 | For example: 102 | 103 | ```python 104 | import os 105 | specprod = 'iron' 106 | desi_root = os.environ['DESI_ROOT'] 107 | datadir = f'{desi_root}/spectro/redux/{specprod}' 108 | print(f'Using data in {datadir}') 109 | ``` 110 | 111 | If you follow that pattern in all of your own analysis notebooks and scripts, it will be much easier to re-run your 112 | work on future productions. 113 | -------------------------------------------------------------------------------- /01_getting_started/EDR/README.md: -------------------------------------------------------------------------------- 1 | # Getting Started 2 | 3 | Welcome to the DESI EDR Getting Started tutorials! 4 | 5 | **NOTE**: These tutorials are provided as a reference for what was released with the DESI Early Data Release, 6 | but we recommend that any new users start with [Data Release 1](../README.md) which has both better 7 | data and updated tutorials. 8 | 9 | In addition to tutorial Jupyter Notebooks, this folder includes the following: 10 | - [Instructions for installing DESI Jupyter kernels at NERSC](installing_EDR_kernels_at_NERSC.md) 11 | - [Links to reproduce figures from the EDR Overview paper](figures_from_EDR_paper.md) 12 | 13 | ## Other useful references 14 | 15 | ### DESI Data Documentation Website 16 | 17 | The [DESI Data Documentation](https://data.desi.lbl.gov/doc/) website describes data access, data format, data releases, and includes links to technical papers as well as information on data license and acknowledgments. 18 | 19 | ### DESI Helpdesk 20 | 21 | If you have a question, please visit the [DESI User Forum](https://help.desi.lbl.gov). 22 | 23 | ## Tutorials and data at NOIRLab 24 | 25 | Outside NERSC, [NOIRLab provides access and tutorials to some DESI data](https://datalab.noirlab.edu/desi/index.php). 26 | Although only the HEALPixel coadds from the [Early Data Release](https://data.desi.lbl.gov/doc/releases/) are served at present, this provides a convenient way to retrieve spectra that could otherwise be spread among many separate files. 27 | -------------------------------------------------------------------------------- /01_getting_started/EDR/figures_from_EDR_paper.md: -------------------------------------------------------------------------------- 1 | For the purpose of replicating the primary figures featured in the Early Data Release paper, please refer to the [EDR paper repository](https://github.com/desihub/edrpaper). 2 | 3 | Within this repository, you will discover a variety of Python scripts housed under bin/, as well as Jupyter notebooks stored under nb/. These resources are at your disposal to independently generate the plots. 4 | -------------------------------------------------------------------------------- /01_getting_started/EDR/installing_EDR_kernels_at_NERSC.md: -------------------------------------------------------------------------------- 1 | # Installing DESI jupyter kernels at NERSC 2 | 3 | If you're a member of the DESI collaboration, please refer to the [collaboration wiki for instructions](https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC). 4 | 5 | For NERSC users who are not part of the DESI collaboration, please follow the steps outlined below: 6 | 7 | 1. Log into perlmutter.nersc.gov and set up the DESI code environment. Switch the environment variables to the publicly accessible EDR location using the following commands: 8 | 9 | ``` 10 | source /global/common/software/desi/desi_environment.sh 23.1 11 | module switch desitree/edr 12 | ``` 13 | 14 | 2. Determine your shell by using the command: echo ${SHELL}. 15 | 16 | 3. Depending on your shell, select the appropriate kernel version: 17 | * For /bin/bash, /bin/sh, or /bin/zsh, use the "bash" kernels. For example, `desi-edr-23.1-bash`. 18 | * For /bin/tcsh or /bin/csh, use the "tcsh" kernels. For example, `desi-edr-23.1-tcsh`. 19 | 20 | 4. Copy the chosen kernel to your home directory using the following command: 21 | ``` 22 | mkdir -p ~/.local/share/jupyter/kernels 23 | cp -R /global/common/software/desi/kernels/desi-edr-23.1-bash ~/.local/share/jupyter/kernels 24 | ``` 25 | This step will install the DESI EDR 23.1 kernel. Other available kernels are `desi-22.5`, `desi-23.1` and `desi-main`, 26 | 27 | 5. Access https://jupyter.nersc.gov/hub/login and initiate a "Perlmutter" server (Note: Do not start a "Spin" server). 28 | 29 | 6. Once the server loads your home directory and kernel definitions, navigate to a new directory where you'll save your notebooks. 30 | 31 | 7. To create a new notebook, navigate through File -> New Launcher and then click on the kernel version, such as DESI EDR 23.1. 32 | 33 | 34 | -------------------------------------------------------------------------------- /01_getting_started/README.md: -------------------------------------------------------------------------------- 1 | # Getting Started 2 | 3 | Welcome to the **DESI Getting Started Tutorials**! 4 | 5 | This resource is designed to guide you through the fundamentals of working with DESI data, providing references and tutorials to help you navigate and make the most of the DESI ecosystem. 6 | 7 | The current set of tutorials is based upon DESI Data Release 1 (DR1) data. 8 | Previous tutorials for the Early Data Release (EDR) are archived under the 9 | [EDR/](./EDR) directory. 10 | 11 | Start with [00_Setup.md](./00_Setup.md) for initial software setup and how to access example data, then proceed with the other tutorials. 12 | 13 | ## Tutorial Overview 14 | 15 | - [00_Setup.md](./00_Setup.md): Set-up instructions to run the file-based tutorials either locally or at NERSC. 16 | - [01_QuickStart.ipynb](./01_QuickStart.ipynb): Rapid overview of DESI DR1 file access from redshift catalogs through spectra. 17 | - [02_CoreConcepts.md](./02_CoreConcepts.md): Explanation of key concepts to better understand DESI data and tutorials (e.g., survey, program, healpix, tile, specprod). 18 | - [03_DataOrganization.ipynb](./03_DataOrganization.ipynb): Navigation of the DESI data file structure under a spectroscopic production run ("specprod") 19 | - [04_RedshiftCatalogs.ipynb](./04_RedshiftCatalogs.ipynb): In-depth usage of the DESI redshift catalog files such as object/target selection, data quality cuts, choosing the best of multiple spectra, and spectral classification 20 | - [05_Spectra.ipynb](./05_Spectra.ipynb): Broad usage of DESI spectra files including reading, slicing, combining, and writing spectra data 21 | - [06_PublicDatabaseAccess.ipynb](./06_PublicDatabaseAccess.ipynb): Usage of searchable databases of DESI catalogs and spectra accessible publicly covering sample selection, classification per target class or spectral type, retrieval and visualization of spectra with SPARCL. 22 | 23 | ## Additional Resources 24 | 25 | ### 1. DESI Data Documentation 26 | 27 | The [DESI Data Documentation website](https://data.desi.lbl.gov/doc/) offers detailed information about: 28 | - [Data access and formats](https://data.desi.lbl.gov/doc/access/) 29 | - [Data releases](https://data.desi.lbl.gov/doc/releases/) 30 | - [Technical papers and key cosmology results](https://data.desi.lbl.gov/doc/papers/) 31 | - [Data license and acknowledgments](https://data.desi.lbl.gov/doc/acknowledgments/) 32 | 33 | ### 2. DESI Helpdesk 34 | 35 | Have questions or need assistance? 36 | Visit the [DESI User Forum](https://help.desi.lbl.gov) to ask questions and find solutions provided by the DESI community and support team. 37 | 38 | ### 3. Tutorials and Data at NOIRLab 39 | 40 | Looking for additional tutorials and data outside NERSC? 41 | [NOIRLab's Astro Data Lab](https://datalab.noirlab.edu/desi/index.php) provides both public (anonymous) and authenticated services to access DESI data including a database containing the main pipeline catalogs (e.g., redshift, target, photometry catalogs). The full-depth ("healpix coadds") spectra are available via [SPectra Analysis and Retrievable Catalog Lab (SPARCL)](https://astrosparcl.datalab.noirlab.edu/) for fast search and retrieval. Other types of spectra (per exposure, per night, per tile) are only available at NERSC using files ([05_Spectra.ipynb](./05_Spectra.ipynb)). 42 | 43 | In addition to the [06_PublicDatabaseAccess.ipynb](./06_PublicDatabaseAccess.ipynb) tutorial provided here, additional [tutorials](https://github.com/astro-datalab/notebooks-latest/tree/master/03_ScienceExamples/DESI) are available to help you explore and analyze DESI data effectively. 44 | 45 | ### 4. Public TAP handle for, e.g., TOPCAT or Astroquery 46 | 47 | The Astro Data Lab created a Table Access Protocol (TAP) handle that provides a convenient access layer for the 48 | DESI catalog database tables. TAP-aware clients (such as TOPCAT) can point to `https://datalab.noirlab.edu/tap`, 49 | select the `desi_dr1` database, and see the database tables and descriptions. 50 | 51 | Descriptions of the associated tables can also be found in the Astro Data Lab [table browser](https://datalab.noirlab.edu/query.php?name=desi_dr1.zpix) 52 | and on the DESI Data Documentation [Database page](https://data.desi.lbl.gov/doc/access/database/). 53 | 54 | Below is an example code snippet featuring [Astroquery](https://astroquery.readthedocs.io/en/latest/), a coordinated package of astropy: 55 | ``` 56 | from astroquery.utils.tap.core import Tap 57 | 58 | ## Point TAP service to Astro Data Lab URL 59 | datalab_tap_url = "https://datalab.noirlab.edu/tap" 60 | 61 | ## Create TAP service object 62 | tap_service = Tap(url=datalab_tap_url) 63 | 64 | ## Example query in ADQL (similar to SQL but Astronomy tailored) 65 | query = """ 66 | SELECT TOP 10 targetid, survey, program, z, zwarn, spectype 67 | FROM desi_dr1.zpix 68 | WHERE mean_fiber_ra BETWEEN 150 AND 151 69 | AND mean_fiber_dec BETWEEN 2 AND 3 70 | """ 71 | 72 | ## Run query 73 | job = tap_service.launch_job(query) 74 | 75 | ## Save output into result 76 | result = job.get_results() 77 | ``` 78 | 79 | For additional tips and example usage, consult: https://astroquery.readthedocs.io/en/latest/utils/tap.html -------------------------------------------------------------------------------- /02_digging_deeper/README.md: -------------------------------------------------------------------------------- 1 | # Digging Deeper with DESI Data 2 | 3 | This directory contains tutorials on detailed topics as a followup to the 4 | the [01_getting_started](../01_getting_started) tutorials. These currently include 5 | 6 | * [RedrockOutputs.ipynb](RedrockOutputs.ipynb) - Working with the spectral classifications and redshift 7 | catalogs output by the Redrock algorithm. This includes how to generate the best fit model for each 8 | spectrum, and how to find the details such as the Nth best fit. 9 | * [ResolutionMatrix.ipynb](ResolutionMatrix.ipynb) - How to work with the "Resolution Matrix" encoding 10 | the per-spectrum per-wavelength effective instrument resolution. 11 | 12 | Also see 13 | 14 | * The [database](../database/README.md) directory for tutorials specifically about working with the 15 | DESI database(s) instead of files. 16 | * [Targeting bits tutorial](https://github.com/desihub/desitarget/blob/main/doc/nb/target-selection-bits-and-bitmasks.ipynb) 17 | in the [desitarget](https://github.com/desihub/tutorials) repository 18 | * NOIRLab [Astro Data Lab](https://github.com/astro-datalab/notebooks-latest/tree/master/03_ScienceExamples/DESI/) example notebooks. 19 | * Scripts and notebooks for generating the plots for data releases are available in the GitHub 20 | [edrpaper](https://github.com/desihub/edrpaper) and [dr1paper](https://github.com/desihub/dr1paper) repositories. 21 | 22 | 23 | Future tutorials might include 24 | 25 | * Downloading data locally 26 | * [FastSpecFit](https://github.com/desihub/fastspecfit) output 27 | * More plotting options 28 | * etc. 29 | 30 | -------------------------------------------------------------------------------- /DataQA/sframe-r6-00104282.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/sframe-r6-00104282.png -------------------------------------------------------------------------------- /DataQA/sframe-r6-00104284.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/sframe-r6-00104284.png -------------------------------------------------------------------------------- /DataQA/sframe-r6-00104285.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/sframe-r6-00104285.png -------------------------------------------------------------------------------- /DataQA/sframe-r6-00104286.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/sframe-r6-00104286.png -------------------------------------------------------------------------------- /DataQA/sframe-r6-beforeafter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/sframe-r6-beforeafter.png -------------------------------------------------------------------------------- /DataQA/sframe-r7-00104284.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/sframe-r7-00104284.png -------------------------------------------------------------------------------- /DataQA/sframe-r8-00104284.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/sframe-r8-00104284.png -------------------------------------------------------------------------------- /DataQA/sframe-sky-r6-00104284.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/sframe-sky-r6-00104284.png -------------------------------------------------------------------------------- /DataQA/skyline-6563-xy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/skyline-6563-xy.png -------------------------------------------------------------------------------- /DataQA/skymodel-r6-00104284.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/skymodel-r6-00104284.png -------------------------------------------------------------------------------- /DataQA/skymodel-r6-beforeafter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/skymodel-r6-beforeafter.png -------------------------------------------------------------------------------- /DataQA/skyresiduals-focalplane.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/skyresiduals-focalplane.png -------------------------------------------------------------------------------- /DataQA/tile-qa-7951-thru20211014-PR1452.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/tile-qa-7951-thru20211014-PR1452.png -------------------------------------------------------------------------------- /DataQA/tile-qa-7951-thru20211014.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/tile-qa-7951-thru20211014.png -------------------------------------------------------------------------------- /DataQA/tile7951-night20211014-fiber3000.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/DataQA/tile7951-night20211014-fiber3000.png -------------------------------------------------------------------------------- /LICENSE.rst: -------------------------------------------------------------------------------- 1 | Copyright (c) 2017, DESI Collaboration 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, 5 | are permitted provided that the following conditions are met: 6 | 7 | * Redistributions of source code must retain the above copyright notice, this 8 | list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above copyright notice, this 10 | list of conditions and the following disclaimer in the documentation and/or 11 | other materials provided with the distribution. 12 | * Neither the name of the DESI Collaboration nor the names of its contributors 13 | may be used to endorse or promote products derived from this software without 14 | specific prior written permission. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 17 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 18 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR 20 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 21 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 22 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON 23 | ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 25 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # DESI tutorials 2 | 3 | Welcome to the repository of tutorials to guide you through the process of analyzing DESI data. 4 | We recommend starting with the tutorials under [01_getting_started/](01_getting_started/) 5 | and then explore additional topics under [02_digging_deeper/](02_digging_deeper/). 6 | 7 | Other directories in this repository cover detailed topics and tutorials presented at DESI collaboration meetings; these are not actively maintained and may not work with the latest data and software releases. Some tutorials require substantial amounts of data or are specific to the 8 | [NERSC](https://nersc.gov) computing center used by the DESI collaboration, but may still be useful to read even without NERSC access. 9 | 10 | See [01_getting_started/00_Setup.md](01_getting_started/00_Setup.md) for instructions on installing the necessary Python libraries and downloading example data. 11 | 12 | # Other tutorials and resources 13 | 14 | ## Tutorials in other DESI code repositories 15 | 16 | Tutorials hosted in other packages include (but haven't been recently vetted): 17 | * [How to run survey simulations](https://github.com/desihub/surveysim/blob/master/doc/tutorial.rst). 18 | * [How to convert an SED into a simulated DESI spectrum](https://github.com/desihub/specsim/blob/master/docs/nb/SimulationExamples.ipynb) 19 | * [How to run fiber assignment](https://desi.lbl.gov/DocDB/cgi-bin/private/ShowDocument?docid=2742) 20 | * [How to run quicksurvey catalog-level simulations](https://github.com/desihub/quicksurvey_example) 21 | * [How to make all-sky plots](https://github.com/desihub/desiutil/blob/master/doc/nb/SkyMapExamples.ipynb) 22 | * [Working with DESI target bits (Main, CMX, SV)](https://github.com/desihub/desitarget/blob/master/doc/nb/target-selection-bits-and-bitmasks.ipynb) 23 | 24 | ## Astro Data Lab 25 | 26 | The NOIRLab Astro Data Lab serves a copy of the DESI DR1 and EDR databases as `desi_dr1` and `desi_edr`. 27 | These are accessible to users without a NERSC account. 28 | Various modes of data access are described [here](https://datalab.noirlab.edu/desi/access.php). 29 | For public access, there is a Table Access Protocol (TAP) handle that provides a convenient access layer for the 30 | DESI catalog database tables. TAP-aware clients (such as TOPCAT) can point to `https://datalab.noirlab.edu/tap`, 31 | select the `desi_dr1` database, and see the database tables and descriptions. 32 | 33 | Descriptions of the associated tables can also be found in the Data Lab [table browser](https://datalab.noirlab.edu/query.php?name=desi_dr1.zpix) 34 | and on the DESI Data Documentation [Database page](https://data.desi.lbl.gov/doc/access/database/). 35 | 36 | ## SPARCL 37 | 38 | The [SPectra Analysis & Retrievable Catalog Lab (SPARCL)](https://astrosparcl.datalab.noirlab.edu) contains 39 | DESI DR1 and EDR spectra that were coadded per healpix and coadded across cameras. It is a searchable database that can be used via the Astro Data Lab 40 | JupyterLab Notebook. Alternatively, the Python client can be installed locally using: 41 | ``` 42 | pip install sparclclient 43 | ``` 44 | The client can be loaded within a Python session or program via: 45 | ``` 46 | >>> from sparcl.client import SparclClient 47 | >>> client = SparclClient() 48 | ``` 49 | 50 | There are instructions and useful examples in the [How-to-use-SPARCL tutorial notebook](https://github.com/astro-datalab/notebooks-latest/blob/master/04_HowTos/SPARCL/How_to_use_SPARCL.ipynb). 51 | 52 | # Acknowledgments 53 | 54 | Any use of DESI data whether via NERSC or external databases requires [DESI Data Acknowledgments](https://data.desi.lbl.gov/doc/acknowledgments/). 55 | 56 | # For authors of tutorials 57 | 58 | Include at the beginning of the tutorial what is needed as a prerequisite for running the tutorial, 59 | e.g. specific codes, environment variables, datasets. It's OK to link elsewhere for detailed 60 | instructions (e.g. for how to install DESI code in general). 61 | 62 | # Bug reporting 63 | 64 | If you find a bug in the tutorials, please file a ticket at https://github.com/desihub/tutorials or ask for help on the DESI user forum at https://help.desi.lbl.gov . 65 | -------------------------------------------------------------------------------- /archive/README.rst: -------------------------------------------------------------------------------- 1 | This directory contains archived tutorials that are no longer actively 2 | maintained. If someone would like to update and revive one of these 3 | tutorials, please do so! 4 | -------------------------------------------------------------------------------- /archive/dc17a-truth.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Connecting data challenge outputs to input truth #\n", 8 | "\n", 9 | "This tutorial documents how to connect reference run outputs back to the input truth. It uses the \"19.12\" reference run available at NERSC in\n", 10 | "```\n", 11 | "/global/cfs/cdirs/desi/datachallenge/reference_runs/19.12\n", 12 | "```\n", 13 | "See https://desi.lbl.gov/trac/wiki/Pipeline/GettingStarted/Laptop/JuneMeeting for\n", 14 | "instructions on installing the DESI code. The bottom of this tutorial lists the code versions used while testing this.\n", 15 | "\n", 16 | "Stephen Bailey
\n", 17 | "January 2020" 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "## Warning\n", 25 | "\n", 26 | "As of 2020-02-05, this notebook is known to be broken. See [#40](https://github.com/desihub/tutorials/issues/40) for details." 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "metadata": {}, 32 | "source": [ 33 | "## Basic setup ##\n", 34 | "\n", 35 | "The first portion of this tutorial doesn't require any DESI-specific code,\n", 36 | "so just load the minimal set of external dependencies needed here." 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": null, 42 | "metadata": {}, 43 | "outputs": [], 44 | "source": [ 45 | "import os\n", 46 | "import numpy as np\n", 47 | "import astropy.table\n", 48 | "import desitutorials\n", 49 | "reference_run = '19.12'\n", 50 | "%pylab inline" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "For convenience, move into the directory containing the reference run data. The following is where this is located at NERSC." 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": null, 63 | "metadata": {}, 64 | "outputs": [], 65 | "source": [ 66 | "os.chdir(os.path.join(os.environ['DESI_ROOT'], 'datachallenge', 'reference_runs', reference_run))" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "## Comparing the redshift catalog to the input truth ##\n", 74 | "\n", 75 | "Read in the redshift catalog, targets, and truth files." 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": null, 81 | "metadata": {}, 82 | "outputs": [], 83 | "source": [ 84 | "zcat = astropy.table.Table.read('spectro/redux/mini/zcatalog-mini.fits', hdu=1)\n", 85 | "targets_bright = astropy.table.Table.read('targets/targets-bright.fits', hdu=1)\n", 86 | "targets_dark = astropy.table.Table.read('targets/targets-dark.fits', hdu=1)\n", 87 | "truth_bright = astropy.table.Table.read('targets/truth-bright.fits', hdu=1)\n", 88 | "truth_dark = astropy.table.Table.read('targets/truth-dark.fits', hdu=1)" 89 | ] 90 | }, 91 | { 92 | "cell_type": "code", 93 | "execution_count": null, 94 | "metadata": {}, 95 | "outputs": [], 96 | "source": [ 97 | "#\n", 98 | "# Strip trailing spaces from TEMPLATETYPE column.\n", 99 | "#\n", 100 | "truth_bright['TEMPLATETYPE'][:] = np.char.strip(truth_bright['TEMPLATETYPE'][:])\n", 101 | "truth_dark['TEMPLATETYPE'][:] = np.char.strip(truth_dark['TEMPLATETYPE'][:])\n", 102 | "print('Updated set of template types:', set(truth_dark['TEMPLATETYPE']) | set(truth_bright['TEMPLATETYPE']))" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "Let's look at what columns each of these tables has, and then merge the `zcat` and `truth` tables together." 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "metadata": {}, 116 | "outputs": [], 117 | "source": [ 118 | "def print_colnames(table_name, tx):\n", 119 | " print('{}.colnames:'.format(table_name))\n", 120 | " for i in range(0, len(tx.colnames), 8):\n", 121 | " print(' '+' '.join(tx.colnames[i:i+8]))\n", 122 | "\n", 123 | "print_colnames('zcat', zcat)\n", 124 | "print_colnames('targets', targets_dark)\n", 125 | "print_colnames('truth', truth_dark)" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "ztruth = astropy.table.join(zcat, truth_dark, keys='TARGETID')\n", 135 | "print_colnames('ztruth', ztruth)" 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "metadata": {}, 141 | "source": [ 142 | "Make a histogram of $\\Delta v = c (z - z_{\\mathrm{true}}) / (1 + z_{\\mathrm{true}})$ for each object type:" 143 | ] 144 | }, 145 | { 146 | "cell_type": "code", 147 | "execution_count": null, 148 | "metadata": {}, 149 | "outputs": [], 150 | "source": [ 151 | "dv = 3e5 * (ztruth['Z'] - ztruth['TRUEZ']) / (1 + ztruth['TRUEZ'])\n", 152 | "figure(figsize=(9,6))\n", 153 | "for i, objtype in enumerate(set(ztruth['TEMPLATETYPE'])):\n", 154 | " subplot(2,3,1+i)\n", 155 | " ii = ztruth['TEMPLATETYPE'] == objtype\n", 156 | " hist(dv[ii], 50, (-100, 100))\n", 157 | " xlabel('{} dv [km/s]'.format(objtype))\n", 158 | "tight_layout()" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "## Comparing spectra to truth\n", 166 | "\n", 167 | "For this next section, we will use some `desispec` code so load that now and set\n", 168 | "some environment variables so that `desispec.io` can find the data files. This\n", 169 | "assumes that we are still in the reference run data directory." 170 | ] 171 | }, 172 | { 173 | "cell_type": "code", 174 | "execution_count": null, 175 | "metadata": {}, 176 | "outputs": [], 177 | "source": [ 178 | "import astropy.io.fits\n", 179 | "import desispec.io\n", 180 | "os.environ['DESI_SPECTRO_REDUX'] = os.path.join(os.getcwd(), 'spectro', 'redux')\n", 181 | "os.environ['SPECPROD'] = 'mini'" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "metadata": {}, 187 | "source": [ 188 | "Read some spectra and find out what bricks they are on" 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "metadata": {}, 195 | "outputs": [], 196 | "source": [ 197 | "specfile = desispec.io.findfile('spectra', groupname='5299', nside=64)\n", 198 | "spectra = desispec.io.read_spectra(specfile)" 199 | ] 200 | }, 201 | { 202 | "cell_type": "code", 203 | "execution_count": null, 204 | "metadata": {}, 205 | "outputs": [], 206 | "source": [ 207 | "itarget = 7\n", 208 | "plot(spectra.wave['b'], spectra.flux['b'][itarget], color='b')\n", 209 | "plot(spectra.wave['r'], spectra.flux['r'][itarget], color='r')\n", 210 | "plot(spectra.wave['z'], spectra.flux['z'][itarget], color='k')" 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": null, 216 | "metadata": {}, 217 | "outputs": [], 218 | "source": [ 219 | "spectra.fibermap['OBJTYPE', 'TARGETID', 'BRICKNAME', 'TARGET_RA', 'TARGET_DEC'][itarget]" 220 | ] 221 | }, 222 | { 223 | "cell_type": "markdown", 224 | "metadata": {}, 225 | "source": [ 226 | "What's the truth and redrock fit results for this target?" 227 | ] 228 | }, 229 | { 230 | "cell_type": "code", 231 | "execution_count": null, 232 | "metadata": {}, 233 | "outputs": [], 234 | "source": [ 235 | "ii = ztruth['TARGETID'] == spectra.fibermap['TARGETID'][itarget]\n", 236 | "ztruth['TEMPLATETYPE', 'TRUEZ', 'SPECTYPE', 'Z', 'ZERR', 'ZWARN'][ii]" 237 | ] 238 | }, 239 | { 240 | "cell_type": "markdown", 241 | "metadata": {}, 242 | "source": [ 243 | "The actual true input spectra were not included with dc17a, but they are available at\n", 244 | "NERSC in:\n", 245 | "```\n", 246 | "/project/projectdirs/desi/datachallenge/dc17a-twopct/targets/{group}/truth-{brickname}.fits\n", 247 | "```\n", 248 | "where `group = brickname[0:3]`, i.e. the first 3 characters of the brickname.\n", 249 | "\n", 250 | "**NOTE**: future data challenges will group the input truth files by healpix like the output spectra, but dc17a did not.\n", 251 | "\n", 252 | "The above target came from brickname 0084m085. If you are running locally (not at NERSC), go get that file now, e.g.\n", 253 | "```\n", 254 | "mkdir -p targets/008/\n", 255 | "scp edison.nersc.gov:/scratch2/scratchdirs/sjbailey/desi/dc17a/targets/008/truth-0084m085.fits targets/008/\n", 256 | "```\n", 257 | "Let's see what is in that file:" 258 | ] 259 | }, 260 | { 261 | "cell_type": "code", 262 | "execution_count": null, 263 | "metadata": {}, 264 | "outputs": [], 265 | "source": [ 266 | "datadir = '/global/cfs/cdirs/desi/datachallenge/dc17a-twopct'\n", 267 | "fx = astropy.io.fits.open(datadir+'/targets/008/truth-0084m085.fits')\n", 268 | "print(fx.info())\n", 269 | "truewave = fx['WAVE'].data\n", 270 | "trueflux = fx['FLUX'].data\n", 271 | "truemeta = fx['TRUTH'].data" 272 | ] 273 | }, 274 | { 275 | "cell_type": "markdown", 276 | "metadata": {}, 277 | "source": [ 278 | "It has the true wavelength, flux, and a TRUTH table which is just a subset\n", 279 | "of the rows in the the truth file that we previously read. Let's get the index for\n", 280 | "the spectrum that we plotted above" 281 | ] 282 | }, 283 | { 284 | "cell_type": "code", 285 | "execution_count": null, 286 | "metadata": {}, 287 | "outputs": [], 288 | "source": [ 289 | "itruth = np.where(truemeta['TARGETID'] == spectra.fibermap['TARGETID'][itarget])[0][0]" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "Does that look right?\n", 297 | "\n", 298 | "**NOTE**: the truth templates are stored in units of erg/(s cm2 Angstrom) while\n", 299 | "the output spectra are inunits of 1e-17 erg/(s cm2 Angstrom). We might standardize that\n", 300 | "in the future..." 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": null, 306 | "metadata": {}, 307 | "outputs": [], 308 | "source": [ 309 | "itarget = 7\n", 310 | "plot(spectra.wave['b'], spectra.flux['b'][itarget], color='b')\n", 311 | "plot(spectra.wave['r'], spectra.flux['r'][itarget], color='r')\n", 312 | "plot(spectra.wave['z'], spectra.flux['z'][itarget], color='k')\n", 313 | "plot(truewave, 1e17*trueflux[itruth], 'y-')\n", 314 | "ylim(-1, 3)" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "**BUG**: Note the unphysical jump in the template around 9200 Angstrom;\n", 322 | "That is the cause of many redshift failures, though redrock did get this one right. That step has been fixed in later software releases." 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "metadata": {}, 328 | "source": [ 329 | "## Further work\n", 330 | "This tutorial used `astropy.table.Table` to read the catalogs\n", 331 | "and `astropy.table.join` to merge them in a database-like manner.\n", 332 | "This is convenient, but won't scale to the full DESI dataset. Reading\n", 333 | "files with `fitsio` and matching via numpy logic will be more scalable,\n", 334 | "but is not yet documented here.\n", 335 | "\n", 336 | "**TODO**: add example reading with fitsio and matching tables with numpy.\n", 337 | "\n", 338 | "**TODO**: add a section on finding and querying the spectro catalog database." 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "metadata": {}, 344 | "source": [ 345 | "## Code versions used ##\n", 346 | "This is at the bottom of the tutorial so that python can introspect what was actually\n", 347 | "used instead of just what we thought was used" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "metadata": {}, 354 | "outputs": [], 355 | "source": [ 356 | "print('Code versions used for this tutorial:')\n", 357 | "desitutorials.print_code_versions()" 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": null, 363 | "metadata": {}, 364 | "outputs": [], 365 | "source": [] 366 | } 367 | ], 368 | "metadata": { 369 | "anaconda-cloud": {}, 370 | "kernelspec": { 371 | "display_name": "DESI 19.12", 372 | "language": "python", 373 | "name": "desi-19.12" 374 | }, 375 | "language_info": { 376 | "codemirror_mode": { 377 | "name": "ipython", 378 | "version": 3 379 | }, 380 | "file_extension": ".py", 381 | "mimetype": "text/x-python", 382 | "name": "python", 383 | "nbconvert_exporter": "python", 384 | "pygments_lexer": "ipython3", 385 | "version": "3.6.6" 386 | } 387 | }, 388 | "nbformat": 4, 389 | "nbformat_minor": 4 390 | } 391 | -------------------------------------------------------------------------------- /archive/input.yaml: -------------------------------------------------------------------------------- 1 | # mock target configuration file 2 | targets: 3 | ELG: { 4 | target_type: ELG, 5 | mockfile: '{DESI_ROOT}/mocks/DarkSky/v1.0.1/elg_0_inpt.fits', 6 | format: gaussianfield, 7 | density: 2400, 8 | } 9 | LRG: { 10 | target_type: LRG, 11 | mockfile: '{DESI_ROOT}/mocks/DarkSky/v1.0.1/lrg_0_inpt.fits', 12 | format: gaussianfield, 13 | density: 480, 14 | } 15 | QSO: { 16 | target_type: QSO, 17 | mockfile: '{DESI_ROOT}/mocks/DarkSky/v1.0.1/qso_0_inpt.fits', 18 | format: gaussianfield, 19 | zmax_qso: 1.8, 20 | use_simqso: True, 21 | density: 120, 22 | } 23 | LYA: { 24 | target_type: QSO, 25 | mockfile: '{DESI_ROOT}/mocks/lya_forest/develop/london/v4.0/master.fits', 26 | format: CoLoRe, 27 | nside_lya: 16, 28 | zmin_lya: 1.8, 29 | density: 50, 30 | use_simqso: True, 31 | balprob: 0.0, 32 | add_dla: True, 33 | } 34 | MWS_MAIN: { 35 | target_type: STAR, 36 | mockfile: '{DESI_ROOT}/mocks/mws/galaxia/alpha/v0.0.6/healpix', 37 | nside_galaxia: 8, 38 | format: galaxia, 39 | } 40 | WD: { 41 | target_type: WD, 42 | mockfile: '{DESI_ROOT}/mocks/mws/wd/v1.0.0/mock_wd.fits', 43 | format: mws_wd, 44 | } 45 | SKY: { 46 | target_type: SKY, 47 | mockfile: '{DESI_ROOT}/mocks/uniformsky/0.2/uniformsky-2048-0.2.fits', 48 | format: uniformsky, 49 | } 50 | -------------------------------------------------------------------------------- /database/README.md: -------------------------------------------------------------------------------- 1 | # README 2 | 3 | This database folder includes the following tutorial notebooks, which require NERSC access. 4 | 5 | For publicly accessible databases, please refer to the tutorial [01_getting_started/06_PublicDatabaseAccess.ipynb](../01_getting_started/06_PublicDatabaseAccess.ipynb). However, the functionality and examples contained in the notebooks listed below can still be informative and adapted to work with the public databases. 6 | 7 | ## How_to_use_SPARCL_at_NERSC.ipynb 8 | 9 | This notebook demonstrates how to use the spectral database SPARCL 10 | together with the DESI production database to find and retrieve DESI DR1 (Iron) spectra. 11 | It also shows how to convert the results to a list of Spectrum1D objects and to plot some 12 | example spectra. 13 | 14 | ## spectroscopic-production-database.ipynb 15 | 16 | This notebook demonstrates how to use the DESI database of tables including documentation of 17 | the database content and it showcases a broad range of example queries ranging from simple to 18 | more complex use cases. 19 | -------------------------------------------------------------------------------- /fiberassign/FiberAssignDECaLS.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# fiberassign on DECaLS (DR8)" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": null, 13 | "metadata": {}, 14 | "outputs": [], 15 | "source": [ 16 | "import os, sys, subprocess, fitsio\n", 17 | "from collections import Counter\n", 18 | "\n", 19 | "import numpy as np\n", 20 | "from astropy.table import Table\n", 21 | "from astropy.io import fits\n", 22 | "\n", 23 | "import desimodel.io\n", 24 | "import desimodel.focalplane\n", 25 | "import desimodel.footprint\n", 26 | "from desitarget.targetmask import desi_mask, obsconditions\n", 27 | "import matplotlib.pyplot as plt" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "## Learning goals\n", 35 | "\n", 36 | "In this notebook you will learn how to:\n", 37 | "\n", 38 | "* Use observational data from DR8 to prepare it for fiberassign.\n", 39 | "* Run fiberassign using the files generated above.\n", 40 | "* Explore the outputs of fiberassign.\n", 41 | "\n", 42 | "\n", 43 | "See https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC for instructions on configuring jupyter kernels with pre-installed DESI software at NERSC. This tutorial was last tested with the 22.2 kernel on April 29 2022.\n", 44 | "\n", 45 | "This notebook reuses code in the [main fiberassign notebook](https://github.com/desihub/tutorials/blob/master/FiberAssign.ipynb)." 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": {}, 52 | "outputs": [], 53 | "source": [ 54 | "# You can set this to a local path if you have the software installed and a copy of the data.\n", 55 | "# workdir = os.path.join(os.environ['HOME'], 'scratch', 'desi', 'tutorials', 'fiberassign_obs')\n", 56 | "workdir = os.path.join(os.environ['SCRATCH'], 'desi', 'test', 'fiberassign_obs')\n", 57 | "os.makedirs(workdir, exist_ok=True)\n", 58 | "os.chdir(workdir)\n", 59 | "outdir = os.path.join(workdir, 'output')\n", 60 | "os.makedirs(outdir, exist_ok=True)" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "Paths and filenames with the observational data" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "# Change this if you are running locally.\n", 77 | "# paths = {\n", 78 | "# \"targets\": \"/home/kisner/scratch/desi/tutorials\", \n", 79 | "# \"skies\": \"/home/kisner/scratch/desi/tutorials\", \n", 80 | "# \"gfas\": \"/home/kisner/scratch/desi/tutorials\",\n", 81 | "# }\n", 82 | "\n", 83 | "paths = {\"targets\": \"/global/cfs/cdirs/desi/target/catalogs/dr8/0.31.1/targets/main/resolve/\", \n", 84 | " \"skies\": \"/global/cfs/cdirs/desi/target/catalogs/dr8/0.32.0/skies/\", \n", 85 | " \"gfas\": \"/global/cfs/cdirs/desi/target/catalogs/dr8/0.32.0/gfas/\",\n", 86 | "}\n", 87 | "\n", 88 | "names = {\"targets\": \"dr8-hp-10,66,69,75,87,105.fits\", \n", 89 | " \"skies\":\"dr8-0.32.0.fits\", \n", 90 | " \"gfas\": \"dr8-0.32.0.fits\"}" 91 | ] 92 | }, 93 | { 94 | "cell_type": "markdown", 95 | "metadata": {}, 96 | "source": [ 97 | "Some more files needed to run fiberassign" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": null, 103 | "metadata": {}, 104 | "outputs": [], 105 | "source": [ 106 | "mtlfile = os.path.join(workdir, 'mtl.fits')\n", 107 | "truthfile = os.path.join(workdir, 'truth.fits')\n", 108 | "targetcutfile = os.path.join(workdir, 'targets.fits') \n", 109 | "skycutfile = os.path.join(workdir, 'sky.fits') \n", 110 | "targetfile = os.path.join(paths[\"targets\"], \"targets-{}\".format(names[\"targets\"]))\n", 111 | "skyfile = os.path.join(paths[\"skies\"], \"skies-{}\".format(names[\"skies\"]))\n", 112 | "gfafile = os.path.join(paths[\"gfas\"], \"gfas-{}\".format(names[\"gfas\"]))\n", 113 | "tilefile = os.path.join(workdir, \"tiles.fits\")" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": null, 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "# tile selection\n", 123 | "program = \"dark\"\n", 124 | "\n", 125 | "tiles = desimodel.io.load_tiles()\n", 126 | "bright = tiles['PROGRAM']=='BRIGHT'\n", 127 | " \n", 128 | "small = ((tiles['RA']>32) & (tiles['RA']<37) & (tiles['DEC']<3) & (tiles['DEC']>-3))\n", 129 | "\n", 130 | "if program==\"bright\":\n", 131 | " Table(tiles[(bright)&(small)]).write(tilefile, overwrite=True)\n", 132 | "else:\n", 133 | " Table(tiles[(~bright) & (small)]).write(tilefile, overwrite=True)\n", 134 | "\n", 135 | "print(\"Wrote tiles to {}\".format(tilefile))" 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "metadata": {}, 142 | "outputs": [], 143 | "source": [ 144 | "# target selection results\n", 145 | "print('Started reading {}'.format(targetfile))\n", 146 | "targetdata = fitsio.read(targetfile, 'TARGETS')\n", 147 | "ii = (targetdata['RA']>30) & (targetdata['RA']<39) & (targetdata['DEC']<5) & (targetdata['DEC']>-5)\n", 148 | "targetdata = targetdata[ii]\n", 149 | "print('Done reading target data')" 150 | ] 151 | }, 152 | { 153 | "cell_type": "code", 154 | "execution_count": null, 155 | "metadata": {}, 156 | "outputs": [], 157 | "source": [ 158 | "plt.scatter(targetdata['RA'], targetdata['DEC'],s=0.1, alpha=0.1)\n", 159 | "plt.xlabel('RA [deg]')\n", 160 | "plt.ylabel('DEC [deg]')" 161 | ] 162 | }, 163 | { 164 | "cell_type": "code", 165 | "execution_count": null, 166 | "metadata": {}, 167 | "outputs": [], 168 | "source": [ 169 | "# write down the targets to disk\n", 170 | "\n", 171 | "if (not os.path.exists(targetcutfile)):\n", 172 | " Table(targetdata).write(targetcutfile, overwrite=True)\n", 173 | " print('Done writing target cut data')" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": null, 179 | "metadata": {}, 180 | "outputs": [], 181 | "source": [ 182 | "# compute sky file\n", 183 | "if (not os.path.exists(skycutfile)):\n", 184 | " skydata = fitsio.read(skyfile)\n", 185 | " ii = (skydata['RA']>30) & (skydata['RA']<39) & (skydata['DEC']<5) & (skydata['DEC']>-5)\n", 186 | " jj = (skydata['DESI_TARGET'] & desi_mask.SKY)!=0\n", 187 | " skydata = skydata[ii&jj]\n", 188 | " Table(skydata).write(skycutfile, overwrite=True)\n", 189 | " print('Done writing sky cut data')\n", 190 | "skydata = Table.read(skycutfile)" 191 | ] 192 | }, 193 | { 194 | "cell_type": "code", 195 | "execution_count": null, 196 | "metadata": {}, 197 | "outputs": [], 198 | "source": [ 199 | "plt.scatter(skydata['RA'], skydata['DEC'],s=0.1, alpha=0.1)\n", 200 | "plt.xlabel('RA [deg]')\n", 201 | "plt.ylabel('DEC [deg]')" 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "metadata": {}, 208 | "outputs": [], 209 | "source": [ 210 | "#compute MTL\n", 211 | "if not os.path.exists(mtlfile):\n", 212 | " print('computing mtl')\n", 213 | " import desitarget.mtl\n", 214 | " mtl = desitarget.mtl.make_mtl(targetdata, 'DARK|GRAY')\n", 215 | "\n", 216 | " mtl.meta['EXTNAME'] = 'MTL'\n", 217 | " mtl.write(mtlfile)\n", 218 | " \n", 219 | " #print some stats\n", 220 | " print('MWS_TARGETS: {}'.format(np.count_nonzero(mtl['MWS_TARGET']!=0)))\n", 221 | " print('BGS_TARGETS: {}'.format(np.count_nonzero(mtl['BGS_TARGET']!=0)))\n", 222 | " print('DESI_TARGETS: {}'.format(np.count_nonzero(mtl['DESI_TARGET']!=0)))\n", 223 | " print('finished computing mtl')\n", 224 | "mtl = Table.read(mtlfile)" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": null, 230 | "metadata": {}, 231 | "outputs": [], 232 | "source": [ 233 | "plt.scatter(mtl['RA'], mtl['DEC'], s=0.1, alpha=0.1)\n", 234 | "plt.xlabel('RA [deg]')\n", 235 | "plt.ylabel('DEC [deg]')" 236 | ] 237 | }, 238 | { 239 | "cell_type": "markdown", 240 | "metadata": {}, 241 | "source": [ 242 | "By default, fiberassign uses the real focalplane geometry and properties at the current time. During operations, if a fiber broke yesterday and you run fiberassign today with an up-to-date desimodel data checkout, then that fiber will not be assigned. For this tutorial, we will run fiberassign with a *rundate* set to a time in the past before the start of commissioning. This will give us a nominal focalplane layout with all positioners working." 243 | ] 244 | }, 245 | { 246 | "cell_type": "code", 247 | "execution_count": null, 248 | "metadata": {}, 249 | "outputs": [], 250 | "source": [ 251 | "assign_date = \"2022-01-01T00:00:00+00:00\"" 252 | ] 253 | }, 254 | { 255 | "cell_type": "code", 256 | "execution_count": null, 257 | "metadata": {}, 258 | "outputs": [], 259 | "source": [ 260 | "cmd = 'fiberassign --overwrite --mtl mtl.fits --sky sky.fits'\n", 261 | "cmd += ' --rundate {}'.format(assign_date)\n", 262 | "cmd += ' --footprint {}'.format(tilefile)\n", 263 | "cmd += ' --outdir {}'.format(outdir)\n", 264 | "\n", 265 | "print(cmd)" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": null, 271 | "metadata": {}, 272 | "outputs": [], 273 | "source": [ 274 | "print('RUNNING: '+cmd)\n", 275 | "try:\n", 276 | " results = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)\n", 277 | " print(results.decode())\n", 278 | " print('--- SUCCESS ---')\n", 279 | "except subprocess.CalledProcessError as ex:\n", 280 | " print('--- ERROR {} ---'.format(ex.returncode))\n", 281 | " print(ex.output.decode())" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": null, 287 | "metadata": {}, 288 | "outputs": [], 289 | "source": [ 290 | "tiles = Table.read(\"tiles.fits\")\n", 291 | "targets = Table.read(\"mtl.fits\")" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "metadata": {}, 298 | "outputs": [], 299 | "source": [ 300 | "# Gather all outputs\n", 301 | "assignments = list()\n", 302 | "for tileid in tiles['TILEID']:\n", 303 | " outtile = '{}/fiberassign-{:06d}.fits'.format(outdir, tileid)\n", 304 | " assignments.append(Table.read(outtile, hdu=1))\n", 305 | " \n", 306 | "assigned_targetids = np.concatenate([tmp['TARGETID'] for tmp in assignments])\n", 307 | "isAssigned = np.in1d(targets['TARGETID'], assigned_targetids)\n", 308 | "\n", 309 | "plt.figure(figsize=(12,8))\n", 310 | "plt.plot(targets['RA'][isAssigned], targets['DEC'][isAssigned], 'k,')\n", 311 | "plt.title('Targets assigned to fibers')\n", 312 | "plt.xlabel('RA [deg]')\n", 313 | "plt.ylabel('DEC [deg]')" 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "metadata": {}, 320 | "outputs": [], 321 | "source": [ 322 | "plt.figure(figsize=(12,8))\n", 323 | "plt.plot(targets['RA'][~isAssigned], targets['DEC'][~isAssigned], 'k,')\n", 324 | "plt.title('Targets left unassigned to fibers')\n", 325 | "plt.xlabel('RA [deg]')\n", 326 | "plt.ylabel('DEC [deg]')" 327 | ] 328 | }, 329 | { 330 | "cell_type": "markdown", 331 | "metadata": {}, 332 | "source": [ 333 | "We can now run some basic QA on these outputs. We run this on the merged outputs so that we have access to more target information." 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "metadata": {}, 340 | "outputs": [], 341 | "source": [ 342 | "cmd = 'fba_run_qa --prefix fiberassign-'\n", 343 | "cmd += ' --rundate {}'.format(assign_date)\n", 344 | "cmd += ' --footprint {}'.format(tilefile)\n", 345 | "cmd += ' --dir {}'.format(outdir)\n", 346 | "\n", 347 | "print(cmd)" 348 | ] 349 | }, 350 | { 351 | "cell_type": "code", 352 | "execution_count": null, 353 | "metadata": {}, 354 | "outputs": [], 355 | "source": [ 356 | "print('RUNNING: '+cmd)\n", 357 | "try:\n", 358 | " results = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)\n", 359 | " print(results.decode())\n", 360 | " print('--- SUCCESS ---')\n", 361 | "except subprocess.CalledProcessError as ex:\n", 362 | " print('--- ERROR {} ---'.format(ex.returncode))\n", 363 | " print(ex.output.decode())" 364 | ] 365 | }, 366 | { 367 | "cell_type": "code", 368 | "execution_count": null, 369 | "metadata": {}, 370 | "outputs": [], 371 | "source": [ 372 | "# Load the results back in and print\n", 373 | "import json\n", 374 | "from pprint import PrettyPrinter\n", 375 | "\n", 376 | "qa_file = os.path.join(outdir, \"qa.json\")\n", 377 | "qa_data = None\n", 378 | "with open(qa_file, \"r\") as f:\n", 379 | " qa_data = json.load(f)\n", 380 | "\n", 381 | "pp = PrettyPrinter(indent=2)\n", 382 | "pp.pprint(qa_data)" 383 | ] 384 | }, 385 | { 386 | "cell_type": "code", 387 | "execution_count": null, 388 | "metadata": {}, 389 | "outputs": [], 390 | "source": [] 391 | } 392 | ], 393 | "metadata": { 394 | "kernelspec": { 395 | "display_name": "DESI 22.2", 396 | "language": "python", 397 | "name": "desi-22.2" 398 | }, 399 | "language_info": { 400 | "codemirror_mode": { 401 | "name": "ipython", 402 | "version": 3 403 | }, 404 | "file_extension": ".py", 405 | "mimetype": "text/x-python", 406 | "name": "python", 407 | "nbconvert_exporter": "python", 408 | "pygments_lexer": "ipython3", 409 | "version": "3.9.7" 410 | } 411 | }, 412 | "nbformat": 4, 413 | "nbformat_minor": 4 414 | } 415 | -------------------------------------------------------------------------------- /fiberassign/FiberAssignMocks.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# fiberassign on Mock Catalogs" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "**NOTE**: this tutorial takes ~30 minutes just to run all the steps" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "import os, sys, subprocess, fitsio, shutil\n", 24 | "from pkg_resources import resource_filename\n", 25 | "from collections import Counter\n", 26 | "\n", 27 | "import numpy as np\n", 28 | "from astropy.table import Table\n", 29 | "from astropy.io import fits\n", 30 | "\n", 31 | "import desimodel.io\n", 32 | "import desimodel.focalplane\n", 33 | "import desimodel.footprint\n", 34 | "from desitarget.targetmask import desi_mask, obsconditions\n", 35 | "import matplotlib.pyplot as plt" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "## Learning goals\n", 43 | "\n", 44 | "In this notebook you will learn how to:\n", 45 | "\n", 46 | "* Generate mock files from simulations to feed into fiberassign.\n", 47 | "* Run fiberassign using the files generated above.\n", 48 | "* Explore the outputs of fiberassign to estimate its efficiency.\n", 49 | "\n", 50 | "\n", 51 | "See https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC for instructions on configuring jupyter kernels with pre-installed DESI software at NERSC. This tutorial was last tested with the 22.2 kernel on April 2022.\n", 52 | "\n", 53 | "This notebook reuses code in the [main fiberassign notebook](https://github.com/desihub/tutorials/blob/master/FiberAssign.ipynb)." 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "workdir = os.path.join(os.environ['SCRATCH'], 'desi', 'test', 'fiberassign_mock')\n", 63 | "os.makedirs(workdir, exist_ok=True)\n", 64 | "outdir = os.path.join(workdir, 'output')\n", 65 | "os.makedirs(outdir, exist_ok=True)\n", 66 | "shutil.copy(resource_filename('desitarget.mock', 'data/select-mock-targets.yaml'), workdir)\n", 67 | "os.chdir(workdir)" 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "# This is the content of the configuration file to be passed to select_mock_targets\n", 77 | "!cat select-mock-targets.yaml" 78 | ] 79 | }, 80 | { 81 | "cell_type": "markdown", 82 | "metadata": {}, 83 | "source": [ 84 | "Some more files needed to run fiberassign" 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "metadata": {}, 91 | "outputs": [], 92 | "source": [ 93 | "mtlfile = os.path.join(workdir, 'mtl.fits')\n", 94 | "truthfile = os.path.join(workdir, 'truth.fits')\n", 95 | "starfile = os.path.join(workdir, 'std.fits')\n", 96 | "targetcutfile = os.path.join(workdir, 'targets.fits') \n", 97 | "skycutfile = os.path.join(workdir, 'sky.fits') \n", 98 | "tilefile = os.path.join(workdir, \"tiles.fits\")" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": null, 104 | "metadata": {}, 105 | "outputs": [], 106 | "source": [ 107 | "# tile selection\n", 108 | "program = \"dark\"\n", 109 | "\n", 110 | "tiles = desimodel.io.load_tiles()\n", 111 | "bright = tiles['PROGRAM']=='BRIGHT'\n", 112 | " \n", 113 | "small = ((tiles['RA']>12) & (tiles['RA']<20) & (tiles['DEC']<1) & (tiles['DEC']>-1))\n", 114 | "\n", 115 | "if program==\"bright\":\n", 116 | " Table(tiles[(bright)&(small)]).write(tilefile, overwrite=True)\n", 117 | "else:\n", 118 | " Table(tiles[(~bright) & (small)]).write(tilefile, overwrite=True)\n", 119 | "\n", 120 | "print(\"Wrote tiles to {}\".format(tilefile))" 121 | ] 122 | }, 123 | { 124 | "cell_type": "code", 125 | "execution_count": null, 126 | "metadata": {}, 127 | "outputs": [], 128 | "source": [ 129 | "!pwd\n", 130 | "!ls" 131 | ] 132 | }, 133 | { 134 | "cell_type": "code", 135 | "execution_count": null, 136 | "metadata": {}, 137 | "outputs": [], 138 | "source": [ 139 | "cmd = \"srun -A desi -N 2 -n 16 -c 8\" # -c 8 not 4 for hyperthreading\n", 140 | "cmd += \" -C haswell -t 01:00:00 --qos interactive \"\n", 141 | "cmd += \" mpi_select_mock_targets --no-spectra --nproc 4 --nside 32 --seed 10\"\n", 142 | "cmd += \" -c ./select-mock-targets.yaml \"\n", 143 | "cmd += \" --output_dir ./ \"\n", 144 | "cmd += \" --tiles ./tiles.fits\"\n", 145 | "print(cmd)" 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "metadata": {}, 152 | "outputs": [], 153 | "source": [ 154 | "# To run target selection comment out these lines. it should take ~30 minutes\n", 155 | "print('RUNNING: '+cmd)\n", 156 | "try:\n", 157 | " results = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)\n", 158 | " print(results.decode())\n", 159 | " print('--- SUCCESS ---')\n", 160 | "except subprocess.CalledProcessError as ex:\n", 161 | " print('--- ERROR {} ---'.format(ex.returncode))\n", 162 | " print(ex.output.decode())\n", 163 | " raise ex" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": {}, 170 | "outputs": [], 171 | "source": [ 172 | "cmd = 'join_mock_targets --overwrite --mockdir ./'\n", 173 | "print('RUNNING: '+cmd)\n", 174 | "try:\n", 175 | " results = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)\n", 176 | " print(results.decode())\n", 177 | " print('--- SUCCESS ---')\n", 178 | "except subprocess.CalledProcessError as ex:\n", 179 | " print('--- ERROR {} ---'.format(ex.returncode))\n", 180 | " print(ex.output.decode())\n", 181 | " raise ex" 182 | ] 183 | }, 184 | { 185 | "cell_type": "code", 186 | "execution_count": null, 187 | "metadata": {}, 188 | "outputs": [], 189 | "source": [ 190 | "mtldata = Table.read('mtl-dark.fits')\n", 191 | "plt.scatter(mtldata['RA'], mtldata['DEC'],s=0.1, alpha=0.1)\n", 192 | "plt.xlabel('RA')\n", 193 | "plt.ylabel('DEC')" 194 | ] 195 | }, 196 | { 197 | "cell_type": "markdown", 198 | "metadata": {}, 199 | "source": [ 200 | "By default, fiberassign uses the real focalplane geometry and properties at the current time. During operations, if a fiber broke yesterday and you run fiberassign today with an up-to-date desimodel data checkout, then that fiber will not be assigned. For this tutorial, we will run fiberassign with a *rundate* set to a time in the past before the start of commissioning. This will give us a nominal focalplane layout with all positioners working." 201 | ] 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": null, 206 | "metadata": {}, 207 | "outputs": [], 208 | "source": [ 209 | "assign_date = \"2022-01-01T00:00:00+00:00\"" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": null, 215 | "metadata": {}, 216 | "outputs": [], 217 | "source": [ 218 | "# Run fiberassign\n", 219 | "cmd = 'fiberassign --overwrite --mtl mtl-dark.fits --sky sky.fits'\n", 220 | "cmd += ' --rundate {}'.format(assign_date)\n", 221 | "cmd += ' --footprint ./tiles.fits'\n", 222 | "cmd += ' --outdir ./output/'\n", 223 | "cmd = cmd.format(outdir=outdir)" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "metadata": {}, 230 | "outputs": [], 231 | "source": [ 232 | "print('RUNNING: '+cmd)\n", 233 | "try:\n", 234 | " results = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)\n", 235 | " print(results.decode())\n", 236 | " print('--- SUCCESS ---')\n", 237 | "except subprocess.CalledProcessError as ex:\n", 238 | " print('--- ERROR {} ---'.format(ex.returncode))\n", 239 | " print(ex.output.decode())\n", 240 | " raise ex" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": null, 246 | "metadata": {}, 247 | "outputs": [], 248 | "source": [ 249 | "!ls output" 250 | ] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "execution_count": null, 255 | "metadata": {}, 256 | "outputs": [], 257 | "source": [ 258 | "tiles = Table.read(\"tiles.fits\")\n", 259 | "targets = Table.read(\"mtl-dark.fits\")" 260 | ] 261 | }, 262 | { 263 | "cell_type": "code", 264 | "execution_count": null, 265 | "metadata": {}, 266 | "outputs": [], 267 | "source": [ 268 | "# Gather all outputs\n", 269 | "assignments = list()\n", 270 | "for tileid in tiles['TILEID']:\n", 271 | " fafile = '{}/fiberassign-{:06d}.fits'.format(outdir, tileid)\n", 272 | " assignments.append(Table.read(fafile, hdu=1))\n", 273 | " \n", 274 | "assigned_targetids = np.concatenate([tmp['TARGETID'] for tmp in assignments])\n", 275 | "isAssigned = np.in1d(targets['TARGETID'], assigned_targetids)\n", 276 | "\n", 277 | "plt.figure(figsize=(12,4))\n", 278 | "plt.plot(targets['RA'][isAssigned], targets['DEC'][isAssigned], 'k,')\n", 279 | "plt.title('Targets assigned to fibers')\n", 280 | "plt.xlabel('RA [deg]')\n", 281 | "plt.ylabel('DEC [deg]')" 282 | ] 283 | }, 284 | { 285 | "cell_type": "code", 286 | "execution_count": null, 287 | "metadata": {}, 288 | "outputs": [], 289 | "source": [ 290 | "plt.figure(figsize=(12,4))\n", 291 | "plt.plot(targets['RA'][~isAssigned], targets['DEC'][~isAssigned], 'k,')\n", 292 | "plt.title('Targets left unassigned to fibers')\n", 293 | "plt.xlabel('RA [deg]')\n", 294 | "plt.ylabel('DEC [deg]')" 295 | ] 296 | }, 297 | { 298 | "cell_type": "code", 299 | "execution_count": null, 300 | "metadata": {}, 301 | "outputs": [], 302 | "source": [] 303 | } 304 | ], 305 | "metadata": { 306 | "kernelspec": { 307 | "display_name": "DESI 22.2", 308 | "language": "python", 309 | "name": "desi-22.2" 310 | }, 311 | "language_info": { 312 | "codemirror_mode": { 313 | "name": "ipython", 314 | "version": 3 315 | }, 316 | "file_extension": ".py", 317 | "mimetype": "text/x-python", 318 | "name": "python", 319 | "nbconvert_exporter": "python", 320 | "pygments_lexer": "ipython3", 321 | "version": "3.9.7" 322 | } 323 | }, 324 | "nbformat": 4, 325 | "nbformat_minor": 4 326 | } 327 | -------------------------------------------------------------------------------- /meetings/Dec2020/3pcf/DESI_3PCF_Tutorial_Slides.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/3pcf/DESI_3PCF_Tutorial_Slides.pdf -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/SDSS_QSO_spec.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/SDSS_QSO_spec.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/ann.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/ann.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/ann_layers.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/ann_layers.jpg -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/architecture.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/boxes.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/boxes.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/computational_graph_hpc.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/computational_graph_hpc.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/computational_graph_simple.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/computational_graph_simple.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/general_logistic_unit.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/general_logistic_unit.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/optimization_gd.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/optimization_gd.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/Images/target.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/ML_TensorFlow/Images/target.png -------------------------------------------------------------------------------- /meetings/Dec2020/ML_TensorFlow/setup.txt: -------------------------------------------------------------------------------- 1 | To set up the QuasarNET kernel: 2 | 3 | 4 | # 0. Move to the directory where you would like QuasarNET to be stored 5 | 6 | 7 | # 1. Load python 8 | 9 | module load python 10 | 11 | 12 | # 2. Create a Conda environment 13 | 14 | conda create -n qnet_tutorial python=3 scipy numpy matplotlib h5py ipykernel pandas scikit-learn 15 | source activate qnet_tutorial 16 | pip install fitsio 17 | 18 | 19 | # 3. Set up a Jupyter kernel 20 | 21 | python -m ipykernel install --user --name qnet_tutorial --display-name qnet_tutorial 22 | 23 | 24 | # 4. Install TensorFlow 25 | 26 | pip install tensorflow==2.2.0 27 | 28 | 29 | # 5. Clone the QuasarNET repo and install 30 | 31 | git clone https://github.com/ngbusca/QuasarNET.git QuasarNET_tutorial 32 | cd QuasarNET_tutorial 33 | python setup.py install 34 | 35 | 36 | Once you have followed these steps, then open up the Jupyter notebook for the tutorial, and choose the "qnet_tutorial" option from the top right corner. -------------------------------------------------------------------------------- /meetings/Dec2020/README.md: -------------------------------------------------------------------------------- 1 | # Tutorials from DESI December 2020 Virtual workshop 2 | - Tutorial on [Large Scale Structure Catalogs](https://github.com/desihub/tutorials/blob/master/Dec2020/LSS.ipynb) by Ashley Ross 3 | - Tutorial on [Fitting Galaxy Spectra](https://github.com/desihub/tutorials/tree/master/Dec2020/galaxies) by ChangHoon Hahn 4 | - Tutorial on [Machine Learning using Tensorflow](https://github.com/desihub/tutorials/tree/master/Dec2020/ML_TensorFlow) by James Farr 5 | - Tutorial on [Quick Quasars and Picca](https://github.com/desihub/tutorials/tree/master/Dec2020/quickquasars) by Oleg Burgueno and Alma Gonzalez 6 | - Tutorial on [Three point correlation functions](https://github.com/desihub/tutorials/tree/master/Dec2020/3pcf) by Zachary Slepian 7 | -------------------------------------------------------------------------------- /meetings/Dec2020/galaxies/coadd-66003-20200315-1-00055654.few.fits: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/galaxies/coadd-66003-20200315-1-00055654.few.fits -------------------------------------------------------------------------------- /meetings/Dec2020/galaxies/csp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/galaxies/csp.png -------------------------------------------------------------------------------- /meetings/Dec2020/galaxies/csp_simple.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/galaxies/csp_simple.png -------------------------------------------------------------------------------- /meetings/Dec2020/galaxies/fitting_redshifts.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Getting started on jupyter.nersc.gov\n", 8 | "Lets first setup the DESI environment and install the DESI-specific kernel\n", 9 | "```bash\n", 10 | "source /global/common/software/desi/desi_environment.sh 19.12\n", 11 | "${DESIMODULES}/install_jupyter_kernel.sh 19.12\n", 12 | "```\n", 13 | "See [https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC](https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC) for more details. \n", 14 | "\n", 15 | "Now we can import DESI-specific python packages from [desihub](https://github.com/desihub/). \n", 16 | "\n", 17 | "\n", 18 | "# Reading DESI spectra\n", 19 | "For instance we can use `desispec` to read in some DESI BGS coadded spectra from Comissioning that I've included in the repo" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": {}, 26 | "outputs": [], 27 | "source": [ 28 | "from desispec.io import read_spectra" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "metadata": {}, 35 | "outputs": [], 36 | "source": [ 37 | "# -- plotting -- \n", 38 | "import matplotlib as mpl\n", 39 | "import matplotlib.pyplot as plt\n", 40 | "mpl.rcParams['font.family'] = 'serif'\n", 41 | "mpl.rcParams['axes.linewidth'] = 1.5\n", 42 | "mpl.rcParams['axes.xmargin'] = 1\n", 43 | "mpl.rcParams['xtick.labelsize'] = 'x-large'\n", 44 | "mpl.rcParams['xtick.major.size'] = 5\n", 45 | "mpl.rcParams['xtick.major.width'] = 1.5\n", 46 | "mpl.rcParams['ytick.labelsize'] = 'x-large'\n", 47 | "mpl.rcParams['ytick.major.size'] = 5\n", 48 | "mpl.rcParams['ytick.major.width'] = 1.5\n", 49 | "mpl.rcParams['legend.frameon'] = False" 50 | ] 51 | }, 52 | { 53 | "cell_type": "code", 54 | "execution_count": null, 55 | "metadata": {}, 56 | "outputs": [], 57 | "source": [ 58 | "spectra = read_spectra('coadd-66003-20200315-1-00055654.few.fits') # handful of BGS galaxies from Tile 66003 observed on 03/15/2020, Petal #1, Exposure #00055654" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": null, 64 | "metadata": {}, 65 | "outputs": [], 66 | "source": [ 67 | "fig = plt.figure(figsize=(15,10))\n", 68 | "sub = fig.add_subplot(211)\n", 69 | "sub.plot(spectra.wave['brz'], spectra.flux['brz'][2])\n", 70 | "sub.set_xlim(3.6e3, 9.8e3)\n", 71 | "sub.set_ylabel('flux ($10^{-17} erg/s/\\AA/s/cm^2$)', fontsize=25)\n", 72 | "sub.set_ylim(-5, 20)\n", 73 | "\n", 74 | "sub = fig.add_subplot(212)\n", 75 | "sub.plot(spectra.wave['brz'], spectra.flux['brz'][6])\n", 76 | "sub.set_xlabel('wavelength ($\\AA$)', fontsize=25)\n", 77 | "sub.set_xlim(3.6e3, 9.8e3)\n", 78 | "#sub.set_ylabel('flux ($10^{-17} erg/s/\\AA/s/cm^2$)', fontsize=25)\n", 79 | "sub.set_ylim(-5, 20)" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "# Fitting Redshifts using `redrock`\n", 87 | "The main goal of DESI is to measure the redshifts of millions of galaxies. Redshifts will be measured for galaxy spectra, like the ones above using `redrock`: https://github.com/desihub/redrock\n", 88 | "\n", 89 | "`redrock` can be easily run on DESI spectra on the command line" 90 | ] 91 | }, 92 | { 93 | "cell_type": "code", 94 | "execution_count": null, 95 | "metadata": {}, 96 | "outputs": [], 97 | "source": [ 98 | "f_spec = 'coadd-66003-20200315-1-00055654.few.fits'\n", 99 | "f_rr_h5 = 'redrock.coadd.h5'\n", 100 | "f_rr = 'zbest.coadd.fits'\n", 101 | "print(f_rr)\n", 102 | "!rrdesi -o $f_rr_h5 -z $f_rr $f_spec" 103 | ] 104 | }, 105 | { 106 | "cell_type": "markdown", 107 | "metadata": {}, 108 | "source": [ 109 | "# `redrock` outputs\n", 110 | "Lets take a look at what `redrock` outputs" 111 | ] 112 | }, 113 | { 114 | "cell_type": "code", 115 | "execution_count": null, 116 | "metadata": {}, 117 | "outputs": [], 118 | "source": [ 119 | "from astropy.table import Table" 120 | ] 121 | }, 122 | { 123 | "cell_type": "code", 124 | "execution_count": null, 125 | "metadata": {}, 126 | "outputs": [], 127 | "source": [ 128 | "zbest = Table.read('zbest.coadd.fits', hdu=1)" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "metadata": {}, 135 | "outputs": [], 136 | "source": [ 137 | "zbest" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "metadata": {}, 144 | "outputs": [], 145 | "source": [ 146 | "import redrock.templates" 147 | ] 148 | }, 149 | { 150 | "cell_type": "code", 151 | "execution_count": null, 152 | "metadata": {}, 153 | "outputs": [], 154 | "source": [ 155 | "templates = dict()\n", 156 | "for filename in redrock.templates.find_templates():\n", 157 | " t = redrock.templates.Template(filename)\n", 158 | " templates[(t.template_type, t.sub_type)] = t" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "metadata": {}, 164 | "source": [ 165 | "## `redrock` galaxy templates\n", 166 | "`redrock` fits galaxy spectra with a linear combination of PCA templates. Here's what the galaxy templates look like:" 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "metadata": {}, 173 | "outputs": [], 174 | "source": [ 175 | "fig = plt.figure(figsize=(15,5))\n", 176 | "sub = fig.add_subplot(111)\n", 177 | "for i in range(templates[('GALAXY', '')].flux.shape[0]):\n", 178 | " sub.plot(templates[('GALAXY', '')].wave, templates[('GALAXY', '')].flux[i])\n", 179 | "sub.set_xlim(templates[('GALAXY', '')].wave.min(), templates[('GALAXY', '')].wave.max())\n", 180 | "sub.set_ylim(-0.02, 0.02)" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": {}, 186 | "source": [ 187 | "Since the `redrock` output file contains the coefficients of the PCA templates, we can use these templates to reconstruct the best-fit `redrock` fit" 188 | ] 189 | }, 190 | { 191 | "cell_type": "code", 192 | "execution_count": null, 193 | "metadata": {}, 194 | "outputs": [], 195 | "source": [ 196 | "i = 2\n", 197 | "z = zbest['Z'][i]\n", 198 | "targetid = zbest['TARGETID'][i]\n", 199 | "ncoeff = templates[(zbest['SPECTYPE'][i].strip(), zbest['SUBTYPE'][i].strip())].flux.shape[0]\n", 200 | "coeff = zbest['COEFF'][i][0:ncoeff]" 201 | ] 202 | }, 203 | { 204 | "cell_type": "code", 205 | "execution_count": null, 206 | "metadata": {}, 207 | "outputs": [], 208 | "source": [ 209 | "print('z_redrock = %.3f' % z)" 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": null, 215 | "metadata": {}, 216 | "outputs": [], 217 | "source": [ 218 | "tflux = templates[(zbest['SPECTYPE'][i].strip(), zbest['SUBTYPE'][i].strip())].flux.T.dot(coeff)\n", 219 | "twave = templates[(zbest['SPECTYPE'][i].strip(), zbest['SUBTYPE'][i].strip())].wave * (1+z)" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "fig = plt.figure(figsize=(15,10))\n", 229 | "sub = fig.add_subplot(211)\n", 230 | "\n", 231 | "for band in spectra.bands:\n", 232 | " sub.plot(spectra.wave[band], spectra.flux[band][i], 'C0', alpha=0.5)\n", 233 | "\n", 234 | "for icoeff in range(ncoeff): \n", 235 | " sub.plot(twave, templates[(zbest['SPECTYPE'][i].strip(), zbest['SUBTYPE'][i].strip())].flux[icoeff] * coeff[icoeff], ls=':', lw=0.5)\n", 236 | "sub.plot(twave, tflux, 'r-')\n", 237 | "sub.set_ylim(-5, 40)\n", 238 | "sub.set_xlim(3500, 10000)" 239 | ] 240 | }, 241 | { 242 | "cell_type": "markdown", 243 | "metadata": {}, 244 | "source": [ 245 | "For more details on the redrock output check out the following tutorials: \n", 246 | "- https://github.com/desihub/tutorials/blob/master/redrock/RedrockOutputs.ipynb\n", 247 | "- https://github.com/desihub/tutorials/blob/master/redrock/RedrockPlotSpec.md" 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": null, 253 | "metadata": {}, 254 | "outputs": [], 255 | "source": [] 256 | } 257 | ], 258 | "metadata": { 259 | "kernelspec": { 260 | "display_name": "DESI 19.12", 261 | "language": "python", 262 | "name": "desi-19.12" 263 | }, 264 | "language_info": { 265 | "codemirror_mode": { 266 | "name": "ipython", 267 | "version": 3 268 | }, 269 | "file_extension": ".py", 270 | "mimetype": "text/x-python", 271 | "name": "python", 272 | "nbconvert_exporter": "python", 273 | "pygments_lexer": "ipython3", 274 | "version": "3.6.6" 275 | } 276 | }, 277 | "nbformat": 4, 278 | "nbformat_minor": 4 279 | } 280 | -------------------------------------------------------------------------------- /meetings/Dec2020/galaxies/sfh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/galaxies/sfh.png -------------------------------------------------------------------------------- /meetings/Dec2020/galaxies/ssp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/galaxies/ssp.png -------------------------------------------------------------------------------- /meetings/Dec2020/galaxies/zh.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2020/galaxies/zh.png -------------------------------------------------------------------------------- /meetings/Dec2020/quickquasars/README.md: -------------------------------------------------------------------------------- 1 | # tutorial_DESI2020 2 | This is a tutorial of quick quasars and picca for the DESI meeting 2020 3 | 4 | We will start with a few introduction slides found here: https://docs.google.com/presentation/d/1sIiYZH0U9yj3bIPFMKlQC9fZvcBgYkNk6rl-zao_i3A/edit?usp=sharing 5 | 6 | # Requirements 7 | 8 | The DESI Master kernel in JupyterLab 9 | 10 | -Here are the instructions to install the DESI kernel: https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC at NERSC 11 | 12 | Set up picca 13 | 14 | -https://github.com/igmhub/picca.git 15 | 16 | -You can follow any of the options described here https://desi.lbl.gov/trac/wiki/LymanAlphaWG/Picca_at_NERSC to set up picca at NERSC. 17 | -------------------------------------------------------------------------------- /meetings/Dec2020/quickquasars/scripts/quickquasars_script.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -l 2 | ## Originally taken from https://github.com/desihub/desisim/blob/master/etc/quickquasar_slurm_script.sh 3 | 4 | #SBATCH -C haswell 5 | #SBATCH --partition=debug 6 | #SBATCH --account=desi 7 | #SBATCH --nodes=40 8 | #SBATCH --time=00:30:00 9 | #SBATCH --job-name=lyasim 10 | #SBATCH --output=lyasim.log 11 | 12 | # HOWTO: 13 | # 1) copy this script 14 | # 2) choose your options to quickquasars (z range, downsampling, target selection, DESI footprint) in the command below 15 | # 3) change idir/outdir, output, nodes, nthreads, time 16 | # 4) verify nodes=XX below is same as --nodes=XX above 17 | # 5) sbatch yourjob.sh 18 | # 19 | # example: 20 | # for the Dec 20 tutorial set downsampling=0.1 to get ~270K quasars. 21 | # To get a QSO density (z>2.1) ~ 50/deg2, using London mocks, set downsampling=0.4, 22 | 23 | source /project/projectdirs/desi/software/desi_environment.sh master 24 | 25 | 26 | seed=123 27 | downsampling=0.1 28 | idir=/global/cfs/projectdirs/desi/mocks/lya_forest/london/v9.0/v9.0.0 29 | outdir=$SCRATCH/mocks/quickquasar/tutorial-0.3-4/ 30 | nodes=40 # CHECK MATCHING #SBATCH --nodes ABOVE !!!! 31 | nthreads=4 # TO BE TUNED ; CAN HIT NODE MEMORY LIMIT ; 4 is max on edison for nside=16 and ~50 QSOs/deg2 32 | 33 | if [ ! -d $outdir ] ; then 34 | mkdir -p $outdir 35 | fi 36 | if [ ! -d $outdir/logs ] ; then 37 | mkdir -p $outdir/logs 38 | fi 39 | if [ ! -d $outdir/spectra-16 ] ; then 40 | mkdir -p $outdir/spectra-16 41 | fi 42 | 43 | echo "get list of skewers to run ..." 44 | 45 | files=`\ls -1 $idir/*/*/transmission*` 46 | nfiles=`echo $files | wc -w` 47 | nfilespernode=$((nfiles/nodes+1)) 48 | 49 | echo "n files =" $nfiles 50 | echo "n files per node =" $nfilespernode 51 | 52 | first=1 53 | last=$nfilespernode 54 | for node in `seq $nodes` ; do 55 | echo "starting node $node" 56 | 57 | # list of files to run 58 | if (( $node == $nodes )) ; then 59 | last="" 60 | fi 61 | 62 | echo ${first}-${last} 63 | tfiles=`echo $files | cut -d " " -f ${first}-${last}` 64 | first=$(( first + nfilespernode )) 65 | last=$(( last + nfilespernode )) 66 | command="srun -N 1 -n 1 -c $nthreads quickquasars --exptime 4000. -i $tfiles --zmin 1.8 --nproc $nthreads --outdir $outdir/spectra-16 --downsampling $downsampling --zbest --bbflux --desi-footprint --metals-from-file --balprob 0.16 --dla file --add-LYB --seed $seed" 67 | echo $command 68 | echo "log in $outdir/logs/node-$node.log" 69 | 70 | $command >& $outdir/logs/node-$node.log & 71 | 72 | done 73 | 74 | wait 75 | echo "END" 76 | -------------------------------------------------------------------------------- /meetings/Dec2021/README.md: -------------------------------------------------------------------------------- 1 | # Notebooks and code associated with the tutorials from December 2021 Collaboration Meeting 2 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/.gitignore: -------------------------------------------------------------------------------- 1 | *.out 2 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/README.md: -------------------------------------------------------------------------------- 1 | # mpi4py 2 | In this directory there are examples of mpi4py scripts (Point to Point, Scatter, Gather and Broadcasting). To run a python script, simply run 3 | `mpirun -np 2 python ` 4 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/Screen Shot 2016-04-09 at 6.54.01 PM.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/Dec2021/mpi4py/Screen Shot 2016-04-09 at 6.54.01 PM.png -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/bcast.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Example of brodcasting 3 | (c) Mehdi Rezaie 4 | 5 | ''' 6 | 7 | import numpy as np 8 | from mpi4py import MPI 9 | 10 | comm = MPI.COMM_WORLD 11 | rank = comm.Get_rank() 12 | 13 | if rank == 0: 14 | # create a data 15 | data = { 16 | 'key1' : [10, 10.1, 10+11j], 17 | 'key2' : ('Mehdi' , 'Rezaie'), 18 | 'key3' : np.array([1, 2, 3]), 19 | 'key4' : True 20 | } 21 | else: 22 | data = None 23 | 24 | data = comm.bcast(data, root=0) # broadcast 25 | 26 | if rank == 0:print("bcast finished") 27 | 28 | print("data on rank {} is : {}".format(comm.rank, data)) 29 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/ex1_helloworld.py: -------------------------------------------------------------------------------- 1 | """ Example1. Print Hello World by rank 0, otherwise Hey. 2 | 3 | run with 4 | $> mpirun -n 2 python 5 | """ 6 | 7 | from mpi4py import MPI 8 | 9 | comm = MPI.COMM_WORLD 10 | 11 | rank = comm.Get_rank() 12 | size = comm.Get_size() 13 | 14 | if rank==0: 15 | print(f"Hello World from {rank}") 16 | else: 17 | print(f"Hey from {rank}") 18 | 19 | 20 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/ex2_p2p.py: -------------------------------------------------------------------------------- 1 | """ Example2. Point to Point Communication 2 | 3 | run with 4 | $> mpirun -n 2 python 5 | """ 6 | 7 | from mpi4py import MPI 8 | 9 | comm = MPI.COMM_WORLD 10 | 11 | rank = comm.Get_rank() 12 | size = comm.Get_size() 13 | 14 | assert size>3, "this example requires at least 3 processes" 15 | 16 | if rank==0: 17 | data = {'survey':'desi', 'year':2021} 18 | comm.send(data, dest=1) 19 | 20 | data = {'survey':'jwst', 'year':2025} 21 | comm.send(data, dest=2) 22 | 23 | if rank in [1, 2]: 24 | data = comm.recv(source=0) 25 | print(f"rank:{rank}, data:{data}") 26 | 27 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/ex3_bcast.py: -------------------------------------------------------------------------------- 1 | """ Example3. Broadcasting 2 | 3 | run with 4 | $> mpirun -n 2 python 5 | """ 6 | 7 | from mpi4py import MPI 8 | 9 | comm = MPI.COMM_WORLD 10 | 11 | rank = comm.Get_rank() 12 | size = comm.Get_size() 13 | 14 | if rank==0: 15 | data = {'a':[1, 2, 3], 16 | 'b':2.+3j, 17 | 'c':'this is a sentence!'} 18 | else: 19 | data = None 20 | 21 | print(f'before bcast rank: {rank}, data: {data}') 22 | data = comm.bcast(data, root=0) 23 | print(f'after bcast rank: {rank}, data: {data}') 24 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/ex4_scatter.py: -------------------------------------------------------------------------------- 1 | """ Example4. Scattering 2 | 3 | run with 4 | $> mpirun -n 2 python 5 | """ 6 | 7 | from mpi4py import MPI 8 | 9 | comm = MPI.COMM_WORLD 10 | 11 | rank = comm.Get_rank() 12 | size = comm.Get_size() 13 | 14 | num_int = 4 15 | 16 | assert size==num_int, f"({num_int}) != ({size})" 17 | 18 | if rank==0: 19 | data = [i for i in range(num_int)] 20 | else: 21 | data = None 22 | 23 | data = comm.scatter(data, root=0) 24 | print(f'rank: {rank}, data:{data}') 25 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/ex5_gather.py: -------------------------------------------------------------------------------- 1 | """ Example5. Gathering 2 | 3 | run with 4 | $> mpirun -n 2 python 5 | """ 6 | 7 | from mpi4py import MPI 8 | 9 | comm = MPI.COMM_WORLD 10 | 11 | rank = comm.Get_rank() 12 | size = comm.Get_size() 13 | 14 | data = [rank, rank*rank] 15 | print(f"before gather, rank: {rank}, data: {data}") 16 | 17 | comm.Barrier() 18 | data = comm.gather(data, root=0) 19 | 20 | if rank==0: 21 | for i in range(size): 22 | assert data[i] = [i, i*i] 23 | else: 24 | assert data is None 25 | 26 | print(f"after gather, rank: {rank}, data: {data}") 27 | 28 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/ex6_simple_pi.py: -------------------------------------------------------------------------------- 1 | """ Example6. Calculate Pi (3.1415..) with a simple 2 | code based on trapzoidal method. 3 | 4 | run with 5 | $> python 6 | """ 7 | import numpy as np 8 | from time import time 9 | 10 | def f(x): 11 | return 4.0/(1.0+x*x) 12 | 13 | def trap(local_a,local_b,local_n,h): 14 | # trapzoidal method 15 | estimate = (f(local_a)+f(local_b))/2.0 16 | for i in np.arange(1,local_n): 17 | x = local_a+float(i)*h 18 | estimate += f(x) 19 | return estimate*h 20 | 21 | b = 1.0 22 | a = 0.0 23 | n = 1000000 24 | h = (b-a)/float(n) 25 | 26 | start = time() 27 | pi = trap(a, b, n, h) 28 | end = time() 29 | 30 | print(f'Pi=%.6f (true)'%np.pi) 31 | print("Pi=%.6f (%d steps in %.3f secs)" %(pi, n, end-start)) 32 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/ex7_mpi_pi.py: -------------------------------------------------------------------------------- 1 | """ Example6. Calculate Pi (3.1415..) with a simple 2 | code based on trapzoidal method. 3 | 4 | run with 5 | $> python 6 | """ 7 | 8 | import numpy as np 9 | from mpi4py import MPI 10 | from time import time 11 | 12 | def f(x): 13 | return 4.0/(1.0+x*x) 14 | 15 | def trap(local_a,local_b,local_n,h): 16 | # trapzoidal method 17 | estimate = (f(local_a)+f(local_b))/2.0 18 | for i in np.arange(1,local_n): 19 | x = local_a+float(i)*h 20 | estimate += f(x) 21 | return estimate*h 22 | 23 | comm = MPI.COMM_WORLD 24 | size = comm.Get_size() 25 | rank = comm.Get_rank() 26 | 27 | b = 1.0 28 | a = 0.0 29 | n = 1000000 30 | h = (b-a)/float(n) 31 | 32 | if rank==0: 33 | start = time() 34 | 35 | local_n = int(n/size) 36 | local_a = a + rank*local_n*h 37 | local_b = local_a + local_n*h 38 | 39 | local_pi = trap(local_a, local_b, local_n, h) 40 | 41 | comm.Barrier() 42 | local_pi = comm.gather(local_pi, root=0) 43 | 44 | if rank==0: 45 | pi = sum(local_pi) 46 | end = time() 47 | print(f'Pi=%.6f (true)'%np.pi) 48 | print("Pi=%.6f (%d steps in %.3f secs)" %(pi, n, end-start)) 49 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/gather.py: -------------------------------------------------------------------------------- 1 | ''' 2 | example of gathering 3 | (c) Mehdi Rezaie 4 | 5 | ''' 6 | 7 | 8 | from mpi4py import MPI 9 | 10 | comm = MPI.COMM_WORLD 11 | size = comm.Get_size() 12 | rank = comm.Get_rank() 13 | 14 | 15 | 16 | # create data 17 | data = [(rank+1)**2, rank**2] 18 | 19 | print("before gather, data on rank %d is: "%rank, data) 20 | 21 | comm.Barrier() 22 | data = comm.gather(data, root=0) 23 | 24 | if rank == 0: 25 | for i in range(size): 26 | assert data[i] == [(i+1)**2,i**2] 27 | else: 28 | assert data is None 29 | 30 | print("data on rank: %d is: "%rank, data) 31 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/helloworld.c: -------------------------------------------------------------------------------- 1 | /* 2 | Example of Hello World using C 3 | 4 | Author: Mehdi Rezaie 5 | medirz90@icloud.com 6 | 7 | */ 8 | 9 | #include 10 | #include 11 | 12 | int main(int argc, char** argv) { 13 | 14 | MPI_Init(NULL, NULL); 15 | 16 | int size; 17 | MPI_Comm_size(MPI_COMM_WORLD, &size); 18 | 19 | int rank; 20 | MPI_Comm_rank(MPI_COMM_WORLD, &rank); 21 | 22 | char processor_name[MPI_MAX_PROCESSOR_NAME]; 23 | 24 | int name_len; 25 | MPI_Get_processor_name(processor_name, &name_len); 26 | 27 | if (rank == 0) 28 | { 29 | printf("Hello world from processor %s, rank %d out of %d processors\n", 30 | processor_name, rank, size); 31 | } 32 | else{ 33 | printf("Goodbye world from processor %s, rank %d out of %d processors\n", 34 | processor_name, rank, size); 35 | } 36 | MPI_Finalize(); 37 | } 38 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/helloworld.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Example of MPI Hello 3 | (c) Mehdi Rezaie 4 | 5 | ''' 6 | 7 | from mpi4py import MPI 8 | 9 | comm = MPI.COMM_WORLD 10 | size = comm.Get_size() 11 | rank = comm.Get_rank() 12 | name = MPI.Get_processor_name() 13 | 14 | 15 | print("Hi from rank {}".format(rank)) 16 | 17 | comm.Barrier() # keep all processes 18 | 19 | if rank ==0: 20 | print("Hello from rank {} from size {} on {}".format(rank, size, name)) 21 | 22 | 23 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/p2p.py: -------------------------------------------------------------------------------- 1 | ''' 2 | example of point to point 3 | (c) Mehdi Rezaie 4 | 5 | run with : 6 | mpirun --oversubscribe -np 3 python p2p.py 7 | ''' 8 | 9 | import numpy as np 10 | from mpi4py import MPI 11 | 12 | comm = MPI.COMM_WORLD 13 | rank = comm.Get_rank() 14 | size = comm.Get_size() 15 | 16 | 17 | if rank == 0: 18 | data = np.arange(10) 19 | comm.send(data[:5], dest=1, tag=13) 20 | comm.send(data[5:], dest=2, tag=14) 21 | print("Rank %d has : "%rank, data) 22 | elif rank == 1: 23 | data = comm.recv(source=0, tag=13) 24 | print("Rank %d Message Received, data is: "%rank, data) 25 | elif rank == 2: 26 | data = comm.recv(source=0, tag=14) 27 | print("Rank %d Message Received, data is: "%rank, data) 28 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/pi_mpi_gather.py: -------------------------------------------------------------------------------- 1 | ''' 2 | python pi_mpi.py 3 | last edit April 23 4 | 5 | mpirun --oversubscribe -np 5 python pi_mpi_gather.py 6 | ''' 7 | 8 | import numpy as np 9 | from time import time 10 | 11 | def f(x): 12 | return 4.0/(1.0+x*x) 13 | 14 | def trap(local_a, local_b, local_n,h): 15 | # trapzoidal algorithm 16 | estimate = (f(local_a)+f(local_b))/2.0 17 | for i in np.arange(1,local_n): 18 | x = local_a+float(i)*h 19 | estimate += f(x) 20 | estimate *= h 21 | return estimate 22 | 23 | 24 | from mpi4py import MPI 25 | comm = MPI.COMM_WORLD 26 | size = comm.Get_size() 27 | rank = comm.Get_rank() 28 | 29 | if rank ==0:start = time() 30 | 31 | b = 1.0 32 | a = 0.0 33 | n = 10000000 34 | h = (b-a)/float(n) 35 | 36 | # split data 37 | local_n = int(n/size) 38 | local_a = a+rank*float(local_n)*h 39 | local_b = local_a+float(local_n)*h 40 | 41 | 42 | local_int = trap(local_a, local_b, local_n, h) 43 | 44 | comm.Barrier() 45 | local_int = comm.gather(local_int, root=0) 46 | 47 | if rank == 0: 48 | end = time() 49 | print("Pi with %d steps is %f in %f secs" %(n, sum(local_int), end-start)) 50 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/pi_mpi_p2p.py: -------------------------------------------------------------------------------- 1 | ''' 2 | python pi_mpi.py 3 | (c) Mehdi Rezaie 4 | mpirun -np 5 python pi_mpi_p2p.py 5 | ''' 6 | 7 | from time import time 8 | import numpy as np 9 | 10 | def f(x): 11 | return 4.0/(1.0+x*x) 12 | 13 | def trap(local_a,local_b,local_n,h): 14 | estimate = (f(local_a)+f(local_b))/2.0 15 | for i in np.arange(1,local_n): 16 | x = local_a+float(i)*h 17 | estimate += f(x) 18 | # 19 | estimate *= h 20 | return estimate 21 | 22 | 23 | from mpi4py import MPI 24 | comm = MPI.COMM_WORLD 25 | size = comm.Get_size() 26 | rank = comm.Get_rank() 27 | 28 | if rank ==0:start = time() 29 | 30 | b = 1.0 31 | a = 0.0 32 | n = 10000000 33 | h = (b-a)/float(n) 34 | 35 | local_n = int(n/size) 36 | local_a = a+rank*float(local_n)*h 37 | local_b = local_a+float(local_n)*h 38 | 39 | 40 | local_int = trap(local_a,local_b,local_n,h) 41 | 42 | if rank != 0: 43 | comm.send(local_int,dest=0,tag = 0) 44 | else: 45 | total_int = local_int 46 | for node in np.arange(1,size): 47 | local_int = comm.recv(source=node,tag=0) 48 | total_int += local_int 49 | 50 | comm.Barrier() 51 | 52 | if rank == 0: 53 | end = time() 54 | print("Pi with %d steps is %f in %f secs" %(n, total_int, end-start)) 55 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/pi_simple.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Pi with a trapzoidal method 3 | no MPI 4 | 5 | (c) Mehdi Rezaie 6 | ''' 7 | from time import time 8 | import numpy as np 9 | 10 | def f(x): 11 | return 4.0/(1.0+x*x) 12 | 13 | def trap(local_a,local_b,local_n,h): 14 | estimate = (f(local_a)+f(local_b))/2.0 15 | for i in np.arange(1,local_n): 16 | x = local_a+float(i)*h 17 | estimate += f(x) 18 | # 19 | estimate *= h 20 | return estimate 21 | 22 | 23 | start = time() 24 | 25 | b = 1.0 26 | a = 0.0 27 | n = 10000000 28 | h = (b-a)/float(n) 29 | total_int = trap(a,b,n,h) 30 | end = time() 31 | print("Pi with %d steps is %f in %f secs" %(n, total_int, end-start)) 32 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/scatter.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Scatter 3 | 4 | (c) Mehdi Rezaie 5 | ''' 6 | 7 | from mpi4py import MPI 8 | 9 | comm = MPI.COMM_WORLD 10 | size = comm.Get_size() 11 | rank = comm.Get_rank() 12 | 13 | if rank == 0: 14 | data = [(i+1)**2 for i in range(size)] 15 | print('generated data set is: ',data) 16 | else: 17 | data = None 18 | 19 | data = comm.scatter(data, root=0) 20 | assert data == (rank+1)**2 21 | 22 | print("data on rank %d is: "%comm.rank, data) 23 | -------------------------------------------------------------------------------- /meetings/Dec2021/mpi4py/scatter_gather.py: -------------------------------------------------------------------------------- 1 | ''' 2 | Scatter Gather 3 | 4 | (c) Mehdi Rezaie 5 | ''' 6 | import numpy as np 7 | from mpi4py import MPI 8 | 9 | comm = MPI.COMM_WORLD 10 | size = comm.Get_size() 11 | rank = comm.Get_rank() 12 | 13 | if rank == 0: 14 | data = [(i+1)**2 for i in range(10)] 15 | print('generated data set is: ',data) 16 | data = [data[:5],data[5:]] 17 | else: 18 | data = None 19 | 20 | data = comm.scatter(data, root=0) 21 | 22 | 23 | print("data on rank %d is: "%comm.rank, data) 24 | if rank != 0: 25 | data = [2*d for d in data] 26 | 27 | comm.Barrier() 28 | data = comm.gather(data, root=0) 29 | 30 | if rank == 0: 31 | data = [d for di in data for d in di] 32 | for i in range(5, 10): 33 | assert data[i] == 2*(i+1)**2 34 | else: 35 | assert data is None 36 | 37 | print("data on rank: %d is: "%rank, data) 38 | -------------------------------------------------------------------------------- /meetings/Dec2022/README.md: -------------------------------------------------------------------------------- 1 | # Code and notebooks for the December 2022 Collaboration meeting tutorials 2 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/library.py: -------------------------------------------------------------------------------- 1 | import random 2 | import math 3 | 4 | def estimate_pi(n): 5 | c = 0 6 | for i in range(n): 7 | x = random.random() 8 | y = random.random() 9 | if math.sqrt(x*x + y*y) < 1: 10 | c += 1 11 | return c * 4.0 / n 12 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/pi-mpi.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | import time 4 | from library import estimate_pi 5 | 6 | def main(): 7 | parser = argparse.ArgumentParser(description="Monte Carlo Pi Estimatator (mpi4py)") 8 | parser.add_argument("-n", type=int, default=20_000_000, help="number of samples") 9 | args = parser.parse_args() 10 | 11 | from mpi4py import MPI 12 | comm = MPI.COMM_WORLD 13 | 14 | n = args.n 15 | p = comm.size 16 | 17 | comm.barrier() 18 | start = time.time() 19 | estimate_pi(n // p) 20 | comm.barrier() 21 | end = time.time() 22 | 23 | if comm.rank == 0: 24 | print(f"Elapsed time: {end - start}") 25 | 26 | if __name__ == "__main__": 27 | main() 28 | 29 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/pi-multiprocessing.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | import time 4 | import multiprocessing as mp 5 | 6 | from library import estimate_pi 7 | 8 | 9 | def main(): 10 | parser = argparse.ArgumentParser( 11 | description="Monte Carlo Pi Estimatator (threading)" 12 | ) 13 | parser.add_argument("-n", type=int, default=20_000_000, help="number of samples") 14 | parser.add_argument("-p", type=int, default=4, help="number of parallel processes") 15 | args = parser.parse_args() 16 | 17 | n = args.n 18 | p = args.p 19 | 20 | if n % p != 0: 21 | print(f"Warning: {n=} is not evenly divisble by {p=}") 22 | 23 | mp.set_start_method("spawn") 24 | start = time.time() 25 | with mp.Pool(processes=p) as pool: 26 | results = pool.map(estimate_pi, [n//p] * p) 27 | end = time.time() 28 | 29 | print(f"Elapsed time: {end - start}") 30 | 31 | if __name__ == "__main__": 32 | main() 33 | 34 | 35 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/pi-numba.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | import time 4 | 5 | import numba 6 | 7 | from library import estimate_pi 8 | 9 | # Apply numba JIT wrapper to estimate_pi function from our library 10 | # estimate_pi will be compiled to machine code when first called 11 | # Note not all python functions can be JIT wrapped 12 | estimate_pi = numba.jit(estimate_pi) 13 | 14 | def main(): 15 | parser = argparse.ArgumentParser(description="Monte Carlo Pi Estimatator (Numba)") 16 | parser.add_argument("-n", type=int, default=20_000_000, help="number of samples") 17 | args = parser.parse_args() 18 | 19 | n = args.n 20 | 21 | estimate_pi(100) 22 | 23 | start = time.time() 24 | result = estimate_pi(n) 25 | end = time.time() 26 | 27 | print(f"Elapsed time: {end - start}") 28 | 29 | 30 | if __name__ == "__main__": 31 | main() 32 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/pi-numpy.py: -------------------------------------------------------------------------------- 1 | #/usr/bin/env python3 2 | import argparse 3 | import time 4 | import numpy as np 5 | 6 | def estimate_pi(n): 7 | x = np.random.random((n, 2)) 8 | r = np.linalg.norm(x, axis=1) 9 | c = np.sum(r < 1) 10 | return 4.0 * c / n 11 | 12 | def main(): 13 | parser = argparse.ArgumentParser(description="Monte Carlo Pi Estimatator (NumPy)") 14 | parser.add_argument("-n", type=int, default=20_000_000, help="number of samples") 15 | args = parser.parse_args() 16 | 17 | n = args.n 18 | 19 | start = time.time() 20 | result = estimate_pi(n) 21 | end = time.time() 22 | 23 | print(f"Elapsed time: {end - start}") 24 | 25 | if __name__ == "__main__": 26 | main() 27 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/pi-serial.c: -------------------------------------------------------------------------------- 1 | // Example usage: 2 | // > gcc -o pi-serial-c pi-serial.c 3 | // > ./pi-serial-c 4 | // 5 | // Example output: 6 | // Result: 3.141237 7 | // Elapsed time: 0.294798 8 | // 9 | #include 10 | #include 11 | #include 12 | #include 13 | 14 | double drandom() { 15 | // Generate a uniform random number [0, 1). 16 | return (double) rand() / RAND_MAX; 17 | } 18 | 19 | double estimate_pi(unsigned long n) { 20 | unsigned long m = 0; 21 | double x, y; 22 | for(int i = 0; i < n; ++i) { 23 | x = drandom(); 24 | y = drandom(); 25 | if (sqrt(x*x + y*y) < 1.0) m++; 26 | } 27 | return (double) m * 4.0 / (double) n; 28 | } 29 | 30 | int main() { 31 | unsigned long n = 20 * 1000 * 1000; 32 | 33 | clock_t start = clock(); 34 | double result = estimate_pi(n); 35 | clock_t end = clock(); 36 | double elapsed_time = (double) (end - start) / CLOCKS_PER_SEC; 37 | printf("Result: %f\n", result); 38 | printf("Elapsed time: %f\n", elapsed_time); 39 | return 0; 40 | } 41 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/pi-serial.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | import time 4 | 5 | from library import estimate_pi 6 | 7 | 8 | def main(): 9 | parser = argparse.ArgumentParser(description="Monte Carlo Pi Estimatator (Serial)") 10 | parser.add_argument("-n", type=int, default=20_000_000, help="number of samples") 11 | args = parser.parse_args() 12 | 13 | n = args.n 14 | 15 | start = time.time() 16 | result = estimate_pi(n) 17 | end = time.time() 18 | 19 | print(f"Elapsed time: {end - start}") 20 | 21 | 22 | if __name__ == "__main__": 23 | main() 24 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/pi-threads.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import argparse 3 | from threading import Thread 4 | import time 5 | 6 | from library import estimate_pi 7 | 8 | 9 | def main(): 10 | parser = argparse.ArgumentParser( 11 | description="Monte Carlo Pi Estimatator (threading)" 12 | ) 13 | parser.add_argument("-n", type=int, default=20_000_000, help="number of samples") 14 | parser.add_argument("-p", type=int, default=4, help="number of parallel threads") 15 | args = parser.parse_args() 16 | 17 | n = args.n 18 | p = args.p 19 | 20 | if n % p != 0: 21 | print(f"Warning: {n=} is not evenly divisble by {p=}") 22 | 23 | import threading 24 | 25 | # Define a wrapper function that appends results from individual threads 26 | # to a shared (nonlocal) list in the main thread 27 | results = [] 28 | def estimate_pi_wrapper(n): 29 | nonlocal results 30 | start = time.time() 31 | results.append(estimate_pi(n)) 32 | end = time.time() 33 | print(f"{threading.current_thread().getName()} Elapsed time: {end - start}") 34 | 35 | # Create threads that will perform portion of calculation 36 | t = [Thread(target=estimate_pi_wrapper, args=(n // p,)) for i in range(p)] 37 | 38 | start = time.time() 39 | # Start threads 40 | [t[i].start() for i in range(p)] 41 | # Wait for threads to complete 42 | [t[i].join() for i in range(p)] 43 | end = time.time() 44 | 45 | print(f"Elapsed time: {end - start}") 46 | 47 | 48 | if __name__ == "__main__": 49 | main() 50 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/sleep-serial.py: -------------------------------------------------------------------------------- 1 | import time 2 | 3 | def iotask(n): 4 | time.sleep(n) 5 | 6 | n = 5 7 | 8 | start = time.time() 9 | iotask(n) 10 | end = time.time() 11 | 12 | print(end - start) 13 | 14 | 15 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/compute-pi-examples/sleep-threads.py: -------------------------------------------------------------------------------- 1 | from threading import Thread 2 | import time 3 | 4 | def iotask(n): 5 | time.sleep(n) 6 | 7 | n = 5 8 | p = 4 9 | 10 | t = [ 11 | Thread(target=iotask, args=(n/p,)) 12 | for i in range(p) 13 | ] 14 | 15 | start = time.time() 16 | [t[i].start() for i in range(p)] 17 | [t[i].join() for i in range(p)] 18 | end = time.time() 19 | 20 | print(end - start) 21 | 22 | 23 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/data-task-examples/data-cupy-mpi4py.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | import time 5 | 6 | import cupy as cp 7 | import numpy as np 8 | from tqdm import tqdm 9 | 10 | 11 | def process_data(i, a): 12 | w, v = np.linalg.eigh(a) 13 | return i, w 14 | 15 | def main(): 16 | parser = argparse.ArgumentParser() 17 | parser.add_argument('-n', type=int, default=1000) 18 | parser.add_argument('--ntasks', type=int, default=128) 19 | args = parser.parse_args() 20 | n = args.n 21 | ntasks = args.ntasks 22 | 23 | from mpi4py import MPI 24 | comm = MPI.COMM_WORLD 25 | rank = comm.rank 26 | size = comm.size 27 | 28 | # have each rank select a different device 29 | num_devices = cp.cuda.runtime.getDeviceCount() 30 | cp.cuda.runtime.setDevice(rank % num_devices) 31 | 32 | device_pci_bus_id = cp.cuda.Device().pci_bus_id 33 | print(f'{rank=} {device_pci_bus_id=}', flush=True) 34 | 35 | comm.barrier() 36 | 37 | data = cp.empty((ntasks, n, n), dtype=float) 38 | 39 | if rank == 0: 40 | print('cupy-mpi4py') 41 | print(f'{n=} {ntasks=} {size=}') 42 | 43 | # create a random set of data 44 | for i in range(ntasks): 45 | cp.random.seed(i) 46 | b = cp.random.rand(n, n) 47 | a = b.dot(b.T) 48 | data[i] = a 49 | 50 | # bcast data from root MPI rank 51 | comm.Bcast([data, MPI.DOUBLE], root=0) 52 | 53 | start = time.time() 54 | 55 | results = [] 56 | for i in range(rank, ntasks, size): 57 | result = process_data(i, data[i]) 58 | results.append(result) 59 | 60 | # gather results to root MPI rank 61 | results = comm.gather(results, root=0) 62 | 63 | if rank == 0: 64 | end = time.time() 65 | elapsed = end - start 66 | rate = ntasks / elapsed 67 | print(f'Processed {ntasks} tasks in {elapsed:.2f}s ({rate:.2f}it/s)') 68 | 69 | # flatten list of lists 70 | results = [r for sub in results for r in sub] 71 | 72 | # check results 73 | if {r[0] for r in results} == set(range(ntasks)): 74 | print(f'completed all tasks!') 75 | else: 76 | print('uh oh, something went wrong.') 77 | 78 | if __name__ == "__main__": 79 | main() 80 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/data-task-examples/data-cupy.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | 5 | import numpy as np 6 | import cupy as cp 7 | from tqdm import tqdm 8 | 9 | def process_data(i, a): 10 | w, v = np.linalg.eigh(a) 11 | return i, w 12 | 13 | def main(): 14 | parser = argparse.ArgumentParser() 15 | parser.add_argument('-n', type=int, default=1000) 16 | parser.add_argument('--ntasks', type=int, default=128) 17 | args = parser.parse_args() 18 | n = args.n 19 | ntasks = args.ntasks 20 | 21 | device_pci_bus_id = cp.cuda.Device().pci_bus_id 22 | 23 | print('cupy') 24 | print(f'{n=} {ntasks=} {device_pci_bus_id=}') 25 | 26 | # create a random set of data 27 | data = list() 28 | for i in range(ntasks): 29 | cp.random.seed(i) 30 | b = cp.random.rand(n, n) 31 | a = b.dot(b.T) 32 | data.append(a) 33 | 34 | results = [] 35 | # process tasks 36 | for i in tqdm(range(ntasks), ncols=80): 37 | results.append(process_data(i, data[i])) 38 | 39 | # check results 40 | if {r[0] for r in results} == set(range(ntasks)): 41 | print(f'completed all tasks!') 42 | else: 43 | print('uh oh, something went wrong.') 44 | 45 | if __name__ == "__main__": 46 | main() 47 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/data-task-examples/data-mpi4py.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | import time 5 | 6 | import numpy as np 7 | from tqdm import tqdm 8 | 9 | def process_data(i, a): 10 | w, v = np.linalg.eigh(a) 11 | return i, w 12 | 13 | 14 | if __name__ == "__main__": 15 | 16 | parser = argparse.ArgumentParser() 17 | parser.add_argument('-n', type=int, default=1000) 18 | parser.add_argument('--ntasks', type=int, default=128) 19 | args = parser.parse_args() 20 | n = args.n 21 | ntasks = args.ntasks 22 | 23 | from mpi4py import MPI 24 | comm = MPI.COMM_WORLD 25 | rank = comm.rank 26 | size = comm.size 27 | 28 | if rank == 0: 29 | print(f'mpi4py') 30 | print(f'{n=} {ntasks=} {size=}') 31 | 32 | # create a random set of data 33 | data = list() 34 | for i in range(ntasks): 35 | np.random.seed(i) 36 | b = np.random.rand(n, n) 37 | a = b.dot(b.T) 38 | data.append(a) 39 | else: 40 | data = None 41 | 42 | start = time.time() 43 | 44 | # bcast data from root MPI rank 45 | data = comm.bcast(data, root=0) 46 | 47 | results = [] 48 | 49 | # each mpi rank only processes a subset of the data 50 | # the range is offset by each process mpi rank 51 | # and uses a step equal to the size of the communicator 52 | # For example, with 2 MPI ranks and 5 data tasks 53 | # each MPI rank will process the following indices 54 | # rank 0: [0, 2, 4] 55 | # rank 1: [1, 3] 56 | for i in range(rank, ntasks, size): 57 | result = process_data(i, data[i]) 58 | results.append(result) 59 | 60 | # gather results to root MPI rank 61 | results = comm.gather(results, root=0) 62 | 63 | if rank == 0: 64 | end = time.time() 65 | elapsed = end - start 66 | rate = ntasks / elapsed 67 | print(f'Processed {ntasks} tasks in {elapsed:.2f}s ({rate:.2f}it/s)') 68 | 69 | # flatten list of lists 70 | results = [r for sub in results for r in sub] 71 | 72 | # check results 73 | if {r[0] for r in results} == set(range(ntasks)): 74 | print(f'completed all tasks!') 75 | else: 76 | print('uh oh, something went wrong.') 77 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/data-task-examples/data-multiprocessing.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | import itertools 5 | import multiprocessing 6 | import os 7 | 8 | import numpy as np 9 | from tqdm import tqdm 10 | 11 | 12 | def process_data(i, a): 13 | w, v = np.linalg.eigh(a) 14 | return i, w 15 | 16 | def _process_data_unpack(args): 17 | # helper function for unpacking args 18 | return process_data(*args) 19 | 20 | def main(): 21 | parser = argparse.ArgumentParser() 22 | parser.add_argument('-n', type=int, default=1000) 23 | parser.add_argument('--ntasks', type=int, default=128) 24 | parser.add_argument('--nproc', type=int, default=None) 25 | args = parser.parse_args() 26 | n = args.n 27 | ntasks = args.ntasks 28 | nproc = args.nproc 29 | 30 | # if nproc is not specified, use the number of physical cores 31 | if nproc is None: 32 | # sched_getaffinity returns number of logical cores 33 | # divide by two to get number of physical cores 34 | cpus = os.sched_getaffinity(0) 35 | nproc = len(cpus) // 2 36 | 37 | print('multiprocessing') 38 | print(f'{n=} {ntasks=} {nproc=}') 39 | 40 | # create a random set of data 41 | data = list() 42 | for i in range(ntasks): 43 | np.random.seed(i) 44 | b = np.random.rand(n, n) 45 | a = b.dot(b.T) 46 | data.append(a) 47 | 48 | results = [] 49 | # We advise using the "spawn" start method at NERSC 50 | # See the following page for more multiprocessing tips: 51 | # https://docs.nersc.gov/development/languages/python/parallel-python/#tips-for-using-multiprocessing-at-nersc 52 | multiprocessing.set_start_method("spawn") 53 | # create a pool of nproc workers 54 | with multiprocessing.Pool(processes=nproc) as pool: 55 | tasks_args = enumerate(data) 56 | # typically you would then use pool.starmap like so: 57 | # results = pool.starmap(process_data, tasks_args) 58 | # which would be a lot simpler. 59 | # Here I'm using pool.imap with a helper function to unpack task_args 60 | # in order to use a progress bar for visualization 61 | it = pool.imap(_process_data_unpack, tasks_args) 62 | for result in tqdm(it, total=ntasks, ncols=80): 63 | results.append(result) 64 | 65 | # check results 66 | if {r[0] for r in results} == set(range(ntasks)): 67 | print(f'completed all tasks!') 68 | else: 69 | print('uh oh, something went wrong.') 70 | 71 | if __name__ == "__main__": 72 | main() -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/data-task-examples/data-serial.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | 5 | import numpy as np 6 | from tqdm import tqdm 7 | 8 | def process_data(i, a): 9 | w, v = np.linalg.eigh(a) 10 | return i, w 11 | 12 | def main(): 13 | parser = argparse.ArgumentParser() 14 | parser.add_argument('-n', type=int, default=1000) 15 | parser.add_argument('--ntasks', type=int, default=128) 16 | args = parser.parse_args() 17 | n = args.n 18 | ntasks = args.ntasks 19 | 20 | print('serial') 21 | print(f'{n=} {ntasks=}') 22 | 23 | # create a random set of data 24 | data = list() 25 | for i in range(ntasks): 26 | np.random.seed(i) 27 | b = np.random.rand(n, n) 28 | a = b.dot(b.T) 29 | data.append(a) 30 | 31 | results = [] 32 | # process tasks 33 | for i in tqdm(range(ntasks), ncols=80): 34 | results.append(process_data(i, data[i])) 35 | 36 | # check results 37 | if {r[0] for r in results} == set(range(ntasks)): 38 | print(f'completed all tasks!') 39 | else: 40 | print('uh oh, something went wrong.') 41 | 42 | if __name__ == "__main__": 43 | main() 44 | -------------------------------------------------------------------------------- /meetings/Dec2022/using-perlmutter/data-task-examples/scan-num-threads.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | n=${1:-1000} 4 | setup="import numpy as np; n = $n; b = np.random.rand(n, n); a = b.dot(b.T)" 5 | benchmark="np.linalg.eigh(a)" 6 | for nthreads in 1 2 4 8 16 32 64 128 256; do \ 7 | echo -n "nthreads=$nthreads | n=$n | " 8 | export OMP_NUM_THREADS=$nthreads 9 | python -m timeit -s "$setup" "$benchmark" 10 | done 11 | -------------------------------------------------------------------------------- /meetings/Dec2023/NERSC/README.md: -------------------------------------------------------------------------------- 1 | # Exercises for How to Use NERSC Effectively Tutorial 2 | 3 | Stephen Bailey
4 | December 2023 Hawaii Collaboration Meeting 5 | 6 | See [DESI-8059](https://desi.lbl.gov/DocDB/cgi-bin/private/ShowDocument?docid=8059) for tutorial slides. 7 | 8 | Solutions for each exercise are in the solutions/ subdirectory, but 9 | you are highly encouraged to try solving the exercises yourself before 10 | looking at the solutions. 11 | 12 | ## Getting started 13 | 14 | On perlmutter.nersc.gov, setup a python environment that has 15 | numpy, scipy, and astropy, e.g. 16 | ``` 17 | source /global/cfs/cdirs/desi/desi_environment.sh main 18 | 19 | -or- 20 | 21 | source /global/common/software/desi/users/adematti/cosmodesi_environment.sh main 22 | ``` 23 | 24 | Get a copy of the DESI tutorials 25 | ``` 26 | mkdir -p $CFS/desi/users/$USER 27 | cd $CFS/desi/users/$USER 28 | git clone https://github.com/desihub/tutorials 29 | cd meetings/Dec2023/NERSC 30 | ``` 31 | 32 | Get an interactive node 33 | ``` 34 | salloc -N 1 -C cpu -t 02:00:00 -q interactive 35 | ``` 36 | 37 | ## Exercise 1: Adding parallelism 38 | 39 | `exercise1.py` is a non-parallelized code to process coadded spectra 40 | files. It reads the `R_FLUX` HDU, and subtracts a median-filtered 41 | continuum from each spectrum and prints the amount of read time + processing 42 | time. It doesn't write any output; this is just a toy example. e.g. 43 | ``` 44 | python exercise1.py /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-8-338-thru20210414.fits 45 | 46 | coadd-8-338-thru20210414.fits 0.5 + 0.6 sec 47 | Total time 1.1 sec 48 | ``` 49 | 50 | The file `coaddfiles.txt` has a list of 128 coadd files; 51 | processing these serially would take 1-2 minutes. 52 | ``` 53 | python exercise1.py $(cat coaddfiles.txt) 54 | ``` 55 | Note: The `$(cat coaddfiles.txt)` syntax just converts the contents of 56 | coaddfiles.txt into command line arguments. 57 | 58 | ### Exercise 1a: GNU parallel 59 | 60 | Use GNU parallel to run example1.py on all input files in coaddfiles.txt. 61 | 62 | Hint: See tutorial slide 8. 63 | 64 | This should run in 10-15 seconds for all files. 65 | 66 | ### Exercise 1b: multiprocessing 67 | 68 | Make a copy of exercise1.py and update it to use multiprocessing. 69 | This can process all files in 2-5 seconds. 70 | 71 | Hint: See tutorial slides 9-11. 72 | 73 | Bonus: did you try parallelizing over files or over spectra within a file? 74 | Try implementing the other form of parallelism. Which was faster? 75 | 76 | ### Exercise 1c: MPI 77 | 78 | Make a copy of exercise1.py and update it to use MPI. This would be run with an 79 | `srun` prefix like: 80 | ``` 81 | srun -n 128 -c 2 python exercise1_mpi.py $(cat coaddfiles.txt) 82 | ``` 83 | This can process all files in 2-5 seconds. 84 | 85 | Hint: See tutorial slides 14-15. 86 | 87 | ### Exercise 1 Discussion 88 | 89 | Which parallelism method ran fastest? Which did you find easiest to implement? 90 | 91 | ## Exercise 2: Tuning parallelism 92 | 93 | In this exercise we'll study the tradeoffs on different levels of parallelism, 94 | using MPI + multiprocessing. 95 | 96 | `exercise2.py` is similar to `exercise1.py`, but it uses MPI to parallelize 97 | over input files, and multiprocessing to parallelize over spectra within a file. 98 | It includes command line options to load the list of input files and how many 99 | multiprocessing processes to use. Run it like 100 | ``` 101 | srun -n 32 -c 8 python exercise2.py --filelist coaddfiles.txt --nproc 4 102 | ``` 103 | That will use 32 MPI ranks to process 32 files in parallel, and for each 104 | file it will use 4 multiprocessing processes to work on 4 spectra in parallel 105 | (out of 500 per file). We chose 32x4=128 to match the number of physical cores. 106 | 107 | Note 1: in this case multiprocessing over spectra has enough overhead that it 108 | would be faster to not parallelize that at all, but this is a toy example 109 | to study parallelism tradeoffs. 110 | 111 | Note 2: srun+starting python can have an overhead of up to a minute depending 112 | upon the load at NERSC, which gets in the way of timing toy examples like this. 113 | For this study, just use the "Total time" reported by the script itself, 114 | not the actual wallclock time. 115 | 116 | Try running this with different levels of MPI vs. multiprocessing parallelism 117 | while keeping the total number of cores constant, and -c twice as large as 118 | --nproc, e.g. 119 | ``` 120 | srun -n 8 -c 32 python exercise2.py --filelist coaddfiles.txt --nproc 16 121 | srun -n 16 -c 16 python exercise2.py --filelist coaddfiles.txt --nproc 8 122 | srun -n 32 -c 8 python exercise2.py --filelist coaddfiles.txt --nproc 4 123 | srun -n 64 -c 4 python exercise2.py --filelist coaddfiles.txt --nproc 2 124 | srun -n 128 -c 2 python exercise2.py --filelist coaddfiles.txt --nproc 1 125 | ``` 126 | 127 | Record the time for each run and compare. What combination was most effective? 128 | 129 | ### Exercise 2 Discussion 130 | 131 | You probably got something like 132 | ``` 133 | MPI x nproc Runtime 134 | ----------- ------- 135 | 8 x 16 550 136 | 16 x 8 256 137 | 32 x 4 124 138 | 64 x 2 65 139 | 128 x 1 28 140 | ``` 141 | 142 | This case was constructed to be especially "unfair" to multiprocessing, 143 | but it shows a >10x performance difference just by changing the balance 144 | of how the cores are used, without changing the code itself at all! 145 | 146 | ### Exercise 2 Bonus 1 - hyperthreading 147 | 148 | The Perlmutter CPUs have 2x hyperthreading, which allows you to run 2x more 149 | processes than there are physical cores (128). 150 | Try running `exercise2.py --nproc 1` which turns off multiprocessing, and 151 | compare performance when using 128 MPI ranks (the number of physical CPU cores) 152 | vs. 256 (with 2x hyperthreading). Did using hyperthreading help? 153 | 154 | ``` 155 | srun -n 128 -c 2 python exercise2.py --filelist coaddfiles.txt --nproc 1 156 | srun -n 256 -c 1 python exercise2.py --filelist coaddfiles.txt --nproc 1 157 | ``` 158 | 159 | ### Exercise 2 Bonus 2 - multiple nodes 160 | 161 | Log out of your current interactive session and get a new one with 2 nodes. 162 | We'll check whether using 2 nodes runs faster, and whether it runs enough 163 | faster to be worth being charged for 2 nodes instead of 1. 164 | 165 | ``` 166 | # get a new interactive session with 2 nodes 167 | salloc -N 2 -C cpu -t 01:00:00 -q interactive 168 | 169 | # run on 1 node 170 | srun -N 1 -n 128 -c 2 python exercise2.py --filelist coaddfiles.txt --nproc 1 171 | 172 | # run twice as many ranks, spread across 2 nodes 173 | srun -N 2 -n 256 -c 2 python exercise2.py --filelist coaddfiles.txt --nproc 1 174 | ``` 175 | 176 | *Discussion*: running on 2 nodes was probably faster, but probably not 2x faster 177 | to fully compensate for being charged for 2x more nodes. If it is close to 178 | the same number of node-hours charged, the faster wall-clock time my be worth 179 | your human time. But for larger runs, optimizing how many nodes should be 180 | used per job can save thousands of node-hours being charged. 181 | 182 | ## Exercise 3 183 | 184 | Apply these methods to your own code, either to add parallelism, or to study 185 | what parallelism parameters are optimal for your code before you do your 186 | next big run. 187 | 188 | If you add parallelism and it still doesn't run faster, something is wrong. 189 | Reach out for help and keep working on it before proceeding with big runs. 190 | 191 | If you are already parallelized and changing parallelism parameters doesn't 192 | have a big effect, that's ok. This is worth a few hours of testing with 193 | various parameters, but if you don't see big differences then it's ok to 194 | proceed with your big runs. Or maybe your will find a big speedup! 195 | 196 | 197 | 198 | -------------------------------------------------------------------------------- /meetings/Dec2023/NERSC/coaddfiles.txt: -------------------------------------------------------------------------------- 1 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-8-338-thru20210414.fits 2 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-0-338-thru20210414.fits 3 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-5-338-thru20210414.fits 4 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-4-338-thru20210414.fits 5 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-2-338-thru20210414.fits 6 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-6-338-thru20210414.fits 7 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-1-338-thru20210414.fits 8 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-7-338-thru20210414.fits 9 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-9-338-thru20210414.fits 10 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/338/20210414/coadd-3-338-thru20210414.fits 11 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-7-263-thru20210504.fits 12 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-0-263-thru20210504.fits 13 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-1-263-thru20210504.fits 14 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-2-263-thru20210504.fits 15 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-5-263-thru20210504.fits 16 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-9-263-thru20210504.fits 17 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-6-263-thru20210504.fits 18 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-8-263-thru20210504.fits 19 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-3-263-thru20210504.fits 20 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/263/20210504/coadd-4-263-thru20210504.fits 21 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-9-334-thru20210419.fits 22 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-7-334-thru20210419.fits 23 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-1-334-thru20210419.fits 24 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-2-334-thru20210419.fits 25 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-4-334-thru20210419.fits 26 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-6-334-thru20210419.fits 27 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-5-334-thru20210419.fits 28 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-0-334-thru20210419.fits 29 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-8-334-thru20210419.fits 30 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/334/20210419/coadd-3-334-thru20210419.fits 31 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-1-80675-thru20210201.fits 32 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-8-80675-thru20210201.fits 33 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-4-80675-thru20210201.fits 34 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-0-80675-thru20210201.fits 35 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-9-80675-thru20210201.fits 36 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-2-80675-thru20210201.fits 37 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-7-80675-thru20210201.fits 38 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-6-80675-thru20210201.fits 39 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80675/20210201/coadd-5-80675-thru20210201.fits 40 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-1-69-thru20210417.fits 41 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-0-69-thru20210417.fits 42 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-3-69-thru20210417.fits 43 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-9-69-thru20210417.fits 44 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-7-69-thru20210417.fits 45 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-4-69-thru20210417.fits 46 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-5-69-thru20210417.fits 47 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-8-69-thru20210417.fits 48 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-2-69-thru20210417.fits 49 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/69/20210417/coadd-6-69-thru20210417.fits 50 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-1-80647-thru20210320.fits 51 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-8-80647-thru20210320.fits 52 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-3-80647-thru20210320.fits 53 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-0-80647-thru20210320.fits 54 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-9-80647-thru20210320.fits 55 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-4-80647-thru20210320.fits 56 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-6-80647-thru20210320.fits 57 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-5-80647-thru20210320.fits 58 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-7-80647-thru20210320.fits 59 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80647/20210320/coadd-2-80647-thru20210320.fits 60 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-8-80635-thru20210105.fits 61 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-2-80635-thru20210105.fits 62 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-4-80635-thru20210105.fits 63 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-9-80635-thru20210105.fits 64 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-5-80635-thru20210105.fits 65 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-0-80635-thru20210105.fits 66 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-6-80635-thru20210105.fits 67 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-7-80635-thru20210105.fits 68 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80635/20210105/coadd-1-80635-thru20210105.fits 69 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-4-81022-thru20210404.fits 70 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-0-81022-thru20210404.fits 71 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-6-81022-thru20210404.fits 72 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-2-81022-thru20210404.fits 73 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-1-81022-thru20210404.fits 74 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-3-81022-thru20210404.fits 75 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-7-81022-thru20210404.fits 76 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-9-81022-thru20210404.fits 77 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-8-81022-thru20210404.fits 78 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/81022/20210404/coadd-5-81022-thru20210404.fits 79 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-9-326-thru20210407.fits 80 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-3-326-thru20210407.fits 81 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-2-326-thru20210407.fits 82 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-7-326-thru20210407.fits 83 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-1-326-thru20210407.fits 84 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-8-326-thru20210407.fits 85 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-0-326-thru20210407.fits 86 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-6-326-thru20210407.fits 87 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-5-326-thru20210407.fits 88 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/326/20210407/coadd-4-326-thru20210407.fits 89 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-7-34-thru20210504.fits 90 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-5-34-thru20210504.fits 91 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-0-34-thru20210504.fits 92 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-3-34-thru20210504.fits 93 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-9-34-thru20210504.fits 94 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-2-34-thru20210504.fits 95 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-6-34-thru20210504.fits 96 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-8-34-thru20210504.fits 97 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-1-34-thru20210504.fits 98 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/34/20210504/coadd-4-34-thru20210504.fits 99 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-6-251-thru20210418.fits 100 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-4-251-thru20210418.fits 101 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-8-251-thru20210418.fits 102 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-5-251-thru20210418.fits 103 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-1-251-thru20210418.fits 104 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-7-251-thru20210418.fits 105 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-3-251-thru20210418.fits 106 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-2-251-thru20210418.fits 107 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-0-251-thru20210418.fits 108 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/251/20210418/coadd-9-251-thru20210418.fits 109 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-6-368-thru20210419.fits 110 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-7-368-thru20210419.fits 111 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-2-368-thru20210419.fits 112 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-3-368-thru20210419.fits 113 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-8-368-thru20210419.fits 114 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-9-368-thru20210419.fits 115 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-0-368-thru20210419.fits 116 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-1-368-thru20210419.fits 117 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-4-368-thru20210419.fits 118 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/368/20210419/coadd-5-368-thru20210419.fits 119 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-9-439-thru20210419.fits 120 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-0-439-thru20210419.fits 121 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-6-439-thru20210419.fits 122 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-7-439-thru20210419.fits 123 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-5-439-thru20210419.fits 124 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-4-439-thru20210419.fits 125 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-3-439-thru20210419.fits 126 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-2-439-thru20210419.fits 127 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-8-439-thru20210419.fits 128 | /dvs_ro/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/439/20210419/coadd-1-439-thru20210419.fits 129 | -------------------------------------------------------------------------------- /meetings/Dec2023/NERSC/exercise1.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Subtract the continuum using a median-filter from spectra in files 5 | """ 6 | 7 | import os, sys, time 8 | import numpy as np 9 | from scipy.signal import medfilt 10 | from astropy.io import fits 11 | 12 | def subtract_continuum(filename): 13 | """ 14 | Subtract a median-filter continuum from each spectrum in flux 15 | 16 | Args: 17 | flux[nspec,nwave]: 2D input flux for nspec spectra x nwave wavelengths 18 | 19 | Returns: 20 | continuum-subtracted newflux[nspec,nwave] 21 | """ 22 | t0 = time.time() 23 | 24 | # Read R-band flux from filename 25 | flux = fits.getdata(filename, 'R_FLUX') 26 | t1 = time.time() 27 | 28 | # subtract median-filter continuum 29 | # Note: medfilt could do this as a single 2D operation much faster, 30 | # but we'll loop to mimic some work per spectrum 31 | newflux = np.zeros(flux.shape) 32 | for i in range(flux.shape[0]): 33 | newflux[i] = flux[i] - medfilt(flux[i], 15) 34 | 35 | # print timing just to show we're doing something 36 | t2 = time.time() 37 | 38 | basename = os.path.basename(filename) 39 | print(f'{basename} {t1-t0:.1f} + {t2-t1:.1f} sec') 40 | 41 | return newflux 42 | 43 | def main(): 44 | start_time = time.time() 45 | for filename in sys.argv[1:]: 46 | subtract_continuum(filename) 47 | 48 | total_time = time.time() - start_time 49 | print(f'Total time {total_time:.1f} sec') 50 | 51 | if __name__ == '__main__': 52 | main() 53 | -------------------------------------------------------------------------------- /meetings/Dec2023/NERSC/exercise2.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Subtract the continuum using a median-filter from spectra in files. 5 | 6 | MPI-parallelize over files, multiprocess parallelize over spectra 7 | within a file. 8 | """ 9 | 10 | import os, sys, time 11 | import multiprocessing 12 | import argparse 13 | import numpy as np 14 | from scipy.signal import medfilt 15 | from astropy.io import fits 16 | 17 | # wrapper function to call with multiprocessing.Pool.map 18 | # subtracts median-filter continuum from 1D spectrum 19 | def _sub1d(flux1d): 20 | return flux1d - medfilt(flux1d, 31) 21 | 22 | def subtract_continuum(filename, nproc): 23 | """ 24 | Subtract a median-filter continuum from each spectrum in flux 25 | 26 | Args: 27 | flux[nspec,nwave]: 2D input flux for nspec spectra x nwave wavelengths 28 | 29 | Returns: 30 | continuum-subtracted newflux[nspec,nwave] 31 | """ 32 | t0 = time.time() 33 | 34 | # Read R-band flux from filename 35 | flux = fits.getdata(filename, 'R_FLUX') 36 | t1 = time.time() 37 | 38 | # subtract median-filter continuum 39 | # Note: medfilt could do this as a single 2D operation much faster, 40 | # but we'll loop to mimic some work per spectrum 41 | if nproc > 1: 42 | with multiprocessing.Pool(nproc) as pool: 43 | newflux = pool.map(_sub1d, flux) 44 | 45 | newflux = np.vstack(newflux) # list of 1D arrays -> 2D array 46 | else: 47 | newflux = np.zeros(flux.shape) 48 | for i in range(flux.shape[0]): 49 | newflux[i] = _sub1d(flux[i]) 50 | 51 | # print timing just to show we're doing something 52 | t2 = time.time() 53 | basename = os.path.basename(filename) 54 | print(f'{basename} {t1-t0:.1f} + {t2-t1:.1f} sec using {nproc} processes') 55 | 56 | return newflux 57 | 58 | def main(): 59 | # IMPORTANT: when used with multiprocessing, mpi4py imports must be 60 | # inside a function, not at top level module (!?!) 61 | from mpi4py import MPI 62 | from mpi4py.futures import MPICommExecutor 63 | 64 | start_time = time.time() 65 | 66 | # argparse to get --nproc number of multiprocesses to use 67 | parser = argparse.ArgumentParser() 68 | parser.add_argument('--nproc', type=int, 69 | default=multiprocessing.cpu_count(), 70 | help='Number of multiprocessing processes to use') 71 | parser.add_argument('--filelist', type=str, required=True, 72 | help='Text file with list of input coadd files to process') 73 | 74 | args = parser.parse_args() 75 | 76 | # connect to MPI communicator to find out which rank this is 77 | comm = MPI.COMM_WORLD 78 | rank = comm.rank 79 | size = comm.size 80 | 81 | # required for MPI + multiprocessing to play nicely together 82 | multiprocessing.set_start_method('spawn') 83 | 84 | # Rank 0 create arglist of (filename, nproc) 85 | if rank == 0: 86 | arglist = list() 87 | for filename in open(args.filelist).readlines(): 88 | filename = filename.strip() # remove trailing newline 89 | arglist.append( (filename, args.nproc) ) 90 | else: 91 | arglist = None 92 | 93 | # MPI parallelize over files; only rank 0 needs arglist 94 | with MPICommExecutor(comm, root=0) as pool: 95 | newflux = pool.starmap(subtract_continuum, arglist) 96 | 97 | # with MPICommExecutor, only rank 0 gets the results 98 | if rank == 0: 99 | newflux = np.vstack(list(newflux)) # list of 1D arrays -> 2D array 100 | 101 | comm.barrier() # wait for all ranks to finish 102 | total_time = time.time() - start_time 103 | if rank == 0: 104 | print(f'Total time {total_time:.1f} sec') 105 | 106 | if __name__ == '__main__': 107 | main() 108 | -------------------------------------------------------------------------------- /meetings/Dec2023/NERSC/solutions/exercise1_gnuparallel.txt: -------------------------------------------------------------------------------- 1 | # to run exercise1.py using gnu parallel: 2 | 3 | module load parallel 4 | cat coaddfiles.txt | time parallel python exercise1.py {} 5 | -------------------------------------------------------------------------------- /meetings/Dec2023/NERSC/solutions/exercise1_mpi.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Subtract the continuum using a median-filter from spectra in files. 5 | 6 | MPI-parallelize over files 7 | """ 8 | 9 | import os, sys, time 10 | import argparse 11 | import numpy as np 12 | from scipy.signal import medfilt 13 | from astropy.io import fits 14 | 15 | from mpi4py import MPI 16 | from mpi4py.futures import MPICommExecutor 17 | 18 | def subtract_continuum(filename): 19 | """ 20 | Subtract a median-filter continuum from each spectrum in flux 21 | 22 | Args: 23 | flux[nspec,nwave]: 2D input flux for nspec spectra x nwave wavelengths 24 | 25 | Returns: 26 | continuum-subtracted newflux[nspec,nwave] 27 | """ 28 | t0 = time.time() 29 | 30 | # Read R-band flux from filename 31 | flux = fits.getdata(filename, 'R_FLUX') 32 | 33 | # subtract median-filter continuum 34 | # Note: medfilt could do this as a single 2D operation much faster, 35 | # but we'll loop to mimic some work per spectrum 36 | newflux = np.zeros(flux.shape) 37 | for i in range(flux.shape[0]): 38 | newflux[i] = flux[i] - medfilt(flux[i], 31) 39 | 40 | # print timing just to show we're doing something 41 | dt = time.time() - t0 42 | basename = os.path.basename(filename) 43 | print(f'{basename} {dt:.1f} sec') 44 | 45 | return newflux 46 | 47 | def main(): 48 | start_time = time.time() 49 | 50 | # connect to MPI communicator to find out which rank this is 51 | comm = MPI.COMM_WORLD 52 | rank = comm.rank 53 | size = comm.size 54 | 55 | # MPI parallelize over files 56 | filenames = sys.argv[1:] 57 | with MPICommExecutor(comm, root=0) as pool: 58 | newflux = pool.map(subtract_continuum, filenames) 59 | 60 | # with MPICommExecutor, only rank 0 gets the results 61 | if rank == 0: 62 | newflux = np.vstack(list(newflux)) # list of 1D arrays -> 2D array 63 | 64 | comm.barrier() # wait for all ranks to finish 65 | total_time = time.time() - start_time 66 | if rank == 0: 67 | print(f'Total time {total_time:.1f} sec') 68 | 69 | if __name__ == '__main__': 70 | main() 71 | -------------------------------------------------------------------------------- /meetings/Dec2023/NERSC/solutions/exercise1_multiproc_files.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Subtract the continuum using a median-filter from spectra in files 5 | 6 | Multiprocess parallelize over files. 7 | """ 8 | 9 | import os, sys, time 10 | import multiprocessing 11 | import numpy as np 12 | from scipy.signal import medfilt 13 | from astropy.io import fits 14 | 15 | def subtract_continuum(filename): 16 | """ 17 | Subtract a median-filter continuum from each spectrum in flux 18 | 19 | Args: 20 | flux[nspec,nwave]: 2D input flux for nspec spectra x nwave wavelengths 21 | 22 | Returns: 23 | continuum-subtracted newflux[nspec,nwave] 24 | """ 25 | t0 = time.time() 26 | 27 | # Read R-band flux from filename 28 | flux = fits.getdata(filename, 'R_FLUX') 29 | 30 | # subtract median-filter continuum 31 | # Note: medfilt could do this as a single 2D operation much faster, 32 | # but we'll loop to mimic some work per spectrum 33 | newflux = np.zeros(flux.shape) 34 | for i in range(flux.shape[0]): 35 | newflux[i] = flux[i] - medfilt(flux[i], 15) 36 | 37 | # print timing just to show we're doing something 38 | dt = time.time() - t0 39 | basename = os.path.basename(filename) 40 | print(f'{basename} {dt:.1f} sec') 41 | 42 | return newflux 43 | 44 | def main(): 45 | start_time = time.time() 46 | 47 | filenames = sys.argv[1:] 48 | with multiprocessing.Pool() as pool: 49 | newfluxes = pool.map(subtract_continuum, filenames) 50 | 51 | total_time = time.time() - start_time 52 | print(f'Total time {total_time:.1f} sec') 53 | 54 | if __name__ == '__main__': 55 | main() 56 | -------------------------------------------------------------------------------- /meetings/Dec2023/NERSC/solutions/exercise1_multiproc_spectra.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Subtract the continuum using a median-filter from spectra in files. 5 | 6 | Do each file serially, and multiprocess parallelize over spectra within 7 | each file. 8 | """ 9 | 10 | import os, sys, time 11 | import multiprocessing 12 | import numpy as np 13 | from scipy.signal import medfilt 14 | from astropy.io import fits 15 | 16 | # wrapper function to call with multiprocessing.Pool.map 17 | # subtracts median-filter continuum from 1D spectrum 18 | def _sub1d(flux1d): 19 | return flux1d - medfilt(flux1d, 15) 20 | 21 | def subtract_continuum(filename): 22 | """ 23 | Subtract a median-filter continuum from each spectrum in flux 24 | 25 | Args: 26 | flux[nspec,nwave]: 2D input flux for nspec spectra x nwave wavelengths 27 | 28 | Returns: 29 | continuum-subtracted newflux[nspec,nwave] 30 | """ 31 | t0 = time.time() 32 | 33 | # Read R-band flux from filename 34 | flux = fits.getdata(filename, 'R_FLUX') 35 | 36 | # subtract median-filter continuum 37 | # Note: medfilt could do this as a single 2D operation much faster, 38 | # but we'll loop to mimic some work per spectrum 39 | with multiprocessing.Pool() as pool: 40 | newflux = pool.map(_sub1d, flux) 41 | 42 | newflux = np.vstack(newflux) # list of 1D arrays -> 2D array 43 | 44 | # print timing just to show we're doing something 45 | dt = time.time() - t0 46 | basename = os.path.basename(filename) 47 | print(f'{basename} {dt:.1f} sec') 48 | 49 | return newflux 50 | 51 | def main(): 52 | start_time = time.time() 53 | for filename in sys.argv[1:]: 54 | subtract_continuum(filename) 55 | 56 | total_time = time.time() - start_time 57 | print(f'Total time {total_time:.1f} sec') 58 | 59 | if __name__ == '__main__': 60 | main() 61 | -------------------------------------------------------------------------------- /meetings/Dec2024/DataIntro/00_GettingStarted.md: -------------------------------------------------------------------------------- 1 | # Introduction to Year 3 Data and Using NERSC Effectively 2 | 3 | Stephen Bailey
4 | DESI Cancun Meeting December 2024
5 | ECS Tutorial 6 | 7 | *Parts of this tutorial are adapted from the [intro_to_DESI_EDR_files](https://github.com/desihub/tutorials/blob/main/getting_started/intro_to_DESI_EDR_files.ipynb) tutorial originally developed by 8 | Ragadeepika Pucha (U.Arizona), Anthony Kremin (Berkeley Lab), Stéphanie Juneau (NOIRLab), Jaime E. Forero-Romero (Uniandes), and DESI Data Team* 9 | 10 | ## Goals 11 | 12 | This tutorial covers the following topics covering a range of expertise 13 | 14 | | File | Description | 15 | |:------ |:---------- | 16 | | 01_CoreConcepts.ipynb | Basic terminology for undersanding how DESI data are organized. | 17 | | 02_DataOrganization.ipynb | Tour of what files are what in a DESI data production. | 18 | | 03_RedshiftCatalogs.ipynb | Using Redshift catalogs | 19 | | 04_Spectra.ipynb | Using DESI spectra | 20 | | 05_NERSC.md | Tips for using NERSC effectively | 21 | 22 |
23 | 24 | ## Code Setup 25 | 26 | All of these tutorials can be read at https://github.com/desihub/tutorials/tree/main/meetings/Dec2024/DataIntro . If you want to interact with them and explore some more, you can run the notebooks using https://jupyter.nersc.gov, but you'll need to setup the DESI kernels first so that your Jupyter notebook can find the DESI code. 27 | 28 | Open a terminal (either from Jupyter File -> New -> Terminal or `ssh perlmutter.nersc.gov`) and run the following commands 29 | 30 | ``` 31 | source /global/common/software/desi/desi_environment.sh 24.11 32 | ${DESIMODULES}/install_jupyter_kernel.sh 24.11 33 | ${DESIMODULES}/install_jupyter_kernel.sh main 34 | ``` 35 | 36 | See https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC for more details and debugging issues. 37 | 38 | From the same terminal, get the code 39 | ``` 40 | mkdir -p /global/cfs/cdirs/desi/users/$USER 41 | cd /global/cfs/cdirs/desi/users/$USER 42 | git clone https://github.com/desihub/tutorials 43 | ``` 44 | 45 | You can put the git clone wherever you want on disk, but we recommend putting it 46 | somewhere under `/global/cfs/cdirs/desi/users/$USER` instead of `$HOME` because then 47 | any plots or files you generate can be shared with your collaborators (they can't 48 | see your `$HOME` directory) and those files/plots can also be downloaded to your 49 | laptop using https://data.desi.lbl.gov/desi/users/ . 50 | 51 | ## Starting jupyter.nersc.gov 52 | 53 | Browse to https://jupyter.nersc.gov and login with your NERSC credentials. 54 | 55 | Start a Perlmutter Login Node server (leftmost option). 56 | 57 | On the lefthand panel "FILE BROWSER", click the folder icon to go back to the `/` root directory, then browse to `/global/cfs/cdirs/desi/users/(your_user_name)` or wherever you put the code. 58 | 59 | **Pro tip**: click the star icon to add this directory to your favorites to make it easier to browse there in the future. 60 | 61 | Navigate to `tutorials/meetings/Dec2024/DataIntro` and select notebooks and Markdown (md) files to view. -------------------------------------------------------------------------------- /meetings/Dec2024/DataIntro/01_CoreConcepts.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "7189769a-dc2c-4936-b2b4-f0ebdffff2f7", 6 | "metadata": {}, 7 | "source": [ 8 | "# Core Concepts" 9 | ] 10 | }, 11 | { 12 | "cell_type": "code", 13 | "execution_count": 1, 14 | "id": "5d020cfd-944a-4eed-bb5f-481a442045dc", 15 | "metadata": {}, 16 | "outputs": [ 17 | { 18 | "data": { 19 | "text/html": [ 20 | "\n", 21 | "\n" 22 | ], 23 | "text/plain": [ 24 | "" 25 | ] 26 | }, 27 | "metadata": {}, 28 | "output_type": "display_data" 29 | } 30 | ], 31 | "source": [ 32 | "%%html\n", 33 | "\n", 34 | "" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "id": "af395972-0c77-42b9-b52e-a23fd33b2647", 40 | "metadata": {}, 41 | "source": [ 42 | "## SURVEY and PROGRAM\n", 43 | "\n", 44 | "DESI observations are organized by\n", 45 | " * **SURVEY** = Phases of DESI observations with a common goal, e.g. \"main\" or \"sv1\"\n", 46 | " * **PROGRAM** = Subsets of SURVEYs split by observing conditions, e.g. \"dark\" or \"bright\"\n", 47 | "\n", 48 | "**Why this matters**: Data processing groups data by SURVEY and PROGRAM on disk,\n", 49 | "and does not combine data across SURVEY and PROGRAM even if it is the same\n", 50 | "object on the sky. This keeps the different scientific goals of DESI\n", 51 | "independent of each other. When analyzing DESI data, you need to know what\n", 52 | "SURVEY and PROGRAM you care about.\n", 53 | "\n", 54 | "Primary **SURVEY**s and **PROGRAMS** in DESI are\n", 55 | "\n", 56 | "| SURVEY | Purpose |\n", 57 | "| :---- | :--- |\n", 58 | "| Survey Validation 1 (sv1) | Tune cuts for target selection; extra high S/N data |\n", 59 | "| Survey Validation 3 (sv3) | Many overlapping observations to get all targets on a given patch of sky (\"highly complete\") |\n", 60 | "| Main (main) | The core cosmology program of DESI |\n", 61 | "\n", 62 | "| PROGRAM | Purpose |\n", 63 | "| :---- | :--- |\n", 64 | "| dark | Best observing conditions for faintest targets: ELG, LRG, QSO |\n", 65 | "| bright | Moon up / poor seeing / poor transparency: BGS, MWS |\n", 66 | "| backup | Very bad conditions: bright stars |\n", 67 | "\n" 68 | ] 69 | }, 70 | { 71 | "cell_type": "markdown", 72 | "id": "0faba066-bb9e-46dc-9e26-206cb741246b", 73 | "metadata": {}, 74 | "source": [ 75 | "## Tiles and Healpix" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "id": "23be5201-bcf4-4333-982d-fa9e426a1b36", 81 | "metadata": {}, 82 | "source": [ 83 | "A DESI \"tile\" is a specific pointing of the telescope and fiber positioners to\n", 84 | "observe a specific set of objects on the sky. Tiles are associated with a\n", 85 | "SURVEY and PROGRAM. Tiles are observed with one more more exposures on\n", 86 | "one more more nights until they have achieved a consistent signal-to-noise (S/N) goal\n", 87 | "set by the SURVEY+PROGRAM. Since a single tile cannot observe all the targets on\n", 88 | "a given patch of sky, the DESI tiles overlap so that if a given target is not observed\n", 89 | "on one tile, it gets another chance on a future overlapping tile.\n", 90 | "\n", 91 | "Some targets are observed on multiple tiles to get more S/N than they would get on\n", 92 | "a single tile, e.g. Lyman-alpha QSOs at z>2. In this case we want to coadd data\n", 93 | "across tiles. Some science studies also want all spectra in a single patch of sky\n", 94 | "and it would be a pain to have to look up and read N>>1 seperate tile files just to\n", 95 | "get those spectra. For these reasons, data processing also groups spectra by\n", 96 | "\"healpix\", which is a diamond-shaped tesselation of the sky. All spectra in a given (healpix, survey, program) are grouped together into files and coadded.\n", 97 | "\n", 98 | "**Why this matters**: If you want the highest S/N data combined across all observations, you want to use the healpix data. If you need to track performance vs. time or are working with\n", 99 | "custom observations on a special tile, you want the tile data.\n", 100 | "\n", 101 | "**Digging Deeper**: The [DESI_petal_healpix_rosette](https://github.com/desihub/tutorials/blob/main/getting_started/DESI_petal_healpix_rosette.ipynb) tutorial explores these ideas in more detail including reading and plotting targets grouped by tiles vs. healpix." 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "id": "e8a5f986-8b02-4acd-a6ac-78391b9295ac", 107 | "metadata": {}, 108 | "source": [ 109 | "## Petals, Spectrographs, Cameras, and Fibers\n", 110 | "\n", 111 | "The DESI focal plane is divided into 10 separate \"petals\".\n", 112 | "Each petal has 500 fibers which map to a single \"spectrograph\".\n", 113 | "The petal number [0-9] is the same as the spectrograph number [0-9]\n", 114 | "and in practice these are used interchangeably.\n", 115 | "The `10*500=5000` DESI fibers are mapped to the spectrographs such that\n", 116 | "```\n", 117 | "PETAL = SPECTROGRAPH = FIBER//500\n", 118 | "```\n", 119 | "\n", 120 | "Each spectrograph has 3 \"cameras\" which split the light by blue (b), red (r), and near-infrared (z) bands. These cameras are named by the band+spectrograph, e.g. \"b0\", \"r1\", \"z9\".\n", 121 | "\n", 122 | "\n", 123 | "**Caveat**: if you get involved in hardware operations, there is a different numbering\n", 124 | "scheme for the hardware spectrographs `smN` developed while they were being manufactured,\n", 125 | "before they were plugged in to the petals. Most people do not need to know about this\n", 126 | "distinction." 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "id": "d169df64-af15-4857-ad98-9f26f3ef74b9", 132 | "metadata": {}, 133 | "source": [ 134 | "## Spectra and Catalogs" 135 | ] 136 | }, 137 | { 138 | "cell_type": "markdown", 139 | "id": "a86390e1-574a-4660-ab83-5f8b809a1c66", 140 | "metadata": {}, 141 | "source": [ 142 | "The core DESI data are *spectra*, i.e. flux vs. wavelength.\n", 143 | "When we measure quantities from spectra like the redshift or the flux in emission lines,\n", 144 | "these measurements can be grouped in to tables in *catalogs*. Many analyses can be\n", 145 | "performed on catalogs generated by others without ever needing to read the much larger spectra files." 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "id": "aa994b31-749d-4ba0-8281-c243dbc0da68", 151 | "metadata": {}, 152 | "source": [ 153 | "## Mountains and spectroscopic productions" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "id": "39ed2f77-9c8f-41ab-bed2-061f1844660b", 159 | "metadata": {}, 160 | "source": [ 161 | "DESI data processing runs are named after mountains, alphabetically increasing with time.\n", 162 | "A given mountain or \"spectroscopic production run\" (specprod) represents a self-consistent processing of the data with a set of code tags.\n", 163 | "\n", 164 | "Productions are located at NERSC under\n", 165 | "```\n", 166 | "/global/cfs/cdirs/desi/spectro/redux/$SPECPROD\n", 167 | "```\n", 168 | "\n", 169 | "It is good practice for all of your scripts and notebooks to set the production directory once at the very top instead of hardcoding e.g. \"kibo\" many places. This makes it easier to switch from one production to a newer one e.g. \"loa\". It's even better to reference this to an environment variable $DESI_ROOT (=/global/cfs/cdirs/desi at NERSC) so that you can copy a subset of the data to your laptop or home institution and still have the same scripts work without a bunch of NERSC-specific hardcoded paths.\n", 170 | "\n", 171 | "For example:" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": 5, 177 | "id": "bb80d5a1-7d11-42c9-b6fe-b4ba3a622185", 178 | "metadata": {}, 179 | "outputs": [ 180 | { 181 | "name": "stdout", 182 | "output_type": "stream", 183 | "text": [ 184 | "Using data in /global/cfs/cdirs/desi/spectro/redux/loa\n" 185 | ] 186 | } 187 | ], 188 | "source": [ 189 | "import os\n", 190 | "specprod = 'loa'\n", 191 | "desi_root = os.environ['DESI_ROOT']\n", 192 | "datadir = f'{desi_root}/spectro/redux/{specprod}'\n", 193 | "print(f'Using data in {datadir}')" 194 | ] 195 | }, 196 | { 197 | "cell_type": "markdown", 198 | "id": "815bf3c5-8f3e-4901-8d72-264c1d6f07e5", 199 | "metadata": {}, 200 | "source": [ 201 | "If you follow that pattern in all of your notebooks and scripts, it will be much easier to re-run your\n", 202 | "analysis on future productions." 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": null, 208 | "id": "c36c6c6e-d6b7-4f2d-94f9-894fa65e9391", 209 | "metadata": {}, 210 | "outputs": [], 211 | "source": [] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": null, 216 | "id": "c4accf96-efc2-4b2c-880a-95bceb2a0244", 217 | "metadata": {}, 218 | "outputs": [], 219 | "source": [] 220 | } 221 | ], 222 | "metadata": { 223 | "kernelspec": { 224 | "display_name": "DESI 24.11", 225 | "language": "python", 226 | "name": "desi-24.11" 227 | }, 228 | "language_info": { 229 | "codemirror_mode": { 230 | "name": "ipython", 231 | "version": 3 232 | }, 233 | "file_extension": ".py", 234 | "mimetype": "text/x-python", 235 | "name": "python", 236 | "nbconvert_exporter": "python", 237 | "pygments_lexer": "ipython3", 238 | "version": "3.10.14" 239 | } 240 | }, 241 | "nbformat": 4, 242 | "nbformat_minor": 5 243 | } 244 | -------------------------------------------------------------------------------- /meetings/Dec2024/DataIntro/05_NERSC.md: -------------------------------------------------------------------------------- 1 | # Using NERSC Effectively 2 | 3 | With apologies, I (Stephen) didn't finish this part of the tutorial. 4 | Please see tutorials from previous meetings: 5 | 6 | * [DESI-8059](https://desi.lbl.gov/DocDB/cgi-bin/private/ShowDocument?docid=8059), Stephen @ Hawaii Dec 2023 7 | * [DESI-8379](https://desi.lbl.gov/DocDB/cgi-bin/private/ShowDocument?docid=8379), Anthony @ Marseilles July 2024 -------------------------------------------------------------------------------- /meetings/June2021/README.md: -------------------------------------------------------------------------------- 1 | # Tutorials for DESI ECS June Pre-meeting 2 | 3 | - `git-intro.ipynb` notebook contains a (short) tutorial for Git and GitHub, presenting 4 | - an overview of some of the concepts, 5 | - a short exercise for getting a sense of some of the git commands, and 6 | - further resources and exercises. 7 | 8 | - `nersc_computing` folder contains a notebook (`computing_at_nersc.ipynb`) with a tutorial for computing at NERSC including: 9 | - introduction to nersc, 10 | - module management, 11 | - environment creation, 12 | - working with Jupyter at NERSC, 13 | - adding kernels to Jupyter, and 14 | - submitting jobs 15 | 16 | - `Intro_to_a_DESI_Release.ipynb` notebook contains a tutorial on accessing and working with data from a DESI data release: 17 | - file locations and environment variables 18 | - Summary files 19 | - Directory structure of redshifts 20 | - Opening and viewing co-added spectra 21 | - Using desispec tools to view spectra 22 | - Viewing redshift information 23 | - Combining redshift and spectral information 24 | 25 | This folder also contains auxiliary files that will be helpful during the tutorial. 26 | -------------------------------------------------------------------------------- /meetings/June2021/images/threecommits.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/meetings/June2021/images/threecommits.png -------------------------------------------------------------------------------- /meetings/June2021/nersc_computing/example_batch.batch: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | #SBATCH -A desi 3 | #SBATCH -N 1 4 | #SBATCH -t 00:10:00 5 | #SBATCH -C haswell 6 | #SBATCH -q debug 7 | 8 | srun python test_script.py -------------------------------------------------------------------------------- /meetings/June2021/nersc_computing/kernel.json: -------------------------------------------------------------------------------- 1 | { 2 | "argv": [ 3 | "/global/cfs/cdirs/desi/users/hiramk/meetings/June2021/Tutorials/nersc_computing/myenv.sh", 4 | "-m", 5 | "ipykernel_launcher", 6 | "-f", 7 | "{connection_file}" 8 | ], 9 | "display_name": "MyEnv", 10 | "language": "python" 11 | } 12 | -------------------------------------------------------------------------------- /meetings/June2021/nersc_computing/load_myenv.sh: -------------------------------------------------------------------------------- 1 | module load python 2 | source activate myenv -------------------------------------------------------------------------------- /meetings/June2021/nersc_computing/myenv.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -l 2 | module load python 3 | source activate myenv 4 | # You can add whatever you want in here module loads, definitions, etc. works as the other example of shell file. 5 | 6 | # This last line is important and should always be the last one! 7 | exec python -m ipykernel_launcher "$@" 8 | 9 | 10 | -------------------------------------------------------------------------------- /meetings/June2021/nersc_computing/test_script.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import time,sys 3 | 4 | # This script emullates a job that takes around 30 seconds to run. 5 | print("Hello! Welcome to the tutorial.") 6 | print("I'm currently working :)") 7 | for i in range(5): 8 | print(f"Iteration {i} of 5"), 9 | sys.stdout.flush() 10 | time.sleep(6) 11 | print("Thanks for your patience!") 12 | print("done") -------------------------------------------------------------------------------- /meetings/June2022/README.md: -------------------------------------------------------------------------------- 1 | # Tutorials from June 2022 Collaboration meeting 2 | -------------------------------------------------------------------------------- /mpl/desi.mplstyle: -------------------------------------------------------------------------------- 1 | # uses the definitions from #https://github.com/desihub/tutorials/blob/main/Dec2021/Everest_Tutorial_Dec2021.ipynb 2 | 3 | font.size : 22 4 | axes.linewidth : 3.0 5 | xtick.major.size : 6.0 6 | xtick.minor.size : 4.0 7 | xtick.major.width : 3.0 8 | xtick.minor.width : 1.5 9 | xtick.direction : in 10 | xtick.minor.visible : True 11 | xtick.top : True 12 | ytick.major.size : 6.0 13 | ytick.minor.size : 4.0 14 | ytick.major.width : 3.0 15 | ytick.minor.width : 1.5 16 | ytick.direction : in 17 | ytick.minor.visible : True 18 | ytick.right : True -------------------------------------------------------------------------------- /onskydata/spectro_nz.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# n(z) histograms from Spectroscopic Reduction" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "This tutorial is a worked example of how to make an n(z) histogram\n", 15 | "for a production run of the spectroscopic pipeline. It shows you how the per-tile per-night per-spectrograph\n", 16 | "\"zbest\" redshift files are organized, and gives you some details about working\n", 17 | "with commissioning (CMX) and survey validation (SV) target bits and \"fibermaps\"." 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": null, 23 | "metadata": {}, 24 | "outputs": [], 25 | "source": [ 26 | "#- Basic imports\n", 27 | "import sys, os, glob\n", 28 | "import numpy as np\n", 29 | "import fitsio\n", 30 | "from astropy.table import Table\n", 31 | "\n", 32 | "from desitarget.cmx.cmx_targetmask import cmx_mask\n", 33 | "from desitarget.sv1.sv1_targetmask import desi_mask\n", 34 | "\n", 35 | "import matplotlib.pyplot as plt" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "## Spectro pipeline production directories\n", 43 | "\n", 44 | "Official DESI spectro pipeline runs are located at NERSC under `/global/cfs/cdirs/desi/spectro/redux`,\n", 45 | "grouped by a \"specprod\" name. For example:\n", 46 | "* \"daily\" : the daily spectroscopic reductions after the data transfer from KPNO.\n", 47 | "* \"andes\" : production run from May 2020.\n", 48 | "* \"blanc\" : production run from Jan 2021.\n", 49 | "* ...\n", 50 | "\n", 51 | "Let's use the \"blanc\" run. If you want to browse the outputs on the web, see\n", 52 | "https://data.desi.lbl.gov/desi/spectro/redux/blanc . Search the DESI wiki for \"collaboration username\" (in quotes) to find\n", 53 | "the authentication info." 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "specprod = 'blanc'\n", 63 | "specprod_dir = os.path.join(os.environ['DESI_SPECTRO_REDUX'], specprod)" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "In andes, we did not make a combined redshift catalog across all tiles and all nights (sorry...).\n", 71 | "Instead, we have to load the individual \"zbest\" files for each tile. These are grouped by\n", 72 | "tiles/TILEID/YEARMMDD/zbest-SPECTROGRAPH-TILEID-NIGHT.fits.\n", 73 | "See https://desi.lbl.gov/trac/wiki/TargetSelectionWG/SV0 for human-friendly summary of what was observed when." 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": null, 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [ 82 | "zbfiles = sorted(glob.glob(specprod_dir+'/tiles/*/20*/zbest*.fits'))\n", 83 | "print('{} zbest files found'.format(len(zbfiles)))" 84 | ] 85 | }, 86 | { 87 | "cell_type": "code", 88 | "execution_count": null, 89 | "metadata": {}, 90 | "outputs": [], 91 | "source": [ 92 | "#- Print a few of them as an example\n", 93 | "zbfiles[0::50]" 94 | ] 95 | }, 96 | { 97 | "cell_type": "markdown", 98 | "metadata": {}, 99 | "source": [ 100 | "Even a relatively small production like andes has a lot of data, which motivates why you should\n", 101 | "get used to using NERSC (e.g. via Jupyter) instead of always starting by downloading the data\n", 102 | "locally." 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "metadata": {}, 109 | "outputs": [], 110 | "source": [ 111 | "!du -hs $specprod_dir/tiles" 112 | ] 113 | }, 114 | { 115 | "cell_type": "markdown", 116 | "metadata": {}, 117 | "source": [ 118 | "## Reading the zbest files\n", 119 | "\n", 120 | "Now let's loop over all the zbest files, load their redshifts table (ZBEST), plus their \"FIBERMAP\" table that gives\n", 121 | "further information about each target, and accumulate their good redshifts.\n", 122 | "\n", 123 | "### Target selection bitmasks\n", 124 | "\n", 125 | "Target selection bitmasks identify which targets are which.\n", 126 | "\n", 127 | "Since we have a combination of commissioning (CMX) and early survey validation (SV0) tiles,\n", 128 | "as well as the post-COVID restart of survey validation (SV1), we need to adjust the bitmasks\n", 129 | "depending on the input.\n", 130 | "\n", 131 | "For more details on working with targeting bits, see\n", 132 | "https://github.com/desihub/desitarget/blob/master/doc/nb/target-selection-bits-and-bitmasks.ipynb" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "metadata": {}, 139 | "outputs": [], 140 | "source": [ 141 | "#- redshift lists that we will extend\n", 142 | "zbgs = list()\n", 143 | "zlrg = list()\n", 144 | "zelg = list()\n", 145 | "zqso = list()\n", 146 | "nstar = 0 #- Sorry, MWS, we're just going to count stars\n", 147 | "\n", 148 | "#- Since a TARGETID could be observed more than once, keep track of ones that we've already\n", 149 | "#- seen before, and only take the first redshift that we find\n", 150 | "previous_targetids = list()\n", 151 | "\n", 152 | "num_zbfiles = len(zbfiles)\n", 153 | "for i, filename in enumerate(zbfiles):\n", 154 | " if i%100 == 0:\n", 155 | " print(f'{i}/{num_zbfiles}')\n", 156 | "\n", 157 | " #- The ZBEST HDU contains the redshift fits\n", 158 | " zb = fitsio.read(filename, 'ZBEST')\n", 159 | " \n", 160 | " #- The FIBERMAP HDU contains information about which targets are assigned to which positioners\n", 161 | " fm = fitsio.read(filename, 'FIBERMAP')\n", 162 | "\n", 163 | " #- ZBEST has one entry per target, while FIBERMAP has one entry per target per exposure\n", 164 | " #- Get one CMX_TARGET entry per target, from the fibermap\n", 165 | " targetid, ii = np.unique(fm['TARGETID'], return_index=True)\n", 166 | " assert np.all(zb['TARGETID'] == fm['TARGETID'][ii])\n", 167 | " \n", 168 | " # Choose CMX/SV0 or SV1 bitmasks on the fly.\n", 169 | " if 'SV1_DESI_TARGET' in fm.dtype.names:\n", 170 | " targetcol = 'SV1_DESI_TARGET'\n", 171 | " bgsMask = desi_mask.mask('BGS_ANY')\n", 172 | " lrgMask = desi_mask.mask('LRG')\n", 173 | " elgMask = desi_mask.mask('ELG')\n", 174 | " qsoMask = desi_mask.mask('QSO')\n", 175 | " elif 'CMX_TARGET' in fm.dtype.names:\n", 176 | " targetcol = 'CMX_TARGET'\n", 177 | " bgsMask = cmx_mask.mask('SV0_BGS')\n", 178 | " lrgMask = cmx_mask.mask('MINI_SV_LRG|SV0_LRG')\n", 179 | " elgMask = cmx_mask.mask('MINI_SV_ELG|SV0_ELG')\n", 180 | " qsoMask = cmx_mask.mask('MINI_SV_QSO|SV0_QSO')\n", 181 | " \n", 182 | " desi_target = fm[targetcol][ii]\n", 183 | "\n", 184 | " #- ZWARN==0 means no warnings means redrock thinks it's a good redshift\n", 185 | " #- identify ZWARN==0 per spectral classification\n", 186 | " isGal = (zb['SPECTYPE'] == 'GALAXY') & (zb['ZWARN'] == 0)\n", 187 | " isQSO = (zb['SPECTYPE'] == 'QSO') & (zb['ZWARN'] == 0)\n", 188 | " isStar = (zb['SPECTYPE'] == 'STAR') & (zb['ZWARN'] == 0)\n", 189 | " \n", 190 | " #- Good targets that we haven't seen in a previous zbest file (e.g. from an earlier night)\n", 191 | " isNew = np.in1d(targetid, list(previous_targetids), invert=True) & (zb['ZWARN'] == 0)\n", 192 | "\n", 193 | " #- Count stars\n", 194 | " nstar += np.count_nonzero(isStar & isNew)\n", 195 | "\n", 196 | " #- collect redshifts\n", 197 | " isBGS = isNew & isGal & ((desi_target & bgsMask) != 0)\n", 198 | " isLRG = isNew & isGal & ((desi_target & lrgMask) != 0)\n", 199 | " isELG = isNew & isGal & ((desi_target & elgMask) != 0)\n", 200 | " isQSO = isNew & isQSO & ((desi_target & qsoMask) != 0)\n", 201 | "\n", 202 | " zbgs.append(zb['Z'][isBGS])\n", 203 | " zlrg.append(zb['Z'][isLRG])\n", 204 | " zelg.append(zb['Z'][isELG])\n", 205 | " zqso.append(zb['Z'][isQSO])\n", 206 | "\n", 207 | " #- Keep track of ones we've seen before\n", 208 | " previous_targetids.extend(targetid[isNew])\n", 209 | "\n", 210 | "#- convert those into 1D arrays of redshifts per target class\n", 211 | "zbgs = np.concatenate(zbgs)\n", 212 | "zlrg = np.concatenate(zlrg)\n", 213 | "zelg = np.concatenate(zelg)\n", 214 | "zqso = np.concatenate(zqso)" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": null, 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "print('# Star:', nstar)\n", 224 | "print('# BGS: ', len(zbgs))\n", 225 | "print('# LRG: ', len(zlrg))\n", 226 | "print('# ELG: ', len(zelg))\n", 227 | "print('# QSO: ', len(zqso))" 228 | ] 229 | }, 230 | { 231 | "cell_type": "markdown", 232 | "metadata": {}, 233 | "source": [ 234 | "## Let's make some plots" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "metadata": {}, 241 | "outputs": [], 242 | "source": [ 243 | "plt.figure(figsize=(8,6))\n", 244 | "plt.subplot(411)\n", 245 | "n = plt.hist(zbgs, 35, (0, 3.5), color='C4')[0]\n", 246 | "plt.text(3.6, int(0.8*np.max(n)), f'{len(zbgs)} BGS', ha='right')\n", 247 | "plt.title(f'{specprod} n(z)')\n", 248 | "\n", 249 | "plt.subplot(412)\n", 250 | "n = plt.hist(zlrg, 35, (0, 3.5), color='C3')[0]\n", 251 | "plt.text(3.6, int(0.8*np.max(n)), f'{len(zlrg)} LRG', ha='right')\n", 252 | "\n", 253 | "plt.subplot(413)\n", 254 | "n = plt.hist(zelg, 35, (0, 3.5), color='C2')[0]\n", 255 | "plt.text(3.6, int(0.8*np.max(n)), f'{len(zelg)} ELG', ha='right')\n", 256 | "\n", 257 | "plt.subplot(414)\n", 258 | "n = plt.hist(zqso, 35, (0, 3.5), color='C0')[0]\n", 259 | "plt.text(3.6, int(0.8*np.max(n)), f'{len(zqso)} QSO', ha='right')\n", 260 | "\n", 261 | "plt.xlabel('redshift')\n", 262 | "plt.tight_layout()" 263 | ] 264 | }, 265 | { 266 | "cell_type": "markdown", 267 | "metadata": {}, 268 | "source": [ 269 | "## Exercises\n", 270 | " * make a radial velocity histogram for stars instead of just counting them\n", 271 | " * explore the contents of the FIBERMAP and the ZBEST HDUs to make plots like\n", 272 | " * fraction of ZWARN==0 (good) vs. FLUX_R\n", 273 | " * g-r color vs. redshift\n", 274 | " * histogram number of exposures per target" 275 | ] 276 | } 277 | ], 278 | "metadata": { 279 | "kernelspec": { 280 | "display_name": "DESI 22.2", 281 | "language": "python", 282 | "name": "desi-22.2" 283 | }, 284 | "language_info": { 285 | "codemirror_mode": { 286 | "name": "ipython", 287 | "version": 3 288 | }, 289 | "file_extension": ".py", 290 | "mimetype": "text/x-python", 291 | "name": "python", 292 | "nbconvert_exporter": "python", 293 | "pygments_lexer": "ipython3", 294 | "version": "3.9.7" 295 | } 296 | }, 297 | "nbformat": 4, 298 | "nbformat_minor": 4 299 | } 300 | -------------------------------------------------------------------------------- /py/desitutorials.py: -------------------------------------------------------------------------------- 1 | #- Utility code that is specific to DESI tutorials, 2 | #- but also useful across multiple tutorials. 3 | 4 | from __future__ import absolute_import, division, print_function 5 | 6 | _standard_repos = [ 7 | 'numpy', 'scipy', 'astropy', 'yaml', 'matplotlib', 8 | 'requests', 'fitsio', 'h5py', 'healpy', 'psycopg2', 9 | 'desiutil', 'desispec', 'desitarget', 'desimodel', 'desisim', 10 | 'desisurvey', 'surveysim', 11 | 'redrock', 'specter', 'speclite', 'specsim', 12 | ] 13 | 14 | def get_code_versions(module_names=None): 15 | ''' 16 | Returns a dict of ver[package] = code_version for loaded DESI code 17 | 18 | Note: similar to desiutil.depend.add_dependencies 19 | ''' 20 | import sys 21 | import importlib 22 | import collections 23 | 24 | if module_names is None: 25 | module_names = _standard_repos 26 | 27 | versions = collections.OrderedDict() 28 | versions['python'] = ".".join(map(str, sys.version_info[0:3])) 29 | for module in module_names: 30 | if module in sys.modules: 31 | # already loaded, but we need a reference to the module object 32 | x = importlib.import_module(module) 33 | if hasattr(x, '__version__'): 34 | versions[module] = x.__version__ 35 | elif hasattr(x, '__path__'): 36 | # e.g. redmonster doesn't set __version__ 37 | versions[module] = 'unknown ({})'.format(x.__path__[0]) 38 | elif hasattr(x, '__file__'): 39 | versions[module] = 'unknown ({})'.format(x.__file__) 40 | else: 41 | versions[module] = 'unknown' 42 | 43 | return versions 44 | 45 | def print_code_versions(module_names=None): 46 | ''' 47 | print code versions for loaded desi code 48 | ''' 49 | for module, version in get_code_versions(module_names).items(): 50 | print('{:12s}: {}'.format(module, version)) 51 | -------------------------------------------------------------------------------- /redrock/RedrockNthBestFit.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "e652900c-badd-49a1-910e-97b06afef568", 6 | "metadata": {}, 7 | "source": [ 8 | "# Retrieving the Nth best redrock fit\n", 9 | "\n", 10 | "This tutorial describes how to look up the Nth best redrock fit coefficients. \n", 11 | "\n", 12 | "See https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC to configure Jupyter at NERSC to be able to run this\n", 13 | "tutorial using jupyter.nersc.gov and the fuji data in `/global/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80618/20210328/`. This notebook runs on kernel 22.2. \n", 14 | "\n", 15 | "The bottom of this tutorial lists the code versions used while testing this.\n", 16 | "\n", 17 | "Raffaella Capasso
\n", 18 | "July 2022" 19 | ] 20 | }, 21 | { 22 | "cell_type": "markdown", 23 | "id": "4b7f4b4c-46d7-422f-a32e-73af07cd2188", 24 | "metadata": {}, 25 | "source": [ 26 | "## Basic imports" 27 | ] 28 | }, 29 | { 30 | "cell_type": "code", 31 | "execution_count": null, 32 | "id": "38a0010d-e4a1-4e1b-80fe-911728110d67", 33 | "metadata": {}, 34 | "outputs": [], 35 | "source": [ 36 | "import os\n", 37 | "from astropy.table import Table\n", 38 | "import numpy as np\n", 39 | "import desispec.io\n", 40 | "\n", 41 | "import matplotlib.pyplot as plt\n", 42 | "import redrock.templates\n", 43 | "import h5py\n", 44 | "import sys" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "id": "fe574d42-22e0-4ff6-a18c-d2052671cf4b", 50 | "metadata": {}, 51 | "source": [ 52 | "Add the tutorials directory to the python search path so that we can import desitutorials" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": null, 58 | "id": "204cb49d-ef87-4cef-8c0f-8a645df9b6b5", 59 | "metadata": {}, 60 | "outputs": [], 61 | "source": [ 62 | "sys.path.append(os.path.abspath(os.getcwd()+'/../py/'))\n", 63 | "\n", 64 | "import desitutorials" 65 | ] 66 | }, 67 | { 68 | "cell_type": "markdown", 69 | "id": "6786e793-0aa2-47d7-a61b-078a711e27e2", 70 | "metadata": {}, 71 | "source": [ 72 | "Move into the directory that contains the reference run data and set some environment variables so that desispec can find the files. If you are running locally (*e.g.* on your laptop instead of at jupyter.nersc.gov, replace these with wherever you installed the reference run data." 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "id": "242df01f-d1ea-4261-b978-cbc09d7f2bac", 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [ 82 | "os.chdir('/global/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80618/20210328/')" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "id": "d126dd94-0cb0-4a42-b86f-e249919b0ee7", 88 | "metadata": {}, 89 | "source": [ 90 | "Import redrock templates" 91 | ] 92 | }, 93 | { 94 | "cell_type": "code", 95 | "execution_count": null, 96 | "id": "4bccc8e2-d767-4baf-9b72-3695e36881af", 97 | "metadata": {}, 98 | "outputs": [], 99 | "source": [ 100 | "templates = dict()\n", 101 | "for filename in redrock.templates.find_templates():\n", 102 | " # print('test')\n", 103 | " t = redrock.templates.Template(filename) #check if you can add a \"verbose=False\"\n", 104 | " templates[(t.template_type, t.sub_type)] = t\n" 105 | ] 106 | }, 107 | { 108 | "cell_type": "markdown", 109 | "id": "fbc02bc3-d82d-47ed-8b27-9c230827f834", 110 | "metadata": {}, 111 | "source": [ 112 | "Read in the spectra and the matching zbest file. Also read the fibermap and trim to a unique set that row-matches the zbest file." 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": null, 118 | "id": "01999738-bf29-4e10-b720-e1894304037e", 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "zbest = Table.read('redrock-0-80618-thru20210328.fits', hdu=1).filled('')\n", 123 | "spectra = desispec.io.read_spectra('coadd-0-80618-thru20210328.fits')\n", 124 | "\n", 125 | "fm = Table.read('redrock-0-80618-thru20210328.fits', hdu='FIBERMAP')\n", 126 | "ii=sorted(np.unique(fm['TARGETID'], return_index=True)[1])\n", 127 | "assert np.all(fm['TARGETID'] == zbest['TARGETID'])\n", 128 | "fm = fm[ii]" 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": null, 134 | "id": "b98fd807-d997-4bb1-bc77-cad56324e62c", 135 | "metadata": {}, 136 | "outputs": [], 137 | "source": [ 138 | "goodELGs = (zbest['SPECTYPE'] == 'GALAXY') \n", 139 | "goodELGs &= (zbest['ZWARN'] == 0) & (zbest['DELTACHI2']>5)\n", 140 | "goodELGs &= (fm['OBJTYPE'] == 'TGT')" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "id": "4587074e-9269-45a3-b432-c1e4a5362ec3", 147 | "metadata": {}, 148 | "outputs": [], 149 | "source": [ 150 | "i = np.where(goodELGs)[0][0]\n", 151 | "z = zbest['Z'][i]\n", 152 | "targetid = zbest['TARGETID'][i]\n", 153 | "spectype = zbest['SPECTYPE'][i].strip()\n", 154 | "subtype = zbest['SUBTYPE'][i].strip()\n", 155 | "fulltype = (spectype, subtype)\n", 156 | "ncoeff = templates[fulltype].flux.shape[0]\n", 157 | "coeff = zbest['COEFF'][i][0:ncoeff]\n", 158 | "\n", 159 | "tflux = templates[fulltype].flux.T.dot(coeff)\n", 160 | "twave = templates[fulltype].wave * (1+z)\n", 161 | "\n" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": null, 167 | "id": "7426c244-f68c-4777-a67f-b0d3992cf27c", 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "print(z,targetid)" 172 | ] 173 | }, 174 | { 175 | "cell_type": "code", 176 | "execution_count": null, 177 | "id": "c0acdf22-5983-4a65-a6b7-849a984dea37", 178 | "metadata": {}, 179 | "outputs": [], 180 | "source": [ 181 | "ispec = np.where(spectra.fibermap['TARGETID'] == targetid)[0][0]\n", 182 | "\n", 183 | "plt.figure(figsize=(8,5))\n", 184 | "plt.subplot(211)\n", 185 | "maxflux = 0.0\n", 186 | "for band in spectra.bands:\n", 187 | " plt.plot(spectra.wave[band], spectra.flux[band][ispec], 'k-', alpha=0.5)\n", 188 | " maxflux = max(maxflux, np.max(spectra.flux[band][ispec]))\n", 189 | "\n", 190 | "plt.plot(twave, tflux, 'r-')\n", 191 | "plt.xlim(3500, 10000)\n", 192 | "# ylim(-1, 0.9*maxflux)\n", 193 | "plt.ylim(-1, 8)\n", 194 | "\n", 195 | "#- zoom in on [OII]\n", 196 | "plt.subplot(212)\n", 197 | "for band in spectra.bands:\n", 198 | " plt.plot(spectra.wave[band], spectra.flux[band][ispec], 'k-', alpha=0.5)\n", 199 | "\n", 200 | "plt.plot(twave, tflux, 'r-')\n", 201 | "plt.xlim(3727*(1+z)-100, 3727*(1+z)+100)\n", 202 | "plt.ylim(-1,8)" 203 | ] 204 | }, 205 | { 206 | "cell_type": "markdown", 207 | "id": "d0d1abdb-fcca-421b-b91f-5fd89bef4c97", 208 | "metadata": {}, 209 | "source": [ 210 | "## Look at the best 2nd fit" 211 | ] 212 | }, 213 | { 214 | "cell_type": "code", 215 | "execution_count": null, 216 | "id": "edd2eca8-46b1-4fae-a8a7-37943effef61", 217 | "metadata": {}, 218 | "outputs": [], 219 | "source": [ 220 | "filename = 'rrdetails-0-80618-thru20210328.h5'\n", 221 | "\n", 222 | "data = h5py.File(filename, mode='r')\n", 223 | "zfit = Table(np.array(data['zfit'][str(targetid)]['zfit']))\n", 224 | "zfit['znum', 'z', 'spectype', 'zwarn', 'chi2', 'deltachi2']" 225 | ] 226 | }, 227 | { 228 | "cell_type": "code", 229 | "execution_count": null, 230 | "id": "8909ba23-d830-4836-bf50-c59171eb2579", 231 | "metadata": {}, 232 | "outputs": [], 233 | "source": [ 234 | "coeff_best=zfit['coeff'][0]\n", 235 | "coeff_2nd=zfit['coeff'][1]\n", 236 | "z_best=zfit['z'][0]\n", 237 | "z_2nd=zfit['z'][1]\n", 238 | "\n", 239 | "targetids = np.array(data['targetids'])\n", 240 | "z_table = np.array(data['zscan']['GALAXY']['redshifts']) #scanned z\n", 241 | "zchi2 = np.array(data['zscan']['GALAXY']['zchi2']) #500 targets per z\n", 242 | "chi2_penalty = np.array(data['zscan']['GALAXY']['penalty'])\n", 243 | "\n", 244 | "tflux_best = templates[fulltype].flux.T.dot(coeff_best)\n", 245 | "twave_best = templates[fulltype].wave * (1+z_best)\n", 246 | "\n", 247 | "tflux_2nd = templates[fulltype].flux.T.dot(coeff_2nd)\n", 248 | "twave_2nd = templates[fulltype].wave * (1+z_2nd)" 249 | ] 250 | }, 251 | { 252 | "cell_type": "code", 253 | "execution_count": null, 254 | "id": "87584b20-d1fa-4f0d-99c1-feb0c67174a6", 255 | "metadata": {}, 256 | "outputs": [], 257 | "source": [ 258 | "plt.figure(figsize=(8,5))\n", 259 | "\n", 260 | "ymin=-1.\n", 261 | "ymax=8.\n", 262 | "\n", 263 | "plt.subplot(211)\n", 264 | "\n", 265 | "for band in spectra.bands:\n", 266 | " plt.plot(spectra.wave[band], spectra.flux[band][ispec], 'k-', alpha=0.5)\n", 267 | "\n", 268 | "# plot(wave, flux, 'k-', alpha=0.5)\n", 269 | "plt.plot(twave, tflux, 'r-')\n", 270 | "plt.xlim(3500, 10000)\n", 271 | "plt.ylim(ymin, ymax)\n", 272 | "\n", 273 | "\n", 274 | "#- zoom in on [OII]\n", 275 | "plt.subplot(212)\n", 276 | "for band in spectra.bands:\n", 277 | " plt.plot(spectra.wave[band], spectra.flux[band][ispec], 'k-', alpha=0.5)\n", 278 | "\n", 279 | "# plot(wave, flux, 'k-', alpha=0.5)\n", 280 | "plt.plot(twave_best, tflux_best, 'r-')\n", 281 | "plt.plot(twave_2nd, tflux_2nd, 'b-')\n", 282 | "plt.xlim(3727*(1+z)-100, 3727*(1+z)+100)\n", 283 | "plt.ylim(ymin, ymax)" 284 | ] 285 | }, 286 | { 287 | "cell_type": "markdown", 288 | "id": "dbca34dc-91ba-4103-a565-e974aa47e625", 289 | "metadata": {}, 290 | "source": [ 291 | "## Code versions" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "id": "8d963b94-b393-42cb-86ca-f3cfea5af3df", 298 | "metadata": {}, 299 | "outputs": [], 300 | "source": [ 301 | "print('Code versions used for this tutorial:')\n", 302 | "desitutorials.print_code_versions()" 303 | ] 304 | } 305 | ], 306 | "metadata": { 307 | "kernelspec": { 308 | "display_name": "DESI 22.2", 309 | "language": "python", 310 | "name": "desi-22.2" 311 | }, 312 | "language_info": { 313 | "codemirror_mode": { 314 | "name": "ipython", 315 | "version": 3 316 | }, 317 | "file_extension": ".py", 318 | "mimetype": "text/x-python", 319 | "name": "python", 320 | "nbconvert_exporter": "python", 321 | "pygments_lexer": "ipython3", 322 | "version": "3.9.7" 323 | } 324 | }, 325 | "nbformat": 4, 326 | "nbformat_minor": 5 327 | } 328 | -------------------------------------------------------------------------------- /redrock/RedrockOutputs.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "tags": [] 7 | }, 8 | "source": [ 9 | "# Understanding redrock output\n", 10 | "\n", 11 | "This tutorial describes how to combine the zbest fit coefficients with the\n", 12 | "redrock templates to see how the best fit template compares to the actual data.\n", 13 | "\n", 14 | "See https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC to configure Jupyter at NERSC to be able to run this\n", 15 | "tutorial using jupyter.nersc.gov and the fuji data in `/global/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80618/20210328/`. This notebook runs on kernel 22.2. \n", 16 | "\n", 17 | "If running locally, see https://desi.lbl.gov/trac/wiki/Pipeline/GettingStarted/Laptop for installing code and download the the fuji reference run data set from NERSC at `/global/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80618/20210328/`.\n", 18 | "\n", 19 | "The bottom of this tutorial lists the code versions used while testing this.\n", 20 | "\n", 21 | "Stephen Bailey
\n", 22 | "December 2020 (last mofified in July 2022)" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "## Basic setup" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": {}, 36 | "outputs": [], 37 | "source": [ 38 | "import os\n", 39 | "from astropy.table import Table\n", 40 | "import numpy as np\n", 41 | "import desispec.io\n", 42 | "import sys\n", 43 | "import matplotlib.pyplot as plt\n", 44 | "# import redrock.io" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "Add the tutorials directory to the python search path so that we can import desitutorials" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": {}, 58 | "outputs": [], 59 | "source": [ 60 | "sys.path.append(os.path.abspath(os.getcwd()+'/../py/'))\n", 61 | "import desitutorials" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": {}, 67 | "source": [ 68 | "For convenience, move into the directory that contains the reference run data and set some environment variables so that desispec can find the files. If you are running locally (*e.g.* on your laptop instead of at jupyter.nersc.gov, replace these with wherever you installed the reference run data." 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": {}, 75 | "outputs": [], 76 | "source": [ 77 | "os.chdir('/global/cfs/cdirs/desi/spectro/redux/fuji/tiles/cumulative/80618/20210328/')\n", 78 | "# os.environ['DESI_SPECTRO_REDUX'] = os.path.join(os.getcwd(), 'spectro', 'redux')\n", 79 | "# os.environ['SPECPROD'] = 'mini'" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "## Read in some spectra and the matching zbest file" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": null, 92 | "metadata": {}, 93 | "outputs": [], 94 | "source": [ 95 | "# specfile = desispec.io.findfile('spectra', groupname='5299', nside=64)\n", 96 | "# zbestfile = desispec.io.findfile('zbest', groupname='5299', nside=64)\n", 97 | "zbest = Table.read('redrock-0-80618-thru20210328.fits', hdu=1).filled('')\n", 98 | "spectra = desispec.io.read_spectra('coadd-0-80618-thru20210328.fits')" 99 | ] 100 | }, 101 | { 102 | "cell_type": "markdown", 103 | "metadata": {}, 104 | "source": [ 105 | "Also read the fibermap and trim to a unique set that row-matches the zbest file;\n", 106 | "we'll use this later to pick targets targeted as ELGs." 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "metadata": {}, 113 | "outputs": [], 114 | "source": [ 115 | "fm = Table.read('redrock-0-80618-thru20210328.fits', hdu='FIBERMAP')\n", 116 | "ii=sorted(np.unique(fm['TARGETID'], return_index=True)[1])\n", 117 | "assert np.all(fm['TARGETID'] == zbest['TARGETID'])\n", 118 | "fm = fm[ii]" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "What did we get?" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "metadata": {}, 132 | "outputs": [], 133 | "source": [ 134 | "zbest['SPECTYPE', 'Z'][0:5]" 135 | ] 136 | }, 137 | { 138 | "cell_type": "code", 139 | "execution_count": null, 140 | "metadata": {}, 141 | "outputs": [], 142 | "source": [ 143 | "for spectype in set(zbest['SPECTYPE']):\n", 144 | " ii = (zbest['SPECTYPE'] == spectype)\n", 145 | " plt.hist(zbest['Z'][ii], 41, (-0.1,4), label=spectype.strip())\n", 146 | "plt.legend()" 147 | ] 148 | }, 149 | { 150 | "cell_type": "markdown", 151 | "metadata": {}, 152 | "source": [ 153 | "Note that redrock does not distinguish between ELGs and LRGs: both are just `spectype=='GALAXY'`." 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "## Compare redrock templates to spectra\n", 161 | "\n", 162 | "`redrock.io.read_templates` returns a list of templates, then convert those into a dict of template type -> Template object. Ignore INFO messages about using default values for redshift ranges." 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": null, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "# tx = redrock.io.read_templates()\n", 172 | "import redrock.templates\n", 173 | "\n", 174 | "templates = dict()\n", 175 | "for filename in redrock.templates.find_templates():\n", 176 | " t = redrock.templates.Template(filename)\n", 177 | " templates[(t.template_type, t.sub_type)] = t" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "Pick an ELG target with confident redshift and use its zbest coefficients to construct the redrock template fit for that\n", 185 | "object." 186 | ] 187 | }, 188 | { 189 | "cell_type": "code", 190 | "execution_count": null, 191 | "metadata": {}, 192 | "outputs": [], 193 | "source": [ 194 | "from desitarget.targets import desi_mask" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": null, 200 | "metadata": {}, 201 | "outputs": [], 202 | "source": [ 203 | "goodELGs = (zbest['SPECTYPE'] == 'GALAXY') & (0.01 < zbest['Z']) & (zbest['Z'] < 1.0)\n", 204 | "goodELGs &= (zbest['ZWARN'] == 0) & (zbest['DELTACHI2']>5)\n", 205 | "# goodELGs &= ((fm['DESI_TARGET'] & desi_mask.ELG) != 0)" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [ 214 | "i = np.where(goodELGs)[0][0]\n", 215 | "z = zbest['Z'][i]\n", 216 | "targetid = zbest['TARGETID'][i]\n", 217 | "spectype = zbest['SPECTYPE'][i].strip()\n", 218 | "subtype = zbest['SUBTYPE'][i].strip()\n", 219 | "fulltype = (spectype, subtype)\n", 220 | "ncoeff = templates[fulltype].flux.shape[0]\n", 221 | "coeff = zbest['COEFF'][i][0:ncoeff]" 222 | ] 223 | }, 224 | { 225 | "cell_type": "code", 226 | "execution_count": null, 227 | "metadata": {}, 228 | "outputs": [], 229 | "source": [ 230 | "tflux = templates[fulltype].flux.T.dot(coeff)\n", 231 | "twave = templates[fulltype].wave * (1+z)" 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": {}, 238 | "outputs": [], 239 | "source": [ 240 | "ispec = np.where(spectra.fibermap['TARGETID'] == targetid)[0][0]\n", 241 | "\n", 242 | "plt.figure(figsize=(8,5))\n", 243 | "plt.subplot(211)\n", 244 | "maxflux = 0.0\n", 245 | "for band in spectra.bands:\n", 246 | " plt.plot(spectra.wave[band], spectra.flux[band][ispec], 'k-', alpha=0.5)\n", 247 | " maxflux = max(maxflux, np.max(spectra.flux[band][ispec]))\n", 248 | "\n", 249 | "plt.plot(twave, tflux, 'r-')\n", 250 | "plt.xlim(3500, 10000)\n", 251 | "# ylim(-1, 0.9*maxflux)\n", 252 | "plt.ylim(-1, 8)\n", 253 | "\n", 254 | "#- zoom in on [OII]\n", 255 | "plt.subplot(212)\n", 256 | "for band in spectra.bands:\n", 257 | " plt.plot(spectra.wave[band], spectra.flux[band][ispec], 'k-', alpha=0.5)\n", 258 | "\n", 259 | "plt.plot(twave, tflux, 'r-')\n", 260 | "plt.xlim(3727*(1+z)-100, 3727*(1+z)+100)\n", 261 | "plt.ylim(-1,8)" 262 | ] 263 | }, 264 | { 265 | "cell_type": "markdown", 266 | "metadata": {}, 267 | "source": [ 268 | "That wasn't quite correct because the redrock templates are high resolution\n", 269 | "before any instrumental resolution is applied. Let's update that to include\n", 270 | "the spectral resolution." 271 | ] 272 | }, 273 | { 274 | "cell_type": "code", 275 | "execution_count": null, 276 | "metadata": {}, 277 | "outputs": [], 278 | "source": [ 279 | "from desispec.interpolation import resample_flux\n", 280 | "from desispec.resolution import Resolution\n", 281 | "R = Resolution(spectra.resolution_data['b'][ispec])\n", 282 | "txflux = R.dot(resample_flux(spectra.wave['b'], twave, tflux))" 283 | ] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": null, 288 | "metadata": {}, 289 | "outputs": [], 290 | "source": [ 291 | "plt.plot(spectra.wave['b'], spectra.flux['b'][ispec])\n", 292 | "plt.plot(spectra.wave['b'], txflux)\n", 293 | "plt.ylim(0,6)\n", 294 | "plt.xlim(3727*(1+z)-100, 3727*(1+z)+100)" 295 | ] 296 | }, 297 | { 298 | "cell_type": "markdown", 299 | "metadata": {}, 300 | "source": [ 301 | "## Future work\n", 302 | "\n", 303 | "**TODO**: document redrock rr*.h5 files, e.g. the chi2 vs. redshift and how to get the second, third, etc. best fits" 304 | ] 305 | }, 306 | { 307 | "cell_type": "markdown", 308 | "metadata": {}, 309 | "source": [ 310 | "## Code versions" 311 | ] 312 | }, 313 | { 314 | "cell_type": "code", 315 | "execution_count": null, 316 | "metadata": {}, 317 | "outputs": [], 318 | "source": [ 319 | "print('Code versions used for this tutorial:')\n", 320 | "desitutorials.print_code_versions()" 321 | ] 322 | } 323 | ], 324 | "metadata": { 325 | "anaconda-cloud": {}, 326 | "kernelspec": { 327 | "display_name": "DESI 22.2", 328 | "language": "python", 329 | "name": "desi-22.2" 330 | }, 331 | "language_info": { 332 | "codemirror_mode": { 333 | "name": "ipython", 334 | "version": 3 335 | }, 336 | "file_extension": ".py", 337 | "mimetype": "text/x-python", 338 | "name": "python", 339 | "nbconvert_exporter": "python", 340 | "pygments_lexer": "ipython3", 341 | "version": "3.9.7" 342 | } 343 | }, 344 | "nbformat": 4, 345 | "nbformat_minor": 4 346 | } 347 | -------------------------------------------------------------------------------- /redrock/RedrockPlotSpec.md: -------------------------------------------------------------------------------- 1 | # Redrock plotspec 2 | 3 | This page documents how to run `rrplot` viewer to get plots like: 4 | 5 | ![redrock.PlotSpec screenshot](rrplotspec.png) 6 | 7 | It allows you to explore the chi2 vs. redshift scans and the N best fits 8 | for each spectrum. 9 | 10 | ## Installation 11 | 12 | Install the redrock + desispec code into an isolated conda environment: 13 | ``` 14 | conda create -n rrdesi python=3 numpy scipy astropy numba ipython h5py matplotlib pyyaml 15 | pip install speclite 16 | source activate rrdesi 17 | 18 | git clone https://github.com/desihub/redrock-templates 19 | git clone https://github.com/desihub/redrock 20 | git clone https://github.com/desihub/desiutil 21 | git clone https://github.com/desihub/desispec 22 | 23 | for repo in desiutil desispec redrock; do 24 | cd $repo 25 | python setup.py install 26 | cd .. 27 | done 28 | 29 | export RR_TEMPLATE_DIR=$(pwd)/redrock-templates 30 | ``` 31 | 32 | ## Get example data 33 | 34 | Available from NERSC at 35 | ``` 36 | /project/projectdirs/desi/datachallenge/dc17a-twopct/dc17a-lite.tar.gz 37 | ``` 38 | 39 | ## Run rrplot 40 | 41 | First go to the top level directory where you have those files. 42 | 43 | It might work for you to run `rrplot` from the command line, e.g. 44 | ``` 45 | specdir=spectro/redux/dc17a2/spectra-64/172/17242/ 46 | rrplot --specfile $specdir/spectra* --rrfile $specdir/r*.h5 47 | ``` 48 | 49 | For some people, this flashes a matplotlib window and immediately exits. 50 | In that case, you need to spawn it from within an `ipython --pylab` session: 51 | 52 | ``` 53 | #- Find where you installed rrplot 54 | import os 55 | rrplot = None 56 | for p in os.environ['PATH'].split(':'): 57 | rrplot = os.path.join(p, 'rrplot') 58 | if os.path.exists(rrplot): 59 | break 60 | 61 | if rrplot is None: 62 | print('ERROR: unable to find rrplot in your $PATH') 63 | else: 64 | print('Using '+rrplot) 65 | 66 | #- Input files 67 | specfile = 'spectro/redux/dc17a2/spectra-64/172/17242/spectra-64-17242.fits' 68 | rrfile = 'spectro/redux/dc17a2/spectra-64/172/17242/rr-64-17242.h5' 69 | 70 | #- Now actually run it 71 | %run $rrplot --specfile $specfile --rrfile $rrfile 72 | ``` 73 | 74 | -------------------------------------------------------------------------------- /redrock/rrplotspec.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/desihub/tutorials/3c97d1d31d0dd3eada3a6eab700ae0130243da51/redrock/rrplotspec.png -------------------------------------------------------------------------------- /simulations/README.md: -------------------------------------------------------------------------------- 1 | # simulations 2 | 3 | This directory is retained as a placeholder. If you are looking for `redshift-database.ipynb`, see `database/spectroscopic-production-database.ipynb`. 4 | -------------------------------------------------------------------------------- /testing/auto_test.html: -------------------------------------------------------------------------------- 1 | Fri 18 Nov 2022 11:11:32 AM PST 2 |

3 | desi-22.5 4 |

5 |
../archive/dc17a-truth.ipynb
6 | "An Exception was encountered at 'In [1]'." 7 |

8 |
../archive/GFA_targets.ipynb
9 | "An Exception was encountered at 'In [2]'." 10 | " raise Exception(\"Please enter an integer between 0 and 9\")\n", 11 |

12 |
../archive/Intro_to_DESI_spectra.ipynb
13 | "An Exception was encountered at 'In [8]'." 14 |

15 |
../archive/quickgen.ipynb
16 | "An Exception was encountered at 'In [2]'." 17 | "File \u001b[0;32m/global/common/software/desi/perlmutter/desiconda/20220119-2.0.1/conda/lib/python3.9/site-packages/cupy/cuda/__init__.py:84\u001b[0m, in \u001b[0;36mis_available\u001b[0;34m()\u001b[0m\n\u001b[1;32m 82\u001b[0m _available \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[1;32m 83\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 84\u001b[0m _available \u001b[38;5;241m=\u001b[39m \u001b[43mruntime\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgetDeviceCount\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m0\u001b[39m\n\u001b[1;32m 85\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 86\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (\u001b[38;5;129;01mnot\u001b[39;00m runtime\u001b[38;5;241m.\u001b[39mis_hip \u001b[38;5;129;01mand\u001b[39;00m e\u001b[38;5;241m.\u001b[39margs[\u001b[38;5;241m0\u001b[39m] \u001b[38;5;241m!=\u001b[39m\n\u001b[1;32m 87\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcudaErrorNoDevice: no CUDA-capable device is detected\u001b[39m\u001b[38;5;124m'\u001b[39m):\n", 18 |

19 |
../archive/simulating-desi-spectra.ipynb
20 | "An Exception was encountered at 'In [14]'." 21 |

22 |
../archive/survey-simulations.ipynb
23 | "An Exception was encountered at 'In [11]'." 24 |

25 |
../fiberassign/FiberAssignAlgorithms_Part1.ipynb
26 | "An Exception was encountered at 'In [1]'." 27 |

28 |
../fiberassign/FiberAssignAlgorithms_Part2.ipynb
29 | "An Exception was encountered at 'In [1]'." 30 |

31 |
../fiberassign/FiberAssignDECaLS.ipynb
32 | "An Exception was encountered at 'In [6]'." 33 |

34 |
../fiberassign/FiberAssign.ipynb
35 |

36 |
../fiberassign/FiberAssignMocks.ipynb
37 | "An Exception was encountered at 'In [2]'." 38 |

39 |
../fiberassign/restrict_reach.ipynb
40 | "An Exception was encountered at 'In [1]'." 41 |

42 |
../onskydata/EDR_Tutorial.ipynb
43 |

44 |
../onskydata/Intro_to_DESI_SV_spectra.ipynb
45 | "An Exception was encountered at 'In [39]'." 46 |

47 |
../onskydata/spectro_completeness.ipynb
48 |

49 |
../onskydata/spectro_nz.ipynb
50 |

51 |
../redrock/RedrockBOSSDemo.ipynb
52 |

53 |
../redrock/RedrockNthBestFit.ipynb
54 |

55 |
../redrock/RedrockOutputs.ipynb
56 |

57 |
../simulations/redshift-database.ipynb
58 |

59 |
../meetings/Dec2020/LSS.ipynb
60 | "An Exception was encountered at 'In [4]'." 61 |

62 |
../meetings/Dec2021/Everest_Tutorial_Dec2021.ipynb
63 |

64 |
../meetings/Dec2021/SpectroperfectionismExample.ipynb
65 |

66 |
../meetings/June2021/git-intro.ipynb
67 | "An Exception was encountered at 'In [24]'." 68 |

69 |
../meetings/June2021/Intro_to_a_DESI_Release.ipynb
70 | -------------------------------------------------------------------------------- /testing/readme.txt: -------------------------------------------------------------------------------- 1 | The script will run papermill in whatever desi kernel you wish on all jupyter notebooks with fall within one or two directories of the main tutorial directories. It will produce an output html file called auto_test.html which will be in the 'testing' directory. This will contain a list of the notebooks and the first input line at which they failed. 2 | 3 | To run the automated test use: 4 | 5 | ./test_papermill.sh kernel 6 | e.g. 7 | ./test_papermill.sh desi-main 8 | ./test_papermill.sh desi-22.5 9 | 10 | If you don't know what kernels you have avalible use: 11 | jupyter kernelspec list 12 | 13 | To install follow intructions here: 14 | https://desi.lbl.gov/trac/wiki/Computing/JupyterAtNERSC 15 | 16 | for help/issues with this script contact becky.canning@port.ac.uk 17 | 18 | -------------------------------------------------------------------------------- /testing/test_papermill.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # EDR DOC sprint, 17th Nov 2022 4 | # (e-mail to becky.canning@port.ac.uk) 5 | # 6 | # Running the tutorials with papermill 7 | # 8 | 9 | date > auto_test.html 10 | 11 | # read in the kernal to use 12 | kernel=$1 13 | echo "

" >> auto_test.html 14 | echo $kernel >> auto_test.html 15 | 16 | # make list of the notebooks 17 | # uncomment t run for all 18 | ls ../*/*.ipynb > list_notebooks.tmp 19 | ls ../*/*/*.ipynb >> list_notebooks.tmp 20 | 21 | while read line; 22 | do 23 | echo "testing $line" 24 | 25 | echo "papermill $line test_auto_output.ipynb -k $kernel" | sh 26 | echo "

" >> auto_test.html 27 | echo "
"$line"
" >> auto_test.html 28 | grep Exception test_auto_output.ipynb >> auto_test.html 29 | 30 | done < list_notebooks.tmp 31 | 32 | sed 's///g' auto_test.html | sed 's/<\/a>//g' > auto_test_2.html 33 | 34 | # clean up 35 | rm auto_test.html 36 | rm list_notebooks.tmp 37 | rm test_auto_output.ipynb 38 | mv auto_test_2.html auto_test.html 39 | 40 | 41 | --------------------------------------------------------------------------------