├── .gitignore ├── LICENSE ├── README.md ├── conda_start.txt ├── des_data_corr.ipynb ├── des_data_goodpixels.ipynb ├── des_data_intro.ipynb ├── mdet_meas_tools.py ├── shape_measurement_101.ipynb └── shape_measurement_102.ipynb /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 95 | __pypackages__/ 96 | 97 | # Celery stuff 98 | celerybeat-schedule 99 | celerybeat.pid 100 | 101 | # SageMath parsed files 102 | *.sage.py 103 | 104 | # Environments 105 | .env 106 | .venv 107 | env/ 108 | venv/ 109 | ENV/ 110 | env.bak/ 111 | venv.bak/ 112 | 113 | # Spyder project settings 114 | .spyderproject 115 | .spyproject 116 | 117 | # Rope project settings 118 | .ropeproject 119 | 120 | # mkdocs documentation 121 | /site 122 | 123 | # mypy 124 | .mypy_cache/ 125 | .dmypy.json 126 | dmypy.json 127 | 128 | # Pyre type checker 129 | .pyre/ 130 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2021, Matthew R. Becker 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # shape-measurement-tutorials 2 | 3 | tutorials on shape measurement, DES data, and other shape measurement-adjacent subjects 4 | -------------------------------------------------------------------------------- /conda_start.txt: -------------------------------------------------------------------------------- 1 | How to install conda 2 | 3 | 1. install xcode tools 4 | 5 | run the following in the terminal 6 | 7 | xcode-select —install 8 | 9 | it will bring up a window, click "install" to proceed 10 | 11 | 2. Download and install miniforge 12 | 13 | run the following commands in the terminal, one at a time 14 | 15 | curl -fsSLo Miniforge3.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-$(uname -m).sh 16 | bash Miniforge3.sh 17 | 18 | The second command will ask you several questions. Type "yes" when asked and otherwise accept the defaults. 19 | After these commands are done, close your terminal window and restart it. When you do, you should see "(base)" somewhere 20 | in the text before the space where you can write. (This text is called the command prompt.) 21 | 22 | 3. Make your first conda environment! A conda environment is a sandboxed environment where you can install packages/code in an 23 | isolated fashion. You can have more than one environment locally. Any time you want to use an environment, it needs to be 24 | activated. 25 | 26 | Here is some example code to show you how it is done (the stuff after "#"s are comments, no need to run those) 27 | 28 | conda create -n my-new-env-name python=3.8 jupyter notebook matplotlib scipy galsim numpy ipython nb_conda_kernels 29 | # ^ this command runs for a while 30 | # then you type 31 | conda activate my-new-env-name 32 | # finally you can do things like start a notebook like this 33 | jupyter notebook 34 | # ^this command will print out a web address and launch jupyter in your browser 35 | # from the main jupyter screen, you can then launch a notebook by going to the new dropdown and selecting a kernel (the same as your conda env) 36 | # you can also reopen notebooks you have already saved 37 | 38 | 39 | 40 | -------------------------------------------------------------------------------- /des_data_corr.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "13a6ff05", 6 | "metadata": {}, 7 | "source": [ 8 | "# Measuring Background Pixel Correlations\n", 9 | "\n", 10 | "Let's build on the last tutorial and measure the correlation properties of the background pixels we've found." 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 2, 16 | "id": "6041f640", 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import os\n", 21 | "import fitsio\n", 22 | "import yaml\n", 23 | "import esutil\n", 24 | "import numpy as np\n", 25 | "\n", 26 | "import matplotlib.pyplot as plt" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "id": "a2ff3de5", 32 | "metadata": {}, 33 | "source": [ 34 | "## What is the correlation anyways?\n", 35 | "\n", 36 | "To get a handle on this, let's step back and talk about variance and covariance. Suppose we have two quantities, `A` and `B`. Formally in statistics, these would be called random variables, but for our purposes let's not worry about the jargon. We are casually familiar with the computing the mean. However, the formal definition of the mean is an integral like this\n", 37 | "\n", 38 | "$$\n", 39 | "E[A] = \\int dA\\, A P(A)\n", 40 | "$$\n", 41 | "\n", 42 | "The usual definition of the mean\n", 43 | "\n", 44 | "$$\n", 45 | "\\langle a\\rangle = \\frac{1}{N}\\sum a_i\n", 46 | "$$\n", 47 | "\n", 48 | "is known as an *estimator* of the mean. (Here the `a_i` are draws from the distribution $P(A)$. For example, if $P(A)$ is a Gaussian distribution, then the `a_i` would be a set of numbers whose histogram approaches the shape of a Gaussian as we increase the size of the set.) An estimator is something we compute from the data we observe as opposed to something defined by $P(A)$ like $E[A]$ above. It turns out that our estimator for the mean is unbiased since \n", 49 | "\n", 50 | "$$\n", 51 | "E[\\langle a \\rangle] = \\int da_0 ... \\int da_N\\, \\frac{1}{N}\\sum a_i P(a_0) ... P(a_N) = \\frac{1}{N}\\sum E[A] = E[A]\n", 52 | "$$\n", 53 | "\n", 54 | "However, this is not a general property of estimators. \n", 55 | "\n", 56 | "Now that we've define the mean, let's define the variance. This quantity is \n", 57 | "\n", 58 | "$$\n", 59 | "Var[A] = E[(A - E[A])^2] = \\int dA\\, (A - E[A])^2 P(A)\n", 60 | "$$\n", 61 | "\n", 62 | "The standard deviation is simply $Std[A] = \\sqrt{Var[A]}$. There are again estimators for the standard deviation. You are likely familiar with the standard one with a factor of N-1 in the denominator. We won't go through the math here, but this odd looking N-1 is needed to make the estimator an unbiased estimate of the standard deviation.\n", 63 | "\n", 64 | "Finally, we can define the covariance\n", 65 | "\n", 66 | "$$\n", 67 | "Cov[A, B] = E[(A-E[A])(B-E[B])] = \\int dAdB\\, (A-E[A])(B-E[B]) P(A, B) \n", 68 | "$$\n", 69 | "\n", 70 | "where now this is a double integral and we have the joint distribution of $A$ and $B$, $P(A,B)$.\n", 71 | "\n", 72 | "### Exercise 1\n", 73 | "\n", 74 | "Compute $Cov[A,A]$. Do you recognize this quantity? What is the general property or rule here?\n", 75 | "\n", 76 | "## Correlation, finally.\n", 77 | "\n", 78 | "Now that we understand the covariance, the correlation is defined as\n", 79 | "\n", 80 | "$$\n", 81 | "Corr[A, B] = \\frac{Cov[A,B]}{Std[A]\\,Std[B]}\n", 82 | "$$\n", 83 | "\n", 84 | "The correlation has a nice property that it is always bounded between between -1 and 1. Positive values are indicate correlation (i.e., as A goes up, B goes up too) and negative values indicate anticorrelation (i.e., as A goes up, B goes down).\n", 85 | "\n", 86 | "## Measuring the correlation of pixels in an image\n", 87 | "\n", 88 | "For an image, the concept of correlation is a bit more complicated. Instead of two sets of values A and B (e.g., height and age or something), we a full image worth of pixels. Further, we might have more than one image to handle. To handle these issues, we need to make some definitions. First, we're going to measure the correlation of pixels separated by some distance, say one pixel. The idea here is that we are asking the question, if a given pixel goes up, do the pixels next to it go up or down? (By up and down here I really mean bigger or less than the mean.)\n", 89 | "\n", 90 | "In math, we have for pixel value $V_ij$ and a separation of one pixel spacing in $j$ a quantity like\n", 91 | "\n", 92 | "$$\n", 93 | "C_{i,j+1} = \\langle Cov[V_{i,j}, V_{i,j+1}]\\rangle\n", 94 | "$$\n", 95 | "\n", 96 | "where the average is over all possible values of $i,j$. We can of course measure this for any offset in $\\Delta i,\\Delta j$. It is helpful to arrange these values into a 3x3 matrix where the center element is the above quantity with the offsets at zero and then each outer element is the quantity above with the offset given by the offset to the location in the matrix. Visually we have a grid of values like this\n", 97 | "\n", 98 | "$$\n", 99 | "\\begin{array}{ccc}\n", 100 | "C_{i-1,j+1} & C_{i,j+1} & C_{i+1,j+1}\\\\\n", 101 | "C_{i-1,j} & C_{i,j} & C_{i+1,j}\\\\\n", 102 | "C_{i-1,j-1} & C_{i,j-1} & C_{i+1,j-1}\\\\\n", 103 | "\\end{array}\n", 104 | "$$\n", 105 | "\n", 106 | "Finally, in our case, we have pixels which we want to ignore and so we cannot simply use all of the pixels we find. I have put a function below that does the computation we'd like." 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": 11, 112 | "id": "f8a45705", 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "import numba\n", 117 | "\n", 118 | "@numba.njit()\n", 119 | "def _meas_cov(im, msk, cov, nrm):\n", 120 | " for ic in range(im.shape[0]):\n", 121 | " io_min = max(ic-1, 0)\n", 122 | " io_max = min(ic+1, im.shape[0]-1) + 1\n", 123 | " \n", 124 | " for jc in range(im.shape[1]):\n", 125 | " jo_min = max(jc-1, 0)\n", 126 | " jo_max = min(jc+1, im.shape[1]-1) + 1\n", 127 | " \n", 128 | " if not msk[ic, jc]:\n", 129 | " continue\n", 130 | " \n", 131 | " for io in range(io_min, io_max):\n", 132 | " i = io - ic + 1\n", 133 | " \n", 134 | " for jo in range(jo_min, jo_max):\n", 135 | " j = jo - jc + 1\n", 136 | " \n", 137 | " if not msk[io, jo]:\n", 138 | " continue\n", 139 | "\n", 140 | " cov[i, j] += (im[ic, jc] * im[io, jo])\n", 141 | " nrm[i, j] += 1\n", 142 | " \n", 143 | " for i in range(3):\n", 144 | " for j in range(3):\n", 145 | " cov[i, j] /= nrm[i, j]\n", 146 | "\n", 147 | "\n", 148 | "def meas_cov(im, msk):\n", 149 | " \"\"\"Measure the one-offset covariance of pixels in an image, ignoring bad ones.\n", 150 | " \n", 151 | " Parameters\n", 152 | " ----------\n", 153 | " im : np.ndarray\n", 154 | " The image.\n", 155 | " msk : np.ndarray\n", 156 | " The mask where True is a good pixel and False is a bad pixel.\n", 157 | " \n", 158 | " Returns\n", 159 | " -------\n", 160 | " cov : np.ndarray\n", 161 | " The 3x3 matrix of covariances between the pixels.\n", 162 | " \"\"\"\n", 163 | " cov = np.zeros((3, 3))\n", 164 | " nrm = np.zeros((3, 3))\n", 165 | " mn = np.mean(im[msk])\n", 166 | " \n", 167 | " _meas_cov(im.astype(np.float64)-mn, msk, cov, nrm)\n", 168 | " return cov" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "id": "ecc68854", 174 | "metadata": {}, 175 | "source": [ 176 | "### Exercise 2: Measure covariance matrix of Gaussian random image\n", 177 | "\n", 178 | "1. Make an image of Gaussian random draws and measure the covariance matrix. I have put some code below to get you started.\n", 179 | "2. Try different sizes (e.g., 10x10, 100x100, 1000x1000). What do you notice about the results in relationship to the correct answer as you increase the image size? What is the correct answer?" 180 | ] 181 | }, 182 | { 183 | "cell_type": "code", 184 | "execution_count": 24, 185 | "id": "a3a4bfca", 186 | "metadata": {}, 187 | "outputs": [], 188 | "source": [ 189 | "im = np.random.normal(size=(1000, 1000))\n", 190 | "msk = np.ones_like(im)\n", 191 | "\n", 192 | "# use the function above" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "id": "642dcb19", 198 | "metadata": {}, 199 | "source": [ 200 | "## Exercise 3: Measure the covariance matrix for a single epoch image\n", 201 | "\n", 202 | "Now let's move on to a single epoch image. I have put a bunch of code below to help in making the cuts etc." 203 | ] 204 | }, 205 | { 206 | "cell_type": "code", 207 | "execution_count": 25, 208 | "id": "d0e4b53a", 209 | "metadata": {}, 210 | "outputs": [], 211 | "source": [ 212 | "def read_wcs(pth, ext=0):\n", 213 | " hdr = fitsio.read_header(pth, ext=ext)\n", 214 | " dct = {}\n", 215 | " for k in hdr.keys():\n", 216 | " try:\n", 217 | " dct[k.lower()] = hdr[k]\n", 218 | " except Exception:\n", 219 | " pass\n", 220 | " return esutil.wcsutil.WCS(dct)" 221 | ] 222 | }, 223 | { 224 | "cell_type": "code", 225 | "execution_count": 26, 226 | "id": "f8902d96", 227 | "metadata": {}, 228 | "outputs": [], 229 | "source": [ 230 | "meds_dir = \"/cosmo/scratch/mrbecker/MEDS_DIR\"\n", 231 | "tilename = \"DES0124-3332\"\n", 232 | "band = \"i\"\n", 233 | "yaml_pth = os.path.join(\n", 234 | " meds_dir, \n", 235 | " \"des-pizza-slices-y6-v8/pizza_cutter_info/%s_%s_pizza_cutter_info.yaml\" % (\n", 236 | " tilename, band\n", 237 | " )\n", 238 | ")\n", 239 | "\n", 240 | "with open(yaml_pth, \"r\") as fp:\n", 241 | " info = yaml.safe_load(fp.read())" 242 | ] 243 | }, 244 | { 245 | "cell_type": "code", 246 | "execution_count": 32, 247 | "id": "90c7c50c", 248 | "metadata": {}, 249 | "outputs": [], 250 | "source": [ 251 | "coadd_wcs = read_wcs(info['image_path'], ext=info['image_ext'])\n", 252 | "coadd_image = fitsio.read(info['image_path'], ext=info['image_ext'])\n", 253 | "coadd_weight = fitsio.read(info['weight_path'], ext=info['weight_ext'])\n", 254 | "coadd_bmask = fitsio.read(info['bmask_path'], ext=info['bmask_ext'])\n", 255 | "coadd_seg = fitsio.read(info['seg_path'], ext=info['seg_ext'])" 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": 48, 261 | "id": "5b8717e8", 262 | "metadata": {}, 263 | "outputs": [], 264 | "source": [ 265 | "def get_mask_and_image(si, coadd_wcs, coadd_weight, coadd_bmask, coadd_seg):\n", 266 | " # read all of the data here\n", 267 | " se_wcs = read_wcs(si['image_path'], ext=si['image_ext'])\n", 268 | " se_image = fitsio.read(si['image_path'], ext=si['image_ext'])\n", 269 | " se_weight = fitsio.read(si['weight_path'], ext=si['weight_ext'])\n", 270 | " se_bmask = fitsio.read(si['bmask_path'], ext=si['bmask_ext'])\n", 271 | " se_bkg = fitsio.read(si['bkg_path'], ext=si['bkg_ext'])\n", 272 | " se_image -= se_bkg\n", 273 | " se_image *= si['scale']\n", 274 | " \n", 275 | " # handle the WCS transforms\n", 276 | " xind, yind = np.meshgrid(np.arange(se_image.shape[1]), np.arange(se_image.shape[0]))\n", 277 | " xind = xind.ravel()\n", 278 | " yind = yind.ravel()\n", 279 | "\n", 280 | " # map to ra, dec\n", 281 | " ra, dec = se_wcs.image2sky(x=xind+1, y=yind+1)\n", 282 | "\n", 283 | " # now map back to coadd pixel coordinates\n", 284 | " x_c, y_c = coadd_wcs.sky2image(ra, dec)\n", 285 | "\n", 286 | " # now take nearest pixel\n", 287 | " x_c = (x_c + 0.5).astype(np.int32)\n", 288 | " y_c = (y_c + 0.5).astype(np.int32)\n", 289 | "\n", 290 | " # finally, to get back to the indices into a coadd image, we need to subtract \n", 291 | " # one per our discussion above\n", 292 | " x_i = x_c - 1\n", 293 | " y_i = y_c - 1\n", 294 | " \n", 295 | " # record unphysical rows and cols\n", 296 | " pmsk = (\n", 297 | " (x_i >= 0)\n", 298 | " & (x_i < 10000)\n", 299 | " & (y_i >= 0)\n", 300 | " & (y_i < 10000)\n", 301 | " ).reshape(se_image.shape)\n", 302 | " \n", 303 | " # now clip to proper range\n", 304 | " x_i = np.clip(x_i, 0, 9999)\n", 305 | " y_i = np.clip(y_i, 0, 9999)\n", 306 | " \n", 307 | " # now grab values\n", 308 | " se_coadd_seg = coadd_seg[y_i, x_i].reshape(se_image.shape)\n", 309 | " se_coadd_bmask = coadd_bmask[y_i, x_i].reshape(se_image.shape)\n", 310 | " se_coadd_weight = coadd_weight[y_i, x_i].reshape(se_image.shape) \n", 311 | " \n", 312 | " \n", 313 | " msk = (\n", 314 | " (se_coadd_seg == 0) # selects sky pixels\n", 315 | " & (se_coadd_bmask == 0) # selects pixels without defects\n", 316 | " & (se_bmask == 0)\n", 317 | " & (se_coadd_weight > 0) # selects pixels with finite variance\n", 318 | " & (se_weight > 0)\n", 319 | " & pmsk # select SE pixels that actuall fell onto the coadd image\n", 320 | " ) \n", 321 | " \n", 322 | " return msk, se_image" 323 | ] 324 | }, 325 | { 326 | "cell_type": "markdown", 327 | "id": "ce4ea7c1", 328 | "metadata": {}, 329 | "source": [ 330 | "Your task is loop through all of the SE images for the coadd tile (each entry in `info['src_info']`) and measure the covariance matrices. Scale each one by the variance to compute the correlation matrix. Record them in a list. Then come up with some way to display the results in a plot that shows how different they are.\n", 331 | "\n", 332 | "The code snippet below will measure the covariance matrix for a single entry of the list." 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": 50, 338 | "id": "f39fffff", 339 | "metadata": {}, 340 | "outputs": [ 341 | { 342 | "name": "stdout", 343 | "output_type": "stream", 344 | "text": [ 345 | "[[0.00274254 0.00329531 0.00263506]\n", 346 | " [0.00277856 1. 0.00277856]\n", 347 | " [0.00263506 0.00329531 0.00274254]]\n" 348 | ] 349 | } 350 | ], 351 | "source": [ 352 | "se_ind = 5 # 5th image is index 4\n", 353 | "si = info['src_info'][se_ind]\n", 354 | "\n", 355 | "msk, im = get_image_mask(si, coadd_wcs, coadd_weight, coadd_bmask, coadd_seg)\n", 356 | "cov = meas_cov(im, msk)\n", 357 | "corr = cov / cov[1, 1]\n", 358 | "print(corr)" 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": null, 364 | "id": "1399e263", 365 | "metadata": {}, 366 | "outputs": [], 367 | "source": [ 368 | "# here is the skeleton of a loop to get you started\n", 369 | "results = []\n", 370 | "for i in range(len(info['src_info'])):\n", 371 | " si = info['src_info'][i]\n", 372 | "\n", 373 | " results.append(corr)" 374 | ] 375 | }, 376 | { 377 | "cell_type": "markdown", 378 | "id": "0d6319b2", 379 | "metadata": {}, 380 | "source": [ 381 | "## Save the Data\n", 382 | "\n", 383 | "You can use python pickle to save the data for now. If your data is in the python variable `results`, then this looks like" 384 | ] 385 | }, 386 | { 387 | "cell_type": "code", 388 | "execution_count": null, 389 | "id": "fb14fc84", 390 | "metadata": {}, 391 | "outputs": [], 392 | "source": [ 393 | "import pickle\n", 394 | "\n", 395 | "# write the data\n", 396 | "with open(\"test.pkl\", \"wb\") as fp:\n", 397 | " pickle.dump(results, fp)\n", 398 | "\n", 399 | "# read the data\n", 400 | "with open(\"test.pkl\", \"rb\") as fp:\n", 401 | " new_res = pickle.load(fp)" 402 | ] 403 | } 404 | ], 405 | "metadata": { 406 | "kernelspec": { 407 | "display_name": "Python [conda env:des-pixcorr] *", 408 | "language": "python", 409 | "name": "conda-env-des-pixcorr-py" 410 | }, 411 | "language_info": { 412 | "codemirror_mode": { 413 | "name": "ipython", 414 | "version": 3 415 | }, 416 | "file_extension": ".py", 417 | "mimetype": "text/x-python", 418 | "name": "python", 419 | "nbconvert_exporter": "python", 420 | "pygments_lexer": "ipython3", 421 | "version": "3.9.6" 422 | } 423 | }, 424 | "nbformat": 4, 425 | "nbformat_minor": 5 426 | } 427 | -------------------------------------------------------------------------------- /des_data_goodpixels.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "13a6ff05", 6 | "metadata": {}, 7 | "source": [ 8 | "# Finding Good Background Pixels in DES Data\n", 9 | "\n", 10 | "We're gonna go through some of the basics on how to access good pixels which are also background pixels in DES data." 11 | ] 12 | }, 13 | { 14 | "cell_type": "code", 15 | "execution_count": 1, 16 | "id": "6041f640", 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [ 20 | "import os\n", 21 | "import fitsio\n", 22 | "import yaml\n", 23 | "import esutil\n", 24 | "import numpy as np\n", 25 | "\n", 26 | "import matplotlib.pyplot as plt" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "id": "a2ff3de5", 32 | "metadata": {}, 33 | "source": [ 34 | "## What is a good background pixel anyways?\n", 35 | "\n", 36 | "We will define the good background pixels as\n", 37 | "\n", 38 | "1. not associated with any detected objects\n", 39 | "2. positive weight\n", 40 | "3. no bit flags set\n", 41 | "\n", 42 | "Let's discuss these in more detail.\n", 43 | "\n", 44 | "### How can I tell if a pixel is associated with any detected objects?\n", 45 | "\n", 46 | "For this task, we will use the segmentation maps associated with the coadd and single-epoch images. Remember that \n", 47 | "a segementation image has integer values and marks areas of the image associated with each detection. For our purposes, we only want pixels where the segmentation map is 0.\n", 48 | "\n", 49 | "### What does it mean for a pixel to have positive weight?\n", 50 | "\n", 51 | "Remember that the weight map is the inverse of the variance of the data in the pixel. This format means that pixels with large variance have very small weights. Sometimes, we set the value of the weight map to zero by hand in order to indicate that a given pixel should be ignored. Thus we want to demand that the weight map is greater than zero.\n", 52 | "\n", 53 | "### What does it mean for a pixel to have no bit flags set?\n", 54 | "\n", 55 | "As images are processed, numerical operations are performed on the pixels and sets of pixels are flagged as being from artifacts, defects, etc. This information is typically stored in what is known as a bit mask image. A bit mask image is an image of integers. For each pixel, the underlying binary representation of the number is used to store flags indicating different meanings. If not bit flags are set, then value of this field should be 0.\n", 56 | "\n", 57 | "## How does this apply to DES data?\n", 58 | "\n", 59 | "For DES data, we will demand these conditions of both the single-epoch image and the coadd image in the same part of the sky. To do this we will have to map all of the single-epoch pixels to their nearest location in the coadd image. \n", 60 | "\n", 61 | "Below I've put some code in the help guide you through this task. The steps will be as follows.\n", 62 | "\n", 63 | "1. Read in the coadd image and single-epoch image data.\n", 64 | "2. Map all of the single-epoch pixels to their nearest coadd pixel.\n", 65 | "4. Make the proper set of cuts on all of these quantities. \n", 66 | "5. Visualize the resulting image.\n", 67 | "\n", 68 | "### 1. Read in the Image Data\n", 69 | "\n", 70 | "We'll use the wcs reading function from the last tutorial." 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": 11, 76 | "id": "d0e4b53a", 77 | "metadata": {}, 78 | "outputs": [], 79 | "source": [ 80 | "def read_wcs(pth, ext=0):\n", 81 | " hdr = fitsio.read_header(pth, ext=ext)\n", 82 | " dct = {}\n", 83 | " for k in hdr.keys():\n", 84 | " try:\n", 85 | " dct[k.lower()] = hdr[k]\n", 86 | " except Exception:\n", 87 | " pass\n", 88 | " return esutil.wcsutil.WCS(dct)" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "id": "b9f1438b", 94 | "metadata": {}, 95 | "source": [ 96 | "Now let's get the images and the WCS solutions. First we grab the info dictionary from the YAML to get the paths." 97 | ] 98 | }, 99 | { 100 | "cell_type": "code", 101 | "execution_count": 2, 102 | "id": "f8902d96", 103 | "metadata": {}, 104 | "outputs": [], 105 | "source": [ 106 | "meds_dir = \"/cosmo/scratch/mrbecker/MEDS_DIR\"\n", 107 | "tilename = \"DES0124-3332\"\n", 108 | "band = \"i\"\n", 109 | "yaml_pth = os.path.join(\n", 110 | " meds_dir, \n", 111 | " \"des-pizza-slices-y6-v8/pizza_cutter_info/%s_%s_pizza_cutter_info.yaml\" % (\n", 112 | " tilename, band\n", 113 | " )\n", 114 | ")\n", 115 | "\n", 116 | "with open(yaml_pth, \"r\") as fp:\n", 117 | " info = yaml.safe_load(fp.read())" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 10, 123 | "id": "ccd9dee4", 124 | "metadata": {}, 125 | "outputs": [], 126 | "source": [ 127 | "import os\n", 128 | "\n", 129 | "meds_dir = os.environ.get(\"MEDS_DIR\")\n", 130 | "tilename = \"DES2359-6331\"\n", 131 | "band = \"i\"\n", 132 | "yaml_pth = os.path.join(\n", 133 | " meds_dir, \n", 134 | " \"des-pizza-slices-y6-v6/pizza_cutter_info/%s_%s_pizza_cutter_info.yaml\" % (\n", 135 | " tilename, band\n", 136 | " )\n", 137 | ")\n", 138 | "\n", 139 | "with open(yaml_pth, \"r\") as fp:\n", 140 | " info = yaml.safe_load(fp.read())" 141 | ] 142 | }, 143 | { 144 | "cell_type": "markdown", 145 | "id": "ea794081", 146 | "metadata": {}, 147 | "source": [ 148 | "And we read the stuff:" 149 | ] 150 | }, 151 | { 152 | "cell_type": "code", 153 | "execution_count": 15, 154 | "id": "90c7c50c", 155 | "metadata": {}, 156 | "outputs": [], 157 | "source": [ 158 | "coadd_wcs = read_wcs(info['image_path'], ext=info['image_ext'])\n", 159 | "coadd_image = fitsio.read(info['image_path'], ext=info['image_ext'])\n", 160 | "coadd_weight = read_wcs(info['weight_path'], ext=info['weight_ext'])\n", 161 | "coadd_bmask = read_wcs(info['bmask_path'], ext=info['bmask_ext'])\n", 162 | "coadd_seg = read_wcs(info['seg_path'], ext=info['seg_ext'])" 163 | ] 164 | }, 165 | { 166 | "cell_type": "markdown", 167 | "id": "50d5981c", 168 | "metadata": {}, 169 | "source": [ 170 | "We will look at the 5th single-epoch image." 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": 18, 176 | "id": "5b8717e8", 177 | "metadata": {}, 178 | "outputs": [], 179 | "source": [ 180 | "se_ind = 4 # 5th image is index 4\n", 181 | "si = info['src_info'][se_ind]\n", 182 | "se_wcs = read_wcs(si['image_path'], ext=si['image_ext'])\n", 183 | "se_image = fitsio.read(si['image_path'], ext=si['image_ext'])\n", 184 | "se_weight = fitsio.read(si['weight_path'], ext=si['weight_ext'])\n", 185 | "se_bmask = fitsio.read(si['bmask_path'], ext=si['bmask_ext'])" 186 | ] 187 | }, 188 | { 189 | "cell_type": "markdown", 190 | "id": "3f89602c", 191 | "metadata": {}, 192 | "source": [ 193 | "### 2. Map all of the single-epoch pixels to the nearest coadd pixel\n", 194 | "\n", 195 | "To get you started, I have build a list of the pixel indices for the single-epoch image pixels below." 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 22, 201 | "id": "26e543d2", 202 | "metadata": {}, 203 | "outputs": [], 204 | "source": [ 205 | "xind, yind = np.meshgrid(se_image.shape[1], se_image.shape[0])\n", 206 | "xind = xind.ravel()\n", 207 | "yind = yind.ravel()" 208 | ] 209 | }, 210 | { 211 | "cell_type": "code", 212 | "execution_count": null, 213 | "id": "12363efd", 214 | "metadata": {}, 215 | "outputs": [], 216 | "source": [ 217 | "# do the rest here, computing the coadd indices for each single-epoch pixel\n", 218 | "# when you do this, you'll need to cut out indices less than zero or greater than or equal to the dimenions\n", 219 | "# this can be done with mask arrays\n", 220 | "# msk = (x >= 0) & (x < coadd_image.shape[1])\n", 221 | "# x = x[msk]" 222 | ] 223 | }, 224 | { 225 | "cell_type": "markdown", 226 | "id": "f099ceb4", 227 | "metadata": {}, 228 | "source": [ 229 | "### 3. Make the cuts on the proper quantities\n", 230 | "\n", 231 | "Once you have the pixel locations remapped, then you can make cuts using the same mask array syntax as above." 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "id": "62a0b6b3", 238 | "metadata": {}, 239 | "outputs": [], 240 | "source": [] 241 | }, 242 | { 243 | "cell_type": "markdown", 244 | "id": "720165f9", 245 | "metadata": {}, 246 | "source": [ 247 | "### 4. Visualize the Data\n", 248 | "\n", 249 | "Make a plot of the good and bad pixels in the image." 250 | ] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "execution_count": null, 255 | "id": "ece5c40f", 256 | "metadata": {}, 257 | "outputs": [], 258 | "source": [] 259 | } 260 | ], 261 | "metadata": { 262 | "kernelspec": { 263 | "display_name": "Python [conda env:anl] *", 264 | "language": "python", 265 | "name": "conda-env-anl-py" 266 | }, 267 | "language_info": { 268 | "codemirror_mode": { 269 | "name": "ipython", 270 | "version": 3 271 | }, 272 | "file_extension": ".py", 273 | "mimetype": "text/x-python", 274 | "name": "python", 275 | "nbconvert_exporter": "python", 276 | "pygments_lexer": "ipython3", 277 | "version": "3.8.10" 278 | } 279 | }, 280 | "nbformat": 4, 281 | "nbformat_minor": 5 282 | } 283 | -------------------------------------------------------------------------------- /mdet_meas_tools.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | """ 3 | This file has a bunch of collected utilities for doing shear measurement tests 4 | with metadetect. 5 | """ 6 | import sys 7 | import time 8 | import logging 9 | 10 | import numpy as np 11 | import fitsio 12 | 13 | from esutil.numpy_util import combine_arrlist 14 | import ngmix 15 | from metadetect.metadetect import do_metadetect 16 | 17 | import multiprocessing 18 | import contextlib 19 | 20 | import tqdm 21 | import schwimmbad 22 | import yaml 23 | 24 | MDET_CONFIG = yaml.safe_load("""\ 25 | metacal: 26 | psf: fitgauss 27 | types: [noshear, 1p, 1m, 2p, 2m] 28 | use_noise_image: True 29 | 30 | psf: 31 | lm_pars: 32 | maxfev: 2000 33 | ftol: 1.0e-05 34 | xtol: 1.0e-05 35 | model: gauss 36 | 37 | # we try many times because if this fails we get no psf info 38 | # for the entire patch 39 | ntry: 10 40 | 41 | sx: 42 | # Minimum contrast parameter for deblending 43 | deblend_cont: 1.0e-05 44 | 45 | # in sky sigma 46 | detect_thresh: 0.8 47 | 48 | # minimum number of pixels above threshold 49 | minarea: 4 50 | 51 | filter_type: conv 52 | # 7x7 convolution mask of a gaussian PSF with FWHM = 3.0 pixels. 53 | filter_kernel: 54 | - [0.004963, 0.021388, 0.051328, 0.068707, 0.051328, 0.021388, 0.004963] 55 | - [0.021388, 0.092163, 0.221178, 0.296069, 0.221178, 0.092163, 0.021388] 56 | - [0.051328, 0.221178, 0.530797, 0.710525, 0.530797, 0.221178, 0.051328] 57 | - [0.068707, 0.296069, 0.710525, 0.951108, 0.710525, 0.296069, 0.068707] 58 | - [0.051328, 0.221178, 0.530797, 0.710525, 0.530797, 0.221178, 0.051328] 59 | - [0.021388, 0.092163, 0.221178, 0.296069, 0.221178, 0.092163, 0.021388] 60 | - [0.004963, 0.021388, 0.051328, 0.068707, 0.051328, 0.021388, 0.004963] 61 | 62 | weight: 63 | fwhm: 1.2 # arcsec 64 | 65 | meds: 66 | box_padding: 2 67 | box_type: iso_radius 68 | max_box_size: 64 69 | min_box_size: 32 70 | rad_fac: 2 71 | rad_min: 4 72 | 73 | # check for an edge hit 74 | bmask_flags: 536870912 # 2**29 75 | """) 76 | 77 | 78 | @contextlib.contextmanager 79 | def backend_pool(backend, n_workers=None): 80 | """Context manager to build a schwimmbad `pool` object with the `map` method. 81 | 82 | Parameters 83 | ---------- 84 | backend : str 85 | One of 'sequential', `loky`, or 'mpi'. 86 | n_workers : int, optional 87 | The number of workers to use. Defaults to 1 for the 'sequential' backend, 88 | the cpu count for the 'loky' backend, and the size of the default global 89 | communicator for the 'mpi' backend. 90 | """ 91 | try: 92 | if backend == "sequential": 93 | pool = schwimmbad.JoblibPool(1, backend=backend, verbose=0) 94 | else: 95 | if backend == "mpi": 96 | from mpi4py import MPI 97 | pool = schwimmbad.choose_pool( 98 | mpi=True, 99 | processes=n_workers or MPI.COMM_WORLD.Get_size(), 100 | ) 101 | else: 102 | pool = schwimmbad.JoblibPool( 103 | n_workers or multiprocessing.cpu_count(), 104 | backend=backend, 105 | verbose=100, 106 | ) 107 | yield pool 108 | finally: 109 | if "pool" in locals(): 110 | pool.close() 111 | 112 | 113 | def cut_nones(presults, mresults): 114 | """Cut entries that are None in a pair of lists. Any entry that is None 115 | in either list will exclude the item in the other. 116 | 117 | Parameters 118 | ---------- 119 | presults : list 120 | One the list of things. 121 | mresults : list 122 | The other list of things. 123 | 124 | Returns 125 | ------- 126 | pcut : list 127 | The cut list. 128 | mcut : list 129 | The cut list. 130 | """ 131 | prr_keep = [] 132 | mrr_keep = [] 133 | for pr, mr in zip(presults, mresults): 134 | if pr is None or mr is None: 135 | continue 136 | prr_keep.append(pr) 137 | mrr_keep.append(mr) 138 | 139 | return prr_keep, mrr_keep 140 | 141 | 142 | def _run_boostrap(x1, y1, x2, y2, wgts): 143 | rng = np.random.RandomState(seed=100) 144 | mvals = [] 145 | cvals = [] 146 | for _ in tqdm.trange(500, leave=False): 147 | ind = rng.choice(len(y1), replace=True, size=len(y1)) 148 | _wgts = wgts[ind].copy() 149 | _wgts /= np.sum(_wgts) 150 | mvals.append(np.mean(y1[ind] * _wgts) / np.mean(x1[ind] * _wgts) - 1) 151 | cvals.append(np.mean(y2[ind] * _wgts) / np.mean(x2[ind] * _wgts)) 152 | 153 | return ( 154 | np.mean(y1 * wgts) / np.mean(x1 * wgts) - 1, np.std(mvals), 155 | np.mean(y2 * wgts) / np.mean(x2 * wgts), np.std(cvals)) 156 | 157 | 158 | def _run_jackknife(x1, y1, x2, y2, wgts, jackknife): 159 | n_per = x1.shape[0] // jackknife 160 | n = n_per * jackknife 161 | x1j = np.zeros(jackknife) 162 | y1j = np.zeros(jackknife) 163 | x2j = np.zeros(jackknife) 164 | y2j = np.zeros(jackknife) 165 | wgtsj = np.zeros(jackknife) 166 | 167 | loc = 0 168 | for i in range(jackknife): 169 | wgtsj[i] = np.sum(wgts[loc:loc+n_per]) 170 | x1j[i] = np.sum(x1[loc:loc+n_per] * wgts[loc:loc+n_per]) / wgtsj[i] 171 | y1j[i] = np.sum(y1[loc:loc+n_per] * wgts[loc:loc+n_per]) / wgtsj[i] 172 | x2j[i] = np.sum(x2[loc:loc+n_per] * wgts[loc:loc+n_per]) / wgtsj[i] 173 | y2j[i] = np.sum(y2[loc:loc+n_per] * wgts[loc:loc+n_per]) / wgtsj[i] 174 | 175 | loc += n_per 176 | 177 | mbar = np.mean(y1 * wgts) / np.mean(x1 * wgts) - 1 178 | cbar = np.mean(y2 * wgts) / np.mean(x2 * wgts) 179 | mvals = np.zeros(jackknife) 180 | cvals = np.zeros(jackknife) 181 | for i in range(jackknife): 182 | _wgts = np.delete(wgtsj, i) 183 | mvals[i] = ( 184 | np.sum(np.delete(y1j, i) * _wgts) / np.sum(np.delete(x1j, i) * _wgts) 185 | - 1 186 | ) 187 | cvals[i] = ( 188 | np.sum(np.delete(y2j, i) * _wgts) / np.sum(np.delete(x2j, i) * _wgts) 189 | ) 190 | 191 | return ( 192 | mbar, 193 | np.sqrt((n - n_per) / n * np.sum((mvals-mbar)**2)), 194 | cbar, 195 | np.sqrt((n - n_per) / n * np.sum((cvals-cbar)**2)), 196 | ) 197 | 198 | 199 | def _estimate_m_and_c( 200 | presults, 201 | mresults, 202 | g_true, 203 | swap12=False, 204 | step=0.01, 205 | weights=None, 206 | jackknife=None, 207 | ): 208 | """Estimate m and c from paired lensing simulations. 209 | 210 | Parameters 211 | ---------- 212 | presults : list of iterables 213 | A list of iterables, each with g1p, g1m, g1, g2p, g2m, g2 214 | from running metadetect with a `g1` shear in the 1-component and 215 | 0 true shear in the 2-component. 216 | mresults : list of iterables 217 | A list of iterables, each with g1p, g1m, g1, g2p, g2m, g2 218 | from running metadetect with a -`g1` shear in the 1-component and 219 | 0 true shear in the 2-component. 220 | g_true : float 221 | The true value of the shear on the 1-axis in the simulation. The other 222 | axis is assumd to havea true value of zero. 223 | swap12 : bool, optional 224 | If True, swap the roles of the 1- and 2-axes in the computation. 225 | step : float, optional 226 | The step used in metadetect for estimating the response. Default is 227 | 0.01. 228 | weights : list of weights, optional 229 | Weights to apply to each sample. Will be normalized if not already. 230 | jackknife : int, optional 231 | The number of jackknife sections to use for error estimation. Default of 232 | None will do no jackknife and default to bootstrap error bars. 233 | 234 | Returns 235 | ------- 236 | m : float 237 | Estimate of the multiplicative bias. 238 | merr : float 239 | Estimat of the 1-sigma standard error in `m`. 240 | c : float 241 | Estimate of the additive bias. 242 | cerr : float 243 | Estimate of the 1-sigma standard error in `c`. 244 | """ 245 | 246 | prr_keep, mrr_keep = cut_nones(presults, mresults) 247 | 248 | def _get_stuff(rr): 249 | _a = np.vstack(rr) 250 | g1p = _a[:, 0] 251 | g1m = _a[:, 1] 252 | g1 = _a[:, 2] 253 | g2p = _a[:, 3] 254 | g2m = _a[:, 4] 255 | g2 = _a[:, 5] 256 | 257 | if swap12: 258 | g1p, g1m, g1, g2p, g2m, g2 = g2p, g2m, g2, g1p, g1m, g1 259 | 260 | return ( 261 | g1, (g1p - g1m) / 2 / step * g_true, 262 | g2, (g2p - g2m) / 2 / step) 263 | 264 | g1p, R11p, g2p, R22p = _get_stuff(prr_keep) 265 | g1m, R11m, g2m, R22m = _get_stuff(mrr_keep) 266 | 267 | if weights is not None: 268 | wgts = np.array(weights).astype(np.float64) 269 | else: 270 | wgts = np.ones(len(g1p)).astype(np.float64) 271 | wgts /= np.sum(wgts) 272 | 273 | msk = ( 274 | np.isfinite(g1p) & 275 | np.isfinite(R11p) & 276 | np.isfinite(g1m) & 277 | np.isfinite(R11m) & 278 | np.isfinite(g2p) & 279 | np.isfinite(R22p) & 280 | np.isfinite(g2m) & 281 | np.isfinite(R22m)) 282 | g1p = g1p[msk] 283 | R11p = R11p[msk] 284 | g1m = g1m[msk] 285 | R11m = R11m[msk] 286 | g2p = g2p[msk] 287 | R22p = R22p[msk] 288 | g2m = g2m[msk] 289 | R22m = R22m[msk] 290 | wgts = wgts[msk] 291 | 292 | x1 = (R11p + R11m)/2 293 | y1 = (g1p - g1m) / 2 294 | 295 | x2 = (R22p + R22m) / 2 296 | y2 = (g2p + g2m) / 2 297 | 298 | if jackknife: 299 | return _run_jackknife(x1, y1, x2, y2, wgts, jackknife) 300 | else: 301 | return _run_boostrap(x1, y1, x2, y2, wgts) 302 | 303 | 304 | def estimate_m_and_c( 305 | pdata, 306 | mdata, 307 | g_true=0.02, 308 | swap12=False, 309 | step=0.01, 310 | weights=None, 311 | jackknife=None, 312 | ): 313 | """Estimate m and c from paired lensing simulations. 314 | 315 | Parameters 316 | ---------- 317 | pdata : np.ndarray 318 | The sim data from the plus simulations. 319 | mdata : np.ndarray 320 | The sim data form the minus simulations. 321 | g_true : float, optional 322 | The true value of the shear on the 1-axis in the simulation. The other 323 | axis is assumd to havea true value of zero. Defualt value is 0.02. 324 | swap12 : bool, optional 325 | If True, swap the roles of the 1- and 2-axes in the computation. 326 | step : float, optional 327 | The step used in metadetect for estimating the response. Default is 328 | 0.01. 329 | weights : list of weights, optional 330 | Weights to apply to each sample. Will be normalized if not already. 331 | jackknife : int, optional 332 | The number of jackknife sections to use for error estimation. Default of 333 | None will do no jackknife and default to bootstrap error bars. 334 | 335 | Returns 336 | ------- 337 | m : float 338 | Estimate of the multiplicative bias. 339 | merr : float 340 | Estimat of the 1-sigma standard error in `m`. 341 | c : float 342 | Estimate of the additive bias. 343 | cerr : float 344 | Estimate of the 1-sigma standard error in `c`. 345 | """ 346 | pres = [ 347 | ( 348 | pdata["g1p"][i], pdata["g1m"][i], pdata["g1"][i], 349 | pdata["g2p"][i], pdata["g2m"][i], pdata["g2"][i], 350 | ) 351 | for i in range(pdata.shape[0]) 352 | ] 353 | 354 | mres = [ 355 | ( 356 | mdata["g1p"][i], mdata["g1m"][i], mdata["g1"][i], 357 | mdata["g2p"][i], mdata["g2m"][i], mdata["g2"][i], 358 | ) 359 | for i in range(pdata.shape[0]) 360 | ] 361 | 362 | return _estimate_m_and_c( 363 | pres, 364 | mres, 365 | g_true, 366 | swap12=swap12, 367 | step=step, 368 | weights=weights, 369 | jackknife=jackknife, 370 | ) 371 | 372 | 373 | def measure_shear_metadetect(res, *, s2n_cut, t_ratio_cut, ormask_cut, mfrac_cut): 374 | """Measure the shear parameters for metadetect. 375 | 376 | NOTE: Returns None if nothing can be measured. 377 | 378 | Parameters 379 | ---------- 380 | res : dict 381 | The metadetect results. 382 | s2n_cut : float 383 | The cut on `wmom_s2n`. Typically 10. 384 | t_ratio_cut : float 385 | The cut on `t_ratio_cut`. Typically 1.2. 386 | ormask_cut : bool 387 | If True, cut on the `ormask` flags. 388 | mfrac_cut : float or None 389 | If not None, cut objects with a masked fraction higher than this 390 | value. 391 | 392 | Returns 393 | ------- 394 | g1p : float 395 | The mean 1-component shape for the plus metadetect measurement. 396 | g1m : float 397 | The mean 1-component shape for the minus metadetect measurement. 398 | g1 : float 399 | The mean 1-component shape for the zero-shear metadetect measurement. 400 | g2p : float 401 | The mean 2-component shape for the plus metadetect measurement. 402 | g2m : float 403 | The mean 2-component shape for the minus metadetect measurement. 404 | g2 : float 405 | The mean 2-component shape for the zero-shear metadetect measurement. 406 | """ 407 | def _mask(data): 408 | _cut_msk = ( 409 | (data['flags'] == 0) 410 | & (data['wmom_s2n'] > s2n_cut) 411 | & (data['wmom_T_ratio'] > t_ratio_cut) 412 | ) 413 | if ormask_cut: 414 | _cut_msk = _cut_msk & (data['ormask'] == 0) 415 | if mfrac_cut is not None: 416 | _cut_msk = _cut_msk & (data["mfrac"] <= mfrac_cut) 417 | return _cut_msk 418 | 419 | op = res['1p'] 420 | q = _mask(op) 421 | if not np.any(q): 422 | return None 423 | g1p = op['wmom_g'][q, 0] 424 | 425 | om = res['1m'] 426 | q = _mask(om) 427 | if not np.any(q): 428 | return None 429 | g1m = om['wmom_g'][q, 0] 430 | 431 | o = res['noshear'] 432 | q = _mask(o) 433 | if not np.any(q): 434 | return None 435 | g1 = o['wmom_g'][q, 0] 436 | g2 = o['wmom_g'][q, 1] 437 | 438 | op = res['2p'] 439 | q = _mask(op) 440 | if not np.any(q): 441 | return None 442 | g2p = op['wmom_g'][q, 1] 443 | 444 | om = res['2m'] 445 | q = _mask(om) 446 | if not np.any(q): 447 | return None 448 | g2m = om['wmom_g'][q, 1] 449 | 450 | return ( 451 | np.mean(g1p), np.mean(g1m), np.mean(g1), 452 | np.mean(g2p), np.mean(g2m), np.mean(g2)) 453 | 454 | 455 | def _run_mdet(obs, seed): 456 | obs.mfrac = np.zeros_like(obs.image) 457 | 458 | mbobs = ngmix.MultiBandObsList() 459 | obslist = ngmix.ObsList() 460 | obslist.append(obs) 461 | mbobs.append(obslist) 462 | 463 | return do_metadetect(MDET_CONFIG, mbobs, np.random.RandomState(seed=seed)) 464 | 465 | 466 | def _run_sim_pair(args): 467 | num, backend, sim_func, sim_kwargs, start, seed = args 468 | pobs = sim_func(g1=0.02, g2=0.0, seed=seed, **sim_kwargs) 469 | mobs = sim_func(g1=-0.02, g2=0.0, seed=seed, **sim_kwargs) 470 | 471 | pres = _run_mdet(pobs, seed+1024768) 472 | mres = _run_mdet(mobs, seed+1024769) 473 | 474 | if pres is None or mres is None: 475 | return None, None 476 | 477 | fkeys = ["g1p", "g1m", "g1", "g2p", "g2m", "g2"] 478 | dtype = [] 479 | for key in fkeys: 480 | dtype.append((key, "f8")) 481 | 482 | pgm = measure_shear_metadetect( 483 | pres, s2n_cut=10, t_ratio_cut=1.2, 484 | ormask_cut=False, mfrac_cut=None, 485 | ) 486 | mgm = measure_shear_metadetect( 487 | mres, s2n_cut=10, t_ratio_cut=1.2, 488 | ormask_cut=False, mfrac_cut=None, 489 | ) 490 | if pgm is None or mgm is None: 491 | return None, None 492 | 493 | datap = [pgm] 494 | datam = [mgm] 495 | 496 | if backend == "mpi": 497 | print( 498 | "[% 10ds] did %04d" % (time.time() - start, num+1), 499 | flush=True, 500 | ) 501 | 502 | return np.array(datap, dtype=dtype), np.array(datam, dtype=dtype) 503 | 504 | 505 | def run_mdet_sims( 506 | sim_func, sim_kwargs, seed, n_sims, 507 | log_level='warning', backend='sequential', n_workers=None 508 | ): 509 | """Run simulation(s) and analyze them with metadetect. 510 | 511 | Parameters 512 | ---------- 513 | sim_func : callable 514 | A function accepting only keyword args with the following signature: 515 | 516 | def sim_func(*, g1, g2, seed, extra kwargs here...): 517 | # do computations here 518 | 519 | It should make a simulation and return an ngmix observation for it. 520 | See the shape_measurement_102.ipynb notebook for an example. 521 | sim_kwargs : dict 522 | any extra sim kwargs to pass to `sim_func`. 523 | seed : int 524 | An RNG seed for seeding the simulations. 525 | n_sims : int 526 | The number of simulations to run. 527 | log_level : str, optional 528 | The logging level for the sim. Set to 'debug' if you'd like more output. 529 | Only works if `backend` is 'sequential'. 530 | backend : str, optional 531 | Set to 'loky' to run simulations in parallel. The default is 'sequential'. 532 | n_workers : int, optional 533 | The number of workers to use when running in parallel. Default of None 534 | will choose a correct number based on the local system and the setting 535 | for `backend`. 536 | 537 | Returns 538 | ------- 539 | pdata : np.ndarray 540 | The sim data from the plus simulations. 541 | mdata : np.ndarray 542 | The sim data form the minus simulations. 543 | """ 544 | 545 | start = time.time() 546 | 547 | if backend == "sequential": 548 | logging.basicConfig(stream=sys.stdout) 549 | for code in ["ngmix", "metadetect"]: 550 | logging.getLogger(code).setLevel( 551 | getattr(logging, log_level.upper())) 552 | 553 | if backend == "mpi": 554 | from mpi4py import MPI 555 | comm = MPI.COMM_WORLD 556 | rank = comm.Get_rank() 557 | else: 558 | rank = 0 559 | 560 | if rank == 0: 561 | rng = np.random.RandomState(seed=seed) 562 | sim_rng_seeds = rng.randint(low=1, high=2**29, size=n_sims) 563 | 564 | args = [] 565 | for i, rng_seed in enumerate(sim_rng_seeds): 566 | args.append(( 567 | i, 568 | backend, 569 | sim_func, 570 | sim_kwargs, 571 | start, 572 | rng_seed, 573 | )) 574 | else: 575 | args = [] 576 | 577 | with backend_pool(backend, n_workers=n_workers) as pool: 578 | outputs = pool.map(_run_sim_pair, args) 579 | 580 | if rank == 0: 581 | pdata, mdata = zip(*outputs) 582 | pdata, mdata = cut_nones(pdata, mdata) 583 | if len(pdata) > 0 and len(mdata) > 0: 584 | pdata = combine_arrlist(list(pdata)) 585 | mdata = combine_arrlist(list(mdata)) 586 | 587 | m, msd, c, csd = estimate_m_and_c( 588 | pdata, 589 | mdata, 590 | ) 591 | 592 | print("""\ 593 | # of sims: {n_sims} 594 | noise cancel m : {m: f} +/- {msd: f} [1e-3, 3-sigma] 595 | noise cancel c : {c: f} +/- {csd: f} [1e-5, 3-sigma]""".format( 596 | n_sims=len(pdata), 597 | m=m/1e-3, 598 | msd=msd/1e-3 * 3, 599 | c=c/1e-5, 600 | csd=csd/1e-5 * 3, 601 | ), 602 | flush=True, 603 | ) 604 | 605 | return pdata, mdata 606 | else: 607 | return None, None 608 | 609 | 610 | def write_sim_data(filename, pdata, mdata): 611 | """Write sim data to a file. 612 | 613 | Parameters 614 | ---------- 615 | filename : str 616 | The full path and name of the file to write. The name should end in `.fits`. 617 | pdata : np.ndarray 618 | The sim data from the plus simulations. 619 | mdata : np.ndarray 620 | The sim data form the minus simulations. 621 | """ 622 | with fitsio.FITS(filename, 'rw', clobber=True) as fits: 623 | fits.write(pdata, extname='plus') 624 | fits.write(mdata, extname='minus') 625 | 626 | 627 | def read_sim_data(filename): 628 | """Read sim data from a path. 629 | 630 | Parameters 631 | ---------- 632 | filename : str 633 | The full path and name of the file to read. The name should end in `.fits`. 634 | 635 | Returns 636 | ------- 637 | pdata : np.ndarray 638 | The sim data from the plus simulations. 639 | mdata : np.ndarray 640 | The sim data form the minus simulations. 641 | """ 642 | with fitsio.FITS(filename, 'r', clobber=True) as fits: 643 | pdata = fits['plus'].read() 644 | mdata = fits['minus'].read() 645 | 646 | return pdata, mdata 647 | -------------------------------------------------------------------------------- /shape_measurement_101.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "030f1314", 6 | "metadata": {}, 7 | "source": [ 8 | "# Shape Measurement 101\n", 9 | "\n", 10 | "This notebook has the first part of a small-ish tutorial on galaxy shape measurement. The goals of this tutorial are as follows\n", 11 | "\n", 12 | "1. Become familiar with the concept of what we mean by a galaxy's shape.\n", 13 | "2. Form an intuitive understand of what happens to an object's shape when it is sheared and then observed with a telescope.\n", 14 | "3. Be able to use the `galsim` package to simulate this process.\n", 15 | "\n", 16 | "In order to run the code in this tutorial, you will need the following packages installed locally\n", 17 | "\n", 18 | " - `galsim`\n", 19 | " - `numpy`\n", 20 | " - `matplotlib`\n", 21 | " \n", 22 | "I suggest using `conda`. You can run the command\n", 23 | "\n", 24 | "```\n", 25 | "conda install galsim numpy metaplotlib\n", 26 | "```\n", 27 | "\n", 28 | "in your environment to get things going.\n" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "id": "73d866d4", 34 | "metadata": {}, 35 | "source": [ 36 | "## A. Galaxy Surface Brightness Profiles\n", 37 | "\n", 38 | "They key quantity we'll be working with in this tutorial is the *surface brightness* of a galaxy. This quantity can roughly be thought of as the value of an image of the galaxy as a function of the position in the galaxy. (There are of course more technical definitions but this definition is good enough for now.) We will call this quantity $I(x,y)$.\n", 39 | "\n", 40 | "Here is a small snippet of code that draws a \"galaxy\" whose surface brightness profile is 1 if the distance from the center is less than 0.5 and zero otherwise." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "id": "85b0bc64", 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "import numpy as np\n", 51 | "import matplotlib.pyplot as plt\n", 52 | "\n", 53 | "%matplotlib notebook" 54 | ] 55 | }, 56 | { 57 | "cell_type": "code", 58 | "execution_count": null, 59 | "id": "17ee6a3e", 60 | "metadata": { 61 | "scrolled": false 62 | }, 63 | "outputs": [], 64 | "source": [ 65 | "# this is the array of pixel edges so that pixel i\n", 66 | "# zero goes from edges[i] to edges[i+1]\n", 67 | "edges = np.linspace(-1, 1, 257)\n", 68 | "\n", 69 | "# we use pixel centers to compute the profile and draw images\n", 70 | "# the indexing edges[1:] removes the first element of the array\n", 71 | "# the indexing edges[:-1] removes the last element of the array\n", 72 | "# thus we are averaging the edges of each pixel to get the center\n", 73 | "vals = (edges[:-1] + edges[1:])/2\n", 74 | "\n", 75 | "# this command constructs 2d arrays of the x and y values across the image\n", 76 | "# it is very useful!\n", 77 | "x, y = np.meshgrid(vals, vals)\n", 78 | "\n", 79 | "# now we compute the profile \n", 80 | "# and set it to 1 if the radius is less than 0.5\n", 81 | "im = np.zeros((256, 256))\n", 82 | "r = np.sqrt(x*x + y*y)\n", 83 | "msk = r < 0.5\n", 84 | "im[msk] = 1\n", 85 | "\n", 86 | "# this set of commands makes a plot of the object\n", 87 | "fig, axs = plt.subplots()\n", 88 | "axs.pcolormesh(edges, edges, im, cmap='viridis')\n", 89 | "axs.grid(False)\n", 90 | "axs.set_aspect(1.0)\n", 91 | "axs.set_xlabel(\"x\")\n", 92 | "axs.set_ylabel(\"y\")" 93 | ] 94 | }, 95 | { 96 | "cell_type": "markdown", 97 | "id": "4178111e", 98 | "metadata": {}, 99 | "source": [ 100 | "## B. How do we define the shape of a galaxy?\n", 101 | "\n", 102 | "There are a lot of ways to think about this question, but let's start with the simplest and one of the most common, weighted moments. We define the *moments* of the surface brightness profile as:\n", 103 | "\n", 104 | "$$\n", 105 | "\\langle M_x\\rangle = \\frac{\\int I(x,y) x}{\\int I(x,y)}\n", 106 | "$$\n", 107 | "\n", 108 | "\n", 109 | "$$\n", 110 | "\\langle M_y\\rangle = \\frac{\\int I(x,y) y}{\\int I(x,y)}\n", 111 | "$$\n", 112 | "\n", 113 | "\n", 114 | "$$\n", 115 | "\\langle M_{xx}\\rangle = \\frac{\\int I(x,y) (x - M_x)^2}{\\int I(x,y)}\n", 116 | "$$\n", 117 | "\n", 118 | "\n", 119 | "$$\n", 120 | "\\langle M_{xy}\\rangle = \\frac{\\int I(x,y) (x - M_x)(y-M_y)}{\\int I(x,y)}\n", 121 | "$$\n", 122 | "\n", 123 | "\n", 124 | "$$\n", 125 | "\\langle M_{yy}\\rangle = \\frac{\\int I(x,y) (y - M_y)^2}{\\int I(x,y)}\n", 126 | "$$\n", 127 | "\n", 128 | "For those of you who have studied statistics, you'll recognize these as the mean ($M_x$, $M_y$), variance ($M_{xx}$, $M_{yy}$), and covariance ($M_{xy}$). However, you do not need to be deeply familiar with these concepts in order to continue with this tutorial." 129 | ] 130 | }, 131 | { 132 | "cell_type": "markdown", 133 | "id": "e71c67f7", 134 | "metadata": {}, 135 | "source": [ 136 | "### Exercise 1: Compute the Moments!\n", 137 | "\n", 138 | "In this exercise, we are going to compute the moments of simplified version of a Gaussian surface brightness profile. This simplified profile is\n", 139 | "\n", 140 | "$$\n", 141 | "I(x,y) \\propto \\exp\\left(-\\frac{(x - \\mu_x)^2}{2\\sigma_x^2} - \\frac{(y - \\mu_y)^2}{2\\sigma_y^2}\\right)\n", 142 | "$$\n", 143 | "\n", 144 | "The exercise is to do the following.\n", 145 | "\n", 146 | "1. Write a function to compute the profile at a given position.\n", 147 | "2. Use `matplotlib` to visualize that profile.\n", 148 | "3. Compute the moments of the profile.\n", 149 | "\n", 150 | "Use values $\\sigma_x = 0.5$, $\\sigma_y = 0.25$, $\\mu_x=-0.2$, $\\mu_y=0.3$. \n", 151 | "\n", 152 | "Here are some questions to answer:\n", 153 | "\n", 154 | "1. As you change $\\sigma_x$ and $\\sigma_y$, what relationship do you notice between the moments and those values?\n", 155 | "2. As you change $\\mu_x$ and $\\mu_y$, what relationship do you notice between the moments and those values?\n", 156 | "3. What is the behavior of $M_{xy}$ and why do you think it does that?\n", 157 | "\n", 158 | "To help get you started, I have put some code below that sets up the coordinates and 2d arrays of them for doing computations and plots. You do not have to use these, but you may find them helpful." 159 | ] 160 | }, 161 | { 162 | "cell_type": "code", 163 | "execution_count": null, 164 | "id": "540f27d4", 165 | "metadata": {}, 166 | "outputs": [], 167 | "source": [ 168 | "x_min = -2.0\n", 169 | "x_max = 2.0\n", 170 | "y_min = -2.0\n", 171 | "y_max = 2.0\n", 172 | "edges = np.linspace(x_min, x_max, 513)\n", 173 | "vals = (edges[:-1] + edges[1:])/2\n", 174 | "x, y = np.meshgrid(vals, vals)" 175 | ] 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "id": "f0a69399", 180 | "metadata": {}, 181 | "source": [ 182 | "Also, remember that an integral $V = \\int_L^H dx f(x)$ can be estimated from a sum:\n", 183 | "\n", 184 | "$$\n", 185 | "V \\approx \\sum_i f(x_i) \\Delta x\n", 186 | "$$\n", 187 | "\n", 188 | "where the $x_i$ have been defined on a suitable grid of points from $L$ to $H$ with spacing $\\Delta x$." 189 | ] 190 | }, 191 | { 192 | "cell_type": "code", 193 | "execution_count": null, 194 | "id": "72fa43c1", 195 | "metadata": {}, 196 | "outputs": [], 197 | "source": [ 198 | "# do your work here!" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "id": "d1be988e", 204 | "metadata": {}, 205 | "source": [ 206 | "## C. Using `galsim`\n", 207 | "\n", 208 | "It turns out, as you may have seen from the previous exercise, that coding up surface brightness profiles with parameters and making grids of positions for visualizing them is quite tedious. There must be a better way!\n", 209 | "\n", 210 | "For us, this better way is a package called `galsim`. I am going to show you how to do simple things in `galsim`.\n", 211 | "\n", 212 | "First, let's make an object with a Gaussian surface brightness profile in `galsim`." 213 | ] 214 | }, 215 | { 216 | "cell_type": "code", 217 | "execution_count": null, 218 | "id": "b02bf9f4", 219 | "metadata": {}, 220 | "outputs": [], 221 | "source": [ 222 | "import galsim\n", 223 | "\n", 224 | "obj = galsim.Gaussian(sigma=1)" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "id": "deceaf55", 230 | "metadata": {}, 231 | "source": [ 232 | "Here we have specified the width of the Gaussian to be 1.0 (`sigma=1`). In `galsim`, sufrace brightness profiles tend to be symmetric so in fact we have specified `sigma` for both the x- and y-axis here.\n", 233 | "\n", 234 | "Notice that this object has a bunch of properties and methods you can use:" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "id": "924a21e9", 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [ 244 | "# the python built-in function `dir` is super helpful\n", 245 | "# it shows you all of the things attached to your object (these things are called attributes)\n", 246 | "# here are some conventions to watch out for\n", 247 | "#\n", 248 | "# - attributes that start with two underscores `__` usually have special behavior defined by the python language\n", 249 | "# - attributes that start with one underscore `_` are usually meant to be private to the object you are using\n", 250 | "# This means generally you should not ever access them or use them in any way.\n", 251 | "\n", 252 | "print(dir(obj))" 253 | ] 254 | }, 255 | { 256 | "cell_type": "markdown", 257 | "id": "1fafcbb4", 258 | "metadata": {}, 259 | "source": [ 260 | "To access one of these attributes, you simply put a period after your object and then the attributes name:" 261 | ] 262 | }, 263 | { 264 | "cell_type": "code", 265 | "execution_count": null, 266 | "id": "561d737d", 267 | "metadata": {}, 268 | "outputs": [], 269 | "source": [ 270 | "print(obj.sigma)" 271 | ] 272 | }, 273 | { 274 | "cell_type": "markdown", 275 | "id": "eee60041", 276 | "metadata": {}, 277 | "source": [ 278 | "We'll come back to some of these later.\n", 279 | "\n", 280 | "In `galsim`, objects like `obj` above are abstract representations of a surface brightness profile. What this kind of weird statement means in practice is that you can do a lot of manipulations on the object (e.g., adding them together, etc.) and `galsim` tracks those operations, generating new objects for you along the way. Finally, when you need an image of your object, then `galsim` does the bulk of the computational work.\n", 281 | "\n", 282 | "To draw an image of an object, you have to specify what kind of coordinate grid to use. There are a multitude of ways to specify this so we are going to stick with a single convention: We will always use square images with a fixed, equal size for the grid spacing in each direction. Drawing an object with `galsim` according to this convention is done as follows:" 283 | ] 284 | }, 285 | { 286 | "cell_type": "code", 287 | "execution_count": null, 288 | "id": "473f9498", 289 | "metadata": {}, 290 | "outputs": [], 291 | "source": [ 292 | "im = obj.drawImage(nx=53, ny=53, scale=0.25)" 293 | ] 294 | }, 295 | { 296 | "cell_type": "markdown", 297 | "id": "c4ab57c8", 298 | "metadata": {}, 299 | "source": [ 300 | "Here we have specified 53 pixels on the x-axis (`nx=53`), 53 pixels on the y-axis (`ny=53`), and a grid spacing (also known as the pixel scale in `galsim`) of 0.25 (`scale=0.25`). \n", 301 | "\n", 302 | "Notice that the Python type of the returned image `im` is not a `numpy` array like you got above:" 303 | ] 304 | }, 305 | { 306 | "cell_type": "code", 307 | "execution_count": null, 308 | "id": "48291fdd", 309 | "metadata": {}, 310 | "outputs": [], 311 | "source": [ 312 | "print(type(im))" 313 | ] 314 | }, 315 | { 316 | "cell_type": "markdown", 317 | "id": "bc67efe8", 318 | "metadata": {}, 319 | "source": [ 320 | "`galsim` has its own image type which can be very useful for certain operations. However, many times one simply wants the underlying `numpy` array. This array can be accessed via the `.array` attribute:" 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "execution_count": null, 326 | "id": "f3e0eef5", 327 | "metadata": {}, 328 | "outputs": [], 329 | "source": [ 330 | "print(type(im.array))" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "id": "d70fb5a8", 336 | "metadata": {}, 337 | "source": [ 338 | "### Exercise 2: Drawing objects with `galsim`\n", 339 | "\n", 340 | "With all of this information in hand, we can now get to the actual exercise!\n", 341 | "\n", 342 | "1. Draw a Gaussian object with `galsim` using a scale of 0.1, `sigma=0.2`, and an image size `nx=ny=7`. Make a plot of the image.\n", 343 | "2. Do the same as 1, but set the image size to `nx=ny=8`.\n", 344 | "3. According to the code, `galsim` is supposed to be drawing the same object with the same grid size. Compare the plots from 1 and 2. What do you notice is different about the images? Why do you think this is the case?\n", 345 | "4. For odd-sized images, what is the formula that relates where the center of the object lands to the size of the image?" 346 | ] 347 | }, 348 | { 349 | "cell_type": "code", 350 | "execution_count": null, 351 | "id": "744e41e1", 352 | "metadata": {}, 353 | "outputs": [], 354 | "source": [ 355 | "# do your work here!\n" 356 | ] 357 | }, 358 | { 359 | "cell_type": "markdown", 360 | "id": "ed4d09e7", 361 | "metadata": {}, 362 | "source": [ 363 | "Given the results above, it is clear that even and odd sized images can cause counter intuitive differences in how `galsim` draws (or sometimes we say \"renders\") the objects surface brightness profile. To avoid ambiguities, we will stick with odd-sized images (e.g., 53 but not 52) in the rest of the tutorial." 364 | ] 365 | }, 366 | { 367 | "cell_type": "markdown", 368 | "id": "79711cf6", 369 | "metadata": {}, 370 | "source": [ 371 | "### Exercise 3: Compute the moments of an object drawn with `galsim`\n", 372 | "\n", 373 | "Let's combine the two skills we learned above to simulate an object with `galsim` and then compute the moments. This task will require us to understand what `(x,y)` values to assign to the different image locations from galsim. There are many ways we could go about this, but let's stick to something simple. Every `galsim` image has a `bounds` attribute:" 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": null, 379 | "id": "911d1e99", 380 | "metadata": {}, 381 | "outputs": [], 382 | "source": [ 383 | "obj = galsim.Gaussian(sigma=1)\n", 384 | "im = obj.drawImage(nx=53, ny=53, scale=0.25)\n", 385 | "\n", 386 | "print(dir(im.bounds))" 387 | ] 388 | }, 389 | { 390 | "cell_type": "markdown", 391 | "id": "2af4bbef", 392 | "metadata": {}, 393 | "source": [ 394 | "From the bounds attribute, we can extract the pixel grid locations:" 395 | ] 396 | }, 397 | { 398 | "cell_type": "code", 399 | "execution_count": null, 400 | "id": "6640f8e9", 401 | "metadata": {}, 402 | "outputs": [], 403 | "source": [ 404 | "print(\"xmin:\", im.bounds.xmin)\n", 405 | "print(\"xmax:\", im.bounds.xmax)\n", 406 | "print(\"ymin:\", im.bounds.ymin)\n", 407 | "print(\"ymax:\", im.bounds.ymax)" 408 | ] 409 | }, 410 | { 411 | "cell_type": "markdown", 412 | "id": "57949caf", 413 | "metadata": {}, 414 | "source": [ 415 | "The pixel grid locations are the integer indexes of the pixels as one would count them. These indexes are of course different from the $(x,y)$coordinate values. Interestingly, `galsim` starts the pixel grid at 1 instead of starting things at 0 like python. In order to keep things consistent with python, we are always going to subtract 1 from the `galsim` pixel grid locations.\n", 416 | "\n", 417 | "With this information, we can now generate the pixel grid locations for the image:" 418 | ] 419 | }, 420 | { 421 | "cell_type": "code", 422 | "execution_count": null, 423 | "id": "5cf9d469", 424 | "metadata": {}, 425 | "outputs": [], 426 | "source": [ 427 | "xmin_zero = im.bounds.xmin - 1\n", 428 | "xmax_zero = im.bounds.xmax - 1\n", 429 | "ymin_zero = im.bounds.ymin - 1\n", 430 | "ymax_zero = im.bounds.ymax - 1\n", 431 | "\n", 432 | "x, y = np.meshgrid(np.arange(xmin_zero, xmax_zero+1), np.arange(ymin_zero, ymax_zero+1))" 433 | ] 434 | }, 435 | { 436 | "cell_type": "markdown", 437 | "id": "de446bb2", 438 | "metadata": {}, 439 | "source": [ 440 | "Here note that we had to add 1 to the max value since `np.arange` follows the python convention of indexes from `i` to `j` spanning the values `i`, `i+1`, `i+2`, ..., `j-1`. We can verify this by examining the shape and the values of our arrays:" 441 | ] 442 | }, 443 | { 444 | "cell_type": "code", 445 | "execution_count": null, 446 | "id": "d058cd59", 447 | "metadata": {}, 448 | "outputs": [], 449 | "source": [ 450 | "print(x.shape, y.shape) # both are 53x53 which is the size of the image" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": null, 456 | "id": "37bb000a", 457 | "metadata": {}, 458 | "outputs": [], 459 | "source": [ 460 | "print((x.min(), x.max()), (y.min(), y.max())) # both are (0, 52) since galsim went from 1 to 53 and we subtracted 1" 461 | ] 462 | }, 463 | { 464 | "cell_type": "markdown", 465 | "id": "e7e46acd", 466 | "metadata": {}, 467 | "source": [ 468 | "Finally, we have generated the pixel grid locations, but not the $(x,y)$ coordinates. To do that, we need to account for the grid spacing by multiplying:" 469 | ] 470 | }, 471 | { 472 | "cell_type": "code", 473 | "execution_count": null, 474 | "id": "64217b24", 475 | "metadata": {}, 476 | "outputs": [], 477 | "source": [ 478 | "x = x * 0.25 # this factor is what we passed for `scale` when drawing the image above\n", 479 | "y = y * 0.25" 480 | ] 481 | }, 482 | { 483 | "cell_type": "markdown", 484 | "id": "fbeca524", 485 | "metadata": {}, 486 | "source": [ 487 | "We can print the range again to see the final results:" 488 | ] 489 | }, 490 | { 491 | "cell_type": "code", 492 | "execution_count": null, 493 | "id": "b4196a67", 494 | "metadata": {}, 495 | "outputs": [], 496 | "source": [ 497 | "print((x.min(), x.max()), (y.min(), y.max()))" 498 | ] 499 | }, 500 | { 501 | "cell_type": "markdown", 502 | "id": "3782ed23", 503 | "metadata": {}, 504 | "source": [ 505 | "With this information, do the following:\n", 506 | "\n", 507 | "1. Use Galsim to draw a Gaussian via the following command `obj = galsim.Gaussian(sigma=2).shear(g1=0.1, g2=0.1)` and then draw this object on a grid that is 513 pixels on each side with a scale of 0.1.\n", 508 | "2. Use the example above to compute the coordinates of the image pixels.\n", 509 | "3. Use your new knowledge of object moments to compute all five of the moments defined previously.\n", 510 | "4. Make a plot of the image. In this case, it may be easier to use the function `axs.imshow` since we have not computed the pixel coordinate edges. This command will always produce a plot in pixel grid locations.\n", 511 | "5. What is different about the moments compared to the object we drew previously in terms of how the object looks and $M_{xy}$?" 512 | ] 513 | }, 514 | { 515 | "cell_type": "markdown", 516 | "id": "875a1998", 517 | "metadata": {}, 518 | "source": [ 519 | "## D. Galaxies, Shears, and Telescopes\n", 520 | "\n", 521 | "Stepping back a bit, let's learn about how our Universe relates to the things we did above. \n", 522 | "\n", 523 | "**galaxies**: We introduced the concept of an object's surface brightness profile above, defined as the value of an image of the object at a given location $(x,y)$. Of course, in our Universe, objects don't actually follow the simple Gaussian profile we used above. Instead, galaxies form out of the gravitational collapse of dark matter and baryons into dark matter halos with galaxies at their centers. This process is exceedingly complex. For our work, instead of using the true surface brightness profile of a galaxy, we use simplified approximations, like a Gaussian profile. These approximations help us in multiple ways, including making our computations faster in some cases and making the results of our computations easier to understand. So, TL;DR, the surface brightness profiles we used above and below are simplified models for true galaxies that help make science easier to do. The issue of how we ensure our methods will work on galaxies the universe generates is a serious one, but something we won't discuss here.\n", 524 | "\n", 525 | "**shears**: Now that we know what a surface brightness profile is and we have some moments computed, we are ready to think about shear. A *shear* is a specific mathematical transformation of a galaxy surface brightness profile. (Instead of coding this transformation up ourselves, we are going to have `galsim` do this for us. Yay!) This operation is highly relevant to our own Universe due to an effect called *weak gravitational lensing*. It turns out one of the foundational predictions of General Relativity is that the paths of light rays change in response to the presence of matter (e.g., us and everything else) near them. Weak gravitational lensing is this effect computed from all of the matter in the Universe along the line-of-sight from us to distant galaxies observed by telescopes. Fractionally, this effect is indeed very weak, causing a relative change of only 1% in the moments of the surface brightness profiles of distant galaxies. The goal of a shape measurement method is to extract the amplitude of this shear, which carries information about the matter along the line of sight and other properties of the Universe.\n", 526 | "\n", 527 | "**telescopes**: After the light from a galaxy has passed through various structures along the line-of-sight, it enters our atmosphere and then the telescope. The combination of the atmosphere and the telescope blurs the image of the galaxy. We characterize this blurring through a quantity called the *point-spread function* or PSF. The PSF describes how light from a single, point-like object is spread out over the image. A galaxy is an extended object, which you can roughly think of as composed of an infinite number of points of light coming towards. To predict what the galaxy will look like after the atmosphere and telescope, we can take the surface brightness at each point on the galaxy, spread it out according to the PSF, and then deposit that spread out light onto our image. Mathematically, this operation is called a *convolution* and we say that the galaxy has been *convolved with the PSF*. Let's not worry about the mathematical details of convolutions for now. Instead, we will let `galsim` do them for us!\n", 528 | "\n", 529 | "### Convolving a Galaxy with the PSF using `galsim`\n", 530 | "\n", 531 | "Let's learn how to use `galsim` to convolve a galaxy with the PSF. First we make the galaxy and the psf:" 532 | ] 533 | }, 534 | { 535 | "cell_type": "code", 536 | "execution_count": null, 537 | "id": "0bbbea84", 538 | "metadata": {}, 539 | "outputs": [], 540 | "source": [ 541 | "gal = galsim.Gaussian(sigma=1)\n", 542 | "psf = galsim.Gaussian(sigma=0.5)" 543 | ] 544 | }, 545 | { 546 | "cell_type": "markdown", 547 | "id": "4a69b284", 548 | "metadata": {}, 549 | "source": [ 550 | "Now we can use the `galsim.Convolve` object to build the convolution like this:" 551 | ] 552 | }, 553 | { 554 | "cell_type": "code", 555 | "execution_count": null, 556 | "id": "a6af9954", 557 | "metadata": {}, 558 | "outputs": [], 559 | "source": [ 560 | "observed_gal = galsim.Convolve(gal, psf)" 561 | ] 562 | }, 563 | { 564 | "cell_type": "markdown", 565 | "id": "9054586f", 566 | "metadata": {}, 567 | "source": [ 568 | "Importantly, the convolved object has many of the same attributes as the original objects, including `.drawImage`." 569 | ] 570 | }, 571 | { 572 | "cell_type": "code", 573 | "execution_count": null, 574 | "id": "3a614cc5", 575 | "metadata": {}, 576 | "outputs": [], 577 | "source": [ 578 | "print(dir(observed_gal))" 579 | ] 580 | }, 581 | { 582 | "cell_type": "markdown", 583 | "id": "c0310382", 584 | "metadata": {}, 585 | "source": [ 586 | "### Exercise 4: Understanding Convolutions\n", 587 | "\n", 588 | "For this exercise do the following:\n", 589 | "\n", 590 | "1. Using a pixel grid with 53 pixels on each side and scale of 0.25, plot the PSF, galaxy and the convolution of the two. Use the objects in the example convolution we just discussed.\n", 591 | "2. Measure the moments of each of the PSF, galaxy, and the convolution.\n", 592 | "3. Plot the moment $M_{xx}$ of the convolved object as a function of the moment $M_{xx}$ of the PSF as you change the PSF size `sigma` from 0.5 to 1.0. What is the relationship between the two?\n", 593 | "4. Repeat 3 but this time changing the size `sigma` of the galaxy from 1 to 2 and measuring $M_{xx}$ of the convolved object with respect to the moment $M_{xx}$ of the input galaxy. Before you make the plot, write down what do you expect to see and why?\n", 594 | "5. Repeat 1 through 4, but this time using `gal = galsim.Exponential(half_light_radius=1)` and varying the `half_light_radius` instead of `sigma`. Do you see the same relationship? Why can you conclude about how this might approximately work in general?" 595 | ] 596 | }, 597 | { 598 | "cell_type": "markdown", 599 | "id": "27a8d698", 600 | "metadata": {}, 601 | "source": [ 602 | "#### Great job! If you've gotten here you can now simulate and draw useful combinations of objects and PSFs for weak lensing. Yay!" 603 | ] 604 | }, 605 | { 606 | "cell_type": "code", 607 | "execution_count": null, 608 | "id": "3fe4c3b9", 609 | "metadata": {}, 610 | "outputs": [], 611 | "source": [] 612 | } 613 | ], 614 | "metadata": { 615 | "kernelspec": { 616 | "display_name": "Python [conda env:anl] *", 617 | "language": "python", 618 | "name": "conda-env-anl-py" 619 | }, 620 | "language_info": { 621 | "codemirror_mode": { 622 | "name": "ipython", 623 | "version": 3 624 | }, 625 | "file_extension": ".py", 626 | "mimetype": "text/x-python", 627 | "name": "python", 628 | "nbconvert_exporter": "python", 629 | "pygments_lexer": "ipython3", 630 | "version": "3.8.10" 631 | } 632 | }, 633 | "nbformat": 4, 634 | "nbformat_minor": 5 635 | } 636 | -------------------------------------------------------------------------------- /shape_measurement_102.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "030f1314", 6 | "metadata": {}, 7 | "source": [ 8 | "# Shape Measurement 102\n", 9 | "\n", 10 | "This notebook has the first part of a small-ish tutorial on galaxy shape measurement. The goals of this tutorial are as follows\n", 11 | "\n", 12 | "1. Become familiar with how weak lensing shear changes the shapes of objects.\n", 13 | "2. Understand the basic principles behind `metacalibration`-based shape measurements.\n", 14 | "3. Learn how to use `metadetection` and `ngmix` to measure the shapes of objects.\n", 15 | "\n", 16 | "**Please do the 101 tutorial before doing this one!**\n", 17 | "\n", 18 | "In order to run the code in this tutorial, you will need the following packages installed locally\n", 19 | "\n", 20 | " - `galsim`\n", 21 | " - `numpy`\n", 22 | " - `matplotlib`\n", 23 | " - `metadetect`\n", 24 | " - `ngmix`\n", 25 | " - `meds`\n", 26 | " - `joblib`\n", 27 | " - `schwimmbad`\n", 28 | " - `tqdm`\n", 29 | " - `joblib`\n", 30 | " - `fitsio`\n", 31 | " - `pyyaml`\n", 32 | " - `des-sxdes`\n", 33 | " \n", 34 | "I suggest using `conda`. You can run the command\n", 35 | "\n", 36 | "```\n", 37 | "conda install galsim numpy metaplotlib metadetect ngmix meds joblib schwimmbad tqdm fitsio pyyaml des-sxdes\n", 38 | "```\n", 39 | "\n", 40 | "in your environment to get things going.\n" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "id": "4178111e", 46 | "metadata": {}, 47 | "source": [ 48 | "## A. How does a weak lensing shear change the shape of an object?\n", 49 | "\n", 50 | "Remember last time we learned about the moments of a surface brightness profile $I(x,y$\n", 51 | "\n", 52 | "$$\n", 53 | "\\langle M_x\\rangle = \\frac{\\int I(x,y) x}{\\int I(x,y)}\\\\\n", 54 | "\\langle M_y\\rangle = \\frac{\\int I(x,y) y}{\\int I(x,y)}\\\\\n", 55 | "\\langle M_{xx}\\rangle = \\frac{\\int I(x,y) (x - M_x)^2}{\\int I(x,y)}\\\\\n", 56 | "\\langle M_{xy}\\rangle = \\frac{\\int I(x,y) (x - M_x)(y-M_y)}{\\int I(x,y)}\\\\\n", 57 | "\\langle M_{yy}\\rangle = \\frac{\\int I(x,y) (y - M_y)^2}{\\int I(x,y)}\n", 58 | "$$\n", 59 | "\n", 60 | "It turns out, one can compute specific combinations of these moments that are particularly sensitive to the shearing operation we discussed last time. These combinations are\n", 61 | "\n", 62 | "$$\n", 63 | "\\langle M_{r}\\rangle = M_{xx} + M_{yy}\\\\\n", 64 | "\\langle M_{+}\\rangle = M_{xx} - M_{yy}\\\\\n", 65 | "\\langle M_{\\times}\\rangle = 2 M_{xy}\\\\\n", 66 | "$$\n", 67 | "\n", 68 | "which can then be combined into \n", 69 | "\n", 70 | "$$\n", 71 | "e_1 = \\frac{M_{+}}{M_{r}}\\\\\n", 72 | "e_2 = \\frac{M_{\\times}}{M_{r}}\\\\\n", 73 | "$$\n", 74 | "\n", 75 | "I have not made the mathematical origin of these particular combinations clear at all, so this should be confusing. The key thing is to remember that one can compute moments and then use the equations above to compute the shape of the object $(e_1, e_2)$. Below we'll explore how shears change these moments.\n", 76 | "\n", 77 | "### Exercise 1: How does it change?\n", 78 | "\n", 79 | "In this exercise, we are going to draw objects with several different shears, make plots of them, and then explore how the object's moments change with shear. \n", 80 | "\n", 81 | "To do this, we'll want to use `galsim` to shear an object. This can be done as follows:" 82 | ] 83 | }, 84 | { 85 | "cell_type": "code", 86 | "execution_count": null, 87 | "id": "01bba66a", 88 | "metadata": {}, 89 | "outputs": [], 90 | "source": [ 91 | "import galsim\n", 92 | "\n", 93 | "obj = galsim.Gaussian(sigma=1)\n", 94 | "sheared_obj = obj.shear(g1=0.01, g2=0.0)" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "id": "e70f0f44", 100 | "metadata": {}, 101 | "source": [ 102 | "We have called the `.shear` command which returns a new object that has been sheared by the amount given. If we print the underlying representation of this object out, we can see that `galsim` is applying a transformation to the original object, which hints at what a shear actually means:" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "id": "c633144c", 109 | "metadata": {}, 110 | "outputs": [], 111 | "source": [ 112 | "print(repr(sheared_obj))" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "id": "0c99dec7", 118 | "metadata": {}, 119 | "source": [ 120 | "With these preliminaries out of the way, do the following.\n", 121 | "\n", 122 | "1. For each of the shears $(g_1, g_2) = \\{(0, 0), (0.1, 0), (-0.1, 0), (0, 0.1), (0, -0.1)\\}$, use `galsim` to make the sheared object and draw an image of it. Describe to yourself how the object changes in relationship to the shear you applied. Why might we have used $+$ and $\\times$ when describing the moments above?\n", 123 | "2. For a grid of shears on the 1-axis with $g_1$ going from -0.2 to 0.2 in steps of 0.025, measure the shapes `(e_1, e_2)` as defined above. Set $g_2=0$. What do you notice about the relationship between the shear and the shape when the shear is small?\n", 124 | "3. Repeat 2 but with the roles of $g_1$ and $g_2$ swapped so that $g_1=0$ and the grid of shears is in $g_2$.\n", 125 | "4. What is your guess for the generic relationship between shapes and shears?" 126 | ] 127 | }, 128 | { 129 | "cell_type": "code", 130 | "execution_count": null, 131 | "id": "69add774", 132 | "metadata": {}, 133 | "outputs": [], 134 | "source": [ 135 | "# do your work here" 136 | ] 137 | }, 138 | { 139 | "cell_type": "markdown", 140 | "id": "c92bc4b4", 141 | "metadata": {}, 142 | "source": [ 143 | "## B. Counterfactual Images and Metacalibration\n", 144 | "\n", 145 | "In the last exercise, you should have discovered how the measured shape of an object changes as it is sheared by different amounts. In particular, for small or *weak* gravitational shears, we expect that\n", 146 | "\n", 147 | "$$\n", 148 | "e_i \\approx e^{0} + R_i g_i\n", 149 | "$$\n", 150 | "\n", 151 | "where $R_i$ is called the *response*. The response is formally defined as a derivative $\\left.de/dg\\right.|_{g=0}$, but this is a detail that is not needed to complete this tutorial. They key bit of information here is that the response is the slope of the linear relationship between the observed shape and the underlying shear, for small shears.\n", 152 | "\n", 153 | "### Exercise 2: Compute the response numerically with `galsim`!\n", 154 | "\n", 155 | "In this exercise, we will use galsim to simulate the same object with different shears. Then by measuring the moments of those simulated profiles, we will estimate the response.\n", 156 | "\n", 157 | "1. Using `galsim`, create a Gaussian object with `sigma=1` and a shear of `g1=0.01` and `g2=0`. Draw this object onto a grid of pixels with `scale=0.2` and measure the shape $(e_1, e_2)$.\n", 158 | "2. Repeat 2 above but with `g1=-0.01` and `g2=0.0`. \n", 159 | "3. Combine the results from 1 and 2 to compute $R_1$ via a finite-difference derivative\n", 160 | "$$\n", 161 | "R_1 = \\frac{e_1^{g1=+h} - e_1^{g1=-h}}{2h}\n", 162 | "$$\n", 163 | "This is called a central difference derivative is typically an accurate but efficient method for this kind of numerical work.\n", 164 | "\n", 165 | "4. Do the same steps 1-3 but with the roles of the shears/shapes on the 1-axis swapped with those on the 2-axis. In other words, use shears with `g1=0`, `g2=+/-0.01` and compute $R_2$ by using a central difference derivative with $e_2$.\n", 166 | "5. Now make the same object but apply the shear `g1=0.02` and `g2=0.0`. Measure its shape with the same pixel scale. Estimate the underlying shear applied to the object by inverting the linear relationship above\n", 167 | "$$\n", 168 | "g_{i}^{\\rm obs} \\approx e_i/R_i\n", 169 | "$$\n", 170 | "How close did you get?" 171 | ] 172 | }, 173 | { 174 | "cell_type": "code", 175 | "execution_count": null, 176 | "id": "56f1e184", 177 | "metadata": {}, 178 | "outputs": [], 179 | "source": [ 180 | "# do your work here" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "id": "83083d00", 186 | "metadata": {}, 187 | "source": [ 188 | "## C. Metadetection/Metacalibration in real data with `ngmix`\n", 189 | "\n", 190 | "Congrats! By completing exercise 2 of this tutorial you have coded up an extremely simple, but non-trivial form of the `metacalibration` weak lensing shape measurement technique!\n", 191 | "\n", 192 | "Moving from here to applying this to real observed galaxies involves a lot of complications. Namely\n", 193 | "\n", 194 | "1. We don't know the true underlying surface brightness profile of real galaxies. Thus we cannot directly compute the response from the unsheared surface brightness profile like we did above.\n", 195 | "2. Real galaxies are observed through telescopes and we need to account for this effect when computing the response.\n", 196 | "3. Images of real galaxies have noise in them. If we take an image with noise in it and shear the image, we will get correlated noise which can cause bias in our shape measurements.\n", 197 | "\n", 198 | "The solutions to these problems are the subject of years of research, so I'll give you the answers.\n", 199 | "\n", 200 | "1. Instead of using the true surface brightness profile of an object, we use the observed image of the object itself. We can do this because the shear applied by nature is small, so that the difference between the response computed using the observed object and that using the true object is even smaller. This process generates a *counter-factual* image of the object at another shear, hence the title above.\n", 201 | "2. We measure the PSF of the telescope and atmosphere using stars around the galaxy. We can then use this observed PSF to remove these effects via a process called deconvolution. This process cannot fully recover the galaxy as it would look without the telescope and atmosphere, but it can remove enough of these effects for our purposes.\n", 202 | "3. We can apply special noise corrections to our images which account for the correlated noise. The details of this process are unimportant except to say that if we don't we see 5% to 10% biases our measurements.\n", 203 | "\n", 204 | "All of these algorithms have been coded up into two packages, `metadetect` and `ngmix`. These are the main tools you will be using. The next exercise introduces them to you and generates the first kinds of simulations you'll run to test shape measurement.\n", 205 | "\n", 206 | "### Exercise 3: Making Sims w/ `ngmix` and `galsim`\n", 207 | "\n", 208 | "In this exercise, we will write a function to create a simple simulation of a single object using `ngmix` and `galsim`. Your goal tasks are to\n", 209 | "\n", 210 | "1. Fill in the parts of the function below to make the sim.\n", 211 | "2. Run the simulation with different shears and different psf sizes, measuring the output shape using the moments code above. Try psf sizes from 0.8 to 1.2 in steps of 0.1. What is the relationship between the slope of output shape vs input shape and the size of the PSF?\n", 212 | "\n", 213 | "The simulation code will output an `ngmix.Observation` object. You can read more about them in the docs: https://github.com/esheldon/ngmix/blob/master/ngmix/observation.py#L66. Here is a short set of things you can do:\n", 214 | "\n", 215 | "```\n", 216 | "# get the image in this observation as a numpy array\n", 217 | "obs.image \n", 218 | "\n", 219 | "# get the image of the PSF as a numpy array\n", 220 | "obs.psf.image\n", 221 | "```\n", 222 | "\n", 223 | "Note that these objects are read only, so you cannot modify the images etc once they are made without special syntax." 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "id": "33df84cf", 230 | "metadata": {}, 231 | "outputs": [], 232 | "source": [ 233 | "import ngmix\n", 234 | "import numpy as np\n", 235 | "\n", 236 | "def sim_func(*, g1, g2, seed, psf_fwhm):\n", 237 | " # this is an RNG you can use to draw random numbers as needed\n", 238 | " # always use this RNG and not np.random directly\n", 239 | " # doing this makes sure the code is reproducible\n", 240 | " rng = np.random.RandomState(seed=seed)\n", 241 | "\n", 242 | " # make an Exponential object in galsim with a half light radius of 0.5\n", 243 | " gal = # fill this in\n", 244 | " \n", 245 | " # make a Gaussian object in galsim with a fwhm of `psf_fwhm`\n", 246 | " psf = # fill this in\n", 247 | " \n", 248 | " # apply the input shear `g1`, `g2` to the galaxy `gal`\n", 249 | " sheared_gal = # fill this in\n", 250 | " \n", 251 | " # here we are going to apply a random shift to the object's center\n", 252 | " dx, dy = 2.0 * (rng.uniform(size=2)-0.5) * 0.2\n", 253 | " sheared_gal = sheared_gal.shift(dx, dy)\n", 254 | " \n", 255 | " # convolve the sheared galaxy with the psf\n", 256 | " obj = # fill this in\n", 257 | " \n", 258 | " # render the object and the PSF on odd sized images of 53 pixels on a side with \n", 259 | " # a pixel scale of 0.2\n", 260 | " obj_im = # fill this in\n", 261 | " psf_im = # fill this in\n", 262 | " \n", 263 | " # now we are going to add noise to the object image and setup the ngmix data\n", 264 | " cen = (53-1)/2\n", 265 | " nse_sd = 1e-5\n", 266 | " nse = rng.normal(size=obj_im.array.shape, scale=nse_sd)\n", 267 | " nse_im = rng.normal(size=obj_im.array.shape, scale=nse_sd)\n", 268 | " jac = ngmix.jacobian.DiagonalJacobian(scale=0.2, row=cen+dy/0.2, col=cen+dx/0.2)\n", 269 | " psf_jac = ngmix.jacobian.DiagonalJacobian(scale=0.2, row=cen, col=cen)\n", 270 | " \n", 271 | " # we have to add a little noise to the PSf to make things stable\n", 272 | " target_psf_s2n = 500.0\n", 273 | " target_psf_noise = np.sqrt(np.sum(psf_im.array ** 2)) / target_psf_s2n \n", 274 | " psf_obs = ngmix.Observation(\n", 275 | " image=psf_im.array,\n", 276 | " weight=np.ones_like(psf_im.array)/target_psf_noise**2,\n", 277 | " jacobian=psf_jac,\n", 278 | " )\n", 279 | " \n", 280 | " # here we build the final observation\n", 281 | " obj_obs = ngmix.Observation(\n", 282 | " image=obj_im.array + nse,\n", 283 | " noise=nse_im,\n", 284 | " weight=np.ones_like(nse_im) / nse_sd**2,\n", 285 | " jacobian=psf_jac,\n", 286 | " bmask=np.zeros_like(nse_im, dtype=np.int32),\n", 287 | " ormask=np.zeros_like(nse_im, dtype=np.int32),\n", 288 | " psf=psf_obs,\n", 289 | " )\n", 290 | "\n", 291 | " return obj_obs" 292 | ] 293 | }, 294 | { 295 | "cell_type": "code", 296 | "execution_count": null, 297 | "id": "ab3377f6", 298 | "metadata": {}, 299 | "outputs": [], 300 | "source": [ 301 | "# do your work here" 302 | ] 303 | }, 304 | { 305 | "cell_type": "markdown", 306 | "id": "fa58a7a3", 307 | "metadata": {}, 308 | "source": [ 309 | "### Exercise 4: Run metadetect!\n", 310 | "\n", 311 | "In this exercise you will run metadetection for real! Yay! Before we do that, we need to understand a few things about the results. We are aiming to measure the shear and are using the estimate $g^{obs}$ as defined above. There is a standard way to characterize the bias in this estimate, which in our notation is\n", 312 | "\n", 313 | "$$\n", 314 | "g^{obs} = (1+m)g^{true} + c\n", 315 | "$$\n", 316 | "\n", 317 | "Here $m$ is called the *multiplicative bias* and is the fractional error in our shear estimate. The quantity $c$ is called the additive bias and denotes any mean bias in our shear estimate. For metadetect, we expect \n", 318 | "\n", 319 | "$$\n", 320 | "m\\approx 0.4\\times10^{-3}\\\\\n", 321 | "c\\approx 0\n", 322 | "$$\n", 323 | "\n", 324 | "Due to measurement noise in the simulations, we usually check that these conditions are true to within the $3\\sigma$ error bar.\n", 325 | "\n", 326 | "In the python source file `mdet_meas_tools.py`, I have put a bunch of code that you will use to run the simulation you made above through metadetection. The main functions to worry about are\n", 327 | "\n", 328 | " - `run_mdet_sims` - used to make the measurements\n", 329 | " - `write_sim_data` - used to write the simulation data to disk\n", 330 | " - `read_sim_data` - used to read the simulation data from disk\n", 331 | " - `estimate_m_and_c` - used to estimate $m$ and $c$ from a sim\n", 332 | " \n", 333 | "Read the doc strings on these functions for instructions on how to use them. Note that you need to pass a python dictionary for the `sim_kwargs` input and it should have the form `{\"psf_fwhm\": 0.9}`.\n", 334 | "\n", 335 | "Do the following:\n", 336 | "\n", 337 | "1. Use `run_mdet_sims` to run metadetection on your sim and verify that $m$ and $c$ meet the requirements above.\n", 338 | "2. Use the utilities to write your data to and then read your data from disk.\n", 339 | "3. Run your data through `estimate_m_and_c` and confirm you get the same answer as in 1." 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": null, 345 | "id": "294148d0", 346 | "metadata": {}, 347 | "outputs": [], 348 | "source": [ 349 | "# do your work here!" 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "id": "fd43baa4", 356 | "metadata": {}, 357 | "outputs": [], 358 | "source": [] 359 | } 360 | ], 361 | "metadata": { 362 | "kernelspec": { 363 | "display_name": "Python [conda env:anl] *", 364 | "language": "python", 365 | "name": "conda-env-anl-py" 366 | }, 367 | "language_info": { 368 | "codemirror_mode": { 369 | "name": "ipython", 370 | "version": 3 371 | }, 372 | "file_extension": ".py", 373 | "mimetype": "text/x-python", 374 | "name": "python", 375 | "nbconvert_exporter": "python", 376 | "pygments_lexer": "ipython3", 377 | "version": "3.7.8" 378 | } 379 | }, 380 | "nbformat": 4, 381 | "nbformat_minor": 5 382 | } 383 | --------------------------------------------------------------------------------