├── .DS_Store ├── .gitattributes ├── .gitignore ├── LICENSE ├── README.md ├── README.pdf ├── cosi_2016_balloon_data ├── README.md ├── imaging │ └── COSIpy_RL_Imaging_Crab_Balloon.ipynb └── spectral-fit │ └── spectral_fit.ipynb ├── cosipy-classic ├── .DS_Store ├── COSIpy_dc1.py ├── COSIpy_tools_dc1.py ├── fit_dc1.py ├── gam_lecr_pdcompatible.spc ├── priors_dc1.py └── response_dc1.py ├── data_products ├── .DS_Store ├── .gitattributes ├── 1809keV_imaging_response.npz ├── 511keV_imaging_response.npz ├── Al26_10xFlux_Only.inc1.id1.extracted.tra.gz ├── Al26_10xFlux_and_Ling.inc1.id1.extracted.tra.gz ├── CenA_BG_10x.tra.gz ├── CenA_only_10x.tra.gz ├── Continuum_imaging_response.npz ├── Crab_BG_10x.tra.gz ├── Crab_COSIBalloonData.tra.gz ├── Crab_only_10x.tra.gz ├── CygX1_BG_10x.tra.gz ├── CygX1_only_10x.tra.gz ├── DC1_combined_10x.tra.gz ├── GC511_10xFlux_and_Ling.inc1.id1.extracted.tra.gz ├── GC511_10xFlux_only.inc1.id1.extracted.tra.gz ├── Point_sources_10x.tra.gz ├── Point_sources_10x_BG.tra.gz ├── README.md ├── Scaled_Ling_BG_1x.npz ├── Scaled_Ling_BG_1x.tra.gz ├── Vela_BG_10x.tra.gz ├── Vela_only_10x.tra.gz └── flight_bg_all_v1_fine_6deg.npz ├── imaging ├── README.md ├── RL-DataChallenge-511keV_10xFlux-Ling.ipynb ├── RL-DataChallenge-Al26_10xFlux-Ling.ipynb ├── RL-DataChallenge-Point_Sources-10XFlux-Ling.ipynb ├── SPI_DIRBE_Orig_Knoedl_240um.fits ├── imaging_suggested_parameters.pdf ├── plots │ ├── 26Al_LingBG_1809keVresponse_RLimage.png │ ├── 26Al_noBG_1809keVresponse_RLimage.png │ ├── 511keV_LingBG_511keVresponse_RLimage.png │ ├── 511keV_noBG_511keVresponse_RLimage.png │ ├── PointSources_LingBG_continuumresponse_ebin2_RLimage.png │ └── PointSources_noBG_continuumresponse_ebin2_RLimage.png └── stanmodel.stan ├── requirements.txt ├── spectral-fit ├── .gitattributes ├── README.md └── spectral_fit.ipynb └── xspec-tutorial ├── Figures ├── effective_area.png ├── source_and_background_grp8ch_spectrum.png ├── source_and_background_spectrum.png ├── unfolded_crab_simulation.png ├── unfolded_crab_simulation_with_background.png └── unfolded_spectrum.png ├── README.md ├── area.arf ├── background.pha ├── groups8ch.dat ├── response.rmf └── source_and_background.pha /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/.DS_Store -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | *.sim filter=lfs diff=lfs merge=lfs -text 2 | *.sim.gz filter=lfs diff=lfs merge=lfs -text 3 | *.tra.gz filter=lfs diff=lfs merge=lfs -text 4 | *.evta.gz filter=lfs diff=lfs merge=lfs -text 5 | *.rsp.gz filter=lfs diff=lfs merge=lfs -text 6 | *.h5 filter=lfs diff=lfs merge=lfs -text 7 | *.rsp filter=lfs diff=lfs merge=lfs -text 8 | *.evta filter=lfs diff=lfs merge=lfs -text 9 | *.tra filter=lfs diff=lfs merge=lfs -text 10 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Byte-compiled / optimized / DLL files 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # C extensions 7 | *.so 8 | 9 | # Distribution / packaging 10 | .Python 11 | build/ 12 | develop-eggs/ 13 | dist/ 14 | downloads/ 15 | eggs/ 16 | .eggs/ 17 | lib/ 18 | lib64/ 19 | parts/ 20 | sdist/ 21 | var/ 22 | wheels/ 23 | pip-wheel-metadata/ 24 | share/python-wheels/ 25 | *.egg-info/ 26 | .installed.cfg 27 | *.egg 28 | MANIFEST 29 | 30 | # PyInstaller 31 | # Usually these files are written by a python script from a template 32 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 33 | *.manifest 34 | *.spec 35 | 36 | # Installer logs 37 | pip-log.txt 38 | pip-delete-this-directory.txt 39 | 40 | # Unit test / coverage reports 41 | htmlcov/ 42 | .tox/ 43 | .nox/ 44 | .coverage 45 | .coverage.* 46 | .cache 47 | nosetests.xml 48 | coverage.xml 49 | *.cover 50 | *.py,cover 51 | .hypothesis/ 52 | .pytest_cache/ 53 | 54 | # Translations 55 | *.mo 56 | *.pot 57 | 58 | # Django stuff: 59 | *.log 60 | local_settings.py 61 | db.sqlite3 62 | db.sqlite3-journal 63 | 64 | # Flask stuff: 65 | instance/ 66 | .webassets-cache 67 | 68 | # Scrapy stuff: 69 | .scrapy 70 | 71 | # Sphinx documentation 72 | docs/_build/ 73 | 74 | # PyBuilder 75 | target/ 76 | 77 | # Jupyter Notebook 78 | .ipynb_checkpoints 79 | 80 | # IPython 81 | profile_default/ 82 | ipython_config.py 83 | 84 | # pyenv 85 | .python-version 86 | 87 | # pipenv 88 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 89 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 90 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 91 | # install all needed dependencies. 92 | #Pipfile.lock 93 | 94 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow 95 | __pypackages__/ 96 | 97 | # Celery stuff 98 | celerybeat-schedule 99 | celerybeat.pid 100 | 101 | # SageMath parsed files 102 | *.sage.py 103 | 104 | # Environments 105 | .env 106 | .venv 107 | env/ 108 | venv/ 109 | ENV/ 110 | env.bak/ 111 | venv.bak/ 112 | 113 | # Spyder project settings 114 | .spyderproject 115 | .spyproject 116 | 117 | # Rope project settings 118 | .ropeproject 119 | 120 | # mkdocs documentation 121 | /site 122 | 123 | # mypy 124 | .mypy_cache/ 125 | .dmypy.json 126 | dmypy.json 127 | 128 | # Pyre type checker 129 | .pyre/ 130 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "[]" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright [yyyy] [name of copyright owner] 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # COSI Data Challenge 2022 2 | 3 | Welcome to the first COSI Data Challenge! This is the first of many COSI Data Challenges to be released on a yearly basis in preparation for the launch of the COSI Small Explorer mission ([Tomsick et al. 2019](https://ui.adsabs.harvard.edu/abs/2019BAAS...51g..98T/abstract)) in 2027. The main goals of the COSI Data Challenges are to facilitate the development of the COSI data pipeline and analysis tools, and to provide resources to the astrophysics community to become familiar with COSI data. This first COSI Data Challenge was funded through NASA’s Astrophysics Research and Analysis (APRA) for the release of high-level analysis tools for the COSI Balloon instrument ([Kierans et al. 2017](https://ui.adsabs.harvard.edu/abs/2017arXiv170105558K/abstract)), and thus the COSI Balloon model and flight data will be the focus this year. Future COSI Data Challenges will be released for the SMEX mission with increasingly more sophisticated tools and a larger range of astrophysical models and simulated sources each year. By the time we’re ready to fly COSI, we will have simulated all of the main science objectives, developed the tools required to analyze each case, and have educated a broader community to perform the analyses. 4 | 5 | ## Installation Instructions 6 | 7 | ### Already have COSItools installed: 8 | If you already have the COSItools installed on your computer, type `cosi` to navigate to the COSItools directory and activate the COSI python environment ("python-env"). Clone the cosi-data-challenge-1 repository. 9 | 10 | The file [requirements.txt](requirements.txt) lists all of the dependencies for running the data challenge, and they can all be installed using pip: `pip install -r requirements.txt`. 11 | 12 | You will also need to add the [cosipy-classic](cosipy-classic) directory to your python path, e.g. `export PYTHONPATH=$PYTHONPATH:/path/to/COSItools/cosi-data-challenge-1/cosipy-classic`. 13 | 14 | Some of the data products are large and we are using the Git Large File Server. You will need to install git-lfs: https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage. After Git LFS has been successfully installed, navigate to your local cosi-data-challenge-1 directory and `git lfs pull` to download all of the data products. If this step is not performed, the file names will appear locally, but they will only be placeholders and the size will be only a few hundred kB. The total size of the data products should be 4.6 GB. 15 | 16 | You should now be able to start a python session, or open one of the provided notebooks, and import the COSIpy_dc1 module, i.e. `from COSIpy_dc1 import *`, without issue. If you have any issues with the initialization, please see **Getting Help** below. 17 | 18 | ### New to COSItools: 19 | Head to the feature/initialsetup branch of the cosi-setup Git repository and follow the readme guide: https://github.com/cositools/cosi-setup/tree/feature/initialsetup. Note that when making the installation you must include the option "--extras=cosi-data-challenge-1". This installation will include MEGAlib, ROOT, Geant4, Git LFS, and all packages in the [requirements.txt](requirements.txt) file. After successful installation (this will take time), you will want to activate the cosi python environment by typing `cosi`. 20 | 21 | You will also need to add the [cosipy-classic](cosipy-classic) directory to your python path, e.g. `export PYTHONPATH=$PYTHONPATH:/path/to/COSItools/cosi-data-challenge-1/cosipy-classic`. 22 | 23 | ## Getting Help 24 | 25 | Since this is the first release of COSIpy, there will likely be issues. We hope to get feedback from the community on what worked well, what didn't, and how can we improve for next time. Please submit a New Issue in git if you have issues with the code. If you have general feedback, or need further assistance, please reach out to the COSI Data Challenge team lead: [Chris Karwin: christopher.m.karwin@nasa.gov](mailto:christopher.m.karwin@nasa.gov). 26 | 27 | 28 | ## Getting Started 29 | The COSI pipeline tools, COSItools, are divided into two programs (see figure below): MEGAlib and COSIpy. MEGAlib (https://megalibtoolkit.com and https://github.com/zoglauer/megalib) is the Medium Energy Gamma-ray Astronomy Library, which is a general purpose tool that is state-of-the-art for MeV telescopes. MEGAlib performs the data calibration, event identification, reconstruction, as well as detailed source simulations. There are some high-level analysis tools in MEGAlib, but most of the COSI science analysis will be performed in COSIpy. COSIpy is the user-facing, python-based, high-level analysis tool for COSI data. 30 | 31 | Screen Shot 2022-10-16 at 10 19 29 PM 32 | 33 | This Data Challenge will serve to introduce the community to COSIpy and general Compton telescope analysis. We have prepared Jupyter Notebooks to walk the user through the analyses which are provided under [spectral-fit](spectral-fit) and [imaging](imaging); however, we suggest reading through the below description before attempting the notebooks. 34 | 35 | COSIpy was first developed by Thomas Siegert in 2019 to perform 511 keV image analysis from the 2016 COSI balloon flight ([Siegert et al. 2020](https://ui.adsabs.harvard.edu/abs/2020ApJ...897...45S/abstract)). Since then, it has been used for point source imaging and spectral extraction (e.g. [Zoglauer et al. 2021](https://ui.adsabs.harvard.edu/abs/2021arXiv210213158Z/abstract)), Aluminum-26 (Al-26) spectral fitting ([Beechert et al. 2022](https://ui.adsabs.harvard.edu/abs/2022ApJ...928..119B/abstract)) and Al-26 imaging, all using data from the COSI Balloon 2016 flight. These analyses and the current Data Challenge use what we refer to as “COSIpy-classic.” The team is currently working on improved response handling and streamlined tools built from the bottom up, and the new and improved COSIpy will be the focus of the coming years’ Data Challenges! With that in mind, there are still known issues and limitations with COSIpy-classic that we will call out throughout this work. 36 | 37 | ## The Simulated Data 38 | 39 | For the first Data Challenge, we wanted to give the users a basic look at COSI data analysis, so we chose 3 straightforward examples based on the COSI science goals:
40 | 1. Extracting the spectra from the Crab, Cen A, Cyg X-1, and Vela.
41 | 2. Imaging bright point sources, such as the Crab and Cyg-X1.
42 | 3. Imaging diffuse emission from 511 keV and the Al-26 1.8 MeV gamma-ray line.
43 | 44 | For each of these examples, we have provided a detailed description of the simulated sources and data products in the [data_products](data_products) directory. Each of the sources was simulated at 10x the astrophysical flux since the balloon flight had limited observation time, and because there were multiple detector failures during the balloon flight which reduced the effective area significantly. 45 | 46 | ## General Compton Telescope Analysis Procedure 47 | 48 | Analysis of MeV data is challenging due to high backgrounds and the complicated instrument responses of Compton telescopes like COSI. Thus, here we provide a basic description of Compton telescope data analysis as an introduction for new users before diving into the Data Challenge analysis. For those who are new to Compton telescopes, we encourage you to read the review: [Kierans, Takahashi & Kanbach 2022](https://ui.adsabs.harvard.edu/abs/2022arXiv220807819K/abstract). 49 | 50 | Compton telescopes perform single-photon detection, a technique whereby each photon is measured as a number of energy deposits in the detector volume (shown in blue in the figure below). The order of interactions must first be sequenced in time to reconstruct the scattering path and determine the original direction of the photon. This photon origin is constrained to a circle on the sky (a so-called Compton event circle) defined by an opening angle equal to the Compton scattering angle $(\phi)$ of the first interaction. A typical event in a Compton telescope is shown in the figure below: a photon Compton scatters twice in the detector volume and undergoes a photoelectric absorption as its third and final interaction. The position of each interaction is recorded and the incident energy of the photon is fully contained in the active detector volume. The event sequencing and reconstruction are performed in the MEGAlib tool; we will start our analysis with events that are already defined by their total energy deposit in the detector $(E_0)$, and the Compton scattering angle. 51 | 52 | Screen Shot 2022-10-16 at 11 00 42 PM 53 | 54 | As shown in the figure above, a Compton telescope can image a source distribution by finding the overlap of event circles from multiple source photons. A point-source image can be recovered by performing deconvolution techniques. This List-mode imaging approach ([Zoglauer 2005](https://megalibtoolkit.com/documents/Zoglauer_PhD.pdf)) is implemented in MEGAlib and can be used for strong point sources, for example in laboratory measurements. However, this approach assumes a simplified detector response and cannot be used for the most sensitive analyses. 55 | 56 | When performing analyses of astrophysical sources with high backgrounds, a more sophisticated description of the data space is required. This data space, along with the fundamentals of modern Compton telescope analysis techniques, was pioneered by the COMPTEL mission ([Schönfelder et al. 1993](https://ui.adsabs.harvard.edu/abs/1993ApJS...86..657S/abstract) & [Diehl et al. 1992](https://ui.adsabs.harvard.edu/abs/1992NASCP3137...95D/abstract)) and is sometimes referred to as the COMPTEL Data Space, or simply the Compton Data Space (CDS). In the CDS, each event is defined by a minimum of 5 parameters. Three parameters describe the geometry of each event: the Compton scatter angle $(\phi)$ of the first interaction, and the polar and azimuthal angles of the scattered photon direction $(\chi, \psi)$. These angles as defined in instrument coordinates are shown in the figure below. The other two parameters in the CDS are the total photon energy $(E_0)$ and the event time $(t)$, but these are not always explicitly written. 57 | 58 | Screen Shot 2022-10-16 at 11 02 38 PM 59 | 60 | The scattering angles $(\phi, \chi, \psi)$ span the three orthogonal axes of the CDS. As photons from a point source at location $(\chi_0, \psi_0)$ scatter in the detector, the CDS is populated in the shape of a cone whose apex lies at the source location, as shown in the figure on the right. This is the point spread function of a Compton telescope. The opening angle of the CDS cone is 90º since the Compton scatter angle is equal (within measurement error) to the deviation of the scattered photon direction. An extended source will appear as a broadened cone. The more familiar Angular Resolution Measure (ARM) for Compton telescopes is a 1-dimensional projection of the width of the CDS cone walls, representing the angular resolution. 61 | 62 | All analyses with COSIpy start with reconstructed events defined in a photon list (MEGAlib’s .tra files). After reading in the data from the .tra file, the first step of any analysis is to bin the data into the CDS. For this Data Challenge, we are using 6º bin sizes for each of the scattering angles. This is because the angular resolution of the COSI Balloon instrument is ~6º at best, and smaller bins would be more computationally demanding. We are also using 10 energy bins for the continuum analyses, 2 hour time bins for the spectral analysis, and 30 minute time bins for the imaging analyses. For the narrow-line sources, such as 511 keV and Al-26, we use only 1 energy bin centered on the gamma-ray line of interest. 63 | 64 | As a visual representation of the data (D) in the CDS, see the figure below. The three axes are the 3 scatter angles $(\phi, \chi, \psi)$. Each $(\phi, \chi, \psi)$ bin contains the number of events, or counts, with that geometry. These counts are illustrated with the red color fill (this is just a random distribution and not representative of what real data looks like in the CDS). The CDS is filled in this way for each energy and time bin, represented by the subscript $E,t$ in the figure. In COSIpy-classic, $\chi$ and $\psi$ are binned into in 1145 FISBEL bins. FISBEL stands for Fixed Integral Square Bins in Equi-Longitude, and is MEGAlib’s spherical axis binning that has approximately equal solid angle for each pixel. 65 | 66 | Screen Shot 2022-10-16 at 11 03 38 PM 67 | 68 | Now that we have a better understanding of the Compton telescope data space and have binned our data in the CDS, we’re ready to perform spectral analysis and imaging with COSI. There are 3 key pieces needed: 69 | 1. Response Matrix 70 | 2. Sky Model 71 | 3. Background Model 72 | 73 | We will describe these components here and explain how they are used for general fitting procedures with COSI data. When running through the analysis notebooks, pay attention to when these are initialized. 74 | 75 | ### Response Matrix 76 | 77 | The response matrix (R) for Compton telescopes represents the probability that a photon with energy $E_0$ originating from Galactic coordinates $(l,b)$ interacts in the detector and is recorded as an event with measured energy E’ and scattering angles $(\phi,\chi,\psi)$. The probability distribution is normalized to match the total effective area of the instrument, defining the conversion from counts to physical parameters. The matrix that encodes this information is 2 dimensions larger $(l,b)$ than the CDS and describes the transformation between image space and the CDS, taking into account the accurate response of the instrument. We build the response matrix through extensive MEGAlib simulations: the instrument is situated at the center of an isotropically-emitting source and thus records photons from all surrounding source locations. With a sufficiently large number of incident photons, we can populate the entire CDS of possible scattering angles and thereby obtain a representation of each incident photon in the CDS. 78 | 79 | Screen Shot 2022-10-16 at 11 05 26 PM 80 | 81 | ### Sky Model 82 | 83 | For forward-folding analysis methods, we assume a source sky distribution, referred to as the sky (or signal) model (S). The sky model in image space can be, for example, a simple point source or a complicated diffuse model. The sky model also contains spectral and polarization assumptions about the source. This model is convolved with the instrument response matrix and knowledge of the instrument aspect pointing during observations to determine the representation of the sky model in the CDS. Below, we show this for a simple point source and regain the expected cone-shape in the CDS. 84 | 85 | Screen Shot 2022-10-16 at 11 06 10 PM 86 | 87 | ### Background Model 88 | 89 | We require an accurate estimate of the backgrounds during observations. This background model (B) can be achieved in a number of ways. One approach is to use the measured flight data from source-starved regions. Alternatively, one can perform full bottom-up simulations of the gamma-ray background at balloon-flight altitudes, including atmospheric contamination and instrumental activation. For this first Data Challenge, we use the latter approach. The simulation is further described in [data_products](data_products); in future Data Challenges, we will employ multiple background-model approaches. 90 | 91 | With the background model generated from simulations, we subsequently bin it in the same CDS that we used for the data and the source model. We have already performed this step for you, and have provided a .npz file, which is a zipped numpy array of the background simulation in the CDS. 92 | 93 | Screen Shot 2022-10-16 at 11 06 45 PM 94 | 95 | ### Fitting General Principle 96 | 97 | Finally, we have all of our components and can perform the analysis. When model fitting, we can optionally free multiple spectral, polarization, and location parameters. For simplicity, in this Data Challenge we will only fit for the amplitudes of the source and background models ($\alpha$ and $\beta$, respectively) that best describe the measured data: **D = $\alpha$ S + $\beta$ B**. This fit is shown schematically in the figure below and is performed for each energy bin and time bin independently. This procedure for spectral fitting and spatial model fitting is maximizing the likelihood in the CDS. 98 | 99 | Screen Shot 2022-10-16 at 11 07 20 PM 100 | 101 | If we don’t know what the source should look like, instead of providing a sky model we can remain agnostic to an assumed source distribution and perform image deconvolution. The data is represented as the sum of the response convolved with the source sky distribution (which we want to find) and the background: 102 | 103 | Screen Shot 2022-10-16 at 11 08 00 PM 104 | 105 | Since the response is non-invertible, we must use iterative deconvolutions to reveal the sky distribution. In COSIpy-classic, we introduce you to a modified Richardson-Lucy algorithm, which is a special case of the expectation maximization algorithm developed for COMPTEL’s images of diffuse gamma-ray emission ([Knödlseder et al. 1999](https://ui.adsabs.harvard.edu/abs/1999A%26A...345..813K/abstract)). The Richardson-Lucy algorithm starts from an initial isotropic (i.e. unbiased) image, and iteratively compares the data in the CDS to the sky distribution convolved with the transpose of the response matrix R: 106 | 107 | Screen Shot 2022-10-16 at 11 08 28 PM 108 | 109 | This will evolve into the maximum likelihood solution. There are other image deconvolution techniques that will be used for COSI imaging, and those will be introduced in subsequent Data Challenges. 110 | 111 | In order to gain intuition for how to interpret this equation, consider that we are trying to find the number of photons in an image pixel given our model expectation $M = R \cdot S + B$. Here, $M$ are the model counts, calculated from a linear combination of the instrumental background $B$ and the sky morphology $S$, to which the image response function $R$ is applied to convert from image space to data space.
112 | In such a counting experiment, the measured data $D$ given the model expectation $M$ follows a Poisson distribution, so that the likelihood of measuring $D$ photons is given by $P(D|M) = \prod \frac{\exp(-M)M^D}{D!}$. The product here is taken over all sky pixels.
113 | Taking the logarithm of the likelihood and plugging in the model expectation results in $L \equiv \ln P(D|S,B) = \sum \left(-(R \cdot S + B) + D \ln(R \cdot B) \right)$.
114 | Assuming, for the moment, that the background is known, we can optimise the likelihood with respect to the wanted sky signal $S$, such that
115 | $0 = \nabla_S L = -I \cdot R^T + \frac{D \cdot R^T}{R \cdot S + B} = \left(1 - \frac{D}{R \cdot S + B} \right) \cdot R^T$.
116 | Finally, this equation can be solved iteratively, (for each pixel simultaneously) by assuming a starting sky distribution (map) for $S$, similar to Newton's method for example, which will result in the RL equation above. 117 | 118 | ## Next Steps 119 | 120 | Now that you have a better understanding of the general approach to Compton telescope analyses, you’re ready to start on the analysis examples. First, we recommend you review the detailed descriptions of the simulations in the [data_products](data_products) directory. The easiest analysis is the spectral fitting, and we recommend you start there: [spectral_fit](spectral-fit). The Richardson-Lucy imaging is computationally intensive, and still largely hard-coded in this release, and thus we recommend you work through these notebooks second: [imaging](imaging). Please note that you should use a workstation with a large memory allocation for the imaging analyses, if one is available to you. A personal laptop with only 16 GB of memory, for example, will be limiting. Finally, we have also included some of the COSI Balloon data for analysis of the Crab Nebula as seen during the 2016 flight. After you’ve run through the spectral fitting and image deconvolution with the simulated data, you should be ready to analyze this real flight data: [balloon data](cosi_2016_balloon_data)! 121 | 122 | As mentioned previously, please don't hesitate to reach out to the COSI Data Challenge team if you have any questions, concerns, issues, or suggestions. Email [Chris Karwin](mailto:christopher.m.karwin@nasa.gov). 123 | 124 | ## Bug Report 125 | 126 | * COSIpy classic requires pystan version 2.19.1.1., which has compatibility issues with Ubuntu 22.04 and currently cannot be installed. This only impacts the imaging notebooks with the simulated data. They can still be ran without modification, and will produce qualitatively similar images for the point sources and 511; however, the Al26 notebook will not work without pytan. In order to install COSITools on Ubuntu 22.04 you must remove pystan from requirements.txt. 127 | 128 | * January 20, 2023: The Al-26 notebook was tested and executes in its current form on Ubuntu 20.04. We've heard reports that the image deconvolution fails at early iterations (~10) on Mac M1 (and does not run on Ubuntu 22.04; see pystan note above). This is under investigation. Note that the other imaging notebooks should still be working ok. Please attempt this notebook as you wish, try changing parameters in the algorithm, or refer to the 511 keV imaging notebook for another example of diffuse imaging. 129 | -------------------------------------------------------------------------------- /README.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/README.pdf -------------------------------------------------------------------------------- /cosi_2016_balloon_data/README.md: -------------------------------------------------------------------------------- 1 | # COSI 2016 Balloon Data 2 | 3 | Here we will perform a spectral fit and imaging of the Crab using the 2016 COSI balloon data. The analyses will mostly follow the notebooks used for the simulated data. For more details on the steps of the spectral analysis, see the main spectral-fit README [(here)](../../spectral-fit/README.md), and likewise for the imaging analysis README [(here)](../../imaging/README.md). This README summarizes the key differences that are needed for analyzing the balloon data. 4 | 5 | ## The Balloon Flight 6 | COSI launched from Wanaka, New Zealand on NASA's Super Pressure Balloon on May 16th, 2016. After a 46 day flight, COSI landed in Southern Peru. The trajectory of the mission is shown in the image below. After one full circumnavigation, COSI floated north and an anomaly in the balloon resulted in the zagged nature of the path for the second half of the flight, indicated with the red curve. For more details of the COSI Balloon instrument, see ([Kierans et al. 2017](https://ui.adsabs.harvard.edu/abs/2017arXiv170105558K/abstract)). 7 | 8 | Screen Shot 2022-10-26 at 7 06 49 PMScreen Shot 2022-10-26 at 7 09 08 PM 9 | 10 | 11 | ## Data Selection 12 | The data has been extracted using a $60^\circ$ pointing cut with respect to the position of the Crab. In practice, this means that we only use times for which the location of the Crab was within $60^\circ$ of COSI's zenith. This can be seen in the images below. The plot on the left shows the pointing in Galactic coordinates as a function of time for the simulated data (which doesn't use a pointing cut), and the plot on the right is for the balloon data, with the $60^\circ$ pointing cut. The green star shows the position of the Crab. 13 | 14 | Screen Shot 2022-10-26 at 6 40 01 PMScreen Shot 2022-10-26 at 6 29 17 PM 15 | 16 | Additionally, we do not consider the first 21 days of the flight, when the instrument background rate was relatively high and variable. Note that the Crab was mostly outside the field of view during these first 3 weeks. We also cut the last day of the mission because of high background rates. This can be seen in the plots below, which show the elevation of the Crab above COSI's horizon. The left plot is for the simulated data, using the entire flight time, and the right plot is for the balloon data with the time cuts applied. As can be seen, the Crab was in COSI's field of view mainly during the latter part of the mission. 17 | 18 | Screen Shot 2022-10-26 at 6 42 54 PMScreen Shot 2022-10-26 at 6 44 29 PM 19 | 20 | Otherwise, the data selection is identical to that of the simulated data. For more details on the 2016 COSI balloon background rates see [Siegert et al. 2020](https://iopscience.iop.org/article/10.3847/1538-4357/ab9607). 21 | 22 | 23 | ## Background 24 | For the data challenge, the simulated background is constant in time. With the actual balloon data, however, the background rate varies significantly, mostly driven by changes in the latitude (i.e. geomagnetic cutoff) and the altitude (i.e. atmospheric depth) of the balloon. Additionally, the application of the pointing cut produces drastic variation in the light curve as the Crab moves in and out of COSI's field of view. To accomodate these variations, in the fit we apply a so-called "tracer". The tracer is a factor that is applied to the background response and effectively normalizes the background rate to the actual data in each time bin. More specifically, for each time bin, the tracer is calculated by summing the counts over the energy and Compton data space dimensions, and then normalizing by the mean of this sum over all time bins. The difference between the background rates and models with and without a tracer can be seen below, for the balloon data analysis (right) and simulated data analysis (left), respectively. 25 | 26 | Screen Shot 2022-10-26 at 6 53 00 PM 27 | 28 | ## Results of the Spectral Fit 29 | 30 | Results for the Crab fit are shown below. Upper limits are shown for bins with SNR < 3. The fit will output a .dat file, where you can examine the numerical results, including the SNR of each bin. We have verified that bins 4, 5, and 6 are consistent with the physical values (based on our simulated Crab spectrum). The other bins (and most notably, bin 3) are within roughly a factor of ~2 of expectations. It is important to note, however, that the extracted spectrum has a significant dependence on the background model being used. Here we are using a highly simplified constant Ling model (with tracer). Additionally, we haven't accounted for activation backgrounds, nor other astrophysical sources (i.e. 511, $^{26}$ Al, Cen A, etc.). Thus, the precision of this result should not be over-interpreted at this point. The COSI team is currently working on developing improved response handling, background models, and analysis tools, which will be part of future Data Challenges! 31 | 32 | Screen Shot 2022-10-26 at 8 21 03 PM 33 | 34 | ## Results of the Imaging 35 | 36 | **IMPORTANT NOTE:** The color scale is in arbitrary units. The COSI team is currently working on determining the level of the flux. 37 | 38 | The primary outputs of the imaging, after 100 iterations, are shown below. Note that there is an artifact in the southern pole, where bright emission can be seen. This feature should be surpressed with more iterations. The cyan star in the map shows the known position of the Crab, which overlaps with the brightest pixel in the Richardson-Lucy image (and the dimmer surrounding pixels). 39 | 40 | Screen Shot 2022-10-26 at 8 44 09 PMScreen Shot 2022-10-26 at 8 44 24 PM 41 | 42 | 43 | Screen Shot 2022-10-27 at 1 19 24 PM 44 | 45 | 46 | -------------------------------------------------------------------------------- /cosi_2016_balloon_data/spectral-fit/spectral_fit.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "b0f27a80", 6 | "metadata": {}, 7 | "source": [ 8 | "# Welcome to spectral fitting with COSIpy classic\n", 9 | "\n", 10 | "In this notebook, we'll perform a spectral fit of the Crab nebula using the 2016 balloon flight data." 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "id": "644d001a", 16 | "metadata": {}, 17 | "source": [ 18 | "## Import packages.\n", 19 | "\n", 20 | "We're using the COSIpy classic functions in COSIpy_dc1.py, response_dc1.py, and fit_dc1.py." 21 | ] 22 | }, 23 | { 24 | "cell_type": "code", 25 | "execution_count": null, 26 | "id": "5e288c39", 27 | "metadata": {}, 28 | "outputs": [], 29 | "source": [ 30 | "from COSIpy_dc1 import *\n", 31 | "import response_dc1 as response\n", 32 | "from fit_dc1 import *" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "id": "5c0734c0", 38 | "metadata": {}, 39 | "source": [ 40 | "## Define file names." 41 | ] 42 | }, 43 | { 44 | "cell_type": "code", 45 | "execution_count": null, 46 | "id": "1f802109", 47 | "metadata": {}, 48 | "outputs": [], 49 | "source": [ 50 | "data_dir = '../../data_products' # directory containing data & response files\n", 51 | "filename_data = 'Crab_COSIBalloonData.tra.gz' # 2016 balloon data\n", 52 | "response_filename = data_dir + '/Continuum_imaging_response.npz' # detector response\n", 53 | "background_filename = data_dir + '/Scaled_Ling_BG_1x.npz' # background response\n", 54 | "background_mode = 'from file'" 55 | ] 56 | }, 57 | { 58 | "cell_type": "markdown", 59 | "id": "e4eaa5cc", 60 | "metadata": {}, 61 | "source": [ 62 | "## Define inputs." 63 | ] 64 | }, 65 | { 66 | "cell_type": "code", 67 | "execution_count": null, 68 | "id": "3dfb9466", 69 | "metadata": {}, 70 | "outputs": [], 71 | "source": [ 72 | "l_crab,b_crab = 184.55746, -5.78436 # Galactic longitude & latitude of Crab\n", 73 | "\n", 74 | "ul = 3 # SNR limit for upper limits on spectral fit" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "id": "f9554d91", 80 | "metadata": {}, 81 | "source": [ 82 | "## Read in data and define analysis object.\n", 83 | "\n", 84 | "Read in the data set and create the main cosipy-classic “analysis_data\" object, which provides various functionalities to study the specified file. This cell usually takes a few minutes to run." 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "id": "9d1673e4", 91 | "metadata": {}, 92 | "outputs": [], 93 | "source": [ 94 | "analysis_data = COSIpy(data_dir,filename_data) # create analysis object\n", 95 | "analysis_data.read_COSI_DataSet() # read in data" 96 | ] 97 | }, 98 | { 99 | "cell_type": "markdown", 100 | "id": "14c74a0e", 101 | "metadata": {}, 102 | "source": [ 103 | "# Bin the data\n", 104 | "The data are binned into time, energy, ϕ and FISBEL. FISBEL is a unique index which specifies the χ and ψ dimensions of the CDS.\n", 105 | "\n", 106 | "Calling \"get_binned_data()\" may take several minutes, depending on the size of the dataset and the number of bins. Keep an eye on memory here: if your time bins are very small, for example, this could be an expensive operation." 107 | ] 108 | }, 109 | { 110 | "cell_type": "code", 111 | "execution_count": null, 112 | "id": "61d63bc2", 113 | "metadata": {}, 114 | "outputs": [], 115 | "source": [ 116 | "#Define the bin sizes\n", 117 | "Delta_T = 7200 # time bin size in seconds\n", 118 | "energy_bin_edges = np.array([150, 220, 325, 480, 520, 765, 1120, 1650, 2350, 3450, 5000]) # energy bin edges in keV\n", 119 | "pixel_size = 6. # pixel size in degrees\n", 120 | "\n", 121 | "analysis_data.dataset.time_binning_tags(time_bin_size=Delta_T) # time binning\n", 122 | "analysis_data.dataset.init_binning(energy_bin_edges=energy_bin_edges,pixel_size=pixel_size) # energy and pixel binning\n", 123 | "analysis_data.dataset.get_binned_data() # bin data" 124 | ] 125 | }, 126 | { 127 | "cell_type": "markdown", 128 | "id": "fee98e0b", 129 | "metadata": {}, 130 | "source": [ 131 | "## Examine the shape of the binned data.\n", 132 | "\n", 133 | "The binned data are contained in \"analysis_combined.dataset.binned_data\". This is a 4-dimensional object representing the 5 dimensions of the Compton data space (time, energy, ϕ, FISBEL).\n", 134 | "\n", 135 | "This prints the shape of the binned data, the total time in the dataset, the number of time bins that have counts in them, and the number of counts in each time bin. Due to this energy range being so background dominated, the number of counts in each bin is very similar." 136 | ] 137 | }, 138 | { 139 | "cell_type": "code", 140 | "execution_count": null, 141 | "id": "9b1aff12", 142 | "metadata": { 143 | "scrolled": true, 144 | "tags": [] 145 | }, 146 | "outputs": [], 147 | "source": [ 148 | "print('Number of bins in each dimension (time, energy, ϕ, FISBEL):')\n", 149 | "print(analysis_data.dataset.binned_data.shape)\n", 150 | "print()\n", 151 | "print('Total time in dataset (s):')\n", 152 | "print(analysis_data.dataset.times.total_time)\n", 153 | "print()\n", 154 | "print('Number of populated time bins:')\n", 155 | "print(analysis_data.dataset.times.n_ph)\n", 156 | "print()\n", 157 | "print('Number of counts in each time bin: ')\n", 158 | "print(analysis_data.dataset.times.n_ph_t)" 159 | ] 160 | }, 161 | { 162 | "cell_type": "markdown", 163 | "id": "f52cf82e", 164 | "metadata": {}, 165 | "source": [ 166 | "## Plot raw spectrum & light curve." 167 | ] 168 | }, 169 | { 170 | "cell_type": "code", 171 | "execution_count": null, 172 | "id": "4c9b12d6", 173 | "metadata": {}, 174 | "outputs": [], 175 | "source": [ 176 | "analysis_data.dataset.plot_raw_spectrum()\n", 177 | "plt.xscale('log')\n", 178 | "\n", 179 | "analysis_data.dataset.plot_lightcurve()" 180 | ] 181 | }, 182 | { 183 | "cell_type": "markdown", 184 | "id": "52ac27e0", 185 | "metadata": {}, 186 | "source": [ 187 | "## Define the pointing object with the COSIpy pointing class.\n", 188 | "\n", 189 | "The pointings refer to the direction/orientation of the telescope at each point in time. This cell usually takes a few minutes to run." 190 | ] 191 | }, 192 | { 193 | "cell_type": "code", 194 | "execution_count": null, 195 | "id": "c971d763", 196 | "metadata": {}, 197 | "outputs": [], 198 | "source": [ 199 | "pointing_data = Pointing(dataset=analysis_data.dataset) # definition of pointings (balloon stability + Earth rotation)" 200 | ] 201 | }, 202 | { 203 | "cell_type": "markdown", 204 | "id": "770538ed", 205 | "metadata": {}, 206 | "source": [ 207 | "## Visualize the paths of the Crab through the field-of-view.\n", 208 | "\n", 209 | "This isn't necessary for the spectral fitting, but is illustrative for understanding the pointings and exposure of the point source." 210 | ] 211 | }, 212 | { 213 | "cell_type": "code", 214 | "execution_count": null, 215 | "id": "ef2e752e", 216 | "metadata": {}, 217 | "outputs": [], 218 | "source": [ 219 | "plt.plot(pointing_data.zpoins[:,0]+360,pointing_data.zpoins[:,1],'o', label=\"COSI zenith pointing\")\n", 220 | "plt.plot(l_crab,b_crab,'*g',markersize=10, label=\"Crab\")\n", 221 | "plt.xlabel('Longitude [deg]')\n", 222 | "plt.ylabel('Latitude [deg]')\n", 223 | "plt.legend()" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "id": "fffacde3", 230 | "metadata": {}, 231 | "outputs": [], 232 | "source": [ 233 | "analysis_data.plot_elevation([l_crab],[b_crab],['Crab'])" 234 | ] 235 | }, 236 | { 237 | "cell_type": "markdown", 238 | "id": "43966556", 239 | "metadata": { 240 | "tags": [] 241 | }, 242 | "source": [ 243 | "# Define a tracer.\n", 244 | "Since background of the balloon data varies with time, here we define a \"tracer\" for our background model. \n", 245 | "The tracer normalizes the background model to the data in each time bin. " 246 | ] 247 | }, 248 | { 249 | "cell_type": "code", 250 | "execution_count": null, 251 | "id": "552f7666-1cc5-4241-8da4-fc8d951d2fad", 252 | "metadata": { 253 | "tags": [] 254 | }, 255 | "outputs": [], 256 | "source": [ 257 | "# Define tracer\n", 258 | "tracer = np.sum(analysis_data.dataset.binned_data,axis=(1,2,3))\n", 259 | "tracer = tracer/np.mean(tracer)" 260 | ] 261 | }, 262 | { 263 | "cell_type": "markdown", 264 | "id": "fac885bd-220e-4daf-a95f-9c08ff6d03c6", 265 | "metadata": { 266 | "tags": [] 267 | }, 268 | "source": [ 269 | "# Define the BG model." 270 | ] 271 | }, 272 | { 273 | "cell_type": "code", 274 | "execution_count": null, 275 | "id": "e6994c89-aee7-4e24-817e-4b2e9d770cb8", 276 | "metadata": { 277 | "tags": [] 278 | }, 279 | "outputs": [], 280 | "source": [ 281 | "# Ling BG simulation to model atmospheric background\n", 282 | "background_data = BG(dataset=analysis_data.dataset,mode=background_mode,filename=background_filename,tracer=tracer) # read in background" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "id": "7b04eaf8", 288 | "metadata": {}, 289 | "source": [ 290 | "# Read in the Response Matrix\n", 291 | "\n", 292 | "This usually takes a few minutes." 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": null, 298 | "id": "fbb0796f", 299 | "metadata": {}, 300 | "outputs": [], 301 | "source": [ 302 | "# continuum response\n", 303 | "rsp = response.SkyResponse(filename=response_filename,pixel_size=pixel_size) # read in detector response" 304 | ] 305 | }, 306 | { 307 | "cell_type": "markdown", 308 | "id": "b4a5de88", 309 | "metadata": {}, 310 | "source": [ 311 | "## Explore the shape of the data space.\n", 312 | "\n", 313 | "The shape of the response spans (Galactic latitude $b$, Galactic longitude $\\ell$, Compton scattering angle $\\phi$, FISBEL, energy). The shape of the data and background objects span (time, energy, Compton scattering angle, FISBEL), as explained above." 314 | ] 315 | }, 316 | { 317 | "cell_type": "code", 318 | "execution_count": null, 319 | "id": "3ba4e173", 320 | "metadata": {}, 321 | "outputs": [], 322 | "source": [ 323 | "print('Shape of response matrix (b, l, ϕ, FISBEL, energy):')\n", 324 | "print(rsp.rsp.response_grid_normed_efinal.shape)\n", 325 | "print()\n", 326 | "print('Shape of binned data (time, energy, ϕ, FISBEL):')\n", 327 | "print(analysis_data.dataset.binned_data.shape)\n", 328 | "print()\n", 329 | "print('Shape of background model (time, energy, ϕ, FISBEL):')\n", 330 | "print(np.shape(background_data.bg_model))" 331 | ] 332 | }, 333 | { 334 | "cell_type": "markdown", 335 | "id": "f10f4ef2", 336 | "metadata": {}, 337 | "source": [ 338 | "# Calculate the point source response for the Crab." 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": null, 344 | "id": "114612b6", 345 | "metadata": {}, 346 | "outputs": [], 347 | "source": [ 348 | "rsp.calculate_PS_response(analysis_data.dataset,pointing_data,l_crab,b_crab,1,background=background_data,pixel_size=pixel_size,lookup=False)" 349 | ] 350 | }, 351 | { 352 | "cell_type": "markdown", 353 | "id": "b0d35fb2", 354 | "metadata": {}, 355 | "source": [ 356 | "## Plot light curves for the data, background & sky models.\n", 357 | "\n", 358 | "This is plotted for the 220-325 keV energy bin. The sky model is normalized to 1." 359 | ] 360 | }, 361 | { 362 | "cell_type": "code", 363 | "execution_count": null, 364 | "id": "086c81ae", 365 | "metadata": {}, 366 | "outputs": [], 367 | "source": [ 368 | "plt.plot(np.sum(analysis_data.dataset.binned_data[:,1,:,:],axis=(1,2)), label=\"Data\") # binned data light curve\n", 369 | "plt.plot(np.sum(background_data.bg_model_reduced[1],axis=1), label=\"Background model\") # background model\n", 370 | "plt.plot(np.sum(rsp.sky_response[1],axis=1)*1000, label=\"Sky model\") # sky model\n", 371 | "plt.xlabel('Time Bins')\n", 372 | "plt.ylabel('Counts per Time Bin')\n", 373 | "plt.legend()" 374 | ] 375 | }, 376 | { 377 | "cell_type": "markdown", 378 | "id": "d9228957", 379 | "metadata": {}, 380 | "source": [ 381 | "# Extract the spectrum for the Crab.\n", 382 | "\n", 383 | "For each energy bin individually, this determines the coefficients for the sky and background models that best match the data. It can take a few hours to run!" 384 | ] 385 | }, 386 | { 387 | "cell_type": "code", 388 | "execution_count": null, 389 | "id": "e368c0c6", 390 | "metadata": { 391 | "scrolled": true, 392 | "tags": [] 393 | }, 394 | "outputs": [], 395 | "source": [ 396 | "result_crab = fit(analysis_data.dataset,pointing_data,rsp,background_data) # create fitting object\n", 397 | "result_crab.fit(iters=2000) # perform spectral fit using emcee (uses pointing definition, background model, & point source response)" 398 | ] 399 | }, 400 | { 401 | "cell_type": "markdown", 402 | "id": "1919d52e", 403 | "metadata": {}, 404 | "source": [ 405 | "## Plot the final count spectrum of the Crab.\n", 406 | "\n", 407 | "Below is the spectrum (in counts/keV) of the Crab nebula!\n", 408 | "\n", 409 | "The extracted spectrum data is saved as a .dat file." 410 | ] 411 | }, 412 | { 413 | "cell_type": "code", 414 | "execution_count": null, 415 | "id": "7435d206", 416 | "metadata": {}, 417 | "outputs": [], 418 | "source": [ 419 | "result_crab.plot_extracted_spectrum('crab_spectrum.dat')\n", 420 | "result_crab.plot_extracted_spectrum('crab_spectrum.dat',ul=3)" 421 | ] 422 | } 423 | ], 424 | "metadata": { 425 | "kernelspec": { 426 | "display_name": "Python 3", 427 | "language": "python", 428 | "name": "python3" 429 | }, 430 | "language_info": { 431 | "codemirror_mode": { 432 | "name": "ipython", 433 | "version": 3 434 | }, 435 | "file_extension": ".py", 436 | "mimetype": "text/x-python", 437 | "name": "python", 438 | "nbconvert_exporter": "python", 439 | "pygments_lexer": "ipython3", 440 | "version": "3.8.10" 441 | } 442 | }, 443 | "nbformat": 4, 444 | "nbformat_minor": 5 445 | } 446 | -------------------------------------------------------------------------------- /cosipy-classic/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/cosipy-classic/.DS_Store -------------------------------------------------------------------------------- /cosipy-classic/COSIpy_tools_dc1.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from astropy.io import fits 3 | import matplotlib.pyplot as plt 4 | import astropy.units as u 5 | import sys 6 | import time 7 | import scipy.interpolate as interpol 8 | import matplotlib 9 | from matplotlib import ticker, cm 10 | from scipy.ndimage import gaussian_filter 11 | from tqdm import tqdm_notebook as tqdm 12 | import os, glob 13 | 14 | import pandas as pd 15 | 16 | import ROOT as M 17 | # Load MEGAlib into ROOT 18 | M.gSystem.Load("$(MEGAlib)/lib/libMEGAlib.so") 19 | # Initialize MEGAlib 20 | G = M.MGlobal() 21 | G.Initialize() 22 | 23 | 24 | 25 | def one_func(x,y,grid=False): 26 | """Returns an array of ones for any given input array (or scalar) x. 27 | :param: x Input array (or scalar, tuple, list) 28 | :param: y Optional second array or value 29 | :option: grid Standard keyword to work with RectBivarSpline 30 | """ 31 | if isinstance(x, (list, tuple, np.ndarray)): 32 | return np.ones(len(x)) 33 | else: 34 | return 1. 35 | 36 | 37 | 38 | def GreatCircle(l1,b1,l2,b2,deg=True): 39 | """ 40 | Calculate the Great Circle length on a sphere from longitude/latitude pairs to others 41 | in units of rad on a unit sphere 42 | :param: l1 longitude of point 1 (or several) 43 | :param: b1 latitude of point 1 (or several) 44 | :param: l2 longitude of point 2 45 | :param: b2 latitude of point 2 46 | :option: deg Default True to convert degree input to radians for trigonometric function use 47 | If False, radian input is assumed 48 | """ 49 | if deg == True: 50 | l1,b1,l2,b2 = np.deg2rad(l1),np.deg2rad(b1),np.deg2rad(l2),np.deg2rad(b2) 51 | 52 | return np.sin(b1)*np.sin(b2) + np.cos(b1)*np.cos(b2)*np.cos(l1-l2) 53 | 54 | 55 | 56 | def GreatCircleGrid(l1,b1,l2,b2,deg=True): 57 | """ 58 | Calculate the Great Circle length on a sphere from longitude/latitude pairs to others 59 | in units of rad on a unit sphere 60 | :param: l1 longitude of points Ai 61 | :param: b1 latitude of points Ai 62 | :param: l2 longitude of point Bj 63 | :param: b2 latitude of point Bj 64 | :option: deg Default True to convert degree input to radians for trigonometric function use 65 | If False, radian input is assumed 66 | """ 67 | if deg == True: 68 | l1,b1,l2,b2 = np.deg2rad(l1),np.deg2rad(b1),np.deg2rad(l2),np.deg2rad(b2) 69 | 70 | L1, L2 = np.meshgrid(l1,l2) 71 | B1, B2 = np.meshgrid(b1,b2) 72 | 73 | return np.sin(B1)*np.sin(B2) + np.cos(B1)*np.cos(B2)*np.cos(L1-L2) 74 | 75 | 76 | 77 | def zenaziGrid(scx_l, scx_b, scy_l, scy_b, scz_l, scz_b, src_l, src_b): 78 | """ 79 | # from spimodfit zenazi function (with rotated axes (optical axis for COSI = z) 80 | # calculate angular distance wrt optical axis in zenith (theta) and 81 | # azimuth (phi): (zenazi function) 82 | # input: spacecraft pointing directions sc(xyz)_l/b; source coordinates src_l/b 83 | # output: source coordinates in spacecraft system frame 84 | 85 | Calculate zenith and azimuth angle of a point (a source) given the orientations 86 | of an instrument (or similar) in a certain coordinate frame (e.g. galactic). 87 | Each point in galactic coordinates can be uniquely mapped into zenith/azimuth of 88 | an instrument/observer/..., by using three Great Circles in x/y/z and retrieving 89 | the correct angles 90 | 91 | :param: scx_l longitude of x-direction/coordinate 92 | :param: scx_b latitude of x-direction/coordinate 93 | :param: scy_l longitude of y-direction/coordinate 94 | :param: scy_b latitude of y-direction/coordinate 95 | :param: scz_l longitude of z-direction/coordinate 96 | :param: scz_b latitude of z-direction/coordinate 97 | :param: src_l SOURCEgrid longitudes 98 | :param: src_b SOURCEgrid latitudes 99 | 100 | """ 101 | # # make matrices for response calculation on a pre-defined grid 102 | SCZ_L, SRC_L = np.meshgrid(scz_l,src_l) 103 | SCZ_B, SRC_B = np.meshgrid(scz_b,src_b) 104 | # Zenith is the distance from the optical axis (here z) 105 | costheta = GreatCircleGrid(scz_l,scz_b,src_l,src_b) 106 | # Azimuth is the combination of the remaining two 107 | 108 | SCX_L, SRC_L = np.meshgrid(scx_l,src_l) 109 | SCX_B, SRC_B = np.meshgrid(scx_b,src_b) 110 | cosx = GreatCircle(SCX_L,SCX_B,SRC_L,SRC_B) 111 | SCY_L, SRC_L = np.meshgrid(scy_l,src_l) 112 | SCY_B, SRC_B = np.meshgrid(scy_b,src_b) 113 | cosy = GreatCircle(SCY_L,SCY_B,SRC_L,SRC_B) 114 | 115 | # theta = zenith 116 | theta = np.rad2deg(np.arccos(costheta)) 117 | # phi = azimuth 118 | phi = np.rad2deg(np.arctan2(cosx,cosy)) 119 | 120 | # make azimuth going from 0 to 360 deg 121 | if phi.size == 1: 122 | if (phi < 0): 123 | phi += 360 124 | else: 125 | phi[phi < 0] += 360 126 | 127 | return theta,phi 128 | 129 | 130 | 131 | def cashstat(data,model): 132 | return -2*np.sum(data*np.log(model)-model-(data*np.nan_to_num(np.log(data))-data)) 133 | 134 | 135 | -------------------------------------------------------------------------------- /cosipy-classic/fit_dc1.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import sys 3 | import os 4 | import matplotlib.pyplot as plt 5 | import matplotlib as mpl 6 | 7 | from tqdm.autonotebook import tqdm 8 | from IPython.display import HTML 9 | 10 | import warnings 11 | warnings.filterwarnings('ignore') 12 | 13 | import pandas as pd 14 | from shapely.geometry import Polygon 15 | 16 | import pickle 17 | 18 | # spectral fits 19 | from priors_dc1 import * 20 | import time 21 | import emcee 22 | import corner 23 | import scipy.optimize as op 24 | 25 | 26 | 27 | class fit(): 28 | """ 29 | Fitting class that includes the dataset to be analysed, the pointings, the response, and a background model. 30 | :option: bg_only: Default = False: performs a background only fit when True 31 | :option: priors: Default: uninformative (normal) priors for the sky; 32 | :option: verbose: Default = False: more verbose output 33 | """ 34 | 35 | def __init__(self, 36 | dataset, # COSIpy dataset 37 | pointings, # POINTINGS object 38 | response, # SkyResponse object for certain source position 39 | background, # BG object including cuts and tracer 40 | bg_only=False, # True performs BG only fit 41 | priors=None, # can set priors for sky components 42 | verbose=False, # more verbose output 43 | reduced=True): # use full data space 44 | 45 | # init objects 46 | self.dataset = dataset 47 | self.pointings = pointings 48 | self.response = response 49 | self.background = background 50 | 51 | self.count_data = self.reduce_dataset_CDS() # construct data set with reduced CDS (ignore bins that are always zero) 52 | 53 | self.bg_only = bg_only 54 | self.priors = priors 55 | self.verbose = verbose 56 | 57 | self.data_per_energy_bin = self.make_dictionaries_for_stan() # make dictionaries for fit with Stan 58 | 59 | 60 | def reduce_dataset_CDS(self): 61 | 62 | count_data_reduced = [] # init list of data sets per bin, will be irregularly-shaped because CDS population depends on energy 63 | 64 | for i in range(self.dataset.energies.n_energy_bins): 65 | 66 | yp_tmp = self.dataset.binned_data[:,i,:,:].reshape(self.dataset.times.n_ph,self.background.bg_model.shape[2]*self.background.bg_model.shape[3]) # reshape count data to reduce CDS if possible, this combines the 3 CDS angles into a 1D array for all times at the chosen energy 67 | 68 | yp_tmp = yp_tmp[:,self.background.calc_this[i]] # reshape count data grid the same way as background and choose only non-zero indices 69 | 70 | count_data_reduced.append(yp_tmp) # append reduced data set 71 | 72 | return count_data_reduced 73 | 74 | 75 | 76 | def make_dictionaries_for_stan(self): 77 | """ 78 | Create dictionaries that can be read in by the Stan model to fit the data 79 | """ 80 | 81 | all_dicts = [] # init dictionaries for each energy bin 82 | 83 | # loop over energy bins 84 | for i in range(self.dataset.energies.n_energy_bins): 85 | Np, Nrsp = self.background.bg_model_reduced[i].shape # initialise sizes of arrays 86 | N = Np*Nrsp # total number of data points 87 | 88 | Nsky = 1 # right now, only one sky model allowed to be fitted 89 | 90 | if np.any(self.priors == None): # standard priors scaling the initial flux as in response calculation 91 | 92 | mu_flux_scl = np.array([0.]) # mean, prior centroids for sky, we use 10 because we are ignorant; this has to be an array because it could be more than one 93 | sigma_flux_scl = np.array([1e8]) # std, same for the width (so, easily 0 but also high values possible) 94 | 95 | # priors for background model components 96 | mu_Abg = 1. # for the moment set to a useful value if bg model is ~normalised to data, initially normalised to 1, so mean would be 1, variance very large (uninformative) 97 | sigma_Abg = 1e4 # same 98 | 99 | else: 100 | 101 | # set priors yourself 102 | mu_flux_scl = np.array([self.priors[0,0]]) 103 | sigma_flux_scl = np.array([self.priors[0,1]]) 104 | 105 | mu_Abg = self.priors[1,0] # for the moment set to a useful value if bg model is ~normalised to data 106 | sigma_Abg = self.priors[1,1] # same 107 | 108 | # dictionary for data set and prior 109 | data2D = dict(N = Nrsp, # number of CDS bins 110 | Np = Np, # number of observations 111 | Nsky = Nsky, # number of sky models (now: 1) 112 | Ncuts = self.background.Ncuts, # number of background cuts / renormalisations 113 | bg_cuts = self.background.bg_cuts, # bg cuts at 114 | bg_idx_arr = self.background.idx_arr, # bg cut indices 115 | y = self.count_data[i].ravel().astype(int), # data 116 | bg_model = self.background.bg_model_reduced[i], # background model 117 | conv_sky = self.response.sky_response[i].reshape(Nsky,Np,Nrsp),# this has to be reshaped because it could be more than one 118 | mu_flux = mu_flux_scl, # priors for sky (mean) 119 | sigma_flux = sigma_flux_scl, # std 120 | mu_Abg = mu_Abg, # BG mean 121 | sigma_Abg = sigma_Abg) # std 122 | 123 | all_dicts.append(data2D) # append dictionary for energy bin 124 | 125 | return all_dicts 126 | 127 | 128 | def fit(self,iters=1000,pars=['flux','Abg']): 129 | """ 130 | Fitting COSIpy fit object of a data set with pointing definition, background model, and (now only) point source response. 131 | Fitting background only is only possible when object is initialised with bg_only = True. 132 | :option: par Parameters to save in fit object: pars=['flux','Abg','model_tot','model_bg','model_sky','ppc'], default pars=['flux','Abg'], 133 | i.e. no models will be saved, only the fitted parameters. 134 | :option: iters Number of iterations for fit, default 1000. 135 | Saves fitting results in .fit_pars, including all posterior distributions. 136 | Creates .diff_rate and .diff_rate_err (1sigma uncertainty) that includes the differential count rate in units of ph/s/keV for all energy bins 137 | """ 138 | 139 | # init arrays 140 | self.fit_pars = [] 141 | self.diff_rate = np.zeros(self.dataset.energies.n_energy_bins) 142 | self.diff_rate_err = np.zeros((self.dataset.energies.n_energy_bins,2)) 143 | 144 | self.systematics = np.array([9.61348192, 4.5787978 , 1.15368693, 1.22003529, 1, 1, 1, 1, 1, 1]) # standard systematics (from sky-only fits) 145 | 146 | for i in range(self.dataset.energies.n_energy_bins): 147 | print("working on energy bin " + str(i)) 148 | init = np.array([np.sum(self.data_per_energy_bin[i]['y'])*0.05,0.99]) # initial guess for emcee fitting (1 bg parameter only) 149 | init_var = init*1e-4 150 | 151 | # here standard emcee workflow 152 | ndim, nwalkers = 2, 10 153 | pos = [init + np.random.randn(ndim)*init_var for i in range(nwalkers)] 154 | 155 | sampler = emcee.EnsembleSampler(nwalkers, 156 | ndim, 157 | COSImodfit, 158 | args = (self.data_per_energy_bin[i]['y'], #data 159 | self.data_per_energy_bin[i]['conv_sky'].ravel(), # sky model 160 | self.data_per_energy_bin[i]['bg_model'].ravel())) # bg model) 161 | 162 | _ = sampler.run_mcmc(pos, iters, progress=True) 163 | 164 | # extract samples 165 | samples = sampler.get_chain() 166 | samplesf = sampler.flatchain 167 | n_par = 2 168 | 169 | n_samples = iters 170 | n_walkers = nwalkers 171 | 172 | burnin = int(0.5*n_samples) 173 | 174 | spec_params = np.zeros((n_par,7)) 175 | 176 | for p in range(n_par): 177 | mean_val = np.mean(samples[burnin:,:,p]) 178 | std_val = np.std(samples[burnin:,:,p]) 179 | median_val = np.median(samples[burnin:,:,p]) 180 | ub1_val = np.percentile(samples[burnin:,:,p],50+68.3/2) 181 | lb1_val = np.percentile(samples[burnin:,:,p],50-68.3/2) 182 | ub3_val = np.percentile(samples[burnin:,:,p],50+99.73/2) 183 | lb3_val = np.percentile(samples[burnin:,:,p],50-99.73/2) 184 | spec_params[p,:] = [mean_val,std_val,lb3_val,lb1_val,median_val,ub1_val,ub3_val] 185 | 186 | self.fit_pars.append(spec_params) # append fitting results 187 | 188 | norm_tmp = self.dataset.energies.energy_bin_wid[i]*2/self.response.flux_norm # calculate rates 189 | 190 | self.diff_rate[i] = self.fit_pars[i][0,4]/norm_tmp # median value is representative for spectrum 191 | 192 | # 1sigma error bars 193 | self.diff_rate_err[i,1] = self.fit_pars[i][0,5]/norm_tmp - self.diff_rate[i] # upper boundary uncertainty 194 | self.diff_rate_err[i,0] = self.fit_pars[i][0,3]/norm_tmp - self.diff_rate[i] # lower boundary uncertainty 195 | 196 | 197 | def plot_extracted_spectrum(self,save_file,ul=None,with_systematics=False,col1='black',col2='gray'): 198 | 199 | if with_systematics == True: 200 | syst_scl = self.systematics 201 | else: 202 | syst_scl = np.ones(len(self.dataset.energies.energy_bin_cen)) 203 | 204 | plt.figure(figsize=(10.24,7.68)) 205 | 206 | snr = self.diff_rate/np.nanmax(np.abs(self.diff_rate_err.T)*syst_scl,axis=0) 207 | if ul == None: 208 | plt.errorbar(self.dataset.energies.energy_bin_cen, 209 | self.diff_rate, 210 | xerr=self.dataset.energies.energy_bin_wid, 211 | yerr=np.abs(self.diff_rate_err.T)*syst_scl[None,:], 212 | fmt='o',linewidth=3,color=col1,markersize=10, 213 | label=r'COSI Data Fit ($1\sigma$)') 214 | else: 215 | # mark data points which have SNR < ul with downward arrows 216 | # snr = self.diff_rate/np.nanmax(np.abs(self.diff_rate_err.T)*syst_scl,axis=0) 217 | ul_idx = np.where(snr < ul)[0] 218 | gl_idx = np.where(snr >= ul)[0] 219 | plt.errorbar(self.dataset.energies.energy_bin_cen[gl_idx], 220 | self.diff_rate[gl_idx], 221 | xerr=self.dataset.energies.energy_bin_wid[gl_idx], 222 | yerr=np.abs(self.diff_rate_err.T)[:,gl_idx]*syst_scl[None,gl_idx], 223 | fmt='o',linewidth=3,color=col1,markersize=10, 224 | label=r'COSI Data Fit ($1\sigma$)') 225 | plt.errorbar(self.dataset.energies.energy_bin_cen[ul_idx], 226 | np.max(np.abs(self.diff_rate_err.T),axis=0)[ul_idx]*ul*syst_scl[ul_idx], 227 | xerr=self.dataset.energies.energy_bin_wid[ul_idx], 228 | yerr=np.zeros(len(ul_idx)), 229 | fmt='v',linewidth=3,color=col2,markersize=15, 230 | label=r'COSI Data Fit ({0}$\sigma$ upper limit)'.format(ul)) 231 | 232 | plt.xlabel('Energy [keV]') 233 | plt.ylabel('Counts [cnts/keV]') 234 | plt.xscale('log') 235 | plt.yscale('log') 236 | plt.xlim(150,5000) 237 | plt.legend() 238 | 239 | # Write data: 240 | save_path = os.getcwd() 241 | save_file = os.path.join(save_path,save_file) 242 | d = {"ebin_center[keV]":self.dataset.energies.energy_bin_cen,\ 243 | "xerr[keV]":self.dataset.energies.energy_bin_wid,\ 244 | "rate[ct/keV]":self.diff_rate,\ 245 | "rate_err_low[ct/keV]":np.abs(self.diff_rate_err.T)[0],\ 246 | "rate_err_high[ct/keV]":np.abs(self.diff_rate_err.T)[1],\ 247 | "SNR":snr} 248 | df = pd.DataFrame(data=d) 249 | df.to_csv(save_file,float_format='%10.5e',index=False,sep="\t",\ 250 | columns=["ebin_center[keV]","xerr[keV]","rate[ct/keV]",\ 251 | "rate_err_low[ct/keV]","rate_err_high[ct/keV]","SNR"]) 252 | 253 | def COSImodfit(theta, data, sky_model, background_model, eval=False): 254 | """ 255 | Returns: 256 | cash-stat for model fit 257 | 258 | one-parameter background model used only 259 | 260 | Parameters: 261 | :param theta: Array of to-be-fitted parameters 262 | :param other: ... 263 | """ 264 | 265 | total_model = theta[0]*sky_model + theta[1]*background_model 266 | 267 | stat = -2*np.sum(total_model - data*(np.log(total_model))) 268 | if np.isnan(stat) | (theta[0] < 0) | (theta[1] < 0): 269 | return -np.inf 270 | else: 271 | return stat 272 | 273 | 274 | -------------------------------------------------------------------------------- /cosipy-classic/priors_dc1.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | from scipy.special import erf as erf 3 | 4 | def normal_prior(par,p): 5 | mu = p[0] 6 | sigma = p[1] 7 | return -0.5*(par-mu)**2/sigma**2 8 | 9 | 10 | def uniform_prior(par,p): 11 | lo = p[0] 12 | hi = p[1] 13 | if lo <= par <= hi: 14 | return 0.0 15 | return -np.inf 16 | 17 | 18 | def truncated_normal_prior(par,p): 19 | mu = p[0] 20 | sigma = p[1] 21 | lo = p[2] 22 | hi = p[3] 23 | if lo <= par <= hi: 24 | return -0.5*(par-mu)**2/sigma**2 - np.log(sigma) - np.log(erf((hi-mu)/(np.sqrt(2)*sigma)) - erf((lo-mu)/(np.sqrt(2)*sigma))) 25 | return -np.inf 26 | -------------------------------------------------------------------------------- /data_products/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/data_products/.DS_Store -------------------------------------------------------------------------------- /data_products/.gitattributes: -------------------------------------------------------------------------------- 1 | *.gz filter=lfs diff=lfs merge=lfs -text 2 | *.npz filter=lfs diff=lfs merge=lfs -text 3 | *.tra.gz filter=lfs diff=lfs merge=lfs -text 4 | -------------------------------------------------------------------------------- /data_products/1809keV_imaging_response.npz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:0eba9a1263cb4e3819441d5a9aac789d20f74845187fa6ba0de5cde9b344638e 3 | size 22922668 4 | -------------------------------------------------------------------------------- /data_products/511keV_imaging_response.npz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:2446f527d497617eb01daf665a9f2132eec98d8115ec7136ac09b2da9e85b6fc 3 | size 14286724 4 | -------------------------------------------------------------------------------- /data_products/Al26_10xFlux_Only.inc1.id1.extracted.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:3ea8bbd0a995ec68976d115c671c88bfc47b3e9886a5fefa3c7a54799d66ec40 3 | size 2790705 4 | -------------------------------------------------------------------------------- /data_products/Al26_10xFlux_and_Ling.inc1.id1.extracted.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:7b21d239cbc31805a20a5cddbe3cce6a28a9a8edca7cc86ccbe1ac4f5a79e892 3 | size 78949381 4 | -------------------------------------------------------------------------------- /data_products/CenA_BG_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:ba78d1c06f5a9037d4ae7c16f613c4fa8cca1c4dcc1d73353b5660e1b6ae1357 3 | size 728209547 4 | -------------------------------------------------------------------------------- /data_products/CenA_only_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:91001c6d0f36437c1386556a4205ca9e47b26ef8e623c84d104f03591d5c4d0d 3 | size 2937109 4 | -------------------------------------------------------------------------------- /data_products/Continuum_imaging_response.npz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:2a1e230b2012a49644954b7d29fe0cbb8488202e652c7c2bd7ddf2236b9f60a9 3 | size 399702457 4 | -------------------------------------------------------------------------------- /data_products/Crab_BG_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:a5cfe75f53298f058e4af2fdb4f7e5d486f221d31668c40dd2e5f568a1d79393 3 | size 734312199 4 | -------------------------------------------------------------------------------- /data_products/Crab_COSIBalloonData.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:0a9a3b8ce64d087bdcf9eafa30d4a2e5c8f26c385262e2d853b7f546520fd7b4 3 | size 73800035 4 | -------------------------------------------------------------------------------- /data_products/Crab_only_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:31ab65d85ca862da94f2140c2227a00225a68a03d123d097536866a3becb5ecb 3 | size 9039774 4 | -------------------------------------------------------------------------------- /data_products/CygX1_BG_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:ab7736883c06e0451c15e731597403452c69e513653fb0f8b126d7e80a58c0e8 3 | size 727015596 4 | -------------------------------------------------------------------------------- /data_products/CygX1_only_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:b16ec507dc9a6b75d73a0f1fce3248d33260192fdad53196a813bb7938dab92d 3 | size 1743483 4 | -------------------------------------------------------------------------------- /data_products/DC1_combined_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:e9bbb8d1f424da977c80ff8355799254f7f4676d1715680373f4b47ae401a240 3 | size 744830773 4 | -------------------------------------------------------------------------------- /data_products/GC511_10xFlux_and_Ling.inc1.id1.extracted.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:aae7018d680843a8fcbc03c21537318fe8b61f37bbec432a89b9bd0869d90efd 3 | size 268674013 4 | -------------------------------------------------------------------------------- /data_products/GC511_10xFlux_only.inc1.id1.extracted.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:60fd058f7de897f432b3d61e97e31df38a95e81ab2da94c6b8f334398d00fa6b 3 | size 2677885 4 | -------------------------------------------------------------------------------- /data_products/Point_sources_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:5617ed1499a18cb497be55da7a4647101a0f9c07aa33959282110fc9b8cfc8ed 3 | size 13993396 4 | -------------------------------------------------------------------------------- /data_products/Point_sources_10x_BG.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:05525622fd02061f9bf7b2cb1b13b33b46cdd2a170623785f4f9463e01a51cae 3 | size 739265884 4 | -------------------------------------------------------------------------------- /data_products/README.md: -------------------------------------------------------------------------------- 1 | # Data Challenge Simulations 2 | 3 | For the first Data Challenge, we wanted to give users a basic introduction to COSI data analysis through 3 straightforward examples of COSI's science goals: 4 | - Extracting energy spectra from the Crab, Cen A, Cygnus X-1, and Vela 5 | - Imaging bright point sources, such as the Crab and Cygnus X-1 6 | - Imaging diffuse emission from the positron-electron annihilation 511 keV and the Al-26 1.8 MeV gamma-ray lines 7 | 8 | For each of these examples, we have provided a detailed description of the simulated sources and data products here in the data_products directory. Each of the sources was simulated at 10x the true astrophysical flux. Having a strong signal simplifies the analysis and allows us to focus on the workflow of the procedures. The COSI SMEX mission is expected to be 50x more sensitive than the balloon-borne mission. 9 | 10 | The simulations were all performed in MEGAlib with an accurate mass model of the COSI Balloon instrument. The [COSIBalloon.9Detector.geo.setup](https://github.com/cositools/massmodel-cosi-balloon/blob/main/COSIBalloon.9Detector.geo.setup) model was used, which accounts for the failure of three GeD detectors at different times during flight. Each of the continuum simulations was performed for 100 keV – 10 MeV, and an energy range selection of <5 MeV was used in MEGAlib’s mimrec event selection tool. 11 | 12 | The source simulations include the real flight aspect information so that the balloon path and source exposure time is accurate. This can be seen in the below plot showing the Galactic longitude and latitude of the COSI balloon's zenith direction as a function of time. 13 | 14 | 15 | 16 | Furthermore, the transmission probability of the source photons in the atmosphere is calculated for each instance of the simulation. The probability of transmission is taken at a constant altitude of 33 km, and is shown as a function of zenith angle and energy in the below figure. 17 | 18 | 19 | 20 | ## Simulation Tools: 21 | 22 | The tools that were used for the simulations (including the source library, orientation file, transmission probability file, etc.) are available in the main branch of cositools/cosi-data-challenges ([link](https://github.com/cositools/cosi-data-challenges/tree/feature/initialsetup)). 23 | 24 | ## Data Products: 25 | 26 | We have included many combinations of the source simulations and background to allow for flexibility and further testing with these files. All files are either in the .npz zipped numpy array format for the CDS-binned response matrix or background model, or in MEGAlib's photon list .tra.gz format. MEGAlib was used to perform all of these simulations, and the source models are described in detail below. 27 | 28 | Within this directory, 1 background model and 3 response matrices are provided: 29 | - **Scaled_Ling_BG_1x.npz**: background model generated from C. Karwin's scaled 1x Ling background simulation 30 | - **Continuum_imaging_response.npz**: 6º response used for spectral analysis and imaging continuum sources 31 | - **511keV_imaging_response.npz**: 6º imaging response required for RL imaging of Galactic positron annihilation 32 | - **1809keV_imaging_response.npz**: 6º imaging response required for RL imaging of Galactic Al-26 33 | 34 | There is a combined simulation of all sources and background: 35 | - **DC1_combined_10x.tra.gz**: 4 point sources with 10x flux (Crab, Cygnus X-1, Cen A, Vela), 511 keV & Al-26 lines, 1x Ling background 36 | 37 | There is each of the sources individually with the background: 38 | - **Point_sources_10x_BG.tra.gz**: 4 point sources with 10x flux (Crab, Cygnus X-1, Cen A, Vela) and 1x Ling background 39 | - **Crab_BG_10x.tra.gz**: Crab with 10x flux and 1x Ling background 40 | - **CygX1_BG_10x.tra.gz**: Cygnus X-1 with 10x flux and 1x Ling background 41 | - **CenA_BG_10x.tra.gz**: Cen A with 10x flux and 1x Ling background 42 | - **Vela_BG_10x.tra.gz**: Vela with 10x flux and 1x Ling background 43 | - **GC511_10xFlux_and_Ling.inc1.id1.extracted.tra.gz**: 511 keV emission with 10x flux with Ling background 44 | - **Al26_10xFlux_and_Ling.inc1.id1.extracted.tra.gz**: Al-26 emission with 10x flux with Ling background 45 | 46 | Each of the sources is also included without background: 47 | - **Point_sources_10x.tra.gz**: 4 point sources with 10x flux (Crab, Cygnus X-1, Cen A, Vela) 48 | - **Crab_only_10x.tra.gz**: Crab with 10x flux 49 | - **CygX1_only_10x.tra.gz**: Cygnus X-1 with 10x flux 50 | - **CenA_only_10x.tra.gz**: Cen A with 10x flux 51 | - **Vela_only_10x.tra.gz**: Vela with 10x flux 52 | - **GC_511_10xFlux_only.inc1.id1.extracted.tra.gz**: 511 keV emission with 10x flux 53 | - **Al26_10xFlux_Only.inc1.id1.extracted.tra.gz**: Al-26 emission with 10x flux 54 | 55 | And finally, there is 1 background simulation corresponding to the .npz background model above (this is not required for any analysis, but is included here for posterity): 56 | - **Scaled_Ling_BG_1x.tra.gz**: C. Karwin's scaled 1x Ling background simulation 57 | 58 | All files have the same start and stop time, in unix time: \ 59 | start time: 1463443400.0 s \ 60 | stop time: 1467475400.0 s \ 61 | total time: 4032000.0 s = 46.67 days 62 | 63 | This corresponds to May 17 2016 00:03:20 GMT to July 02 2016 16:03:20 GMT, covering the full COSI Balloon flight in 2016. 64 | 65 | As described in the main cosi-data-challenge-1 README, these files are stored on Git’s Large File Server. One must install git-lfs to access them. 66 | 67 | ## Science Background 68 | 69 | ### Point Sources 70 | 71 | There are four bright point sources included in these simulations: the Crab nebula, Cygnus X-1, Centaurus A, and Vela. 72 | 73 | The Crab nebula is considered both a pulsar wind nebula (PWN) and supernova remnant. The PWN surrounds the Crab Pulsar, a rapidly rotating and magnetized neutron star in the Milky Way constellation Taurus. The supernova remnant was produced by SN 1054. The Crab entered COSI-balloon's field of view for only ~12 of the 46 days of the 2016 flight ([Sleator 2019](https://www.proquest.com/docview/2313733159?pq-origsite=gscholar&fromopenview=true)); the balloon remained largely in Earth's Southern Hemisphere and the Crab is more easily viewed from the Northern Hemisphere. Nevertheless, as the brightest persistent $\gamma$-ray source in the sky, the Crab is detectable in the balloon data and is simulated for the data challenge with 10x its true flux of 0.049 ph cm $^{-2}$ s $^{-1}$ (100 keV-10 MeV). 74 | 75 | Cygnus X-1 is a bright hard X-ray source in the Cygnus constellation of the Milky Way. It is believed to be a black hole in an X-ray binary system. Cygnus X-1 emits in COSI's bandpass as well, and like the Crab is simulated here at 10x its true flux of 0.041 ph cm $^{-2}$ s $^{-1}$ (100 keV-10 MeV). This data challenge thus helps establish expectations for COSI-balloon observations of Cygnus X-1 during the 2016 flight. 76 | 77 | Centaurus A is a galaxy in the constellation of Centaurus. Also called NGC 5128, Centaurus A has a supermassive black hole which emits radio waves, X-rays, and gamma-rays. It is simulated with 10x its true flux of 0.0036 ph cm $^{-2}$ s $^{-1}$ (100 keV-10 MeV) for the data challenge analyses. 78 | 79 | The Vela pulsar is simulated based on a power law extrapolation of the corresponding Fermi-LAT source (4FGL J0835.3-4510) to lower energy. We have verified that the extrapolation is consistent with the observations from OSSE+COMPTEL+EGRET, e.g. see [Pavlov+01](https://iopscience.iop.org/article/10.1086/320342). It is fainter than the other three sources but is included as a source which could be observed by an instrument like COSI-balloon with more observation time. 80 | 81 | ### Positron Annihilation at 511 keV 82 | 83 | A strong 511 keV signal emanating from the direction of the Galactic Center $(\ell = 0^{\circ}, b = 0^{\circ})$ was first discovered in the 1970s on a series of balloon missions [(Johnson III et al. 1972,](https://adsabs.harvard.edu/full/1972ApJ...172L...1J?TB_iframe=true&width=370.8&height=658.8) [ Leventhal et al. 1978,](https://adsabs.harvard.edu/full/record/seri/ApJ../0225/1978ApJ...225L..11L.html) [ Johnson III & Haymes 1973,](https://adsabs.harvard.edu/pdf/1973ApJ...184..103J) [ Haymes et al. 1975,](https://adsabs.harvard.edu/full/record/seri/ApJ../0201/1975ApJ...201..593H.html) [ Ling et al. 1977,](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/JA082i010p01463) [ Albernhe et al. 1981)](https://adsabs.harvard.edu/pdf/1981A%26A....94..214A). Despite decades of observation since the initial measurements, several questions remain regarding the nature of this abundant positron-electron annihilation. As MeV gamma-ray instruments, COSI-balloon and the COSI satellite are uniquely equipped to study this signal, which traces one of the biggest unsolved mysteries in gamma-ray astrophysics. 84 | 85 | 1. What is the underlying nature of the emission morphology? 86 | 87 | The 511 keV image from the INTEGRAL SPI satellite ([Bouchet et al. 2010](https://iopscience.iop.org/article/10.1088/0004-637X/720/2/1772/meta), shown below as a significance map) reveals two seemingly distinct components: an extended "disk" which traces the Galactic Plane and a central, bright "bulge" around the Galactic Center. The notable difference between these structures is not understood. The extended disk emission may suggest that positrons are propagating away from and annihilating at a distance from their production sites, thereby smearing out the emission into a diffuse presentation. It is also possible, however, that there could be a collection of point-like sources emitting positrons which annihilate close to their progenitors; it is the collection of these sources together which may form a total diffuse structure. However, no individual point source of positrons has been detected. It is likely that positrons propagate away from their production sites and slow down to low enough energies to form positronium. 88 | 89 | Imaging with fine angular resolution can help disentangle this dichotomy. The detection of an individual point source, for example, could revolutionize our understanding of the morphology. High-resolution spectroscopy is also critical to understanding the transport and eventual annihilation sites of these positrons: a significant ortho-Positronum (o-Ps; $\leq 511$ keV) component of the spectrum would suggest annihilation in colder regions of the interstellar medium (ISM). A stronger signature at the line energy of 511 keV (para-Positronium, p-Ps) indicates annihilation in warmer regions of the ISM. Data from SPI currently favor the latter scenario, though additional measurements of this o-Ps to p-Ps fraction are necessary. 90 | 91 | Bouchet_2010_SPI_511keV 92 | 93 | 94 | 2. Where are all of these positrons coming from? 95 | 96 | The bulge emission exhibits a positron annihilation rate of $10^{43} e^+ s^{-1}$. Positrons from the $\beta^+$ decay of nucleosynthesis products may account for the $\sim 10^{42} e^+ s^{-1}$ in the disk and some of the bulge, but the origin of the remaining positrons in the bulge is unknown. Important to note is that there is no lack of potential positron sources; rather, there are too many possible sources to explain the emission! Positrons are readily created in a wide variety of astrophysical objects and processes. Stellar flares, massive stars, supernovae (core-collapse and Type Ia), classical novae, and neutron star mergers all synthesize radioactive isotopes which can $\beta^+$ decay. Secondary interactions with cosmic rays produce positrons, positrons are created through pair creation channels in strong photon and magnetic fields, evaporating black holes may potentially produce positrons, dark matter density profiles may trace positron annihilation,...and more. 97 | 98 | Ultimately, we can boil the "positron puzzle" down to uncertainty around the *source, transport, and sink* (the annihilation itself) of these particles. 99 | 100 | The COSI-balloon flight in 2016 detected the 511 keV signature of the positron puzzle with $7.2\sigma$ significance [(Kierans et al. 2020)](https://iopscience.iop.org/article/10.3847/1538-4357/ab89a9/meta). It also clearly imaged the bright bulge emission near the Galactic Center [(Siegert et al. 2020)](https://iopscience.iop.org/article/10.3847/1538-4357/ab9607/meta). Both measurements are consistent with those from SPI and indicate additional extended emission. 101 | 102 | In this data challenge, you will image the Galactic 511 keV emission, simulated at 10x its true flux (10x flux = $1.1 \times 10^{-2}$ ph cm $^{-2}$ s $^{-1}$) for robust statistics, as seen during the COSI-balloon flight in 2016. You should expect to see the bright bulge emission near the Galactic Center (but not the disk emission, which is too weak for COSI-balloon to see during its 46-day flight; SPI was able to detect the disk's $\sim 1$ ph/week with over a decade of observation time). 103 | 104 | 105 | ### Aluminum-26 Decay at 1.8 MeV 106 | 107 | As MeV gamma-ray instruments, COSI-balloon and the COSI satellite are uniquely equipped to study the 1.809 MeV signature emission from this radioisotope, which traces stellar nucleosynthesis over millions of years. 108 | 109 | In the 1980s, NASA's High Energy Astrophysical Observatory (HEAO-3) satellite mission detected 1.809 MeV emission emanating from the direction of the Galactic Center [Mahoney et al. 1984](https://adsabs.harvard.edu/pdf/1984ApJ...286..578M). This marked the discovery of Galactic Al-26. The spectrum is shown below. 110 | 111 | Mahoney_1984_HEAO-3_1809keV 112 | 113 | Subsequent observations by the Compton telescope (COMPTEL) on-board NASA's Compton Gamma Ray Observatory (CGRO) yielded the first image of Al-26 emission ([Oberlack et al. 1996](https://ui.adsabs.harvard.edu/abs/1996A%26AS..120C.311O/abstract), [Oberlack 1997](https://ui.adsabs.harvard.edu/abs/1997PhDT.........8O/abstract), [Pluschke et al. 2001](https://ui.adsabs.harvard.edu/abs/2001ESASP.459...55P/abstract)). Emission is concentrated in the Inner Galaxy $(|\ell| \leq 30^{\circ}, |b| \leq 10^{\circ}$) with enhanced emission in regions of massive star activity, including Cygnus, Carina, and Vela. 114 | 115 | The SPectrometer on INTEGRAL (SPI) largely corroborated the features seen in the COMPTEL image with over a decade of observation time from ESA's INTEGRAL satellite ([Bouchet et al. 2015](https://iopscience.iop.org/article/10.1088/0004-637X/801/2/142/meta)). Emission is concentrated in the Inner Galaxy with a reported flux of $\sim 3.3 \times 10^{-4}$ ph cm $^{-2}$ s $^{-1}$. As in the COMPTEL image, there is enhanced emission in regions of massive star activity, including Perseus/Taurus, Cygnus/Cepheus, Carina, Vela, and Scorpius-Centaurus. 116 | 117 | COMPTEL_1 8MeV_image | SPI_1 8MeV_image \ 118 | COMPTEL 1.8 MeV image (Pluschke et al. 2001) | SPI 1.8 MeV image (Bouchet et al. 2015) 119 | 120 | The COSI-balloon flight in 2016 measured the 1.809 MeV signature of Al-26 with $3.7\sigma$ significance, corresponding to about 106 Al-26 photons [(Beechert et al. 2022)](https://iopscience.iop.org/article/10.3847/1538-4357/ac56dc/meta). The reported Inner Galaxy flux of $(8.6 \pm 2.5) \times 10^{-4}$ ph cm $^{-2}$ s $^{-1}$ and line centroid of $1811.2 \pm 1.8$ keV are consistent with results from SPI and COMPTEL within 2 $\sigma$ uncertainties. Future observations with the COSI satellite (significantly increased observation time, greater effective area at 1.8 MeV, better constraints on high-latitude emission, and finer angular resolution) will comprise an important comparison to the balloon measurement, which was the first measurement of Al-26 on a compact Compton telescope. 121 | 122 | Furthermore, the COSI satellite's full-sky observations with fine angular resolution have potential to more closely study individual regions of massive star activity; in particular, resolving individual sites of emission within Cygnus is a promising goal of the mission. Detailed imaging and spectroscopic studies of the region may inform better understanding of the dynamics of Al-26 after it is produced and ejected from massive stars. 123 | 124 | In this data challenge, you will image the Galactic Al-26 emission (traced by the DIRBE 240 um image) as seen during the COSI-balloon flight in 2016. The flux is simulated at 10x the observed Al-26 flux (10x Inner Galaxy flux = $3.3 \times 10^{-3}$ ph cm $^{-2}$ s $^{-1}$; 10x total map flux = $1.2 \times 10^{-2}$ ph cm $^{-2}$ s $^{-1}$) for robust statistics. You should expect to see extended emission along the Galactic Plane, similar to that revealed by COMPTEL and SPI's 1.8 MeV images. The massive star regions of Cygnus, Carina, and Vela will not be as easily identifiable in this simulation of the balloon flight; for those, be sure to participate in the data challenge (and real data analysis) of the COSI satellite! 125 | 126 | 127 | 128 | ## Source Models 129 | 130 | ### Point Sources 131 | 132 | There are four bright point sources that are included in these simulations: the Crab nebula, Cygnus X-1, Centaurus A, and Vela. The spectra and flux for each of these sources was determined from the literature. As explained above, we have used 10x the flux values for these sources in order to simplify the analysis. Please keep this in mind when interpreting the results of your analyses in this Data Challenge. 133 | 134 | Crab: 135 |   (l,b) = (184.56, -5.78) 136 |   Spectral shape: Band function from [Jourdain et al. 2020](https://ui.adsabs.harvard.edu/abs/2020ApJ...899..131J/abstract) 137 |   Flux = 0.48977 ph cm $^{-2}$ s $^{-1}$ between 100 keV and 10 MeV 138 | Cen A: 139 |   (l,b) = (309.52, 19.42) 140 |   Spectral shape: SED from [HESS+LAT collaboration 2018](https://ui.adsabs.harvard.edu/abs/2018A%26A...619A..71H/abstract) 141 |   Flux: 0.03609 ph cm $^{-2}$ s $^{-1}$ between 100 keV and 10 MeV 142 | Cygnus X-1: 143 |   (l,b) = (71.33, 3.07) 144 |   Spectral shape: SED from [Kantzas+21](https://ui.adsabs.harvard.edu/abs/2021MNRAS.500.2112K/abstract) 145 |   Flux = 0.40644 ph cm $^{-2}$ s $^{-1}$ between 100 keV - 10 MeV 146 | Vela: 147 |   (l,b) = (263.55, -2.79) 148 |   Spectral shape: Power law extrapolation of the Fermi-LAT Vela pulsar (4FGL J0835.3-4510), see [Abdollahi+20](https://iopscience.iop.org/article/10.3847/1538-4365/ab6bcb)
149 |   Flux = 0.00120 ph cm $^{-2}$ s $^{-1}$ between 100 keV - 10 MeV 150 | 151 | The input spectra for these point sources is shown below. Note that the plot shows the true spectra, whereas the flux values reported above are for the 10x simulations. 152 | 153 | Screen Shot 2022-10-17 at 2 17 37 AM 154 | 155 | The simulations are run in MEGAlib’s cosima tool. A list-mode image created in mimrec confirms the correct point source locations: 156 | 157 | Screen Shot 2022-10-17 at 2 21 06 AM 158 | 159 | ### Positron Annihilation at 511 keV 160 | 161 | The morphology of the 511 keV emission from positron annihilation is not well constrained. For this first Data Challenge, we have used the model defined in [Knödlseder et al. 2005](https://ui.adsabs.harvard.edu/abs/2005A%26A...441..513K/abstract), where the emission was fit with a 2-D asymmetric Gaussian spatial model with the following parameters (the reported flux is the true value and not the 10x value): 162 | 163 | Screen Shot 2022-10-17 at 2 21 06 AM 164 | 165 | The emission was simulated as a 511 keV mono-energetic source, and the image from mimrec confirms the extended emission in the Galactic Center. 166 | 167 | 168 | 169 | ### Aluminum-26 Decay at 1.8 MeV 170 | 171 | The Diffuse Infrared Background Experiment (DIRBE) 240 um map has been shown to be a good tracer for the Al-26 emission, as measured by COMPTEL and INTEGRAL/SPI. We thus assume the DIRBE 240 um distribution as the spatial sky model for the Al-26 emission and normalize the Inner Galaxy flux of this map to SPI's measured Al-26 flux in this region: $3.3 \times 10^{−4}$ ph cm $^{-2}$ s $^{-1}$ ([Diehl et al. 2006](https://ui.adsabs.harvard.edu/abs/2006Natur.439...45D/abstract)). This model is described further in [Beechert et al. 2022](https://ui.adsabs.harvard.edu/abs/2022ApJ...928..119B/abstract). 172 | 173 | 174 | 175 | The emission was simulated with a 1.8 MeV mono-energetic source, and the image from mimrec confirms the extended emission along the Galactic disk. 176 | 177 | 178 | 179 | Note that the list-mode imaging method employed in MEGAlib is not optimized for diffuse sources, and the extra structure out of the Galactic plane is an imaging artifact. 180 | 181 | ### Background Radiation 182 | 183 | The background radiation model used for the Data Challenge is based on the semi-empirical model from [Ling 1975](https://ui.adsabs.harvard.edu/abs/1975JGR....80.3241L/abstract), which has been used for all COSI balloon analyses. The model describes the angular and atmospheric depth dependence of gamma-rays within 0.3 – 10 MeV and includes 3 primary gamma-ray sources: continuum emission (Bremsstrahlung from cosmic ray interactions in the atmosphere), a 511 keV line component (from electron-positron annihilation in the atmosphere, and a cosmic diffuse component (down-scattered gamma-rays from the extragalactic background and Galactic plane). The Ling model requires a description of the atmosphere’s density, interaction depth, and mass absorption coefficients, which are obtained through an [NRLMSISE-00 atmospheric model](https://ccmc.gsfc.nasa.gov/modelweb/models/nrlmsise00.php). An altitude of 33.5 km was assumed for these models - altitude drops and changes in longitude and latitude were not taken into account for this simulation. 184 | 185 | The amplitude of the Ling background was scaled so the total integrated background spectrum from simulation matched closely to what was measured during flight, as can been seen in the figure below. The flight data (“All Data” label) shows a time-variable background count rate that was influenced by the geomagnetic cutoff and balloon altitude drops. The cyan line shows the scaled Ling background model. As a reference, the count rate from the 1x flux point sources is shown, and the signal is at most a few percent of the background count rate. 186 | 187 | 188 | 189 | An image of the background simulation traces the exposure map, since the orientation of the COSI Balloon in Galactic coordinates was included in the simulation. Note that a $90^\circ$ Earth limb cut has also been applied to the data. 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207 | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | -------------------------------------------------------------------------------- /data_products/Scaled_Ling_BG_1x.npz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:ce05ebebe5e392fa39e1fac9c743107f4f804b7609ffb0b364ed9315f96ff498 3 | size 710116 4 | -------------------------------------------------------------------------------- /data_products/Scaled_Ling_BG_1x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:0ab13a456df9ded6cb6b5d48d5afd26932e79189f0e1eee61381a93d37346cfc 3 | size 725272567 4 | -------------------------------------------------------------------------------- /data_products/Vela_BG_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:57a0f73bfbaac90d54f23015963fa2805e6dfd4aed37d54b73573f2122d74aa8 3 | size 725547021 4 | -------------------------------------------------------------------------------- /data_products/Vela_only_10x.tra.gz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:6269d3e4c347280010625d320eef9fcf0c78ea94a2e33b39d36eb978e8a401ab 3 | size 274568 4 | -------------------------------------------------------------------------------- /data_products/flight_bg_all_v1_fine_6deg.npz: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:70858d1234c632c50b9d6ac26672ba2a70493c9bac5ba4df89485c5c7fbdaee2 3 | size 747207 4 | -------------------------------------------------------------------------------- /imaging/README.md: -------------------------------------------------------------------------------- 1 | # Welcome to image deconvolution with COSIpy-classic 2 | 3 | In this example, we'll use a Richardson-Lucy deconvolution algorithm to image the full simulated sky as observed during the balloon flight. There are three notebooks in this directory: 4 | 1. [RL-DataChallenge-Point_Sources-10XFlux-Ling.ipynb](RL-DataChallenge-Point_Sources-10XFlux-Ling.ipynb) which images 4 simulated point sources. 5 | 2. [RL-DataChallenge-511keV_10xFlux-Ling.ipynb](RL-DataChallenge-511keV_10xFlux-Ling.ipynb) performs the diffuse imaging of the 511 keV emission. 6 | 3. [RL-DataChallenge-Al26_10xFlux-Ling.ipynb](RL-DataChallenge-Al26_10xFlux-Ling.ipynb) performs the diffuse imaging of the Al-26 1.8 MeV decay line. 7 | 8 | The three notebooks are almost identical in their execution, but different data sets will be uploaded, different response matrices will be used, and slightly different imaging parameters will be required. Please refer to the [data_products](../data_products) README for more details on the scientific background for these sources, and the simulated source models. This README should act as a guide offering additional information for each step of the analysis. The following sections align with the different steps within the notebooks. 9 | 10 | **Before diving in**, you should be aware that these notebooks take signficant time and computer memory to run, more so than the spectral fitting notebook. Calculating the imaging response takes ~1 hour on a personal computer with 16 GB of memory, and to actually perform the Richardson-Lucy image deconvolution as it currently is written will take >75 GB of memory - this will likely be >6 hours even on powerful workstations. The imaging may not be doable with a personal computer and it is recommended that you complete this analysis on a workstation with more memory, if accessible. 11 | 12 | It's worth noting that the memory-intensive nature of this analysis is a consequence of using the original COSIpy-classic code, for which there wasn't a dedicated effort to reduce the computing requirements. Additionally, the Richardson-Lucy imaging is still being developed for COSI analysis, and the code is largely hard-coded in the notebooks. Future versions of Data Challenges will address these issues. 13 | 14 | **Note as of January 20, 2023**: The Al-26 notebook was tested and executes in its current form on Ubuntu 20.04. We've heard reports that the image deconvolution fails at early iterations (~10) on Mac M1 and does not run on Ubuntu 22.04, the latter of which cannot support pystan. This is under investigation. Note that the other imaging notebooks should still be working ok. Please attempt the Al-26 notebook as you wish, try changing parameters in the algorithm, or refer to the 511 keV imaging notebook for another example of diffuse imaging. 15 | 16 | ## Intial Setup 17 | 18 | These cells import relevant packages, define file names, and read in the data files. 19 | 20 | ## Bin the data 21 | 22 | ### Define the bins for the Compton data space 23 | **Time bins:** The balloon environment is background-dominated, and the rates depend mainly on the latitude (i.e. geomagnetic cutoff) and the altitude (i.e. atmospheric depth). When analyzing the COSI flight data, we account for this changing background rate by binning the data set into time bins which we can individually fit with varying backgrounds. For the simulated data in this Data challenge, the background rate is constant with time, and therefore this time binning is less important. We include it here for flexibility. 24 | 25 | Through extensive testing, **1800 second** (30 minute) time bins were found to strike a practical balance between a sufficiently precise treatment of background variability and computational means for the imaging analysis. 26 | 27 | You can feel free to play with the time binning. You may choose to decrease the size to 900 s, or increase it to 3600 s, for example, to see if there's an effect on the image (e.g. does it look less/more blurred, respectively, as you lump more data into more/fewer time bins?) 28 | 29 | **Energy bins:** We need to define the energy bins exactly as they are defined in the response. 30 | 31 | For point source imaging, we use a continuum response simulation which spans several energy bins across COSI's 0.2-5 MeV bandpass: **[150, 220, 325, 480, 520, 765, 1120, 1650, 2350, 3450, 5000] keV**. 32 | 33 | For positron-electron annihilation at 511 keV, we use a response simulation with only one energy bin around the 511 keV signature: **501-521 keV**. 34 | 35 | For Al-26, we use a response simulation with only one energy bin around the 1809 keV photopeak signature: **1803-1817 keV**. 36 | 37 | **Sky pixel size:** As with the energy binning, the pixel size here must match that of the response. The response matrix that we are providing for this COSI-balloon analysis assumes $6^{\circ} \times 6^{\circ}$ resolution. 38 | 39 | ### Binning 40 | Calling `.get_binned_data()` will loop through all of the events in the simulated data set to fill the bins of the Compton Data Sapce (CDS; [../README.md](../README.md)). 41 | 42 | ### Examining the shape 43 | 44 | The binned data are contained in "analysis1.dataset.binned_data." This is a 4-dimensional object representing the 5 dimensions of the CDS: (time, energy, $\phi$, FISBEL). 45 | 46 | The number of bins in each dimension are shown by calling "shape." 47 | 48 | Per the binning definitions above, there are 2240 time bins, 10 energy bins for the continuum analysis or 1 bin for line analysis (as governed by those in the response), 30 $\phi$ bins $(6^{\circ}$ bin size spanning the full $0-180^{\circ}$ range of possible Compton scattering angles), and 1145 FISBEL bins. 49 | 50 | FISBEL is a unique index which specifies the $\chi$ and $\psi$ dimensions of the CDS that specify the direction of the scattered photon in the first interaction. 51 | 52 | How do we end up with 1145 FISBEL bins? Consider a sphere which is $4 \pi( 180^{\circ}/ \pi)^2 = 41252.96 \textrm{ deg}^{2}$ \ 53 | Given our $6 \textrm{ deg}^{2}$ binning, we have $41252.96 \textrm{ deg}^{2}$ / $6 \textrm{ deg}^{2}$ $\sim$ 1145 bins. 54 | 55 | The notebook will show you how to get the shape of the data set, and how to extract the bin sizes. 56 | 57 | ### Inspecting the data 58 | 59 | Since we have the simulated data read and binned into the CDS, we can now make raw spectra, light curves, and other projections of the data. Two examples of this are shown in the notebook. The spectrum isn't entirely enlightening when looking at the single bin of the line analyses, but in the point source notebook with the continuum response, we can see the total simulated spectrum. This is for the duration of the balloon flight (total time = 4031996 seconds = 46.6 days), and the majority of photons in this spectrum are from the background simulation. The spectrum shows a clear 511 keV line, which fills the narrow 4th bin of the full continuum spectrum, which has contributions from the Ling background and from the Galactic center source simulation. The light curve is dominated by background radiation, but in the point source notebook one can see the variability in the latter half of the flight due to the bright Crab nebula within the FOV. 60 | 61 | ## Pointing Class 62 | 63 | The pointing class handles the aspect information of the COSI-balloon instrument. During flight, COSI freely floated on the balloon platform. This means that, unlike a space or ground-based telescope with well-defined pointings and slewing schedule, its orientation was largely dependent on the unconstrained path of the balloon. It was a zenith-pointing instrument, meaning that its vertical orientation pointed straight above the hanging instrument, towards the balloon above it. The exception to this freedom is that during the day time, COSI's azimuthal orientation was fixed such that its solar panels remained oriented facing the Sun. At nighttime, though, the instrument freely rotated about its azimuth. 64 | 65 | This is all to say that COSI's orientation changed rapidly during flight. We had a differential GPS onboard, which recorded the yaw, pitch, and roll of the balloon payload every second. In the COSI-balloon calibrations performed in MEGAlib, this is converted to the X, Y and Z pointing of the COSI-balloon in Galactic coordinates. This aspect information is contained in the .tra.gz simulation file and the pointing information for each event is read in during the `.read_COSI_DataSet()` command. This pointing class bins this aspect information into a list of 'stable' pointings for which the change in the aspect is below a certain angular threshold. By default, this threshold is set to 5 degrees. This information is required when creating the sky model or image response. 66 | 67 | In the point source imaging [notebook](RL-DataChallenge-Point_Sources-10XFlux-Ling.ipynb), we include some visuals to help you understand the pointings. For example, all of the Z pointings (i.e. COSI's zenith) are plotted in Galactic coordinates, with the Crab nebula position overlaid. From this, you can see the path that COSI traced across the sky. This can be compared with the flight path shown in the main [README](../README.md). We also can plot the elevation of any source within COSI's FOV. The elevation for the Crab position is shown, and we can see the source move in and out of the field of view. In this plot, the "horizon" lies at the maximum extent of what COSI can see beyond zenith, which is ~60 deg from zenith; therefore, COSI's zenith lies 60 deg above the horizon. The Crab is more visible in the latter part of the flight when COSI floated further north. We notice, too, that the Crab is always somewhat off-axis; it is never directly overhead the instrument at zenith. 68 | 69 | ## Background Model 70 | 71 | As discussed in [data_products](../data_products), we model the background using extensive simulations of Earth's atmospheric $\gamma$-ray background based on the Ling model. The simulations use an accurate mass model of the COSI-balloon instrument during flight and follow the true orientation of the instrument as it traveled along its flight path. The simulations were performed in MEGAlib, and we have provided an .npz background response file which contains the Ling model simulation binned into the Compton Data Space. Defining the background model here loads this response. 72 | 73 | ## Read in Response Matrix 74 | 75 | The instrument response matrix is created through large simulations in MEGAlib's ResponseCreator program. There are different response matrices for the point sources and the diffuse line emission based on the energy binning. The point sources use the continuum response, which spans the full energy range of the COSI-balloon, whereas the 511 keV and Al-26 analyses only have one energy bin around the line of interest. 76 | 77 | After we read in the response, which is a 5D numpy array (.npz format), we can explore the shape of the data space to better understand the connections between the response matrix, the data set, and the background model. 78 | 79 | The shape of the response spans (Galactic latitude $b$, Galactic longitude $\ell$, Compton scattering angle $\phi$, FISBEL, energy). The size of each dimension depends on the chosen pixel size. Here, we've chosen $6^{\circ}$ pixels. 80 | 81 | Galactic latitude $b \in [-90^{\circ}, 90^{\circ}] \rightarrow$ 30 bins.\ 82 | Galactic longitude $\ell \in [-180^{\circ}, 180^{\circ}] \rightarrow$ 60 bins.\ 83 | Compton scattering angle $\phi \in [0^{\circ}, 180^{\circ}] \rightarrow$ 30 bins (`analysis1.dataset.phis.n_phi_bins`).\ 84 | See above for explanation of 1145 FISBEL bins (`rsp.rsp.n_fisbel_bins`).\ 85 | The continuum response has 10 energy bins. 86 | 87 | The shape of the data and background objects span (time, energy, Compton scattering angle, FISBEL). 88 | 89 | Given the time bin size `Delta_T` which we defined at the beginning of the notebook, there are 2240 time bins. 90 | 91 | The notebook will show you how to find these shapes and bin sizes. 92 | 93 | ## RL Imaging 94 | 95 | Up until this point, the analysis steps have been completely generic and parallel what is required for standard COSI analysis. The spectral fitting notebook almost identically follows the above steps. All subsequent explanations are specific to the image deconvolution algorithims, and as you can see the RL algorithim and supporting functions are hard-coded in this notebook. These tools will be integrated into the COSIpy library for the next data challenge iteration. 96 | 97 | 98 | ## Setup for imaging / Define the imaging response 99 | 100 | **Defining a grid on the sky to make images** \ 101 | Here we initialize a regular 2-D sky grid (6 deg x 6 deg, per the previously defined pixel size). This grid is used later in the notebook when generating images with the RL algorithm. 102 | 103 | **Convert sky grid to zenith/azimuth pairs for all pointings** \ 104 | The response is a function of coordinates: R(Z, A; $\phi$, $\psi$, $\chi$) 105 | 106 | Z = zenith (angle in instrument coordinates) \ 107 | A = azimuth (angle in instrument coordinates) \ 108 | $(\phi, \psi, \chi)$ are the CDS angles. 109 | 110 | Z and A point to a location on the sky via a unique mapping of (Z, A) $\leftrightarrow$ $(\ell, b)$, i.e. Galactic longitude and latitude, that is changing with time. This mapping is calculated with "Great Circles," trigonometric functions, etc. See the "zenaziGrid()" function in COSIpy_tools_dc1.py. Note that the attitude is what defines the (Z, A) $\leftrightarrow$ $(\ell, b)$ transformation. The attitude is just a rotation matrix, which can be specified multiple ways. Currently the .tra.gz file contains the attitude for every event, specified by the orientation of the spacecraft axes (given in Galactic coordinates). In the future, this will be part of a separate “spacecraft file” that contains the spacecraft location/orientation information (i.e. Earth’s latitude/longitude + attitude quaternion) necessary for data analysis. 111 | 112 | Here, we convert the sky grid to said zenith/azimuth pairs for later convolution with the response. 113 | 114 | **Get observation indices for non-zero bins** \ 115 | To speed up the imaging, we only analyze bins with non-zero counts in both the sky response and background response. We also select only one energy bin for the analysis. In the case of point source imaging, we choose the 320-480 keV bin ("ebin = 2") of the continuum response because it has the highest effective area. The 511 keV and 1809 keV responses only have one energy bin, and thus the index is necessarily "ebin = 0." 116 | 117 | **Reduce the response dimensions** \ 118 | This cell reduces our energy dimension to the sole energy bin selected above and reduces the length of the CDS array to only the non-zero entries. 119 | 120 | **Function to get the response of an image for arbitrary time binning** \ 121 | For each point in time, we calculate the response. This takes into account that within the selected time bin width Delta_T, the instrument moves with respect to the fixed Galactic coordinate system, and applies weighting whenever the change in aspect angle exceeds 5 degrees (defined by the Pointing class). 122 | 123 | **Exposure map** \ 124 | Here we plot the exposure map, which is the response multiplied by the actual observation time in the field of view within each Delta_T bin. For a more complete understanding of COSI-balloon's observations, we divide by the pixel size on the sky. This allows for estimates of the effective area (exposure) of extended sources versus point sources. 125 | 126 | ## Set up for RL Algorithim 127 | 128 | There are many definitions required for the RL algorithim, including defining regions of the sky with poor exposure, defining the number of time bins, selecting the energy bin for imaging, defining an intial starting map (here defined to be an isotropic map), etc. See the details in the notebook for more information. Be aware that the stan model that is loaded often will print out a large warning, but the cell does run correctly. 129 | 130 | ## Richardson-Lucy Algorithim 131 | 132 | The steps follow the algorithm as outlined in [Knödlseder et al. 1999](https://ui.adsabs.harvard.edu/abs/1999A%26A...345..813K/abstract). Refer to that paper for a mathematical description of the algorithm. 133 | 134 | The total memory used during these iterations is about 94 GB for the continuum response, 75 GB for the 511 keV, and 105 GB for the Al-26!! You might be limited on your personal computer, and at the very least, you might not be able to do much else with your machine while this is running. 135 | 136 | #### Adjustable parameters 137 | There are three parameters at the beginning of this RL cell which we encourage you to adjust. In fact, it is often necessary to adjust these parameters depending on the data being studied. A list of suggested paramters for different cases is provided [here](imaging_suggested_parameters.pdf). 138 | 139 | - map_init\ 140 | This is the flux value of the initial, isotropic map. Typically, a value of 0.01 works well. For stronger sources, you can try increasing it to 0.1 or 1.0. As an example, running the algorithm on a source-only (no background) simulation of the Crab, Cen A, Cygnus X-1, and Vela works well with map_init = 0.01. However, when imaging these sources each simulated with 10X their true flux values, the algorithm fails at 0.01 and works when map_init = 1.0. 141 | 142 | - iterations\ 143 | This is the number of RL iterations. You can set this to a small value, say 50, as you get used to using the algorithm. In our testing, though, for fully converged images we usually let the algorithm run for 150 iterations. ***This can take anywhere from several hours (usually simulations without background) to overnight (simulations with background) to run.*** 144 | 145 | - afl_scl\ 146 | This is a scaling factor for the delta map (a map which encodes the changes between each iteration) which we call the "acceleration parameter." This allows the delta map to be afl_scl times stronger than the original RL algorithm suggests (c.f. [Knoedlseder+1999](https://ui.adsabs.harvard.edu/abs/1999A%26A...345..813K/abstract)).\ 147 | The default value here is 2000, though 1000 also works well. If you find that the algorithm returns "Fit failed" messages after running for awhile, for example, lowering this acceleration parameter to 1000 can help. 148 | 149 | Other parameters you can adjust: 150 | - mu_Abg, sigma_Abg\ 151 | There is a prior in the background fit defined by mu_Abg $\pm$ sigma_Abg. By default, mu_Abg and sigma_Abg are set to fitted_bg and most testing has been done with this setting. You can try constraining the fit by decreasing sigma_Abg, for example, to sigma_Abg/2., sigma_Abg/10., which would enable to fit to vary by 50%, 10% of the initial guess. 152 | 153 | - delta_map_tot_old\ 154 | You can change the exponent of the denominator. By default, it is set to 0.25 to help avoid exposure edge effects. All testing has been done with this fourth root. However, you can try setting it to 0, 0.5, etc. to see what happens. You can also try smoothing delta_map_tot_old with a Gaussian filter. 155 | 156 | ## Confirming algorithim convergence 157 | 158 | After the RL algorithm is done running, you can plot the fit likelihoods and the likelihoods of maps (vs. initial map, i.e. basically only background). The likelihood is a proxy of algorithm convergence; if the likelihood remains unchanged after a certain number of iterations, then further iterations show an equally likely distribution of photons as the sky as previous images. Thus, when the likelihood plots plateau, we can say that the algorithm has converged. 159 | 160 | Wildly oscillating likelihoods, for example, indicate instability in the algorithm. Try changing the suggested parameters and run the algorithm again. 161 | 162 | Note that the flux may increase without bound as the number of iterations increases. This is an intrinsic consequence of the RL algorithm itself. The algorithm may settle into imaging artifacts as the iterations progress, particularly in areas of poor exposure, and amplify the flux in these regions. This is why determining the "final" iteration of a RL algorithm, and thereby the "correct" image, is an open question. The user must consider different metrics, including the likelihood here, to make the best judgment. 163 | 164 | ## Final image 165 | 166 | **IMPORTANT NOTE:** The color scales for all images are in arbitrary units. The COSI team is currently working on determining the level of the flux. 167 | 168 | Sample final images are included in the "plots" folder. 169 | 170 | **Point Source Notebook:** \ 171 | The Crab nebula is the only easily visible source in this combined simulation of 10x flux Crab, 10x flux Cygnus X-1, 10x flux Centaurus A, 10x Vela, and 1x flux Ling background (scaled to the observed 2016 flight background level). A sample final image is shown below: 172 | PointSources_LingBG_continuumresponse_ebin2_RLimage 173 | 174 | You can play with the color scaling to try to enhance the appearance of the other sources. Vela is likely too dim to be seen, however. 175 | 176 | You can also try running this notebook without including the Ling background. Change the loaded .tra.gz file at the beginning, adjust RL parameters as necessary, and see if the four point sources are more easily resolved without background contamination! 177 | 178 | As another suggestion, what happens if you run this notebook using the 511 keV response? The 1809 keV response? A different energy bin of the continuum response? 179 | 180 | You can try combining these four point sources and Ling background with the 10x 511 keV and 10x Al-26 simulations for a full combined imaging test, using all three response simulations too. 181 | 182 | **Positron Annihilation at 511 keV:** \ 183 | We clearly see the "bulge" emission of positron-electron annihilation at the center of the Milky Way: 184 | 185 | 511keV_LingBG_511keVresponse_RLimage 186 | 187 | This was also seen in the published image of real COSI-balloon flight data [(Siegert et al. 2020)](https://iopscience.iop.org/article/10.3847/1538-4357/ab9607/meta): 188 | 189 | Siegert_2020_COSI_511keV 190 | 191 | The extended disk emission seen in the SPI image (see [Science Background](../data_products/README.md)) is not visible here. This is expected; SPI saw about 1 photon per week from the disk and has over a decade of observation time. There is not enough data in the 46-day balloon flight to image the disk. 192 | 193 | However, we can still probe the emission morphology of the bulge by fitting a 2-D Gaussian, for example, to our simulated image. Constraining the parameters of this fit is important for modeling the physics (positron propogation, point sources of positrons, etc.) behind this enduring mystery. 194 | 195 | **Al-26:** \ 196 | As expected, we observe extended Al-26 emission along the Galactic Plane. There is concentrated emission in the Inner Galaxy. The RL algorithm therefore behaves as expected. 197 | 198 | 26Al_LingBG_1809keVresponse_RLimage 199 | 200 | Given that only ~100 Al-26 photons were detected during the COSI-balloon flight $(3.7 \sigma$ significance, [Beechert et al. 2022](https://iopscience.iop.org/article/10.3847/1538-4357/ac56dc/meta)), imaging the emission at its true flux instead of 10x strength would likely result in only imaging artifacts. Consider the following calculation. For $n$ spatial bins each with measurement significance $n_i$, the total significance of a measurement is $$s = (\sum_{i = 1}^{n} s_i^2)^{1/2}.$$ 201 | 202 | Even if requiring only a weak $2\sigma$ measurement in each bin, for example, the maximum $n$ number of bins for a total $3.7\sigma$ measurement is approximately 3.4. In other words, a $3.7\sigma$ significant measurement distributed across the broad, diffuse Al-26 emission which spans the Galactic Plane would result in few spatial bins with meaningful significance. 203 | 204 | As with the positron annihilation, we can fit the simulated 10x flux image with a 2-D Gaussian to characterize the morphology of Galactic Al-26. 205 | 206 | Future data challenges will image Al-26 as seen with the COSI satellite. Increased observation time, increased effective area, finer angular resolution, and observations at high Galactic latitudes (extending above and below the Galactic Plane) have great potential to advance understanding of this radioisotope. 207 | 208 | 209 | 210 | 211 | -------------------------------------------------------------------------------- /imaging/SPI_DIRBE_Orig_Knoedl_240um.fits: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/imaging/SPI_DIRBE_Orig_Knoedl_240um.fits -------------------------------------------------------------------------------- /imaging/imaging_suggested_parameters.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/imaging/imaging_suggested_parameters.pdf -------------------------------------------------------------------------------- /imaging/plots/26Al_LingBG_1809keVresponse_RLimage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/imaging/plots/26Al_LingBG_1809keVresponse_RLimage.png -------------------------------------------------------------------------------- /imaging/plots/26Al_noBG_1809keVresponse_RLimage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/imaging/plots/26Al_noBG_1809keVresponse_RLimage.png -------------------------------------------------------------------------------- /imaging/plots/511keV_LingBG_511keVresponse_RLimage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/imaging/plots/511keV_LingBG_511keVresponse_RLimage.png -------------------------------------------------------------------------------- /imaging/plots/511keV_noBG_511keVresponse_RLimage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/imaging/plots/511keV_noBG_511keVresponse_RLimage.png -------------------------------------------------------------------------------- /imaging/plots/PointSources_LingBG_continuumresponse_ebin2_RLimage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/imaging/plots/PointSources_LingBG_continuumresponse_ebin2_RLimage.png -------------------------------------------------------------------------------- /imaging/plots/PointSources_noBG_continuumresponse_ebin2_RLimage.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/imaging/plots/PointSources_noBG_continuumresponse_ebin2_RLimage.png -------------------------------------------------------------------------------- /imaging/stanmodel.stan: -------------------------------------------------------------------------------- 1 | data { 2 | 3 | int N; // number of response matrix points times number of hours in data set 4 | int Nh; // number of hours of observation 5 | int y[Nh*N]; // y observations 6 | int Nsky; // number of sky models 7 | 8 | real bg_model[Nh,N]; // BG model 9 | real conv_sky[Nsky,Nh,N]; // SKY model(s) 10 | real acceleration_factor_limit; 11 | 12 | // background re-normalisation times 13 | int Ncuts; 14 | int bg_cuts[Nh]; 15 | int bg_idx_arr[Nh]; 16 | 17 | //priors on the fitted parameters 18 | real mu_flux[Nsky]; 19 | real sigma_flux[Nsky]; 20 | real mu_Abg[Ncuts]; 21 | real sigma_Abg[Ncuts]; 22 | 23 | } 24 | 25 | 26 | //transformed data { 27 | // // data 28 | // int data_values[Nh*N]; 29 | // 30 | // for (nh in 1:Nh) { 31 | // for (nn in 1:N) { 32 | // data_values[N*(nh-1)+nn] = y[nh,nn]; 33 | // } 34 | // } 35 | // 36 | //} 37 | 38 | 39 | parameters { 40 | 41 | real flux[Nsky]; // 511 keV line flux of conved sky model 42 | real Abg[Ncuts]; // background model amplitude(s) 43 | 44 | } 45 | 46 | 47 | transformed parameters { 48 | 49 | // model 50 | real model_values[Nh*N]; 51 | 52 | for (nh in 1:Nh) { 53 | for (nn in 1:N) { 54 | model_values[N*(nh-1)+nn] = Abg[bg_idx_arr[nh]] * bg_model[nh,nn]; 55 | for (ns in 1:Nsky) { 56 | model_values[N*(nh-1)+nn] += flux[ns] * conv_sky[ns,nh,nn]; 57 | } 58 | } 59 | } 60 | 61 | } 62 | 63 | 64 | 65 | model { 66 | 67 | // normal priors 68 | flux ~ normal(mu_flux,sigma_flux); 69 | Abg ~ normal(mu_Abg,sigma_Abg); 70 | 71 | //print(flux); 72 | //print(Abg); 73 | //print(model_values); 74 | 75 | // likelihood 76 | y ~ poisson(model_values); 77 | 78 | } 79 | 80 | 81 | // generated quantities { 82 | 83 | // vector[Nh*N] ppc; 84 | 85 | // vector[Nh] model_tot = rep_vector(0,Nh); 86 | // vector[Nh] model_bg = rep_vector(0,Nh); 87 | // matrix[Nsky,Nh] model_sky = rep_matrix(0,Nsky,Nh); 88 | 89 | // // create posterior samples for PPC 90 | // // and 91 | // // generate the posterior of the 92 | // for (nh in 1:Nh) { 93 | // for (nn in 1:N) { 94 | // ppc[N*(nh-1)+nn] = poisson_rng(model_values[N*(nh-1)+nn]); 95 | // // fitted model, summed over phi/psi/chi-dimension (only hours left) 96 | // // sky 97 | // for (ns in 1:Nsky) { 98 | // model_sky[ns,nh] += flux[ns] * conv_sky[ns,nh,nn]; 99 | // } 100 | // // bg 101 | // model_bg[nh] += Abg[bg_idx_arr[nh]] * bg_model[nh,nn]; 102 | // // total 103 | // for (ns in 1:Nsky) { 104 | // model_tot[nh] += flux[ns] * conv_sky[ns,nh,nn]; 105 | // } 106 | // model_tot[nh] += Abg[bg_idx_arr[nh]] * bg_model[nh,nn]; 107 | // } 108 | // } 109 | 110 | //} -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | tqdm==4.64.1 2 | pandas==1.5.0 3 | shapely==1.8.5.post1 4 | scipy==1.9.2 5 | emcee==3.1.3 6 | corner==2.2.1 7 | jupyter 8 | ipywidgets 9 | astropy 10 | pystan==2.19.1.1 11 | matplotlib 12 | numpy==1.23 13 | -------------------------------------------------------------------------------- /spectral-fit/.gitattributes: -------------------------------------------------------------------------------- 1 | *.npz filter=lfs diff=lfs merge=lfs -text 2 | *.gz filter=lfs diff=lfs merge=lfs -text 3 | -------------------------------------------------------------------------------- /spectral-fit/README.md: -------------------------------------------------------------------------------- 1 | # Welcome to spectral fitting with COSIpy-classic 2 | 3 | In this notebook, we'll perform a spectral fit on the Crab nebula and Centaurus A using simulated balloon flight data. There are other simulated sources in the data set - Cygnus X-1, Vela, the 511 keV emission from positron annihilation and the Al-26 1.8 MeV decay line - but these are not explored in this notebook. Please refer to the [data_products](../data_products) README for more details on the scientific background for these sources, and the simulated source models. This README should act as a guide offering additional information for each step of the analysis. The following sections align with the different steps within the notebook. 4 | 5 | Before diving in, you should be aware that this notebook takes signficant time and computer memory to run. The final spectral fit can take 1 hour on a personal computer with 16 GB of memory. If you have access to a workstation with more memory, that will save you time. 6 | 7 | ## Intial Setup 8 | 9 | These cells import relevant packages, define file names, and read in the data files. 10 | 11 | ## Bin the data 12 | 13 | ### Define the bins for the Compton data space 14 | **Time bins:** The balloon environment is background-dominated, and the rates depend mainly on the latitude (i.e. geomagnetic cutoff) and the altitude (i.e. atmospheric depth). When analyzing the COSI flight data, we account for this changing background rate by binning the data set into time bins which we can individually fit with varying backgrounds. For the simulated data in this Data challenge, the background rate is constant with time, and therefore this time binning is less important. We include it here for flexibility. 15 | 16 | Through testing with the flight data, **7200 second** (2 hr) time bins were found to strike a practical balance between a sufficiently precise treatment of background variability and computational means for the spectral analysis. 17 | 18 | **Energy bins:** We need to define the energy bins exactly as they are defined in the response. 19 | 20 | For point source analysis, we use a continuum response simulation which spans several energy bins across COSI's 0.2-5 MeV bandpass: **[150, 220, 325, 480, 520, 765, 1120, 1650, 2350, 3450, 5000] keV**. 21 | 22 | **Sky pixel size:** As with the energy binning, the pixel size here must match that of the response. The response matrix that we are providing for this COSI-balloon analysis assumes **$6^{\circ} \times 6^{\circ}$ resolution**. 23 | 24 | ### Binning 25 | Calling `.get_binned_data()` will loop through all of the events in the simulated data set to fill the bins of the Compton Data Sapce (CDS; [../README.md](../README.md)). 26 | 27 | ### Examining the shape 28 | 29 | The binned data are contained in `analysis1.dataset.binned_data`. This is a 4-dimensional object representing the 5 dimensions of the CDS: (time, energy, $\phi$, FISBEL). 30 | 31 | The number of bins in each dimension are shown by calling "shape." 32 | 33 | Per the binning definitions above, there are 560 time bins, 10 energy bins (as governed by those in the response), 30 $\phi$ bins $(6^{\circ}$ bin size spanning the full $0-180^{\circ}$ range of possible Compton scattering angles), and 1145 FISBEL bins. 34 | 35 | FISBEL is a unique index which specifies the $\chi$ and $\psi$ dimensions of the CDS that specify the direction of the scattered photon in the first interaction. 36 | 37 | How do we end up with 1145 FISBEL bins? Consider a sphere which is $4 \pi( 180^{\circ}/ \pi)^2 = 41252.96 \textrm{ deg}^{2}$ \ 38 | Given our $6 \textrm{ deg}^{2}$ binning, we have $41252.96 \textrm{ deg}^{2}$ / $6 \textrm{ deg}^{2}$ $\sim$ 1145 bins. 39 | 40 | The notebook will show you how to get the shape of the data set, and how to extract the bin sizes. 41 | 42 | ### Inspecting the data 43 | 44 | Since we have the simulated data read and binned into the CDS, we can now make raw spectra, light curves, and other projections of the data. Two examples of this are shown in the notebook. The spectrum shows the total simulated spectrum for the duration of the balloon flight (total time = 4031996 seconds = 46.6 days), and the majority of photons in this spectrum are from the background simulation. The spectrum shows a clear 511 keV line, which fills the narrow 4th bin of the full continuum spectrum, which has contributions from the Ling background and from the Galactic center source simulation. The light curve is dominated by background radiation but one can see the variability in the latter half of the flight due to the bright Crab nebula within the FOV. 45 | 46 | ## Pointing Class 47 | 48 | The pointing class handles the aspect information of the COSI-balloon instrument. During flight, COSI freely floated on the balloon platform. This means that, unlike a space or ground-based telescope with well-defined pointings and slewing schedule, its orientation was largely dependent on the unconstrained path of the balloon. It was a zenith-pointing instrument, meaning that its vertical orientation pointed straight above the hanging instrument, towards the balloon above it. The exception to this freedom is that during the day time, COSI's azimuthal orientation was fixed such that its solar panels remained oriented facing the Sun. At nighttime, though, the instrument freely rotated about its azimuth. 49 | 50 | This is all to say that COSI's orientation changed rapidly during flight. We had a differential GPS onboard, which recorded the yaw, pitch, and roll of the balloon payload every second. In the COSI-balloon calibrations performed in MEGAlib, this is converted to the X, Y and Z pointing of the COSI-balloon in Galactic coordinates. This aspect information is contained in the .tra.gz simulation file and the pointing information for each event is read in during the `.read_COSI_DataSet()` command. This pointing class bins this aspect information into a list of 'stable' pointings for which the change in the aspect is below a certain angular threshold. By default, this threshold is set to 5 degrees. This information is required when creating the sky model or image response. 51 | 52 | In the notebook we have some visuals to help you understand the pointings. For example, all of the Z pointings (i.e. COSI's zenith) are plotted in Galactic coordinates, with the Crab nebula and Cen A position overlaid. From this, you can see the path that COSI traced across the sky. This can be compared with the flight path shown in the main [README](../README.md). We also can plot the elevation of any source within COSI's FOV. The elevation for the Crab and Cen A position are shown, and we can see the sources move in and out of the field of view. In these plots, the "horizon" lies at the maximum extent of what COSI can see beyond zenith, which is ~60 deg from zenith; therefore, COSI's zenith lies 60 deg above the horizon. The Crab is more visible in the latter part of the flight when COSI floated further north. We notice, too, that the Crab is always somewhat off-axis; it is never directly overhead the instrument at zenith. Cen A, however, is much higher in COSI's FOV at the beginning of the flight, and at times is directly overhead at zenith. 53 | 54 | ## Background Model 55 | 56 | As discussed in [data_products](../data_products), we model the background using extensive simulations of Earth's atmospheric $\gamma$-ray background based on the Ling model. The simulations use an accurate mass model of the COSI-balloon instrument during flight and follow the true orientation of the instrument as it traveled along its flight path. The simulations were performed in MEGAlib, and we have provided an .npz background response file which contains the Ling model simulation binned into the Compton Data Space. Defining the background model here loads this response and it is scaled to match the mean rate in the data set. 57 | 58 | ## Read in Response Matrix 59 | 60 | The instrument response matrix is created through large simulations in MEGAlib's ResponseCreator program. This point source analysis uses the continuum response, which spans the full energy range of the COSI-balloon. After we read in the response, which is a 5D numpy array (.npz format), we can explore the shape of the data space to better understand the connections between the response matrix, the data set, and the background model. 61 | 62 | The shape of the response spans (Galactic latitude $b$, Galactic longitude $\ell$, Compton scattering angle $\phi$, FISBEL, energy). The size of each dimension depends on the chosen pixel size. Here, we've chosen $6^{\circ}$ pixels. 63 | 64 | Galactic latitude $b \in [-90^{\circ}, 90^{\circ}] \rightarrow$ 30 bins.\ 65 | Galactic longitude $\ell \in [-180^{\circ}, 180^{\circ}] \rightarrow$ 60 bins.\ 66 | Compton scattering angle $\phi \in [0^{\circ}, 180^{\circ}] \rightarrow$ 30 bins (`analysis1.dataset.phis.n_phi_bins`).\ 67 | See above for explanation of 1145 FISBEL bins (`rsp.rsp.n_fisbel_bins`).\ 68 | The continuum response has 10 energy bins. 69 | 70 | The shape of the data and background objects span (time, energy, Compton scattering angle, FISBEL). 71 | 72 | Given the time bin size `Delta_T` which we defined at the beginning of the notebook, there are 560 time bins. 73 | 74 | The notebook will show you how to find these shapes and bin sizes. 75 | 76 | ## Point Source Response 77 | 78 | Up until this point, the analysis steps have been completely generic and parallel what is required for standard COSI analysis. The imaging notebook almost identically follows the above steps. All subsequent explanations are specific to the spectral fitting of point sources. 79 | 80 | We'll start by looking at the Crab Nebula. The first step is to create the point source response for the Crab Nebula observation. The `rsp.calculate_PS_response` command combines the instrument response matrix and the pointing information to determine the response from the Crab location (read in with `l_crab` and `b_crab` parameters) for each instance in time. The background response is also read in here since it is used to find any empty bins in the CDS that can be removed from the fitting analysis. 81 | 82 | The counts for the simulated data, the background model, and the Crab point source sky response are all shown as a function of the time in the notebook. The data set, as mentioned previously, is dominated by the background simulation; however, in the latter half of the flight the counts from the bright Crab moving in and out of the FOV is seen. The background model is constant in time for the simulated data (a tracer can be used for flight data with variable backgrounds), and the normalization is scaled to match the simulated data set, as can be seen visually here. The point source response for the Crab shows this clearly (here it is normalized to 1, but we scale the counts for visibility). All components are thus prepared for the fit. 83 | 84 | 85 | ## Fitting the Crab Spectrum 86 | 87 | For each energy, we fit the coefficients for the sky and background models that best match the data for each time bin (see [README](../README.md) for more information). This is currently done for each individual energy bin with no consideration of neighboring bins. Given the number of bins in our data space, it can take about an hour to run the full fit. 88 | 89 | The final spectrum you obtain should appear as below: 90 | 91 | Screen Shot 2022-10-27 at 9 19 39 AM 92 | 93 | The narrow 4th energy bin containing the 511 keV line is too low. This is expected because the background model does not account for the strong 511 keV Galactic source, which is present in the data set. We have included a direct comparison with the expected spectrum from the Crab simulation itself, shown with the grey lines. Other than the outlier bin at 511 keV, the fit is generally in good agreement with the expected results. This serves as proof-of-concept for our spectral fitting tool in COSIpy-classic. 94 | 95 | # Cen A Analysis 96 | 97 | Let's take a look the spectral fit for the second brightest source: Cen A. Since we’ve already loaded the COSI-balloon simulated data, response matrix, and background model, we only need to redefine the point source response using Cen A’s coordinates and repeat the fit. Again, the fit will take about an hour, depending on your computer performance. 98 | 99 | ## First attempt at fitting the Cen A Spectrum 100 | 101 | After recomputing the point source response, you will see that the extracted Cen A spectrum does not adequately match expectations from simulations. You should find the following fit results: 102 | 103 | CenA_DC1_combined_10x_Spectrum 104 | 105 | The upper limits indicate where the signal-to-noise ratio (SNR) < 3. The error bars for the Cen A spectrum are very large, and almost all energy bins have upper limits. 106 | 107 | The poor fit is attributed to unwanted contamination from the bright simulated Crab source. To fix this, we would need to include the Crab in the background model. Because the elevation of the Crab in COSI's field of view is changing over time, our background model would also need to be time-dependent. However, the current fitting algorithm only includes one background parameter, so we cannot handle a time-dependent background at this point. We explore this hypothesis by considering a data set which includes the simulated Cen A with background, but without the other simulated sources. 108 | 109 | We repeat the steps used in the Crab analysis: we read in a new data set, define the background model, and redefine the point source response. We're then ready to reattempt the Cen A spectral analysis with this simplified example. 110 | 111 | ## Second attempt at fitting the Cen A Spectrum 112 | 113 | With only Cen A in our data set and no contamination from the Crab, the extracted spectrum from the fit more closely matches expectations from simulations: 114 | 115 | CenA_BG_10x_Spectrum 116 | 117 | This is an important demonstration of the need for an accurate background model. Sophisticated background models which include contaminating bright sources and which encode time varaibility of the background level during flight are critical to obtaining trustworthy results. Developing robust background models will be explored in subsequent Data Challenges. 118 | -------------------------------------------------------------------------------- /spectral-fit/spectral_fit.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "b0f27a80", 6 | "metadata": {}, 7 | "source": [ 8 | "# Welcome to spectral fitting with COSIpy classic\n", 9 | "\n", 10 | "In this notebook, we'll perform a spectral fit on the Crab nebula and Centaurus A using simulated balloon flight data.\n", 11 | "\n", 12 | "There are other simulated sources in the data set - Cygnus X-1, Vela, the 511 keV emission from positron annihilation and the Al-26 1.8 MeV decay line - but these are not explored in this notebook." 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "id": "644d001a", 18 | "metadata": {}, 19 | "source": [ 20 | "## Import packages.\n", 21 | "\n", 22 | "We're using the COSIpy classic functions in COSIpy_dc1.py, response_dc1.py, and fit_dc1.py." 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": null, 28 | "id": "5e288c39", 29 | "metadata": {}, 30 | "outputs": [], 31 | "source": [ 32 | "from COSIpy_dc1 import *\n", 33 | "import response_dc1 as response\n", 34 | "from fit_dc1 import *" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "id": "5c0734c0", 40 | "metadata": {}, 41 | "source": [ 42 | "## Define file names.\n", 43 | "\n", 44 | "DC1_combined_10x.tra.gz is a simulation of 4 point sources (Crab, Cen A, Cyg X-1, Vela), the 511 keV & Al26 lines, and Ling background.\n", 45 | "\n", 46 | "CenA_BG_10x.tra.gz is a simulation of only Cen A and Ling background." 47 | ] 48 | }, 49 | { 50 | "cell_type": "code", 51 | "execution_count": null, 52 | "id": "1f802109", 53 | "metadata": {}, 54 | "outputs": [], 55 | "source": [ 56 | "data_dir = '../data_products' # directory containing data & response files\n", 57 | "filename_combined = 'DC1_combined_10x.tra.gz' # combined simulation\n", 58 | "filename_cenA = 'CenA_BG_10x.tra.gz' # Cen A simulation\n", 59 | "response_filename = data_dir + '/Continuum_imaging_response.npz' # detector response\n", 60 | "background_filename = data_dir + '/Scaled_Ling_BG_1x.npz' # background response\n", 61 | "background_mode = 'from file'" 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "id": "e4eaa5cc", 67 | "metadata": {}, 68 | "source": [ 69 | "## Define inputs.\n", 70 | "\n", 71 | "You can perform the spectral fit on another point source, such as Cyg X-1 or Vela, by changing the latitude and longitude" 72 | ] 73 | }, 74 | { 75 | "cell_type": "code", 76 | "execution_count": null, 77 | "id": "3dfb9466", 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [ 81 | "l_crab,b_crab = 184.55746, -5.78436 # Galactic longitude & latitude of Crab\n", 82 | "l_cenA,b_cenA = 309.51584, 19.41709 # Galactic longitude & latitude of Cen A\n", 83 | "\n", 84 | "ul = 3 # SNR limit for upper limits on spectral fit" 85 | ] 86 | }, 87 | { 88 | "cell_type": "markdown", 89 | "id": "f9554d91", 90 | "metadata": {}, 91 | "source": [ 92 | "## Read in simulation and define analysis object.\n", 93 | "\n", 94 | "Read in the data set and create the main cosipy-classic “analysis_combined\" object, which provides various functionalities to study the specified file. This cell usually takes a few minutes to run." 95 | ] 96 | }, 97 | { 98 | "cell_type": "code", 99 | "execution_count": null, 100 | "id": "9d1673e4", 101 | "metadata": {}, 102 | "outputs": [], 103 | "source": [ 104 | "analysis_combined = COSIpy(data_dir,filename_combined) # create analysis object\n", 105 | "analysis_combined.read_COSI_DataSet() # read in data" 106 | ] 107 | }, 108 | { 109 | "cell_type": "markdown", 110 | "id": "14c74a0e", 111 | "metadata": {}, 112 | "source": [ 113 | "# Bin the data\n", 114 | "The data are binned into time, energy, ϕ and FISBEL. FISBEL is a unique index which specifies the χ and ψ dimensions of the CDS.\n", 115 | "\n", 116 | "Calling \"get_binned_data()\" may take several minutes, depending on the size of the dataset and the number of bins. Keep an eye on memory here: if your time bins are very small, for example, this could be an expensive operation." 117 | ] 118 | }, 119 | { 120 | "cell_type": "code", 121 | "execution_count": null, 122 | "id": "61d63bc2", 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "#Define the bin sizes\n", 127 | "Delta_T = 7200 # time bin size in seconds\n", 128 | "energy_bin_edges = np.array([150, 220, 325, 480, 520, 765, 1120, 1650, 2350, 3450, 5000]) # energy bin edges in keV\n", 129 | "pixel_size = 6. # pixel size in degrees\n", 130 | "\n", 131 | "analysis_combined.dataset.time_binning_tags(time_bin_size=Delta_T) # time binning\n", 132 | "analysis_combined.dataset.init_binning(energy_bin_edges=energy_bin_edges,pixel_size=pixel_size) # energy and pixel binning\n", 133 | "analysis_combined.dataset.get_binned_data() # bin data" 134 | ] 135 | }, 136 | { 137 | "cell_type": "markdown", 138 | "id": "fee98e0b", 139 | "metadata": {}, 140 | "source": [ 141 | "## Examine the shape of the binned data.\n", 142 | "\n", 143 | "The binned data are contained in \"analysis_combined.dataset.binned_data\". This is a 4-dimensional object representing the 5 dimensions of the Compton data space (time, energy, ϕ, FISBEL).\n", 144 | "\n", 145 | "This prints the shape of the binned data, the total time in the dataset, the number of time bins that have counts in them, and the number of counts in each time bin. Due to this energy range being so background dominated, the number of counts in each bin is very similar." 146 | ] 147 | }, 148 | { 149 | "cell_type": "code", 150 | "execution_count": null, 151 | "id": "9b1aff12", 152 | "metadata": { 153 | "scrolled": true, 154 | "tags": [] 155 | }, 156 | "outputs": [], 157 | "source": [ 158 | "print('Number of bins in each dimension (time, energy, ϕ, FISBEL):')\n", 159 | "print(analysis_combined.dataset.binned_data.shape)\n", 160 | "print()\n", 161 | "print('Total time in dataset (s):')\n", 162 | "print(analysis_combined.dataset.times.total_time)\n", 163 | "print()\n", 164 | "print('Number of populated time bins:')\n", 165 | "print(analysis_combined.dataset.times.n_ph)\n", 166 | "print()\n", 167 | "print('Number of counts in each time bin: ')\n", 168 | "print(analysis_combined.dataset.times.n_ph_t)" 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "id": "f52cf82e", 174 | "metadata": {}, 175 | "source": [ 176 | "## Plot raw spectrum & light curve." 177 | ] 178 | }, 179 | { 180 | "cell_type": "code", 181 | "execution_count": null, 182 | "id": "4c9b12d6", 183 | "metadata": {}, 184 | "outputs": [], 185 | "source": [ 186 | "analysis_combined.dataset.plot_raw_spectrum()\n", 187 | "plt.xscale('log')\n", 188 | "\n", 189 | "analysis_combined.dataset.plot_lightcurve()" 190 | ] 191 | }, 192 | { 193 | "cell_type": "markdown", 194 | "id": "52ac27e0", 195 | "metadata": {}, 196 | "source": [ 197 | "## Define the pointing object with the COSIpy pointing class.\n", 198 | "\n", 199 | "The pointings refer to the direction/orientation of the telescope at each point in time. This cell usually takes a few minutes to run." 200 | ] 201 | }, 202 | { 203 | "cell_type": "code", 204 | "execution_count": null, 205 | "id": "c971d763", 206 | "metadata": {}, 207 | "outputs": [], 208 | "source": [ 209 | "pointing_combined = Pointing(dataset=analysis_combined.dataset) # definition of pointings (balloon stability + Earth rotation)" 210 | ] 211 | }, 212 | { 213 | "cell_type": "markdown", 214 | "id": "770538ed", 215 | "metadata": {}, 216 | "source": [ 217 | "## Visualize the paths of the Crab & Cen A through the field-of-view.\n", 218 | "\n", 219 | "This isn't necessary for the spectral fitting, but is illustrative for understanding the pointings and exposure of the point sources." 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": null, 225 | "id": "ef2e752e", 226 | "metadata": {}, 227 | "outputs": [], 228 | "source": [ 229 | "plt.plot(pointing_combined.zpoins[:,0]+360,pointing_combined.zpoins[:,1],'o', label=\"COSI zenith pointing\")\n", 230 | "plt.plot(l_crab,b_crab,'*g',markersize=10, label=\"Crab\")\n", 231 | "plt.plot(l_cenA,b_cenA,'*r',markersize=10, label=\"Cen A\")\n", 232 | "plt.xlabel('Longitude [deg]')\n", 233 | "plt.ylabel('Latitude [deg]')\n", 234 | "plt.legend()" 235 | ] 236 | }, 237 | { 238 | "cell_type": "code", 239 | "execution_count": null, 240 | "id": "fffacde3", 241 | "metadata": {}, 242 | "outputs": [], 243 | "source": [ 244 | "analysis_combined.plot_elevation([l_crab],[b_crab],['Crab'])\n", 245 | "\n", 246 | "analysis_combined.plot_elevation([l_cenA],[b_cenA],['Cen A'])" 247 | ] 248 | }, 249 | { 250 | "cell_type": "markdown", 251 | "id": "43966556", 252 | "metadata": {}, 253 | "source": [ 254 | "# Define the BG model." 255 | ] 256 | }, 257 | { 258 | "cell_type": "code", 259 | "execution_count": null, 260 | "id": "431b879f", 261 | "metadata": {}, 262 | "outputs": [], 263 | "source": [ 264 | "# Ling BG simulation to model atmospheric background\n", 265 | "background_combined = BG(dataset=analysis_combined.dataset,mode=background_mode,filename=background_filename) # read in background" 266 | ] 267 | }, 268 | { 269 | "cell_type": "markdown", 270 | "id": "7b04eaf8", 271 | "metadata": {}, 272 | "source": [ 273 | "# Read in the Response Matrix\n", 274 | "\n", 275 | "This usually takes a few minutes." 276 | ] 277 | }, 278 | { 279 | "cell_type": "code", 280 | "execution_count": null, 281 | "id": "fbb0796f", 282 | "metadata": {}, 283 | "outputs": [], 284 | "source": [ 285 | "# continuum response\n", 286 | "rsp = response.SkyResponse(filename=response_filename,pixel_size=pixel_size) # read in detector response" 287 | ] 288 | }, 289 | { 290 | "cell_type": "markdown", 291 | "id": "b4a5de88", 292 | "metadata": {}, 293 | "source": [ 294 | "## Explore the shape of the data space.\n", 295 | "\n", 296 | "The shape of the response spans (Galactic latitude $b$, Galactic longitude $\\ell$, Compton scattering angle $\\phi$, FISBEL, energy). The shape of the data and background objects span (time, energy, Compton scattering angle, FISBEL), as explained above." 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": null, 302 | "id": "3ba4e173", 303 | "metadata": {}, 304 | "outputs": [], 305 | "source": [ 306 | "print('Shape of response matrix (b, l, ϕ, FISBEL, energy):')\n", 307 | "print(rsp.rsp.response_grid_normed_efinal.shape)\n", 308 | "print()\n", 309 | "print('Shape of binned data (time, energy, ϕ, FISBEL):')\n", 310 | "print(analysis_combined.dataset.binned_data.shape)\n", 311 | "print()\n", 312 | "print('Shape of background model (time, energy, ϕ, FISBEL):')\n", 313 | "print(np.shape(background_combined.bg_model))" 314 | ] 315 | }, 316 | { 317 | "cell_type": "markdown", 318 | "id": "f10f4ef2", 319 | "metadata": {}, 320 | "source": [ 321 | "# Calculate the point source response for the Crab." 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": null, 327 | "id": "114612b6", 328 | "metadata": {}, 329 | "outputs": [], 330 | "source": [ 331 | "rsp.calculate_PS_response(analysis_combined.dataset,pointing_combined,l_crab,b_crab,1,background=background_combined,pixel_size=pixel_size,lookup=False)" 332 | ] 333 | }, 334 | { 335 | "cell_type": "markdown", 336 | "id": "b0d35fb2", 337 | "metadata": {}, 338 | "source": [ 339 | "## Plot light curves for the data, background & sky models.\n", 340 | "\n", 341 | "This is plotted for the 220-325 keV energy bin. The sky model is normalized to 1." 342 | ] 343 | }, 344 | { 345 | "cell_type": "code", 346 | "execution_count": null, 347 | "id": "086c81ae", 348 | "metadata": {}, 349 | "outputs": [], 350 | "source": [ 351 | "plt.plot(np.sum(analysis_combined.dataset.binned_data[:,1,:,:],axis=(1,2)), label=\"Data\") # binned data light curve\n", 352 | "plt.plot(np.sum(background_combined.bg_model_reduced[1],axis=1), label=\"Background model\") # background model\n", 353 | "plt.plot(np.sum(rsp.sky_response[1],axis=1)*1000, label=\"Sky model\") # sky model\n", 354 | "plt.xlabel('Time Bins')\n", 355 | "plt.ylabel('Counts per Time Bin')\n", 356 | "plt.legend()" 357 | ] 358 | }, 359 | { 360 | "cell_type": "markdown", 361 | "id": "d9228957", 362 | "metadata": {}, 363 | "source": [ 364 | "# Extract the spectrum for the Crab.\n", 365 | "\n", 366 | "For each energy bin individually, this determines the coefficients for the sky and background models that best match the data. It can take a few hours to run!" 367 | ] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "execution_count": null, 372 | "id": "e368c0c6", 373 | "metadata": { 374 | "scrolled": true, 375 | "tags": [] 376 | }, 377 | "outputs": [], 378 | "source": [ 379 | "result_crab = fit(analysis_combined.dataset,pointing_combined,rsp,background_combined) # create fitting object\n", 380 | "result_crab.fit(iters=2000) # perform spectral fit using emcee (uses pointing definition, background model, & point source response)" 381 | ] 382 | }, 383 | { 384 | "cell_type": "markdown", 385 | "id": "1919d52e", 386 | "metadata": {}, 387 | "source": [ 388 | "## Plot the final count spectrum of the Crab.\n", 389 | "\n", 390 | "Below is the spectrum (in counts/keV) of the Crab nebula!\n", 391 | "\n", 392 | "The fitted value for the 480-520 keV bin is a bit low, which is likely due to the 511 keV line in the simulation.\n", 393 | "\n", 394 | "The extracted spectrum data is saved as a .dat file." 395 | ] 396 | }, 397 | { 398 | "cell_type": "code", 399 | "execution_count": null, 400 | "id": "7435d206", 401 | "metadata": {}, 402 | "outputs": [], 403 | "source": [ 404 | "result_crab.plot_extracted_spectrum('crab_spectrum.dat')" 405 | ] 406 | }, 407 | { 408 | "cell_type": "markdown", 409 | "id": "47f4b1d1", 410 | "metadata": {}, 411 | "source": [ 412 | "# Analysis of Cen A\n", 413 | "\n", 414 | "Now that we have successfully recovered the Crab spectrum from the full flight simulation, we will look at a slightly weaker source and discover some of the limitations with the current COSIpy classic implementation.\n", 415 | "\n", 416 | "Since we’ve already loaded the COSI-balloon simulated data, response matrix and background model, we only need to redefine the point source sky model using Cen A’s coordinates and repeat the fit." 417 | ] 418 | }, 419 | { 420 | "cell_type": "markdown", 421 | "id": "d833fa16", 422 | "metadata": {}, 423 | "source": [ 424 | "## Calculate the point source response for Cen A." 425 | ] 426 | }, 427 | { 428 | "cell_type": "code", 429 | "execution_count": null, 430 | "id": "d886441d", 431 | "metadata": {}, 432 | "outputs": [], 433 | "source": [ 434 | "rsp.calculate_PS_response(analysis_combined.dataset,pointing_combined,l_cenA,b_cenA,1,background=background_combined,pixel_size=pixel_size,lookup=False)" 435 | ] 436 | }, 437 | { 438 | "cell_type": "markdown", 439 | "id": "9344c62c", 440 | "metadata": {}, 441 | "source": [ 442 | "## Extract the spectrum for Cen A.\n", 443 | "\n", 444 | "This can take a few hours to run!" 445 | ] 446 | }, 447 | { 448 | "cell_type": "code", 449 | "execution_count": null, 450 | "id": "5c23aea0", 451 | "metadata": { 452 | "scrolled": true, 453 | "tags": [] 454 | }, 455 | "outputs": [], 456 | "source": [ 457 | "result_cenA = fit(analysis_combined.dataset,pointing_combined,rsp,background_combined) # create fitting object\n", 458 | "result_cenA.fit(iters=2000) # perform spectral fit using emcee (uses pointing definition, background model, & point source response)" 459 | ] 460 | }, 461 | { 462 | "cell_type": "markdown", 463 | "id": "c039286b", 464 | "metadata": {}, 465 | "source": [ 466 | "## Plot the spectrum of Cen A.\n", 467 | "\n", 468 | "Below is the spectrum (in counts/keV) of the Cen A, with the extracted spectrum data saved as a .dat file.\n", 469 | "\n", 470 | "The first plot includes error bars on each point, and the second shows upper limits where the signal-to-noise ratio (SNR) > 3.\n", 471 | "\n", 472 | "The error bars for Cen A are very large, and almost all energy bins have upper limits. We believe this is due to the Crab's brightness interfering with the fit. To fix this, we need to include the Crab in the background model. Because the elevation of the Crab in COSI's field of view is changing over time, our background model would now be time-dependent. However, our current fitting algorithm only includes one background parameter, so we cannot handle a time-dependent background. Future versions of COSIpy will fix this issue." 473 | ] 474 | }, 475 | { 476 | "cell_type": "code", 477 | "execution_count": null, 478 | "id": "46d164bc", 479 | "metadata": {}, 480 | "outputs": [], 481 | "source": [ 482 | "result_cenA.plot_extracted_spectrum('cenA_spectrum1.dat')\n", 483 | "result_cenA.plot_extracted_spectrum('cenA_spectrum1.dat',ul=ul)" 484 | ] 485 | }, 486 | { 487 | "cell_type": "markdown", 488 | "id": "dd5475f7", 489 | "metadata": {}, 490 | "source": [ 491 | "## Read in Cen A simulation and define analysis object.\n", 492 | "\n", 493 | "To investigate why the fit is failing using the combined simulation, we will redo the Cen A analysis, but we will use a simulation that excludes the Crab and all other sources. The 'CenA_BG_10x.tra.gz' file includes only Cen A and the Ling background, and by repeating the above analysis, we will show that we can recover the Cen A count spectrum when the background model is static. For this analysis, we will only include the required cells. Please refer to the Crab analysis above for the details of each step.\n", 494 | "\n", 495 | "This cell usually takes a few minutes to run." 496 | ] 497 | }, 498 | { 499 | "cell_type": "code", 500 | "execution_count": null, 501 | "id": "abe7803a", 502 | "metadata": {}, 503 | "outputs": [], 504 | "source": [ 505 | "analysis_cenA = COSIpy(data_dir,filename_cenA) # create analysis object\n", 506 | "analysis_cenA.read_COSI_DataSet() # read in data" 507 | ] 508 | }, 509 | { 510 | "cell_type": "markdown", 511 | "id": "0e543b96", 512 | "metadata": {}, 513 | "source": [ 514 | "## Bin the data." 515 | ] 516 | }, 517 | { 518 | "cell_type": "code", 519 | "execution_count": null, 520 | "id": "dfa93fd7", 521 | "metadata": {}, 522 | "outputs": [], 523 | "source": [ 524 | "analysis_cenA.dataset.time_binning_tags(time_bin_size=Delta_T) # time binning\n", 525 | "analysis_cenA.dataset.init_binning(energy_bin_edges=energy_bin_edges,pixel_size=pixel_size) # energy and pixel binning\n", 526 | "analysis_cenA.dataset.get_binned_data() # bin data" 527 | ] 528 | }, 529 | { 530 | "cell_type": "markdown", 531 | "id": "4dbfcdbf", 532 | "metadata": {}, 533 | "source": [ 534 | "## Define the pointing object.\n", 535 | "\n", 536 | "This cell usually takes a few minutes to run" 537 | ] 538 | }, 539 | { 540 | "cell_type": "code", 541 | "execution_count": null, 542 | "id": "a2f70a98", 543 | "metadata": {}, 544 | "outputs": [], 545 | "source": [ 546 | "pointing_cenA = Pointing(dataset=analysis_cenA.dataset)" 547 | ] 548 | }, 549 | { 550 | "cell_type": "markdown", 551 | "id": "6a92c1c3", 552 | "metadata": {}, 553 | "source": [ 554 | "## Define the background model." 555 | ] 556 | }, 557 | { 558 | "cell_type": "code", 559 | "execution_count": null, 560 | "id": "bf84ad9e", 561 | "metadata": {}, 562 | "outputs": [], 563 | "source": [ 564 | "background_cenA = BG(dataset=analysis_cenA.dataset,mode=background_mode,filename=background_filename) # read in background" 565 | ] 566 | }, 567 | { 568 | "cell_type": "markdown", 569 | "id": "88f3a996", 570 | "metadata": {}, 571 | "source": [ 572 | "## Calculate the point source response for Cen A." 573 | ] 574 | }, 575 | { 576 | "cell_type": "code", 577 | "execution_count": null, 578 | "id": "ed85434e", 579 | "metadata": {}, 580 | "outputs": [], 581 | "source": [ 582 | "rsp.calculate_PS_response(analysis_cenA.dataset,pointing_cenA,l_cenA,b_cenA,1,background=background_cenA,pixel_size=pixel_size,lookup=False)" 583 | ] 584 | }, 585 | { 586 | "cell_type": "markdown", 587 | "id": "50204ade", 588 | "metadata": {}, 589 | "source": [ 590 | "## Plot light curves for data, background & sky models.\n", 591 | "\n", 592 | "This is plotted for the 220-325 keV energy bin. The sky model is normalized to 1.\n", 593 | "\n", 594 | "You can see that the data rates appear flatter here compared to the light curve above for the combined simulation. The Crab was mainly in COSI's field of view during the second half of the flight, and since we no longer have those additional counts, there is less variation in this light curve and sky model." 595 | ] 596 | }, 597 | { 598 | "cell_type": "code", 599 | "execution_count": null, 600 | "id": "0ca701e9", 601 | "metadata": {}, 602 | "outputs": [], 603 | "source": [ 604 | "plt.plot(np.sum(analysis_cenA.dataset.binned_data[:,1,:,:],axis=(1,2)), label=\"Data\") # binned data light curve\n", 605 | "plt.plot(np.sum(background_cenA.bg_model_reduced[1],axis=1), label=\"Background model\") # background model\n", 606 | "plt.plot(np.sum(rsp.sky_response[1],axis=1)*1000, label=\"Sky model\") # sky model\n", 607 | "plt.xlabel('Time Bins')\n", 608 | "plt.ylabel('Counts per Time Bin')\n", 609 | "plt.legend()" 610 | ] 611 | }, 612 | { 613 | "cell_type": "markdown", 614 | "id": "4354e97c", 615 | "metadata": {}, 616 | "source": [ 617 | "## Extract the spectrum for Cen A.\n", 618 | "\n", 619 | "This can take a few hours to run!" 620 | ] 621 | }, 622 | { 623 | "cell_type": "code", 624 | "execution_count": null, 625 | "id": "adf06888", 626 | "metadata": { 627 | "scrolled": true, 628 | "tags": [] 629 | }, 630 | "outputs": [], 631 | "source": [ 632 | "result_cenA2 = fit(analysis_cenA.dataset,pointing_cenA,rsp,background_cenA) # create fitting object\n", 633 | "result_cenA2.fit(iters=2000) # perform spectral fit using emcee (uses pointing definition, background model, & point source response)" 634 | ] 635 | }, 636 | { 637 | "cell_type": "markdown", 638 | "id": "d8aa056b", 639 | "metadata": {}, 640 | "source": [ 641 | "## Plot the final count spectrum of Cen A.\n", 642 | "\n", 643 | "Below is the spectrum (in counts/keV) of Centaurus A!\n", 644 | "\n", 645 | "When we use a simulation of only Cen A and background, we are able to recover the correct spectrum. This emphasizes the need for more sophisticated background models which include time variability and bright sources, and this will be a focus for the next Data Challenge." 646 | ] 647 | }, 648 | { 649 | "cell_type": "code", 650 | "execution_count": null, 651 | "id": "23a56f50", 652 | "metadata": {}, 653 | "outputs": [], 654 | "source": [ 655 | "result_cenA2.plot_extracted_spectrum('cenA_only_spectrum1.dat')\n", 656 | "result_cenA2.plot_extracted_spectrum('cenA_only_spectrum1.dat',ul=ul)" 657 | ] 658 | } 659 | ], 660 | "metadata": { 661 | "kernelspec": { 662 | "display_name": "Python 3", 663 | "language": "python", 664 | "name": "python3" 665 | }, 666 | "language_info": { 667 | "codemirror_mode": { 668 | "name": "ipython", 669 | "version": 3 670 | }, 671 | "file_extension": ".py", 672 | "mimetype": "text/x-python", 673 | "name": "python", 674 | "nbconvert_exporter": "python", 675 | "pygments_lexer": "ipython3", 676 | "version": "3.8.10" 677 | } 678 | }, 679 | "nbformat": 4, 680 | "nbformat_minor": 5 681 | } 682 | -------------------------------------------------------------------------------- /xspec-tutorial/Figures/effective_area.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/Figures/effective_area.png -------------------------------------------------------------------------------- /xspec-tutorial/Figures/source_and_background_grp8ch_spectrum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/Figures/source_and_background_grp8ch_spectrum.png -------------------------------------------------------------------------------- /xspec-tutorial/Figures/source_and_background_spectrum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/Figures/source_and_background_spectrum.png -------------------------------------------------------------------------------- /xspec-tutorial/Figures/unfolded_crab_simulation.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/Figures/unfolded_crab_simulation.png -------------------------------------------------------------------------------- /xspec-tutorial/Figures/unfolded_crab_simulation_with_background.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/Figures/unfolded_crab_simulation_with_background.png -------------------------------------------------------------------------------- /xspec-tutorial/Figures/unfolded_spectrum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/Figures/unfolded_spectrum.png -------------------------------------------------------------------------------- /xspec-tutorial/README.md: -------------------------------------------------------------------------------- 1 | # Using XSPEC for COSI spectral analysis and simulations 2 | 3 | This tutorial is meant to serve as a starting point for using the X-ray spectral fitting package (XSPEC) with COSI data. We will start with an extracted data set from the 2016 COSI balloon flight during its observation of GRB 160530A (source_and_background.pha). This tutorial will walk you through the fit of this data set, and then will show an example of how to simulate sources using the provided instrument response and background files. In future Data Challenges, we will release tools with which users can make their own response files and conduct XSPEC analyses with more flexibility. 4 | 5 | XSPEC is an extremely mature software package used by a large portion of the high-energy community. It contains a wide range of models and allows users to define custom models as needed. Once the basic commands are known, it is easy to use, and spectral analysis can be performed efficiently. It has a very convenient spectral simulation tool that will be helpful to the COSI science team and will broaden the COSI community. Although not discussed in this tutorial, there is a pyXSPEC package that allows for automated scripts to be written. The [main XSPEC website](https://heasarc.gsfc.nasa.gov/xanadu/xspec/) provides a more extensive manual. XSPEC is installed by default as part of the HEASoft tools and in turn as part of the COSItools. 6 | 7 | Here, we assume that the user has extracted a spectral file (source_and_background.pha), a background file (background.pha), a response matrix (response.rmf), and an ancillary response file (area.arf) - these are provided in the above download. **Command#1** below gives an example of using the FTOOL `fdump` to see what is inside the file. The useful information is in the FITS file binary table extension (see more details about FITS files [here](https://fits.gsfc.nasa.gov/fits_documentation.html)). For example, there are two fields in each row of data (CHANNEL and COUNTS), and there are 502 rows in the table. 8 | 9 | It is important to check that you have specified the names of the background file, the response matrix, and the ancillary response file in the header under the keywords BACKFILE, RESPFILE, and ANCRFILE. When you load the file into XSPEC, these files are read in automatically from the same directory. The easiest way to put the file names in the header is to use the FTOOL `fv`, search for the keywords, and type or copy the filenames in. 10 | 11 | Both the `fdump` and `fv` commands are part of FTOOLS and can be used in the regular command line or from inside XSPEC. 12 | 13 | **Command#1** 14 | ``` 15 | [MacBook-Pro-4:cosi/2022/tutorial] jat% fdump source_and_background.pha 16 | ``` 17 | The output will be as follows: 18 | ``` 19 | Name of optional output file[STDOUT] 20 | Names of columns[-] 21 | Lists of rows[-] 22 | SIMPLE = T / file does conform to FITS standard 23 | BITPIX = -32 / number of bits per data pixel 24 | NAXIS = 0 / number of data axes 25 | EXTEND = T / FITS dataset may contain extensions 26 | COMMENT FITS (Flexible Image Transport System) format is defined in 'Astronomy 27 | COMMENT and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H 28 | TELESCOP= 'COSI ' 29 | INSTRUME= 'COSI ' 30 | END 31 | 32 | XTENSION= 'BINTABLE' / binary table extension 33 | BITPIX = 8 / 8-bit bytes 34 | NAXIS = 2 / 2-dimensional binary table 35 | NAXIS1 = 8 / width of table in bytes 36 | NAXIS2 = 502 / number of rows in table 37 | PCOUNT = 0 / size of special data area 38 | GCOUNT = 1 / one data group (required keyword) 39 | TFIELDS = 2 / number of fields in each row 40 | TTYPE1 = 'CHANNEL ' / label for field 1 41 | TFORM1 = 'J ' / data format of field: 4-byte INTEGER 42 | TTYPE2 = 'COUNTS ' / label for field 2 43 | TFORM2 = 'J ' / data format of field: 4-byte INTEGER 44 | TUNIT2 = 'count ' / physical unit of field 45 | EXTNAME = 'SPECTRUM' / name of this binary table extension 46 | TELESCOP= 'COSI ' / telescope/mission name 47 | INSTRUME= 'COSI ' / instrument/detector name 48 | EXTNAME = 'SPECTRUM' / name of extension 49 | FILTER = 'NONE ' / filter type if any 50 | EXPOSURE= 38.912 / integration time in seconds 51 | BACKFILE= 'background.pha' / background filename 52 | BACKSCAL= 1. / background scaling factor 53 | CORRFILE= 'NONE ' / associated correction filename 54 | CORRSCAL= 1. / correction file scaling factor 55 | RESPFILE= 'response.rmf' / associated rmf filename 56 | ANCRFILE= 'area.arf' / associated arf filename 57 | AREASCAL= 1. / area scaling factor 58 | STAT_ERR= 0 / no statistical error specified 59 | SYS_ERR = 0 / no systematic error specified 60 | GROUPING= 0 / no grouping of the data has been defined 61 | QUALITY = 0 / no data quality information specified 62 | HDUCLASS= 'OGIP ' / format conforms to OGIP standard 63 | HDUCLAS1= 'SPECTRUM' / PHA dataset 64 | HDUVERS = '1.2.1 ' / version of format 65 | POISSERR= T / Poissonian errors to be assumed 66 | CHANTYPE= 'PI ' / channel type (PHA or PI) 67 | DETCHANS= 502 / total number of detector channels 68 | HISTORY File modified by user 'jat' with fv on 2022-06-06T22:37:16 69 | END 70 | 71 | CHANNEL COUNTS 72 | count 73 | 1 0 0 74 | 2 1 0 75 | 3 2 0 76 | 4 3 0 77 | 5 4 0 78 | … 79 | ``` 80 | 81 | ## Part 1: Standard spectral fitting with XSPEC 82 | 83 | You will need to add the path to HEASoft to your login file. For csh, tsh shells: 84 | ``` 85 | setenv HEADAS /path/to/your/COSItools/external/heasoft_v6.30.1/x86_64-apple-darwin21.5.0 86 | alias heasoft "source $HEADAS/headas-init.csh" 87 | ``` 88 | For sh, ash, ksh, bash, and zsh: 89 | ``` 90 | export HEADAS="/path/to/your/COSItools/external/heasoft_v6.30.1/x86_64-apple-darwin21.5.0" 91 | alias heasoft="source $HEADAS/headas-init.csh" 92 | ``` 93 | 94 | Then, type `heasoft` to initialize HEASOFT and `xspec` to start an XSPEC session. 95 | ``` 96 | [MacBook-Pro-4:cosi/2022/tutorial] jat% xspec 97 | ``` 98 | You will see the xspec version returned. In our example, we have XSPEC 12.12.0: 99 | ``` 100 | XSPEC version: 12.12.0 101 | Build Date/Time: Fri Sep 3 10:58:24 2021 102 | ``` 103 | 104 | Then, load the GRB160530A spectral file with the `data` command. 105 | ``` 106 | XSPEC12>data source_and_background.pha 107 | ``` 108 | The following information about the spectral file will be printed to the command line: 109 | ``` 110 | 1 spectrum in use 111 | 112 | Spectral Data File: source_and_background.pha Spectrum 1 113 | Net count rate (cts/s) for Spectrum:1 1.590e+01 +/- 9.214e-01 (49.8 % total) 114 | Assigned to Data Group 1 and Plot Group 1 115 | Noticed Channels: 1-502 116 | Telescope: COSI Instrument: COSI Channel Type: PI 117 | Exposure Time: 38.91 sec 118 | Using fit statistic: chi 119 | Using Background File background.pha 120 | Background Exposure Time: 573 sec 121 | Using Response (RMF) File response.rmf for Source 1 122 | Using Auxiliary Response (ARF) File area.arf 123 | ``` 124 | To see what the spectrum looks like, use the following commands: 125 | `XSPEC12>iplot` (with this command, the use enters “pgplot”) 126 | `PLT>device /xs` (this makes plots go to the computer screen; note that d /xs also works) 127 | `PLT>time off` (turns off the timestamp at the bottom of the window) 128 | `PLT>lw 5` (makes line widths five times thicker) 129 | `PLT>lw 5 on 1` (makes the data group 1 point five times thicker) 130 | `PLT>font roman` (uses times roman fonts rather than the default) 131 | `PLT>plot` 132 | 133 | You should see a spectrum that looks exactly like the one shown below. 134 | ![Figure 1](Figures/source_and_background_spectrum.png) 135 | 136 | Now, you can leave pgplot by typing `exit` or `quit`. And for future reference, to end an XSPEC session and return to a regular command line, the command `exit` is used. 137 | 138 | The next step is to bin the spectrum and to remove the channels outside the spectral range. Here, we will use a standard set of bins (provided in groups8ch.dat). Other techniques for binning are described in an Appendix. We bin the spectra and mark the bad channels using the FTOOL grppha (once again, this can be done from within XSPEC or from a regular command line): 139 | 140 | ``` 141 | XSPEC12>grppha source_and_background.pha 142 | Please enter output filename[] source_and_background_grp8ch.pha 143 | 144 | ------------------------- 145 | MANDATORY KEYWORDS/VALUES 146 | ------------------------- 147 | -------------------------------------------------------------------- 148 | -------------------------------------------------------------------- 149 | EXTNAME - SPECTRUM Name of this BINTABLE 150 | TELESCOP - COSI Mission/Satellite name 151 | INSTRUME - COSI Instrument/Detector 152 | FILTER - NONE Instrument filter in use 153 | EXPOSURE - 38.912 Integration time (in secs) of PHA data 154 | AREASCAL - 1.0000 Area scaling factor 155 | BACKSCAL - 1.0000 Background scaling factor 156 | BACKFILE - background.pha Associated background file 157 | CORRSCAL - 1.0000 Correlation scaling factor 158 | CORRFILE - NONE Associated correlation file 159 | RESPFILE - response.rmf Associated redistribution matrix file 160 | ANCRFILE - area.arf Associated ancillary response file 161 | POISSERR - TRUE Whether Poissonian errors apply 162 | CHANTYPE - PI Whether channels have been corrected 163 | TLMIN1 - 0 First legal Detector channel 164 | DETCHANS - 502 No. of legal detector channels 165 | NCHAN - 502 No. of detector channels in dataset 166 | PHAVERSN - 1.2.1 OGIP FITS version number 167 | STAT_ERR - FALSE Statistical Error 168 | SYS_ERR - FALSE Fractional Systematic Error 169 | QUALITY - TRUE Quality Flag 170 | GROUPING - FALSE Grouping Flag 171 | -------------------------------------------------------------------- 172 | -------------------------------------------------------------------- 173 | ``` 174 | And now adding the channels to be removed: 175 | 176 | `GRPPHA[] bad 0-180` (these channels will not change as long as you are using groups8ch.dat) 177 | 178 | `GRPPHA[] bad 377-501` (these channels will not change as long as you are using groups8ch.dat) 179 | 180 | And reading in the pre-defined binning: 181 | ``` 182 | GRPPHA[] group groups8ch.dat 183 | GRPPHA[] exit 184 | ``` 185 | (Note that you will see text in the square brackets. The text comes from the grppha parameter file. If the text in the square brackets is the command that you want, then you can press return instead of typing in the command.) 186 | 187 | Now, read in the file that was created after grouping 188 | 189 | `XSPEC12> data source_and_background_grp8ch.pha` 190 | 191 | Remove the bins outside the spectral range with 192 | 193 | `XSPEC12> ignore bad` 194 | 195 | We also convert the x-axis to energy using 196 | 197 | `XSPEC12> setplot energy` 198 | 199 | We plot the spectrum using 200 | ``` 201 | XSPEC12> plot ldata 202 | XSPEC12> iplot 203 | ``` 204 | (and the other commands in pgplot above) and the figure below shows the result. 205 | 206 | ![Figure 2](Figures/source_and_background_grp8ch_spectrum.png) 207 | 208 | Now, we are ready to specify the model (we’ll use a power-law here) and fit the GRB spectrum. 209 | ``` 210 | XSPEC12> model powerlaw 211 | ``` 212 | There is an option to choose starting values of parameters, but for such a simple model, it is fine just to accept the default values by pressing “return.” Even though the starting parameters are not close to the correct values, we proceed by typing 213 | ``` 214 | XSPEC12> fit 215 | ``` 216 | The output will look like the following: 217 | ``` 218 | Parameters 219 | Chi-Squared |beta|/N Lvl 1:PhoIndex 2:norm 220 | 62.1745 32.638 -1 1.03130 0.479344 221 | 58.6047 10.0731 -1 1.06020 0.583287 222 | 55.1013 10.8614 -1 1.08810 0.708488 223 | 51.7511 9.45984 -1 1.11522 0.855737 224 | 48.5738 7.90947 -1 1.14154 1.02710 225 | 45.5754 6.53869 -1 1.16706 1.22482 226 | 42.7575 5.35105 -1 1.19176 1.45110 227 | 40.1189 4.31784 -1 1.21563 1.70802 228 | 37.6563 3.41706 -1 1.23867 1.99750 229 | 35.3644 2.63735 -1 1.26089 2.32124 230 | Number of trials exceeded: continue fitting? 231 | ``` 232 | If the fit doesn’t converge in 10 iterations, then you need to type “y” or press the return key to continue with the fit. You can also use the command “query yes” to override the 10 iteration limit, or type “fit 100” to change the number of iterations before continuing from 10 to 100. After the fit does converge, XSPEC will show you the best-fit parameter values and minimum Chi-Squared value (we'll address these below). 233 | 234 | You can see the model on the data and the residuals using 235 | ``` 236 | XSPEC12> plot ldata del 237 | ``` 238 | 239 | And you can see the spectrum converted to flux units (sometimes called the “unfolded” spectrum) using 240 | ``` 241 | XSPEC12> plot eeuf del 242 | ``` 243 | The figure below shows the GRB spectrum fitted with a power-law and the residuals. After typing 244 | ``` 245 | XSPEC12> iplot 246 | ``` 247 | we invoke other commands, including 248 | 249 | `PLT> lw 5 on 1` (to make the data points thick) 250 | 251 | `PLT> lw 5 on 2` (to make the model thick) 252 | 253 | `PLT> lw 5 on 3` (to make the residual points thick) 254 | 255 | `PLT> r y 100 800` (to set the y-range to be 100 to 800) 256 | 257 | `PLT> window 2` (to switch to the bottom panel) 258 | 259 | `PLT> r y -2.9 2.9` (to set the y-range to be from -2.9 to 2.9) 260 | 261 | ![Figure 3](Figures/unfolded_spectrum.png) 262 | 263 | The parameters for the fit are below. Please note that the default uncertainties are approximations. 264 | 265 | ``` 266 | ======================================================================== 267 | Model powerlaw<1> Source No.: 1 Active/On 268 | Model Model Component Parameter Unit Value 269 | par comp 270 | 1 1 powerlaw PhoIndex 1.80459 +/- 0.116120 271 | 2 1 powerlaw norm 73.5619 +/- 51.3469 272 | ________________________________________________________________________ 273 | 274 | Using energies from responses. 275 | 276 | Fit statistic : Chi-Squared 8.55 using 8 bins. 277 | 278 | Test statistic : Chi-Squared 8.55 using 8 bins. 279 | Null hypothesis probability of 2.00e-01 with 6 degrees of freedom 280 | 281 | Bayes is off 282 | ``` 283 | It is necessary to use the “error” command to calculate reliable errors. 284 | 285 | ``` 286 | XSPEC12>error 2.7 1 287 | Parameter Confidence Range (2.7) 288 | 1 1.62291 2.00488 (-0.181687,0.200285) 289 | ``` 290 | 291 | In the error command above, the 2.7 refers to the value by which the fit statistic changes at the extremes of the confidence range, and 2.7 corresponds to 90% confidence errors. The "1" after 2.7 refers to the parameter number; in this case, we are interested in the error on the photon index. Also, note that the Chi-Squared value is 8.55 for 6 degrees of freedom. With such a small number of degrees of freedom, a reduced Chi-Squared of 1.43 would be considered to be an acceptable fit (using the Chi-Squared distribution, there is a probability of 20% that the true model would have a Chi-Squared of 1.43). 292 | 293 | This is the final spectrum of GRB 160530A from the 2016 COSI balloon flight, as published in [Sleator 2019](https://escholarship.org/uc/item/0zn566rj). 294 | 295 | Before moving on to another spectrum or a simulation (see Part 2), you should use the following command 296 | ``` 297 | XSPEC12> data none 298 | ``` 299 | to clear the previous data file. A command is to clear the model is 300 | ``` 301 | XSPEC12> model none 302 | ``` 303 | but loading a new model automatically clears the old model, so there are few (if any) cases when it is necessary to use it. 304 | 305 | 306 | ## Part 2: Spectral simulations 307 | 308 | One of the very nice features of XSPEC is the simplicity for making simulated spectra. All that is needed is a rmf, an arf, and a background file. If you have a source plus background file, you can start by reading that into XSPEC (if you only have a background file, then you can start by loading that instead, basically as a dummy file). You should use an unbinned spectrum such as the following 309 | ``` 310 | XSPEC12> data source_and_background.pha 311 | ``` 312 | Now, load the model that you would like to simulate. Here, we will use the model that [Roques et al. 2019](https://ui.adsabs.harvard.edu/abs/2019ApJ...870...92R/abstract) used to fit the Crab. This model is a Band function (often used for GRBs). You will have to manually input the parameters for the model as follows: 313 | ``` 314 | XSPEC12>model grbm 315 | 316 | Input parameter value, delta, min, bot, top, and max values for ... 317 | -1 0.01( 0.01) -10 -3 2 5 318 | 1:grbm:alpha>-1.98 319 | -2 0.01( 0.02) -10 -5 2 10 320 | 2:grbm:beta>-2.32 321 | 300 10( 3) 10 50 1000 10000 322 | 3:grbm:tem>489 323 | 1 0.01( 0.01) 0 0 1e+20 1e+24 324 | 4:grbm:norm>0.000745 325 | 326 | ======================================================================== 327 | Model grbm<1> Source No.: 1 Active/On 328 | Model Model Component Parameter Unit Value 329 | par comp 330 | 1 1 grbm alpha -1.98000 +/- 0.0 331 | 2 1 grbm beta -2.32000 +/- 0.0 332 | 3 1 grbm tem keV 489.000 +/- 0.0 333 | 4 1 grbm norm 7.45000E-04 +/- 0.0 334 | ________________________________________________________________________ 335 | ``` 336 | The first time you load the model, you will type in the parameters as in the example above. Then you can save the model a command like the following: 337 | ``` 338 | XSPEC12>save model your_filename 339 | ``` 340 | The next time you want to load the model, you can use the command: 341 | ``` 342 | XSPEC12>@your_filename.xcm 343 | ``` 344 | Now, we are ready to perform the simulation. 345 | ``` 346 | XSPEC12>fakeit 347 | Use counting statistics in creating fake data? (y): 348 | Input optional fake file prefix: 349 | Fake data file name (source_and_background.fak): source_and_background_crabr.pha 350 | Exposure time, correction norm, bkg exposure time (38.9120, 1.00000, 573.000): 1000000,1,1000000 351 | ``` 352 | 353 | Here, we have made a simulated spectrum with a 1 Ms exposure time called source_and_background_crabr.pha (source plus background). XSPEC also makes a background file (source_and_background_crabr_bkg.pha). We have chosen it to also have an exposure time of 1 Ms. 354 | 355 | Now, we need to use grppha again to mark the bad channels and to rebin (we will use the same standard bins we used previously). 356 | ``` 357 | XSPEC12>grppha source_and_background_crabr.pha 358 | Please enter output filename[]source_and_background_crabr_grp8ch.pha 359 | 360 | ------------------------- 361 | MANDATORY KEYWORDS/VALUES 362 | ------------------------- 363 | -------------------------------------------------------------------- 364 | -------------------------------------------------------------------- 365 | EXTNAME - SPECTRUM Name of this BINTABLE 366 | TELESCOP - COSI Mission/Satellite name 367 | INSTRUME - COSI Instrument/Detector 368 | FILTER - Instrument filter in use 369 | EXPOSURE - 1.00000E+06 Integration time (in secs) of PHA data 370 | AREASCAL - 1.0000 Area scaling factor 371 | BACKSCAL - 1.0000 Background scaling factor 372 | BACKFILE - source_and_background_crabr_bkg.pha 373 | CORRSCAL - 1.0000 Correlation scaling factor 374 | CORRFILE - NONE Associated correlation file 375 | RESPFILE - response.rmf Associated redistribution matrix file 376 | ANCRFILE - area.arf Associated ancillary response file 377 | POISSERR - TRUE Whether Poissonian errors apply 378 | CHANTYPE - PI Whether channels have been corrected 379 | TLMIN1 - 0 First legal Detector channel 380 | DETCHANS - 502 No. of legal detector channels 381 | NCHAN - 502 No. of detector channels in dataset 382 | PHAVERSN - 1.2.1 OGIP FITS version number 383 | STAT_ERR - FALSE Statistical Error 384 | SYS_ERR - FALSE Fractional Systematic Error 385 | QUALITY - TRUE Quality Flag 386 | GROUPING - TRUE Grouping Flag 387 | -------------------------------------------------------------------- 388 | -------------------------------------------------------------------- 389 | GRPPHA[] bad 0-180 390 | GRPPHA[] bad 377-501 391 | GRPPHA[] group groups8ch.dat 392 | GRPPHA[] exit 393 | ``` 394 | 395 | Now, read in the file that was created after grouping 396 | ``` 397 | XSPEC12> data source_and_background_crabr_grp8ch.pha 398 | ``` 399 | Remove the bins outside the spectral range with 400 | ``` 401 | XSPEC12> ignore bad 402 | ``` 403 | We also convert the x-axis to energy using 404 | ``` 405 | XSPEC12> setplot energy 406 | ``` 407 | 408 | We first fit it with the Band function (grbm). The Chi-Squared was 2.95 for 4 degrees of freedom. Only one of the four grbm parameters are well-constrained, indicating that the statistical quality of the spectrum is not high enough to require all the free parameters in the grbm model. Thus, we switch to a power-law, and the Chi-Squared is 2.95 for 4 degrees of freedom, indicating that the fit is not improved by the additional free parameters in the Band function. The conclusion is that this COSI spectrum would provide a good measurement of the slope of the Crab but that it would not allow us to measure the change in slope that was measured by INTEGRAL/SPI (Roques et al. 2019). Of course, keep in mind that the files we are using here are for the COSI-Balloon instrument with the balloon background, so they should not be used to draw any conclusions about the SMEX. 409 | 410 | The power-law results and the unfolded spectrum are shown below. Your spectrum will not look exactly the same. Each simulated spectrum will look somewhat different because of the random number generator. 411 | ``` 412 | ======================================================================== 413 | Model powerlaw<1> Source No.: 1 Active/On 414 | Model Model Component Parameter Unit Value 415 | par comp 416 | 1 1 powerlaw PhoIndex 2.34058 +/- 5.16072E-02 417 | 2 1 powerlaw norm 30.5604 +/- 9.05992 418 | ________________________________________________________________________ 419 | 420 | Using energies from responses. 421 | 422 | Fit statistic : Chi-Squared 2.95 using 8 bins. 423 | 424 | Test statistic : Chi-Squared 2.95 using 8 bins. 425 | Null hypothesis probability of 8.15e-01 with 6 degrees of freedom 426 | 427 | Bayes is off 428 | 429 | XSPEC12>error 2.7 1 430 | Parameter Confidence Range (2.7) 431 | 1 2.25552 2.42918 (-0.0850525,0.0885994) 432 | ``` 433 | ![Figure 4](Figures/unfolded_crab_simulation.png) 434 | 435 | ## Part 3: A couple more useful XSPEC tools 436 | 437 | XSPEC can be quite useful for troubleshooting issues with energy spectra. If something looks suspicious when loading a file for spectral fitting or performing a simulation, it is very easy to see what XSPEC is using for the effective area and for the background. The following command shows the effective area, which can be seen in Figure 5. 438 | ``` 439 | XSPEC12>plot eff 440 | ``` 441 | ![Figure 5](Figures/effective_area.png) 442 | 443 | Also, when plotting the spectrum, the command 444 | ``` 445 | XSPEC12>setplot back 446 | ``` 447 | adds the background spectrum to the plot. In Figure 6, the bottom points are the Crab spectrum after background subtraction (i.e. the source spectrum). The top points (with the crosses) show the background (for example, you can see the 511 keV line). 448 | 449 | ![Figure 6](Figures/unfolded_crab_simulation_with_background.png) 450 | 451 | The tutorial above describes very basic uses of XSPEC. Additional instructions can be added for advanced users (hopefully, everyone performing COSI spectral analysis will eventually be an advanced user). Some ideas for more advanced topics are: 452 | - More complicated models 453 | - Setting starting values and limiting ranges for parameters 454 | - Freezing and thawing parameters 455 | - Spectral binning 456 | 457 | 458 | -------------------------------------------------------------------------------- /xspec-tutorial/area.arf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/area.arf -------------------------------------------------------------------------------- /xspec-tutorial/background.pha: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/background.pha -------------------------------------------------------------------------------- /xspec-tutorial/groups8ch.dat: -------------------------------------------------------------------------------- 1 | 181 204 24 2 | 205 230 26 3 | 231 255 25 4 | 256 260 5 5 | 261 286 26 6 | 287 311 25 7 | 312 336 25 8 | 337 376 40 9 | -------------------------------------------------------------------------------- /xspec-tutorial/response.rmf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/response.rmf -------------------------------------------------------------------------------- /xspec-tutorial/source_and_background.pha: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cositools/cosi-data-challenge-1/059056ed5a8bf6cc5aae953dad2f715189a10c8f/xspec-tutorial/source_and_background.pha --------------------------------------------------------------------------------