├── README.md └── notebooks └── workflow_demo1 ├── workflow_demo1_unrendered.ipynb └── workflow_demo1_rendered.ipynb /README.md: -------------------------------------------------------------------------------- 1 | # ccsp_supplement 2 | 3 | Tutorials and supplementary content for Roman Community Contributed Science Products. 4 | -------------------------------------------------------------------------------- /notebooks/workflow_demo1/workflow_demo1_unrendered.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "04942737-7b27-41de-ab74-929e784fd2fe", 6 | "metadata": {}, 7 | "source": [ 8 | "# An Example ASDF Design Workflow for CCSPs" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "id": "e11569a7-f499-491f-abd3-d0a5782c0447", 14 | "metadata": {}, 15 | "source": [ 16 | "## Overview\n", 17 | "\n", 18 | "In this notebook, we'll illustrate one possible method for starting to incorporate ASDF file design into your workflow and pipeline.\n", 19 | "\n", 20 | "We'll assume here that you're starting with a FITS file, though there is no need to do so. We'll extract the WCS from this FITS file's header and convert it into something ASDF-flavored, and then we'll extract the rest of the metadata from the FITS header. (But if you need more a complicated WCS than FITS can easily support, or if you are making alterations to a Roman ASDF file, then you probably *shouldn't* start with a FITS File.)\n", 21 | "\n", 22 | "The rest of the example design process would, in full, go something like this:\n", 23 | "- Design an ASDF tree structure and sample content for your product, passing nested dictionaries into an ASDF file object.\n", 24 | "- Save the resultant ASDF file object to an ASDF file: your initial sample file (without a corresponding schema).\n", 25 | "- Draft a schema that matches the initial sample file you just designed, in consultation with MAST and the RAD maintainers. Send the initial sample file and draft schema to MAST and the RAD maintainers in a Github issue to the RAD repository, which will evolve into a pull request in consultation with MAST and the RAD maintainers. **Note:** We'll largely elide the details of this step in this notebook. For more information on the schema-writing part of the process, see the [File Design Guidelines](https://outerspace.stsci.edu/spaces/ISWG/pages/356669707/File+Design+for+PITs).\n", 26 | "- Install the appropriate branch/fork of the RAD and roman_datamodels (with MAST and the RAD's help) into your pipeline environment.\n", 27 | "- Pass your ASDF tree into a Roman data model object. Save the result as your revised sample file, which now tags your schema, and deliver it to MAST for validation.\n", 28 | "\n", 29 | "**Note:** this is not the only possible workflow. For example, some people prefer to design the schema first, before constructing the ASDF file object." 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "id": "4d25895c-629c-421d-a123-7168740862a5", 35 | "metadata": {}, 36 | "source": [ 37 | "## Imports and installations" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "id": "2cac22ca-0538-4edb-aab3-72a64a87f04c", 43 | "metadata": {}, 44 | "source": [ 45 | "Later, we'll need a particular branch in a particular fork of the `roman_datamodels` repository, where we've set up an ASDF schema and Roman data model for the product we'll be converting. You should only use this fork in the context of this specific notebook; when you're ready to start working on your own data, install the main branch of [roman_datamodels](https://github.com/spacetelescope/roman_datamodels) proper." 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "id": "3642a987-2178-448d-8750-ed2637414c2c", 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "%%capture captured\n", 56 | "!pip install git+https://github.com/adrianlucy/roman_datamodels.git@ccsp_schemas_for_notebook" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "id": "e771e420-8dc4-4f93-8f60-bb5c9cea99d4", 62 | "metadata": {}, 63 | "source": [ 64 | "You may need to restart your kernel for that to take effect.\n", 65 | "\n", 66 | "That installation, incidentally, installed everything else that we need as dependencies. So now let's run our imports:" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": null, 72 | "id": "7da07c96-10f3-4e2b-be24-3be2bcefce82", 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "from astropy.io import fits # For loading the FITS file that we'll convert\n", 77 | "import numpy as np # For array and matrix wrangling\n", 78 | "\n", 79 | "import asdf # For building the ASDF tree\n", 80 | "\n", 81 | "from astropy.time import Time # For passing Time objects into the ASDF tree\n", 82 | "\n", 83 | "import gwcs # For building the WCS\n", 84 | "from gwcs import coordinate_frames as cf # For building the WCS\n", 85 | "from astropy.modeling import models # For building the WCS\n", 86 | "from astropy import coordinates as coord # For building the WCS\n", 87 | "\n", 88 | "import roman_datamodels.datamodels as rdm # Only used for the final step" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "id": "975c29a7-89eb-4853-8e0c-d52cffd0892f", 94 | "metadata": {}, 95 | "source": [ 96 | "## Opening the FITS file" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "id": "fdb9db35-d507-4cd2-8c9e-dcf7094d03cd", 102 | "metadata": {}, 103 | "source": [ 104 | "Next, we'll open the FITS file that we want to convert to ASDF. For the purposes of this tutorial, we'll use a simple 2D image file from the [FIMS-SPEAR mission](https://outerspace.stsci.edu/spaces/SPEARFIMS/overview). The main science data array is in the primary HDU, which is accompanied by concomitant images of net photon counts and an exposure map in HDU1 and HDU2, respectively." 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": null, 110 | "id": "d85642bf-71ef-495a-af87-6812ebcd4cd2", 111 | "metadata": {}, 112 | "outputs": [], 113 | "source": [ 114 | "# For best practice, in your pipeline this would be `with fits.open(\"filename\"):`\n", 115 | "# Get what you need, then close the file.\n", 116 | "hdul = fits.open(\"https://archive.stsci.edu/mccm/fims-spear/spear/vela/mccm_fims-spear_spear-ap100_vela_long-c-iv_v1.0_img.fits\")\n", 117 | "\n", 118 | "hdul.info()" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "id": "7461fa57-a454-49ec-8683-0fd6e6100805", 124 | "metadata": {}, 125 | "source": [ 126 | "Let's take a look at the headers for each HDU. We see that most of the metadata is in the primary header, and that the concomitant images share the primary array's World Coordinate System (WCS). The WCS maps the array pixel coordinates to world coordinates like wavelength, time, or (in this case) sky coordinates." 127 | ] 128 | }, 129 | { 130 | "cell_type": "code", 131 | "execution_count": null, 132 | "id": "48740c00-41bc-44df-b186-be86e664cbcb", 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "header0 = hdul[0].header\n", 137 | "header0" 138 | ] 139 | }, 140 | { 141 | "cell_type": "code", 142 | "execution_count": null, 143 | "id": "c4b53735-8db7-41b3-a84e-a7244044ee6c", 144 | "metadata": {}, 145 | "outputs": [], 146 | "source": [ 147 | "hdul[1].header" 148 | ] 149 | }, 150 | { 151 | "cell_type": "code", 152 | "execution_count": null, 153 | "id": "f601eb75-8d7c-49e3-925d-9ae91b3c7c41", 154 | "metadata": {}, 155 | "outputs": [], 156 | "source": [ 157 | "hdul[2].header" 158 | ] 159 | }, 160 | { 161 | "cell_type": "markdown", 162 | "id": "7e8f3b58-426a-43aa-a3a1-fa99c4c95cea", 163 | "metadata": {}, 164 | "source": [ 165 | "## Converting FITS WCS to GWCS" 166 | ] 167 | }, 168 | { 169 | "cell_type": "markdown", 170 | "id": "2a88e8d2-a0c3-4be7-8513-0b3fe2b371c5", 171 | "metadata": {}, 172 | "source": [ 173 | "`GWCS` allows for storing both simple WCS solutions and more complex WCS solutions with distortions and/or nonlinear components. Since we are converting a simple FITS WCS in this example, we'll use `gwcs.FITSImagingWCSTransform`. Note that Python and GWCS use 0-based indexing, while FITS uses 1-based indexing, so we'll need a 1 pixel offset to the `CRPIXj` pixel coordinates." 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": null, 179 | "id": "55117648-687d-47de-8157-214378a293f2", 180 | "metadata": {}, 181 | "outputs": [], 182 | "source": [ 183 | "# The header only has PC1_1 and PC2_2, so use the default values for PC1_2 and PC2_1\n", 184 | "pc1_2 = 0.0\n", 185 | "pc2_1 = 0.0\n", 186 | "\n", 187 | "# Construct the transform based on the FITS header\n", 188 | "fwcs_transform = gwcs.FITSImagingWCSTransform(\n", 189 | " models.Pix2Sky_Gnomonic(), # Apply a gnomonic projection\n", 190 | " crpix=[header0['CRPIX1']-1, header0['CRPIX2']-1], # correct for FITS 1-based indexing\n", 191 | " crval=[header0['CRVAL1'], header0['CRVAL2']],\n", 192 | " cdelt=[header0['CDELT1'], header0['CDELT2']],\n", 193 | " pc=[[header0['PC1_1'], pc1_2], [pc2_1, header0['PC2_2']]], # set the PCi_j matrix\n", 194 | ")\n", 195 | "\n", 196 | "# Define the frames\n", 197 | "pixel_frame = cf.Frame2D(axes_names=('x','y'), name='pixel') # Input pixel frame\n", 198 | "sky_frame = cf.CelestialFrame(reference_frame=coord.ICRS(),\n", 199 | " axes_names=('ra','dec'),\n", 200 | " name='icrs',\n", 201 | " axis_physical_types = ('pos.eq.ra', 'pos.eq.dec')) # Use the UCDs for RA/Dec in the axis_physical_types\n", 202 | "\n", 203 | "# Put the transform and the frames together\n", 204 | "pipeline = [(pixel_frame, fwcs_transform), (sky_frame, None)]\n", 205 | "\n", 206 | "# Pass them into a gwcs WCS object\n", 207 | "wcs = gwcs.WCS(pipeline)\n", 208 | "print(wcs)" 209 | ] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "id": "16ed2786-56a4-43a0-a55f-0e8502a7c5f5", 214 | "metadata": {}, 215 | "source": [ 216 | "If you're having trouble constructing a WCS for your data, your MAST and SOC contacts are happy to help, and can consult with the `gwcs` developers on your behalf as needed." 217 | ] 218 | }, 219 | { 220 | "cell_type": "markdown", 221 | "id": "c29a1d7f-275a-4d82-bef0-6804c72d56e2", 222 | "metadata": {}, 223 | "source": [ 224 | "## Assembling metadata dictionaries" 225 | ] 226 | }, 227 | { 228 | "cell_type": "markdown", 229 | "id": "886c35f1-34e3-453e-8bea-f927b451ea78", 230 | "metadata": {}, 231 | "source": [ 232 | "Next, we'll pass a variety of metadata (including the WCS object we created above) from the FITS header into Python dictionaries, before we construct our ASDF file object.\n", 233 | "\n", 234 | "Let's take a look at the [archival common metadata table in the expandable linked here](https://outerspace.stsci.edu/spaces/ISWG/pages/356669707/File+Design+for+PITs#FileDesignforPITs-pits_common) (or if you prefer to look at schemas, see [ccsp_minimal](https://rad.readthedocs.io/en/latest/generated/schemas/CCSP/ccsp_minimal-1.0.0.html) and [ccsp_custom_product](https://rad.readthedocs.io/en/latest/generated/schemas/CCSP/ccsp_custom_product-1.0.0.html)) and start filling out what we can for those keys, supplemented by the other, unique metadata found in this FITS header." 235 | ] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "id": "c7ac02b4-8577-40d7-91be-f0fd96c57589", 240 | "metadata": {}, 241 | "source": [ 242 | "ASDF data and metadata can be nested in a hierarchical tree structure, collecting related information together. So we'll start with small dictionaries, and then pass those dictionaries into a top-level dictionary in a nested structure." 243 | ] 244 | }, 245 | { 246 | "cell_type": "code", 247 | "execution_count": null, 248 | "id": "364c21bf-7a04-4ada-bf28-da2da5eb42dc", 249 | "metadata": {}, 250 | "outputs": [], 251 | "source": [ 252 | "ccsp = {\n", 253 | " \"name\": \"SPEAR\",\n", 254 | " \"investigator\": \"Jerry Edelstein\",\n", 255 | " \"archive_lead\": \"Martin Sirk\",\n", 256 | " \"doi\": header0['DOI'],\n", 257 | " \"file_version\": header0['VER'],\n", 258 | " \"data_release_id\": \"DR1\",\n", 259 | " \"license\": header0['LICENSE'],\n", 260 | " \"license_url\": header0['LICENURL'],\n", 261 | " \"target_name\": header0['TARG'],\n", 262 | " \"intent\": \"SCIENCE\",\n", 263 | " \"target_keywords\": \"Supernova remnants\", # Chosen from https://astrothesaurus.org/concept-select/\n", 264 | " \"target_keywords_id\": 1667, # From the https://astrothesaurus.org/concept-select/ entry above\n", 265 | "}\n", 266 | "\n", 267 | "instrument = {\n", 268 | " \"name\": header0['INSTRUME'],\n", 269 | " \"detector\": None,\n", 270 | " \"optical_element\": header0['FILTER'],\n", 271 | "}\n", 272 | "\n", 273 | "target_coordinates = {\n", 274 | " \"reference_frame\": header0['RADESYS'],\n", 275 | " \"ra\": header0['RA_TARG'],\n", 276 | " \"dec\": header0['DEC_TARG'],\n", 277 | "}\n", 278 | "\n", 279 | "wavelength = {\n", 280 | " \"band\": \"UV\",\n", 281 | " \"minimum\": float(header0['LAM-MIN']),\n", 282 | " \"maximum\": float(header0['LAM-MAX']),\n", 283 | "}" 284 | ] 285 | }, 286 | { 287 | "cell_type": "markdown", 288 | "id": "ecbb7b48-a5a1-446b-8fac-6616e20d4819", 289 | "metadata": {}, 290 | "source": [ 291 | "To populate the start and end date-times of this product, we'll convert the values from the FITS header into astropy Time objects before passing them into the dictionary, for compatibility with the ASDF Time schema.\n", 292 | "\n", 293 | "And the exposure time in the FITS header that best matches the mandatory `exposure_time` sub-key's definition is `EXP-SLIT`, so we'll use that as the characteristic exposure time under `exposure`." 294 | ] 295 | }, 296 | { 297 | "cell_type": "code", 298 | "execution_count": null, 299 | "id": "b8db0f9c-ef5e-427b-b95d-64ae14e9e2d7", 300 | "metadata": {}, 301 | "outputs": [], 302 | "source": [ 303 | "start = Time(header0['DATE-BEG'], format='isot', scale=header0['TIMESYS'].lower())\n", 304 | "end = Time(header0['DATE-END'], format='isot', scale=header0['TIMESYS'].lower())\n", 305 | "\n", 306 | "exposure = {\n", 307 | " \"start_time\": start,\n", 308 | " \"end_time\": end,\n", 309 | " \"exposure_time\": header0['EXP-SLIT'],\n", 310 | "}" 311 | ] 312 | }, 313 | { 314 | "cell_type": "markdown", 315 | "id": "a76aae32-8339-4cf0-b651-7c40ce732649", 316 | "metadata": {}, 317 | "source": [ 318 | "We have a lot of detail on the statistics of the exposure map, beyond the scope of the `exposure` parent key defined in [ccsp_custom_product](https://rad.readthedocs.io/en/latest/generated/schemas/CCSP/ccsp_custom_product-1.0.0.html), so we'll create an `exposure_stats` tree to group this information.\n", 319 | "\n", 320 | "Because there is no character count limit on ASDF keywords, we can be slightly more verbose than we would be in FITS (e.g., `pixel_exposure_time` instead of `EXP-PIX`). Still, the definition of these keys is left to the schema." 321 | ] 322 | }, 323 | { 324 | "cell_type": "code", 325 | "execution_count": null, 326 | "id": "0e593b4f-98a4-470a-8cda-5026f4fff7dc", 327 | "metadata": {}, 328 | "outputs": [], 329 | "source": [ 330 | "exposure_stats = {\n", 331 | " \"median_exposure_time\": header0['EXP-SLIT'],\n", 332 | " \"max_exposure_time\": header0['EXP-SMAX'],\n", 333 | " \"min_exposure_time\": header0['EXP-SMIN'],\n", 334 | " \"pixel_exposure_time\": header0['EXP-PIX'],\n", 335 | "}" 336 | ] 337 | }, 338 | { 339 | "cell_type": "markdown", 340 | "id": "13cf52be-6c33-4444-bd6b-b46fa12ff0c2", 341 | "metadata": {}, 342 | "source": [ 343 | "We'll also use our WCS object to get the sky coordinate boundaries of the image, to pass into an `s_region` keyword that will make this product discoverable in coordinate cone searches upon ingestion into MAST." 344 | ] 345 | }, 346 | { 347 | "cell_type": "code", 348 | "execution_count": null, 349 | "id": "7d28b610-e9df-4d87-910b-b54549cb461f", 350 | "metadata": {}, 351 | "outputs": [], 352 | "source": [ 353 | "# Assumes a fully-illuminated 2D image\n", 354 | "def s_region_fullchip(wcs, data):\n", 355 | " s_region_parts = ['POLYGON', 'ICRS']\n", 356 | " naxis1, naxis2 = data.shape # Get pixel image shape\n", 357 | "\n", 358 | " s_region_parts.extend([\n", 359 | " str(wcs(0,0)[0]), # RA of 1st vertex\n", 360 | " str(wcs(0,0)[1]), # Dec of 1st vertex\n", 361 | " str(wcs(naxis1-1, 0)[0]), # RA of 2nd vertex\n", 362 | " str(wcs(naxis1-1, 0)[1]), # Dec of 2nd vertex\n", 363 | " str(wcs(naxis1-1, naxis2-1)[0]), # RA of 3rd vertex\n", 364 | " str(wcs(naxis1-1, naxis2-1)[1]), # Dec of 3rd vertex\n", 365 | " str(wcs(0, naxis2-1)[0]), # RA of 4th vertex\n", 366 | " str(wcs(0, naxis2-1)[1]) # Dec of 4th vertex\n", 367 | " ])\n", 368 | "\n", 369 | " s_region = \" \".join(s_region_parts)\n", 370 | "\n", 371 | " return s_region\n", 372 | "\n", 373 | "s_region = s_region_fullchip(wcs, hdul[0].data)\n", 374 | "\n", 375 | "print(s_region)" 376 | ] 377 | }, 378 | { 379 | "cell_type": "markdown", 380 | "id": "4eeda61e-6738-470b-bd1e-86bf12536143", 381 | "metadata": {}, 382 | "source": [ 383 | "## Assembling the ASDF tree\n", 384 | "\n", 385 | "Now that we've deconstructed the FITS file and mapped much of its contents to Python dictionaries, we can construct the ASDF tree in a readable way. " 386 | ] 387 | }, 388 | { 389 | "cell_type": "code", 390 | "execution_count": null, 391 | "id": "39dde8cd-646d-4807-98f0-9677cc5d944e", 392 | "metadata": {}, 393 | "outputs": [], 394 | "source": [ 395 | "# Make the ASDF tree\n", 396 | "af = asdf.AsdfFile({\n", 397 | " \"meta\": {\n", 398 | " \"wcs\": wcs,\n", 399 | " \"ccsp\": ccsp,\n", 400 | " \"exposure\": exposure,\n", 401 | " \"exposure_stats\": exposure_stats,\n", 402 | " \"instrument\": instrument,\n", 403 | " \"telescope\": header0['TELESCOP'],\n", 404 | " \"target_coordinates\": target_coordinates,\n", 405 | " \"s_region\": s_region,\n", 406 | " \"pixel_scale\": header0['CDELT1']/3600., # Convert deg to arcsec\n", 407 | " \"wavelength\": wavelength,\n", 408 | " \"aperture\": header0['APERTURE'],\n", 409 | " \"grasp\": header0['GRASP']\n", 410 | " \n", 411 | " },\n", 412 | " \"data\": hdul[0].data,\n", 413 | " \"count\": hdul[1].data,\n", 414 | " \"exp\": hdul[2].data,\n", 415 | "})" 416 | ] 417 | }, 418 | { 419 | "cell_type": "markdown", 420 | "id": "5a60bb6c-0010-45c4-881b-e8e0cb75ef14", 421 | "metadata": {}, 422 | "source": [ 423 | "Now we can write that ASDF file object to an ASDF file on-disk:" 424 | ] 425 | }, 426 | { 427 | "cell_type": "code", 428 | "execution_count": null, 429 | "id": "56113652-1674-4a0e-ba72-e6412a790353", 430 | "metadata": {}, 431 | "outputs": [], 432 | "source": [ 433 | "af.write_to(\"sample_file.asdf\")" 434 | ] 435 | }, 436 | { 437 | "cell_type": "markdown", 438 | "id": "4834db28-27eb-4775-9217-6c29fa353ab6", 439 | "metadata": {}, 440 | "source": [ 441 | "At this point, you would start writing a data product schema compatible with this sample, following [the guidelines](https://outerspace.stsci.edu/spaces/ISWG/pages/356669707/File+Design+for+PITs#FileDesignforPITs-asdfASDFFileDesign). Then, you would share this file and its corresponding schema with MAST and the RAD maintainers in a [Github issue](github.com/spacetelescope/rad/issues/new) to the RAD repository." 442 | ] 443 | }, 444 | { 445 | "cell_type": "markdown", 446 | "id": "41f64710-56fb-42e7-bc28-478bf5c7bca9", 447 | "metadata": {}, 448 | "source": [ 449 | "Taking a look at this sample file, we see that it's pretty good, but many of the keys aren't clearly defined:" 450 | ] 451 | }, 452 | { 453 | "cell_type": "code", 454 | "execution_count": null, 455 | "id": "9728938a-066f-4d75-bafd-93776dbec765", 456 | "metadata": {}, 457 | "outputs": [], 458 | "source": [ 459 | "testaf_sample = asdf.open('sample_file.asdf')\n", 460 | "testaf_sample.info(max_rows=None, max_cols=None)" 461 | ] 462 | }, 463 | { 464 | "cell_type": "markdown", 465 | "id": "fb880c87-5cce-4555-b0be-108ad5041131", 466 | "metadata": {}, 467 | "source": [ 468 | "That's because key definitions and units in ASDF are externalized to the schemas, and we haven't yet linked this file to a schema:" 469 | ] 470 | }, 471 | { 472 | "cell_type": "code", 473 | "execution_count": null, 474 | "id": "5050d775-c95f-4bed-bd3e-ee6037337a2e", 475 | "metadata": {}, 476 | "outputs": [], 477 | "source": [ 478 | "print(af.schema_info(\"description\",\"roman.data\"))" 479 | ] 480 | }, 481 | { 482 | "cell_type": "markdown", 483 | "id": "058480f8-9c9a-4115-98b2-a8b655b6ab16", 484 | "metadata": {}, 485 | "source": [ 486 | "## Converting to a DataModel and tagging the schema" 487 | ] 488 | }, 489 | { 490 | "cell_type": "markdown", 491 | "id": "d3d8e2ec-3833-4f91-95a6-1f8b77b8f745", 492 | "metadata": {}, 493 | "source": [ 494 | "Now that we have the file contents mapped to ASDF, you would work on making a DataModel for the data. This would involve:\n", 495 | "- defining a schema\n", 496 | "- registering this schema using roman_datamodels\n", 497 | "\n", 498 | "Since the above changes are implemented by modifying `rad` and `roman_datamodels`, and these examples haven't been merged into a release, the following code will only work by installing the fork at `git+https://github.com/adrianlucy/roman_datamodels.git@ccsp_schemas_for_notebook`, which we did at the top of this notebook.\n", 499 | "\n", 500 | "First let's look at the added schema by providing its URI and asking asdf to load the resource." 501 | ] 502 | }, 503 | { 504 | "cell_type": "code", 505 | "execution_count": null, 506 | "id": "eebd34e7-9ed4-4bf3-a916-c859bbaa9944", 507 | "metadata": {}, 508 | "outputs": [], 509 | "source": [ 510 | "resource = asdf.get_config().resource_manager[\"asdf://stsci.edu/datamodels/roman/schemas/CCSP/EXAMPLE/example_spear_pointed_image-1.0.0\"]\n", 511 | "print(resource.decode(\"ascii\"))" 512 | ] 513 | }, 514 | { 515 | "cell_type": "markdown", 516 | "id": "090ecf1a-406f-4585-a13e-bc49eced4408", 517 | "metadata": {}, 518 | "source": [ 519 | "In the above schema note that:\n", 520 | "- the common `ccsp_custom_product-1.0.0` schema is **referenced**\n", 521 | "- the ASDF structure is described (and constrained) by the schema (for example, `data` must be a 2-dimensional image with float32 values and units of photons/cm^2/s/sr.\n", 522 | "\n", 523 | "The addition of this schema to RAD and a small modification to roman_datamodels allows us to use this schema for a new DataModel `ExampleSpearPointedImageModel`. Let's create a new instance of that model with the tree we constructed above." 524 | ] 525 | }, 526 | { 527 | "cell_type": "code", 528 | "execution_count": null, 529 | "id": "f46aae34-3724-4440-8209-99a721fb7e8b", 530 | "metadata": {}, 531 | "outputs": [], 532 | "source": [ 533 | "model = rdm.ExampleSpearPointedImageModel.create_from_model(af.tree)\n", 534 | "\n", 535 | "model.info(max_rows=None, max_cols=None)" 536 | ] 537 | }, 538 | { 539 | "cell_type": "markdown", 540 | "id": "3459b94d-f586-4dd6-a894-f7f11c585c98", 541 | "metadata": {}, 542 | "source": [ 543 | "Let's try validating the model we've created against its schema:" 544 | ] 545 | }, 546 | { 547 | "cell_type": "code", 548 | "execution_count": null, 549 | "id": "fc5717ad-05bc-4734-8ba3-235a8273ee4e", 550 | "metadata": {}, 551 | "outputs": [], 552 | "source": [ 553 | "try:\n", 554 | " model.validate()\n", 555 | "except asdf.exceptions.ValidationError as err:\n", 556 | " print(f\"ValidationError({err.message})\")" 557 | ] 558 | }, 559 | { 560 | "cell_type": "markdown", 561 | "id": "ab14d9b0-a24b-4f05-8150-d94befb385c8", 562 | "metadata": {}, 563 | "source": [ 564 | "Oops! We've forgotten to add the required `file_date` key. Let's do that, and try again:" 565 | ] 566 | }, 567 | { 568 | "cell_type": "code", 569 | "execution_count": null, 570 | "id": "67da8c19-4a55-4f47-b88f-454c37fd8a5c", 571 | "metadata": {}, 572 | "outputs": [], 573 | "source": [ 574 | "model[\"meta\"][\"file_date\"] = Time(Time.now(), format='isot')" 575 | ] 576 | }, 577 | { 578 | "cell_type": "code", 579 | "execution_count": null, 580 | "id": "3bbc34fc-e890-49f9-ad16-e6889cd4c478", 581 | "metadata": {}, 582 | "outputs": [], 583 | "source": [ 584 | "try:\n", 585 | " model.validate()\n", 586 | "except asdf.exceptions.ValidationError as err:\n", 587 | " print(f\"ValidationError({err.message})\")" 588 | ] 589 | }, 590 | { 591 | "cell_type": "markdown", 592 | "id": "d1d44ca0-510e-41c3-bfbb-a9d88259c524", 593 | "metadata": {}, 594 | "source": [ 595 | "Now that our model is valid we can save it to an ASDF file:" 596 | ] 597 | }, 598 | { 599 | "cell_type": "code", 600 | "execution_count": null, 601 | "id": "8ebd60f8-ffc4-47d7-9ae7-ac13cb4dd2e1", 602 | "metadata": {}, 603 | "outputs": [], 604 | "source": [ 605 | "model.save(\"ccsp_example_spear_vela_long-c-iv_v0.0_img.asdf\")" 606 | ] 607 | }, 608 | { 609 | "cell_type": "markdown", 610 | "id": "14987435-1cea-49e2-8be6-9f70b01d1886", 611 | "metadata": {}, 612 | "source": [ 613 | "## Examining the results" 614 | ] 615 | }, 616 | { 617 | "cell_type": "markdown", 618 | "id": "2cd0ddc1-67aa-4688-8643-b5e7d8c973bf", 619 | "metadata": {}, 620 | "source": [ 621 | "We're done, but let's take some time to look at what we've created. If we open that file up with `roman_datamodels`, we see that the `roman` key is tagged, and the keys throughout its tree are now commented with titles pulled from the schema associated with the tag:" 622 | ] 623 | }, 624 | { 625 | "cell_type": "code", 626 | "execution_count": null, 627 | "id": "ce07db47-0a02-4ee5-9e5f-69077175d668", 628 | "metadata": {}, 629 | "outputs": [], 630 | "source": [ 631 | "testdm = rdm.open('ccsp_example_spear_vela_long-c-iv_v0.0_img.asdf')" 632 | ] 633 | }, 634 | { 635 | "cell_type": "code", 636 | "execution_count": null, 637 | "id": "9c24e429-6bfb-4f36-8adf-948aa3821152", 638 | "metadata": { 639 | "scrolled": true 640 | }, 641 | "outputs": [], 642 | "source": [ 643 | "testdm.info(max_rows=None, max_cols=None)" 644 | ] 645 | }, 646 | { 647 | "cell_type": "markdown", 648 | "id": "3d9838ac-d601-42b4-a146-34a62a399514", 649 | "metadata": {}, 650 | "source": [ 651 | "And we can also retrieve the description and units of these keys programatically:" 652 | ] 653 | }, 654 | { 655 | "cell_type": "code", 656 | "execution_count": null, 657 | "id": "fe3a3e31-cd40-4152-a2a7-80f84d19cc8c", 658 | "metadata": {}, 659 | "outputs": [], 660 | "source": [ 661 | "print(testdm.schema_info(\"description\",\"roman.meta.target_coordinates.ra\"))\n", 662 | "print(testdm.schema_info(\"unit\",\"roman.meta.target_coordinates.ra\"))" 663 | ] 664 | }, 665 | { 666 | "cell_type": "markdown", 667 | "id": "483f3ad2-eccd-47e1-a3ce-d95475eb7e65", 668 | "metadata": {}, 669 | "source": [ 670 | "And because the right branch of `roman_datamodels` is installed into our Python environment, the schema is also similarly registered by the ASDF package while we're in this environment:" 671 | ] 672 | }, 673 | { 674 | "cell_type": "code", 675 | "execution_count": null, 676 | "id": "333aff5b-4b6e-4fa8-ade9-5fd7f66d5fc5", 677 | "metadata": {}, 678 | "outputs": [], 679 | "source": [ 680 | "testaf = asdf.open('ccsp_example_spear_vela_long-c-iv_v0.0_img.asdf')" 681 | ] 682 | }, 683 | { 684 | "cell_type": "code", 685 | "execution_count": null, 686 | "id": "4e45b114-8c28-4217-85ff-7df11f6193b6", 687 | "metadata": {}, 688 | "outputs": [], 689 | "source": [ 690 | "testaf.info(max_rows=None, max_cols=None)" 691 | ] 692 | }, 693 | { 694 | "cell_type": "code", 695 | "execution_count": null, 696 | "id": "91589f49-8ce9-4328-b082-20cc581b005a", 697 | "metadata": {}, 698 | "outputs": [], 699 | "source": [ 700 | "# This only works because `roman_datamodels` is installed in our environment,\n", 701 | "# serving to connect the file to its tagged schema\n", 702 | "print(testaf.schema_info(\"description\",\"roman.meta.target_coordinates.ra\"))\n", 703 | "print(testaf.schema_info(\"unit\",\"roman.meta.target_coordinates.ra\"))" 704 | ] 705 | }, 706 | { 707 | "cell_type": "code", 708 | "execution_count": null, 709 | "id": "5ee7b595-9dd4-4277-8d9d-6fe2812ebb91", 710 | "metadata": {}, 711 | "outputs": [], 712 | "source": [ 713 | "# Let's finally close the FITS file, now that we're done with it\n", 714 | "hdul.close()" 715 | ] 716 | } 717 | ], 718 | "metadata": { 719 | "kernelspec": { 720 | "display_name": "Python 3 (ipykernel)", 721 | "language": "python", 722 | "name": "python3" 723 | }, 724 | "language_info": { 725 | "codemirror_mode": { 726 | "name": "ipython", 727 | "version": 3 728 | }, 729 | "file_extension": ".py", 730 | "mimetype": "text/x-python", 731 | "name": "python", 732 | "nbconvert_exporter": "python", 733 | "pygments_lexer": "ipython3", 734 | "version": "3.14.0" 735 | } 736 | }, 737 | "nbformat": 4, 738 | "nbformat_minor": 5 739 | } 740 | -------------------------------------------------------------------------------- /notebooks/workflow_demo1/workflow_demo1_rendered.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "04942737-7b27-41de-ab74-929e784fd2fe", 6 | "metadata": {}, 7 | "source": [ 8 | "# An Example ASDF Design Workflow for CCSPs" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "id": "e11569a7-f499-491f-abd3-d0a5782c0447", 14 | "metadata": {}, 15 | "source": [ 16 | "## Overview\n", 17 | "\n", 18 | "In this notebook, we'll illustrate one possible method for starting to incorporate ASDF file design into your workflow and pipeline.\n", 19 | "\n", 20 | "We'll assume here that you're starting with a FITS file, though there is no need to do so. We'll extract the WCS from this FITS file's header and convert it into something ASDF-flavored, and then we'll extract the rest of the metadata from the FITS header. (But if you need more a complicated WCS than FITS can easily support, or if you are making alterations to a Roman ASDF file, then you probably *shouldn't* start with a FITS File.)\n", 21 | "\n", 22 | "The rest of the example design process would, in full, go something like this:\n", 23 | "- Design an ASDF tree structure and sample content for your product, passing nested dictionaries into an ASDF file object.\n", 24 | "- Save the resultant ASDF file object to an ASDF file: your initial sample file (without a corresponding schema).\n", 25 | "- Draft a schema that matches the initial sample file you just designed, in consultation with MAST and the RAD maintainers. Send the initial sample file and draft schema to MAST and the RAD maintainers in a Github issue to the RAD repository, which will evolve into a pull request in consultation with MAST and the RAD maintainers. **Note:** We'll largely elide the details of this step in this notebook. For more information on the schema-writing part of the process, see the [File Design Guidelines](https://outerspace.stsci.edu/spaces/ISWG/pages/356669707/File+Design+for+PITs).\n", 26 | "- Install the appropriate branch/fork of the RAD and roman_datamodels (with MAST and the RAD's help) into your pipeline environment.\n", 27 | "- Pass your ASDF tree into a Roman data model object. Save the result as your revised sample file, which now tags your schema, and deliver it to MAST for validation.\n", 28 | "\n", 29 | "**Note:** this is not the only possible workflow. For example, some people prefer to design the schema first, before constructing the ASDF file object." 30 | ] 31 | }, 32 | { 33 | "cell_type": "markdown", 34 | "id": "4d25895c-629c-421d-a123-7168740862a5", 35 | "metadata": {}, 36 | "source": [ 37 | "## Imports and installations" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "id": "2cac22ca-0538-4edb-aab3-72a64a87f04c", 43 | "metadata": {}, 44 | "source": [ 45 | "Later, we'll need a particular branch in a particular fork of the `roman_datamodels` repository, where we've set up an ASDF schema and Roman data model for the product we'll be converting. You should only use this fork in the context of this specific notebook; when you're ready to start working on your own data, install the main branch of [roman_datamodels](https://github.com/spacetelescope/roman_datamodels) proper." 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 1, 51 | "id": "3642a987-2178-448d-8750-ed2637414c2c", 52 | "metadata": {}, 53 | "outputs": [], 54 | "source": [ 55 | "%%capture captured\n", 56 | "!pip install git+https://github.com/adrianlucy/roman_datamodels.git@ccsp_schemas_for_notebook" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "id": "e771e420-8dc4-4f93-8f60-bb5c9cea99d4", 62 | "metadata": {}, 63 | "source": [ 64 | "You may need to restart your kernel for that to take effect.\n", 65 | "\n", 66 | "That installation, incidentally, installed everything else that we need as dependencies. So now let's run our imports:" 67 | ] 68 | }, 69 | { 70 | "cell_type": "code", 71 | "execution_count": 2, 72 | "id": "7da07c96-10f3-4e2b-be24-3be2bcefce82", 73 | "metadata": {}, 74 | "outputs": [], 75 | "source": [ 76 | "from astropy.io import fits # For loading the FITS file that we'll convert\n", 77 | "import numpy as np # For array and matrix wrangling\n", 78 | "\n", 79 | "import asdf # For building the ASDF tree\n", 80 | "\n", 81 | "from astropy.time import Time # For passing Time objects into the ASDF tree\n", 82 | "\n", 83 | "import gwcs # For building the WCS\n", 84 | "from gwcs import coordinate_frames as cf # For building the WCS\n", 85 | "from astropy.modeling import models # For building the WCS\n", 86 | "from astropy import coordinates as coord # For building the WCS\n", 87 | "\n", 88 | "import roman_datamodels.datamodels as rdm # Only used for the final step" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "id": "975c29a7-89eb-4853-8e0c-d52cffd0892f", 94 | "metadata": {}, 95 | "source": [ 96 | "## Opening the FITS file" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "id": "fdb9db35-d507-4cd2-8c9e-dcf7094d03cd", 102 | "metadata": {}, 103 | "source": [ 104 | "Next, we'll open the FITS file that we want to convert to ASDF. For the purposes of this tutorial, we'll use a simple 2D image file from the [FIMS-SPEAR mission](https://outerspace.stsci.edu/spaces/SPEARFIMS/overview). The main science data array is in the primary HDU, which is accompanied by concomitant images of net photon counts and an exposure map in HDU1 and HDU2, respectively." 105 | ] 106 | }, 107 | { 108 | "cell_type": "code", 109 | "execution_count": 3, 110 | "id": "d85642bf-71ef-495a-af87-6812ebcd4cd2", 111 | "metadata": {}, 112 | "outputs": [ 113 | { 114 | "name": "stdout", 115 | "output_type": "stream", 116 | "text": [ 117 | "Filename: /Users/alucy/.astropy/cache/download/url/4fb09bbd351a92dc11a9029b6feea2d1/contents\n", 118 | "No. Name Ver Type Cards Dimensions Format\n", 119 | " 0 PRIMARY 1 PrimaryHDU 52 (512, 512) float32 \n", 120 | " 1 COUNT_IMAGE 1 ImageHDU 12 (512, 512) float32 \n", 121 | " 2 EXPOSURE_MAP 1 ImageHDU 13 (512, 512) float32 \n" 122 | ] 123 | } 124 | ], 125 | "source": [ 126 | "# For best practice, in your pipeline this would be `with fits.open(\"filename\"):`\n", 127 | "# Get what you need, then close the file.\n", 128 | "hdul = fits.open(\"https://archive.stsci.edu/mccm/fims-spear/spear/vela/mccm_fims-spear_spear-ap100_vela_long-c-iv_v1.0_img.fits\")\n", 129 | "\n", 130 | "hdul.info()" 131 | ] 132 | }, 133 | { 134 | "cell_type": "markdown", 135 | "id": "7461fa57-a454-49ec-8683-0fd6e6100805", 136 | "metadata": {}, 137 | "source": [ 138 | "Let's take a look at the headers for each HDU. We see that most of the metadata is in the primary header, and that the concomitant images share the primary array's World Coordinate System (WCS). The WCS maps the array pixel coordinates to world coordinates like wavelength, time, or (in this case) sky coordinates." 139 | ] 140 | }, 141 | { 142 | "cell_type": "code", 143 | "execution_count": 4, 144 | "id": "48740c00-41bc-44df-b186-be86e664cbcb", 145 | "metadata": {}, 146 | "outputs": [ 147 | { 148 | "data": { 149 | "text/plain": [ 150 | "SIMPLE = T / conforms to FITS standard \n", 151 | "BITPIX = -32 / array data type \n", 152 | "NAXIS = 2 / number of array dimensions \n", 153 | "NAXIS1 = 512 \n", 154 | "NAXIS2 = 512 \n", 155 | "EXTEND = T \n", 156 | "DATE = '2023-04-12' / Creation date (CCYY-MM-DD) of FITS header \n", 157 | "BUNIT = 'photon/(cm2*s*sr)' / Physical unit of the array values \n", 158 | "PC1_1 = 1.0 / Transformation matrix \n", 159 | "PC2_2 = 1.0 / Transformation matrix \n", 160 | "CTYPE1 = 'RA---TAN' / X-axis \n", 161 | "CRVAL1 = 129.0 / Origin coordinate \n", 162 | "CRPIX1 = 256.5 / Origin pixel index (1 based) \n", 163 | "CDELT1 = -0.0244141 / Degrees/pixel \n", 164 | "CUNIT1 = 'deg ' / Coordinate units \n", 165 | "CTYPE2 = 'DEC--TAN' / Y-axis \n", 166 | "CRVAL2 = -46.25 / Origin coordinate \n", 167 | "CRPIX2 = 256.5 / Origin pixel index (1 based) \n", 168 | "CDELT2 = 0.0244141 / Degrees/pixel \n", 169 | "CUNIT2 = 'deg ' / Coordinate units \n", 170 | "RADESYS = 'ICRS ' / Coordinate reference frame \n", 171 | "TELESCOP= 'FIMS-SPEAR' / Observatory name \n", 172 | "INSTRUME= 'FIMS-SPEAR' / Instrument Name \n", 173 | "TEAM = 'SPEAR ' / Mission team who produced this product \n", 174 | "FILTER = 'L-Band_C-IV' / Disperser band and extracted emission feature \n", 175 | "APERTURE= '100 ' / Slit shutter aperture \n", 176 | "LAMBDA = 1549.05 / Wavelength of Line Species for image \n", 177 | "LAM-MIN = 1547 / Nominal minimum wavelength for databases \n", 178 | "LAM-MAX = 1552 / Nominal maximum wavelength for databases \n", 179 | "GRASP = 3.13019E-07 / Grasp [cm-2 sr-1] per exon at line wavelength \n", 180 | "DATE-BEG= '2004-01-27T04:16:00' / Start of observation \n", 181 | "DATE-END= '2004-01-27T19:14:00' / End of observation \n", 182 | "DATE-AVG= '2004-01-27T07:40:00' / Mid-point of observation \n", 183 | "TIMESYS = 'UTC ' / Time system for data-time keywords \n", 184 | "EXP-SLIT= 25.36313787917133 / Median exposure time [s] per slit width^2 \n", 185 | "EXP-SMAX= 99.90161508114453 / Max exposure time [s] per slit width^2 \n", 186 | "EXP-SMIN= 0.1083651593083156 / Min exposure time [s] per slit width^2 \n", 187 | "EXP-PIX = 2.943404912948608 / Median exposure time [s] per pixel \n", 188 | "TARG = 'Vela ' / Target \n", 189 | "RA_TARG = 129.0 / Target right ascension coordinate [degrees] \n", 190 | "DEC_TARG= -46.25 / Target declination coordinate [degrees] \n", 191 | "VER = '1.0 ' / Version \n", 192 | "DOI = 'doi:10.17909/dsbe-kj54' / Digital Object Identifier for mission \n", 193 | "LICENSE = 'CC BY 4.0' / License \n", 194 | "LICENURL= 'https://creativecommons.org/licenses/by/4.0/' / License URL \n", 195 | "CHECKSUM= 'BAgmE7gmBAgmB7gm' / HDU checksum updated 2023-04-12T17:22:29 \n", 196 | "DATASUM = '3763279621' / data unit checksum updated 2023-04-12T17:22:29 \n", 197 | "COMMENT Continuum-subtracted emission line image \n", 198 | "COMMENT Primary array is integrated intensity in line units \n", 199 | "COMMENT Primary array sets values with exp time < 1 s/pix to NaN \n", 200 | "HISTORY Created by M.M. Sirk, UC Berkeley on 2008-03-19, re-written 2008-04-04 \n", 201 | "HISTORY Revised by MAST STScI on 2023-04-12 " 202 | ] 203 | }, 204 | "execution_count": 4, 205 | "metadata": {}, 206 | "output_type": "execute_result" 207 | } 208 | ], 209 | "source": [ 210 | "header0 = hdul[0].header\n", 211 | "header0" 212 | ] 213 | }, 214 | { 215 | "cell_type": "code", 216 | "execution_count": 5, 217 | "id": "c4b53735-8db7-41b3-a84e-a7244044ee6c", 218 | "metadata": {}, 219 | "outputs": [ 220 | { 221 | "data": { 222 | "text/plain": [ 223 | "XTENSION= 'IMAGE ' / Image extension \n", 224 | "BITPIX = -32 / array data type \n", 225 | "NAXIS = 2 / number of array dimensions \n", 226 | "NAXIS1 = 512 \n", 227 | "NAXIS2 = 512 \n", 228 | "PCOUNT = 0 / number of parameters \n", 229 | "GCOUNT = 1 / number of groups \n", 230 | "EXTNAME = 'COUNT_IMAGE' / extension name \n", 231 | "BUNIT = 'photon ' / Physical unit of the array values [Count/pixel]\n", 232 | "CHECKSUM= '9mCKEkBH9kBHEkBH' / HDU checksum updated 2023-04-12T17:22:30 \n", 233 | "DATASUM = '328920997' / data unit checksum updated 2023-04-12T17:22:30 \n", 234 | "COMMENT Continuum-subtracted photon count map " 235 | ] 236 | }, 237 | "execution_count": 5, 238 | "metadata": {}, 239 | "output_type": "execute_result" 240 | } 241 | ], 242 | "source": [ 243 | "hdul[1].header" 244 | ] 245 | }, 246 | { 247 | "cell_type": "code", 248 | "execution_count": 6, 249 | "id": "f601eb75-8d7c-49e3-925d-9ae91b3c7c41", 250 | "metadata": {}, 251 | "outputs": [ 252 | { 253 | "data": { 254 | "text/plain": [ 255 | "XTENSION= 'IMAGE ' / Image extension \n", 256 | "BITPIX = -32 / array data type \n", 257 | "NAXIS = 2 / number of array dimensions \n", 258 | "NAXIS1 = 512 \n", 259 | "NAXIS2 = 512 \n", 260 | "PCOUNT = 0 / number of parameters \n", 261 | "GCOUNT = 1 / number of groups \n", 262 | "EXTNAME = 'EXPOSURE_MAP' / extension name \n", 263 | "BUNIT = 's ' / Physical unit of the array values [s/pixel] \n", 264 | "CHECKSUM= '4p284n164n164n16' / HDU checksum updated 2023-04-12T17:22:30 \n", 265 | "DATASUM = '738137009' / data unit checksum updated 2023-04-12T17:22:30 \n", 266 | "COMMENT Exposure time map. Array values correspond to net exposure time in pixel\n", 267 | "COMMENT . " 268 | ] 269 | }, 270 | "execution_count": 6, 271 | "metadata": {}, 272 | "output_type": "execute_result" 273 | } 274 | ], 275 | "source": [ 276 | "hdul[2].header" 277 | ] 278 | }, 279 | { 280 | "cell_type": "markdown", 281 | "id": "7e8f3b58-426a-43aa-a3a1-fa99c4c95cea", 282 | "metadata": {}, 283 | "source": [ 284 | "## Converting FITS WCS to GWCS" 285 | ] 286 | }, 287 | { 288 | "cell_type": "markdown", 289 | "id": "2a88e8d2-a0c3-4be7-8513-0b3fe2b371c5", 290 | "metadata": {}, 291 | "source": [ 292 | "`GWCS` allows for storing both simple WCS solutions and more complex WCS solutions with distortions and/or nonlinear components. Since we are converting a simple FITS WCS in this example, we'll use `gwcs.FITSImagingWCSTransform`. Note that Python and GWCS use 0-based indexing, while FITS uses 1-based indexing, so we'll need a 1 pixel offset to the `CRPIXj` pixel coordinates." 293 | ] 294 | }, 295 | { 296 | "cell_type": "code", 297 | "execution_count": 7, 298 | "id": "55117648-687d-47de-8157-214378a293f2", 299 | "metadata": {}, 300 | "outputs": [ 301 | { 302 | "name": "stdout", 303 | "output_type": "stream", 304 | "text": [ 305 | " From Transform \n", 306 | "----- -----------------------\n", 307 | "pixel FITSImagingWCSTransform\n", 308 | " icrs None\n" 309 | ] 310 | } 311 | ], 312 | "source": [ 313 | "# The header only has PC1_1 and PC2_2, so use the default values for PC1_2 and PC2_1\n", 314 | "pc1_2 = 0.0\n", 315 | "pc2_1 = 0.0\n", 316 | "\n", 317 | "# Construct the transform based on the FITS header\n", 318 | "fwcs_transform = gwcs.FITSImagingWCSTransform(\n", 319 | " models.Pix2Sky_Gnomonic(), # Apply a gnomonic projection\n", 320 | " crpix=[header0['CRPIX1']-1, header0['CRPIX2']-1], # correct for FITS 1-based indexing\n", 321 | " crval=[header0['CRVAL1'], header0['CRVAL2']],\n", 322 | " cdelt=[header0['CDELT1'], header0['CDELT2']],\n", 323 | " pc=[[header0['PC1_1'], pc1_2], [pc2_1, header0['PC2_2']]], # set the PCi_j matrix\n", 324 | ")\n", 325 | "\n", 326 | "# Define the frames\n", 327 | "pixel_frame = cf.Frame2D(axes_names=('x','y'), name='pixel') # Input pixel frame\n", 328 | "sky_frame = cf.CelestialFrame(reference_frame=coord.ICRS(),\n", 329 | " axes_names=('ra','dec'),\n", 330 | " name='icrs',\n", 331 | " axis_physical_types = ('pos.eq.ra', 'pos.eq.dec')) # Use the UCDs for RA/Dec in the axis_physical_types\n", 332 | "\n", 333 | "# Put the transform and the frames together\n", 334 | "pipeline = [(pixel_frame, fwcs_transform), (sky_frame, None)]\n", 335 | "\n", 336 | "# Pass them into a gwcs WCS object\n", 337 | "wcs = gwcs.WCS(pipeline)\n", 338 | "print(wcs)" 339 | ] 340 | }, 341 | { 342 | "cell_type": "markdown", 343 | "id": "16ed2786-56a4-43a0-a55f-0e8502a7c5f5", 344 | "metadata": {}, 345 | "source": [ 346 | "If you're having trouble constructing a WCS for your data, your MAST and SOC contacts are happy to help, and can consult with the `gwcs` developers on your behalf as needed." 347 | ] 348 | }, 349 | { 350 | "cell_type": "markdown", 351 | "id": "c29a1d7f-275a-4d82-bef0-6804c72d56e2", 352 | "metadata": {}, 353 | "source": [ 354 | "## Assembling metadata dictionaries" 355 | ] 356 | }, 357 | { 358 | "cell_type": "markdown", 359 | "id": "886c35f1-34e3-453e-8bea-f927b451ea78", 360 | "metadata": {}, 361 | "source": [ 362 | "Next, we'll pass a variety of metadata (including the WCS object we created above) from the FITS header into Python dictionaries, before we construct our ASDF file object.\n", 363 | "\n", 364 | "Let's take a look at the [archival common metadata table in the expandable linked here](https://outerspace.stsci.edu/spaces/ISWG/pages/356669707/File+Design+for+PITs#FileDesignforPITs-pits_common) (or if you prefer to look at schemas, see [ccsp_minimal](https://rad.readthedocs.io/en/latest/generated/schemas/CCSP/ccsp_minimal-1.0.0.html) and [ccsp_custom_product](https://rad.readthedocs.io/en/latest/generated/schemas/CCSP/ccsp_custom_product-1.0.0.html)) and start filling out what we can for those keys, supplemented by the other, unique metadata found in this FITS header." 365 | ] 366 | }, 367 | { 368 | "cell_type": "markdown", 369 | "id": "c7ac02b4-8577-40d7-91be-f0fd96c57589", 370 | "metadata": {}, 371 | "source": [ 372 | "ASDF data and metadata can be nested in a hierarchical tree structure, collecting related information together. So we'll start with small dictionaries, and then pass those dictionaries into a top-level dictionary in a nested structure." 373 | ] 374 | }, 375 | { 376 | "cell_type": "code", 377 | "execution_count": 8, 378 | "id": "364c21bf-7a04-4ada-bf28-da2da5eb42dc", 379 | "metadata": {}, 380 | "outputs": [], 381 | "source": [ 382 | "ccsp = {\n", 383 | " \"name\": \"SPEAR\",\n", 384 | " \"investigator\": \"Jerry Edelstein\",\n", 385 | " \"archive_lead\": \"Martin Sirk\",\n", 386 | " \"doi\": header0['DOI'],\n", 387 | " \"file_version\": header0['VER'],\n", 388 | " \"data_release_id\": \"DR1\",\n", 389 | " \"license\": header0['LICENSE'],\n", 390 | " \"license_url\": header0['LICENURL'],\n", 391 | " \"target_name\": header0['TARG'],\n", 392 | " \"intent\": \"SCIENCE\",\n", 393 | " \"target_keywords\": \"Supernova remnants\", # Chosen from https://astrothesaurus.org/concept-select/\n", 394 | " \"target_keywords_id\": 1667, # From the https://astrothesaurus.org/concept-select/ entry above\n", 395 | "}\n", 396 | "\n", 397 | "instrument = {\n", 398 | " \"name\": header0['INSTRUME'],\n", 399 | " \"detector\": None,\n", 400 | " \"optical_element\": header0['FILTER'],\n", 401 | "}\n", 402 | "\n", 403 | "target_coordinates = {\n", 404 | " \"reference_frame\": header0['RADESYS'],\n", 405 | " \"ra\": header0['RA_TARG'],\n", 406 | " \"dec\": header0['DEC_TARG'],\n", 407 | "}\n", 408 | "\n", 409 | "wavelength = {\n", 410 | " \"band\": \"UV\",\n", 411 | " \"minimum\": float(header0['LAM-MIN']),\n", 412 | " \"maximum\": float(header0['LAM-MAX']),\n", 413 | "}" 414 | ] 415 | }, 416 | { 417 | "cell_type": "markdown", 418 | "id": "ecbb7b48-a5a1-446b-8fac-6616e20d4819", 419 | "metadata": {}, 420 | "source": [ 421 | "To populate the start and end date-times of this product, we'll convert the values from the FITS header into astropy Time objects before passing them into the dictionary, for compatibility with the ASDF Time schema.\n", 422 | "\n", 423 | "And the exposure time in the FITS header that best matches the mandatory `exposure_time` sub-key's definition is `EXP-SLIT`, so we'll use that as the characteristic exposure time under `exposure`." 424 | ] 425 | }, 426 | { 427 | "cell_type": "code", 428 | "execution_count": 9, 429 | "id": "b8db0f9c-ef5e-427b-b95d-64ae14e9e2d7", 430 | "metadata": {}, 431 | "outputs": [], 432 | "source": [ 433 | "start = Time(header0['DATE-BEG'], format='isot', scale=header0['TIMESYS'].lower())\n", 434 | "end = Time(header0['DATE-END'], format='isot', scale=header0['TIMESYS'].lower())\n", 435 | "\n", 436 | "exposure = {\n", 437 | " \"start_time\": start,\n", 438 | " \"end_time\": end,\n", 439 | " \"exposure_time\": header0['EXP-SLIT'],\n", 440 | "}" 441 | ] 442 | }, 443 | { 444 | "cell_type": "markdown", 445 | "id": "a76aae32-8339-4cf0-b651-7c40ce732649", 446 | "metadata": {}, 447 | "source": [ 448 | "We have a lot of detail on the statistics of the exposure map, beyond the scope of the `exposure` parent key defined in [ccsp_custom_product](https://rad.readthedocs.io/en/latest/generated/schemas/CCSP/ccsp_custom_product-1.0.0.html), so we'll create an `exposure_stats` tree to group this information.\n", 449 | "\n", 450 | "Because there is no character count limit on ASDF keywords, we can be slightly more verbose than we would be in FITS (e.g., `pixel_exposure_time` instead of `EXP-PIX`). Still, the definition of these keys is left to the schema." 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": 10, 456 | "id": "0e593b4f-98a4-470a-8cda-5026f4fff7dc", 457 | "metadata": {}, 458 | "outputs": [], 459 | "source": [ 460 | "exposure_stats = {\n", 461 | " \"median_exposure_time\": header0['EXP-SLIT'],\n", 462 | " \"max_exposure_time\": header0['EXP-SMAX'],\n", 463 | " \"min_exposure_time\": header0['EXP-SMIN'],\n", 464 | " \"pixel_exposure_time\": header0['EXP-PIX'],\n", 465 | "}" 466 | ] 467 | }, 468 | { 469 | "cell_type": "markdown", 470 | "id": "13cf52be-6c33-4444-bd6b-b46fa12ff0c2", 471 | "metadata": {}, 472 | "source": [ 473 | "We'll also use our WCS object to get the sky coordinate boundaries of the image, to pass into an `s_region` keyword that will make this product discoverable in coordinate cone searches upon ingestion into MAST." 474 | ] 475 | }, 476 | { 477 | "cell_type": "code", 478 | "execution_count": 11, 479 | "id": "7d28b610-e9df-4d87-910b-b54549cb461f", 480 | "metadata": {}, 481 | "outputs": [ 482 | { 483 | "name": "stdout", 484 | "output_type": "stream", 485 | "text": [ 486 | "POLYGON ICRS 139.072956118736 -52.03248109358212 118.92704388126397 -52.03248109358211 120.95392087678839 -39.757741620879415 137.04607912321157 -39.757741620879415\n" 487 | ] 488 | } 489 | ], 490 | "source": [ 491 | "# Assumes a fully-illuminated 2D image\n", 492 | "def s_region_fullchip(wcs, data):\n", 493 | " s_region_parts = ['POLYGON', 'ICRS']\n", 494 | " naxis1, naxis2 = data.shape # Get pixel image shape\n", 495 | "\n", 496 | " s_region_parts.extend([\n", 497 | " str(wcs(0,0)[0]), # RA of 1st vertex\n", 498 | " str(wcs(0,0)[1]), # Dec of 1st vertex\n", 499 | " str(wcs(naxis1-1, 0)[0]), # RA of 2nd vertex\n", 500 | " str(wcs(naxis1-1, 0)[1]), # Dec of 2nd vertex\n", 501 | " str(wcs(naxis1-1, naxis2-1)[0]), # RA of 3rd vertex\n", 502 | " str(wcs(naxis1-1, naxis2-1)[1]), # Dec of 3rd vertex\n", 503 | " str(wcs(0, naxis2-1)[0]), # RA of 4th vertex\n", 504 | " str(wcs(0, naxis2-1)[1]) # Dec of 4th vertex\n", 505 | " ])\n", 506 | "\n", 507 | " s_region = \" \".join(s_region_parts)\n", 508 | "\n", 509 | " return s_region\n", 510 | "\n", 511 | "s_region = s_region_fullchip(wcs, hdul[0].data)\n", 512 | "\n", 513 | "print(s_region)" 514 | ] 515 | }, 516 | { 517 | "cell_type": "markdown", 518 | "id": "4eeda61e-6738-470b-bd1e-86bf12536143", 519 | "metadata": {}, 520 | "source": [ 521 | "## Assembling the ASDF tree\n", 522 | "\n", 523 | "Now that we've deconstructed the FITS file and mapped much of its contents to Python dictionaries, we can construct the ASDF tree in a readable way. " 524 | ] 525 | }, 526 | { 527 | "cell_type": "code", 528 | "execution_count": 12, 529 | "id": "39dde8cd-646d-4807-98f0-9677cc5d944e", 530 | "metadata": {}, 531 | "outputs": [], 532 | "source": [ 533 | "# Make the ASDF tree\n", 534 | "af = asdf.AsdfFile({\n", 535 | " \"meta\": {\n", 536 | " \"wcs\": wcs,\n", 537 | " \"ccsp\": ccsp,\n", 538 | " \"exposure\": exposure,\n", 539 | " \"exposure_stats\": exposure_stats,\n", 540 | " \"instrument\": instrument,\n", 541 | " \"telescope\": header0['TELESCOP'],\n", 542 | " \"target_coordinates\": target_coordinates,\n", 543 | " \"s_region\": s_region,\n", 544 | " \"pixel_scale\": header0['CDELT1']/3600., # Convert deg to arcsec\n", 545 | " \"wavelength\": wavelength,\n", 546 | " \"aperture\": header0['APERTURE'],\n", 547 | " \"grasp\": header0['GRASP']\n", 548 | " \n", 549 | " },\n", 550 | " \"data\": hdul[0].data,\n", 551 | " \"count\": hdul[1].data,\n", 552 | " \"exp\": hdul[2].data,\n", 553 | "})" 554 | ] 555 | }, 556 | { 557 | "cell_type": "markdown", 558 | "id": "5a60bb6c-0010-45c4-881b-e8e0cb75ef14", 559 | "metadata": {}, 560 | "source": [ 561 | "Now we can write that ASDF file object to an ASDF file on-disk:" 562 | ] 563 | }, 564 | { 565 | "cell_type": "code", 566 | "execution_count": 13, 567 | "id": "56113652-1674-4a0e-ba72-e6412a790353", 568 | "metadata": {}, 569 | "outputs": [], 570 | "source": [ 571 | "af.write_to(\"sample_file.asdf\")" 572 | ] 573 | }, 574 | { 575 | "cell_type": "markdown", 576 | "id": "4834db28-27eb-4775-9217-6c29fa353ab6", 577 | "metadata": {}, 578 | "source": [ 579 | "At this point, you would start writing a data product schema compatible with this sample, following [the guidelines](https://outerspace.stsci.edu/spaces/ISWG/pages/356669707/File+Design+for+PITs#FileDesignforPITs-asdfASDFFileDesign). Then, you would share this file and its corresponding schema with MAST and the RAD maintainers in a [Github issue](github.com/spacetelescope/rad/issues/new) to the RAD repository." 580 | ] 581 | }, 582 | { 583 | "cell_type": "markdown", 584 | "id": "41f64710-56fb-42e7-bc28-478bf5c7bca9", 585 | "metadata": {}, 586 | "source": [ 587 | "Taking a look at this sample file, we see that it's pretty good, but many of the keys aren't clearly defined:" 588 | ] 589 | }, 590 | { 591 | "cell_type": "code", 592 | "execution_count": 14, 593 | "id": "9728938a-066f-4d75-bafd-93776dbec765", 594 | "metadata": {}, 595 | "outputs": [ 596 | { 597 | "name": "stdout", 598 | "output_type": "stream", 599 | "text": [ 600 | "root (AsdfObject)\n", 601 | "├─asdf_library (Software)\n", 602 | "│ ├─author (str): The ASDF Developers\n", 603 | "│ ├─homepage (str): http://github.com/asdf-format/asdf\n", 604 | "│ ├─name (str): asdf\n", 605 | "│ └─version (str): 5.1.0\n", 606 | "├─history (dict)\n", 607 | "│ └─extensions (list)\n", 608 | "│ ├─[0] (ExtensionMetadata)\n", 609 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 610 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/core/extensions/core-1.6.0\n", 611 | "│ │ ├─manifest_software (Software)\n", 612 | "│ │ │ ├─name (str): asdf_standard\n", 613 | "│ │ │ └─version (str): 1.4.0\n", 614 | "│ │ └─software (Software)\n", 615 | "│ │ ├─name (str): asdf\n", 616 | "│ │ └─version (str): 5.1.0\n", 617 | "│ ├─[1] (ExtensionMetadata)\n", 618 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 619 | "│ │ ├─extension_uri (str): asdf://astropy.org/astropy/extensions/units-1.0.0\n", 620 | "│ │ └─software (Software)\n", 621 | "│ │ ├─name (str): asdf-astropy\n", 622 | "│ │ └─version (str): 0.9.0\n", 623 | "│ ├─[2] (ExtensionMetadata)\n", 624 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 625 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.4.0\n", 626 | "│ │ ├─manifest_software (Software)\n", 627 | "│ │ │ ├─name (str): asdf_wcs_schemas\n", 628 | "│ │ │ └─version (str): 0.5.0\n", 629 | "│ │ └─software (Software)\n", 630 | "│ │ ├─name (str): gwcs\n", 631 | "│ │ └─version (str): 0.26.1\n", 632 | "│ ├─[3] (ExtensionMetadata)\n", 633 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 634 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/transform/extensions/transform-1.6.0\n", 635 | "│ │ ├─manifest_software (Software)\n", 636 | "│ │ │ ├─name (str): asdf_transform_schemas\n", 637 | "│ │ │ └─version (str): 0.6.0\n", 638 | "│ │ └─software (Software)\n", 639 | "│ │ ├─name (str): asdf-astropy\n", 640 | "│ │ └─version (str): 0.9.0\n", 641 | "│ ├─[4] (ExtensionMetadata)\n", 642 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 643 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/astronomy/coordinates/extensions/coordinates-1.0.0\n", 644 | "│ │ ├─manifest_software (Software)\n", 645 | "│ │ │ ├─name (str): asdf_coordinates_schemas\n", 646 | "│ │ │ └─version (str): 0.4.0\n", 647 | "│ │ └─software (Software)\n", 648 | "│ │ ├─name (str): asdf-astropy\n", 649 | "│ │ └─version (str): 0.9.0\n", 650 | "│ └─[5] (ExtensionMetadata)\n", 651 | "│ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 652 | "│ ├─extension_uri (str): asdf://asdf-format.org/astronomy/extensions/astronomy-1.1.0\n", 653 | "│ ├─manifest_software (Software)\n", 654 | "│ │ ├─name (str): asdf_standard\n", 655 | "│ │ └─version (str): 1.4.0\n", 656 | "│ └─software (Software)\n", 657 | "│ ├─name (str): asdf-astropy\n", 658 | "│ └─version (str): 0.9.0\n", 659 | "├─count (NDArrayType)\n", 660 | "│ ├─shape (tuple)\n", 661 | "│ │ ├─[0] (int): 512\n", 662 | "│ │ └─[1] (int): 512\n", 663 | "│ └─dtype (Float32DType): >f4\n", 664 | "├─data (NDArrayType)\n", 665 | "│ ├─shape (tuple)\n", 666 | "│ │ ├─[0] (int): 512\n", 667 | "│ │ └─[1] (int): 512\n", 668 | "│ └─dtype (Float32DType): >f4\n", 669 | "├─exp (NDArrayType)\n", 670 | "│ ├─shape (tuple)\n", 671 | "│ │ ├─[0] (int): 512\n", 672 | "│ │ └─[1] (int): 512\n", 673 | "│ └─dtype (Float32DType): >f4\n", 674 | "└─meta (dict)\n", 675 | " ├─aperture (str): 100\n", 676 | " ├─ccsp (dict)\n", 677 | " │ ├─archive_lead (str): Martin Sirk\n", 678 | " │ ├─data_release_id (str): DR1\n", 679 | " │ ├─doi (str): doi:10.17909/dsbe-kj54\n", 680 | " │ ├─file_version (str): 1.0\n", 681 | " │ ├─intent (str): SCIENCE\n", 682 | " │ ├─investigator (str): Jerry Edelstein\n", 683 | " │ ├─license (str): CC BY 4.0\n", 684 | " │ ├─license_url (str): https://creativecommons.org/licenses/by/4.0/\n", 685 | " │ ├─name (str): SPEAR\n", 686 | " │ ├─target_keywords (str): Supernova remnants\n", 687 | " │ ├─target_keywords_id (int): 1667\n", 688 | " │ └─target_name (str): Vela\n", 689 | " ├─exposure (dict)\n", 690 | " │ ├─end_time (Time): 2004-01-27T19:14:00.000\n", 691 | " │ ├─exposure_time (float): 25.36313787917133\n", 692 | " │ └─start_time (Time): 2004-01-27T04:16:00.000\n", 693 | " ├─exposure_stats (dict)\n", 694 | " │ ├─max_exposure_time (float): 99.90161508114453\n", 695 | " │ ├─median_exposure_time (float): 25.36313787917133\n", 696 | " │ ├─min_exposure_time (float): 0.1083651593083156\n", 697 | " │ └─pixel_exposure_time (float): 2.943404912948608\n", 698 | " ├─grasp (float): 3.13019e-07\n", 699 | " ├─instrument (dict)\n", 700 | " │ ├─detector (NoneType): None\n", 701 | " │ ├─name (str): FIMS-SPEAR\n", 702 | " │ └─optical_element (str): L-Band_C-IV\n", 703 | " ├─pixel_scale (float): -6.7816944444444445e-06\n", 704 | " ├─s_region (str): POLYGON ICRS 139.072956118736 -52.03248109358212 118.92704388126397 -52.03248109358211 120.95392087678839 -39.757741620879415 137.04607912321157 -39.757741620879415\n", 705 | " ├─target_coordinates (dict)\n", 706 | " │ ├─dec (float): -46.25\n", 707 | " │ ├─ra (float): 129.0\n", 708 | " │ └─reference_frame (str): ICRS\n", 709 | " ├─telescope (str): FIMS-SPEAR\n", 710 | " ├─wavelength (dict)\n", 711 | " │ ├─band (str): UV\n", 712 | " │ ├─maximum (float): 1552.0\n", 713 | " │ └─minimum (float): 1547.0\n", 714 | " └─wcs (WCS)\n" 715 | ] 716 | } 717 | ], 718 | "source": [ 719 | "testaf_sample = asdf.open('sample_file.asdf')\n", 720 | "testaf_sample.info(max_rows=None, max_cols=None)" 721 | ] 722 | }, 723 | { 724 | "cell_type": "markdown", 725 | "id": "fb880c87-5cce-4555-b0be-108ad5041131", 726 | "metadata": {}, 727 | "source": [ 728 | "That's because key definitions and units in ASDF are externalized to the schemas, and we haven't yet linked this file to a schema:" 729 | ] 730 | }, 731 | { 732 | "cell_type": "code", 733 | "execution_count": 15, 734 | "id": "5050d775-c95f-4bed-bd3e-ee6037337a2e", 735 | "metadata": {}, 736 | "outputs": [ 737 | { 738 | "name": "stdout", 739 | "output_type": "stream", 740 | "text": [ 741 | "None\n" 742 | ] 743 | } 744 | ], 745 | "source": [ 746 | "print(af.schema_info(\"description\",\"roman.data\"))" 747 | ] 748 | }, 749 | { 750 | "cell_type": "markdown", 751 | "id": "058480f8-9c9a-4115-98b2-a8b655b6ab16", 752 | "metadata": {}, 753 | "source": [ 754 | "## Converting to a DataModel and tagging the schema" 755 | ] 756 | }, 757 | { 758 | "cell_type": "markdown", 759 | "id": "d3d8e2ec-3833-4f91-95a6-1f8b77b8f745", 760 | "metadata": {}, 761 | "source": [ 762 | "Now that we have the file contents mapped to ASDF, you would work on making a DataModel for the data. This would involve:\n", 763 | "- defining a schema\n", 764 | "- registering this schema using roman_datamodels\n", 765 | "\n", 766 | "Since the above changes are implemented by modifying `rad` and `roman_datamodels`, and these examples haven't been merged into a release, the following code will only work by installing the fork at `git+https://github.com/adrianlucy/roman_datamodels.git@ccsp_schemas_for_notebook`, which we did at the top of this notebook.\n", 767 | "\n", 768 | "First let's look at the added schema by providing its URI and asking asdf to load the resource." 769 | ] 770 | }, 771 | { 772 | "cell_type": "code", 773 | "execution_count": 16, 774 | "id": "eebd34e7-9ed4-4bf3-a916-c859bbaa9944", 775 | "metadata": {}, 776 | "outputs": [ 777 | { 778 | "name": "stdout", 779 | "output_type": "stream", 780 | "text": [ 781 | "%YAML 1.1\n", 782 | "---\n", 783 | "$schema: asdf://stsci.edu/datamodels/roman/schemas/rad_schema-1.0.0\n", 784 | "id: asdf://stsci.edu/datamodels/roman/schemas/CCSP/EXAMPLE/example_spear_pointed_image-1.0.0\n", 785 | "\n", 786 | "title: Example SPEAR pointed image CCSP product\n", 787 | "\n", 788 | "datamodel_name: ExampleSpearPointedImageModel\n", 789 | "\n", 790 | "type: object\n", 791 | "properties:\n", 792 | " meta:\n", 793 | " allOf:\n", 794 | " - $ref: asdf://stsci.edu/datamodels/roman/schemas/CCSP/ccsp_custom_product-1.0.0\n", 795 | " - required:\n", 796 | " [\n", 797 | " exposure,\n", 798 | " instrument,\n", 799 | " telescope,\n", 800 | " target_coordinates,\n", 801 | " s_region,\n", 802 | " pixel_scale,\n", 803 | " wavelength,\n", 804 | " ]\n", 805 | " - type: object\n", 806 | " properties:\n", 807 | " wcs:\n", 808 | " title: World Coordinate System (WCS)\n", 809 | " description: |\n", 810 | " WCS for the data array and all concomitant arrays\n", 811 | " tag: tag:stsci.edu:gwcs/wcs-*\n", 812 | " aperture:\n", 813 | " title: Slit shutter aperture mode\n", 814 | " type: string\n", 815 | " exposure_stats:\n", 816 | " title: Exposure map statistical detail\n", 817 | " type: object\n", 818 | " properties:\n", 819 | " median_exposure_time:\n", 820 | " title: Median exposure time (s) per slit width^2\n", 821 | " type: number\n", 822 | " unit: \"s\"\n", 823 | " max_exposure_time:\n", 824 | " title: Max exposure time (s) per slit width^2\n", 825 | " type: number\n", 826 | " unit: \"s\"\n", 827 | " min_exposure_time:\n", 828 | " title: Min exposure time (s) per slit width^2\n", 829 | " type: number\n", 830 | " unit: \"s\"\n", 831 | " pixel_exposure_time:\n", 832 | " title: Median exposure time (s) per pixel\n", 833 | " type: number\n", 834 | " unit: \"s\"\n", 835 | " required: [\n", 836 | " median_exposure_time,\n", 837 | " max_exposure_time, \n", 838 | " min_exposure_time,\n", 839 | " pixel_exposure_time,\n", 840 | " ]\n", 841 | " grasp:\n", 842 | " title: Grasp (cm^-2 sr^-1) per exon at line wavelength\n", 843 | " type: number\n", 844 | " unit: \"cm**-2.sr**-1\"\n", 845 | " required: [wcs, aperture, exposure_stats, grasp]\n", 846 | " data:\n", 847 | " title: Science Data (photon/(cm^2 s sr))\n", 848 | " description: |\n", 849 | " Integrated intensity in line units. Values with exposure time\n", 850 | " < 1 s/pix are set to NaN.\n", 851 | " tag: tag:stsci.edu:asdf/core/ndarray-1.*\n", 852 | " datatype: float32\n", 853 | " exact_datatype: true\n", 854 | " ndim: 2\n", 855 | " unit: \"photon/(cm**2.s.sr)\"\n", 856 | " count:\n", 857 | " title: Count (photon)\n", 858 | " description: |\n", 859 | " Continuum-subtracted photon count map.\n", 860 | " tag: tag:stsci.edu:asdf/core/ndarray-1.*\n", 861 | " datatype: float32\n", 862 | " exact_datatype: true\n", 863 | " ndim: 2\n", 864 | " unit: \"photon\"\n", 865 | " exp:\n", 866 | " title: Exposure Map (s)\n", 867 | " description: |\n", 868 | " Exposure time map. Array values correspond to net exposure time\n", 869 | " in pixel.\n", 870 | " tag: tag:stsci.edu:asdf/core/ndarray-1.*\n", 871 | " datatype: float32\n", 872 | " exact_datatype: true\n", 873 | " ndim: 2\n", 874 | " unit: \"s\"\n", 875 | "required: [data, count, exp, meta]\n", 876 | "flowStyle: block\n", 877 | "\n" 878 | ] 879 | } 880 | ], 881 | "source": [ 882 | "resource = asdf.get_config().resource_manager[\"asdf://stsci.edu/datamodels/roman/schemas/CCSP/EXAMPLE/example_spear_pointed_image-1.0.0\"]\n", 883 | "print(resource.decode(\"ascii\"))" 884 | ] 885 | }, 886 | { 887 | "cell_type": "markdown", 888 | "id": "090ecf1a-406f-4585-a13e-bc49eced4408", 889 | "metadata": {}, 890 | "source": [ 891 | "In the above schema note that:\n", 892 | "- the common `ccsp_custom_product-1.0.0` schema is **referenced**\n", 893 | "- the ASDF structure is described (and constrained) by the schema (for example, `data` must be a 2-dimensional image with float32 values and units of photons/cm^2/s/sr.\n", 894 | "\n", 895 | "The addition of this schema to RAD and a small modification to roman_datamodels allows us to use this schema for a new DataModel `ExampleSpearPointedImageModel`. Let's create a new instance of that model with the tree we constructed above." 896 | ] 897 | }, 898 | { 899 | "cell_type": "code", 900 | "execution_count": 17, 901 | "id": "f46aae34-3724-4440-8209-99a721fb7e8b", 902 | "metadata": {}, 903 | "outputs": [ 904 | { 905 | "name": "stdout", 906 | "output_type": "stream", 907 | "text": [ 908 | "root (AsdfObject)\n", 909 | "└─roman (ExampleSpearPointedImage) # Example SPEAR pointed image CCSP product\n", 910 | " ├─meta (dict)\n", 911 | " │ ├─wcs (WCS) # World Coordinate System (WCS)\n", 912 | " │ ├─ccsp (dict) # Community Contributed Science Product (CCSP) information\n", 913 | " │ │ ├─name (str): SPEAR # CCSP name\n", 914 | " │ │ ├─investigator (str): Jerry Edelstein # CCSP Principal Investigator\n", 915 | " │ │ ├─archive_lead (str): Martin Sirk # CCSP staff lead for MAST ingest\n", 916 | " │ │ ├─doi (str): doi:10.17909/dsbe-kj54 # CCSP Digital Object Identifier\n", 917 | " │ │ ├─file_version (str): 1.0 # Version of this file\n", 918 | " │ │ ├─data_release_id (str): DR1 # CCSP collection data release\n", 919 | " │ │ ├─license (str): CC BY 4.0 # License for use of these data.\n", 920 | " │ │ ├─license_url (str): https://creativecommons.org/licenses/by/4.0/ # URL of license for use of these data.\n", 921 | " │ │ ├─target_name (str): Vela # Target name\n", 922 | " │ │ ├─intent (str): SCIENCE # Observation intent\n", 923 | " │ │ ├─target_keywords (str): Supernova remnants # UAT keywords\n", 924 | " │ │ └─target_keywords_id (int): 1667 # UAT keyword IDs\n", 925 | " │ ├─exposure (dict) # Exposure time summary information\n", 926 | " │ │ ├─start_time (Time): 2004-01-27T04:16:00.000 # Start time\n", 927 | " │ │ ├─end_time (Time): 2004-01-27T19:14:00.000 # End time\n", 928 | " │ │ └─exposure_time (float): 25.36313787917133 # Exposure time (s)\n", 929 | " │ ├─exposure_stats (dict) # Exposure map statistical detail\n", 930 | " │ │ ├─median_exposure_time (float): 25.36313787917133 # Median exposure time (s) per slit width^2\n", 931 | " │ │ ├─max_exposure_time (float): 99.90161508114453 # Max exposure time (s) per slit width^2\n", 932 | " │ │ ├─min_exposure_time (float): 0.1083651593083156 # Min exposure time (s) per slit width^2\n", 933 | " │ │ └─pixel_exposure_time (float): 2.943404912948608 # Median exposure time (s) per pixel\n", 934 | " │ ├─instrument (dict) # Instrument observing configuration\n", 935 | " │ │ ├─name (str): FIMS-SPEAR # Instrument name\n", 936 | " │ │ ├─detector (NoneType): None # Detector\n", 937 | " │ │ └─optical_element (str): L-Band_C-IV # Optical element\n", 938 | " │ ├─telescope (str): FIMS-SPEAR # Telescope name\n", 939 | " │ ├─target_coordinates (dict) # Position for this data product\n", 940 | " │ │ ├─reference_frame (str): ICRS\n", 941 | " │ │ ├─ra (float): 129.0 # Right ascension (deg)\n", 942 | " │ │ └─dec (float): -46.25 # Declination (deg)\n", 943 | " │ ├─s_region (str): POLYGON ICRS 139.072956118736 -52.03248109358212 118.92704388126397 -52.03248109358211 120.95392087678839 -39.757741620879415 137.04607912321157 -39.757741620879415 # STC-S formatted sky footprint\n", 944 | " │ ├─pixel_scale (float): -6.7816944444444445e-06 # Pixel scale of spatial axes (arcsec/pixel)\n", 945 | " │ ├─wavelength (dict) # Wavelength information\n", 946 | " │ │ ├─band (str): UV # Waveband\n", 947 | " │ │ ├─minimum (float): 1547.0 # Minimum wavelength\n", 948 | " │ │ └─maximum (float): 1552.0 # Maximum wavelength\n", 949 | " │ ├─aperture (str): 100 # Slit shutter aperture mode\n", 950 | " │ ├─grasp (float): 3.13019e-07 # Grasp (cm^-2 sr^-1) per exon at line wavelength\n", 951 | " │ └─model_type (str): ExampleSpearPointedImageModel\n", 952 | " ├─data (ndarray) # Science Data (photon/(cm^2 s sr))\n", 953 | " │ ├─shape (tuple)\n", 954 | " │ │ ├─[0] (int): 512\n", 955 | " │ │ └─[1] (int): 512\n", 956 | " │ └─dtype (Float32DType): >f4\n", 957 | " ├─count (ndarray) # Count (photon)\n", 958 | " │ ├─shape (tuple)\n", 959 | " │ │ ├─[0] (int): 512\n", 960 | " │ │ └─[1] (int): 512\n", 961 | " │ └─dtype (Float32DType): >f4\n", 962 | " └─exp (ndarray) # Exposure Map (s)\n", 963 | " ├─shape (tuple)\n", 964 | " │ ├─[0] (int): 512\n", 965 | " │ └─[1] (int): 512\n", 966 | " └─dtype (Float32DType): >f4\n" 967 | ] 968 | } 969 | ], 970 | "source": [ 971 | "model = rdm.ExampleSpearPointedImageModel.create_from_model(af.tree)\n", 972 | "\n", 973 | "model.info(max_rows=None, max_cols=None)" 974 | ] 975 | }, 976 | { 977 | "cell_type": "markdown", 978 | "id": "3459b94d-f586-4dd6-a894-f7f11c585c98", 979 | "metadata": {}, 980 | "source": [ 981 | "Let's try validating the model we've created against its schema:" 982 | ] 983 | }, 984 | { 985 | "cell_type": "code", 986 | "execution_count": 18, 987 | "id": "fc5717ad-05bc-4734-8ba3-235a8273ee4e", 988 | "metadata": {}, 989 | "outputs": [ 990 | { 991 | "name": "stdout", 992 | "output_type": "stream", 993 | "text": [ 994 | "ValidationError('file_date' is a required property)\n" 995 | ] 996 | } 997 | ], 998 | "source": [ 999 | "try:\n", 1000 | " model.validate()\n", 1001 | "except asdf.exceptions.ValidationError as err:\n", 1002 | " print(f\"ValidationError({err.message})\")" 1003 | ] 1004 | }, 1005 | { 1006 | "cell_type": "markdown", 1007 | "id": "ab14d9b0-a24b-4f05-8150-d94befb385c8", 1008 | "metadata": {}, 1009 | "source": [ 1010 | "Oops! We've forgotten to add the required `file_date` key. Let's do that, and try again:" 1011 | ] 1012 | }, 1013 | { 1014 | "cell_type": "code", 1015 | "execution_count": 19, 1016 | "id": "67da8c19-4a55-4f47-b88f-454c37fd8a5c", 1017 | "metadata": {}, 1018 | "outputs": [], 1019 | "source": [ 1020 | "model[\"meta\"][\"file_date\"] = Time(Time.now(), format='isot')" 1021 | ] 1022 | }, 1023 | { 1024 | "cell_type": "code", 1025 | "execution_count": 20, 1026 | "id": "3bbc34fc-e890-49f9-ad16-e6889cd4c478", 1027 | "metadata": {}, 1028 | "outputs": [], 1029 | "source": [ 1030 | "try:\n", 1031 | " model.validate()\n", 1032 | "except asdf.exceptions.ValidationError as err:\n", 1033 | " print(f\"ValidationError({err.message})\")" 1034 | ] 1035 | }, 1036 | { 1037 | "cell_type": "markdown", 1038 | "id": "d1d44ca0-510e-41c3-bfbb-a9d88259c524", 1039 | "metadata": {}, 1040 | "source": [ 1041 | "Now that our model is valid we can save it to an ASDF file:" 1042 | ] 1043 | }, 1044 | { 1045 | "cell_type": "code", 1046 | "execution_count": 21, 1047 | "id": "8ebd60f8-ffc4-47d7-9ae7-ac13cb4dd2e1", 1048 | "metadata": {}, 1049 | "outputs": [ 1050 | { 1051 | "data": { 1052 | "text/plain": [ 1053 | "PosixPath('ccsp_example_spear_vela_long-c-iv_v0.0_img.asdf')" 1054 | ] 1055 | }, 1056 | "execution_count": 21, 1057 | "metadata": {}, 1058 | "output_type": "execute_result" 1059 | } 1060 | ], 1061 | "source": [ 1062 | "model.save(\"ccsp_example_spear_vela_long-c-iv_v0.0_img.asdf\")" 1063 | ] 1064 | }, 1065 | { 1066 | "cell_type": "markdown", 1067 | "id": "14987435-1cea-49e2-8be6-9f70b01d1886", 1068 | "metadata": {}, 1069 | "source": [ 1070 | "## Examining the results" 1071 | ] 1072 | }, 1073 | { 1074 | "cell_type": "markdown", 1075 | "id": "2cd0ddc1-67aa-4688-8643-b5e7d8c973bf", 1076 | "metadata": {}, 1077 | "source": [ 1078 | "We're done, but let's take some time to look at what we've created. If we open that file up with `roman_datamodels`, we see that the `roman` key is tagged, and the keys throughout its tree are now commented with titles pulled from the schema associated with the tag:" 1079 | ] 1080 | }, 1081 | { 1082 | "cell_type": "code", 1083 | "execution_count": 22, 1084 | "id": "ce07db47-0a02-4ee5-9e5f-69077175d668", 1085 | "metadata": {}, 1086 | "outputs": [], 1087 | "source": [ 1088 | "testdm = rdm.open('ccsp_example_spear_vela_long-c-iv_v0.0_img.asdf')" 1089 | ] 1090 | }, 1091 | { 1092 | "cell_type": "code", 1093 | "execution_count": 23, 1094 | "id": "9c24e429-6bfb-4f36-8adf-948aa3821152", 1095 | "metadata": { 1096 | "scrolled": true 1097 | }, 1098 | "outputs": [ 1099 | { 1100 | "name": "stdout", 1101 | "output_type": "stream", 1102 | "text": [ 1103 | "root (AsdfObject)\n", 1104 | "├─asdf_library (Software)\n", 1105 | "│ ├─author (str): The ASDF Developers\n", 1106 | "│ ├─homepage (str): http://github.com/asdf-format/asdf\n", 1107 | "│ ├─name (str): asdf\n", 1108 | "│ └─version (str): 5.1.0\n", 1109 | "├─history (AsdfDictNode)\n", 1110 | "│ └─extensions (AsdfListNode)\n", 1111 | "│ ├─0 (ExtensionMetadata)\n", 1112 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1113 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/core/extensions/core-1.6.0\n", 1114 | "│ │ ├─manifest_software (Software)\n", 1115 | "│ │ │ ├─name (str): asdf_standard\n", 1116 | "│ │ │ └─version (str): 1.4.0\n", 1117 | "│ │ └─software (Software)\n", 1118 | "│ │ ├─name (str): asdf\n", 1119 | "│ │ └─version (str): 5.1.0\n", 1120 | "│ ├─1 (ExtensionMetadata)\n", 1121 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1122 | "│ │ ├─extension_uri (str): asdf://astropy.org/astropy/extensions/units-1.0.0\n", 1123 | "│ │ └─software (Software)\n", 1124 | "│ │ ├─name (str): asdf-astropy\n", 1125 | "│ │ └─version (str): 0.9.0\n", 1126 | "│ ├─2 (ExtensionMetadata)\n", 1127 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1128 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.4.0\n", 1129 | "│ │ ├─manifest_software (Software)\n", 1130 | "│ │ │ ├─name (str): asdf_wcs_schemas\n", 1131 | "│ │ │ └─version (str): 0.5.0\n", 1132 | "│ │ └─software (Software)\n", 1133 | "│ │ ├─name (str): gwcs\n", 1134 | "│ │ └─version (str): 0.26.1\n", 1135 | "│ ├─3 (ExtensionMetadata)\n", 1136 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1137 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/transform/extensions/transform-1.6.0\n", 1138 | "│ │ ├─manifest_software (Software)\n", 1139 | "│ │ │ ├─name (str): asdf_transform_schemas\n", 1140 | "│ │ │ └─version (str): 0.6.0\n", 1141 | "│ │ └─software (Software)\n", 1142 | "│ │ ├─name (str): asdf-astropy\n", 1143 | "│ │ └─version (str): 0.9.0\n", 1144 | "│ ├─4 (ExtensionMetadata)\n", 1145 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1146 | "│ │ ├─extension_uri (str): asdf://stsci.edu/datamodels/roman/extensions/static-1.1.0\n", 1147 | "│ │ ├─manifest_software (Software)\n", 1148 | "│ │ │ ├─name (str): rad\n", 1149 | "│ │ │ └─version (str): 0.1.0.dev1159+g049cb984f\n", 1150 | "│ │ └─software (Software)\n", 1151 | "│ │ ├─name (str): roman_datamodels\n", 1152 | "│ │ └─version (str): 0.1.0.dev729+g446c92943\n", 1153 | "│ ├─5 (ExtensionMetadata)\n", 1154 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1155 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/astronomy/coordinates/extensions/coordinates-1.0.0\n", 1156 | "│ │ ├─manifest_software (Software)\n", 1157 | "│ │ │ ├─name (str): asdf_coordinates_schemas\n", 1158 | "│ │ │ └─version (str): 0.4.0\n", 1159 | "│ │ └─software (Software)\n", 1160 | "│ │ ├─name (str): asdf-astropy\n", 1161 | "│ │ └─version (str): 0.9.0\n", 1162 | "│ └─6 (ExtensionMetadata)\n", 1163 | "│ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1164 | "│ ├─extension_uri (str): asdf://asdf-format.org/astronomy/extensions/astronomy-1.1.0\n", 1165 | "│ ├─manifest_software (Software)\n", 1166 | "│ │ ├─name (str): asdf_standard\n", 1167 | "│ │ └─version (str): 1.4.0\n", 1168 | "│ └─software (Software)\n", 1169 | "│ ├─name (str): asdf-astropy\n", 1170 | "│ └─version (str): 0.9.0\n", 1171 | "└─roman (ExampleSpearPointedImage) # Example SPEAR pointed image CCSP product\n", 1172 | " ├─count (NDArrayType) # Count (photon)\n", 1173 | " │ ├─shape (tuple)\n", 1174 | " │ │ ├─[0] (int): 512\n", 1175 | " │ │ └─[1] (int): 512\n", 1176 | " │ └─dtype (Float32DType): >f4\n", 1177 | " ├─data (NDArrayType) # Science Data (photon/(cm^2 s sr))\n", 1178 | " │ ├─shape (tuple)\n", 1179 | " │ │ ├─[0] (int): 512\n", 1180 | " │ │ └─[1] (int): 512\n", 1181 | " │ └─dtype (Float32DType): >f4\n", 1182 | " ├─exp (NDArrayType) # Exposure Map (s)\n", 1183 | " │ ├─shape (tuple)\n", 1184 | " │ │ ├─[0] (int): 512\n", 1185 | " │ │ └─[1] (int): 512\n", 1186 | " │ └─dtype (Float32DType): >f4\n", 1187 | " └─meta (AsdfDictNode)\n", 1188 | " ├─aperture (str): 100 # Slit shutter aperture mode\n", 1189 | " ├─ccsp (AsdfDictNode) # Community Contributed Science Product (CCSP) information\n", 1190 | " │ ├─archive_lead (str): Martin Sirk # CCSP staff lead for MAST ingest\n", 1191 | " │ ├─data_release_id (str): DR1 # CCSP collection data release\n", 1192 | " │ ├─doi (str): doi:10.17909/dsbe-kj54 # CCSP Digital Object Identifier\n", 1193 | " │ ├─file_version (str): 1.0 # Version of this file\n", 1194 | " │ ├─intent (str): SCIENCE # Observation intent\n", 1195 | " │ ├─investigator (str): Jerry Edelstein # CCSP Principal Investigator\n", 1196 | " │ ├─license (str): CC BY 4.0 # License for use of these data.\n", 1197 | " │ ├─license_url (str): https://creativecommons.org/licenses/by/4.0/ # URL of license for use of these data.\n", 1198 | " │ ├─name (str): SPEAR # CCSP name\n", 1199 | " │ ├─target_keywords (str): Supernova remnants # UAT keywords\n", 1200 | " │ ├─target_keywords_id (int): 1667 # UAT keyword IDs\n", 1201 | " │ └─target_name (str): Vela # Target name\n", 1202 | " ├─exposure (AsdfDictNode) # Exposure time summary information\n", 1203 | " │ ├─end_time (Time): 2004-01-27T19:14:00.000 # End time\n", 1204 | " │ ├─exposure_time (float): 25.36313787917133 # Exposure time (s)\n", 1205 | " │ └─start_time (Time): 2004-01-27T04:16:00.000 # Start time\n", 1206 | " ├─exposure_stats (AsdfDictNode) # Exposure map statistical detail\n", 1207 | " │ ├─max_exposure_time (float): 99.90161508114453 # Max exposure time (s) per slit width^2\n", 1208 | " │ ├─median_exposure_time (float): 25.36313787917133 # Median exposure time (s) per slit width^2\n", 1209 | " │ ├─min_exposure_time (float): 0.1083651593083156 # Min exposure time (s) per slit width^2\n", 1210 | " │ └─pixel_exposure_time (float): 2.943404912948608 # Median exposure time (s) per pixel\n", 1211 | " ├─file_date (Time): 2025-11-20 19:47:41.705000 # File creation date\n", 1212 | " ├─grasp (float): 3.13019e-07 # Grasp (cm^-2 sr^-1) per exon at line wavelength\n", 1213 | " ├─instrument (AsdfDictNode) # Instrument observing configuration\n", 1214 | " │ ├─detector (NoneType): None # Detector\n", 1215 | " │ ├─name (str): FIMS-SPEAR # Instrument name\n", 1216 | " │ └─optical_element (str): L-Band_C-IV # Optical element\n", 1217 | " ├─model_type (str): ExampleSpearPointedImageModel\n", 1218 | " ├─pixel_scale (float): -6.7816944444444445e-06 # Pixel scale of spatial axes (arcsec/pixel)\n", 1219 | " ├─s_region (str): POLYGON ICRS 139.072956118736 -52.03248109358212 118.92704388126397 -52.03248109358211 120.95392087678839 -39.757741620879415 137.04607912321157 -39.757741620879415 # STC-S formatted sky footprint\n", 1220 | " ├─target_coordinates (AsdfDictNode) # Position for this data product\n", 1221 | " │ ├─dec (float): -46.25 # Declination (deg)\n", 1222 | " │ ├─ra (float): 129.0 # Right ascension (deg)\n", 1223 | " │ └─reference_frame (str): ICRS\n", 1224 | " ├─telescope (str): FIMS-SPEAR # Telescope name\n", 1225 | " ├─wavelength (AsdfDictNode) # Wavelength information\n", 1226 | " │ ├─band (str): UV # Waveband\n", 1227 | " │ ├─maximum (float): 1552.0 # Maximum wavelength\n", 1228 | " │ └─minimum (float): 1547.0 # Minimum wavelength\n", 1229 | " └─wcs (WCS) # World Coordinate System (WCS)\n" 1230 | ] 1231 | } 1232 | ], 1233 | "source": [ 1234 | "testdm.info(max_rows=None, max_cols=None)" 1235 | ] 1236 | }, 1237 | { 1238 | "cell_type": "markdown", 1239 | "id": "3d9838ac-d601-42b4-a146-34a62a399514", 1240 | "metadata": {}, 1241 | "source": [ 1242 | "And we can also retrieve the description and units of these keys programatically:" 1243 | ] 1244 | }, 1245 | { 1246 | "cell_type": "code", 1247 | "execution_count": 24, 1248 | "id": "fe3a3e31-cd40-4152-a2a7-80f84d19cc8c", 1249 | "metadata": {}, 1250 | "outputs": [ 1251 | { 1252 | "name": "stdout", 1253 | "output_type": "stream", 1254 | "text": [ 1255 | "{'description': Characteristic right ascension in degrees; typically the\n", 1256 | "center of the spatial image, or the target coordinates\n", 1257 | "of a spectrum or light curve.\n", 1258 | "}\n", 1259 | "{'unit': deg}\n" 1260 | ] 1261 | } 1262 | ], 1263 | "source": [ 1264 | "print(testdm.schema_info(\"description\",\"roman.meta.target_coordinates.ra\"))\n", 1265 | "print(testdm.schema_info(\"unit\",\"roman.meta.target_coordinates.ra\"))" 1266 | ] 1267 | }, 1268 | { 1269 | "cell_type": "markdown", 1270 | "id": "483f3ad2-eccd-47e1-a3ce-d95475eb7e65", 1271 | "metadata": {}, 1272 | "source": [ 1273 | "And because the right branch of `roman_datamodels` is installed into our Python environment, the schema is also similarly registered by the ASDF package while we're in this environment:" 1274 | ] 1275 | }, 1276 | { 1277 | "cell_type": "code", 1278 | "execution_count": 25, 1279 | "id": "333aff5b-4b6e-4fa8-ade9-5fd7f66d5fc5", 1280 | "metadata": {}, 1281 | "outputs": [], 1282 | "source": [ 1283 | "testaf = asdf.open('ccsp_example_spear_vela_long-c-iv_v0.0_img.asdf')" 1284 | ] 1285 | }, 1286 | { 1287 | "cell_type": "code", 1288 | "execution_count": 26, 1289 | "id": "4e45b114-8c28-4217-85ff-7df11f6193b6", 1290 | "metadata": {}, 1291 | "outputs": [ 1292 | { 1293 | "name": "stdout", 1294 | "output_type": "stream", 1295 | "text": [ 1296 | "root (AsdfObject)\n", 1297 | "├─asdf_library (Software)\n", 1298 | "│ ├─author (str): The ASDF Developers\n", 1299 | "│ ├─homepage (str): http://github.com/asdf-format/asdf\n", 1300 | "│ ├─name (str): asdf\n", 1301 | "│ └─version (str): 5.1.0\n", 1302 | "├─history (dict)\n", 1303 | "│ └─extensions (list)\n", 1304 | "│ ├─[0] (ExtensionMetadata)\n", 1305 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1306 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/core/extensions/core-1.6.0\n", 1307 | "│ │ ├─manifest_software (Software)\n", 1308 | "│ │ │ ├─name (str): asdf_standard\n", 1309 | "│ │ │ └─version (str): 1.4.0\n", 1310 | "│ │ └─software (Software)\n", 1311 | "│ │ ├─name (str): asdf\n", 1312 | "│ │ └─version (str): 5.1.0\n", 1313 | "│ ├─[1] (ExtensionMetadata)\n", 1314 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1315 | "│ │ ├─extension_uri (str): asdf://astropy.org/astropy/extensions/units-1.0.0\n", 1316 | "│ │ └─software (Software)\n", 1317 | "│ │ ├─name (str): asdf-astropy\n", 1318 | "│ │ └─version (str): 0.9.0\n", 1319 | "│ ├─[2] (ExtensionMetadata)\n", 1320 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1321 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/astronomy/gwcs/extensions/gwcs-1.4.0\n", 1322 | "│ │ ├─manifest_software (Software)\n", 1323 | "│ │ │ ├─name (str): asdf_wcs_schemas\n", 1324 | "│ │ │ └─version (str): 0.5.0\n", 1325 | "│ │ └─software (Software)\n", 1326 | "│ │ ├─name (str): gwcs\n", 1327 | "│ │ └─version (str): 0.26.1\n", 1328 | "│ ├─[3] (ExtensionMetadata)\n", 1329 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1330 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/transform/extensions/transform-1.6.0\n", 1331 | "│ │ ├─manifest_software (Software)\n", 1332 | "│ │ │ ├─name (str): asdf_transform_schemas\n", 1333 | "│ │ │ └─version (str): 0.6.0\n", 1334 | "│ │ └─software (Software)\n", 1335 | "│ │ ├─name (str): asdf-astropy\n", 1336 | "│ │ └─version (str): 0.9.0\n", 1337 | "│ ├─[4] (ExtensionMetadata)\n", 1338 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1339 | "│ │ ├─extension_uri (str): asdf://stsci.edu/datamodels/roman/extensions/static-1.1.0\n", 1340 | "│ │ ├─manifest_software (Software)\n", 1341 | "│ │ │ ├─name (str): rad\n", 1342 | "│ │ │ └─version (str): 0.1.0.dev1159+g049cb984f\n", 1343 | "│ │ └─software (Software)\n", 1344 | "│ │ ├─name (str): roman_datamodels\n", 1345 | "│ │ └─version (str): 0.1.0.dev729+g446c92943\n", 1346 | "│ ├─[5] (ExtensionMetadata)\n", 1347 | "│ │ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1348 | "│ │ ├─extension_uri (str): asdf://asdf-format.org/astronomy/coordinates/extensions/coordinates-1.0.0\n", 1349 | "│ │ ├─manifest_software (Software)\n", 1350 | "│ │ │ ├─name (str): asdf_coordinates_schemas\n", 1351 | "│ │ │ └─version (str): 0.4.0\n", 1352 | "│ │ └─software (Software)\n", 1353 | "│ │ ├─name (str): asdf-astropy\n", 1354 | "│ │ └─version (str): 0.9.0\n", 1355 | "│ └─[6] (ExtensionMetadata)\n", 1356 | "│ ├─extension_class (str): asdf.extension._manifest.ManifestExtension\n", 1357 | "│ ├─extension_uri (str): asdf://asdf-format.org/astronomy/extensions/astronomy-1.1.0\n", 1358 | "│ ├─manifest_software (Software)\n", 1359 | "│ │ ├─name (str): asdf_standard\n", 1360 | "│ │ └─version (str): 1.4.0\n", 1361 | "│ └─software (Software)\n", 1362 | "│ ├─name (str): asdf-astropy\n", 1363 | "│ └─version (str): 0.9.0\n", 1364 | "└─roman (ExampleSpearPointedImage) # Example SPEAR pointed image CCSP product\n", 1365 | " ├─count (NDArrayType) # Count (photon)\n", 1366 | " │ ├─shape (tuple)\n", 1367 | " │ │ ├─[0] (int): 512\n", 1368 | " │ │ └─[1] (int): 512\n", 1369 | " │ └─dtype (Float32DType): >f4\n", 1370 | " ├─data (NDArrayType) # Science Data (photon/(cm^2 s sr))\n", 1371 | " │ ├─shape (tuple)\n", 1372 | " │ │ ├─[0] (int): 512\n", 1373 | " │ │ └─[1] (int): 512\n", 1374 | " │ └─dtype (Float32DType): >f4\n", 1375 | " ├─exp (NDArrayType) # Exposure Map (s)\n", 1376 | " │ ├─shape (tuple)\n", 1377 | " │ │ ├─[0] (int): 512\n", 1378 | " │ │ └─[1] (int): 512\n", 1379 | " │ └─dtype (Float32DType): >f4\n", 1380 | " └─meta (dict)\n", 1381 | " ├─aperture (str): 100 # Slit shutter aperture mode\n", 1382 | " ├─ccsp (dict) # Community Contributed Science Product (CCSP) information\n", 1383 | " │ ├─archive_lead (str): Martin Sirk # CCSP staff lead for MAST ingest\n", 1384 | " │ ├─data_release_id (str): DR1 # CCSP collection data release\n", 1385 | " │ ├─doi (str): doi:10.17909/dsbe-kj54 # CCSP Digital Object Identifier\n", 1386 | " │ ├─file_version (str): 1.0 # Version of this file\n", 1387 | " │ ├─intent (str): SCIENCE # Observation intent\n", 1388 | " │ ├─investigator (str): Jerry Edelstein # CCSP Principal Investigator\n", 1389 | " │ ├─license (str): CC BY 4.0 # License for use of these data.\n", 1390 | " │ ├─license_url (str): https://creativecommons.org/licenses/by/4.0/ # URL of license for use of these data.\n", 1391 | " │ ├─name (str): SPEAR # CCSP name\n", 1392 | " │ ├─target_keywords (str): Supernova remnants # UAT keywords\n", 1393 | " │ ├─target_keywords_id (int): 1667 # UAT keyword IDs\n", 1394 | " │ └─target_name (str): Vela # Target name\n", 1395 | " ├─exposure (dict) # Exposure time summary information\n", 1396 | " │ ├─end_time (Time): 2004-01-27T19:14:00.000 # End time\n", 1397 | " │ ├─exposure_time (float): 25.36313787917133 # Exposure time (s)\n", 1398 | " │ └─start_time (Time): 2004-01-27T04:16:00.000 # Start time\n", 1399 | " ├─exposure_stats (dict) # Exposure map statistical detail\n", 1400 | " │ ├─max_exposure_time (float): 99.90161508114453 # Max exposure time (s) per slit width^2\n", 1401 | " │ ├─median_exposure_time (float): 25.36313787917133 # Median exposure time (s) per slit width^2\n", 1402 | " │ ├─min_exposure_time (float): 0.1083651593083156 # Min exposure time (s) per slit width^2\n", 1403 | " │ └─pixel_exposure_time (float): 2.943404912948608 # Median exposure time (s) per pixel\n", 1404 | " ├─file_date (Time): 2025-11-20 19:47:41.705000 # File creation date\n", 1405 | " ├─grasp (float): 3.13019e-07 # Grasp (cm^-2 sr^-1) per exon at line wavelength\n", 1406 | " ├─instrument (dict) # Instrument observing configuration\n", 1407 | " │ ├─detector (NoneType): None # Detector\n", 1408 | " │ ├─name (str): FIMS-SPEAR # Instrument name\n", 1409 | " │ └─optical_element (str): L-Band_C-IV # Optical element\n", 1410 | " ├─model_type (str): ExampleSpearPointedImageModel\n", 1411 | " ├─pixel_scale (float): -6.7816944444444445e-06 # Pixel scale of spatial axes (arcsec/pixel)\n", 1412 | " ├─s_region (str): POLYGON ICRS 139.072956118736 -52.03248109358212 118.92704388126397 -52.03248109358211 120.95392087678839 -39.757741620879415 137.04607912321157 -39.757741620879415 # STC-S formatted sky footprint\n", 1413 | " ├─target_coordinates (dict) # Position for this data product\n", 1414 | " │ ├─dec (float): -46.25 # Declination (deg)\n", 1415 | " │ ├─ra (float): 129.0 # Right ascension (deg)\n", 1416 | " │ └─reference_frame (str): ICRS\n", 1417 | " ├─telescope (str): FIMS-SPEAR # Telescope name\n", 1418 | " ├─wavelength (dict) # Wavelength information\n", 1419 | " │ ├─band (str): UV # Waveband\n", 1420 | " │ ├─maximum (float): 1552.0 # Maximum wavelength\n", 1421 | " │ └─minimum (float): 1547.0 # Minimum wavelength\n", 1422 | " └─wcs (WCS) # World Coordinate System (WCS)\n" 1423 | ] 1424 | } 1425 | ], 1426 | "source": [ 1427 | "testaf.info(max_rows=None, max_cols=None)" 1428 | ] 1429 | }, 1430 | { 1431 | "cell_type": "code", 1432 | "execution_count": 27, 1433 | "id": "91589f49-8ce9-4328-b082-20cc581b005a", 1434 | "metadata": {}, 1435 | "outputs": [ 1436 | { 1437 | "name": "stdout", 1438 | "output_type": "stream", 1439 | "text": [ 1440 | "{'description': Characteristic right ascension in degrees; typically the\n", 1441 | "center of the spatial image, or the target coordinates\n", 1442 | "of a spectrum or light curve.\n", 1443 | "}\n", 1444 | "{'unit': deg}\n" 1445 | ] 1446 | } 1447 | ], 1448 | "source": [ 1449 | "# This only works because `roman_datamodels` is installed in our environment,\n", 1450 | "# serving to connect the file to its tagged schema\n", 1451 | "print(testaf.schema_info(\"description\",\"roman.meta.target_coordinates.ra\"))\n", 1452 | "print(testaf.schema_info(\"unit\",\"roman.meta.target_coordinates.ra\"))" 1453 | ] 1454 | }, 1455 | { 1456 | "cell_type": "code", 1457 | "execution_count": 28, 1458 | "id": "5ee7b595-9dd4-4277-8d9d-6fe2812ebb91", 1459 | "metadata": {}, 1460 | "outputs": [], 1461 | "source": [ 1462 | "# Let's finally close the FITS file, now that we're done with it\n", 1463 | "hdul.close()" 1464 | ] 1465 | } 1466 | ], 1467 | "metadata": { 1468 | "kernelspec": { 1469 | "display_name": "Python 3 (ipykernel)", 1470 | "language": "python", 1471 | "name": "python3" 1472 | }, 1473 | "language_info": { 1474 | "codemirror_mode": { 1475 | "name": "ipython", 1476 | "version": 3 1477 | }, 1478 | "file_extension": ".py", 1479 | "mimetype": "text/x-python", 1480 | "name": "python", 1481 | "nbconvert_exporter": "python", 1482 | "pygments_lexer": "ipython3", 1483 | "version": "3.14.0" 1484 | } 1485 | }, 1486 | "nbformat": 4, 1487 | "nbformat_minor": 5 1488 | } 1489 | --------------------------------------------------------------------------------