├── docs ├── rtd_requirements.txt ├── citation.rst ├── api.rst ├── whatsnew │ ├── index.rst │ └── changelog.rst ├── index.rst ├── make.bat ├── intro.rst ├── conf.py ├── Makefile └── tutorial.rst ├── CITATION.rst ├── drms ├── tests │ ├── __init__.py │ ├── test_exceptions.py │ ├── test_main.py │ ├── test_json.py │ ├── test_jsoc_basic.py │ ├── test_client.py │ ├── test_init.py │ ├── test_jsoc_info.py │ ├── test_jsoc_email.py │ ├── test_utils.py │ ├── test_kis_basic.py │ ├── conftest.py │ ├── test_jsoc_query.py │ ├── test_to_datetime.py │ ├── test_series_info.py │ ├── test_jsoc_export.py │ └── test_config.py ├── _dev │ ├── __init__.py │ └── scm_version.py ├── data │ └── README.rst ├── exceptions.py ├── version.py ├── main.py ├── CITATION.rst ├── __init__.py ├── utils.py ├── config.py └── json.py ├── setup.py ├── changelog ├── 156.doc.rst └── README.rst ├── .gitattributes ├── examples ├── README.txt ├── list_hmi_series.py ├── list_keywords.py ├── skip_export_from_id.py ├── export_print_urls.py ├── plot_hmi_lightcurve.py ├── export_as_is.py ├── export_fits.py ├── export_jpg.py ├── export_tar.py ├── export_movie.py ├── plot_aia_lightcurve.py ├── plot_synoptic_mr.py ├── plot_polarfield.py ├── cutout_export_request.py └── plot_hmi_modes.py ├── .rtd-environment.yml ├── 158.breaking.rst ├── .codecov.yaml ├── .black ├── .editorconfig ├── .codespellrc ├── MANIFEST.in ├── .readthedocs.yml ├── .isort.cfg ├── .readthedocs.yaml ├── .flake8 ├── .github └── workflows │ ├── label_sync.yml │ ├── sub_package_update.yml │ └── ci.yml ├── .coveragerc ├── .pre-commit-config.yaml ├── LICENSE.rst ├── .cruft.json ├── pytest.ini ├── .ruff.toml ├── tox.ini ├── pyproject.toml ├── CHANGELOG.rst ├── README.rst └── .gitignore /docs/rtd_requirements.txt: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /CITATION.rst: -------------------------------------------------------------------------------- 1 | drms/CITATION.rst -------------------------------------------------------------------------------- /drms/tests/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | This module contains package tests. 3 | """ 4 | -------------------------------------------------------------------------------- /docs/citation.rst: -------------------------------------------------------------------------------- 1 | .. _drms-citation: 2 | 3 | .. include:: ../drms/CITATION.rst 4 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | from setuptools import setup 3 | 4 | setup() 5 | -------------------------------------------------------------------------------- /changelog/156.doc.rst: -------------------------------------------------------------------------------- 1 | Added export status codes to docstring of `drms.ExportResult.status`. 2 | -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | *fits binary 2 | *fit binary 3 | *fts binary 4 | *fit.gz binary 5 | *fits.gz binary 6 | *fts.gz binary 7 | -------------------------------------------------------------------------------- /examples/README.txt: -------------------------------------------------------------------------------- 1 | *************** 2 | Example Gallery 3 | *************** 4 | 5 | This gallery contains examples of how to use ``drms``. 6 | -------------------------------------------------------------------------------- /.rtd-environment.yml: -------------------------------------------------------------------------------- 1 | name: drms 2 | channels: 3 | - conda-forge 4 | dependencies: 5 | - python=3.12 6 | - pip 7 | - graphviz!=2.42.*,!=2.43.* 8 | -------------------------------------------------------------------------------- /docs/api.rst: -------------------------------------------------------------------------------- 1 | .. _drms-api-reference: 2 | 3 | ************* 4 | API Reference 5 | ************* 6 | 7 | .. automodapi:: drms 8 | :no-heading: 9 | -------------------------------------------------------------------------------- /158.breaking.rst: -------------------------------------------------------------------------------- 1 | Increased minimum version of Python to 3.12. 2 | Increased minimum version of NumPy to 1.26.0. 3 | Increased minimum version of pandas to 2.2.0. 4 | -------------------------------------------------------------------------------- /docs/whatsnew/index.rst: -------------------------------------------------------------------------------- 1 | .. _drms-release-history: 2 | 3 | *************** 4 | Release History 5 | *************** 6 | 7 | .. toctree:: 8 | :maxdepth: 1 9 | 10 | changelog 11 | -------------------------------------------------------------------------------- /.codecov.yaml: -------------------------------------------------------------------------------- 1 | comment: off 2 | coverage: 3 | status: 4 | project: 5 | default: 6 | threshold: 0.2% 7 | 8 | codecov: 9 | require_ci_to_pass: false 10 | notify: 11 | wait_for_ci: true 12 | -------------------------------------------------------------------------------- /drms/_dev/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | This package contains utilities that are only used when developing in a 3 | copy of the source repository. 4 | 5 | These files are not installed, and should not be assumed to exist at 6 | runtime. 7 | """ 8 | -------------------------------------------------------------------------------- /docs/whatsnew/changelog.rst: -------------------------------------------------------------------------------- 1 | .. _drms-changelog: 2 | 3 | ************** 4 | Full Changelog 5 | ************** 6 | 7 | .. changelog:: 8 | :towncrier: ../../ 9 | :towncrier-skip-if-empty: 10 | :changelog_file: ../../CHANGELOG.rst 11 | -------------------------------------------------------------------------------- /.black: -------------------------------------------------------------------------------- 1 | target-version = ['py310'] 2 | exclude = ''' 3 | ( 4 | /( 5 | \.eggs 6 | | \.git 7 | | \.mypy_cache 8 | | \.tox 9 | | \.venv 10 | | _build 11 | | build 12 | | dist 13 | | docs 14 | | .history 15 | )/ 16 | ) 17 | ''' 18 | -------------------------------------------------------------------------------- /drms/data/README.rst: -------------------------------------------------------------------------------- 1 | Data directory 2 | ============== 3 | 4 | This directory contains data files included with the package source 5 | code distribution. Note that this is intended only for relatively small files 6 | - large files should be externally hosted and downloaded as needed. 7 | -------------------------------------------------------------------------------- /.editorconfig: -------------------------------------------------------------------------------- 1 | # https://editorconfig.org 2 | 3 | root = true 4 | 5 | # utf, UNIX-style new line 6 | [*] 7 | charset = utf-8 8 | end_of_line = lf 9 | insert_final_newline = true 10 | trim_trailing_whitespace = true 11 | 12 | [*.{py,rst,md}] 13 | indent_style = space 14 | indent_size = 4 15 | 16 | [*.yml] 17 | indent_style = space 18 | indent_size = 2 19 | -------------------------------------------------------------------------------- /.codespellrc: -------------------------------------------------------------------------------- 1 | [codespell] 2 | skip = *.asdf,*.fits,*.fts,*.header,*.json,*.xsh,*cache*,*egg*,*extern*,.git,.idea,.tox,_build,*truncated,*.svg,.asv_env,.history 3 | ignore-words-list = 4 | afile, 5 | alog, 6 | nd, 7 | nin, 8 | observ, 9 | ot, 10 | precess 11 | precessed, 12 | requestor, 13 | sav, 14 | te, 15 | upto, 16 | parms, 17 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | # Exclude specific files 2 | # All files which are tracked by git and not explicitly excluded here are included by setuptools_scm 3 | # Prune folders 4 | prune build 5 | prune docs/_build 6 | prune docs/api 7 | global-exclude *.pyc *.o 8 | 9 | # This subpackage is only used in development checkouts 10 | # and should not be included in built tarballs 11 | prune drms/_dev 12 | -------------------------------------------------------------------------------- /.readthedocs.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | build: 3 | os: ubuntu-20.04 4 | tools: 5 | python: "3.11" 6 | apt_packages: 7 | - graphviz 8 | 9 | sphinx: 10 | builder: html 11 | configuration: docs/conf.py 12 | fail_on_warning: false 13 | 14 | python: 15 | install: 16 | - method: pip 17 | extra_requirements: 18 | - all 19 | - docs 20 | path: . 21 | -------------------------------------------------------------------------------- /drms/tests/test_exceptions.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | import drms 4 | 5 | 6 | @pytest.mark.parametrize( 7 | "exception_class", 8 | [drms.DrmsError, drms.DrmsQueryError, drms.DrmsExportError, drms.DrmsOperationNotSupported], 9 | ) 10 | def test_exception_class(exception_class): 11 | with pytest.raises(RuntimeError): 12 | raise exception_class 13 | with pytest.raises(drms.DrmsError): 14 | raise exception_class 15 | -------------------------------------------------------------------------------- /drms/_dev/scm_version.py: -------------------------------------------------------------------------------- 1 | # Try to use setuptools_scm to get the current version; this is only used 2 | # in development installations from the git repository. 3 | from pathlib import Path 4 | 5 | try: 6 | from setuptools_scm import get_version 7 | 8 | version = get_version(root=Path("../.."), relative_to=__file__) 9 | except ImportError: 10 | raise 11 | except Exception as e: 12 | raise ValueError("setuptools_scm can not determine version.") from e 13 | -------------------------------------------------------------------------------- /.isort.cfg: -------------------------------------------------------------------------------- 1 | [settings] 2 | balanced_wrapping = true 3 | skip = 4 | docs/conf.py 5 | drms/__init__.py 6 | default_section = THIRDPARTY 7 | include_trailing_comma = true 8 | known_astropy = astropy, asdf 9 | known_sunpy = sunpy 10 | known_first_party = drms 11 | length_sort = false 12 | length_sort_sections = stdlib 13 | line_length = 110 14 | multi_line_output = 3 15 | no_lines_before = LOCALFOLDER 16 | sections = STDLIB, THIRDPARTY, ASTROPY, SUNPY, FIRSTPARTY, LOCALFOLDER 17 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | .. _drms-index: 2 | 3 | ********************** 4 | ``drms`` documentation 5 | ********************** 6 | 7 | :Github: https://github.com/sunpy/drms 8 | :PyPI: https://pypi.org/project/drms/ 9 | 10 | Python library for accessing HMI, AIA and MDI data from the Joint Science Operations Center (JSOC) or other Data Record Management System (DRMS) servers. 11 | 12 | .. toctree:: 13 | :maxdepth: 2 14 | :caption: Contents: 15 | 16 | intro 17 | tutorial 18 | api 19 | generated/gallery/index 20 | citation 21 | whatsnew/index 22 | -------------------------------------------------------------------------------- /drms/exceptions.py: -------------------------------------------------------------------------------- 1 | __all__ = [ 2 | "DrmsError", 3 | "DrmsExportError", 4 | "DrmsOperationNotSupported", 5 | "DrmsQueryError", 6 | ] 7 | 8 | 9 | class DrmsError(RuntimeError): 10 | """ 11 | Unspecified DRMS run-time error. 12 | """ 13 | 14 | 15 | class DrmsQueryError(DrmsError): 16 | """ 17 | DRMS query error. 18 | """ 19 | 20 | 21 | class DrmsExportError(DrmsError): 22 | """ 23 | DRMS data export error. 24 | """ 25 | 26 | 27 | class DrmsOperationNotSupported(DrmsError): 28 | """ 29 | Operation is not supported by DRMS server. 30 | """ 31 | -------------------------------------------------------------------------------- /.readthedocs.yaml: -------------------------------------------------------------------------------- 1 | version: 2 2 | 3 | build: 4 | os: ubuntu-lts-latest 5 | tools: 6 | python: "mambaforge-latest" 7 | jobs: 8 | post_checkout: 9 | - git fetch --unshallow || true 10 | pre_install: 11 | - git update-index --assume-unchanged .rtd-environment.yml docs/conf.py 12 | 13 | conda: 14 | environment: .rtd-environment.yml 15 | 16 | sphinx: 17 | builder: html 18 | configuration: docs/conf.py 19 | fail_on_warning: false 20 | 21 | formats: 22 | - htmlzip 23 | 24 | python: 25 | install: 26 | - method: pip 27 | extra_requirements: 28 | - docs 29 | path: . 30 | -------------------------------------------------------------------------------- /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | ignore = 3 | # missing-whitespace-around-operator 4 | E225 5 | # missing-whitespace-around-arithmetic-operator 6 | E226 7 | # line-too-long 8 | E501 9 | # unused-import 10 | F401 11 | # undefined-local-with-import-star 12 | F403 13 | # redefined-while-unused 14 | F811 15 | # Line break occurred before a binary operator 16 | W503, 17 | # Line break occurred after a binary operator 18 | W504 19 | max-line-length = 110 20 | exclude = 21 | .git 22 | __pycache__ 23 | docs/conf.py 24 | build 25 | drms/__init__.py 26 | rst-directives = 27 | plot 28 | -------------------------------------------------------------------------------- /drms/tests/test_main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | 3 | import pytest 4 | 5 | from drms.main import main, parse_args 6 | 7 | 8 | def helper(args, name, expected): 9 | args = parse_args(args) 10 | assert getattr(args, name) == expected 11 | 12 | 13 | def test_version(): 14 | with pytest.raises(SystemExit): 15 | helper(["--version"], "version", expected=True) 16 | 17 | 18 | def test_server(): 19 | helper(["fake_url_server"], "server", "fake_url_server") 20 | helper([], "server", "jsoc") 21 | 22 | 23 | def test_email(): 24 | helper(["--email", "fake@gmail.com"], "email", "fake@gmail.com") 25 | helper([], "email", None) 26 | 27 | 28 | def test_main_empty(): 29 | sys.argv = [ 30 | "drms", 31 | ] 32 | main() 33 | -------------------------------------------------------------------------------- /.github/workflows/label_sync.yml: -------------------------------------------------------------------------------- 1 | name: Label Sync 2 | on: 3 | workflow_dispatch: 4 | schedule: 5 | # ┌───────── minute (0 - 59) 6 | # │ ┌───────── hour (0 - 23) 7 | # │ │ ┌───────── day of the month (1 - 31) 8 | # │ │ │ ┌───────── month (1 - 12 or JAN-DEC) 9 | # │ │ │ │ ┌───────── day of the week (0 - 6 or SUN-SAT) 10 | - cron: '0 0 * * *' # run every day at midnight UTC 11 | 12 | # Give permissions to write issue labels 13 | permissions: 14 | issues: write 15 | 16 | jobs: 17 | label_sync: 18 | runs-on: ubuntu-latest 19 | name: Label Sync 20 | steps: 21 | - uses: srealmoreno/label-sync-action@850ba5cef2b25e56c6c420c4feed0319294682fd 22 | with: 23 | config-file: https://raw.githubusercontent.com/sunpy/.github/main/labels.yml 24 | -------------------------------------------------------------------------------- /drms/version.py: -------------------------------------------------------------------------------- 1 | # NOTE: First try _dev.scm_version if it exists and setuptools_scm is installed 2 | # This file is not included in wheels/tarballs, so otherwise it will 3 | # fall back on the generated _version module. 4 | try: 5 | try: 6 | from ._dev.scm_version import version 7 | except ImportError: 8 | from ._version import version 9 | except Exception: # NOQA: BLE001 10 | import warnings 11 | 12 | warnings.warn( 13 | f'could not determine {__name__.split(".")[0]} package version; this indicates a broken installation', 14 | stacklevel=3, 15 | ) 16 | del warnings 17 | version = "0.0.0" 18 | 19 | from packaging.version import parse as _parse 20 | 21 | _version = _parse(version) 22 | major, minor, bugfix = [*_version.release, 0][:3] 23 | release = not _version.is_devrelease 24 | -------------------------------------------------------------------------------- /drms/main.py: -------------------------------------------------------------------------------- 1 | import sys 2 | import argparse 3 | 4 | 5 | def main(): 6 | import drms # noqa: PLC0415 7 | 8 | args = parse_args(sys.argv[1:]) 9 | client = drms.Client(server=args.server, email=args.email) 10 | drms.logger.info(f"client: {client}") 11 | 12 | 13 | def parse_args(args): 14 | import drms # noqa: PLC0415 15 | 16 | parser = argparse.ArgumentParser(description="drms, access HMI, AIA and MDI data with python") 17 | parser.add_argument( 18 | "--version", 19 | action="version", 20 | version=f"drms v{drms.__version__}", 21 | help="show package version and exit", 22 | ) 23 | parser.add_argument("--email", help="email address for data export requests") 24 | parser.add_argument("server", nargs="?", default="jsoc", help="DRMS server, default is JSOC") 25 | 26 | return parser.parse_args(args) 27 | -------------------------------------------------------------------------------- /.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | omit = 3 | drms/conftest.py 4 | drms/*setup_package* 5 | drms/extern/* 6 | drms/version* 7 | */drms/conftest.py 8 | */drms/*setup_package* 9 | */drms/extern/* 10 | */drms/version* 11 | 12 | [report] 13 | exclude_lines = 14 | # Have to re-enable the standard pragma 15 | pragma: no cover 16 | # Don't complain about packages we have installed 17 | except ImportError 18 | # Don't complain if tests don't hit assertions 19 | raise AssertionError 20 | raise NotImplementedError 21 | # Don't complain about script hooks 22 | def main(.*): 23 | # Ignore branches that don't pertain to this version of Python 24 | pragma: py{ignore_python_version} 25 | # Don't complain about IPython completion helper 26 | def _ipython_key_completions_ 27 | # typing.TYPE_CHECKING is False at runtime 28 | if TYPE_CHECKING: 29 | # Ignore typing overloads 30 | @overload 31 | -------------------------------------------------------------------------------- /drms/tests/test_json.py: -------------------------------------------------------------------------------- 1 | from unittest.mock import patch 2 | 3 | import pytest 4 | 5 | from drms.client import Client 6 | from drms.json import HttpJsonRequest, JsocInfoConstants 7 | 8 | 9 | @pytest.mark.remote_data() 10 | def test_jsocinfoconstants(): 11 | assert isinstance(JsocInfoConstants.all, str) 12 | assert JsocInfoConstants.all == "**ALL**" 13 | client = Client() 14 | client.query("hmi.synoptic_mr_720s[2150]", key=JsocInfoConstants.all, seg="synopMr") 15 | 16 | 17 | def test_request_headers(): 18 | with patch("drms.json.urlopen") as mock: 19 | HttpJsonRequest("http://example.com", "latin1") 20 | 21 | actual_request = mock.call_args[0][0] 22 | assert actual_request.headers["User-agent"] 23 | assert "drms/" in actual_request.headers["User-agent"] 24 | assert "python/" in actual_request.headers["User-agent"] 25 | assert actual_request.full_url == "http://example.com" 26 | -------------------------------------------------------------------------------- /drms/tests/test_jsoc_basic.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | 4 | @pytest.mark.jsoc() 5 | @pytest.mark.remote_data() 6 | def test_series_list_all(jsoc_client): 7 | slist = jsoc_client.series() 8 | assert isinstance(slist, list) 9 | assert "hmi.v_45s" in (s.lower() for s in slist) 10 | assert "hmi.m_720s" in (s.lower() for s in slist) 11 | assert "hmi.ic_720s" in (s.lower() for s in slist) 12 | assert "aia.lev1" in (s.lower() for s in slist) 13 | assert "aia.lev1_euv_12s" in (s.lower() for s in slist) 14 | assert "mdi.fd_v" in (s.lower() for s in slist) 15 | 16 | 17 | @pytest.mark.jsoc() 18 | @pytest.mark.remote_data() 19 | @pytest.mark.parametrize("schema", ["aia", "hmi", "mdi"]) 20 | def test_series_list_schemata(jsoc_client, schema): 21 | regex = rf"{schema}\." 22 | slist = jsoc_client.series(regex) 23 | assert len(slist) > 0 24 | for sname in slist: 25 | assert sname.startswith(f"{schema}.") 26 | -------------------------------------------------------------------------------- /docs/make.bat: -------------------------------------------------------------------------------- 1 | @ECHO OFF 2 | 3 | pushd %~dp0 4 | 5 | REM Command file for Sphinx documentation 6 | 7 | if "%SPHINXBUILD%" == "" ( 8 | set SPHINXBUILD=sphinx-build 9 | ) 10 | set SOURCEDIR=. 11 | set BUILDDIR=_build 12 | 13 | if "%1" == "" goto help 14 | 15 | %SPHINXBUILD% >NUL 2>NUL 16 | if errorlevel 9009 ( 17 | echo. 18 | echo.The 'sphinx-build' command was not found. Make sure you have Sphinx 19 | echo.installed, then set the SPHINXBUILD environment variable to point 20 | echo.to the full path of the 'sphinx-build' executable. Alternatively you 21 | echo.may add the Sphinx directory to PATH. 22 | echo. 23 | echo.If you don't have Sphinx installed, grab it from 24 | echo.http://sphinx-doc.org/ 25 | exit /b 1 26 | ) 27 | 28 | %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 29 | goto end 30 | 31 | :help 32 | %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% 33 | 34 | :end 35 | popd 36 | -------------------------------------------------------------------------------- /drms/tests/test_client.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | import drms 4 | from drms.config import ServerConfig 5 | 6 | 7 | def test_client_init_defaults(): 8 | c = drms.Client() 9 | assert isinstance(c._server, ServerConfig) 10 | assert c._server.name.lower() == "jsoc" 11 | assert c.email is None 12 | 13 | 14 | @pytest.mark.parametrize("server_name", ["jsoc", "kis"]) 15 | def test_client_registered_servers(server_name): 16 | c = drms.Client(server_name) 17 | assert isinstance(c._server, ServerConfig) 18 | assert c._server.name.lower() == server_name 19 | assert c.email is None 20 | 21 | 22 | def test_client_custom_config(): 23 | cfg = ServerConfig(name="TEST") 24 | c = drms.Client(cfg) 25 | assert isinstance(c._server, ServerConfig) 26 | assert c._server.name == "TEST" 27 | 28 | 29 | def test_repr(): 30 | assert repr(drms.Client()) == "" 31 | assert repr(drms.Client("kis")) == "" 32 | -------------------------------------------------------------------------------- /examples/list_hmi_series.py: -------------------------------------------------------------------------------- 1 | """ 2 | ============================================== 3 | Print all HMI series and prime keys with notes 4 | ============================================== 5 | 6 | This example shows how to find and display the HMI series names, prime keys and corresponding notes. 7 | """ 8 | 9 | import textwrap 10 | 11 | import drms 12 | 13 | ############################################################################### 14 | # First we will create a `drms.Client`, using the JSOC baseurl. 15 | client = drms.Client() 16 | 17 | ############################################################################### 18 | # Get all available HMI series and print their names, prime keys and notes. 19 | 20 | hmi_series = client.series(regex=r"hmi\.", full=True) 21 | 22 | # Print series names, prime-keys (pkeys) and notes 23 | for series in hmi_series.index: 24 | print("Series:", hmi_series.name[series]) 25 | print(" Notes:", (f"\n{8 * ' '}").join(textwrap.wrap(hmi_series.note[series]))) 26 | -------------------------------------------------------------------------------- /.pre-commit-config.yaml: -------------------------------------------------------------------------------- 1 | exclude: ".*(.fits|.fts|.fit|.header|.txt|tca.*|extern.*|drms/extern)$|^CITATION.rst$" 2 | 3 | repos: 4 | - repo: https://github.com/astral-sh/ruff-pre-commit 5 | rev: "v0.14.3" 6 | hooks: 7 | - id: ruff 8 | args: ['--fix', '--unsafe-fixes'] 9 | - id: ruff-format 10 | - repo: https://github.com/PyCQA/isort 11 | rev: 7.0.0 12 | hooks: 13 | - id: isort 14 | - repo: https://github.com/PyCQA/isort 15 | rev: 6.1.0 16 | hooks: 17 | - id: isort 18 | - repo: https://github.com/pre-commit/pre-commit-hooks 19 | rev: v6.0.0 20 | hooks: 21 | - id: check-ast 22 | - id: check-case-conflict 23 | - id: trailing-whitespace 24 | - id: check-yaml 25 | - id: debug-statements 26 | - id: check-added-large-files 27 | args: ["--enforce-all", "--maxkb=1054"] 28 | - id: end-of-file-fixer 29 | - id: mixed-line-ending 30 | - repo: https://github.com/codespell-project/codespell 31 | rev: v2.4.1 32 | hooks: 33 | - id: codespell 34 | ci: 35 | autofix_prs: false 36 | autoupdate_schedule: "quarterly" 37 | -------------------------------------------------------------------------------- /examples/list_keywords.py: -------------------------------------------------------------------------------- 1 | """ 2 | ================================== 3 | List all keywords for a HMI series 4 | ================================== 5 | 6 | This example shows how to display the keywords for a HMI series. 7 | """ 8 | 9 | import drms 10 | 11 | ############################################################################### 12 | # First we will create a `drms.Client`, using the JSOC baseurl. 13 | 14 | client = drms.Client() 15 | 16 | ############################################################################### 17 | # Query series info 18 | 19 | series_info = client.info("hmi.v_45s") 20 | 21 | ############################################################################### 22 | # Print keyword info 23 | 24 | print(f"Listing keywords for {series_info.name}:\n") 25 | for keyword in sorted(series_info.keywords.index): 26 | keyword_info = series_info.keywords.loc[keyword] 27 | print(keyword) 28 | print(f" type ....... {keyword_info.type} ") 29 | print(f" recscope ... {keyword_info.recscope} ") 30 | print(f" defval ..... {keyword_info.defval} ") 31 | print(f" units ...... {keyword_info.units} ") 32 | print(f" note ....... {keyword_info.note} ") 33 | -------------------------------------------------------------------------------- /drms/tests/test_init.py: -------------------------------------------------------------------------------- 1 | import re 2 | 3 | import pytest 4 | 5 | import drms 6 | 7 | 8 | @pytest.mark.parametrize( 9 | "symbol", 10 | [ 11 | "__bibtex__", 12 | "__citation__", 13 | "__version__", 14 | "client", 15 | "Client", 16 | "config", 17 | "DrmsError", 18 | "DrmsExportError", 19 | "DrmsOperationNotSupported", 20 | "DrmsQueryError", 21 | "exceptions", 22 | "ExportRequest", 23 | "HttpJsonClient", 24 | "HttpJsonRequest", 25 | "JsocInfoConstants", 26 | "json", 27 | "main", 28 | "Path", 29 | "register_server", 30 | "SeriesInfo", 31 | "ServerConfig", 32 | "to_datetime", 33 | "utils", 34 | "version", 35 | ], 36 | ) 37 | def test_symbols(symbol): 38 | assert symbol in dir(drms) 39 | 40 | 41 | def test_version(): 42 | assert isinstance(drms.__version__, str) 43 | version = drms.__version__.split("+")[0] 44 | # Check to make sure it isn't empty 45 | assert version 46 | # To match the 0.6 in 0.6.dev3 or v0.6 47 | m = re.match(r"v*\d+\.\d+\.*", version) 48 | assert m is not None 49 | 50 | 51 | def test_bibtex(): 52 | assert isinstance(drms.__citation__, str) 53 | m = re.match(r".*Glogowski2019.*", drms.__citation__) 54 | assert m is not None 55 | -------------------------------------------------------------------------------- /LICENSE.rst: -------------------------------------------------------------------------------- 1 | Copyright (c) 2013-2023 The SunPy developers 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are 6 | met: 7 | 8 | * Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | 11 | * Redistributions in binary form must reproduce the above copyright 12 | notice, this list of conditions and the following disclaimer in the 13 | documentation and/or other materials provided with the distribution. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 16 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 17 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 18 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 19 | HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 20 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 21 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 22 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 23 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /examples/skip_export_from_id.py: -------------------------------------------------------------------------------- 1 | """ 2 | ================================== 3 | Exporting from existing RequestIDs 4 | ================================== 5 | 6 | This example takes a RequestID of an already existing export request, prints 7 | the corresponding "Request URL" and downloads the available files. 8 | 9 | Note that you can also use RequestIDs from export requests, that were 10 | submitted using the JSOC website. 11 | """ 12 | 13 | from pathlib import Path 14 | 15 | import drms 16 | 17 | ############################################################################### 18 | # First we will create a `drms.Client`, using the JSOC baseurl. 19 | 20 | client = drms.Client() 21 | 22 | # Export request ID 23 | request_id = "JSOC_20201101_198" 24 | 25 | # Querying the server using the entered RequestID. 26 | print(f"Looking up export request {request_id}...") 27 | result = client.export_from_id(request_id) 28 | 29 | # Print request URL and number of available files. 30 | print(f"\nRequest URL: {result.request_url}") 31 | print(f"{len(result.urls)} file(s) available for download.\n") 32 | 33 | # Create download directory if it does not exist yet. 34 | out_dir = Path("downloads") / request_id 35 | if not out_dir.exists(): 36 | Path(out_dir).mkdir(parents=True) 37 | 38 | # Download all available files. 39 | result.download(out_dir) 40 | print("Download finished.") 41 | print(f"\nDownload directory:\n {Path.resolve(out_dir)}\n") 42 | -------------------------------------------------------------------------------- /examples/export_print_urls.py: -------------------------------------------------------------------------------- 1 | """ 2 | ============================== 3 | Getting the urls of a download 4 | ============================== 5 | 6 | This example prints the download URLs for files returned from an 'as-is' data 7 | export request. 8 | 9 | Note that there is no "Request URL" for method 'url_quick'. 10 | """ 11 | 12 | import os 13 | 14 | import drms 15 | 16 | ############################################################################### 17 | # First we will create a `drms.Client`, using the JSOC baseurl. 18 | 19 | client = drms.Client() 20 | 21 | # This example requires a registered export email address. You can register 22 | # JSOC exports at: http://jsoc.stanford.edu/ajax/register_email.html 23 | # You must supply your own email. 24 | email = os.environ["JSOC_EMAIL"] 25 | 26 | ############################################################################### 27 | # Construct the DRMS query string: "Series[timespan][wavelength]" 28 | 29 | qstr = "hmi.ic_720s[2015.01.01_00:00:00_TAI/10d@1d]{continuum}" 30 | 31 | # Submit export request, defaults to method='url_quick' and protocol='as-is' 32 | print(f"Data export query:\n {qstr}\n") 33 | print("Submitting export request...") 34 | result = client.export(qstr, email=email) 35 | print(f"{len(result.urls)} file(s) available for download.\n") 36 | 37 | # Print download URLs. 38 | for _, row in result.urls[["record", "url"]].iterrows(): 39 | print(f"REC(ORD): {row.record}") 40 | print(f"URL: {row.url}\n") 41 | -------------------------------------------------------------------------------- /drms/tests/test_jsoc_info.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | 4 | @pytest.mark.jsoc() 5 | @pytest.mark.remote_data() 6 | @pytest.mark.parametrize( 7 | ("series", "pkeys", "segments"), 8 | [ 9 | ("hmi.v_45s", ["T_REC", "CAMERA"], ["Dopplergram"]), 10 | ("hmi.m_720s", ["T_REC", "CAMERA"], ["magnetogram"]), 11 | ("hmi.v_sht_2drls", ["LMIN", "NACOEFF"], ["split", "rot", "err"]), 12 | ], 13 | ) 14 | def test_series_info_basic(jsoc_client, series, pkeys, segments): 15 | si = jsoc_client.info(series) 16 | assert si.name.lower() == series 17 | for k in pkeys: 18 | assert k in si.primekeys 19 | assert k in si.keywords.index 20 | for s in segments: 21 | assert s in si.segments.index 22 | 23 | 24 | @pytest.mark.jsoc() 25 | @pytest.mark.remote_data() 26 | @pytest.mark.parametrize( 27 | ("series", "pkeys"), 28 | [ 29 | ("hmi.v_45s", ["T_REC", "CAMERA"]), 30 | ("hmi.m_720s", ["T_REC", "CAMERA"]), 31 | ("hmi.v_sht_2drls", ["LMIN", "NACOEFF"]), 32 | ("aia.lev1", ["T_REC", "FSN"]), 33 | ("aia.lev1_euv_12s", ["T_REC", "WAVELNTH"]), 34 | ("aia.response", ["T_START", "WAVE_STR"]), 35 | ("iris.lev1", ["T_OBS", "FSN"]), 36 | ("mdi.fd_m_lev182", ["T_REC"]), 37 | ], 38 | ) 39 | def test_series_primekeys(jsoc_client, series, pkeys): 40 | pkey_list = jsoc_client.pkeys(series) 41 | key_list = jsoc_client.keys(series) 42 | for k in pkeys: 43 | assert k in pkey_list 44 | assert k in key_list 45 | -------------------------------------------------------------------------------- /drms/CITATION.rst: -------------------------------------------------------------------------------- 1 | Acknowledging or Citing drms 2 | ============================ 3 | 4 | If you use drms in your scientific work, we would appreciate citing it in your publications. 5 | The continued growth and development of drms is dependent on the community being aware of drms. 6 | 7 | Please add the following line within your methods, conclusion or acknowledgements sections: 8 | 9 | *This research used version X.Y.Z (software citation) of the drms open source software package (project citation).* 10 | 11 | The project citation should be to the `drms paper`_, and the software citation should be the specific `Zenodo DOI`_ for the version used in your work. 12 | 13 | Here is the Bibtex entry: 14 | 15 | .. code:: bibtex 16 | 17 | @ARTICLE{Glogowski2019, 18 | doi = {10.21105/joss.01614}, 19 | url = {https://doi.org/10.21105/joss.01614}, 20 | year = {2019}, 21 | publisher = {The Open Journal}, 22 | volume = {4}, 23 | number = {40}, 24 | pages = {1614}, 25 | author = {Kolja Glogowski and Monica G. Bobra and Nitin Choudhary and Arthur B. Amezcua and Stuart J. Mumford}, 26 | title = {drms: A Python package for accessing HMI and AIA data}, 27 | journal = {Journal of Open Source Software} 28 | } 29 | 30 | You can also get this information with ``drms.__citation__``. 31 | 32 | Or as, "Glogowski et al., (2019). drms: A Python package for accessing HMI and AIA data. Journal of Open Source Software, 4(40), 1614, https://doi.org/10.21105/joss.01614." 33 | 34 | .. _drms paper: https://doi.org/10.21105/joss.01614 35 | .. _Zenodo DOI: https://doi.org/10.5281/zenodo.3369966 36 | -------------------------------------------------------------------------------- /.cruft.json: -------------------------------------------------------------------------------- 1 | { 2 | "template": "https://github.com/sunpy/package-template", 3 | "commit": "c359c134fbf9e3f11302c2019fb58ac11cf14cdf", 4 | "checkout": null, 5 | "context": { 6 | "cookiecutter": { 7 | "package_name": "drms", 8 | "module_name": "drms", 9 | "short_description": "Access HMI, AIA and MDI data from the Standford JSOC DRMS", 10 | "author_name": "The SunPy Community", 11 | "author_email": "sunpy@googlegroups.com", 12 | "project_url": "https://sunpy.org", 13 | "github_repo": "sunpy/drms", 14 | "sourcecode_url": "https://github.com/sunpy/drms", 15 | "download_url": "https://pypi.org/project/drms", 16 | "documentation_url": "https://docs.sunpy.org/projects/drms", 17 | "changelog_url": "https://docs.sunpy.org/projects/drms/en/stable/whatsnew/changelog.html", 18 | "issue_tracker_url": "https://github.com/sunpy/drms/issues", 19 | "license": "BSD 2-Clause", 20 | "minimum_python_version": "3.12", 21 | "use_compiled_extensions": "n", 22 | "enable_dynamic_dev_versions": "y", 23 | "include_example_code": "n", 24 | "include_cruft_update_github_workflow": "y", 25 | "use_extended_ruff_linting": "y", 26 | "_sphinx_theme": "sunpy", 27 | "_parent_project": "", 28 | "_install_requires": "", 29 | "_copy_without_render": [ 30 | "docs/_templates", 31 | "docs/_static", 32 | ".github/workflows/sub_package_update.yml" 33 | ], 34 | "_template": "https://github.com/sunpy/package-template", 35 | "_commit": "c359c134fbf9e3f11302c2019fb58ac11cf14cdf" 36 | } 37 | }, 38 | "directory": null 39 | } 40 | -------------------------------------------------------------------------------- /examples/plot_hmi_lightcurve.py: -------------------------------------------------------------------------------- 1 | """ 2 | ========================================= 3 | Downloading and plotting a HMI lightcurve 4 | ========================================= 5 | 6 | This example shows how to download HMI data from JSOC and make a lightcurve plot. 7 | """ 8 | 9 | import matplotlib.pyplot as plt 10 | 11 | import drms 12 | 13 | ############################################################################### 14 | # First we will create a `drms.Client`, using the JSOC baseurl. 15 | 16 | client = drms.Client() 17 | 18 | ############################################################################### 19 | # Construct the DRMS query string: "Series[timespan]" 20 | 21 | qstr = "hmi.ic_720s[2010.05.01_TAI-2016.04.01_TAI@6h]" 22 | 23 | # Send request to the DRMS server 24 | print(f"Querying keyword data...\n -> {qstr}") 25 | result = client.query(qstr, key=["T_REC", "DATAMEAN", "DATARMS"]) 26 | print(f" -> {len(result)} lines retrieved.") 27 | 28 | ############################################################################### 29 | # Now to plot the image. 30 | 31 | # Convert T_REC strings to datetime and use it as index for the series 32 | result.index = drms.to_datetime(result.pop("T_REC")) 33 | 34 | # Note: DATARMS contains the standard deviation, not the RMS! 35 | t = result.index 36 | avg = result.DATAMEAN / 1e3 37 | std = result.DATARMS / 1e3 38 | 39 | # Create plot 40 | fig, ax = plt.subplots(1, 1, figsize=(15, 7)) 41 | ax.set_title(qstr, fontsize="medium") 42 | ax.fill_between( 43 | t, 44 | avg + std, 45 | avg - std, 46 | edgecolor="none", 47 | facecolor="b", 48 | alpha=0.3, 49 | interpolate=True, 50 | ) 51 | ax.plot(t, avg, color="b") 52 | ax.set_xlabel("Time") 53 | ax.set_ylabel("Disk-averaged continuum intensity [kDN/s]") 54 | fig.tight_layout() 55 | 56 | plt.show() 57 | -------------------------------------------------------------------------------- /examples/export_as_is.py: -------------------------------------------------------------------------------- 1 | """ 2 | ========================= 3 | Exporting data as 'as-is' 4 | ========================= 5 | 6 | This example shows how to submit an 'url_quick' / 'as-is' data export request 7 | to JSOC and how to download the requested files. 8 | 9 | The export protocol 'as-is', should be preferred over other protocols, 10 | because it minimizes the server load. The only reason to use 11 | protocol='fits' is, when keywords in the FITS header are really needed. 12 | """ 13 | 14 | import os 15 | from pathlib import Path 16 | 17 | import drms 18 | 19 | ############################################################################### 20 | # First we will create a `drms.Client`, using the JSOC baseurl. 21 | 22 | client = drms.Client() 23 | 24 | # This example requires a registered export email address. You can register 25 | # JSOC exports at: http://jsoc.stanford.edu/ajax/register_email.html 26 | # You must supply your own email. 27 | email = os.environ["JSOC_EMAIL"] 28 | 29 | # Create download directory if it does not exist yet. 30 | out_dir = Path("downloads") 31 | if not out_dir.exists(): 32 | Path(out_dir).mkdir(parents=True) 33 | 34 | ############################################################################### 35 | # Construct the DRMS query string: "Series[harpnum][timespan]{data segments}" 36 | 37 | qstr = "hmi.sharp_720s[7451][2020.09.27_00:00:00_TAI]{continuum, magnetogram, field}" 38 | print(f"Data export query:\n {qstr}\n") 39 | 40 | # Submit export request, defaults to method='url_quick' and protocol='as-is' 41 | print("Submitting export request...") 42 | result = client.export(qstr, email=email) 43 | print(f"{len(result.urls)} file(s) available for download.\n") 44 | 45 | # Download selected files. 46 | result.download(out_dir) 47 | print("Download finished.") 48 | print(f'\nDownload directory:\n "{out_dir.resolve()}"\n') 49 | -------------------------------------------------------------------------------- /drms/tests/test_jsoc_email.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | import drms 4 | from drms.config import ServerConfig 5 | from drms.exceptions import DrmsOperationNotSupported 6 | 7 | # Invalid email addresses used for testing 8 | invalid_emails = [ 9 | "notregistered@example.com", 10 | "not-valid", 11 | "", 12 | ] 13 | 14 | 15 | @pytest.mark.jsoc() 16 | @pytest.mark.remote_data() 17 | @pytest.mark.parametrize("email", invalid_emails) 18 | def test_email_invalid_check(email): 19 | c = drms.Client("jsoc") 20 | assert not c.check_email(email) 21 | 22 | 23 | @pytest.mark.jsoc() 24 | @pytest.mark.remote_data() 25 | @pytest.mark.parametrize("email", invalid_emails) 26 | def test_email_invalid_set(email): 27 | c = drms.Client("jsoc") 28 | with pytest.raises(ValueError, match="Email address is invalid or not registered"): 29 | c.email = email 30 | 31 | 32 | @pytest.mark.jsoc() 33 | @pytest.mark.remote_data() 34 | @pytest.mark.parametrize("email", invalid_emails) 35 | def test_email_invalid_init(email): 36 | with pytest.raises(ValueError, match="Email address is invalid or not registered"): 37 | drms.Client("jsoc", email=email) 38 | 39 | 40 | @pytest.mark.jsoc() 41 | @pytest.mark.remote_data() 42 | def test_email_cmdopt_check(email): 43 | c = drms.Client("jsoc") 44 | assert c.check_email(email) 45 | 46 | 47 | @pytest.mark.jsoc() 48 | @pytest.mark.remote_data() 49 | def test_email_cmdopt_set(email): 50 | c = drms.Client("jsoc") 51 | c.email = email 52 | assert c.email == email 53 | 54 | 55 | @pytest.mark.jsoc() 56 | @pytest.mark.remote_data() 57 | def test_email_cmdopt_init(email): 58 | c = drms.Client("jsoc", email=email) 59 | assert c.email == email 60 | 61 | 62 | def test_query_invalid(): 63 | cfg = ServerConfig(name="TEST") 64 | with pytest.raises(DrmsOperationNotSupported): 65 | drms.Client(cfg, email="user@example.com") 66 | -------------------------------------------------------------------------------- /pytest.ini: -------------------------------------------------------------------------------- 1 | [pytest] 2 | minversion = 7.0 3 | testpaths = 4 | drms 5 | docs 6 | norecursedirs = 7 | .tox 8 | build 9 | docs/_build 10 | docs/generated 11 | *.egg-info 12 | examples 13 | drms/_dev 14 | .history 15 | drms/extern 16 | doctest_plus = enabled 17 | doctest_optionflags = 18 | NORMALIZE_WHITESPACE 19 | FLOAT_CMP 20 | ELLIPSIS 21 | text_file_format = rst 22 | addopts = 23 | --doctest-rst 24 | -p no:unraisableexception 25 | -p no:threadexception 26 | markers = 27 | remote_data: marks this test function as needing remote data. 28 | jsoc: marks the test function as needing a connection to JSOC. 29 | kis: marks the test function as needing a connection to KIS. 30 | flaky: from sunpy 31 | remote_data_strict = True 32 | log_cli=true 33 | log_level=INFO 34 | filterwarnings = 35 | # Turn all warnings into errors so they do not pass silently. 36 | error 37 | # Do not fail on pytest config issues (i.e. missing plugins) but do show them 38 | always::pytest.PytestConfigWarning 39 | # A list of warnings to ignore follows. If you add to this list, you MUST 40 | # add a comment or ideally a link to an issue that explains why the warning 41 | # is being ignored 42 | # Lmao Pandas 43 | ignore:(?s).*Pyarrow will become a required dependency of pandas:DeprecationWarning 44 | # This is due to dependencies building with a numpy version different from 45 | # the local installed numpy version, but should be fine 46 | # See https://github.com/numpy/numpy/issues/15748#issuecomment-598584838 47 | ignore:numpy.ufunc size changed:RuntimeWarning 48 | ignore:numpy.ndarray size changed:RuntimeWarning 49 | # https://github.com/pytest-dev/pytest-cov/issues/557 50 | ignore:The --rsyncdir command line argument and rsyncdirs config variable are deprecated.:DeprecationWarning 51 | ignore:.*is deprecated and slated for removal in Python 3:DeprecationWarning 52 | -------------------------------------------------------------------------------- /examples/export_fits.py: -------------------------------------------------------------------------------- 1 | """ 2 | ====================== 3 | Exporting data as fits 4 | ====================== 5 | 6 | This example shows how to submit a data export request using the 'fits' 7 | protocol and how to download the requested files. 8 | 9 | Note that the 'as-is' protocol should be used instead of 'fits', if 10 | record keywords in the FITS headers are not needed, as it greatly 11 | reduces the server load. 12 | """ 13 | 14 | import os 15 | from pathlib import Path 16 | 17 | import drms 18 | 19 | ############################################################################### 20 | # First we will create a `drms.Client`, using the JSOC baseurl. 21 | 22 | client = drms.Client() 23 | 24 | # This example requires a registered export email address. You can register 25 | # JSOC exports at: http://jsoc.stanford.edu/ajax/register_email.html 26 | # You must supply your own email. 27 | email = os.environ["JSOC_EMAIL"] 28 | 29 | # Use 'as-is' instead of 'fits', if record keywords are not needed in the 30 | # FITS header. This greatly reduces the server load! 31 | export_protocol = "fits" 32 | 33 | # Create download directory if it does not exist yet. 34 | out_dir = Path("downloads") 35 | if not out_dir.exists(): 36 | Path(out_dir).mkdir(parents=True) 37 | 38 | ############################################################################### 39 | # Construct the DRMS query string: "Series[harpnum][timespan]{data segments}" 40 | 41 | qstr = "hmi.sharp_720s[12519][2024.12.30_22:24:00_TAI/1d@8h]{continuum, magnetogram, field}" 42 | print(f"Data export query:\n {qstr}\n") 43 | 44 | # Submit export request using the 'fits' protocol 45 | print("Submitting export request...") 46 | result = client.export(qstr, method="url", protocol=export_protocol, email=email) 47 | 48 | # Print request URL. 49 | print(f"\nRequest URL: {result.request_url}") 50 | print(f"{len(result.urls)} file(s) available for download.\n") 51 | 52 | # Download selected files. 53 | result.download(out_dir) 54 | print("Download finished.") 55 | print(f'\nDownload directory:\n "{out_dir.absolute()}"\n') 56 | -------------------------------------------------------------------------------- /changelog/README.rst: -------------------------------------------------------------------------------- 1 | ========= 2 | Changelog 3 | ========= 4 | 5 | .. note:: 6 | 7 | This README was adapted from the pytest changelog readme under the terms of the MIT licence. 8 | 9 | This directory contains "news fragments" which are short files that contain a small **ReST**-formatted text that will be added to the next ``CHANGELOG``. 10 | 11 | The ``CHANGELOG`` will be read by users, so this description should be aimed at drms users instead of describing internal changes which are only relevant to the developers. 12 | 13 | Make sure to use full sentences with correct case and punctuation, for example:: 14 | 15 | Add support for Helioprojective coordinates in `sunpy.coordinates.frames`. 16 | 17 | Please try to use Sphinx intersphinx using backticks. 18 | 19 | Each file should be named like ``.[.].rst``, where ```` is a pull request number, ``COUNTER`` is an optional number if a PR needs multiple entries with the same type and ```` is one of: 20 | 21 | * ``breaking``: A change which requires users to change code and is not backwards compatible. (Not to be used for removal of deprecated features.) 22 | * ``feature``: New user facing features and any new behavior. 23 | * ``bugfix``: Fixes a reported bug. 24 | * ``doc``: Documentation addition or improvement, like rewording an entire session or adding missing docs. 25 | * ``docfix``: Correction to existing documentation, such as fixing a typo or adding a missing input parameter. 26 | * ``removal``: Feature deprecation and/or feature removal. 27 | * ``trivial``: A change which has no user facing effect or is tiny change. 28 | 29 | So for example: ``123.feature.rst``, ``456.bugfix.rst``. 30 | 31 | If you are unsure what pull request type to use, don't hesitate to ask in your PR. 32 | 33 | Note that the ``towncrier`` tool will automatically reflow your text, so it will work best if you stick to a single paragraph, but multiple sentences and links are OK and encouraged. 34 | You can install ``towncrier`` and then run ``towncrier --draft`` if you want to get a preview of how your change will look in the final release notes. 35 | -------------------------------------------------------------------------------- /drms/tests/test_utils.py: -------------------------------------------------------------------------------- 1 | import pandas as pd 2 | import pytest 3 | 4 | from drms.utils import _extract_series_name, _pd_to_datetime_coerce, _pd_to_numeric_coerce, _split_arg 5 | 6 | 7 | @pytest.mark.parametrize( 8 | ("in_obj", "expected"), 9 | [ 10 | ("", []), 11 | ("asd", ["asd"]), 12 | ("aa,bb,cc", ["aa", "bb", "cc"]), 13 | ("aa, bb, cc", ["aa", "bb", "cc"]), 14 | (" aa,bb, cc, dd", ["aa", "bb", "cc", "dd"]), 15 | ("aa,\tbb,cc, dd ", ["aa", "bb", "cc", "dd"]), 16 | ([], []), 17 | (["a", "b", "c"], ["a", "b", "c"]), 18 | (("a", "b", "c"), ["a", "b", "c"]), 19 | ], 20 | ) 21 | def test_split_arg(in_obj, expected): 22 | res = _split_arg(in_obj) 23 | assert len(res) == len(expected) 24 | for i in range(len(res)): 25 | assert res[i] == expected[i] 26 | 27 | 28 | @pytest.mark.parametrize( 29 | ("ds_string", "expected"), 30 | [ 31 | ("hmi.v_45s", "hmi.v_45s"), 32 | ("hmi.v_45s[2010.05.01_TAI]", "hmi.v_45s"), 33 | ("hmi.v_45s[2010.05.01_TAI/365d@1d]", "hmi.v_45s"), 34 | ("hmi.v_45s[2010.05.01_TAI/365d@1d][?QUALITY>=0?]", "hmi.v_45s"), 35 | ("hmi.v_45s[2010.05.01_TAI/1d@6h]{Dopplergram}", "hmi.v_45s"), 36 | ], 37 | ) 38 | def test_extract_series(ds_string, expected): 39 | assert _extract_series_name(ds_string) == expected 40 | 41 | 42 | @pytest.mark.parametrize( 43 | ("arg", "exp"), 44 | [ 45 | (pd.Series(["1.0", "2", -3]), pd.Series([1.0, 2.0, -3.0])), 46 | (pd.Series(["1.0", "apple", -3]), pd.Series([1.0, float("nan"), -3.0])), 47 | ], 48 | ) 49 | def test_pd_to_numeric_coerce(arg, exp): 50 | assert _pd_to_numeric_coerce(arg).equals(exp) 51 | 52 | 53 | @pytest.mark.parametrize( 54 | ("arg", "exp"), 55 | [ 56 | ( 57 | pd.Series(["2016-04-01 00:00:00", "2016-04-01 06:00:00"]), 58 | pd.Series([pd.Timestamp("2016-04-01 00:00:00"), pd.Timestamp("2016-04-01 06:00:00")]), 59 | ), 60 | ], 61 | ) 62 | def test_pd_to_datetime_coerce(arg, exp): 63 | assert _pd_to_datetime_coerce(arg).equals(exp) 64 | -------------------------------------------------------------------------------- /examples/export_jpg.py: -------------------------------------------------------------------------------- 1 | """ 2 | =============== 3 | Exporting JPEGs 4 | =============== 5 | 6 | This example shows how to export image data as JPEG file, using the 'jpg' 7 | protocol. 8 | 9 | The 'jpg' protocol accepts additional protocol arguments, like color 10 | table, color scaling or pixel binning. For a list of available color 11 | tables, see http://jsoc.stanford.edu/ajax/exportdata.html and select the 12 | JPEG protocol. 13 | """ 14 | 15 | import os 16 | from pathlib import Path 17 | 18 | import drms 19 | 20 | ############################################################################### 21 | # First we will create a `drms.Client`, using the JSOC baseurl. 22 | 23 | client = drms.Client() 24 | 25 | # This example requires a registered export email address. You can register 26 | # JSOC exports at: http://jsoc.stanford.edu/ajax/register_email.html 27 | # You must supply your own email. 28 | email = os.environ["JSOC_EMAIL"] 29 | 30 | # Arguments for 'jpg' protocol 31 | jpg_args = { 32 | "ct": "aia_304.lut", # color table 33 | "min": 4, # min value 34 | "max": 800, # max value 35 | "scaling": "log", # color scaling 36 | "size": 2, # binning (1 -> 4k, 2 -> 2k, 4 -> 1k) 37 | } 38 | 39 | # Create download directory if it does not exist yet. 40 | out_dir = Path("downloads") 41 | if not out_dir.exists(): 42 | Path(out_dir).mkdir(parents=True) 43 | 44 | ############################################################################### 45 | # Construct the DRMS query string: "Series[timespan][wavelength]{data segments}" 46 | 47 | qstr = "aia.lev1_euv_12s[2012-08-31T19:48:01Z][304]{image}" 48 | print(f"Data export query:\n {qstr}\n") 49 | 50 | # Submit export request using the 'jpg' protocol with custom protocol_args 51 | print("Submitting export request...") 52 | result = client.export(qstr, protocol="jpg", protocol_args=jpg_args, email=email) 53 | 54 | # Print request URL. 55 | print(f"\nRequest URL: {result.request_url}") 56 | print(f"{len(result.urls)} file(s) available for download.\n") 57 | 58 | # Download selected files. 59 | result.download(out_dir) 60 | print("Download finished.") 61 | print(f"\nDownload directory:\n {out_dir.resolve()}\n") 62 | -------------------------------------------------------------------------------- /examples/export_tar.py: -------------------------------------------------------------------------------- 1 | """ 2 | ==================================== 3 | Downloading data as a tar collection 4 | ==================================== 5 | 6 | This example shows how to submit a data export request using the 'url-tar' 7 | method, which provides a single TAR archive containing all requested files. 8 | 9 | Here we use this method to download data from the 10 | 'hmi.rdvflows_fd15_frame' series, which stores directories of text files 11 | for each record. This is currently the only way to download directory 12 | data segments using the Python DRMS client. The export protocol in this 13 | case is 'as-is'. You might change the protocol to 'fits', if you are 14 | downloading FITS files instead of text files. 15 | """ 16 | 17 | import os 18 | from pathlib import Path 19 | 20 | import drms 21 | 22 | ############################################################################### 23 | # First we will create a `drms.Client`, using the JSOC baseurl. 24 | 25 | client = drms.Client() 26 | 27 | # This example requires a registered export email address. You can register 28 | # JSOC exports at: http://jsoc.stanford.edu/ajax/register_email.html 29 | # You must supply your own email. 30 | email = os.environ["JSOC_EMAIL"] 31 | 32 | # Create download directory if it does not exist yet. 33 | out_dir = Path("downloads") 34 | if not out_dir.exists(): 35 | Path(out_dir).mkdir(parents=True) 36 | 37 | ############################################################################### 38 | # Construct the DRMS query string: "Series[Carrington rotation][Carrington longitude]{data segments}" 39 | 40 | qstr = "hmi.rdvflows_fd15_frame[2150][360]{Ux, Uy}" 41 | print(f"Data export query:\n {qstr}\n") 42 | 43 | # Submit export request using the 'url-tar' method, protocol default: 'as-is' 44 | print("Submitting export request...") 45 | result = client.export(qstr, method="url-tar", email=email) 46 | 47 | # Print request URL. 48 | print(f"\nRequest URL: {result.request_url}") 49 | print(f"{len(result.urls)} file(s) available for download.\n") 50 | 51 | # Download selected files. 52 | dr = result.download(out_dir) 53 | print("Download finished.") 54 | print(f'\nDownloaded file:\n "{dr.download[0]}"\n') 55 | -------------------------------------------------------------------------------- /docs/intro.rst: -------------------------------------------------------------------------------- 1 | .. _drms-introduction: 2 | 3 | ************ 4 | Introduction 5 | ************ 6 | 7 | The ``drms`` Python package can be used to access HMI, AIA and MDI data which are stored in a DRMS database system. 8 | 9 | DRMS stands for *Data Record Management System* and is a system that was developed by the `Joint Science Operation Center `__ (JSOC), headquartered at Stanford University, to handle the data produced by the AIA and HMI instruments aboard the `Solar Dynamics Observatory `__ spacecraft. 10 | 11 | By default the ``drms`` library uses the HTTP/JSON interface provided by JSOC and has similar functionality to the `JSOC Lookdata `__ website. 12 | It can be used to query metadata, submit data export requests and download data files. 13 | 14 | This module also works well for local `NetDRMS `__ sites, as long as the site runs a web server providing the needed CGI programs ``show_series`` and ``jsoc_info`` (for the data export functionality, additional CGIs, like ``jsoc_fetch``, are needed). 15 | 16 | Installation 17 | ============ 18 | 19 | If you are using `miniforge`_ (which is conda but using the conda-forge channel): 20 | 21 | .. code-block:: bash 22 | 23 | conda install drms 24 | 25 | Otherwise the ``drms`` Python package can be installed from `PyPI`_ using: 26 | 27 | .. code-block:: bash 28 | 29 | pip install drms 30 | 31 | .. note:: 32 | If you do not use a Python distribution, like `miniforge`_, 33 | and did not create an isolated Python environment using `Virtualenv`_, 34 | you might need to add ``--user`` to the ``pip`` command: 35 | 36 | .. code-block:: bash 37 | 38 | pip install --user drms 39 | 40 | .. _PyPI: https://pypi.org/project/drms/ 41 | .. _conda-forge: https://anaconda.org/conda-forge/drms 42 | .. _miniforge: https://github.com/conda-forge/miniforge#miniforge3 43 | .. _Virtualenv: https://virtualenv.pypa.io/en/latest/ 44 | 45 | Acknowledgements 46 | ================ 47 | 48 | Kolja Glogowski has received funding from the European Research Council under the European Union's Seventh Framework Programme (FP/2007-2013) / ERC Grant Agreement no. 307117. 49 | -------------------------------------------------------------------------------- /.ruff.toml: -------------------------------------------------------------------------------- 1 | target-version = "py310" 2 | line-length = 120 3 | exclude = [ 4 | ".git,", 5 | "__pycache__", 6 | "build", 7 | "drms/version.py", 8 | ] 9 | 10 | [lint] 11 | select = [ 12 | "E", 13 | "F", 14 | "W", 15 | "UP", 16 | "PT", 17 | "BLE", 18 | "A", 19 | "C4", 20 | "INP", 21 | "PIE", 22 | "T20", 23 | "RET", 24 | "TID", 25 | "PTH", 26 | "PD", 27 | "PLC", 28 | "PLE", 29 | "FLY", 30 | "NPY", 31 | "PERF", 32 | "RUF", 33 | ] 34 | extend-ignore = [ 35 | # pycodestyle (E, W) 36 | "E501", # ignore line length will use a formatter instead 37 | # pytest (PT) 38 | "PT001", # Always use pytest.fixture() 39 | "PT023", # Always use () on pytest decorators 40 | # flake8-pie (PIE) 41 | "PIE808", # Disallow passing 0 as the first argument to range 42 | # flake8-use-pathlib (PTH) 43 | "PTH123", # open() should be replaced by Path.open() 44 | # Ruff (RUF) 45 | "RUF003", # Ignore ambiguous quote marks, doesn't allow ' in comments 46 | "RUF012", # Mutable class attributes should be annotated with `typing.ClassVar` 47 | "RUF013", # PEP 484 prohibits implicit `Optional` 48 | "RUF015", # Prefer `next(iter(...))` over single element slice 49 | ] 50 | 51 | [lint.per-file-ignores] 52 | "setup.py" = [ 53 | "INP001", # File is part of an implicit namespace package. 54 | ] 55 | "conftest.py" = [ 56 | "INP001", # File is part of an implicit namespace package. 57 | ] 58 | "docs/conf.py" = [ 59 | "E402" # Module imports not at top of file 60 | ] 61 | "docs/*.py" = [ 62 | "INP001", # File is part of an implicit namespace package. 63 | ] 64 | "examples/**.py" = [ 65 | "T201", # allow use of print in examples 66 | "INP001", # File is part of an implicit namespace package. 67 | ] 68 | "__init__.py" = [ 69 | "E402", # Module level import not at top of cell 70 | "F401", # Unused import 71 | "F403", # from {name} import * used; unable to detect undefined names 72 | "F405", # {name} may be undefined, or defined from star imports 73 | ] 74 | "test_*.py" = [ 75 | "E402", # Module level import not at top of cell 76 | ] 77 | "examples/plot_hmi_modes.py" = [ 78 | "E741", # Ambiguous variable name 79 | ] 80 | "drms/json.py" = [ 81 | "A005", # Module `json` shadows a Python standard-library module 82 | ] 83 | 84 | [lint.pydocstyle] 85 | convention = "numpy" 86 | -------------------------------------------------------------------------------- /examples/export_movie.py: -------------------------------------------------------------------------------- 1 | """ 2 | ================= 3 | Exporting a movie 4 | ================= 5 | 6 | This example shows how to export movies from image data, using the 'mp4' 7 | protocol. 8 | 9 | The 'mp4' protocol accepts additional protocol arguments, like color 10 | table, color scaling or pixel binning. For a list of available color 11 | tables, see http://jsoc.stanford.edu/ajax/exportdata.html and select the 12 | MP4 protocol. 13 | """ 14 | 15 | import os 16 | from pathlib import Path 17 | 18 | import drms 19 | 20 | ############################################################################### 21 | # First we will create a `drms.Client`, using the JSOC baseurl. 22 | 23 | client = drms.Client() 24 | 25 | # This example requires a registered export email address. You can register 26 | # JSOC exports at: http://jsoc.stanford.edu/ajax/register_email.html 27 | # You must supply your own email. 28 | email = os.environ["JSOC_EMAIL"] 29 | 30 | # Create download directory if it does not exist yet. 31 | out_dir = Path("downloads") 32 | if not out_dir.exists(): 33 | Path(out_dir).mkdir(parents=True) 34 | 35 | ############################################################################### 36 | # Construct the DRMS query string: "Series[timespan]{segment}" 37 | 38 | qstr = "hmi.M_720s[2025.01.29_23:12:00_TAI/5d@1h]{magnetogram}" 39 | print(f"Data export query:\n {qstr}\n") 40 | 41 | ############################################################################### 42 | # Arguments for 'mp4' protocol 43 | 44 | mp4_args = { 45 | "ct": "grey.sao", # color table 46 | "min": -1500, # min value 47 | "max": 1500, # max value 48 | "scaling": "mag", # color scaling 49 | "size": 8, # binning (1 -> 4k, 2 -> 2k, 4 -> 1k, 8 -> 512) 50 | } 51 | 52 | # Submit export request using the 'mp4' protocol with custom protocol_args 53 | print("Submitting export request...") 54 | result = client.export(qstr, protocol="mp4", protocol_args=mp4_args, email=email) 55 | result.wait(sleep=10) 56 | 57 | # Print request URL. 58 | print(f"\nRequest URL: {result.request_url}") 59 | print(f"{len(result.urls)} file(s) available for download.\n") 60 | 61 | # Download movie file only: index=0 62 | result.download(out_dir, index=0) 63 | print("Download finished.") 64 | print(f"\nDownload directory:\n {out_dir.resolve()}\n") 65 | -------------------------------------------------------------------------------- /drms/__init__.py: -------------------------------------------------------------------------------- 1 | """ 2 | drms 3 | ==== 4 | 5 | The ``drms`` library provides an easy-to-use interface for accessing HMI, AIA and MDI data with Python. 6 | It uses the publicly accessible JSOC DRMS server by default, but can also be used with local NetDRMS sites. 7 | More information, including a detailed tutorial, is available in the Documentation. 8 | 9 | * Homepage: https://github.com/sunpy/drms 10 | * Documentation: https://docs.sunpy.org/projects/drms/en/stable/ 11 | """ 12 | 13 | import logging 14 | from pathlib import Path 15 | 16 | logger = logging.getLogger(__name__) 17 | logging.basicConfig( 18 | level=logging.INFO, 19 | format="%(asctime)s - %(name)s - %(levelname)s: %(message)s", 20 | datefmt="%Y-%m-%d %H:%M:%S", 21 | ) 22 | 23 | from .client import Client, ExportRequest, SeriesInfo 24 | from .config import ServerConfig, register_server 25 | from .exceptions import DrmsError, DrmsExportError, DrmsOperationNotSupported, DrmsQueryError 26 | from .json import HttpJsonClient, HttpJsonRequest, JsocInfoConstants 27 | from .utils import to_datetime 28 | from .version import version as __version__ 29 | 30 | 31 | def _get_bibtex(): 32 | import textwrap # noqa: PLC0415 33 | 34 | # Set the bibtex entry to the article referenced in CITATION.rst 35 | citation_file = Path(__file__).parent / "CITATION.rst" 36 | # Explicitly specify UTF-8 encoding in case the system's default encoding is problematic 37 | with citation_file.open(encoding="utf-8") as citation: 38 | # Extract the first bibtex block: 39 | ref = citation.read().partition(".. code:: bibtex\n\n")[2] 40 | lines = ref.split("\n") 41 | # Only read the lines which are indented 42 | lines = lines[: [line.startswith(" ") for line in lines].index(False)] 43 | return textwrap.dedent("\n".join(lines)) 44 | 45 | 46 | __citation__ = __bibtex__ = _get_bibtex() 47 | 48 | __all__ = [ 49 | "Client", 50 | "DrmsError", 51 | "DrmsExportError", 52 | "DrmsOperationNotSupported", 53 | "DrmsQueryError", 54 | "ExportRequest", 55 | "HttpJsonClient", 56 | "HttpJsonRequest", 57 | "JsocInfoConstants", 58 | "SeriesInfo", 59 | "ServerConfig", 60 | "__bibtex__", 61 | "__citation__", 62 | "__version__", 63 | "logger", 64 | "register_server", 65 | "to_datetime", 66 | ] 67 | -------------------------------------------------------------------------------- /drms/tests/test_kis_basic.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | import drms 4 | 5 | 6 | @pytest.mark.kis() 7 | @pytest.mark.remote_data() 8 | def test_series_list_all(kis_client): 9 | slist = kis_client.series() 10 | assert isinstance(slist, list) 11 | assert "hmi.v_45s" in (s.lower() for s in slist) 12 | assert "hmi.m_720s" in (s.lower() for s in slist) 13 | assert "hmi.ic_720s" in (s.lower() for s in slist) 14 | assert "mdi.fd_v" in (s.lower() for s in slist) 15 | 16 | 17 | @pytest.mark.kis() 18 | @pytest.mark.remote_data() 19 | @pytest.mark.parametrize("schema", ["hmi", "mdi"]) 20 | def test_series_list_schemata(kis_client, schema): 21 | regex = rf"{schema}\." 22 | slist = kis_client.series(regex=regex) 23 | assert len(slist) > 0 24 | for sname in slist: 25 | assert sname.startswith(f"{schema}.") 26 | 27 | 28 | @pytest.mark.kis() 29 | @pytest.mark.remote_data() 30 | @pytest.mark.parametrize( 31 | ("series", "pkeys", "segments"), 32 | [ 33 | ("hmi.v_45s", ["T_REC", "CAMERA"], ["Dopplergram"]), 34 | ("hmi.m_720s", ["T_REC", "CAMERA"], ["magnetogram"]), 35 | ("hmi.v_sht_2drls", ["LMIN", "NACOEFF"], ["split", "rot", "err"]), 36 | ], 37 | ) 38 | def test_series_info_basic(kis_client, series, pkeys, segments): 39 | si = kis_client.info(series) 40 | assert si.name.lower() == series 41 | for k in pkeys: 42 | assert k in si.primekeys 43 | assert k in si.keywords.index 44 | for s in segments: 45 | assert s in si.segments.index 46 | 47 | 48 | @pytest.mark.kis() 49 | @pytest.mark.remote_data() 50 | def test_query_basic(kis_client): 51 | keys, segs = kis_client.query("hmi.v_45s[2013.07.03_08:42_TAI/3m]", key="T_REC, CRLT_OBS", seg="Dopplergram") 52 | assert len(keys) == 4 53 | for k in ["T_REC", "CRLT_OBS"]: 54 | assert k in keys.columns 55 | assert len(segs) == 4 56 | assert "Dopplergram" in segs.columns 57 | assert ((keys.CRLT_OBS - 3.14159).abs() < 0.0001).all() 58 | 59 | 60 | @pytest.mark.kis() 61 | @pytest.mark.remote_data() 62 | def test_not_supported_email(kis_client): 63 | with pytest.raises(drms.DrmsOperationNotSupported): 64 | kis_client.email = "name@example.com" 65 | 66 | 67 | @pytest.mark.kis() 68 | @pytest.mark.remote_data() 69 | def test_not_supported_export(kis_client): 70 | with pytest.raises(drms.DrmsOperationNotSupported): 71 | kis_client.export("hmi.v_45s[2010.05.01_TAI]") 72 | with pytest.raises(drms.DrmsOperationNotSupported): 73 | kis_client.export_from_id("KIS_20120101_123") 74 | -------------------------------------------------------------------------------- /drms/tests/conftest.py: -------------------------------------------------------------------------------- 1 | import os 2 | from urllib.error import URLError, HTTPError 3 | from urllib.request import urlopen 4 | 5 | import pytest 6 | 7 | import drms 8 | from drms.utils import create_request_with_header 9 | 10 | # Test URLs, used to check if a online site is reachable 11 | jsoc_testurl = "http://jsoc.stanford.edu/" 12 | kis_testurl = "http://drms.leibniz-kis.de/" 13 | 14 | 15 | class lazily_cached: 16 | """ 17 | Lazily evaluated function call with cached result. 18 | """ 19 | 20 | def __init__(self, f, *args, **kwargs): 21 | self.func = lambda: f(*args, **kwargs) 22 | 23 | def __call__(self): 24 | if not hasattr(self, "result"): 25 | self.result = self.func() 26 | return self.result 27 | 28 | 29 | def site_reachable(url, timeout=60): 30 | """ 31 | Checks if the given URL is accessible. 32 | """ 33 | try: 34 | urlopen(create_request_with_header(url), timeout=timeout) 35 | except (URLError, HTTPError): 36 | return False 37 | return True 38 | 39 | 40 | # Create lazily evaluated, cached site checks for JSOC and KIS. 41 | jsoc_reachable = lazily_cached(site_reachable, jsoc_testurl) 42 | kis_reachable = lazily_cached(site_reachable, kis_testurl) 43 | 44 | 45 | def pytest_runtest_setup(item): 46 | # Skip JSOC online site tests if the site is not reachable. 47 | if item.get_closest_marker("jsoc") is not None: 48 | if not jsoc_reachable(): 49 | pytest.skip("JSOC is not reachable") 50 | 51 | # Skip KIS online site tests if the site is not reachable. 52 | if item.get_closest_marker("kis") is not None: 53 | if not kis_reachable(): 54 | pytest.skip("KIS is not reachable") 55 | 56 | 57 | @pytest.fixture() 58 | def email(): 59 | """ 60 | Email address for tests. 61 | """ 62 | email = os.environ.get("JSOC_EMAIL", None) 63 | if email is None: 64 | pytest.skip("No email address specified; use the JSOC_EMAIL environmental variable to enable export tests") 65 | return email 66 | 67 | 68 | @pytest.fixture() 69 | def jsoc_client(): 70 | """ 71 | Client fixture for JSOC online tests, does not use email. 72 | """ 73 | return drms.Client("jsoc") 74 | 75 | 76 | @pytest.fixture() 77 | def jsoc_client_export(email): 78 | """ 79 | Client fixture for JSOC online tests, uses email if specified. 80 | """ 81 | return drms.Client("jsoc", email=email) 82 | 83 | 84 | @pytest.fixture() 85 | def kis_client(): 86 | """ 87 | Client fixture for KIS online tests. 88 | """ 89 | return drms.Client("kis") 90 | -------------------------------------------------------------------------------- /examples/plot_aia_lightcurve.py: -------------------------------------------------------------------------------- 1 | """ 2 | ========================================= 3 | Downloading and plotting a AIA lightcurve 4 | ========================================= 5 | 6 | This example shows how to download AIA data from JSOC and make a lightcurve plot. 7 | """ 8 | 9 | import matplotlib.pyplot as plt 10 | 11 | import drms 12 | 13 | ############################################################################### 14 | # First we will create a `drms.Client`, using the JSOC baseurl. 15 | 16 | client = drms.Client() 17 | 18 | ############################################################################### 19 | # Some keywords we are interested in; you can use client.keys(series) to get a 20 | # list of all available keywords of a series. 21 | 22 | keys = [ 23 | "T_REC", 24 | "T_OBS", 25 | "DATAMIN", 26 | "DATAMAX", 27 | "DATAMEAN", 28 | "DATARMS", 29 | "DATASKEW", 30 | "DATAKURT", 31 | "QUALITY", 32 | ] 33 | 34 | ############################################################################### 35 | # Get detailed information about the series. Some keywords from 36 | # aia.lev1_euv_12s are links to keywords in aia.lev1 and unfortunately some 37 | # entries (like note) are missing for linked keywords, so we are using the 38 | # entries from aia.lev1 in this case. 39 | 40 | print("Querying series info...") 41 | series_info = client.info("aia.lev1_euv_12s") 42 | series_info_lev1 = client.info("aia.lev1") 43 | for key in keys: 44 | linkinfo = series_info.keywords.loc[key].linkinfo 45 | if linkinfo is not None and linkinfo.startswith("lev1->"): 46 | note_str = series_info_lev1.keywords.loc[key].note 47 | else: 48 | note_str = series_info.keywords.loc[key].note 49 | print(f"{key:>10} : {note_str}") 50 | 51 | ############################################################################### 52 | # Construct the DRMS query string: "Series[timespan][wavelength]" 53 | 54 | qstr = "aia.lev1_euv_12s[2014-01-01T00:00:01Z/365d@1d][335]" 55 | 56 | # Get keyword values for the selected timespan and wavelength 57 | print(f"Querying keyword data...\n -> {qstr}") 58 | result = client.query(qstr, key=keys) 59 | print(f" -> {len(result)} lines retrieved.") 60 | 61 | # Only use entries with QUALITY==0 62 | result = result[result.QUALITY == 0] 63 | print(f" -> {len(result)} lines after QUALITY selection.") 64 | 65 | # Convert T_REC strings to datetime and use it as index for the series 66 | result.index = drms.to_datetime(result.T_REC) 67 | 68 | ############################################################################### 69 | # Create some simple plots 70 | 71 | ax = result[["DATAMIN", "DATAMAX", "DATAMEAN", "DATARMS", "DATASKEW"]].plot(figsize=(8, 10), subplots=True) 72 | ax[0].set_title(qstr, fontsize="medium") 73 | plt.tight_layout() 74 | plt.show() 75 | -------------------------------------------------------------------------------- /examples/plot_synoptic_mr.py: -------------------------------------------------------------------------------- 1 | """ 2 | ============================================ 3 | Downloading and plotting a HMI synoptic data 4 | ============================================ 5 | 6 | This example shows how to download HMI synoptic data from JSOC and make a plot. 7 | """ 8 | 9 | import matplotlib.pyplot as plt 10 | 11 | from astropy.io import fits 12 | 13 | import drms 14 | 15 | ############################################################################### 16 | # First we will create a `drms.Client`, using the JSOC baseurl. 17 | 18 | client = drms.Client() 19 | 20 | ############################################################################### 21 | # Construct the DRMS query string: "Series[Carrington rotation]" 22 | 23 | qstr = "hmi.synoptic_mr_720s[2150]" 24 | 25 | # Send request to the DRMS server 26 | print(f"Querying keyword data...\n -> {qstr}") 27 | segname = "synopMr" 28 | results, filenames = client.query(qstr, key=drms.JsocInfoConstants.all, seg=segname) 29 | print(f" -> {len(results)} lines retrieved.") 30 | 31 | # Use only the first line of the query result 32 | results = results.iloc[0] 33 | fname = f"http://jsoc.stanford.edu{filenames[segname][0]}" 34 | 35 | # Read the data segment 36 | # Note: HTTP downloads get cached in ~/.astropy/cache/downloads 37 | print(f"Reading data from {fname}...") 38 | a = fits.getdata(fname) 39 | ny, nx = a.shape 40 | 41 | ############################################################################### 42 | # Now to plot the image. 43 | 44 | # Convert pixel to world coordinates using WCS keywords 45 | xmin = (1 - results.CRPIX1) * results.CDELT1 + results.CRVAL1 46 | xmax = (nx - results.CRPIX1) * results.CDELT1 + results.CRVAL1 47 | ymin = (1 - results.CRPIX2) * results.CDELT2 + results.CRVAL2 48 | ymax = (ny - results.CRPIX2) * results.CDELT2 + results.CRVAL2 49 | 50 | # Convert to Carrington longitude 51 | xmin = results.LON_LAST - xmin 52 | xmax = results.LON_LAST - xmax 53 | 54 | # Compute the plot extent used with imshow 55 | extent = ( 56 | xmin - abs(results.CDELT1) / 2, 57 | xmax + abs(results.CDELT1) / 2, 58 | ymin - abs(results.CDELT2) / 2, 59 | ymax + abs(results.CDELT2) / 2, 60 | ) 61 | 62 | # Aspect ratio for imshow in respect to the extent computed above 63 | aspect = abs((xmax - xmin) / nx * ny / (ymax - ymin)) 64 | 65 | # Create plot 66 | fig, ax = plt.subplots(1, 1, figsize=(13.5, 6)) 67 | ax.set_title(f"{qstr}, Time: {results.T_START} ... {results.T_STOP}", fontsize="medium") 68 | ax.imshow( 69 | a, 70 | vmin=-300, 71 | vmax=300, 72 | origin="lower", 73 | interpolation="nearest", 74 | cmap="gray", 75 | extent=extent, 76 | aspect=aspect, 77 | ) 78 | ax.invert_xaxis() 79 | ax.set_xlabel("Carrington longitude") 80 | ax.set_ylabel("Sine latitude") 81 | fig.tight_layout() 82 | 83 | plt.show() 84 | -------------------------------------------------------------------------------- /examples/plot_polarfield.py: -------------------------------------------------------------------------------- 1 | """ 2 | ================================================ 3 | Downloading and plotting a HMI polar field data 4 | ================================================ 5 | 6 | This example shows how to download HMI polar field data from JSOC and make a plot. 7 | """ 8 | 9 | import matplotlib.pyplot as plt 10 | import numpy as np 11 | 12 | import drms 13 | 14 | ############################################################################### 15 | # First we will create a `drms.Client`, using the JSOC baseurl. 16 | 17 | client = drms.Client() 18 | 19 | ############################################################################### 20 | # Construct the DRMS query string: "Series[timespan][wavelength]" 21 | 22 | qstr = "hmi.meanpf_720s[2010.05.01_TAI-2016.04.01_TAI@12h]" 23 | 24 | # Send request to the DRMS server 25 | print(f"Querying keyword data...\n -> {qstr}") 26 | result = client.query(qstr, key=["T_REC", "CAPN2", "CAPS2"]) 27 | print(f" -> {len(result)} lines retrieved.") 28 | 29 | # Convert T_REC strings to datetime and use it as index for the series 30 | result.index = drms.to_datetime(result.pop("T_REC")) 31 | 32 | # Determine smallest timestep 33 | dt = np.diff(result.index.to_pydatetime()).min() 34 | 35 | # Make sure the time series contains all time steps (fills gaps with NaNs) 36 | a = result.asfreq(dt) 37 | 38 | ############################################################################### 39 | # Plot the magnetic field values. 40 | 41 | # Compute 30d moving average and standard deviation using a boxcar window 42 | win_size = int(30 * 24 * 3600 / dt.total_seconds()) 43 | a_avg = a.rolling(win_size, min_periods=1, center=True).mean() 44 | a_std = a.rolling(win_size, min_periods=1, center=True).std() 45 | 46 | # Plot results 47 | t = a.index.to_pydatetime() 48 | n, mn, sn = a.CAPN2, a_avg.CAPN2, a_std.CAPN2 49 | s, ms, ss = a.CAPS2, a_avg.CAPS2, a_std.CAPS2 50 | 51 | # Plot smoothed data 52 | fig, ax = plt.subplots(1, 1, figsize=(15, 7)) 53 | ax.set_title(qstr, fontsize="medium") 54 | ax.plot(t, n, "b", alpha=0.5, label="North pole") 55 | ax.plot(t, s, "g", alpha=0.5, label="South pole") 56 | ax.plot(t, mn, "r", label="Moving average") 57 | ax.plot(t, ms, "r", label="") 58 | ax.set_xlabel("Time") 59 | ax.set_ylabel("Mean radial field strength [G]") 60 | ax.legend() 61 | fig.tight_layout() 62 | 63 | # Plot raw data 64 | fig, ax = plt.subplots(1, 1, figsize=(15, 7)) 65 | ax.set_title(qstr, fontsize="medium") 66 | ax.fill_between(t, mn - sn, mn + sn, edgecolor="none", facecolor="b", alpha=0.3, interpolate=True) 67 | ax.fill_between(t, ms - ss, ms + ss, edgecolor="none", facecolor="g", alpha=0.3, interpolate=True) 68 | ax.plot(t, mn, "b", label="North pole") 69 | ax.plot(t, ms, "g", label="South pole") 70 | ax.set_xlabel("Time") 71 | ax.set_ylabel("Mean radial field strength [G]") 72 | ax.legend() 73 | fig.tight_layout() 74 | 75 | plt.show() 76 | -------------------------------------------------------------------------------- /examples/cutout_export_request.py: -------------------------------------------------------------------------------- 1 | """ 2 | =================================== 3 | Exporting data with cutout requests 4 | =================================== 5 | 6 | This example shows how to submit a data export request with a cutout request using ``im_patch``. 7 | """ 8 | 9 | import os 10 | from pathlib import Path 11 | 12 | import drms 13 | 14 | ############################################################################### 15 | # First we will create a `drms.Client`, using the JSOC baseurl. 16 | 17 | client = drms.Client() 18 | 19 | ############################################################################### 20 | # This example requires a registered export email address You can register 21 | # JSOC exports at: http://jsoc.stanford.edu/ajax/register_email.html 22 | # You must supply your own email. 23 | 24 | email = os.environ["JSOC_EMAIL"] 25 | 26 | # Create download directory if it does not exist yet. 27 | out_dir = Path("downloads") 28 | if not out_dir.exists(): 29 | Path(out_dir).mkdir(parents=True) 30 | 31 | ############################################################################### 32 | # Construct the DRMS query string: ``"Series[timespan][wavelength]{data segments}"`` 33 | 34 | qstr = "aia.lev1_euv_12s[2025-01-01T04:33:30.000/1m@12s][171]{image}" 35 | print(f"Data export query:\n {qstr}\n") 36 | 37 | ############################################################################### 38 | # Construct the dictionary specifying that we want to request a cutout. 39 | # This is done via the ``im_patch`` command. 40 | # We request a 345.6 arcsecond cutout (on both sides) centered on the coordinate (-517.2, -246) arcseconds as defined in the helioprojective frame of SDO at time ``t_ref``. 41 | # The ``t`` controls whether tracking is disabled (``1``) or enabled (``0``). 42 | # ``r`` controls the use of sub-pixel registration. 43 | # ``c`` controls whether off-limb pixels are filled with NaNs. 44 | # For additional details about ``im_patch``, see the `documentation `_. 45 | process = { 46 | "im_patch": { 47 | "t_ref": "2025-01-01T04:33:30.000", 48 | "t": 0, 49 | "r": 0, 50 | "c": 0, 51 | "locunits": "arcsec", 52 | "boxunits": "arcsec", 53 | "x": -517.2, 54 | "y": -246, 55 | "width": 345.6, 56 | "height": 345.6, 57 | }, 58 | } 59 | 60 | # Submit export request using the 'fits' protocol 61 | print("Submitting export request...") 62 | result = client.export( 63 | qstr, 64 | method="url", 65 | protocol="fits", 66 | email=email, 67 | process=process, 68 | ) 69 | 70 | # Print request URL. 71 | print(f"\nRequest URL: {result.request_url}") 72 | print(f"{len(result.urls)} file(s) available for download.\n") 73 | 74 | # Download selected files. 75 | result.wait() 76 | result.download(out_dir) 77 | print("Download finished.") 78 | print(f'\nDownload directory:\n "{out_dir.resolve()}"\n') 79 | -------------------------------------------------------------------------------- /drms/utils.py: -------------------------------------------------------------------------------- 1 | import re 2 | import sys 3 | from urllib.request import Request 4 | 5 | import numpy as np 6 | import pandas as pd 7 | from packaging.version import Version 8 | 9 | import drms 10 | 11 | __all__ = ["create_request_with_header", "to_datetime"] 12 | 13 | PD_VERSION = Version(pd.__version__) 14 | 15 | 16 | def create_request_with_header(url): 17 | request = Request(url) 18 | request.add_header("User-Agent", f"drms/{drms.__version__}, python/{sys.version[:5]}") 19 | return request 20 | 21 | 22 | def _pd_to_datetime_coerce(arg): 23 | if PD_VERSION >= Version("2.0.0"): 24 | return pd.to_datetime(arg, errors="coerce", format="mixed", dayfirst=False) 25 | return pd.to_datetime(arg, errors="coerce") 26 | 27 | 28 | def _pd_to_numeric_coerce(arg): 29 | return pd.to_numeric(arg, errors="coerce") 30 | 31 | 32 | def _split_arg(arg): 33 | """ 34 | Split a comma-separated string into a list. 35 | """ 36 | if isinstance(arg, str): 37 | return [it for it in re.split(r"[\s,]+", arg) if it] 38 | return arg 39 | 40 | 41 | def _extract_series_name(ds): 42 | """ 43 | Extract series name from record set. 44 | """ 45 | m = re.match(r"^\s*([\w\.]+).*$", ds) 46 | return m.group(1) if m is not None else None 47 | 48 | 49 | def to_datetime(tstr, *, force=False): 50 | """ 51 | Parse JSOC time strings. 52 | 53 | In general, this is quite complicated, because of the many 54 | different (non-standard) time strings supported by the DRMS. For 55 | more (much more!) details on this matter, see 56 | `Rick Bogart's notes `__. 57 | 58 | The current implementation only tries to convert typical HMI time 59 | strings, with a format like "%Y.%m.%d_%H:%M:%S_TAI", to an ISO time 60 | string, that is then parsed by pandas. Note that "_TAI", as well as 61 | other timezone identifiers like "Z", will not be taken into 62 | account, so the result will be a naive timestamp without any 63 | associated timezone. 64 | 65 | If you know the time string format, it might be better calling 66 | pandas.to_datetime() directly. For handling TAI timestamps, e.g. 67 | converting between TAI and UTC, the astropy.time package can be 68 | used. 69 | 70 | Parameters 71 | ---------- 72 | tstr : str or List[str] or pandas.Series 73 | Datetime strings. 74 | force : bool 75 | Set to True to omit the ``endswith('_TAI')`` check. 76 | 77 | Returns 78 | ------- 79 | result : pandas.Series or pandas.Timestamp 80 | Pandas series or a single Timestamp object. 81 | """ 82 | date = pd.Series(tstr, dtype=object).astype(str) 83 | if force or date.str.endswith("_TAI").any(): 84 | date = date.str.replace("_TAI", "") 85 | date = date.str.replace("_", " ") 86 | if PD_VERSION >= Version("2.0.0"): 87 | regex = False 88 | else: 89 | regex = True 90 | date = date.str.replace(".", "-", regex=regex, n=2) 91 | res = _pd_to_datetime_coerce(date) 92 | res = res.dt.tz_localize(None) 93 | return res.iloc[0] if (len(res) == 1) and np.isscalar(tstr) else res 94 | -------------------------------------------------------------------------------- /examples/plot_hmi_modes.py: -------------------------------------------------------------------------------- 1 | """ 2 | ============================================= 3 | Downloading and plotting solar modes with HMI 4 | ============================================= 5 | 6 | This example shows how to download HMI data from JSOC and make a plot of the solar modes. 7 | """ 8 | 9 | import matplotlib.pyplot as plt 10 | import numpy as np 11 | 12 | import drms 13 | 14 | ############################################################################### 15 | # First we will create a `drms.Client`, using the JSOC baseurl. 16 | 17 | client = drms.Client() 18 | 19 | ############################################################################### 20 | # Construct the DRMS query string: "Series[timespan][wavelength]" 21 | 22 | qstr = "hmi.v_sht_modes[2024.09.19_00:00:00_TAI]" 23 | 24 | # TODO: Add text here. 25 | segname = "m6" # 'm6', 'm18' or 'm36' 26 | 27 | # Send request to the DRMS server 28 | print(f"Querying keyword data...\n -> {qstr}") 29 | result, filenames = client.query(qstr, key=["T_START", "T_STOP", "LMIN", "LMAX", "NDT"], seg=segname) 30 | print(f" -> {len(result)} lines retrieved.") 31 | 32 | # Use only the first line of the query result 33 | result = result.iloc[0] 34 | fname = f"http://jsoc.stanford.edu{filenames[segname][0]}" 35 | 36 | # Read the data segment 37 | print(f"Reading data from {fname}...") 38 | a = np.genfromtxt(fname) 39 | 40 | # For column names, see appendix of Larson & Schou (2015SoPh..290.3221L) 41 | l = a[:, 0].astype(int) 42 | n = a[:, 1].astype(int) 43 | nu = a[:, 2] / 1e3 44 | if a.shape[1] in [24, 48, 84]: 45 | # tan(gamma) present 46 | sig_offs = 5 47 | elif a.shape[1] in [26, 50, 86]: 48 | # tan(gamma) not present 49 | sig_offs = 6 50 | snu = a[:, sig_offs + 2] / 1e3 51 | 52 | ############################################################################### 53 | # Plot the zoomed in on lower l 54 | fig, ax = plt.subplots(1, 1, figsize=(11, 7)) 55 | ax.set_title( 56 | f"Time = {result.T_START} ... {result.T_STOP}, L = {result.LMIN} ... {result.LMAX}, NDT = {result.NDT}", 57 | fontsize="medium", 58 | ) 59 | for ni in np.unique(n): 60 | idx = n == ni 61 | ax.plot(l[idx], nu[idx], "b.-") 62 | ax.set_xlim(0, 120) 63 | ax.set_ylim(0.8, 4.5) 64 | ax.set_xlabel("Harmonic degree") 65 | ax.set_ylabel("Frequency [mHz]") 66 | fig.tight_layout() 67 | 68 | ############################################################################### 69 | # Plot the zoomed in on higher l, n <= 20, with errors 70 | 71 | fig, ax = plt.subplots(1, 1, figsize=(11, 7)) 72 | ax.set_title( 73 | f"Time = {result.T_START} ... {result.T_STOP}, L = {result.LMIN} ... {result.LMAX}, NDT = {result.NDT}", 74 | fontsize="medium", 75 | ) 76 | for ni in np.unique(n): 77 | if ni <= 20: 78 | idx = n == ni 79 | ax.plot(l[idx], nu[idx], "b.", ms=3) 80 | if ni < 10: 81 | ax.plot(l[idx], nu[idx] + 1000 * snu[idx], "g") 82 | ax.plot(l[idx], nu[idx] - 1000 * snu[idx], "g") 83 | else: 84 | ax.plot(l[idx], nu[idx] + 500 * snu[idx], "r") 85 | ax.plot(l[idx], nu[idx] - 500 * snu[idx], "r") 86 | ax.legend( 87 | loc="upper right", 88 | handles=[ 89 | plt.Line2D([0], [0], color="r", label="500 sigma"), 90 | plt.Line2D([0], [0], color="g", label="1000 sigma"), 91 | ], 92 | ) 93 | ax.set_xlim(-5, 305) 94 | ax.set_ylim(0.8, 4.5) 95 | ax.set_xlabel("Harmonic degree") 96 | ax.set_ylabel("Frequency [mHz]") 97 | fig.tight_layout() 98 | 99 | plt.show() 100 | -------------------------------------------------------------------------------- /.github/workflows/sub_package_update.yml: -------------------------------------------------------------------------------- 1 | # This template is taken from the cruft example code, for further information please see: 2 | # https://cruft.github.io/cruft/#automating-updates-with-github-actions 3 | name: Automatic Update from package template 4 | permissions: 5 | contents: write 6 | pull-requests: write 7 | 8 | on: 9 | # Allow manual runs through the web UI 10 | workflow_dispatch: 11 | schedule: 12 | # ┌───────── minute (0 - 59) 13 | # │ ┌───────── hour (0 - 23) 14 | # │ │ ┌───────── day of the month (1 - 31) 15 | # │ │ │ ┌───────── month (1 - 12 or JAN-DEC) 16 | # │ │ │ │ ┌───────── day of the week (0 - 6 or SUN-SAT) 17 | - cron: '0 7 * * 1' # Every Monday at 7am UTC 18 | 19 | jobs: 20 | update: 21 | runs-on: ubuntu-latest 22 | strategy: 23 | fail-fast: true 24 | steps: 25 | - uses: actions/checkout@v5 26 | 27 | - uses: actions/setup-python@v6 28 | with: 29 | python-version: "3.11" 30 | 31 | - name: Install Cruft 32 | run: python -m pip install git+https://github.com/Cadair/cruft@patch-p1 33 | 34 | - name: Check if update is available 35 | continue-on-error: false 36 | id: check 37 | run: | 38 | CHANGES=0 39 | if [ -f .cruft.json ]; then 40 | if ! cruft check; then 41 | CHANGES=1 42 | fi 43 | else 44 | echo "No .cruft.json file" 45 | fi 46 | 47 | echo "has_changes=$CHANGES" >> "$GITHUB_OUTPUT" 48 | 49 | - name: Run update if available 50 | id: cruft_update 51 | if: steps.check.outputs.has_changes == '1' 52 | run: | 53 | git config --global user.email "${{ github.actor }}@users.noreply.github.com" 54 | git config --global user.name "${{ github.actor }}" 55 | 56 | cruft_output=$(cruft update --skip-apply-ask --refresh-private-variables) 57 | echo $cruft_output 58 | git restore --staged . 59 | 60 | if [[ "$cruft_output" == *"Failed to cleanly apply the update, there may be merge conflicts."* ]]; then 61 | echo merge_conflicts=1 >> $GITHUB_OUTPUT 62 | else 63 | echo merge_conflicts=0 >> $GITHUB_OUTPUT 64 | fi 65 | 66 | - name: Check if only .cruft.json is modified 67 | id: cruft_json 68 | if: steps.check.outputs.has_changes == '1' 69 | run: | 70 | git status --porcelain=1 71 | if [[ "$(git status --porcelain=1)" == " M .cruft.json" ]]; then 72 | echo "Only .cruft.json is modified. Exiting workflow early." 73 | echo "has_changes=0" >> "$GITHUB_OUTPUT" 74 | else 75 | echo "has_changes=1" >> "$GITHUB_OUTPUT" 76 | fi 77 | 78 | - name: Create pull request 79 | if: steps.cruft_json.outputs.has_changes == '1' 80 | uses: peter-evans/create-pull-request@v7 81 | with: 82 | token: ${{ secrets.GITHUB_TOKEN }} 83 | add-paths: "." 84 | commit-message: "Automatic package template update" 85 | branch: "cruft/update" 86 | delete-branch: true 87 | draft: ${{ steps.cruft_update.outputs.merge_conflicts == '1' }} 88 | title: "Updates from the package template" 89 | labels: | 90 | No Changelog Entry Needed 91 | body: | 92 | This is an autogenerated PR, which will applies the latest changes from the [SunPy Package Template](https://github.com/sunpy/package-template). 93 | If this pull request has been opened as a draft there are conflicts which need fixing. 94 | 95 | **To run the CI on this pull request you will need to close it and reopen it.** 96 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | min_version = 4.0 3 | requires = 4 | tox-pypi-filter>=0.14 5 | envlist = 6 | py{312,313,314}{,-online,-sunpy} 7 | py314-devdeps 8 | py312-oldestdeps 9 | codestyle 10 | build_docs 11 | 12 | [testenv] 13 | pypi_filter = https://raw.githubusercontent.com/sunpy/sunpy/main/.test_package_pins.txt 14 | # Run the tests in a temporary directory to make sure that we don't import 15 | # the package from the source tree 16 | change_dir = .tmp/{envname} 17 | description = 18 | run tests 19 | oldestdeps: with the oldest supported version of key dependencies 20 | devdeps: with the latest developer version of key dependencies 21 | online: that require remote data 22 | sunpy: run the sunpy jsoc tests to ensure we dont break them 23 | pass_env = 24 | # A variable to tell tests we are on a CI system 25 | CI 26 | # Custom compiler locations (such as ccache) 27 | CC 28 | # Location of locales (needed by sphinx on some systems) 29 | LOCALE_ARCHIVE 30 | # If the user has set a LC override we should follow it 31 | LC_ALL 32 | # HTTP Proxy 33 | HTTP_PROXY 34 | HTTPS_PROXY 35 | NO_PROXY 36 | setenv = 37 | MPLBACKEND = agg 38 | COLUMNS = 180 39 | PYTEST_COMMAND = pytest -vvv -s -ra --pyargs drms --cov-report=xml --cov=drms --cov-config={toxinidir}/setup.cfg {toxinidir}/docs 40 | build_docs,online: HOME = {envtmpdir} 41 | JSOC_EMAIL = jsoc@sunpy.org 42 | deps = 43 | # For packages which publish nightly wheels this will pull the latest nightly 44 | # devdeps: astropy>=0.0.dev0 45 | # Packages without nightly wheels will be built from source like this 46 | # devdeps: git+https://github.com/ndcube/ndcube 47 | oldestdeps: minimum_dependencies 48 | # Extra pluings for the CI 49 | pytest-xdist 50 | pytest-timeout 51 | # These are specific extras we use to run the sunpy tests. 52 | sunpy: sunpy>=0.0.dev0 53 | sunpy: beautifulsoup4 54 | sunpy: mpl-animators 55 | sunpy: reproject 56 | sunpy: pytest-mock 57 | sunpy: python-dateutil 58 | sunpy: scipy 59 | sunpy: tqdm 60 | sunpy: zeep 61 | extras = 62 | tests 63 | commands_pre = 64 | oldestdeps: minimum_dependencies drms --filename requirements-min.txt 65 | oldestdeps: pip install -r requirements-min.txt 66 | pip freeze --all --no-input 67 | commands = 68 | # To amend the pytest command for different factors you can add a line 69 | # which starts with a factor like `online: --remote-data=any \` 70 | # If you have no factors which require different commands this is all you need: 71 | pytest \ 72 | -vvv \ 73 | -r fEs \ 74 | !sunpy: --pyargs drms \ 75 | sunpy: --pyargs sunpy.net.jsoc \ 76 | --cov-report=xml \ 77 | --cov=drms \ 78 | --cov-config={toxinidir}/.coveragerc \ 79 | online: --timeout=120 \ 80 | online,sunpy: --remote-data=any \ 81 | !sunpy: {toxinidir}/docs \ 82 | {posargs} 83 | 84 | [testenv:codestyle] 85 | pypi_filter = 86 | skip_install = true 87 | description = Run all style and file checks with pre-commit 88 | deps = 89 | pre-commit 90 | commands = 91 | pre-commit install-hooks 92 | pre-commit run --color always --all-files --show-diff-on-failure 93 | 94 | [testenv:build_docs] 95 | description = invoke sphinx-build to build the HTML docs 96 | change_dir = 97 | docs 98 | extras = 99 | docs 100 | commands = 101 | pip freeze --all --no-input 102 | sphinx-build -j 1 --color -W --keep-going -b html -d _build/.doctrees . _build/html {posargs} 103 | python -c 'import pathlib; print("Documentation available under file://\{0\}".format(pathlib.Path(r"{toxinidir}") / "docs" / "_build" / "html"/ "index.html"))' 104 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | # Main CI Workflow 2 | name: CI 3 | 4 | on: 5 | push: 6 | branches: 7 | - 'main' 8 | - '*.*' 9 | - '!*backport*' 10 | tags: 11 | - 'v*' 12 | - '!*dev*' 13 | - '!*pre*' 14 | - '!*post*' 15 | pull_request: 16 | # Allow manual runs through the web UI 17 | workflow_dispatch: 18 | schedule: 19 | # ┌───────── minute (0 - 59) 20 | # │ ┌───────── hour (0 - 23) 21 | # │ │ ┌───────── day of the month (1 - 31) 22 | # │ │ │ ┌───────── month (1 - 12 or JAN-DEC) 23 | # │ │ │ │ ┌───────── day of the week (0 - 6 or SUN-SAT) 24 | - cron: '0 7 * * 3' # Every Wed at 07:00 UTC 25 | 26 | concurrency: 27 | group: ${{ github.workflow }}-${{ github.ref }} 28 | cancel-in-progress: true 29 | 30 | jobs: 31 | core: 32 | uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v2 33 | with: 34 | submodules: false 35 | coverage: codecov 36 | toxdeps: "tox-pypi-filter" 37 | posargs: -n auto 38 | envs: | 39 | - linux: py313 40 | secrets: 41 | CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} 42 | 43 | sdist_verify: 44 | runs-on: ubuntu-latest 45 | steps: 46 | - uses: actions/checkout@v5 47 | - uses: actions/setup-python@v6 48 | with: 49 | python-version: '3.13' 50 | - run: python -m pip install -U --user build 51 | - run: python -m build . --sdist 52 | - run: python -m pip install -U --user twine 53 | - run: python -m twine check dist/* 54 | 55 | test: 56 | needs: [core, sdist_verify] 57 | uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v2 58 | with: 59 | submodules: false 60 | coverage: codecov 61 | toxdeps: "tox-pypi-filter" 62 | posargs: -n auto 63 | envs: | 64 | - linux: py314 65 | - windows: py312 66 | - macos: py312 67 | - linux: py312-oldestdeps 68 | - linux: py314-devdeps 69 | secrets: 70 | CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} 71 | 72 | docs: 73 | needs: [core] 74 | uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v2 75 | with: 76 | default_python: '3.13' 77 | submodules: false 78 | pytest: false 79 | toxdeps: "tox-pypi-filter" 80 | libraries: | 81 | apt: 82 | - graphviz 83 | envs: | 84 | - linux: build_docs 85 | 86 | online: 87 | uses: OpenAstronomy/github-actions-workflows/.github/workflows/tox.yml@v1 88 | with: 89 | default_python: '3.12' 90 | submodules: false 91 | coverage: codecov 92 | toxdeps: "tox-pypi-filter" 93 | posargs: -n 1 --dist loadgroup 94 | envs: | 95 | - linux: py312-online 96 | - linux: py312-sunpy 97 | secrets: 98 | CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} 99 | 100 | publish: 101 | # Build wheels on PRs only when labelled. Releases will only be published if tagged ^v.* 102 | # see https://github-actions-workflows.openastronomy.org/en/latest/publish.html#upload-to-pypi 103 | if: | 104 | github.event_name != 'pull_request' || 105 | ( 106 | github.event_name == 'pull_request' && 107 | contains(github.event.pull_request.labels.*.name, 'Run publish') 108 | ) 109 | needs: [test, docs] 110 | uses: OpenAstronomy/github-actions-workflows/.github/workflows/publish_pure_python.yml@v2 111 | with: 112 | python-version: '3.13' 113 | test_extras: 'tests' 114 | test_command: 'pytest -p no:warnings --doctest-rst --pyargs drms' 115 | submodules: false 116 | secrets: 117 | pypi_token: ${{ secrets.pypi_token }} 118 | -------------------------------------------------------------------------------- /drms/tests/test_jsoc_query.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | import drms 4 | from drms.config import ServerConfig 5 | from drms.exceptions import DrmsOperationNotSupported, DrmsQueryError 6 | 7 | 8 | @pytest.mark.jsoc() 9 | @pytest.mark.remote_data() 10 | def test_query_basic(jsoc_client): 11 | keys, segs = jsoc_client.query("hmi.v_45s[2013.07.03_08:42_TAI/3m]", key="T_REC, CRLT_OBS", seg="Dopplergram") 12 | assert len(keys) == 4 13 | for k in ["T_REC", "CRLT_OBS"]: 14 | assert k in keys.columns 15 | assert len(segs) == 4 16 | assert "Dopplergram" in segs.columns 17 | assert ((keys.CRLT_OBS - 3.14159).abs() < 0.0001).all() 18 | 19 | 20 | @pytest.mark.jsoc() 21 | @pytest.mark.remote_data() 22 | def test_query_allmissing(jsoc_client): 23 | with pytest.raises(ValueError, match="At least one key, seg or link must be specified"): 24 | jsoc_client.query("hmi.v_45s[2013.07.03_08:42_TAI/3m]") 25 | 26 | 27 | @pytest.mark.jsoc() 28 | @pytest.mark.remote_data() 29 | def test_query_key(jsoc_client): 30 | keys = jsoc_client.query("hmi.v_45s[2013.07.03_08:42_TAI/3m]", key="T_REC, CRLT_OBS") 31 | assert len(keys) == 4 32 | for k in ["T_REC", "CRLT_OBS"]: 33 | assert k in keys.columns 34 | assert ((keys.CRLT_OBS - 3.14159).abs() < 0.0001).all() 35 | 36 | 37 | @pytest.mark.jsoc() 38 | @pytest.mark.remote_data() 39 | def test_query_seg(jsoc_client): 40 | segs = jsoc_client.query("hmi.v_45s[2013.07.03_08:42_TAI/3m]", seg="Dopplergram") 41 | assert len(segs) == 4 42 | assert "Dopplergram" in segs.columns 43 | 44 | 45 | @pytest.mark.jsoc() 46 | @pytest.mark.remote_data() 47 | def test_query_link(jsoc_client): 48 | links = jsoc_client.query("hmi.B_720s[2013.07.03_08:42_TAI/40m]", link="MDATA") 49 | assert len(links) == 3 50 | assert "MDATA" in links.columns 51 | 52 | 53 | @pytest.mark.jsoc() 54 | @pytest.mark.remote_data() 55 | def test_query_seg_key_link(jsoc_client): 56 | keys, segs, links = jsoc_client.query("hmi.B_720s[2013.07.03_08:42_TAI/40m]", key="foo", link="bar", seg="baz") 57 | assert len(keys) == 3 58 | assert (keys.foo == "Invalid KeyLink").all() 59 | assert len(segs) == 3 60 | assert (segs.baz == "InvalidSegName").all() 61 | assert len(links) == 3 62 | assert (links.bar == "Invalid_Link").all() 63 | 64 | 65 | @pytest.mark.jsoc() 66 | @pytest.mark.remote_data() 67 | def test_query_pkeys(jsoc_client): 68 | keys = jsoc_client.query("hmi.v_45s[2013.07.03_08:42_TAI/3m]", pkeys=True) 69 | pkeys = list(keys.columns.values) 70 | assert pkeys == jsoc_client.pkeys("hmi.v_45s[2013.07.03_08:42_TAI/3m]") 71 | assert len(keys) == 4 72 | 73 | 74 | @pytest.mark.jsoc() 75 | @pytest.mark.remote_data() 76 | def test_query_recindex(jsoc_client): 77 | keys = jsoc_client.query("hmi.V_45s[2013.07.03_08:42_TAI/4m]", key="T_REC", rec_index=True) 78 | index_values = list(keys.index.values) 79 | assert all("hmi.V_45s" in s for s in index_values) 80 | assert len(keys) == 5 81 | 82 | 83 | def test_query_invalid_server(): 84 | cfg = ServerConfig(name="TEST") 85 | c = drms.Client(cfg) 86 | with pytest.raises(DrmsOperationNotSupported): 87 | c.query("hmi.v_45s[2013.07.03_08:42_TAI/3m]", pkeys=True) 88 | 89 | 90 | @pytest.mark.jsoc() 91 | @pytest.mark.remote_data() 92 | def test_query_invalid_series(jsoc_client): 93 | with pytest.raises(DrmsQueryError): 94 | jsoc_client.query("foo", key="T_REC") 95 | 96 | 97 | @pytest.mark.remote_data() 98 | @pytest.mark.parametrize( 99 | "query", 100 | [ 101 | "hmi.v_45s[2014.01.01_00:00:35_TAI-2014.01.01_01:00:35_TAI]", 102 | "hmi.M_720s[2011.04.14_00:30:00_TAI/6h@2h]", 103 | ], 104 | ) 105 | def test_query_hexadecimal_strings(query): 106 | # Exercise the part of client.py that deals with hexadecimal strings 107 | c = drms.Client() 108 | c.query(query, key="**ALL**") 109 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = [ 3 | "setuptools>=62.1", 4 | "setuptools_scm[toml]>=8.0.0", 5 | "wheel", 6 | ] 7 | build-backend = "setuptools.build_meta" 8 | 9 | [project] 10 | name = "drms" 11 | description = "Access HMI, AIA and MDI data from the Standford JSOC DRMS" 12 | requires-python = ">=3.12" 13 | readme = { file = "README.rst", content-type = "text/x-rst" } 14 | license-files = ["licenses/LICENSE.rst"] 15 | authors = [ 16 | { name = "The SunPy Community", email = "sunpy@googlegroups.com" }, 17 | ] 18 | dependencies = [ 19 | "numpy>=1.26.0", 20 | "pandas>=2.2.0", 21 | ] 22 | dynamic = [ "version" ] 23 | 24 | [project.optional-dependencies] 25 | tests = [ 26 | "astropy", # Doctests 27 | "pytest", 28 | "pytest-astropy", 29 | "pytest-doctestplus", 30 | "pytest-cov", 31 | "pytest-xdist", 32 | ] 33 | docs = [ 34 | "sphinx", 35 | "sphinx-automodapi", 36 | "sunpy-sphinx-theme", 37 | "packaging", 38 | "astropy", 39 | "matplotlib", 40 | "sphinx-changelog", 41 | "sphinx-copybutton", 42 | "sphinx-gallery", 43 | "sphinx-hoverxref", 44 | "sphinxext-opengraph", 45 | ] 46 | 47 | [project.urls] 48 | Homepage = "https://sunpy.org" 49 | "Source Code" = "https://github.com/sunpy/drms" 50 | Download = "https://pypi.org/project/sunpy" 51 | Documentation = "https://docs.sunpy.org/projects/drms" 52 | Changelog = "https://docs.sunpy.org/projects/drms/en/stable/whatsnew/changelog.html" 53 | "Issue Tracker" = "https://github.com/sunpy/drms/issues" 54 | 55 | [tool.setuptools] 56 | zip-safe = false 57 | include-package-data = true 58 | 59 | [tool.setuptools.packages.find] 60 | include = ["drms*"] 61 | exclude = ["drms._dev*"] 62 | 63 | [tool.setuptools_scm] 64 | version_file = "drms/_version.py" 65 | 66 | [tool.gilesbot] 67 | [tool.gilesbot.pull_requests] 68 | enabled = true 69 | 70 | [tool.gilesbot.towncrier_changelog] 71 | enabled = true 72 | verify_pr_number = true 73 | changelog_skip_label = "No Changelog Entry Needed" 74 | help_url = "https://github.com/sunpy/drms/blob/main/changelog/README.rst" 75 | 76 | changelog_missing_long = "There isn't a changelog file in this pull request. Please add a changelog file to the `changelog/` directory following the instructions in the changelog [README](https://github.com/sunpy/drms/blob/main/changelog/README.rst)." 77 | 78 | type_incorrect_long = "The changelog file you added is not one of the allowed types. Please use one of the types described in the changelog [README](https://github.com/sunpy/drms/blob/main/changelog/README.rst)" 79 | 80 | number_incorrect_long = "The number in the changelog file you added does not match the number of this pull request. Please rename the file." 81 | 82 | # TODO: This should be in towncrier.toml but Giles currently only works looks in 83 | # pyproject.toml we should move this back when it's fixed. 84 | [tool.towncrier] 85 | package = "drms" 86 | filename = "CHANGELOG.rst" 87 | directory = "changelog/" 88 | issue_format = "`#{issue} `__" 89 | title_format = "{version} ({project_date})" 90 | 91 | [[tool.towncrier.type]] 92 | directory = "breaking" 93 | name = "Breaking Changes" 94 | showcontent = true 95 | 96 | [[tool.towncrier.type]] 97 | directory = "deprecation" 98 | name = "Deprecations" 99 | showcontent = true 100 | 101 | [[tool.towncrier.type]] 102 | directory = "removal" 103 | name = "Removals" 104 | showcontent = true 105 | 106 | [[tool.towncrier.type]] 107 | directory = "feature" 108 | name = "New Features" 109 | showcontent = true 110 | 111 | [[tool.towncrier.type]] 112 | directory = "bugfix" 113 | name = "Bug Fixes" 114 | showcontent = true 115 | 116 | [[tool.towncrier.type]] 117 | directory = "doc" 118 | name = "Documentation" 119 | showcontent = true 120 | 121 | [[tool.towncrier.type]] 122 | directory = "trivial" 123 | name = "Internal Changes" 124 | showcontent = true 125 | -------------------------------------------------------------------------------- /drms/tests/test_to_datetime.py: -------------------------------------------------------------------------------- 1 | import numpy as np 2 | import pandas as pd 3 | import pytest 4 | 5 | import drms 6 | 7 | data_tai = [ 8 | ("2010.05.01_TAI", pd.Timestamp("2010-05-01 00:00:00")), 9 | ("2010.05.01_00:00_TAI", pd.Timestamp("2010-05-01 00:00:00")), 10 | ("2010.05.01_00:00:00_TAI", pd.Timestamp("2010-05-01 00:00:00")), 11 | ("2010.05.01_01:23:45_TAI", pd.Timestamp("2010-05-01 01:23:45")), 12 | ("2013.12.21_23:32_TAI", pd.Timestamp("2013-12-21 23:32:00")), 13 | ("2013.12.21_23:32:34_TAI", pd.Timestamp("2013-12-21 23:32:34")), 14 | ] 15 | data_tai_in = [data[0] for data in data_tai] 16 | data_tai_out = pd.Series([data[1] for data in data_tai]) 17 | 18 | 19 | @pytest.mark.parametrize(("time_string", "expected"), data_tai) 20 | def test_tai_string(time_string, expected): 21 | assert drms.to_datetime(time_string) == expected 22 | 23 | 24 | @pytest.mark.parametrize( 25 | ("time_string", "expected"), 26 | [ 27 | ("2010-05-01T00:00Z", pd.Timestamp("2010-05-01 00:00:00")), 28 | ("2010-05-01T00:00:00Z", pd.Timestamp("2010-05-01 00:00:00")), 29 | ("2010-05-01T01:23:45Z", pd.Timestamp("2010-05-01 01:23:45")), 30 | ("2013-12-21T23:32Z", pd.Timestamp("2013-12-21 23:32:00")), 31 | ("2013-12-21T23:32:34Z", pd.Timestamp("2013-12-21 23:32:34")), 32 | ("2010-05-01 00:00Z", pd.Timestamp("2010-05-01 00:00:00")), 33 | ("2010-05-01 00:00:00Z", pd.Timestamp("2010-05-01 00:00:00")), 34 | ("2010-05-01 01:23:45Z", pd.Timestamp("2010-05-01 01:23:45")), 35 | ("2013-12-21 23:32Z", pd.Timestamp("2013-12-21 23:32:00")), 36 | ("2013-12-21 23:32:34Z", pd.Timestamp("2013-12-21 23:32:34")), 37 | ], 38 | ) 39 | def test_z_string(time_string, expected): 40 | assert drms.to_datetime(time_string) == expected 41 | 42 | 43 | @pytest.mark.xfail(reason="pandas does not support leap seconds") 44 | @pytest.mark.parametrize( 45 | ("time_string", "expected"), 46 | [ 47 | ("2012-06-30T23:59:60Z", "2012-06-30 23:59:60"), 48 | ("2015-06-30T23:59:60Z", "2015-06-30 23:59:60"), 49 | ("2016-12-31T23:59:60Z", "2016-12-31 23:59:60"), 50 | ], 51 | ) 52 | def test_z_leap_string(time_string, expected): 53 | assert drms.to_datetime(time_string) == expected 54 | 55 | 56 | @pytest.mark.parametrize( 57 | ("time_string", "expected"), 58 | [ 59 | ("2013.12.21_23:32:34_TAI", pd.Timestamp("2013-12-21 23:32:34")), 60 | ("2013.12.21_23:32:34_UTC", pd.Timestamp("2013-12-21 23:32:34")), 61 | ("2013.12.21_23:32:34Z", pd.Timestamp("2013-12-21 23:32:34")), 62 | ], 63 | ) 64 | def test_force_string(time_string, expected): 65 | assert drms.to_datetime(time_string, force=True) == expected 66 | 67 | 68 | @pytest.mark.parametrize( 69 | ("time_series", "expected"), 70 | [ 71 | (data_tai_in, data_tai_out), 72 | (pd.Series(data_tai_in), data_tai_out), 73 | (tuple(data_tai_in), data_tai_out), 74 | (np.array(data_tai_in), data_tai_out), 75 | ], 76 | ) 77 | def test_time_series(time_series, expected): 78 | assert drms.to_datetime(time_series).equals(expected) 79 | 80 | 81 | data_invalid = [ 82 | ("2010.05.01_TAI", False), 83 | ("2010.05.01_00:00_TAI", False), 84 | ("", True), 85 | ("1600", True), 86 | ("foo", True), 87 | ("2013.12.21_23:32:34_TAI", False), 88 | ] 89 | data_invalid_in = [data[0] for data in data_invalid] 90 | data_invalid_out = pd.Series([data[1] for data in data_invalid]) 91 | 92 | 93 | @pytest.mark.parametrize(("time_string", "expected"), data_invalid) 94 | def test_corner_case(time_string, expected): 95 | assert pd.isna(drms.to_datetime(time_string)) == expected 96 | assert isinstance(drms.to_datetime([]), pd.Series) 97 | assert drms.to_datetime([]).empty 98 | 99 | 100 | @pytest.mark.parametrize( 101 | ("time_series", "expected"), 102 | [ 103 | (data_invalid_in, data_invalid_out), 104 | (pd.Series(data_invalid_in), data_invalid_out), 105 | (tuple(data_invalid_in), data_invalid_out), 106 | (np.array(data_invalid_in), data_invalid_out), 107 | ], 108 | ) 109 | def test_corner_case_series(time_series, expected): 110 | assert pd.isna(drms.to_datetime(time_series)).equals(expected) 111 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | 0.9.0 (2025-02-05) 2 | ================== 3 | 4 | New Features 5 | ------------ 6 | 7 | - Added timeout keyword to :meth:`drms.client.ExportRequest.download` which also will use the socket value, if it is set. (`#137 `__) 8 | 9 | 10 | 0.8.0 (2024-07-23) 11 | ================== 12 | 13 | Backwards Incompatible Changes 14 | ------------------------------ 15 | 16 | - Increased minimum version of Python to 3.10.0 (`#116 `__) 17 | - The return from `drms.JsocInfoConstants` is now a string, there is no need to do ``.value`` on it. (`#116 `__) 18 | 19 | 20 | 0.7.1 (2023-12-28) 21 | ================== 22 | 23 | Bug Fixes 24 | --------- 25 | 26 | - Incorrect setup of the logger is now fixed. (`#113 `__) 27 | - Fixed missing environment variable in the docs. (`#113 `__) 28 | 29 | 0.7.0 (2023-11-17) 30 | ================== 31 | 32 | Backwards Incompatible Changes 33 | ------------------------------ 34 | 35 | - Dropped Python 3.8 support. (`#90 `__) 36 | - Updated all of the sphinx anchors to be more consistent. 37 | This means that any use of the old anchors (intersphinx links to sunpy doc pages) will need to be updated. (`#90 `__) 38 | - Removed ``verbose`` keyword argument from `drms.Client`. 39 | Now all prints are done via the logging module. (`#90 `__) 40 | - ``drms.const`` was renamed to `drms.JsocInfoConstants` 41 | It is also now a `Enum`. (`#90 `__) 42 | - Renamed keywords or arguments from ``requestor`` to ``requester``. (`#90 `__) 43 | - Removed ``debug`` keyword argument from `drms.HttpJsonClient` 44 | Now all prints are done via the logging module. (`#90 `__) 45 | - Removed all FTP options. (`#90 `__) 46 | - All keywords have to be passed by reference, no more positional keywords arguments are allowed. (`#90 `__) 47 | 48 | 49 | Trivial/Internal Changes 50 | ------------------------ 51 | 52 | - Added "ruff" to the pre-commit and fixed the errors. (`#90 `__) 53 | 54 | 55 | 0.6.4 (2023-06-09) 56 | ================== 57 | 58 | Bug Fixes 59 | --------- 60 | 61 | - Modified :meth:`drms.client.Client._convert_numeric_keywords` to use a row-centric approach for handling hexadecimal strings. (`#102 `__) 62 | - Modified :meth:`drms.utils.to_datetime` to work with Pandas 2.0. (`#103 `__) 63 | - Fixed pandas 2.0.0 warning. (`#97 `__) 64 | 65 | 66 | 0.6.3 (2022-10-13) 67 | ================== 68 | 69 | Bug Fixes 70 | --------- 71 | 72 | - Updated indexing in a function to prevent FutureWarnings from pandas. (`#73 `__) 73 | 74 | 75 | Trivial/Internal Changes 76 | ------------------------ 77 | 78 | - Updated the init of `drms.json.HttpJsonRequest` to raise a nicer message if the URL fails to open. (`#76 `__) 79 | 80 | 81 | 0.6.2 (2021-05-15) 82 | ================== 83 | 84 | Trivial 85 | ------- 86 | 87 | - Tidy up of internal code that has no user facing changes. 88 | 89 | 90 | 0.6.1 (2021-01-23) 91 | ================== 92 | 93 | Bug Fixes 94 | --------- 95 | 96 | - Fixed issue with downloads not having the primekeys substituted with their correct values in downloaded filenames. (`#52 `__) 97 | 98 | 99 | 0.6.0 (2020-11-01) 100 | ================== 101 | 102 | Improved Documentation 103 | ---------------------- 104 | 105 | - Examples has been formatted into an online gallery. 106 | 107 | Backwards Incompatible Changes 108 | ------------------------------ 109 | 110 | - Python 2 support has been dropped, only Python 3.7 or higher is supported. 111 | 112 | Deprecations and Removals 113 | ------------------------- 114 | 115 | - ``Client.get()`` has been removed, use :meth:`drms.client.Client.query()` instead. 116 | 117 | Support for Processing Keywords 118 | -------------------------------- 119 | 120 | - :meth:`drms.client.Client.export` now accepts a ``process`` keyword argument 121 | - This allows users to specify additional server-side processing options such as image cutouts 122 | - See the "Processing" section of the `JSOC Data Export page `__ for more information. 123 | -------------------------------------------------------------------------------- /drms/tests/test_series_info.py: -------------------------------------------------------------------------------- 1 | from collections import OrderedDict 2 | 3 | import pandas as pd 4 | 5 | import drms 6 | 7 | 8 | def test_parse_keywords(): 9 | info = [ 10 | { 11 | "recscope": "variable", 12 | "units": "none", 13 | "name": "cparms_sg000", 14 | "defval": "compress Rice", 15 | "note": "", 16 | "type": "string", 17 | }, 18 | { 19 | "recscope": "variable", 20 | "units": "none", 21 | "name": "mean_bzero", 22 | "defval": "0", 23 | "note": "", 24 | "type": "double", 25 | }, 26 | { 27 | "recscope": "variable", 28 | "units": "none", 29 | "name": "mean_bscale", 30 | "defval": "0.25", 31 | "note": "", 32 | "type": "double", 33 | }, 34 | { 35 | "recscope": "variable", 36 | "units": "TAI", 37 | "name": "MidTime", 38 | "defval": "-4712.01.01_11:59_TAI", 39 | "note": "Midpoint of averaging interval", 40 | "type": "time", 41 | }, 42 | ] 43 | exp = OrderedDict( 44 | [ 45 | ("name", ["cparms_sg000", "mean_bzero", "mean_bscale", "MidTime"]), 46 | ("type", ["string", "double", "double", "time"]), 47 | ("recscope", ["variable", "variable", "variable", "variable"]), 48 | ("defval", ["compress Rice", "0", "0.25", "-4712.01.01_11:59_TAI"]), 49 | ("units", ["none", "none", "none", "TAI"]), 50 | ("note", ["", "", "", "Midpoint of averaging interval"]), 51 | ("linkinfo", [None, None, None, None]), 52 | ("is_time", [False, False, False, True]), 53 | ("is_integer", [False, False, False, False]), 54 | ("is_real", [False, True, True, False]), 55 | ("is_numeric", [False, True, True, False]), 56 | ], 57 | ) 58 | 59 | exp = pd.DataFrame(data=exp) 60 | exp.index = exp.pop("name") 61 | assert drms.SeriesInfo._parse_keywords(info).equals(exp) 62 | 63 | 64 | def test_parse_links(): 65 | links = [ 66 | {"name": "BHARP", "kind": "DYNAMIC", "note": "Bharp", "target": "hmi.Bharp_720s"}, 67 | {"name": "MHARP", "kind": "DYNAMIC", "note": "Mharp", "target": "hmi.Mharp_720s"}, 68 | ] 69 | exp = OrderedDict( 70 | [ 71 | ("name", ["BHARP", "MHARP"]), 72 | ("target", ["hmi.Bharp_720s", "hmi.Mharp_720s"]), 73 | ("kind", ["DYNAMIC", "DYNAMIC"]), 74 | ("note", ["Bharp", "Mharp"]), 75 | ], 76 | ) 77 | exp = pd.DataFrame(data=exp) 78 | exp.index = exp.pop("name") 79 | assert drms.SeriesInfo._parse_links(links).equals(exp) 80 | 81 | 82 | def test_parse_segments(): 83 | segments = [ 84 | { 85 | "type": "int", 86 | "dims": "VARxVAR", 87 | "units": "Gauss", 88 | "protocol": "fits", 89 | "note": "magnetogram", 90 | "name": "magnetogram", 91 | }, 92 | { 93 | "type": "char", 94 | "dims": "VARxVAR", 95 | "units": "Enumerated", 96 | "protocol": "fits", 97 | "note": "Mask for the patch", 98 | "name": "bitmap", 99 | }, 100 | { 101 | "type": "int", 102 | "dims": "VARxVAR", 103 | "units": "m/s", 104 | "protocol": "fits", 105 | "note": "Dopplergram", 106 | "name": "Dopplergram", 107 | }, 108 | ] 109 | exp = OrderedDict( 110 | [ 111 | ("name", ["magnetogram", "bitmap", "Dopplergram"]), 112 | ("type", ["int", "char", "int"]), 113 | ("units", ["Gauss", "Enumerated", "m/s"]), 114 | ("protocol", ["fits", "fits", "fits"]), 115 | ("dims", ["VARxVAR", "VARxVAR", "VARxVAR"]), 116 | ("note", ["magnetogram", "Mask for the patch", "Dopplergram"]), 117 | ], 118 | ) 119 | 120 | exp = pd.DataFrame(data=exp) 121 | exp.index = exp.pop("name") 122 | assert drms.SeriesInfo._parse_segments(segments).equals(exp) 123 | 124 | 125 | def test_repr(): 126 | info = { 127 | "primekeys": ["CarrRot", "CMLon"], 128 | "retention": 1800, 129 | "tapegroup": 1, 130 | "archive": 1, 131 | "primekeysinfo": [], 132 | "unitsize": 1, 133 | "note": "Temporal averages of HMI Vgrams over 1/3 CR", 134 | "dbindex": [None], 135 | "links": [], 136 | "segments": [], 137 | "keywords": [], 138 | } 139 | assert repr(drms.SeriesInfo(info)) == "" 140 | assert repr(drms.SeriesInfo(info, name="hmi")) == "" 141 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | ==== 2 | drms 3 | ==== 4 | 5 | Access HMI, AIA and MDI data from the Standford JSOC DRMS. 6 | 7 | `Docs `__ | 8 | `Tutorial `__ | 9 | `Github `__ | 10 | `PyPI `__ 11 | 12 | |JOSS| |Zenodo| 13 | 14 | .. |JOSS| image:: https://joss.theoj.org/papers/10.21105/joss.01614/status.svg 15 | :target: https://doi.org/10.21105/joss.01614 16 | .. |Zenodo| image:: https://zenodo.org/badge/58651845.svg 17 | :target: https://zenodo.org/badge/latestdoi/58651845 18 | 19 | The ``drms`` module provides an easy-to-use interface for accessing HMI, AIA and MDI data with Python. 20 | It uses the publicly accessible `JSOC `__ DRMS server by default, but can also be used with local `NetDRMS `__ sites. 21 | More information, including a detailed tutorial, is available in the `Documentation `__. 22 | 23 | Getting Help 24 | ------------ 25 | 26 | For more information or to ask questions about ``drms``, check out: 27 | 28 | - `drms Documentation `__ 29 | - `SunPy Chat `__ 30 | 31 | Usage of Generative AI 32 | ---------------------- 33 | 34 | We expect authentic engagement in our community. 35 | Be wary of posting output from Large Language Models or similar generative AI as comments on GitHub or any other platform, as such comments tend to be formulaic and low quality content. 36 | If you use generative AI tools as an aid in developing code or documentation changes, ensure that you fully understand the proposed changes and can explain why they are the correct approach and an improvement to the current state. 37 | 38 | License 39 | ------- 40 | 41 | This project is Copyright (c) The SunPy Community and licensed under 42 | the terms of the BSD 2-Clause license. This package is based upon 43 | the `Openastronomy packaging guide `_ 44 | which is licensed under the BSD 3-clause licence. See the licenses folder for 45 | more information. 46 | 47 | Contributing 48 | ------------ 49 | 50 | We love contributions! drms is open source, 51 | built on open source, and we'd love to have you hang out in our community. 52 | 53 | **Imposter syndrome disclaimer**: We want your help. No, really. 54 | 55 | There may be a little voice inside your head that is telling you that you're not 56 | ready to be an open source contributor; that your skills aren't nearly good 57 | enough to contribute. What could you possibly offer a project like this one? 58 | 59 | We assure you - the little voice in your head is wrong. If you can write code at 60 | all, you can contribute code to open source. Contributing to open source 61 | projects is a fantastic way to advance one's coding skills. Writing perfect code 62 | isn't the measure of a good developer (that would disqualify all of us!); it's 63 | trying to create something, making mistakes, and learning from those 64 | mistakes. That's how we all improve, and we are happy to help others learn. 65 | 66 | Being an open source contributor doesn't just mean writing code, either. You can 67 | help out by writing documentation, tests, or even giving feedback about the 68 | project (and yes - that includes giving feedback about the contribution 69 | process). Some of these contributions may be the most valuable to the project as 70 | a whole, because you're coming to the project with fresh eyes, so you can see 71 | the errors and assumptions that seasoned contributors have glossed over. 72 | 73 | Note: This disclaimer was originally written by 74 | `Adrienne Lowe `_ for a 75 | `PyCon talk `_, and was adapted by 76 | drms based on its use in the README file for the 77 | `MetPy project `_. 78 | 79 | Citation 80 | -------- 81 | If you use ``drms`` in your work, please cite our `paper `__. 82 | 83 | .. code-block:: bibtex 84 | 85 | @article{Glogowski2019, 86 | doi = {10.21105/joss.01614}, 87 | url = {https://doi.org/10.21105/joss.01614}, 88 | year = {2019}, 89 | publisher = {The Open Journal}, 90 | volume = {4}, 91 | number = {40}, 92 | pages = {1614}, 93 | author = {Kolja Glogowski and Monica G. Bobra and Nitin Choudhary and Arthur B. Amezcua and Stuart J. Mumford}, 94 | title = {drms: A Python package for accessing HMI and AIA data}, 95 | journal = {Journal of Open Source Software} 96 | } 97 | 98 | Code of Conduct (CoC) 99 | --------------------- 100 | 101 | When you are interacting with the SunPy community you are asked to follow our `code of conduct `__. 102 | 103 | Acknowledgements 104 | ---------------- 105 | 106 | Kolja Glogowski has received funding from the European Research Council under the European Union's Seventh Framework Programme (FP/2007-2013) / ERC Grant Agreement no. 307117. 107 | -------------------------------------------------------------------------------- /drms/tests/test_jsoc_export.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | import drms 4 | 5 | 6 | @pytest.mark.jsoc() 7 | @pytest.mark.remote_data() 8 | @pytest.mark.parametrize("method", ["url_quick", "url"]) 9 | def test_export_asis_basic(jsoc_client_export, method): 10 | r = jsoc_client_export.export( 11 | "hmi.v_sht_2drls[2024.09.19_00:00:00_TAI]{split,rot,err}", 12 | protocol="as-is", 13 | method=method, 14 | requester=False, 15 | ) 16 | 17 | assert isinstance(r, drms.ExportRequest) 18 | assert r.wait(timeout=60) 19 | assert r.has_succeeded() 20 | assert r.protocol == "as-is" 21 | assert len(r.urls) == 9 # 3 files per segment 22 | 23 | for record in r.urls.record: 24 | record = record.lower() 25 | assert record.startswith("hmi.v_sht_2drls[2024.09.19_00:00:00_tai]") 26 | assert record.endswith(("{split}", "{rot}", "{err}")) 27 | 28 | for filename in r.urls.filename: 29 | assert filename.endswith(("err.2d", "rot.2d", "splittings.out")) 30 | 31 | for url in r.urls.url: 32 | assert url.endswith(("err.2d", "rot.2d", "splittings.out")) 33 | 34 | 35 | @pytest.mark.jsoc() 36 | @pytest.mark.remote_data() 37 | def test_export_fits_basic(jsoc_client_export): 38 | r = jsoc_client_export.export( 39 | "hmi.sharp_720s[4864][2014.11.30_00:00_TAI]{continuum, magnetogram}", 40 | protocol="fits", 41 | method="url", 42 | requester=False, 43 | ) 44 | 45 | assert isinstance(r, drms.ExportRequest) 46 | assert r.wait(timeout=60) 47 | assert r.has_succeeded() 48 | assert r.protocol == "fits" 49 | assert len(r.urls) == 2 # 1 file per segment 50 | 51 | for record in r.urls.record: 52 | record = record.lower() 53 | assert record.startswith("hmi.sharp_720s[4864]") 54 | assert record.endswith("2014.11.30_00:00:00_tai]") 55 | 56 | for filename in r.urls.filename: 57 | assert filename.endswith(("continuum.fits", "magnetogram.fits")) 58 | 59 | for url in r.urls.url: 60 | assert url.endswith(("continuum.fits", "magnetogram.fits")) 61 | 62 | 63 | @pytest.mark.jsoc() 64 | @pytest.mark.remote_data() 65 | def test_export_im_patch(jsoc_client_export): 66 | # TODO: check that this has actually done the export/processing properly? 67 | # NOTE: processing exports seem to fail silently on the server side if 68 | # the correct names/arguments are not passed. Not clear how to check 69 | # that this has not happened. 70 | process = { 71 | "im_patch": { 72 | "t_ref": "2025-01-01T04:33:30.000", 73 | "t": 0, 74 | "r": 0, 75 | "c": 0, 76 | "locunits": "arcsec", 77 | "boxunits": "arcsec", 78 | "x": -517.2, 79 | "y": -246, 80 | "width": 345.6, 81 | "height": 345.6, 82 | }, 83 | } 84 | req = jsoc_client_export.export( 85 | "aia.lev1_euv_12s[2025-01-01T04:33:30.000/1m@12s][171]{image}", 86 | method="url", 87 | protocol="fits", 88 | process=process, 89 | requester=False, 90 | ) 91 | 92 | assert isinstance(req, drms.ExportRequest) 93 | assert req.wait(timeout=60) 94 | assert req.has_succeeded() 95 | assert req.protocol == "fits" 96 | 97 | for record in req.urls.record: 98 | record = record.lower() 99 | assert record.startswith("aia.lev1_euv_12s_mod") 100 | 101 | for filename in req.urls.filename: 102 | assert filename.endswith("image.fits") 103 | 104 | for url in req.urls.url: 105 | assert url.endswith("image.fits") 106 | 107 | 108 | @pytest.mark.jsoc() 109 | @pytest.mark.remote_data() 110 | def test_export_rebin(jsoc_client_export): 111 | # TODO: check that this has actually done the export/processing properly? 112 | # NOTE: processing exports seem to fail silently on the server side if 113 | # the correct names/arguments are not passed. Not clear how to check 114 | # that this has not happened. 115 | req = jsoc_client_export.export( 116 | "hmi.M_720s[2020-10-17_22:12:00_TAI/24m]{magnetogram}", 117 | method="url", 118 | protocol="fits", 119 | process={"rebin": {"method": "boxcar", "scale": 0.25}}, 120 | requester=False, 121 | ) 122 | 123 | assert isinstance(req, drms.ExportRequest) 124 | assert req.wait(timeout=60) 125 | assert req.has_succeeded() 126 | assert req.protocol == "fits" 127 | 128 | for record in req.urls.record: 129 | record = record.lower() 130 | assert record.startswith("hmi.m_720s_mod") 131 | 132 | for filename in req.urls.filename: 133 | assert filename.endswith("magnetogram.fits") 134 | 135 | for url in req.urls.url: 136 | assert url.endswith("magnetogram.fits") 137 | 138 | 139 | @pytest.mark.jsoc() 140 | @pytest.mark.remote_data() 141 | def test_export_invalid_process(jsoc_client_export): 142 | with pytest.raises(ValueError, match="foobar is not one of the allowed processing options"): 143 | jsoc_client_export.export( 144 | "aia.lev1_euv_12s[2015-10-17T04:33:30.000/1m@12s][171]{image}", 145 | process={"foobar": {}}, 146 | ) 147 | 148 | 149 | @pytest.mark.jsoc() 150 | @pytest.mark.remote_data() 151 | def test_export_email(jsoc_client): 152 | with pytest.raises(ValueError, match=r"The email argument is required, when no default email address was set."): 153 | jsoc_client.export("hmi.v_45s[2016.04.01_TAI/1d@6h]{Dopplergram}") 154 | -------------------------------------------------------------------------------- /drms/config.py: -------------------------------------------------------------------------------- 1 | from urllib.parse import urljoin 2 | 3 | __all__ = ["ServerConfig", "register_server"] 4 | 5 | 6 | class ServerConfig: 7 | """ 8 | DRMS Server configuration. 9 | 10 | Additional keyword arguments can be used to add additional entries 11 | to config. In case a keyword argument already exists in the config 12 | dictionary, the config entry will be replaced by the kwargs value. 13 | 14 | Available config keys are: 15 | name 16 | cgi_baseurl 17 | cgi_show_series 18 | cgi_jsoc_info 19 | cgi_jsoc_fetch 20 | cgi_check_address 21 | cgi_show_series_wrapper 22 | show_series_wrapper_dbhost 23 | url_show_series 24 | url_jsoc_info 25 | url_jsoc_fetch 26 | url_check_address 27 | url_show_series_wrapper 28 | encoding 29 | http_download_baseurl 30 | 31 | Parameters 32 | ---------- 33 | name : str 34 | Server configuration name. 35 | config : dict 36 | Dictionary containing configuration entries (see below for a 37 | list of available entries). 38 | """ 39 | 40 | _valid_keys = ( 41 | "name", 42 | "cgi_baseurl", 43 | "cgi_show_series", 44 | "cgi_jsoc_info", 45 | "cgi_jsoc_fetch", 46 | "cgi_check_address", 47 | "cgi_show_series_wrapper", 48 | "show_series_wrapper_dbhost", 49 | "url_show_series", 50 | "url_jsoc_info", 51 | "url_jsoc_fetch", 52 | "url_check_address", 53 | "url_show_series_wrapper", 54 | "encoding", 55 | "http_download_baseurl", 56 | ) 57 | 58 | def __init__(self, config=None, **kwargs): 59 | self._d = d = config.copy() if config is not None else {} 60 | d.update(kwargs) 61 | 62 | for k in d: 63 | if k not in self._valid_keys: 64 | raise ValueError(f"Invalid server config key: {k}") 65 | 66 | if "name" not in d: 67 | raise ValueError('Server config entry "name" is missing') 68 | 69 | # encoding defaults to latin1 70 | if "encoding" not in d: 71 | d["encoding"] = "latin1" 72 | 73 | # Generate URL entries from CGI entries, if cgi_baseurl exists and 74 | # the specific URL entry is not already set. 75 | if "cgi_baseurl" in d: 76 | cgi_baseurl = d["cgi_baseurl"] 77 | cgi_keys = [k for k in self._valid_keys if k.startswith("cgi") and k != "cgi_baseurl"] 78 | for k in cgi_keys: 79 | url_key = f"url{k[3:]}" 80 | cgi_value = d.get(k) 81 | if d.get(url_key) is None and cgi_value is not None: 82 | d[url_key] = urljoin(cgi_baseurl, cgi_value) 83 | 84 | def __repr__(self): 85 | return f"" 86 | 87 | def __dir__(self): 88 | return dir(type(self)) + list(self.__dict__.keys()) + list(self._valid_keys) 89 | 90 | def __getattr__(self, name): 91 | if name in self._valid_keys: 92 | return self._d.get(name) 93 | return object.__getattribute__(self, name) 94 | 95 | def __setattr__(self, name, value): 96 | if name in self._valid_keys: 97 | if not isinstance(value, str): 98 | raise ValueError(f"{name} config value must be a string") 99 | self._d[name] = value 100 | else: 101 | object.__setattr__(self, name, value) 102 | 103 | def copy(self): 104 | return ServerConfig(self._d) 105 | 106 | def to_dict(self): 107 | return self._d 108 | 109 | def check_supported(self, op): 110 | """ 111 | Check if an operation is supported by the server. 112 | """ 113 | if op == "series": 114 | return (self.cgi_show_series is not None) or (self.cgi_show_series_wrapper is not None) 115 | if op == "info": 116 | return self.cgi_jsoc_info is not None 117 | if op == "query": 118 | return self.cgi_jsoc_info is not None 119 | if op == "email": 120 | return self.cgi_check_address is not None 121 | if op == "export": 122 | return (self.cgi_jsoc_info is not None) and (self.cgi_jsoc_fetch is not None) 123 | raise ValueError(f"Unknown operation: {op!r}") 124 | 125 | 126 | def register_server(config): 127 | """ 128 | Register a server configuration. 129 | """ 130 | global _server_configs 131 | name = config.name.lower() 132 | if name in _server_configs: 133 | raise RuntimeError(f"ServerConfig {name} already registered") 134 | _server_configs[config.name.lower()] = config 135 | 136 | 137 | # Registered servers 138 | _server_configs = {} 139 | 140 | # Register public JSOC DRMS server. 141 | register_server( 142 | ServerConfig( 143 | name="JSOC", 144 | cgi_baseurl="http://jsoc.stanford.edu/cgi-bin/ajax/", 145 | cgi_show_series="show_series", 146 | cgi_jsoc_info="jsoc_info", 147 | cgi_jsoc_fetch="jsoc_fetch", 148 | cgi_check_address="checkAddress.sh", 149 | cgi_show_series_wrapper="showextseries", 150 | show_series_wrapper_dbhost="hmidb2", 151 | http_download_baseurl="http://jsoc.stanford.edu/", 152 | ), 153 | ) 154 | 155 | # Register KIS DRMS server. 156 | register_server( 157 | ServerConfig( 158 | name="KIS", 159 | cgi_baseurl="http://drms.leibniz-kis.de/cgi-bin/", 160 | cgi_show_series="show_series", 161 | cgi_jsoc_info="jsoc_info", 162 | ), 163 | ) 164 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ### Python: https://raw.githubusercontent.com/github/gitignore/master/Python.gitignore 2 | 3 | # Byte-compiled / optimized / DLL files 4 | __pycache__/ 5 | *.py[cod] 6 | *$py.class 7 | tmp/ 8 | 9 | # C extensions 10 | *.so 11 | 12 | # Distribution / packaging 13 | .Python 14 | pip-wheel-metadata/ 15 | build/ 16 | develop-eggs/ 17 | dist/ 18 | downloads/ 19 | eggs/ 20 | .eggs/ 21 | lib/ 22 | lib64/ 23 | parts/ 24 | sdist/ 25 | var/ 26 | wheels/ 27 | share/python-wheels/ 28 | *.egg-info/ 29 | .installed.cfg 30 | *.egg 31 | MANIFEST 32 | drms/_version.py 33 | # PyInstaller 34 | # Usually these files are written by a python script from a template 35 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 36 | *.manifest 37 | *.spec 38 | 39 | # Installer logs 40 | pip-log.txt 41 | pip-delete-this-directory.txt 42 | 43 | # Unit test / coverage reports 44 | htmlcov/ 45 | .tox/ 46 | .nox/ 47 | .coverage 48 | .coverage.* 49 | .cache 50 | nosetests.xml 51 | coverage.xml 52 | *.cover 53 | *.py,cover 54 | .hypothesis/ 55 | .pytest_cache/ 56 | cover/ 57 | 58 | # Translations 59 | *.mo 60 | *.pot 61 | 62 | # Django stuff: 63 | *.log 64 | local_settings.py 65 | db.sqlite3 66 | db.sqlite3-journal 67 | 68 | # Flask stuff: 69 | instance/ 70 | .webassets-cache 71 | 72 | # Scrapy stuff: 73 | .scrapy 74 | 75 | # Sphinx documentation 76 | docs/_build/ 77 | # automodapi 78 | docs/api 79 | docs/sg_execution_times.rst 80 | 81 | # PyBuilder 82 | .pybuilder/ 83 | target/ 84 | 85 | # Jupyter Notebook 86 | .ipynb_checkpoints 87 | 88 | # IPython 89 | profile_default/ 90 | ipython_config.py 91 | 92 | # pyenv 93 | # For a library or package, you might want to ignore these files since the code is 94 | # intended to run in multiple environments; otherwise, check them in: 95 | # .python-version 96 | 97 | # pipenv 98 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 99 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 100 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 101 | # install all needed dependencies. 102 | #Pipfile.lock 103 | 104 | # pdm 105 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 106 | #pdm.lock 107 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 108 | # in version control. 109 | # https://pdm.fming.dev/#use-with-ide 110 | .pdm.toml 111 | 112 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 113 | __pypackages__/ 114 | 115 | # Celery stuff 116 | celerybeat-schedule 117 | celerybeat.pid 118 | 119 | # SageMath parsed files 120 | *.sage.py 121 | 122 | # Environments 123 | .env 124 | .venv 125 | env/ 126 | venv/ 127 | ENV/ 128 | env.bak/ 129 | venv.bak/ 130 | 131 | # Rope project settings 132 | .ropeproject 133 | 134 | # mkdocs documentation 135 | /site 136 | 137 | # mypy 138 | .mypy_cache/ 139 | 140 | # Pyre type checker 141 | .pyre/ 142 | 143 | # IDE 144 | # PyCharm 145 | .idea 146 | 147 | # Spyder project settings 148 | .spyderproject 149 | .spyproject 150 | 151 | ### VScode: https://raw.githubusercontent.com/github/gitignore/master/Global/VisualStudioCode.gitignore 152 | .vscode/* 153 | .vs/* 154 | 155 | ### https://raw.github.com/github/gitignore/master/Global/OSX.gitignore 156 | .DS_Store 157 | .AppleDouble 158 | .LSOverride 159 | 160 | # Icon must ends with two \r. 161 | Icon 162 | 163 | # Thumbnails 164 | ._* 165 | 166 | # Files that might appear on external disk 167 | .Spotlight-V100 168 | .Trashes 169 | 170 | ### Linux: https://raw.githubusercontent.com/github/gitignore/master/Global/Linux.gitignore 171 | *~ 172 | 173 | # temporary files which can be created if a process still has a handle open of a deleted file 174 | .fuse_hidden* 175 | 176 | # KDE directory preferences 177 | .directory 178 | 179 | # Linux trash folder which might appear on any partition or disk 180 | .Trash-* 181 | 182 | # .nfs files are created when an open file is removed but is still being accessed 183 | .nfs* 184 | 185 | # pytype static type analyzer 186 | .pytype/ 187 | 188 | # General 189 | .DS_Store 190 | .AppleDouble 191 | .LSOverride 192 | 193 | # Icon must end with two \r 194 | Icon 195 | 196 | 197 | # Thumbnails 198 | ._* 199 | 200 | # Files that might appear in the root of a volume 201 | .DocumentRevisions-V100 202 | .fseventsd 203 | .Spotlight-V100 204 | .TemporaryItems 205 | .Trashes 206 | .VolumeIcon.icns 207 | .com.apple.timemachine.donotpresent 208 | 209 | # Directories potentially created on remote AFP share 210 | .AppleDB 211 | .AppleDesktop 212 | Network Trash Folder 213 | Temporary Items 214 | .apdisk 215 | 216 | ### Windows: https://raw.githubusercontent.com/github/gitignore/master/Global/Windows.gitignore 217 | 218 | # Windows thumbnail cache files 219 | Thumbs.db 220 | ehthumbs.db 221 | ehthumbs_vista.db 222 | 223 | # Dump file 224 | *.stackdump 225 | 226 | # Folder config file 227 | [Dd]esktop.ini 228 | 229 | # Recycle Bin used on file shares 230 | $RECYCLE.BIN/ 231 | 232 | # Windows Installer files 233 | *.cab 234 | *.msi 235 | *.msix 236 | *.msm 237 | *.msp 238 | 239 | # Windows shortcuts 240 | *.lnk 241 | 242 | ### Extra Python Items and SunPy Specific 243 | .hypothesis 244 | .pytest_cache 245 | docs/_build 246 | docs/api/ 247 | docs/generated 248 | docs/whatsnew/latest_changelog.txt 249 | drms/_version.py 250 | examples/**/*.asdf 251 | examples/**/*.csv 252 | examples/jsoc.stanford.edu/ 253 | jsoc.stanford.edu/ 254 | docs/sg_execution_times.rst 255 | 256 | # Release script 257 | .github_cache 258 | 259 | # Misc Stuff 260 | .history 261 | *.orig 262 | .tmp 263 | node_modules/ 264 | package-lock.json 265 | package.json 266 | .prettierrc 267 | 268 | # Log files generated by 'vagrant up' 269 | *.log 270 | -------------------------------------------------------------------------------- /drms/tests/test_config.py: -------------------------------------------------------------------------------- 1 | import pytest 2 | 3 | from drms.config import ServerConfig, _server_configs, register_server 4 | 5 | 6 | def test_create_config_basic(): 7 | cfg = ServerConfig(name="TEST") 8 | valid_keys = ServerConfig._valid_keys 9 | assert "name" in valid_keys 10 | assert "encoding" in valid_keys 11 | for k in valid_keys: 12 | v = getattr(cfg, k) 13 | if k == "name": 14 | assert v == "TEST" 15 | elif k == "encoding": 16 | assert v == "latin1" 17 | else: 18 | assert v is None 19 | assert repr(cfg) == "" 20 | 21 | 22 | def test_create_config_missing_name(): 23 | with pytest.raises(ValueError, match='Server config entry "name" is missing'): 24 | ServerConfig() 25 | 26 | 27 | def test_copy_config(): 28 | cfg = ServerConfig(name="TEST") 29 | assert cfg.name == "TEST" 30 | 31 | cfg2 = cfg.copy() 32 | assert cfg2 is not cfg 33 | assert cfg2.name == "TEST" 34 | 35 | cfg.name = "MUH" 36 | assert cfg.name != cfg2.name 37 | 38 | 39 | def test_register_server(): 40 | cfg = ServerConfig(name="TEST") 41 | 42 | assert "test" not in _server_configs 43 | register_server(cfg) 44 | assert "test" in _server_configs 45 | 46 | del _server_configs["test"] 47 | assert "test" not in _server_configs 48 | 49 | 50 | def test_register_server_existing(): 51 | assert "jsoc" in _server_configs 52 | cfg = ServerConfig(name="jsoc") 53 | with pytest.raises(RuntimeError): 54 | register_server(cfg) 55 | assert "jsoc" in _server_configs 56 | 57 | 58 | def test_config_jsoc(): 59 | assert "jsoc" in _server_configs 60 | 61 | cfg = _server_configs["jsoc"] 62 | assert cfg.name.lower() == "jsoc" 63 | assert isinstance(cfg.encoding, str) 64 | assert isinstance(cfg.cgi_show_series, str) 65 | assert isinstance(cfg.cgi_jsoc_info, str) 66 | assert isinstance(cfg.cgi_jsoc_fetch, str) 67 | assert isinstance(cfg.cgi_check_address, str) 68 | assert isinstance(cfg.cgi_show_series_wrapper, str) 69 | assert isinstance(cfg.show_series_wrapper_dbhost, str) 70 | assert cfg.http_download_baseurl.startswith("http://") 71 | 72 | baseurl = cfg.cgi_baseurl 73 | assert baseurl.startswith("http://") 74 | assert cfg.url_show_series.startswith(baseurl) 75 | assert cfg.url_jsoc_info.startswith(baseurl) 76 | assert cfg.url_jsoc_fetch.startswith(baseurl) 77 | assert cfg.url_check_address.startswith(baseurl) 78 | assert cfg.url_show_series_wrapper.startswith(baseurl) 79 | 80 | 81 | def test_config_kis(): 82 | assert "kis" in _server_configs 83 | cfg = _server_configs["kis"] 84 | 85 | assert cfg.name.lower() == "kis" 86 | assert isinstance(cfg.encoding, str) 87 | 88 | assert isinstance(cfg.cgi_show_series, str) 89 | assert isinstance(cfg.cgi_jsoc_info, str) 90 | assert cfg.cgi_jsoc_fetch is None 91 | assert cfg.cgi_check_address is None 92 | assert cfg.cgi_show_series_wrapper is None 93 | assert cfg.show_series_wrapper_dbhost is None 94 | assert cfg.http_download_baseurl is None 95 | 96 | baseurl = cfg.cgi_baseurl 97 | assert baseurl.startswith("http://") 98 | assert cfg.url_show_series.startswith(baseurl) 99 | assert cfg.url_jsoc_info.startswith(baseurl) 100 | assert cfg.url_jsoc_fetch is None 101 | assert cfg.url_check_address is None 102 | assert cfg.url_show_series_wrapper is None 103 | 104 | 105 | @pytest.mark.parametrize( 106 | ("server_name", "operation", "expected"), 107 | [ 108 | ("jsoc", "series", True), 109 | ("jsoc", "info", True), 110 | ("jsoc", "query", True), 111 | ("jsoc", "email", True), 112 | ("jsoc", "export", True), 113 | ("kis", "series", True), 114 | ("kis", "info", True), 115 | ("kis", "query", True), 116 | ("kis", "email", False), 117 | ("kis", "export", False), 118 | ], 119 | ) 120 | def test_supported(server_name, operation, expected): 121 | cfg = _server_configs[server_name] 122 | assert cfg.check_supported(operation) == expected 123 | 124 | 125 | @pytest.mark.parametrize( 126 | ("server_name", "operation"), 127 | [("jsoc", "bar"), ("kis", "foo")], 128 | ) 129 | def test_supported_invalid_operation(server_name, operation): 130 | cfg = _server_configs[server_name] 131 | with pytest.raises(ValueError, match="Unknown operation:"): 132 | cfg.check_supported(operation) 133 | 134 | 135 | def test_create_config_invalid_key(): 136 | with pytest.raises(ValueError, match="Invalid server config key: foo"): 137 | ServerConfig(foo="bar") 138 | 139 | 140 | def test_getset_attr(): 141 | cfg = ServerConfig(name="TEST") 142 | assert cfg.name == "TEST" 143 | assert cfg.__dict__ == {"_d": {"encoding": "latin1", "name": "TEST"}} 144 | with pytest.raises(AttributeError, match="'ServerConfig' object has no attribute 'foo'"): 145 | _ = cfg.foo 146 | cfg.name = "NewTest" 147 | assert cfg.name == "NewTest" 148 | with pytest.raises(ValueError, match="name config value must be a string"): 149 | cfg.name = 123 150 | cfg.__sizeof__ = 127 151 | assert cfg.__sizeof__ == 127 152 | 153 | 154 | def test_to_dict(): 155 | cfg = ServerConfig(name="TEST") 156 | _dict = cfg.to_dict() 157 | assert isinstance(_dict, dict) 158 | assert _dict == {"encoding": "latin1", "name": "TEST"} 159 | 160 | 161 | def test_inbuilt_dir(): 162 | cfg = ServerConfig(name="TEST") 163 | valid_keys = ServerConfig._valid_keys 164 | list_attr = dir(cfg) 165 | assert isinstance(list_attr, list) 166 | assert set(dir(object)).issubset(set(list_attr)) 167 | assert set(valid_keys).issubset(set(list_attr)) 168 | assert set(cfg.__dict__.keys()).issubset(set(list_attr)) 169 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file does only contain a selection of the most common options. For a 4 | # full list see the documentation: 5 | # http://www.sphinx-doc.org/en/master/config 6 | 7 | import os 8 | import datetime 9 | from pathlib import Path 10 | 11 | from sunpy_sphinx_theme import PNG_ICON 12 | 13 | from packaging.version import Version 14 | 15 | # -- Project information ----------------------------------------------------- 16 | 17 | # The full version, including alpha/beta/rc tags 18 | from drms import __version__ 19 | 20 | _version = Version(__version__) 21 | version = release = str(_version) 22 | # Avoid "post" appearing in version string in rendered docs 23 | if _version.is_postrelease: 24 | version = release = _version.base_version 25 | # Avoid long githashes in rendered Sphinx docs 26 | elif _version.is_devrelease: 27 | version = release = f"{_version.base_version}.dev{_version.dev}" 28 | is_development = _version.is_devrelease 29 | is_release = not (_version.is_prerelease or _version.is_devrelease) 30 | 31 | project = "drms" 32 | author = "The SunPy Community" 33 | copyright = f"{datetime.datetime.now().year}, {author}" # noqa: A001 34 | 35 | # -- General configuration --------------------------------------------------- 36 | 37 | # Wrap large function/method signatures 38 | maximum_signature_line_length = 80 39 | 40 | # Add any Sphinx extension module names here, as strings. They can be 41 | # extensions coming with Sphinx (named "sphinx.ext.*") or your custom 42 | # ones. 43 | extensions = [ 44 | "hoverxref.extension", 45 | "sphinx_copybutton", 46 | "sphinx_gallery.gen_gallery", 47 | "sphinx.ext.autodoc", 48 | "sphinx.ext.coverage", 49 | "sphinx.ext.doctest", 50 | "sphinx.ext.inheritance_diagram", 51 | "sphinx.ext.intersphinx", 52 | "sphinx.ext.napoleon", 53 | "sphinx.ext.todo", 54 | "sphinx.ext.viewcode", 55 | "sphinx.ext.mathjax", 56 | "sphinx_automodapi.automodapi", 57 | "sphinx_automodapi.smart_resolver", 58 | "sphinx_changelog", 59 | ] 60 | 61 | # Add any paths that contain templates here, relative to this directory. 62 | # templates_path = ["_templates"] 63 | 64 | # List of patterns, relative to source directory, that match files and 65 | # directories to ignore when looking for source files. 66 | # This pattern also affects html_static_path and html_extra_path. 67 | automodapi_toctreedirnm = "generated/api" 68 | exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] 69 | 70 | # The suffix(es) of source filenames. 71 | source_suffix = {".rst": "restructuredtext"} 72 | 73 | # The master toctree document. 74 | master_doc = "index" 75 | 76 | # Treat everything in single ` as a Python reference. 77 | default_role = "py:obj" 78 | 79 | # -- Options for intersphinx extension --------------------------------------- 80 | 81 | intersphinx_mapping = { 82 | "python": ( 83 | "https://docs.python.org/3/", 84 | (None, "http://www.astropy.org/astropy-data/intersphinx/python3.inv"), 85 | ), 86 | "numpy": ( 87 | "https://numpy.org/doc/stable/", 88 | (None, "http://www.astropy.org/astropy-data/intersphinx/numpy.inv"), 89 | ), 90 | "scipy": ( 91 | "https://docs.scipy.org/doc/scipy/reference/", 92 | (None, "http://www.astropy.org/astropy-data/intersphinx/scipy.inv"), 93 | ), 94 | "matplotlib": ("https://matplotlib.org/stable", None), 95 | "astropy": ("https://docs.astropy.org/en/stable/", None), 96 | "pandas": ("https://pandas.pydata.org/pandas-docs/stable/", None), 97 | "sunpy": ("https://docs.sunpy.org/en/stable/", None), 98 | } 99 | 100 | # -- Options for HTML output ------------------------------------------------- 101 | 102 | # The theme to use for HTML and HTML Help pages. See the documentation for 103 | # a list of builtin themes. 104 | html_theme = "sunpy" 105 | 106 | # Render inheritance diagrams in SVG 107 | graphviz_output_format = "svg" 108 | 109 | graphviz_dot_args = [ 110 | "-Nfontsize=10", 111 | "-Nfontname=Helvetica Neue, Helvetica, Arial, sans-serif", 112 | "-Efontsize=10", 113 | "-Efontname=Helvetica Neue, Helvetica, Arial, sans-serif", 114 | "-Gfontsize=10", 115 | "-Gfontname=Helvetica Neue, Helvetica, Arial, sans-serif", 116 | ] 117 | 118 | # Add any paths that contain custom static files (such as style sheets) here, 119 | # relative to this directory. They are copied after the builtin static files, 120 | # so a file named "default.css" will overwrite the builtin "default.css". 121 | # html_static_path = ["_static"] 122 | 123 | # By default, when rendering docstrings for classes, sphinx.ext.autodoc will 124 | # make docs with the class-level docstring and the class-method docstrings, 125 | # but not the __init__ docstring, which often contains the parameters to 126 | # class constructors across the scientific Python ecosystem. The option below 127 | # will append the __init__ docstring to the class-level docstring when rendering 128 | # the docs. For more options, see: 129 | # https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autoclass_content 130 | autoclass_content = "both" 131 | 132 | # -- Other options ---------------------------------------------------------- 133 | 134 | # JSOC email os env 135 | # see https://github.com/sunpy/sunpy/wiki/Home:-JSOC 136 | os.environ["JSOC_EMAIL"] = "jsoc@sunpy.org" 137 | 138 | # -- Options for hoverxref ----------------------------------------------------- 139 | 140 | if os.environ.get("READTHEDOCS"): 141 | hoverxref_api_host = "https://readthedocs.org" 142 | if os.environ.get("PROXIED_API_ENDPOINT"): 143 | # Use the proxied API endpoint 144 | # - A RTD thing to avoid a CSRF block when docs are using a 145 | # custom domain 146 | hoverxref_api_host = "/_" 147 | 148 | hoverxref_tooltip_maxwidth = 600 # RTD main window is 696px 149 | hoverxref_auto_ref = True 150 | hoverxref_mathjax = True 151 | hoverxref_domains = ["py"] 152 | hoverxref_role_types = { 153 | # roles with py domain 154 | "attr": "tooltip", 155 | "class": "tooltip", 156 | "const": "tooltip", 157 | "data": "tooltip", 158 | "exc": "tooltip", 159 | "func": "tooltip", 160 | "meth": "tooltip", 161 | "mod": "tooltip", 162 | "obj": "tooltip", 163 | # roles with std domain 164 | "confval": "tooltip", 165 | "hoverxref": "tooltip", 166 | "ref": "tooltip", 167 | "term": "tooltip", 168 | } 169 | 170 | # -- Options for sphinx-copybutton --------------------------------------------- 171 | 172 | # Python Repl + continuation, Bash, ipython and qtconsole + continuation, jupyter-console + continuation 173 | copybutton_prompt_text = r">>> |\.\.\. |\$ |In \[\d*\]: | {2,5}\.\.\.: | {5,8}: " 174 | copybutton_prompt_is_regexp = True 175 | 176 | # -- Sphinx Gallery ------------------------------------------------------------ 177 | 178 | sphinx_gallery_conf = { 179 | "backreferences_dir": Path("generated") / "modules", 180 | "filename_pattern": "^((?!skip_).)*$", 181 | "examples_dirs": Path("..") / "examples", 182 | "within_subsection_order": "ExampleTitleSortKey", 183 | "gallery_dirs": Path("generated") / "gallery", 184 | "default_thumb_file": PNG_ICON, 185 | "abort_on_example_error": False, 186 | "plot_gallery": "True", 187 | "remove_config_comments": True, 188 | "doc_module": ("sunpy"), 189 | "only_warn_on_example_error": True, 190 | "matplotlib_animations": True, 191 | } 192 | -------------------------------------------------------------------------------- /docs/Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line. 5 | SPHINXOPTS = 6 | SPHINXBUILD = sphinx-build 7 | PAPER = 8 | BUILDDIR = build 9 | 10 | # User-friendly check for sphinx-build 11 | ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) 12 | $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don\'t have Sphinx installed, grab it from http://sphinx-doc.org/) 13 | endif 14 | 15 | # Internal variables. 16 | PAPEROPT_a4 = -D latex_paper_size=a4 17 | PAPEROPT_letter = -D latex_paper_size=letter 18 | ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 19 | # the i18n builder cannot share the environment and doctrees with the others 20 | I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . 21 | 22 | .PHONY: help 23 | help: 24 | @echo "Please use \`make ' where is one of" 25 | @echo " html to make standalone HTML files" 26 | @echo " dirhtml to make HTML files named index.html in directories" 27 | @echo " singlehtml to make a single large HTML file" 28 | @echo " pickle to make pickle files" 29 | @echo " json to make JSON files" 30 | @echo " htmlhelp to make HTML files and a HTML help project" 31 | @echo " qthelp to make HTML files and a qthelp project" 32 | @echo " applehelp to make an Apple Help Book" 33 | @echo " devhelp to make HTML files and a Devhelp project" 34 | @echo " epub to make an epub" 35 | @echo " epub3 to make an epub3" 36 | @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" 37 | @echo " latexpdf to make LaTeX files and run them through pdflatex" 38 | @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" 39 | @echo " text to make text files" 40 | @echo " man to make manual pages" 41 | @echo " texinfo to make Texinfo files" 42 | @echo " info to make Texinfo files and run them through makeinfo" 43 | @echo " gettext to make PO message catalogs" 44 | @echo " changes to make an overview of all changed/added/deprecated items" 45 | @echo " xml to make Docutils-native XML files" 46 | @echo " pseudoxml to make pseudoxml-XML files for display purposes" 47 | @echo " linkcheck to check all external links for integrity" 48 | @echo " doctest to run all doctests embedded in the documentation (if enabled)" 49 | @echo " coverage to run coverage check of the documentation (if enabled)" 50 | @echo " dummy to check syntax errors of document sources" 51 | 52 | .PHONY: clean 53 | clean: 54 | rm -rf $(BUILDDIR)/* generated 55 | 56 | .PHONY: html 57 | html: 58 | $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html 59 | @echo 60 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." 61 | 62 | .PHONY: dirhtml 63 | dirhtml: 64 | $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml 65 | @echo 66 | @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." 67 | 68 | .PHONY: singlehtml 69 | singlehtml: 70 | $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml 71 | @echo 72 | @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." 73 | 74 | .PHONY: pickle 75 | pickle: 76 | $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle 77 | @echo 78 | @echo "Build finished; now you can process the pickle files." 79 | 80 | .PHONY: json 81 | json: 82 | $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json 83 | @echo 84 | @echo "Build finished; now you can process the JSON files." 85 | 86 | .PHONY: htmlhelp 87 | htmlhelp: 88 | $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp 89 | @echo 90 | @echo "Build finished; now you can run HTML Help Workshop with the" \ 91 | ".hhp project file in $(BUILDDIR)/htmlhelp." 92 | 93 | .PHONY: qthelp 94 | qthelp: 95 | $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp 96 | @echo 97 | @echo "Build finished; now you can run "qcollectiongenerator" with the" \ 98 | ".qhcp project file in $(BUILDDIR)/qthelp, like this:" 99 | @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/drms.qhcp" 100 | @echo "To view the help file:" 101 | @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/drms.qhc" 102 | 103 | .PHONY: applehelp 104 | applehelp: 105 | $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp 106 | @echo 107 | @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." 108 | @echo "N.B. You won't be able to view it unless you put it in" \ 109 | "~/Library/Documentation/Help or install it in your application" \ 110 | "bundle." 111 | 112 | .PHONY: devhelp 113 | devhelp: 114 | $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp 115 | @echo 116 | @echo "Build finished." 117 | @echo "To view the help file:" 118 | @echo "# mkdir -p $$HOME/.local/share/devhelp/drms" 119 | @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/drms" 120 | @echo "# devhelp" 121 | 122 | .PHONY: epub 123 | epub: 124 | $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub 125 | @echo 126 | @echo "Build finished. The epub file is in $(BUILDDIR)/epub." 127 | 128 | .PHONY: epub3 129 | epub3: 130 | $(SPHINXBUILD) -b epub3 $(ALLSPHINXOPTS) $(BUILDDIR)/epub3 131 | @echo 132 | @echo "Build finished. The epub3 file is in $(BUILDDIR)/epub3." 133 | 134 | .PHONY: latex 135 | latex: 136 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 137 | @echo 138 | @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." 139 | @echo "Run \`make' in that directory to run these through (pdf)latex" \ 140 | "(use \`make latexpdf' here to do that automatically)." 141 | 142 | .PHONY: latexpdf 143 | latexpdf: 144 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 145 | @echo "Running LaTeX files through pdflatex..." 146 | $(MAKE) -C $(BUILDDIR)/latex all-pdf 147 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 148 | 149 | .PHONY: latexpdfja 150 | latexpdfja: 151 | $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex 152 | @echo "Running LaTeX files through platex and dvipdfmx..." 153 | $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja 154 | @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." 155 | 156 | .PHONY: text 157 | text: 158 | $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text 159 | @echo 160 | @echo "Build finished. The text files are in $(BUILDDIR)/text." 161 | 162 | .PHONY: man 163 | man: 164 | $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man 165 | @echo 166 | @echo "Build finished. The manual pages are in $(BUILDDIR)/man." 167 | 168 | .PHONY: texinfo 169 | texinfo: 170 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 171 | @echo 172 | @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." 173 | @echo "Run \`make' in that directory to run these through makeinfo" \ 174 | "(use \`make info' here to do that automatically)." 175 | 176 | .PHONY: info 177 | info: 178 | $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo 179 | @echo "Running Texinfo files through makeinfo..." 180 | make -C $(BUILDDIR)/texinfo info 181 | @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." 182 | 183 | .PHONY: gettext 184 | gettext: 185 | $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale 186 | @echo 187 | @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." 188 | 189 | .PHONY: changes 190 | changes: 191 | $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes 192 | @echo 193 | @echo "The overview file is in $(BUILDDIR)/changes." 194 | 195 | .PHONY: linkcheck 196 | linkcheck: 197 | $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck 198 | @echo 199 | @echo "Link check complete; look for any errors in the above output " \ 200 | "or in $(BUILDDIR)/linkcheck/output.txt." 201 | 202 | .PHONY: doctest 203 | doctest: 204 | $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest 205 | @echo "Testing of doctests in the sources finished, look at the " \ 206 | "results in $(BUILDDIR)/doctest/output.txt." 207 | 208 | .PHONY: coverage 209 | coverage: 210 | $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage 211 | @echo "Testing of coverage in the sources finished, look at the " \ 212 | "results in $(BUILDDIR)/coverage/python.txt." 213 | 214 | .PHONY: xml 215 | xml: 216 | $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml 217 | @echo 218 | @echo "Build finished. The XML files are in $(BUILDDIR)/xml." 219 | 220 | .PHONY: pseudoxml 221 | pseudoxml: 222 | $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml 223 | @echo 224 | @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." 225 | 226 | .PHONY: dummy 227 | dummy: 228 | $(SPHINXBUILD) -b dummy $(ALLSPHINXOPTS) $(BUILDDIR)/dummy 229 | @echo 230 | @echo "Build finished. Dummy builder generates no files." 231 | -------------------------------------------------------------------------------- /drms/json.py: -------------------------------------------------------------------------------- 1 | import json as _json 2 | import socket 3 | from enum import Enum 4 | from urllib.parse import urlencode, quote_plus 5 | from urllib.request import HTTPError, urlopen 6 | 7 | from drms import logger 8 | from .config import ServerConfig, _server_configs 9 | from .utils import _split_arg, create_request_with_header 10 | 11 | __all__ = ["HttpJsonClient", "HttpJsonRequest", "JsocInfoConstants"] 12 | 13 | 14 | # TODO: When we support 3.11, we can use StrEnum instead of Enum 15 | class JsocInfoConstants(str, Enum): 16 | """ 17 | Constants for DRMS queries. 18 | """ 19 | 20 | all = "**ALL**" 21 | none = "**NONE**" 22 | recdir = "*recdir*" 23 | dirmtime = "*dirmtime*" 24 | logdir = "*logdir*" 25 | recnum = "*recnum*" 26 | sunum = "*sunum*" 27 | size = "*size*" 28 | online = "*online*" 29 | retain = "*retain*" 30 | archive = "*archive*" 31 | 32 | 33 | class HttpJsonRequest: 34 | """ 35 | Class for handling HTTP/JSON requests. 36 | 37 | Use `HttpJsonClient` to create an instance. 38 | """ 39 | 40 | def __init__(self, url, encoding, timeout=60): 41 | timeout = socket.getdefaulttimeout() or timeout 42 | self._encoding = encoding 43 | try: 44 | self._http = urlopen(create_request_with_header(url), timeout=timeout) 45 | except HTTPError as e: 46 | e.msg = f"Failed to open URL: {e.url} with {e.code} - {e.msg}" 47 | raise e 48 | self._data_str = None 49 | self._data = None 50 | 51 | def __repr__(self): 52 | return f"" 53 | 54 | @property 55 | def url(self): 56 | return self._http.url 57 | 58 | @property 59 | def raw_data(self): 60 | if self._data_str is None: 61 | self._data_str = self._http.read() 62 | return self._data_str 63 | 64 | @property 65 | def data(self): 66 | if self._data is None: 67 | self._data = _json.loads(self.raw_data.decode(self._encoding)) 68 | return self._data 69 | 70 | 71 | class HttpJsonClient: 72 | """ 73 | HTTP/JSON communication with the DRMS server CGIs. 74 | 75 | Parameters 76 | ---------- 77 | server : str or drms.config.ServerConfig 78 | Registered server ID or ServerConfig instance. 79 | Defaults to JSOC. 80 | """ 81 | 82 | def __init__(self, server="jsoc"): 83 | if isinstance(server, ServerConfig): 84 | self._server = server 85 | else: 86 | self._server = _server_configs[server.lower()] 87 | 88 | def __repr__(self): 89 | return f"" 90 | 91 | def _json_request(self, url): 92 | logger.debug(f"URL for request: {url}") 93 | return HttpJsonRequest(url, self._server.encoding) 94 | 95 | @property 96 | def server(self): 97 | return self._server 98 | 99 | def show_series(self, ds_filter=None): 100 | """ 101 | List available data series. 102 | 103 | Parameters 104 | ---------- 105 | ds_filter : str, None, optional 106 | Name filter regexp. 107 | Default is None, which returns all available series. 108 | 109 | Returns 110 | ------- 111 | result : dict 112 | """ 113 | query = "?" if ds_filter is not None else "" 114 | if ds_filter is not None: 115 | query += urlencode({"filter": ds_filter}) 116 | req = self._json_request(self._server.url_show_series + query) 117 | return req.data 118 | 119 | def show_series_wrapper(self, ds_filter=None, *, info=False): 120 | """ 121 | List available data series. 122 | 123 | This is an alternative to show_series, which needs to be used 124 | to get a list of all available series provided by JSOC. There 125 | is currently no support for retrieving primekeys using this 126 | CGI. 127 | 128 | Parameters 129 | ---------- 130 | ds_filter : str, None, optional 131 | Name filter regexp. 132 | Default is None, which returns all available series. 133 | info : bool 134 | If False (default), the result only contains series names. 135 | If set to True, the result includes a description for each 136 | series. 137 | 138 | Returns 139 | ------- 140 | result : dict 141 | """ 142 | query_args = {"dbhost": self._server.show_series_wrapper_dbhost} 143 | if ds_filter is not None: 144 | query_args["filter"] = ds_filter 145 | if info: 146 | query_args["info"] = "1" 147 | query = f"?{urlencode(query_args)}" 148 | req = self._json_request(self._server.url_show_series_wrapper + query) 149 | return req.data 150 | 151 | def series_struct(self, ds): 152 | """ 153 | Get information about the content of a data series. 154 | 155 | Parameters 156 | ---------- 157 | ds : str 158 | Name of the data series. 159 | 160 | Returns 161 | ------- 162 | result : dict 163 | Dictionary containing information about the data series. 164 | """ 165 | query = f"?{urlencode({'op': 'series_struct', 'ds': ds})}" 166 | req = self._json_request(self._server.url_jsoc_info + query) 167 | return req.data 168 | 169 | def rs_summary(self, ds): 170 | """ 171 | Get summary (i.e. count) of a given record set. 172 | 173 | Parameters 174 | ---------- 175 | ds : str 176 | Record set query (only one series). 177 | 178 | Returns 179 | ------- 180 | result : dict 181 | Dictionary containing 'count', 'status' and 'runtime'. 182 | """ 183 | query = f"?{urlencode({'op': 'rs_summary', 'ds': ds})}" 184 | req = self._json_request(self._server.url_jsoc_info + query) 185 | return req.data 186 | 187 | def rs_list(self, ds, *, key=None, seg=None, link=None, recinfo=False, n=None, uid=None): 188 | """ 189 | Get detailed information about a record set. 190 | 191 | Parameters 192 | ---------- 193 | ds : str 194 | Record set query. 195 | key : str, list or None 196 | List of requested keywords, optional. 197 | seg : str, list or None 198 | List of requested segments, optional. 199 | link : str or None 200 | List of requested Links, optional. 201 | recinfo : bool 202 | Request record info for each record in the record set. 203 | n : int or None 204 | Record set limit. For positive values, the first n records 205 | of the record set are returned, for negative values the 206 | last abs(n) records. If set to None (default), no limit is 207 | applied. 208 | uid : str or None 209 | Session ID used when calling rs_list CGI, optional. 210 | 211 | Returns 212 | ------- 213 | result : dict 214 | Dictionary containing the requested record set information. 215 | """ 216 | if key is None and seg is None and link is None: 217 | raise ValueError("At least one key, seg or link must be specified") 218 | d = {"op": "rs_list", "ds": ds} 219 | if key is not None: 220 | d["key"] = ",".join(_split_arg(key)) 221 | if seg is not None: 222 | d["seg"] = ",".join(_split_arg(seg)) 223 | if link is not None: 224 | d["link"] = ",".join(_split_arg(link)) 225 | if recinfo: 226 | d["R"] = "1" 227 | if n is not None: 228 | d["n"] = f"{int(n)}" 229 | if uid is not None: 230 | d["userhandle"] = uid 231 | query = f"?{urlencode(d)}" 232 | req = self._json_request(self._server.url_jsoc_info + query) 233 | return req.data 234 | 235 | def check_address(self, email): 236 | """ 237 | Check if an email address is registered for export data requests. 238 | 239 | Parameters 240 | ---------- 241 | email : str 242 | Email address to be verified. 243 | 244 | Returns 245 | ------- 246 | result : dict 247 | Dictionary containing 'status' and 'msg'. 248 | 249 | Some status codes are: 250 | - 2: Email address is valid and registered 251 | - 4: Email address has neither been validated nor registered 252 | - -2: Not a valid email address 253 | """ 254 | query = "?" + urlencode({"address": quote_plus(email), "checkonly": "1"}) 255 | req = self._json_request(self._server.url_check_address + query) 256 | return req.data 257 | 258 | def exp_request(self, *args, **kwargs): 259 | """ 260 | Request data export. 261 | 262 | Parameters 263 | ---------- 264 | ds : str 265 | Data export record set query. 266 | notify : str 267 | Registered email address. 268 | method : str 269 | Export method. Supported methods are: 'url_quick', 'url', 270 | and 'url-tar'. Default is 'url_quick'. 271 | protocol : str 272 | Export protocol. Supported protocols are: 'as-is', 'fits', 273 | 'jpg', 'mpg' and 'mp4'. Default is 'as-is'. 274 | protocol_args : dict or None 275 | Extra protocol arguments for protocols 'jpg', 'mpg' and 276 | 'mp4'. Valid arguments are: 'ct', 'scaling', 'min', 'max' 277 | and 'size'. 278 | filenamefmt : str, None 279 | Custom filename format string for exported files. This is 280 | ignored for 'url_quick'/'as-is' data exports. 281 | process : `dict`, None 282 | Dictionary of processing commands. Each entry is also a `dict` 283 | containing all of the applicable options for that processing 284 | command. 285 | n : int or None 286 | Limits the number of records requested. For positive 287 | values, the first n records of the record set are returned, 288 | for negative values the last abs(n) records. If set to None 289 | (default), no limit is applied. 290 | requester : str, None or bool 291 | Export user ID. Default is None, in which case the user 292 | name is determined from the email address. If set to False, 293 | the requester argument will be omitted in the export 294 | request. 295 | 296 | Returns 297 | ------- 298 | result : dict 299 | Dictionary containing the server response to the export 300 | request. 301 | """ 302 | req = self._json_request(self._exp_request_url(*args, **kwargs)) 303 | return req.data 304 | 305 | def _exp_request_url( 306 | self, 307 | ds, 308 | notify, 309 | *, 310 | method="url_quick", 311 | protocol="as-is", 312 | protocol_args=None, 313 | filenamefmt=None, 314 | n=None, 315 | process=None, 316 | requester=None, 317 | ): 318 | method = method.lower() 319 | method_list = ["url_quick", "url", "url-tar"] 320 | if method not in method_list: 321 | raise ValueError( 322 | "Method {} is not supported, valid methods are: {}".format( 323 | method, 324 | ", ".join(str(s) for s in method_list), 325 | ), 326 | ) 327 | 328 | protocol = protocol.lower() 329 | img_protocol_list = ["jpg", "mpg", "mp4"] 330 | protocol_list = ["as-is", "fits", *img_protocol_list] 331 | if protocol not in protocol_list: 332 | raise ValueError( 333 | "Protocol {} is not supported, valid protocols are: {}".format( 334 | protocol, 335 | ", ".join(str(s) for s in protocol_list), 336 | ), 337 | ) 338 | 339 | # method "url_quick" is meant to be used with "as-is", change method 340 | # to "url" if protocol is not "as-is" 341 | if method == "url_quick" and protocol != "as-is": 342 | method = "url" 343 | 344 | if protocol in img_protocol_list: 345 | extra_keys = {"ct": "grey.sao", "scaling": "MINMAX", "size": 1} 346 | if protocol_args is not None: 347 | for k, v in protocol_args.items(): 348 | if k.lower() == "ct": 349 | extra_keys["ct"] = v 350 | elif k == "scaling": 351 | extra_keys[k] = v 352 | elif k == "size": 353 | extra_keys[k] = int(v) 354 | elif k in ["min", "max"]: 355 | extra_keys[k] = float(v) 356 | else: 357 | raise ValueError(f"Unknown protocol argument: {k}") 358 | protocol += ",CT={ct},scaling={scaling},size={size}".format(**extra_keys) 359 | if "min" in extra_keys: 360 | protocol += f",min={extra_keys['min']:g}" 361 | if "max" in extra_keys: 362 | protocol += f",max={extra_keys['max']:g}" 363 | else: 364 | if protocol_args is not None: 365 | raise ValueError(f"protocol_args not supported for protocol {protocol}") 366 | 367 | d = { 368 | "op": "exp_request", 369 | "format": "json", 370 | "ds": ds, 371 | "notify": notify, 372 | "method": method, 373 | "protocol": protocol, 374 | } 375 | 376 | if filenamefmt is not None: 377 | d["filenamefmt"] = filenamefmt 378 | 379 | n = int(n) if n is not None else 0 380 | d["process=n"] = f"{n}" 381 | if process is not None: 382 | allowed_processes = [ 383 | "im_patch", 384 | "resize", 385 | "rebin", 386 | "aia_scale_aialev1", 387 | "aia_scale_orig", 388 | "aia_scale_other", 389 | "Maproj", 390 | "HmiB2ptr", 391 | ] 392 | process_strings = {} 393 | for p, opts in process.items(): 394 | if p not in allowed_processes: 395 | raise ValueError(f"{p} is not one of the allowed processing options: {allowed_processes}") 396 | process_strings[p] = ",".join([f"{k}={v}" for k, v in opts.items()]) 397 | processes = "|".join([f"{k},{v}" for k, v in process_strings.items()]) 398 | d["process=n"] = f"{d['process=n']}|{processes}" 399 | 400 | if requester is None: 401 | d["requester"] = notify.split("@")[0] 402 | elif requester is not False: 403 | d["requester"] = requester 404 | 405 | query = "?" + urlencode(d) 406 | return self._server.url_jsoc_fetch + query 407 | 408 | def exp_status(self, requestid): 409 | """ 410 | Query data export status. 411 | 412 | Parameters 413 | ---------- 414 | requestid : str 415 | Request identifier returned by exp_request. 416 | 417 | Returns 418 | ------- 419 | result : dict 420 | Dictionary containing the export request status. 421 | """ 422 | query = f"?{urlencode({'op': 'exp_status', 'requestid': requestid})}" 423 | req = self._json_request(self._server.url_jsoc_fetch + query) 424 | return req.data 425 | -------------------------------------------------------------------------------- /docs/tutorial.rst: -------------------------------------------------------------------------------- 1 | .. _drms-tutorial: 2 | 3 | ******** 4 | Tutorial 5 | ******** 6 | 7 | This tutorial gives an introduction on how to use the ``drms`` Python library. 8 | More detailed information on the different classes and functions can be found in the :ref:`API Reference `. 9 | 10 | Basic usage 11 | =========== 12 | 13 | We start with looking at data series that are available from `JSOC `__ and perform some basic DRMS queries to obtain keyword data (metadata) and segment file (data) locations. 14 | This is essentially what you can do on the `JSOC Lookdata `__ website. 15 | 16 | To be able to access the JSOC DRMS from Python, we first need to import the ``drms`` module and create an instance of the `~drms.client.Client` class: 17 | 18 | .. code-block:: python 19 | 20 | >>> import drms 21 | >>> client = drms.Client() # doctest: +REMOTE_DATA 22 | 23 | All available data series can be now retrieved by calling :meth:`drms.client.Client.series`. 24 | HMI series names start with ``"hmi."``, AIA series names with ``"aia."`` and the names of MDI series with ``"mdi."``. 25 | 26 | The first (optional) parameter of this method takes a regular expression that allows you to filter the result. 27 | If for example, you want to obtain a list of HMI series, with a name that start with the string ``"m_"``, you can write: 28 | 29 | .. code-block:: python 30 | 31 | >>> client.series(r'hmi\.m_') # doctest: +REMOTE_DATA 32 | ['hmi.M_45s', 'hmi.M_45s_dcon', 'hmi.M_720s', 'hmi.M_720s_dcon', 'hmi.M_720s_dconS', 'hmi.m_720s_mod', 'hmi.m_720s_nrt'] 33 | 34 | Keep in mind to escape the dot character (``'.'``), like it is shown in the example above, if you want to include it in your filter string. 35 | Also note that series names are handled in a case-insensitive way. 36 | 37 | DRMS records can be selected by creating a query string that contains a series name, followed by one or more fields, which are surrounded by square brackets. 38 | Each of those fields corresponds to a specific primekey, that is specified in the series definition. 39 | A complete set of primekeys represents a unique identifier for a record in that particular series. 40 | For more detailed information on building record set queries, including additional non-primekey fields, see the `JSOC Help `__ page about this. 41 | 42 | With the ``drms`` module you can use :meth:`drms.client.Client.pkeys` to obtain a list of all primekeys of a series: 43 | 44 | .. code-block:: python 45 | 46 | >>> client.pkeys('hmi.m_720s') # doctest: +REMOTE_DATA 47 | ['T_REC', 'CAMERA'] 48 | >>> client.pkeys('hmi.v_sht_modes') # doctest: +REMOTE_DATA 49 | ['T_START', 'LMIN', 'LMAX', 'NDT'] 50 | 51 | A list of all (regular) keywords can be obtained using :meth:`drms.client.Client.keys`. 52 | You can also use :meth:`drms.client.Client.info` to get more detailed information about a series: 53 | 54 | .. code-block:: python 55 | 56 | >>> series_info = client.info('hmi.v_sht_2drls') # doctest: +REMOTE_DATA 57 | >>> series_info.segments # doctest: +REMOTE_DATA 58 | type units protocol dims note 59 | name 60 | split string none generic calculated splittings 61 | rot string none generic rotation profile 62 | err string none generic errors 63 | mesh string none generic radial grid points 64 | parms string none generic input parameters 65 | log string none generic standard output 66 | 67 | All table-like structures, returned by routines in the ``drms`` module, are `Pandas DataFrames `__. 68 | If you are new to `Pandas `__, you should have a look at the introduction to `Pandas Data Structures `__. 69 | 70 | Record set queries, used to obtain keyword data and get the location of data segments, can be performed using :meth:`drms.client.Client.query`. 71 | To get, for example, the record time and the mean value for some of the HMI Dopplergrams that were recorded on April 1, 2016, together with the spacecraft's radial velocity in respect to the Sun, you can write: 72 | 73 | .. code-block:: python 74 | 75 | >>> query = client.query('hmi.v_45s[2016.04.01_TAI/1d@6h]', 76 | ... key='T_REC, DATAMEAN, OBS_VR') # doctest: +REMOTE_DATA 77 | >>> query # doctest: +REMOTE_DATA 78 | T_REC DATAMEAN OBS_VR 79 | 0 2016.04.01_00:00:00_TAI 3313.104980 3309.268006 80 | 1 2016.04.01_06:00:00_TAI 878.075195 887.864139 81 | 2 2016.04.01_12:00:00_TAI -2289.062500 -2284.690263 82 | 3 2016.04.01_18:00:00_TAI 128.609283 137.836168 83 | 84 | JSOC time strings can be converted to a naive `~datetime.datetime` representation using :meth:`drms.utils.to_datetime`: 85 | 86 | .. code-block:: python 87 | 88 | >>> timestamps = drms.to_datetime(query.T_REC) # doctest: +REMOTE_DATA 89 | >>> timestamps # doctest: +REMOTE_DATA 90 | 0 2016-04-01 00:00:00 91 | 1 2016-04-01 06:00:00 92 | 2 2016-04-01 12:00:00 93 | 3 2016-04-01 18:00:00 94 | Name: T_REC, dtype: datetime64[ns] 95 | 96 | For most of the HMI and MDI data sets, the `TAI `__ time standard is used which, in contrast to `UTC `__, does not make use of any leap seconds. 97 | The TAI standard is currently not supported by the Python standard libraries. 98 | If you need to convert timestamps between TAI and UTC, you can use `Astropy `__: 99 | 100 | .. code-block:: python 101 | 102 | >>> from astropy.time import Time 103 | >>> start_time = Time(timestamps[0], format='datetime', scale='tai') # doctest: +REMOTE_DATA 104 | >>> start_time # doctest: +REMOTE_DATA 105 |