├── .gitignore ├── README.md ├── alerce_logo_rgb_01.png ├── alercereaduser.json ├── alercereaduser_v2.json ├── alercereaduser_v3.json ├── alercereaduser_v4.json ├── api ├── ALeRCE AVRO-STAMP Access API.md ├── ALeRCE ZTF DB API.md └── README.md ├── example_data ├── CSAGN_coordinates.csv ├── MorphGal.dat ├── MorphGal_readme.txt ├── catalina.csv ├── ephs_153935.pkl ├── ephs_153935_20210301-20210402.pkl ├── watchlist.csv └── xmatch_test.csv ├── notebooks ├── ADASS_XXXII_Nov2022_tutorial │ ├── ALeRCE_AGN_starter.ipynb │ ├── ALeRCE_Client_starter.ipynb │ ├── ALeRCE_ML_Stamp_Classifier.ipynb │ ├── ALeRCE_SN_Starter.ipynb │ ├── README.md │ └── figures │ │ ├── SanchezSaez+2020_confusion.png │ │ ├── SanchezSaez+2020_recall.png │ │ ├── Villar+19.png │ │ ├── ml │ │ ├── alerce_explorer.png │ │ ├── alerce_explorer_object.png │ │ ├── alerce_frontend.png │ │ ├── betas.png │ │ ├── class_samples.png │ │ ├── cm.png │ │ ├── cnn_architecture.png │ │ ├── cnn_table.png │ │ ├── hierarchical_rf.png │ │ ├── sanchez_paper_confusion_matrix.png │ │ ├── sn_hunter.png │ │ └── taxonomy.png │ │ ├── recall_agn_qso_blazar_for_mag.png │ │ ├── recall_agn_qso_blazar_for_ndet.png │ │ └── taxonomy_tree.png ├── ALeRCE+externalcatalog_xmatch.ipynb ├── ALeRCE_AGN_Starter.ipynb ├── ALeRCE_API_PanSTARRS_galaxies.ipynb ├── ALeRCE_API_check_pixels.ipynb ├── ALeRCE_Asteroid_Starter.ipynb ├── ALeRCE_Client_starter.ipynb ├── ALeRCE_ML_Light_Curve_Classifier.ipynb ├── ALeRCE_ML_Stamp_Classifier.ipynb ├── ALeRCE_NLP_tests.ipynb ├── ALeRCE_Other_Periodograms.ipynb ├── ALeRCE_Other_Watchlist.ipynb ├── ALeRCE_SN_Fast.ipynb ├── ALeRCE_SN_Forecasting.ipynb ├── ALeRCE_SN_Starter.ipynb ├── ALeRCE_Transients_DESI.ipynb ├── ALeRCE_Tutorial_workshop_Sep2020 │ ├── ALeRCE_Hands_On_Tutorial_newDB_collab.ipynb │ ├── ALeRCE_ZTF_ActiveGalaxies_newDB_colab.ipynb │ ├── ALeRCE_ZTF_Asteroids_newDB_colab.ipynb │ ├── ALeRCE_ZTF_Machine_Learning_newDB.ipynb │ ├── ALeRCE_ZTF_Supernova_newDB_colab.ipynb │ ├── ALeRCE_ZTF_VariableStars_newDB_colab.ipynb │ ├── ALeRCE_pandas_sql_tutorial_newDB_colab.ipynb │ ├── README.md │ ├── data │ │ └── ephs_10586_20200101-20200102.pkl │ └── figures │ │ ├── dists_contours.png │ │ └── ml │ │ ├── alerce_explorer.png │ │ ├── alerce_explorer_object.png │ │ ├── alerce_frontend.png │ │ ├── betas.png │ │ ├── class_samples.png │ │ ├── cm.png │ │ ├── cnn_architecture.png │ │ ├── cnn_table.png │ │ ├── hierarchical_rf.png │ │ ├── sanchez_paper_confusion_matrix.png │ │ ├── sn_hunter.png │ │ └── taxonomy.png ├── ALeRCE_VS_Starter.ipynb ├── ALeRCE_ZTF_Stamp_animation.ipynb ├── CosmicStreams_Dec2023_tutorial │ └── ALeRCE_Basics_CosmicStreams.ipynb ├── EVIC_Dec2022_tutorial │ ├── ALeRCE_Client_starter.ipynb │ ├── ALeRCE_ML_Light_Curve_Classifier.ipynb │ ├── ALeRCE_ML_Stamp_Classifier.ipynb │ └── figures │ │ ├── SanchezSaez+2020_confusion.png │ │ ├── SanchezSaez+2020_recall.png │ │ ├── Villar+19.png │ │ ├── ml │ │ ├── alerce_explorer.png │ │ ├── alerce_explorer_object.png │ │ ├── alerce_frontend.png │ │ ├── betas.png │ │ ├── class_samples.png │ │ ├── cm.png │ │ ├── cnn_architecture.png │ │ ├── cnn_table.png │ │ ├── hierarchical_rf.png │ │ ├── sanchez_paper_confusion_matrix.png │ │ ├── sn_hunter.png │ │ └── taxonomy.png │ │ ├── recall_agn_qso_blazar_for_mag.png │ │ ├── recall_agn_qso_blazar_for_ndet.png │ │ └── taxonomy_tree.png ├── IAUS375_Dec2022_Blazars_tutorial │ ├── ALeRCE_Blazar_starter.ipynb │ └── readme.md ├── IAU_GA_2024 │ ├── ALeRCE_inspect_features.ipynb │ ├── README.md │ └── alercereaduser_v4.json ├── LSSTC_Broker_workshop_Apr2021 │ ├── ALeRCE_Extragalactic_notebook_brokers_workshop_2021.ipynb │ ├── ALeRCE_Galactic_notebook_brokers_workshop_2021.ipynb │ ├── ALeRCE_SolarSystem_notebook_brokers_workshop_2021.ipynb │ ├── ALeRCE_Training_sets_notebook_brokers_workshop_2021.ipynb │ └── README.md ├── README.md ├── coords_utils.py ├── figures │ ├── SanchezSaez+2020_confusion.png │ ├── SanchezSaez+2020_recall.png │ ├── Villar+19.png │ ├── recall_agn_qso_blazar_for_mag.png │ ├── recall_agn_qso_blazar_for_ndet.png │ └── taxonomy_tree.png └── variablestars_figures │ ├── RRL_GAIA_VS_Notebook.png │ ├── SanchezSaez+2020_confusion.png │ ├── SanchezSaez+2020_rrl.png │ └── taxonomy_tree.png └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | .ipynb_checkpoints 2 | __pycache__ -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | See more information about the ALeRCE broker, including its frontend, in [http://alerce.science/](http://alerce.science/) 4 | 5 | 6 | # ALeRCE use cases 7 | 8 | A collection of jupyter notebooks and scripts on how to access the ALeRCE database, API, etc. 9 | 10 | For faster cloning please use: `git clone --depth=1 https://github.com/alercebroker/usecases.git` 11 | 12 | ## Context 13 | 14 | The ALeRCE broker is a Chilean-led initiative to build a community broker for LSST and other large etendue survey telescopes which produce public streams of alerts. ALeRCE is developed by many institutions in Chile and in the U.S., taking advantage of the large network of collaborators which we have established around data science and astronomy during the last decade. The main motivation of ALeRCE is to facilitate the follow-up and the exploration of the LSST and other observatories alert streams. 15 | 16 | ## Scientific aim 17 | 18 | ALeRCE aims to facilitate the study of stationary (non--moving) variable and transient objects. We will do this by providing real-time filtered streams of aggregated, annotated and classified alerts, but also by providing alert exploration and analysis tools that can help researchers look for patterns and outliers within large populations of events. We also aim to provide forecasting tools which can help with the optimization of follow--up resources. 19 | 20 | ## This repository 21 | 22 | In this repository we show how to access the ALeRCE ZTF database, focused on different science cases. You can find several introductory jupyter notebooks in the [notebooks](https://github.com/alercebroker/usecases/tree/master/notebooks) directory, which connect to the database, query some tables and does some processing and visualization of the data. If you would like to beta test this please note that ALeRCE is still under development, with very simple taxonomy and classification models. 23 | 24 | ***LSST PCW 2020: note that we are undergoing major changes in our database design and we are beta releasing some notebooks which use this new API, which is much richer than our stable version. These notebooks have the "newDB" string on them.*** 25 | 26 | Please report any problems to francisco dot forster at gmail dot com. 27 | 28 | ## How can I contribute? 29 | 30 | The success of ALeRCE depends on being able to build a community of users which can connect LSST with the different follow up resources. Therefore, we are very happy to receive and publish contributed notebooks in this repository showing how your science case can benefit from having a real-time access to an annotated and classified database of ZTF alerts. Please feel free to join this repository and add your science cases! 31 | -------------------------------------------------------------------------------- /alerce_logo_rgb_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/alerce_logo_rgb_01.png -------------------------------------------------------------------------------- /alercereaduser.json: -------------------------------------------------------------------------------- 1 | { 2 | "params" : { 3 | "dbname" : "ztf", 4 | "user" : "alerceread", 5 | "host": "psql.alerce.online", 6 | "password" : "alerce2019" 7 | } 8 | } 9 | -------------------------------------------------------------------------------- /alercereaduser_v2.json: -------------------------------------------------------------------------------- 1 | { 2 | "params" : { 3 | "dbname" : "ztf_v2", 4 | "user" : "alerceread", 5 | "host": "db.alerce.online", 6 | "password" : "alerce2019" 7 | } 8 | } 9 | -------------------------------------------------------------------------------- /alercereaduser_v3.json: -------------------------------------------------------------------------------- 1 | { 2 | "params" : { 3 | "dbname" : "new_pipeline_ts", 4 | "user" : "alerceread", 5 | "host": "13.58.88.2", 6 | "password" : "alerce2020" 7 | } 8 | } 9 | -------------------------------------------------------------------------------- /alercereaduser_v4.json: -------------------------------------------------------------------------------- 1 | { 2 | "params" : { 3 | "dbname" : "ztf", 4 | "user" : "alerceread", 5 | "host": "54.205.99.47", 6 | "password" : "w*C*u8AXZ4e%d+zv" 7 | } 8 | } 9 | -------------------------------------------------------------------------------- /api/ALeRCE AVRO-STAMP Access API.md: -------------------------------------------------------------------------------- 1 | # ALeRCE AVRO/Stamps Access API 2 | 3 | API to get an specified avro/stamp from alert files of an ZTF Object. This API uses **GET requests** parameters. 4 | 5 | ## avro.alerce.online/get_avro 6 | Get an specified avro 7 | ``` 8 | curl http://avro.alerce.online/get_stamp?oid=ZTF19abguqsi&candid=928474644915015004 9 | ``` 10 | 11 | Parameters: 12 | - oid: ZTF Object ID. 13 | - candid: ZTF Alert ID. 14 | 15 | ## avro.alerce.online/get_stamp 16 | Get an specific stamp 17 | ``` 18 | curl http://avro.alerce.online/get_stamp?oid=ZTF19abguqsi&candid=928474644915015004&format=png&type=science 19 | ``` 20 | Parameters: 21 | - oid: ZTF Object ID. 22 | - candid: ZTF Alert ID. 23 | - format: ["png","fits"] 24 | - type: ["science","template","difference"] 25 | -------------------------------------------------------------------------------- /api/ALeRCE ZTF DB API.md: -------------------------------------------------------------------------------- 1 | # ALeRCE ZTF DB API 2 | This API gives access to ALeRCE Annotated ZTF API, with corrected magnitudes, objects statistics and the object data. 3 | 4 | The access point to the API is [ztf.alerce.online], querying the API is done with **http POST requests** with parameters in **JSON** format. 5 | 6 | ## Query the DB 7 | ### ztf.alerce.online/query 8 | This will query the DB objects and get the stats for the matching objects. 9 | 10 | The current fields to query the db are the following: 11 | 12 | ``` 13 | { 14 | total: number (if not set the total is counted and the query is slower), 15 | records_per_pages: number (default 20), 16 | page: number (default 1), 17 | sortBy: string columnName (default nobs), 18 | query_parameters:{ 19 | filters:{ 20 | //ZTF Object id 21 | oid: "ZTFXXXXXX", 22 | //Number of detections 23 | nobs: { 24 | min: int 25 | max: int 26 | }, 27 | //Late Classifier (Random Forest) 28 | classrf: ["CEPH","DSCT","EB","LPV","RRL","SNe","Other"] or int, 29 | pclassrf: float [0-1], 30 | //Early Classifier (Stamp Classifier) 31 | classearly: ["AGN","SN","VS","asteroid","bogus"] or int, 32 | pclassearly: float [0-1], 33 | }, 34 | //Coordinate based search (RA,DEC) and Search Radius. 35 | coordinates:{ 36 | ra: float degrees, 37 | dec: float degrees, 38 | sr: float degrees 39 | }, 40 | dates:{ 41 | //First detection (Discovery date) 42 | firstmjd: { 43 | min: float mjd, 44 | max: float mjd 45 | } 46 | } 47 | } 48 | ``` 49 | 50 | The response contains the following: 51 | ``` 52 | { 53 | "total": int, 54 | "num_pages": int, 55 | "page": int, 56 | "result": { 57 | : 58 | } 59 | 60 | } 61 | ``` 62 | Where ObjectId is the ZTF id and ObjectStats is a JSON document with the stats. 63 | 64 | #### Example 65 | Get the recent objects 66 | ``` 67 | curl -X POST ztf.alerce.online/query -d @- << EOF 68 | { 69 | "query_parameters":{ 70 | "dates":{ 71 | "firstmjd":{ 72 | "min": 58682 73 | } 74 | } 75 | } 76 | } 77 | EOF 78 | ``` 79 | 80 | Get the last 100 objects classified as SNe 81 | ``` 82 | curl -X POST ztf.alerce.online/query -d @- << EOF 83 | { 84 | "records_per_pages":100, 85 | "query_parameters": { 86 | "filters": {"classearly": 2}, 87 | "dates":{ 88 | "firstmjd":{ 89 | "min":58680 90 | } 91 | } 92 | }, 93 | "sortBy": "pclassearly", 94 | "total":100 95 | } 96 | EOF 97 | ``` 98 | 99 | 100 | 101 | ### ztf.alerce.online/get_sql 102 | Get the SQL Query done to the ZTF DB using the same parameters as **/query** 103 | 104 | ### Example 105 | ``` 106 | curl -X POST ztf.alerce.online/get_sql -d @- << EOF 107 | { 108 | "query_parameters":{ 109 | "dates":{ 110 | "firstmjd":{ 111 | "min": 58682 112 | } 113 | } 114 | } 115 | } 116 | EOF 117 | ``` 118 | 119 | 120 | ## Query an object 121 | To get an specific ZTF Object information. 122 | ### ztf.alerce.online/get_detection 123 | Get all the detections for an object. 124 | 125 | ``` 126 | curl -X POST ztf.alerce.online/get_detections -d @- << EOF 127 | { 128 | "oid":"ZTF18abbvavt" 129 | } 130 | EOF 131 | ``` 132 | 133 | 134 | 135 | ### ztf.alerce.online/get_non_detection 136 | Get non detections of an object. 137 | 138 | ``` 139 | curl -X POST ztf.alerce.online/get_non_detections -d @- << EOF 140 | { 141 | "oid":"ZTF18abbvavt" 142 | } 143 | EOF 144 | ``` 145 | 146 | ### ztf.alerce.online/get_stats 147 | Get stats for an object. 148 | ``` 149 | curl -X POST ztf.alerce.online/get_stats -d @- << EOF 150 | { 151 | "oid":"ZTF18abbvavt" 152 | } 153 | EOF 154 | ``` 155 | 156 | ### ztf.alerce.online/get_probabilities 157 | Get probabilities of the models for an object. 158 | ``` 159 | curl -X POST ztf.alerce.online/get_probabilities -d @- << EOF 160 | { 161 | "oid":"ZTF18abbvavt" 162 | } 163 | EOF 164 | ``` 165 | 166 | ### ztf.alerce.online/get_features 167 | Get features computed for an object (list of features pending). 168 | ``` 169 | curl -X POST ztf.alerce.online/get_features -d @- << EOF 170 | { 171 | "oid":"ZTF18abbvavt" 172 | } 173 | EOF 174 | ``` 175 | -------------------------------------------------------------------------------- /api/README.md: -------------------------------------------------------------------------------- 1 | # ALeRCE API documentation 2 | 3 | 4 | 1. [Database access](https://github.com/alercebroker/usecases/blob/master/api/ALeRCE%20ZTF%20DB%20API.md) 5 | 6 | 2. [Stamps server](https://github.com/alercebroker/usecases/blob/master/api/ALeRCE%20AVRO-STAMP%20Access%20API.md) 7 | 8 | 3. [API python wrapper example notebook](https://github.com/alercebroker/usecases/blob/master/notebooks/ALeRCE_ZTF_API_examples.ipynb) 9 | -------------------------------------------------------------------------------- /example_data/CSAGN_coordinates.csv: -------------------------------------------------------------------------------- 1 | ra,dec 2 | 123.33041666666665,46.14694444444445 3 | 144.55124999999998,7.727777777777779 4 | 150.58374999999998,45.157500000000006 5 | 173.12125,3.9580555555555557 6 | 194.81958333333333,55.251944444444455 7 | 221.97583333333336,28.55666666666667 8 | 236.28125,17.164166666666667 9 | 237.5716666666667,41.650555555555556 10 | 238.6675,36.49777777777778 11 | 244.29749999999996,6.6425 12 | 118.66791666666664,32.68475 13 | 119.36791666666669,24.919472222222222 14 | 121.25125,34.04044444444445 15 | 125.13875000000003,38.40566666666667 16 | 128.10541666666666,37.12683333333333 17 | 131.81708333333333,37.53852777777778 18 | 142.57375,47.12269444444444 19 | 146.5870833333333,33.79624999999999 20 | 148.61499999999998,48.94413888888889 21 | 150.73416666666668,47.84108333333333 22 | 156.55833333333337,52.63111111111111 23 | 157.07375,21.252249999999997 24 | 169.94833333333332,23.594416666666667 25 | 170.67958333333328,36.6948888888889 26 | 172.79625000000001,37.619277777777775 27 | 174.27875,1.6633888888888892 28 | 181.17583333333334,27.903222222222226 29 | 188.0633333333333,13.342305555555557 30 | 189.58166666666668,41.405666666666676 31 | 194.48833333333334,32.49155555555556 32 | 220.51166666666666,43.61919444444445 33 | 224.48083333333335,43.84319444444445 34 | 229.01791666666668,35.84038888888889 35 | 233.56416666666667,30.576249999999998 36 | 239.62249999999997,27.28730555555555 37 | 241.92916666666667,43.47141666666668 38 | -------------------------------------------------------------------------------- /example_data/MorphGal_readme.txt: -------------------------------------------------------------------------------- 1 | J/ApJS/217/27 Morphologies of z<0.01 SDSS-DR7 galaxies (Ann+, 2015) 2 | ================================================================================ 3 | A catalog of visually classified galaxies in the local (z ~ 0.01) universe. 4 | Ann H.B., Seo M., Ha D.K. 5 | 6 | =2015ApJS..217...27A 7 | ================================================================================ 8 | ADC_Keywords: Galaxies, nearby ; Morphology ; Photometry, SDSS 9 | Keywords: catalogs; galaxies: general; galaxies: statistics; galaxies: structure 10 | 11 | Abstract: 12 | The morphological types of 5836 galaxies were classified by a visual 13 | inspection of color images using the Sloan Digital Sky Survey Data 14 | Release 7 to produce a morphology catalog of a representative sample 15 | of local galaxies with z<0.01. The sample galaxies are almost complete 16 | for galaxies brighter than r_pet_=17.77. Our classification system is 17 | basically the same as that of the Third Reference Catalog of Bright 18 | Galaxies with some simplifications for giant galaxies. On the other 19 | hand, we distinguish the fine features of dwarf elliptical (dE)-like 20 | galaxies to classify five subtypes: dE, blue-cored dwarf ellipticals, 21 | dwarf spheroidals (dSph), blue dwarf ellipticals (dE_blue_), and dwarf 22 | lenticulars (dS0). In addition, we note the presence of nucleation in 23 | dE, dSph, and dS0. Elliptical galaxies and lenticular galaxies 24 | contribute only ~1.5 and 4.9% of local galaxies, respectively, whereas 25 | spirals and irregulars contribute ~32.1 and ~42.8%, respectively. The 26 | dE_blue_ galaxies, which are a recently discovered population of 27 | galaxies, contribute a significant fraction of dwarf galaxies. There 28 | seem to be structural differences between dSph and dE galaxies. The 29 | dSph galaxies are fainter and bluer with a shallower surface 30 | brightness gradient than dE galaxies. They also have a lower fraction 31 | of galaxies with small axis ratios (b/a<~0.4) than dE galaxies. The 32 | mean projected distance to the nearest neighbor galaxy is ~260 kpc. 33 | About 1% of local galaxies have no neighbors with comparable 34 | luminosity within a projected distance of 2Mpc. 35 | 36 | Description: 37 | This paper presents a catalog of the morphological types of galaxies 38 | whose redshifts are less than z=0.01. The morphological types are 39 | determined by a visual inspection of the color images provided by SDSS 40 | DR7 (II/294). The majority of galaxies in the present sample come from 41 | the KIAS-VAGC (Choi et al. 2010JKAS...43..191C) which is based on the 42 | spectroscopic target galaxies of the SDSS DR7 complemented by the 43 | bright galaxies with known redshifts from various catalogs. 44 | 45 | File Summary: 46 | -------------------------------------------------------------------------------- 47 | FileName Lrecl Records Explanations 48 | -------------------------------------------------------------------------------- 49 | ReadMe 80 . This file 50 | table1.dat 121 5840 Catalog of the morphological types of local galaxies 51 | -------------------------------------------------------------------------------- 52 | 53 | See also: 54 | II/294 : The SDSS Photometric Catalog, Release 7 (Adelman-McCarthy+, 2009) 55 | VII/250 : The 2dF Galaxy Redshift Survey (2dFGRS) (2dFGRS Team, 1998-2003) 56 | VII/98 : Catalogue of 2810 nearby galaxies (Kraan-Korteweg, 1986) 57 | J/MNRAS/446/3749 : SDSS nearby galaxies morphologies (Yoshino+, 2015) 58 | J/MNRAS/445/630 : Early-type galaxies in Ursa Major cluster (Pak+, 2014) 59 | J/MNRAS/435/2835 : Morphological types from Galaxy Zoo 2 (Willett+, 2013) 60 | J/AJ/146/151 : Morphology catalog of nearby galaxies from SDSS (Oh+, 2013) 61 | J/AJ/145/101 : Updated nearby galaxy catalog (Karachentsev+, 2013) 62 | J/MNRAS/429/2264 : Anatomy of Ursa Majoris (Karachentsev+, 2013) 63 | J/AJ/144/4 : Properties of dwarf galaxies in the LG (McConnachie+, 2012) 64 | J/ApJ/743/123 : Motions of galaxies in Coma I cloud (Karachentsev+, 2011) 65 | J/MNRAS/412/2498 : Galaxy groups in the local universe (Makarov+, 2011) 66 | J/MNRAS/412/727 : SDSS red gal. automated morph. classification (Cheng+, 2011) 67 | J/ApJ/722/L120 : Central surface brightness of SDSS galaxies (Fathi, 2010) 68 | J/ApJ/711/361 : Local Group dE galaxies. II. (Geha+, 2010) 69 | J/ApJS/186/427 : Detailed morphology of SDSS galaxies (Nair+, 2010) 70 | J/MNRAS/398/1129 : Central galaxies in groups and clusters (Guo+, 2009) 71 | J/ApJS/182/216 : Surface photometry of Virgo ellipticals (Kormendy+, 2009) 72 | J/ApJ/660/1186 : VCC galaxies classification with SDSS (Lisker+, 2007) 73 | J/AJ/128/163 : Galaxy morphological classification (Lotz+, 2004) 74 | J/ApJS/147/29 : BRHalpha data of blue compact dwarf gal. (Gil De Paz+, 2003) 75 | J/ApJS/147/1 : Classification of nearby galaxies (Conselice+, 2003) 76 | J/ApJ/588/218 : i*g* photometry of SDSS EDR galaxies (Abraham+, 2003) 77 | J/AJ/90/1681 : The Virgo Cluster Catalog (VCC) (Binggeli+, 1985) 78 | http://www.sdss.org/ : SDSS home page 79 | 80 | Byte-by-byte Description of file: table1.dat 81 | -------------------------------------------------------------------------------- 82 | Bytes Format Units Label Explanations 83 | -------------------------------------------------------------------------------- 84 | 1- 18 A18 --- ID SDSS identifier (ObjID or SDSSJHHMMSS.ss+DDM) 85 | 20- 29 F10.6 deg RAdeg Right Ascension in decimal degrees (J2000) 86 | 31- 40 F10.6 deg DEdeg Declination in decimal degrees (J2000) 87 | 42- 50 F9.6 --- z [-0.003/0.01] Heliocentric spectroscopic redshift 88 | 52- 58 A7 --- Morph Morphology from this study 89 | 60- 62 I3 --- T [-11/13] Numeric morphology index (1) 90 | 64- 68 F5.2 Mpc Dist [0/42.2] Distance (2) 91 | 70- 75 F6.2 mag rMag [-22.3/0.5] Absolute SDSS DR7 r band magnitude (3) 92 | 77- 81 F5.2 mag u-r [-2.3/15.5]?=-9 Extinction corrected model (u-r) 93 | color index 94 | 83- 89 F7.1 pix amaj ?=-9 Isophotal semi-major axis measured at 95 | {mu}_r_=25mag/arcsec^2^ 96 | 91- 95 F5.2 --- b/a [0/1]?=-9 Isophotal axis ratio 97 | 97-121 A25 --- Name Primary source identifier (4) 98 | -------------------------------------------------------------------------------- 99 | Note (1): For practical purposes, the numerical type T borrowed from RC3 is 100 | used with some simplification (see section 3.1): 101 | * ellipticals: T=-5 102 | * lenticulars: T=-3 103 | * spirals have their Hubble stages from T=0 (S0/a) to T=9 (Sm) 104 | * Irregulars: T=10 (Im), T=11 (dI), and 105 | T=12 (irregulars containing small BCD-like components) 106 | * Blue compact dwarf (BCD): T=13 107 | * dwarf ellipticals: T=-6 (dE), T=-7 (dEbc), T=-8 (dSph), T=-9 (dEblue), 108 | T=-10 (dS0, dS0p), and T=-11 for transition type dwarf (dEs/dI dI/dEs), 109 | Note (2): Derived from the radial velocity relative to the Local Group. 110 | Note (3): We use the model magnitude from SDSS DR7 corrected for the galactic 111 | extinction (Schlegel et al. 1998ApJ...500..525S). The SDSS model 112 | magnitude is a magnitude derived by best fitting model (exponential 113 | profile or de Vaucouleurs profile) to the observed images. 114 | Note (4): The first entry in NED is taken. In cases of no NED name, we make a 115 | name using the coordinates of the galaxy similar to the SDSS 116 | object name. 117 | -------------------------------------------------------------------------------- 118 | 119 | History: 120 | From electronic version of the journal 121 | 122 | ================================================================================ 123 | (End) Greg Schwarz [AAS], Emmanuelle Perret [CDS] 11-May-2015 124 | -------------------------------------------------------------------------------- /example_data/ephs_153935.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/example_data/ephs_153935.pkl -------------------------------------------------------------------------------- /example_data/ephs_153935_20210301-20210402.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/example_data/ephs_153935_20210301-20210402.pkl -------------------------------------------------------------------------------- /example_data/watchlist.csv: -------------------------------------------------------------------------------- 1 | id_source,ra,dec 2 | source_1,160.18301441363647,33.0164673528409 3 | source_2,174.21524897555545,44.83789535222221 4 | -------------------------------------------------------------------------------- /example_data/xmatch_test.csv: -------------------------------------------------------------------------------- 1 | id,ra,dec 2 | 1,123.33041666666665,46.14694444444445 3 | 2,144.55124999999998,7.727777777777779 4 | 3,150.58374999999998,45.157500000000006 5 | 4,173.12125,3.9580555555555557 6 | 5,194.81958333333333,55.251944444444455 7 | 6,221.97583333333336,28.55666666666667 8 | 7,236.28125,17.164166666666667 9 | 8,237.5716666666667,41.650555555555556 10 | 9,238.6675,36.49777777777778 11 | 10,244.29749999999996,6.6425 12 | 11,118.66791666666664,32.68475 13 | 12,119.36791666666669,24.919472222222222 14 | 13,121.25125,34.04044444444445 15 | 14,125.13875000000003,38.40566666666667 16 | 15,128.10541666666666,37.12683333333333 17 | 16,131.81708333333333,37.53852777777778 18 | 17,142.57375,47.12269444444444 19 | 18,146.5870833333333,33.79624999999999 20 | 19,148.61499999999998,48.94413888888889 21 | 20,150.73416666666668,47.84108333333333 22 | 21,156.55833333333337,52.63111111111111 23 | 22,157.07375,21.252249999999997 24 | 23,169.94833333333332,23.594416666666667 25 | 24,170.67958333333328,36.6948888888889 26 | 25,172.79625000000001,37.619277777777775 27 | 26,174.27875,1.6633888888888892 28 | 27,181.17583333333334,27.903222222222226 29 | 28,188.0633333333333,13.342305555555557 30 | 29,189.58166666666668,41.405666666666676 31 | 30,194.48833333333334,32.49155555555556 32 | 31,220.51166666666666,43.61919444444445 33 | 32,224.48083333333335,43.84319444444445 34 | 33,229.01791666666668,35.84038888888889 35 | 34,233.56416666666667,30.576249999999998 36 | 35,239.62249999999997,27.28730555555555 37 | 36,241.92916666666667,43.47141666666668 38 | -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/ALeRCE_Client_starter.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# ALeRCE client starter" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "Francisco Förster\n", 15 | "\n", 16 | "Last modification: 20221104\n", 17 | "\n", 18 | "Very simple examples on how to interact with the ALeRCE client https://alerce.readthedocs.io/en/latest/index.html.\n", 19 | "\n", 20 | "See https://alerce.readthedocs.io/en/latest/apis.html for the API documentation.\n", 21 | "\n", 22 | "We recommend that you run this notebook from the following [link](https://colab.research.google.com/github/alercebroker/usecases/blob/master/notebooks/ADASS_XXXII_Nov2022_tutorial/ALeRCE_Client_starter.ipynb)." 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "Load libraries" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": { 36 | "ExecuteTime": { 37 | "end_time": "2022-12-08T14:40:34.946647Z", 38 | "start_time": "2022-12-08T14:40:34.399891Z" 39 | } 40 | }, 41 | "outputs": [], 42 | "source": [ 43 | "import matplotlib.pyplot as plt\n", 44 | "import pandas as pd\n", 45 | "import matplotlib.colors" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": null, 51 | "metadata": { 52 | "ExecuteTime": { 53 | "end_time": "2022-12-08T14:40:34.949308Z", 54 | "start_time": "2022-12-08T14:40:34.947845Z" 55 | } 56 | }, 57 | "outputs": [], 58 | "source": [ 59 | "#!pip install alerce" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": { 66 | "ExecuteTime": { 67 | "end_time": "2022-12-08T14:40:35.322086Z", 68 | "start_time": "2022-12-08T14:40:34.950184Z" 69 | } 70 | }, 71 | "outputs": [], 72 | "source": [ 73 | "from alerce.core import Alerce" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "# Initialize alerce api object" 81 | ] 82 | }, 83 | { 84 | "cell_type": "markdown", 85 | "metadata": {}, 86 | "source": [ 87 | "Let's start the ALeRCE client" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": { 94 | "ExecuteTime": { 95 | "end_time": "2022-12-08T14:40:35.324525Z", 96 | "start_time": "2022-12-08T14:40:35.323005Z" 97 | } 98 | }, 99 | "outputs": [], 100 | "source": [ 101 | "client = Alerce()" 102 | ] 103 | }, 104 | { 105 | "cell_type": "markdown", 106 | "metadata": {}, 107 | "source": [ 108 | "We will explore different methods from the client:\n", 109 | "\n", 110 | "* Query global properties of an individual object\n", 111 | "* Query properties per band of an individual object\n", 112 | "* Query detections of an individual object\n", 113 | "* Query image stamps\n", 114 | "* Crossmatch with objects in the vicinity\n", 115 | "* Query non detections of an individual object\n", 116 | "* Query features of an individual object\n", 117 | "* Query probabilities of an individual object\n", 118 | "* Query global properties of a group of objects" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "# Query properties for an individual object" 126 | ] 127 | }, 128 | { 129 | "cell_type": "markdown", 130 | "metadata": {}, 131 | "source": [ 132 | "We will now query the global properties of one object based on a given object id, ZTF20aaelulu in this case. You can see this object in the website https://alerce.online/object/ZTF20aaelulu.\n" 133 | ] 134 | }, 135 | { 136 | "cell_type": "code", 137 | "execution_count": null, 138 | "metadata": { 139 | "ExecuteTime": { 140 | "end_time": "2022-12-08T14:40:36.218269Z", 141 | "start_time": "2022-12-08T14:40:35.325609Z" 142 | } 143 | }, 144 | "outputs": [], 145 | "source": [ 146 | "oid = \"ZTF20aaelulu\"\n", 147 | "query_results = client.query_objects(\n", 148 | " oid=oid,\n", 149 | " format='pandas')\n", 150 | "properties = query_results # save for later\n", 151 | "query_results" 152 | ] 153 | }, 154 | { 155 | "cell_type": "markdown", 156 | "metadata": { 157 | "ExecuteTime": { 158 | "end_time": "2022-01-18T14:44:08.210148Z", 159 | "start_time": "2022-01-18T14:44:08.193187Z" 160 | } 161 | }, 162 | "source": [ 163 | "The column names are the following" 164 | ] 165 | }, 166 | { 167 | "cell_type": "code", 168 | "execution_count": null, 169 | "metadata": { 170 | "ExecuteTime": { 171 | "end_time": "2022-12-08T14:40:36.221406Z", 172 | "start_time": "2022-12-08T14:40:36.219195Z" 173 | } 174 | }, 175 | "outputs": [], 176 | "source": [ 177 | "\", \".join(query_results)" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "The columns are described in here https://alerce.readthedocs.io/en/latest/models.html" 185 | ] 186 | }, 187 | { 188 | "cell_type": "markdown", 189 | "metadata": {}, 190 | "source": [ 191 | "# Query properties per band for an indivual object" 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": null, 197 | "metadata": { 198 | "ExecuteTime": { 199 | "end_time": "2022-12-08T14:40:36.413184Z", 200 | "start_time": "2022-12-08T14:40:36.222906Z" 201 | } 202 | }, 203 | "outputs": [], 204 | "source": [ 205 | "query_results = client.query_magstats(\n", 206 | " oid=oid,\n", 207 | " format='pandas')\n", 208 | "query_results" 209 | ] 210 | }, 211 | { 212 | "cell_type": "markdown", 213 | "metadata": { 214 | "ExecuteTime": { 215 | "end_time": "2022-01-18T15:43:44.730607Z", 216 | "start_time": "2022-01-18T15:43:44.706074Z" 217 | } 218 | }, 219 | "source": [ 220 | "You can see that the results are different statistics in two rows, one per band. The columns are described in here https://alerce.readthedocs.io/en/latest/models.html" 221 | ] 222 | }, 223 | { 224 | "cell_type": "code", 225 | "execution_count": null, 226 | "metadata": { 227 | "ExecuteTime": { 228 | "end_time": "2022-12-08T14:40:36.417949Z", 229 | "start_time": "2022-12-08T14:40:36.415042Z" 230 | } 231 | }, 232 | "outputs": [], 233 | "source": [ 234 | "\", \".join(query_results)" 235 | ] 236 | }, 237 | { 238 | "cell_type": "markdown", 239 | "metadata": {}, 240 | "source": [ 241 | "# Query detections and non-detections for an individual object" 242 | ] 243 | }, 244 | { 245 | "cell_type": "markdown", 246 | "metadata": { 247 | "ExecuteTime": { 248 | "end_time": "2022-10-12T19:21:12.710843Z", 249 | "start_time": "2022-10-12T19:21:12.704041Z" 250 | } 251 | }, 252 | "source": [ 253 | "Now we obtain a list of all the detections and non-detections (limiting magnitude) in two bands. \n", 254 | "\n", 255 | "Note that both dataframes include a unique identifier for the telescope (`tid`), which hints at the multi-stream nature of ALeRCE (soon to be available)." 256 | ] 257 | }, 258 | { 259 | "cell_type": "code", 260 | "execution_count": null, 261 | "metadata": { 262 | "ExecuteTime": { 263 | "end_time": "2022-12-08T14:40:36.822421Z", 264 | "start_time": "2022-12-08T14:40:36.418958Z" 265 | }, 266 | "scrolled": false 267 | }, 268 | "outputs": [], 269 | "source": [ 270 | "detections = client.query_detections(\n", 271 | " oid=oid,\n", 272 | " format='pandas')\n", 273 | "nondetections = client.query_non_detections(\n", 274 | " oid=oid,\n", 275 | " format='pandas')\n", 276 | "display(detections)\n", 277 | "display(nondetections)" 278 | ] 279 | }, 280 | { 281 | "cell_type": "markdown", 282 | "metadata": {}, 283 | "source": [ 284 | "The most important columns for the detections are the time (`mjd`), the unique detection identifier (`candid`), the band (`fid`), the difference magnitude (`magpsf`) and its error (`sigmapsf`). \n", 285 | "\n", 286 | "The columns for the non-detections are the time (`mjd`), the band (`fid`), and the limiting magnitude (`diffmaglim`).\n", 287 | "\n", 288 | "All the field are explained in here The columns are described in here https://alerce.readthedocs.io/en/latest/models.html." 289 | ] 290 | }, 291 | { 292 | "cell_type": "markdown", 293 | "metadata": {}, 294 | "source": [ 295 | "# Query image stamps\n", 296 | "\n", 297 | "We can also query the image stamps associated to a specific object and candid." 298 | ] 299 | }, 300 | { 301 | "cell_type": "markdown", 302 | "metadata": {}, 303 | "source": [ 304 | "We will use the first candid from the previously queried detections." 305 | ] 306 | }, 307 | { 308 | "cell_type": "code", 309 | "execution_count": null, 310 | "metadata": { 311 | "ExecuteTime": { 312 | "end_time": "2022-12-08T14:40:36.833734Z", 313 | "start_time": "2022-12-08T14:40:36.823417Z" 314 | } 315 | }, 316 | "outputs": [], 317 | "source": [ 318 | "stamps = client.get_stamps(oid, detections.iloc[0].candid)\n", 319 | "print(stamps)" 320 | ] 321 | }, 322 | { 323 | "cell_type": "markdown", 324 | "metadata": {}, 325 | "source": [ 326 | "The image stamps are a triplet of science, reference and difference images. Let's look at the first image:" 327 | ] 328 | }, 329 | { 330 | "cell_type": "code", 331 | "execution_count": null, 332 | "metadata": { 333 | "ExecuteTime": { 334 | "end_time": "2022-12-08T14:40:36.993399Z", 335 | "start_time": "2022-12-08T14:40:36.834713Z" 336 | } 337 | }, 338 | "outputs": [], 339 | "source": [ 340 | "fig, ax = plt.subplots(ncols=3)\n", 341 | "for i in range(3):\n", 342 | " ax[i].imshow(stamps[i].data)" 343 | ] 344 | }, 345 | { 346 | "cell_type": "markdown", 347 | "metadata": {}, 348 | "source": [ 349 | "We can also directly plot the images stamps using the plot_stamps command." 350 | ] 351 | }, 352 | { 353 | "cell_type": "code", 354 | "execution_count": null, 355 | "metadata": { 356 | "ExecuteTime": { 357 | "end_time": "2022-12-08T14:40:36.997758Z", 358 | "start_time": "2022-12-08T14:40:36.994594Z" 359 | } 360 | }, 361 | "outputs": [], 362 | "source": [ 363 | "client.plot_stamps(oid, detections.iloc[0].candid)" 364 | ] 365 | }, 366 | { 367 | "cell_type": "markdown", 368 | "metadata": {}, 369 | "source": [ 370 | "If `candid` is not given the last available value will be used." 371 | ] 372 | }, 373 | { 374 | "cell_type": "code", 375 | "execution_count": null, 376 | "metadata": { 377 | "ExecuteTime": { 378 | "end_time": "2022-12-08T14:40:37.783302Z", 379 | "start_time": "2022-12-08T14:40:36.998552Z" 380 | } 381 | }, 382 | "outputs": [], 383 | "source": [ 384 | "client.plot_stamps(oid)\n", 385 | "print(detections.candid.max())" 386 | ] 387 | }, 388 | { 389 | "cell_type": "markdown", 390 | "metadata": {}, 391 | "source": [ 392 | "# Crossmatch with objects in the vicinity" 393 | ] 394 | }, 395 | { 396 | "cell_type": "markdown", 397 | "metadata": {}, 398 | "source": [ 399 | "We can also crossmatch the object's position with objects from the catsHTM collection of catalogs (https://github.com/maayane/catsHTM). The catalogs included in catsHTM are:\n", 400 | "\n", 401 | "* 2MASS (input name: TMASS)\n", 402 | "* 2MASSxsc (input name: TMASSxsc) - 2MASS extended source catalog\n", 403 | "* AKARI (input name: AKARI)\n", 404 | "* APASS (input name: APASS) - AAVSO All Sky Photometric Sky Survey (~5.5x10^7 sources)\n", 405 | "* Cosmos (input name: Cosmos) - Sources in the Cosmos field\n", 406 | "* DECaLS (input name: DECaLS) - DECaLS DR5 release\n", 407 | "* FIRST (input name: FIRST) - (~9.5x10^5 sources)\n", 408 | "* GAIA/DR1 (input name: GAIADR1) - (~1.1x10^9 sources).\n", 409 | "* GAIA/DR2 (input name: GAIADR2) - NEW! (~1.6x10^9 sources)\n", 410 | "* GAIA/EDR3 (input name: GAIAEDR3) - NEW! (~1.8x10^9 sources)\n", 411 | "* GALEX (input name: GALEX) - GALAEX/GR6Plus7 (~1.7x10^8 sources).\n", 412 | "* HSC/v2 (input name: HSCv2)- Hubble source catalog\n", 413 | "* IPHAS/DR2 (input name: IPHAS)\n", 414 | "* NED redshifts (input name: NEDz)\n", 415 | "* NVSS (input name: NVSS) - (~1.8x10^6 sources)\n", 416 | "* HYPERLEDA (input name: PGC)\n", 417 | "* PS1 (input name: PS1) - Pan-STARRS (~2.6x10^9 sources; A cleaned version of the PS1 stack catalog; some missing tiles below declination of zero [being corrected])\n", 418 | "* The PTF photometric catalog (input name: PTFpc)\n", 419 | "* ROSATfsc (input name: ROSATfsc) - ROSAT faint source catalog\n", 420 | "* SDSS/DR10 (input name: SDSSDR10)- Primary sources from SDSS/DR10 (last photometric release)\n", 421 | "* Skymapper DR1 (input name: Skymapper)\n", 422 | "* SpecSDSS/DR14 (input name: SpecSDSS) - SDSS spectroscopic catalog\n", 423 | "* Spitzer/SAGE (input name SAGE)\n", 424 | "* Spitzer/IRAC (input name IRACgc) - Spitzer IRAC galactic center survey\n", 425 | "* UCAC4 (input name: UCAC4) - (~1.1x10^8 sources)\n", 426 | "* UKIDSS/DR10 (input name: UKIDSS)\n", 427 | "* USNOB1 (not yet available)\n", 428 | "* VISTA/Viking/DR3 (not yet available)\n", 429 | "* VST/ATLAS/DR3 (input name: VSTatlas)\n", 430 | "* VST/KiDS/DR3 (input name: VSTkids)\n", 431 | "* WISE (input name: WISE) - ~5.6x10^8 sources\n", 432 | "* XMM (input name: XMM)- 7.3x10^5 sources 3XMM-DR7 (Rosen et al. 2016; A&A 26, 590)\n", 433 | "* ZTF-DR1 stellar variability catalog (input name: ztfSrcLCDR1)\n", 434 | "* ZTF-DR1 variable star candidates (input name: ztfSrcLCDR1)" 435 | ] 436 | }, 437 | { 438 | "cell_type": "code", 439 | "execution_count": null, 440 | "metadata": { 441 | "ExecuteTime": { 442 | "end_time": "2022-12-08T14:41:11.737939Z", 443 | "start_time": "2022-12-08T14:41:11.575178Z" 444 | } 445 | }, 446 | "outputs": [], 447 | "source": [ 448 | "ra = properties.meanra\n", 449 | "dec = properties.meandec\n", 450 | "radius = 30 # arcsec\n", 451 | "cone_objects = {}\n", 452 | "for catalog_name in [\"GAIA/DR1\", \"SpecSDSS\", \"NEDz\", \"GALEX\", \"SDSSDR10\"]:\n", 453 | " cone_objects[catalog_name] = client.catshtm_conesearch(ra, dec, radius, catalog_name, format=\"pandas\")\n", 454 | "cone_objects" 455 | ] 456 | }, 457 | { 458 | "cell_type": "markdown", 459 | "metadata": {}, 460 | "source": [ 461 | "We obtained many crossmatches from the [catsHTM](https://arxiv.org/abs/1805.02666_) catalog. Let's look at the catalogs where matches were found:" 462 | ] 463 | }, 464 | { 465 | "cell_type": "markdown", 466 | "metadata": {}, 467 | "source": [ 468 | "Let's plot the location of all crossmatches at the SDSS/DR10 xmatches, highlighting objects with redshift if there are any from the table SpecSDSS." 469 | ] 470 | }, 471 | { 472 | "cell_type": "code", 473 | "execution_count": null, 474 | "metadata": { 475 | "ExecuteTime": { 476 | "end_time": "2022-12-08T14:40:38.568835Z", 477 | "start_time": "2022-12-08T14:40:34.419Z" 478 | } 479 | }, 480 | "outputs": [], 481 | "source": [ 482 | "fig, ax = plt.subplots(figsize=(10, 10))\n", 483 | "cmap = plt.cm.tab20\n", 484 | "norm = matplotlib.colors.Normalize(vmin=0, vmax=len(cone_objects.keys()))\n", 485 | "for idx, i in enumerate(cone_objects.keys()):\n", 486 | " try:\n", 487 | " aux = cone_objects[i].rename({\"RA\": \"ra\", \"Dec\": \"dec\"}, axis=1)\n", 488 | " ax.scatter(aux.ra, aux.dec, color=cmap(norm(idx)), label=i, s=50)\n", 489 | " if i == \"SpecSDSS\": # the table with spectroscopic redshifts\n", 490 | " s = 500\n", 491 | " for idxrow, row in aux.iterrows():\n", 492 | " ax.text(row.ra, row.dec, \"z=%.5f\" % row.z, fontsize=20)\n", 493 | " except:\n", 494 | " print(i)\n", 495 | " \n", 496 | "ax.scatter(ra, dec, c='k', marker='o', s=100, label=oid)\n", 497 | "ax.set_xlim(ax.get_xlim()[::-1])\n", 498 | "plt.legend()" 499 | ] 500 | }, 501 | { 502 | "cell_type": "markdown", 503 | "metadata": {}, 504 | "source": [ 505 | "# Query features for an individual object" 506 | ] 507 | }, 508 | { 509 | "cell_type": "markdown", 510 | "metadata": {}, 511 | "source": [ 512 | "Now we will query the features used by our light curve classifier. These are hand made statistics or contextual information based on the object's light curve." 513 | ] 514 | }, 515 | { 516 | "cell_type": "code", 517 | "execution_count": null, 518 | "metadata": { 519 | "ExecuteTime": { 520 | "end_time": "2022-12-08T14:40:38.569386Z", 521 | "start_time": "2022-12-08T14:40:34.421Z" 522 | } 523 | }, 524 | "outputs": [], 525 | "source": [ 526 | "query_results = client.query_features(\n", 527 | " oid=oid,\n", 528 | " format='pandas')\n", 529 | "query_results" 530 | ] 531 | }, 532 | { 533 | "cell_type": "markdown", 534 | "metadata": {}, 535 | "source": [ 536 | "You can see that there are 178 rows, where each row has a feature name (`name`), a value (`value`), a band id (`fid`), and a feature version (`version`).\n", 537 | "\n", 538 | "A detailed explanation of all the features can be found in http://alerce.science/features/." 539 | ] 540 | }, 541 | { 542 | "cell_type": "markdown", 543 | "metadata": {}, 544 | "source": [ 545 | "We can pivot these dataframe to make the features appear as columns. To do this we will add an auxiliary columns that contains the feature name and filter id all in one string, as well as adding the object identifier as a column." 546 | ] 547 | }, 548 | { 549 | "cell_type": "code", 550 | "execution_count": null, 551 | "metadata": { 552 | "ExecuteTime": { 553 | "end_time": "2022-12-08T14:40:38.569875Z", 554 | "start_time": "2022-12-08T14:40:34.423Z" 555 | } 556 | }, 557 | "outputs": [], 558 | "source": [ 559 | "query_results[\"oid\"] = oid\n", 560 | "query_results['feature'] = [f\"{name}_{fid}\" for name, fid in zip(query_results.name, query_results.fid)]" 561 | ] 562 | }, 563 | { 564 | "cell_type": "code", 565 | "execution_count": null, 566 | "metadata": { 567 | "ExecuteTime": { 568 | "end_time": "2022-12-08T14:40:38.570287Z", 569 | "start_time": "2022-12-08T14:40:34.424Z" 570 | } 571 | }, 572 | "outputs": [], 573 | "source": [ 574 | "query_results.pivot(index='oid', columns='feature', values='value')" 575 | ] 576 | }, 577 | { 578 | "cell_type": "markdown", 579 | "metadata": {}, 580 | "source": [ 581 | "# Query probabilities for an individual object" 582 | ] 583 | }, 584 | { 585 | "cell_type": "markdown", 586 | "metadata": {}, 587 | "source": [ 588 | "Similarly, we can query the classification probabilities for a given object. \n", 589 | "\n", 590 | "Note that an object can be classified by different classifiers with different versions, which is shown in the columns `classifier_name` and `classifier_version`, respectively." 591 | ] 592 | }, 593 | { 594 | "cell_type": "code", 595 | "execution_count": null, 596 | "metadata": { 597 | "ExecuteTime": { 598 | "end_time": "2022-12-08T14:40:38.570959Z", 599 | "start_time": "2022-12-08T14:40:34.426Z" 600 | } 601 | }, 602 | "outputs": [], 603 | "source": [ 604 | "query_results = client.query_probabilities(\n", 605 | " oid=oid,\n", 606 | " format='pandas')\n", 607 | "query_results" 608 | ] 609 | }, 610 | { 611 | "cell_type": "markdown", 612 | "metadata": {}, 613 | "source": [ 614 | "We see that we get many rows for a single object. This shows all the probabilities associated to all the ALeRCE classifiers and the classes in their associated taxonomies. The columns indicate the name of the classifier (`classifier_name`), its version (`classifier_version`), the class (`class_name`), the probability (`probability`) and the ranking (`ranking`) growing from most to least likely. " 615 | ] 616 | }, 617 | { 618 | "cell_type": "markdown", 619 | "metadata": {}, 620 | "source": [ 621 | "Let's check the unique classifier versions associated to each classifier:" 622 | ] 623 | }, 624 | { 625 | "cell_type": "code", 626 | "execution_count": null, 627 | "metadata": { 628 | "ExecuteTime": { 629 | "end_time": "2022-12-08T14:40:38.571400Z", 630 | "start_time": "2022-12-08T14:40:34.427Z" 631 | } 632 | }, 633 | "outputs": [], 634 | "source": [ 635 | "for clf in query_results.classifier_name.unique():\n", 636 | " mask = query_results.classifier_name == clf\n", 637 | " print(clf, query_results.loc[mask].classifier_version.unique())" 638 | ] 639 | }, 640 | { 641 | "cell_type": "markdown", 642 | "metadata": {}, 643 | "source": [ 644 | "And now the unique classes associated to each classifier:" 645 | ] 646 | }, 647 | { 648 | "cell_type": "code", 649 | "execution_count": null, 650 | "metadata": { 651 | "ExecuteTime": { 652 | "end_time": "2022-12-08T14:40:38.571769Z", 653 | "start_time": "2022-12-08T14:40:34.428Z" 654 | } 655 | }, 656 | "outputs": [], 657 | "source": [ 658 | "for clf in query_results.classifier_name.unique():\n", 659 | " mask = query_results.classifier_name == clf\n", 660 | " print(clf, query_results.loc[mask].class_name.unique())" 661 | ] 662 | }, 663 | { 664 | "cell_type": "markdown", 665 | "metadata": { 666 | "ExecuteTime": { 667 | "end_time": "2022-01-18T16:16:49.599582Z", 668 | "start_time": "2022-01-18T16:16:49.583059Z" 669 | } 670 | }, 671 | "source": [ 672 | "The classifiers are two classifiers, the `lc_classifier` and the `stamp_classifier`, where `lc_classifier` is a hierarchical classifier composed of four independent classifiers: `lc_classifier_top`, `lc_classifier_periodic`, `lc_classifier_stochastic`, and `lc_classifier_transient`. You can see more details about these classifiers in [Sánchez-Sáez+2021](https://ui.adsabs.harvard.edu/abs/2021AJ....161..141S/abstract) and [Carrasco-Davis+2021](https://ui.adsabs.harvard.edu/abs/2020arXiv200803309C/abstract). \n", 673 | "\n", 674 | "`lc_classifier`:\n", 675 | "* AGN\n", 676 | "* Blazar\n", 677 | "* CEP\n", 678 | "* CV/Nova\n", 679 | "* DSCT\n", 680 | "* E\n", 681 | "* LPV\n", 682 | "* Periodic-Other\n", 683 | "* QSO\n", 684 | "* RRL\n", 685 | "* SLSN\n", 686 | "* SNIa\n", 687 | "* SNIbc\n", 688 | "* SNII\n", 689 | "* YSO\n", 690 | "\n", 691 | "`stamp_classifier`:\n", 692 | "* AGN\n", 693 | "* asteroid\n", 694 | "* bogus\n", 695 | "* SN\n", 696 | "* VS\n", 697 | "\n", 698 | "\n", 699 | "`lc_classifier_top`:\n", 700 | "* Periodic\n", 701 | "* Stochastic\n", 702 | "* Transient\n", 703 | "\n", 704 | "`lc_classifier_periodic`:\n", 705 | "* CEP\n", 706 | "* DSCT\n", 707 | "* E\n", 708 | "* LPV\n", 709 | "* Periodic-Other\n", 710 | "* RRL\n", 711 | "\n", 712 | "`lc_classifier_stochastic`:\n", 713 | "* AGN\n", 714 | "* Blazar\n", 715 | "* CV/Nova\n", 716 | "* QSO\n", 717 | "* YSO\n", 718 | "\n", 719 | "`lc_classifier_transient`:\n", 720 | "* SLSN\n", 721 | "* SNIa\n", 722 | "* SNIbc\n", 723 | "* SNII" 724 | ] 725 | }, 726 | { 727 | "cell_type": "markdown", 728 | "metadata": {}, 729 | "source": [ 730 | "# Query global properties of a set of objects" 731 | ] 732 | }, 733 | { 734 | "cell_type": "markdown", 735 | "metadata": {}, 736 | "source": [ 737 | "## Query objects based on the most likely class" 738 | ] 739 | }, 740 | { 741 | "cell_type": "markdown", 742 | "metadata": {}, 743 | "source": [ 744 | "We will query the top 200 objects classified SNIa according to the light curve classifier. In here we ask for the top 200 objects in pandas format. By default this query asks for objects with classification `ranking=1`. We will ask for the results to be ordered by probability in descending order (`DESC`)." 745 | ] 746 | }, 747 | { 748 | "cell_type": "code", 749 | "execution_count": null, 750 | "metadata": { 751 | "ExecuteTime": { 752 | "end_time": "2022-12-08T14:40:38.572207Z", 753 | "start_time": "2022-12-08T14:40:34.429Z" 754 | } 755 | }, 756 | "outputs": [], 757 | "source": [ 758 | "query_results = client.query_objects(\n", 759 | " classifier=\"lc_classifier\",\n", 760 | " class_name=\"SNIa\",\n", 761 | " page_size=200,\n", 762 | " order_by='probability',\n", 763 | " order_mode='DESC',\n", 764 | " format='pandas')" 765 | ] 766 | }, 767 | { 768 | "cell_type": "code", 769 | "execution_count": null, 770 | "metadata": { 771 | "ExecuteTime": { 772 | "end_time": "2022-12-08T14:40:38.572574Z", 773 | "start_time": "2022-12-08T14:40:34.430Z" 774 | } 775 | }, 776 | "outputs": [], 777 | "source": [ 778 | "query_results" 779 | ] 780 | }, 781 | { 782 | "cell_type": "markdown", 783 | "metadata": {}, 784 | "source": [ 785 | "Note that now the columns `class`, `classifier`, and `probability` are included." 786 | ] 787 | }, 788 | { 789 | "cell_type": "markdown", 790 | "metadata": {}, 791 | "source": [ 792 | "## Query objects by classified ranking" 793 | ] 794 | }, 795 | { 796 | "cell_type": "markdown", 797 | "metadata": {}, 798 | "source": [ 799 | "When an object is classified by our classifiers, a `ranking` column is available to quickly extract the most likely class (`ranking=1`). We can also extract objects that were classified as SNIa as the 2nd or 3rd most likely classes. Note that not asking for a ranking is equivalent to asking for `ranking=1` (the most likely class). " 800 | ] 801 | }, 802 | { 803 | "cell_type": "code", 804 | "execution_count": null, 805 | "metadata": { 806 | "ExecuteTime": { 807 | "end_time": "2022-12-08T14:40:38.573100Z", 808 | "start_time": "2022-12-08T14:40:34.431Z" 809 | } 810 | }, 811 | "outputs": [], 812 | "source": [ 813 | "fig, ax = plt.subplots()\n", 814 | "\n", 815 | "for ranking in [1, 2, 3]:\n", 816 | " query_results = client.query_objects(\n", 817 | " classifier=\"lc_classifier\",\n", 818 | " class_name=\"SNIa\",\n", 819 | " ranking=ranking,\n", 820 | " page_size=200,\n", 821 | " order_by='probability',\n", 822 | " order_mode='DESC', format='pandas')\n", 823 | " query_results.probability.plot.hist(bins=20, ax=ax, lw=2, log=True,\n", 824 | " alpha=0.5, histtype='step', label=\"ranking=%i\" % ranking)\n", 825 | "query_results = client.query_objects(\n", 826 | " classifier=\"lc_classifier\",\n", 827 | " class_name=\"SNIa\",\n", 828 | " page_size=200,\n", 829 | " order_by='probability',\n", 830 | " order_mode='DESC',\n", 831 | " format='pandas')\n", 832 | "query_results.probability.plot.hist(bins=20, ax=ax, lw=2, log=True, histtype='step', label=\"no ranking\")\n", 833 | "\n", 834 | "ax.set_xlabel(\"probability\")\n", 835 | "plt.legend()" 836 | ] 837 | }, 838 | { 839 | "cell_type": "markdown", 840 | "metadata": {}, 841 | "source": [ 842 | "You can see that the probabilities when no ranking is specified and `ranking=1` overlap because `ranking=1` is the default value. Also, the typical probabilities of `ranking=1` > `ranking=2` > `ranking=3`. " 843 | ] 844 | }, 845 | { 846 | "cell_type": "markdown", 847 | "metadata": {}, 848 | "source": [ 849 | "# Let's now generate a link to look at all the previous objects" 850 | ] 851 | }, 852 | { 853 | "cell_type": "markdown", 854 | "metadata": {}, 855 | "source": [ 856 | "Here we restrict the number of objects to 200, which is the maximum number that the ALeRCE explorer can accept. " 857 | ] 858 | }, 859 | { 860 | "cell_type": "code", 861 | "execution_count": null, 862 | "metadata": { 863 | "ExecuteTime": { 864 | "end_time": "2022-12-08T14:40:38.573569Z", 865 | "start_time": "2022-12-08T14:40:34.433Z" 866 | } 867 | }, 868 | "outputs": [], 869 | "source": [ 870 | "suffix = \"&count=true&page=1&perPage=1000&sortDesc=true&selectedClassifier=lc_classifier\"\n", 871 | "url = \"https://alerce.online/?\" + \"&\".join(\"oid=%s\" % i for i in query_results.oid.iloc[:200]) + suffix\n", 872 | "print(url)" 873 | ] 874 | }, 875 | { 876 | "cell_type": "markdown", 877 | "metadata": {}, 878 | "source": [ 879 | "Open the link in your browser to see the explore the objects." 880 | ] 881 | }, 882 | { 883 | "cell_type": "code", 884 | "execution_count": null, 885 | "metadata": {}, 886 | "outputs": [], 887 | "source": [] 888 | } 889 | ], 890 | "metadata": { 891 | "kernelspec": { 892 | "display_name": "Python 3", 893 | "language": "python", 894 | "name": "python3" 895 | }, 896 | "language_info": { 897 | "codemirror_mode": { 898 | "name": "ipython", 899 | "version": 3 900 | }, 901 | "file_extension": ".py", 902 | "mimetype": "text/x-python", 903 | "name": "python", 904 | "nbconvert_exporter": "python", 905 | "pygments_lexer": "ipython3", 906 | "version": "3.8.8" 907 | }, 908 | "toc": { 909 | "base_numbering": 1, 910 | "nav_menu": {}, 911 | "number_sections": true, 912 | "sideBar": true, 913 | "skip_h1_title": false, 914 | "title_cell": "Table of Contents", 915 | "title_sidebar": "Contents", 916 | "toc_cell": false, 917 | "toc_position": { 918 | "height": "calc(100% - 180px)", 919 | "left": "10px", 920 | "top": "150px", 921 | "width": "388.991px" 922 | }, 923 | "toc_section_display": true, 924 | "toc_window_display": true 925 | }, 926 | "varInspector": { 927 | "cols": { 928 | "lenName": 16, 929 | "lenType": 16, 930 | "lenVar": 40 931 | }, 932 | "kernels_config": { 933 | "python": { 934 | "delete_cmd_postfix": "", 935 | "delete_cmd_prefix": "del ", 936 | "library": "var_list.py", 937 | "varRefreshCmd": "print(var_dic_list())" 938 | }, 939 | "r": { 940 | "delete_cmd_postfix": ") ", 941 | "delete_cmd_prefix": "rm(", 942 | "library": "var_list.r", 943 | "varRefreshCmd": "cat(var_dic_list()) " 944 | } 945 | }, 946 | "types_to_exclude": [ 947 | "module", 948 | "function", 949 | "builtin_function_or_method", 950 | "instance", 951 | "_Feature" 952 | ], 953 | "window_display": false 954 | } 955 | }, 956 | "nbformat": 4, 957 | "nbformat_minor": 2 958 | } 959 | -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/README.md: -------------------------------------------------------------------------------- 1 | # ADASS XXXII ALeRCE tutorial 2 | 3 | Welcome to the ALeRCE tutorial github page. For a detailed explanation of the tutorial, see https://alerce.science/alerce-adass-tutorial-2022/. Also, see the accompanying slides [here](https://docs.google.com/presentation/d/1sATyMXlXBEUHwfjHzWAXpYroWF98WgkQdqM4mOG3WDk/edit?usp=sharing). 4 | 5 | ## List of Notebooks: 6 | 7 | [ALeRCE client notebook](https://colab.research.google.com/github/alercebroker/usecases/blob/master/notebooks/ADASS_XXXII_Nov2022_tutorial/ALeRCE_Client_starter.ipynb) 8 | The ALeRCE client is a library to interact programmatically with the ALeRCE API. It allows you to interact more easily with the API via a Python library. You can query the light curve, probabilities, or features associated with different objects via simple commands in Python. 9 | 10 | [ALeRCE SN Starter notebook](https://colab.research.google.com/github/alercebroker/usecases/blob/master/notebooks/ALeRCE_SN_Starter.ipynb) 11 | This notebook shows how to use the ALeRCE client to query and visualize SN related data. This includes object light curves, features and classification probabilities. 12 | 13 | [ALeRCE AGN Starter notebook](https://colab.research.google.com/github/alercebroker/usecases/blob/master/notebooks/ADASS_XXXII_Nov2022_tutorial/ALeRCE_AGN_starter.ipynb) 14 | The notebook is focused on Active Galactic Nuclei (AGN) mostly using the results from the light curve classifier. After a brief introduction, we will use the ALeRCE client to retrieve the light curve for one particular AGN, and then for a sample of sources. Then, we will learn how to use the ALeRCE database to retrieve large samples of AGN. 15 | 16 | [ALeRCE ML notebook (bonus)](https://colab.research.google.com/github/alercebroker/usecases/blob/master/notebooks/ADASS_XXXII_Nov2022_tutorial/ALeRCE_ML_Stamp_Classifier.ipynb) 17 | In this notebook you will explore the data used in the Stamp classifier and an explanation of the model is given. The link to the notebook is the following: 18 | 19 | -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/SanchezSaez+2020_confusion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/SanchezSaez+2020_confusion.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/SanchezSaez+2020_recall.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/SanchezSaez+2020_recall.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/Villar+19.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/Villar+19.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/alerce_explorer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/alerce_explorer.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/alerce_explorer_object.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/alerce_explorer_object.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/alerce_frontend.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/alerce_frontend.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/betas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/betas.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/class_samples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/class_samples.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/cm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/cm.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/cnn_architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/cnn_architecture.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/cnn_table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/cnn_table.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/hierarchical_rf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/hierarchical_rf.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/sanchez_paper_confusion_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/sanchez_paper_confusion_matrix.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/sn_hunter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/sn_hunter.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/taxonomy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/ml/taxonomy.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/recall_agn_qso_blazar_for_mag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/recall_agn_qso_blazar_for_mag.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/recall_agn_qso_blazar_for_ndet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/recall_agn_qso_blazar_for_ndet.png -------------------------------------------------------------------------------- /notebooks/ADASS_XXXII_Nov2022_tutorial/figures/taxonomy_tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ADASS_XXXII_Nov2022_tutorial/figures/taxonomy_tree.png -------------------------------------------------------------------------------- /notebooks/ALeRCE+externalcatalog_xmatch.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "b5a4f6af", 6 | "metadata": {}, 7 | "source": [ 8 | "# Notebook to crossmatch an external catalog with the ALeRCE database" 9 | ] 10 | }, 11 | { 12 | "cell_type": "markdown", 13 | "id": "036397c3", 14 | "metadata": {}, 15 | "source": [ 16 | "Francisco Förster\n", 17 | "2024/01/30" 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 10, 23 | "id": "1a84e0ce", 24 | "metadata": { 25 | "ExecuteTime": { 26 | "end_time": "2024-01-29T23:59:24.803253Z", 27 | "start_time": "2024-01-29T23:59:24.797754Z" 28 | } 29 | }, 30 | "outputs": [], 31 | "source": [ 32 | "import pandas as pd\n", 33 | "import psycopg2\n", 34 | "import requests" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "id": "914936d1", 40 | "metadata": {}, 41 | "source": [ 42 | "## Open a connection using the public credentials" 43 | ] 44 | }, 45 | { 46 | "cell_type": "code", 47 | "execution_count": 11, 48 | "id": "53ea7c2b", 49 | "metadata": { 50 | "ExecuteTime": { 51 | "end_time": "2024-01-29T23:59:25.241439Z", 52 | "start_time": "2024-01-29T23:59:25.190976Z" 53 | } 54 | }, 55 | "outputs": [], 56 | "source": [ 57 | "credentials_file = \"https://raw.githubusercontent.com/alercebroker/usecases/master/alercereaduser_v4.json\"\n", 58 | "params = requests.get(credentials_file).json()[\"params\"]" 59 | ] 60 | }, 61 | { 62 | "cell_type": "code", 63 | "execution_count": 12, 64 | "id": "be80b049", 65 | "metadata": { 66 | "ExecuteTime": { 67 | "end_time": "2024-01-29T23:59:26.856000Z", 68 | "start_time": "2024-01-29T23:59:25.540850Z" 69 | } 70 | }, 71 | "outputs": [], 72 | "source": [ 73 | "conn = psycopg2.connect(dbname=params[\"dbname\"], user=params[\"user\"], host=params[\"host\"], password=params[\"password\"])" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "id": "3cc774c8", 79 | "metadata": { 80 | "ExecuteTime": { 81 | "end_time": "2024-01-29T21:29:44.360464Z", 82 | "start_time": "2024-01-29T21:29:44.356089Z" 83 | } 84 | }, 85 | "source": [ 86 | "## Example external catalog" 87 | ] 88 | }, 89 | { 90 | "cell_type": "code", 91 | "execution_count": 13, 92 | "id": "eada2072", 93 | "metadata": { 94 | "ExecuteTime": { 95 | "end_time": "2024-01-29T23:59:28.675379Z", 96 | "start_time": "2024-01-29T23:59:27.956525Z" 97 | } 98 | }, 99 | "outputs": [ 100 | { 101 | "data": { 102 | "text/html": [ 103 | "
\n", 104 | "\n", 117 | "\n", 118 | " \n", 119 | " \n", 120 | " \n", 121 | " \n", 122 | " \n", 123 | " \n", 124 | " \n", 125 | " \n", 126 | " \n", 127 | " \n", 128 | " \n", 129 | " \n", 130 | " \n", 131 | " \n", 132 | " \n", 133 | " \n", 134 | " \n", 135 | " \n", 136 | " \n", 137 | " \n", 138 | " \n", 139 | " \n", 140 | "
id_sourceradec
0source_1160.18301433.016467
1source_2174.21524944.837895
\n", 141 | "
" 142 | ], 143 | "text/plain": [ 144 | " id_source ra dec\n", 145 | "0 source_1 160.183014 33.016467\n", 146 | "1 source_2 174.215249 44.837895" 147 | ] 148 | }, 149 | "execution_count": 13, 150 | "metadata": {}, 151 | "output_type": "execute_result" 152 | } 153 | ], 154 | "source": [ 155 | "# The catalog should have id_source, ra, and dec columns\n", 156 | "df = pd.read_csv(\"https://github.com/alercebroker/usecases/blob/master/example_data/watchlist.csv?raw=True\")\n", 157 | "df.head()" 158 | ] 159 | }, 160 | { 161 | "cell_type": "code", 162 | "execution_count": 14, 163 | "id": "fc8d6388", 164 | "metadata": { 165 | "ExecuteTime": { 166 | "end_time": "2024-01-29T23:59:28.733613Z", 167 | "start_time": "2024-01-29T23:59:28.729228Z" 168 | } 169 | }, 170 | "outputs": [], 171 | "source": [ 172 | "def ztf_crossmatch(conn, df, search_radius=1):\n", 173 | " \n", 174 | " '''\n", 175 | " conn: connection to database\n", 176 | " df: external catalog dataframe (with columns id_source, ra, dec)\n", 177 | " search_radius: external radius in arcsec (default=1)\n", 178 | " \n", 179 | " The output is a dataframe with the source id, ra, and dec, \n", 180 | " as well as the ALeRCE database meanra, meandec, the crossmatch distance \n", 181 | " in degrees and the time of first detection according to the ALeRCE database\n", 182 | " '''\n", 183 | " \n", 184 | " # fix the dataframe\n", 185 | " objects = []\n", 186 | " for _,row in df.iterrows():\n", 187 | " objects.append(f\"(\\'{row.id_source}\\', {row.ra}, {row.dec})\")\n", 188 | " objects_str = \",\\n\".join(objects)\n", 189 | " \n", 190 | " # convert the radius into degrees\n", 191 | " search_radius = search_radius / 3600\n", 192 | " \n", 193 | " # prepare the query\n", 194 | "\n", 195 | " query = \"\"\"\n", 196 | " WITH catalog ( source_id, ra, dec) AS (\n", 197 | " VALUES\n", 198 | " {values}\n", 199 | " )\n", 200 | " SELECT \n", 201 | " c.source_id, c.ra, c.dec, o.oid, o.meanra, o.meandec, q3c_dist(c.ra,c.dec,o.meanra,o.meandec), \n", 202 | " o.firstmjd\n", 203 | " \n", 204 | " FROM object o, catalog c\n", 205 | " /*\n", 206 | " * It is REALLY important to first use the catalog then the object ra,dec for speed. The radius is in degrees.\n", 207 | " */\n", 208 | " WHERE\n", 209 | " q3c_join(c.ra, c.dec,o.meanra, o.meandec, {radius})\n", 210 | " \"\"\"\n", 211 | " \n", 212 | " # final query string\n", 213 | " query_str = query.format(values=objects_str, radius=search_radius) # radius in degrees\n", 214 | " \n", 215 | " # do the query\n", 216 | " try:\n", 217 | " matches = pd.read_sql(query_str,conn)\n", 218 | " matches[\"q3c_dist\"] = matches.q3c_dist * 3600\n", 219 | " matches.rename({\"q3c_dist\": \"dist_arcsec\"}, axis=1, inplace=True)\n", 220 | " return matches\n", 221 | " except:\n", 222 | " print(\"Error accessing the database. Most common causes are timeout errors or wrongly formatted input query.\")" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "id": "5c0a9804", 228 | "metadata": {}, 229 | "source": [ 230 | "## Do the crossmatch\n", 231 | "\n", 232 | "Note that the default timeout is 2 min, if your catalog is too large you should split in smaller catalogs and do many queries." 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": 15, 238 | "id": "8050b22f", 239 | "metadata": { 240 | "ExecuteTime": { 241 | "end_time": "2024-01-29T23:59:30.090544Z", 242 | "start_time": "2024-01-29T23:59:29.341466Z" 243 | } 244 | }, 245 | "outputs": [ 246 | { 247 | "name": "stderr", 248 | "output_type": "stream", 249 | "text": [ 250 | "/home/fforster/anaconda3/lib/python3.8/site-packages/pandas/io/sql.py:761: UserWarning: pandas only support SQLAlchemy connectable(engine/connection) ordatabase string URI or sqlite3 DBAPI2 connectionother DBAPI2 objects are not tested, please consider using SQLAlchemy\n", 251 | " warnings.warn(\n" 252 | ] 253 | }, 254 | { 255 | "data": { 256 | "text/html": [ 257 | "
\n", 258 | "\n", 271 | "\n", 272 | " \n", 273 | " \n", 274 | " \n", 275 | " \n", 276 | " \n", 277 | " \n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | " \n", 288 | " \n", 289 | " \n", 290 | " \n", 291 | " \n", 292 | " \n", 293 | " \n", 294 | " \n", 295 | " \n", 296 | " \n", 297 | " \n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | "
source_idradecoidmeanrameandecdist_arcsecfirstmjd
0source_1160.18301433.016467ZTF18aaacsup160.18297733.0162940.63358558423.484560
1source_2174.21524944.837895ZTF18aaapcut174.21536144.8379970.46376758440.470775
\n", 310 | "
" 311 | ], 312 | "text/plain": [ 313 | " source_id ra dec oid meanra meandec \\\n", 314 | "0 source_1 160.183014 33.016467 ZTF18aaacsup 160.182977 33.016294 \n", 315 | "1 source_2 174.215249 44.837895 ZTF18aaapcut 174.215361 44.837997 \n", 316 | "\n", 317 | " dist_arcsec firstmjd \n", 318 | "0 0.633585 58423.484560 \n", 319 | "1 0.463767 58440.470775 " 320 | ] 321 | }, 322 | "execution_count": 15, 323 | "metadata": {}, 324 | "output_type": "execute_result" 325 | } 326 | ], 327 | "source": [ 328 | "results = ztf_crossmatch(conn, df)\n", 329 | "results" 330 | ] 331 | }, 332 | { 333 | "cell_type": "markdown", 334 | "id": "13a01960", 335 | "metadata": {}, 336 | "source": [ 337 | "## Close the connection" 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": 16, 343 | "id": "c6a8bbbd", 344 | "metadata": { 345 | "ExecuteTime": { 346 | "end_time": "2024-01-29T23:59:32.790951Z", 347 | "start_time": "2024-01-29T23:59:32.786730Z" 348 | } 349 | }, 350 | "outputs": [], 351 | "source": [ 352 | "conn.close()" 353 | ] 354 | }, 355 | { 356 | "cell_type": "code", 357 | "execution_count": null, 358 | "id": "a1a5ca05", 359 | "metadata": {}, 360 | "outputs": [], 361 | "source": [] 362 | } 363 | ], 364 | "metadata": { 365 | "kernelspec": { 366 | "display_name": "Python 3", 367 | "language": "python", 368 | "name": "python3" 369 | }, 370 | "language_info": { 371 | "codemirror_mode": { 372 | "name": "ipython", 373 | "version": 3 374 | }, 375 | "file_extension": ".py", 376 | "mimetype": "text/x-python", 377 | "name": "python", 378 | "nbconvert_exporter": "python", 379 | "pygments_lexer": "ipython3", 380 | "version": "3.8.8" 381 | }, 382 | "toc": { 383 | "base_numbering": 1, 384 | "nav_menu": {}, 385 | "number_sections": true, 386 | "sideBar": true, 387 | "skip_h1_title": false, 388 | "title_cell": "Table of Contents", 389 | "title_sidebar": "Contents", 390 | "toc_cell": false, 391 | "toc_position": {}, 392 | "toc_section_display": true, 393 | "toc_window_display": false 394 | }, 395 | "varInspector": { 396 | "cols": { 397 | "lenName": 16, 398 | "lenType": 16, 399 | "lenVar": 40 400 | }, 401 | "kernels_config": { 402 | "python": { 403 | "delete_cmd_postfix": "", 404 | "delete_cmd_prefix": "del ", 405 | "library": "var_list.py", 406 | "varRefreshCmd": "print(var_dic_list())" 407 | }, 408 | "r": { 409 | "delete_cmd_postfix": ") ", 410 | "delete_cmd_prefix": "rm(", 411 | "library": "var_list.r", 412 | "varRefreshCmd": "cat(var_dic_list()) " 413 | } 414 | }, 415 | "types_to_exclude": [ 416 | "module", 417 | "function", 418 | "builtin_function_or_method", 419 | "instance", 420 | "_Feature" 421 | ], 422 | "window_display": false 423 | } 424 | }, 425 | "nbformat": 4, 426 | "nbformat_minor": 5 427 | } 428 | -------------------------------------------------------------------------------- /notebooks/ALeRCE_API_PanSTARRS_galaxies.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Getting the three nearest PanSTARRS sources from avro packets" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "Francisco Förster\n", 15 | "\n", 16 | "Last modification: 20210118" 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "# Introduction\n", 24 | "\n", 25 | "ZTF objects have a unique identifier which we call `oid`, each object has many associated alerts, one per detection, and each alert as a unique identifier `candid`.\n", 26 | "\n", 27 | "ALeRCE does not store all the information contained in the alerts in our postgresql database. However, we provide access to the individual alerts and their contents through our API. Each alert contains information about the three nearest PanSTARRS sources, so if you access any alert from a given object you can get these objects.\n", 28 | "\n", 29 | "In this example you can see how we get the nearest PanSTARRS objects given an `oid`.\n", 30 | "\n", 31 | "We will use the ALeRCE client and the requests library to interact with the API." 32 | ] 33 | }, 34 | { 35 | "cell_type": "code", 36 | "execution_count": 6, 37 | "metadata": { 38 | "ExecuteTime": { 39 | "end_time": "2022-01-18T21:43:08.394932Z", 40 | "start_time": "2022-01-18T21:43:08.392127Z" 41 | } 42 | }, 43 | "outputs": [], 44 | "source": [ 45 | "from alerce.core import Alerce" 46 | ] 47 | }, 48 | { 49 | "cell_type": "code", 50 | "execution_count": 7, 51 | "metadata": { 52 | "ExecuteTime": { 53 | "end_time": "2022-01-18T21:43:09.105338Z", 54 | "start_time": "2022-01-18T21:43:09.103531Z" 55 | } 56 | }, 57 | "outputs": [], 58 | "source": [ 59 | "import requests" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": 8, 65 | "metadata": { 66 | "ExecuteTime": { 67 | "end_time": "2022-01-18T21:43:14.757627Z", 68 | "start_time": "2022-01-18T21:43:14.753467Z" 69 | } 70 | }, 71 | "outputs": [], 72 | "source": [ 73 | "client = Alerce()" 74 | ] 75 | }, 76 | { 77 | "cell_type": "markdown", 78 | "metadata": {}, 79 | "source": [ 80 | "# Get any candidate id from the object id" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": 9, 86 | "metadata": { 87 | "ExecuteTime": { 88 | "end_time": "2022-01-18T21:43:16.186767Z", 89 | "start_time": "2022-01-18T21:43:16.185079Z" 90 | } 91 | }, 92 | "outputs": [], 93 | "source": [ 94 | "oid = \"ZTF19acqebwl\" #\"ZTF19abguqsi\"" 95 | ] 96 | }, 97 | { 98 | "cell_type": "markdown", 99 | "metadata": {}, 100 | "source": [ 101 | "Extract the minimum id of thos objects that have image stamps, i.e. they also have an associated avro file" 102 | ] 103 | }, 104 | { 105 | "cell_type": "code", 106 | "execution_count": 20, 107 | "metadata": { 108 | "ExecuteTime": { 109 | "end_time": "2022-01-18T21:45:22.127959Z", 110 | "start_time": "2022-01-18T21:45:21.965053Z" 111 | } 112 | }, 113 | "outputs": [], 114 | "source": [ 115 | "candids = client.query_detections(oid=oid, format='pandas')\n", 116 | "candid = candids.loc[candids.has_stamp].candid.min()" 117 | ] 118 | }, 119 | { 120 | "cell_type": "markdown", 121 | "metadata": {}, 122 | "source": [ 123 | "# Get avro info from oid and candid" 124 | ] 125 | }, 126 | { 127 | "cell_type": "code", 128 | "execution_count": 22, 129 | "metadata": { 130 | "ExecuteTime": { 131 | "end_time": "2022-01-18T21:45:25.278655Z", 132 | "start_time": "2022-01-18T21:45:24.790192Z" 133 | } 134 | }, 135 | "outputs": [], 136 | "source": [ 137 | "params = {\"oid\": oid, \"candid\": candid}\n", 138 | "url = \"http://avro.alerce.online\"\n", 139 | "r = requests.get(url = \"%s/get_avro_info\" % url, params = params)\n", 140 | "response = r.json()" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": 24, 146 | "metadata": { 147 | "ExecuteTime": { 148 | "end_time": "2022-01-18T21:45:29.094971Z", 149 | "start_time": "2022-01-18T21:45:29.092682Z" 150 | } 151 | }, 152 | "outputs": [ 153 | { 154 | "data": { 155 | "text/plain": [ 156 | "'aimage, aimagerat, bimage, bimagerat, candid, chinr, chipsf, classtar, clrcoeff, clrcounc, clrmed, clrrms, dec, decnr, diffmaglim, distnr, distpsnr1, distpsnr2, distpsnr3, drb, drbversion, dsdiff, dsnrms, elong, exptime, fid, field, fwhm, isdiffpos, jd, jdendhist, jdendref, jdstarthist, jdstartref, magap, magapbig, magdiff, magfromlim, maggaia, maggaiabright, magnr, magpsf, magzpsci, magzpscirms, magzpsciunc, mindtoedge, nbad, ncovhist, ndethist, neargaia, neargaiabright, nframesref, nid, nmatches, nmtchps, nneg, objectidps1, objectidps2, objectidps3, pdiffimfilename, pid, programid, programpi, ra, ranr, rb, rbversion, rcid, rfid, scorr, seeratio, sgmag1, sgmag2, sgmag3, sgscore1, sgscore2, sgscore3, sharpnr, sigmagap, sigmagapbig, sigmagnr, sigmapsf, simag1, simag2, simag3, sky, srmag1, srmag2, srmag3, ssdistnr, ssmagnr, ssnamenr, ssnrms, sumrat, szmag1, szmag2, szmag3, tblid, tooflag, xpos, ypos, zpclrcov, zpmed'" 157 | ] 158 | }, 159 | "execution_count": 24, 160 | "metadata": {}, 161 | "output_type": "execute_result" 162 | } 163 | ], 164 | "source": [ 165 | "# Show dictionary keys\n", 166 | "\", \".join(list(response['candidate']))" 167 | ] 168 | }, 169 | { 170 | "cell_type": "markdown", 171 | "metadata": {}, 172 | "source": [ 173 | "# get distance, star galaxy score and object id of the the three nearest PS1 objects" 174 | ] 175 | }, 176 | { 177 | "cell_type": "code", 178 | "execution_count": 28, 179 | "metadata": { 180 | "ExecuteTime": { 181 | "end_time": "2022-01-18T21:48:05.003285Z", 182 | "start_time": "2022-01-18T21:48:05.000678Z" 183 | } 184 | }, 185 | "outputs": [ 186 | { 187 | "name": "stdout", 188 | "output_type": "stream", 189 | "text": [ 190 | "objectidps1 118801391366765195\n", 191 | "sgscore1 0.026624999940395355\n", 192 | "distpsnr1 0.44015640020370483\n", 193 | "objectidps2 118801391381720313\n", 194 | "sgscore2 0.9841669797897339\n", 195 | "distpsnr2 15.294926643371582\n", 196 | "objectidps3 118801391410508642\n", 197 | "sgscore3 0.5\n", 198 | "distpsnr3 19.648544311523438\n" 199 | ] 200 | } 201 | ], 202 | "source": [ 203 | "for i in range(3):\n", 204 | " for key in ['objectidps%i' % (i+1), 'sgscore%i' % (i+1), 'distpsnr%i' % (i+1)]:\n", 205 | " print(key, response['candidate'][key])" 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [] 214 | } 215 | ], 216 | "metadata": { 217 | "kernelspec": { 218 | "display_name": "Python 3", 219 | "language": "python", 220 | "name": "python3" 221 | }, 222 | "language_info": { 223 | "codemirror_mode": { 224 | "name": "ipython", 225 | "version": 3 226 | }, 227 | "file_extension": ".py", 228 | "mimetype": "text/x-python", 229 | "name": "python", 230 | "nbconvert_exporter": "python", 231 | "pygments_lexer": "ipython3", 232 | "version": "3.8.8" 233 | }, 234 | "toc": { 235 | "base_numbering": 1, 236 | "nav_menu": {}, 237 | "number_sections": true, 238 | "sideBar": true, 239 | "skip_h1_title": false, 240 | "title_cell": "Table of Contents", 241 | "title_sidebar": "Contents", 242 | "toc_cell": false, 243 | "toc_position": {}, 244 | "toc_section_display": true, 245 | "toc_window_display": false 246 | }, 247 | "varInspector": { 248 | "cols": { 249 | "lenName": 16, 250 | "lenType": 16, 251 | "lenVar": 40 252 | }, 253 | "kernels_config": { 254 | "python": { 255 | "delete_cmd_postfix": "", 256 | "delete_cmd_prefix": "del ", 257 | "library": "var_list.py", 258 | "varRefreshCmd": "print(var_dic_list())" 259 | }, 260 | "r": { 261 | "delete_cmd_postfix": ") ", 262 | "delete_cmd_prefix": "rm(", 263 | "library": "var_list.r", 264 | "varRefreshCmd": "cat(var_dic_list()) " 265 | } 266 | }, 267 | "types_to_exclude": [ 268 | "module", 269 | "function", 270 | "builtin_function_or_method", 271 | "instance", 272 | "_Feature" 273 | ], 274 | "window_display": false 275 | } 276 | }, 277 | "nbformat": 4, 278 | "nbformat_minor": 2 279 | } 280 | -------------------------------------------------------------------------------- /notebooks/ALeRCE_API_check_pixels.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Given a list of object ids, extracts their avros to get pixel coordinates of their detections, plot their coordinates, stamps and overplot in aladin\n", 8 | "\n", 9 | "\n", 10 | "Francisco Förster\n", 11 | "\n", 12 | "Last modification: 20210118\n", 13 | "\n", 14 | "\n", 15 | "In this notebook we explore a series of supernova candidates that were located around the same galaxy and around the same time, but not exactly in the same position. \n", 16 | "\n", 17 | "We found that the x, y position in the CCDs where all very similar, which suggests that in fact these candidates were due to a region of bad pixels in the CCD." 18 | ] 19 | }, 20 | { 21 | "cell_type": "code", 22 | "execution_count": 1, 23 | "metadata": { 24 | "ExecuteTime": { 25 | "end_time": "2022-01-18T21:36:24.665106Z", 26 | "start_time": "2022-01-18T21:36:23.839561Z" 27 | } 28 | }, 29 | "outputs": [], 30 | "source": [ 31 | "import matplotlib.pyplot as plt\n", 32 | "import pandas as pd\n", 33 | "import ipyaladin as ipyal # see installation instructions here: https://github.com/cds-astro/ipyaladin\n", 34 | "from astropy.table import Table, Column\n", 35 | "from astropy import coordinates\n", 36 | "import numpy as np" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 2, 42 | "metadata": { 43 | "ExecuteTime": { 44 | "end_time": "2022-01-18T21:36:24.704234Z", 45 | "start_time": "2022-01-18T21:36:24.673627Z" 46 | } 47 | }, 48 | "outputs": [], 49 | "source": [ 50 | "import requests" 51 | ] 52 | }, 53 | { 54 | "cell_type": "code", 55 | "execution_count": 3, 56 | "metadata": { 57 | "ExecuteTime": { 58 | "end_time": "2022-01-18T21:36:24.715912Z", 59 | "start_time": "2022-01-18T21:36:24.712935Z" 60 | } 61 | }, 62 | "outputs": [], 63 | "source": [ 64 | "from alerce.core import Alerce\n", 65 | "client = Alerce()" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "# The list of candidates" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 4, 78 | "metadata": { 79 | "ExecuteTime": { 80 | "end_time": "2022-01-18T21:36:25.447703Z", 81 | "start_time": "2022-01-18T21:36:25.444022Z" 82 | } 83 | }, 84 | "outputs": [], 85 | "source": [ 86 | "oids = [\"ZTF19adbtqgy\", \"ZTF20aacbijc\", \"ZTF20aaehjjm\", \"ZTF20aaebiae\", \"ZTF20aaejaax\"]" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": { 92 | "ExecuteTime": { 93 | "end_time": "2020-01-08T02:30:27.397935Z", 94 | "start_time": "2020-01-08T02:30:27.390608Z" 95 | } 96 | }, 97 | "source": [ 98 | "# Check avro files for pixel position" 99 | ] 100 | }, 101 | { 102 | "cell_type": "code", 103 | "execution_count": 5, 104 | "metadata": { 105 | "ExecuteTime": { 106 | "end_time": "2022-01-18T21:36:31.175722Z", 107 | "start_time": "2022-01-18T21:36:27.296767Z" 108 | } 109 | }, 110 | "outputs": [ 111 | { 112 | "name": "stdout", 113 | "output_type": "stream", 114 | "text": [ 115 | "ZTF19adbtqgy 1092289313615015065\n", 116 | "ZTF20aacbijc 1097444403615015104\n", 117 | "ZTF20aaehjjm 1100451663615015164\n", 118 | "ZTF20aaebiae 1100187173615015124\n", 119 | "ZTF20aaejaax 1101179013615015179\n" 120 | ] 121 | }, 122 | { 123 | "data": { 124 | "text/html": [ 125 | "
\n", 126 | "\n", 139 | "\n", 140 | " \n", 141 | " \n", 142 | " \n", 143 | " \n", 144 | " \n", 145 | " \n", 146 | " \n", 147 | " \n", 148 | " \n", 149 | " \n", 150 | " \n", 151 | " \n", 152 | " \n", 153 | " \n", 154 | " \n", 155 | " \n", 156 | " \n", 157 | " \n", 158 | " \n", 159 | " \n", 160 | " \n", 161 | " \n", 162 | " \n", 163 | " \n", 164 | " \n", 165 | " \n", 166 | " \n", 167 | " \n", 168 | " \n", 169 | " \n", 170 | " \n", 171 | " \n", 172 | " \n", 173 | " \n", 174 | " \n", 175 | " \n", 176 | " \n", 177 | " \n", 178 | " \n", 179 | " \n", 180 | " \n", 181 | " \n", 182 | " \n", 183 | " \n", 184 | " \n", 185 | " \n", 186 | " \n", 187 | " \n", 188 | " \n", 189 | " \n", 190 | " \n", 191 | " \n", 192 | " \n", 193 | " \n", 194 | " \n", 195 | " \n", 196 | " \n", 197 | " \n", 198 | " \n", 199 | " \n", 200 | " \n", 201 | " \n", 202 | " \n", 203 | " \n", 204 | " \n", 205 | " \n", 206 | " \n", 207 | " \n", 208 | " \n", 209 | " \n", 210 | " \n", 211 | " \n", 212 | " \n", 213 | " \n", 214 | " \n", 215 | " \n", 216 | " \n", 217 | " \n", 218 | " \n", 219 | " \n", 220 | " \n", 221 | "
candidjdradecxposyposrbdrb
oid
ZTF19adbtqgy10922893136150150652.458847e+06132.49984570.3117041682.0258792919.7492680.3657140.999991
ZTF20aacbijc10974444036150151042.458852e+06132.51178770.3136861681.5345462919.6352540.3957140.999998
ZTF20aaehjjm11004516636150151642.458855e+06132.51114870.3123821681.7347412919.5634770.4214290.999996
ZTF20aaebiae11001871736150151242.458855e+06132.51588070.2925081681.7526862919.5683590.3942861.000000
ZTF20aaejaax11011790136150151792.458856e+06132.52199370.2922651681.8149412919.8168950.6414291.000000
\n", 222 | "
" 223 | ], 224 | "text/plain": [ 225 | " candid jd ra dec \\\n", 226 | "oid \n", 227 | "ZTF19adbtqgy 1092289313615015065 2.458847e+06 132.499845 70.311704 \n", 228 | "ZTF20aacbijc 1097444403615015104 2.458852e+06 132.511787 70.313686 \n", 229 | "ZTF20aaehjjm 1100451663615015164 2.458855e+06 132.511148 70.312382 \n", 230 | "ZTF20aaebiae 1100187173615015124 2.458855e+06 132.515880 70.292508 \n", 231 | "ZTF20aaejaax 1101179013615015179 2.458856e+06 132.521993 70.292265 \n", 232 | "\n", 233 | " xpos ypos rb drb \n", 234 | "oid \n", 235 | "ZTF19adbtqgy 1682.025879 2919.749268 0.365714 0.999991 \n", 236 | "ZTF20aacbijc 1681.534546 2919.635254 0.395714 0.999998 \n", 237 | "ZTF20aaehjjm 1681.734741 2919.563477 0.421429 0.999996 \n", 238 | "ZTF20aaebiae 1681.752686 2919.568359 0.394286 1.000000 \n", 239 | "ZTF20aaejaax 1681.814941 2919.816895 0.641429 1.000000 " 240 | ] 241 | }, 242 | "execution_count": 5, 243 | "metadata": {}, 244 | "output_type": "execute_result" 245 | } 246 | ], 247 | "source": [ 248 | "df = []\n", 249 | "for oid in oids:\n", 250 | " \n", 251 | " # get candid\n", 252 | " candid = client.query_detections(oid=oid, format='pandas').candid.min()\n", 253 | " print(oid, candid)\n", 254 | " \n", 255 | " # get avro\n", 256 | " params = {\"oid\": oid, \"candid\": candid}\n", 257 | " url = \"http://avro.alerce.online\"\n", 258 | " r = requests.get(url = \"%s/get_avro_info\" % url, params = params)\n", 259 | " response = r.json()\n", 260 | " \n", 261 | " # extract selected fields\n", 262 | " fields = [\"jd\", \"ra\", \"dec\", \"xpos\", \"ypos\", \"rb\", \"drb\"]\n", 263 | " df.append(pd.DataFrame(data = [[oid, candid] + [response['candidate'][field] for field in fields]],\n", 264 | " columns=[\"oid\", \"candid\", \"jd\", \"ra\", \"dec\", \"xpos\", \"ypos\", \"rb\", \"drb\"]))\n", 265 | "df = pd.concat(df)\n", 266 | "df.set_index(\"oid\", inplace=True)\n", 267 | "df" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "metadata": {}, 273 | "source": [ 274 | "# Plot the x, y positions" 275 | ] 276 | }, 277 | { 278 | "cell_type": "code", 279 | "execution_count": 6, 280 | "metadata": { 281 | "ExecuteTime": { 282 | "end_time": "2022-01-18T21:36:39.738134Z", 283 | "start_time": "2022-01-18T21:36:39.619099Z" 284 | } 285 | }, 286 | "outputs": [ 287 | { 288 | "data": { 289 | "text/plain": [ 290 | "Text(0, 0.5, 'ypos')" 291 | ] 292 | }, 293 | "execution_count": 6, 294 | "metadata": {}, 295 | "output_type": "execute_result" 296 | }, 297 | { 298 | "data": { 299 | "image/png": "iVBORw0KGgoAAAANSUhEUgAAAusAAAGGCAYAAADCe88kAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0vUlEQVR4nO3deZhlVX3v//en54GpgWZsJgUVJYLSAVQ0qGHQeEVNVCAqjoQbNWp+DhjvjQnGxCQajQma4BAxgwSUKQleRBxIVJAWmccGRZqxZbAZe/z+/ti79VBUNd3Vdersrnq/nmc/dc7aa+3zXV3Vpz+9a+19UlVIkiRJ6p4pgy5AkiRJ0vAM65IkSVJHGdYlSZKkjjKsS5IkSR1lWJckSZI6yrAuSZIkdZRhXZI0KST56yTXJbkiyZlJthqh37uSXJXk6iTv7ml/ddu2JsnCnvYZSf4pyZVJLk9ySM++/dv2xUk+nSRt+9ZJzk9yY/t1Xtuett/its5n9xzr2Lb/jUmO7cdrjPDnMSvJD9u5XZ3kTzfkz13SxjGsS5ImnCSHJPnSkObzgX2q6pnADcAHhxm3D/A24ABgX+BlSfZsd18FvAq4cMiwtwFU1a8BhwKfSLL239fPtvv3arcj2vYTgAuqai/ggvY5wEt6+h7XjifJ1sCHgQPb2j68NnyP1Wusw3LgRVW1L7AfcESSg55gjKQxYliXJE0KVfWNqlrVPr0IWDBMt72Bi6vq4bbvd2kCOlV1bVVdP8yYpwPfavvcDdwPLEyyI7BFVV1UzScQfhl4RTvmSOCU9vEpQ9q/XI2LgK3a4xwOnF9V91bVfTT/8ThijF+DJO9Lckl7xv1P2zlVVT3Yjp3ebn6iojRODOuSpMnozcDXh2m/Cnh+km2SzAFeCuzyBMe6HHh5kmlJ9gD2b8fsDCzp6bekbQPYvqruaB/fCWzfPt4ZuHWYMetqH5PXSHIYzdn2A2jOoO+f5AUASaYmuQy4m+Y/DRev489D0hiaNugCJEkaK0kuBmYCmwFbtwET4ANVdV7b50PAKuBfh46vqmuT/CXwDeAh4DJg9RO87BdpzsgvAm4Bvr8eY3pfs5L09Uz1er7GYe324/b5ZjTh/cKqWg3s167zPzPJPlV1Vd8KlvRLhnVJ0oRRVQdCs2YdeGNVvbF3f5I3Ai8DXtwuGxnuGF8AvtD2/3Mee+Z6uP6rgPf0vMb3adbE38djl9osAG5rH9+VZMequqNdgnJ3234bjz2Tv3bMbcAhQ9q/07aP1WsE+Iuq+sd1zPX+JN+mWRdvWJfGgctgJEmTQpIjgPcDL6+qh9fRb7v2664069X/7QmOOyfJ3PbxocCqqrqmXYKyLMlB7R1a3gCc3Q47B1h7R5djh7S/ob1jy0HAL9rjnAcclmRee2HpYcB5fXiNNyfZrJ3Lzkm2SzK/PaNOktk0F9Fet64/E0ljxzPrkqTJ4u9plsic397d8KKqOj7JTsDnq+qlbb+vJdkGWAm8varuB0jySuDvgPnAfyW5rKoOB7YDzkuyhuYM9et7XvP3gS8Bs2nWyK9dJ/8x4LQkb6FZOvOatv1cmnXyi4GHgTcBVNW9ST4CXNL2O7Gq7h3j1/hGkr2BH7R/Pg8CrwPmAqckmUpzku+0qvrPJ/rDljQ2MsJvASVJkiQNmMtgJEmSpI4yrEuSJEkd5Zr1ddh2221r9913H3QZkjQqP/rRj35eVfMHXcd48T1b0qZspPdsw/o67L777ixatGjQZUjSqCS5ZdA1jCffsyVtykZ6z3YZjCRJktRRhnVJkiSpowzrkiRJUkcZ1iVJkqSOMqxLkiRJHWVYlyRJkjrKsC5JkiR1lGFdkiRJ6ijDutRxVcXKNasHXYYkaYxUraFq1aDL0CbCTzCVOmr56lV89EcXcPpNV7B89Sr2nrc9f3bg4Txr250HXZokaRRqzcPUA38Gj5wDrKKm70O2OJFMf/qgS1OHeWZd6qh3/vdZnH7TFTy6ehUFXHPfXfzuN7/CT5bdO+jSJEmjUPcf3wb1FcAaWHkFde8x1Oo7Bl2aOsywLnXQkgfv58Lbb+bR1Y/9NemK1av4wrU/HFBVkqTRqlWLYcVlNEG9d8dK6uF/GURJ2kQY1qUOuuWB+5kx9fGr1FZXce19dw+gIknSRln1E8hwq49Xwsrrxr0cbToM61IHPXnLbVi++vEXH02fMoV9t91xABVJkjbKtL2gVg6zYwZM33fcy9Gmw7AuddAOczbnpbs+jVlDzq7PmDKNNz/tgAFVJUkarUzbHWYeDMzqbYXMInOOGVBV2hQY1qWO+uvnvozjn3EQW8+cw4wpU3neDrtzxhFvYMFmWw66NEnSKGSrv4W5b4BsBcyEGS8g25xOpm476NLUYd66UeqoaVOm8K5nPp93PfP5gy5FkjQGkhlk8/fC5u8ddCnahHhmXZIkSeoow7okSZLUUYZ1SZIkqaMM65IkSVJHGdYlSZKkjjKsS5IkSR1lWJckSZI6yrAuSZIkdZRhXZIkSeoow7okSZLUUdMGXYA0Wa2pldz+i7/l7ge+zOp6iLkz9mW3rU9k7ox9Bl2aJEnqCM+sSwPy03vez50PfJ7VtQxYzUMrLuW6u17Doyt/OujSJElSRxjWpQFYsfpu7n34XKoefUz7mlrOnQ/844CqksZekiOSXJ9kcZIThtm/a5JvJ/lxkiuSvLRn3wfbcdcnOXx8K5ekbjCsSwOwfOVPmJIZw+xZzUMrrh73eqR+SDIVOAl4CfB04OgkTx/S7f8Ap1XVs4CjgM+0Y5/ePn8GcATwmfZ4kjSpGNalAZg5bXfW1Iph9kxlzvShWUbaZB0ALK6qm6tqBXAqcOSQPgVs0T7eEri9fXwkcGpVLa+qnwCL2+NJ0qRiWJcGYMa07Zk353CmZNZj2qdkJjtu8XsDqkoaczsDt/Y8X9K29foT4HVJlgDnAu/cgLEkOS7JoiSLli5dOlZ1S1JnGNalAdljm0+w3WZvZEo2A8LcGfvy1O2+wqzpewy6NGk8HQ18qaoWAC8F/jnJev/bVFUnV9XCqlo4f/78vhUpSYPirRulAZmS6ewy7wR2mXcCVUWSQZckjbXbgF16ni9o23q9hWZNOlX1gySzgG3Xc6wkTXieWZc6wKCuCeoSYK8keySZQXPB6DlD+vwMeDFAkr2BWcDStt9RSWYm2QPYC/jhuFUuSR3hmfUxdN0Pb+Sb/3Ihq1et4YVHPY9fe/7ehjBJk1ZVrUryDuA8YCrwxaq6OsmJwKKqOgf4/4DPJXkPzcWmb6yqAq5OchpwDbAKeHtVrR7MTCRpcAzrY+RLHz6Vr37iP1nx6Aqo4pv//F0OPfYQ/uDv3zro0iRpYKrqXJoLR3vb/rjn8TXA80YY+1Hgo30tUJI6zmUwY2DJjXdw+l+fw/KHl1Nriip49KHlfONL3+b6RTcNujxJkiRtogzrY+CH515K1ePbVzy6kh+cc8n4FyRJkqQJwbA+BmbMmsGUqY//o5w6dQoz58wcQEWSJEmaCAzrY+DgVx1ADXNqfcq0qRzy2ucOoCJJkiRNBIb1MbDV/C354L/8ATNnz2D25rOYvdksZsyazh+c9FZ23GP7QZcnSZKkTZR3gxkjB7/yQP799pO5+Nwfs2b1Gg54ybPYYpvNB12WJEmSNmGG9TE0d8u5vOjogwddhiRJkiYIl8FIkiRJHWVYlyRJkjrKsC5JkiR1lGFdkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcZ1iVJkqSOMqxLkiRJHWVYlyRJkjrKsC5JkiR1lGFdkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcNLKwneWqSy3q2ZUnenWTrJOcnubH9Om+YsbslubQdd3WS43v2fTTJrUkeHDJmZpJ/T7I4ycVJdh+HaUqSJEmjNrCwXlXXV9V+VbUfsD/wMHAmcAJwQVXtBVzQPh/qDuA57dgDgROS7NTu+w/ggGHGvAW4r6r2BD4J/OUYTkeSJEkac11ZBvNi4KaqugU4EjilbT8FeMXQzlW1oqqWt09n0jOPqrqoqu4Y5jV6j/tV4MVJMjblS5IkSWOvK2H9KOAr7ePte8L2ncD2ww1IskuSK4Bbgb+sqtuf4DV2bvtSVauAXwDbDHPc45IsSrJo6dKlGz4TSZIkaYwMPKwnmQG8HDh96L6qKqCGG1dVt1bVM4E9gWOTDBvqN1RVnVxVC6tq4fz588fikJIkSdKoDDysAy8BLq2qu9rndyXZEaD9eve6Brdn1K8Cnv8Er3MbsEt73GnAlsA9G1G3JEmS1FddCOtH86slMADnAMe2j48Fzh46IMmCJLPbx/OAg4Hrn+B1eo/7O8C32jP3kiRJUicNNKwnmQscCpzR0/wx4NAkNwK/2T4nycIkn2/77A1cnORy4LvAx6vqyrbfXyVZAsxJsiTJn7RjvgBsk2Qx8IcMf5cZSZIkqTOmDfLFq+ohhlzkWVX30NwdZmjfRcBb28fnA88c4ZjvB94/TPujwKs3vmpJkiRpfHRhGYwkSZKkYRjWJUmSpI4yrEuSJEkdZViXJEmSOsqwLkmSJHWUYV2SJEnqKMO6JEmS1FGGdUmSJKmjDOuSJElSRxnWJUmSpI4yrEuSJEkdZViXJEmSOsqwLkmSJHWUYV2SJEnqKMO6JEmS1FGGdUmSJKmjDOuSJElSRxnWJUmSpI4yrEuS+ibJEUmuT7I4yQnD7P9kksva7YYk9/fsW92z75xxLVySOmLaoAuQJE1MSaYCJwGHAkuAS5KcU1XXrO1TVe/p6f9O4Fk9h3ikqvYbp3IladQeuO9BvvTH/86Fp32fTJ3Cb77uBbz+w69m9txZG31sw7okqV8OABZX1c0ASU4FjgSuGaH/0cCHx6k2SRoTq1au4l3P+z/ccfNdrFqxCoCz/u7rXPHdq/m7i/6CJBt1fJfBSJL6ZWfg1p7nS9q2x0myG7AH8K2e5llJFiW5KMkr+lalJG2E7599CT9fcs8vgzrAyuUr+dm1t3HZt6/a6OMb1iVJXXAU8NWqWt3TtltVLQSOAT6V5MlDByU5rg30i5YuXTpetUrSL92w6CYeefDRx7WvXL6SxT/+6UYf37AuSeqX24Bdep4vaNuGcxTwld6Gqrqt/Xoz8B0eu559bZ+Tq2phVS2cP3/+WNQsSRtkpz13ZNbcmY9rnzFrBjvssd1GH9+wLknql0uAvZLskWQGTSB/3F1dkjwNmAf8oKdtXpKZ7eNtgecx8lp3SRqYQ177XGbMmv6YtelTpk5hzhazOehlz97o4xvWJUl9UVWrgHcA5wHXAqdV1dVJTkzy8p6uRwGnVlX1tO0NLEpyOfBt4GO9d5GRpK6Ys/lsPvU/f8ZTD9iTadOnMnX6VPY5+Gn87ff+jOkzpm/08fPY90b1WrhwYS1atGjQZUjSqCT5Ubvme1LwPVvSoD207GGSMGfz2Rs8dqT3bG/dKEmSJI2BuVvMGfNjugxGkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcZ1iVJkqSOMqxLkiRJHWVYlyRJkjrKsC5JkiR1lGFdkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcZ1iVJkqSOMqxLkiRJHWVYlyRJkjrKsC5JkiR1lGFdkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcZ1iVJkqSOMqxLkiRJHWVYlyRJkjrKsC5JkiR1lGFdkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcZ1iVJkqSOMqxLkiRJHWVYlyRJkjrKsC5JkiR11MDCepKnJrmsZ1uW5N1Jtk5yfpIb26/zhhm7W5JL23FXJzm+Z9/+Sa5MsjjJp5Okbf+TJLf1vN5Lx3O+kiRJ0oYaWFivquurar+q2g/YH3gYOBM4AbigqvYCLmifD3UH8Jx27IHACUl2avd9FngbsFe7HdEz7pNrX7Oqzu3DtCRJkqQx05VlMC8GbqqqW4AjgVPa9lOAVwztXFUrqmp5+3Qm7TyS7AhsUVUXVVUBXx5uvCRJkrQp6EpYPwr4Svt4+6q6o318J7D9cAOS7JLkCuBW4C+r6nZgZ2BJT7clbdta70hyRZIvDre8RpIkSeqSgYf1JDOAlwOnD93Xnh2v4cZV1a1V9UxgT+DYJMOG+h6fBZ4M7EezjOYTI9RzXJJFSRYtXbp0vechSZIkjbWBh3XgJcClVXVX+/yudjnL2mUtd69rcHtG/Srg+cBtwIKe3QvaNqrqrqpaXVVrgM8BB4xwvJOramFVLZw/f/5GTEuSJEnaOF0I60fzqyUwAOcAx7aPjwXOHjogyYIks9vH84CDgevb5TPLkhzU3gXmDWvHr/0PQOuVNAFfkiRJ6qyBhvUkc4FDgTN6mj8GHJrkRuA32+ckWZjk822fvYGLk1wOfBf4eFVd2e77feDzwGLgJuDrbftftbd0vAJ4IfCe/s1MkiRJ2njTBvniVfUQsM2Qtnto7g4ztO8i4K3t4/OBZ45wzEXAPsO0v34MSpYkSZLGTReWwUiSJEkahmFdkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcN9G4wkqRuSTIVmFlVD/e0bQW8BdgaOLXnVrmSpD4zrEuSev0jcBDtLXCTTAf+B3h6u/8Pkzynqi4bTHmSNLm4DEaS1Otgmk+SXut3aIL624HnAncBJwygLkmalAzrkqReOwI/6Xn+W8DVVfXZqroIOBl4zvoeLMkRSa5PsjjJ40J+kk8muazdbkhyf8++Y5Pc2G7Hjn5KkrTpchmMJKlXgKk9zw8Bzuh5fgew3XodqFn/fhJwKLAEuCTJOVV1zdo+VfWenv7vBJ7VPt4a+DCwECjgR+3Y+0YxJ0naZG3wmfUkb0/yzXXs/0aS39u4siRJA/IT4HCAJM+jOdP+7Z79OwG/WM9jHQAsrqqbq2oFcCpw5Dr6Hw18pX18OHB+Vd3bBvTzgSPWexaSNEGMZhnMG4Eb17H/BuDNo6pGkjRo/wQcmeQq4D+Bu4HzevYfCFy3nsfaGbi15/mStu1xkuwG7AF8a0PGJjkuyaIki5YuXbqeZUnSpmM0YX0vYF237bq67SNJ2vT8Lc3yk+XAj4FXrr2NY5JtaO4Uc24fXvco4KtVtXpDBlXVyVW1sKoWzp8/vw9lSdJgjWbN+nRg1jr2z3qC/ZKkjqqqAj7SbkP33cN6rldv3Qbs0vN8Qds2nKNo7jjTO/aQIWO/swGvLUkTwmjOrN9Ac7HQSA4DbhpdOZKkLkmybZJtRzn8EmCvJHskmUETyM8Z2inJ04B5wA96ms8DDksyL8k8mn9bzhs6VpImutGE9a/QvIF+pH3zBZoPzkjypzRvqP82VgVKksZXkp2SnNLeRvEu4K4k9yX5UpJh15wPp6pWAe+gCdnXAqdV1dVJTkzy8p6uR9F8Mmr1jL2X5uz+Je12YtsmSZNKet4b129A82l23wB+A7iXX11o9DSaj6L+b+DQ9sr/TdrChQtr0aJFgy5DkkYlyY+qauEGjtkVuAjYAbiM5jokaD4Y6Vk0t248qKpuHfYAA+R7tqRN2Ujv2Rt8Zr2qVtKcPT+B5ur8Z7XbrcD7gd+cCEFdkiapj9AsSXlZVT27ql7fbvvTfEDS1gyznl2S1B+j+lCkNrD/VbtJkiaOw4DPVNXj7vhSVV9P8lngmPEvS5Imp9GsWZckTVzzWPdnadwIbDU+pUiSRhXWk+yS5ItJliRZkeRFbfv8tv3Xx7ZMSdI4WcJjb5k41AvaPpKkcbDBYT3JHsAi4LdpLjyaunZfVS0FFgJvHasCJUnj6nTg1Un+IsmWaxuTbJHkz4HXAP8+sOokaZIZzZr1jwJrgH2AR2g+irrXucD/2si6JEmD8RHg+cAHgPcmub1t34nm5Mz3gD8bUG2SNOmMZhnMb9JcfHQrMNx9H2+h+aQ5SdImpqoeplkG83s0t+l9qN3OA44DXlhVjwysQEmaZEZzZn0LmvvsjmTGKI8rSeqA9sOMPtdukqQBGs2Z9VuBZ6xj/0HA4tGVI0nqkiSzk8wedB2SNFmNJqyfAbw5yT49bQWQ5LeBVwOnjUFtkqQBSLJdks+069UfBB5Mckfbtv2g65OkyWQ0Yf2jNLftuhj4F5qgfkKSH9CE9MuBT4xZhZKkcdPe8evHwPHAL4Cz2+3+tu3SJE8aWIGSNMlscFivqmXAc4DP09ymMcChwFOBz9BcfPToWBYpSRo3nwC2AV5VVXtX1avabW+aW/ZuA3x8oBVK0iQyqgtB28D+LuBdSebTBPalVTXc3WEkSZuOFwMnVdVZQ3dU1ZlJPgu8edyrkqRJaoPDepLnAr8FPIXmzjDLgOuA/wIuGtPqJEnjrYAb17H/Boa/ba8kqQ/WO6wn2QL4CnAEzZn0of4oyX8Bv1tVD4xRfZKk8fVd4IXAP4yw/xDgO+NVjCRNdhuyZv2rwEtoPr3uTcD+wF7t1zcB3wdehh9DLUmbsncDByX5RJLt1ja2d4j5G+DAto8kaRys15n1JIfTfHLpJ6rqfcN0+TFwSpKPA+9JcmhVnT+GdUqSxscFwCyaQP7uJPe37Vu1X38OfCt5zC9Yq6qePE71SdKksr7LYI4GbgHe/wT93k9zt4BjAMO6JG16foZr0iWpM9Y3rO8PnPVEd3upqjVJzqI5Cy9J2sRU1SGDrkGS9Cvru2Z9Z+D69ex7PbBgdOVIkgYpyQ1JPuAnlUpSN6xvWN8CWN87vDwAbDa6ciRJA7YS+Avg1iRnJXlZktF82rUkaQys7xvwFDZsDaNv7JK0CaqqZwDPBU6huYXj2TTB/aNJvIhUksbZhnwo0kuT7LAe/fYfbTGSpMGrqouAi5K8C3gt8Bbgg8AJSb4LfB74WlUtH2CZkjQpbEhYP6bd1od3EpCkTVxVPQz8E/BPSZ4CfJjm7mC/Afxdkn8G/qaqfjbAMiVpQlvfsP7CvlYhSeqkJFOBl9OcXT+C5mTMt4HlwDuAtyU5pqrOHlyVkjRxrVdYr6rv9rsQSVJ3JHkaTUB/PbAdcDfwceBzVXVT22dP4DTgr2jWtkuSxtiGLIORJE1wSd4CvBk4qG36JnAycHZVrertW1WLk3yaZg27JKkPDOuSpF6fA+4EPkZzFv2nT9D/GuCf+12UJE1WhnVJUq9XAf9RVavXp3NV/RD4YX9LkqTJy7AuSfqlqjpr0DVIkn7FDy+SJEmSOsqwLkmSJHWUYV2SJEnqKMO6JEmS1FGGdUmSJKmjDOuSJElSRxnWJUmSpI4yrEuSJEkdZViXJEmSOsqwLkmSJHWUYV2SJEnqKMO6JEmS1FGGdUmSJKmjBhbWkzw1yWU927Ik706ydZLzk9zYfp03zNjdklzajrs6yfE9+/ZPcmWSxUk+nSRt+xMeV5IkSeqSgYX1qrq+qvarqv2A/YGHgTOBE4ALqmov4IL2+VB3AM9pxx4InJBkp3bfZ4G3AXu12xFt+/ocV5IkSeqMriyDeTFwU1XdAhwJnNK2nwK8YmjnqlpRVcvbpzNp55FkR2CLqrqoqgr4cs/4JzyuJEmS1CVdCetHAV9pH29fVXe0j+8Eth9uQJJdklwB3Ar8ZVXdDuwMLOnptqRtW+/jSpIkSV0x8LCeZAbwcuD0ofvas+M13LiqurWqngnsCRybZL3D97qOm+S4JIuSLFq6dOn6HlKSJEkacwMP68BLgEur6q72+V3tcpa1y1ruXtfg9oz6VcDzgduABT27F7Rt633cqjq5qhZW1cL58+ePckqSJEnSxutCWD+aXy2BATgHOLZ9fCxw9tABSRYkmd0+ngccDFzfLnNZluSg9i4wb+gZ/4THlSSNrSRHJLm+vUPXsBf2J3lNkmvau3v9W0/76p47hp0zflVLUndMG+SLJ5kLHAr8Xk/zx4DTkrwFuAV4Tdt3IXB8Vb0V2Bv4RJICAny8qq5sx/8+8CVgNvD1dhvxuJKk/kgyFTiJ5n1+CXBJknOq6pqePnsBHwSeV1X3Jdmu5xCPtHf9kqRJa6BhvaoeArYZ0nYPzd1hhvZdBLy1fXw+8MwRjrkI2GeY9mGPK0nqmwOAxVV1M0CSU2nuzHVNT5+3ASdV1X0AVbXOpY+SNNl0YRmMJGli2pnmjl1r9d6ha62nAE9J8r0kFyU5omffrPaC/4uSvKLPtUpSJw30zLokadKbRvMBdofQ3BTgwiS/VlX3A7tV1W1JngR8K8mVVXVT7+AkxwHHAey6667jWrgkjQfPrEuS+uU2YJee57136FprCXBOVa2sqp8AN9CEd6rqtvbrzcB3gGcNfQHv4CVpojOsS5L65RJgryR7tJ+pcRTNnbl6nUVzVp0k29Isi7k5ybwkM3van8dj17pL0qTgMhhJUl9U1aok7wDOA6YCX6yqq5OcCCyqqnPafYcluQZYDbyvqu5J8lzgH5OsoTmx9LHeu8hI0mRhWJck9U1VnQucO6Ttj3seF/CH7dbb5/vAr41HjZLUZS6DkSRJkjrKsC5JkiR1lGFdkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcZ1iVJkqSOMqxLkiRJHeWHIkmSJqVfLH+E/7rlOn7+6EMcsP2uHLjdLiQZdFmS9BiGdUnSpHPp0iW84YJ/Z00Vj65eyexp01k4fwGff+GrmT5l6qDLk6RfchmMJGlSWVPF/77wTB5atYJHVq+kgIdXreSSu2/l9MVXDLo8SXoMw7okaVK57v67eXDl8se1P7J6FafddPkAKpKkkRnWJUmTShh5Xbpr1iV1jWFdkjSpPHWr+Wwxfdbj2mdPnc5rnvzMAVQkSSMzrEuSJpUpCZ/9jVex2fQZzJk2nakJc6ZN5zk77Mqrn7zvoMuTpMfwbjCSpElnv2134vuvfAfn/qy5deOB2+3K/vN3dhmMpM4xrEuSJqXNZ8zktXt6Jl1St7kMRpIkSeoow7okSZLUUYZ1SZIkqaMM65IkSVJHGdYlSZKkjjKsS5IkSR1lWJckSZI6yrAuSZIkdZRhXZIkSeoow7okSZLUUYZ1SZIkqaMM65IkSVJHGdYlSZKkjjKsS5IkSR1lWJckSZI6yrAuSZIkdZRhXZIkSeoow7okSZLUUYZ1SZIkqaMM65IkSVJHGdYlSZKkjjKsS5IkSR1lWJckSZI6yrAuSZIkdZRhXZIkSeoow7okSZLUUYZ1SZIkqaMM65IkSVJHGdYlSZKkjjKsS5IkSR1lWJckSZI6yrAuSZIkdZRhXZIkSeoow7okqW+SHJHk+iSLk5wwQp/XJLkmydVJ/q2n/dgkN7bbseNXtSR1x7RBFyBJmpiSTAVOAg4FlgCXJDmnqq7p6bMX8EHgeVV1X5Lt2vatgQ8DC4ECftSOvW+85yFJg+SZdUlSvxwALK6qm6tqBXAqcOSQPm8DTlobwqvq7rb9cOD8qrq33Xc+cMQ41S1JnWFYlyT1y87ArT3Pl7RtvZ4CPCXJ95JclOSIDRhLkuOSLEqyaOnSpWNYuiR1g2FdkjRI04C9gEOAo4HPJdlqfQdX1clVtbCqFs6fP78/FUrSAA0srCd5apLLerZlSd6dZOsk57cXFJ2fZN4wY/dL8oP2YqQrkry2Z9+Lklya5KokpySZ1rYfkuQXPa/3x+M5X0mahG4Ddul5vqBt67UEOKeqVlbVT4AbaML7+oyVpAlvYGG9qq6vqv2qaj9gf+Bh4EzgBOCCqtoLuKB9PtTDwBuq6hk0axg/lWSrJFOAU4Cjqmof4Bag9w4C/732NavqxL5NTpIEcAmwV5I9kswAjgLOGdLnLJqz6iTZlmZZzM3AecBhSea1J20Oa9skaVLpyjKYFwM3VdUtNBcfndK2nwK8Ymjnqrqhqm5sH98O3A3MB7YBVlTVDW3X84Hf7m/pkqThVNUq4B00Ifta4LSqujrJiUle3nY7D7gnyTXAt4H3VdU9VXUv8BGawH8JcGLbJkmTSldu3XgU8JX28fZVdUf7+E5g+3UNTHIAMAO4ieb2XtOSLKyqRcDv8Nhfoz4nyeXA7cB7q+rqYY53HHAcwK677jr6GUmSqKpzgXOHtP1xz+MC/rDdho79IvDFftcoSV028DPr7a9GXw6cPnRf+yZe6xi7I/DPwJuqak3b/yjgk0l+CDwArG67XwrsVlX7An9H86vXx/FiJUmSJHXFwMM68BLg0qq6q31+VxvC14bxu4cblGQL4L+AD1XVRWvbq+oHVfX8qjoAuJDmYiWqallVPdg+PheY3q6PlCRJkjqpC2H9aH61BAaai4/WXhR6LHD20AHt2fgzgS9X1VeH7Fv76XczgQ8A/9A+3yFJ2scH0Mz9njGdiSRJkjSGBhrWk8yl+RjqM3qaPwYcmuRG4Dfb5yRZmOTzbZ/XAC8A3thzK8b92n3vS3ItcAXwH1X1rbb9d4Cr2jXrn6a5Y8yIS2wkSZKkQYt5dWQLFy6sRYsWDboMSRqVJD+qqoWDrmO8+J4taVM20nt2F5bBSJIkSRqGYV2SJEnqKMO6JEmS1FGGdUmSJKmjDOuSJElSRxnWJUmSpI4yrEuSJEkdZViXJEmSOsqwLkmSJHWUYV2SJEnqKMO6JEmS1FGGdUmSJKmjDOuSJElSRxnWJUmSpI4yrEuSJEkdZViXJEmSOsqwLkmSJHWUYV2SJEnqKMO6JEmS1FGGdUmSJKmjDOuSJElSRxnWJUmSpI4yrEuSpAmtqnhgxY3cv/xK1tTKQZcjbZBpgy5AkiSpXx5ccTOL7noHy1cvJUyBTGXf+X/O9nMOGXRp0nrxzLokSZqQ1tQqLr7zzTy86lZW1yOsqodYtWYZP777vTy08meDLk9aL4Z1SePizofO539uezXfvOX5XHLn21m24vpBlyRpgrvnkYtYteYRoB7TXrWKWx/42mCKkjaQYV1S392y7FQuW/pBlq24lhVr7mPpIxfyg9tfxwMrbhx0aZImsOWr72FoUAcoVvHo6rvHvyBpFAzrkvpqTa3k+ns/xZp6tKe1WF2PcsN9fzewuiRNfFvP2p9i9ePap2YO82cfPICKpA3nBaaS+urRVXcP+48lFPcvv2Lc69lUVK2BFT+gVlxCpmwDs19GpswbdFlSpzyy8ibue/jrFMXWc45g9vS9HrN/zvQFLNjsldz24NmsrkcAmJJZzJ2+OzvOPWwQJUsbzLAuqa9mTJ03QliH2dN2GudqNg1VK6j73gYrL4d6mGIWPPgJmPd5MmPhoMuTOuGOZf/Abb/4FFWrgOKOZSex0xbvZKct3/6Yfs/Y5kNsM/vXuWXZqaxe8zA7bfZb7Lr5a5iS6YMpXNpAhnVJfTVtyhx23uxIbnvwPx6zFGZKZrHnVscPsLLuqodPhxU/Btb+eT0KBXX/u2D+f5O4glGT26Mrb+G2X3ySquW/bKtaze3LPs28OUcwe/qTf9mehB3nHs6Ocw8fRKnSRvMdX1LfPWObP2LBZkcyJTOZkplMn7Ilz9jmQ2w35wWDLq2bHjmDXwX1HvUQrLpu3MuRuub+R86HGubC0VrN/Q+fN4CKpP7xzLqkvpuS6eyz7f9l763fx8o1v2Dm1G1Jpg66rO5a55lzz7FIkHYb2pon+PsjbXr8iZY0bqZOmcWsadsb1J9AZr8GmD3Mji1h2lPHvR6pa+bNOXy4rA6ZwrzZLxn3eqR+MqxLUtfMfhXMfB5NYJ8GzIFsRuadRDJcQpEml5nTFrDLVv+XMPMx24ItT2DW9N0GXZ40plwGI0kdk0yFrU6ClVfAyktgyjYw8zAyZe6gS5M6Y/vNX8dWs1/E/Y98Ayi2mn0YM6ftPOiypDFnWJekDkoCM/ZtNknDmjltJ7bf/I2DLkPqK5fBSJIkSR1lWJckSZI6yrAuSZIkdZRhXZIkSeoow7okSZLUUYZ1SZIkqaMM65IkSVJHGdYlSZKkjjKsS5IkSR1lWJckSZI6KlU16Bo6K8lS4JaNOMS2wM/HqJxNyWSdN0zeuU/WeUO3575bVc0fdBHjZQzes7ukyz9XY2Gizw8m/hyd39gb9j3bsN5HSRZV1cJB1zHeJuu8YfLOfbLOGyb33NU/E/3naqLPDyb+HJ3f+HEZjCRJktRRhnVJkiSpowzr/XXyoAsYkMk6b5i8c5+s84bJPXf1z0T/uZro84OJP0fnN05csy5JkiR1lGfWJUmSpI4yrA+R5ItJ7k5yVU/bR5JckeSyJN9IslPb/rtt+5VJvp9k3xGO+aUkP2nHX5Zkv7Y9ST6dZHF7nGf3jDk2yY3tdmyfp92lea/u6X9On6c93vN+WpIfJFme5L1DxhyR5Pr2z+SEPk659zW7Mveftse9LMmiPk557euN57xHHD+I77nGz3A/Z237O5Ncl+TqJH/Vtk1Pckr7c3Jtkg8+0XGGeb1D2p+9q5N8tz+zeszrjdv8kryv5+/WVe2/E1v3b3bjPr8tk/xHksvb476pfzN7zOuO5xznJTmzfT/8YZJ9+jezdde1IfNLskuSbye5pu3/rhFeKxkh14yJqnLr2YAXAM8Grupp26Ln8R8A/9A+fi4wr338EuDiEY75JeB3hml/KfB1IMBBa8cDWwM3t1/ntY/nTfR5t/senMDf7+2AXwc+Cry3p30qcBPwJGAGcDnw9Mkw93bfT4FtJ+j3fNjxg/qeu43fNsLP2QuBbwIz2+fbtV+PAU5tH89p/07sPtJxhnmtrYBrgF17jztR5jfkdf8X8K2JND/gj4C/bB/PB+4FZkywOf418OH28dOACzaF+QE7As9u2zcHbmCY92rWkWvGYvPM+hBVdSHNX5TetmU9T+cC1bZ/v6rua9svAhZs4MsdCXy5GhcBWyXZETgcOL+q7m2Pfz5wxIbPZv11ZN7jbjznXVV3V9UlwMohuw4AFlfVzVW1AjiV5s+orzoy93E3zvMeafxAvucaP8P9nAH/G/hYVS1v+9y9tjswN8k0YDawAli2juMMdQxwRlX9bMhx+2ac59fraOArG1H6ehnn+RWweZIAm7X9V43FPNb5ouM7x6cD32r7XwfsnmT7sZjHSMZiflV1R1Vd2vZ9ALgW2HmYl+trrjGsr6ckH01yK/C7wB8P0+UtNP+rGslH21+NfDLJzLZtZ+DWnj5L2raR2sfdOM8bYFaSRUkuSvKKjSx/1Po075F05vsN4z53aN4kv5HkR0mOG0XJY2Ic5t07vlPfc42bpwDPT3Jxku8m+fW2/avAQ8AdwM+Aj1fVhgTYpwDzknyn/Xv0hrEte4Pq6Mf8AEgyh+bE1dfGquAN1K/5/T2wN3A7cCXwrqpaM4Z1b4h+zfFy4FUASQ4AdmPDT/SNhVHPL8nuwLOAi4c5bl/f0w3r66mqPlRVuwD/Cryjd1+SF9L8Q/yBEYZ/kObXPr9Os7RlpH6dM4B571bNJ4YdA3wqyZNHW/vGmKzfbxjI3A+uqmfTLBN5e5IXjLb2jdHPea/HeE0O02h+Pg4C3gec1p5NPQBYDewE7AH8f0metIHH3R/4LZrfzP7fJE8Zy8I3oI5+zG+t/wV8bzRBf4z0a36HA5e14/cD/j7JFmNX9gbp1xw/RnO2+TLgncCP2+ONt1HNL8lmNP9JfPeQ38SOC8P6hvtX4LfXPknyTODzwJFVdc9wA9pfo1T7a5d/ovmhALgN2KWn64K2baT2QRqPeVNVa7/eDHyH5n+xgzSW8x5JF7/fMD5z7/2e3w2cuT5j+mxM5z3C+K5+z9VfS2iWq1RV/RBYA2xLc3Li/1XVyvbvwfeADfmY8yXAeVX1UFX9HLgQGPZi6D7r1/zWOopxWAKzDv2a35t6jrsY+AnNf/4HoS9zrKplVfWmqtoPeAPN2vybx7z6J7bB80synSao/2tVnTHCcfv6nm5YXw9J9up5eiRwXdu+K3AG8PqqumEd43dsvwZ4BbD2yuRzgDe0VxEfBPyiqu4AzgMOS3P19DzgsLZtXI33vNv5zmzHbAs8j+aiqXHVx3mP5BJgryR7JJlB8w9S3++EM5zxnnuSuUk2X/uY5mf9if68xly/5r2O8Z35nmtcnUVzgRvtme8ZwM9pfu3+orZ9Ls1Zv+s24LhnAwcnmdYuFTmQZm3teDuL/syPJFsCv0Ez10E5i/7M72fAi9vx2wNPZTBBFvo0xyRbte91AG8FLhzEGWo2cH7te/oXgGur6m/WcdyR8tzYqD5fjbupbTT/a7+D5kK4JTS/uv4azT++VwD/Aezc9v08cB/Nr68uAxb1HOdcYKf28bdo1qFdBfwLsFnbHuAkmrtCXAks7Bn/ZmBxu71pMsyb5s4ZV9KsbbsSeMsEm/cO7WssA+5vH2/R7nspzVXmNwEfmoA/68POneZuKJe329XjMfdxnve6xo/799xt/LYRfs5mtD8fVwGXAi9q+24GnN7+HbgGeN+6jtO2Hw8c39Pvfe3Yq2h+VT/R5vdG2rt1TLTvH83Si2/0vIe8bgLO8Tnt+931NCcw+nqHu7GaH3AwzXVVV/Cr9/GXDjO/EfPcWGx+gqkkSZLUUS6DkSRJkjrKsC5JkiR1lGFdkiRJ6ijDuiRJktRRhnVJkiSpowzrkiRJUkcZ1qUxluSQJDVkezDJpUnek2TaCONe0vZd3X6YjiRJmuQM61L/fAV4Pc1HK/8pMB34G+AzI/R/C3ArsIrm46clSdIk54ciSWMsySHAt2k+Ae3jPe1zaT6eeWdg+6pa2rNvPnAb8BHgWe32pPIvqCRJk5pn1qVxUlUPARfRfCzxk4fsfj0wDfhn4EvA7sCLx7E8SZpQkkxL8r0kDyV52pB9x7XLDk9sn38nyU+TPCnJ2Ul+kWRZkjOTPGmYY89N8hdJbkqyPMmdSb6cZLch/aYkeXeSK5I80B7z+iRfSDK9v38CmigM69L4WhvS7x3S/mbgu1X1U+Bc4O62TZI0ClW1CjgGWAGcmmQmQJJnAJ8C/odmieJac4HvtP0/CHwBeCnwvSQ7rO3UhuzzgBOAS4H30Cx7fA1wcZIFPcf8EPBJ4KfAB4D3AWcCzwFmjuF0NYENe6GbpDExJ8m2NGfSdwCOp1ne8sOqumFtpyQHAs+gXadeVauS/BtwfJJ5VXXf+JcuSZu+qrolyVuArwGfSPI+4N+BR4HfrarVPd23Bf62qt69tiHJhcAZwJ/QvIcDvBF4HvDXVfX+nr7fBP4T+Aua35YCvBK4tqpePqS0E8ZifpocXLMujbGeNevDOQN4e1Xd2dP/ZJqzPztU1YNt277AZcA7quqkftYrSRNdks8A/xv4PvBc4Ler6oye/d8BfgPYsff9ud13HbBVVe3QPj8XOBzYdujJlCQ/pvkN6lZVtSbJt4FnA79VVf/Tr/lpYnMZjNQ/JwOH0vwa9QM0S18W0JzRAX550elRNL963SHJnkn2BB4CFtPcIUaStHH+ELiJJqh/rjeo97h/aFBvXQts375fA+wB3D7Cbz2vBjanOUsP8Ec07/n/neS2JP+a5JgkMzZmMppcXAYj9c+NVfXN9vHXk/wPzRrJf6AJ6ACvpnlj/612e5wk+1XVZX2uVZImsn2BtZ9fsU+Sae2a9r6qqh8keTLNmfgXttsxwP9JcnBVDb1+SXocz6xL46Sqvk9zt5fXJnlu2/xm4Haa0D50+11gNZ5dl6RRS7IFzQWgP6e54PM5PPbC0rW26r2QtMfewN3tHb0AbgZ2SrLVMH2fDixrXwuAqnqwqr5WVe+oqmcAb2+P6Xu71othXRpfH6EJ4CcmeQrwfOBrVfXVYbZ/A/4bOGbtXQwkSRvsZGA34HVV9efAV4ETkrxwmL6PufAzySuBpwJn9TSfRZOfhvZ9Cc1NBM6pqjVt27Y83qXt1603dCKanFwGI42jqlqc5FSas+afbZu/to4hXwMOobmjwKn9rU6SJpb2TjCvBf68qr7VNr8N+HXgX5I8s6ruadt/DrwqyU401xHtBfw+cBfN3WDW+hJwLPCBJLsDFwJ79vT9o56+1ya5CLiY5reoOwLH0d5OcgynqgnMM+vS+Pto+/VFwFKas+cjORMovOe6JG2Q9oOQPk1zB5gPr22vqvuBo4HtgH/qGfIQzcmRmcDHgLcC/w84uKru6Bm/kmYN+seAA2ju2f464HTgwKq6teeYnwC2BP6A5gTN8cAPgedU1eVjNVdNbN66UZIkTWrtrRt3r6rdB1yK9DieWZckSZI6yrAuSZIkdZRhXZIkSeoo16xLkiRJHeWZdUmSJKmjDOuSJElSRxnWJUmSpI4yrEuSJEkdZViXJEmSOsqwLkmSJHXU/w/z8wacklLZUQAAAABJRU5ErkJggg==\n", 300 | "text/plain": [ 301 | "
" 302 | ] 303 | }, 304 | "metadata": { 305 | "needs_background": "light" 306 | }, 307 | "output_type": "display_data" 308 | } 309 | ], 310 | "source": [ 311 | "fig, ax = plt.subplots(ncols=2, figsize=(12, 6))\n", 312 | "ax[0].scatter(df.ra, df.dec, c = df.jd)\n", 313 | "ax[0].set_xlabel(\"RA\", fontsize=18)\n", 314 | "ax[0].set_ylabel(\"Dec\", fontsize=18)\n", 315 | "ax[1].scatter(df.xpos, df.ypos, c = df.jd)\n", 316 | "ax[1].set_xlabel(\"xpos\", fontsize=18)\n", 317 | "ax[1].set_ylabel(\"ypos\", fontsize=18)" 318 | ] 319 | }, 320 | { 321 | "cell_type": "markdown", 322 | "metadata": {}, 323 | "source": [ 324 | "The positions are all very similar in x,y!" 325 | ] 326 | }, 327 | { 328 | "cell_type": "markdown", 329 | "metadata": {}, 330 | "source": [ 331 | "# Show stamps" 332 | ] 333 | }, 334 | { 335 | "cell_type": "code", 336 | "execution_count": 7, 337 | "metadata": { 338 | "ExecuteTime": { 339 | "end_time": "2022-01-18T21:37:03.454691Z", 340 | "start_time": "2022-01-18T21:37:03.442202Z" 341 | } 342 | }, 343 | "outputs": [ 344 | { 345 | "data": { 346 | "text/html": [ 347 | "\n", 348 | "
ZTF oid: ZTF19adbtqgy, candid: 1092289313615015065
\n", 349 | "
     \n", 350 | " Science\n", 351 | "             \n", 352 | " Template\n", 353 | "             \n", 354 | " Difference\n", 355 | "
\n", 356 | "
\n", 357 | "
\n", 358 | "
\n", 359 | "
\n", 360 | " " 361 | ], 362 | "text/plain": [ 363 | "" 364 | ] 365 | }, 366 | "metadata": {}, 367 | "output_type": "display_data" 368 | }, 369 | { 370 | "data": { 371 | "text/html": [ 372 | "\n", 373 | "
ZTF oid: ZTF20aacbijc, candid: 1097444403615015104
\n", 374 | "
     \n", 375 | " Science\n", 376 | "             \n", 377 | " Template\n", 378 | "             \n", 379 | " Difference\n", 380 | "
\n", 381 | "
\n", 382 | "
\n", 383 | "
\n", 384 | "
\n", 385 | " " 386 | ], 387 | "text/plain": [ 388 | "" 389 | ] 390 | }, 391 | "metadata": {}, 392 | "output_type": "display_data" 393 | }, 394 | { 395 | "data": { 396 | "text/html": [ 397 | "\n", 398 | "
ZTF oid: ZTF20aaehjjm, candid: 1100451663615015164
\n", 399 | "
     \n", 400 | " Science\n", 401 | "             \n", 402 | " Template\n", 403 | "             \n", 404 | " Difference\n", 405 | "
\n", 406 | "
\n", 407 | "
\n", 408 | "
\n", 409 | "
\n", 410 | " " 411 | ], 412 | "text/plain": [ 413 | "" 414 | ] 415 | }, 416 | "metadata": {}, 417 | "output_type": "display_data" 418 | }, 419 | { 420 | "data": { 421 | "text/html": [ 422 | "\n", 423 | "
ZTF oid: ZTF20aaebiae, candid: 1100187173615015124
\n", 424 | "
     \n", 425 | " Science\n", 426 | "             \n", 427 | " Template\n", 428 | "             \n", 429 | " Difference\n", 430 | "
\n", 431 | "
\n", 432 | "
\n", 433 | "
\n", 434 | "
\n", 435 | " " 436 | ], 437 | "text/plain": [ 438 | "" 439 | ] 440 | }, 441 | "metadata": {}, 442 | "output_type": "display_data" 443 | }, 444 | { 445 | "data": { 446 | "text/html": [ 447 | "\n", 448 | "
ZTF oid: ZTF20aaejaax, candid: 1101179013615015179
\n", 449 | "
     \n", 450 | " Science\n", 451 | "             \n", 452 | " Template\n", 453 | "             \n", 454 | " Difference\n", 455 | "
\n", 456 | "
\n", 457 | "
\n", 458 | "
\n", 459 | "
\n", 460 | " " 461 | ], 462 | "text/plain": [ 463 | "" 464 | ] 465 | }, 466 | "metadata": {}, 467 | "output_type": "display_data" 468 | } 469 | ], 470 | "source": [ 471 | "for oid, row in df.iterrows():\n", 472 | " client.plot_stamps(oid, row.candid)" 473 | ] 474 | }, 475 | { 476 | "cell_type": "markdown", 477 | "metadata": {}, 478 | "source": [ 479 | "# Display in Aladin" 480 | ] 481 | }, 482 | { 483 | "cell_type": "code", 484 | "execution_count": 8, 485 | "metadata": { 486 | "ExecuteTime": { 487 | "end_time": "2022-01-18T21:37:07.653627Z", 488 | "start_time": "2022-01-18T21:37:07.647075Z" 489 | } 490 | }, 491 | "outputs": [ 492 | { 493 | "data": { 494 | "application/vnd.jupyter.widget-view+json": { 495 | "model_id": "990acb2ecae248bcbf75be1e3c38484f", 496 | "version_major": 2, 497 | "version_minor": 0 498 | }, 499 | "text/plain": [ 500 | "Aladin(fov=0.15, options=['allow_full_zoomout', 'coo_frame', 'fov', 'full_screen', 'log', 'overlay_survey', 'o…" 501 | ] 502 | }, 503 | "metadata": {}, 504 | "output_type": "display_data" 505 | } 506 | ], 507 | "source": [ 508 | "aladin= ipyal.Aladin(target='%s %s' % (df.iloc[0].ra, df.iloc[0].dec), fov=0.15, survey='P/PanSTARRS/DR1/color-z-zg-g')\n", 509 | "aladin" 510 | ] 511 | }, 512 | { 513 | "cell_type": "markdown", 514 | "metadata": {}, 515 | "source": [ 516 | "Add all the candidates to Aladin" 517 | ] 518 | }, 519 | { 520 | "cell_type": "code", 521 | "execution_count": 9, 522 | "metadata": { 523 | "ExecuteTime": { 524 | "end_time": "2022-01-18T21:37:10.841388Z", 525 | "start_time": "2022-01-18T21:37:10.785579Z" 526 | } 527 | }, 528 | "outputs": [], 529 | "source": [ 530 | "cand = Table.from_pandas(pd.DataFrame(\n", 531 | " data={\"MAIN_ID\": df.index.tolist(), \"ra\": df.ra.to_list(), \"dec\": df.dec.to_list()}))\n", 532 | "aladin.add_table(cand)" 533 | ] 534 | }, 535 | { 536 | "cell_type": "markdown", 537 | "metadata": {}, 538 | "source": [ 539 | "Note that you can click in individual candidates to see their object ids." 540 | ] 541 | }, 542 | { 543 | "cell_type": "code", 544 | "execution_count": null, 545 | "metadata": {}, 546 | "outputs": [], 547 | "source": [] 548 | } 549 | ], 550 | "metadata": { 551 | "kernelspec": { 552 | "display_name": "Python 3", 553 | "language": "python", 554 | "name": "python3" 555 | }, 556 | "language_info": { 557 | "codemirror_mode": { 558 | "name": "ipython", 559 | "version": 3 560 | }, 561 | "file_extension": ".py", 562 | "mimetype": "text/x-python", 563 | "name": "python", 564 | "nbconvert_exporter": "python", 565 | "pygments_lexer": "ipython3", 566 | "version": "3.8.8" 567 | }, 568 | "toc": { 569 | "base_numbering": 1, 570 | "nav_menu": {}, 571 | "number_sections": true, 572 | "sideBar": true, 573 | "skip_h1_title": false, 574 | "title_cell": "Table of Contents", 575 | "title_sidebar": "Contents", 576 | "toc_cell": false, 577 | "toc_position": {}, 578 | "toc_section_display": true, 579 | "toc_window_display": false 580 | }, 581 | "varInspector": { 582 | "cols": { 583 | "lenName": 16, 584 | "lenType": 16, 585 | "lenVar": 40 586 | }, 587 | "kernels_config": { 588 | "python": { 589 | "delete_cmd_postfix": "", 590 | "delete_cmd_prefix": "del ", 591 | "library": "var_list.py", 592 | "varRefreshCmd": "print(var_dic_list())" 593 | }, 594 | "r": { 595 | "delete_cmd_postfix": ") ", 596 | "delete_cmd_prefix": "rm(", 597 | "library": "var_list.r", 598 | "varRefreshCmd": "cat(var_dic_list()) " 599 | } 600 | }, 601 | "types_to_exclude": [ 602 | "module", 603 | "function", 604 | "builtin_function_or_method", 605 | "instance", 606 | "_Feature" 607 | ], 608 | "window_display": false 609 | } 610 | }, 611 | "nbformat": 4, 612 | "nbformat_minor": 2 613 | } 614 | -------------------------------------------------------------------------------- /notebooks/ALeRCE_ML_Light_Curve_Classifier.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# ALeRCE light curve classifier\n", 8 | "Author: Ignacio Reyes, Last updated: 10/08/2021\n", 9 | "### **Notebook under development**" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "## Table of contents:\n", 17 | "* [Required libraries](#Required-libraries)\n", 18 | "* [The classifier](#The-classifier)\n", 19 | " * [Brief explanation of the data](#Brief-explanation-of-the-data)\n", 20 | " * [Model description](#Model-description)\n", 21 | " * [Results](#Results)\n", 22 | " * [ALeRCE Explorer](#ALeRCE-Explorer)\n", 23 | "* [Query model predictions](#Query-model-predictions)\n", 24 | " * [Which objects have we found?](#Which-objects-have-we-found?)\n", 25 | "* [Spatial distribution of the predictions](#Spatial-distribution-of-the-predictions)\n", 26 | " * [Galactic sources](#Galactic-sources-(LPV,-CEP,-YSO))\n", 27 | " * [Extragalactic sources](#Extragalactic-sources-(QSO,-AGN,-Blazar,-SNe))\n", 28 | "* [Visualizing a sample of predicted objects](#Visualizing-a-sample-of-predicted-objects)" 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "## Required libraries" 36 | ] 37 | }, 38 | { 39 | "cell_type": "code", 40 | "execution_count": null, 41 | "metadata": {}, 42 | "outputs": [], 43 | "source": [ 44 | "!pip install alerce" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": {}, 51 | "outputs": [], 52 | "source": [ 53 | "from IPython.core.display import display, HTML\n", 54 | "display(HTML(\"\"))\n", 55 | "import numpy as np\n", 56 | "import pandas as pd\n", 57 | "import matplotlib.pyplot as plt\n", 58 | "from alerce.core import Alerce\n", 59 | "import psycopg2\n", 60 | "import json" 61 | ] 62 | }, 63 | { 64 | "cell_type": "markdown", 65 | "metadata": {}, 66 | "source": [ 67 | "# The classifier\n", 68 | "This notebook shows the light curve classifier from ALeRCE, described in detail in \"Alert Classification for the ALeRCE Broker System: The Light Curve Classifier\" by Sánchez-Sáez et. al (2020, under review). The goal of this notebook is to show how to access the predictions made by the model from ALeRCE's database. The code to replicate the results from the paper is contained in another GitHub repository." 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "## Brief explanation of the data\n", 76 | "ALeRCE's database contains all the alerts generated by the Zwicky Transient Facility (ZTF) public stream. An alert is generated by ZTF each time an area of the sky is observed and a difference in brightness is detected. All alerts from the same coordinates are grouped into the same object with an object id (oid). Up to this date, the database contains more than 100 million alerts and 35 million objects.\n", 77 | "\n", 78 | "Most ZTF detections come from two filters, g and r. Let's see an example of a light curve from ZTF." 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "alerce_client = Alerce()" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": {}, 94 | "outputs": [], 95 | "source": [ 96 | "def plot_light_curve(df, title='', period=None):\n", 97 | " def colormap(i):\n", 98 | " if i == 1:\n", 99 | " return '#56e03a'\n", 100 | " elif i == 2:\n", 101 | " return '#d42f4b'\n", 102 | " else:\n", 103 | " return '#000000'\n", 104 | " for fid in [1, 2]:\n", 105 | " fid_df = df[df.fid == fid]\n", 106 | " if len(fid_df) == 0:\n", 107 | " continue\n", 108 | " if period is not None:\n", 109 | " mjd = (fid_df.mjd % period) / period\n", 110 | " else:\n", 111 | " mjd = fid_df.mjd\n", 112 | " plt.errorbar(\n", 113 | " mjd,\n", 114 | " fid_df.magpsf,\n", 115 | " yerr=fid_df.sigmapsf,\n", 116 | " fmt='*',\n", 117 | " c=colormap(fid))\n", 118 | " plt.gca().invert_yaxis()\n", 119 | " if period is not None:\n", 120 | " plt.xlabel(f'Phase (period={period:.3f})')\n", 121 | " else:\n", 122 | " plt.xlabel('Date [mjd]')\n", 123 | " plt.ylabel('Apparent magnitude [mag]')\n", 124 | " plt.title(title)" 125 | ] 126 | }, 127 | { 128 | "cell_type": "code", 129 | "execution_count": null, 130 | "metadata": {}, 131 | "outputs": [], 132 | "source": [ 133 | "fig = plt.figure(figsize=(14, 10))\n", 134 | "fig.patch.set_facecolor('white')\n", 135 | "oid = 'ZTF20aaelulu'\n", 136 | "light_curve = alerce_client.query_detections(oid, format='pandas')\n", 137 | "plot_light_curve(light_curve, title=f'Light curve example {oid}')" 138 | ] 139 | }, 140 | { 141 | "cell_type": "markdown", 142 | "metadata": {}, 143 | "source": [ 144 | "The training set for the light curve classifier was built by cross matching ZTF with many astronomical catalogs with objects from known categories. The current training set of ALeRCE contains more than 1700 supernovae from many subclasses, almost 35k stochastic sources (QSO, AGN, Blazar, YSO, CV/Nova) and 87k light curves from periodic stars." 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": {}, 150 | "source": [ 151 | "## Model description\n", 152 | "When a light curve has more than 5 alerts in any of its bands it's classified by our light curve classifier. The model uses Machine Learning, combining feature extraction of the light curves and a model build of 4 Random Forest classifiers. Each object is classified in 15 astrophysical categories according to the following taxonomy.\n", 153 | "\n", 154 | "![ALeRCE_taxonomy](colab/figures/ml/taxonomy.png)" 155 | ] 156 | }, 157 | { 158 | "cell_type": "markdown", 159 | "metadata": {}, 160 | "source": [ 161 | "Our classifier takes the light curve of an object, information about non-detections (previous ZTF observations where no alerts were generated), the metadata available from ZTF (e.g. coordinates) and cross match data from WISE if available. Using that data, more than 150 features are computed.\n", 162 | "\n", 163 | "Some of the computed features are:\n", 164 | "* Irregular autoregressive model from Eyheramendy et al. (2018).\n", 165 | "* Mexican hat power spectrum from Arévalo et al. (2012).\n", 166 | "* Parameters from a damped random walk model from Graham et al. (2017).\n", 167 | "* Structure function from Schmidt et al. (2010).\n", 168 | "* Multiband periodogram from Mondrik et al. (2015)." 169 | ] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "metadata": {}, 174 | "source": [ 175 | "Once the features are computed they are classified by 4 Random Forest models, grouped in a hierarchy with two levels. The random forest on the top level classifies each object into 3 main categories: stochastic, transient and periodic sources. After that, each object is processed by 3 random forest classifiers in the bottom level, each one specialized in one category of objects.\n", 176 | "\n", 177 | "For example, the Transient Random Forest provides as output a vector with 4 probabilities, which corresponds to the probability of being a SN type Ia, SN type Ibc, SN type II or superluminous supernova.\n", 178 | "\n", 179 | "Finally, the outputs of the 4 classsifiers are combined using the total probabilities rule. That means that the probability of an object being a cepheid according to our classifier is the product of the object being a periodic star (computed by the random forest of the top level) and the probability of the object being a cepheid (computed by the periodic random forest).\n", 180 | "\n", 181 | "![Hierarchical_Random_Forest](colab/figures/ml/hierarchical_rf.png)" 182 | ] 183 | }, 184 | { 185 | "cell_type": "markdown", 186 | "metadata": {}, 187 | "source": [ 188 | "## Results\n", 189 | "\n", 190 | "The following confusion matrix is a summary of the performance of the model over the test set. This figure shows that the classifier is able to separate among the different classes. Of course some classes are difficult to distinguish, but the mistakes are \"reasonable\" in the sense that most of them choose a similar class. For example, most misclassified supernovae type Ia are labeled as another kind of supernova.\n", 191 | "\n", 192 | "The black boxes in the figure indicate the confusion inside of one of the three main categories: transient sources, periodic stars and stochastic objects.\n", 193 | "![Confusion_matrix](colab/figures/ml/sanchez_paper_confusion_matrix.png)" 194 | ] 195 | }, 196 | { 197 | "cell_type": "markdown", 198 | "metadata": {}, 199 | "source": [ 200 | "## ALeRCE Explorer\n", 201 | "All the data in ALeRCE's database, including model predictions, can be seen using the ALeRCE Explorer. The explorer is available in https://alerce.online\n", 202 | "\n", 203 | "This webpage is currently being updated, so the shown predictions can be out of date.\n", 204 | "\n", 205 | "![ALeRCE_Explorer](colab/figures/ml/alerce_explorer.png)" 206 | ] 207 | }, 208 | { 209 | "cell_type": "markdown", 210 | "metadata": {}, 211 | "source": [ 212 | "Once an object is selected, all available information about it is shown. The figure in the bottom center shows the probabilities given by the light curve classifier.\n", 213 | "\n", 214 | "![alerce_explorer_object](colab/figures/ml/alerce_explorer_object.png)" 215 | ] 216 | }, 217 | { 218 | "cell_type": "code", 219 | "execution_count": null, 220 | "metadata": {}, 221 | "outputs": [], 222 | "source": [ 223 | "!wget https://github.com/alercebroker/usecases/raw/master/alercereaduser_v4.json\n", 224 | "!mv alercereaduser_v4.json ..\n", 225 | "credentials_file = \"../alercereaduser_v4.json\"\n", 226 | "with open(credentials_file) as jsonfile:\n", 227 | " params = json.load(jsonfile)[\"params\"]\n", 228 | "conn = psycopg2.connect(\n", 229 | " dbname=params['dbname'], \n", 230 | " user=params['user'], \n", 231 | " host=params['host'], \n", 232 | " password=params['password'])" 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": null, 238 | "metadata": {}, 239 | "outputs": [], 240 | "source": [ 241 | "# Heavy query (~ 2 min)\n", 242 | "query = \"select * from probability where classifier_name = 'lc_classifier' and ranking = 1;\"\n", 243 | "bhrf_predictions = pd.read_sql_query(query, conn)\n", 244 | "bhrf_predictions" 245 | ] 246 | }, 247 | { 248 | "cell_type": "markdown", 249 | "metadata": {}, 250 | "source": [ 251 | "# Query model predictions\n", 252 | "## Which objects have we found?" 253 | ] 254 | }, 255 | { 256 | "cell_type": "code", 257 | "execution_count": null, 258 | "metadata": {}, 259 | "outputs": [], 260 | "source": [ 261 | "pred_classes_count = bhrf_predictions.groupby('class_name').count()['oid']\n", 262 | "pred_classes_count = pred_classes_count.sort_values(ascending=False)\n", 263 | "fig = plt.figure(figsize=(14, 10))\n", 264 | "fig.patch.set_facecolor('white')\n", 265 | "plt.bar(pred_classes_count.index.values, pred_classes_count.values)\n", 266 | "plt.gca().set_yscale('log')\n", 267 | "plt.title('Predicted class frequency')" 268 | ] 269 | }, 270 | { 271 | "cell_type": "markdown", 272 | "metadata": {}, 273 | "source": [ 274 | "## Spatial distribution of the predictions\n", 275 | "### Galactic sources (LPV, CEP, YSO)" 276 | ] 277 | }, 278 | { 279 | "cell_type": "code", 280 | "execution_count": null, 281 | "metadata": {}, 282 | "outputs": [], 283 | "source": [ 284 | "galactic_sources = bhrf_predictions[bhrf_predictions['class_name'].isin(\n", 285 | " ['LPV', 'CEP', 'YSO'])]\n", 286 | "galactic_sources" 287 | ] 288 | }, 289 | { 290 | "cell_type": "code", 291 | "execution_count": null, 292 | "metadata": {}, 293 | "outputs": [], 294 | "source": [ 295 | "# Heavy query (~ 8 min)\n", 296 | "query = \"select oid, name, value from feature where name = 'gal_l' or name = 'gal_b';\"\n", 297 | "features = pd.read_sql_query(query, conn)\n", 298 | "features" 299 | ] 300 | }, 301 | { 302 | "cell_type": "code", 303 | "execution_count": null, 304 | "metadata": {}, 305 | "outputs": [], 306 | "source": [ 307 | "gal_l = features[features.name == 'gal_l'].copy()\n", 308 | "gal_l.rename(columns={'value': 'gal_l'}, inplace=True)\n", 309 | "gal_l.drop(columns=['name'], inplace=True)\n", 310 | "gal_l.set_index('oid', inplace=True)\n", 311 | "\n", 312 | "gal_b = features[features.name == 'gal_b'].copy()\n", 313 | "gal_b.rename(columns={'value': 'gal_b'}, inplace=True)\n", 314 | "gal_b.drop(columns=['name'], inplace=True)\n", 315 | "gal_b.set_index('oid', inplace=True)\n", 316 | "\n", 317 | "coordinates = pd.concat([gal_l, gal_b], sort=True, axis=1)\n", 318 | "coordinates" 319 | ] 320 | }, 321 | { 322 | "cell_type": "code", 323 | "execution_count": null, 324 | "metadata": {}, 325 | "outputs": [], 326 | "source": [ 327 | "n_per_class = 1000 # sample size\n", 328 | "replace = False # with replacement\n", 329 | "fn = lambda obj: obj.loc[np.random.choice(obj.index, n_per_class, replace),:]\n", 330 | "galactic_sources_subset = galactic_sources.groupby('class_name', as_index=False).apply(fn)\n", 331 | "galactic_sources_subset = galactic_sources_subset.droplevel(level=0)\n", 332 | "galactic_sources_subset\n" 333 | ] 334 | }, 335 | { 336 | "cell_type": "code", 337 | "execution_count": null, 338 | "metadata": {}, 339 | "outputs": [], 340 | "source": [ 341 | "coordinates_subset = coordinates.loc[galactic_sources_subset.oid]\n", 342 | "fig = plt.figure(figsize=(14, 10))\n", 343 | "fig.patch.set_facecolor('white')\n", 344 | "plt.scatter(\n", 345 | " coordinates_subset.values[:, 0],\n", 346 | " coordinates_subset.values[:, 1],\n", 347 | " alpha=0.2,\n", 348 | " s=galactic_sources_subset.probability*100\n", 349 | ")\n", 350 | "plt.xlabel('Galactic latitude [deg]')\n", 351 | "plt.ylabel('Galactic longitude [deg]')\n", 352 | "plt.title('Spatial distribution of predictions: LPV, CEP, YSO')" 353 | ] 354 | }, 355 | { 356 | "cell_type": "markdown", 357 | "metadata": {}, 358 | "source": [ 359 | "### Extragalactic sources (QSO, AGN, Blazar, SNe)" 360 | ] 361 | }, 362 | { 363 | "cell_type": "code", 364 | "execution_count": null, 365 | "metadata": {}, 366 | "outputs": [], 367 | "source": [ 368 | "extragalactic_sources = bhrf_predictions[bhrf_predictions['class_name'].isin(\n", 369 | " ['QSO', 'Blazar', 'SNIa', 'SNII', 'AGN', 'SNIbc', 'SLSN'])]\n", 370 | "extragalactic_sources" 371 | ] 372 | }, 373 | { 374 | "cell_type": "code", 375 | "execution_count": null, 376 | "metadata": {}, 377 | "outputs": [], 378 | "source": [ 379 | "n_per_class = 100 # sample size\n", 380 | "replace = False # with replacement\n", 381 | "fn = lambda obj: obj.loc[np.random.choice(obj.index, n_per_class, replace),:]\n", 382 | "extragalactic_sources_subset = extragalactic_sources.groupby('class_name', as_index=False).apply(fn)\n", 383 | "extragalactic_sources_subset = extragalactic_sources_subset.droplevel(level=0)\n", 384 | "extragalactic_sources_subset" 385 | ] 386 | }, 387 | { 388 | "cell_type": "code", 389 | "execution_count": null, 390 | "metadata": {}, 391 | "outputs": [], 392 | "source": [ 393 | "coordinates_subset = coordinates.loc[extragalactic_sources_subset.oid]\n", 394 | "fig = plt.figure(figsize=(14, 10))\n", 395 | "fig.patch.set_facecolor('white')\n", 396 | "plt.scatter(\n", 397 | " coordinates_subset.values[:, 0],\n", 398 | " coordinates_subset.values[:, 1],\n", 399 | " alpha=0.4,\n", 400 | " s=extragalactic_sources_subset.probability*100\n", 401 | ")\n", 402 | "plt.xlabel('Galactic latitude [deg]')\n", 403 | "plt.ylabel('Galactic longitude [deg]')\n", 404 | "plt.title('Spatial distribution of extragalactic pred. sources')" 405 | ] 406 | }, 407 | { 408 | "cell_type": "markdown", 409 | "metadata": {}, 410 | "source": [ 411 | "## Visualizing a sample of predicted objects" 412 | ] 413 | }, 414 | { 415 | "cell_type": "code", 416 | "execution_count": null, 417 | "metadata": {}, 418 | "outputs": [], 419 | "source": [ 420 | "n_per_class = 1 # sample size\n", 421 | "minimum_probability = 0.3\n", 422 | "replace = False # with replacement\n", 423 | "fn = lambda obj: obj.loc[np.random.choice(obj.index, n_per_class, replace),:]\n", 424 | "filtered_bhrf_predictions = bhrf_predictions[bhrf_predictions.probability >= minimum_probability]\n", 425 | "sample_per_class = filtered_bhrf_predictions.groupby('class_name', as_index=False).apply(fn)\n", 426 | "sample_per_class = sample_per_class.droplevel(level=0)\n", 427 | "sample_per_class" 428 | ] 429 | }, 430 | { 431 | "cell_type": "code", 432 | "execution_count": null, 433 | "metadata": {}, 434 | "outputs": [], 435 | "source": [ 436 | "fig = plt.figure(figsize=(14, 25))\n", 437 | "fig.patch.set_facecolor('white')\n", 438 | "for index in range(len(sample_per_class)):\n", 439 | " astro_class = sample_per_class['class_name'].values[index]\n", 440 | " oid = sample_per_class['oid'].values[index]\n", 441 | " light_curve = alerce_client.query_detections(oid, format='pandas')\n", 442 | " plt.subplot(8, 2, index+1)\n", 443 | " period = None\n", 444 | " if astro_class in ['CEP', 'E', 'DSCT', 'LPV', 'Periodic-Other', 'RRL']:\n", 445 | " query = f\"select oid, name, value from feature where name = 'Multiband_period' and oid = '{oid}';\"\n", 446 | " period_df = pd.read_sql_query(query, conn)\n", 447 | " if len(period_df) > 0:\n", 448 | " period = float(period_df.value.values[0])\n", 449 | " plot_light_curve(light_curve, title=f'{astro_class} candidate {oid}', period=period)\n", 450 | "plt.tight_layout()" 451 | ] 452 | }, 453 | { 454 | "cell_type": "code", 455 | "execution_count": null, 456 | "metadata": {}, 457 | "outputs": [], 458 | "source": [] 459 | } 460 | ], 461 | "metadata": { 462 | "colab": { 463 | "name": "ALeRCE_ZTF_light_curve_classifier_newDB.ipynb", 464 | "provenance": [] 465 | }, 466 | "kernelspec": { 467 | "display_name": "Python 3 (ipykernel)", 468 | "language": "python", 469 | "name": "python3" 470 | }, 471 | "language_info": { 472 | "codemirror_mode": { 473 | "name": "ipython", 474 | "version": 3 475 | }, 476 | "file_extension": ".py", 477 | "mimetype": "text/x-python", 478 | "name": "python", 479 | "nbconvert_exporter": "python", 480 | "pygments_lexer": "ipython3", 481 | "version": "3.8.10" 482 | } 483 | }, 484 | "nbformat": 4, 485 | "nbformat_minor": 1 486 | } 487 | -------------------------------------------------------------------------------- /notebooks/ALeRCE_Other_Watchlist.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Watchlist example notebook\n", 8 | "\n", 9 | "`Author: Francisco Förster, Last update: 20210411`" 10 | ] 11 | }, 12 | { 13 | "cell_type": "markdown", 14 | "metadata": {}, 15 | "source": [ 16 | "Here we will do a xmatch against a list of positions in the sky, given in the file ```watchlist.csv```" 17 | ] 18 | }, 19 | { 20 | "cell_type": "code", 21 | "execution_count": 1, 22 | "metadata": { 23 | "ExecuteTime": { 24 | "end_time": "2021-04-11T21:19:05.153813Z", 25 | "start_time": "2021-04-11T21:19:04.813677Z" 26 | } 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "import pandas as pd\n", 31 | "import psycopg2\n", 32 | "import requests" 33 | ] 34 | }, 35 | { 36 | "cell_type": "markdown", 37 | "metadata": {}, 38 | "source": [ 39 | "# Get credentials to open a direct connection with the database" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 2, 45 | "metadata": { 46 | "ExecuteTime": { 47 | "end_time": "2021-04-11T21:19:05.271345Z", 48 | "start_time": "2021-04-11T21:19:05.155531Z" 49 | } 50 | }, 51 | "outputs": [], 52 | "source": [ 53 | "credentials_file = \"https://raw.githubusercontent.com/alercebroker/usecases/master/alercereaduser_v4.json\"\n", 54 | "params = requests.get(credentials_file).json()[\"params\"]" 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": 3, 60 | "metadata": { 61 | "ExecuteTime": { 62 | "end_time": "2021-04-11T21:19:06.380633Z", 63 | "start_time": "2021-04-11T21:19:05.279776Z" 64 | } 65 | }, 66 | "outputs": [], 67 | "source": [ 68 | "conn = psycopg2.connect(dbname=params[\"dbname\"], user=params[\"user\"], host=params[\"host\"], password=params[\"password\"])" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "# Get watchlist of object positions" 76 | ] 77 | }, 78 | { 79 | "cell_type": "code", 80 | "execution_count": 4, 81 | "metadata": { 82 | "ExecuteTime": { 83 | "end_time": "2021-04-11T21:19:08.246902Z", 84 | "start_time": "2021-04-11T21:19:06.382781Z" 85 | } 86 | }, 87 | "outputs": [ 88 | { 89 | "data": { 90 | "text/html": [ 91 | "
\n", 92 | "\n", 105 | "\n", 106 | " \n", 107 | " \n", 108 | " \n", 109 | " \n", 110 | " \n", 111 | " \n", 112 | " \n", 113 | " \n", 114 | " \n", 115 | " \n", 116 | " \n", 117 | " \n", 118 | " \n", 119 | " \n", 120 | " \n", 121 | " \n", 122 | " \n", 123 | " \n", 124 | " \n", 125 | " \n", 126 | " \n", 127 | " \n", 128 | "
id_sourceradec
0source_1160.18301433.016467
1source_2174.21524944.837895
\n", 129 | "
" 130 | ], 131 | "text/plain": [ 132 | " id_source ra dec\n", 133 | "0 source_1 160.183014 33.016467\n", 134 | "1 source_2 174.215249 44.837895" 135 | ] 136 | }, 137 | "execution_count": 4, 138 | "metadata": {}, 139 | "output_type": "execute_result" 140 | } 141 | ], 142 | "source": [ 143 | "df = pd.read_csv(\"https://github.com/alercebroker/usecases/blob/master/example_data/watchlist.csv?raw=True\")\n", 144 | "df.head()" 145 | ] 146 | }, 147 | { 148 | "cell_type": "markdown", 149 | "metadata": {}, 150 | "source": [ 151 | "# Prepare the query string for watchlist for object within 1 degress of given positions, and happening within the last 10 days" 152 | ] 153 | }, 154 | { 155 | "cell_type": "code", 156 | "execution_count": 5, 157 | "metadata": { 158 | "ExecuteTime": { 159 | "end_time": "2021-04-11T21:19:08.255447Z", 160 | "start_time": "2021-04-11T21:19:08.248711Z" 161 | } 162 | }, 163 | "outputs": [ 164 | { 165 | "data": { 166 | "text/plain": [ 167 | "\"('source_1', 160.18301441363647, 33.0164673528409),\\n('source_2', 174.21524897555543, 44.83789535222221)\"" 168 | ] 169 | }, 170 | "execution_count": 5, 171 | "metadata": {}, 172 | "output_type": "execute_result" 173 | } 174 | ], 175 | "source": [ 176 | "objects = []\n", 177 | "for _,row in df.iterrows():\n", 178 | " objects.append(f\"(\\'{row.id_source}\\', {row.ra}, {row.dec})\")\n", 179 | "objects_str = \",\\n\".join(objects)\n", 180 | "objects_str" 181 | ] 182 | }, 183 | { 184 | "cell_type": "markdown", 185 | "metadata": { 186 | "ExecuteTime": { 187 | "end_time": "2021-04-02T17:36:00.076713Z", 188 | "start_time": "2021-04-02T17:36:00.067559Z" 189 | } 190 | }, 191 | "source": [ 192 | "Create query string" 193 | ] 194 | }, 195 | { 196 | "cell_type": "code", 197 | "execution_count": 6, 198 | "metadata": { 199 | "ExecuteTime": { 200 | "end_time": "2021-04-11T21:19:08.759189Z", 201 | "start_time": "2021-04-11T21:19:08.257235Z" 202 | } 203 | }, 204 | "outputs": [], 205 | "source": [ 206 | "from astropy.time import Time\n", 207 | "nt = Time.now()\n", 208 | "last_mjd_discovery = nt.mjd - 10" 209 | ] 210 | }, 211 | { 212 | "cell_type": "code", 213 | "execution_count": 7, 214 | "metadata": { 215 | "ExecuteTime": { 216 | "end_time": "2021-04-11T21:19:08.763613Z", 217 | "start_time": "2021-04-11T21:19:08.760570Z" 218 | } 219 | }, 220 | "outputs": [], 221 | "source": [ 222 | "search_radius = 1 # degrees\n", 223 | "query = \"\"\"\n", 224 | "WITH catalog ( source_id, ra, dec) AS (\n", 225 | " VALUES\n", 226 | " {values}\n", 227 | ")\n", 228 | "SELECT \n", 229 | " c.source_id, c.ra, c.dec, o.oid, o.meanra, o.meandec, q3c_dist(c.ra,c.dec,o.meanra,o.meandec), \n", 230 | " o.firstmjd\n", 231 | "\n", 232 | "FROM object o, catalog c\n", 233 | " /*\n", 234 | " * It is REALLY important to first use the catalog then the object ra,dec for speed. The radius is in degrees.\n", 235 | " */\n", 236 | "WHERE\n", 237 | " q3c_join(c.ra, c.dec,o.meanra, o.meandec, {radius})\n", 238 | " AND o.firstmjd > %s\n", 239 | "\"\"\" % (last_mjd_discovery)\n", 240 | "query_str = query.format(values = objects_str, radius = search_radius) # radius in degrees" 241 | ] 242 | }, 243 | { 244 | "cell_type": "code", 245 | "execution_count": 8, 246 | "metadata": { 247 | "ExecuteTime": { 248 | "end_time": "2021-04-11T21:19:10.736640Z", 249 | "start_time": "2021-04-11T21:19:08.765726Z" 250 | } 251 | }, 252 | "outputs": [ 253 | { 254 | "name": "stdout", 255 | "output_type": "stream", 256 | "text": [ 257 | "(235, 8)\n" 258 | ] 259 | }, 260 | { 261 | "data": { 262 | "text/html": [ 263 | "
\n", 264 | "\n", 277 | "\n", 278 | " \n", 279 | " \n", 280 | " \n", 281 | " \n", 282 | " \n", 283 | " \n", 284 | " \n", 285 | " \n", 286 | " \n", 287 | " \n", 288 | " \n", 289 | " \n", 290 | " \n", 291 | " \n", 292 | " \n", 293 | " \n", 294 | " \n", 295 | " \n", 296 | " \n", 297 | " \n", 298 | " \n", 299 | " \n", 300 | " \n", 301 | " \n", 302 | " \n", 303 | " \n", 304 | " \n", 305 | " \n", 306 | " \n", 307 | " \n", 308 | " \n", 309 | " \n", 310 | " \n", 311 | " \n", 312 | " \n", 313 | " \n", 314 | " \n", 315 | " \n", 316 | " \n", 317 | " \n", 318 | " \n", 319 | " \n", 320 | " \n", 321 | " \n", 322 | " \n", 323 | " \n", 324 | " \n", 325 | " \n", 326 | " \n", 327 | " \n", 328 | " \n", 329 | " \n", 330 | " \n", 331 | " \n", 332 | " \n", 333 | " \n", 334 | " \n", 335 | " \n", 336 | " \n", 337 | " \n", 338 | " \n", 339 | " \n", 340 | " \n", 341 | " \n", 342 | " \n", 343 | " \n", 344 | " \n", 345 | " \n", 346 | " \n", 347 | " \n", 348 | " \n", 349 | " \n", 350 | " \n", 351 | " \n", 352 | " \n", 353 | " \n", 354 | " \n", 355 | " \n", 356 | " \n", 357 | " \n", 358 | " \n", 359 | " \n", 360 | " \n", 361 | " \n", 362 | " \n", 363 | " \n", 364 | " \n", 365 | " \n", 366 | " \n", 367 | " \n", 368 | " \n", 369 | " \n", 370 | " \n", 371 | " \n", 372 | " \n", 373 | " \n", 374 | " \n", 375 | " \n", 376 | " \n", 377 | " \n", 378 | " \n", 379 | " \n", 380 | " \n", 381 | " \n", 382 | " \n", 383 | " \n", 384 | " \n", 385 | " \n", 386 | " \n", 387 | " \n", 388 | " \n", 389 | " \n", 390 | " \n", 391 | " \n", 392 | " \n", 393 | " \n", 394 | " \n", 395 | " \n", 396 | " \n", 397 | " \n", 398 | " \n", 399 | " \n", 400 | " \n", 401 | " \n", 402 | " \n", 403 | " \n", 404 | " \n", 405 | " \n", 406 | " \n", 407 | " \n", 408 | " \n", 409 | " \n", 410 | " \n", 411 | " \n", 412 | " \n", 413 | " \n", 414 | "
source_idradecoidmeanrameandecq3c_distfirstmjd
0source_1160.18301433.016467ZTF21aarnexs160.06229633.6705910.66185259306.251771
1source_1160.18301433.016467ZTF21aarnfde159.65098732.3927430.76775559306.251771
2source_1160.18301433.016467ZTF21aarnfdl160.05376033.2048480.21727659306.251771
3source_1160.18301433.016467ZTF21aarnfdp160.96010533.0151540.65160659306.251771
4source_1160.18301433.016467ZTF21aarnfdj159.55712732.9245970.53306559306.251771
...........................
230source_2174.21524944.837895ZTF21aauhmmw174.13297644.5153770.32778259315.238819
231source_2174.21524944.837895ZTF21aauhmrg173.90348545.6978320.88748659315.238819
232source_2174.21524944.837895ZTF21aauhprj174.66790644.0819640.82207459315.239282
233source_2174.21524944.837895ZTF21aauhpri174.26073644.1973260.64139059315.239282
234source_2174.21524944.837895ZTF21aauhprk174.00599743.9247480.92531259315.239282
\n", 415 | "

235 rows × 8 columns

\n", 416 | "
" 417 | ], 418 | "text/plain": [ 419 | " source_id ra dec oid meanra meandec \\\n", 420 | "0 source_1 160.183014 33.016467 ZTF21aarnexs 160.062296 33.670591 \n", 421 | "1 source_1 160.183014 33.016467 ZTF21aarnfde 159.650987 32.392743 \n", 422 | "2 source_1 160.183014 33.016467 ZTF21aarnfdl 160.053760 33.204848 \n", 423 | "3 source_1 160.183014 33.016467 ZTF21aarnfdp 160.960105 33.015154 \n", 424 | "4 source_1 160.183014 33.016467 ZTF21aarnfdj 159.557127 32.924597 \n", 425 | ".. ... ... ... ... ... ... \n", 426 | "230 source_2 174.215249 44.837895 ZTF21aauhmmw 174.132976 44.515377 \n", 427 | "231 source_2 174.215249 44.837895 ZTF21aauhmrg 173.903485 45.697832 \n", 428 | "232 source_2 174.215249 44.837895 ZTF21aauhprj 174.667906 44.081964 \n", 429 | "233 source_2 174.215249 44.837895 ZTF21aauhpri 174.260736 44.197326 \n", 430 | "234 source_2 174.215249 44.837895 ZTF21aauhprk 174.005997 43.924748 \n", 431 | "\n", 432 | " q3c_dist firstmjd \n", 433 | "0 0.661852 59306.251771 \n", 434 | "1 0.767755 59306.251771 \n", 435 | "2 0.217276 59306.251771 \n", 436 | "3 0.651606 59306.251771 \n", 437 | "4 0.533065 59306.251771 \n", 438 | ".. ... ... \n", 439 | "230 0.327782 59315.238819 \n", 440 | "231 0.887486 59315.238819 \n", 441 | "232 0.822074 59315.239282 \n", 442 | "233 0.641390 59315.239282 \n", 443 | "234 0.925312 59315.239282 \n", 444 | "\n", 445 | "[235 rows x 8 columns]" 446 | ] 447 | }, 448 | "execution_count": 8, 449 | "metadata": {}, 450 | "output_type": "execute_result" 451 | } 452 | ], 453 | "source": [ 454 | "matches = pd.read_sql(query_str,conn)\n", 455 | "print(matches.shape)\n", 456 | "matches" 457 | ] 458 | }, 459 | { 460 | "cell_type": "markdown", 461 | "metadata": {}, 462 | "source": [ 463 | "# Query objects within 10 degress of given positions, first detected within the last 10 days, with SN probabilities > 0.4, and ranking=1 in the stamp classifier" 464 | ] 465 | }, 466 | { 467 | "cell_type": "code", 468 | "execution_count": 9, 469 | "metadata": { 470 | "ExecuteTime": { 471 | "end_time": "2021-04-11T21:19:10.741996Z", 472 | "start_time": "2021-04-11T21:19:10.738665Z" 473 | } 474 | }, 475 | "outputs": [], 476 | "source": [ 477 | "search_radius = 10 # degrees\n", 478 | "query = \"\"\"\n", 479 | "WITH catalog ( source_id, ra, dec) AS (\n", 480 | " VALUES\n", 481 | " {values}\n", 482 | "),\n", 483 | "sn (oid, classifier_name, class_name, probability, ranking) AS (\n", 484 | " SELECT\n", 485 | " o.oid, p.classifier_name, p.class_name, p.probability, p.ranking\n", 486 | " FROM\n", 487 | " probability p\n", 488 | " INNER JOIN \n", 489 | " object o\n", 490 | " ON \n", 491 | " o.oid=p.oid\n", 492 | " WHERE\n", 493 | " p.classifier_name='stamp_classifier'\n", 494 | " AND p.class_name IN ('SN')\n", 495 | " AND p.ranking=1\n", 496 | " AND p.probability > 0.4\n", 497 | " AND o.firstmjd > {last_mjd_discovery}\n", 498 | ")\n", 499 | "\n", 500 | "SELECT \n", 501 | " c.source_id, c.ra, c.dec, o.oid, o.meanra, o.meandec, q3c_dist(c.ra,c.dec,o.meanra,o.meandec), \n", 502 | " o.firstmjd, sn.classifier_name, sn.class_name, sn.probability, sn.ranking\n", 503 | "\n", 504 | "FROM object o INNER JOIN sn ON sn.oid=o.oid, catalog c\n", 505 | " /*\n", 506 | " * It is REALLY important to first use the catalog then the object ra,dec for speed. The radius is in degrees.\n", 507 | " */\n", 508 | "WHERE\n", 509 | " q3c_join(c.ra, c.dec,o.meanra, o.meandec, {radius})\n", 510 | " AND o.firstmjd > {last_mjd_discovery}\n", 511 | "ORDER BY\n", 512 | " c.source_id\n", 513 | "\"\"\"\n", 514 | "query_str = query.format(values = objects_str, radius = search_radius, last_mjd_discovery=last_mjd_discovery) # radius in degrees" 515 | ] 516 | }, 517 | { 518 | "cell_type": "code", 519 | "execution_count": 10, 520 | "metadata": { 521 | "ExecuteTime": { 522 | "end_time": "2021-04-11T21:19:18.998801Z", 523 | "start_time": "2021-04-11T21:19:10.743634Z" 524 | } 525 | }, 526 | "outputs": [ 527 | { 528 | "name": "stdout", 529 | "output_type": "stream", 530 | "text": [ 531 | "(23, 12)\n" 532 | ] 533 | }, 534 | { 535 | "data": { 536 | "text/html": [ 537 | "
\n", 538 | "\n", 551 | "\n", 552 | " \n", 553 | " \n", 554 | " \n", 555 | " \n", 556 | " \n", 557 | " \n", 558 | " \n", 559 | " \n", 560 | " \n", 561 | " \n", 562 | " \n", 563 | " \n", 564 | " \n", 565 | " \n", 566 | " \n", 567 | " \n", 568 | " \n", 569 | " \n", 570 | " \n", 571 | " \n", 572 | " \n", 573 | " \n", 574 | " \n", 575 | " \n", 576 | " \n", 577 | " \n", 578 | " \n", 579 | " \n", 580 | " \n", 581 | " \n", 582 | " \n", 583 | " \n", 584 | " \n", 585 | " \n", 586 | " \n", 587 | " \n", 588 | " \n", 589 | " \n", 590 | " \n", 591 | " \n", 592 | " \n", 593 | " \n", 594 | " \n", 595 | " \n", 596 | " \n", 597 | " \n", 598 | " \n", 599 | " \n", 600 | " \n", 601 | " \n", 602 | " \n", 603 | " \n", 604 | " \n", 605 | " \n", 606 | " \n", 607 | " \n", 608 | " \n", 609 | " \n", 610 | " \n", 611 | " \n", 612 | " \n", 613 | " \n", 614 | " \n", 615 | " \n", 616 | " \n", 617 | " \n", 618 | " \n", 619 | " \n", 620 | " \n", 621 | " \n", 622 | " \n", 623 | " \n", 624 | " \n", 625 | " \n", 626 | " \n", 627 | " \n", 628 | " \n", 629 | " \n", 630 | " \n", 631 | " \n", 632 | " \n", 633 | " \n", 634 | " \n", 635 | " \n", 636 | " \n", 637 | " \n", 638 | " \n", 639 | " \n", 640 | " \n", 641 | " \n", 642 | " \n", 643 | " \n", 644 | " \n", 645 | " \n", 646 | " \n", 647 | " \n", 648 | " \n", 649 | " \n", 650 | " \n", 651 | " \n", 652 | " \n", 653 | " \n", 654 | " \n", 655 | " \n", 656 | " \n", 657 | " \n", 658 | " \n", 659 | " \n", 660 | " \n", 661 | " \n", 662 | " \n", 663 | " \n", 664 | " \n", 665 | " \n", 666 | " \n", 667 | " \n", 668 | " \n", 669 | " \n", 670 | " \n", 671 | " \n", 672 | " \n", 673 | " \n", 674 | " \n", 675 | " \n", 676 | " \n", 677 | " \n", 678 | " \n", 679 | " \n", 680 | " \n", 681 | " \n", 682 | " \n", 683 | " \n", 684 | " \n", 685 | " \n", 686 | " \n", 687 | " \n", 688 | " \n", 689 | " \n", 690 | " \n", 691 | " \n", 692 | " \n", 693 | " \n", 694 | " \n", 695 | " \n", 696 | " \n", 697 | " \n", 698 | " \n", 699 | " \n", 700 | " \n", 701 | " \n", 702 | " \n", 703 | " \n", 704 | " \n", 705 | " \n", 706 | " \n", 707 | " \n", 708 | " \n", 709 | " \n", 710 | " \n", 711 | " \n", 712 | " \n", 713 | " \n", 714 | " \n", 715 | " \n", 716 | " \n", 717 | " \n", 718 | " \n", 719 | " \n", 720 | " \n", 721 | " \n", 722 | " \n", 723 | " \n", 724 | " \n", 725 | " \n", 726 | " \n", 727 | " \n", 728 | " \n", 729 | " \n", 730 | " \n", 731 | " \n", 732 | " \n", 733 | " \n", 734 | " \n", 735 | " \n", 736 | " \n", 737 | " \n", 738 | " \n", 739 | " \n", 740 | " \n", 741 | " \n", 742 | " \n", 743 | " \n", 744 | " \n", 745 | " \n", 746 | " \n", 747 | " \n", 748 | " \n", 749 | " \n", 750 | " \n", 751 | " \n", 752 | " \n", 753 | " \n", 754 | " \n", 755 | " \n", 756 | " \n", 757 | " \n", 758 | " \n", 759 | " \n", 760 | " \n", 761 | " \n", 762 | " \n", 763 | " \n", 764 | " \n", 765 | " \n", 766 | " \n", 767 | " \n", 768 | " \n", 769 | " \n", 770 | " \n", 771 | " \n", 772 | " \n", 773 | " \n", 774 | " \n", 775 | " \n", 776 | " \n", 777 | " \n", 778 | " \n", 779 | " \n", 780 | " \n", 781 | " \n", 782 | " \n", 783 | " \n", 784 | " \n", 785 | " \n", 786 | " \n", 787 | " \n", 788 | " \n", 789 | " \n", 790 | " \n", 791 | " \n", 792 | " \n", 793 | " \n", 794 | " \n", 795 | " \n", 796 | " \n", 797 | " \n", 798 | " \n", 799 | " \n", 800 | " \n", 801 | " \n", 802 | " \n", 803 | " \n", 804 | " \n", 805 | " \n", 806 | " \n", 807 | " \n", 808 | " \n", 809 | " \n", 810 | " \n", 811 | " \n", 812 | " \n", 813 | " \n", 814 | " \n", 815 | " \n", 816 | " \n", 817 | " \n", 818 | " \n", 819 | " \n", 820 | " \n", 821 | " \n", 822 | " \n", 823 | " \n", 824 | " \n", 825 | " \n", 826 | " \n", 827 | " \n", 828 | " \n", 829 | " \n", 830 | " \n", 831 | " \n", 832 | " \n", 833 | " \n", 834 | " \n", 835 | " \n", 836 | " \n", 837 | " \n", 838 | " \n", 839 | " \n", 840 | " \n", 841 | " \n", 842 | " \n", 843 | " \n", 844 | " \n", 845 | " \n", 846 | " \n", 847 | " \n", 848 | " \n", 849 | " \n", 850 | " \n", 851 | " \n", 852 | " \n", 853 | " \n", 854 | " \n", 855 | " \n", 856 | " \n", 857 | " \n", 858 | " \n", 859 | " \n", 860 | " \n", 861 | " \n", 862 | " \n", 863 | " \n", 864 | " \n", 865 | " \n", 866 | " \n", 867 | " \n", 868 | " \n", 869 | " \n", 870 | " \n", 871 | " \n", 872 | " \n", 873 | " \n", 874 | " \n", 875 | " \n", 876 | " \n", 877 | " \n", 878 | " \n", 879 | " \n", 880 | " \n", 881 | " \n", 882 | " \n", 883 | " \n", 884 | " \n", 885 | " \n", 886 | " \n", 887 | " \n", 888 | " \n", 889 | " \n", 890 | " \n", 891 | " \n", 892 | " \n", 893 | " \n", 894 | " \n", 895 | " \n", 896 | " \n", 897 | " \n", 898 | " \n", 899 | " \n", 900 | " \n", 901 | " \n", 902 | " \n", 903 | " \n", 904 | " \n", 905 | " \n", 906 | " \n", 907 | " \n", 908 | " \n", 909 | " \n", 910 | " \n", 911 | " \n", 912 | " \n", 913 | " \n", 914 | " \n", 915 | " \n", 916 | "
source_idradecoidmeanrameandecq3c_distfirstmjdclassifier_nameclass_nameprobabilityranking
0source_1160.18301433.016467ZTF21aasvhwh166.77065932.8000885.53392559306.372836stamp_classifierSN0.7078141
1source_1160.18301433.016467ZTF21aasuggn166.21281331.6229345.28184859308.227685stamp_classifierSN0.4301961
2source_1160.18301433.016467ZTF21aastzsn148.78683535.3796859.71118959310.249132stamp_classifierSN0.7931071
3source_1160.18301433.016467ZTF21aappwvm165.75794331.9747414.81542659308.227685stamp_classifierSN0.6615271
4source_1160.18301433.016467ZTF21aatuynj154.10868524.8178339.76673959312.234653stamp_classifierSN0.8162491
5source_1160.18301433.016467ZTF21aasuhnu160.41848626.5399216.47976359308.269213stamp_classifierSN0.7544781
6source_1160.18301433.016467ZTF21aalwdzx151.07030638.9731649.46966359314.267894stamp_classifierSN0.4125411
7source_1160.18301433.016467ZTF21aatxuyp157.57244037.9752245.39437559314.277234stamp_classifierSN0.7715511
8source_1160.18301433.016467ZTF21aardzlu155.94949642.2758949.84395059306.248542stamp_classifierSN0.6517081
9source_1160.18301433.016467ZTF21aaroydq163.57345134.2388473.07611259306.304271stamp_classifierSN0.5248961
10source_1160.18301433.016467ZTF21aardzpu158.15070638.9602626.16655759308.225984stamp_classifierSN0.4357251
11source_1160.18301433.016467ZTF21aarowtc161.39018339.8233606.87564259306.248542stamp_classifierSN0.5701811
12source_1160.18301433.016467ZTF21aarnbox154.13394039.0142537.73622559306.303229stamp_classifierSN0.4311221
13source_1160.18301433.016467ZTF21aasgnqa150.83851838.4081639.29979259308.264919stamp_classifierSN0.4715521
14source_2174.21524944.837895ZTF21aarohyu180.16705751.8409328.03699559307.262998stamp_classifierSN0.8363881
15source_2174.21524944.837895ZTF21aarwlht163.52835345.6120587.56104559307.238032stamp_classifierSN0.6772321
16source_2174.21524944.837895ZTF21aatwunv175.47711135.8816129.00741759315.197338stamp_classifierSN0.4897511
17source_2174.21524944.837895ZTF21aarxsfm174.66828541.7582953.09718659307.261493stamp_classifierSN0.8781251
18source_2174.21524944.837895ZTF21aarovmb170.25485553.1746508.72868159306.298993stamp_classifierSN0.4616721
19source_2174.21524944.837895ZTF21aardszu175.15426736.6699558.19868759307.233808stamp_classifierSN0.6973911
20source_2174.21524944.837895ZTF21aasyydo174.84518742.1185912.75742959311.175648stamp_classifierSN0.7173401
21source_2174.21524944.837895ZTF21aaszaoy176.27683153.6160358.87977459311.179491stamp_classifierSN0.4841961
22source_2174.21524944.837895ZTF21aarxuih182.87732241.8875356.94949459307.235220stamp_classifierSN0.5725131
\n", 917 | "
" 918 | ], 919 | "text/plain": [ 920 | " source_id ra dec oid meanra meandec \\\n", 921 | "0 source_1 160.183014 33.016467 ZTF21aasvhwh 166.770659 32.800088 \n", 922 | "1 source_1 160.183014 33.016467 ZTF21aasuggn 166.212813 31.622934 \n", 923 | "2 source_1 160.183014 33.016467 ZTF21aastzsn 148.786835 35.379685 \n", 924 | "3 source_1 160.183014 33.016467 ZTF21aappwvm 165.757943 31.974741 \n", 925 | "4 source_1 160.183014 33.016467 ZTF21aatuynj 154.108685 24.817833 \n", 926 | "5 source_1 160.183014 33.016467 ZTF21aasuhnu 160.418486 26.539921 \n", 927 | "6 source_1 160.183014 33.016467 ZTF21aalwdzx 151.070306 38.973164 \n", 928 | "7 source_1 160.183014 33.016467 ZTF21aatxuyp 157.572440 37.975224 \n", 929 | "8 source_1 160.183014 33.016467 ZTF21aardzlu 155.949496 42.275894 \n", 930 | "9 source_1 160.183014 33.016467 ZTF21aaroydq 163.573451 34.238847 \n", 931 | "10 source_1 160.183014 33.016467 ZTF21aardzpu 158.150706 38.960262 \n", 932 | "11 source_1 160.183014 33.016467 ZTF21aarowtc 161.390183 39.823360 \n", 933 | "12 source_1 160.183014 33.016467 ZTF21aarnbox 154.133940 39.014253 \n", 934 | "13 source_1 160.183014 33.016467 ZTF21aasgnqa 150.838518 38.408163 \n", 935 | "14 source_2 174.215249 44.837895 ZTF21aarohyu 180.167057 51.840932 \n", 936 | "15 source_2 174.215249 44.837895 ZTF21aarwlht 163.528353 45.612058 \n", 937 | "16 source_2 174.215249 44.837895 ZTF21aatwunv 175.477111 35.881612 \n", 938 | "17 source_2 174.215249 44.837895 ZTF21aarxsfm 174.668285 41.758295 \n", 939 | "18 source_2 174.215249 44.837895 ZTF21aarovmb 170.254855 53.174650 \n", 940 | "19 source_2 174.215249 44.837895 ZTF21aardszu 175.154267 36.669955 \n", 941 | "20 source_2 174.215249 44.837895 ZTF21aasyydo 174.845187 42.118591 \n", 942 | "21 source_2 174.215249 44.837895 ZTF21aaszaoy 176.276831 53.616035 \n", 943 | "22 source_2 174.215249 44.837895 ZTF21aarxuih 182.877322 41.887535 \n", 944 | "\n", 945 | " q3c_dist firstmjd classifier_name class_name probability ranking \n", 946 | "0 5.533925 59306.372836 stamp_classifier SN 0.707814 1 \n", 947 | "1 5.281848 59308.227685 stamp_classifier SN 0.430196 1 \n", 948 | "2 9.711189 59310.249132 stamp_classifier SN 0.793107 1 \n", 949 | "3 4.815426 59308.227685 stamp_classifier SN 0.661527 1 \n", 950 | "4 9.766739 59312.234653 stamp_classifier SN 0.816249 1 \n", 951 | "5 6.479763 59308.269213 stamp_classifier SN 0.754478 1 \n", 952 | "6 9.469663 59314.267894 stamp_classifier SN 0.412541 1 \n", 953 | "7 5.394375 59314.277234 stamp_classifier SN 0.771551 1 \n", 954 | "8 9.843950 59306.248542 stamp_classifier SN 0.651708 1 \n", 955 | "9 3.076112 59306.304271 stamp_classifier SN 0.524896 1 \n", 956 | "10 6.166557 59308.225984 stamp_classifier SN 0.435725 1 \n", 957 | "11 6.875642 59306.248542 stamp_classifier SN 0.570181 1 \n", 958 | "12 7.736225 59306.303229 stamp_classifier SN 0.431122 1 \n", 959 | "13 9.299792 59308.264919 stamp_classifier SN 0.471552 1 \n", 960 | "14 8.036995 59307.262998 stamp_classifier SN 0.836388 1 \n", 961 | "15 7.561045 59307.238032 stamp_classifier SN 0.677232 1 \n", 962 | "16 9.007417 59315.197338 stamp_classifier SN 0.489751 1 \n", 963 | "17 3.097186 59307.261493 stamp_classifier SN 0.878125 1 \n", 964 | "18 8.728681 59306.298993 stamp_classifier SN 0.461672 1 \n", 965 | "19 8.198687 59307.233808 stamp_classifier SN 0.697391 1 \n", 966 | "20 2.757429 59311.175648 stamp_classifier SN 0.717340 1 \n", 967 | "21 8.879774 59311.179491 stamp_classifier SN 0.484196 1 \n", 968 | "22 6.949494 59307.235220 stamp_classifier SN 0.572513 1 " 969 | ] 970 | }, 971 | "execution_count": 10, 972 | "metadata": {}, 973 | "output_type": "execute_result" 974 | } 975 | ], 976 | "source": [ 977 | "matches = pd.read_sql(query_str,conn)\n", 978 | "print(matches.shape)\n", 979 | "matches.head(100)" 980 | ] 981 | }, 982 | { 983 | "cell_type": "markdown", 984 | "metadata": {}, 985 | "source": [ 986 | "Close the connection" 987 | ] 988 | }, 989 | { 990 | "cell_type": "code", 991 | "execution_count": 11, 992 | "metadata": { 993 | "ExecuteTime": { 994 | "end_time": "2021-04-11T21:19:19.003542Z", 995 | "start_time": "2021-04-11T21:19:19.000766Z" 996 | } 997 | }, 998 | "outputs": [], 999 | "source": [ 1000 | "conn.close()" 1001 | ] 1002 | }, 1003 | { 1004 | "cell_type": "markdown", 1005 | "metadata": {}, 1006 | "source": [ 1007 | "# Open the resulting objects in the ALeRCE explorer" 1008 | ] 1009 | }, 1010 | { 1011 | "cell_type": "code", 1012 | "execution_count": 12, 1013 | "metadata": { 1014 | "ExecuteTime": { 1015 | "end_time": "2021-04-11T21:19:19.016548Z", 1016 | "start_time": "2021-04-11T21:19:19.005683Z" 1017 | } 1018 | }, 1019 | "outputs": [ 1020 | { 1021 | "name": "stdout", 1022 | "output_type": "stream", 1023 | "text": [ 1024 | "https://alerce.online/?oid=ZTF21aasvhwh&oid=ZTF21aasuggn&oid=ZTF21aastzsn&oid=ZTF21aappwvm&oid=ZTF21aatuynj&oid=ZTF21aasuhnu&oid=ZTF21aalwdzx&oid=ZTF21aatxuyp&oid=ZTF21aardzlu&oid=ZTF21aaroydq&oid=ZTF21aardzpu&oid=ZTF21aarowtc&oid=ZTF21aarnbox&oid=ZTF21aasgnqa&oid=ZTF21aarohyu&oid=ZTF21aarwlht&oid=ZTF21aatwunv&oid=ZTF21aarxsfm&oid=ZTF21aarovmb&oid=ZTF21aardszu&oid=ZTF21aasyydo&oid=ZTF21aaszaoy&oid=ZTF21aarxuih&count=true&page=1&perPage=1000&sortDesc=false\n" 1025 | ] 1026 | } 1027 | ], 1028 | "source": [ 1029 | "url = \"https://alerce.online/?\" + \"&\".join([\"oid=%s\" % oid for oid in matches.oid.values]) + \"&count=true&page=1&perPage=1000&sortDesc=false\"\n", 1030 | "print(url)" 1031 | ] 1032 | }, 1033 | { 1034 | "cell_type": "code", 1035 | "execution_count": null, 1036 | "metadata": {}, 1037 | "outputs": [], 1038 | "source": [] 1039 | } 1040 | ], 1041 | "metadata": { 1042 | "kernelspec": { 1043 | "display_name": "Python 3", 1044 | "language": "python", 1045 | "name": "python3" 1046 | }, 1047 | "language_info": { 1048 | "codemirror_mode": { 1049 | "name": "ipython", 1050 | "version": 3 1051 | }, 1052 | "file_extension": ".py", 1053 | "mimetype": "text/x-python", 1054 | "name": "python", 1055 | "nbconvert_exporter": "python", 1056 | "pygments_lexer": "ipython3", 1057 | "version": "3.6.5" 1058 | }, 1059 | "toc": { 1060 | "base_numbering": 1, 1061 | "nav_menu": {}, 1062 | "number_sections": true, 1063 | "sideBar": true, 1064 | "skip_h1_title": false, 1065 | "title_cell": "Table of Contents", 1066 | "title_sidebar": "Contents", 1067 | "toc_cell": false, 1068 | "toc_position": {}, 1069 | "toc_section_display": true, 1070 | "toc_window_display": true 1071 | } 1072 | }, 1073 | "nbformat": 4, 1074 | "nbformat_minor": 4 1075 | } 1076 | -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/ALeRCE_Hands_On_Tutorial_newDB_collab.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "colab_type": "text", 7 | "id": "aARiWZB804Yh" 8 | }, 9 | "source": [ 10 | "# ALeRCE Hands On Tutorial\n", 11 | "\n", 12 | "In this tutorial we will put in practice what we have learned in this workshop. The idea is that you try to use the tools we provide and adapt them to your specific science workflow.\n", 13 | "\n", 14 | "As a way to test your knowledge using ALeRCE's services we ask you to complete **3 challenges**.\n", 15 | "\n", 16 | "\n", 17 | "1. Plot lightcurve using *alerce* client\n", 18 | "2. Plot any combination of attributes from any table\n", 19 | "3. Download a dataframe as CSV and open it again with pandas" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": { 25 | "colab_type": "text", 26 | "id": "o48zzNVY0zo8" 27 | }, 28 | "source": [ 29 | "We are going to be using the alerce client so first we need to install it" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": { 36 | "colab": null, 37 | "colab_type": "code", 38 | "id": "cDYh15Vin1vZ" 39 | }, 40 | "outputs": [], 41 | "source": [ 42 | "!pip install git+https://git@github.com/alercebroker/alerce_client_new#alerce" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": { 48 | "colab_type": "text", 49 | "id": "KQd1JdMUtrZy" 50 | }, 51 | "source": [ 52 | "# First Challenge: Plot a light curve using the ALeRCE client" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": { 58 | "colab_type": "text", 59 | "id": "YGWC0LfSt8HX" 60 | }, 61 | "source": [ 62 | "# Second Challenge: Query a list of objects and plot some attributes" 63 | ] 64 | }, 65 | { 66 | "cell_type": "markdown", 67 | "metadata": { 68 | "colab_type": "text", 69 | "id": "T_H86iLMzpKp" 70 | }, 71 | "source": [ 72 | "# Third Challenge: Save and download data using pandas" 73 | ] 74 | }, 75 | { 76 | "cell_type": "markdown", 77 | "metadata": { 78 | "colab_type": "text", 79 | "id": "JTCG_nynIZSz" 80 | }, 81 | "source": [ 82 | "**Congratulations, you completed the challenge!**\n", 83 | "\n", 84 | "---\n", 85 | "\n" 86 | ] 87 | } 88 | ], 89 | "metadata": { 90 | "colab": { 91 | "collapsed_sections": null, 92 | "name": "ALeRCE_Hands_On_Tutorial_newDB_collab.ipynb", 93 | "provenance": null 94 | }, 95 | "kernelspec": { 96 | "display_name": "Python 3", 97 | "language": "python", 98 | "name": "python3" 99 | }, 100 | "language_info": { 101 | "codemirror_mode": { 102 | "name": "ipython", 103 | "version": 3 104 | }, 105 | "file_extension": ".py", 106 | "mimetype": "text/x-python", 107 | "name": "python", 108 | "nbconvert_exporter": "python", 109 | "pygments_lexer": "ipython3", 110 | "version": "3.8.5" 111 | }, 112 | "name": "ALeRCE_Hands_On_Tutorial_newDB_collab.ipynb" 113 | }, 114 | "nbformat": 4, 115 | "nbformat_minor": 1 116 | } 117 | -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/README.md: -------------------------------------------------------------------------------- 1 | # ALeRCE Tutorial Workshop - 23,24 September 2020 2 | 3 | This was a two-day workshop with the aim of learning how to use the tools by the ALeRCE broker, as well as to learn to code a notebook focused on your own science case. 4 | 5 | Details on the workshop, the schedule and access to the slides can be found at: 6 | 7 | http://workshops.alerce.online/alerce-broker-tutorial/ 8 | 9 | The talks were recorded and are available in Youtube: 10 | 11 | [https://www.youtube.com/playlist?list=PLFA428AMRhwAfmfgEc4MP8m4PiNeqbxZv](https://www.youtube.com/playlist?list=PLFA428AMRhwAfmfgEc4MP8m4PiNeqbxZv) 12 | 13 | 14 | Note that these notebooks work in Google Colab and the pipeline that was used was alercereaduser_v3.json 15 | -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/data/ephs_10586_20200101-20200102.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/data/ephs_10586_20200101-20200102.pkl -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/dists_contours.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/dists_contours.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/alerce_explorer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/alerce_explorer.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/alerce_explorer_object.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/alerce_explorer_object.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/alerce_frontend.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/alerce_frontend.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/betas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/betas.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/class_samples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/class_samples.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/cm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/cm.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/cnn_architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/cnn_architecture.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/cnn_table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/cnn_table.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/hierarchical_rf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/hierarchical_rf.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/sanchez_paper_confusion_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/sanchez_paper_confusion_matrix.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/sn_hunter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/sn_hunter.png -------------------------------------------------------------------------------- /notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/taxonomy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/ALeRCE_Tutorial_workshop_Sep2020/figures/ml/taxonomy.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/SanchezSaez+2020_confusion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/SanchezSaez+2020_confusion.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/SanchezSaez+2020_recall.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/SanchezSaez+2020_recall.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/Villar+19.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/Villar+19.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/alerce_explorer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/alerce_explorer.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/alerce_explorer_object.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/alerce_explorer_object.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/alerce_frontend.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/alerce_frontend.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/betas.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/betas.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/class_samples.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/class_samples.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/cm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/cm.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/cnn_architecture.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/cnn_architecture.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/cnn_table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/cnn_table.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/hierarchical_rf.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/hierarchical_rf.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/sanchez_paper_confusion_matrix.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/sanchez_paper_confusion_matrix.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/sn_hunter.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/sn_hunter.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/ml/taxonomy.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/ml/taxonomy.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/recall_agn_qso_blazar_for_mag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/recall_agn_qso_blazar_for_mag.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/recall_agn_qso_blazar_for_ndet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/recall_agn_qso_blazar_for_ndet.png -------------------------------------------------------------------------------- /notebooks/EVIC_Dec2022_tutorial/figures/taxonomy_tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/EVIC_Dec2022_tutorial/figures/taxonomy_tree.png -------------------------------------------------------------------------------- /notebooks/IAUS375_Dec2022_Blazars_tutorial/readme.md: -------------------------------------------------------------------------------- 1 | # IAUS 375: : The Multimessenger Chakra of Blazar Jets - ALeRCE Tutorial 2 | 3 | This tutorial will be focused on science that can be done with ALeRCE to study Blazars. After an introduction to ALeRCE and its general services, 4 | we will use the following Jupyter Notebook: 5 | 6 | [ALeRCE Blazar Starter notebook](https://colab.research.google.com/github/alercebroker/usecases/blob/master/notebooks/IAUS375_Dec2022_Blazars_tutorial/ALeRCE_Blazar_starter.ipynb) 7 | 8 | The notebook is focused on Blazars mostly using the results from the light curve classifier. After a brief introduction, we will use the ALeRCE client to retrieve the light curve for one particular Blazar, and then for a sample of sources. Then, we will learn how to use the ALeRCE database to retrieve large samples of Blazars and AGN, and to compare their properties. 9 | -------------------------------------------------------------------------------- /notebooks/IAU_GA_2024/README.md: -------------------------------------------------------------------------------- 1 | # IAU GA 2024 ALeRCE tutorial 2 | 3 | Welcome to the ALeRCE tutorial github page. See the accompanying slides [here](https://docs.google.com/presentation/d/118kXeGJlyLVHVh_vFEVGCD15EYpN_C30GnWJHF_L0ms/edit?usp=sharing) and setup instructions [here](https://docs.google.com/presentation/d/1kXo-adiF8wSznCyAAa9_Dtv1jZs8ZZZ4fSs98SkClMI/edit?usp=sharing). 4 | 5 | 6 | -------------------------------------------------------------------------------- /notebooks/IAU_GA_2024/alercereaduser_v4.json: -------------------------------------------------------------------------------- 1 | { 2 | "params" : { 3 | "dbname" : "ztf", 4 | "user" : "alerceread", 5 | "host": "54.205.99.47", 6 | "password" : "w*C*u8AXZ4e%d+zv" 7 | } 8 | } 9 | -------------------------------------------------------------------------------- /notebooks/LSSTC_Broker_workshop_Apr2021/README.md: -------------------------------------------------------------------------------- 1 | # ALeRCE notebooks for the LSSTC Broker Workshop 2021 - 13,14,15 April 2021 2 | 3 | The notebooks were presented in the LSSTC Broker Workshop 2021 (Part II) as part of the ALeRCE Broker presentations. During this workshop, ~70% tutorials and demos were focused on science cases and interfaces (filtering, machine-learning classifications and cross-matches, multi-messenger/wavelength analyses, APIs, batch processing). A very small number of talks on topics like LSST DR0 & cadence, ZTF II, PLAsTiCC 2, TOMs. 4 | 5 | All the information about the workshop, including the schedule are available at: 6 | 7 | * [LSSTC Enabling Science 2021 Broker Workshop Part II](http://workshops.alerce.online/lsstc-enabling-science-2020-broker-workshop-part-ii/) 8 | 9 | Available material for all the brokers involved in this workshop can be found at: 10 | 11 | * [Broker Workshop 2021 Tutorials](https://github.com/broker-workshop/tutorials) 12 | 13 | The full list of recorded talks can be found in the following link: 14 | 15 | * [Link to Youtube channel](https://www.youtube.com/playlist?list=PLFA428AMRhwADMFsNDC1dttTyIukPhADi) 16 | 17 | The ALeRCE notebooks work in Google Colab and the pipeline that was used is alercereaduser_v4.json 18 | -------------------------------------------------------------------------------- /notebooks/README.md: -------------------------------------------------------------------------------- 1 | # ALeRCE notebooks 2 | 3 | These notebooks are tools to facilitate the use of the ALeRCE API and DB. These are based on different scientific cases, including Variable Stars (VS), Active Galactic Nuclei (AGN), Supernova (SN), Asteroids, as well as related tools (Other). There are also notebooks based on the ALeRCE stamp and light curve classifiers (ML). 4 | -------------------------------------------------------------------------------- /notebooks/figures/SanchezSaez+2020_confusion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/figures/SanchezSaez+2020_confusion.png -------------------------------------------------------------------------------- /notebooks/figures/SanchezSaez+2020_recall.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/figures/SanchezSaez+2020_recall.png -------------------------------------------------------------------------------- /notebooks/figures/Villar+19.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/figures/Villar+19.png -------------------------------------------------------------------------------- /notebooks/figures/recall_agn_qso_blazar_for_mag.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/figures/recall_agn_qso_blazar_for_mag.png -------------------------------------------------------------------------------- /notebooks/figures/recall_agn_qso_blazar_for_ndet.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/figures/recall_agn_qso_blazar_for_ndet.png -------------------------------------------------------------------------------- /notebooks/figures/taxonomy_tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/figures/taxonomy_tree.png -------------------------------------------------------------------------------- /notebooks/variablestars_figures/RRL_GAIA_VS_Notebook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/variablestars_figures/RRL_GAIA_VS_Notebook.png -------------------------------------------------------------------------------- /notebooks/variablestars_figures/SanchezSaez+2020_confusion.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/variablestars_figures/SanchezSaez+2020_confusion.png -------------------------------------------------------------------------------- /notebooks/variablestars_figures/SanchezSaez+2020_rrl.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/variablestars_figures/SanchezSaez+2020_rrl.png -------------------------------------------------------------------------------- /notebooks/variablestars_figures/taxonomy_tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/alercebroker/usecases/991df543749ec02420ec693a4039fd32f05a2d5f/notebooks/variablestars_figures/taxonomy_tree.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | alerce_xmatch 2 | websocket-client 3 | --------------------------------------------------------------------------------