├── 1.Data_Processing.ipynb ├── 2.Batch_Data_Processing.ipynb ├── 3.Model_building_training_and_deployment.ipynb ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── aws-ml-healthcare-worshop.yaml ├── data ├── .DS_Store ├── .ipynb_checkpoints │ └── sample_report_1-checkpoint.pdf ├── mtsamples.csv ├── processed_combined_wide.csv └── sample_report_1.pdf └── util ├── .ipynb_checkpoints └── preprocess-checkpoint.py ├── Pipeline.py ├── __pycache__ └── preprocess.cpython-36.pyc ├── classification_report.py └── preprocess.py /1.Data_Processing.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 1. Data Processing - Build a data processing pipeline to process electronic medical reports (EMR) using Amazon Textract and Comprehend Medical\n", 8 | "\n", 9 | "In this notebook, we will walkthrough on how to build a data processing pipeline that will process electronic medical reports (EMR) in PDF format to extract relevant medical information by using the following AWS services:\n", 10 | "\n", 11 | "- [Textract](https://aws.amazon.com/textract/): To extract text from the PDF medical report\n", 12 | "- [Comprehend Medical](https://aws.amazon.com/comprehend/medical/): To extract relevant medical information from the output of textract\n", 13 | "\n", 14 | "## Contents\n", 15 | "\n", 16 | "1. [Objective](#Objective)\n", 17 | "1. [Setup Environment](#Setup-Environment)\n", 18 | "1. [Extract text using Amazon Textract](#Step-1:-Process-PDF-with-Amazon-Textract)\n", 19 | "1. [Extract medical information using Amazon Comprehend Medical](#Step-2:-Extract-medical-information-with-Amazon-Comprehend-Medical)\n", 20 | "1. [Clean up resources](#Clean-up-resources)" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "---\n", 28 | "\n", 29 | "# Objective\n", 30 | "\n", 31 | "The objective of this section of the workshop is to learn how to use Amazon Textract and Comprehend Medical to extract the medical information from an electronic medical report in PDF format." 32 | ] 33 | }, 34 | { 35 | "cell_type": "markdown", 36 | "metadata": {}, 37 | "source": [ 38 | "---\n", 39 | "\n", 40 | "# Setup environment\n", 41 | "\n", 42 | "Before be begin, let us setup our environment. We will need the following:\n", 43 | "\n", 44 | "* Amazon Textract Results Parser `textract-trp` to process our Textract results.\n", 45 | "* Python libraries \n", 46 | "* Pre-processing functions that will help with processing and visualization of our results. For the purpose of this workshop, we have provided a pre-processing function library that can be found in [util/preprocess.py](./util/preprocess.py)\n", 47 | "\n", 48 | "Note: `textract-trp` will require Python 3.6 or newer." 49 | ] 50 | }, 51 | { 52 | "cell_type": "code", 53 | "execution_count": null, 54 | "metadata": { 55 | "scrolled": true 56 | }, 57 | "outputs": [], 58 | "source": [ 59 | "!pip install textract-trp" 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "import boto3\n", 69 | "import time\n", 70 | "import sagemaker\n", 71 | "import os \n", 72 | "import trp\n", 73 | "from util.preprocess import *\n", 74 | "import pandas as pd\n", 75 | "bucket = sagemaker.Session().default_bucket()\n", 76 | "prefix = 'sagemaker/medical_notes'" 77 | ] 78 | }, 79 | { 80 | "cell_type": "markdown", 81 | "metadata": {}, 82 | "source": [ 83 | "---\n", 84 | "\n", 85 | "# Step 1: Process PDF with Amazon Textract\n", 86 | "\n", 87 | "In this section we will be extracting the text from a medical report in PDF format using Textract. To facilitate this workshop, we have generated a [sample PDF medical report](./data/sample_report_1.pdf) using the [MTSample dataset](https://www.kaggle.com/tboyle10/medicaltranscriptions) from kaggle." 88 | ] 89 | }, 90 | { 91 | "cell_type": "markdown", 92 | "metadata": {}, 93 | "source": [ 94 | "## About Textract\n", 95 | "Amazon Textract can detect lines of text and the words that make up a line of text. Textract can handle documents in either synchronous or asynchronous processing:\n", 96 | "+ [synchronous API](https://docs.aws.amazon.com/textract/latest/dg/sync.html): supports *The input document must be an image in `JPEG` or `PNG` format*. Single page document analysis can be performed using a Textract synchronous operation.\n", 97 | " 1. *`detect_document_text`*: detects text in the input document. \n", 98 | " 2. *`analyze_document`*: analyzes an input document for relationships between detected items.\n", 99 | "+ [asynchronous API](https://docs.aws.amazon.com/textract/latest/dg/async.html): *can analyze text in documents that are in `JPEG`, `PNG`, and `PDF` format. Multi page processing is an asynchronous operation. The documents are stored in an Amazon S3 bucket. Use DocumentLocation to specify the bucket name and file name of the document.*\n", 100 | " 1. for context analysis:\n", 101 | " 1. *`start_document_text_detection`*: starts the asynchronous detection of text in a document. \n", 102 | " 2. *`get_document_text_detection`*: gets the results for an Amazon Textract asynchronous operation that detects text in a document.\n", 103 | " 2. for relationships between detected items :\n", 104 | " 1. *`start_document_analysis`*: starts the asynchronous analysis of relationship in a document. \n", 105 | " 2. *`get_document_analysis`*: Gets the results for an Amazon Textract asynchronous operation that analyzes text in a document\n", 106 | " \n", 107 | "For detailed api, refer to documentation [here](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/textract.html#Textract.Client.analyze_document).\n", 108 | "\n", 109 | "In this demo, as the input is in pdf format and has multiple pages, we will be using the multi page textract operation, we will need to upload our sample medical record to an S3 bucket. Run the next cell to upload our sample medical report." 110 | ] 111 | }, 112 | { 113 | "cell_type": "code", 114 | "execution_count": null, 115 | "metadata": {}, 116 | "outputs": [], 117 | "source": [ 118 | "fileName = 'sample_report_1.pdf'\n", 119 | "fileUploadPath = os.path.join('./data', fileName)\n", 120 | "textractObjectName = os.path.join(prefix, 'data', fileName)\n", 121 | "\n", 122 | "# Upload medical report file\n", 123 | "boto3.Session().resource('s3').Bucket(bucket).Object(textractObjectName).upload_file(fileUploadPath)" 124 | ] 125 | }, 126 | { 127 | "cell_type": "markdown", 128 | "metadata": {}, 129 | "source": [ 130 | "## Start text detection asynchonously in the pdf\n", 131 | "In the next step, we will start the asynchronous textract operation by calling the `start_document_analysis()` function. The function will kickoff an asynchronous job that will process our medical report file in the stipulated S3 bucket." 132 | ] 133 | }, 134 | { 135 | "cell_type": "code", 136 | "execution_count": null, 137 | "metadata": {}, 138 | "outputs": [], 139 | "source": [ 140 | "textract = boto3.client('textract')\n", 141 | "response = textract.start_document_analysis(\n", 142 | " DocumentLocation={\n", 143 | " 'S3Object': {\n", 144 | " 'Bucket': bucket,\n", 145 | " 'Name': textractObjectName\n", 146 | " }},\n", 147 | " FeatureTypes=[\n", 148 | " 'TABLES',\n", 149 | " ]\n", 150 | " )\n", 151 | "\n", 152 | "textractJobId = response[\"JobId\"]\n", 153 | "print('job id is: ',textractJobId)" 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "## Monitor the job status\n", 161 | "\n", 162 | "As the job is kicked off in the background, we can monitor the progress of the job by calling the `get_document_analysis()` function and passing the job id of the job that we created. \n", 163 | "\n", 164 | "Run the next cell and wait for the Textract Job status to return a SUCCEEDED status.\n", 165 | "the outcome is in json format" 166 | ] 167 | }, 168 | { 169 | "cell_type": "code", 170 | "execution_count": null, 171 | "metadata": {}, 172 | "outputs": [], 173 | "source": [ 174 | "%%time\n", 175 | "time.sleep(5)\n", 176 | "response = textract.get_document_analysis(JobId=textractJobId)\n", 177 | "status = response[\"JobStatus\"]\n", 178 | "\n", 179 | "while(status == \"IN_PROGRESS\"):\n", 180 | " time.sleep(5)\n", 181 | " response = textract.get_document_analysis(JobId=textractJobId)\n", 182 | " status = response[\"JobStatus\"]\n", 183 | " print(\"Textract Job status: {}\".format(status))" 184 | ] 185 | }, 186 | { 187 | "cell_type": "markdown", 188 | "metadata": {}, 189 | "source": [ 190 | "## Extract textract results\n", 191 | "Now that we've successfully extracted the text from the medical report, let us extract the textract results and consolidate the text so that we can pass it to Comprehend Medical to start extract medical information from the report." 192 | ] 193 | }, 194 | { 195 | "cell_type": "code", 196 | "execution_count": null, 197 | "metadata": {}, 198 | "outputs": [], 199 | "source": [ 200 | "%%time\n", 201 | "pages = []\n", 202 | "\n", 203 | "time.sleep(5)\n", 204 | "\n", 205 | "response = textract.get_document_analysis(JobId=textractJobId)\n", 206 | "\n", 207 | "pages.append(response)\n", 208 | "\n", 209 | "nextToken = None\n", 210 | "if('NextToken' in response):\n", 211 | " nextToken = response['NextToken']\n", 212 | "\n", 213 | "while(nextToken):\n", 214 | " time.sleep(5)\n", 215 | "\n", 216 | " response = textract.get_document_analysis(JobId=textractJobId, NextToken=nextToken)\n", 217 | "\n", 218 | " pages.append(response)\n", 219 | " print(\"Resultset page recieved: {}\".format(len(pages)))\n", 220 | " nextToken = None\n", 221 | " if('NextToken' in response):\n", 222 | " nextToken = response['NextToken']" 223 | ] 224 | }, 225 | { 226 | "cell_type": "markdown", 227 | "metadata": {}, 228 | "source": [ 229 | "### Output from Textract\n", 230 | "\n", 231 | "Let's take a look at the output from textract by using the trp library to extract and format the textract results." 232 | ] 233 | }, 234 | { 235 | "cell_type": "code", 236 | "execution_count": null, 237 | "metadata": {}, 238 | "outputs": [], 239 | "source": [ 240 | "doc = trp.Document(pages)\n", 241 | "print(\"Total length of document is\",len(doc.pages))\n", 242 | "idx=1\n", 243 | "for page in doc.pages:\n", 244 | " print(color.BOLD + f\"Results from page {idx}: \\n\" + color.END, page.text)\n", 245 | " idx=idx+1\n" 246 | ] 247 | }, 248 | { 249 | "cell_type": "markdown", 250 | "metadata": {}, 251 | "source": [ 252 | "---\n", 253 | "\n", 254 | "# Step 2: Extract medical information with Amazon Comprehend Medical\n", 255 | "\n", 256 | "## About Amazon Comprehend Medical\n", 257 | "\n", 258 | "Comprehend Medical detects useful information in unstructured clinical text. As much as 75% of all health record data is found in unstructured text such as physician's notes, discharge summaries, test results, and case notes. Amazon Comprehend Medical uses Natural Language Processing (NLP) models to sort through text for valuable information. \n", 259 | "\n", 260 | "Using Amazon Comprehend Medical, you can quickly and accurately gather information, such as medical condition, medication, dosage, strength, and frequency from a variety of sources like doctors’ notes. Amazon Comprehend Medical uses advanced machine learning models to accurately and quickly identify medical information, such as medical conditions and medications, and determines their relationship to each other, for instance, medicine dosage and strength. Amazon Comprehend Medical can also link the detected information to medical ontologies such as ICD-10-CM or RxNorm\n", 261 | "\n", 262 | "Currently, Amazon Comprehend Medical only detects medical entities in English language texts.\n", 263 | "\n", 264 | "![Image of Comprehend Medical](https://d1.awsstatic.com/diagrams/product-page-diagram-Ontology-Linking_How-It-Works@2x.f2dde99f71240451d64b24bdd202573ff9a26d35.png)\n", 265 | "\n", 266 | "With Amazon Comprehend Medical, you can perform the following on your documents:\n", 267 | "\n", 268 | "- [Detect Entities (Version 2)](https://docs.aws.amazon.com/comprehend/latest/dg/extracted-med-info-V2.html) - Examine unstructured clinical text to detect textual references to medical information such as medical condition, treatment, tests and results, and medications. This version uses a new model and changes the way some entities are returned in the output. For more information, see [DetectEntitiesV2](https://docs.aws.amazon.com/comprehend/latest/dg/API_medical_DetectEntitiesV2.html).\n", 269 | "\n", 270 | "- [Detect PHI (Verdion 1)](https://docs.aws.amazon.com/comprehend/latest/dg/how-medical-phi.html) —Examine unstructured clinical text to detect textual references to protected health information (PHI) such as names and addresses.\n", 271 | "\n", 272 | "\n", 273 | "In this workshop, we will be using the detect entities function ([detect_entities_v2](https://docs.aws.amazon.com/comprehend/latest/dg/extracted-med-info-V2.html)) to extract medical conditions. In the following cell, we will be processing the text on each page in batches of 20,000 UTF-8 characters. This is because Comprehend Medical has a maximum document size of 20,000 bytes (reference: https://docs.aws.amazon.com/comprehend/latest/dg/guidelines-and-limits-med.html). Once we've processed the text, we will then stich up the response into a into a single variable where we can either save to a csv or use for our analysis." 274 | ] 275 | }, 276 | { 277 | "cell_type": "code", 278 | "execution_count": null, 279 | "metadata": {}, 280 | "outputs": [], 281 | "source": [ 282 | "maxLength=20000\n", 283 | "\n", 284 | "comprehendResponse = []\n", 285 | "comprehend_medical_client = boto3.client(service_name='comprehendmedical', region_name='us-east-1')\n", 286 | "\n", 287 | "for page in doc.pages:\n", 288 | " pageText = page.text\n", 289 | " \n", 290 | " for i in range(0, len(pageText), maxLength):\n", 291 | " response = comprehend_medical_client.detect_entities_v2(Text=pageText[0+i:maxLength+i])\n", 292 | " comprehendResponse.append(response)\n", 293 | " patient_string = \"\"" 294 | ] 295 | }, 296 | { 297 | "cell_type": "markdown", 298 | "metadata": {}, 299 | "source": [ 300 | "## Review comprehend results\n", 301 | "The output of *detect_entities_v2* can detect the following entities:\n", 302 | "\n", 303 | "\n", 304 | "- `MEDICAL_CONDITION`: The signs, symptoms, and diagnosis of medical conditions.\n", 305 | "- `Score` - The level of confidence that Amazon Comprehend Medical has in the accuracy of the detection\n", 306 | "- `Trait` - Contextual information for the entity\n", 307 | "\n", 308 | "Other information extracted by Comprehend Medical:\n", 309 | "- `MEDICATION`: Medication and dosage information for the patient.\n", 310 | "- `PROTECTED_HEALTH_INFORMATION`: patient's personal information, e.g. name, age, gender\n", 311 | "- `TEST_TREATMENT_PROCEDURE`: the procedures that are used to determine a medical condition.\n", 312 | "- `TIME_EXPRESSION`: Entities related to time when they are associated with a detected entity.\n", 313 | "\n", 314 | "For this workshop, we will be using the MEDICAL_CONDITION entity to train our machine learning model. Let us take a look at some of the data." 315 | ] 316 | }, 317 | { 318 | "cell_type": "code", 319 | "execution_count": null, 320 | "metadata": {}, 321 | "outputs": [], 322 | "source": [ 323 | "## use our pre-defined util function extractMC_v2 to extract all the medical conditions, confidence score, trait from json file \n", 324 | "df_cm=extractMC_v2(comprehendResponse[0])\n", 325 | "df_cm['ID']=1\n", 326 | "df_cm.head(10)" 327 | ] 328 | }, 329 | { 330 | "cell_type": "markdown", 331 | "metadata": {}, 332 | "source": [ 333 | "---\n", 334 | "\n", 335 | "# Clean up resources\n", 336 | "\n", 337 | "As some resources will be used in the following step, We will clean up the resource until the end of the lab. You may uncomment the following sentence if this is the last step of your lab " 338 | ] 339 | }, 340 | { 341 | "cell_type": "code", 342 | "execution_count": null, 343 | "metadata": {}, 344 | "outputs": [], 345 | "source": [ 346 | "##boto3.Session().resource('s3').Bucket(bucket).Object(textractObjectName).delete()" 347 | ] 348 | } 349 | ], 350 | "metadata": { 351 | "kernelspec": { 352 | "display_name": "conda_python3", 353 | "language": "python", 354 | "name": "conda_python3" 355 | }, 356 | "language_info": { 357 | "codemirror_mode": { 358 | "name": "ipython", 359 | "version": 3 360 | }, 361 | "file_extension": ".py", 362 | "mimetype": "text/x-python", 363 | "name": "python", 364 | "nbconvert_exporter": "python", 365 | "pygments_lexer": "ipython3", 366 | "version": "3.6.10" 367 | } 368 | }, 369 | "nbformat": 4, 370 | "nbformat_minor": 4 371 | } 372 | -------------------------------------------------------------------------------- /2.Batch_Data_Processing.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 2. Batch Data Processing - Batch processing of Electronic Medical Reports (EMR) with Amazon Comprehend Medical\n", 8 | "\n", 9 | "In the previous module, [1.Data_Processing](./1.Data_processing.ipynb), we learnt how to extract medical information from a single PDF medical report using Textract and Comprehend Medical. In this step, we will be preparing our dataset for the classification machine learning model that we will be building. We will use the same pre-processing methodology from the previous step to process a batch of medical reports. But to minimise processing time and cost for this lab, we will skipping the Textract step and focusing on be processing a batch of medical reports in textual format using Comprehend Medical. We use this output from Comprehend Medical as our dataset to train our machine learning model.\n", 10 | "\n", 11 | "## Contents\n", 12 | "\n", 13 | "1. [Objective](#Objective)\n", 14 | "1. [Background](#Background)\n", 15 | "1. [Setup Environment](#Setup-Environment)\n", 16 | "1. [Load and Explore Data](#Load-and-Explore-Data)\n", 17 | "1. [Data Sampling for modeling](#Data-Sampling-for-modeling)\n", 18 | "1. [Combine the dataset](#Combine-the-dataset)\n", 19 | "1. [Save the processed file](#Save-the-processed-file)" 20 | ] 21 | }, 22 | { 23 | "cell_type": "markdown", 24 | "metadata": {}, 25 | "source": [ 26 | "---\n", 27 | "## Objective \n", 28 | "This notebook is the preprocessing step to prepare a batch of medical records for model training. Specifically, you will use Comprehend Medical to extract medical terms from doctors's transcripts and organize them into data frame." 29 | ] 30 | }, 31 | { 32 | "cell_type": "markdown", 33 | "metadata": {}, 34 | "source": [ 35 | "---\n", 36 | "\n", 37 | "## Background\n", 38 | "\n", 39 | "**Dataset**: Medical transcription data scraped from mtsamples.com. The target in the preprocess is to extract the medical conditions from doctors' notes, and then organized into dataset for modeling. In the next step, we will use the processed dataset to correctly classify the medical specialties based on the transcription text. In real life, the model can be used for automatic reference to respective specialist.\n", 40 | "\n", 41 | "**Amazon Comprehend Medical**: Comprehend Medical detects useful information in unstructured clinical text. As much as 75% of all health record data is found in unstructured text such as physician's notes, discharge summaries, test results, and case notes. Amazon Comprehend Medical uses Natural Language Processing (NLP) models to sort through text for valuable information.\n", 42 | "\n", 43 | "**Supported Languages**: Amazon Comprehend Medical only detects medical entities in English language texts." 44 | ] 45 | }, 46 | { 47 | "cell_type": "markdown", 48 | "metadata": {}, 49 | "source": [ 50 | "---\n", 51 | "## Setup Environment\n", 52 | "\n", 53 | "Before we begin, let us setup our environment, will be doing the following:\n", 54 | "\n", 55 | "- **import** some useful libraries (as in any Python notebook)\n", 56 | "- **configure** the S3 bucket and folder where data should be stored (to keep our environment tidy)\n", 57 | "- **connect** to Amazon Comprehend(with [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)) and SageMaker in particular (with the [sagemaker SDK](https://sagemaker.readthedocs.io/en/stable/)), to use the cloud services\n", 58 | "\n", 59 | "Note: While `boto3` is the general AWS SDK for Python, `sagemaker` provides some powerful, higher-level interfaces designed specifically for ML workflows." 60 | ] 61 | }, 62 | { 63 | "cell_type": "code", 64 | "execution_count": null, 65 | "metadata": {}, 66 | "outputs": [], 67 | "source": [ 68 | "import numpy as np # For matrix operations and numerical processing\n", 69 | "import pandas as pd # For munging tabular data\n", 70 | "import time\n", 71 | "import os\n", 72 | "from util.preprocess import * # helper function for classification reports\n", 73 | "\n", 74 | "# setting up SageMaker parameters\n", 75 | "import sagemaker\n", 76 | "import boto3\n", 77 | "\n", 78 | "import matplotlib.pyplot as plt\n", 79 | "import seaborn as sns\n", 80 | "\n", 81 | "boto_session = boto3.Session()\n", 82 | "region = boto_session.region_name\n", 83 | "bucket_name = sagemaker.Session().default_bucket()\n", 84 | "bucket_prefix = \"emr-mtSample\" # Location in the bucket to store our files\n", 85 | "sgmk_session = sagemaker.Session()\n", 86 | "\n", 87 | "sgmk_client = boto_session.client(\"sagemaker\") ## API for sagemaker\n", 88 | "cm_client = boto3.client(service_name='comprehendmedical', use_ssl=True, region_name = 'us-east-1') ## API for comprehend medical" 89 | ] 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "metadata": {}, 94 | "source": [ 95 | "---\n", 96 | "## Load and Explore Data\n", 97 | "\n", 98 | "For this workshop, we have already included the MTSamples dataset in your local folder (`./data/mtsample.csv`). You can find the raw dataset is available at [kaggle](https://www.kaggle.com/tboyle10/medicaltranscriptions). \n", 99 | "\n", 100 | "**Columns in the dataset**:\n", 101 | "\n", 102 | "* `description`: Short description of transcription (string)\n", 103 | "* `medical_specialty`: Medical specialty classification of transcription (string)\n", 104 | "* `sample_name`: Transcription title\n", 105 | "* `transcription`: Transcribed doctors' notes\n", 106 | "* `keywords`: Relevant keywords from transcription\n", 107 | "\n", 108 | "In the next section of our workshop, [3.Model building, training and deployment](./3.Model_building_training_and_development.ipynb), we will be building a machine learning model to classify medical reports based on the transcription. But before we can do that, we need to extract data from the MTSamples dataset. To train our model, we will be extracting the medical information from the `transcription` column using Amazon Comprehend Medical. We will also be using the `medical_specialty`column as our labels for our classification model." 109 | ] 110 | }, 111 | { 112 | "cell_type": "code", 113 | "execution_count": null, 114 | "metadata": {}, 115 | "outputs": [], 116 | "source": [ 117 | "df=pd.read_csv(\"./data/mtsamples.csv\")\n", 118 | "df.head()" 119 | ] 120 | }, 121 | { 122 | "cell_type": "markdown", 123 | "metadata": {}, 124 | "source": [ 125 | "### Clean up dataset\n", 126 | "\n", 127 | "Before we explore and process the dataset, let us check for empty columns and remove them." 128 | ] 129 | }, 130 | { 131 | "cell_type": "code", 132 | "execution_count": null, 133 | "metadata": {}, 134 | "outputs": [], 135 | "source": [ 136 | "df.isnull().sum(axis=0) ## check for missing information" 137 | ] 138 | }, 139 | { 140 | "cell_type": "markdown", 141 | "metadata": {}, 142 | "source": [ 143 | "Remove the *33* rows with `transcription` is null." 144 | ] 145 | }, 146 | { 147 | "cell_type": "code", 148 | "execution_count": null, 149 | "metadata": {}, 150 | "outputs": [], 151 | "source": [ 152 | "df=df[df['transcription'].isnull()==False].reset_index()\n", 153 | "df.isnull().sum(axis=0) " 154 | ] 155 | }, 156 | { 157 | "cell_type": "markdown", 158 | "metadata": {}, 159 | "source": [ 160 | "### Explore dataset by medical speciality\n", 161 | "\n", 162 | "Let us do some data exploration and observe the distribution of medical reports by medical speciality" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": null, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "## add patient_id for reference\n", 172 | "df['id']=df.index\n", 173 | "sns.set(rc={'figure.figsize':(15,10)})\n", 174 | "sns.countplot(y='medical_specialty',order=df['medical_specialty'].value_counts().index, data=df) #df.medical_specialty.value_counts()" 175 | ] 176 | }, 177 | { 178 | "cell_type": "markdown", 179 | "metadata": {}, 180 | "source": [ 181 | "---\n", 182 | "## Data Sampling for modeling\n", 183 | "\n", 184 | "#### Business Question to be addressed here:\n", 185 | "Based on the medical description of this patient, should he/she go just for *Consulation* or be referred to *Surgery* specialist. \n", 186 | "\n", 187 | "#### ML problem to resolve:\n", 188 | "Binary classfication-*Consulation* or *Surgery* - for patient based on medical conditions.\n", 189 | "\n", 190 | "#### Why we do data sampling at this step? \n", 191 | "\n", 192 | "For demo purpose and time concern, we will sample 200 records from both categories (in total, we have 1000+ medical reports that have been classified under the **Surgery** and 400+ under the **Consult** medical speciality) to conduct binary classification for step 3.\n" 193 | ] 194 | }, 195 | { 196 | "cell_type": "markdown", 197 | "metadata": {}, 198 | "source": [ 199 | "### Data Sampling 1- Sampling from Surgery\n", 200 | "\n", 201 | "Let's randomly select 200 patients from the **Surgery** category and extract the medical conditions using Amazon Comprehend Medical." 202 | ] 203 | }, 204 | { 205 | "cell_type": "code", 206 | "execution_count": null, 207 | "metadata": {}, 208 | "outputs": [], 209 | "source": [ 210 | "%%time\n", 211 | "nSample=20 ## . We recommend 20 for time consideration\n", 212 | "medical_specialty=' Surgery'\n", 213 | "df_phyList1,patient_ids=subpopulation_comprehend(df, medical_specialty,sampleSize=nSample)" 214 | ] 215 | }, 216 | { 217 | "cell_type": "markdown", 218 | "metadata": {}, 219 | "source": [ 220 | "### Challenge 1: Do a batch processing using Amazon Comprehend Medical \n", 221 | "\n", 222 | "In this challenge, you are required to write your code within for-loop. You are expected to extract all the medical_conditions for each patient, together with the confidence score \n", 223 | "\n" 224 | ] 225 | }, 226 | { 227 | "cell_type": "code", 228 | "execution_count": null, 229 | "metadata": {}, 230 | "outputs": [], 231 | "source": [ 232 | "## Function to process a multiple records\n", 233 | "def extractMCbatch(transcriptionList,patientIDList):\n", 234 | " df_final = pd.DataFrame()\n", 235 | " \n", 236 | " if(len(transcriptionList)!=len(patientIDList)):\n", 237 | " return(\"Error! different length!\")\n", 238 | " \n", 239 | " ## In this for loop, gererate a wide dataframe with extracted medical condition from each item, together with the corresponding ID \n", 240 | " for item,patient_id in zip(transcriptionList,patientIDList):\n", 241 | " print(\"processing patient_id:\",patient_id )\n", 242 | " \n", 243 | " ###########################################################################\n", 244 | " ## TODO: WRITE YOUR CODE HERE TO GENERATE A ROW FOR THE ITEM FOR THE PATIENT!\n", 245 | " ###########################################################################\n", 246 | " \n", 247 | " # remove the duplicated entries if any\n", 248 | " df_final=df_final.sort_values(by=['ID','MEDICAL_CONDITION']).drop_duplicates(['ID','MEDICAL_CONDITION'],keep='last')\n", 249 | "\n", 250 | " return df_final" 251 | ] 252 | }, 253 | { 254 | "cell_type": "markdown", 255 | "metadata": {}, 256 | "source": [ 257 | "#### Stucked?\n", 258 | "Feel free to check the hint/recording section in [*step 4*](https://www.aiml-loft.wwps.aws.dev/workshops/module-medical-document-processing-and-classification/step2/) on the website " 259 | ] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "metadata": {}, 264 | "source": [ 265 | "### Test your function *extractMCbatch* and visualize it" 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": null, 271 | "metadata": {}, 272 | "outputs": [], 273 | "source": [ 274 | "df_extracted_surg=extractMCbatch(df_phyList1,patient_ids)\n", 275 | "\n", 276 | "## plot the results\n", 277 | "topN=20 ## the number for top conditions\n", 278 | "threshold_score=0.9 ##the threshold of confidence score\n", 279 | "df_surg_plot=mc_barplot(df_extracted_surg, threshold_score,topN)" 280 | ] 281 | }, 282 | { 283 | "cell_type": "markdown", 284 | "metadata": {}, 285 | "source": [ 286 | "### Data Sampling 2 - *Consult - History and Phy*\n", 287 | "\n", 288 | "Let us pick another medical speciality we want to try and classify. Let us pick 200 medical reports from the next most popular medical speciality - *Consult - History and Phy*." 289 | ] 290 | }, 291 | { 292 | "cell_type": "code", 293 | "execution_count": null, 294 | "metadata": {}, 295 | "outputs": [], 296 | "source": [ 297 | "%%time\n", 298 | "nSample=20 ##\n", 299 | "medical_specialty=' Consult - History and Phy.'\n", 300 | "df_phyList,patient_ids=subpopulation_comprehend(df, medical_specialty,sampleSize=nSample)" 301 | ] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "execution_count": null, 306 | "metadata": {}, 307 | "outputs": [], 308 | "source": [ 309 | "## organize into a dataframe\n", 310 | "df_extracted_consult=extractMCbatch(df_phyList,patient_ids)\n", 311 | "## plot the results\n", 312 | "topN=20 ## the number for top conditions\n", 313 | "threshold_score=0.9\n", 314 | "mc_barplot(df_extracted_consult, threshold_score,topN)" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "---\n", 322 | "## Combine the dataset\n", 323 | "\n", 324 | "Until now, you have successfully generated 2 dataframes! One is for referal to *`Surgery`* specialist and the other for just for *`Consultation`*. \n", 325 | "\n", 326 | "After sampling the 2 medical specialities for classification, let's consolidate this data and save it for the next part of the workshop.\n", 327 | "\n", 328 | "### Gaps:\n", 329 | "After taking a closer look at the dataset, you will observe that the dataset is in long format, meaning that each row represents a single medical condition for one patients. If a patient *John* has 10 medical conditions, there will be 10 rows. Thus, you may expect varied number of rows of each patient. \n", 330 | "### Solutions:\n", 331 | "To make the dataset easier for ML algorithm to handle, we need to convert them into wide format, one row for one patient. Instead of keeping all the existing medical conditions, let's select the top `20` medical conditions from each category as input features. Note that `20` here is an arbitrary number which you can increase or decrease based on your experience.\n", 332 | "\n", 333 | "In the following cells, function *`retrieve_mcList(df, nFeature=20,threshold=0.9)`* helps to retrieve the features from each subset with `nFeature`(default=20) as specified number of features and `threshold`(default=0.9) as the confidence threshold. Outputs from *`retrieve_mcList()`*:\n", 334 | "\n", 335 | "+ top medical conditions list,\n", 336 | "+ cleaned dataframe through converting to lower case, merg *etc*.\n", 337 | "\n", 338 | "\n", 339 | "`Target column`: as it is a binary classification problem, we defined a new column called `Label`, representing whether the patient need to go through surgery (`True`) or not (`False`). " 340 | ] 341 | }, 342 | { 343 | "cell_type": "code", 344 | "execution_count": null, 345 | "metadata": {}, 346 | "outputs": [], 347 | "source": [ 348 | "df_extracted_surg.head()" 349 | ] 350 | }, 351 | { 352 | "cell_type": "code", 353 | "execution_count": null, 354 | "metadata": {}, 355 | "outputs": [], 356 | "source": [ 357 | "df_extracted_consult.head()" 358 | ] 359 | }, 360 | { 361 | "cell_type": "code", 362 | "execution_count": null, 363 | "metadata": {}, 364 | "outputs": [], 365 | "source": [ 366 | "mcList1, df_grp1=retrieve_mcList(df_extracted_surg, 20)" 367 | ] 368 | }, 369 | { 370 | "cell_type": "code", 371 | "execution_count": null, 372 | "metadata": {}, 373 | "outputs": [], 374 | "source": [ 375 | "mcList2, df_grp2=retrieve_mcList(df_extracted_consult)" 376 | ] 377 | }, 378 | { 379 | "cell_type": "code", 380 | "execution_count": null, 381 | "metadata": {}, 382 | "outputs": [], 383 | "source": [ 384 | "## Label the target column \"Label\": True--> go to see Surgery specialist, False --> only Consultation\n", 385 | "df_grp1['Label']=True # group one is Labeled as True \n", 386 | "df_grp2['Label']=False # group two is Labeled as False \n", 387 | "\n", 388 | "\n", 389 | "df_combined=df_grp1.append(df_grp2) ## append two data frames \n", 390 | "mcLists=list(set(mcList1+mcList2))\n", 391 | "\n", 392 | "df_combined2=df_mc_generator(df_combined,mcLists ,colname_other=['ID',\"Label\"] )" 393 | ] 394 | }, 395 | { 396 | "cell_type": "markdown", 397 | "metadata": {}, 398 | "source": [ 399 | "---\n", 400 | "## Save the processed file" 401 | ] 402 | }, 403 | { 404 | "cell_type": "code", 405 | "execution_count": null, 406 | "metadata": {}, 407 | "outputs": [], 408 | "source": [ 409 | "df_combined2.to_csv(\"./data/processed_combined_wide_short.csv\",index=False)" 410 | ] 411 | } 412 | ], 413 | "metadata": { 414 | "kernelspec": { 415 | "display_name": "conda_python3", 416 | "language": "python", 417 | "name": "conda_python3" 418 | }, 419 | "language_info": { 420 | "codemirror_mode": { 421 | "name": "ipython", 422 | "version": 3 423 | }, 424 | "file_extension": ".py", 425 | "mimetype": "text/x-python", 426 | "name": "python", 427 | "nbconvert_exporter": "python", 428 | "pygments_lexer": "ipython3", 429 | "version": "3.6.10" 430 | } 431 | }, 432 | "nbformat": 4, 433 | "nbformat_minor": 4 434 | } 435 | -------------------------------------------------------------------------------- /3.Model_building_training_and_deployment.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 3. Building an XGBoost model to classify electronic medical records (EMR) using Sagemaker\n", 8 | "\n", 9 | "In the previous module of the workshop, we prepared our dataset by extracting medical records that had a medical speciality of *`Surgery`* and *`Consultation`*. In this module, we will be using this dataset to create a classification model that will look at the transcription and categorize them into the following two categories - undergoing *`Consultation`* only or refer to a specialist for *`Sugery`*.\n", 10 | "\n", 11 | "The goal of this experiment is to do a **Next step Prediction** which aims at predicting the speciality needed for a patient with certain diseases. In practice, the model could be used to analyze a medical transcription in real-time and provide a recommended referals to respective specialist. \n", 12 | "\n", 13 | "The input is the EMR as a raw text file with doctor's notes about the patient, including his/her age, compaints described in free way, patient's history and so on. It is unstructured - different sections of oen patient anamnesis may abscent in another's.\n", 14 | "\n", 15 | "The value on the solution might be found in helping a doctor to find the optimal solution for diasnostics order. Patient can save time and money, and doctor can serve a patient more efficiently on sparing time for unnecessary diagnostics. Moreover, in difficult cases the algorithm may help a doctor to find a diagnosys faster, which in some cases may be extremely valuable, up to saving lives.\n", 16 | "\n", 17 | "Theoretically some regularities found by the algorithm may help medical researchers to find the idea of treating some deseases, based on their unobvious interconnections with some symptoms.\n", 18 | "\n", 19 | "---\n", 20 | "\n", 21 | "## Contents\n", 22 | "\n", 23 | "1. [Objective](#Objective)\n", 24 | "1. [Setup Environment](#Setup-Environment)\n", 25 | "1. [Load and Explore the Dataset](#Load-and-Explore-Dataset)\n", 26 | "1. [Prepare Dataset for Model Training](#Prepare-Dataset-for-Model-Training)\n", 27 | "1. [Understand the Algorithm](#Understand-the-Algorithm)\n", 28 | "1. [Train the Model](#Train-the-Model)\n", 29 | "1. [Deploy and Evaluate the Model](#Deploy-and-Evaluate-the-Model)\n", 30 | "1. [Challenges](#Challenges)\n", 31 | "1. [Hyperparameter Optimization](#Hyperparameter-Optimization)\n", 32 | "1. [Bonus Activtity](#Bonus-Activtity)\n", 33 | "1. [Conclusion](#Conclusion)\n", 34 | "1. [Clean up resources](#Clean-up-resources)\n", 35 | "\n" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": {}, 41 | "source": [ 42 | "---\n", 43 | "\n", 44 | "## Objective \n", 45 | "\n", 46 | "In this section of the workshop we will walkthrough an example of using Sagemaker to train a machine learning model to classify medical documents using a SageMaker's built-in algorithm- XGboost. \n", 47 | " \n", 48 | "**Bonus activity**: The workshop will also include a bonus activity whereby you will learn how to use SageMaker's automatic Hyperparameter Optimization (HPO) capabilities to improve the model's performance with needing to tune the hyperparameters manually.\n", 49 | "\n", 50 | "**Note**: Teaching in-depth data science approaches for tabular data is outside this scope, however we recommend that you use this notebook as a reference to adapt to your needs for future projects." 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "---\n", 58 | "## Setup Environment\n", 59 | "\n", 60 | "Before we can begin, we will need to setup up notebook environment by performing some of the following environment setup:\n", 61 | "\n", 62 | "- **import** some useful libraries (as in any Python notebook)\n", 63 | "- **configure** the S3 bucket and folder where data should be stored (to keep our environment tidy)\n", 64 | "- **connect** to AWS in general (with [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)) and SageMaker in particular (with the [sagemaker SDK](https://sagemaker.readthedocs.io/en/stable/)), to use the cloud services\n", 65 | "- **Upgrad** SageMaker to the latest version\n", 66 | "\n", 67 | "While `boto3` is the general AWS SDK for Python, `sagemaker` provides some powerful, higher-level interfaces designed specifically for ML workflows." 68 | ] 69 | }, 70 | { 71 | "cell_type": "code", 72 | "execution_count": null, 73 | "metadata": { 74 | "scrolled": true 75 | }, 76 | "outputs": [], 77 | "source": [ 78 | "pip install --upgrade sagemaker" 79 | ] 80 | }, 81 | { 82 | "cell_type": "code", 83 | "execution_count": null, 84 | "metadata": {}, 85 | "outputs": [], 86 | "source": [ 87 | "import numpy as np # For matrix operations and numerical processing\n", 88 | "import pandas as pd # For munging tabular data\n", 89 | "pd.set_option('display.max_colwidth', None)\n", 90 | "\n", 91 | "import time\n", 92 | "import os\n", 93 | "\n", 94 | "# import self-defined functions\n", 95 | "from util.classification_report import generate_classification_report, predict_from_numpy_V2 # helper function for classification reports\n", 96 | "from util.Pipeline import extractTextract, extractMedical\n", 97 | "from util.preprocess import *\n", 98 | "\n", 99 | "# setting up SageMaker parameters\n", 100 | "import pkg_resources\n", 101 | "pkg_resources.require(\"sagemaker>2.9.2\") \n", 102 | "import sagemaker\n", 103 | "import boto3\n", 104 | "\n", 105 | "import matplotlib.pyplot as plt\n", 106 | "import seaborn as sns\n", 107 | "\n", 108 | "boto_session = boto3.Session()\n", 109 | "region = boto_session.region_name\n", 110 | "bucket_name = sagemaker.Session().default_bucket()\n", 111 | "bucket_prefix = \"emr-mtSample\" # Location in the bucket to store our files\n", 112 | "sgmk_session = sagemaker.Session()\n", 113 | "sgmk_client = boto_session.client(\"sagemaker\")\n", 114 | "sgmk_role = sagemaker.get_execution_role()" 115 | ] 116 | }, 117 | { 118 | "cell_type": "markdown", 119 | "metadata": {}, 120 | "source": [ 121 | "---\n", 122 | "## Load and Explore Dataset\n", 123 | "\n", 124 | "To begin, we will load the dataset from our previous notebook [2.Batch_Data_Processing](./2.Batch_Data_Processing.ipynb). This dataset contains labelled data based on the medical speciality - Surgery or not Surgery and the medical conditions that were extracted from the electronic medical reports.\n", 125 | "You can find the processed dataset in the following location '/data/processed_combined_wide.csv'.\n", 126 | "\n", 127 | "**Note**: The original raw dataset that this workshop is based on is available at [kaggle](https://www.kaggle.com/tboyle10/medicaltranscriptions).\n", 128 | "\n", 129 | "*Demographics:*\n", 130 | "* `ID`: id of the patients (int)\n", 131 | "* `Label`: the patient needs surgery? (bool)\n", 132 | "* `nontender`: medical condition extracted from doctos'notes. the number indicate confidence of the symptom (float)\n", 133 | "* the rest columns are other medical condistions ..." 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": { 140 | "scrolled": true 141 | }, 142 | "outputs": [], 143 | "source": [ 144 | "df_wide=pd.read_csv(\"./data/processed_combined_wide.csv\")\n", 145 | "df_wide.head()" 146 | ] 147 | }, 148 | { 149 | "cell_type": "markdown", 150 | "metadata": {}, 151 | "source": [ 152 | "#### Explore correlation between the input variables and output one\n", 153 | "\n", 154 | "Let us start by looking at the correlation between the input features and our label (Surgry/Not Surgery).\n" 155 | ] 156 | }, 157 | { 158 | "cell_type": "code", 159 | "execution_count": null, 160 | "metadata": {}, 161 | "outputs": [], 162 | "source": [ 163 | "corrPlot(df_wide)" 164 | ] 165 | }, 166 | { 167 | "cell_type": "markdown", 168 | "metadata": {}, 169 | "source": [ 170 | "#### Observations\n", 171 | "+ Observation 1: *`wound`* and *`hypertension`* is postively and negatively correlated with Surgery. \n", 172 | "+ Observation 2: *`nausea` and *`vomitting`* is positively correlated\n", 173 | "+ Did you observe any more from the plot?" 174 | ] 175 | }, 176 | { 177 | "cell_type": "markdown", 178 | "metadata": {}, 179 | "source": [ 180 | "---\n", 181 | "## Prepare-Dataset-for-Model-Training\n", 182 | "\n", 183 | "Next, we will start training the model. But before we can proceeed, we need to:\n", 184 | "\n", 185 | "1. Suffle and split the data into **Training (80%)**, **Validation (10%)**, and **Test (10%)** sets\n", 186 | "2. Convert the data to the format the algorithm expects (e.g. CSV)\n", 187 | "3. Upload the data to S3\n", 188 | "4. Create `s3_input` objects defining the data sources for the SageMaker SDK\n", 189 | "\n", 190 | "The training and validation datasets will be used during the training (and tuning) phase, while the 'holdout' test set will be used afterwards to evaluate the model.\n", 191 | "\n", 192 | "Please note that to train the SageMaker XGBoost algorithm, it expects data in the **libSVM** or **CSV** formats with the following format:\n", 193 | "\n", 194 | "- The target variable in the first column, and\n", 195 | "- No header row\n", 196 | "\n", 197 | "You can find more information about this [here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html#InputOutput-XGBoost)." 198 | ] 199 | }, 200 | { 201 | "cell_type": "code", 202 | "execution_count": null, 203 | "metadata": {}, 204 | "outputs": [], 205 | "source": [ 206 | "# remove the id column \n", 207 | "df_combined_model=df_wide.iloc[:,1:] \n", 208 | "# Shuffle and splitting dataset\n", 209 | "train_data, validation_data, test_data = np.split(df_combined_model.sample(frac=1, random_state=123), \n", 210 | " [int(0.8 * len(df_combined_model)), int(0.9*len(df_combined_model))],) \n", 211 | "\n", 212 | "# Create CSV files for Train / Validation / Test\n", 213 | "train_data.to_csv(\"data/train.csv\", index=False, header=False)\n", 214 | "validation_data.to_csv(\"data/validation.csv\", index=False, header=False)\n", 215 | "test_data.to_csv(\"data/test.csv\", index=False, header=True)" 216 | ] 217 | }, 218 | { 219 | "cell_type": "markdown", 220 | "metadata": {}, 221 | "source": [ 222 | "### Upload dataset to S3" 223 | ] 224 | }, 225 | { 226 | "cell_type": "code", 227 | "execution_count": null, 228 | "metadata": {}, 229 | "outputs": [], 230 | "source": [ 231 | "# Upload CSV files to S3 for SageMaker training\n", 232 | "train_uri = sgmk_session.upload_data(\n", 233 | " path=\"data/train.csv\",\n", 234 | " bucket=bucket_name,\n", 235 | " key_prefix=bucket_prefix\n", 236 | ")\n", 237 | "val_uri = sgmk_session.upload_data(\n", 238 | " path=\"data/validation.csv\",\n", 239 | " bucket=bucket_name,\n", 240 | " key_prefix=bucket_prefix\n", 241 | ")\n", 242 | "\n", 243 | "\n", 244 | "# Create s3_inputs\n", 245 | "\n", 246 | "s3_input_train = sagemaker.TrainingInput(s3_data=train_uri, content_type=\"csv\")\n", 247 | "s3_input_validation = sagemaker.TrainingInput(s3_data=val_uri, content_type=\"csv\")\n", 248 | "\n", 249 | "print(f\"{s3_input_train.config}\\n\\n{s3_input_validation.config}\")" 250 | ] 251 | }, 252 | { 253 | "cell_type": "markdown", 254 | "metadata": {}, 255 | "source": [ 256 | "## Understand the Algorithm\n", 257 | "**`XGBoost`** stands for e**X**treme **G**radient **Boosting**. It implements the gradient boosting decision tree algorithm, which is an approach where new models are created that predict the residuals or errors of prior models and then added together to make the final prediction. It is called gradient boosting because it uses a gradient descent algorithm to minimize the loss when adding new models. Check detailed documentation for built-in *XGBoost* [here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost.html).\n", 258 | "\n", 259 | "*\"The name xgboost, though, actually refers to the engineering goal to push the limit of computations resources for boosted tree algorithms. Which is the reason why many people use xgboost.\"* -- Tianqi Chen, Creator of *XGBoost*\n", 260 | "\n", 261 | "The two major advantages of using XGBoost are:\n", 262 | "\n", 263 | " 1. Fast Execution Speed: Generally, XGBoost is faster when compared to other implementations of gradient boosting.\n", 264 | " 2. High Model Performance: XGBoost has exceled in either structured or tabular datasets on classification and regression predictive modeling problems.\n" 265 | ] 266 | }, 267 | { 268 | "cell_type": "code", 269 | "execution_count": null, 270 | "metadata": {}, 271 | "outputs": [], 272 | "source": [ 273 | "from sagemaker import image_uris \n", 274 | "from sagemaker.image_uris import retrieve\n", 275 | "\n", 276 | "training_image = retrieve(\"xgboost\",region=region, version=\"1.0-1\")\n", 277 | "\n", 278 | "print(training_image)" 279 | ] 280 | }, 281 | { 282 | "cell_type": "markdown", 283 | "metadata": {}, 284 | "source": [ 285 | "### Define Hyperparameters & Algorithm\n", 286 | "In the following step, we will use the [sagemaker.estimator.Estimator()](https://sagemaker.readthedocs.io/en/v1.72.0/api/training/estimators.html) function to configure the following:\n", 287 | "\n", 288 | "* image_name - Training image to use(image_name), in this case we will be using the xgboost training image\n", 289 | "* train_instance_type - Type of instance to use.\n", 290 | "* train_instance_count - The number of instances to run the training job. For suitable algorithms that support distributed training, set an instance count of more than 1.\n", 291 | "* role - IAM role used to run the training job\n", 292 | "* train_use_spot_instances - Specify whether to use spot instances. For more information about spot training, refer to the following url: https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html\n", 293 | "* train_max_run - Timeout in seconds for training (default: 24 * 60 * 60). After this amount of time Amazon SageMaker terminates the job regardless of its current status.\n", 294 | "* train_max_wait - Timeout in seconds waiting for spot training instances\n", 295 | "* hyperparameters - Our hyperparameters used to train the model\n", 296 | "\n", 297 | "For more information about the available xgboost hyperparamters, please refer to the following documentation [here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost_hyperparameters.html)." 298 | ] 299 | }, 300 | { 301 | "cell_type": "code", 302 | "execution_count": null, 303 | "metadata": { 304 | "scrolled": true 305 | }, 306 | "outputs": [], 307 | "source": [ 308 | "hyperparameters = {\n", 309 | " \"num_round\": \"150\", # int: [1,300]\n", 310 | " \"max_depth\": \"5\", # int: [1,10]\n", 311 | " \"alpha\": \"2.5\", # float: [0,5]\n", 312 | " \"eta\": \"0.5\", # float: [0,1]\n", 313 | " \"objective\": \"binary:logistic\",\n", 314 | "}\n", 315 | "\n", 316 | "# Instantiate an XGBoost estimator object\n", 317 | "estimator = sagemaker.estimator.Estimator(\n", 318 | " image_uri=training_image, # XGBoost algorithm container\n", 319 | " instance_type=\"ml.m5.xlarge\", # type of training instance\n", 320 | " instance_count=1, # number of instances to be used\n", 321 | " role=sgmk_role, # IAM role to be used\n", 322 | " use_spot_instances=True, # Use spot instances to reduce cost\n", 323 | " max_run=20*60, # Maximum allowed active runtime\n", 324 | " max_wait=30*60, # Maximum clock time (including spot delays)\n", 325 | " hyperparameters=hyperparameters\n", 326 | ")" 327 | ] 328 | }, 329 | { 330 | "cell_type": "markdown", 331 | "metadata": {}, 332 | "source": [ 333 | "---\n", 334 | "## Train-the-Model\n", 335 | "\n", 336 | "To start the training job, we will call the `estimator.fit()` function. This will start a Sagemaker training job in the background. You can also see your training job within the AWS console by going to Sagemaker -> Training jobs.\n", 337 | "\n", 338 | "Once the training job is completed, proceed to the next step." 339 | ] 340 | }, 341 | { 342 | "cell_type": "code", 343 | "execution_count": null, 344 | "metadata": { 345 | "scrolled": true 346 | }, 347 | "outputs": [], 348 | "source": [ 349 | "# start a training (fitting) job\n", 350 | "estimator.fit({ \"train\": s3_input_train, \"validation\": s3_input_validation })" 351 | ] 352 | }, 353 | { 354 | "cell_type": "markdown", 355 | "metadata": {}, 356 | "source": [ 357 | "## Deploy and Evaluate the Model\n", 358 | "Now that we've trained our xgboost model, let us proceed with deploying our model (hosting it behind a real-time endpoint) so that we can start running predictions in real-time. This can be done using the `estimator.deploy()` function. You can find more information about model deployment here - https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-deployment.html.\n", 359 | "\n", 360 | "For the input to the deployment function, we will specify the following:\n", 361 | "\n", 362 | "* initial_instance_count - Minimum number of EC2 instances to deploy to an endpoint for prediction.\n", 363 | "* instance_type - Type of EC2 instance to deploy to an endpoint for prediction, for example, ‘ml.c4.xlarge’.\n", 364 | "\n", 365 | "This deployment might take up to 8 minutes, and by default the code will wait for the deployment to complete.\n", 366 | "If you like, you can instead:\n", 367 | "+ Un-comment the wait=False parameter\n", 368 | "+ Use the Endpoints page of the SageMaker Console to check the status of the deployment\n", 369 | "+ Skip over the Evaluation section below (which won't run until the deployment is complete), and start the Hyperparameter Optimization job - which will take a while to run too, so can be started in parallel" 370 | ] 371 | }, 372 | { 373 | "cell_type": "code", 374 | "execution_count": null, 375 | "metadata": {}, 376 | "outputs": [], 377 | "source": [ 378 | "predictor = estimator.deploy(\n", 379 | " #endpoint_name=auto_ml_job_name,\n", 380 | " initial_instance_count=1,\n", 381 | " instance_type=\"ml.m5.large\",\n", 382 | " #inference_response_keys=inference_response_keys,\n", 383 | " predictor_cls=sagemaker.predictor.Predictor,\n", 384 | " #serializer = sagemaker.serializers.CSVSerializer()\n", 385 | " #wait=False\n", 386 | ")" 387 | ] 388 | }, 389 | { 390 | "cell_type": "markdown", 391 | "metadata": {}, 392 | "source": [ 393 | "## Run predictions\n", 394 | "\n", 395 | "Once the Sagemaker endpoint has been deployed, we can now run some prediction to test our endpoint. Let us test our endpoint by running some predictions on our test data and evaluating the results." 396 | ] 397 | }, 398 | { 399 | "cell_type": "code", 400 | "execution_count": null, 401 | "metadata": {}, 402 | "outputs": [], 403 | "source": [ 404 | "# (This cell will error, until the endpoint deployment above is finished!)\n", 405 | "\n", 406 | "# Get predictions for the test set:\n", 407 | "predictions = predict_from_numpy_V2(predictor, test_data.drop([\"Label\"], axis=1))\n", 408 | "\n", 409 | "print(predictions)" 410 | ] 411 | }, 412 | { 413 | "cell_type": "code", 414 | "execution_count": null, 415 | "metadata": {}, 416 | "outputs": [], 417 | "source": [ 418 | "from imp import reload\n", 419 | "from util import classification_report\n", 420 | "reload(classification_report)\n", 421 | "\n", 422 | "predictions = predict_from_numpy_V2(predictor, test_data.drop([\"Label\"], axis=1))" 423 | ] 424 | }, 425 | { 426 | "cell_type": "code", 427 | "execution_count": null, 428 | "metadata": { 429 | "scrolled": true 430 | }, 431 | "outputs": [], 432 | "source": [ 433 | "# helper function for classification reports\n", 434 | "\n", 435 | "generate_classification_report(\n", 436 | " y_real=test_data['Label'].values, \n", 437 | " y_predict_proba=predictions, \n", 438 | " decision_threshold=0.5,\n", 439 | " class_names_list=[\"Negative\", \"Positive\"],\n", 440 | " title=\"Initial model\",\n", 441 | ")" 442 | ] 443 | }, 444 | { 445 | "cell_type": "markdown", 446 | "metadata": {}, 447 | "source": [ 448 | "---\n", 449 | "## Challenges" 450 | ] 451 | }, 452 | { 453 | "cell_type": "markdown", 454 | "metadata": {}, 455 | "source": [ 456 | "### Challenge 2a: Change the Hyperparameter \n", 457 | "Now that we've trained our model using some pre-defined hyperparamters, let us try and see if we can improve the model performance by mannually change the HyperParameter. For example, you can change the number of `num_round`, `max_depth` and the rest of the hyperparameter that we plan to tune.\n", 458 | "\n" 459 | ] 460 | }, 461 | { 462 | "cell_type": "code", 463 | "execution_count": null, 464 | "metadata": {}, 465 | "outputs": [], 466 | "source": [ 467 | "##!! TO DO: You are expected to fill the hyperparameters bellow and Run the Model training\n", 468 | "\n", 469 | "hyperparameters2 = {\n", 470 | " \"num_round\": \"\", ## TO WRITE \n", 471 | " \"max_depth\": \"\", ## TO WRITE \n", 472 | " \"alpha\": \"\", ## TO WRITE \n", 473 | " \"eta\": \"\", ## TO WRITE \n", 474 | " \"objective\": \"binary:logistic\",\n", 475 | "}\n", 476 | "\n", 477 | "# Instantiate an XGBoost estimator object\n", 478 | "estimator2 = sagemaker.estimator.Estimator(\n", 479 | " image_uri=training_image, # XGBoost algorithm container\n", 480 | " instance_type=\"ml.m5.xlarge\", # type of training instance\n", 481 | " instance_count=1, # number of instances to be used\n", 482 | " role=sgmk_role, # IAM role to be used\n", 483 | " use_spot_instances=True, # Use spot instances to reduce cost\n", 484 | " max_run=20*60, # Maximum allowed active runtime\n", 485 | " max_wait=30*60, # Maximum clock time (including spot delays)\n", 486 | " hyperparameters=hyperparameters2\n", 487 | ")\n", 488 | "estimator2.fit({ \"train\": s3_input_train, \"validation\": s3_input_validation }) ## model fitting \n", 489 | "estimator2.create_model() ## create a new model from the training job" 490 | ] 491 | }, 492 | { 493 | "cell_type": "markdown", 494 | "metadata": {}, 495 | "source": [ 496 | "### Create model from the training job\n", 497 | "\n", 498 | "After the training job is done, the model is not saved yet. Check training jobs and models in your SageMaker Console. To create a model from a training job, refer to the documentation for *[create_model API](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_model)*" 499 | ] 500 | }, 501 | { 502 | "cell_type": "code", 503 | "execution_count": null, 504 | "metadata": {}, 505 | "outputs": [], 506 | "source": [ 507 | "##!! TO DO: get the model artifact information\n", 508 | "\n", 509 | "## hint 1: create a primary container with the trained model\n", 510 | "## hint 2: check the API estimator2.create_model().model_data " 511 | ] 512 | }, 513 | { 514 | "cell_type": "code", 515 | "execution_count": null, 516 | "metadata": {}, 517 | "outputs": [], 518 | "source": [ 519 | "##!! TO DO: create a model from the container\n", 520 | "\n", 521 | "## hint 3: estimator2.create_model().model_data " 522 | ] 523 | }, 524 | { 525 | "cell_type": "markdown", 526 | "metadata": {}, 527 | "source": [ 528 | "### Stucked?\n", 529 | "Feel free to check the [video/solution in Step 3](https://www.aiml-loft.wwps.aws.dev/workshops/module-medical-document-processing-and-classification/step3/), or reach out to any of us. " 530 | ] 531 | }, 532 | { 533 | "cell_type": "markdown", 534 | "metadata": {}, 535 | "source": [ 536 | "### Challenge 2b: Update the endpoint \n", 537 | "\n", 538 | "Now, we've trained 2 models using pre-defined hyperparameter and deployed the first model. For the second model, instead of deploying a new model, can you update the endpoint with the newly built model? \n", 539 | "\n", 540 | "Hint 1: create a model with API *create_model*: https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-deploy-model.html \n", 541 | "\n", 542 | "The API that you are going to use *update_endpoint*: https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html#sagemaker.predictor.Predictor.update_endpoint\n" 543 | ] 544 | }, 545 | { 546 | "cell_type": "code", 547 | "execution_count": null, 548 | "metadata": { 549 | "scrolled": true 550 | }, 551 | "outputs": [], 552 | "source": [ 553 | "%%time\n", 554 | "\n", 555 | "###########################################################################\n", 556 | "## !! TO DO: WRITE YOUR CODE HERE TO UPDATE THE END POINT ########################\n", 557 | "###########################################################################" 558 | ] 559 | }, 560 | { 561 | "cell_type": "code", 562 | "execution_count": null, 563 | "metadata": {}, 564 | "outputs": [], 565 | "source": [ 566 | "#predictions = predict_from_numpy(predictor, test_data.drop([\"Label\"], axis=1))\n", 567 | "predictions = predict_from_numpy_V2(predictor\n", 568 | " , test_data.drop([\"Label\"], axis=1))\n", 569 | "generate_classification_report(\n", 570 | " y_real=test_data['Label'].values, \n", 571 | " y_predict_proba=predictions, \n", 572 | " decision_threshold=0.5,\n", 573 | " class_names_list=[\"Negative\", \"Positive\"],\n", 574 | " title=\"updated model\",\n", 575 | ")" 576 | ] 577 | }, 578 | { 579 | "cell_type": "markdown", 580 | "metadata": {}, 581 | "source": [ 582 | "---\n", 583 | "## Hyperparameter Optimization\n", 584 | "\n", 585 | "Now that we've trained our model using some pre-defined hyperparamters, let us try and see if we can improve the model performance by using SageMaker HyperParameter Optimization (HPO) by automating the search for an optimal hyperparameter. Specifically, we **specify a range**, or a list of possible values in the case of categorical hyperparameters, for each of the hyperparameter that we plan to tune.\n", 586 | "\n", 587 | "SageMaker hyperparameter tuning will automatically launch **multiple training jobs** with different hyperparameter settings, evaluate results of those training jobs based on a predefined \"objective metric\", and select the hyperparameter settings for future attempts based on previous results. For each hyperparameter tuning job, we will specify the maximum number of HPO tries (`max_jobs`) and how many of these can happen in parallel (`max_parallel_jobs`).\n", 588 | "\n", 589 | "Tip: `max_parallel_jobs` creates a **trade-off between performance and speed** (better hyperparameter values vs how long it takes to find these values). If `max_parallel_jobs` is large, then HPO is faster, but the discovered values may not be optimal. Smaller `max_parallel_jobs` will increase the chance of finding optimal values, but HPO will take more time to finish.\n", 590 | "\n", 591 | "Next we'll specify the objective metric that we'd like to tune and its definition, which includes the regular expression (Regex) needed to extract that metric from the CloudWatch logs of the training job. Since we are using built-in XGBoost algorithm here, it emits two predefined metrics: **validation:auc** and **train:auc**, and we elected to monitor *validation:auc* as you can see below. In this case (because it's pre-built for us), we only need to specify the metric name. For more information on parameter tuning of XGboost, please refer [here](https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html) \n", 592 | "\n", 593 | "Depending on the number of tries, HPO can find a better performing model faster, compared to simply trying different hyperparameters by trial and error or grid search. You can learn more in-depth details about SageMaker HPO [here](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-how-it-works.html).\n", 594 | "\n", 595 | "For more information on Sagemaker HPO please refer to the documentation [here](https://sagemaker.readthedocs.io/en/stable/tuner.html).\n", 596 | "\n", 597 | "**Note:** with the default settings below, the hyperparameter tuning job can take up to ~20 minutes to complete." 598 | ] 599 | }, 600 | { 601 | "cell_type": "code", 602 | "execution_count": null, 603 | "metadata": {}, 604 | "outputs": [], 605 | "source": [ 606 | "from sagemaker.tuner import IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner\n", 607 | "\n", 608 | "# set up hyperparameter ranges\n", 609 | "ranges = {\n", 610 | " \"num_round\": IntegerParameter(100, 300),\n", 611 | " \"max_depth\": IntegerParameter(1, 10),\n", 612 | " \"alpha\": ContinuousParameter(0, 5),\n", 613 | " \"eta\": ContinuousParameter(0, 1),\n", 614 | "}\n", 615 | "\n", 616 | "# set up the objective metric\n", 617 | "objective = \"validation:auc\"\n", 618 | "#objective = \"validation:accuracy\"\n", 619 | "# instantiate a HPO object\n", 620 | "tuner = HyperparameterTuner(\n", 621 | " estimator=estimator, # the SageMaker estimator object\n", 622 | " hyperparameter_ranges=ranges, # the range of hyperparameters\n", 623 | " max_jobs=10, # total number of HPO jobs\n", 624 | " max_parallel_jobs=2, # how many HPO jobs can run in parallel\n", 625 | " strategy=\"Bayesian\", # the internal optimization strategy of HPO\n", 626 | " objective_metric_name=objective, # the objective metric to be used for HPO\n", 627 | " objective_type=\"Maximize\", # maximize or minimize the objective metric\n", 628 | ") \n" 629 | ] 630 | }, 631 | { 632 | "cell_type": "code", 633 | "execution_count": null, 634 | "metadata": {}, 635 | "outputs": [], 636 | "source": [ 637 | "%%time\n", 638 | "# start HPO\n", 639 | "tuner.fit({ \"train\": s3_input_train, \"validation\": s3_input_validation })" 640 | ] 641 | }, 642 | { 643 | "cell_type": "code", 644 | "execution_count": null, 645 | "metadata": {}, 646 | "outputs": [], 647 | "source": [ 648 | "%%time\n", 649 | "# wait, until HPO is finished\n", 650 | "hpo_state = \"InProgress\"\n", 651 | "\n", 652 | "while hpo_state == \"InProgress\":\n", 653 | " hpo_state = sgmk_client.describe_hyper_parameter_tuning_job(\n", 654 | " HyperParameterTuningJobName=tuner.latest_tuning_job.job_name)[\"HyperParameterTuningJobStatus\"]\n", 655 | " print(\"-\", end=\"\")\n", 656 | " time.sleep(60) # poll once every 1 min\n", 657 | "\n", 658 | "print(\"\\nHPO state:\", hpo_state)\n", 659 | "\n" 660 | ] 661 | }, 662 | { 663 | "cell_type": "code", 664 | "execution_count": null, 665 | "metadata": { 666 | "scrolled": true 667 | }, 668 | "outputs": [], 669 | "source": [ 670 | "%%time\n", 671 | "# deploy the best model from HPO\n", 672 | "hpo_predictor = tuner.deploy(initial_instance_count=1, instance_type=\"ml.m4.xlarge\",predictor_cls=sagemaker.predictor.Predictor,\n", 673 | " serializer = sagemaker.serializers.CSVSerializer())" 674 | ] 675 | }, 676 | { 677 | "cell_type": "code", 678 | "execution_count": null, 679 | "metadata": { 680 | "scrolled": true 681 | }, 682 | "outputs": [], 683 | "source": [ 684 | "hpo_predictor.deserializer=sagemaker.deserializers.CSVDeserializer()" 685 | ] 686 | }, 687 | { 688 | "cell_type": "code", 689 | "execution_count": null, 690 | "metadata": {}, 691 | "outputs": [], 692 | "source": [ 693 | "# getting the predicted probabilities of the best model\n", 694 | "hpo_predictions = predict_from_numpy_V2(hpo_predictor, test_data.drop([\"Label\"], axis=1))\n", 695 | "print(hpo_predictions)\n", 696 | "\n", 697 | "# generate report for the best model\n", 698 | "generate_classification_report(\n", 699 | " y_real=test_data[\"Label\"].values, \n", 700 | " y_predict_proba=hpo_predictions, \n", 701 | " decision_threshold=0.5,\n", 702 | " class_names_list=[\"Consultation\",\"Surgery\"],\n", 703 | " title=\"Best model (with HPO)\",\n", 704 | ")" 705 | ] 706 | }, 707 | { 708 | "cell_type": "markdown", 709 | "metadata": {}, 710 | "source": [ 711 | "---\n", 712 | "---\n", 713 | "## Bonus Activtity\n", 714 | "\n", 715 | "### A simplified pipeline to process an Electronic Health Record\n", 716 | "Here, we will combine Textract, Comprehend Medical and SageMaker endpoint to process an electronic medical resport. " 717 | ] 718 | }, 719 | { 720 | "cell_type": "code", 721 | "execution_count": null, 722 | "metadata": {}, 723 | "outputs": [], 724 | "source": [ 725 | "from imp import reload\n", 726 | "from util.Pipeline import extractTextract, extractMedical" 727 | ] 728 | }, 729 | { 730 | "cell_type": "markdown", 731 | "metadata": {}, 732 | "source": [ 733 | "### Step 1: Extract data from Textract" 734 | ] 735 | }, 736 | { 737 | "cell_type": "code", 738 | "execution_count": null, 739 | "metadata": {}, 740 | "outputs": [], 741 | "source": [ 742 | "PDFprefix = 'sagemaker/medical_notes'\n", 743 | "fileName = 'sample_report_1.pdf'\n", 744 | "#fileUploadPath = os.path.join('./data', fileName)\n", 745 | "textractObjectName = os.path.join(PDFprefix, 'data', fileName)\n", 746 | "print(\"EHR file to be processed is at \", textractObjectName)\n", 747 | "\n", 748 | "doc=extractTextract(bucket_name,textractObjectName)" 749 | ] 750 | }, 751 | { 752 | "cell_type": "markdown", 753 | "metadata": {}, 754 | "source": [ 755 | "### Step 2: Extract data from Comprehend Medical" 756 | ] 757 | }, 758 | { 759 | "cell_type": "code", 760 | "execution_count": null, 761 | "metadata": {}, 762 | "outputs": [], 763 | "source": [ 764 | "comprehendResponse=extractMedical(doc)\n", 765 | "df_cm=extractMC_v2(comprehendResponse[0])" 766 | ] 767 | }, 768 | { 769 | "cell_type": "markdown", 770 | "metadata": {}, 771 | "source": [ 772 | "### Step 3: Organize the extracted json file into dataframe" 773 | ] 774 | }, 775 | { 776 | "cell_type": "code", 777 | "execution_count": null, 778 | "metadata": {}, 779 | "outputs": [], 780 | "source": [ 781 | "mclist, df_cm2=retrieve_mcList(df_cm, nFeature=20,threshold=0.9)\n", 782 | "df_cm2=df_mc_generator_slim(df_cm2)\n", 783 | "df_cm2" 784 | ] 785 | }, 786 | { 787 | "cell_type": "markdown", 788 | "metadata": {}, 789 | "source": [ 790 | "### Step 4: Prediction with the endpoint" 791 | ] 792 | }, 793 | { 794 | "cell_type": "code", 795 | "execution_count": null, 796 | "metadata": {}, 797 | "outputs": [], 798 | "source": [ 799 | "sample_pred = predict_from_numpy_V2(predictor, df_cm2.drop([\"ID\"], axis=1))\n", 800 | "sample_pred_hpo = predict_from_numpy_V2(hpo_predictor, df_cm2.drop([\"ID\"], axis=1))\n", 801 | "print(f\"Predicted probability for sugery for the patient is : {round(sample_pred[0],2)} from model1,\\n \\t \\t \\t \\t \\t\\t {round(sample_pred_hpo[0],2)} from model2 after HPO \")" 802 | ] 803 | }, 804 | { 805 | "cell_type": "markdown", 806 | "metadata": {}, 807 | "source": [ 808 | "---\n", 809 | "## Conclusion\n", 810 | "SageMaker built-in algorithms are great for getting a first model fast, and combining them with SageMaker HPO can really boost their accuracy. As we mentioned here, the best way to success with a built-in algorithm is to **read the [algorithm's doc pages](https://docs.aws.amazon.com/sagemaker/latest/dg/algos.html) carefully** - to understand what data format and parameters it needs!\n", 811 | "\n", 812 | "In our run, we have used a built-in algorithm XGBoost to train a classification model based on doctors' transcriptions. The first model with self-defined HyperParameters showed AUC of ~0.945, and the optimized HPO model exhibited an AUC of ~0.951: fairly higher !\n", 813 | "\n", 814 | "\n", 815 | "After that, we demonstrated a simple pipeline to process an electronic patient's record with the endpoint. " 816 | ] 817 | }, 818 | { 819 | "cell_type": "markdown", 820 | "metadata": {}, 821 | "source": [ 822 | "---\n", 823 | "## Clean up resources\n", 824 | "### Delete the endpoint and configuration if needed" 825 | ] 826 | }, 827 | { 828 | "cell_type": "code", 829 | "execution_count": null, 830 | "metadata": {}, 831 | "outputs": [], 832 | "source": [ 833 | "predictor.delete_endpoint(delete_endpoint_config=True)\n", 834 | "hpo_predictor.delete_endpoint(delete_endpoint_config=True)" 835 | ] 836 | }, 837 | { 838 | "cell_type": "markdown", 839 | "metadata": {}, 840 | "source": [ 841 | "### Delete the generated files S3 bucket files\n", 842 | "+ check your [S3 bucket](https://s3.console.aws.amazon.com/s3/home?region=ap-southeast-1) for the content information\n", 843 | "+ delete all the saved models and training jobs in the bucket, under folder */emr-mtSample*\n", 844 | "+ delete the pdf files in S3 bucket under folder */sagemaker/medical_notes*\n" 845 | ] 846 | }, 847 | { 848 | "cell_type": "code", 849 | "execution_count": null, 850 | "metadata": {}, 851 | "outputs": [], 852 | "source": [ 853 | "## Delete all the content in the emr-mtSample folder. Check S3 before deleting it\n", 854 | "s3 = boto3.resource('s3')\n", 855 | "bucket = s3.Bucket(bucket_name)\n", 856 | "bucket.objects.filter(Prefix=bucket_prefix).delete()" 857 | ] 858 | }, 859 | { 860 | "cell_type": "code", 861 | "execution_count": null, 862 | "metadata": {}, 863 | "outputs": [], 864 | "source": [ 865 | "### Delete all the content in the PDF folder \n", 866 | "\n", 867 | "bucket.objects.filter(Prefix=PDFprefix).delete()" 868 | ] 869 | }, 870 | { 871 | "cell_type": "markdown", 872 | "metadata": {}, 873 | "source": [ 874 | "### Best Practice:\n", 875 | " 1. Delete the bucket if this is the last lab of your workshop\n", 876 | " 2. Shut down your notebook instance if you are not planning to explore more labs" 877 | ] 878 | } 879 | ], 880 | "metadata": { 881 | "kernelspec": { 882 | "display_name": "conda_python3", 883 | "language": "python", 884 | "name": "conda_python3" 885 | }, 886 | "language_info": { 887 | "codemirror_mode": { 888 | "name": "ipython", 889 | "version": 3 890 | }, 891 | "file_extension": ".py", 892 | "mimetype": "text/x-python", 893 | "name": "python", 894 | "nbconvert_exporter": "python", 895 | "pygments_lexer": "ipython3", 896 | "version": "3.6.10" 897 | } 898 | }, 899 | "nbformat": 4, 900 | "nbformat_minor": 4 901 | } 902 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | ## Code of Conduct 2 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 3 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 4 | opensource-codeofconduct@amazon.com with any additional questions or comments. 5 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing Guidelines 2 | 3 | Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional 4 | documentation, we greatly value feedback and contributions from our community. 5 | 6 | Please read through this document before submitting any issues or pull requests to ensure we have all the necessary 7 | information to effectively respond to your bug report or contribution. 8 | 9 | 10 | ## Reporting Bugs/Feature Requests 11 | 12 | We welcome you to use the GitHub issue tracker to report bugs or suggest features. 13 | 14 | When filing an issue, please check existing open, or recently closed, issues to make sure somebody else hasn't already 15 | reported the issue. Please try to include as much information as you can. Details like these are incredibly useful: 16 | 17 | * A reproducible test case or series of steps 18 | * The version of our code being used 19 | * Any modifications you've made relevant to the bug 20 | * Anything unusual about your environment or deployment 21 | 22 | 23 | ## Contributing via Pull Requests 24 | Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: 25 | 26 | 1. You are working against the latest source on the *main* branch. 27 | 2. You check existing open, and recently merged, pull requests to make sure someone else hasn't addressed the problem already. 28 | 3. You open an issue to discuss any significant work - we would hate for your time to be wasted. 29 | 30 | To send us a pull request, please: 31 | 32 | 1. Fork the repository. 33 | 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change. 34 | 3. Ensure local tests pass. 35 | 4. Commit to your fork using clear commit messages. 36 | 5. Send us a pull request, answering any default questions in the pull request interface. 37 | 6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation. 38 | 39 | GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and 40 | [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). 41 | 42 | 43 | ## Finding contributions to work on 44 | Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start. 45 | 46 | 47 | ## Code of Conduct 48 | This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). 49 | For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact 50 | opensource-codeofconduct@amazon.com with any additional questions or comments. 51 | 52 | 53 | ## Security issue notifications 54 | If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. 55 | 56 | 57 | ## Licensing 58 | 59 | See the [LICENSE](LICENSE) file for our project's licensing. We will ask you to confirm the licensing of your contribution. 60 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 7 | the Software, and to permit persons to whom the Software is furnished to do so. 8 | 9 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 10 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 11 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 12 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 13 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 14 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 15 | 16 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # AWS ML Healthcare Workshop 2 | 3 | ## Introduction 4 | 5 | 6 | #### Machine learning in healthcare 7 | The advent of Machine learning is undoubtfully speeding up the medical development, such as new drug discovery and manufacturing, automating the diagnosis through computer vision, personalized treatments and improvements to organization of patients’ health records. 8 | 9 | The availability of large amount of clinical data is opening up a lot new opportunities for digital transformation in the healthcare industry. Such as patient management optimization, AI predictive diagnosis and forecasting healthcare trajectories. However, patients’ health records have long been captured and kept as hard copies. In fact, nearly 20% of the doctors still prefer to use paper medical records. Such a phenomenon have greatly affected the pace of digitalization and knowledge sharing, thus limiting the ability to realize the potential from the wealth of medical data. When medical records are digitized, doctors’ notes are often captured in free text and is usually difficult to comprehend and extract meaningful data for analysis. To address the challenges of digitalization of medical records, we have developed a workshop that will guide you to: 10 | 11 | 1) Build an medical document processing pipeline with Amazon Textract and Comprehend Medical 12 | 2) Build, train and deploy a classification machine learning model with medical data extracted from the medical document processing pipeline 13 | 14 | 15 | ## Objectives: 16 | 17 | By the end of this workshop, you would have learnt how to: 18 | 1) Build, train and deploy a classification model with medical notes using Amazon SageMaker notebook 19 | 2) Extract textual data from PDF reports using Amazon Textract 20 | 3) Extract medical data from textual doctor’s notes using Amazon Sagemaker Comprehend 21 | 4) Build a pipeline of automatic processing of medical in pdf format and predicting further treatment with Amazon Lambda, Amazon Textract, Amazon Comprehend Medical and SageMaker Endpoint 22 | 23 | The template to be deployed contains all of the codes and data you need to finish the workshop. To understand AWS ML services better, you are highly encouraged to write some of the code yourselves. The notebooks have some cells where the you are asked to complete a challenge. 24 | 25 | 26 | 27 | 28 | 29 | ## Lab Instruction 30 | 31 | [Workshop Link Placeholder] 32 | 33 | ## Deploy your Working Environment 34 | 35 | The first step is to deploy a Cloudformation template that will perform most of the initial setup for you. 36 | 37 | 1. Download the cloudformation template 38 | 39 | Go to the following URL https://raw.githubusercontent.com/dalacan/aws-ml-healthcare-workshop/master/aws-ml-healthcare-worshop.yaml, right click 'Save As' and download the cloudformation template. 40 | 41 | **Note** Make sure you save the file as a .yaml file. 42 | 43 | 2. Create a new cloud formationstack 44 | 45 | In another browser window or tab, login to your AWS account. Once you have done that, open the link below in a new tab to start the process of deploying the items you need via CloudFormation. 46 | 47 | [![Launch Stack](https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home#/stacks/new?stackName=HealthcareWorkshop) 48 | 49 | 3. Upload the cloud formation template 50 | 51 | Select `Upload a template`, click `Choose file` and select the cloudformation template file you've just downloaded and then click `Next`. 52 | 53 | ![CloudformationWizard1](static/images/step3.png) 54 | 55 | 4. Specify stack details 56 | 57 | In this section, **optionally** specify the following options: 58 | 59 | 1. Stack Name - Change the stack name to something more relevant if required. 60 | 2. Notebook Name - Change the name of your SageMaker notebook which you will be using if required. 61 | 3. Volumen Size - Set the size of SageMaker EBS volume (default is 10GB). If you expect to load a larger dataset (i.e. if you want to reuse this lab to experiment with larger dataset), increase this accordingly. 62 | 63 | When you're done, click the `Next` button at the bottom of the page 64 | ![CloudformationWizard2](static/images/step4.png) 65 | 66 | 5. Configure stack options 67 | 68 | All of the defaults in this section will be sufficient to complete the lab. If you have any custom requirements, please alter as required. Once you're done, click the `Next` button to continue. 69 | 70 | Finally, in the next section, scroll to the bottom of the page and check the checkbox to enable the template to create IAM resources and click the `Create stack` button. 71 | 72 | ![CloudformationWizard3](static/images/step5a.png) 73 | 74 | It will take a few minutes to provision the resources required for the lab. Once it is completed, navigate to the `SageMaker` service by clicking `Services` in tht top of the console and then search for `SageMaker` and click on the service. 75 | 76 | ![SageMaker](static/images/step5b.png) 77 | 78 | 6. Launch the SageMaker notebook 79 | 80 | Click on Notebook instance and open the `aws-ml-healthcare-workshop` notebook (or the name of the notebook you provided in Cloudformation) by clicking `Open JupyterLab` 81 | 82 | ![SageMaker](static/images/step6.png) 83 | 84 | 85 | ## Cleanup 86 | Once you're done with the lab, please make sure you follow the instructions at the end of the notebook to delete all the resources you created in your AWS account. Once you have done that, go to the **CloudFormation** service in the AWS console and delete the `HealthcareWorkshop` stack. 87 | 88 | --- 89 | 90 | ## References 91 | -------------------------------------------------------------------------------- /aws-ml-healthcare-worshop.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | AWSTemplateFormatVersion: '2010-09-09' 3 | 4 | Description: AWS ML Healthcare Workshop 5 | 6 | Metadata: 7 | AWS::CloudFormation::Interface: 8 | ParameterGroups: 9 | - 10 | Label: 11 | default: "Notebook Configuration" 12 | Parameters: 13 | - NotebookName 14 | - NotebookInstanceType 15 | - VolumeSize 16 | - CodeRepository 17 | 18 | Parameters: 19 | 20 | NotebookName: 21 | Type: String 22 | Default: aws-ml-healthcare-workshop 23 | Description: Enter the name of the SageMaker notebook instance. Default is aws-ml-healthcare-workshop. 24 | 25 | VolumeSize: 26 | Type: Number 27 | Default: 5 28 | MinValue: 5 29 | MaxValue: 16384 30 | ConstraintDescription: Must be an integer between 5 (GB) and 16384 (16 TB). 31 | Description: Enter the size of the EBS volume in GB. 32 | 33 | CodeRepository: 34 | Type: String 35 | Default: https://github.com/dalacan/aws-ml-healthcare-workshop.git 36 | Description: AWS ML Healthcare Workshop repository 37 | 38 | NotebookInstanceType: 39 | Type: String 40 | Default: ml.t2.medium 41 | Description: Enter the SageMaker notebook instance type. Default is ml.t2.medium 42 | 43 | Resources: 44 | # SageMaker Execution Role 45 | SageMakerIamRole: 46 | Type: "AWS::IAM::Role" 47 | Properties: 48 | AssumeRolePolicyDocument: 49 | Version: "2012-10-17" 50 | Statement: 51 | - 52 | Effect: Allow 53 | Principal: 54 | Service: sagemaker.amazonaws.com 55 | Action: sts:AssumeRole 56 | Path: "/" 57 | ManagedPolicyArns: 58 | - "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess" 59 | - "arn:aws:iam::aws:policy/AmazonTextractFullAccess" 60 | - "arn:aws:iam::aws:policy/ComprehendFullAccess" 61 | - "arn:aws:iam::aws:policy/ComprehendMedicalFullAccess" 62 | - "arn:aws:iam::aws:policy/AmazonS3FullAccess" 63 | - "arn:aws:iam::aws:policy/IAMFullAccess" 64 | 65 | # SageMaker notebook 66 | NotebookInstance: 67 | Type: "AWS::SageMaker::NotebookInstance" 68 | Properties: 69 | InstanceType: !Ref NotebookInstanceType 70 | NotebookInstanceName: !Ref NotebookName 71 | RoleArn: !GetAtt SageMakerIamRole.Arn 72 | VolumeSizeInGB: !Ref VolumeSize 73 | DefaultCodeRepository: !Ref CodeRepository 74 | -------------------------------------------------------------------------------- /data/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-textract-and-comprehend-medical-document-processing/669ef2de621c2c4d18f81405056d4bb8c9f61f69/data/.DS_Store -------------------------------------------------------------------------------- /data/.ipynb_checkpoints/sample_report_1-checkpoint.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-textract-and-comprehend-medical-document-processing/669ef2de621c2c4d18f81405056d4bb8c9f61f69/data/.ipynb_checkpoints/sample_report_1-checkpoint.pdf -------------------------------------------------------------------------------- /data/processed_combined_wide.csv: -------------------------------------------------------------------------------- 1 | ID,Label,nontender,foreign body,edema,alert,murmur,chest pain,vomiting,hiatal hernia,distress,hemostasis,carpal tunnel syndrome,endometriosis,weakness,pain,mass,inflammation,polyp,bleeding,hypertension,supple,fever,stenosis,wound,cyanosis,infection,erythema,normocephalic,fracture,lesion,ulceration,nausea,cough,tumor,soft,shortness of breath,injury,diabetes 2 | 337,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8745948076248169,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8758465051651001,0.0,0.0,0.4131222069263458,0.0 3 | 1054,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7162202596664429,0.0,0.0,0.0,0.968464732170105,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9141705632209778,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 4 | 225,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9799316525459291,0.0,0.0,0.0,0.0,0.0,0.0,0.9937908053398132,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 5 | 1086,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6410290598869324,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9292075634002686,0.0,0.0,0.0,0.0,0.0,0.776443362236023,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 6 | 304,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4490427970886231,0.0,0.9831297993659972,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9947522878646852,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 7 | 1070,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9578970074653624,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 8 | 563,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4033505022525787,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9885196089744568,0.0,0.0,0.0,0.0,0.7519121170043945,0.0,0.9843950271606444,0.0,0.0,0.0,0.9306575059890748,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 9 | 1080,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.43729513883590704,0.986423671245575,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7442220449447632,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 10 | 286,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6366420388221741,0.0,0.0,0.0,0.8197255134582521,0.6542521715164185,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 11 | 227,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6355668306350708,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 12 | 967,1,0.0,0.0,0.0,0.0,0.0,0.8920240998268127,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7061194181442261,0.0,0.0,0.0,0.0,0.0,0.0,0.8148785829544067,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 13 | 659,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6915370225906372,0.0,0.0,0.9772849082946776,0.9414882063865662,0.0,0.0,0.0,0.9424986243247986,0.0,0.0,0.0,0.0,0.6075635552406311,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 14 | 332,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8563896417617798,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 15 | 582,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7150526642799377,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7420989871025085,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 16 | 1022,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 17 | 1031,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8804498314857483,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 18 | 433,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 19 | 489,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8218086957931519,0.0,0.0,0.0,0.3773014545440674,0.0,0.0,0.0,0.0,0.8433253765106201,0.0,0.0,0.0,0.0,0.9079804420471193,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 20 | 1048,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9958113431930542,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 21 | 268,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9170490503311156,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 22 | 406,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5055738687515259,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9212989807128906,0.0,0.0,0.0,0.0,0.0,0.0,0.9330489039421082,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 23 | 1265,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5412341356277466,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 24 | 597,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6685250401496887,0.0,0.0,0.0,0.99830961227417,0.7714146971702576,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.910359025001526,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 25 | 613,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9836032390594482,0.0,0.0,0.0,0.0,0.979081392288208,0.0,0.0,0.0,0.0,0.9101989865303041,0.0,0.9968721270561218,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.841008722782135,0.0,0.0,0.9660457372665404,0.0 26 | 908,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 27 | 404,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8244494795799255,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.588036298751831,0.0,0.0,0.0,0.0,0.9511459469795228,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4801688194274902,0.0 28 | 745,1,0.0,0.0,0.0,0.0,0.0,0.4893645644187927,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 29 | 256,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.36466532945632935,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 30 | 888,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.592643678188324,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 31 | 1172,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8678246140480042,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 32 | 615,1,0.0,0.0,0.0,0.0,0.0,0.0,0.9985941052436828,0.0,0.0,0.8068608045578003,0.0,0.978669285774231,0.0,0.3727966248989105,0.0,0.0,0.0,0.0,0.0,0.0,0.9912802577018738,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9991158843040466,0.0,0.0,0.0,0.0,0.8802235722541809,0.0 33 | 1094,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6539973616600037,0.9737154245376588,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 34 | 880,1,0.0,0.0,0.9882240295410156,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9891341328620912,0.9707226753234864,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9976710677146912,0.0,0.0,0.0,0.9848970174789428,0.0,0.0,0.0,0.0 35 | 1213,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4938248991966248,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 36 | 420,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 37 | 219,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5847300887107849,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 38 | 1050,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 39 | 440,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7411140203475952,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 40 | 1212,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9384378790855408,0.7618566155433655,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9672259092330932,0.0 41 | 1063,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6435815691947937,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7111477851867676,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 42 | 326,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 43 | 1235,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9065274596214294,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4664733707904816,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 44 | 878,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7200319170951843,0.9388562440872192,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 45 | 1128,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7159039378166199,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 46 | 610,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9976759552955629,0.0,0.0,0.0,0.9735616445541382,0.0,0.0,0.0,0.0,0.0,0.0,0.9965289235115052,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 47 | 368,1,0.0,0.0,0.0,0.0,0.0,0.0,0.9967574477195741,0.0,0.0,0.9153442978858948,0.0,0.0,0.0,0.6943514347076416,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9962782263755798,0.0,0.0,0.0,0.0,0.0,0.0 48 | 279,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9570212364196776,0.0,0.0,0.0,0.9870920181274414,0.0,0.0,0.0,0.0,0.6245230436325073,0.0,0.9949458241462708,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 49 | 482,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6424610018730164,0.0,0.0,0.0,0.0,0.9399652481079102,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 50 | 903,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.972706139087677,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 51 | 525,1,0.0,0.0,0.9829005599021912,0.0,0.0,0.0,0.0,0.0,0.0,0.5684897899627686,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8853998184204102,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 52 | 297,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9593913555145264,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9894776940345764,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 53 | 658,1,0.0,0.0,0.0,0.8115906119346619,0.0,0.0,0.0,0.0,0.9962276220321656,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 54 | 1226,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.3633253276348114,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8638107776641846,0.0,0.0,0.0,0.0,0.9272415637969972,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 55 | 507,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9927356243133544,0.0,0.0,0.0,0.9780617952346802,0.0,0.0,0.0,0.0,0.9379663467407228,0.0,0.9886142015457152,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 56 | 334,1,0.9850618243217468,0.0,0.0,0.0,0.0,0.0,0.9898471236228944,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9989891648292542,0.0,0.0,0.0,0.9902225732803344,0.9889901876449584,0.0,0.0,0.0,0.0,0.9051790237426758,0.0,0.9967690706253052,0.0,0.9987865090370178,0.0,0.0,0.0,0.0,0.9865716695785522,0.0,0.0,0.0 57 | 866,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9850348234176636,0.0,0.0,0.0,0.994628131389618,0.0,0.0,0.0,0.0,0.8224507570266724,0.0,0.9882630705833436,0.0,0.0,0.9732576608657836,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 58 | 940,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 59 | 259,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7682535648345947,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6585469841957092,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 60 | 1013,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8668486475944519,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9660758972167968,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 61 | 715,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.982990026473999,0.9577236771583556,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9952778816223145,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 62 | 887,1,0.0,0.0,0.0,0.0,0.0,0.9430243372917176,0.0,0.0,0.0,0.0,0.0,0.0,0.9897408485412598,0.9980797767639159,0.0,0.0,0.0,0.9857511520385742,0.0,0.0,0.9977450966835022,0.0,0.0,0.0,0.9679391384124756,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9846011400222778,0.0,0.0 63 | 857,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 64 | 1135,1,0.0,0.0,0.0,0.0,0.0,0.8349402546882629,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9984821677207948,0.7118296027183533,0.0,0.9961885809898376,0.0,0.0 65 | 780,1,0.0,0.7713832259178162,0.9971152544021606,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9950783252716064,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9070927500724792,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 66 | 1160,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4236750602722168,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 67 | 414,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7017966508865356,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 68 | 272,1,0.0,0.7514050602912903,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8509963750839233,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 69 | 258,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6676121950149536,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.31265226006507874,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 70 | 238,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4153994917869568,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8166817426681519,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 71 | 759,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 72 | 487,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8263570070266724,0.0,0.0,0.0,0.0,0.9272395968437196,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 73 | 825,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.907734215259552,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9891920685768129,0.0,0.0,0.0,0.7313584685325623,0.0,0.0,0.0,0.0,0.0,0.0,0.0 74 | 699,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 75 | 592,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 76 | 1231,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8322871327400208,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.685095489025116,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 77 | 378,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7659980058670044,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 78 | 1125,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7386569380760193,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 79 | 452,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9960349202156068,0.0,0.0,0.0,0.0,0.0,0.0,0.9964151382446288,0.9883686304092408,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 80 | 976,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9724149107933044,0.0,0.9686858057975768,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 81 | 631,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9964744448661804,0.624212920665741,0.0,0.0,0.971465528011322,0.0,0.0,0.0,0.0,0.0,0.0,0.9948603510856628,0.0,0.0,0.0,0.8689402937889099,0.0,0.0,0.0,0.0,0.0,0.0,0.551917314529419,0.0 82 | 359,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6553434133529663,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 83 | 853,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 84 | 827,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8009155988693237,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7222871780395508,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9566400051116944,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 85 | 379,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7568726539611816,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.986170530319214,0.9995544552803041,0.0,0.0,0.0,0.0,0.0,0.9949941039085388,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9573240876197816,0.0 86 | 1069,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7997692227363586,0.0,0.0,0.0,0.0,0.8870148658752441,0.0,0.987263023853302,0.0,0.0,0.0,0.9274089932441713,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 87 | 277,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.39254650473594666,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8180550932884216,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 88 | 1110,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5389034152030945,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 89 | 586,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6834443211555481,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 90 | 1152,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9976882934570312,0.0,0.0,0.0,0.9599367380142212,0.0,0.0,0.0,0.0,0.0,0.0,0.9962337613105774,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 91 | 1035,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 92 | 372,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.921350121498108,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8123026490211487,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 93 | 212,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.645904541015625,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 94 | 426,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6963062882423401,0.0,0.9527121782302856,0.0,0.0,0.9322569370269777,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 95 | 977,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8631901741027832,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8156597018241882,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 96 | 694,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6886130571365356,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 97 | 213,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 98 | 1024,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9674212336540222,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 99 | 672,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6054945588111877,0.0,0.0,0.0,0.0,0.7361539006233215,0.0,0.9908555746078492,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9653146862983704 100 | 930,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 101 | 552,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4537114202976227,0.0,0.0,0.0,0.0,0.3788434565067291,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7679307460784912,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7610792517662048,0.0,0.0,0.0,0.0 102 | 311,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6795103549957275,0.7658042311668396,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9510716795921326,0.0,0.0,0.0,0.0 103 | 369,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7511634230613708,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9544965028762816,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 104 | 1218,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6088843941688538,0.0,0.9819955825805664,0.0,0.0,0.0,0.2600046992301941,0.7987059354782104,0.0,0.0,0.0,0.0,0.0,0.0,0.0 105 | 298,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6406288146972656,0.0,0.0,0.0,0.5713314414024353,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 106 | 382,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9712303876876832,0.0,0.0,0.0,0.0,0.0,0.0,0.99128657579422,0.0,0.0,0.0,0.9803054332733154,0.0,0.0,0.0,0.8099786043167114,0.0,0.0,0.8147233724594116,0.0 107 | 1174,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7286914587020874,0.0,0.9466617107391356,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 108 | 709,1,0.0,0.6322572231292725,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7159754633903503,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9671344757080078,0.0,0.0,0.9348090291023254,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 109 | 385,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5393296480178833,0.0,0.0,0.0,0.9916229844093324,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8881240487098694,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 110 | 1167,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6414190530776978,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8982887268066406,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 111 | 561,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5853676795959473,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 112 | 899,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9384775757789612,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 113 | 828,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 114 | 1003,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9723733067512512,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 115 | 383,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9889256358146667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8157887458801271,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 116 | 1046,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.34728220105171204,0.0,0.0,0.0,0.9828314781188964,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9926464557647704,0.0,0.0,0.0,0.0,0.0,0.0 117 | 1247,1,0.0,0.0,0.0,0.7992490530014038,0.0,0.0,0.9988316893577576,0.0,0.9939678907394408,0.7221013307571411,0.0,0.0,0.8302438855171204,0.0,0.0,0.0,0.0,0.8808948993682861,0.0,0.0,0.0,0.8410378098487854,0.6854658722877502,0.0,0.9821920990943908,0.0,0.0,0.0,0.0,0.0,0.999215841293335,0.0,0.0,0.0,0.0,0.8708691596984863,0.0 118 | 1252,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7952470779418945,0.7380697131156921,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 119 | 184,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6056357622146606,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 120 | 528,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5939322113990784,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 121 | 633,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 122 | 243,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 123 | 877,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7964003682136536,0.0,0.0,0.0,0.0,0.0,0.0,0.9906347393989564,0.9808968305587769,0.9636537432670592,0.984763205051422,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9695960879325868,0.9831103086471558,0.0,0.0,0.0,0.0,0.0,0.0,0.0 124 | 394,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8452602028846741,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9556735157966614,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5344921350479126,0.0 125 | 479,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.980017066001892,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6581085920333862,0.0,0.0,0.0,0.0,0.0,0.9736681580543518,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 126 | 807,1,0.0,0.0,0.0,0.8837515115737915,0.0,0.0,0.9984610080718994,0.0,0.9906942844390868,0.932692050933838,0.0,0.0,0.8977546691894531,0.0,0.0,0.0,0.0,0.9811757206916808,0.0,0.0,0.0,0.9329255223274232,0.816145122051239,0.0,0.8864151835441589,0.0,0.0,0.0,0.0,0.0,0.9984281063079834,0.0,0.0,0.0,0.0,0.0,0.0 127 | 606,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 128 | 783,1,0.0,0.7847884297370911,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.969343900680542,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 129 | 957,1,0.0,0.3970134258270264,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9320454001426696,0.0,0.0,0.0,0.0,0.301692396402359,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 130 | 730,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5487715005874634,0.0,0.0,0.0,0.9876234531402588,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 131 | 1164,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.558883011341095,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 132 | 842,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9513232111930848,0.0,0.0,0.0,0.0 133 | 986,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 134 | 660,1,0.0,0.42039233446121216,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5093064904212952,0.0,0.0,0.0,0.0,0.852265477180481,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 135 | 436,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5797843933105469,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 136 | 991,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9864315390586852,0.9536970853805542,0.9835890531539916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9648069739341736,0.0,0.0,0.96013343334198,0.0,0.0,0.0,0.0 137 | 343,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9977195858955384,0.0,0.0,0.0,0.9671587347984314,0.0,0.0,0.0,0.0,0.0,0.0,0.9968480467796326,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 138 | 642,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5290694236755371,0.0,0.0,0.0,0.0,0.0,0.8643751740455627,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 139 | 266,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5260283350944519,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9219139814376832,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 140 | 282,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9929308891296388,0.0,0.0,0.9920086860656738,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.993979811668396,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 141 | 1201,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9872373342514038,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 142 | 882,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 143 | 926,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8086258769035339,0.0,0.9854555726051332,0.0,0.0,0.5426726937294006,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 144 | 358,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8370157480239868,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 145 | 1026,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6206848621368408,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9837430119514464,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 146 | 914,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 147 | 488,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9476606845855712,0.0,0.0,0.0,0.0,0.8722436428070068,0.0,0.0,0.0,0.0,0.8160474300384521,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 148 | 564,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7362957000732422,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 149 | 771,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 150 | 317,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7410116791725159,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 151 | 1165,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5601537823677063,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8633912205696106,0.0,0.0,0.0,0.0,0.0,0.0,0.9915961623191832,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 152 | 810,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6135095357894897,0.0,0.0,0.0,0.0,0.6148456931114197,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6314182281494141,0.0 153 | 630,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9573026895523072,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 154 | 919,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9694586992263794,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 155 | 973,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9396433830261229,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 156 | 641,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6514409184455872,0.0,0.0,0.0,0.0,0.987722098827362,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 157 | 1134,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 158 | 518,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9743205904960632,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 159 | 896,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4956449270248413,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 160 | 982,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.958996057510376,0.7326705455780029,0.9881736636161804,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 161 | 476,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7114882469177246,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8355204463005066,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 162 | 765,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8658614754676819,0.0,0.0,0.0,0.0,0.0,0.9047607779502868,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 163 | 1060,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 164 | 532,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 165 | 1068,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9775301814079284,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5925321578979492,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 166 | 838,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8880597352981567,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9720067977905272,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 167 | 855,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.978114366531372,0.0,0.0,0.0,0.0,0.0,0.0,0.9894312620162964,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 168 | 760,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6805782318115234,0.0,0.0,0.0,0.9411664605140686,0.0,0.0,0.0,0.8241273164749146,0.0,0.0,0.0,0.0,0.6271464228630066,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9798712134361268 169 | 1266,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4222136437892914,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9891144037246704,0.0,0.0,0.0,0.0,0.0,0.0,0.9844073057174684,0.0,0.0,0.0,0.8648215532302856,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 170 | 1004,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 171 | 1148,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7228710055351257,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.917699933052063,0.0,0.0,0.0,0.0,0.0,0.8156079649925232,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 172 | 195,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4453928768634796,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 173 | 197,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9868359565734864,0.0,0.0,0.0,0.7847900390625,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 174 | 1108,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9625878930091858,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 175 | 498,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9072924256324768,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8956637978553772,0.0 176 | 492,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8521896600723267,0.0,0.0,0.0,0.988835632801056,0.0,0.0,0.0,0.9869151711463928,0.0,0.0,0.0,0.0,0.7309504151344299,0.0,0.9685614705085754,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 177 | 1178,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9039342999458312,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.91375070810318,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 178 | 912,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9677367210388184,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 179 | 980,1,0.0,0.0,0.9967323541641236,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9784138202667236,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 180 | 637,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6711181402206421,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 181 | 1077,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9873260855674744,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6245903968811035,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 182 | 527,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9194707274436952,0.0,0.0,0.0,0.0,0.7440087795257568,0.0,0.0,0.0,0.0,0.0,0.984128713607788,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 183 | 669,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 184 | 1185,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7683551907539368,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9338914752006532,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 185 | 755,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4241640269756317,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8668075203895569,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 186 | 608,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 187 | 841,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9544718861579896,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7827689647674559,0.9733846187591552,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9251095056533812,0.0,0.0,0.0,0.0,0.0,0.0,0.0 188 | 829,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.964185118675232,0.9378972053527832,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 189 | 1171,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 190 | 546,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8256381154060364,0.0,0.0,0.0,0.0,0.8894447684288025,0.0,0.0,0.9149484634399414,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5731514692306519,0.0,0.0,0.0,0.0 191 | 437,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9945782423019408,0.0,0.0,0.0,0.9972156286239624,0.0,0.0,0.0,0.0,0.8147572875022888,0.0,0.9983288645744324,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 192 | 685,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8494566082954407,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 193 | 514,1,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9770126342773438,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 194 | 4081,0,0.9975377321243286,0.0,0.9995926022529602,0.9173840880393982,0.996932864189148,0.0,0.0,0.0,0.9991676807403564,0.0,0.0,0.0,0.0,0.0,0.9882987737655641,0.0,0.0,0.0,0.9982446432113647,0.9964824914932252,0.9814311265945436,0.0,0.0,0.9996500015258788,0.0,0.0,0.9987105131149292,0.0,0.9984992742538452,0.0,0.0,0.0,0.0,0.9712203741073608,0.0,0.0,0.0 195 | 4086,0,0.996911883354187,0.0,0.9995729327201844,0.761818528175354,0.9968563318252563,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9937593936920166,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9996343851089478,0.0,0.0,0.998519241809845,0.0,0.9984831213951112,0.0,0.0,0.0,0.0,0.9912818670272828,0.0,0.0,0.0 196 | 4087,0,0.9979105591773988,0.0,0.0,0.912973940372467,0.0,0.0,0.0,0.0,0.9988214373588562,0.0,0.0,0.0,0.0,0.0,0.9986101388931274,0.0,0.0,0.0,0.0,0.9901066422462464,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9990136623382568,0.0,0.0,0.0,0.0,0.9952998161315918,0.0,0.0,0.0 197 | 4090,0,0.9964461922645568,0.0,0.0,0.9655802845954896,0.9971549510955809,0.0,0.9973838925361632,0.0,0.9992207288742064,0.0,0.0,0.0,0.0,0.0,0.9850580096244812,0.0,0.0,0.0,0.0,0.9955759048461914,0.0,0.0,0.0,0.0,0.0,0.0,0.9979954957962036,0.0,0.0,0.0,0.0,0.9976108074188232,0.0,0.993071973323822,0.0,0.0,0.0 198 | 4092,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 199 | 4093,0,0.0,0.0,0.998824179172516,0.0,0.0,0.0,0.9987974166870116,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9988490343093872,0.0,0.0,0.0,0.0,0.0,0.9949839115142822,0.0,0.0,0.9993589520454408,0.0,0.9984822869300842,0.9959768652915956,0.0,0.0,0.0,0.9880940318107604,0.99839848279953,0.0,0.918362319469452,0.0,0.0,0.0 200 | 4094,0,0.0,0.0,0.9957126379013062,0.9622663855552672,0.0,0.9390177726745604,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9992745518684388,0.9913968443870544,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 201 | 4096,0,0.0,0.0,0.9993796348571776,0.8813627362251282,0.0,0.9524356126785278,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9973111152648926,0.0,0.0,0.0,0.9964665174484252,0.9963816404342652,0.9962486624717712,0.0,0.0,0.9997425675392152,0.0,0.0,0.9924383759498596,0.0,0.0,0.0,0.0,0.9841222763061525,0.0,0.0,0.9933159947395324,0.0,0.9893783926963806 202 | 4101,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 203 | 4102,0,0.0,0.0,0.0,0.0,0.0,0.9255284667015076,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9978094696998596,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 204 | 4105,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9993174076080322,0.0,0.0,0.0,0.0,0.0,0.9953811168670654,0.0,0.0,0.0,0.9967544674873352,0.0,0.0,0.0,0.0,0.0,0.9822415709495544,0.0,0.9973675608634948,0.0,0.9973052740097046,0.0,0.0,0.0,0.0,0.0,0.9790357947349548,0.0,0.0 205 | 4107,0,0.0,0.0,0.9992658495903016,0.0,0.0,0.0,0.998612642288208,0.0,0.9956544637680054,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9889287352561952,0.0,0.0,0.9996840953826904,0.0,0.0,0.0,0.0,0.0,0.0,0.9983336329460144,0.0,0.0,0.0,0.0,0.0,0.0 206 | 4112,0,0.0,0.0,0.973042905330658,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9948298335075378,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 207 | 4114,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9970685839653016,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9866504669189452 208 | 4115,0,0.0,0.0,0.9979623556137084,0.0,0.0,0.0,0.0,0.0,0.9931957721710204,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9947441816329956,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9678574204444884,0.0,0.0,0.0 209 | 4117,0,0.0,0.0,0.0,0.0,0.9969307780265808,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9933167695999146,0.0,0.0,0.0,0.0,0.9949363470077516,0.0,0.0,0.0,0.0,0.0,0.0,0.994933307170868,0.0,0.0,0.0,0.0,0.0,0.0,0.9857051372528076,0.0,0.0,0.0 210 | 4119,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9986244440078736,0.0,0.0,0.0,0.0,0.0,0.9889739751815796,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9997420907020568,0.0,0.0,0.994961142539978,0.0,0.0,0.0,0.0,0.0,0.0,0.8703204393386841,0.0,0.0,0.0 211 | 4122,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9839690923690796,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 212 | 4123,0,0.997327208518982,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9988522529602052,0.0,0.0,0.0,0.0,0.0,0.9985517859458924,0.0,0.0,0.0,0.0,0.988627016544342,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9990262985229492,0.0,0.0,0.0,0.0,0.9914544820785522,0.0,0.0,0.0 213 | 4129,0,0.9965695142745972,0.0,0.9897594451904296,0.9477128386497498,0.0,0.0,0.0,0.0,0.9977645874023438,0.0,0.0,0.0,0.0,0.9873762130737304,0.0,0.0,0.0,0.0,0.0,0.9937375783920288,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9989485144615172,0.0,0.0,0.0,0.0,0.9816973805427552,0.9930375218391418,0.0,0.0 214 | 4130,0,0.9925794005393982,0.0,0.0,0.719750702381134,0.0,0.9728755950927734,0.0,0.0,0.995936155319214,0.0,0.0,0.0,0.0,0.6494130492210388,0.0,0.0,0.0,0.0,0.9987403750419616,0.0,0.9982383251190186,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.998881995677948,0.0,0.9935548901557922,0.0,0.0,0.9804256558418274 215 | 4133,0,0.0,0.0,0.0,0.0,0.996896743774414,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9923175573349,0.0,0.0,0.0,0.0,0.9945053458213806,0.0,0.0,0.0,0.0,0.0,0.0,0.9950336217880248,0.0,0.0,0.0,0.0,0.0,0.0,0.9856303334236144,0.0,0.0,0.0 216 | 4135,0,0.9979060888290404,0.0,0.0,0.5505645871162415,0.0,0.967944085597992,0.0,0.0,0.9985733032226562,0.0,0.0,0.0,0.0,0.0,0.9891583919525146,0.0,0.0,0.0,0.9977237582206726,0.9961596727371216,0.0,0.0,0.0,0.999703586101532,0.0,0.0,0.998040735721588,0.0,0.0,0.0,0.0,0.0,0.0,0.9955887794494628,0.0,0.0,0.976332187652588 217 | 4136,0,0.0,0.0,0.0,0.0,0.0,0.0,0.99865061044693,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9983475208282472,0.0,0.0,0.0,0.0,0.0,0.0 218 | 4140,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9959514141082764,0.0,0.0,0.0,0.9830081462860109,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9899152517318726,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 219 | 4147,0,0.99766206741333,0.0,0.9990272521972656,0.9237180948257446,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9991987347602844,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9997529387474059,0.0,0.0,0.9954126477241516,0.0,0.0,0.0,0.0,0.8869203329086304,0.0,0.9873859882354736,0.0,0.0,0.9615063071250916 220 | 4153,0,0.0,0.0,0.9981160163879396,0.7091554403305054,0.0,0.0,0.0,0.0,0.9979654550552368,0.0,0.0,0.0,0.0,0.0,0.9992802739143372,0.0,0.9657157659530641,0.9603884220123292,0.0,0.9934669137001038,0.9963985681533812,0.0,0.0,0.9994099140167236,0.9775184988975524,0.0,0.9988096952438354,0.0,0.0,0.0,0.0,0.0,0.0,0.9851179122924804,0.0,0.0,0.0 221 | 4156,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9867375493049622,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 222 | 4160,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 223 | 4163,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 224 | 4166,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9948866963386536,0.0,0.0,0.0,0.0,0.0,0.9087240695953368,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 225 | 4172,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9959303736686708,0.0,0.0,0.0,0.0,0.0 226 | 4174,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9509610533714294,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.996738851070404,0.0,0.0,0.0,0.0,0.0,0.0 227 | 4176,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 228 | 4182,0,0.9973229765892028,0.0,0.0,0.0,0.964098572731018,0.0,0.9989374279975892,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9897902607917786,0.0,0.99826979637146,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9985378980636596,0.9979990124702454,0.0,0.9848083257675172,0.9943242073059082,0.0,0.8462409377098083 229 | 4183,0,0.0,0.0,0.0,0.0,0.9805865287780762,0.9605437517166138,0.9981583952903748,0.0,0.9978018403053284,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9769617915153505,0.0,0.9858304858207704,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9976884126663208,0.9843824505805968,0.0,0.0,0.9957544803619384,0.0,0.9875617027282716 230 | 4185,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9990817308425904,0.0,0.0,0.0,0.0,0.0,0.9516971707344056,0.0,0.0,0.0,0.9594281911849976,0.9982374906539916,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8296383619308472,0.0,0.0,0.0 231 | 4188,0,0.9965291619300842,0.0,0.0,0.9170680642127992,0.9974811673164368,0.0,0.0,0.0,0.9985734224319458,0.0,0.0,0.0,0.0,0.9830499291419984,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9463927745819092,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9938610792160034,0.0,0.0,0.0 232 | 4195,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9714592695236206,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8560107350349426,0.0 233 | 4201,0,0.0,0.0,0.0,0.0,0.9825565218925476,0.0,0.9960397481918336,0.0,0.9973822236061096,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9917912483215332,0.0,0.993216335773468,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9973164200782776,0.982381284236908,0.9829874038696288,0.0,0.995880961418152,0.0,0.0 234 | 4202,0,0.0,0.0,0.0,0.680209219455719,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9583544731140136,0.9766407012939452,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 235 | 4206,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9314308762550354,0.0,0.0,0.9724978804588318,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8841766119003296,0.0 236 | 4212,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 237 | 4213,0,0.0,0.0,0.0,0.0,0.0,0.987121045589447,0.0,0.9865525364875792,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9986900687217712,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9960059523582458,0.0,0.0 238 | 4215,0,0.9956285953521729,0.0,0.992440938949585,0.8844022154808044,0.9991160035133362,0.9638156294822692,0.9991457462310792,0.0,0.9988362193107604,0.0,0.0,0.0,0.9979386925697328,0.9711942672729492,0.99689781665802,0.0,0.0,0.0,0.999439299106598,0.9944912195205688,0.9974259734153748,0.0,0.0,0.0,0.9917857050895692,0.0,0.9976389408111572,0.0,0.9992665648460388,0.0,0.9980251789093018,0.9941065311431884,0.0,0.9941495656967164,0.977038025856018,0.0,0.0 239 | 4219,0,0.0,0.0,0.0,0.0,0.0,0.9777833819389344,0.9994552731513976,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9994033575057985,0.9978675842285156,0.0,0.0,0.9965215921401978,0.0,0.0 240 | 4223,0,0.9942445158958436,0.0,0.9986151456832886,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9978450536727904,0.9850652813911438,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9995983242988586,0.0,0.0,0.9923290610313416,0.0,0.9976062774658204,0.0,0.0,0.0,0.0,0.9889013767242432,0.0,0.0,0.0 241 | 4228,0,0.0,0.0,0.9983537197113036,0.9287968277931212,0.0,0.0,0.0,0.0,0.9992343187332152,0.0,0.0,0.0,0.0,0.95391845703125,0.9972745776176452,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9996213912963868,0.0,0.0,0.0,0.0,0.9993649125099182,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 242 | 4229,0,0.7158084511756897,0.0,0.0,0.9805477857589722,0.9993176460266112,0.0,0.0,0.0,0.9992637038230896,0.0,0.0,0.0,0.0,0.9689627289772034,0.9973575472831726,0.0,0.0,0.0,0.0,0.9791105389595032,0.0,0.0,0.9871448874473572,0.0,0.0,0.9988636970520021,0.0,0.0,0.9976293444633484,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 243 | 4231,0,0.0,0.0,0.0,0.0,0.0,0.0,0.9993140697479248,0.0,0.0,0.0,0.0,0.0,0.9986095428466796,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9991563558578492,0.9971206188201904,0.0,0.0,0.0,0.0,0.0 244 | 4232,0,0.0,0.0,0.0,0.0,0.0,0.0,0.9981131553649902,0.0,0.0,0.0,0.0,0.0,0.0,0.996417760848999,0.0,0.0,0.0,0.9912559986114502,0.0,0.0,0.9934222102165222,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9904279708862304,0.0,0.0,0.995682954788208,0.0,0.924874484539032 245 | 4236,0,0.9937551021575928,0.0,0.9979384541511536,0.9701701402664183,0.9878050088882446,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9896711707115172,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9963739514350892,0.0,0.0,0.0,0.0,0.9813823699951172,0.0,0.0,0.0 246 | 4237,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.969148814678192,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9933428168296814,0.9972305893898009,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 247 | 4240,0,0.9950256943702698,0.946503221988678,0.9988952875137328,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8191061615943909,0.998454451560974,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9989014863967896,0.0,0.0,0.9971246123313904,0.0,0.0,0.0,0.0,0.9830873012542723,0.0,0.0,0.0 248 | 4241,0,0.9930307269096376,0.0,0.9995749592781068,0.9912289381027222,0.9971497654914856,0.0,0.0,0.0,0.9988958835601808,0.0,0.0,0.0,0.0,0.0,0.9972628355026244,0.0,0.0,0.9730979204177856,0.0,0.0,0.0,0.0,0.0,0.9996581077575684,0.7467968463897705,0.9960126876831056,0.9983372688293456,0.0,0.99592387676239,0.0,0.0,0.0,0.0,0.8721677660942078,0.0,0.0,0.0 249 | 4243,0,0.9939910769462584,0.0,0.9977527260780334,0.9712350964546204,0.9870759844779968,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9651142358779908,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9880256056785583,0.0,0.0,0.0,0.0,0.9809298515319824,0.0,0.0,0.0 250 | 4245,0,0.9946309328079224,0.0,0.991657018661499,0.9699670672416688,0.9947091341018676,0.0,0.0,0.0,0.999414563179016,0.0,0.0,0.0,0.0,0.0,0.9940811991691588,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9993444085121156,0.0,0.0,0.9738805890083312,0.0,0.0,0.0,0.0,0.0,0.0,0.9926922917366028,0.0,0.0,0.0 251 | 4246,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9958093166351318,0.9776210188865662,0.0,0.9986131191253662,0.0,0.0,0.0,0.0,0.0,0.0,0.9901976585388184,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 252 | 4251,0,0.0,0.0,0.0,0.8343344330787659,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9926687479019164 253 | 4252,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9993866682052612,0.0,0.0,0.0,0.0,0.8264229893684387,0.9921904206275941,0.972056210041046,0.0,0.0,0.0,0.0,0.0,0.9603761434555054,0.0,0.0,0.0,0.0,0.9983636736869812,0.0,0.0,0.0,0.0,0.0,0.0,0.9609065055847168,0.0,0.0,0.9936732053756714 254 | 4253,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9866292476654052,0.9782627820968628,0.0,0.0,0.0,0.0,0.9992448091506958,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9946168065071106 255 | 4256,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9931302666664124,0.0,0.0,0.0,0.0,0.9980227947235109,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9902267456054688 256 | 4258,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9928184747695924,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 257 | 4259,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9923783540725708,0.0,0.0,0.0,0.0,0.0,0.9981923699378968,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9915818572044371,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.992203950881958 258 | 4260,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9974121451377868,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9987001419067384,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 259 | 4262,0,0.0,0.0,0.0,0.0,0.0,0.8738419413566589,0.9957001209259032,0.9462274312973022,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9947698712348938,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9959447979927064,0.0,0.9670356512069702,0.0,0.0,0.0,0.0 260 | 4263,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9877482652664183,0.0,0.0,0.0,0.0,0.0,0.9977009892463684,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9709103107452391 261 | 4265,0,0.9968116879463196,0.0,0.9989792108535768,0.8028582930564879,0.8047490119934082,0.0,0.99881774187088,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9964373111724854,0.9874401092529296,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9987642765045166,0.0,0.9907336235046388,0.0,0.0,0.0 262 | 4266,0,0.9970412850379944,0.0,0.0,0.0,0.0,0.0,0.998435080051422,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9939903020858764,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9988811612129213,0.0,0.0,0.0,0.0,0.9975529313087464,0.0,0.0,0.9953635931015016,0.0,0.0,0.0 263 | 4269,0,0.9965791106224059,0.0,0.998840034008026,0.0,0.9950608611106871,0.9534841775894164,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9970932006835938,0.0,0.0,0.0,0.0,0.9993351101875304,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9942541718482972,0.0,0.0,0.0 264 | 4270,0,0.0,0.0,0.9881895780563354,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9932520985603333,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 265 | 4271,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.997334361076355,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 266 | 4272,0,0.9954460859298706,0.0,0.9995433688163756,0.0,0.0,0.929330348968506,0.0,0.0,0.0,0.0,0.0,0.0,0.9929158687591552,0.968636155128479,0.9917322397232056,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9995263814926147,0.0,0.0,0.9916986227035522,0.0,0.0,0.0,0.0,0.9978320002555848,0.0,0.0,0.9939175248146056,0.0,0.0 267 | 4273,0,0.99809068441391,0.0,0.0,0.0,0.0,0.9273728728294371,0.9996218681335448,0.0,0.9993270635604858,0.0,0.0,0.0,0.0,0.9921431541442872,0.9972265362739564,0.0,0.0,0.0,0.9940224289894104,0.9968931674957277,0.9969063401222228,0.0,0.0,0.0,0.0,0.9988785386085509,0.0,0.0,0.0,0.0,0.999642252922058,0.9684346318244934,0.0,0.9962790608406068,0.9960338473320008,0.0,0.0 268 | 4277,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 269 | 4280,0,0.9917275905609132,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9986763596534728,0.0,0.0,0.0,0.0,0.96873277425766,0.0,0.0,0.0,0.0,0.9990707635879515,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.998581051826477,0.0,0.0,0.9569268226623536,0.0,0.0,0.0 270 | 4281,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.6550976634025574,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9640130400657654,0.0,0.0,0.0,0.9590667486190796,0.0,0.0,0.0,0.0 271 | 4282,0,0.9726621508598328,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 272 | 4283,0,0.0,0.0,0.0,0.0,0.0,0.0,0.9966647028923036,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9554243683815002,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9965131878852844,0.0,0.0,0.0,0.0,0.0,0.0 273 | 4284,0,0.0,0.0,0.0,0.7409618496894836,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9284103512763976,0.982372224330902,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 274 | 4285,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9783498048782348,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 275 | 4290,0,0.0,0.0,0.0,0.0,0.9981054067611694,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9853342771530152,0.0,0.0,0.0,0.0,0.9988396763801576,0.0,0.9955050349235536,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9985456466674804,0.0,0.0,0.993093967437744,0.0,0.0 276 | 4292,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9061157107353209,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9652937054634094,0.0 277 | 4293,0,0.0,0.0,0.998428225517273,0.0,0.0,0.9620833992958068,0.9888478517532348,0.0,0.0,0.0,0.0,0.0,0.0,0.9668589234352112,0.0,0.0,0.0,0.0,0.999128758907318,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9943962097167968,0.0,0.0,0.0,0.0,0.0,0.0,0.9909098744392396,0.0,0.0,0.0 278 | 4297,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 279 | 4302,0,0.0,0.0,0.0,0.0,0.0,0.0,0.958627700805664,0.0,0.0,0.0,0.0,0.0,0.0,0.9891878962516784,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9816306233406068,0.0,0.0,0.0,0.0,0.0,0.0 280 | 4304,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9171552062034608,0.0,0.0,0.0,0.0,0.9969252943992616,0.9947595000267028,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 281 | 4307,0,0.0,0.0,0.0,0.9631641507148744,0.0,0.0,0.0,0.0,0.996807336807251,0.0,0.0,0.0,0.9878841042518616,0.9985414743423462,0.0,0.0,0.0,0.9255155324935912,0.9991100430488586,0.0,0.0,0.0,0.0,0.0,0.9965338706970216,0.0,0.0,0.982174515724182,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 282 | 4310,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9757294058799744,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8858920335769653,0.0 283 | 4312,0,0.0,0.0,0.0,0.0,0.0,0.0,0.9949017763137816,0.0,0.0,0.0,0.0,0.0,0.9863874912261964,0.0,0.0,0.0,0.0,0.0,0.0,0.9958863854408264,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9923931360244752,0.0,0.0,0.9480738639831544,0.0,0.0,0.0 284 | 4315,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9494127035140992,0.0,0.0,0.957765817642212,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 285 | 4317,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9984943866729736,0.0,0.0,0.0,0.9804180860519408,0.9965882301330566,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9428630471229552,0.0 286 | 4318,0,0.0,0.0,0.0,0.0,0.0,0.9382194876670836,0.9989805817604064,0.0,0.0,0.0,0.0,0.0,0.0,0.9907550215721129,0.0,0.0,0.0,0.0,0.999398946762085,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.999316930770874,0.0,0.0,0.0,0.0,0.0,0.9840595722198486 287 | 4322,0,0.9882947206497192,0.0,0.0,0.0,0.9337171316146852,0.0,0.9991806149482728,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9990642666816713,0.0,0.0,0.0,0.0,0.9993398785591124,0.0,0.0,0.9956762194633484,0.0,0.0,0.0,0.0,0.9979339838027954,0.0,0.9494867324829102,0.0,0.0,0.990314781665802 288 | 4327,0,0.0,0.0,0.0,0.8054326772689819,0.0,0.9496755003929138,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.975269615650177,0.0,0.0,0.0,0.9796068072319032,0.9991499185562134,0.0,0.0,0.0,0.0,0.9997286200523376,0.0,0.0,0.9955289959907532,0.0,0.0,0.0,0.0,0.0,0.0,0.9884524941444396,0.9950224161148072,0.0,0.9952475428581238 289 | 4329,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 290 | 4330,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 291 | 4339,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9880546927452089,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5878257751464844,0.0 292 | 4341,0,0.0,0.0,0.0,0.0,0.0,0.9748920798301696,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8074784874916077,0.9682630300521852,0.0,0.0,0.0,0.9975653886795044,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9983201622962952,0.0,0.0 293 | 4342,0,0.0,0.0,0.0,0.0,0.0,0.9095599055290222,0.997869610786438,0.0,0.9992677569389344,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9990585446357728,0.9952502250671388,0.0,0.0,0.0,0.0,0.0,0.0,0.9971994161605836,0.0,0.0,0.0,0.9980692267417908,0.0,0.0,0.0,0.9954116940498352,0.0,0.9883174300193788 294 | 4344,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9658799171447754,0.9958613514900208,0.0,0.0,0.0,0.0,0.9971471428871156,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9887873530387878 295 | 4345,0,0.0,0.0,0.0,0.0,0.9840211272239684,0.0,0.9960145950317384,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.996282398700714,0.0,0.0,0.0,0.0,0.9907004237174988,0.9701716899871826,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 296 | 4348,0,0.0,0.0,0.0,0.0,0.9984715580940248,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9885000586509703,0.0,0.0,0.0,0.0,0.0,0.0,0.9941710829734802,0.0,0.0,0.0,0.0,0.0,0.9956852197647096,0.0,0.9961311817169191,0.0,0.9988906979560852,0.9988860487937928,0.0,0.0,0.0,0.0,0.0 297 | 4353,0,0.9095882177352904,0.0,0.0,0.0,0.9994081258773804,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9979660511016846,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9986672401428224,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9666244983673096,0.0,0.0,0.9979624748229979,0.0,0.9802306294441224,0.0,0.0,0.0 298 | 4354,0,0.0,0.0,0.9966551065444946,0.0,0.0,0.0,0.9989554882049559,0.0,0.9993909597396852,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9929443001747132,0.0,0.0,0.0,0.0,0.9941245913505554,0.0,0.0,0.0,0.0,0.9987952709197998,0.0,0.0,0.0,0.0,0.0,0.0 299 | 4355,0,0.9953946471214294,0.0,0.0,0.9530481696128844,0.0,0.9498204588890076,0.998970866203308,0.0,0.998735010623932,0.0,0.0,0.0,0.0,0.9894121289253236,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9982287287712096,0.9951253533363342,0.0,0.0,0.0,0.9987241625785828,0.0,0.0,0.9904158115386964,0.9975708127021791,0.0,0.0 300 | 4356,0,0.9919614195823668,0.0,0.0,0.0,0.9974963068962096,0.9010663628578186,0.0,0.0,0.998029887676239,0.0,0.0,0.0,0.0,0.9267282485961914,0.9972027540206908,0.0,0.0,0.0,0.9980419874191284,0.9924768805503844,0.0,0.0,0.0,0.0,0.993501365184784,0.0,0.9933431148529052,0.9905825853347778,0.0,0.0,0.0,0.9730292558670044,0.0,0.9802205562591552,0.0,0.0,0.9888595342636108 301 | 4358,0,0.0,0.0,0.0,0.0,0.0,0.0,0.9995971322059632,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.99644273519516,0.9968480467796326,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.999704897403717,0.0,0.0,0.0,0.0,0.0,0.0 302 | 4361,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9424153566360474,0.0,0.0,0.0,0.0,0.0,0.995107650756836,0.9956995248794556,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.981347918510437,0.0,0.0,0.0 303 | 4363,0,0.0,0.0,0.0,0.9536016583442688,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.989066481590271,0.0,0.0,0.0,0.933516561985016,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9968763589859008,0.9964220523834229,0.0,0.0,0.9313277602195741,0.0,0.0,0.0,0.0,0.0,0.0,0.0 304 | 4365,0,0.0,0.0,0.0,0.9688974022865297,0.9920896291732788,0.0,0.9983116388320924,0.0,0.9981119632720948,0.0,0.0,0.0,0.0,0.8318285346031189,0.0,0.0,0.0,0.0,0.9981417655944824,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.996154248714447,0.0,0.0,0.9830328822135924,0.0,0.0,0.0 305 | 4366,0,0.9974403381347656,0.0,0.9982276558876038,0.9670243859291076,0.0,0.8651847243309021,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9956919550895692,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9932695031166076,0.0,0.0,0.0 306 | 4369,0,0.0,0.0,0.9497841000556946,0.0,0.0,0.0,0.9877303242683412,0.0,0.0,0.0,0.0,0.0,0.0,0.9699538350105286,0.0,0.0,0.0,0.0,0.0,0.9953320622444152,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.990720510482788,0.9891128540039062,0.0,0.0,0.0,0.0,0.0 307 | 4370,0,0.9967440366744996,0.0,0.9989033937454224,0.5390135645866394,0.0,0.9775614142417908,0.0,0.0,0.9983752965927124,0.0,0.0,0.0,0.0,0.5038776397705078,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9978185892105104,0.0,0.0,0.0,0.0,0.999461829662323,0.9977450370788574,0.0,0.9841374158859252,0.99433833360672,0.0,0.0 308 | 4373,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7962594628334045,0.0,0.0,0.0,0.0,0.9984793066978456,0.0,0.0,0.0,0.7487519979476929,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 309 | 4378,0,0.0,0.0,0.999334990978241,0.0,0.0,0.7904200553894043,0.0,0.0,0.9992713332176208,0.0,0.0,0.0,0.0,0.9439085721969604,0.0,0.0,0.0,0.0,0.0,0.9954485297203064,0.0,0.0,0.0,0.9997712969779968,0.0,0.0,0.0,0.0,0.0,0.0,0.9993910789489746,0.0,0.0,0.9816120266914368,0.9915627241134644,0.0,0.0 310 | 4379,0,0.0,0.0,0.9968246221542358,0.0,0.0,0.0,0.99733304977417,0.0,0.9988387227058412,0.0,0.0,0.0,0.0,0.9907647371292114,0.9463278651237488,0.0,0.0,0.0,0.9988433122634888,0.9882007837295532,0.9952046871185304,0.0,0.0,0.99964702129364,0.0,0.0,0.0,0.0,0.9025091528892516,0.0,0.9951357245445251,0.0,0.0,0.0,0.0,0.0,0.0 311 | 4387,0,0.9865959286689758,0.0,0.9990723133087158,0.4285808503627777,0.6010024547576904,0.9845964312553406,0.9990440011024476,0.0,0.9981337189674376,0.0,0.0,0.0,0.0,0.9805389642715454,0.9981194138526917,0.0,0.0,0.0,0.99784255027771,0.0,0.9952351450920104,0.0,0.0,0.9997770190238952,0.0,0.0,0.9966809153556824,0.0,0.0,0.0,0.9983797073364258,0.0,0.0,0.95725417137146,0.9984934329986572,0.0,0.0 312 | 4391,0,0.0,0.0,0.9986855387687684,0.4961142241954804,0.0,0.9590725898742676,0.998913049697876,0.0,0.9957780838012696,0.0,0.0,0.0,0.0,0.9795743227005004,0.0,0.0,0.0,0.0,0.9961063265800476,0.0,0.0,0.0,0.0,0.9997952580451964,0.0,0.0,0.0,0.0,0.9961703419685364,0.0,0.9988922476768494,0.0,0.0,0.8989785313606262,0.9975937008857728,0.0,0.993811309337616 313 | 4394,0,0.99261873960495,0.0,0.0,0.0,0.0,0.9809179306030272,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9985979199409484,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9990122318267822,0.0,0.0,0.0,0.0,0.982654869556427,0.9938020110130309,0.0,0.9944118857383728 314 | 4398,0,0.0,0.0,0.0,0.0,0.0,0.0,0.9951651096343994,0.0,0.9946675300598145,0.0,0.0,0.0,0.0,0.9787140488624572,0.0,0.0,0.0,0.0,0.999409556388855,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9953047037124634,0.0,0.0,0.0,0.0,0.0,0.0 315 | 4399,0,0.0,0.9204591512680054,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9984124898910522,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 316 | 4401,0,0.0,0.9286266565322876,0.0,0.0,0.0,0.0,0.9949575066566468,0.0,0.999226212501526,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9892103672027588,0.0,0.0,0.0,0.0,0.0,0.9824851751327516,0.0,0.0,0.0,0.0,0.0,0.9706951975822448,0.0,0.0,0.0,0.0,0.0 317 | 4407,0,0.0,0.0,0.9982444047927856,0.95409494638443,0.0,0.9147833585739136,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9943002462387084,0.9941691160202026,0.9995928406715392,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9995459914207458,0.0,0.9597983956336976,0.9930753707885742,0.0,0.9902377724647522 318 | 4408,0,0.996119737625122,0.0,0.9852109551429749,0.9795008897781372,0.0,0.0,0.9925692677497864,0.0,0.9991868138313292,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9411648511886596,0.0,0.998469889163971,0.9962211847305298,0.0,0.0,0.0,0.9996683597564696,0.0,0.0,0.0,0.0,0.0,0.0,0.9978533387184144,0.0,0.0,0.9939714074134828,0.0,0.0,0.0 319 | 4409,0,0.0,0.0,0.0,0.9785191416740416,0.0,0.0,0.9987687468528748,0.0,0.998824417591095,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.994434893131256,0.9993396401405334,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9984198808670044,0.9990707039833068,0.0,0.0,0.0,0.0,0.0 320 | 4410,0,0.0,0.0,0.0,0.0,0.0,0.9716328978538512,0.9988247752189636,0.0,0.9984512329101562,0.0,0.0,0.0,0.9972568154335022,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.997050642967224,0.0,0.0,0.0,0.0,0.0,0.9983649849891664,0.0,0.0,0.0,0.9980964064598083,0.0,0.0 321 | 4413,0,0.995353937149048,0.0,0.9771319031715392,0.0,0.9985561966896056,0.0,0.0,0.0,0.9964836835861206,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9989370703697203,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9813151955604552,0.0,0.0,0.0 322 | 4414,0,0.9948892593383788,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9797454476356506,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9853641390800476,0.0,0.0,0.0 323 | 4417,0,0.9944538474082948,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9833985567092896,0.999225616455078,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9785903692245485,0.0,0.0,0.0 324 | 4422,0,0.9966040849685668,0.9149872064590454,0.0,0.0,0.0,0.0,0.0,0.0,0.9992442131042479,0.0,0.0,0.0,0.0,0.8653165698051453,0.9924253225326538,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.997688055038452,0.0,0.0,0.0,0.0,0.0,0.9977226853370668,0.0,0.993960976600647,0.0,0.0,0.0 325 | 4425,0,0.987901210784912,0.0,0.9987805485725404,0.0,0.9968942403793336,0.9522094130516052,0.9994472861289978,0.0,0.0,0.0,0.0,0.0,0.9973215460777284,0.8231527209281921,0.9974690675735474,0.9922515749931335,0.0,0.0,0.9993658661842346,0.0,0.0,0.0,0.0,0.0,0.9978128671646118,0.0,0.0,0.0,0.9915031194686891,0.0,0.9993046522140504,0.9967975020408629,0.0,0.90118807554245,0.9958077669143676,0.0,0.0 326 | 4427,0,0.0,0.0,0.0,0.9339793920516968,0.0,0.0,0.0,0.0,0.9989441037178041,0.0,0.0,0.0,0.0,0.9895570874214172,0.0,0.9734237790107728,0.0,0.9757125973701476,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9903623461723328,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 327 | 4433,0,0.9204651713371276,0.0,0.0,0.9165496826171876,0.0,0.0,0.0,0.0,0.99892657995224,0.0,0.0,0.0,0.0,0.9193233847618104,0.9922665953636168,0.0,0.0,0.0,0.0,0.9749547243118286,0.0,0.0,0.0,0.0,0.0,0.9964736104011536,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 328 | 4437,0,0.0,0.0,0.9992051720619202,0.0,0.0,0.0,0.0,0.0,0.9921565055847168,0.0,0.0,0.0,0.0,0.0,0.999146580696106,0.0,0.9952762126922609,0.0,0.0,0.0,0.9971418976783752,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9920772314071656,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 329 | 4443,0,0.9912892580032348,0.0,0.0,0.0,0.9992116689682008,0.9647815823554992,0.9422985315322876,0.0,0.9871616959571838,0.0,0.0,0.0,0.0,0.9963307976722716,0.0,0.0,0.0,0.0,0.0,0.9945122003555298,0.9976818561553956,0.0,0.0,0.0,0.9919781684875488,0.0,0.9975881576538086,0.0,0.999369204044342,0.0,0.9923838973045348,0.992482841014862,0.0,0.988024652004242,0.99245023727417,0.0,0.0 330 | 4444,0,0.9907549023628236,0.0,0.0,0.7808677554130554,0.99674654006958,0.9493510127067566,0.0,0.0,0.9959920048713684,0.0,0.0,0.0,0.0,0.0,0.997798502445221,0.0,0.0,0.9782729148864746,0.0,0.995649755001068,0.0,0.0,0.7228522300720215,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9829444289207458,0.989698588848114,0.0,0.989284336566925 331 | 4450,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.999476969242096,0.9830082654953004,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 332 | 4451,0,0.0,0.0,0.9938171505928041,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9993374943733216,0.0,0.0,0.0,0.0,0.9919893145561218,0.0,0.0,0.0,0.999283492565155,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9604745507240297,0.9962918758392334,0.0,0.0 333 | 4455,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9992696642875672,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 334 | 4457,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 335 | 4462,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9682486653327942 336 | 4464,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 337 | 4465,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9958282113075256,0.0,0.0,0.0,0.0,0.9757556319236756,0.0,0.0,0.0,0.0,0.0,0.0,0.9953006505966188,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 338 | 4466,0,0.9958504438400269,0.0,0.9994175434112548,0.9807392358779908,0.9954212307929992,0.9249537587165833,0.0,0.0,0.9970526695251464,0.0,0.0,0.0,0.9428759813308716,0.0,0.0,0.0,0.0,0.0,0.0,0.9619278311729432,0.0,0.8677845597267151,0.0,0.9995966553688048,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9941131472587584,0.0,0.0,0.0 339 | 4469,0,0.9965689182281494,0.0,0.9991236329078674,0.0,0.0,0.0,0.0,0.0,0.9993261098861694,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9997633099555968,0.0,0.0,0.9982838034629822,0.0,0.0,0.0,0.0,0.0,0.0,0.9911439418792723,0.0,0.0,0.0 340 | 4470,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 341 | 4473,0,0.9971712231636048,0.0,0.0,0.9292067289352416,0.9957499504089355,0.9807919859886168,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9860801100730896,0.0,0.0,0.0,0.0,0.0,0.9959271550178528,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9946522712707521,0.0,0.9880902767181396,0.9984116554260254,0.0,0.0 342 | 4474,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 343 | 4477,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.987166464328766,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 344 | 4478,0,0.0,0.0,0.0,0.985791563987732,0.0,0.9196180701255798,0.9985806941986084,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9978795051574708,0.0,0.9712803959846495,0.0,0.7544327974319458,0.0,0.0,0.9876093864440918,0.0,0.0,0.0,0.0,0.9991205334663392,0.9973762035369872,0.0,0.9408379197120668,0.0,0.0,0.0 345 | 4479,0,0.9970712661743164,0.0,0.998539924621582,0.0,0.0,0.9792360663414,0.9993376135826112,0.0,0.0,0.0,0.0,0.0,0.9967249035835266,0.5913842916488647,0.0,0.0,0.0,0.0,0.998485267162323,0.0,0.0,0.0,0.0,0.9998206496238708,0.0,0.0,0.998296082019806,0.0,0.0,0.0,0.9989732503890992,0.0,0.0,0.9835276007652284,0.9980619549751282,0.0,0.0 346 | 4481,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9519366025924684,0.0,0.0,0.9932717680931092 347 | 4484,0,0.9967098236083984,0.0,0.999643087387085,0.7560529112815857,0.998793601989746,0.0,0.0,0.0,0.9984582662582396,0.0,0.0,0.0,0.0,0.0,0.9982072114944458,0.0,0.0,0.9503793716430664,0.996537446975708,0.9912354946136476,0.0,0.0,0.0,0.9996525049209596,0.9920365810394288,0.0,0.0,0.0,0.9547344446182252,0.0,0.0,0.0,0.0,0.9903941750526428,0.0,0.0,0.0 348 | 4485,0,0.9841483831405641,0.0,0.0,0.9733527302742004,0.9957256317138672,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9977377653121948,0.0,0.0,0.0,0.9991369843482972,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9990585446357728,0.0,0.9908864498138428,0.0,0.0,0.0,0.0,0.0,0.0 349 | 4487,0,0.9978081583976746,0.0,0.9993163347244264,0.0,0.9992730021476746,0.0,0.0,0.0,0.9962568283081056,0.0,0.0,0.0,0.0,0.0,0.9990390539169312,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.99979168176651,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9954588413238524,0.0,0.0,0.98639976978302 350 | 4489,0,0.9500147104263306,0.0,0.9993643164634703,0.8895315527915955,0.0,0.9815664291381836,0.0,0.0,0.0,0.0,0.0,0.0,0.981181025505066,0.0,0.977152407169342,0.0,0.9787930846214294,0.98924058675766,0.9979444146156312,0.0,0.0,0.0,0.0,0.9997802376747132,0.9948534369468688,0.0,0.0,0.0,0.0,0.0,0.9987850785255432,0.0,0.9943737983703612,0.0,0.0,0.0,0.9905779361724854 351 | 4497,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9869427680969238,0.0,0.8826259970664978,0.0,0.9973605275154114,0.0,0.0,0.0,0.0,0.9892606735229492,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9651840925216676,0.0,0.0,0.9875565171241759 352 | 4498,0,0.0,0.0,0.9820207953453064,0.0,0.0,0.9316129088401794,0.9987577199935912,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9960243701934814,0.99080091714859,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9991196990013124,0.9974573254585266,0.0,0.0,0.99446302652359,0.0,0.0 353 | 4500,0,0.9972368478775024,0.0,0.9649906754493712,0.0,0.0,0.9741753935813904,0.9991379976272584,0.0,0.0,0.0,0.0,0.0,0.996510922908783,0.0,0.0,0.0,0.0,0.0,0.9985833168029784,0.996670424938202,0.8476964831352234,0.0,0.0,0.9994687438011168,0.0,0.0,0.9983439445495604,0.0,0.0,0.0,0.9989765882492064,0.0,0.0,0.9845504164695741,0.995237112045288,0.0,0.0 354 | 4502,0,0.997042953968048,0.0,0.9990217685699464,0.0,0.0,0.8687757849693298,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9856989979743958,0.0,0.9990480542182922,0.0,0.0,0.9997795224189758,0.0,0.0,0.9983803033828736,0.0,0.0,0.0,0.0,0.0,0.0,0.9875956773757936,0.9879276156425476,0.0,0.0 355 | 4505,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 356 | 4506,0,0.0,0.0,0.9976356029510498,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9099527597427368,0.0,0.0,0.0,0.9990993738174438,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 357 | 4512,0,0.9952847361564636,0.0,0.9992246627807616,0.0,0.9945264458656312,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.990935742855072,0.994658887386322,0.0,0.0,0.9995079040527344,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9795056581497192,0.0,0.9907943606376648,0.0,0.0,0.0 358 | 4515,0,0.9963828325271606,0.0,0.9989259839057922,0.0,0.0,0.9157451391220092,0.9991034865379332,0.0,0.0,0.0,0.0,0.0,0.979425013065338,0.0,0.0,0.0,0.0,0.0,0.9824000597000122,0.99461430311203,0.0,0.0,0.0,0.9997900128364564,0.0,0.0,0.9985675811767578,0.0,0.0,0.0,0.9989476799964904,0.0,0.0,0.9787858724594116,0.994151532649994,0.0,0.0 359 | 4516,0,0.9968998432159424,0.0,0.9957693815231324,0.0,0.9960021376609802,0.0,0.9985972046852112,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7963827848434448,0.0,0.0,0.0,0.998244047164917,0.9959990978240968,0.0,0.0,0.0,0.0,0.96290522813797,0.0,0.0,0.0,0.0,0.0,0.9981393814086914,0.0,0.0,0.986128568649292,0.0,0.0,0.997059404850006 360 | 4518,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.998728334903717,0.0,0.0,0.0,0.0,0.0,0.9164012670516968,0.0,0.971339225769043,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9501384496688844,0.0,0.0,0.9964085221290588,0.8959142565727234,0.9719404578208924,0.9971256852149964,0.0,0.0 361 | 4519,0,0.0,0.0,0.9993327260017396,0.8596173524856567,0.0,0.0,0.0,0.0,0.9988578557968141,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9957014918327332,0.0,0.0,0.0,0.9998351335525512,0.0,0.0,0.9970489144325256,0.0,0.9641205072402954,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 362 | 4520,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9888545274734496,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9930127263069152,0.0,0.0,0.9982748031616212,0.0,0.0 363 | 4523,0,0.995326042175293,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9980528354644777,0.0,0.9917820692062378,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9948264956474304,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9864290952682496,0.0,0.0,0.0 364 | 4525,0,0.0,0.0,0.9972705245018004,0.0,0.9983285069465636,0.9732614755630492,0.0,0.0,0.0,0.0,0.0,0.0,0.9995099306106568,0.0,0.9976207613945008,0.0,0.0,0.0,0.997783362865448,0.0,0.997266173362732,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9976455569267272,0.0,0.0,0.9972467422485352,0.0,0.9953131675720216 365 | 4530,0,0.0,0.0,0.9960095882415771,0.0,0.9988377690315248,0.8884617686271667,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9784774184226991,0.0,0.0,0.0 366 | 4531,0,0.0,0.0,0.8931040167808533,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9089359641075134,0.9887603521347046,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 367 | 4533,0,0.9978779554367064,0.0,0.0,0.0,0.9987560510635376,0.9543508887290956,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9956563711166382,0.0,0.0,0.0,0.9981443881988524,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9914841055870056,0.996204674243927,0.0,0.0 368 | 4534,0,0.0,0.0,0.0,0.9513267278671264,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9984669089317322,0.886768639087677,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9295616745948792,0.0,0.0,0.0,0.9739262461662292,0.0,0.0,0.0,0.0 369 | 4535,0,0.0,0.0,0.0,0.893271267414093,0.0,0.95586359500885,0.999251663684845,0.0,0.9975610971450806,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9977325201034546,0.0,0.9982460737228394,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9991212487220764,0.9996283054351808,0.0,0.0,0.9925408363342284,0.0,0.0 370 | 4536,0,0.0,0.0,0.0,0.0,0.0,0.9162667393684388,0.0,0.0,0.9977273344993592,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9950578212738036,0.993947982788086,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9755735993385316,0.9980382323265076,0.0,0.0 371 | 4537,0,0.0,0.0,0.9919249415397644,0.0,0.0,0.9609695672988892,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9947487711906432,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 372 | 4544,0,0.0,0.0,0.0,0.0,0.0,0.9403304457664491,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9606561660766602,0.0,0.0,0.0,0.0,0.9983347058296204,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9924803972244264,0.0,0.0 373 | 4547,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9931718111038208,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8761487007141113,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.4666653275489807,0.0 374 | 4548,0,0.0,0.0,0.9993411898612976,0.9567309021949768,0.0,0.0,0.0,0.0,0.9981302618980408,0.986394226551056,0.0,0.0,0.0,0.9950860142707824,0.9985328912734984,0.0,0.0,0.0,0.9956673383712769,0.992514193058014,0.0,0.0,0.0,0.9995854496955872,0.0,0.0,0.9952741861343384,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 375 | 4550,0,0.9969594478607178,0.0,0.0,0.0,0.9988046884536744,0.9667805433273317,0.9990791082382202,0.0,0.0,0.0,0.0,0.0,0.9978952407836914,0.5421189665794373,0.9847421646118164,0.0,0.0,0.9577844738960266,0.0,0.9961864352226256,0.9851539731025696,0.0,0.0,0.0,0.992069125175476,0.0,0.0,0.0,0.9992018342018129,0.0,0.992923378944397,0.9971168041229248,0.0,0.9927651286125184,0.9921053647994996,0.0,0.0 376 | 4551,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9982314705848694,0.9987589120864868,0.0,0.0,0.0,0.0,0.0,0.992456316947937,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 377 | 4553,0,0.9942833781242372,0.0,0.0,0.8625645041465759,0.0,0.971182346343994,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9968371391296388,0.0,0.0,0.0,0.9981802701950072,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9655733704566956,0.9948446750640868,0.0,0.9939846396446228 378 | 4555,0,0.8669775724411011,0.0,0.9995378255844116,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9980826377868652,0.0,0.0,0.0,0.9988166093826294,0.0,0.0,0.0,0.6192555427551271,0.9996383190155028,0.0,0.0,0.0,0.0,0.999060332775116,0.0,0.0,0.9513209462165833,0.8961997628211975,0.9820731878280641,0.0,0.0,0.0 379 | 4556,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9964823722839355,0.9957758784294128,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9979350566864014,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9868932962417604 380 | 4559,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.5248376131057739,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 381 | 4562,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9828090071678162,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 382 | 4564,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.997757613658905,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9763270616531372 383 | 4566,0,0.9927629828453064,0.0,0.998824179172516,0.0,0.0,0.0,0.0,0.0,0.9992411136627196,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9992495179176332,0.0,0.0,0.0,0.0,0.9996907711029052,0.0,0.0,0.997461199760437,0.0,0.0,0.0,0.0,0.0,0.0,0.9930122494697572,0.0,0.0,0.9833557605743408 384 | 4568,0,0.0,0.0,0.0,0.0,0.0,0.9722948670387268,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9952007532119752,0.0,0.0 385 | 4571,0,0.0,0.0,0.9963964819908142,0.97039133310318,0.9277663230895996,0.0,0.0,0.0,0.9986122846603394,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9943865537643432,0.9866864681243896,0.0,0.0,0.9983080625534058,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9719314575195312,0.0,0.9557806849479676,0.9913750290870668,0.0,0.0 386 | 4572,0,0.9970800280570984,0.0,0.987379252910614,0.0,0.998923361301422,0.9504491090774536,0.9987359642982484,0.0,0.0,0.0,0.0,0.0,0.0,0.9986728429794312,0.0,0.0,0.0,0.0,0.0,0.9937906861305236,0.0,0.0,0.8411729335784912,0.9988000392913818,0.0,0.9968993663787842,0.0,0.0,0.0,0.0,0.9989563226699828,0.9663669466972352,0.0,0.9917783141136168,0.0,0.0,0.0 387 | 4574,0,0.0,0.0,0.0,0.0,0.0,0.0,0.9941634535789491,0.0,0.9850153923034668,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 388 | 4575,0,0.99548077583313,0.0,0.9995585083961488,0.6881191730499268,0.999014973640442,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9943999648094176,0.0,0.0,0.3728523254394531,0.9996920824050904,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9929083585739136,0.0,0.7830172181129456,0.0 389 | 4584,0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9993537068367004,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9949175119400024 390 | 4585,0,0.973491907119751,0.0,0.998855471611023,0.0,0.9966128468513488,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.7391393184661865,0.0,0.0,0.0,0.0,0.9992699027061462,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9906448125839232,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 391 | 4590,0,0.0,0.0,0.9982523322105408,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9942137598991394,0.9982701539993286,0.0,0.0,0.0,0.9980411529541016,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9970955848693848,0.0,0.0,0.0,0.9983332753181458,0.0,0.0,0.0,0.0,0.0,0.0 392 | 4592,0,0.9957043528556824,0.0,0.0,0.0,0.0,0.0,0.9993763566017152,0.0,0.0,0.0,0.0,0.0,0.9962879419326782,0.0,0.0,0.0,0.0,0.0,0.99809592962265,0.0,0.9562206268310548,0.0,0.0,0.9996892213821412,0.0,0.0,0.9983710646629332,0.0,0.0,0.0,0.9993025064468384,0.0,0.0,0.9776806831359864,0.9952490329742432,0.0,0.0 393 | 4594,0,0.0,0.0,0.0,0.872241735458374,0.0,0.8335050940513611,0.9622510075569152,0.0,0.9982208609580994,0.0,0.0,0.0,0.0,0.995834231376648,0.0,0.0,0.0,0.0,0.997969090938568,0.0,0.99777489900589,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.992228090763092,0.0,0.0,0.0,0.9930166006088256,0.0,0.0 394 | -------------------------------------------------------------------------------- /data/sample_report_1.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-textract-and-comprehend-medical-document-processing/669ef2de621c2c4d18f81405056d4bb8c9f61f69/data/sample_report_1.pdf -------------------------------------------------------------------------------- /util/.ipynb_checkpoints/preprocess-checkpoint.py: -------------------------------------------------------------------------------- 1 | import seaborn as sns 2 | import matplotlib.pyplot as plt 3 | import pandas as pd 4 | import boto3, botocore 5 | 6 | 7 | class color: 8 | PURPLE = '\033[95m' 9 | CYAN = '\033[96m' 10 | DARKCYAN = '\033[36m' 11 | BLUE = '\033[94m' 12 | GREEN = '\033[92m' 13 | YELLOW = '\033[93m' 14 | RED = '\033[91m' 15 | BOLD = '\033[1m' 16 | UNDERLINE = '\033[4m' 17 | END = '\033[0m' 18 | 19 | 20 | def retrieve_mcList(df, nFeature=20,threshold=0.9): 21 | ## change all terms to lower case 22 | df['MEDICAL_CONDITION']=df['MEDICAL_CONDITION'].str.lower() 23 | ## 24 | df=df.replace(['hemostatic','hematoma','Hemostasis'],'hemostasis') 25 | df=df.replace(['wounds','masses','lesions','polyps'],['wound' ,'mass','lesion','polyp']) 26 | mcList=df[df.Score>=threshold].MEDICAL_CONDITION.value_counts().index[:nFeature].to_list() 27 | 28 | 29 | return mcList, df 30 | 31 | 32 | 33 | def mc_barplot(df, threshold_score=0.9,topN=20): 34 | df_mcs_vc = df.MEDICAL_CONDITION[df.Score>0.9].value_counts() 35 | 36 | df_mcs_top = df_mcs_vc[:topN,] 37 | plt.figure(figsize=(20,5)) 38 | chart=sns.barplot(df_mcs_top.index, df_mcs_top.values, alpha=0.8) 39 | chart.set_xticklabels(chart.get_xticklabels(), rotation=45) 40 | plt.title(f'top {topN} medical conditions in the patients') 41 | plt.ylabel('Number of Occurrences', fontsize=12) 42 | plt.xlabel('occurance', fontsize=14) 43 | plt.show() 44 | return df_mcs_vc 45 | 46 | 47 | ## extract medical conditions in a batch 48 | 49 | 50 | import pandas as pd 51 | def extractMC_v2(json_file): 52 | list_icd10=[] 53 | medical_conditions=[] 54 | scores=[] 55 | traits=[] 56 | for row in json_file['Entities']: 57 | # 58 | if row['Category'] == "MEDICAL_CONDITION": 59 | medical_conditions.append(row['Text'])# += row['Text'] + ' ' 60 | scores.append(row['Score']) 61 | trait='NaN' 62 | if row['Traits']: 63 | #print(row['Traits'],row['Text'] ) 64 | trait=row['Traits'][0]['Name'] 65 | 66 | traits.append(trait) 67 | df_mc = pd.DataFrame({'MEDICAL_CONDITION': pd.Series(medical_conditions), 'Score':pd.Series(scores),'Trait':pd.Series(traits)}) 68 | return df_mc 69 | 70 | 71 | def extractMCbatch(transcriptionList,patientIDList): 72 | df_final = pd.DataFrame() 73 | #patient_id=100 74 | #assert(len(transcriptionList)==len(patientIDList)): 75 | if(len(transcriptionList)!=len(patientIDList)): 76 | return("Error! different length!") 77 | 78 | 79 | for item,patient_id in zip(transcriptionList,patientIDList): 80 | 81 | df_ind = extractMC_v2(item) 82 | df_ind['ID']=patient_id 83 | patient_id=patient_id+1 84 | df_final=df_final.append(df_ind) 85 | 86 | 87 | # remove the duplicated entries 88 | df_final=df_final.sort_values(by=['ID','MEDICAL_CONDITION']).drop_duplicates(['ID','MEDICAL_CONDITION'],keep='last') 89 | 90 | #print(df_final.shape) 91 | 92 | return df_final 93 | 94 | 95 | from tqdm import tqdm 96 | import boto3 97 | 98 | ## this function will extract the subpopulation 99 | def subpopulation_comprehend(df, medical_specialty,sampleSize=200): 100 | ## select the sub population 101 | df_sub=df[df.medical_specialty==medical_specialty ].reset_index() 102 | #df_sub.head() 103 | 104 | ## sample from the population 105 | df_sub_sub=df_sub.sample(n=sampleSize, random_state=123) 106 | print("original data shape is ",df_sub_sub.shape) 107 | 108 | ## remove missing entries 109 | df_sub_sub=df_sub_sub[df_sub_sub.transcription.notna()==True] 110 | print("data shape after removing missing entries is ",df_sub_sub.shape) 111 | 112 | #patient_ids=df_sub_sub['id'].to_list() 113 | 114 | 115 | cm = boto3.client(service_name='comprehendmedical', use_ssl=True, region_name = 'us-east-1') 116 | #idx=0 117 | #print("df_sub_sub['transcription'] ", len(df_sub_sub['transcription'])) 118 | patient_ids=df_sub_sub['id'].to_list() 119 | ## comprehend processing 120 | # transcrption_list=df_sub_sub['id'].to_list() 121 | transcrption_list=[] 122 | for text in tqdm(df_sub_sub['transcription']): 123 | #print(idx) 124 | #print("----------------") 125 | #print("analyzing:", text) 126 | comprehend_result = cm.detect_entities_v2(Text = text) 127 | #print(len(comprehend_result)) 128 | transcrption_list.append(comprehend_result) 129 | 130 | 131 | return transcrption_list, patient_ids 132 | 133 | 134 | def corrPlot(df): 135 | plt.figure(figsize=(15,15)) 136 | corr = df.iloc[:,1:].corr() ## skip the 1st column as it is the patient_id 137 | ax = sns.heatmap( 138 | corr, 139 | vmin=-1, vmax=1, center=0, 140 | cmap=sns.diverging_palette(20, 220, n=200), 141 | square=True 142 | ) 143 | ax.set_xticklabels(ax.get_xticklabels(), 144 | rotation=45, 145 | horizontalalignment='right' 146 | ); 147 | 148 | return 149 | 150 | ##### function to interate the medical conditions and then convert to a wide formate 151 | 152 | def dataframe_convert(df_raw,df_final, condition ): 153 | 154 | #step1: get the sub dataframe 155 | df_sub = df_raw[df_raw.MEDICAL_CONDITION==condition] 156 | 157 | #step2: iterate the sub dataframe and fill the information into the native ones 158 | for index, row in df_sub.iterrows(): 159 | #print(row) 160 | sid=row.ID 161 | 162 | #print("condition is:",condition) 163 | df_final.loc[df_final.ID == sid,condition]=row.Score 164 | #print("Processed: ",df_final.columns) 165 | return df_final 166 | 167 | 168 | 169 | #### predict for single record ## 170 | def predict_from_numpy(predictor, data): 171 | # Configure predictor for CSV input: 172 | predictor.content_type = "text/csv" 173 | predictor.serializer = sagemaker.predictor.csv_serializer 174 | # Fetch result and load back to numpy: 175 | return np.fromstring(predictor.predict(data).decode("utf-8"), sep=",") 176 | 177 | ##### the function to convert dataframe of medical conditions from long format to wide format 178 | 179 | 180 | colname_mc=['nontender', 'foreign body', 'edema', 'alert', 'murmur', 181 | 'chest pain', 'vomiting', 'hiatal hernia', 'distress', 'hemostasis', 182 | 'carpal tunnel syndrome', 'endometriosis', 'weakness', 'pain', 'mass', 183 | 'inflammation', 'polyp', 'bleeding', 'hypertension', 'supple', 'fever', 184 | 'stenosis', 'wound', 'cyanosis', 'infection', 'erythema', 185 | 'normocephalic', 'fracture', 'lesion', 'ulceration', 'nausea', 'cough', 186 | 'tumor', 'soft', 'shortness of breath', 'injury', 'diabetes'] 187 | 188 | 189 | 190 | 191 | def df_mc_generator(df_mcs,colname_mc=colname_mc ,colname_other=['ID',"Label"] ): 192 | 193 | ## remove duplicate rows 194 | df_1 = df_mcs.drop_duplicates(subset=['ID']).copy() 195 | 196 | ## generate an empty dataframe first 197 | column_names=colname_other+colname_mc 198 | ## column names 199 | df_combined=pd.DataFrame(columns=column_names) 200 | ## copy ID and positive data from the original df 201 | df_combined[colname_other]=df_1[colname_other] 202 | 203 | ## loop to fill in the information for each condition 204 | for con in colname_mc: 205 | #print(df_combined.columns) 206 | df_combined = dataframe_convert(df_mcs,df_combined, con ) 207 | df_combined = df_combined.fillna(0) 208 | df_combined["Label"] = df_combined["Label"].astype(int) 209 | 210 | return df_combined 211 | 212 | def df_mc_generator_slim(df_mcs,colname_mc=colname_mc ,colname_other=['ID'] ): 213 | 214 | ## remove duplicate rows 215 | df_1 = df_mcs.drop_duplicates(subset=['ID']).copy() 216 | #print(colname_mc) 217 | ## generate an empty dataframe first 218 | column_names=colname_other+colname_mc 219 | ## column names 220 | df_combined=pd.DataFrame(columns=column_names) 221 | ## copy ID and positive data from the original df 222 | df_combined[colname_other]=df_1[colname_other] 223 | 224 | ## loop to fill in the information for each condition 225 | for con in colname_mc: 226 | #print(df_combined.columns) 227 | df_combined = dataframe_convert(df_mcs,df_combined, con ) 228 | df_combined = df_combined.fillna(0) 229 | #df_combined["Label"] = df_combined["Label"].astype(int) 230 | 231 | return df_combined 232 | -------------------------------------------------------------------------------- /util/Pipeline.py: -------------------------------------------------------------------------------- 1 | import seaborn as sns 2 | import matplotlib.pyplot as plt 3 | import pandas as pd 4 | import boto3, botocore 5 | import trp 6 | import time 7 | #from preprocess import extractMC_v2 8 | 9 | ###################################################################### 10 | ############# functions to extract with Textract###################### 11 | textract = boto3.client('textract') 12 | 13 | def extractTextract(bucket,textractObjectName): 14 | 15 | response = textract.start_document_analysis( 16 | DocumentLocation={ 17 | 'S3Object': { 18 | 'Bucket': bucket, 19 | 'Name': textractObjectName 20 | }}, 21 | FeatureTypes=[ 22 | 'TABLES', 23 | ] 24 | ) 25 | 26 | textractJobId = response["JobId"] 27 | print('job id is: ',textractJobId) 28 | time.sleep(15) 29 | response = textract.get_document_analysis(JobId=textractJobId) 30 | status = response["JobStatus"] 31 | 32 | while(status == "IN_PROGRESS"): 33 | time.sleep(5) 34 | response = textract.get_document_analysis(JobId=textractJobId) 35 | status = response["JobStatus"] 36 | print("Textract Job status: {}".format(status)) 37 | 38 | pages=extract_text(textractJobId,response) 39 | doc = trp.Document(pages) 40 | return doc 41 | 42 | 43 | def extract_text(textractJobId,response): 44 | pages = [] 45 | 46 | time.sleep(5) 47 | 48 | response = textract.get_document_analysis(JobId=textractJobId) 49 | 50 | pages.append(response) 51 | 52 | nextToken = None 53 | if('NextToken' in response): 54 | nextToken = response['NextToken'] 55 | 56 | while(nextToken): 57 | time.sleep(5) 58 | 59 | response = textract.get_document_analysis(JobId=textractJobId, NextToken=nextToken) 60 | 61 | pages.append(response) 62 | print("Resultset page recieved: {}".format(len(pages))) 63 | nextToken = None 64 | if('NextToken' in response): 65 | nextToken = response['NextToken'] 66 | 67 | return pages 68 | 69 | 70 | ######################################################################## 71 | ############# functions to extract with comprehend###################### 72 | maxLength=10000 73 | 74 | 75 | comprehend_medical_client = boto3.client(service_name='comprehendmedical', region_name='us-east-1') 76 | 77 | 78 | def extractMedical(doc): 79 | comprehendResponse = [] 80 | for page in doc.pages: 81 | pageText = page.text 82 | 83 | for i in range(0, len(pageText), maxLength): 84 | response = comprehend_medical_client.detect_entities_v2(Text=pageText[0+i:maxLength+i]) 85 | comprehendResponse.append(response) 86 | patient_string = "" 87 | 88 | #df_cm=extractMC_v2(comprehendResponse[0]) 89 | return comprehendResponse 90 | 91 | ############################################################################################# 92 | ############# functions to convert all medical conditions to 1 record ######################## 93 | -------------------------------------------------------------------------------- /util/__pycache__/preprocess.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/aws-samples/amazon-textract-and-comprehend-medical-document-processing/669ef2de621c2c4d18f81405056d4bb8c9f61f69/util/__pycache__/preprocess.cpython-36.pyc -------------------------------------------------------------------------------- /util/classification_report.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | """ 3 | Set of helper functions to generate classification reports. 4 | """ 5 | 6 | 7 | 8 | import numpy as np 9 | import matplotlib.pyplot as plt 10 | import itertools 11 | from sklearn import metrics 12 | import sagemaker 13 | 14 | 15 | 16 | # Function to plot a nice comfusion matrix 17 | def plot_confusion_matrix(confusion_matrix, 18 | class_names_list=['Class1', 'Class2'], 19 | axis=None, 20 | title='Confusion matrix', 21 | plot_style='ggplot', 22 | colormap=plt.cm.Blues): 23 | 24 | if axis is None: # for standalone plot 25 | plt.figure() 26 | ax = plt.gca() 27 | else: # for plots inside a subplot 28 | ax = axis 29 | 30 | plt.style.use(plot_style) 31 | 32 | # normalizing matrix to [0,100%] 33 | confusion_matrix_norm = (confusion_matrix.astype('float') / 34 | confusion_matrix.sum(axis=1)[:, np.newaxis]) 35 | confusion_matrix_norm = np.round(100 * confusion_matrix_norm, 2) 36 | 37 | ax.imshow(confusion_matrix_norm, 38 | interpolation='nearest', 39 | cmap=colormap, 40 | vmin=0, # to make sure colors are scaled between [0,100%] 41 | vmax=100) 42 | 43 | ax.set_title(title) 44 | tick_marks = np.arange(len(class_names_list)) 45 | ax.set_xticks(tick_marks) 46 | ax.set_xticklabels(class_names_list, rotation=0) 47 | ax.set_yticks(tick_marks) 48 | ax.set_yticklabels(class_names_list) 49 | 50 | for i, j in itertools.product(range(confusion_matrix.shape[0]), 51 | range(confusion_matrix.shape[1])): 52 | ax.text(j, i, 53 | str(confusion_matrix[i, j])+'\n('+str(confusion_matrix_norm[i,j])+'%)', 54 | horizontalalignment="center", 55 | color="white" if confusion_matrix_norm[i, j] > 50 else "black") 56 | 57 | ax.set_ylabel('True label') 58 | ax.set_xlabel('Predicted label') 59 | ax.grid(False) 60 | 61 | if axis is None: # for standalone plots 62 | plt.tight_layout() 63 | plt.show() 64 | 65 | 66 | 67 | 68 | # Function to plot a nice preceision/recall curve for a binary classification model 69 | def plot_precision_recall_curve(y_real, 70 | y_predict, 71 | axis=None, 72 | plot_style='ggplot'): 73 | 74 | if axis is None: # for standalone plot 75 | plt.figure() 76 | ax = plt.gca() 77 | else: # for plots inside a subplot 78 | ax = axis 79 | 80 | plt.style.use(plot_style) 81 | 82 | metrics_P, metrics_R, _ = metrics.precision_recall_curve(y_real, y_predict) 83 | metrics_AP = metrics.average_precision_score(y_real, y_predict) 84 | 85 | ax.set_aspect(aspect=0.95) 86 | ax.step(metrics_R, metrics_P, color='b', where='post', linewidth=0.7) 87 | ax.fill_between(metrics_R, metrics_P, step='post', alpha=0.2, color='b') 88 | ax.set_xlabel('Recall') 89 | ax.set_ylabel('Precision') 90 | ax.set_ylim([0.0, 1.05]) 91 | ax.set_xlim([0.0, 1.05]) 92 | ax.set_title('Precision-Recall curve: AP={0:0.3f}'.format(metrics_AP)) 93 | 94 | if axis is None: # for standalone plots 95 | plt.tight_layout() 96 | plt.show() 97 | 98 | 99 | 100 | 101 | # Function to plot a nice ROC curve for a binary classification model 102 | def plot_roc_curve(y_real, 103 | y_predict, 104 | axis=None, 105 | plot_style='ggplot'): 106 | 107 | if axis is None: # for standalone plot 108 | plt.figure() 109 | ax = plt.gca() 110 | else: # for plots inside a subplot 111 | ax = axis 112 | 113 | plt.style.use(plot_style) 114 | 115 | metrics_FPR, metrics_TPR, _ = metrics.roc_curve(y_real, y_predict) 116 | metrics_AUC = metrics.roc_auc_score(y_real, y_predict) 117 | 118 | ax.set_aspect(aspect=0.95) 119 | ax.plot(metrics_FPR, 120 | metrics_TPR, 121 | color='b', 122 | linewidth=0.7) 123 | 124 | ax.fill_between(metrics_FPR, 125 | metrics_TPR, 126 | step='post', 127 | alpha=0.2, 128 | color='b') 129 | 130 | ax.plot([0, 1], [0, 1], color='k', linestyle='--', linewidth=1) 131 | ax.set_xlim([-0.05, 1.0]) 132 | ax.set_ylim([0.0, 1.05]) 133 | ax.set_xlabel('False Positive Rate') 134 | ax.set_ylabel('True Positive Rate') 135 | ax.set_title('ROC curve: AUC={0:0.3f}'.format(metrics_AUC)) 136 | 137 | if axis is None: # for standalone plots 138 | plt.tight_layout() 139 | plt.show() 140 | 141 | 142 | 143 | 144 | # Function to plot text inside an image 145 | def plot_text(text, axis=None): 146 | 147 | if axis is None: # for standalone plot 148 | plt.figure() 149 | ax = plt.gca() 150 | else: # for plots inside a subplot 151 | ax = axis 152 | 153 | # set background white 154 | ax.set_axis_off() 155 | ax.set_frame_on(True) 156 | ax.grid(False) 157 | 158 | ax.text(x=0.8, 159 | y=0, 160 | s=text, 161 | horizontalalignment="right", 162 | color="black") 163 | 164 | 165 | 166 | 167 | # Function to generate a complete classification report 168 | def generate_classification_report(y_real, 169 | y_predict_proba, 170 | decision_threshold=0.5, 171 | class_names_list=None, 172 | title='Model report', 173 | plot_style='ggplot'): 174 | 175 | plt.style.use(plot_style) 176 | 177 | if class_names_list is None: 178 | class_names_list = ['Class 0', 'Class 1'] 179 | 180 | # find out how many classes we have in the test set 181 | number_of_classes = len(np.unique(y_real)) 182 | 183 | ml_report = 'Number of classes: ' + str(number_of_classes) + '\n' 184 | 185 | for i,class_name in enumerate(class_names_list): 186 | ml_report += str(i) + ': ' + str(class_name) + '\n' 187 | 188 | ml_report += '\nDecision threshold: ' + str(decision_threshold) + '\n' 189 | 190 | ml_report += '\n---------------------Performance--------------------\n\n' 191 | 192 | y_decision = y_predict_proba.copy() 193 | y_decision[y_decision>decision_threshold] = 1 194 | y_decision[y_decision<1] = 0 195 | y_decision = y_decision.astype(bool) 196 | 197 | # get initial classification report and more text in it 198 | metrics_report = metrics.classification_report(y_real, y_decision) 199 | metrics_ACC = metrics.accuracy_score(y_real, y_decision) 200 | metrics_report += ('\n Total accuracy = ' + 201 | str(round(metrics_ACC*100,2)) + '%') 202 | ml_report += metrics_report 203 | ml_report += '\n\n\n\n\n' 204 | 205 | # generate graphs 206 | fig, ax = plt.subplots(2, 2, figsize=(12,9)) 207 | plot_text(ml_report, axis=ax[0,0]) 208 | plot_confusion_matrix(metrics.confusion_matrix(y_real, y_decision), 209 | class_names_list=class_names_list, axis=ax[0,1]) 210 | plot_precision_recall_curve(y_real, y_predict_proba, axis=ax[1,0]) 211 | plot_roc_curve(y_real, y_predict_proba, axis=ax[1,1]) 212 | fig.suptitle(title, fontsize=15) 213 | fig.tight_layout() 214 | return 215 | 216 | ## make prediction from numpy array 217 | def predict_from_numpy(predictor, data): 218 | # Configure predictor for CSV input: 219 | predictor.content_type = "text/csv" 220 | predictor.serializer = sagemaker.predictor.csv_serializer 221 | # Fetch result and load back to numpy: 222 | return np.fromstring(predictor.predict(data).decode("utf-8"), sep=",") 223 | 224 | #### predict for single record ## 225 | 226 | def predict_from_numpy_V2(predictor, data): 227 | # Configure predictor for CSV input: 228 | 229 | #predictor.content_type = "text/csv" 230 | 231 | predictor.serializer = sagemaker.serializers.CSVSerializer() 232 | predictor.serializer.CONTENT_TYPE = 'text/csv' 233 | predictor.deserializer = sagemaker.deserializers.CSVDeserializer() 234 | 235 | result=predictor.predict(data.to_numpy())[0] 236 | 237 | result_np = np.array([float(i) for i in result]) 238 | 239 | 240 | 241 | return result_np#np.fromstring(predictor.predict(data.to_numpy()).decode("utf-8"), sep=",") 242 | 243 | -------------------------------------------------------------------------------- /util/preprocess.py: -------------------------------------------------------------------------------- 1 | import seaborn as sns 2 | import matplotlib.pyplot as plt 3 | import pandas as pd 4 | import boto3, botocore 5 | from tqdm import tqdm 6 | import boto3 7 | 8 | class color: 9 | PURPLE = '\033[95m' 10 | CYAN = '\033[96m' 11 | DARKCYAN = '\033[36m' 12 | BLUE = '\033[94m' 13 | GREEN = '\033[92m' 14 | YELLOW = '\033[93m' 15 | RED = '\033[91m' 16 | BOLD = '\033[1m' 17 | UNDERLINE = '\033[4m' 18 | END = '\033[0m' 19 | 20 | ## retrieve the top medical conditions based on confidence score 21 | ## nFeature: number of features to retrieve 22 | ## threshold: confidence score from comprehend 23 | def retrieve_mcList(df, nFeature=20,threshold=0.9): 24 | ## change all terms to lower case 25 | df['MEDICAL_CONDITION']=df['MEDICAL_CONDITION'].str.lower() 26 | ## 27 | df=df.replace(['hemostatic','hematoma','Hemostasis'],'hemostasis') 28 | df=df.replace(['wounds','masses','lesions','polyps'],['wound' ,'mass','lesion','polyp']) 29 | mcList=df[df.Score>=threshold].MEDICAL_CONDITION.value_counts().index[:nFeature].to_list() 30 | 31 | 32 | return mcList, df 33 | 34 | 35 | 36 | def mc_barplot(df, threshold_score=0.9,topN=20): 37 | df_mcs_vc = df.MEDICAL_CONDITION[df.Score>0.9].value_counts() 38 | 39 | df_mcs_top = df_mcs_vc[:topN,] 40 | plt.figure(figsize=(20,5)) 41 | chart=sns.barplot(df_mcs_top.index, df_mcs_top.values, alpha=0.8) 42 | chart.set_xticklabels(chart.get_xticklabels(), rotation=45) 43 | plt.title(f'top {topN} medical conditions in the patients') 44 | plt.ylabel('Number of Occurrences', fontsize=12) 45 | plt.xlabel('occurance', fontsize=14) 46 | plt.show() 47 | return df_mcs_vc 48 | 49 | 50 | 51 | #########*************************########## 52 | 53 | ## Function to extract a single record 54 | def extractMC_v2(json_file): 55 | 56 | ## initialize the list variables for medical_conditions, scores and traits 57 | medical_conditions=[] 58 | scores=[] 59 | traits=[] 60 | for row in json_file['Entities']: 61 | # 62 | if row['Category'] == "MEDICAL_CONDITION": 63 | medical_conditions.append(row['Text'])# += row['Text'] + ' ' 64 | scores.append(row['Score']) 65 | trait='NaN' 66 | if row['Traits']: 67 | #print(row['Traits'],row['Text'] ) 68 | trait=row['Traits'][0]['Name'] 69 | 70 | traits.append(trait) 71 | df_mc = pd.DataFrame({'MEDICAL_CONDITION': pd.Series(medical_conditions), 'Score':pd.Series(scores),'Trait':pd.Series(traits)}) 72 | return df_mc 73 | 74 | ## extract medical conditions in a batch 75 | 76 | def extractMCbatch(transcriptionList,patientIDList): 77 | df_final = pd.DataFrame() 78 | #patient_id=100 79 | #assert(len(transcriptionList)==len(patientIDList)): 80 | if(len(transcriptionList)!=len(patientIDList)): 81 | return("Error! different length!") 82 | 83 | 84 | for item,patient_id in zip(transcriptionList,patientIDList): 85 | 86 | df_ind = extractMC_v2(item) 87 | df_ind['ID']=patient_id 88 | df_final=df_final.append(df_ind) 89 | 90 | 91 | # remove the duplicated entries 92 | df_final=df_final.sort_values(by=['ID','MEDICAL_CONDITION']).drop_duplicates(['ID','MEDICAL_CONDITION'],keep='last') 93 | 94 | #print(df_final.shape) 95 | 96 | return df_final 97 | 98 | 99 | 100 | ## this function will 101 | ## this function will extract the subpopulation given dataframe, medical_speciality 102 | def subpopulation_comprehend(df, medical_specialty,sampleSize=200): 103 | ## select the sub population 104 | df_sub=df[df.medical_specialty==medical_specialty ].reset_index() 105 | #df_sub.head() 106 | 107 | ## sample from the population 108 | df_sub_sub=df_sub.sample(n=sampleSize, random_state=123) 109 | print("original data shape is ",df_sub_sub.shape) 110 | 111 | ## remove missing entries 112 | df_sub_sub=df_sub_sub[df_sub_sub.transcription.notna()==True] 113 | print("data shape after removing missing entries is ",df_sub_sub.shape) 114 | 115 | cm = boto3.client(service_name='comprehendmedical', use_ssl=True, region_name = 'us-east-1') 116 | #idx=0 117 | #print("df_sub_sub['transcription'] ", len(df_sub_sub['transcription'])) 118 | patient_ids=df_sub_sub['id'].to_list() 119 | ## comprehend processing 120 | # transcrption_list=df_sub_sub['id'].to_list() 121 | transcrption_list=[] 122 | for text in tqdm(df_sub_sub['transcription']): 123 | #print(idx) 124 | #print("----------------") 125 | #print("analyzing:", text) 126 | comprehend_result = cm.detect_entities_v2(Text = text) 127 | #print(len(comprehend_result)) 128 | transcrption_list.append(comprehend_result) 129 | 130 | 131 | return transcrption_list, patient_ids 132 | 133 | 134 | def corrPlot(df): 135 | plt.figure(figsize=(15,15)) 136 | corr = df.iloc[:,1:].corr() ## skip the 1st column as it is the patient_id 137 | ax = sns.heatmap( 138 | corr, 139 | vmin=-1, vmax=1, center=0, 140 | cmap=sns.diverging_palette(20, 220, n=200), 141 | square=True 142 | ) 143 | ax.set_xticklabels(ax.get_xticklabels(), 144 | rotation=45, 145 | horizontalalignment='right' 146 | ); 147 | 148 | return 149 | 150 | ##### function to interate the medical conditions and then convert to a wide formate 151 | def dataframe_convert(df_raw,df_final, condition ): 152 | 153 | #step1: get the sub dataframe 154 | df_sub = df_raw[df_raw.MEDICAL_CONDITION==condition] 155 | 156 | #step2: iterate the sub dataframe and fill the information into the native ones 157 | for index, row in df_sub.iterrows(): 158 | #print(row) 159 | sid=row.ID 160 | 161 | #print("condition is:",condition) 162 | df_final.loc[df_final.ID == sid,condition]=row.Score 163 | #print("Processed: ",df_final.columns) 164 | return df_final 165 | 166 | 167 | 168 | 169 | 170 | ##### the function to convert dataframe of medical conditions from long format to wide format 171 | 172 | 173 | colname_mc=['nontender', 'foreign body', 'edema', 'alert', 'murmur', 174 | 'chest pain', 'vomiting', 'hiatal hernia', 'distress', 'hemostasis', 175 | 'carpal tunnel syndrome', 'endometriosis', 'weakness', 'pain', 'mass', 176 | 'inflammation', 'polyp', 'bleeding', 'hypertension', 'supple', 'fever', 177 | 'stenosis', 'wound', 'cyanosis', 'infection', 'erythema', 178 | 'normocephalic', 'fracture', 'lesion', 'ulceration', 'nausea', 'cough', 179 | 'tumor', 'soft', 'shortness of breath', 'injury', 'diabetes'] 180 | 181 | 182 | 183 | 184 | def df_mc_generator(df_mcs,colname_mc=colname_mc ,colname_other=['ID',"Label"] ): 185 | 186 | ## remove duplicate rows 187 | df_1 = df_mcs.drop_duplicates(subset=['ID']).copy() 188 | 189 | ## generate an empty dataframe first 190 | column_names=colname_other+colname_mc 191 | ## column names 192 | df_combined=pd.DataFrame(columns=column_names) 193 | ## copy ID and positive data from the original df 194 | df_combined[colname_other]=df_1[colname_other] 195 | 196 | ## loop to fill in the information for each condition 197 | for con in colname_mc: 198 | #print(df_combined.columns) 199 | df_combined = dataframe_convert(df_mcs,df_combined, con ) 200 | df_combined = df_combined.fillna(0) 201 | df_combined["Label"] = df_combined["Label"].astype(int) 202 | 203 | return df_combined 204 | 205 | def df_mc_generator_slim(df_mcs,colname_mc=colname_mc ,colname_other=['ID'] ): 206 | 207 | ## remove duplicate rows 208 | if 'ID' not in df_mcs: 209 | df_mcs['ID']=0 210 | 211 | df_1 = df_mcs.drop_duplicates(subset=['ID']).copy() 212 | #print(colname_mc) 213 | ## generate an empty dataframe first 214 | column_names=colname_other+colname_mc 215 | ## column names 216 | df_combined=pd.DataFrame(columns=column_names) 217 | ## copy ID and positive data from the original df 218 | df_combined[colname_other]=df_1[colname_other] 219 | 220 | ## loop to fill in the information for each condition 221 | for con in colname_mc: 222 | #print(df_combined.columns) 223 | df_combined = dataframe_convert(df_mcs,df_combined, con ) 224 | df_combined = df_combined.fillna(0) 225 | #df_combined["Label"] = df_combined["Label"].astype(int) 226 | 227 | return df_combined 228 | --------------------------------------------------------------------------------