├── .cfignore ├── .gitignore ├── README.md ├── auth_example.js ├── data └── Mortgage_Full_Records.csv ├── img ├── step_00 │ ├── add_credential.png │ ├── add_service.png │ ├── copy_api_key.png │ ├── copy_credentials.png │ ├── create_api_confirm.png │ ├── create_api_key.png │ ├── empty_project.png │ ├── existing_service_instance.png │ ├── find_wml.png │ ├── machine_learning.png │ ├── new_credential.png │ ├── new_project.png │ ├── project_details.png │ ├── service_credentials.png │ └── settings.png ├── step_01 │ ├── add_to_project.png │ ├── assets_tab.png │ ├── cells.png │ ├── clone_or_download.png │ ├── copy_guid.png │ ├── from_file.png │ ├── mortgage_default_cloud.png │ ├── notebook.png │ └── run_all.png ├── step_02 │ ├── explain.png │ ├── fairness.png │ ├── fairness_graph.png │ ├── mortgage_tile.png │ └── view_transactions.png └── step_03 │ └── ls.png ├── main.js ├── notebooks └── mortgage_default_cloud.ipynb ├── package-lock.json ├── package.json ├── slides.pptx └── views ├── application.hbs ├── application_results.hbs ├── explain.hbs └── index.hbs /.cfignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | node_modules/ 3 | img/ 4 | data/ 5 | notebooks/ 6 | auth_example.js 7 | README.md -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .idea/ 2 | node_modules/ 3 | auth.js -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Build, monitor and infuse AI with IBM Cloud, Watson Machine Learning, and Watson OpenScale 2 | 3 | Welcome! The instructions below have been provided as part of an IBM Developer webcast series. By following along, you will provision IBM Cloud services, train and deploy a machine learning model, monitor that model, and finally deploy an IBM Cloud app that incorporates the model and the monitoring information. The webcast replay can be found [here](https://www.crowdcast.io/e/3part-watson-ai-technology-series/2). 4 | 5 | ## Prerequisites 6 | In this step, you will provision and install the services and software you need for the hands-on lab. 7 | 8 | 1. Sign up for a free [IBM Cloud account](https://cloud.ibm.com/). 9 | 2. Sign up for a free [Watson Studio account](https://dataplatform.ibm.com/). 10 | 3. Provision a free lite instance of [Watson Machine Learning](https://cloud.ibm.com/catalog/services/machine-learning). **Choose either the Dallas or Frankfurt region.** 11 | 4. Provision a free lite instance of [Watson OpenScale](https://cloud.ibm.com/catalog/services/watson-openscale). **Choose the same region (Dallas or Frankfurt) you selected for Watson Machine Learning.** You just need to provision the service for now. **DO NOT RUN THE AUTO-SETUP DURING THE WEBCAST.** It will take too much time to complete and you won't be able to follow along. 12 | 5. Provision a free instance of [Object Storage](https://cloud.ibm.com/catalog/services/cloud-object-storage). 13 | 6. Download and install the [IBM Cloud command line tool](https://github.com/IBM-Cloud/ibm-cloud-cli-release/releases/). This is optional if you don't want to deploy the app in the final section to IBM Cloud. 14 | 15 | ## 0. Environment setup 16 | In this step, you will create the service credentials and projjects you need for the hands-on lab. 17 | 18 | Start by creating credentials for Watson Machine Learning. Navigate to your [IBM Cloud Resources page](https://cloud.ibm.com/resources). From the **Resource list**, expand the **Services** section and click on the instance of Watson Machine Learning you created in the previous section. 19 | 20 | ![Resource list -> Services -> Watson Machine Learning](./img/step_00/find_wml.png) 21 | 22 | From the menu on the left, click on **Service credentials**. 23 | 24 | ![Service credentials](./img/step_00/service_credentials.png) 25 | 26 | Click blue the **New credential** button. 27 | 28 | ![New credential](./img/step_00/new_credential.png) 29 | 30 | Give your credentials a name, and then click the **Add** button. 31 | 32 | ![Add credential](./img/step_00/add_credential.png) 33 | 34 | Your new credentials will appear in the list. Click the **copy button** to copy them to your clipboard. 35 | 36 | ![Copy to clipboard](./img/step_00/copy_credentials.png) 37 | 38 | Open your favorite text editor and paste the credentials into a new file. We will use them in a few different locations in later steps. 39 | 40 | Next, you'll need a Cloud API key. Navigate to the [Cloud user API key page](https://cloud.ibm.com/iam/apikeys) and click the **Create an IBM Cloud API key** button. 41 | 42 | ![Create an IBM Cloud API key](./img/step_00/create_api_key.png) 43 | 44 | Give your key a name and click the **Create** button. 45 | 46 | ![Confirm API key creation](./img/step_00/create_api_confirm.png) 47 | 48 | Click the **Copy** button to copy the key to your clipboard. 49 | 50 | ![Copy API key](./img/step_00/copy_api_key.png) 51 | 52 | Paste the API key into your text editor file for later use. 53 | 54 | Finally, you will need to create a Watson Studio project for your Python notebooks and models. Navigate to the [Watson Studio home page](https://dataplatform.cloud.ibm.com/) and click the **New project** button. 55 | 56 | ![New project](./img/step_00/new_project.png) 57 | 58 | Click the **Create an empty project** tile. 59 | 60 | ![Empty project](./img/step_00/empty_project.png) 61 | 62 | Give your project a name. In the **Define storage** section, select the Object Storage instance you created in the previous section, and then click **Create**. 63 | 64 | ![Project details](./img/step_00/project_details.png) 65 | 66 | Click on the **Settings** tab at the top of the screen. 67 | 68 | ![Project settings](./img/step_00/settings.png) 69 | 70 | Scroll down to the **Associated services** section. Click the **Add service** button and select **Watson** from the list. 71 | 72 | ![Associate Watson services](./img/step_00/add_service.png) 73 | 74 | Click the **Add** button on the **Machine Learning** tile. 75 | 76 | ![Add Machine Learning](./img/step_00/machine_learning.png) 77 | 78 | Select the instance of Watson Machine Learning you created in a previous step from the dropdown, and click the **Select** button. 79 | 80 | ![Existing service instance](./img/step_00/existing_service_instance.png) 81 | 82 | You are now ready to begin the lab. 83 | 84 | ## 1. Create and deploy the machine learning model 85 | 86 | Navigate to [this repo](https://github.com/ericmartens/openscale_app) and clone or download the files to your machine. 87 | 88 | ![Clone or download](./img/step_01/clone_or_download.png) 89 | 90 | Extract the downloaded files on your machine. 91 | 92 | Navigate to your [list of projects in Watson Studio](https://dataplatform.cloud.ibm.com/projects?context=wdp), and click on the project you created in the previous section to open it. 93 | 94 | Click on the **Assets** tab. 95 | 96 | ![Assets tab](./img/step_01/assets_tab.png) 97 | 98 | In the upper section of the screen, click the **Add to project** button. 99 | 100 | ![Add to project](./img/step_01/add_to_project.png) 101 | 102 | Select **Notebook**. 103 | 104 | ![Notebook](./img/step_01/notebook.png) 105 | 106 | Select the **From file** tab, and then click on the **Drag and drop files here or upload** link at the bottom. 107 | 108 | ![Drag and drop file](./img/step_01/from_file.png) 109 | 110 | Navigate to the folder where you extracted the github files, open the **notebooks** folder, select **mortgage\_default_cloud.ipynb**, and then click **Open**. 111 | 112 | ![mortgage_default_cloud.ipynb](./img/step_01/mortgage_default_cloud.png) 113 | 114 | Click the **Create** button to create the notebook in your Watson Studio project. When the notebook has loaded, scroll down to the first two code cells. You will want to replace the values for **WML_CREDENTIALS** and **CLOUD\_API_KEY** in these two cells with the credentials and API key you created in a previous step. Ensure that the Watson Machine Learning credentials are enclosed in the curly braces, and that the Cloud API key is enclosed in quotation marks. 115 | 116 | ![Edit code cells](./img/step_01/cells.png) 117 | 118 | You can now run the cells in the notebook. Click on the **Cell** menu and select **Run All**. You may also choose to run the cells one at a time. 119 | 120 | ![Run all cells](./img/step_01/run_all.png) 121 | 122 | The notebook will pull the training data from GitHub and use it to train a random forest classification model. It will then deploy that model as a web service to your Watson Machine Learning instance. 123 | 124 | Next, it will use the Watson OpenScale Python client to configure explainability, drift, fairness, and accuracy monitoring for the model. Finally, it will submit enough random records from the training data to the model to engage the monitors. 125 | 126 | The notebook is heavily commented to help you understand what each cell is doing. You may read through it for more detailed information. 127 | 128 | When the notebook has finished running, scroll to the very bottom of the screen. You will need to copy the output of the final cell and paste it into your text file for use in the application. 129 | 130 | ![Copy the OpenScale GUID](./img/step_01/copy_guid.png) 131 | 132 | ## 2. Explore Watson OpenScale 133 | 134 | Watson OpenScale monitors production machine learning models for quality, fairness, and drift. It also provides detailed explanations of model predictions. 135 | 136 | Navigate to [Watson OpenScale](https://aiopenscale.cloud.ibm.com/aiopenscale/). Once you have signed in, you will see the Insights Dashboard, which provides an overview of the models you are monitoring. Click on the tile for the Mortgage Default model. 137 | 138 | ![Mortgage Default](./img/step_02/mortgage_tile.png) 139 | 140 | The notebook you ran in the previous step configured the monitors for our model, and provided enough data for the monitors to run. If you are interested in learning more about the monitors OpenScale provides, click [here](https://www.ibm.com/demos/collection/IBM-Watson-OpenScale/). 141 | 142 | The first monitor shown is the **Fairness** monitor. It ensures that our model is not unfairly biased against a particular group. For this use case, we wanted to make sure that our model was treating online applications fairly. For more information, click the **Fairness** monitor section. 143 | 144 | ![Fairness monitor](./img/step_02/fairness.png) 145 | 146 | This screen shows fairness data computed hourly for the past week. Because we just built and fed data to our model, there won't be much data here yet. 147 | 148 | On the left, you can explore the different monitors OpenScale provides. The **Quality** monitor requires scored feedback data, and calculates nine different quality metrics. 149 | 150 | The **Drift** monitor compares the data that has been submitted to the model to the training data, and attempts to identify potential quality issues that may result if the submission data begins to vary too much from the training data. 151 | 152 | When you are finished exploring, return to the **Fairness** graph screen and click on the left-most point on the graph. 153 | 154 | ![Fairness graph](./img/step_02/fairness_graph.png) 155 | 156 | Here we can see the breakdown for predictions made during a specific time period. Click the **View Transactions** button. 157 | 158 | ![View transactions](./img/step_02/view_transactions.png) 159 | 160 | This screen provides a table with all the predictions made by the model during the selected time period. Find one you'd like to get more information about, and click the **Explain** link. 161 | 162 | ![Explain link](./img/step_02/explain.png) 163 | 164 | OpenScale's explanation service can take up to a minute to fully complete. When it finishes, it provides detailed information on which features from the submitted application influenced the model prediction. It can also tell you what changes need to be made to the request to result in a different prediction, and how much the request could change without altering the prediction. 165 | 166 | In the next section, we'll take a look at how these tools might work in a real-world application. 167 | 168 | ## 3. Build an application to use the model and explanations 169 | 170 | > **Note:** If you are having trouble with the ibmcloud command line tool in this section, you may wish to use the [Cloud Foundry command line tool](https://github.com/cloudfoundry/cli#downloads) instead. In that case, login with `cf login`, omit the `ibmcloud` portion from each command below, and skip the `ibmcloud cf install` and `ibmcloud target --cf` steps. You can also run the app from your local machine using [node.js](https://nodejs.org/). 171 | 172 | It's time to use the credentials you've been pasting in your text editor. Navigate to the directory where you extracted the files from the repo, and edit the **auth_example.js** file. Paste your Cloud API key, Watson Machine Learning service credentials, and the Watson OpenScale information from the notebook into the correct locations in the JavaScript file. 173 | 174 | When you are finished, save the file and then rename it to **auth.js**. 175 | 176 | Next, open up a terminal window and run the following command to install the Cloud Foundry tools: 177 | ```ibmcloud cf install``` 178 | 179 | Next, login to your IBM Cloud account: 180 | ```ibmcloud login``` 181 | 182 | IBM employees will login with: 183 | ```ibmcloud login --sso``` 184 | 185 | Follow any prompts from the tool to select your default accounts or workspaces. 186 | 187 | Enter the following command to target Cloud Foundry: 188 | ```ibmcloud target --cf``` 189 | 190 | The command line tool may prompt you to select an account or workspace. 191 | 192 | Within the terminal, navigate to the directory where you extracted the GitHub repo files. The directory file listing should look like this: 193 | ![Directory listing](./img/step_03/ls.png) 194 | 195 | Ensure that you've renamed the edited **auth_example.js** to **auth.js** to avoid breaking the app. 196 | 197 | Enter the following command, replacing **your-app-name** with a unique application name: 198 | ```ibmcloud cf push your-app-name -m 256m``` 199 | 200 | When the command successfully completes, you can navigate to [https://your-app-name.mybluemix.net](https://your-app-name.mybluemix.net) to use the running app. You can change the details of the mortgage application to see how it affects the model's prediction, and then use the OpenScale explainability service to see exactly what is influencing each decision. 201 | 202 | Note that the Lite version of OpenScale does limit the number of explanations you can generate each month. 203 | 204 | ## 4. Next steps 205 | 206 | Tune in May 15 for an AutoAI presentation from Jacques Roy! And subscribe to his excellent YouTube channel, [Byte Size Data Science](https://www.youtube.com/channel/UCC00OTwEMHA2jq50m4oJ95g). 207 | 208 | The model and application can be improved in several ways: 209 | 210 | * More tuning could be done to the model to improve its accuracy. 211 | * Continually feeding data into the model would provide more information for the monitors. [This notebook](https://github.com/emartensibm/mortgage-default/blob/master/mortgage_model_feed.ipynb) will allow you to schedule hourly requests. Before setting this up, take into account the limits of the free plans for Watson Machine Learning and Watson OpenScale. 212 | * The app could certainly use a visual upgrade. 213 | * As discussed in the [app source code](https://github.com/ericmartens/openscale_app/blob/master/main.js), it would be a good idea to render the model result and explanation result pages immediately with progress bars, then update the page when the results are available. 214 | * OpenScale's explanation service provides both LIME and contrastive information. Currently, the app only requests and displays the LIME information. It could be updated to provide both, which would allow the user to see what changes to the record would lead to a different prediction. 215 | -------------------------------------------------------------------------------- /auth_example.js: -------------------------------------------------------------------------------- 1 | //this file contains service credentials for your IBM services. replace the variables below 2 | //with your credentials, then rename this file to auth.js prior to pushing to IBM Cloud 3 | module.exports = { 4 | cloud: { 5 | apikey: "xxxxxPASTE HERExxxxx" 6 | }, 7 | wml: { 8 | "apikey": "key", 9 | "iam_apikey_description": "description", 10 | "iam_apikey_name": "auto-generated-apikey", 11 | "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer", 12 | "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::", 13 | "instance_id": "instance_id", 14 | "url": "https://us-south.ml.cloud.ibm.com", 15 | }, 16 | openscale: { 17 | guid: "xxxxxxxxxxxxxxxxxxxxxx", 18 | subscription: "xxxxxxx-xxxxxxx-xxxxxxx", 19 | scoring_url: "https://us-south.ml.cloud.ibm.com/v3/wml_instances/xxxxx/deployments/xxxxx/online" 20 | } 21 | }; -------------------------------------------------------------------------------- /data/Mortgage_Full_Records.csv: -------------------------------------------------------------------------------- 1 | ID,Income,AppliedOnline,Residence,Yrs_at_Current_Address,Yrs_with_Current_Employer,Number_of_Cards,Creditcard_Debt,Loans,Loan_Amount,SalePrice,Location,MortgageDefault 2 | 100537,45081,YES,Owner Occupier,14,15,2,713,1,8430,140000,L110,NO 3 | 100458,46645,YES,Owner Occupier,19,4,1,884,0,6045,475000,L110,NO 4 | 101432,44202,YES,Owner Occupier,1,23,2,2611,0,12915,162000,L101,NO 5 | 100599,52495,YES,Owner Occupier,18,16,2,2527,1,10375,195000,L100,YES 6 | 100782,43608,YES,Owner Occupier,2,20,1,452,0,7610,410000,L100,YES 7 | 101452,57689,YES,Owner Occupier,25,19,1,331,1,9915,315000,L100,YES 8 | 100494,55421,YES,Owner Occupier,11,3,2,1169,0,10715,245000,L130,YES 9 | 100697,51561,YES,Public Housing,17,5,2,3010,0,16630,359850,L100,NO 10 | 100423,49007,YES,Living with parents/guardian,14,18,1,13360,0,24375,200000,L100,YES 11 | 100695,58878,YES,Owner Occupier,18,9,1,1680,0,6165,260000,L100,YES 12 | 100590,59393,YES,Owner Occupier,34,18,2,2746,0,6845,400000,L100,YES 13 | 100516,45849,YES,Living with parents/guardian,13,9,2,364,1,12210,195000,L100,YES 14 | 100465,44048,YES,Owner Occupier,0,6,1,222,0,10155,299000,L101,NO 15 | 100625,45993,YES,Public Housing,6,7,2,311,0,8670,199000,L100,NO 16 | 101700,55140,YES,Owner Occupier,20,13,2,5095,0,8540,545000,L100,NO 17 | 100350,52973,YES,Public Housing,13,3,1,2535,0,9365,420000,L111,YES 18 | 100288,44956,YES,Owner Occupier,21,21,2,806,1,8150,170000,L100,YES 19 | 101477,43761,YES,Owner Occupier,10,6,1,1418,0,11245,192500,L100,YES 20 | 100596,47019,YES,Private Renting,11,21,1,1138,1,8645,345000,L110,YES 21 | 100377,51945,YES,Public Housing,19,0,1,25,1,7480,290000,L110,YES 22 | 100615,44298,YES,Owner Occupier,16,19,2,763,0,8410,170000,L120,YES 23 | 100416,59463,YES,Owner Occupier,2,9,2,1298,1,4765,222000,L100,NO 24 | 100642,43739,YES,Owner Occupier,3,7,1,2247,1,10275,285000,L100,YES 25 | 100394,44997,YES,Public Housing,6,18,1,1238,1,8070,176000,L100,NO 26 | 100562,45103,YES,Owner Occupier,21,14,1,2327,1,5575,185000,L100,YES 27 | 100785,43823,YES,Living with parents/guardian,14,1,2,2245,1,9795,175000,L130,NO 28 | 101531,45159,YES,Owner Occupier,6,12,1,2310,0,12065,190000,L100,NO 29 | 101394,44997,YES,Public Housing,6,18,1,1238,1,8070,176000,L100,YES 30 | 100505,45153,YES,Owner Occupier,2,13,2,804,1,8900,146000,L130,YES 31 | 100384,53195,YES,Owner Occupier,8,31,2,7757,1,13645,600000,L100,NO 32 | 101784,44261,YES,Owner Occupier,9,22,1,823,1,8010,323600,L100,YES 33 | 100293,48336,YES,Owner Occupier,16,2,2,1227,0,3705,575000,L120,YES 34 | 101746,45194,YES,Owner Occupier,15,10,2,1678,1,12915,273000,L120,NO 35 | 100641,44534,YES,Owner Occupier,9,13,2,805,0,8775,200000,L130,YES 36 | 101448,53201,YES,Owner Occupier,18,29,1,17134,0,16275,197307,L100,NO 37 | 100698,58785,YES,Private Renting,31,16,2,1111,0,7415,275000,L130,NO 38 | 100718,45240,YES,Private Renting,12,5,2,1320,1,5840,368000,L101,NO 39 | 100703,45368,YES,Owner Occupier,4,2,2,711,1,10340,325000,L100,NO 40 | 100323,46955,YES,Owner Occupier,16,1,2,201,1,3380,210000,L120,YES 41 | 100668,59692,YES,Owner Occupier,11,3,1,469,0,10100,190000,L100,NO 42 | 100597,53731,YES,Public Housing,26,16,2,13160,1,17985,365000,L110,NO 43 | 100640,43293,YES,Private Renting,16,7,1,2207,1,8450,260000,L100,YES 44 | 100468,45970,YES,Owner Occupier,11,7,1,579,0,9280,132000,L100,YES 45 | 100486,54938,YES,Owner Occupier,5,15,2,268,0,8225,235000,L130,NO 46 | 100432,44202,YES,Owner Occupier,1,23,2,2611,0,12915,162000,L101,NO 47 | 100386,54603,YES,Owner Occupier,8,30,2,1562,1,7760,150000,L100,NO 48 | 101505,45153,YES,Owner Occupier,2,13,2,804,1,8900,146000,L130,YES 49 | 100581,45841,YES,Public Housing,11,12,2,4159,0,13350,350000,L100,NO 50 | 101698,58785,YES,Private Renting,31,16,2,1111,0,7415,275000,L130,YES 51 | 100448,53201,YES,Owner Occupier,18,29,1,17134,0,16275,197307,L100,NO 52 | 100621,50621,YES,Owner Occupier,24,22,2,1047,0,9940,350000,L120,NO 53 | 100375,50808,YES,Private Renting,19,18,2,457,0,7775,125000,L101,YES 54 | 100353,43908,YES,Public Housing,18,15,1,4242,0,7915,190000,L100,NO 55 | 101789,43151,YES,Private Renting,6,9,1,681,1,7290,320000,L110,NO 56 | 100456,47026,YES,Private Renting,23,18,2,1368,0,6955,145000,L130,NO 57 | 100759,47035,YES,Living with parents/guardian,8,15,1,2003,0,10915,390000,L100,YES 58 | 100470,45949,YES,Private Renting,17,7,1,3865,0,13010,149000,L100,YES 59 | 101570,55565,YES,Owner Occupier,23,0,2,2219,0,11885,440000,L110,YES 60 | 100716,45217,YES,Owner Occupier,17,8,2,445,1,11455,315000,L120,YES 61 | 100729,43146,YES,Owner Occupier,0,10,1,679,1,10420,105000,L101,YES 62 | 100443,43874,YES,Owner Occupier,18,0,2,1320,1,10295,258000,L100,YES 63 | 100691,44898,YES,Owner Occupier,4,22,1,527,0,10270,288500,L100,NO 64 | 100784,44261,YES,Owner Occupier,9,22,1,823,1,8010,323600,L100,YES 65 | 101377,51945,YES,Public Housing,19,0,1,25,1,7480,290000,L110,YES 66 | 101703,45368,YES,Owner Occupier,4,2,2,711,1,10340,325000,L100,YES 67 | 100292,56087,YES,Public Housing,27,18,1,4818,1,8910,95000,L101,NO 68 | 100477,43761,YES,Owner Occupier,10,6,1,1418,0,11245,192500,L100,NO 69 | 100683,45549,YES,Owner Occupier,2,21,1,5791,0,13325,155500,L100,YES 70 | 100650,45283,YES,Public Housing,15,3,1,1460,1,13075,167000,L100,YES 71 | 100588,45266,YES,Public Housing,3,5,1,6805,0,21360,550000,L100,YES 72 | 100700,55140,YES,Owner Occupier,20,13,2,5095,0,8540,545000,L100,YES 73 | 100604,49407,YES,Owner Occupier,23,13,2,1793,0,8950,180000,L100,YES 74 | 101739,43185,YES,Living with parents/guardian,0,8,1,20,0,9420,305000,L130,NO 75 | 100412,56891,YES,Owner Occupier,26,1,1,1543,1,5545,570000,L100,NO 76 | 101388,48511,YES,Owner Occupier,21,25,1,793,0,9550,260000,L120,YES 77 | 100521,44394,YES,Owner Occupier,20,17,1,2041,0,10100,287500,L100,YES 78 | 101399,44617,YES,Owner Occupier,4,10,2,154,1,925,275000,L100,YES 79 | 100733,45318,YES,Private Renting,2,3,1,461,1,7665,305000,L130,YES 80 | 100326,55897,YES,Private Renting,13,2,2,315,0,9690,240000,L120,YES 81 | 100506,43820,YES,Owner Occupier,3,3,2,230,0,12095,395000,L110,NO 82 | 100422,47623,YES,Private Renting,23,7,1,784,1,8630,290000,L100,YES 83 | 100724,43348,YES,Sheltered,12,5,2,608,1,11720,200000,L120,YES 84 | 100761,53102,YES,Owner Occupier,15,1,1,1147,0,7590,215000,L100,YES 85 | 101468,45970,YES,Owner Occupier,11,7,1,579,0,9280,132000,L100,NO 86 | 100679,45584,YES,Private Renting,2,19,1,1256,0,4150,474000,L110,YES 87 | 100473,43750,YES,Private Renting,13,6,1,988,1,7060,443000,L100,NO 88 | 100510,43976,YES,Public Housing,18,12,2,745,0,5145,350000,L100,NO 89 | 100580,44477,YES,Private Renting,19,15,1,351,0,8290,220000,L130,NO 90 | 101470,45949,YES,Private Renting,17,7,1,3865,0,13010,149000,L100,YES 91 | 101353,43908,YES,Public Housing,18,15,1,4242,0,7915,190000,L100,YES 92 | 101386,54603,YES,Owner Occupier,8,30,2,1562,1,7760,150000,L100,NO 93 | 101494,55421,YES,Owner Occupier,11,3,2,1169,0,10715,245000,L130,YES 94 | 100320,44483,YES,Owner Occupier,16,15,2,116,0,7535,245000,L130,YES 95 | 101753,53550,YES,Public Housing,11,23,1,544,1,7260,250000,L120,YES 96 | 101458,46645,YES,Owner Occupier,19,4,1,884,0,6045,475000,L110,NO 97 | 100681,44582,YES,Owner Occupier,1,13,1,1581,1,10410,238000,L101,NO 98 | 101423,49007,YES,Living with parents/guardian,14,18,1,13360,0,24375,200000,L100,YES 99 | 100753,53550,YES,Public Housing,11,23,1,544,1,7260,250000,L120,YES 100 | 100452,57689,YES,Owner Occupier,25,19,1,331,1,9915,315000,L100,YES 101 | 100493,50515,YES,Owner Occupier,3,5,1,1642,1,12725,106300,L130,YES 102 | 101588,45266,YES,Public Housing,3,5,1,6805,0,21360,550000,L100,YES 103 | 101456,47026,YES,Private Renting,23,18,2,1368,0,6955,145000,L130,NO 104 | 100539,47470,YES,Private Renting,9,12,1,2111,0,9255,104000,L130,NO 105 | 101338,45813,YES,Owner Occupier,10,13,1,59,0,7385,103000,L101,YES 106 | 100592,55390,YES,Living with parents/guardian,23,3,1,812,0,7180,231000,L130,NO 107 | 100324,51383,YES,Owner Occupier,19,12,2,1119,1,7955,222500,L101,NO 108 | 100322,50295,YES,Owner Occupier,3,12,1,298,0,6140,950000,L110,NO 109 | 100789,43151,YES,Private Renting,6,9,1,681,1,7290,320000,L110,YES 110 | 100338,45813,YES,Owner Occupier,10,13,1,59,0,7385,103000,L101,YES 111 | 100587,45246,YES,Living with parents/guardian,6,5,1,737,1,5300,175000,L130,YES 112 | 100758,58914,YES,Living with parents/guardian,11,7,1,1562,1,9490,260000,L100,YES 113 | 101599,52495,YES,Owner Occupier,18,16,2,2527,1,10375,195000,L100,YES 114 | 101324,51383,YES,Owner Occupier,19,12,2,1119,1,7955,222500,L101,NO 115 | 101412,56891,YES,Owner Occupier,26,1,1,1543,1,5545,570000,L100,NO 116 | 100799,49866,YES,Owner Occupier,10,24,1,2493,0,9340,2095000,L101,NO 117 | 100531,45159,YES,Owner Occupier,6,12,1,2310,0,12065,190000,L100,NO 118 | 100361,44613,YES,Public Housing,17,11,2,2134,0,4215,500000,L130,YES 119 | 100746,45194,YES,Owner Occupier,15,10,2,1678,1,12915,273000,L120,YES 120 | 101592,55390,YES,Living with parents/guardian,23,3,1,812,0,7180,231000,L130,YES 121 | 100727,51939,YES,Public Housing,5,2,1,382,0,11090,105000,L101,YES 122 | 100344,49745,YES,Private Renting,24,6,1,518,1,5915,170000,L120,NO 123 | 100645,43648,YES,Owner Occupier,14,12,1,2496,1,6095,219000,L100,YES 124 | 101320,44483,YES,Owner Occupier,16,15,2,116,0,7535,245000,L130,YES 125 | 101716,45217,YES,Owner Occupier,17,8,2,445,1,11455,315000,L120,YES 126 | 100693,48414,YES,Public Housing,9,0,2,232,0,9240,250000,L110,YES 127 | 100409,43957,YES,Public Housing,18,13,2,286,1,7810,350000,L110,NO 128 | 100706,46965,YES,Public Housing,21,0,1,83,1,4070,498500,L100,YES 129 | 100378,49075,YES,Owner Occupier,7,16,2,133,0,4900,149900,L101,YES 130 | 100388,48511,YES,Owner Occupier,21,25,1,793,0,9550,260000,L120,YES 131 | 100739,43185,YES,Living with parents/guardian,0,8,1,20,0,9420,305000,L130,NO 132 | 101580,44477,YES,Private Renting,19,15,1,351,0,8290,220000,L130,YES 133 | 100570,55565,YES,Owner Occupier,23,0,2,2219,0,11885,440000,L110,YES 134 | 100399,44617,YES,Owner Occupier,4,10,2,154,1,925,275000,L100,YES 135 | 101537,45081,YES,Owner Occupier,14,15,2,713,1,8430,140000,L110,YES 136 | 100522,43982,NO,Owner Occupier,13,11,2,1055,0,9405,500000,L110,NO 137 | 101756,59944,NO,Owner Occupier,20,11,2,3894,0,9880,750000,L110,NO 138 | 101354,57718,NO,Owner Occupier,25,16,2,1555,1,6285,155000,L130,YES 139 | 100512,45621,NO,Owner Occupier,1,19,1,1878,0,9260,195000,L100,YES 140 | 101430,45066,NO,Private Renting,16,15,1,860,1,6605,255000,L100,YES 141 | 100601,55215,NO,Private Renting,1,6,2,1930,0,9820,905000,L110,NO 142 | 101549,44460,NO,Owner Occupier,4,16,1,3467,1,9290,192000,L100,YES 143 | 101524,57004,NO,Private Renting,14,31,1,3561,0,6720,121000,L100,NO 144 | 100548,44272,NO,Owner Occupier,12,20,1,645,1,10095,170000,L100,YES 145 | 101472,45659,NO,Public Housing,12,17,1,9466,1,17050,340000,L100,NO 146 | 100536,45450,NO,Owner Occupier,15,24,1,4217,0,8040,270000,L100,YES 147 | 100627,44671,NO,Private Renting,16,9,2,3971,1,12050,270000,L130,YES 148 | 101754,50816,NO,Owner Occupier,0,17,2,2031,1,8075,300000,L130,YES 149 | 101749,43876,NO,Private Renting,13,0,2,98,1,5330,252500,L100,NO 150 | 100620,44705,NO,Owner Occupier,0,8,1,230,0,5720,275000,L111,NO 151 | 100569,46168,NO,Private Renting,0,6,1,3762,1,12520,385000,L120,YES 152 | 100756,59944,NO,Owner Occupier,20,11,2,3894,0,9880,750000,L110,NO 153 | 100563,55112,NO,Living with parents/guardian,10,14,2,4113,1,14510,265000,L100,YES 154 | 100677,44728,NO,Private Renting,2,7,2,847,0,11580,237500,L110,NO 155 | 101722,45390,NO,Owner Occupier,3,2,2,154,0,11710,242500,L110,NO 156 | 100439,43890,NO,Owner Occupier,19,18,2,505,1,6960,348500,L100,NO 157 | 101363,44193,NO,Public Housing,18,3,1,260,0,6605,284500,L120,NO 158 | 100325,52540,NO,Sheltered,0,15,2,397,0,8250,215000,L120,YES 159 | 100532,48931,NO,Owner Occupier,11,19,2,652,0,7795,190000,L100,NO 160 | 100589,43362,NO,Owner Occupier,15,4,2,434,1,4320,650000,L391,YES 161 | 101764,44787,NO,Owner Occupier,9,4,2,982,0,8175,295000,L100,NO 162 | 100704,43651,NO,Private Renting,4,11,2,527,1,6090,265000,L100,NO 163 | 100741,45001,NO,Public Housing,11,17,1,2134,1,9285,370000,L110,YES 164 | 100472,45659,NO,Public Housing,12,17,1,9466,1,17050,340000,L100,NO 165 | 100368,43474,NO,Living with parents/guardian,17,1,1,270,1,5315,125000,L100,NO 166 | 101635,44039,NO,Owner Occupier,17,6,2,109,1,6590,235000,L120,YES 167 | 100635,44039,NO,Owner Occupier,17,6,2,109,1,6590,235000,L120,YES 168 | 100411,47833,NO,Owner Occupier,9,6,1,854,0,12920,200000,L130,NO 169 | 100530,43303,NO,Owner Occupier,2,7,2,866,1,10220,75000,L100,NO 170 | 100721,43669,NO,Living with parents/guardian,18,10,1,2518,0,12500,325000,L100,YES 171 | 100676,44597,NO,Owner Occupier,10,12,2,468,0,8630,135000,L100,NO 172 | 100793,50552,NO,Owner Occupier,5,23,1,466,0,6880,380000,L110,YES 173 | 100283,43800,NO,Owner Occupier,0,4,2,725,0,7340,259000,L100,NO 174 | 101694,53307,NO,Owner Occupier,8,27,2,1984,1,9475,130000,L100,NO 175 | 100594,49529,NO,Owner Occupier,9,10,1,342,0,7995,345000,L110,YES 176 | 100337,45209,NO,Owner Occupier,4,9,2,658,0,11420,185000,L101,YES 177 | 100284,45049,NO,Public Housing,6,16,2,1345,1,9085,280000,L110,NO 178 | 101360,44052,NO,Owner Occupier,13,5,2,375,0,9305,235000,L100,YES 179 | 100598,43514,NO,Private Renting,17,0,2,274,0,3425,250000,L130,YES 180 | 100421,56486,NO,Owner Occupier,5,11,1,696,1,4240,110000,L100,NO 181 | 100491,46785,NO,Private Renting,14,4,1,1227,1,6510,300000,L100,NO 182 | 101382,49008,NO,Private Renting,10,16,2,216,0,10910,145000,L130,NO 183 | 100382,49008,NO,Private Renting,10,16,2,216,0,10910,145000,L130,NO 184 | 100633,43120,NO,Public Housing,4,10,1,801,1,6220,215000,L100,YES 185 | 100535,49988,NO,Private Renting,14,21,2,2660,0,9645,210000,L100,NO 186 | 101450,54673,NO,Owner Occupier,15,22,2,3086,1,11875,180000,L100,NO 187 | 100567,57683,NO,Owner Occupier,14,24,1,3275,1,10185,281915,L100,YES 188 | 100273,45706,NO,Owner Occupier,17,16,2,373,1,7275,145000,L100,YES 189 | 100454,43799,NO,Private Renting,11,4,1,145,0,8810,220000,L100,NO 190 | 101405,43482,NO,Owner Occupier,4,1,1,361,1,8375,499250,L100,NO 191 | 100492,53365,NO,Owner Occupier,21,26,1,3819,0,10250,230000,L100,YES 192 | 100415,53791,NO,Owner Occupier,1,15,1,1844,1,11855,173900,L100,YES 193 | 100511,43835,NO,Owner Occupier,3,1,2,1501,1,12220,144000,L100,NO 194 | 100280,44202,NO,Owner Occupier,8,0,2,748,0,10455,170000,L100,NO 195 | 101385,52896,NO,Owner Occupier,19,6,2,1655,1,8975,260000,L110,YES 196 | 100485,45673,NO,Owner Occupier,22,16,1,359,0,6670,185000,L100,NO 197 | 100385,52896,NO,Owner Occupier,19,6,2,1655,1,8975,260000,L110,YES 198 | 101798,47111,NO,Private Renting,1,14,2,1001,1,10510,425000,L110,YES 199 | 100279,44756,NO,Owner Occupier,19,6,1,2117,1,10760,145000,L110,NO 200 | 100578,45750,NO,Public Housing,13,13,1,1930,1,11355,209000,L101,NO 201 | 101797,51385,NO,Owner Occupier,14,10,1,777,0,7430,260000,L101,NO 202 | 100795,53842,NO,Owner Occupier,25,13,2,3115,0,6150,625000,L100,NO 203 | 100527,48061,NO,Private Renting,3,5,2,296,0,9680,190500,L100,YES 204 | 100571,50076,NO,Private Renting,0,17,1,460,1,9385,675000,L130,NO 205 | 100605,49415,NO,Owner Occupier,17,11,1,150,0,6485,180000,L101,YES 206 | 100533,44363,NO,Owner Occupier,5,19,1,1609,1,7325,250000,L110,YES 207 | 100306,43650,NO,Public Housing,8,12,1,267,0,7800,355000,L100,NO 208 | 100390,53559,NO,Owner Occupier,7,19,2,2646,0,10105,537500,L100,NO 209 | 100623,48599,NO,Owner Occupier,16,25,1,1363,1,9385,447000,L100,YES 210 | 101696,43977,NO,Public Housing,14,12,1,4763,1,11145,330000,L120,NO 211 | 100568,55752,NO,Owner Occupier,5,11,2,2048,1,6685,565000,L100,YES 212 | 101704,43651,NO,Private Renting,4,11,2,527,1,6090,265000,L100,NO 213 | 100381,53115,NO,Owner Occupier,12,1,2,674,0,10455,309534,L110,YES 214 | 100603,52173,NO,Public Housing,4,22,2,689,0,9720,140000,L100,YES 215 | 100345,53552,NO,Owner Occupier,19,22,2,1255,1,4430,159900,L100,YES 216 | 100496,44449,NO,Private Renting,9,16,1,4168,0,8450,258000,L100,YES 217 | 101757,54451,NO,Private Renting,14,30,1,3312,1,9440,150000,L100,NO 218 | 101419,51219,NO,Private Renting,14,15,2,1292,1,7005,165000,L111,YES 219 | 100565,46823,NO,Private Renting,22,2,1,94,1,6575,281915,L100,NO 220 | 101707,46119,NO,Private Renting,1,6,1,104,0,11470,145000,L100,YES 221 | 100629,54671,NO,Owner Occupier,25,28,2,3612,0,8610,250000,L120,YES 222 | 101497,55777,NO,Private Renting,1,11,2,453,0,6215,285000,L100,YES 223 | 100657,59276,NO,Public Housing,20,21,2,8000,1,8265,285000,L100,NO 224 | 100490,50799,NO,Owner Occupier,22,19,2,1919,0,6020,185000,L111,YES 225 | 100797,51385,NO,Owner Occupier,14,10,1,777,0,7430,260000,L101,NO 226 | 100272,43593,NO,Owner Occupier,13,0,1,2315,0,12820,180000,L130,NO 227 | 101763,45452,NO,Owner Occupier,12,24,1,5490,1,10040,328000,L100,NO 228 | 100488,54021,NO,Owner Occupier,2,4,1,101,1,12230,355000,L100,YES 229 | 100602,44892,NO,Private Renting,12,2,2,3337,0,11195,175000,L110,NO 230 | 101624,43677,NO,Sheltered,16,9,1,740,0,10915,190000,L100,NO 231 | 100334,44895,NO,Owner Occupier,8,0,2,888,1,8915,310000,L100,YES 232 | 100497,55777,NO,Private Renting,1,11,2,453,0,6215,285000,L100,YES 233 | 100285,44974,NO,Public Housing,14,18,2,2772,0,9515,264000,L130,NO 234 | 100294,55642,NO,Private Renting,4,22,2,240,0,5925,349900,L130,NO 235 | 101731,49789,NO,Owner Occupier,2,4,2,44,0,9340,200000,L100,NO 236 | 101453,45869,NO,Private Renting,21,8,2,71,1,9240,325000,L100,YES 237 | 101532,48931,NO,Owner Occupier,11,19,2,652,0,7795,190000,L100,NO 238 | 100449,45238,NO,Owner Occupier,9,6,2,326,1,5515,160000,L120,YES 239 | 100425,55497,NO,Owner Occupier,7,3,1,637,1,8980,229500,L100,YES 240 | 100760,43794,NO,Owner Occupier,9,6,2,1430,0,9020,143000,L101,NO 241 | 101282,45715,NO,Owner Occupier,8,14,2,772,1,12985,137000,L100,NO 242 | 100671,51859,NO,Private Renting,19,2,2,369,1,5290,360000,L110,YES 243 | 101491,46785,NO,Private Renting,14,4,1,1227,1,6510,300000,L100,NO 244 | 100705,49485,NO,Private Renting,24,10,1,563,1,4455,250000,L100,YES 245 | 100624,43677,NO,Sheltered,16,9,1,740,0,10915,190000,L100,NO 246 | 100702,59201,NO,Owner Occupier,12,25,1,12164,0,14795,545000,L100,YES 247 | 100730,57388,NO,Private Renting,23,22,1,3450,0,5365,105000,L101,YES 248 | 100374,56849,NO,Owner Occupier,4,4,1,117,1,5725,117000,L100,NO 249 | 100403,44831,NO,Owner Occupier,15,2,1,1597,1,8790,161000,L100,YES 250 | 100389,59036,NO,Owner Occupier,7,6,1,495,1,8915,193000,L101,YES 251 | 100369,43322,NO,Private Renting,15,6,2,218,1,6940,325000,L100,NO 252 | 100661,47568,NO,Private Renting,5,11,2,2391,1,12965,285000,L100,YES 253 | 100754,50816,NO,Owner Occupier,0,17,2,2031,1,8075,300000,L130,YES 254 | 101303,43795,NO,Public Housing,17,5,2,1171,0,11285,153000,L100,YES 255 | 101383,54013,NO,Private Renting,9,4,1,996,0,11105,156000,L100,NO 256 | 100662,50833,NO,Public Housing,21,16,1,760,1,11585,270000,L100,YES 257 | 100626,52887,NO,Public Housing,10,9,2,1339,0,8560,185000,L130,YES 258 | 100383,54013,NO,Private Renting,9,4,1,996,0,11105,156000,L100,NO 259 | 101285,44974,NO,Public Housing,14,18,2,2772,0,9515,264000,L130,NO 260 | 101732,47623,NO,Owner Occupier,12,7,2,1358,1,7365,275000,L130,YES 261 | 101701,46501,NO,Private Renting,5,5,2,327,1,9070,325000,L100,NO 262 | 100436,45518,NO,Private Renting,17,22,1,8231,1,16485,260000,L130,NO 263 | 101672,54777,NO,Public Housing,26,17,2,2814,1,10895,260000,L100,NO 264 | 100696,43977,NO,Public Housing,14,12,1,4763,1,11145,330000,L120,NO 265 | 100736,44005,NO,Private Renting,16,8,2,25,1,6920,179000,L130,NO 266 | 101670,56470,NO,Private Renting,31,15,2,390,0,7055,240000,L110,YES 267 | 100330,45217,NO,Private Renting,19,8,1,732,0,5860,240000,L120,YES 268 | 100765,51498,NO,Private Renting,26,9,1,589,1,10150,295000,L100,NO 269 | 101407,43989,NO,Owner Occupier,8,10,1,30,1,5805,410000,L101,NO 270 | 100767,51724,NO,Owner Occupier,15,3,1,88,1,7150,140000,L101,YES 271 | 101760,43794,NO,Owner Occupier,9,6,2,1430,0,9020,143000,L101,NO 272 | 100766,57376,NO,Private Renting,21,27,2,684,1,9845,540000,L110,YES 273 | 100796,58381,NO,Private Renting,16,12,1,829,0,11750,455000,L100,NO 274 | 100453,45869,NO,Private Renting,21,8,2,71,1,9240,325000,L100,YES 275 | 100442,44271,NO,Private Renting,17,20,1,284,1,6185,200000,L130,YES 276 | 100561,45721,NO,Living with parents/guardian,3,14,1,294,1,7925,245000,L100,NO 277 | 100500,50302,NO,Owner Occupier,20,9,2,455,0,10490,535000,L100,YES 278 | 101512,45621,NO,Owner Occupier,1,19,1,1878,0,9260,195000,L100,YES 279 | 101503,44095,NO,Private Renting,13,4,1,368,0,8730,285000,L100,NO 280 | 100549,44460,NO,Owner Occupier,4,16,1,3467,1,9290,192000,L100,YES 281 | 100551,43707,NO,Owner Occupier,8,9,1,1131,0,9365,125000,L100,NO 282 | 101602,44892,NO,Private Renting,12,2,2,3337,0,11195,175000,L110,NO 283 | 100444,43888,NO,Owner Occupier,16,21,1,3820,1,7240,220000,L100,YES 284 | 100414,58026,NO,Living with parents/guardian,17,13,1,6156,0,15905,153000,L100,YES 285 | 100792,44230,NO,Owner Occupier,9,22,2,3106,0,540,350000,L100,NO 286 | 101439,43890,NO,Owner Occupier,19,18,2,505,1,6960,348500,L100,NO 287 | 100343,45687,NO,Public Housing,22,16,2,1064,1,8465,349000,L100,NO 288 | 101389,59036,NO,Owner Occupier,7,6,1,495,1,8915,193000,L101,YES 289 | 100794,53704,NO,Owner Occupier,21,17,2,490,1,2825,425000,L100,YES 290 | 100579,45904,NO,Owner Occupier,22,7,1,60,0,4850,110000,L110,NO 291 | 101347,55554,NO,Public Housing,27,0,1,708,0,8615,235000,L100,NO 292 | 101376,58505,NO,Owner Occupier,11,9,2,668,0,5370,150000,L100,NO 293 | 100755,53660,NO,Private Renting,23,17,1,380,1,7945,378000,L100,NO 294 | 100424,43895,NO,Owner Occupier,7,18,1,446,1,9085,400000,L110,NO 295 | 100405,43482,NO,Owner Occupier,4,1,1,361,1,8375,499250,L100,NO 296 | 100722,45390,NO,Owner Occupier,3,2,2,154,0,11710,242500,L110,NO 297 | 101295,52725,NO,Private Renting,26,9,2,12,0,4415,98000,L100,NO 298 | 100694,53307,NO,Owner Occupier,8,27,2,1984,1,9475,130000,L100,NO 299 | 101272,43593,NO,Owner Occupier,13,0,1,2315,0,12820,180000,L130,NO 300 | 100445,44194,NO,Private Renting,16,2,2,1007,1,8550,565000,L120,YES 301 | 100802,48607,NO,Owner Occupier,19,23,2,985,1,4415,170000,L100,NO 302 | 100665,57520,NO,Private Renting,25,19,1,890,0,11950,430000,L121,NO 303 | 101802,48607,NO,Owner Occupier,19,23,2,985,1,4415,170000,L100,NO 304 | 100311,43330,NO,Owner Occupier,9,6,1,25,1,9630,355000,L100,NO 305 | 100321,43185,NO,Owner Occupier,1,9,2,1646,0,8125,235000,L100,NO 306 | 101593,50445,NO,Public Housing,10,8,1,237,1,9310,250000,L120,YES 307 | 100534,44413,NO,Private Renting,13,16,2,885,0,4665,165000,L100,YES 308 | 100558,45419,NO,Private Renting,1,2,2,231,0,9200,131500,L100,NO 309 | 100745,45313,NO,Owner Occupier,13,3,2,341,1,4065,290000,L130,YES 310 | 101330,45217,NO,Private Renting,19,8,1,732,0,5860,240000,L120,YES 311 | 100752,44395,NO,Owner Occupier,20,16,2,1299,1,8750,290000,L100,YES 312 | 100660,49850,NO,Living with parents/guardian,5,1,2,1265,1,4535,455000,L100,NO 313 | 100737,44306,NO,Owner Occupier,15,19,2,7995,1,13220,275000,L130,YES 314 | 100593,50445,NO,Public Housing,10,8,1,237,1,9310,250000,L120,YES 315 | 100499,47422,NO,Owner Occupier,11,12,2,2010,1,12315,330000,L100,YES 316 | 101365,45115,NO,Private Renting,11,13,2,5189,1,16935,164000,L101,NO 317 | 100347,55554,NO,Public Housing,27,0,1,708,0,8615,235000,L100,NO 318 | 100749,43876,NO,Private Renting,13,0,2,98,1,5330,252500,L100,NO 319 | 101660,49850,NO,Living with parents/guardian,5,1,2,1265,1,4535,455000,L100,NO 320 | 101495,47734,NO,Sheltered,23,6,2,2631,1,14355,150000,L100,NO 321 | 101730,57388,NO,Private Renting,23,22,1,3450,0,5365,105000,L101,YES 322 | 100348,54792,NO,Owner Occupier,22,17,1,1223,0,6915,254500,L130,YES 323 | 100622,49600,NO,Private Renting,5,9,2,3366,1,9065,330000,L100,NO 324 | 101770,53983,NO,Public Housing,1,9,2,2117,0,9585,319000,L100,NO 325 | 100376,58505,NO,Owner Occupier,11,9,2,668,0,5370,150000,L100,NO 326 | 101334,44895,NO,Owner Occupier,8,0,2,888,1,8915,310000,L100,YES 327 | 100519,43408,NO,Owner Occupier,10,4,2,1106,0,10605,182500,L130,NO 328 | 100524,57004,NO,Private Renting,14,31,1,3561,0,6720,121000,L100,NO 329 | 100701,46501,NO,Private Renting,5,5,2,327,1,9070,325000,L100,NO 330 | 101519,43408,NO,Owner Occupier,10,4,2,1106,0,10605,182500,L130,NO 331 | 101421,56486,NO,Owner Occupier,5,11,1,696,1,4240,110000,L100,NO 332 | 100790,44072,NO,Owner Occupier,5,5,1,1250,0,10590,275000,L100,YES 333 | 100595,47161,NO,Public Housing,0,13,2,1096,1,4855,365000,L110,NO 334 | 101600,45537,NO,Private Renting,22,21,2,2698,1,8885,1000000,L110,YES 335 | 100770,53983,NO,Public Housing,1,9,2,2117,0,9585,319000,L100,NO 336 | 100684,45712,NO,Owner Occupier,14,15,2,2216,0,8910,215000,L100,NO 337 | 100282,45715,NO,Owner Occupier,8,14,2,772,1,12985,137000,L100,NO 338 | 101626,52887,NO,Public Housing,10,9,2,1339,0,8560,185000,L130,YES 339 | 100476,44318,NO,Owner Occupier,8,19,1,2096,0,9240,150876,L100,NO 340 | 101627,44671,NO,Private Renting,16,9,2,3971,1,12050,270000,L130,YES 341 | 100402,44684,NO,Owner Occupier,6,9,1,194,1,3965,1290000,L110,NO 342 | 100591,56523,NO,Public Housing,20,10,1,2066,0,9340,157500,L131,YES 343 | 101415,53791,NO,Owner Occupier,1,15,1,1844,1,11855,173900,L100,YES 344 | 100440,44772,NO,Owner Occupier,9,6,1,469,0,7950,1060000,L101,NO 345 | 101682,44270,NO,Owner Occupier,3,22,2,962,1,7845,200000,L111,NO 346 | 100398,44187,NO,Owner Occupier,19,3,1,693,0,9310,185000,L100,NO 347 | 100489,48084,NO,Owner Occupier,10,3,2,555,1,10500,265000,L100,YES 348 | 100670,56470,NO,Private Renting,31,15,2,390,0,7055,240000,L110,YES 349 | 100352,52961,NO,Public Housing,17,4,1,1853,1,11950,125000,L130,YES 350 | 100763,45452,NO,Owner Occupier,12,24,1,5490,1,10040,328000,L100,NO 351 | 100618,45463,NO,Owner Occupier,2,23,2,478,1,13350,215000,L100,NO 352 | 100725,57598,NO,Public Housing,25,16,1,2171,1,5140,207000,L100,NO 353 | 100319,59193,NO,Owner Occupier,26,0,2,2295,1,7350,525000,L100,NO 354 | 100600,45537,NO,Private Renting,22,21,2,2698,1,8885,1000000,L110,YES 355 | 101561,45721,NO,Living with parents/guardian,3,14,1,294,1,7925,245000,L100,NO 356 | 100764,44787,NO,Owner Occupier,9,4,2,982,0,8175,295000,L100,NO 357 | 101306,43650,NO,Public Housing,8,12,1,267,0,7800,355000,L100,NO 358 | 100798,47111,NO,Private Renting,1,14,2,1001,1,10510,425000,L110,YES 359 | 100418,59508,NO,Owner Occupier,2,19,1,3671,1,10595,135000,L130,NO 360 | 100732,47623,NO,Owner Occupier,12,7,2,1358,1,7365,275000,L130,YES 361 | 101578,45750,NO,Public Housing,13,13,1,1930,1,11355,209000,L101,NO 362 | 100672,54777,NO,Public Housing,26,17,2,2814,1,10895,260000,L100,NO 363 | 101492,53365,NO,Owner Occupier,21,26,1,3819,0,10250,230000,L100,YES 364 | 100450,54673,NO,Owner Occupier,15,22,2,3086,1,11875,180000,L100,NO 365 | 100373,49678,NO,Owner Occupier,11,8,2,564,0,7195,275000,L100,NO 366 | 100566,56590,NO,Owner Occupier,27,8,2,332,0,6605,265000,L100,YES 367 | 100346,52118,NO,Private Renting,17,22,2,2771,1,8390,165000,L100,YES 368 | 100451,46949,NO,Public Housing,0,2,2,1634,1,9720,300000,L110,NO 369 | 100688,45654,NO,Public Housing,6,18,2,5471,0,6915,282000,L130,YES 370 | 100731,49789,NO,Owner Occupier,2,4,2,44,0,9340,200000,L100,NO 371 | 100529,50075,NO,Private Renting,4,18,2,3768,0,6095,150000,L110,NO 372 | 100430,45066,NO,Private Renting,16,15,1,860,1,6605,255000,L100,YES 373 | 101720,44682,NO,Private Renting,12,9,1,1192,1,7320,475000,L110,YES 374 | 101280,44202,NO,Owner Occupier,8,0,2,748,0,10455,170000,L100,NO 375 | 101598,43514,NO,Private Renting,17,0,2,274,0,3425,250000,L130,YES 376 | 101684,45712,NO,Owner Occupier,14,15,2,2216,0,8910,215000,L100,NO 377 | 100630,53180,NO,Owner Occupier,9,31,2,11860,0,16335,335000,L100,YES 378 | 100768,52772,NO,Owner Occupier,23,9,2,1348,1,8340,285000,L100,NO 379 | 100363,44193,NO,Public Housing,18,3,1,260,0,6605,284500,L120,NO 380 | 101676,44597,NO,Owner Occupier,10,12,2,468,0,8630,135000,L100,NO 381 | 100407,43989,NO,Owner Occupier,8,10,1,30,1,5805,410000,L101,NO 382 | 101414,58026,NO,Living with parents/guardian,17,13,1,6156,0,15905,153000,L100,YES 383 | 100360,44052,NO,Owner Occupier,13,5,2,375,0,9305,235000,L100,YES 384 | 100329,44781,NO,Owner Occupier,15,4,1,331,0,10165,150000,L100,NO 385 | 100728,57623,NO,Owner Occupier,1,10,1,4133,1,10995,105000,L101,NO 386 | 100303,43795,NO,Public Housing,17,5,2,1171,0,11285,153000,L100,YES 387 | 100495,47734,NO,Sheltered,23,6,2,2631,1,14355,150000,L100,NO 388 | 100781,44294,NO,Owner Occupier,9,20,2,3195,1,10825,211000,L130,YES 389 | 100417,44159,NO,Public Housing,6,4,1,1110,0,7735,210000,L100,YES 390 | 101737,44306,NO,Owner Occupier,15,19,2,7995,1,13220,275000,L130,YES 391 | 100419,51219,NO,Private Renting,14,15,2,1292,1,7005,165000,L111,YES 392 | 100585,44010,NO,Private Renting,4,7,2,542,1,10180,235000,L100,NO 393 | 100720,44682,NO,Private Renting,12,9,1,1192,1,7320,475000,L110,YES 394 | 100342,43202,NO,Owner Occupier,17,7,1,1412,1,8925,650000,L101,NO 395 | 100295,52725,NO,Private Renting,26,9,2,12,0,4415,98000,L100,NO 396 | 101325,52540,NO,Sheltered,0,15,2,397,0,8250,215000,L120,YES 397 | 100310,43972,NO,Private Renting,0,13,1,340,1,7410,525000,L100,YES 398 | 100682,44270,NO,Owner Occupier,3,22,2,962,1,7845,200000,L111,NO 399 | 100586,44378,NO,Private Renting,9,18,2,306,0,7445,582000,L120,YES 400 | 100523,44936,NO,Owner Occupier,1,22,1,4502,0,11015,360000,L110,NO 401 | 100769,58529,NO,Sheltered,0,9,1,1108,0,11775,275000,L100,NO 402 | 100367,43645,NO,Owner Occupier,5,13,2,502,0,4600,875000,L100,NO 403 | 100614,43622,NO,Owner Occupier,8,18,1,3828,1,5625,180000,L100,YES 404 | 100503,44095,NO,Private Renting,13,4,1,368,0,8730,285000,L100,NO 405 | 100664,50631,NO,Private Renting,26,21,2,2132,0,3110,299500,L100,NO 406 | 100666,59179,NO,Living with parents/guardian,29,5,1,5780,1,11885,220000,L111,NO 407 | 101793,50552,NO,Owner Occupier,5,23,1,466,0,6880,380000,L110,YES 408 | 100707,46119,NO,Private Renting,1,6,1,104,0,11470,145000,L100,YES 409 | 101551,43707,NO,Owner Occupier,8,9,1,1131,0,9365,125000,L100,NO 410 | 100757,54451,NO,Private Renting,14,30,1,3312,1,9440,150000,L100,NO 411 | 100526,52545,NO,Owner Occupier,25,14,1,1125,0,8060,147000,L100,YES 412 | 100396,44442,NO,Private Renting,12,16,1,578,0,7600,300000,L100,YES 413 | 100365,45115,NO,Private Renting,11,13,2,5189,1,16935,164000,L101,NO 414 | 100379,56087,NO,Private Renting,14,16,2,1386,1,9100,273500,L120,NO 415 | 100507,43633,NO,Public Housing,14,16,1,6100,0,7330,650000,L110,NO 416 | 101664,50631,NO,Private Renting,26,21,2,2132,0,3110,299500,L100,NO 417 | 100628,50199,NO,Private Renting,5,12,1,1230,0,10070,199000,L100,NO 418 | 100715,44150,NO,Owner Occupier,9,4,2,1038,1,9010,280000,L110,YES 419 | 100487,49409,NO,Owner Occupier,11,12,2,107,1,9745,190000,L130,NO 420 | 100354,57718,NO,Owner Occupier,25,16,2,1555,1,6285,155000,L130,YES 421 | -------------------------------------------------------------------------------- /img/step_00/add_credential.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/add_credential.png -------------------------------------------------------------------------------- /img/step_00/add_service.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/add_service.png -------------------------------------------------------------------------------- /img/step_00/copy_api_key.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/copy_api_key.png -------------------------------------------------------------------------------- /img/step_00/copy_credentials.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/copy_credentials.png -------------------------------------------------------------------------------- /img/step_00/create_api_confirm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/create_api_confirm.png -------------------------------------------------------------------------------- /img/step_00/create_api_key.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/create_api_key.png -------------------------------------------------------------------------------- /img/step_00/empty_project.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/empty_project.png -------------------------------------------------------------------------------- /img/step_00/existing_service_instance.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/existing_service_instance.png -------------------------------------------------------------------------------- /img/step_00/find_wml.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/find_wml.png -------------------------------------------------------------------------------- /img/step_00/machine_learning.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/machine_learning.png -------------------------------------------------------------------------------- /img/step_00/new_credential.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/new_credential.png -------------------------------------------------------------------------------- /img/step_00/new_project.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/new_project.png -------------------------------------------------------------------------------- /img/step_00/project_details.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/project_details.png -------------------------------------------------------------------------------- /img/step_00/service_credentials.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/service_credentials.png -------------------------------------------------------------------------------- /img/step_00/settings.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_00/settings.png -------------------------------------------------------------------------------- /img/step_01/add_to_project.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/add_to_project.png -------------------------------------------------------------------------------- /img/step_01/assets_tab.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/assets_tab.png -------------------------------------------------------------------------------- /img/step_01/cells.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/cells.png -------------------------------------------------------------------------------- /img/step_01/clone_or_download.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/clone_or_download.png -------------------------------------------------------------------------------- /img/step_01/copy_guid.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/copy_guid.png -------------------------------------------------------------------------------- /img/step_01/from_file.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/from_file.png -------------------------------------------------------------------------------- /img/step_01/mortgage_default_cloud.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/mortgage_default_cloud.png -------------------------------------------------------------------------------- /img/step_01/notebook.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/notebook.png -------------------------------------------------------------------------------- /img/step_01/run_all.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_01/run_all.png -------------------------------------------------------------------------------- /img/step_02/explain.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_02/explain.png -------------------------------------------------------------------------------- /img/step_02/fairness.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_02/fairness.png -------------------------------------------------------------------------------- /img/step_02/fairness_graph.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_02/fairness_graph.png -------------------------------------------------------------------------------- /img/step_02/mortgage_tile.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_02/mortgage_tile.png -------------------------------------------------------------------------------- /img/step_02/view_transactions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_02/view_transactions.png -------------------------------------------------------------------------------- /img/step_03/ls.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/img/step_03/ls.png -------------------------------------------------------------------------------- /main.js: -------------------------------------------------------------------------------- 1 | //import dependencies 2 | const axios = require('axios'); 3 | const express = require('express'); 4 | const hbs = require('hbs'); 5 | const auth = require('./auth.js'); 6 | const app = express(); 7 | 8 | //app settings. use handlebars for our view engine, and set up express 9 | app.set('view engine', 'hbs'); 10 | app.use(express.static('dist')); 11 | app.use(express.json()); 12 | app.use(express.urlencoded({extended: true})); 13 | 14 | //routes here 15 | app.get('/', (req, res) => res.render('index.hbs')); 16 | 17 | app.get('/application', (req, res) => { 18 | res.render('application.hbs'); 19 | }); 20 | 21 | app.post('/application_results', (req, res) => { 22 | //take the POST data submitted via the application form and send it to our machine 23 | //learning model for a prediction, then display the prediction details. in a production 24 | //app we'd render the page with a progress bar until the model had returned a prediction 25 | get_prediction(req.body).then((value) => { 26 | console.log(value.values[0]); 27 | let outcome_result = "LOW RISK"; 28 | let outcome_text = "Congratulations! Your application is likely to be approved!"; 29 | if (value.values[0][0] === "YES") { 30 | outcome_result = "HIGH RISK"; 31 | outcome_text = "Unfortunately, your application is not likely to be approved."; 32 | } 33 | let confidence = value.values[0][1][0]; 34 | if (value.values[0][1][1] > confidence) { 35 | confidence = value.values[0][1][1]; 36 | } 37 | confidence *= 100; 38 | res.render('application_results.hbs', {outcome_result, outcome_text, confidence}); 39 | }); 40 | }); 41 | 42 | app.get('/explain', (req, res) => { 43 | //get the last prediction made by the model, and explain it. in a production app, we would need 44 | //to make sure we're explaining the relevant prediction and not just the last one. also, in a 45 | //production app we'd render the page with a loading bar while we waited for the explanation 46 | //service to finish. 47 | get_explanation().then((value) => { 48 | console.log(value); 49 | 50 | //make the confidence and weight values pretty for display 51 | value.predictions[0].probability = Math.round(value.predictions[0].probability * 1000) / 10 52 | value.predictions[1].probability = Math.round(value.predictions[1].probability * 1000) / 10 53 | 54 | if (value.predictions[0].hasOwnProperty("explanation_features")) { 55 | for (let i = 0; i < value.predictions[0].explanation_features.length; i++) { 56 | value.predictions[0].explanation_features[i].weight = Math.round(value.predictions[0].explanation_features[i].weight * 1000) / 10; 57 | } 58 | } 59 | 60 | if (value.predictions[1].hasOwnProperty("explanation_features")) { 61 | for (let i = 0; i < value.predictions[1].explanation_features.length; i++) { 62 | value.predictions[1].explanation_features[i].weight = Math.round(value.predictions[1].explanation_features[i].weight * 1000) / 10; 63 | } 64 | } 65 | 66 | res.render('explain.hbs', {explanation:value}); 67 | }); 68 | }); 69 | 70 | async function get_token() { 71 | //use the Cloud API key to authenticate. in a production app we'd cache this and then refresh 72 | //when necessary, since it's a bit wasteful to get a new token each time we make a call. 73 | console.log('getting token'); 74 | try { 75 | const response = await axios({ 76 | url: `https://iam.cloud.ibm.com/identity/token?grant_type=urn:ibm:params:oauth:grant-type:apikey&apikey=${auth.cloud.apikey}`, 77 | method: 'POST', 78 | headers: { 79 | 'Content-Type': 'application/x-www-form-urlencoded', 80 | 'Accept': 'application/json' 81 | } 82 | }); 83 | return await response.data.access_token; 84 | } catch (error) { 85 | console.error(error); 86 | } 87 | } 88 | 89 | async function get_last_transaction(token) { 90 | //this function gets the scoring_id of the last prediction made by the model. we need it to 91 | //request an explanation. 92 | console.log(`getting transaction using token ${token}`); 93 | try { 94 | const response = await axios({ 95 | url: `https://api.aiopenscale.cloud.ibm.com/v1/data_marts/${auth.openscale.guid}/scoring_transactions?subscription_id=${auth.openscale.subscription}&limit=1`, 96 | headers: { 97 | 'Authorization': `Bearer ${token}` 98 | } 99 | }); 100 | return await response.data; 101 | } catch (error) { 102 | console.error(error); 103 | } 104 | } 105 | 106 | async function request_explanation(token, transaction_id) { 107 | //send a request to the explanation service. this will start an explanation job and return its ID. 108 | //for the sake of time and system load, we're just getting the LIME explanation, and not the full 109 | //contrastive explanation. if you want the contrastive piece, remove the 'cem=false' from the url 110 | //in the call below. 111 | console.log(`Requesting explanation of ${transaction_id}`); 112 | try { 113 | const response = await axios({ 114 | url: `https://api.aiopenscale.cloud.ibm.com/v1/data_marts/${auth.openscale.guid}/explanation_tasks?cem=false`, 115 | method: 'POST', 116 | headers: { 117 | 'Authorization': `Bearer ${token}`, 118 | 'Content-Type': 'application/json' 119 | }, 120 | data: { 121 | "scoring_id":transaction_id 122 | } 123 | }); 124 | return await response.data.metadata.id; 125 | } catch (error) { 126 | console.error(error); 127 | } 128 | } 129 | 130 | async function get_request_status(token, job_id) { 131 | //get the status of the running explanation job. when it's complete, it will return the full 132 | //explanation details. we'll need to call this repeatedly, since the explanation service usually 133 | //takes 10-30 seconds to run. 134 | console.log(`Getting status of explain job id ${job_id}`); 135 | try { 136 | const response = await axios({ 137 | url: `https://api.aiopenscale.cloud.ibm.com/v1/data_marts/${auth.openscale.guid}/explanation_tasks/${job_id}?version=2018-09-17&with_error_details=true`, 138 | headers: { 139 | 'Authorization': `Bearer ${token}`, 140 | 'Content-Type': 'application/json' 141 | } 142 | }); 143 | return await response.data; 144 | } catch (error) { 145 | console.error(error); 146 | } 147 | } 148 | 149 | async function delay(ms) { 150 | //generic promise function so we can wait a set amount of time before we query the explanation 151 | //job status again. 152 | return await new Promise(resolve => setTimeout(resolve, ms)); 153 | } 154 | 155 | async function get_explanation() { 156 | //step by async step to get our explanation. first we authenticate, then get the ID of the last 157 | //prediction. next, we request an explanation of the prediction. finally, we check every three 158 | //seconds to see if the explanation task has finished, then return the final value. 159 | const token = await get_token(); 160 | const transaction = await get_last_transaction(token); 161 | const transaction_id = transaction.values[0][transaction.fields.indexOf('scoring_id')]; 162 | const request_id = await request_explanation(token, transaction_id); 163 | let request_status = 'in_progress'; 164 | let request_output = null; 165 | while (request_status === 'in_progress') { 166 | await delay(3000); 167 | request_output = await get_request_status(token, request_id); 168 | request_status = request_output.entity.status.state; 169 | } 170 | return request_output.entity; 171 | } 172 | 173 | async function get_prediction(payload) { 174 | //getting a prediction value is much simpler than getting an explanation. we just need to 175 | //authenticate, then send our query to the WML REST endpoint for our model. 176 | const token = await get_token(); 177 | return await query_wml(token, payload); 178 | } 179 | 180 | async function query_wml(token, payload) { 181 | //format the data so we can send it to our model 182 | let fields = Object.keys(payload); 183 | let values = Object.values(payload); 184 | 185 | //convert numerical form values to integers. the form passes them as strings. this doesn't 186 | //affect our model, which handles the transformation for us, but will break OpenScale's 187 | //payload logging service unless we get the data types right. 188 | for (let i = 0; i < values.length; i++) { 189 | if (!isNaN(Number(values[i]))) { 190 | values[i] = Number(values[i]); 191 | } 192 | } 193 | 194 | console.log(`querying wml`); 195 | try { 196 | const response = await axios({ 197 | url: auth.openscale.scoring_url, 198 | method: 'POST', 199 | headers: { 200 | 'Content-Type': 'application/json', 201 | 'Accept': 'application/json', 202 | 'Authorization': `Bearer ${token}`, 203 | 'ML-Instance-ID': auth.wml["instance_id"] 204 | }, 205 | data: { 206 | "fields": fields, 207 | "values": [values] 208 | } 209 | }); 210 | return await response.data; 211 | } catch (error) { 212 | console.error(error); 213 | } 214 | } 215 | 216 | const port = process.env.PORT || 3000; 217 | app.listen(port, () => console.log(`App listening at http://localhost:${port}`)); -------------------------------------------------------------------------------- /notebooks/mortgage_default_cloud.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Watson OpenScale Mortgage Default Demo" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "This notebook should be run in a [Watson Studio](https://dataplatform.ibm.com/) project with Python 3.6 or greater. It requires a free lite version of [Watson Machine Learning](https://cloud.ibm.com/catalog/services/machine-learning).\n", 15 | "\n", 16 | "This notebook will train, save and deploy a machine learning model to predict mortgage defaults. Then, it will configure OpenScale to monitor the model." 17 | ] 18 | }, 19 | { 20 | "cell_type": "markdown", 21 | "metadata": {}, 22 | "source": [ 23 | "## Provision services and create credentials" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": {}, 29 | "source": [ 30 | "You will need credentials for Watson Machine Learning. If you already have a WML instance, you may use credentials for it. To provision a new Lite instance of WML, use the [Cloud catalog](https://cloud.ibm.com/catalog/services/machine-learning), give your service a name, and click **Create**. Once your instance is created, click the **Service Credentials** link on the left side of the screen. Click the **New credential** button, give your credentials a name, and click **Add**. Your new credentials can be accessed by clicking the **View credentials** button. Copy and paste your WML credentials into the cell below." 31 | ] 32 | }, 33 | { 34 | "cell_type": "code", 35 | "execution_count": null, 36 | "metadata": {}, 37 | "outputs": [], 38 | "source": [ 39 | "WML_CREDENTIALS = {\n", 40 | " \"apikey\": \"key\",\n", 41 | " \"iam_apikey_description\": \"description\",\n", 42 | " \"iam_apikey_name\": \"auto-generated-apikey\",\n", 43 | " \"iam_role_crn\": \"crn:v1:bluemix:public:iam::::serviceRole:Writer\",\n", 44 | " \"iam_serviceid_crn\": \"crn:v1:bluemix:public:iam-identity::\",\n", 45 | " \"instance_id\": \"instance_id\",\n", 46 | " \"url\": \"https://us-south.ml.cloud.ibm.com\",\n", 47 | "}" 48 | ] 49 | }, 50 | { 51 | "cell_type": "markdown", 52 | "metadata": {}, 53 | "source": [ 54 | "You can generate a Cloud API key [here](https://cloud.ibm.com/iam/apikeys)." 55 | ] 56 | }, 57 | { 58 | "cell_type": "code", 59 | "execution_count": null, 60 | "metadata": {}, 61 | "outputs": [], 62 | "source": [ 63 | "CLOUD_API_KEY = \"xxxxxxxxxxxxxxxxx\"" 64 | ] 65 | }, 66 | { 67 | "cell_type": "markdown", 68 | "metadata": {}, 69 | "source": [ 70 | "If you have already set up an OpenScale datamart, or if you would like to use the free internal PostgreSQL datamart, you can skip the following cell. If you are setting up a new instance of OpenScale and would like to use a paid database service, paste your [Db2](https://cloud.ibm.com/catalog/services/db2-warehouse) or [PostgreSQL](https://cloud.ibm.com/catalog/services/databases-for-postgresql) credentials below." 71 | ] 72 | }, 73 | { 74 | "cell_type": "code", 75 | "execution_count": null, 76 | "metadata": {}, 77 | "outputs": [], 78 | "source": [ 79 | "DB_CREDENTIALS = None" 80 | ] 81 | }, 82 | { 83 | "cell_type": "markdown", 84 | "metadata": {}, 85 | "source": [ 86 | "## Name your model" 87 | ] 88 | }, 89 | { 90 | "cell_type": "markdown", 91 | "metadata": {}, 92 | "source": [ 93 | "You may give your model and deployment a custom name below; however, if you change the values below, be sure to use the same names in all subsequent notebooks in this lab." 94 | ] 95 | }, 96 | { 97 | "cell_type": "code", 98 | "execution_count": null, 99 | "metadata": {}, 100 | "outputs": [], 101 | "source": [ 102 | "MODEL_NAME = 'Mortgage Default'\n", 103 | "DEPLOYMENT_NAME = 'Mortgage Default - Production'" 104 | ] 105 | }, 106 | { 107 | "cell_type": "code", 108 | "execution_count": null, 109 | "metadata": {}, 110 | "outputs": [], 111 | "source": [ 112 | "!pip install --upgrade ibm-ai-openscale --no-cache | tail -n 1" 113 | ] 114 | }, 115 | { 116 | "cell_type": "markdown", 117 | "metadata": {}, 118 | "source": [ 119 | "## Run the notebook" 120 | ] 121 | }, 122 | { 123 | "cell_type": "markdown", 124 | "metadata": {}, 125 | "source": [ 126 | "At this point, you can run all cells in this notebook using the menus above." 127 | ] 128 | }, 129 | { 130 | "cell_type": "markdown", 131 | "metadata": {}, 132 | "source": [ 133 | "Import the scikit-learn framework and check the version. This notebook was developed using sklearn version 0.20.3." 134 | ] 135 | }, 136 | { 137 | "cell_type": "code", 138 | "execution_count": null, 139 | "metadata": {}, 140 | "outputs": [], 141 | "source": [ 142 | "import sklearn\n", 143 | "sklearn.__version__" 144 | ] 145 | }, 146 | { 147 | "cell_type": "markdown", 148 | "metadata": {}, 149 | "source": [ 150 | "Use the provided credentials above to create a new Watson Machine Learning client." 151 | ] 152 | }, 153 | { 154 | "cell_type": "code", 155 | "execution_count": null, 156 | "metadata": {}, 157 | "outputs": [], 158 | "source": [ 159 | "from watson_machine_learning_client import WatsonMachineLearningAPIClient\n", 160 | "\n", 161 | "wml_client = WatsonMachineLearningAPIClient(WML_CREDENTIALS)" 162 | ] 163 | }, 164 | { 165 | "cell_type": "markdown", 166 | "metadata": {}, 167 | "source": [ 168 | "List all models for this instance of Watson Machine Learning." 169 | ] 170 | }, 171 | { 172 | "cell_type": "code", 173 | "execution_count": null, 174 | "metadata": {}, 175 | "outputs": [], 176 | "source": [ 177 | "wml_client.repository.list_models()" 178 | ] 179 | }, 180 | { 181 | "cell_type": "markdown", 182 | "metadata": {}, 183 | "source": [ 184 | "Import the pandas library, download and examine our training data. The data contains an 'ID' field for the loan ID, which will not be used in training the model and is dropped." 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": null, 190 | "metadata": {}, 191 | "outputs": [], 192 | "source": [ 193 | "import pandas as pd\n", 194 | "\n", 195 | "url = 'https://raw.githubusercontent.com/ericmartens/openscale_app/master/data/Mortgage_Full_Records.csv'\n", 196 | "df_raw = pd.read_csv(url)\n", 197 | "df = df_raw.drop('ID', axis=1)\n", 198 | "df.head()" 199 | ] 200 | }, 201 | { 202 | "cell_type": "markdown", 203 | "metadata": {}, 204 | "source": [ 205 | "Import the sklearn libraries we need, including encoders, transformers, scalers, and our random forest classifier." 206 | ] 207 | }, 208 | { 209 | "cell_type": "code", 210 | "execution_count": null, 211 | "metadata": {}, 212 | "outputs": [], 213 | "source": [ 214 | "from sklearn.preprocessing import OneHotEncoder\n", 215 | "from sklearn.ensemble import RandomForestClassifier\n", 216 | "from sklearn.impute import SimpleImputer\n", 217 | "from sklearn.compose import ColumnTransformer\n", 218 | "from sklearn.pipeline import Pipeline\n", 219 | "from sklearn.metrics import classification_report\n", 220 | "from sklearn.model_selection import train_test_split\n", 221 | "from sklearn.preprocessing import MinMaxScaler" 222 | ] 223 | }, 224 | { 225 | "cell_type": "markdown", 226 | "metadata": {}, 227 | "source": [ 228 | "Identify the categorical features, and create a one-hot encoder pipeline for them.\n", 229 | "\n", 230 | "Next, identify the numerical features and use the min-max scaler to scale the values, which will significantly increase our model's accuracy.\n", 231 | "\n", 232 | "Finally, organize the categorical encoder and the scaler into a pipeline so the deployed model can work with our data." 233 | ] 234 | }, 235 | { 236 | "cell_type": "code", 237 | "execution_count": null, 238 | "metadata": {}, 239 | "outputs": [], 240 | "source": [ 241 | "categorical_features = ['AppliedOnline','Residence','Location']\n", 242 | "categorical_transformer = Pipeline(steps=[\n", 243 | " ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),\n", 244 | " ('onehot', OneHotEncoder(handle_unknown='ignore'))])\n", 245 | "\n", 246 | "scaled_features = ['Income','Yrs_at_Current_Address','Yrs_with_Current_Employer',\\\n", 247 | " 'Number_of_Cards','Creditcard_Debt','Loan_Amount','SalePrice']\n", 248 | "scale_transformer = Pipeline(steps=[('scale', MinMaxScaler())])\n", 249 | "\n", 250 | "preprocessor = ColumnTransformer(\n", 251 | " transformers=[\n", 252 | " ('cat', categorical_transformer, categorical_features),\n", 253 | " ('scaler', scale_transformer, scaled_features)\n", 254 | " ]\n", 255 | ")\n", 256 | "\n", 257 | "clf = Pipeline(steps=[('preprocessor', preprocessor),\n", 258 | " ('classifier', RandomForestClassifier())])" 259 | ] 260 | }, 261 | { 262 | "cell_type": "markdown", 263 | "metadata": {}, 264 | "source": [ 265 | "Perform the train/test split, train the model, and score the model quality." 266 | ] 267 | }, 268 | { 269 | "cell_type": "code", 270 | "execution_count": null, 271 | "metadata": {}, 272 | "outputs": [], 273 | "source": [ 274 | "X = df.drop('MortgageDefault', axis=1)\n", 275 | "y = df['MortgageDefault']\n", 276 | "\n", 277 | "X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)\n", 278 | "\n", 279 | "model = clf.fit(X_train, y_train)\n", 280 | "res_predict = model.predict(X_test)\n", 281 | "print(\"model score: %.3f\" % clf.score(X_test, y_test))\n", 282 | "print(classification_report(y_test, res_predict, target_names=[\"False\", \"True\"]))" 283 | ] 284 | }, 285 | { 286 | "cell_type": "markdown", 287 | "metadata": {}, 288 | "source": [ 289 | "## Save the model to Watson Machine Learning" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "Check the list of models in the WML instance, and remove pre-existing versions of this model. This allows the notebook to be re-run to reset all data if necessary." 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": null, 302 | "metadata": {}, 303 | "outputs": [], 304 | "source": [ 305 | "model_deployment_ids = wml_client.deployments.get_uids()\n", 306 | "for deployment_id in model_deployment_ids:\n", 307 | " deployment = wml_client.deployments.get_details(deployment_id)\n", 308 | " model_id = deployment['entity']['deployable_asset']['guid']\n", 309 | " if deployment['entity']['name'] == DEPLOYMENT_NAME:\n", 310 | " print('Deleting deployment id', deployment_id)\n", 311 | " wml_client.deployments.delete(deployment_id)\n", 312 | " print('Deleting model id', model_id)\n", 313 | " wml_client.repository.delete(model_id)\n", 314 | "wml_client.repository.list_models()" 315 | ] 316 | }, 317 | { 318 | "cell_type": "markdown", 319 | "metadata": {}, 320 | "source": [ 321 | "Create the metadata and save the model." 322 | ] 323 | }, 324 | { 325 | "cell_type": "code", 326 | "execution_count": null, 327 | "metadata": {}, 328 | "outputs": [], 329 | "source": [ 330 | "metadata = {\n", 331 | " wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,\n", 332 | " wml_client.repository.ModelMetaNames.EVALUATION_METHOD: \"binary\",\n", 333 | " wml_client.repository.ModelMetaNames.EVALUATION_METRICS: [\n", 334 | " {\n", 335 | " \"name\": \"areaUnderROC\",\n", 336 | " \"value\": 0.7,\n", 337 | " \"threshold\": 0.7\n", 338 | " }\n", 339 | " ]\n", 340 | "}\n", 341 | "\n", 342 | "# Name the columns\n", 343 | "cols=[\"Income\",\"AppliedOnline\",\"Residence\",\"Yrs_at_Current_Address\",\"Yrs_with_Current_Employer\",\\\n", 344 | " \"Number_of_Cards\",\"Creditcard_Debt\",\"Loans\",\"Loan_Amount\",\"SalePrice\",\"Location\"]\n", 345 | " \n", 346 | "saved_model = wml_client.repository.store_model(model=model, meta_props=metadata, \n", 347 | " training_data=X_train, training_target=y_train, \n", 348 | " feature_names=cols, label_column_names=[\"MortgageDefault\"] )\n", 349 | "saved_model" 350 | ] 351 | }, 352 | { 353 | "cell_type": "markdown", 354 | "metadata": {}, 355 | "source": [ 356 | "Get the unique ID for the model so we can deploy it." 357 | ] 358 | }, 359 | { 360 | "cell_type": "code", 361 | "execution_count": null, 362 | "metadata": {}, 363 | "outputs": [], 364 | "source": [ 365 | "model_uid = saved_model['metadata']['guid']\n", 366 | "model_uid" 367 | ] 368 | }, 369 | { 370 | "cell_type": "markdown", 371 | "metadata": {}, 372 | "source": [ 373 | "Deploy the model as a web service with Watson Machine Learning." 374 | ] 375 | }, 376 | { 377 | "cell_type": "code", 378 | "execution_count": null, 379 | "metadata": {}, 380 | "outputs": [], 381 | "source": [ 382 | "print(\"Deploying model...\")\n", 383 | "\n", 384 | "deployment = wml_client.deployments.create(artifact_uid=model_uid, name=DEPLOYMENT_NAME, asynchronous=False)" 385 | ] 386 | }, 387 | { 388 | "cell_type": "code", 389 | "execution_count": null, 390 | "metadata": {}, 391 | "outputs": [], 392 | "source": [ 393 | "deployment_uid = wml_client.deployments.get_uid(deployment)\n", 394 | "\n", 395 | "print(\"Model id: {}\".format(model_uid))\n", 396 | "print(\"Deployment id: {}\".format(deployment_uid))" 397 | ] 398 | }, 399 | { 400 | "cell_type": "markdown", 401 | "metadata": {}, 402 | "source": [ 403 | "# OpenScale Mortgage Default Configuration\n", 404 | "\n", 405 | "This pportion of the notebook will configure OpenScale monitoring for the mortgage default model using the Python client, as opposed to the graphical user interface." 406 | ] 407 | }, 408 | { 409 | "cell_type": "code", 410 | "execution_count": null, 411 | "metadata": {}, 412 | "outputs": [], 413 | "source": [ 414 | "from ibm_ai_openscale import APIClient\n", 415 | "from ibm_ai_openscale.engines import *\n", 416 | "from ibm_ai_openscale.utils import *\n", 417 | "from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature\n", 418 | "from ibm_ai_openscale.supporting_classes.enums import *" 419 | ] 420 | }, 421 | { 422 | "cell_type": "markdown", 423 | "metadata": {}, 424 | "source": [ 425 | "Get the instance ID for Watson OpenScale." 426 | ] 427 | }, 428 | { 429 | "cell_type": "code", 430 | "execution_count": null, 431 | "metadata": {}, 432 | "outputs": [], 433 | "source": [ 434 | "import requests\n", 435 | "from ibm_ai_openscale.utils import get_instance_guid\n", 436 | "\n", 437 | "WOS_GUID = get_instance_guid(api_key=CLOUD_API_KEY)\n", 438 | "WOS_CREDENTIALS = {\n", 439 | " \"instance_guid\": WOS_GUID,\n", 440 | " \"apikey\": CLOUD_API_KEY,\n", 441 | " \"url\": \"https://api.aiopenscale.cloud.ibm.com\"\n", 442 | "}\n", 443 | "\n", 444 | "if WOS_GUID is None:\n", 445 | " print('Watson OpenScale GUID NOT FOUND')\n", 446 | "else:\n", 447 | " print(WOS_GUID)" 448 | ] 449 | }, 450 | { 451 | "cell_type": "markdown", 452 | "metadata": {}, 453 | "source": [ 454 | "Use the Cloud API key and WOS instance ID to create a new OpenScale client." 455 | ] 456 | }, 457 | { 458 | "cell_type": "code", 459 | "execution_count": null, 460 | "metadata": {}, 461 | "outputs": [], 462 | "source": [ 463 | "ai_client = APIClient(aios_credentials=WOS_CREDENTIALS)\n", 464 | "ai_client.version" 465 | ] 466 | }, 467 | { 468 | "cell_type": "markdown", 469 | "metadata": {}, 470 | "source": [ 471 | "Set up the OpenScale datamart. First check for an existing datamart. If none is found, create one using the DB_CREDENTIALS if provided. If no credentials were provided, use the free internal datamart." 472 | ] 473 | }, 474 | { 475 | "cell_type": "code", 476 | "execution_count": null, 477 | "metadata": {}, 478 | "outputs": [], 479 | "source": [ 480 | "try:\n", 481 | " data_mart_details = ai_client.data_mart.get_details()\n", 482 | " if 'internal_database' in data_mart_details and data_mart_details['internal_database']:\n", 483 | " print('Using existing internal datamart')\n", 484 | " else:\n", 485 | " print('Using existing external datamart')\n", 486 | "except:\n", 487 | " if DB_CREDENTIALS is None:\n", 488 | " print('Setting up internal datamart')\n", 489 | " ai_client.data_mart.setup(internal_db=True)\n", 490 | " else:\n", 491 | " print('Setting up external datamart')\n", 492 | " try:\n", 493 | " ai_client.data_mart.setup(db_credentials=DB_CREDENTIALS)\n", 494 | " except:\n", 495 | " print('Setup failed, trying Db2 setup')\n", 496 | " ai_client.data_mart.setup(db_credentials=DB_CREDENTIALS, schema=DB_CREDENTIALS['username'])" 497 | ] 498 | }, 499 | { 500 | "cell_type": "code", 501 | "execution_count": null, 502 | "metadata": {}, 503 | "outputs": [], 504 | "source": [ 505 | "data_mart_details = ai_client.data_mart.get_details()" 506 | ] 507 | }, 508 | { 509 | "cell_type": "markdown", 510 | "metadata": {}, 511 | "source": [ 512 | "Bind the OpenScale datamart to the WML instance. If the binding already exists, this will generate an error message, but will not affect the remainder of the notebook." 513 | ] 514 | }, 515 | { 516 | "cell_type": "code", 517 | "execution_count": null, 518 | "metadata": {}, 519 | "outputs": [], 520 | "source": [ 521 | "binding_uid = ai_client.data_mart.bindings.add('WML Binding', WatsonMachineLearningInstance(WML_CREDENTIALS))\n", 522 | "bindings_details = ai_client.data_mart.bindings.get_details()\n", 523 | "\n", 524 | "ai_client.data_mart.bindings.list()" 525 | ] 526 | }, 527 | { 528 | "cell_type": "code", 529 | "execution_count": null, 530 | "metadata": {}, 531 | "outputs": [], 532 | "source": [ 533 | "print(binding_uid)" 534 | ] 535 | }, 536 | { 537 | "cell_type": "markdown", 538 | "metadata": {}, 539 | "source": [ 540 | "Get the scoring endpoint for the deployed model." 541 | ] 542 | }, 543 | { 544 | "cell_type": "code", 545 | "execution_count": null, 546 | "metadata": {}, 547 | "outputs": [], 548 | "source": [ 549 | "deployment_details = wml_client.deployments.get_details(deployment_uid)\n", 550 | "scoring_endpoint = deployment_details['entity']['scoring_url']\n", 551 | "\n", 552 | "print('Model UID:', model_uid)\n", 553 | "print('Scoring URL:', scoring_endpoint)" 554 | ] 555 | }, 556 | { 557 | "cell_type": "markdown", 558 | "metadata": {}, 559 | "source": [ 560 | "List all the subscribed models." 561 | ] 562 | }, 563 | { 564 | "cell_type": "code", 565 | "execution_count": null, 566 | "metadata": {}, 567 | "outputs": [], 568 | "source": [ 569 | "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", 570 | "ai_client.data_mart.subscriptions.list()" 571 | ] 572 | }, 573 | { 574 | "cell_type": "markdown", 575 | "metadata": {}, 576 | "source": [ 577 | "The credentials below point to the training data for the model, in CSV format. OpenScale uses the training data to train the drift model, and generate distribution statistics for the explainability service and the fairness monitor. If you don't want to provide this information to OpenScale, it is possible to run a custom notebook to create this data." 578 | ] 579 | }, 580 | { 581 | "cell_type": "code", 582 | "execution_count": null, 583 | "metadata": {}, 584 | "outputs": [], 585 | "source": [ 586 | "cos_credentials = {\n", 587 | " \"apikey\": \"yqcPbWZ0AQPHleHVerrR4Wx5e9pymBdMgydbEra5zCif\",\n", 588 | " \"api_key\": \"yqcPbWZ0AQPHleHVerrR4Wx5e9pymBdMgydbEra5zCif\",\n", 589 | " \"url\": \"https://s3.us.cloud-object-storage.appdomain.cloud\",\n", 590 | " \"iam_url\": 'https://iam.bluemix.net/oidc/token',\n", 591 | " \"cos_hmac_keys\": {\n", 592 | " \"access_key_id\": \"2d1be760f19241d695a534960da6eb80\",\n", 593 | " \"secret_access_key\": \"e1252b952f47a6b3f42305b8ffe6f9bd7d10e45f966b9a62\"\n", 594 | " },\n", 595 | " \"endpoints\": \"https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints\",\n", 596 | " \"iam_apikey_description\": \"Auto-generated for key 2d1be760-f192-41d6-95a5-34960da6eb80\",\n", 597 | " \"iam_apikey_name\": \"FastStartLab\",\n", 598 | " \"iam_role_crn\": \"crn:v1:bluemix:public:iam::::serviceRole:Reader\",\n", 599 | " \"iam_serviceid_crn\": \"crn:v1:bluemix:public:iam-identity::a/7d8b3c34272c0980d973d3e40be9e9d2::serviceid:ServiceId-568ba191-a3bf-48f2-a30c-f3a4af7ec61d\",\n", 600 | " \"resource_instance_id\": \"crn:v1:bluemix:public:cloud-object-storage:global:a/7d8b3c34272c0980d973d3e40be9e9d2:2883ef10-23f1-4592-8582-2f2ef4973639::\"\n", 601 | "}" 602 | ] 603 | }, 604 | { 605 | "cell_type": "code", 606 | "execution_count": null, 607 | "metadata": {}, 608 | "outputs": [], 609 | "source": [ 610 | "training_data_reference = {\n", 611 | " 'type': 'cos',\n", 612 | " 'location': {\n", 613 | " 'bucket': 'faststartlab-donotdelete-pr-nhfd4jnhlxgpc7',\n", 614 | " 'file_name': 'Mortgage_Full_Records.csv',\n", 615 | " 'firstlineheader': True,\n", 616 | " 'file_format': 'csv'\n", 617 | " },\n", 618 | " 'connection': cos_credentials,\n", 619 | " 'name': 'training data reference'\n", 620 | "}" 621 | ] 622 | }, 623 | { 624 | "cell_type": "markdown", 625 | "metadata": {}, 626 | "source": [ 627 | "Remove previous subscriptions for this model" 628 | ] 629 | }, 630 | { 631 | "cell_type": "code", 632 | "execution_count": null, 633 | "metadata": {}, 634 | "outputs": [], 635 | "source": [ 636 | "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", 637 | "for subscription in subscriptions_uids:\n", 638 | " sub_name = ai_client.data_mart.subscriptions.get_details(subscription)['entity']['asset']['name']\n", 639 | " if sub_name == MODEL_NAME:\n", 640 | " ai_client.data_mart.subscriptions.delete(subscription)\n", 641 | " print('Deleted existing subscription for', MODEL_NAME)" 642 | ] 643 | }, 644 | { 645 | "cell_type": "markdown", 646 | "metadata": {}, 647 | "source": [ 648 | "Create the subscription in OpenScale so we can monitor the model. Required information includes feature columns, categorical columns, problem types, input types, and output types." 649 | ] 650 | }, 651 | { 652 | "cell_type": "code", 653 | "execution_count": null, 654 | "metadata": {}, 655 | "outputs": [], 656 | "source": [ 657 | "subscription = ai_client.data_mart.subscriptions.add(WatsonMachineLearningAsset(\n", 658 | " model_uid,\n", 659 | " problem_type=ProblemType.BINARY_CLASSIFICATION,\n", 660 | " input_data_type=InputDataType.STRUCTURED,\n", 661 | " label_column='MortgageDefault',\n", 662 | " prediction_column='prediction',\n", 663 | " probability_column='probability',\n", 664 | " transaction_id_column='ID',\n", 665 | " feature_columns = ['AppliedOnline','Residence','Location','Income','Yrs_at_Current_Address','Yrs_with_Current_Employer',\\\n", 666 | " 'Number_of_Cards','Creditcard_Debt','Loan_Amount','Loans','SalePrice'],\n", 667 | " categorical_columns = ['AppliedOnline','Residence','Location'],\n", 668 | " training_data_reference = training_data_reference\n", 669 | "))\n", 670 | "\n", 671 | "if subscription is None:\n", 672 | " print('Subscription already exists; get the existing one')\n", 673 | " subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", 674 | " for sub in subscriptions_uids:\n", 675 | " if ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name'] == MODEL_NAME:\n", 676 | " subscription = ai_client.data_mart.subscriptions.get(sub)" 677 | ] 678 | }, 679 | { 680 | "cell_type": "code", 681 | "execution_count": null, 682 | "metadata": {}, 683 | "outputs": [], 684 | "source": [ 685 | "subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()\n", 686 | "ai_client.data_mart.subscriptions.list()" 687 | ] 688 | }, 689 | { 690 | "cell_type": "markdown", 691 | "metadata": {}, 692 | "source": [ 693 | "Get the subscription uid for our application" 694 | ] 695 | }, 696 | { 697 | "cell_type": "code", 698 | "execution_count": null, 699 | "metadata": {}, 700 | "outputs": [], 701 | "source": [ 702 | "for sub in subscriptions_uids:\n", 703 | " sub_name = ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name']\n", 704 | " if sub_name == MODEL_NAME:\n", 705 | " subscription_id = sub\n", 706 | " print(sub)" 707 | ] 708 | }, 709 | { 710 | "cell_type": "markdown", 711 | "metadata": {}, 712 | "source": [ 713 | "Score the model so we can begin configuring OpenScale monitors." 714 | ] 715 | }, 716 | { 717 | "cell_type": "code", 718 | "execution_count": null, 719 | "metadata": {}, 720 | "outputs": [], 721 | "source": [ 722 | "features = list(df)\n", 723 | "payload_set = df.values.tolist()\n", 724 | "\n", 725 | "scoring_payload = {\n", 726 | " \"fields\": features[:-1],\n", 727 | " \"values\": [],\n", 728 | " \"meta\":{\n", 729 | " \"fields\": [\"ID\"],\n", 730 | " \"values\": []\n", 731 | " }\n", 732 | "}" 733 | ] 734 | }, 735 | { 736 | "cell_type": "code", 737 | "execution_count": null, 738 | "metadata": {}, 739 | "outputs": [], 740 | "source": [ 741 | "import random\n", 742 | "import string\n", 743 | "\n", 744 | "letters = string.digits\n", 745 | "\n", 746 | "for _ in range(0, 201):\n", 747 | " value_to_score = random.choice(payload_set)\n", 748 | " scoring_payload['values'].append(value_to_score[:-1])\n", 749 | " scoring_payload['meta']['values'].append([int(''.join(random.choices(letters, k=8)))])" 750 | ] 751 | }, 752 | { 753 | "cell_type": "code", 754 | "execution_count": null, 755 | "metadata": {}, 756 | "outputs": [], 757 | "source": [ 758 | "predictions = wml_client.deployments.score(scoring_endpoint, scoring_payload)\n", 759 | "print(predictions['values'][0])" 760 | ] 761 | }, 762 | { 763 | "cell_type": "markdown", 764 | "metadata": {}, 765 | "source": [ 766 | "Pause for ten seconds to give OpenScale time to create the datamart schema, then show the number of records in the datamart." 767 | ] 768 | }, 769 | { 770 | "cell_type": "code", 771 | "execution_count": null, 772 | "metadata": {}, 773 | "outputs": [], 774 | "source": [ 775 | "time.sleep(10)\n", 776 | "subscription.payload_logging.get_records_count()" 777 | ] 778 | }, 779 | { 780 | "cell_type": "markdown", 781 | "metadata": {}, 782 | "source": [ 783 | "Enable quality monitoring. Set the alert threshold at 70%, and the minimum records for scoring at 100." 784 | ] 785 | }, 786 | { 787 | "cell_type": "code", 788 | "execution_count": null, 789 | "metadata": {}, 790 | "outputs": [], 791 | "source": [ 792 | "subscription.quality_monitoring.enable(threshold=0.7, min_records=100)" 793 | ] 794 | }, 795 | { 796 | "cell_type": "markdown", 797 | "metadata": {}, 798 | "source": [ 799 | "Enable fairness monitoring. In this case, we'll monitor online applications, since they are under-represented in our training data. The fairness alert threshold will be set at 90%, and we will require at least 200 records for scoring. Note that we have to supply a favourable and unfavourable model outcome, as well as a majority (reference) and minority (monitored) group." 800 | ] 801 | }, 802 | { 803 | "cell_type": "code", 804 | "execution_count": null, 805 | "metadata": {}, 806 | "outputs": [], 807 | "source": [ 808 | "subscription.fairness_monitoring.enable(\n", 809 | " features=[\n", 810 | " Feature(\"AppliedOnline\", majority=['NO'], minority=['YES'], threshold=0.90)\n", 811 | " ],\n", 812 | " favourable_classes=['NO'],\n", 813 | " unfavourable_classes=['YES'],\n", 814 | " min_records=200\n", 815 | ")" 816 | ] 817 | }, 818 | { 819 | "cell_type": "markdown", 820 | "metadata": {}, 821 | "source": [ 822 | "Configure drift monitoring. Set the alert threshold at 5% predicted drop in accuracy, and the minimum records required for scoring at 100. This will begin training the drift monitor model in OpenScale." 823 | ] 824 | }, 825 | { 826 | "cell_type": "code", 827 | "execution_count": null, 828 | "metadata": {}, 829 | "outputs": [], 830 | "source": [ 831 | "subscription.drift_monitoring.enable(threshold=0.05, min_records=100)" 832 | ] 833 | }, 834 | { 835 | "cell_type": "markdown", 836 | "metadata": {}, 837 | "source": [ 838 | "Monitor the creation of the drift monitor. Check every 30 seconds to see if it has been successfully created." 839 | ] 840 | }, 841 | { 842 | "cell_type": "code", 843 | "execution_count": null, 844 | "metadata": {}, 845 | "outputs": [], 846 | "source": [ 847 | "drift_status = None\n", 848 | "while drift_status != 'finished':\n", 849 | " drift_details = subscription.drift_monitoring.get_details()\n", 850 | " drift_status = drift_details['parameters']['config_status']['state']\n", 851 | " if drift_status != 'finished':\n", 852 | " print(drift_status)\n", 853 | " time.sleep(30)\n", 854 | "print(drift_status)" 855 | ] 856 | }, 857 | { 858 | "cell_type": "code", 859 | "execution_count": null, 860 | "metadata": {}, 861 | "outputs": [], 862 | "source": [ 863 | "print(drift_details['parameters'])" 864 | ] 865 | }, 866 | { 867 | "cell_type": "markdown", 868 | "metadata": {}, 869 | "source": [ 870 | "Manually run the fairness monitor." 871 | ] 872 | }, 873 | { 874 | "cell_type": "code", 875 | "execution_count": null, 876 | "metadata": {}, 877 | "outputs": [], 878 | "source": [ 879 | "fairness_run_details = subscription.fairness_monitoring.run(background_mode=False)" 880 | ] 881 | }, 882 | { 883 | "cell_type": "code", 884 | "execution_count": null, 885 | "metadata": {}, 886 | "outputs": [], 887 | "source": [ 888 | "fairness_run_details" 889 | ] 890 | }, 891 | { 892 | "cell_type": "code", 893 | "execution_count": null, 894 | "metadata": {}, 895 | "outputs": [], 896 | "source": [ 897 | "subscription.fairness_monitoring.show_table()" 898 | ] 899 | }, 900 | { 901 | "cell_type": "markdown", 902 | "metadata": {}, 903 | "source": [ 904 | "Manually run the drift monitor" 905 | ] 906 | }, 907 | { 908 | "cell_type": "code", 909 | "execution_count": null, 910 | "metadata": {}, 911 | "outputs": [], 912 | "source": [ 913 | "drift_run_details = subscription.drift_monitoring.run(background_mode=False)" 914 | ] 915 | }, 916 | { 917 | "cell_type": "code", 918 | "execution_count": null, 919 | "metadata": {}, 920 | "outputs": [], 921 | "source": [ 922 | "drift_run_details" 923 | ] 924 | }, 925 | { 926 | "cell_type": "code", 927 | "execution_count": null, 928 | "metadata": {}, 929 | "outputs": [], 930 | "source": [ 931 | "subscription.drift_monitoring.show_table()" 932 | ] 933 | }, 934 | { 935 | "cell_type": "markdown", 936 | "metadata": {}, 937 | "source": [ 938 | "Upload feedback data to run the quality monitor" 939 | ] 940 | }, 941 | { 942 | "cell_type": "code", 943 | "execution_count": null, 944 | "metadata": {}, 945 | "outputs": [], 946 | "source": [ 947 | "feedback_payload_set = df.values.tolist()\n", 948 | "\n", 949 | "feedback_payload = []\n", 950 | "for _ in range(0, 101):\n", 951 | " feedback_payload.append(random.choice(feedback_payload_set))\n", 952 | "print(feedback_payload[0])" 953 | ] 954 | }, 955 | { 956 | "cell_type": "code", 957 | "execution_count": null, 958 | "metadata": {}, 959 | "outputs": [], 960 | "source": [ 961 | "subscription.feedback_logging.store(feedback_payload)" 962 | ] 963 | }, 964 | { 965 | "cell_type": "code", 966 | "execution_count": null, 967 | "metadata": {}, 968 | "outputs": [], 969 | "source": [ 970 | "time.sleep(10)\n", 971 | "subscription.feedback_logging.show_table()" 972 | ] 973 | }, 974 | { 975 | "cell_type": "code", 976 | "execution_count": null, 977 | "metadata": {}, 978 | "outputs": [], 979 | "source": [ 980 | "run_details = subscription.quality_monitoring.run(background_mode=False)" 981 | ] 982 | }, 983 | { 984 | "cell_type": "code", 985 | "execution_count": null, 986 | "metadata": {}, 987 | "outputs": [], 988 | "source": [ 989 | "time.sleep(10)\n", 990 | "subscription.quality_monitoring.show_table()" 991 | ] 992 | }, 993 | { 994 | "cell_type": "markdown", 995 | "metadata": {}, 996 | "source": [ 997 | "Enable explainability." 998 | ] 999 | }, 1000 | { 1001 | "cell_type": "code", 1002 | "execution_count": null, 1003 | "metadata": {}, 1004 | "outputs": [], 1005 | "source": [ 1006 | "from ibm_ai_openscale.supporting_classes import *\n", 1007 | "\n", 1008 | "subscription.explainability.enable()" 1009 | ] 1010 | }, 1011 | { 1012 | "cell_type": "markdown", 1013 | "metadata": {}, 1014 | "source": [ 1015 | "Get a transaction to explain." 1016 | ] 1017 | }, 1018 | { 1019 | "cell_type": "code", 1020 | "execution_count": null, 1021 | "metadata": {}, 1022 | "outputs": [], 1023 | "source": [ 1024 | "transaction_id = subscription.payload_logging.get_table_content(limit=1)['scoring_id'].values[0]\n", 1025 | "\n", 1026 | "print(transaction_id)" 1027 | ] 1028 | }, 1029 | { 1030 | "cell_type": "markdown", 1031 | "metadata": {}, 1032 | "source": [ 1033 | "## Congratulations!\n", 1034 | "\n", 1035 | "You have successfully created the mortgage default model and configured it for monitoring with Watson OpenScale. Continue on with the lab to see how to set up an application to access the model and explain model predictions. Before you go, copy the output of the cell below to your text file for use in the application:" 1036 | ] 1037 | }, 1038 | { 1039 | "cell_type": "code", 1040 | "execution_count": null, 1041 | "metadata": {}, 1042 | "outputs": [], 1043 | "source": [ 1044 | "print('{{\\n\\tguid: \"{}\",\\n\\tsubscription: \"{}\",\\n\\tscoring_url: \"{}\"\\n}}'.format(WOS_GUID,subscription_id, scoring_endpoint))" 1045 | ] 1046 | }, 1047 | { 1048 | "cell_type": "code", 1049 | "execution_count": null, 1050 | "metadata": {}, 1051 | "outputs": [], 1052 | "source": [] 1053 | } 1054 | ], 1055 | "metadata": { 1056 | "kernelspec": { 1057 | "display_name": "Python 3.6", 1058 | "language": "python", 1059 | "name": "python3" 1060 | }, 1061 | "language_info": { 1062 | "codemirror_mode": { 1063 | "name": "ipython", 1064 | "version": 3 1065 | }, 1066 | "file_extension": ".py", 1067 | "mimetype": "text/x-python", 1068 | "name": "python", 1069 | "nbconvert_exporter": "python", 1070 | "pygments_lexer": "ipython3", 1071 | "version": "3.6.9" 1072 | } 1073 | }, 1074 | "nbformat": 4, 1075 | "nbformat_minor": 1 1076 | } 1077 | -------------------------------------------------------------------------------- /package-lock.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "openscale_app_demo", 3 | "version": "1.0.0", 4 | "lockfileVersion": 1, 5 | "requires": true, 6 | "dependencies": { 7 | "accepts": { 8 | "version": "1.3.7", 9 | "resolved": "https://registry.npmjs.org/accepts/-/accepts-1.3.7.tgz", 10 | "integrity": "sha512-Il80Qs2WjYlJIBNzNkK6KYqlVMTbZLXgHx2oT0pU/fjRHyEp+PEfEPY0R3WCwAGVOtauxh1hOxNgIf5bv7dQpA==", 11 | "requires": { 12 | "mime-types": "~2.1.24", 13 | "negotiator": "0.6.2" 14 | } 15 | }, 16 | "array-flatten": { 17 | "version": "1.1.1", 18 | "resolved": "https://registry.npmjs.org/array-flatten/-/array-flatten-1.1.1.tgz", 19 | "integrity": "sha1-ml9pkFGx5wczKPKgCJaLZOopVdI=" 20 | }, 21 | "axios": { 22 | "version": "0.19.2", 23 | "resolved": "https://registry.npmjs.org/axios/-/axios-0.19.2.tgz", 24 | "integrity": "sha512-fjgm5MvRHLhx+osE2xoekY70AhARk3a6hkN+3Io1jc00jtquGvxYlKlsFUhmUET0V5te6CcZI7lcv2Ym61mjHA==", 25 | "requires": { 26 | "follow-redirects": "1.5.10" 27 | } 28 | }, 29 | "body-parser": { 30 | "version": "1.19.0", 31 | "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.19.0.tgz", 32 | "integrity": "sha512-dhEPs72UPbDnAQJ9ZKMNTP6ptJaionhP5cBb541nXPlW60Jepo9RV/a4fX4XWW9CuFNK22krhrj1+rgzifNCsw==", 33 | "requires": { 34 | "bytes": "3.1.0", 35 | "content-type": "~1.0.4", 36 | "debug": "2.6.9", 37 | "depd": "~1.1.2", 38 | "http-errors": "1.7.2", 39 | "iconv-lite": "0.4.24", 40 | "on-finished": "~2.3.0", 41 | "qs": "6.7.0", 42 | "raw-body": "2.4.0", 43 | "type-is": "~1.6.17" 44 | } 45 | }, 46 | "bytes": { 47 | "version": "3.1.0", 48 | "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.0.tgz", 49 | "integrity": "sha512-zauLjrfCG+xvoyaqLoV8bLVXXNGC4JqlxFCutSDWA6fJrTo2ZuvLYTqZ7aHBLZSMOopbzwv8f+wZcVzfVTI2Dg==" 50 | }, 51 | "commander": { 52 | "version": "2.20.3", 53 | "resolved": "https://registry.npmjs.org/commander/-/commander-2.20.3.tgz", 54 | "integrity": "sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ==", 55 | "optional": true 56 | }, 57 | "content-disposition": { 58 | "version": "0.5.3", 59 | "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-0.5.3.tgz", 60 | "integrity": "sha512-ExO0774ikEObIAEV9kDo50o+79VCUdEB6n6lzKgGwupcVeRlhrj3qGAfwq8G6uBJjkqLrhT0qEYFcWng8z1z0g==", 61 | "requires": { 62 | "safe-buffer": "5.1.2" 63 | } 64 | }, 65 | "content-type": { 66 | "version": "1.0.4", 67 | "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.4.tgz", 68 | "integrity": "sha512-hIP3EEPs8tB9AT1L+NUqtwOAps4mk2Zob89MWXMHjHWg9milF/j4osnnQLXBCBFBk/tvIG/tUc9mOUJiPBhPXA==" 69 | }, 70 | "cookie": { 71 | "version": "0.4.0", 72 | "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.4.0.tgz", 73 | "integrity": "sha512-+Hp8fLp57wnUSt0tY0tHEXh4voZRDnoIrZPqlo3DPiI4y9lwg/jqx+1Om94/W6ZaPDOUbnjOt/99w66zk+l1Xg==" 74 | }, 75 | "cookie-signature": { 76 | "version": "1.0.6", 77 | "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.0.6.tgz", 78 | "integrity": "sha1-4wOogrNCzD7oylE6eZmXNNqzriw=" 79 | }, 80 | "debug": { 81 | "version": "2.6.9", 82 | "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz", 83 | "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==", 84 | "requires": { 85 | "ms": "2.0.0" 86 | } 87 | }, 88 | "depd": { 89 | "version": "1.1.2", 90 | "resolved": "https://registry.npmjs.org/depd/-/depd-1.1.2.tgz", 91 | "integrity": "sha1-m81S4UwJd2PnSbJ0xDRu0uVgtak=" 92 | }, 93 | "destroy": { 94 | "version": "1.0.4", 95 | "resolved": "https://registry.npmjs.org/destroy/-/destroy-1.0.4.tgz", 96 | "integrity": "sha1-l4hXRCxEdJ5CBmE+N5RiBYJqvYA=" 97 | }, 98 | "ee-first": { 99 | "version": "1.1.1", 100 | "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", 101 | "integrity": "sha1-WQxhFWsK4vTwJVcyoViyZrxWsh0=" 102 | }, 103 | "encodeurl": { 104 | "version": "1.0.2", 105 | "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-1.0.2.tgz", 106 | "integrity": "sha1-rT/0yG7C0CkyL1oCw6mmBslbP1k=" 107 | }, 108 | "escape-html": { 109 | "version": "1.0.3", 110 | "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz", 111 | "integrity": "sha1-Aljq5NPQwJdN4cFpGI7wBR0dGYg=" 112 | }, 113 | "etag": { 114 | "version": "1.8.1", 115 | "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz", 116 | "integrity": "sha1-Qa4u62XvpiJorr/qg6x9eSmbCIc=" 117 | }, 118 | "express": { 119 | "version": "4.17.1", 120 | "resolved": "https://registry.npmjs.org/express/-/express-4.17.1.tgz", 121 | "integrity": "sha512-mHJ9O79RqluphRrcw2X/GTh3k9tVv8YcoyY4Kkh4WDMUYKRZUq0h1o0w2rrrxBqM7VoeUVqgb27xlEMXTnYt4g==", 122 | "requires": { 123 | "accepts": "~1.3.7", 124 | "array-flatten": "1.1.1", 125 | "body-parser": "1.19.0", 126 | "content-disposition": "0.5.3", 127 | "content-type": "~1.0.4", 128 | "cookie": "0.4.0", 129 | "cookie-signature": "1.0.6", 130 | "debug": "2.6.9", 131 | "depd": "~1.1.2", 132 | "encodeurl": "~1.0.2", 133 | "escape-html": "~1.0.3", 134 | "etag": "~1.8.1", 135 | "finalhandler": "~1.1.2", 136 | "fresh": "0.5.2", 137 | "merge-descriptors": "1.0.1", 138 | "methods": "~1.1.2", 139 | "on-finished": "~2.3.0", 140 | "parseurl": "~1.3.3", 141 | "path-to-regexp": "0.1.7", 142 | "proxy-addr": "~2.0.5", 143 | "qs": "6.7.0", 144 | "range-parser": "~1.2.1", 145 | "safe-buffer": "5.1.2", 146 | "send": "0.17.1", 147 | "serve-static": "1.14.1", 148 | "setprototypeof": "1.1.1", 149 | "statuses": "~1.5.0", 150 | "type-is": "~1.6.18", 151 | "utils-merge": "1.0.1", 152 | "vary": "~1.1.2" 153 | } 154 | }, 155 | "finalhandler": { 156 | "version": "1.1.2", 157 | "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-1.1.2.tgz", 158 | "integrity": "sha512-aAWcW57uxVNrQZqFXjITpW3sIUQmHGG3qSb9mUah9MgMC4NeWhNOlNjXEYq3HjRAvL6arUviZGGJsBg6z0zsWA==", 159 | "requires": { 160 | "debug": "2.6.9", 161 | "encodeurl": "~1.0.2", 162 | "escape-html": "~1.0.3", 163 | "on-finished": "~2.3.0", 164 | "parseurl": "~1.3.3", 165 | "statuses": "~1.5.0", 166 | "unpipe": "~1.0.0" 167 | } 168 | }, 169 | "follow-redirects": { 170 | "version": "1.5.10", 171 | "resolved": "https://registry.npmjs.org/follow-redirects/-/follow-redirects-1.5.10.tgz", 172 | "integrity": "sha512-0V5l4Cizzvqt5D44aTXbFZz+FtyXV1vrDN6qrelxtfYQKW0KO0W2T/hkE8xvGa/540LkZlkaUjO4ailYTFtHVQ==", 173 | "requires": { 174 | "debug": "=3.1.0" 175 | }, 176 | "dependencies": { 177 | "debug": { 178 | "version": "3.1.0", 179 | "resolved": "https://registry.npmjs.org/debug/-/debug-3.1.0.tgz", 180 | "integrity": "sha512-OX8XqP7/1a9cqkxYw2yXss15f26NKWBpDXQd0/uK/KPqdQhxbPa994hnzjcE2VqQpDslf55723cKPUOGSmMY3g==", 181 | "requires": { 182 | "ms": "2.0.0" 183 | } 184 | } 185 | } 186 | }, 187 | "foreachasync": { 188 | "version": "3.0.0", 189 | "resolved": "https://registry.npmjs.org/foreachasync/-/foreachasync-3.0.0.tgz", 190 | "integrity": "sha1-VQKYfchxS+M5IJfzLgBxyd7gfPY=" 191 | }, 192 | "forwarded": { 193 | "version": "0.1.2", 194 | "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.1.2.tgz", 195 | "integrity": "sha1-mMI9qxF1ZXuMBXPozszZGw/xjIQ=" 196 | }, 197 | "fresh": { 198 | "version": "0.5.2", 199 | "resolved": "https://registry.npmjs.org/fresh/-/fresh-0.5.2.tgz", 200 | "integrity": "sha1-PYyt2Q2XZWn6g1qx+OSyOhBWBac=" 201 | }, 202 | "handlebars": { 203 | "version": "4.7.6", 204 | "resolved": "https://registry.npmjs.org/handlebars/-/handlebars-4.7.6.tgz", 205 | "integrity": "sha512-1f2BACcBfiwAfStCKZNrUCgqNZkGsAT7UM3kkYtXuLo0KnaVfjKOyf7PRzB6++aK9STyT1Pd2ZCPe3EGOXleXA==", 206 | "requires": { 207 | "minimist": "^1.2.5", 208 | "neo-async": "^2.6.0", 209 | "source-map": "^0.6.1", 210 | "uglify-js": "^3.1.4", 211 | "wordwrap": "^1.0.0" 212 | } 213 | }, 214 | "hbs": { 215 | "version": "4.1.1", 216 | "resolved": "https://registry.npmjs.org/hbs/-/hbs-4.1.1.tgz", 217 | "integrity": "sha512-6QsbB4RwbpL4cb4DNyjEEPF+suwp+3yZqFVlhILEn92ScC0U4cDCR+FDX53jkfKJPhutcqhAvs+rOLZw5sQrDA==", 218 | "requires": { 219 | "handlebars": "4.7.6", 220 | "walk": "2.3.14" 221 | } 222 | }, 223 | "http-errors": { 224 | "version": "1.7.2", 225 | "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-1.7.2.tgz", 226 | "integrity": "sha512-uUQBt3H/cSIVfch6i1EuPNy/YsRSOUBXTVfZ+yR7Zjez3qjBz6i9+i4zjNaoqcoFVI4lQJ5plg63TvGfRSDCRg==", 227 | "requires": { 228 | "depd": "~1.1.2", 229 | "inherits": "2.0.3", 230 | "setprototypeof": "1.1.1", 231 | "statuses": ">= 1.5.0 < 2", 232 | "toidentifier": "1.0.0" 233 | } 234 | }, 235 | "iconv-lite": { 236 | "version": "0.4.24", 237 | "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz", 238 | "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==", 239 | "requires": { 240 | "safer-buffer": ">= 2.1.2 < 3" 241 | } 242 | }, 243 | "inherits": { 244 | "version": "2.0.3", 245 | "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz", 246 | "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4=" 247 | }, 248 | "ipaddr.js": { 249 | "version": "1.9.1", 250 | "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz", 251 | "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==" 252 | }, 253 | "media-typer": { 254 | "version": "0.3.0", 255 | "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-0.3.0.tgz", 256 | "integrity": "sha1-hxDXrwqmJvj/+hzgAWhUUmMlV0g=" 257 | }, 258 | "merge-descriptors": { 259 | "version": "1.0.1", 260 | "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-1.0.1.tgz", 261 | "integrity": "sha1-sAqqVW3YtEVoFQ7J0blT8/kMu2E=" 262 | }, 263 | "methods": { 264 | "version": "1.1.2", 265 | "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz", 266 | "integrity": "sha1-VSmk1nZUE07cxSZmVoNbD4Ua/O4=" 267 | }, 268 | "mime": { 269 | "version": "1.6.0", 270 | "resolved": "https://registry.npmjs.org/mime/-/mime-1.6.0.tgz", 271 | "integrity": "sha512-x0Vn8spI+wuJ1O6S7gnbaQg8Pxh4NNHb7KSINmEWKiPE4RKOplvijn+NkmYmmRgP68mc70j2EbeTFRsrswaQeg==" 272 | }, 273 | "mime-db": { 274 | "version": "1.44.0", 275 | "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.44.0.tgz", 276 | "integrity": "sha512-/NOTfLrsPBVeH7YtFPgsVWveuL+4SjjYxaQ1xtM1KMFj7HdxlBlxeyNLzhyJVx7r4rZGJAZ/6lkKCitSc/Nmpg==" 277 | }, 278 | "mime-types": { 279 | "version": "2.1.27", 280 | "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.27.tgz", 281 | "integrity": "sha512-JIhqnCasI9yD+SsmkquHBxTSEuZdQX5BuQnS2Vc7puQQQ+8yiP5AY5uWhpdv4YL4VM5c6iliiYWPgJ/nJQLp7w==", 282 | "requires": { 283 | "mime-db": "1.44.0" 284 | } 285 | }, 286 | "minimist": { 287 | "version": "1.2.6", 288 | "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.6.tgz", 289 | "integrity": "sha512-Jsjnk4bw3YJqYzbdyBiNsPWHPfO++UGG749Cxs6peCu5Xg4nrena6OVxOYxrQTqww0Jmwt+Ref8rggumkTLz9Q==" 290 | }, 291 | "ms": { 292 | "version": "2.0.0", 293 | "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz", 294 | "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g=" 295 | }, 296 | "negotiator": { 297 | "version": "0.6.2", 298 | "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-0.6.2.tgz", 299 | "integrity": "sha512-hZXc7K2e+PgeI1eDBe/10Ard4ekbfrrqG8Ep+8Jmf4JID2bNg7NvCPOZN+kfF574pFQI7mum2AUqDidoKqcTOw==" 300 | }, 301 | "neo-async": { 302 | "version": "2.6.1", 303 | "resolved": "https://registry.npmjs.org/neo-async/-/neo-async-2.6.1.tgz", 304 | "integrity": "sha512-iyam8fBuCUpWeKPGpaNMetEocMt364qkCsfL9JuhjXX6dRnguRVOfk2GZaDpPjcOKiiXCPINZC1GczQ7iTq3Zw==" 305 | }, 306 | "on-finished": { 307 | "version": "2.3.0", 308 | "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.3.0.tgz", 309 | "integrity": "sha1-IPEzZIGwg811M3mSoWlxqi2QaUc=", 310 | "requires": { 311 | "ee-first": "1.1.1" 312 | } 313 | }, 314 | "parseurl": { 315 | "version": "1.3.3", 316 | "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz", 317 | "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==" 318 | }, 319 | "path-to-regexp": { 320 | "version": "0.1.7", 321 | "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-0.1.7.tgz", 322 | "integrity": "sha1-32BBeABfUi8V60SQ5yR6G/qmf4w=" 323 | }, 324 | "proxy-addr": { 325 | "version": "2.0.6", 326 | "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.6.tgz", 327 | "integrity": "sha512-dh/frvCBVmSsDYzw6n926jv974gddhkFPfiN8hPOi30Wax25QZyZEGveluCgliBnqmuM+UJmBErbAUFIoDbjOw==", 328 | "requires": { 329 | "forwarded": "~0.1.2", 330 | "ipaddr.js": "1.9.1" 331 | } 332 | }, 333 | "qs": { 334 | "version": "6.7.0", 335 | "resolved": "https://registry.npmjs.org/qs/-/qs-6.7.0.tgz", 336 | "integrity": "sha512-VCdBRNFTX1fyE7Nb6FYoURo/SPe62QCaAyzJvUjwRaIsc+NePBEniHlvxFmmX56+HZphIGtV0XeCirBtpDrTyQ==" 337 | }, 338 | "range-parser": { 339 | "version": "1.2.1", 340 | "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", 341 | "integrity": "sha512-Hrgsx+orqoygnmhFbKaHE6c296J+HTAQXoxEF6gNupROmmGJRoyzfG3ccAveqCBrwr/2yxQ5BVd/GTl5agOwSg==" 342 | }, 343 | "raw-body": { 344 | "version": "2.4.0", 345 | "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-2.4.0.tgz", 346 | "integrity": "sha512-4Oz8DUIwdvoa5qMJelxipzi/iJIi40O5cGV1wNYp5hvZP8ZN0T+jiNkL0QepXs+EsQ9XJ8ipEDoiH70ySUJP3Q==", 347 | "requires": { 348 | "bytes": "3.1.0", 349 | "http-errors": "1.7.2", 350 | "iconv-lite": "0.4.24", 351 | "unpipe": "1.0.0" 352 | } 353 | }, 354 | "safe-buffer": { 355 | "version": "5.1.2", 356 | "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz", 357 | "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==" 358 | }, 359 | "safer-buffer": { 360 | "version": "2.1.2", 361 | "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", 362 | "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==" 363 | }, 364 | "send": { 365 | "version": "0.17.1", 366 | "resolved": "https://registry.npmjs.org/send/-/send-0.17.1.tgz", 367 | "integrity": "sha512-BsVKsiGcQMFwT8UxypobUKyv7irCNRHk1T0G680vk88yf6LBByGcZJOTJCrTP2xVN6yI+XjPJcNuE3V4fT9sAg==", 368 | "requires": { 369 | "debug": "2.6.9", 370 | "depd": "~1.1.2", 371 | "destroy": "~1.0.4", 372 | "encodeurl": "~1.0.2", 373 | "escape-html": "~1.0.3", 374 | "etag": "~1.8.1", 375 | "fresh": "0.5.2", 376 | "http-errors": "~1.7.2", 377 | "mime": "1.6.0", 378 | "ms": "2.1.1", 379 | "on-finished": "~2.3.0", 380 | "range-parser": "~1.2.1", 381 | "statuses": "~1.5.0" 382 | }, 383 | "dependencies": { 384 | "ms": { 385 | "version": "2.1.1", 386 | "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.1.tgz", 387 | "integrity": "sha512-tgp+dl5cGk28utYktBsrFqA7HKgrhgPsg6Z/EfhWI4gl1Hwq8B/GmY/0oXZ6nF8hDVesS/FpnYaD/kOWhYQvyg==" 388 | } 389 | } 390 | }, 391 | "serve-static": { 392 | "version": "1.14.1", 393 | "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-1.14.1.tgz", 394 | "integrity": "sha512-JMrvUwE54emCYWlTI+hGrGv5I8dEwmco/00EvkzIIsR7MqrHonbD9pO2MOfFnpFntl7ecpZs+3mW+XbQZu9QCg==", 395 | "requires": { 396 | "encodeurl": "~1.0.2", 397 | "escape-html": "~1.0.3", 398 | "parseurl": "~1.3.3", 399 | "send": "0.17.1" 400 | } 401 | }, 402 | "setprototypeof": { 403 | "version": "1.1.1", 404 | "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.1.1.tgz", 405 | "integrity": "sha512-JvdAWfbXeIGaZ9cILp38HntZSFSo3mWg6xGcJJsd+d4aRMOqauag1C63dJfDw7OaMYwEbHMOxEZ1lqVRYP2OAw==" 406 | }, 407 | "source-map": { 408 | "version": "0.6.1", 409 | "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz", 410 | "integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==" 411 | }, 412 | "statuses": { 413 | "version": "1.5.0", 414 | "resolved": "https://registry.npmjs.org/statuses/-/statuses-1.5.0.tgz", 415 | "integrity": "sha1-Fhx9rBd2Wf2YEfQ3cfqZOBR4Yow=" 416 | }, 417 | "toidentifier": { 418 | "version": "1.0.0", 419 | "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.0.tgz", 420 | "integrity": "sha512-yaOH/Pk/VEhBWWTlhI+qXxDFXlejDGcQipMlyxda9nthulaxLZUNcUqFxokp0vcYnvteJln5FNQDRrxj3YcbVw==" 421 | }, 422 | "type-is": { 423 | "version": "1.6.18", 424 | "resolved": "https://registry.npmjs.org/type-is/-/type-is-1.6.18.tgz", 425 | "integrity": "sha512-TkRKr9sUTxEH8MdfuCSP7VizJyzRNMjj2J2do2Jr3Kym598JVdEksuzPQCnlFPW4ky9Q+iA+ma9BGm06XQBy8g==", 426 | "requires": { 427 | "media-typer": "0.3.0", 428 | "mime-types": "~2.1.24" 429 | } 430 | }, 431 | "uglify-js": { 432 | "version": "3.9.1", 433 | "resolved": "https://registry.npmjs.org/uglify-js/-/uglify-js-3.9.1.tgz", 434 | "integrity": "sha512-JUPoL1jHsc9fOjVFHdQIhqEEJsQvfKDjlubcCilu8U26uZ73qOg8VsN8O1jbuei44ZPlwL7kmbAdM4tzaUvqnA==", 435 | "optional": true, 436 | "requires": { 437 | "commander": "~2.20.3" 438 | } 439 | }, 440 | "unpipe": { 441 | "version": "1.0.0", 442 | "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz", 443 | "integrity": "sha1-sr9O6FFKrmFltIF4KdIbLvSZBOw=" 444 | }, 445 | "utils-merge": { 446 | "version": "1.0.1", 447 | "resolved": "https://registry.npmjs.org/utils-merge/-/utils-merge-1.0.1.tgz", 448 | "integrity": "sha1-n5VxD1CiZ5R7LMwSR0HBAoQn5xM=" 449 | }, 450 | "vary": { 451 | "version": "1.1.2", 452 | "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz", 453 | "integrity": "sha1-IpnwLG3tMNSllhsLn3RSShj2NPw=" 454 | }, 455 | "walk": { 456 | "version": "2.3.14", 457 | "resolved": "https://registry.npmjs.org/walk/-/walk-2.3.14.tgz", 458 | "integrity": "sha512-5skcWAUmySj6hkBdH6B6+3ddMjVQYH5Qy9QGbPmN8kVmLteXk+yVXg+yfk1nbX30EYakahLrr8iPcCxJQSCBeg==", 459 | "requires": { 460 | "foreachasync": "^3.0.0" 461 | } 462 | }, 463 | "wordwrap": { 464 | "version": "1.0.0", 465 | "resolved": "https://registry.npmjs.org/wordwrap/-/wordwrap-1.0.0.tgz", 466 | "integrity": "sha1-J1hIEIkUVqQXHI0CJkQa3pDLyus=" 467 | } 468 | } 469 | } 470 | -------------------------------------------------------------------------------- /package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "openscale_app_demo", 3 | "version": "1.0.0", 4 | "description": "Create a model, monitor with OpenScale, and connect via Node app", 5 | "main": "main.js", 6 | "scripts": { 7 | "start": "node main.js" 8 | }, 9 | "keywords": [ 10 | "openscale", 11 | "watson", 12 | "ai", 13 | "machine", 14 | "learning" 15 | ], 16 | "author": "emartens@us.ibm.com", 17 | "license": "ISC", 18 | "dependencies": { 19 | "axios": "^0.19.2", 20 | "express": "^4.17.1", 21 | "hbs": "^4.1.1" 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /slides.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/ericmartens/openscale_app/563a2a76b5138dbeebf3656b3a26cec6a9d5810c/slides.pptx -------------------------------------------------------------------------------- /views/application.hbs: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Mortgage Application 9 | 10 | 11 |
12 |

Mortgages R Us

13 |
14 | 15 |
16 |
17 |
18 | 19 | 20 |
21 |
22 |
23 |
24 | 25 | 32 |
33 |
34 |
35 |
36 |
37 |
38 | 39 | 40 |
41 |
42 |
43 |
44 | 45 | 46 |
47 |
48 |
49 |
50 |
51 |
52 | 53 | 54 |
55 |
56 |
57 |
58 | 59 | 60 |
61 |
62 |
63 |
64 |
65 |
66 | 67 | 68 |
69 |
70 |
71 |
72 | 73 | 74 |
75 |
76 |
77 |
78 |
79 |
80 | 81 | 82 |
83 |
84 |
85 |
86 | 87 | 98 |
99 |
100 |
101 | 102 |
103 |
104 | 105 | 106 | 107 | 108 | -------------------------------------------------------------------------------- /views/application_results.hbs: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Explain 9 | 10 | 11 |
12 |

Your Mortgage Application

13 |

Thank you for submitting your application. According to our machine learning model, your application represents:

14 |

{{outcome_result}}

15 |

{{outcome_text}}

16 |

Our model was {{confidence}}% confident in this result.

17 | Explain this result 18 |
19 | 20 | 21 | 22 | 23 | 24 | -------------------------------------------------------------------------------- /views/explain.hbs: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Explain 9 | 10 | 11 |
12 |

Explain My Rating

13 |

Using the LIME 14 | algorithm, Watson OpenScale can provide detailed explanations for any predictions made by 15 | the model. LIME works not just for decision trees and linear regressions, but also for complex 16 | neural networks. 17 |

18 |
19 |
20 |

Is this mortgage application risky?

21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 37 | 44 | 45 | 46 |
{{explanation.predictions.0.value}} - {{explanation.predictions.0.probability}}%{{explanation.predictions.1.value}} - {{explanation.predictions.1.probability}}%
31 |
    32 | {{#each explanation.predictions.0.explanation_features}} 33 |
  • {{this.feature_name}} - {{this.weight}}%
    • 34 | {{/each}} 35 |
36 | 		38 |
    39 | {{#each explanation.predictions.1.explanation_features}} 40 |
  • {{this.feature_name}} - {{this.weight}}
    • 41 | {{/each}} 42 |
43 |
47 | File another application 48 |
49 |
50 |
Feature Values
51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | {{#each explanation.input_features}} 60 | 61 | 62 | 63 | 64 | {{/each}} 65 | 66 |
FeatureValue
{{this.name}}{{this.value}}
67 |
68 |
69 |
70 | 71 | 72 | 73 | 74 | -------------------------------------------------------------------------------- /views/index.hbs: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Foundation Starter Template 9 | 10 | 11 |
12 |

Mortgages R Us

13 | Apply today! 14 |
15 | 16 | 17 | 18 | 19 | --------------------------------------------------------------------------------