├── README.md ├── benchmark ├── corrgroups60 │ ├── index.html │ ├── plot_corrgroups60_decision_tree_keep_absolute_impute__r2.pdf │ ├── plot_corrgroups60_decision_tree_keep_absolute_mask__r2.pdf │ ├── plot_corrgroups60_decision_tree_keep_absolute_resample__r2.pdf │ ├── plot_corrgroups60_decision_tree_keep_negative_impute.pdf │ ├── plot_corrgroups60_decision_tree_keep_negative_mask.pdf │ ├── plot_corrgroups60_decision_tree_keep_negative_resample.pdf │ ├── plot_corrgroups60_decision_tree_keep_positive_impute.pdf │ ├── plot_corrgroups60_decision_tree_keep_positive_mask.pdf │ ├── plot_corrgroups60_decision_tree_keep_positive_resample.pdf │ ├── plot_corrgroups60_decision_tree_remove_absolute_impute__r2.pdf │ ├── plot_corrgroups60_decision_tree_remove_absolute_mask__r2.pdf │ ├── plot_corrgroups60_decision_tree_remove_absolute_resample__r2.pdf │ ├── plot_corrgroups60_decision_tree_remove_negative_impute.pdf │ ├── plot_corrgroups60_decision_tree_remove_negative_mask.pdf │ ├── plot_corrgroups60_decision_tree_remove_negative_resample.pdf │ ├── plot_corrgroups60_decision_tree_remove_positive_impute.pdf │ ├── plot_corrgroups60_decision_tree_remove_positive_mask.pdf │ ├── plot_corrgroups60_decision_tree_remove_positive_resample.pdf │ ├── plot_corrgroups60_gbm_keep_absolute_impute__r2.pdf │ ├── plot_corrgroups60_gbm_keep_absolute_mask__r2.pdf │ ├── plot_corrgroups60_gbm_keep_absolute_resample__r2.pdf │ ├── plot_corrgroups60_gbm_keep_negative_impute.pdf │ ├── plot_corrgroups60_gbm_keep_negative_mask.pdf │ ├── plot_corrgroups60_gbm_keep_negative_resample.pdf │ ├── plot_corrgroups60_gbm_keep_positive_impute.pdf │ ├── plot_corrgroups60_gbm_keep_positive_mask.pdf │ ├── plot_corrgroups60_gbm_keep_positive_resample.pdf │ ├── plot_corrgroups60_gbm_remove_absolute_impute__r2.pdf │ ├── plot_corrgroups60_gbm_remove_absolute_mask__r2.pdf │ ├── plot_corrgroups60_gbm_remove_absolute_resample__r2.pdf │ ├── plot_corrgroups60_gbm_remove_negative_impute.pdf │ ├── plot_corrgroups60_gbm_remove_negative_mask.pdf │ ├── plot_corrgroups60_gbm_remove_negative_resample.pdf │ ├── plot_corrgroups60_gbm_remove_positive_impute.pdf │ ├── plot_corrgroups60_gbm_remove_positive_mask.pdf │ ├── plot_corrgroups60_gbm_remove_positive_resample.pdf │ ├── plot_corrgroups60_random_forest_keep_absolute_impute__r2.pdf │ ├── plot_corrgroups60_random_forest_keep_absolute_mask__r2.pdf │ ├── plot_corrgroups60_random_forest_keep_absolute_resample__r2.pdf │ ├── plot_corrgroups60_random_forest_keep_negative_impute.pdf │ ├── plot_corrgroups60_random_forest_keep_negative_mask.pdf │ ├── plot_corrgroups60_random_forest_keep_negative_resample.pdf │ ├── plot_corrgroups60_random_forest_keep_positive_impute.pdf │ ├── plot_corrgroups60_random_forest_keep_positive_mask.pdf │ ├── plot_corrgroups60_random_forest_keep_positive_resample.pdf │ ├── plot_corrgroups60_random_forest_remove_absolute_impute__r2.pdf │ ├── plot_corrgroups60_random_forest_remove_absolute_mask__r2.pdf │ ├── plot_corrgroups60_random_forest_remove_absolute_resample__r2.pdf │ ├── plot_corrgroups60_random_forest_remove_negative_impute.pdf │ ├── plot_corrgroups60_random_forest_remove_negative_mask.pdf │ ├── plot_corrgroups60_random_forest_remove_negative_resample.pdf │ ├── plot_corrgroups60_random_forest_remove_positive_impute.pdf │ ├── plot_corrgroups60_random_forest_remove_positive_mask.pdf │ └── plot_corrgroups60_random_forest_remove_positive_resample.pdf ├── cric │ ├── index.html │ ├── plot_cric_decision_tree_keep_absolute_impute__roc_auc.pdf │ ├── plot_cric_decision_tree_keep_absolute_mask__roc_auc.pdf │ ├── plot_cric_decision_tree_keep_absolute_resample__roc_auc.pdf │ ├── plot_cric_decision_tree_keep_negative_impute.pdf │ ├── plot_cric_decision_tree_keep_negative_mask.pdf │ ├── plot_cric_decision_tree_keep_negative_resample.pdf │ ├── plot_cric_decision_tree_keep_positive_impute.pdf │ ├── plot_cric_decision_tree_keep_positive_mask.pdf │ ├── plot_cric_decision_tree_keep_positive_resample.pdf │ ├── plot_cric_decision_tree_remove_absolute_impute__roc_auc.pdf │ ├── plot_cric_decision_tree_remove_absolute_mask__roc_auc.pdf │ ├── plot_cric_decision_tree_remove_absolute_resample__roc_auc.pdf │ ├── plot_cric_decision_tree_remove_negative_impute.pdf │ ├── plot_cric_decision_tree_remove_negative_mask.pdf │ ├── plot_cric_decision_tree_remove_negative_resample.pdf │ ├── plot_cric_decision_tree_remove_positive_impute.pdf │ ├── plot_cric_decision_tree_remove_positive_mask.pdf │ ├── plot_cric_decision_tree_remove_positive_resample.pdf │ ├── plot_cric_gbm_keep_absolute_impute__roc_auc.pdf │ ├── plot_cric_gbm_keep_absolute_mask__roc_auc.pdf │ ├── plot_cric_gbm_keep_absolute_resample__roc_auc.pdf │ ├── plot_cric_gbm_keep_negative_impute.pdf │ ├── plot_cric_gbm_keep_negative_mask.pdf │ ├── plot_cric_gbm_keep_negative_resample.pdf │ ├── plot_cric_gbm_keep_positive_impute.pdf │ ├── plot_cric_gbm_keep_positive_mask.pdf │ ├── plot_cric_gbm_keep_positive_resample.pdf │ ├── plot_cric_gbm_remove_absolute_impute__roc_auc.pdf │ ├── plot_cric_gbm_remove_absolute_mask__roc_auc.pdf │ ├── plot_cric_gbm_remove_absolute_resample__roc_auc.pdf │ ├── plot_cric_gbm_remove_negative_impute.pdf │ ├── plot_cric_gbm_remove_negative_mask.pdf │ ├── plot_cric_gbm_remove_negative_resample.pdf │ ├── plot_cric_gbm_remove_positive_impute.pdf │ ├── plot_cric_gbm_remove_positive_mask.pdf │ ├── plot_cric_gbm_remove_positive_resample.pdf │ ├── plot_cric_random_forest_keep_absolute_impute__roc_auc.pdf │ ├── plot_cric_random_forest_keep_absolute_mask__roc_auc.pdf │ ├── plot_cric_random_forest_keep_absolute_resample__roc_auc.pdf │ ├── plot_cric_random_forest_keep_negative_impute.pdf │ ├── plot_cric_random_forest_keep_negative_mask.pdf │ ├── plot_cric_random_forest_keep_negative_resample.pdf │ ├── plot_cric_random_forest_keep_positive_impute.pdf │ ├── plot_cric_random_forest_keep_positive_mask.pdf │ ├── plot_cric_random_forest_keep_positive_resample.pdf │ ├── plot_cric_random_forest_remove_absolute_impute__roc_auc.pdf │ ├── plot_cric_random_forest_remove_absolute_mask__roc_auc.pdf │ ├── plot_cric_random_forest_remove_absolute_resample__roc_auc.pdf │ ├── plot_cric_random_forest_remove_negative_impute.pdf │ ├── plot_cric_random_forest_remove_negative_mask.pdf │ ├── plot_cric_random_forest_remove_negative_resample.pdf │ ├── plot_cric_random_forest_remove_positive_impute.pdf │ ├── plot_cric_random_forest_remove_positive_mask.pdf │ └── plot_cric_random_forest_remove_positive_resample.pdf ├── human │ ├── index.html │ ├── plot_human_decision_tree_human_and_00.pdf │ ├── plot_human_decision_tree_human_and_01.pdf │ ├── plot_human_decision_tree_human_and_11.pdf │ ├── plot_human_decision_tree_human_or_00.pdf │ ├── plot_human_decision_tree_human_or_01.pdf │ ├── plot_human_decision_tree_human_or_11.pdf │ ├── plot_human_decision_tree_human_sum_00.pdf │ ├── plot_human_decision_tree_human_sum_01.pdf │ ├── plot_human_decision_tree_human_sum_11.pdf │ ├── plot_human_decision_tree_human_xor_00.pdf │ ├── plot_human_decision_tree_human_xor_01.pdf │ └── plot_human_decision_tree_human_xor_11.pdf └── independentlinear60 │ ├── index.html │ ├── plot_independentlinear60_decision_tree_keep_absolute_impute__r2.pdf │ ├── plot_independentlinear60_decision_tree_keep_absolute_mask__r2.pdf │ ├── plot_independentlinear60_decision_tree_keep_absolute_resample__r2.pdf │ ├── plot_independentlinear60_decision_tree_keep_negative_impute.pdf │ ├── plot_independentlinear60_decision_tree_keep_negative_mask.pdf │ ├── plot_independentlinear60_decision_tree_keep_negative_resample.pdf │ ├── plot_independentlinear60_decision_tree_keep_positive_impute.pdf │ ├── plot_independentlinear60_decision_tree_keep_positive_mask.pdf │ ├── plot_independentlinear60_decision_tree_keep_positive_resample.pdf │ ├── plot_independentlinear60_decision_tree_remove_absolute_impute__r2.pdf │ ├── plot_independentlinear60_decision_tree_remove_absolute_mask__r2.pdf │ ├── plot_independentlinear60_decision_tree_remove_absolute_resample__r2.pdf │ ├── plot_independentlinear60_decision_tree_remove_negative_impute.pdf │ ├── plot_independentlinear60_decision_tree_remove_negative_mask.pdf │ ├── plot_independentlinear60_decision_tree_remove_negative_resample.pdf │ ├── plot_independentlinear60_decision_tree_remove_positive_impute.pdf │ ├── plot_independentlinear60_decision_tree_remove_positive_mask.pdf │ ├── plot_independentlinear60_decision_tree_remove_positive_resample.pdf │ ├── plot_independentlinear60_gbm_keep_absolute_impute__r2.pdf │ ├── plot_independentlinear60_gbm_keep_absolute_mask__r2.pdf │ ├── plot_independentlinear60_gbm_keep_absolute_resample__r2.pdf │ ├── plot_independentlinear60_gbm_keep_negative_impute.pdf │ ├── plot_independentlinear60_gbm_keep_negative_mask.pdf │ ├── plot_independentlinear60_gbm_keep_negative_resample.pdf │ ├── plot_independentlinear60_gbm_keep_positive_impute.pdf │ ├── plot_independentlinear60_gbm_keep_positive_mask.pdf │ ├── plot_independentlinear60_gbm_keep_positive_resample.pdf │ ├── plot_independentlinear60_gbm_remove_absolute_impute__r2.pdf │ ├── plot_independentlinear60_gbm_remove_absolute_mask__r2.pdf │ ├── plot_independentlinear60_gbm_remove_absolute_resample__r2.pdf │ ├── plot_independentlinear60_gbm_remove_negative_impute.pdf │ ├── plot_independentlinear60_gbm_remove_negative_mask.pdf │ ├── plot_independentlinear60_gbm_remove_negative_resample.pdf │ ├── plot_independentlinear60_gbm_remove_positive_impute.pdf │ ├── plot_independentlinear60_gbm_remove_positive_mask.pdf │ ├── plot_independentlinear60_gbm_remove_positive_resample.pdf │ ├── plot_independentlinear60_random_forest_keep_absolute_impute__r2.pdf │ ├── plot_independentlinear60_random_forest_keep_absolute_mask__r2.pdf │ ├── plot_independentlinear60_random_forest_keep_absolute_resample__r2.pdf │ ├── plot_independentlinear60_random_forest_keep_negative_impute.pdf │ ├── plot_independentlinear60_random_forest_keep_negative_mask.pdf │ ├── plot_independentlinear60_random_forest_keep_negative_resample.pdf │ ├── plot_independentlinear60_random_forest_keep_positive_impute.pdf │ ├── plot_independentlinear60_random_forest_keep_positive_mask.pdf │ ├── plot_independentlinear60_random_forest_keep_positive_resample.pdf │ ├── plot_independentlinear60_random_forest_remove_absolute_impute__r2.pdf │ ├── plot_independentlinear60_random_forest_remove_absolute_mask__r2.pdf │ ├── plot_independentlinear60_random_forest_remove_absolute_resample__r2.pdf │ ├── plot_independentlinear60_random_forest_remove_negative_impute.pdf │ ├── plot_independentlinear60_random_forest_remove_negative_mask.pdf │ ├── plot_independentlinear60_random_forest_remove_negative_resample.pdf │ ├── plot_independentlinear60_random_forest_remove_positive_impute.pdf │ ├── plot_independentlinear60_random_forest_remove_positive_mask.pdf │ └── plot_independentlinear60_random_forest_remove_positive_resample.pdf ├── images └── overview_header.png ├── notebooks ├── Saabas Inconsistencies.ipynb ├── kidney │ ├── CRIC Analysis.ipynb │ └── convergence_plots │ │ ├── .ipynb_checkpoints │ │ ├── Generate and Save Deviations-checkpoint.ipynb │ │ └── Plots-checkpoint.ipynb │ │ ├── Generate and Save Deviations.ipynb │ │ ├── Plots.ipynb │ │ ├── kernel_expl_abs_dev_single_ref_truth_max_dict.p │ │ ├── kernel_expl_abs_dev_single_ref_truth_mean_dict.p │ │ ├── max_kidney_agnostic_convergence.pdf │ │ ├── mean_kidney_agnostic_convergence.pdf │ │ ├── sample_expl_abs_dev_single_ref_truth_max_dict.p │ │ └── sample_expl_abs_dev_single_ref_truth_mean_dict.p ├── mortality │ ├── NHANES I Analysis.ipynb │ ├── NHANES Nonlinearity.ipynb │ ├── data │ │ ├── DU4111.txt │ │ ├── DU4171.txt │ │ ├── DU4233.txt │ │ ├── DU4233_F.sas │ │ ├── DU4800.txt │ │ ├── N92vitl.txt │ │ ├── X_orig.csv │ │ └── y_orig.npy │ └── loadnhanes.py ├── simulation │ ├── .ipynb_checkpoints │ │ ├── Generate and Save Deviations-checkpoint.ipynb │ │ └── Plot Runtime and Convergence-checkpoint.ipynb │ ├── Generate and Save Deviations.ipynb │ ├── Plot Runtime and Convergence.ipynb │ ├── Saabas Inconsistencies.ipynb │ └── scratch │ │ ├── ime_m_lst.npy │ │ ├── ime_std_lst.npy │ │ ├── kernel_shap_m_lst.npy │ │ └── kernel_shap_std_lst.npy └── timeseries │ ├── Explain LSTM->XGB.ipynb │ ├── Explain XGB.ipynb │ ├── Run LSTM->XGB.ipynb │ ├── Run LSTM.ipynb │ ├── Run XGB.ipynb │ └── tbi_downstream_prediction.py └── user_studies └── sickness_scores ├── .ipynb_checkpoints └── Analyse Results-checkpoint.ipynb ├── .psiturk_history ├── Analyse Results.ipynb ├── Debrief.jsx ├── Instructions.jsx ├── Main.jsx ├── Questions.jsx ├── README.md ├── config.txt ├── index.jsx ├── package.json ├── static ├── css │ ├── bootstrap.min.css │ └── style.css ├── favicon.ico ├── fonts │ ├── glyphicons-halflings-regular.eot │ ├── glyphicons-halflings-regular.svg │ ├── glyphicons-halflings-regular.ttf │ └── glyphicons-halflings-regular.woff ├── images │ ├── Example1.png │ ├── Example1_nobar.png │ ├── Example2.png │ ├── Example2_nobar.png │ └── university.png ├── js │ ├── bundle.js │ ├── task.js │ └── utils.js └── lib │ ├── backbone-min.js │ ├── bootstrap.min.js │ ├── d3.v3.min.js │ ├── jquery-min.js │ └── underscore-min.js ├── templates ├── ad.html ├── closepopup.html ├── complete.html ├── consent.html ├── default.html ├── error.html ├── exp.html ├── thanks-mturksubmit.html └── thanks.html └── webpack.config.js /README.md: -------------------------------------------------------------------------------- 1 | # TreeExplainer Paper Repository 2 | 3 |
4 |
5 |
Thank you for your participation in our study! Your anonymous data makes an 49 | important contribution to our understanding of human machine interaction.
50 | 51 |If you have any questions about this research, you may contact slund1@cs.washington.edu.
52 |I feel that I have been adequately debriefed about the nature 65 | of the study. The investigator has explained the purposes of the 66 | research to me, and I feel that any questions I have asked were 67 | satisfactorily answered.
68 | 69 | 72 | 73 |"Sickness score {this.score_names[this.state.score]}" is computed in the following manner:
245 | 246 |250 | If they have a fever: 251 | | 252 |253 | +2 points 254 | | 255 |
258 | If they have a cough: 259 | | 260 |261 | +2 points 262 | | 263 |
273 | Fever: 274 | | 275 |276 | {this.symptoms[this.state.person]['fever'] ? 'YES' : 'NO'} 277 | | 278 |
281 | Cough: 282 | | 283 |284 | {this.symptoms[this.state.person]['cough'] ? 'YES' : 'NO'} 285 | | 286 |
289 | Headache: 290 | | 291 |292 | {this.symptoms[this.state.person]['headache'] ? 'YES' : 'NO'} 293 | | 294 |
313 | 314 | Fever: {this.symptoms[this.state.person]['fever'] ? 'YES' : 'NO'} 315 | |
316 |
317 | 318 | Cough: {this.symptoms[this.state.person]['cough'] ? 'YES' : 'NO'} 319 | |
320 |
321 | 322 | Headache: {this.symptoms[this.state.person]['headache'] ? 'YES' : 'NO'} 323 | |
324 |
Type "R" for Red, "B" for blue, "G" for green.
'); 77 | } 78 | }; 79 | 80 | var response_handler = function(e) { 81 | if (!listening) return; 82 | 83 | var keyCode = e.keyCode, 84 | response; 85 | 86 | switch (keyCode) { 87 | case 82: 88 | // "R" 89 | response="red"; 90 | break; 91 | case 71: 92 | // "G" 93 | response="green"; 94 | break; 95 | case 66: 96 | // "B" 97 | response="blue"; 98 | break; 99 | default: 100 | response = ""; 101 | break; 102 | } 103 | if (response.length>0) { 104 | listening = false; 105 | var hit = response == stim[1]; 106 | var rt = new Date().getTime() - wordon; 107 | 108 | psiTurk.recordTrialData({'phase':"TEST", 109 | 'word':stim[0], 110 | 'color':stim[1], 111 | 'relation':stim[2], 112 | 'response':response, 113 | 'hit':hit, 114 | 'rt':rt} 115 | ); 116 | remove_word(); 117 | next(); 118 | } 119 | }; 120 | 121 | var finish = function() { 122 | $("body").unbind("keydown", response_handler); // Unbind keys 123 | currentview = new Questionnaire(); 124 | }; 125 | 126 | var show_word = function(text, color) { 127 | d3.select("#stim") 128 | .append("div") 129 | .attr("id","word") 130 | .style("color",color) 131 | .style("text-align","center") 132 | .style("font-size","150px") 133 | .style("font-weight","400") 134 | .style("margin","20px") 135 | .text(text); 136 | }; 137 | 138 | var remove_word = function() { 139 | d3.select("#word").remove(); 140 | }; 141 | 142 | 143 | // Load the stage.html snippet into the body of the page 144 | psiTurk.showPage('stage.html'); 145 | 146 | // Register the response handler that is defined above to handle any 147 | // key down events. 148 | $("body").focus().keydown(response_handler); 149 | 150 | // Start the test 151 | next(); 152 | }; 153 | 154 | 155 | /**************** 156 | * Questionnaire * 157 | ****************/ 158 | 159 | var Questionnaire = function() { 160 | 161 | var error_message = "Something went wrong submitting your HIT. This might happen if you lose your internet connection. Press the button to resubmit.
"; 162 | 163 | record_responses = function() { 164 | 165 | psiTurk.recordTrialData({'phase':'postquestionnaire', 'status':'submit'}); 166 | 167 | $('textarea').each( function(i, val) { 168 | psiTurk.recordUnstructuredData(this.id, this.value); 169 | }); 170 | $('select').each( function(i, val) { 171 | psiTurk.recordUnstructuredData(this.id, this.value); 172 | }); 173 | 174 | }; 175 | 176 | prompt_resubmit = function() { 177 | document.body.innerHTML = error_message; 178 | $("#resubmit").click(resubmit); 179 | }; 180 | 181 | resubmit = function() { 182 | document.body.innerHTML = "74 | We are looking for online participants 75 | for a brief experiment about understanding artificial intellegence 76 | computer predictions. 77 | The only requirements are that you are at least 18 years old and are a fluent English 78 | speaker. The task will take less than 2 minutes. 79 |
80 |87 | Otherwise, please click the "Accept HIT" button on the Amazon site 88 | above to begin the task. 89 |
90 | 91 | 92 | {% else %} 93 | 94 | 106 |108 | By clicking the following URL link, you will be taken to the experiment, 109 | including complete instructions and an informed consent agreement. 110 |
111 | 116 |You are being redirected back to the HIT window, which should refresh. 13 | If you do not see the HIT submission form when you return, try refreshing 14 | the page.
15 | 16 | 17 | -------------------------------------------------------------------------------- /user_studies/sickness_scores/templates/complete.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 |28 | You have been invited to take part in research designed to compare 29 | different ways of explaining computer predictions. 30 |
31 |32 | Aggregate information about your responses will be shared publically, and while we 33 | will make no attempt to identify worker identities we cannot guarantee the 34 | privacy of your responses. You also must not be currently working for the University of Washington. 35 |
36 | 37 | 40 |Sorry, our records indicate that you have attempted to complete this HIT, but quit before finishing.
20 |Because this is a Psychology experiment, you can only complete this HIT once.
21 |Please return the HIT so someone else can perform the experiment.
22 | 23 | {% elif errornum == 1010 %} 24 | 25 |Sorry, our records indicate that you have already completed (or attempted to complete) this HIT.
26 |Because this is a Psychology experiment, you can only complete this HIT once.
27 |Please release the HIT so someone else can perform the experiment.
28 | 29 | {% else %} 30 | 31 |Sorry, there was an unexpected error in our processing of your HIT.
32 | 33 | {% endif %} 34 |35 | If you would like to receive partial compensation or feel you 36 | have reached this page in error, please email 37 | {{contact_address}} 38 | and send the following information: 39 |
40 |
42 | Error: {{ errornum }}
43 | Description: {{ errordesc }}
44 | {% if hitId %}HITId: #{{ hitId }}
{% endif %}
45 | {% if assignId %}AssignmentId: #{{ assignId }}
{% endif %}
46 | {% if workerId %}WorkerId: #{{ workerId }}
{% endif %}
47 |
To complete the HIT, simply press the button below.
30 | 31 | {% if using_sandbox %} 32 | 40 | 41 | 44 | 45 |