![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
12 | "\n",
13 | "\n"
14 | ]
15 | },
16 | {
17 | "cell_type": "markdown",
18 | "metadata": {
19 | "id": "NrP6gOUsaVsy"
20 | },
21 | "source": [
22 | "## Classifier Evaluation\n",
23 | "\n",
24 | "### True Positive (TP) \n",
25 | "\n",
26 | "The predicted value matches the actual value\n",
27 | "The actual value was positive and the model predicted a positive value\n",
28 | "\n",
29 | "### True Negative (TN) \n",
30 | "\n",
31 | "The predicted value matches the actual value\n",
32 | "The actual value was negative and the model predicted a negative value\n",
33 | "\n",
34 | "### False Positive (FP) – Type 1 error\n",
35 | "\n",
36 | "The predicted value was falsely predicted\n",
37 | "The actual value was negative but the model predicted a positive value\n",
38 | "Also known as the Type 1 error\n",
39 | "\n",
40 | "### False Negative (FN) – Type 2 error\n",
41 | "\n",
42 | "The predicted value was falsely predicted\n",
43 | "The actual value was positive but the model predicted a negative value\n",
44 | "Also known as the Type 2 error\n",
45 | "\n",
46 | "### Precision\n",
47 | "Precision is a good measure to determine, when the costs of False Positive is high. For instance, email spam detection. In email spam detection, a false positive means that an email that is non-spam (actual negative) has been identified as spam (predicted spam). The email user might lose important emails if the precision is not high for the spam detection model.\n",
48 | "\n",
49 | "$$Precision = \\dfrac{TP}{TP + FP}$$\n",
50 | "\n",
51 | "### Recall\n",
52 | "Recall shall be the model metric we use to select our best model when there is a high cost associated with False Negative.\n",
53 | "in sick patient detection. If a sick patient (Actual Positive) goes through the test and predicted as not sick (Predicted Negative). The cost associated with False Negative will be extremely high if the sickness is contagious.\n",
54 | "\n",
55 | "$$Recall = \\dfrac{TP}{TP + FN}$$\n",
56 | "\n",
57 | "### F1 score\n",
58 | "F1 Score might be a better measure to use if we need to seek a balance between Precision and Recall AND there is an uneven class distribution (large number of Actual Negatives).\n",
59 | "\n",
60 | "$$F1 = 2 \\cdot \\dfrac{Precision \\cdot Recall}{Precision + Recall}$$\n",
61 | "\n",
62 | "### Confusion Matrix\n",
63 | "A Confusion matrix is an N x N matrix used for evaluating the performance of a classification model, where N is the number of target classes. The matrix compares the actual target values with those predicted by the machine learning model. This gives us a holistic view of how well our classification model is performing and what kinds of errors it is making."
64 | ]
65 | },
66 | {
67 | "cell_type": "code",
68 | "execution_count": 6,
69 | "metadata": {
70 | "id": "WVfD0etJidZq",
71 | "vscode": {
72 | "languageId": "python"
73 | }
74 | },
75 | "outputs": [],
76 | "source": [
77 | "from sklearn import (\n",
78 | " linear_model,\n",
79 | " metrics,\n",
80 | " datasets,\n",
81 | " model_selection,\n",
82 | " pipeline,\n",
83 | " preprocessing,\n",
84 | ")"
85 | ]
86 | },
87 | {
88 | "cell_type": "code",
89 | "execution_count": 7,
90 | "metadata": {
91 | "id": "1LqNVHlaikt4",
92 | "vscode": {
93 | "languageId": "python"
94 | }
95 | },
96 | "outputs": [],
97 | "source": [
98 | "x, y = datasets.load_breast_cancer(return_X_y=True)\n",
99 | "x_train, x_test, y_train, y_test = model_selection.train_test_split(\n",
100 | " x, y, test_size=0.1, random_state=42, stratify=y\n",
101 | ")"
102 | ]
103 | },
104 | {
105 | "cell_type": "code",
106 | "execution_count": 8,
107 | "metadata": {
108 | "id": "510UXIxaB5zV",
109 | "vscode": {
110 | "languageId": "python"
111 | }
112 | },
113 | "outputs": [],
114 | "source": [
115 | "# Create the pipeline from the scaler and the estimator\n",
116 | "# Scaler\n",
117 | "scaler = preprocessing.StandardScaler()\n",
118 | "# Estimator\n",
119 | "estimator = linear_model.LogisticRegression(solver=\"liblinear\", random_state=42)\n",
120 | "# Create the pipeline\n",
121 | "pipe = pipeline.Pipeline([(\"scaler\", scaler), (\"estimator\", estimator)])"
122 | ]
123 | },
124 | {
125 | "cell_type": "code",
126 | "execution_count": 9,
127 | "metadata": {
128 | "colab": {
129 | "base_uri": "https://localhost:8080/"
130 | },
131 | "id": "ti3WlBgD40JL",
132 | "outputId": "b20ca8c4-4c5a-467d-acc4-e4644e663b42",
133 | "vscode": {
134 | "languageId": "python"
135 | }
136 | },
137 | "outputs": [
138 | {
139 | "data": {
140 | "text/plain": [
141 | "Pipeline(steps=[('scaler', StandardScaler()),\n",
142 | " ('estimator',\n",
143 | " LogisticRegression(random_state=42, solver='liblinear'))])"
144 | ]
145 | },
146 | "execution_count": 9,
147 | "metadata": {},
148 | "output_type": "execute_result"
149 | }
150 | ],
151 | "source": [
152 | "# fit the pipeline of the x_train and y_train\n",
153 | "pipe.fit(x_train, y_train)"
154 | ]
155 | },
156 | {
157 | "cell_type": "code",
158 | "execution_count": 10,
159 | "metadata": {
160 | "colab": {
161 | "base_uri": "https://localhost:8080/",
162 | "height": 511
163 | },
164 | "id": "nevs86s-i-t0",
165 | "outputId": "a13d6cb1-56b2-4eec-e9d3-7fe3df8ec492",
166 | "vscode": {
167 | "languageId": "python"
168 | }
169 | },
170 | "outputs": [
171 | {
172 | "name": "stdout",
173 | "output_type": "stream",
174 | "text": [
175 | "Accuracy Score: 0.9649122807017544\n",
176 | "True Positive: 35\n",
177 | "True Negative: 20\n",
178 | "False Positive: 1\n",
179 | "False Negative: 1\n",
180 | "Precision Score: 0.9722222222222222\n",
181 | "Precision Score Manual: 0.9722222222222222\n",
182 | "Recall Score: 0.9722222222222222\n",
183 | "Recall Score Manual: 0.9722222222222222\n"
184 | ]
185 | },
186 | {
187 | "name": "stderr",
188 | "output_type": "stream",
189 | "text": [
190 | "/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function plot_confusion_matrix is deprecated; Function `plot_confusion_matrix` is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: ConfusionMatrixDisplay.from_predictions or ConfusionMatrixDisplay.from_estimator.\n",
191 | " warnings.warn(msg, category=FutureWarning)\n"
192 | ]
193 | },
194 | {
195 | "data": {
196 | "text/plain": [
197 | "\n",
13 | "\n",
14 | "Feature selection is a process where you automatically or manually select those features in your data that contribute most to the prediction variable or output in which you are interested.\n",
15 | "\n",
16 | "There are many ways to perform feature selection. Some methods include:\n",
17 | "\n",
18 | "- Using a correlation matrix to select features that are highly correlated with the output variable\n",
19 | "- Using a statistical tests (e.g., $x^2$) to measure the statistical significance of each feature in relation to the label. \n",
20 | "- Using a recursive feature elimination algorithm to automatically select features that are most relevant to the output variable\n",
21 | "\n",
22 | "Feature selection is important in machine learning because it can help you reduce the amount of data you need to work with, which in turn can reduce the amount of time and resources required to train and tune your machine learning model. Additionally, by selecting only the most relevant features, you can improve the interpretability of your machine learning model and make overfitting less likley to occure.\n",
23 | "\n",
24 | "There are many feature selection techniques supported by Scikit Learn. We will go through some of them and if you wish to learn more visit [Scikit Learn's Documentation](https://scikit-learn.org/stable/modules/feature_selection.html)."
25 | ]
26 | },
27 | {
28 | "cell_type": "code",
29 | "execution_count": 8,
30 | "metadata": {
31 | "id": "3euJPnU0fcCU"
32 | },
33 | "outputs": [],
34 | "source": [
35 | "from sklearn import (\n",
36 | " datasets,\n",
37 | " model_selection,\n",
38 | " feature_selection,\n",
39 | " svm,\n",
40 | " metrics,\n",
41 | " pipeline,\n",
42 | " preprocessing,\n",
43 | ")"
44 | ]
45 | },
46 | {
47 | "cell_type": "code",
48 | "execution_count": 9,
49 | "metadata": {
50 | "colab": {
51 | "base_uri": "https://localhost:8080/"
52 | },
53 | "id": "4zwq1k1zfae4",
54 | "outputId": "74223840-9527-44f7-c474-f40dcb1b865f"
55 | },
56 | "outputs": [
57 | {
58 | "data": {
59 | "text/plain": [
60 | "(569, 30)"
61 | ]
62 | },
63 | "execution_count": 9,
64 | "metadata": {},
65 | "output_type": "execute_result"
66 | }
67 | ],
68 | "source": [
69 | "# Load iris dataset\n",
70 | "x, y = datasets.load_breast_cancer(return_X_y=True)\n",
71 | "\n",
72 | "x.shape"
73 | ]
74 | },
75 | {
76 | "cell_type": "markdown",
77 | "metadata": {
78 | "id": "U5FBvhd5-EBE"
79 | },
80 | "source": [
81 | "## Feature Selection Methods"
82 | ]
83 | },
84 | {
85 | "cell_type": "markdown",
86 | "metadata": {
87 | "id": "iZStJ23HkfrE"
88 | },
89 | "source": [
90 | "### Univariate Feature Selection\n",
91 | "\n",
92 | "Univariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator.\n",
93 | "\n",
94 | "We will be using the `SelectKBest` class from Scikit Learn which returing the best $k$ features using some scoring method based on a statistical significance test like $x^2$ (chi-square).\n",
95 | "\n",
96 | "Scikit Learn includes additional scoring functions like:\n",
97 | "\n",
98 | "- For classification: `chi2`, `f_classif`, `mutual_info_classif`\n",
99 | "- For regression: `f_regression`, `mutual_info_regression`\n",
100 | "\n"
101 | ]
102 | },
103 | {
104 | "cell_type": "code",
105 | "execution_count": 10,
106 | "metadata": {
107 | "colab": {
108 | "base_uri": "https://localhost:8080/"
109 | },
110 | "id": "zdIpHV6nwaEb",
111 | "outputId": "da8ea4e4-d5fb-4f37-eea6-9461b7a9b4d9"
112 | },
113 | "outputs": [
114 | {
115 | "data": {
116 | "text/plain": [
117 | "(569, 10)"
118 | ]
119 | },
120 | "execution_count": 10,
121 | "metadata": {},
122 | "output_type": "execute_result"
123 | }
124 | ],
125 | "source": [
126 | "# Define the feature selector by selecing the scoring function and the number of features to select\n",
127 | "feature_selector = feature_selection.SelectKBest(feature_selection.chi2, k=10)\n",
128 | "\n",
129 | "# Use fit transform to train the feature selector and return the best 10 features\n",
130 | "x_new = feature_selector.fit_transform(x, y)\n",
131 | "\n",
132 | "# New X will have 10 features instead of 30\n",
133 | "x_new.shape"
134 | ]
135 | },
136 | {
137 | "cell_type": "markdown",
138 | "metadata": {
139 | "id": "ei4QcwSF0v6w"
140 | },
141 | "source": [
142 | "## Training & Evaluating"
143 | ]
144 | },
145 | {
146 | "cell_type": "code",
147 | "execution_count": 11,
148 | "metadata": {
149 | "id": "G6JJ70wZ0v6x"
150 | },
151 | "outputs": [],
152 | "source": [
153 | "x_train, x_test, y_train, y_test = model_selection.train_test_split(\n",
154 | " x_new, y, test_size=0.3, stratify=y, random_state=42\n",
155 | ")"
156 | ]
157 | },
158 | {
159 | "cell_type": "code",
160 | "execution_count": 12,
161 | "metadata": {
162 | "colab": {
163 | "base_uri": "https://localhost:8080/"
164 | },
165 | "id": "szO8ej9U0v6y",
166 | "outputId": "b497ea9b-99ba-4fd0-a69e-fe6eaad73ce0"
167 | },
168 | "outputs": [
169 | {
170 | "name": "stdout",
171 | "output_type": "stream",
172 | "text": [
173 | "Training model on 10 features\n"
174 | ]
175 | },
176 | {
177 | "name": "stderr",
178 | "output_type": "stream",
179 | "text": [
180 | "/usr/local/lib/python3.7/dist-packages/sklearn/svm/_base.py:289: ConvergenceWarning: Solver terminated early (max_iter=10). Consider pre-processing your data with StandardScaler or MinMaxScaler.\n",
181 | " ConvergenceWarning,\n"
182 | ]
183 | },
184 | {
185 | "data": {
186 | "text/plain": [
187 | "Pipeline(steps=[('scaler', StandardScaler()),\n",
188 | " ('model', SVC(max_iter=10, random_state=10))])"
189 | ]
190 | },
191 | "execution_count": 12,
192 | "metadata": {},
193 | "output_type": "execute_result"
194 | }
195 | ],
196 | "source": [
197 | "print(f\"Training model on {x_train.shape[1]} features\")\n",
198 | "\n",
199 | "# Define a Support Vector Machine classifier with default configuration\n",
200 | "model = pipeline.Pipeline(\n",
201 | " [\n",
202 | " (\"scaler\", preprocessing.StandardScaler()),\n",
203 | " (\"model\", svm.SVC(kernel=\"rbf\", random_state=10, max_iter=10)),\n",
204 | " ]\n",
205 | ")\n",
206 | "\n",
207 | "# Train model using the training dataset\n",
208 | "model.fit(x_train, y_train)"
209 | ]
210 | },
211 | {
212 | "cell_type": "code",
213 | "execution_count": 13,
214 | "metadata": {
215 | "colab": {
216 | "base_uri": "https://localhost:8080/"
217 | },
218 | "id": "oq-XRDaP0v6z",
219 | "outputId": "2c0d747a-6e0e-47ae-e877-5285ebdaf2d9"
220 | },
221 | "outputs": [
222 | {
223 | "name": "stdout",
224 | "output_type": "stream",
225 | "text": [
226 | "Accuracy: 0.7777777777777778\n",
227 | "Precision: 0.8876404494382022\n",
228 | "Recall: 0.7383177570093458\n"
229 | ]
230 | }
231 | ],
232 | "source": [
233 | "# Use the model to predict the testing set to prepare for calculating the metrics\n",
234 | "y_pred = model.predict(x_test)\n",
235 | "\n",
236 | "# Calculate and print relevant scores\n",
237 | "accuracy = metrics.accuracy_score(y_test, y_pred)\n",
238 | "precision = metrics.precision_score(y_test, y_pred)\n",
239 | "recall = metrics.recall_score(y_test, y_pred)\n",
240 | "\n",
241 | "print(\"Accuracy:\", accuracy)\n",
242 | "print(\"Precision:\", precision)\n",
243 | "print(\"Recall:\", recall)"
244 | ]
245 | },
246 | {
247 | "cell_type": "markdown",
248 | "metadata": {
249 | "id": "gkav1r_6QLHz"
250 | },
251 | "source": [
252 | "# Feature selection using Select From Model"
253 | ]
254 | },
255 | {
256 | "cell_type": "markdown",
257 | "metadata": {
258 | "id": "eF3dd6CnQVuS"
259 | },
260 | "source": [
261 | "[SelectFromModel](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html#sklearn.feature_selection.SelectFromModel) is a meta-transformer that can be used alongside any estimator that assigns importance to each feature through a specific attribute (such as coef_, feature_importances_) or via an importance_getter callable after fitting. \n",
262 | "\n",
263 | "The features are considered unimportant and removed if the corresponding importance of the feature values are below the provided threshold parameter. \n",
264 | "\n",
265 | "Apart from specifying the threshold numerically, there are built-in heuristics for finding a threshold using a string argument. Available heuristics are “mean”, “median” and float multiples of these like “0.1*mean”. \n",
266 | "\n",
267 | "In combination with the threshold criteria, one can use the max_features parameter to set a limit on the number of features to select."
268 | ]
269 | },
270 | {
271 | "cell_type": "markdown",
272 | "metadata": {
273 | "id": "jQ7k6NsdV216"
274 | },
275 | "source": [
276 | "## Define and Train the Linear Model"
277 | ]
278 | },
279 | {
280 | "cell_type": "code",
281 | "execution_count": 14,
282 | "metadata": {
283 | "colab": {
284 | "base_uri": "https://localhost:8080/"
285 | },
286 | "id": "2ks-dw3vT7tx",
287 | "outputId": "4c423421-a540-4cc8-8614-ba96ce3c6b58"
288 | },
289 | "outputs": [
290 | {
291 | "name": "stderr",
292 | "output_type": "stream",
293 | "text": [
294 | "/usr/local/lib/python3.7/dist-packages/sklearn/svm/_base.py:1208: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
295 | " ConvergenceWarning,\n"
296 | ]
297 | },
298 | {
299 | "data": {
300 | "text/plain": [
301 | "LinearSVC()"
302 | ]
303 | },
304 | "execution_count": 14,
305 | "metadata": {},
306 | "output_type": "execute_result"
307 | }
308 | ],
309 | "source": [
310 | "lsvc = svm.LinearSVC()\n",
311 | "lsvc.fit(x, y)"
312 | ]
313 | },
314 | {
315 | "cell_type": "markdown",
316 | "metadata": {
317 | "id": "MzkHBxM1V_eN"
318 | },
319 | "source": [
320 | "## Use the Trained Linear Model to Get the Best Features"
321 | ]
322 | },
323 | {
324 | "cell_type": "code",
325 | "execution_count": 16,
326 | "metadata": {
327 | "colab": {
328 | "base_uri": "https://localhost:8080/"
329 | },
330 | "id": "a7YXLQWLUBup",
331 | "outputId": "870be180-5c1b-4c4d-b713-7f3f9135147b"
332 | },
333 | "outputs": [
334 | {
335 | "data": {
336 | "text/plain": [
337 | "(569, 8)"
338 | ]
339 | },
340 | "execution_count": 16,
341 | "metadata": {},
342 | "output_type": "execute_result"
343 | }
344 | ],
345 | "source": [
346 | "feature_selector = feature_selection.SelectFromModel(lsvc, prefit=True)\n",
347 | "x_new = feature_selector.transform(\n",
348 | " x\n",
349 | ") # get the best features using the pretrained model\n",
350 | "x_new.shape"
351 | ]
352 | },
353 | {
354 | "cell_type": "markdown",
355 | "metadata": {
356 | "id": "3dugS7ARWgKE"
357 | },
358 | "source": [
359 | "## Split the Datase to Train and Test "
360 | ]
361 | },
362 | {
363 | "cell_type": "code",
364 | "execution_count": 17,
365 | "metadata": {
366 | "id": "0VRTE-r8UPtb"
367 | },
368 | "outputs": [],
369 | "source": [
370 | "x_train, x_test, y_train, y_test = model_selection.train_test_split(\n",
371 | " x_new, y, test_size=0.3, stratify=y, random_state=42\n",
372 | ")"
373 | ]
374 | },
375 | {
376 | "cell_type": "markdown",
377 | "metadata": {
378 | "id": "nN_DtPG8Vrc9"
379 | },
380 | "source": [
381 | "## Training & Evaluating"
382 | ]
383 | },
384 | {
385 | "cell_type": "code",
386 | "execution_count": 20,
387 | "metadata": {
388 | "colab": {
389 | "base_uri": "https://localhost:8080/"
390 | },
391 | "id": "T9O9CXCCU9ll",
392 | "outputId": "d0dec91b-ff98-4be6-d71a-5f29778877da"
393 | },
394 | "outputs": [
395 | {
396 | "name": "stdout",
397 | "output_type": "stream",
398 | "text": [
399 | "Training model on 8 features\n"
400 | ]
401 | },
402 | {
403 | "name": "stderr",
404 | "output_type": "stream",
405 | "text": [
406 | "/usr/local/lib/python3.7/dist-packages/sklearn/svm/_base.py:289: ConvergenceWarning: Solver terminated early (max_iter=10). Consider pre-processing your data with StandardScaler or MinMaxScaler.\n",
407 | " ConvergenceWarning,\n"
408 | ]
409 | },
410 | {
411 | "data": {
412 | "text/plain": [
413 | "Pipeline(steps=[('scaler', StandardScaler()),\n",
414 | " ('model', SVC(max_iter=10, random_state=10))])"
415 | ]
416 | },
417 | "execution_count": 20,
418 | "metadata": {},
419 | "output_type": "execute_result"
420 | }
421 | ],
422 | "source": [
423 | "print(f\"Training model on {x_train.shape[1]} features\")\n",
424 | "\n",
425 | "# Define a Support Vector Machine classifier with default configuration\n",
426 | "model = pipeline.Pipeline(\n",
427 | " [\n",
428 | " (\"scaler\", preprocessing.StandardScaler()),\n",
429 | " (\"model\", svm.SVC(kernel=\"rbf\", random_state=10, max_iter=10)),\n",
430 | " ]\n",
431 | ")\n",
432 | "\n",
433 | "# Train model using the training dataset\n",
434 | "model.fit(x_train, y_train)"
435 | ]
436 | },
437 | {
438 | "cell_type": "code",
439 | "execution_count": 21,
440 | "metadata": {
441 | "colab": {
442 | "base_uri": "https://localhost:8080/"
443 | },
444 | "id": "EH4fPo7eVCF5",
445 | "outputId": "471791b6-c918-4963-aed8-0dd046ab5d6c"
446 | },
447 | "outputs": [
448 | {
449 | "name": "stdout",
450 | "output_type": "stream",
451 | "text": [
452 | "Accuracy: 0.9298245614035088\n",
453 | "Precision: 0.9279279279279279\n",
454 | "Recall: 0.9626168224299065\n"
455 | ]
456 | }
457 | ],
458 | "source": [
459 | "# Use the model to predict the testing set to prepare for calculating the metrics\n",
460 | "y_pred = model.predict(x_test)\n",
461 | "\n",
462 | "# Calculate and print relevant scores\n",
463 | "accuracy = metrics.accuracy_score(y_test, y_pred)\n",
464 | "precision = metrics.precision_score(y_test, y_pred)\n",
465 | "recall = metrics.recall_score(y_test, y_pred)\n",
466 | "\n",
467 | "print(\"Accuracy:\", accuracy)\n",
468 | "print(\"Precision:\", precision)\n",
469 | "print(\"Recall:\", recall)"
470 | ]
471 | },
472 | {
473 | "cell_type": "markdown",
474 | "metadata": {
475 | "id": "OCF8uyvr5DKS"
476 | },
477 | "source": [
478 | "# Recursive Feature Elimination\n",
479 | "\n",
480 | "Given an external estimator that assigns weights to features (e.g., the coefficients/weights of a linear model), the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through any specific attribute (such as `coef_`, `feature_importances_`). Then, the least important features are pruned from current set of features. That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.\n",
481 | "\n"
482 | ]
483 | },
484 | {
485 | "cell_type": "code",
486 | "execution_count": 26,
487 | "metadata": {
488 | "colab": {
489 | "base_uri": "https://localhost:8080/"
490 | },
491 | "id": "l6AXNXt0805y",
492 | "outputId": "f5ae1bdf-858c-4542-f403-e88bdef48659"
493 | },
494 | "outputs": [
495 | {
496 | "data": {
497 | "text/plain": [
498 | "(569, 12)"
499 | ]
500 | },
501 | "execution_count": 26,
502 | "metadata": {},
503 | "output_type": "execute_result"
504 | }
505 | ],
506 | "source": [
507 | "# We use a linear kernel because RFE requires a model that has a coef_ or feature_importances_ attributes\n",
508 | "svc = svm.SVC(kernel=\"linear\")\n",
509 | "\n",
510 | "# Define the feature selector by selecing the estimator and the number of features to select, and the number of features to remove in each iteration\n",
511 | "rfe = feature_selection.RFE(estimator=svc, n_features_to_select=12, step=1)\n",
512 | "\n",
513 | "# Use fit transform to train the feature selector and return the best 10 features\n",
514 | "x_new = rfe.fit_transform(x, y)\n",
515 | "\n",
516 | "# New X will have 12 features instead of 30\n",
517 | "x_new.shape"
518 | ]
519 | },
520 | {
521 | "cell_type": "markdown",
522 | "metadata": {
523 | "id": "emX3sGCwyc92"
524 | },
525 | "source": [
526 | "## Training & Evaluating"
527 | ]
528 | },
529 | {
530 | "cell_type": "code",
531 | "execution_count": 27,
532 | "metadata": {
533 | "id": "UoVEg8slyZTo"
534 | },
535 | "outputs": [],
536 | "source": [
537 | "use_pruned_features = True # @param {type:\"boolean\"}\n",
538 | "x_final = x_new if use_pruned_features else x\n",
539 | "\n",
540 | "x_train, x_test, y_train, y_test = model_selection.train_test_split(\n",
541 | " x_new, y, test_size=0.3, stratify=y, random_state=42\n",
542 | ")"
543 | ]
544 | },
545 | {
546 | "cell_type": "code",
547 | "execution_count": 28,
548 | "metadata": {
549 | "colab": {
550 | "base_uri": "https://localhost:8080/"
551 | },
552 | "id": "fhKL3ElPyuBu",
553 | "outputId": "860155e7-2d7a-4003-cc80-46b9003dc61c"
554 | },
555 | "outputs": [
556 | {
557 | "name": "stdout",
558 | "output_type": "stream",
559 | "text": [
560 | "Training model on 12 features\n"
561 | ]
562 | },
563 | {
564 | "name": "stderr",
565 | "output_type": "stream",
566 | "text": [
567 | "/usr/local/lib/python3.7/dist-packages/sklearn/svm/_base.py:289: ConvergenceWarning: Solver terminated early (max_iter=10). Consider pre-processing your data with StandardScaler or MinMaxScaler.\n",
568 | " ConvergenceWarning,\n"
569 | ]
570 | },
571 | {
572 | "data": {
573 | "text/plain": [
574 | "Pipeline(steps=[('scaler', StandardScaler()),\n",
575 | " ('model', SVC(max_iter=10, random_state=10))])"
576 | ]
577 | },
578 | "execution_count": 28,
579 | "metadata": {},
580 | "output_type": "execute_result"
581 | }
582 | ],
583 | "source": [
584 | "print(f\"Training model on {x_train.shape[1]} features\")\n",
585 | "\n",
586 | "# Define a Support Vector Machine classifier with default configuration\n",
587 | "model = pipeline.Pipeline(\n",
588 | " [\n",
589 | " (\"scaler\", preprocessing.StandardScaler()),\n",
590 | " (\"model\", svm.SVC(kernel=\"rbf\", random_state=10, max_iter=10)),\n",
591 | " ]\n",
592 | ")\n",
593 | "\n",
594 | "# Train model using the training dataset\n",
595 | "model.fit(x_train, y_train)"
596 | ]
597 | },
598 | {
599 | "cell_type": "code",
600 | "execution_count": 29,
601 | "metadata": {
602 | "colab": {
603 | "base_uri": "https://localhost:8080/"
604 | },
605 | "id": "9dusZZULy_Jq",
606 | "outputId": "41ea1080-e6a6-4ea7-b12a-bf99cf65981e"
607 | },
608 | "outputs": [
609 | {
610 | "name": "stdout",
611 | "output_type": "stream",
612 | "text": [
613 | "Accuracy: 0.935672514619883\n",
614 | "Precision: 0.9363636363636364\n",
615 | "Recall: 0.9626168224299065\n"
616 | ]
617 | }
618 | ],
619 | "source": [
620 | "# Use the model to predict the testing set to prepare for calculating the metrics\n",
621 | "y_pred = model.predict(x_test)\n",
622 | "\n",
623 | "# Calculate and print relevant scores\n",
624 | "accuracy = metrics.accuracy_score(y_test, y_pred)\n",
625 | "precision = metrics.precision_score(y_test, y_pred)\n",
626 | "recall = metrics.recall_score(y_test, y_pred)\n",
627 | "\n",
628 | "print(\"Accuracy:\", accuracy)\n",
629 | "print(\"Precision:\", precision)\n",
630 | "print(\"Recall:\", recall)"
631 | ]
632 | },
633 | {
634 | "cell_type": "markdown",
635 | "metadata": {
636 | "id": "iav2Q9et3SV6"
637 | },
638 | "source": [
639 | "# Recursive Feature Elimination CV"
640 | ]
641 | },
642 | {
643 | "cell_type": "markdown",
644 | "metadata": {
645 | "id": "l4WwtQqk3oDs"
646 | },
647 | "source": [
648 | "Recursive feature elimination with cross-validation to select the number of features.\n",
649 | "\n"
650 | ]
651 | },
652 | {
653 | "cell_type": "code",
654 | "execution_count": 35,
655 | "metadata": {
656 | "colab": {
657 | "base_uri": "https://localhost:8080/"
658 | },
659 | "id": "A-nAsnPC3UWe",
660 | "outputId": "667840db-f73b-475a-e585-7f8aa0d9159a"
661 | },
662 | "outputs": [
663 | {
664 | "data": {
665 | "text/plain": [
666 | "(569, 9)"
667 | ]
668 | },
669 | "execution_count": 35,
670 | "metadata": {},
671 | "output_type": "execute_result"
672 | }
673 | ],
674 | "source": [
675 | "# We use a linear kernel because RFECV requires a model that has a coef_ or feature_importances_ attributes\n",
676 | "svc = svm.SVC(kernel=\"linear\")\n",
677 | "\n",
678 | "# Define the feature selector by selecing the estimator and the number of features to select, and the score for the features to be selected\n",
679 | "feature_selector = feature_selection.RFECV(\n",
680 | " lsvc, scoring=metrics.make_scorer(metrics.precision_score)\n",
681 | ")\n",
682 | "# Use fit transform to train the feature selector\n",
683 | "feature_selector.fit(x, y)\n",
684 | "\n",
685 | "new_x = feature_selector.transform(x)\n",
686 | "\n",
687 | "# New X will have 9 features instead of 30\n",
688 | "new_x.shape"
689 | ]
690 | },
691 | {
692 | "cell_type": "code",
693 | "execution_count": 36,
694 | "metadata": {
695 | "id": "uBFe87Ur3XVK"
696 | },
697 | "outputs": [],
698 | "source": [
699 | "x_train, x_test, y_train, y_test = model_selection.train_test_split(\n",
700 | " new_x, y, test_size=0.2, random_state=42, stratify=y\n",
701 | ")"
702 | ]
703 | },
704 | {
705 | "cell_type": "markdown",
706 | "metadata": {
707 | "id": "j18PyPYG4LiY"
708 | },
709 | "source": [
710 | "## Training & Evaluating"
711 | ]
712 | },
713 | {
714 | "cell_type": "code",
715 | "execution_count": 39,
716 | "metadata": {
717 | "colab": {
718 | "base_uri": "https://localhost:8080/"
719 | },
720 | "id": "AqianVCP3Y0s",
721 | "outputId": "b2319989-1fc6-431d-8eb0-204cf9b8cc2b"
722 | },
723 | "outputs": [
724 | {
725 | "name": "stderr",
726 | "output_type": "stream",
727 | "text": [
728 | "/usr/local/lib/python3.7/dist-packages/sklearn/svm/_base.py:289: ConvergenceWarning: Solver terminated early (max_iter=10). Consider pre-processing your data with StandardScaler or MinMaxScaler.\n",
729 | " ConvergenceWarning,\n"
730 | ]
731 | },
732 | {
733 | "data": {
734 | "text/plain": [
735 | "Pipeline(steps=[('scaler', StandardScaler()),\n",
736 | " ('model', SVC(max_iter=10, random_state=10))])"
737 | ]
738 | },
739 | "execution_count": 39,
740 | "metadata": {},
741 | "output_type": "execute_result"
742 | }
743 | ],
744 | "source": [
745 | "# Define a Support Vector Machine classifier with default configuration\n",
746 | "model = pipeline.Pipeline(\n",
747 | " [\n",
748 | " (\"scaler\", preprocessing.StandardScaler()),\n",
749 | " (\"model\", svm.SVC(kernel=\"rbf\", random_state=10, max_iter=10)),\n",
750 | " ]\n",
751 | ")\n",
752 | "# Train model using the training dataset\n",
753 | "model.fit(x_train, y_train)"
754 | ]
755 | },
756 | {
757 | "cell_type": "code",
758 | "execution_count": 40,
759 | "metadata": {
760 | "colab": {
761 | "base_uri": "https://localhost:8080/"
762 | },
763 | "id": "qEF83uCX3dOQ",
764 | "outputId": "0b999914-1ad7-4cf5-811f-54deace61ebd"
765 | },
766 | "outputs": [
767 | {
768 | "name": "stdout",
769 | "output_type": "stream",
770 | "text": [
771 | "Accuracy: 0.9210526315789473\n",
772 | "Precision: 0.9565217391304348\n",
773 | "Recall: 0.9166666666666666\n"
774 | ]
775 | }
776 | ],
777 | "source": [
778 | "# Use the model to predict the testing set to prepare for calculating the metrics\n",
779 | "y_pred = model.predict(x_test)\n",
780 | "\n",
781 | "# Calculate and print relevant scores\n",
782 | "accuracy = metrics.accuracy_score(y_test, y_pred)\n",
783 | "precision = metrics.precision_score(y_test, y_pred)\n",
784 | "recall = metrics.recall_score(y_test, y_pred)\n",
785 | "\n",
786 | "print(\"Accuracy:\", accuracy)\n",
787 | "print(\"Precision:\", precision)\n",
788 | "print(\"Recall:\", recall)"
789 | ]
790 | },
791 | {
792 | "cell_type": "code",
793 | "execution_count": null,
794 | "metadata": {
795 | "id": "lA9m72tq3eol"
796 | },
797 | "outputs": [],
798 | "source": []
799 | }
800 | ],
801 | "metadata": {
802 | "colab": {
803 | "collapsed_sections": [
804 | "Hd4Lz8wkfcdj",
805 | "gkav1r_6QLHz"
806 | ],
807 | "name": "1 - Feature Selection.ipynb",
808 | "provenance": [],
809 | "toc_visible": true
810 | },
811 | "kernelspec": {
812 | "display_name": "Python 3.9.13 64-bit",
813 | "language": "python",
814 | "name": "python3"
815 | },
816 | "language_info": {
817 | "codemirror_mode": {
818 | "name": "ipython",
819 | "version": 3
820 | },
821 | "file_extension": ".py",
822 | "mimetype": "text/x-python",
823 | "name": "python",
824 | "nbconvert_exporter": "python",
825 | "pygments_lexer": "ipython3",
826 | "version": "3.9.13"
827 | },
828 | "vscode": {
829 | "interpreter": {
830 | "hash": "b667cebad148e7b094a58ee81f940c685de1dd70a003a9ccdca4a5792431bee5"
831 | }
832 | }
833 | },
834 | "nbformat": 4,
835 | "nbformat_minor": 0
836 | }
--------------------------------------------------------------------------------
/LICENSE:
--------------------------------------------------------------------------------
1 | GNU GENERAL PUBLIC LICENSE
2 | Version 3, 29 June 2007
3 |
4 | Copyright (C) 2007 Free Software Foundation, Inc. \n",
12 | "\n",
13 | "\n",
14 | "\n",
15 | "Splitting our datasetes into train/test sets allows us to test our model on unseen examples. However, it might be the case that we got a lucky (or unlucky) split that doesn't represent the model's actual performance. To solve this problem, we'll use a technique called cross-validation, where we use the entire dataset for training and for testing and evaluate the model accordingly.\n",
16 | "\n",
17 | "There are several ways of performing cross-validation, and there are several corresponding iterators defined in scikit-learn. Each defines a `split` method, which will generate arrays of indices from the data set, each array indicating the instances to go into the training or testing set. \n"
18 | ]
19 | },
20 | {
21 | "cell_type": "code",
22 | "execution_count": null,
23 | "metadata": {
24 | "id": "U-L8w5RMoR8c",
25 | "scrolled": true
26 | },
27 | "outputs": [],
28 | "source": [
29 | "import pandas as pd\n",
30 | "import numpy as np\n",
31 | "from sklearn import datasets, svm, metrics, model_selection"
32 | ]
33 | },
34 | {
35 | "cell_type": "code",
36 | "execution_count": null,
37 | "metadata": {
38 | "id": "3H2-Cv3uqIG8"
39 | },
40 | "outputs": [],
41 | "source": [
42 | "x, y = datasets.load_breast_cancer(return_X_y=True)"
43 | ]
44 | },
45 | {
46 | "cell_type": "code",
47 | "execution_count": null,
48 | "metadata": {
49 | "id": "jAHUC3xpqkJX"
50 | },
51 | "outputs": [],
52 | "source": [
53 | "# Define a function to split our dataset into train/test splits using indices\n",
54 | "\n",
55 | "\n",
56 | "def kfold_train_test_split(x, y, train_indices, test_indices):\n",
57 | " return x[train_indices], x[test_indices], y[train_indices], y[test_indices]"
58 | ]
59 | },
60 | {
61 | "cell_type": "markdown",
62 | "metadata": {
63 | "id": "TNI8exLPp61j"
64 | },
65 | "source": [
66 | "### `KFold`\n",
67 | "\n",
68 | "`KFold` is arguably the simplest. It partitions the data into $k$ folds. It does not attempt to keep the proportions of classes. \n"
69 | ]
70 | },
71 | {
72 | "cell_type": "code",
73 | "execution_count": null,
74 | "metadata": {
75 | "id": "QqNihPfzoR8f"
76 | },
77 | "outputs": [],
78 | "source": [
79 | "k_fold = model_selection.KFold(\n",
80 | " n_splits=10\n",
81 | ") # splits the data into 10 splits, using 9 for training and 1 for testing in each iteration\n",
82 | "\n",
83 | "# Empty array to store the scores\n",
84 | "scores = []\n",
85 | "\n",
86 | "for train_indices, test_indices in k_fold.split(x):\n",
87 | " # Split data using our predefined function\n",
88 | " x_train, x_test, y_train, y_test = kfold_train_test_split(\n",
89 | " x, y, train_indices, test_indices\n",
90 | " )\n",
91 | "\n",
92 | " # Train model\n",
93 | " svc = svm.SVC()\n",
94 | " svc.fit(x_train, y_train)\n",
95 | "\n",
96 | " # Predict using test set\n",
97 | " y_pred = svc.predict(x_test)\n",
98 | "\n",
99 | " # Calculate scores\n",
100 | " accuracy = metrics.accuracy_score(y_test, y_pred)\n",
101 | " precision = metrics.precision_score(y_test, y_pred)\n",
102 | " recall = metrics.recall_score(y_test, y_pred)\n",
103 | "\n",
104 | " # Create scores dictionary\n",
105 | " scores_dict = {\"accuracy\": accuracy, \"precision\": precision, \"recall\": recall}\n",
106 | "\n",
107 | " # Append to scores array\n",
108 | " scores.append(scores_dict)"
109 | ]
110 | },
111 | {
112 | "cell_type": "code",
113 | "execution_count": null,
114 | "metadata": {
115 | "colab": {
116 | "base_uri": "https://localhost:8080/",
117 | "height": 357
118 | },
119 | "executionInfo": {
120 | "elapsed": 793,
121 | "status": "ok",
122 | "timestamp": 1631624726269,
123 | "user": {
124 | "displayName": "muntadher alkaabi",
125 | "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gj1adaCWpR-cnwQQK4viQZvo0gyRf1A0y9AkvnPMt0=s64",
126 | "userId": "00221067971932357642"
127 | },
128 | "user_tz": -180
129 | },
130 | "id": "ILHzsRYEr-N3",
131 | "outputId": "fa0cb867-e76c-48d2-cba1-0336886ab8d0"
132 | },
133 | "outputs": [
134 | {
135 | "data": {
136 | "text/html": [
137 | "| \n", 155 | " | accuracy | \n", 156 | "precision | \n", 157 | "recall | \n", 158 | "
|---|---|---|---|
| 0 | \n", 163 | "0.701754 | \n", 164 | "0.392857 | \n", 165 | "1.000000 | \n", 166 | "
| 1 | \n", 169 | "0.912281 | \n", 170 | "0.875000 | \n", 171 | "1.000000 | \n", 172 | "
| 2 | \n", 175 | "0.912281 | \n", 176 | "0.918919 | \n", 177 | "0.944444 | \n", 178 | "
| 3 | \n", 181 | "0.894737 | \n", 182 | "0.828571 | \n", 183 | "1.000000 | \n", 184 | "
| 4 | \n", 187 | "0.964912 | \n", 188 | "0.935484 | \n", 189 | "1.000000 | \n", 190 | "
| 5 | \n", 193 | "0.982456 | \n", 194 | "0.978261 | \n", 195 | "1.000000 | \n", 196 | "
| 6 | \n", 199 | "0.947368 | \n", 200 | "0.952381 | \n", 201 | "0.975610 | \n", 202 | "
| 7 | \n", 205 | "0.947368 | \n", 206 | "0.955556 | \n", 207 | "0.977273 | \n", 208 | "
| 8 | \n", 211 | "0.912281 | \n", 212 | "0.953488 | \n", 213 | "0.931818 | \n", 214 | "
| 9 | \n", 217 | "0.982143 | \n", 218 | "0.977273 | \n", 219 | "1.000000 | \n", 220 | "
| \n", 388 | " | accuracy | \n", 389 | "precision | \n", 390 | "recall | \n", 391 | "
|---|---|---|---|
| 0 | \n", 396 | "0.894737 | \n", 397 | "0.853659 | \n", 398 | "1.000000 | \n", 399 | "
| 1 | \n", 402 | "0.842105 | \n", 403 | "0.809524 | \n", 404 | "0.971429 | \n", 405 | "
| 2 | \n", 408 | "0.894737 | \n", 409 | "0.894737 | \n", 410 | "0.944444 | \n", 411 | "
| 3 | \n", 414 | "0.929825 | \n", 415 | "0.900000 | \n", 416 | "1.000000 | \n", 417 | "
| 4 | \n", 420 | "0.929825 | \n", 421 | "0.900000 | \n", 422 | "1.000000 | \n", 423 | "
| 5 | \n", 426 | "0.929825 | \n", 427 | "0.921053 | \n", 428 | "0.972222 | \n", 429 | "
| 6 | \n", 432 | "0.947368 | \n", 433 | "0.945946 | \n", 434 | "0.972222 | \n", 435 | "
| 7 | \n", 438 | "0.929825 | \n", 439 | "0.921053 | \n", 440 | "0.972222 | \n", 441 | "
| 8 | \n", 444 | "0.929825 | \n", 445 | "0.944444 | \n", 446 | "0.944444 | \n", 447 | "
| 9 | \n", 450 | "0.910714 | \n", 451 | "0.875000 | \n", 452 | "1.000000 | \n", 453 | "
| \n", 619 | " | accuracy | \n", 620 | "precision | \n", 621 | "recall | \n", 622 | "
|---|---|---|---|
| 0 | \n", 627 | "0.982456 | \n", 628 | "0.975610 | \n", 629 | "1.000000 | \n", 630 | "
| 1 | \n", 633 | "0.894737 | \n", 634 | "0.857143 | \n", 635 | "1.000000 | \n", 636 | "
| 2 | \n", 639 | "0.912281 | \n", 640 | "0.868421 | \n", 641 | "1.000000 | \n", 642 | "
| 3 | \n", 645 | "0.842105 | \n", 646 | "0.857143 | \n", 647 | "0.923077 | \n", 648 | "
| 4 | \n", 651 | "0.894737 | \n", 652 | "0.860465 | \n", 653 | "1.000000 | \n", 654 | "
| 5 | \n", 657 | "0.947368 | \n", 658 | "0.918919 | \n", 659 | "1.000000 | \n", 660 | "
| 6 | \n", 663 | "0.912281 | \n", 664 | "0.894737 | \n", 665 | "0.971429 | \n", 666 | "
| 7 | \n", 669 | "0.947368 | \n", 670 | "0.928571 | \n", 671 | "1.000000 | \n", 672 | "
| 8 | \n", 675 | "0.947368 | \n", 676 | "0.944444 | \n", 677 | "0.971429 | \n", 678 | "
| 9 | \n", 681 | "0.894737 | \n", 682 | "0.853659 | \n", 683 | "1.000000 | \n", 684 | "
| \n", 852 | " | accuracy | \n", 853 | "precision | \n", 854 | "recall | \n", 855 | "
|---|---|---|---|
| 0 | \n", 860 | "0.947368 | \n", 861 | "0.945946 | \n", 862 | "0.972222 | \n", 863 | "
| 1 | \n", 866 | "0.964912 | \n", 867 | "0.972222 | \n", 868 | "0.972222 | \n", 869 | "
| 2 | \n", 872 | "0.947368 | \n", 873 | "0.971429 | \n", 874 | "0.944444 | \n", 875 | "
| 3 | \n", 878 | "0.894737 | \n", 879 | "0.894737 | \n", 880 | "0.944444 | \n", 881 | "
| 4 | \n", 884 | "0.929825 | \n", 885 | "0.900000 | \n", 886 | "1.000000 | \n", 887 | "
| 5 | \n", 890 | "0.947368 | \n", 891 | "0.923077 | \n", 892 | "1.000000 | \n", 893 | "
| 6 | \n", 896 | "1.000000 | \n", 897 | "1.000000 | \n", 898 | "1.000000 | \n", 899 | "
| 7 | \n", 902 | "0.947368 | \n", 903 | "0.945946 | \n", 904 | "0.972222 | \n", 905 | "
| 8 | \n", 908 | "0.912281 | \n", 909 | "0.878049 | \n", 910 | "1.000000 | \n", 911 | "
| 9 | \n", 914 | "0.929825 | \n", 915 | "0.900000 | \n", 916 | "1.000000 | \n", 917 | "
\n",
12 | "\n",
13 | "\n",
14 | "[Scikit-learn](http://scikit-learn.org/stable/) is a python-based machine learning library providing implementations of a great many algorithms for supervised and unsupervised learning. In large part, it builds upon the cabilities of NumPy, SciPy, matplotlib, and Pandas.\n",
15 | "\n",
16 | "In the context of supervised learning, the primary objects scikit-learn defines are called **estimators**. Each of these defines a `fit` method, which develops a model from provided training data, and a `predict` method, which uses the model to map a new instance to a suitable target value. Scikit-learn also defines multiple utilities for partitioning and manipulating data sets as well as evaluating models.\n",
17 | "\n",
18 | "Below, we cover some of the basic steps needed to create a model in scikit-learn. These notes are based on material appearing in the *scikit-learn tutorials*.\n",
19 | "\n",
20 | "* [Tutorial](http://scikit-learn.org/stable/tutorial/index.html)\n",
21 | "* [Cheatsheet](https://s3.amazonaws.com/assets.datacamp.com/blog_assets/Scikit_Learn_Cheat_Sheet_Python.pdf)"
22 | ]
23 | },
24 | {
25 | "cell_type": "markdown",
26 | "metadata": {
27 | "id": "GvzKfYgXEqVC"
28 | },
29 | "source": [
30 | "## Datasets\n",
31 | "\n",
32 | "Scikit-learn comes bundled with several pre-defined (typically small) `datasets` that users can explore.\n",
33 | "\n",
34 | " load_boston()\tLoad and return the boston house-prices dataset (regression).\n",
35 | " load_iris()\tLoad and return the iris dataset (classification).\n",
36 | " load_diabetes()\tLoad and return the diabetes dataset (regression).\n",
37 | " load_digits()\tLoad and return the digits dataset (classification).\n",
38 | " load_linnerud()\tLoad and return the linnerud dataset (multivariate regression).\n",
39 | " load_wine()\tLoad and return the wine dataset (classification).\n",
40 | " load_breast_cancer()\tLoad and return the breast cancer wisconsin dataset (classification).\n",
41 | "\n",
42 | "The iris dataset is loaded below, and a description of it is printed."
43 | ]
44 | },
45 | {
46 | "cell_type": "code",
47 | "execution_count": 1,
48 | "metadata": {
49 | "id": "tPAFLH3AEqVD"
50 | },
51 | "outputs": [],
52 | "source": [
53 | "import numpy as np\n",
54 | "import pandas as pd\n",
55 | "\n",
56 | "# using 'from * import ...' allows as to import submodules directly\n",
57 | "from sklearn import (\n",
58 | " datasets,\n",
59 | " model_selection,\n",
60 | " linear_model,\n",
61 | " metrics,\n",
62 | " neighbors,\n",
63 | " tree,\n",
64 | " ensemble,\n",
65 | " preprocessing,\n",
66 | ")\n",
67 | "\n",
68 | "# alternatively, we can import the whole package as such\n",
69 | "import sklearn"
70 | ]
71 | },
72 | {
73 | "cell_type": "code",
74 | "execution_count": null,
75 | "metadata": {
76 | "colab": {
77 | "base_uri": "https://localhost:8080/"
78 | },
79 | "executionInfo": {
80 | "elapsed": 850,
81 | "status": "ok",
82 | "timestamp": 1655312921916,
83 | "user": {
84 | "displayName": "Muntadher Al-kaabi",
85 | "userId": "00221067971932357642"
86 | },
87 | "user_tz": -180
88 | },
89 | "id": "FARFibuOuHaw",
90 | "outputId": "bb6e52cd-d142-4fa0-d857-7efb43aa353a"
91 | },
92 | "outputs": [
93 | {
94 | "name": "stdout",
95 | "output_type": "stream",
96 | "text": [
97 | ".. _iris_dataset:\n",
98 | "\n",
99 | "Iris plants dataset\n",
100 | "--------------------\n",
101 | "\n",
102 | "**Data Set Characteristics:**\n",
103 | "\n",
104 | " :Number of Instances: 150 (50 in each of three classes)\n",
105 | " :Number of Attributes: 4 numeric, predictive attributes and the class\n",
106 | " :Attribute Information:\n",
107 | " - sepal length in cm\n",
108 | " - sepal width in cm\n",
109 | " - petal length in cm\n",
110 | " - petal width in cm\n",
111 | " - class:\n",
112 | " - Iris-Setosa\n",
113 | " - Iris-Versicolour\n",
114 | " - Iris-Virginica\n",
115 | " \n",
116 | " :Summary Statistics:\n",
117 | "\n",
118 | " ============== ==== ==== ======= ===== ====================\n",
119 | " Min Max Mean SD Class Correlation\n",
120 | " ============== ==== ==== ======= ===== ====================\n",
121 | " sepal length: 4.3 7.9 5.84 0.83 0.7826\n",
122 | " sepal width: 2.0 4.4 3.05 0.43 -0.4194\n",
123 | " petal length: 1.0 6.9 3.76 1.76 0.9490 (high!)\n",
124 | " petal width: 0.1 2.5 1.20 0.76 0.9565 (high!)\n",
125 | " ============== ==== ==== ======= ===== ====================\n",
126 | "\n",
127 | " :Missing Attribute Values: None\n",
128 | " :Class Distribution: 33.3% for each of 3 classes.\n",
129 | " :Creator: R.A. Fisher\n",
130 | " :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)\n",
131 | " :Date: July, 1988\n",
132 | "\n",
133 | "The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken\n",
134 | "from Fisher's paper. Note that it's the same as in R, but not as in the UCI\n",
135 | "Machine Learning Repository, which has two wrong data points.\n",
136 | "\n",
137 | "This is perhaps the best known database to be found in the\n",
138 | "pattern recognition literature. Fisher's paper is a classic in the field and\n",
139 | "is referenced frequently to this day. (See Duda & Hart, for example.) The\n",
140 | "data set contains 3 classes of 50 instances each, where each class refers to a\n",
141 | "type of iris plant. One class is linearly separable from the other 2; the\n",
142 | "latter are NOT linearly separable from each other.\n",
143 | "\n",
144 | ".. topic:: References\n",
145 | "\n",
146 | " - Fisher, R.A. \"The use of multiple measurements in taxonomic problems\"\n",
147 | " Annual Eugenics, 7, Part II, 179-188 (1936); also in \"Contributions to\n",
148 | " Mathematical Statistics\" (John Wiley, NY, 1950).\n",
149 | " - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.\n",
150 | " (Q327.D83) John Wiley & Sons. ISBN 0-471-22361-1. See page 218.\n",
151 | " - Dasarathy, B.V. (1980) \"Nosing Around the Neighborhood: A New System\n",
152 | " Structure and Classification Rule for Recognition in Partially Exposed\n",
153 | " Environments\". IEEE Transactions on Pattern Analysis and Machine\n",
154 | " Intelligence, Vol. PAMI-2, No. 1, 67-71.\n",
155 | " - Gates, G.W. (1972) \"The Reduced Nearest Neighbor Rule\". IEEE Transactions\n",
156 | " on Information Theory, May 1972, 431-433.\n",
157 | " - See also: 1988 MLC Proceedings, 54-64. Cheeseman et al\"s AUTOCLASS II\n",
158 | " conceptual clustering system finds 3 classes in the data.\n",
159 | " - Many, many more ...\n"
160 | ]
161 | }
162 | ],
163 | "source": [
164 | "iris_dataset = (\n",
165 | " datasets.load_iris()\n",
166 | ") # sklearn.datasets.load_iris() works exactly the same\n",
167 | "\n",
168 | "print(iris_dataset.DESCR)"
169 | ]
170 | },
171 | {
172 | "cell_type": "markdown",
173 | "metadata": {
174 | "id": "idq1o-aou6WR"
175 | },
176 | "source": [
177 | "We can also use `iris_dataset.data` and `iris_dataset.targets` to create or x & y (inputs & outputs) pairs that will be used for training and testing"
178 | ]
179 | },
180 | {
181 | "cell_type": "code",
182 | "execution_count": null,
183 | "metadata": {
184 | "colab": {
185 | "base_uri": "https://localhost:8080/",
186 | "height": 424
187 | },
188 | "executionInfo": {
189 | "elapsed": 32,
190 | "status": "ok",
191 | "timestamp": 1655312925923,
192 | "user": {
193 | "displayName": "Muntadher Al-kaabi",
194 | "userId": "00221067971932357642"
195 | },
196 | "user_tz": -180
197 | },
198 | "id": "JbXkcYblu6DT",
199 | "outputId": "0ead5436-2f61-4296-84f7-c5325c204a2c"
200 | },
201 | "outputs": [
202 | {
203 | "data": {
204 | "text/html": [
205 | "\n",
206 | " | \n", 226 | " | sepal length (cm) | \n", 227 | "sepal width (cm) | \n", 228 | "petal length (cm) | \n", 229 | "petal width (cm) | \n", 230 | "
|---|---|---|---|---|
| 0 | \n", 235 | "5.1 | \n", 236 | "3.5 | \n", 237 | "1.4 | \n", 238 | "0.2 | \n", 239 | "
| 1 | \n", 242 | "4.9 | \n", 243 | "3.0 | \n", 244 | "1.4 | \n", 245 | "0.2 | \n", 246 | "
| 2 | \n", 249 | "4.7 | \n", 250 | "3.2 | \n", 251 | "1.3 | \n", 252 | "0.2 | \n", 253 | "
| 3 | \n", 256 | "4.6 | \n", 257 | "3.1 | \n", 258 | "1.5 | \n", 259 | "0.2 | \n", 260 | "
| 4 | \n", 263 | "5.0 | \n", 264 | "3.6 | \n", 265 | "1.4 | \n", 266 | "0.2 | \n", 267 | "
| ... | \n", 270 | "... | \n", 271 | "... | \n", 272 | "... | \n", 273 | "... | \n", 274 | "
| 145 | \n", 277 | "6.7 | \n", 278 | "3.0 | \n", 279 | "5.2 | \n", 280 | "2.3 | \n", 281 | "
| 146 | \n", 284 | "6.3 | \n", 285 | "2.5 | \n", 286 | "5.0 | \n", 287 | "1.9 | \n", 288 | "
| 147 | \n", 291 | "6.5 | \n", 292 | "3.0 | \n", 293 | "5.2 | \n", 294 | "2.0 | \n", 295 | "
| 148 | \n", 298 | "6.2 | \n", 299 | "3.4 | \n", 300 | "5.4 | \n", 301 | "2.3 | \n", 302 | "
| 149 | \n", 305 | "5.9 | \n", 306 | "3.0 | \n", 307 | "5.1 | \n", 308 | "1.8 | \n", 309 | "
150 rows × 4 columns
\n", 313 | "LinearRegression()In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
LinearRegression()