`
95 | cd Auto_ViML
96 | pip install -r requirements.txt
97 | ```
98 |
99 | ## Usage
100 |
101 | In the same directory, open a Jupyter Notebook and use this line to import the .py file:
102 |
103 | ```
104 | from autoviml.Auto_ViML import Auto_ViML
105 | ```
106 |
107 | Load a data set (any CSV or text file) into a Pandas dataframe and split it into Train and Test dataframes. If you don't have a test dataframe, you can simple assign the test variable below to '' (empty string):
108 |
109 | ```
110 | model, features, trainm, testm = Auto_ViML(
111 | train,
112 | target,
113 | test,
114 | sample_submission,
115 | hyper_param="GS",
116 | feature_reduction=True,
117 | scoring_parameter="weighted-f1",
118 | KMeans_Featurizer=False,
119 | Boosting_Flag=False,
120 | Binning_Flag=False,
121 | Add_Poly=False,
122 | Stacking_Flag=False,
123 | Imbalanced_Flag=False,
124 | verbose=0,
125 | )
126 | ```
127 |
128 | Finally, it writes your submission file to disk in the current directory called `mysubmission.csv`.
129 | This submission file is ready for you to show it clients or submit it to competitions.
130 | If no submission file was given, but as long as you give it a test file name, it will create a submission file for you named `mySubmission.csv`.
131 | Auto_ViML works on any Multi-Class, Multi-Label Data Set. So you can have many target labels.
132 | You don't have to tell Auto_ViML whether it is a Regression or Classification problem.
133 |
134 | ## Tips for using Auto_ViML:
135 | 1. For Classification problems and imbalanced classes, choose scoring_parameter="balanced_accuracy". It works better.
136 | 2. For Imbalanced Classes (<5% samples in rare class), choose "Imbalanced_Flag"=True. You can also set this flag to True for Regression problems where the target variable might have skewed distributions.
137 | 3. For Multi-Label dataset, the target input target variable can be sent in as a list of variables.
138 | 4. It is recommended that you first set Boosting_Flag=None to get a Linear model. Once you understand that, then you can try to set Boosting_Flag=False to get a Random Forest model. Finally, try Boosting_Flag=True to get an XGBoost model. This is the order that we recommend in order to use Auto_ViML.
139 | 5. Finally try Boosting_Flag="CatBoost" to get a complex but high performing model.
140 | 6. Binning_Flag=True improves a CatBoost model since it adds to the list of categorical vars in data
141 | 7. KMeans_featurizer=True works well in NLP and CatBoost models since it creates cluster variables
142 | 8. Add_Poly=3 improves certain models where there is date-time or categorical and numeric variables
143 | 9. feature_reduction=True is the default and works best. But when you have <10 features in data, set it to False
144 | 10. Do not use Stacking_Flag=True with Linear models since your results may not look great.
145 | 11. Use Stacking_Flag=True only for complex models and as a last step with Boosting_Flag=True or CatBoost
146 | 12. Always set hyper_param ="RS" as input since it runs faster than GridSearchCV and gives better results!
147 | 13. KMeans_Featurizer=True does not work well for small data sets. Use it for data sets > 10,000 rows.
148 | 14. Finally Auto_ViML is meant to be a baseline or challenger solution to your data set. So use it for making quick models that you can compare against or in Hackathons. It is not meant for production!
149 |
150 | ## API
151 |
152 | **Arguments**
153 |
154 | - `train`: could be a datapath+filename or a dataframe. It will detect which is which and load it.
155 | - `test`: could be a datapath+filename or a dataframe. If you don't have any, just leave it as "".
156 | - `submission`: must be a datapath+filename. If you don't have any, just leave it as empty string.
157 | - `target`: name of the target variable in the data set.
158 | - `sep`: if you have a spearator in the file such as "," or "\t" mention it here. Default is ",".
159 | - `scoring_parameter`: if you want your own scoring parameter such as "f1" give it here. If not, it will assume the appropriate scoring param for the problem and it will build the model.
160 | - `hyper_param`: Tuning options are GridSearch ('GS') and RandomizedSearch ('RS'). Default is 'RS'.
161 | - `feature_reduction`: Default = 'True' but it can be set to False if you don't want automatic feature_reduction since in Image data sets like digits and MNIST, you get better results when you don't reduce features automatically. You can always try both and see.
162 | - `KMeans_Featurizer`
163 | - `True`: Adds a cluster label to features based on KMeans. Use for Linear.
164 | - `False (default)` For Random Forests or XGB models, leave it False since it may overfit.
165 | - `Boosting Flag`: you have 4 possible choices (default is False):
166 | - `None` This will build a Linear Model
167 | - `False` This will build a Random Forest or Extra Trees model (also known as Bagging)
168 | - `True` This will build an XGBoost model
169 | - `CatBoost` This will build a CatBoost model (provided you have CatBoost installed)
170 | - `Add_Poly`: Default is 0 which means do-nothing. But it has three interesting settings:
171 | - `1` Add interaction variables only such as x1*x2, x2*x3,...x9\*10 etc.
172 | - `2` Add Interactions and Squared variables such as x1**2, x2**2, etc.
173 | - `3` Adds both Interactions and Squared variables such as x1*x2, x1**2,x2*x3, x2**2, etc.
174 | - `Stacking_Flag`: Default is False. If set to True, it will add an additional feature which is derived from predictions of another model. This is used in some cases but may result in overfitting. So be careful turning this flag "on".
175 | - `Binning_Flag`: Default is False. It set to True, it will convert the top numeric variables into binned variables through a technique known as "Entropy" binning. This is very helpful for certain datasets (especially hard to build models).
176 | - `Imbalanced_Flag`: Default is False. If set to True, it will use SMOTE from Imbalanced-Learn to oversample the "Rare Class" in an imbalanced dataset and make the classes balanced (50-50 for example in a binary classification). This also works for Regression problems where you have highly skewed distributions in the target variable. Auto_ViML creates additional samples using SMOTE for Highly Imbalanced data.
177 | - `verbose`: This has 3 possible states:
178 | - `0` limited output. Great for running this silently and getting fast results.
179 | - `1` more charts. Great for knowing how results were and making changes to flags in input.
180 | - `2` lots of charts and output. Great for reproducing what Auto_ViML does on your own.
181 |
182 | **Return values**
183 |
184 | - `model`: It will return your trained model
185 | - `features`: the fewest number of features in your model to make it perform well
186 | - `train_modified`: this is the modified train dataframe after removing and adding features
187 | - `test_modified`: this is the modified test dataframe with the same transformations as train
188 |
189 | ## Maintainers
190 |
191 | * [@AutoViML](https://github.com/AutoViML)
192 | * [@morenoh149](https://github.com/morenoh149)
193 | * [@hironroy](https://github.com/hironroy)
194 |
195 | ## Contributing
196 |
197 | See [the contributing file](CONTRIBUTING.md)!
198 |
199 | PRs accepted.
200 |
201 | ## License
202 |
203 | Apache License 2.0 © 2020 Ram Seshadri
204 |
205 | ## DISCLAIMER
206 | This project is not an official Google project. It is not supported by Google and Google specifically disclaims all warranties as to its quality, merchantability, or fitness for a particular purpose.
207 |
208 | Platform: UNKNOWN
209 | Classifier: Programming Language :: Python :: 3
210 | Classifier: Operating System :: OS Independent
211 | Description-Content-Type: text/markdown
212 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Auto-ViML
2 |
3 | 
4 | Automatically Build Various Interpretable ML models fast!
5 | [](https://pepy.tech/project/autoviml/week)
6 | [](https://pepy.tech/project/autoviml/month)
7 | [](https://pepy.tech/project/autoviml)
8 | [](https://github.com/RichardLitt/standard-readme)
9 | [](https://pypi.org/project/autoviml)
10 | [](https://pypi.org/project/autoviml)
11 | [](https://github.com/AutoViML/Auto_ViML/blob/master/LICENSE)
12 |
13 | Auto_ViML is pronounced as "auto vimal" (autovimal logo created by Sanket Ghanmare).
14 |
15 | ## Update (Jan 2025)
16 |
17 | - Auto_ViML is now upgraded to version 0.2 which means it now runs on Python 3.12 or greater and also pandas 2.0 - this is a huge upgrade to those working in Colabs, Kaggle and other latest kernels. Please make sure you check the `requirements.txt` file to know which versions are recommended.
18 |
19 |
20 | ## Update (March 2023)
21 |
22 | - Auto_ViML has a new flag to speed up processing using GPU's. Just set the `GPU_flag`=`True` on Colab and other environments. But don't forget to set the runtime type to be "GPU" while running on Colab. Otherwise you will get an error.
23 |
24 |
25 | ## Update (May 2022)
26 |
27 | - Auto_ViML as of version 0.1.710 uses a very high performance library called `imbalanced_ensemble` for Imbalanced dataset problems. It will produce a 5-10% boost in your balanced_accuracy based on my experience with many datasets I have tried.
28 |
29 |
30 | In addition, new features in this version are:
31 |
32 | - SULOV -> Uses the SULOV algorithm for removing highly correlated features automatically.
33 | - Auto_NLP -> AutoViML automatically detects Text variables and does NLP processing using Auto_NLP
34 | - Date Time -> It automatically detects date time variables and generates new features
35 | - `imbalanced_ensemble` library -> Uses imbalanced_ensemble library for imbalanced data. Just set Imbalanced_Flag = True in arguments
36 | - Feature Selection -> We use the same algorithm that featurewiz library uses: SULOV and Recursive XGBoost to select best features fast. See below.
37 |
38 |
39 | ## Table of Contents
40 |
50 |
51 | ## Background
52 | Read this Medium article to learn how to use Auto_ViML effectively.
53 |
54 |
Auto_ViML was designed for building High Performance Interpretable Models with the fewest variables needed.
55 | The "V" in Auto_ViML stands for Variant because it tries multiple models with multiple features to find you the best performing model for your dataset. The "i" in Auto_ViML stands for "interpretable" since Auto_ViML selects the least number of features necessary to build a simpler, more interpretable model. In most cases, Auto_ViML builds models with 20%-99% fewer features than a similar performing model with all included features (this is based on my trials. Your experience may vary).
56 |
57 | Auto_ViML is every Data Scientist's model accelerator tool that:
58 |
59 | - Helps you with data cleaning: you can send in your entire dataframe as is and Auto_ViML will suggest changes to help with missing values, formatting variables, adding variables, etc. It loves dirty data. The dirtier the better!
60 | - Performs Feature Selection: Auto_ViML selects variables automatically by default. This is very helpful when you have hundreds if not thousands of variables since it can readily identify which of those are important variables vs which are unnecessary. You can turn this off as well (see API).
61 |
62 | 
63 | - Removes highly correlated features automatically. If two variables are highly correlated in your dataset, which one should you remove and which one should you keep? The decision is not as easy as it looks. Auto_ViML uses the SULOV algorithm to remove highly correlated features. You can understand SULOV from this picture more intuitively.
64 |
65 | 
66 |
67 | - Generates performance results graphically. Just set verbose = 1 (or) 2 instead of 0 (silent). You will get higher quality of insights as you increase verbosity.
68 | - Handles text, date-time, structs (lists, dictionaries), numeric, boolean, factor and categorical variables all in one model using one straight process.
69 |
70 | Auto_ViML is built using scikit-learn, numpy, pandas and matplotlib. It should run on most Python 3 Anaconda installations. You won't have to import any special libraries other than "XGBoost", "Imbalanced-Learn", "CatBoost", and "featuretools" library. We use "SHAP" library for interpretability.
But if you don't have these libraries, Auto_ViML will install those for you automatically.
71 |
72 | ## Install
73 |
74 | **Prerequsites:**
75 |
76 | - [Anaconda](https://docs.anaconda.com/anaconda/install/)
77 |
78 | To clone Auto_ViML, it is better to create a new environment, and install the required dependencies:
79 |
80 | To install from PyPi:
81 |
82 | $ pip install autoviml --upgrade --ignore-installed
83 |
84 |
85 | ```
86 | pip install git+https://github.com/AutoViML/Auto_ViML.git
87 | ```
88 |
89 | To install from source:
90 |
91 | ```
92 | cd
93 | git clone git@github.com:AutoViML/Auto_ViML.git
94 | # or download and unzip https://github.com/AutoViML/Auto_ViML/archive/master.zip
95 | conda create -n python=3.7 anaconda
96 | conda activate # ON WINDOWS: `source activate `
97 | cd Auto_ViML
98 | pip install -r requirements.txt
99 | ```
100 |
101 | ## Usage
102 |
103 | In the same directory, open a Jupyter Notebook and use this line to import the .py file:
104 |
105 | ```
106 | from autoviml.Auto_ViML import Auto_ViML
107 | ```
108 |
109 | Load a data set (any CSV or text file) into a Pandas dataframe and split it into Train and Test dataframes. If you don't have a test dataframe, you can simple assign the test variable below to '' (empty string):
110 |
111 | ```
112 | model, features, trainm, testm = Auto_ViML(
113 | train,
114 | target,
115 | test,
116 | sample_submission,
117 | hyper_param="GS",
118 | feature_reduction=True,
119 | scoring_parameter="weighted-f1",
120 | KMeans_Featurizer=False,
121 | Boosting_Flag=False,
122 | Binning_Flag=False,
123 | Add_Poly=False,
124 | Stacking_Flag=False,
125 | Imbalanced_Flag=False,
126 | verbose=0,
127 | )
128 | ```
129 |
130 | Finally, it writes your submission file to disk in the current directory called `mysubmission.csv`.
131 | This submission file is ready for you to show it clients or submit it to competitions.
132 | If no submission file was given, but as long as you give it a test file name, it will create a submission file for you named `mySubmission.csv`.
133 | Auto_ViML works on any Multi-Class, Multi-Label Data Set. So you can have many target labels.
134 | You don't have to tell Auto_ViML whether it is a Regression or Classification problem.
135 |
136 | ## Tips for using Auto_ViML:
137 | 1. `scoring_parameter`: For Classification problems and imbalanced classes, choose scoring_parameter="balanced_accuracy". It works better.
138 | 2. `Imbalanced_Flag`: For Imbalanced Classes (<5% samples in rare class), choose "Imbalanced_Flag"=True. You can also set this flag to True for Regression problems where the target variable might have skewed distributions.
139 | 3. `target`: For Multi-Label dataset, the target input target variable can be sent in as a list of variables.
140 | 4. `Boosting_Flag`: It is recommended that you first set Boosting_Flag=None to get a Linear model. Once you understand that, then you can try to set Boosting_Flag=False to get a Random Forest model. Finally, try Boosting_Flag=True to get an XGBoost model. This is the order that we recommend in order to use Auto_ViML. Finally try Boosting_Flag="CatBoost" to get a complex but high performing model.
141 | 5. `Binning_Flag`: Binning_Flag=True improves a CatBoost model since it adds to the list of categorical vars in data
142 | 6. `KMeans_featurizer`: KMeans_featurizer=True works well in NLP and CatBoost models since it creates cluster variables
143 | 7. `Add_Poly`: Add_Poly=3 improves certain models where there is date-time or categorical and numeric variables
144 | 8. `feature_reduction`: feature_reduction=True is the default and works best. But when you have <10 features in data, set it to False
145 | 9. `Stacking_Flag`: Do not set Stacking_Flag=True with Linear models since your results may not look great.
146 | 10. `Stacking_Flag`: Use Stacking_Flag=True only for complex models and as a last step with Boosting_Flag=True or CatBoost
147 | 11. `hyper_param`: Leave hyper_param ="RS" as input since it runs faster than GridSearchCV and gives better results unless you have a small data set and can afford to spend time on hyper tuning.
148 | 12. `KMeans_Featurizer`: KMeans_Featurizer=True does not work well for small data sets. Use it for data sets > 10,000 rows.
149 | 13. `Final thoughts`: Finally Auto_ViML is meant to be a baseline or challenger solution to your data set. So use it for making quick models that you can compare against or in Hackathons. It is not meant for production!
150 |
151 | ## API
152 |
153 | **Arguments**
154 |
155 | - `train`: could be a datapath+filename or a dataframe. It will detect which is which and load it.
156 | - `test`: could be a datapath+filename or a dataframe. If you don't have any, just leave it as "".
157 | - `submission`: must be a datapath+filename. If you don't have any, just leave it as empty string.
158 | - `target`: name of the target variable in the data set.
159 | - `sep`: if you have a spearator in the file such as "," or "\t" mention it here. Default is ",".
160 | - `scoring_parameter`: if you want your own scoring parameter such as "f1" give it here. If not, it will assume the appropriate scoring param for the problem and it will build the model.
161 | - `hyper_param`: Tuning options are GridSearch ('GS') and RandomizedSearch ('RS'). Default is 'RS'.
162 | - `feature_reduction`: Default = 'True' but it can be set to False if you don't want automatic feature_reduction since in Image data sets like digits and MNIST, you get better results when you don't reduce features automatically. You can always try both and see.
163 | - `KMeans_Featurizer`
164 | - `True`: Adds a cluster label to features based on KMeans. Use for Linear.
165 | - `False (default)` For Random Forests or XGB models, leave it False since it may overfit.
166 | - `Boosting Flag`: you have 4 possible choices (default is False):
167 | - `None` This will build a Linear Model
168 | - `False` This will build a Random Forest or Extra Trees model (also known as Bagging)
169 | - `True` This will build an XGBoost model
170 | - `CatBoost` This will build a CatBoost model (provided you have CatBoost installed)
171 | - `Add_Poly`: Default is 0 which means do-nothing. But it has three interesting settings:
172 | - `1` Add interaction variables only such as x1*x2, x2*x3,...x9\*10 etc.
173 | - `2` Add Interactions and Squared variables such as x1**2, x2**2, etc.
174 | - `3` Adds both Interactions and Squared variables such as x1*x2, x1**2,x2*x3, x2**2, etc.
175 | - `Stacking_Flag`: Default is False. If set to True, it will add an additional feature which is derived from predictions of another model. This is used in some cases but may result in overfitting. So be careful turning this flag "on".
176 | - `Binning_Flag`: Default is False. It set to True, it will convert the top numeric variables into binned variables through a technique known as "Entropy" binning. This is very helpful for certain datasets (especially hard to build models).
177 | - `Imbalanced_Flag`: Default is False. Uses imbalanced_ensemble library for imbalanced data. Just set Imbalanced_Flag = True in arguments
178 | - `verbose`: This has 3 possible states:
179 | - `0` limited output. Great for running this silently and getting fast results.
180 | - `1` more charts. Great for knowing how results were and making changes to flags in input.
181 | - `2` lots of charts and output. Great for reproducing what Auto_ViML does on your own.
182 |
183 | **Return values**
184 |
185 | - `model`: It will return your trained model
186 | - `features`: the fewest number of features in your model to make it perform well
187 | - `train_modified`: this is the modified train dataframe after removing and adding features
188 | - `test_modified`: this is the modified test dataframe with the same transformations as train
189 |
190 | ## Maintainers
191 |
192 | * [@AutoViML](https://github.com/AutoViML)
193 | * [@morenoh149](https://github.com/morenoh149)
194 | * [@hironroy](https://github.com/hironroy)
195 |
196 | ## Contributing
197 |
198 | See [the contributing file](CONTRIBUTING.md)!
199 |
200 | PRs accepted.
201 |
202 | ## License
203 |
204 | Apache License 2.0 © 2020 Ram Seshadri
205 |
206 | ## DISCLAIMER
207 | This project is not an official Google project. It is not supported by Google and Google specifically disclaims all warranties as to its quality, merchantability, or fitness for a particular purpose.
208 |
--------------------------------------------------------------------------------
/SULOV.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/AutoViML/Auto_ViML/0761412417b6dac0683425c372fd7e4f8942d815/SULOV.jpg
--------------------------------------------------------------------------------
/autoviml/QuickML_Ensembling.py:
--------------------------------------------------------------------------------
1 | import matplotlib.pyplot as plt
2 | import numpy as np
3 | import pandas as pd
4 | import seaborn as sns
5 |
6 | sns.set(style="white", color_codes=True)
7 | import warnings
8 |
9 | warnings.filterwarnings("ignore")
10 | from sklearn.linear_model import SGDClassifier
11 | from sklearn.model_selection import StratifiedKFold, KFold
12 | from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
13 | from sklearn.ensemble import BaggingRegressor, RandomForestClassifier, RandomForestRegressor
14 | from sklearn.svm import LinearSVC, LinearSVR
15 | from sklearn.ensemble import AdaBoostClassifier, AdaBoostRegressor
16 | from sklearn.tree import DecisionTreeRegressor
17 | from sklearn.linear_model import LassoLarsCV
18 | from sklearn.linear_model import LogisticRegressionCV
19 | from sklearn.naive_bayes import GaussianNB, MultinomialNB
20 | from sklearn.tree import DecisionTreeClassifier
21 | from sklearn.multiclass import OneVsRestClassifier
22 | from sklearn.neighbors import KNeighborsRegressor
23 | import time
24 | import operator
25 |
26 |
27 | #########################################################
28 | def QuickML_Ensembling(X_train, y_train, X_test, y_test='', modeltype='Regression',
29 | Boosting_Flag=False,
30 | scoring='', verbose=0):
31 | """
32 | Quickly builds and runs multiple models for a clean data set(only numerics).
33 | """
34 | start_time = time.time()
35 | seed = 99
36 | FOLDS = 5
37 | model_dict = {}
38 | model_tuples = []
39 | if len(X_train) <= 100000 and X_train.shape[1] < 50:
40 | NUMS = 100
41 | else:
42 | try:
43 | X_train = X_train.sample(frac=0.30, random_state=99)
44 | y_train = y_train[X_train.index]
45 | except:
46 | pass
47 | NUMS = 200
48 | ###### This is where we start performing ensembling of multiple simpler models ###
49 | if modeltype == 'Regression':
50 | if scoring == '':
51 | scoring = 'neg_mean_squared_error'
52 | # scv = ShuffleSplit(n_splits=FOLDS,random_state=seed)
53 | scv = KFold(n_splits=FOLDS, shuffle=False)
54 | if Boosting_Flag is None:
55 | ## Create an ensemble model ####
56 | model5 = AdaBoostRegressor(estimator=DecisionTreeRegressor(
57 | random_state=seed, max_depth=1, min_samples_leaf=2
58 | ), n_estimators=NUMS, random_state=seed)
59 | model_tuples.append(('Adaboost', model5))
60 | elif not Boosting_Flag:
61 | model5 = LassoLarsCV(cv=scv)
62 | model_tuples.append(('LassoLarsCV', model5))
63 | else:
64 | model5 = LassoLarsCV(cv=scv)
65 | model_tuples.append(('LassoLarsCV', model5))
66 | if Boosting_Flag is None:
67 | model6 = BaggingRegressor(DecisionTreeRegressor(random_state=seed),
68 | n_estimators=NUMS, random_state=seed)
69 | model_tuples.append(('Bagging_Regressor', model6))
70 | elif not Boosting_Flag:
71 | model6 = LinearSVR()
72 | model_tuples.append(('Linear_SVR', model6))
73 | else:
74 | model6 = DecisionTreeRegressor(max_depth=5, min_samples_leaf=2)
75 | model_tuples.append(('Decision_Tree', model6))
76 | model7 = KNeighborsRegressor(n_neighbors=8)
77 | model_tuples.append(('KNN_Regressor', model7))
78 | if Boosting_Flag is None:
79 | #### If the Boosting_Flag is True, it means Boosting model is present.
80 | ### So choose a different kind of classifier here
81 | model8 = DecisionTreeRegressor(max_depth=5, min_samples_leaf=2)
82 | model_tuples.append(('Decision_Tree', model8))
83 | elif not Boosting_Flag:
84 | #### If the Boosting_Flag is True, it means Boosting model is present.
85 | ### So choose a different kind of classifier here
86 | model8 = AdaBoostRegressor(base_estimator=DecisionTreeRegressor(
87 | random_state=seed, max_depth=1, min_samples_leaf=2
88 | ), n_estimators=NUMS, random_state=seed)
89 | model_tuples.append(('Adaboost', model8))
90 | else:
91 | model8 = RandomForestRegressor(bootstrap=False,
92 | max_depth=10,
93 | max_features='auto',
94 | min_samples_leaf=2,
95 | n_estimators=200,
96 | random_state=99)
97 | model_tuples.append(('RF_Regressor', model8))
98 | else:
99 | if scoring == '':
100 | scoring = 'accuracy'
101 | num_classes = len(np.unique(y_test))
102 | scv = StratifiedKFold(n_splits=FOLDS, shuffle=True, random_state=seed)
103 | if Boosting_Flag is None:
104 | ## Create an ensemble model ####
105 | model5 = AdaBoostClassifier(base_estimator=DecisionTreeClassifier(
106 | random_state=seed, max_depth=1, min_samples_leaf=2
107 | ), n_estimators=NUMS, random_state=seed)
108 | model_tuples.append(('Adaboost', model5))
109 | elif not Boosting_Flag:
110 | model5 = LinearDiscriminantAnalysis()
111 | model_tuples.append(('Linear_Discriminant', model5))
112 | else:
113 | model5 = LogisticRegressionCV(Cs=[0.001, 0.01, 0.1, 1, 10, 100], tol=0.01,
114 | solver='liblinear', random_state=seed)
115 | model_tuples.append(('Logistic_Regression_CV', model5))
116 | if Boosting_Flag is None:
117 | model6 = DecisionTreeClassifier(max_depth=5, min_samples_leaf=2)
118 | model_tuples.append(('Decision_Tree', model6))
119 | elif not Boosting_Flag:
120 | model6 = LinearSVC()
121 | model_tuples.append(('Linear_SVC', model6))
122 | else:
123 | model6 = DecisionTreeClassifier(max_depth=5, min_samples_leaf=2)
124 | model_tuples.append(('Decision_Tree', model6))
125 | if modeltype == 'Binary_Classification':
126 | model7 = GaussianNB()
127 | else:
128 | model7 = MultinomialNB()
129 | model_tuples.append(('Naive_Bayes', model7))
130 | if Boosting_Flag is None:
131 | #### If the Boosting_Flag is True, it means Boosting model is present.
132 | ### So choose a different kind of classifier here
133 | model8 = RandomForestClassifier(bootstrap=False,
134 | max_depth=10,
135 | max_features='auto',
136 | min_samples_leaf=2,
137 | n_estimators=200,
138 | random_state=99)
139 | model_tuples.append(('Bagging_Classifier', model8))
140 | elif not Boosting_Flag:
141 | #### If the Boosting_Flag is True, it means Boosting model is present.
142 | ### So choose a different kind of classifier here
143 | sgd_best_model = SGDClassifier(alpha=1e-06,
144 | loss='log',
145 | max_iter=1000,
146 | penalty='l2',
147 | learning_rate='constant',
148 | eta0=.1,
149 | random_state=3,
150 | tol=None)
151 | model8 = OneVsRestClassifier(sgd_best_model)
152 | model_tuples.append(('One_vs_Rest_Classifier', model8))
153 | else:
154 | model8 = RandomForestClassifier(bootstrap=False,
155 | max_depth=10,
156 | max_features='auto',
157 | min_samples_leaf=2,
158 | n_estimators=200,
159 | random_state=99)
160 | model_tuples.append(('Bagging_Classifier', model8))
161 | model_dict = dict(model_tuples)
162 | models, results = run_ensemble_models(model_dict, X_train, y_train, X_test, y_test,
163 | scoring, modeltype)
164 | return models, results
165 |
166 |
167 | #########################################################
168 | from sklearn.metrics import balanced_accuracy_score, accuracy_score, precision_score, recall_score, f1_score
169 | from sklearn.metrics import mean_absolute_error, mean_squared_error
170 | import copy
171 | from sklearn.linear_model import LinearRegression
172 | from sklearn.linear_model import LogisticRegression
173 |
174 |
175 | def run_ensemble_models(model_dict, X_train, y_train, X_test, y_test, scoring, modeltype):
176 | start_time = time.time()
177 | model_name, bac_score_list, ac_score_list, p_score_list, r_score_list, f1_score_list = [], [], [], [], [], []
178 | iteration = 0
179 | estimators = []
180 | estim_tuples = []
181 | for key, val in model_dict.items():
182 | estimator_name = key
183 | model_name.append(key)
184 | if str(val).split("(")[0] == 'MultinomialNB':
185 | #### Multinomial models need only positive values!!
186 | val.fit(abs(X_train), y_train)
187 | y_pred = val.predict(abs(X_test))
188 | else:
189 | try:
190 | val.fit(X_train, y_train)
191 | y_pred = val.predict(X_test)
192 | except:
193 | if modeltype != 'Regression':
194 | key = 'Logistic_Regression'
195 | val = LogisticRegression()
196 | else:
197 | key = 'Linear_Regression'
198 | val = LinearRegression()
199 | val.fit(X_train, y_train)
200 | try:
201 | y_pred = val.predict(X_test)
202 | except:
203 | print('Error in ensemble models. Returning...')
204 | stacks = np.array([])
205 | return model_name, stacks
206 | if iteration == 0:
207 | stacks = copy.deepcopy(y_pred)
208 | iteration += 1
209 | else:
210 | stacks = np.c_[stacks, y_pred]
211 | if not isinstance(y_test, str):
212 | if modeltype == 'Regression':
213 | bac_score = np.sqrt(mean_squared_error(y_test, y_pred))
214 | estimators.append((estimator_name, val, bac_score))
215 | estim_tuples.append((estimator_name, bac_score))
216 | bac_score_list.append(bac_score)
217 | ac_score_list.append(mean_squared_error(y_test, y_pred))
218 | p_score_list.append(mean_absolute_error(y_test, y_pred))
219 | model_comparison_df = pd.DataFrame([model_name, bac_score_list, ac_score_list, p_score_list]).T
220 | model_comparison_df.columns = ['model_name', 'RMSE', 'MSE', 'MAE']
221 | model_comparison_df = model_comparison_df.sort_values(by='RMSE', ascending=True)
222 | else:
223 | bac_score = balanced_accuracy_score(y_test, y_pred)
224 | estimators.append((estimator_name, val, bac_score))
225 | estim_tuples.append((estimator_name, bac_score))
226 | bac_score_list.append(balanced_accuracy_score(y_test, y_pred))
227 | ac_score_list.append(accuracy_score(y_test, y_pred))
228 | p_score_list.append(precision_score(y_test, y_pred, average='macro'))
229 | r_score_list.append(recall_score(y_test, y_pred, average='macro'))
230 | f1_score_list.append(f1_score(y_test, y_pred, average='macro'))
231 | model_comparison_df = pd.DataFrame([model_name, bac_score_list, ac_score_list, p_score_list,
232 | r_score_list, f1_score_list]).T
233 | model_comparison_df.columns = ['model_name', 'bal_accuracy_score', 'accuracy_score',
234 | 'ave_precision_score', 'ave_recall_score', 'ave_f1_score']
235 | model_comparison_df = model_comparison_df.sort_values(by='bal_accuracy_score', ascending=False)
236 | if not isinstance(y_test, str):
237 | data_frame = model_comparison_df.set_index('model_name').astype(float)
238 | plt.figure(figsize=(10, 10))
239 | g = sns.heatmap(data_frame, annot=True, fmt='0.2f', cbar=False)
240 | g.set_xticklabels(g.get_xticklabels(), rotation=45, fontsize=12)
241 | g.set_yticklabels(g.get_yticklabels(), rotation=0, fontsize=12)
242 | print('Time taken = %0.0f seconds' % (time.time() - start_time))
243 | g.set_title('QuickML Ensembling Models Results', fontsize=18)
244 | f1_stats = dict(estim_tuples)
245 | try:
246 | if scoring in ['logloss', 'rmse', 'mae', 'mape', 'RMSE', 'neg_mean_squared_error']:
247 | best_model_name = min(f1_stats.items(), key=operator.itemgetter(1))[0]
248 | else:
249 | best_model_name = max(f1_stats.items(), key=operator.itemgetter(1))[0]
250 | print('Based on trying multiple models, Best type of algorithm for this data set is %s' % best_model_name)
251 | except:
252 | print('Could not detect best algorithm type from ensembling. Continuing...')
253 | return model_name, stacks
254 | #########################################################
255 |
--------------------------------------------------------------------------------
/autoviml/QuickML_Stacking.py:
--------------------------------------------------------------------------------
1 | import warnings
2 |
3 | import numpy as np
4 | import pandas as pd
5 |
6 | warnings.filterwarnings("ignore")
7 | from sklearn.model_selection import StratifiedKFold, KFold
8 | from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
9 | from sklearn.svm import LinearSVR
10 | from sklearn.tree import DecisionTreeRegressor
11 | from sklearn.linear_model import LassoCV
12 | from sklearn.model_selection import cross_val_predict
13 | from sklearn.naive_bayes import GaussianNB, MultinomialNB
14 | from sklearn.tree import DecisionTreeClassifier
15 | import time
16 | import copy
17 | from collections import Counter
18 | from collections import OrderedDict
19 |
20 |
21 | #############################################################################
22 | def find_rare_class(classes, verbose=0):
23 | ######### Print the % count of each class in a Target variable #####
24 | """
25 | Works on Multi Class too. Prints class percentages count of target variable.
26 | It returns the name of the Rare class (the one with the minimum class member count).
27 | This can also be helpful in using it as pos_label in Binary and Multi Class problems.
28 | """
29 | counts = OrderedDict(Counter(classes))
30 | total = sum(counts.values())
31 | if verbose >= 1:
32 | print(' Class -> Counts -> Percent')
33 | for cls in counts.keys():
34 | print("%6s: % 7d -> % 5.1f%%" % (cls, counts[cls], counts[cls] / total * 100))
35 | if type(pd.Series(counts).idxmin()) == str:
36 | return pd.Series(counts).idxmin()
37 | else:
38 | return int(pd.Series(counts).idxmin())
39 |
40 |
41 | ################################################################################
42 | def QuickML_Stacking(X_train, y_train, X_test='', modeltype='Regression', Boosting_Flag=False,
43 | scoring='', verbose=0):
44 | """
45 | Quickly build Stacks of multiple model results
46 | Input must be a clean data set (only numeric variables, no categorical or string variables).
47 | """
48 | X_train = copy.deepcopy(X_train)
49 | X_test = copy.deepcopy(X_test)
50 | y_train = copy.deepcopy(y_train)
51 | start_time = time.time()
52 | seed = 99
53 | if len(X_train) <= 100000 or X_train.shape[1] < 50:
54 | NUMS = 100
55 | FOLDS = 5
56 | else:
57 | NUMS = 200
58 | FOLDS = 10
59 | ## create Stacking models
60 | estimators = []
61 | ### This keeps tracks of the number of predict_proba columns generated by each model ####
62 | estimator_length = []
63 | if isinstance(X_test, str):
64 | no_fit = True
65 | else:
66 | no_fit = False
67 | if no_fit:
68 | #### This is where you don't fit the model but just do cross_val_predict ####
69 | if modeltype == 'Regression':
70 | if scoring == '':
71 | scoring = 'neg_mean_squared_error'
72 | scv = KFold(n_splits=FOLDS, shuffle=False)
73 | if Boosting_Flag:
74 | ###### Bagging models if Bagging is chosen ####
75 | # model4 = BaggingRegressor(DecisionTreeRegressor(random_state=seed),
76 | # n_estimators=NUMS,random_state=seed)
77 | model4 = LinearSVR()
78 | results = cross_val_predict(model4, X_train, y_train, cv=scv, n_jobs=-1)
79 | estimators.append(('Linear_SVR', model4))
80 | estimator_length.append(1)
81 | elif Boosting_Flag is None:
82 | #### Tree models if Linear chosen #####
83 | model5 = DecisionTreeRegressor(random_state=seed, min_samples_leaf=2)
84 | results = cross_val_predict(model5, X_train, y_train, cv=scv, n_jobs=-1)
85 | estimators.append(('Decision Trees', model5))
86 | estimator_length.append(1)
87 | else:
88 | #### Linear Models if Boosting is chosen #####
89 | model6 = LassoCV(alphas=np.logspace(-5, -1, 20), cv=scv, random_state=seed)
90 | results = cross_val_predict(model6, X_train, y_train, cv=scv, n_jobs=-1)
91 | estimators.append(('LassoCV Regularization', model6))
92 | estimator_length.append(1)
93 | else:
94 | n_classes = len(Counter(y_train))
95 | if n_classes > 2:
96 | #### In multi-class setting, it makes sense to turn it into binary class in stage-1
97 | #### In stage 2, a complex model will take the inputs of this model and try to predict
98 | rare_class = find_rare_class(y_train)
99 | if rare_class == 0:
100 | majority_class = 1
101 | else:
102 | majority_class = 0
103 | y_train = y_train.map(lambda x: rare_class if x == rare_class else majority_class)
104 | if scoring == '':
105 | scoring = 'accuracy'
106 | scv = StratifiedKFold(n_splits=FOLDS, random_state=seed, shuffle=True)
107 | if Boosting_Flag:
108 | #### Linear Models if Boosting is chosen #####
109 | model4 = LinearDiscriminantAnalysis()
110 | results = cross_val_predict(model4, X_train, y_train, cv=scv, n_jobs=-1,
111 | method='predict_proba')
112 | estimators.append(('Linear Discriminant', model4))
113 | estimator_length.append(results.shape[1])
114 | elif Boosting_Flag is None:
115 | #### Tree models if Linear chosen #####
116 | model6 = DecisionTreeClassifier(min_samples_leaf=2)
117 | results = cross_val_predict(model6, X_train, y_train, cv=scv, n_jobs=-1,
118 | method='predict_proba')
119 | estimators.append(('Decision Tree', model6))
120 | estimator_length.append(results.shape[1])
121 | else:
122 | ###### Naive Bayes models if Bagging is chosen ####
123 | if n_classes <= 2:
124 | try:
125 | model7 = GaussianNB()
126 | except:
127 | model7 = DecisionTreeClassifier(min_samples_leaf=2)
128 | else:
129 | try:
130 | model7 = MultinomialNB()
131 | except:
132 | model7 = DecisionTreeClassifier(min_samples_leaf=2)
133 | results = cross_val_predict(model7, X_train, y_train, cv=scv, n_jobs=-1,
134 | method='predict_proba')
135 | estimators.append(('Naive Bayes', model7))
136 | estimator_length.append(results.shape[1])
137 | else:
138 | #### This is where you fit the model and then predict ########
139 | if modeltype == 'Regression':
140 | if scoring == '':
141 | scoring = 'neg_mean_squared_error'
142 | scv = KFold(n_splits=FOLDS, shuffle=False)
143 | if Boosting_Flag:
144 | ###### Bagging models if Bagging is chosen ####
145 | # model4 = BaggingRegressor(DecisionTreeRegressor(random_state=seed),
146 | # n_estimators=NUMS,random_state=seed)
147 | model4 = LinearSVR()
148 | results = model4.fit(X_train, y_train).predict(X_test)
149 | estimators.append(('Linear_SVR', model4))
150 | estimator_length.append(1)
151 | elif Boosting_Flag is None:
152 | #### Tree models if Linear chosen #####
153 | model5 = DecisionTreeRegressor(random_state=seed, min_samples_leaf=2)
154 | results = model5.fit(X_train, y_train).predict(X_test)
155 | estimators.append(('Decision Trees', model5))
156 | estimator_length.append(1)
157 | else:
158 | #### Linear Models if Boosting is chosen #####
159 | model6 = LassoCV(alphas=np.logspace(-5, -1, 20), cv=scv, random_state=seed)
160 | results = model6.fit(X_train, y_train).predict(X_test)
161 | estimators.append(('LassoCV Regularization', model6))
162 | estimator_length.append(1)
163 | else:
164 | n_classes = len(Counter(y_train))
165 | if n_classes > 2:
166 | #### In multi-class setting, it makes sense to turn it into binary class in stage-1
167 | #### In stage 2, a complex model will take the inputs of this model and try to predict
168 | rare_class = find_rare_class(y_train)
169 | if rare_class == 0:
170 | majority_class = 1
171 | else:
172 | majority_class = 0
173 | y_train = y_train.map(lambda x: rare_class if x == rare_class else majority_class)
174 | if scoring == '':
175 | scoring = 'accuracy'
176 | scv = StratifiedKFold(n_splits=FOLDS, random_state=seed, shuffle=True)
177 | if Boosting_Flag:
178 | #### Linear Models if Boosting is chosen #####
179 | model4 = LinearDiscriminantAnalysis()
180 | results = model4.fit(X_train, y_train).predict_proba(X_test)
181 | estimators.append(('Linear Discriminant', model4))
182 | estimator_length.append(results.shape[1])
183 | elif Boosting_Flag is None:
184 | #### Tree models if Linear chosen #####
185 | model6 = DecisionTreeClassifier(min_samples_leaf=2)
186 | results = model6.fit(X_train, y_train).predict_proba(X_test)
187 | estimators.append(('Decision Tree', model6))
188 | estimator_length.append(results.shape[1])
189 | else:
190 | ###### Naive Bayes models if Bagging is chosen ####
191 | if n_classes <= 2:
192 | try:
193 | model7 = GaussianNB()
194 | except:
195 | model7 = DecisionTreeClassifier(min_samples_leaf=2)
196 | else:
197 | try:
198 | model7 = MultinomialNB()
199 | except:
200 | model7 = DecisionTreeClassifier(min_samples_leaf=2)
201 | results = model7.fit(X_train, y_train).predict_proba(X_test)
202 | estimators.append(('Naive Bayes', model7))
203 | estimator_length.append(results.shape[1])
204 | # stacks = np.c_[results1,results2,results3]
205 | estimators_list = [(tuples[0], tuples[1]) for tuples in estimators]
206 | estimator_names = [tuples[0] for tuples in estimators]
207 | #### Here is where we consolidate the estimator names and their results into one common list ###
208 | ls = []
209 | for x, y in dict(zip(estimator_names, estimator_length)).items():
210 | els = [x + '_' + str(eachy) for eachy in range(y)]
211 | ls += els
212 | if verbose == 1:
213 | print(' Time taken for Stacking: %0.1f seconds' % (time.time() - start_time))
214 | return ls, results
215 | #########################################################
216 |
--------------------------------------------------------------------------------
/autoviml/Transform_KM_Features.py:
--------------------------------------------------------------------------------
1 | from collections import Counter
2 |
3 | import numpy as np
4 | from sklearn.cluster import KMeans
5 |
6 |
7 | class KMeansFeaturizer:
8 | """Transforms numeric data into k-means cluster memberships.
9 | From the Feature Engineering Book by Alice Zheng
10 | https://github.com/alicezheng/feature-engineering-book/blob/master/kmeans_featurizer.py
11 | This transformer runs k-means on the input data and converts each data point
12 | into the id of the closest cluster. If a target variable is present, it is
13 | scaled and included as input to k-means in order to derive clusters that
14 | obey the classification boundary as well as group similar points together.
15 | Parameters
16 | ----------
17 | k: integer, optional, default 100
18 | The number of clusters to group data into.
19 | target_scale: float, [0, infty], optional, default 5.0
20 | The scaling factor for the target variable. Set this to zero to ignore
21 | the target. For classification problems, larger `target_scale` values
22 | will produce clusters that better respect the class boundary.
23 | random_state : integer or numpy.RandomState, optional
24 | This is passed to k-means as the generator used to initialize the
25 | kmeans centers. If an integer is given, it fixes the seed. Defaults to
26 | the global numpy random number generator.
27 | Attributes
28 | ----------
29 | cluster_centers_ : array, [k, n_features]
30 | Coordinates of cluster centers. n_features does count the target column.
31 | """
32 |
33 | def __init__(self, k=100, target_scale=5.0, random_state=None):
34 | self.k = k
35 | self.target_scale = target_scale
36 | self.random_state = random_state
37 | self.km_model_ = None
38 | self.km_model = None
39 | self.cluster_centers_ = None
40 |
41 | def fit(self, X, y=None):
42 | """Runs k-means on the input data and find centroids.
43 | If no target is given (`y` is None) then run vanilla k-means on input
44 | `X`.
45 | If target `y` is given, then include the target (weighted by
46 | `target_scale`) as an extra dimension for k-means clustering. In this
47 | case, run k-means twice, first with the target, then an extra iteration
48 | without.
49 | After fitting, the attribute `cluster_centers_` are set to the k-means
50 | centroids in the input space represented by `X`.
51 | Parameters
52 | ----------
53 | X : array-like or sparse matrix, shape=(n_data_points, n_features)
54 | y : vector of length n_data_points, optional, default None
55 | If provided, will be weighted with `target_scale` and included in
56 | k-means clustering as hint.
57 | """
58 | n_features = X.shape[1]
59 | if y is None:
60 | # No target variable, just do plain k-means
61 | km_model = KMeans(n_clusters=self.k,
62 | n_init=20,
63 | random_state=self.random_state)
64 | km_model.fit(X)
65 |
66 | self.km_model_ = km_model
67 | self.cluster_centers_ = km_model.cluster_centers_
68 | return self
69 |
70 | # There is target information. Apply appropriate scaling and include
71 | # into input data to k-means
72 | data_with_target = np.hstack((X, y[:, np.newaxis] * self.target_scale))
73 |
74 | # Build a pre-training k-means model on data and target
75 | km_model_pretrain = KMeans(n_clusters=self.k,
76 | n_init=20,
77 | random_state=self.random_state)
78 | km_model_pretrain.fit(data_with_target)
79 |
80 | # Run k-means a second time to get the clusters in the original space
81 | # without target info. Initialize using centroids found in pre-training.
82 | # Go through a single iteration of cluster assignment and centroid
83 | # recomputation.
84 | km_model = KMeans(n_clusters=self.k,
85 | init=km_model_pretrain.cluster_centers_[:, :n_features],
86 | n_init=1,
87 | max_iter=1)
88 | km_model.fit(X)
89 |
90 | self.km_model = km_model
91 | self.cluster_centers_ = km_model.cluster_centers_
92 | return self
93 |
94 | def transform(self, X):
95 | """Output the closest cluster id for each input data point.
96 | Parameters
97 | ----------
98 | X : array-like or sparse matrix, shape=(n_data_points, n_features)
99 | Returns
100 | -------
101 | cluster_ids : array, shape[n_data_points,1]
102 | """
103 | clusters = self.km_model.predict(X)
104 | return clusters[:, np.newaxis]
105 |
106 | def fit_transform(self, X, y=None):
107 | """Runs fit followed by transform.
108 | """
109 | self.fit(X, y)
110 | return self.transform(X)
111 |
112 |
113 | import copy
114 |
115 |
116 | def Transform_KM_Features(training_data, training_labels, test_data, km_max=0):
117 | seed = 99
118 | preds = list(training_data)
119 | target = training_labels.name
120 | train_index = training_data.index
121 | test_index = test_data.index
122 | if km_max <= 2:
123 | k_max = 2
124 | else:
125 | k_max = copy.deepcopy(km_max)
126 | ### Calculate the target scale here => the higher the number the better for target accuracy
127 | try:
128 | if training_labels.dtype in [np.float64, np.float32, np.float16]:
129 | target_range = float(abs(training_labels.max() - training_labels.min()))
130 | elif training_labels.dtype in [object, bool]:
131 | target_range = int(len(Counter(training_labels)) + 3)
132 | else:
133 | target_range = int(abs(training_labels.max() - training_labels.min()))
134 | except:
135 | target_range = 5.0
136 | kmf = KMeansFeaturizer(k=k_max, target_scale=target_range, random_state=seed)
137 | kmf_hint = kmf.fit(training_data.values, training_labels.values)
138 | ### Just return it with the cluster column => no need to return the data frame ###
139 | training_cluster_features = kmf_hint.transform(training_data.values)
140 | test_cluster_features = kmf_hint.transform(test_data.values)
141 | npx = np.c_[training_data.values, training_labels.values]
142 | training_with_cluster = np.c_[npx, training_cluster_features]
143 | test_with_cluster = np.c_[test_data, test_cluster_features]
144 | ### We are going to just return the cluster values ######
145 | train_with_cluster_df = training_with_cluster[:, -1]
146 | test_with_cluster_df = test_with_cluster[:, -1]
147 | # train_with_cluster_df = pd.DataFrame(training_with_cluster,index=train_index,
148 | # columns=preds+[target,'cluster'])
149 | # test_with_cluster_df = pd.DataFrame(test_with_cluster,index=test_index,
150 | # columns=preds+['cluster'])
151 | return train_with_cluster_df, test_with_cluster_df
152 |
--------------------------------------------------------------------------------
/autoviml/__init__.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | ################################################################################
3 | # Auto_ViML - Automatically Build Multiple Interpretable ML Models in Single Line of Code
4 | # Python v3.6+
5 | # Created by Ram Seshadri
6 | # Licensed under Apache License v2
7 | ################################################################################
8 | # Version
9 | from .__version__ import __version__, __nlp_version__
10 | from autoviml.Auto_ViML import Auto_ViML
11 | from autoviml.Auto_NLP import Auto_NLP
12 | if __name__ == "__main__":
13 | module_type = 'Running'
14 | else:
15 | module_type = 'Imported'
16 | viml_version_number = __version__
17 | nlp_version_number = __nlp_version__
18 | print("""%s Auto_ViML version: %s. Call using:
19 | m, feats, trainm, testm = Auto_ViML(train, target, test,
20 | sample_submission='',
21 | scoring_parameter='', KMeans_Featurizer=False,
22 | hyper_param='RS',feature_reduction=True,
23 | Boosting_Flag='CatBoost', Binning_Flag=False,
24 | Add_Poly=0, Stacking_Flag=False,Imbalanced_Flag=False,
25 | GPU_flag=False, verbose=1)
26 | """ %(module_type, viml_version_number))
27 | print()
28 | ###########################################################################################
29 | print("""%s Auto_NLP version: %s.. Call using:
30 | train_nlp, test_nlp, nlp_pipeline, predictions = Auto_NLP(
31 | nlp_column, train, test, target, score_type='balanced_accuracy',
32 | modeltype='Classification',top_num_features=200, verbose=0,
33 | build_model=True)""" %(module_type, nlp_version_number))
34 | ########################################################################
35 |
--------------------------------------------------------------------------------
/autoviml/__version__.py:
--------------------------------------------------------------------------------
1 | # -*- coding: utf-8 -*-
2 | """Specifies the version of the Auto_ViML package."""
3 |
4 | __title__ = "Auto_ViML"
5 | __author__ = "Ram Seshadri"
6 | __description__ = "Automatically Build Multiple Interpretable ML Models in Single Line of Code"
7 | __url__ = "https://github.com/AutoViML/Auto_ViML.git"
8 | __version__ = "0.2.02"
9 | __nlp_version__ = "0.1.01"
10 | __license__ = "Apache License 2.0"
11 | __copyright__ = "2020-21 Google"
12 |
--------------------------------------------------------------------------------
/autoviml/classify_method.py:
--------------------------------------------------------------------------------
1 | import random
2 |
3 | import numpy as np
4 | import pandas as pd
5 |
6 | np.random.seed(99)
7 | random.seed(42)
8 | ################################################################################
9 | #### The warnings from Sklearn are so annoying that I have to shut it off #######
10 | import warnings
11 |
12 | warnings.filterwarnings("ignore")
13 | from sklearn.exceptions import DataConversionWarning
14 |
15 | warnings.filterwarnings(action='ignore', category=DataConversionWarning)
16 |
17 |
18 | def warn(*args, **kwargs):
19 | pass
20 |
21 |
22 | warnings.warn = warn
23 | ####################################################################################
24 | from functools import reduce
25 |
26 |
27 | def left_subtract(l1, l2):
28 | lst = []
29 | for i in l1:
30 | if i not in l2:
31 | lst.append(i)
32 | return lst
33 |
34 |
35 | #################################################################################
36 | import copy
37 |
38 |
39 | def EDA_find_remove_columns_with_infinity(df, remove=False):
40 | """
41 | This function finds all columns in a dataframe that have inifinite values (np.inf or -np.inf)
42 | It returns a list of column names. If the list is empty, it means no columns were found.
43 | If remove flag is set, then it returns a smaller dataframe with inf columns removed.
44 | """
45 | nums = df.select_dtypes(include='number').columns.tolist()
46 | dfx = df[nums]
47 | sum_rows = np.isinf(dfx).values.sum()
48 | add_cols = list(dfx.columns.to_series()[np.isinf(dfx).any()])
49 | if sum_rows > 0:
50 | print(' there are %d rows and %d columns with infinity in them...' % (sum_rows, len(add_cols)))
51 | if remove:
52 | ### here you need to use df since the whole dataset is involved ###
53 | nocols = [x for x in df.columns if x not in add_cols]
54 | print(" Shape of dataset before %s and after %s removing columns with infinity" %
55 | (df.shape, (df[nocols].shape,)))
56 | return df[nocols]
57 | else:
58 | ## this will be a list of columns with infinity ####
59 | return add_cols
60 | else:
61 | ## this will be an empty list if there are no columns with infinity
62 | return add_cols
63 |
64 |
65 | ####################################################################################
66 | def classify_columns(df_preds, verbose=0):
67 | """
68 | This actually does Exploratory data analysis - it means this function performs EDA
69 | ######################################################################################
70 | Takes a dataframe containing only predictors to be classified into various types.
71 | DO NOT SEND IN A TARGET COLUMN since it will try to include that into various columns.
72 | Returns a data frame containing columns and the class it belongs to such as numeric,
73 | categorical, date or id column, boolean, nlp, discrete_string and cols to delete...
74 | ####### Returns a dictionary with 10 kinds of vars like the following: # continuous_vars,int_vars
75 | # cat_vars,factor_vars, bool_vars,discrete_string_vars,nlp_vars,date_vars,id_vars,cols_delete
76 | """
77 | train = copy.deepcopy(df_preds)
78 | #### If there are 30 chars are more in a discrete_string_var, it is then considered an NLP variable
79 | max_nlp_char_size = 30
80 | max_cols_to_print = 30
81 | print('#######################################################################################')
82 | print('######################## C L A S S I F Y I N G V A R I A B L E S ####################')
83 | print('#######################################################################################')
84 | print('Classifying variables in data set...')
85 | #### Cat_Limit defines the max number of categories a column can have to be called a categorical colum
86 | cat_limit = 35
87 | float_limit = 15 # Make this limit low so that float variables below this limit become cat vars
88 |
89 | def add(a, b):
90 | return a + b
91 |
92 | sum_all_cols = dict()
93 | orig_cols_total = train.shape[1]
94 | # Types of columns
95 | cols_delete = []
96 | cols_delete = [col for col in list(train) if (len(train[col].value_counts()) == 1) | (
97 | train[col].isnull().sum() / len(train) >= 0.90)]
98 | inf_cols = EDA_find_remove_columns_with_infinity(train)
99 | mixed_cols = [x for x in list(train) if len(train[x].dropna().apply(type).value_counts()) > 1]
100 | if len(mixed_cols) > 0:
101 | print(' Removing %s column(s) due to mixed data type detected...' % mixed_cols)
102 | cols_delete += mixed_cols
103 | cols_delete += inf_cols
104 | train = train[left_subtract(list(train), cols_delete)]
105 | var_df = pd.Series(dict(train.dtypes)).reset_index(drop=False).rename(
106 | columns={0: 'type_of_column'})
107 | sum_all_cols['cols_delete'] = cols_delete
108 |
109 | var_df['bool'] = var_df.apply(
110 | lambda x: 1 if x['type_of_column'] in ['bool', 'object'] and len(train[x['index']].value_counts()) == 2 else 0,
111 | axis=1)
112 | string_bool_vars = list(var_df[(var_df['bool'] == 1)]['index'])
113 | sum_all_cols['string_bool_vars'] = string_bool_vars
114 | var_df['num_bool'] = var_df.apply(lambda x: 1 if x['type_of_column'] in [np.uint8,
115 | np.uint16, np.uint32, np.uint64,
116 | 'int8', 'int16', 'int32', 'int64',
117 | 'float16', 'float32', 'float64'] and len(
118 | train[x['index']].value_counts()) == 2 else 0, axis=1)
119 | num_bool_vars = list(var_df[(var_df['num_bool'] == 1)]['index'])
120 | sum_all_cols['num_bool_vars'] = num_bool_vars
121 | ###### This is where we take all Object vars and split them into diff kinds ###
122 | discrete_or_nlp = var_df.apply(lambda x: 1 if x['type_of_column'] in ['object'] and x[
123 | 'index'] not in string_bool_vars + cols_delete else 0, axis=1)
124 | ######### This is where we figure out whether a string var is nlp or discrete_string var ###
125 | var_df['nlp_strings'] = 0
126 | var_df['discrete_strings'] = 0
127 | var_df['cat'] = 0
128 | var_df['id_col'] = 0
129 | discrete_or_nlp_vars = var_df.loc[discrete_or_nlp == 1]['index'].values.tolist()
130 | copy_discrete_or_nlp_vars = copy.deepcopy(discrete_or_nlp_vars)
131 | if len(discrete_or_nlp_vars) > 0:
132 | for col in copy_discrete_or_nlp_vars:
133 | #### first fill empty or missing vals since it will blowup ###
134 | ### Remember that fillna only works at the dataframe level!
135 | train[[col]] = train[[col]].fillna(' ')
136 | if train[col].map(lambda x: len(x) if type(x) == str else 0).max(
137 | ) >= 50 and len(train[col].value_counts()) >= int(0.9 * len(train)) and col not in string_bool_vars:
138 | var_df.loc[var_df['index'] == col, 'nlp_strings'] = 1
139 | elif train[col].map(lambda x: len(x) if type(x) == str else 0).mean(
140 | ) >= max_nlp_char_size and train[col].map(lambda x: len(x) if type(x) == str else 0).max(
141 | ) < 50 and len(train[col].value_counts()
142 | ) <= int(0.9 * len(train)) and col not in string_bool_vars:
143 | var_df.loc[var_df['index'] == col, 'discrete_strings'] = 1
144 | elif len(train[col].value_counts()) > cat_limit and len(
145 | train[col].value_counts()) <= int(0.9 * len(train)) and col not in string_bool_vars:
146 | var_df.loc[var_df['index'] == col, 'discrete_strings'] = 1
147 | elif len(train[col].value_counts()) > cat_limit and len(train[col].value_counts()
148 | ) == len(train) and col not in string_bool_vars:
149 | var_df.loc[var_df['index'] == col, 'id_col'] = 1
150 | else:
151 | var_df.loc[var_df['index'] == col, 'cat'] = 1
152 | nlp_vars = list(var_df[(var_df['nlp_strings'] == 1)]['index'])
153 | sum_all_cols['nlp_vars'] = nlp_vars
154 | discrete_string_vars = list(var_df[(var_df['discrete_strings'] == 1)]['index'])
155 | sum_all_cols['discrete_string_vars'] = discrete_string_vars
156 | ###### This happens only if a string column happens to be an ID column #######
157 | #### DO NOT Add this to ID_VARS yet. It will be done later. Don't change it easily...
158 | #### Category DTYPE vars are very special = they can be left as is and not disturbed in Python. ###
159 | var_df['dcat'] = var_df.apply(lambda x: 1 if str(x['type_of_column']) == 'category' else 0,
160 | axis=1)
161 | factor_vars = list(var_df[(var_df['dcat'] == 1)]['index'])
162 | sum_all_cols['factor_vars'] = factor_vars
163 | ########################################################################
164 | date_or_id = var_df.apply(lambda x: 1 if x['type_of_column'] in [np.uint8,
165 | np.uint16, np.uint32, np.uint64,
166 | 'int8', 'int16',
167 | 'int32', 'int64'] and x[
168 | 'index'] not in string_bool_vars + num_bool_vars + discrete_string_vars + nlp_vars else 0,
169 | axis=1)
170 | ######### This is where we figure out whether a numeric col is date or id variable ###
171 | var_df['int'] = 0
172 | var_df['date_time'] = 0
173 | ### if a particular column is date-time type, now set it as a date time variable ##
174 | var_df['date_time'] = var_df.apply(lambda x: 1 if x['type_of_column'] in [' 2050:
182 | var_df.loc[var_df['index'] == col, 'id_col'] = 1
183 | else:
184 | try:
185 | pd.to_datetime(train[col], infer_datetime_format=True)
186 | var_df.loc[var_df['index'] == col, 'date_time'] = 1
187 | except:
188 | var_df.loc[var_df['index'] == col, 'id_col'] = 1
189 | else:
190 | if train[col].min() < 1900 or train[col].max() > 2050:
191 | if col not in num_bool_vars:
192 | var_df.loc[var_df['index'] == col, 'int'] = 1
193 | else:
194 | try:
195 | pd.to_datetime(train[col], infer_datetime_format=True)
196 | var_df.loc[var_df['index'] == col, 'date_time'] = 1
197 | except:
198 | if col not in num_bool_vars:
199 | var_df.loc[var_df['index'] == col, 'int'] = 1
200 | else:
201 | pass
202 | int_vars = list(var_df[(var_df['int'] == 1)]['index'])
203 | date_vars = list(var_df[(var_df['date_time'] == 1)]['index'])
204 | id_vars = list(var_df[(var_df['id_col'] == 1)]['index'])
205 | sum_all_cols['int_vars'] = int_vars
206 | copy_date_vars = copy.deepcopy(date_vars)
207 | for date_var in copy_date_vars:
208 | #### This test is to make sure date vars are actually date vars
209 | try:
210 | pd.to_datetime(train[date_var], infer_datetime_format=True)
211 | except:
212 | ##### if not a date var, then just add it to delete it from processing
213 | cols_delete.append(date_var)
214 | date_vars.remove(date_var)
215 | sum_all_cols['date_vars'] = date_vars
216 | sum_all_cols['id_vars'] = id_vars
217 | sum_all_cols['cols_delete'] = cols_delete
218 | ## This is an EXTREMELY complicated logic for cat vars. Don't change it unless you test it many times!
219 | var_df['numeric'] = 0
220 | float_or_cat = var_df.apply(lambda x: 1 if x['type_of_column'] in ['float16',
221 | 'float32', 'float64'] else 0,
222 | axis=1)
223 | ####### We need to make sure there are no categorical vars in float #######
224 | if len(var_df.loc[float_or_cat == 1]) > 0:
225 | for col in var_df.loc[float_or_cat == 1]['index'].values.tolist():
226 | if 2 < len(train[col].value_counts()) <= len(
227 | train) and len(train[col].value_counts()
228 | ) <= float_limit:
229 | var_df.loc[var_df['index'] == col, 'cat'] = 1
230 | else:
231 | if col not in (num_bool_vars + factor_vars):
232 | var_df.loc[var_df['index'] == col, 'numeric'] = 1
233 | cat_vars = list(var_df[(var_df['cat'] == 1)]['index'])
234 | continuous_vars = list(var_df[(var_df['numeric'] == 1)]['index'])
235 |
236 | ######## V E R Y I M P O R T A N T ###################################################
237 | cat_vars_copy = copy.deepcopy(factor_vars)
238 | for cat in cat_vars_copy:
239 | if df_preds[cat].dtype == float:
240 | continuous_vars.append(cat)
241 | factor_vars.remove(cat)
242 | var_df.loc[var_df['index'] == cat, 'dcat'] = 0
243 | var_df.loc[var_df['index'] == cat, 'numeric'] = 1
244 | elif len(df_preds[cat].value_counts()) == df_preds.shape[0]:
245 | id_vars.append(cat)
246 | factor_vars.remove(cat)
247 | var_df.loc[var_df['index'] == cat, 'dcat'] = 0
248 | var_df.loc[var_df['index'] == cat, 'id_col'] = 1
249 |
250 | sum_all_cols['factor_vars'] = factor_vars
251 | ##### There are a couple of extra tests you need to do to remove aberrations in cat_vars ###
252 | cat_vars_copy = copy.deepcopy(cat_vars)
253 | for cat in cat_vars_copy:
254 | if df_preds[cat].dtype == float:
255 | continuous_vars.append(cat)
256 | cat_vars.remove(cat)
257 | var_df.loc[var_df['index'] == cat, 'cat'] = 0
258 | var_df.loc[var_df['index'] == cat, 'numeric'] = 1
259 | elif len(df_preds[cat].value_counts()) == df_preds.shape[0]:
260 | id_vars.append(cat)
261 | cat_vars.remove(cat)
262 | var_df.loc[var_df['index'] == cat, 'cat'] = 0
263 | var_df.loc[var_df['index'] == cat, 'id_col'] = 1
264 | sum_all_cols['cat_vars'] = cat_vars
265 | sum_all_cols['continuous_vars'] = continuous_vars
266 | sum_all_cols['id_vars'] = id_vars
267 | ###### This is where you consolidate the numbers ###########
268 | var_dict_sum = dict(zip(var_df.values[:, 0], var_df.values[:, 2:].sum(1)))
269 | for col, sumval in var_dict_sum.items():
270 | if sumval == 0:
271 | print('%s of type=%s is not classified' % (col, train[col].dtype))
272 | elif sumval > 1:
273 | print('%s of type=%s is classified into more then one type' % (col, train[col].dtype))
274 | else:
275 | pass
276 | ##### If there are more than 1000 unique values, then add it to NLP vars ###
277 | copy_discretes = copy.deepcopy(discrete_string_vars)
278 | for each_discrete in copy_discretes:
279 | if train[each_discrete].nunique() >= 1000:
280 | nlp_vars.append(each_discrete)
281 | discrete_string_vars.remove(each_discrete)
282 | elif 100 < train[each_discrete].nunique() < 1000:
283 | pass
284 | else:
285 | ### If it is less than 100 unique values, then make it categorical var
286 | cat_vars.append(each_discrete)
287 | discrete_string_vars.remove(each_discrete)
288 | sum_all_cols['discrete_string_vars'] = discrete_string_vars
289 | sum_all_cols['cat_vars'] = cat_vars
290 | sum_all_cols['nlp_vars'] = nlp_vars
291 | ############### This is where you print all the types of variables ##############
292 | ####### Returns 8 vars in the following order: continuous_vars,int_vars,cat_vars,
293 | ### string_bool_vars,discrete_string_vars,nlp_vars,date_or_id_vars,cols_delete
294 | if verbose == 1:
295 | print(" Number of Numeric Columns = ", len(continuous_vars))
296 | print(" Number of Integer-Categorical Columns = ", len(int_vars))
297 | print(" Number of String-Categorical Columns = ", len(cat_vars))
298 | print(" Number of Factor-Categorical Columns = ", len(factor_vars))
299 | print(" Number of String-Boolean Columns = ", len(string_bool_vars))
300 | print(" Number of Numeric-Boolean Columns = ", len(num_bool_vars))
301 | print(" Number of Discrete String Columns = ", len(discrete_string_vars))
302 | print(" Number of NLP String Columns = ", len(nlp_vars))
303 | print(" Number of Date Time Columns = ", len(date_vars))
304 | print(" Number of ID Columns = ", len(id_vars))
305 | print(" Number of Columns to Delete = ", len(cols_delete))
306 | if verbose == 2:
307 | print(' Printing upto %d columns max in each category:' % max_cols_to_print)
308 | print(" Numeric Columns : %s" % continuous_vars[:max_cols_to_print])
309 | print(" Integer-Categorical Columns: %s" % int_vars[:max_cols_to_print])
310 | print(" String-Categorical Columns: %s" % cat_vars[:max_cols_to_print])
311 | print(" Factor-Categorical Columns: %s" % factor_vars[:max_cols_to_print])
312 | print(" String-Boolean Columns: %s" % string_bool_vars[:max_cols_to_print])
313 | print(" Numeric-Boolean Columns: %s" % num_bool_vars[:max_cols_to_print])
314 | print(" Discrete String Columns: %s" % discrete_string_vars[:max_cols_to_print])
315 | print(" NLP text Columns: %s" % nlp_vars[:max_cols_to_print])
316 | print(" Date Time Columns: %s" % date_vars[:max_cols_to_print])
317 | print(" ID Columns: %s" % id_vars[:max_cols_to_print])
318 | print(" Columns that will not be considered in modeling: %s" % cols_delete[:max_cols_to_print])
319 | ##### now collect all the column types and column names into a single dictionary to return!
320 |
321 | len_sum_all_cols = reduce(add, [len(v) for v in sum_all_cols.values()])
322 | if len_sum_all_cols == orig_cols_total:
323 | print(' %d Predictors classified...' % orig_cols_total)
324 | # print(' This does not include the Target column(s)')
325 | else:
326 | print('No of columns classified %d does not match %d total cols. Continuing...' % (
327 | len_sum_all_cols, orig_cols_total))
328 | ls = sum_all_cols.values()
329 | flat_list = [item for sublist in ls for item in sublist]
330 | if len(left_subtract(list(train), flat_list)) == 0:
331 | print(' Missing columns = None')
332 | else:
333 | print(' Missing columns = %s' % left_subtract(list(train), flat_list))
334 | return sum_all_cols
335 | ####################################################################################
336 |
--------------------------------------------------------------------------------
/autoviml/custom_scores.py:
--------------------------------------------------------------------------------
1 | import warnings
2 |
3 | import numpy as np
4 | from sklearn.metrics import accuracy_score, confusion_matrix, f1_score, fbeta_score, log_loss
5 | from sklearn.metrics import mean_squared_error, median_absolute_error, mean_absolute_error
6 | from sklearn.metrics import mean_squared_log_error, precision_score, recall_score, roc_auc_score
7 |
8 |
9 | def balanced_accuracy_score(y_true, y_pred, sample_weight=None,
10 | adjusted=False):
11 | matrix = confusion_matrix(y_true, y_pred, sample_weight=sample_weight)
12 | with np.errstate(divide='ignore', invalid='ignore'):
13 | per_class = np.diag(matrix) / matrix.sum(axis=1)
14 | if np.any(np.isnan(per_class)):
15 | warnings.warn('y_pred contains classes not in y_true')
16 | per_class = per_class[~np.isnan(per_class)]
17 | score = np.mean(per_class)
18 | if adjusted:
19 | n_classes = len(per_class)
20 | chance = 1 / n_classes
21 | score -= chance
22 | score /= 1 - chance
23 | return score
24 |
25 |
26 | #####################################################################################
27 | def accu(results, y_cv):
28 | return (results == y_cv).astype(int).sum(axis=0) / (y_cv.shape[0])
29 |
30 |
31 | def rmse(results, y_cv):
32 | return np.sqrt(np.mean((results - y_cv) ** 2, axis=0))
33 |
34 |
35 | # Defining objective functions for HyperOpt here
36 | def gini(truth, predictions):
37 | g = np.asarray(np.c_[truth, predictions, np.arange(len(truth))], dtype=np.float)
38 | g = g[np.lexsort((g[:, 2], -1 * g[:, 1]))]
39 | gs = g[:, 0].cumsum().sum() / g[:, 0].sum()
40 | gs -= (len(truth) + 1) / 2.
41 | return gs / len(truth)
42 |
43 |
44 | def gini_sklearn(truth, predictions):
45 | return gini(truth, predictions) / gini(truth, truth)
46 |
47 |
48 | def gini_meae(truth, predictions):
49 | score = median_absolute_error(truth, predictions)
50 | return score
51 |
52 |
53 | def gini_msle(truth, predictions):
54 | score = mean_squared_log_error(truth, predictions)
55 | return score
56 |
57 |
58 | def gini_mae(truth, predictions):
59 | score = mean_absolute_error(truth, predictions)
60 | return score
61 |
62 |
63 | def gini_mse(truth, predictions):
64 | score = mean_squared_error(truth, predictions)
65 | return score
66 |
67 |
68 | def gini_rmse(truth, predictions):
69 | score = np.sqrt(mean_squared_error(truth, predictions))
70 | return score
71 |
72 |
73 | def gini_accuracy(truth, predictions):
74 | return accuracy_score(truth, predictions)
75 |
76 |
77 | def gini_bal_accuracy(truth, predictions):
78 | return balanced_accuracy_score(truth, predictions)
79 |
80 |
81 | def gini_roc(truth, predictions):
82 | return roc_auc_score(truth, predictions)
83 |
84 |
85 | def gini_precision(truth, predictions, pos_label=1):
86 | return precision_score(truth, predictions, average=None)[pos_label]
87 |
88 |
89 | def gini_average_precision(truth, predictions):
90 | return precision_score(truth, predictions.argmax(axis=1), average='macro')
91 |
92 |
93 | def gini_weighted_precision(truth, predictions):
94 | return precision_score(truth, predictions.argmax(axis=1), average='weighted')
95 |
96 |
97 | def gini_macro_precision(truth, predictions):
98 | return precision_score(truth, predictions.argmax(axis=1), average='macro')
99 |
100 |
101 | def gini_micro_precision(truth, predictions):
102 | return precision_score(truth, predictions.argmax(axis=1), average='micro')
103 |
104 |
105 | def gini_samples_precision(truth, predictions):
106 | return precision_score(truth, predictions.argmax(axis=1), average='samples')
107 |
108 |
109 | def gini_f1(truth, predictions, pos_label=1):
110 | return f1_score(truth, predictions, average=None)[pos_label]
111 |
112 |
113 | def gini_weighted_f1(truth, predictions):
114 | return f1_score(truth, predictions.argmax(axis=1), average='weighted')
115 |
116 |
117 | def gini_macro_f1(truth, predictions):
118 | return f1_score(truth, predictions.argmax(axis=1), average='macro')
119 |
120 |
121 | def gini_micro_f1(truth, predictions):
122 | return f1_score(truth, predictions.argmax(axis=1), average='micro')
123 |
124 |
125 | def gini_samples_f1(truth, predictions):
126 | return f1_score(truth, predictions.argmax(axis=1), average='samples')
127 |
128 |
129 | def gini_log_loss(truth, predictions):
130 | return log_loss(truth, predictions, normalize=True)
131 |
132 |
133 | def gini_recall(truth, predictions, pos_label=1):
134 | return recall_score(truth, predictions, average=None)[pos_label]
135 |
136 |
137 | def gini_weighted_recall(truth, predictions):
138 | return recall_score(truth, predictions.argmax(axis=1), average='weighted')
139 |
140 |
141 | def gini_samples_recall(truth, predictions):
142 | return recall_score(truth, predictions.argmax(axis=1), average='samples')
143 |
144 |
145 | def gini_macro_recall(truth, predictions):
146 | return recall_score(truth, predictions.argmax(axis=1), average='macro')
147 |
148 |
149 | def gini_micro_recall(truth, predictions):
150 | return recall_score(truth, predictions.argmax(axis=1), average='micro')
151 |
152 |
153 | # calculate f2-measure
154 | def f2_measure(y_true, y_pred):
155 | return fbeta_score(y_true, y_pred, beta=2)
156 |
--------------------------------------------------------------------------------
/autoviml/custom_scores_HO.py:
--------------------------------------------------------------------------------
1 | import warnings
2 |
3 | import numpy as np
4 | from sklearn.metrics import accuracy_score
5 | from sklearn.metrics import average_precision_score
6 | #####################################################################################
7 | from sklearn.metrics import confusion_matrix
8 | from sklearn.metrics import f1_score
9 | from sklearn.metrics import fbeta_score
10 | from sklearn.metrics import log_loss
11 | from sklearn.metrics import make_scorer
12 | from sklearn.metrics import mean_squared_error, median_absolute_error, mean_absolute_error
13 | from sklearn.metrics import mean_squared_log_error
14 | from sklearn.metrics import precision_score
15 | from sklearn.metrics import recall_score
16 | from sklearn.metrics import roc_auc_score
17 |
18 |
19 | def balanced_accuracy_score(y_true, y_pred, sample_weight=None,
20 | adjusted=False):
21 | C = confusion_matrix(y_true, y_pred, sample_weight=sample_weight)
22 | with np.errstate(divide='ignore', invalid='ignore'):
23 | per_class = np.diag(C) / C.sum(axis=1)
24 | if np.any(np.isnan(per_class)):
25 | warnings.warn('y_pred contains classes not in y_true')
26 | per_class = per_class[~np.isnan(per_class)]
27 | score = np.mean(per_class)
28 | if adjusted:
29 | n_classes = len(per_class)
30 | chance = 1 / n_classes
31 | score -= chance
32 | score /= 1 - chance
33 | return score
34 |
35 |
36 | #####################################################################################
37 | def accu(results, y_cv):
38 | return (results == y_cv).astype(int).sum(axis=0) / (y_cv.shape[0])
39 |
40 |
41 | def rmse(results, y_cv):
42 | return np.sqrt(np.mean((results - y_cv) ** 2, axis=0))
43 |
44 |
45 | ######## Defining objective functions for HyperOpt here ######################
46 | ### keep all Classification Scorers greater_is_better True but subtract scorer from 1 so it diminishes.
47 | #### This is the only way HyperOpt will find the minimum - so don't change this code anytime!
48 | def gini(truth, predictions):
49 | g = np.asarray(np.c_[truth, predictions, np.arange(len(truth))], dtype=np.float)
50 | g = g[np.lexsort((g[:, 2], -1 * g[:, 1]))]
51 | gs = g[:, 0].cumsum().sum() / g[:, 0].sum()
52 | gs -= (len(truth) + 1) / 2.
53 | return gs / len(truth)
54 |
55 |
56 | def gini_sklearn(truth, predictions):
57 | return gini(truth, predictions) / gini(truth, truth)
58 |
59 |
60 | def gini_meae(truth, predictions):
61 | score = median_absolute_error(truth, predictions)
62 | return score
63 |
64 |
65 | def gini_msle(truth, predictions):
66 | score = np.sqrt(mean_squared_log_error(truth, predictions))
67 | return score
68 |
69 |
70 | def gini_mae(truth, predictions):
71 | score = mean_absolute_error(truth, predictions)
72 | return score
73 |
74 |
75 | def gini_mse(truth, predictions):
76 | score = np.sqrt(mean_squared_error(truth, predictions))
77 | return score
78 |
79 |
80 | def gini_rmse(truth, predictions):
81 | score = np.sqrt(mean_squared_error(truth, predictions))
82 | return score
83 |
84 |
85 | def gini_accuracy(truth, predictions):
86 | return 1 - accuracy_score(truth, predictions)
87 |
88 |
89 | def gini_bal_accuracy(truth, predictions):
90 | return 1 - balanced_accuracy_score(truth, predictions)
91 |
92 |
93 | def gini_roc(truth, predictions):
94 | return 1 - roc_auc_score(truth, predictions)
95 |
96 |
97 | def gini_precision(truth, predictions):
98 | return 1 - precision_score(truth, predictions)
99 |
100 |
101 | def gini_average_precision(truth, predictions):
102 | return 1 - average_precision_score(truth, predictions.argmax(axis=1), average='weighted')
103 |
104 |
105 | def gini_weighted_precision(truth, predictions):
106 | return 1 - precision_score(truth, predictions.argmax(axis=1), average='weighted')
107 |
108 |
109 | def gini_macro_precision(truth, predictions):
110 | return 1 - precision_score(truth, predictions.argmax(axis=1), average='macro')
111 |
112 |
113 | def gini_micro_precision(truth, predictions):
114 | return 1 - precision_score(truth, predictions.argmax(axis=1), average='micro')
115 |
116 |
117 | def gini_samples_precision(truth, predictions):
118 | return 1 - precision_score(truth, predictions.argmax(axis=1), average='samples')
119 |
120 |
121 | def gini_f1(truth, predictions):
122 | return 1 - f1_score(truth, predictions)
123 |
124 |
125 | def gini_weighted_f1(truth, predictions):
126 | return 1 - f1_score(truth, predictions.argmax(axis=1), average='weighted')
127 |
128 |
129 | def gini_macro_f1(truth, predictions):
130 | return 1 - f1_score(truth, predictions.argmax(axis=1), average='macro')
131 |
132 |
133 | def gini_micro_f1(truth, predictions):
134 | return 1 - f1_score(truth, predictions.argmax(axis=1), average='micro')
135 |
136 |
137 | def gini_samples_f1(truth, predictions):
138 | return 1 - f1_score(truth, predictions.argmax(axis=1), average='samples')
139 |
140 |
141 | def gini_log_loss(truth, predictions):
142 | return log_loss(truth, predictions, normalize=True)
143 |
144 |
145 | def gini_recall(truth, predictions):
146 | return 1 - recall_score(truth, predictions)
147 |
148 |
149 | def gini_weighted_recall(truth, predictions):
150 | return 1 - recall_score(truth, predictions.argmax(axis=1), average='weighted')
151 |
152 |
153 | def gini_samples_recall(truth, predictions):
154 | return 1 - recall_score(truth, predictions.argmax(axis=1), average='samples')
155 |
156 |
157 | def gini_macro_recall(truth, predictions):
158 | return 1 - recall_score(truth, predictions.argmax(axis=1), average='macro')
159 |
160 |
161 | def gini_micro_recall(truth, predictions):
162 | return 1 - recall_score(truth, predictions.argmax(axis=1), average='micro')
163 |
164 |
165 | def gini_roc_auc(truth, predictions):
166 | return 1 - roc_auc_score(truth, predictions.argmax(axis=1), average='macro')
167 |
168 |
169 | # calculate f2-measure
170 | def f2_measure(y_true, y_pred):
171 | return fbeta_score(y_true, y_pred, beta=2)
172 |
173 |
174 | ### keep all Regression Scorers greater_is_better True since it leaves them as is and minimizes them
175 | meae_scorer = make_scorer(gini_meae, greater_is_better=True)
176 | msle_scorer = make_scorer(gini_msle, greater_is_better=True)
177 | mae_scorer = make_scorer(gini_mae, greater_is_better=True)
178 | mse_scorer = make_scorer(gini_mse, greater_is_better=True)
179 |
180 | ### keep all Classification Scorers greater_is_better True but subtract scorer from 1 so it diminishes.
181 | #### This is the only way HyperOpt will find the minimum - so don't change this code anytime!
182 | accuracy_scorer = make_scorer(gini_accuracy, greater_is_better=True, needs_proba=False)
183 | bal_accuracy_scorer = make_scorer(gini_bal_accuracy, greater_is_better=True, needs_proba=False)
184 |
185 | gini_scorer = make_scorer(gini_sklearn, greater_is_better=True, needs_proba=True)
186 | roc_scorer = make_scorer(gini_roc, greater_is_better=True, needs_proba=True)
187 |
188 | precision_scorer = make_scorer(gini_precision, greater_is_better=True, needs_proba=False)
189 | average_precision_scorer = make_scorer(gini_average_precision, greater_is_better=True, needs_proba=True)
190 | weighted_precision_scorer = make_scorer(gini_weighted_precision, greater_is_better=True, needs_proba=True)
191 | macro_precision_scorer = make_scorer(gini_macro_precision, greater_is_better=True, needs_proba=True)
192 | micro_precision_scorer = make_scorer(gini_micro_precision, greater_is_better=True, needs_proba=True)
193 | samples_precision_scorer = make_scorer(gini_samples_precision, greater_is_better=True, needs_proba=True)
194 |
195 | f1_scorer = make_scorer(gini_f1, greater_is_better=True, needs_proba=False)
196 | weighted_f1_scorer = make_scorer(gini_weighted_f1, greater_is_better=True, needs_proba=True)
197 | macro_f1_scorer = make_scorer(gini_macro_f1, greater_is_better=True, needs_proba=True)
198 | micro_f1_scorer = make_scorer(gini_micro_f1, greater_is_better=True, needs_proba=True)
199 | samples_f1_scorer = make_scorer(gini_samples_f1, greater_is_better=True, needs_proba=True)
200 |
201 | recall_scorer = make_scorer(gini_recall, greater_is_better=True, needs_proba=False)
202 | weighted_recall_scorer = make_scorer(gini_weighted_recall, greater_is_better=True, needs_proba=True)
203 | macro_recall_scorer = make_scorer(gini_macro_recall, greater_is_better=True, needs_proba=True)
204 | micro_recall_scorer = make_scorer(gini_micro_recall, greater_is_better=True, needs_proba=True)
205 | samples_recall_scorer = make_scorer(gini_samples_recall, greater_is_better=True, needs_proba=True)
206 |
207 | roc_auc_scorer = make_scorer(gini_roc_auc, greater_is_better=True, needs_proba=True)
208 | ### Leave the log-loss scorer as greater_is_better True since it keeps sign and minimizes it.
209 | logloss_scorer = make_scorer(gini_log_loss, greater_is_better=True, needs_proba=True)
210 | ##########################################################################################
211 |
--------------------------------------------------------------------------------
/autoviml/feature_engineering.py:
--------------------------------------------------------------------------------
1 | import pandas as pd
2 | import numpy as np
3 |
4 | ######### NEW And FAST WAY to ADD Feature Engg COLUMNS IN A DATA SET #######
5 | ### Modify Dataframe by adding Features using Feature Tools ####
6 | def add_computational_primitive_features(df, add_types: list, idcolumn=''):
7 | """
8 | ### Modify Dataframe by adding computational primitive Features using Feature Tools ####
9 | ### What are computational primitives? they are to add, subtract, multiply and divide features
10 | ### Inputs:
11 | ### df: Just sent in the data frame df that you want features added to
12 | ### add_types: list of computational types: 'add', 'subtract', 'multiply' and 'divide'. Choose any or all.
13 | ### idcolumn: this is to create an index for the dataframe since FT runs on index variable. You can leave it empty string.
14 | """
15 | try:
16 | import featuretools as ft
17 | except:
18 | print('pip install featuretools and try this again')
19 | df = copy.deepcopy(df)
20 | projectid = 'project_prediction'
21 | dataid = 'project_data'
22 | if idcolumn == '':
23 | indexid = 'index'
24 | make_index = True
25 | else:
26 | indexid = idcolumn
27 | make_index = False
28 | # Make an entityset and add the entity
29 | es = ft.EntitySet(id=projectid)
30 | es.entity_from_dataframe(entity_id=dataid, dataframe=df,
31 | make_index=make_index, index=indexid)
32 |
33 | # Run deep feature synthesis with given input primitives or automatically deep 2
34 | if len(add_types) > 0:
35 | ### Build Features based on given primitive types, add_types which is a list
36 | df_mod, feature_defs = ft.dfs(entityset=es, target_entity=dataid,
37 | trans_primitives=add_types)
38 | else:
39 | ### Perform Deep Feature Synthesis Automatically for Depth 2
40 | df_mod, feature_defs = ft.dfs(entityset=es, target_entity=dataid,
41 | max_depth=2, n_jobs=-1,
42 | verbose=0)
43 | if make_index:
44 | df_mod = df_mod.reset_index(drop=True)
45 | return df_mod
46 |
47 |
48 | def feature_engineering(df, ft_requests, idcol):
49 | """
50 | The Feature Engineering module needs FeatureTools installed to work.
51 | So please do "pip install featuretools" before trying out this module.
52 | It takes a given data set, df and adds features based on the requet types in
53 | ft_requests which can be 'add','subtract','multiply','divide'. If you have
54 | an id_column in the data set, you can provide it as idcol (a string variable).
55 | It will return your modified dataset with 'idcol' as the index. Make sure
56 | you reset the index if you want to return it to its former state.
57 | Also do not send in your entire dataframe! Just send a small dataframe with a few variables.
58 | Once you see how it adds to performance of model, you can add more variables to dataframe.
59 | """
60 | df = copy.deepcopy(df)
61 | if df.shape[1] < 2:
62 | print('More than one column in dataframe required to perform feature engineering. Returning')
63 | return df
64 | ft_dict = dict(zip(['add', 'multiply', 'subtract', 'divide'],
65 | ['add_numeric', 'multiply_numeric',
66 | 'subtract_numeric', 'divide_numeric']))
67 | if len(ft_requests) > 0:
68 | ft_list = []
69 | for ft_one in ft_requests:
70 | if ft_one in ft_dict.keys():
71 | ft_list.append(ft_dict[ft_one])
72 | else:
73 | print(' Cannot perform %s-type feature engineering...' % ft_one)
74 | cols = [x for x in df.columns.tolist() if x not in [idcol]]
75 | for each_ft, count in zip(ft_list, range(len(ft_list))):
76 | if count == 0:
77 | df_mod = add_computational_primitive_features(df, [each_ft], idcol)
78 | print(df_mod.shape)
79 | else:
80 | df_temp = add_computational_primitive_features(df, [each_ft], idcol)
81 | df_temp.drop(cols, axis=1, inplace=True)
82 | df_mod = pd.concat([df_mod, df_temp], axis=1, ignore_index=False)
83 | print(df_mod.shape)
84 | else:
85 | df_mod = add_computational_primitive_features(df, [], idcol)
86 | return df_mod
87 |
88 |
89 | def add_date_time_features(smalldf, startTime, endTime, splitter_date_string="/", splitter_hour_string=":"):
90 | """
91 | If you have start date time stamp and end date time stamp, this module will create additional features for such fields.
92 | You must provide a start date time stamp field and if you have an end date time stamp field, you must use it.
93 | Otherwise, you are better off using the create_date_time_features module which is also in this library.
94 | You must provide the following:
95 | smalldf: Dataframe containing your date time fields
96 | startTime: this is hopefully a string field which converts to a date time stamp easily. Make sure it is a string.
97 | endTime: this also must be a string field which converts to a date time stamp easily. Make sure it is a string.
98 | splitter_date_string: usually there is a string such as '/' or '.' between day/month/year etc. Default is assumed / here.
99 | splitter_hour_string: usually there is a string such as ':' or '.' between hour:min:sec etc. Default is assumed : here.
100 | """
101 | smalldf = smalldf.copy()
102 | add_cols = []
103 | start_date = 'processing' + startTime + '_start_date'
104 | smalldf[start_date] = smalldf[startTime].map(lambda x: x.split(" ")[0])
105 | add_cols.append(start_date)
106 | try:
107 | start_time = 'processing' + startTime + '_start_time'
108 | smalldf[start_time] = smalldf[startTime].map(lambda x: x.split(" ")[1])
109 | add_cols.append(start_time)
110 | except:
111 | ### there is no hour-minutes part of this date time stamp field. You can just skip it if it is not there
112 | pass
113 | end_date = 'processing' + endTime + '_end_date'
114 | smalldf[end_date] = smalldf[endTime].map(lambda x: x.split(" ")[0])
115 | add_cols.append(end_date)
116 | try:
117 | end_time = 'processing' + endTime + '_end_time'
118 | smalldf[end_time] = smalldf[endTime].map(lambda x: x.split(" ")[1])
119 | add_cols.append(end_time)
120 | except:
121 | ### there is no hour-minutes part of this date time stamp field. You can just skip it if it is not there
122 | pass
123 | view_days = 'processing' + startTime + '_elapsed_days'
124 | smalldf[view_days] = (pd.to_datetime(smalldf[end_date]) - pd.to_datetime(smalldf[start_date])).values.astype(int)
125 | add_cols.append(view_days)
126 | try:
127 | view_time = 'processing' + startTime + '_elapsed_time'
128 | smalldf[view_time] = (pd.to_datetime(smalldf[end_time]) - pd.to_datetime(smalldf[start_time])).astype(
129 | 'timedelta64[s]').values
130 | add_cols.append(view_time)
131 | except:
132 | ### In some date time fields this gives an error so skip it in that case
133 | pass
134 | #### The reason we chose endTime here is that startTime is usually taken care of by another library. So better to do this alone.
135 | year = 'processing' + endTime + '_end_year'
136 | smalldf[year] = smalldf[end_date].map(lambda x: str(x).split(splitter_date_string)[0]).values
137 | add_cols.append(year)
138 | #### The reason we chose endTime here is that startTime is usually taken care of by another library. So better to do this alone.
139 | month = 'processing' + endTime + '_end_month'
140 | smalldf[month] = smalldf[end_date].map(lambda x: str(x).split(splitter_date_string)[1]).values
141 | add_cols.append(month)
142 | try:
143 | #### The reason we chose endTime here is that startTime is usually taken care of by another library. So better to do this alone.
144 | daynum = 'processing' + endTime + '_end_day_number'
145 | smalldf[daynum] = smalldf[end_date].map(lambda x: str(x).split(splitter_date_string)[2]).values
146 | add_cols.append(daynum)
147 | except:
148 | ### In some date time fields the day number is not there. If not, just skip it ####
149 | pass
150 | #### In some date time fields, the hour and minute is not there, so skip it in that case if it errors!
151 | try:
152 | start_hour = 'processing' + startTime + '_start_hour'
153 | smalldf[start_hour] = smalldf[start_time].map(lambda x: str(x).split(splitter_hour_string)[0]).values
154 | add_cols.append(start_hour)
155 | start_min = 'processing' + startTime + '_start_hour'
156 | smalldf[start_min] = smalldf[start_time].map(lambda x: str(x).split(splitter_hour_string)[1]).values
157 | add_cols.append(start_min)
158 | except:
159 | ### If it errors, skip it
160 | pass
161 | #### Check if there is a weekday and weekends in date time columns using endTime only
162 | weekday_num = 'processing' + endTime + '_end_weekday_number'
163 | smalldf[weekday_num] = pd.to_datetime(smalldf[end_date]).dt.weekday.values
164 | add_cols.append(weekday_num)
165 | weekend = 'processing' + endTime + '_end_weekend_flag'
166 | smalldf[weekend] = smalldf[weekday_num].map(lambda x: 1 if x in [5, 6] else 0)
167 | add_cols.append(weekend)
168 | #### If everything works well, there should be 13 new columns added by module. All the best!
169 | print('%d columns added using start date=%s and end date=%s processing...' % (len(add_cols), startTime, endTime))
170 | return smalldf
171 |
172 | #################################################################################
173 | def split_one_field_into_many(df, field, splitter, filler, new_names_list):
174 | """
175 | This little function takes any data frame field (string variables only) and splits
176 | it into as many fields as you want in the new_names_list.
177 | You can also specify what string to split on using the splitter argument.
178 | You can also fill Null values that occur due to your splitting by specifying a filler.
179 | if no new_names_list is given, then we use the name of the field itself to split.
180 | """
181 | import warnings
182 | warnings.filterwarnings("ignore")
183 | df = df.copy()
184 | ### First print the maximum number of things in that field
185 | max_things = df[field].map(lambda x: len(x.split(splitter))).max()
186 | if len(new_names_list) == 0:
187 | print(' Max. columns created by splitting %s field is %d.' % (
188 | field, max_things))
189 | else:
190 | print(
191 | ' Max. columns created by splitting %s field is %d but you have given %d variable names only. Selecting first %d' % (
192 | field, max_things, len(new_names_list), len(new_names_list)))
193 | ### This creates a new field that counts the number of things that are in that field.
194 | num_products_viewed = 'count_things_in_' + field
195 | df[num_products_viewed] = df[field].map(lambda x: len(x.split(";"))).values
196 | ### Clean up the field such that it has the right number of split chars otherwise add to it
197 | df[field] = df[field].map(
198 | lambda x: x + splitter * (max_things - len(x.split(";"))) if len(x.split(";")) < max_things else x)
199 | ###### Now you create new fields by split the one large field ########
200 | if new_names_list == '':
201 | new_names_list = [field + '_' + str(i) for i in range(1, max_things + 1)]
202 | try:
203 | for i in range(len(new_names_list)):
204 | df[field].fillna(filler, inplace=True)
205 | df.loc[df[field] == splitter, field] = filler
206 | df[new_names_list[i]] = df[field].map(lambda x: x.split(splitter)[i]
207 | if splitter in x else x)
208 | except:
209 | ### Check if the column is a string column. If not, give an error message.
210 | print('Cannot split the column. Getting an error. Check the column again')
211 | return df
212 | return df, new_names_list
213 |
214 |
215 | #################################################################################
216 | def add_aggregate_primitive_features(dft, agg_types, id_column, ignore_variables: list):
217 | """
218 | ### Modify Dataframe by adding computational primitive Features using Feature Tools ####
219 | ### What are aggregate primitives? they are to "mean""median","mode","min","max", etc. features
220 | ### Inputs:
221 | ### df: Just sent in the data frame df that you want features added to
222 | ### agg_types: list of computational types: 'mean','median','count', 'max', 'min', 'sum', etc.
223 | ### One caveat: these agg_types must be found in the agg_func of numpy or pandas groupby statement.
224 | ### for example: numpy has 'median','prod','sum','std','var', etc. - they will work!
225 | ### idcolumn: this is to create an index for the dataframe since FT runs on index variable. You can leave it empty string.
226 | ### ignore_variables: list of variables to ignore among numeric variables in data since they may be ID variables.
227 | """
228 | import copy
229 | ### Make sure the list of functions they send in are acceptable functions. If not, the aggregate will blow up!
230 | func_set = {'count', 'sum', 'mean', 'mad', 'median', 'min', 'max', 'mode', 'abs', 'prod', 'std', 'var', 'sem',
231 | 'skew', 'kurt', 'quantile', 'cumsum', 'cumprod', 'cummax', 'cummin'}
232 | agg_types = list(set(agg_types).intersection(func_set))
233 | ### If the ignore_variables list is empty, make sure you add the id_column to it so it can be dropped from aggregation.
234 | if len(ignore_variables) == 0:
235 | ignore_variables = [id_column]
236 | ### Select only integer and float variables to do this aggregation on. Be very careful if there are too many vars.
237 | ### This will take time to run in that case.
238 | dft_index = copy.deepcopy(dft[id_column])
239 | dft_cont = copy.deepcopy(dft.select_dtypes('number').drop(ignore_variables, axis=1))
240 | dft_cont[id_column] = dft_index
241 | try:
242 | dft_full = dft_cont.groupby(id_column).agg(agg_types)
243 | except:
244 | ### if for some reason, the groupby blows up, then just return the dataframe as is - no changes!
245 | return dft
246 | cols = [x + '_' + y + '_by_' + id_column for (x, y) in dft_full.columns]
247 | dft_full.columns = cols
248 | ### Not every column has useful values. If it is full of just the same value, remove it
249 | _, list_unique_col_ids = np.unique(dft_full, axis=1, return_index=True)
250 | dft_full = dft_full.iloc[:, list_unique_col_ids]
251 | return dft_full
252 |
253 |
254 | ################################################################################################################################
255 | import copy
256 | import time
257 | import pdb
258 |
259 |
260 | def FE_create_groupby_features(dft, groupby_columns, numeric_columns, agg_types):
261 | """
262 | FE means FEATURE ENGINEERING - That means this function will create new features
263 | Beware: this function will return a smaller dataframe than what you send in since it groups rows by keys.
264 | #########################################################################################################
265 | Function groups rows in a dft dataframe by the groupby_columns and returns multiple columns for the numeric column aggregated.
266 | Do not send in more than one column in the numeric column since beyond the first column it will be ignored!
267 | agg_type can be any numpy function such as mean, median, sum, count, etc.
268 | ##########################################################################################################
269 | Returns: a smaller dataframe with rows grouped by groupby_columns and aggregated for the numeric_column
270 | """
271 | start_time = time.time()
272 | print('Autoviml Feature Engineering: creating groupby features using %s' % groupby_columns)
273 | ########## This is where we create new columns by each numeric column grouped by group-by columns given.
274 | if isinstance(numeric_columns, list):
275 | pass
276 | elif isinstance(numeric_columns, str):
277 | numeric_columns = [numeric_columns]
278 | else:
279 | print(' Numeric column must be a string not a number Try again')
280 | return pd.DataFrame()
281 | grouped_list = pd.DataFrame()
282 | for iteration, numeric_column in zip(range(len(numeric_columns)), numeric_columns):
283 | grouped = dft.groupby(groupby_columns)[[numeric_column]]
284 | try:
285 | agg_type = agg_types[iteration]
286 | except:
287 | print(' No aggregation type given, hence mean is chosen by default')
288 | agg_type = 'mean'
289 | try:
290 | prefix = numeric_column + '_'
291 | if agg_type in ['Sum', 'sum']:
292 | grouped_agg = grouped.sum()
293 | elif agg_type in ['Mean', 'mean', 'Average', 'average']:
294 | grouped_agg = grouped.mean()
295 | elif agg_type in ['count', 'Count']:
296 | grouped_agg = grouped.count()
297 | elif agg_type in ['Median', 'median']:
298 | grouped_agg = grouped.median()
299 | elif agg_type in ['Maximum', 'maximum', 'max', 'Max']:
300 | ## maximum of the amounts
301 | grouped_agg = grouped.max()
302 | elif agg_type in ['Minimum', 'minimum', 'min', 'Min']:
303 | ## maximum of the amounts
304 | grouped_agg = grouped.min()
305 | else:
306 | grouped_agg = grouped.mean()
307 | grouped_sep = grouped_agg.unstack().add_prefix(prefix).fillna(0)
308 | except:
309 | print(' Error in creating groupby features...returning with null dataframe')
310 | grouped_sep = pd.DataFrame()
311 | if iteration == 0:
312 | grouped_list = copy.deepcopy(grouped_sep)
313 | else:
314 | grouped_list = pd.concat([grouped_list, grouped_sep], axis=1)
315 | print(
316 | ' After grouped features added by %s, number of columns = %d' % (numeric_column, grouped_list.shape[1]))
317 | #### once everything is done, you can close it here
318 | print('Time taken for creation of groupby features (in seconds) = %0.0f' % (time.time() - start_time))
319 | try:
320 | grouped_list.columns = grouped_list.columns.get_level_values(1)
321 | grouped_list.columns.name = None ## make sure the name on columns is removed
322 | grouped_list = grouped_list.reset_index() ## make sure the ID column comes back
323 | except:
324 | print(' Error in setting column names. Please reset column names after this step...')
325 | return grouped_list
326 | ################################################################################
327 | from scipy.stats import skew, kurtosis
328 | from sklearn.base import BaseEstimator, TransformerMixin
329 | from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier
330 | from sklearn.metrics import mean_squared_error, accuracy_score
331 |
332 | class EntropyBinningTransformer(BaseEstimator, TransformerMixin):
333 | """
334 | ######### This innovative binning transformer is found only in AutoViML ##############################
335 | The entropy binning approach in AutoViML is a novel and intriguing approach, especially as
336 | it leverages decision trees to determine optimal binning thresholds based on information gain
337 | (for classification) or mean squared error reduction (for regression).
338 |
339 | This method can adapt well to non-linear relationships between the feature and the target.
340 | It also incorporates some heuristics and improvements to enhance its effectiveness and efficiency.
341 |
342 | Heuristics:
343 | Variable Selection: We prioritize binning the most important variables.
344 | This is a sensible heuristic as it focuses computational resources on variables that are
345 | most likely to impact the model. We introduce a parameter to specify the number of top variables to bin.
346 |
347 | Adaptive max_depth: We've used an adaptive max_depth based on the number of continuous variables.
348 | This heuristic helps control the granularity of the binning. We can maintain this by adding a
349 | logic to adjust max_depth based on the number of features.
350 |
351 | Handling of Binned Variables: The code has an option to either keep the original variables
352 | alongside the binned versions or replace them. This flexibility can be useful and we keep it
353 | in the class implementation.
354 |
355 | Post-Binning Smoothing: After binning, applying smoothing techniques (e.g., Laplace smoothing)
356 | can help address bins with very low frequencies, which might be particularly useful for categorical targets.
357 |
358 | Incorporating these aspects, this is an updated version of the original EntropyBinningTransformer method in AutoViML
359 |
360 | Additional Improvements to consider:
361 |
362 | Evaluation and Optimization: Integrating a mechanism to evaluate the effectiveness of binning (e.g.,
363 | using cross-validation scores) and automatically optimize the binning parameters accordingly.
364 | #########################################################################################################
365 | EntropyBinningTransformer is a transformer class for binning numeric variables based on entropy using decision trees,
366 | with options for selecting the most relevant variables for binning based on their predictive power, skewness,
367 | and kurtosis, and for smoothing the bins post-binning to mitigate issues with overfitting and bins with
368 | low frequencies.
369 |
370 | Parameters:
371 | - max_depth: The maximum depth of the decision trees used for binning.
372 | - min_samples_leaf: The minimum number of samples required to be at a leaf node of the decision trees.
373 | - entropy_binning: Whether to apply entropy binning.
374 | - modeltype: Indicates whether the target variable is for 'Regression' or 'Classification'.
375 |
376 | Methods:
377 | - fit: Learns the binning thresholds for each variable.
378 | - transform: Applies the learned thresholds to bin the data, with an option for post-binning smoothing.
379 | - select_top_n_vars: Selects the top N variables for binning based on their predictive power and distribution
380 | characteristics.
381 | #########################################################################################################
382 | #### U S A G E #####
383 | # Initialize the EntropyBinningTransformer
384 | entropy_binner = EntropyBinningTransformer( replace_vars=True, modeltype='Classification')
385 |
386 | # Optionally, select top n variables based on their predictive power
387 | # This step is useful if you want to bin only the most informative variables
388 | top_vars = entropy_binner.select_top_n_vars(X_train, y_train, n=2)
389 | X_train_top_vars = X_train[top_vars]
390 |
391 | # Fit the transformer to the training data
392 | entropy_binner.fit(X_train_top_vars, y_train)
393 |
394 | # Transform the training data and new data using the learned binning thresholds
395 | X_train_binned = entropy_binner.transform(X_train_top_vars)
396 | X_test_binned = entropy_binner.transform(X_test[top_vars])
397 | #########################################################################################################
398 | """
399 | def __init__(self, replace_vars=True, modeltype='Classification', top_n_vars=None):
400 | self.replace_vars = replace_vars
401 | self.modeltype = modeltype
402 | self.max_depth = 10
403 | self.min_samples_leaf = 2
404 | self.top_n_vars = top_n_vars
405 | self.binning_thresholds = {}
406 | self.remvars = []
407 |
408 | def fit(self, X, y):
409 | # Determine top_n variables based on importance, for example, using feature importances from a preliminary model
410 | if self.top_n_vars is not None:
411 | top_vars = self.select_top_n_vars(X, y, n=self.top_n_vars)
412 | else:
413 | top_vars = X.columns
414 |
415 | for col in top_vars:
416 | max_depth = self._adjust_max_depth(len(top_vars))
417 | clf = self.get_decision_tree(max_depth)
418 |
419 | try:
420 | clf.fit(X[col].values.reshape(-1, 1), y)
421 | thresholds = clf.tree_.threshold[clf.tree_.threshold > -2]
422 | self.binning_thresholds[col] = np.sort(thresholds)
423 | except Exception as e:
424 | print(f'Error in {col} during Entropy Binning: {e}')
425 |
426 | return self
427 |
428 | def get_decision_tree(self, max_depth):
429 | if self.modeltype == 'Regression':
430 | return DecisionTreeRegressor(criterion='mse', min_samples_leaf=self.min_samples_leaf,
431 | max_depth=max_depth, random_state=99)
432 | else:
433 | return DecisionTreeClassifier(criterion='entropy', min_samples_leaf=self.min_samples_leaf,
434 | max_depth=max_depth, random_state=99)
435 |
436 | def _adjust_max_depth(self, n_features):
437 | """
438 | Adjust max depth based on the heuristic from the original function.
439 |
440 | Future:
441 | Dynamic Depth Adjustment: Instead of fixed depth adjustments based on the number of variables,
442 | use a more dynamic approach that considers the actual distribution and variability of each variable.
443 |
444 | """
445 | if self.max_depth is not None:
446 | return self.max_depth # Use user-defined max_depth if provided
447 | elif n_features <= 2:
448 | return 2
449 | elif n_features <= 5:
450 | return n_features - 2
451 | elif n_features <= 10:
452 | return 5
453 | else:
454 | return 10 # Default max depth for > 10 features
455 |
456 | def select_top_n_vars(self, X, y, n=2, skew_threshold=1.5, kurtosis_threshold=3):
457 | """
458 | The select_top_n_vars method takes into account the skewness and kurtosis of each variable
459 | in addition to its predictive power. The final score is used to rank the variables
460 | adjusted by the absolute values of skewness and kurtosis to prefer variables that are
461 | both predictive and have less extreme distributions.
462 |
463 | Incorporating distribution-based metrics such as skewness, kurtosis, and outlier
464 | detection into the selection process for variables to bin can indeed provide a more
465 | nuanced approach. Variables with high skewness or kurtosis may benefit more from binning,
466 | as it can help to normalize their effect in predictive models. Additionally, handling
467 | outliers before binning can ensure that bins are more representative of the general
468 | distribution of the data.
469 | """
470 | scores = {}
471 | ### Select only float numeric columns to bin ###########
472 | numvars = X.select_dtypes(include='float').columns.tolist()
473 | ### Let's keep the remaining vars ######
474 | self.remvars = [x for x in list(X) if x not in numvars ]
475 | for col in numvars:
476 | X_col = X[col].values.reshape(-1, 1)
477 |
478 | # Calculate skewness and kurtosis
479 | skewness = skew(X[col])
480 | kurt = kurtosis(X[col], fisher=True) # Fisher's definition is used (normal ==> 0.0)
481 |
482 | # Initialize a simple decision tree
483 | if self.modeltype == 'Regression':
484 | model = DecisionTreeRegressor(max_depth=self.max_depth, random_state=99)
485 | else:
486 | model = DecisionTreeClassifier(max_depth=self.max_depth, random_state=99)
487 |
488 | # Fit the model
489 | model.fit(X_col, y)
490 |
491 | # Predict using the model
492 | predictions = model.predict(X_col)
493 |
494 | # Calculate the performance metric
495 | if self.modeltype == 'Regression':
496 | score = -mean_squared_error(y, predictions) # Negative because lower MSE is better
497 | else:
498 | score = accuracy_score(y, predictions)
499 |
500 | # Store the score, possibly adjusted for skewness and kurtosis
501 | if abs(skewness) > skew_threshold or abs(kurt) > kurtosis_threshold:
502 | scores[col] = score * (1 + abs(skewness) + abs(kurt))
503 | else:
504 | scores[col] = score
505 |
506 | # Sort the variables based on the scores and select the top n
507 | top_n_vars = sorted(scores, key=scores.get, reverse=True)[:max(n, int(X.shape[1] * 0.1), 2)]
508 |
509 | self.remvars += [x for x in numvars if x not in top_n_vars ]
510 | return top_n_vars
511 |
512 | def transform(self, X):
513 | """
514 | Applies the learned binning thresholds and Laplace smoothing to the data, replacing or appending the original
515 | variables with their smoothed, binned versions based on the replace_vars flag.
516 |
517 | Parameters:
518 | - X (pd.DataFrame): The input data to transform.
519 |
520 | Returns:
521 | - X_transformed (pd.DataFrame): The transformed data with variables binned and smoothed according to the learned thresholds.
522 | """
523 | X_transformed = X.copy()
524 | for col, thresholds in self.binning_thresholds.items():
525 | bin_col_name = f'{col}_bin' if not self.replace_vars else col
526 | binned_data = np.digitize(X_transformed[col].values, thresholds)
527 |
528 | # Apply Laplace smoothing to the binned data
529 | smoothed_binned_data = binned_data + 1 # Add-one smoothing
530 |
531 | # Replace or append the smoothed, binned data
532 | X_transformed[bin_col_name] = smoothed_binned_data
533 |
534 | # If replacing, drop the original column if it's not the same as the bin_col_name
535 | if self.replace_vars and bin_col_name != col:
536 | X_transformed.drop(col, axis=1, inplace=True)
537 |
538 | return X_transformed
539 | ################################################################################################
540 |
--------------------------------------------------------------------------------
/autoviml/sulov_method.py:
--------------------------------------------------------------------------------
1 | import random
2 |
3 | import numpy as np
4 | import pandas as pd
5 | import networkx as nx
6 |
7 | np.random.seed(99)
8 | random.seed(42)
9 | ################################################################################
10 | #### The warnings from Sklearn are so annoying that I have to shut it off #######
11 | import warnings
12 | warnings.filterwarnings("ignore")
13 | from sklearn.exceptions import DataConversionWarning
14 | warnings.filterwarnings(action='ignore', category=DataConversionWarning)
15 | def warn(*args, **kwargs):
16 | pass
17 | warnings.warn = warn
18 | ####################################################################################
19 | import copy
20 | from sklearn.feature_selection import mutual_info_regression, mutual_info_classif
21 | from sklearn.feature_selection import SelectKBest
22 | from itertools import combinations
23 | import matplotlib.patches as mpatches
24 | import matplotlib.pyplot as plt
25 | from collections import defaultdict
26 | #################################################################################################
27 | def remove_highly_correlated_vars_fast(df, corr_limit=0.70):
28 | """
29 | This is a simple method to remove highly correlated features fast using Pearson's Correlation.
30 | Use this only for float and integer variables. It will automatically select those only.
31 | It can be used for very large data sets where featurewiz has trouble with memory
32 | """
33 | # Creating correlation matrix
34 | cor_matrix = df.corr().abs().astype(np.float16)
35 | # Selecting upper triangle of correlation matrix
36 | upper_tri = cor_matrix.where(np.triu(np.ones(cor_matrix.shape),
37 | k=1).astype(bool))
38 | # Finding index of feature columns with correlation greater than 0.95
39 | to_drop = [column for column in upper_tri.columns if any(upper_tri[column] > corr_limit)]
40 | print()
41 | print('Highly correlated columns to remove: %s' % to_drop)
42 | return to_drop
43 |
44 |
45 | #####################################################################################
46 | def left_subtract(l1, l2):
47 | lst = []
48 | for i in l1:
49 | if i not in l2:
50 | lst.append(i)
51 | return lst
52 |
53 | #################################################################################
54 | def return_dictionary_list(lst_of_tuples):
55 | """ Returns a dictionary of lists if you send in a list of Tuples"""
56 | orDict = defaultdict(list)
57 | # iterating over list of tuples
58 | for key, val in lst_of_tuples:
59 | orDict[key].append(val)
60 | return orDict
61 |
62 |
63 | ################################################################################
64 | def find_remove_duplicates(list_of_values):
65 | """
66 | # Removes duplicates from a list to return unique values - USED ONLY ONCE
67 | """
68 | output = []
69 | seen = set()
70 | for value in list_of_values:
71 | if value not in seen:
72 | output.append(value)
73 | seen.add(value)
74 | return output
75 |
76 |
77 | ##################################################################################
78 | def FE_remove_variables_using_SULOV_method(df, preds_in, modeltype, target,
79 | corr_limit=0.70, verbose=0, dask_xgboost_flag=False):
80 | """
81 | FE stands for Feature Engineering - it means this function performs feature engineering
82 | ###########################################################################################
83 | ##### SULOV stands for Searching Uncorrelated List Of Variables #############
84 | This highly efficient method removes variables that are highly correlated using a series of
85 | pair-wise correlation knockout rounds. It is extremely fast and hence can work on thousands
86 | of variables in less than a minute, even on a laptop. You need to send in a list of numeric
87 | variables and that's all! The method defines high Correlation as anything over 0.70 (absolute)
88 | but this can be changed. If two variables have absolute correlation higher than this, they
89 | will be marked, and using a process of elimination, one of them will get knocked out:
90 | To decide order of variables to keep, we use mutuail information score to select. MIS returns
91 | a ranked list of these correlated variables: when we select one, we knock out others
92 | that it is correlated to. Then we select next var. This way we knock out correlated variables.
93 | Finally we are left with uncorrelated variables that are also highly important in mutual score.
94 | ######## YOU MUST INCLUDE THE ABOVE MESSAGE IF YOU COPY THIS CODE IN YOUR LIBRARY ##########
95 | """
96 |
97 | df = copy.deepcopy(df)
98 | df_target = df[target]
99 | numvars = df[preds_in].select_dtypes(include='float').columns.tolist()
100 | remvars = left_subtract(preds_in, numvars)
101 | df = df[numvars]
102 | ### for some reason, doing a mass fillna of vars doesn't work! Hence doing it individually!
103 | null_vars = np.array(numvars)[df.isnull().sum() > 0]
104 | for each_num in null_vars:
105 | ### Remember that fillna only works at the dataframe level!
106 | df[[each_num]] = df[[each_num]].fillna(0)
107 | target = copy.deepcopy(target)
108 |
109 | print('#######################################################################################')
110 | print('##### SULOV: Searching for Uncorrelated List Of Variables in %s features ############' % len(numvars))
111 | print('#######################################################################################')
112 | ### This is a shorter version of getting unduplicated and highly correlated vars ##
113 | correlation_dataframe = df.corr().abs().unstack().sort_values().drop_duplicates()
114 | corrdf = pd.DataFrame(correlation_dataframe[:].reset_index())
115 | corrdf.columns = ['var1', 'var2', 'coeff']
116 | corrdf1 = corrdf[corrdf['coeff'] >= corr_limit]
117 | correlated_pair = list(zip(corrdf1['var1'].values.tolist(), corrdf1['var2'].values.tolist()))
118 | corr_pair_dict = dict(return_dictionary_list(correlated_pair))
119 | corr_list = find_remove_duplicates(corrdf1['var1'].values.tolist() + corrdf1['var2'].values.tolist())
120 | keys_in_dict = list(corr_pair_dict.keys())
121 | reverse_correlated_pair = [(y, x) for (x, y) in correlated_pair]
122 | reverse_corr_pair_dict = dict(return_dictionary_list(reverse_correlated_pair))
123 | #### corr_pair_dict is used later to make the network diagram to see which vars are correlated to which
124 | for key, val in reverse_corr_pair_dict.items():
125 | if key in keys_in_dict:
126 | if len(key) > 1:
127 | corr_pair_dict[key] += val
128 | else:
129 | corr_pair_dict[key] = val
130 |
131 | ###### This is for ordering the variables in the highest to lowest importance to target ###
132 | if len(corr_list) == 0:
133 | final_list = list(correlation_dataframe)
134 | print('Selecting all (%d) variables since none of numeric vars are highly correlated...' % len(numvars))
135 | return numvars
136 | else:
137 | if isinstance(target, list):
138 | target = target[0]
139 | max_feats = len(corr_list)
140 | if modeltype == 'Regression':
141 | sel_function = mutual_info_regression
142 | fs = SelectKBest(score_func=sel_function, k=max_feats)
143 | else:
144 | sel_function = mutual_info_classif
145 | fs = SelectKBest(score_func=sel_function, k=max_feats)
146 | ##### you must ensure there are no infinite nor null values in corr_list df ##
147 | df_fit = df[corr_list]
148 | ### Now check if there are any NaN values in the dataset #####
149 |
150 | if df_fit.isnull().sum().sum() > 0:
151 | df_fit = df_fit.dropna()
152 | else:
153 | print(' there are no null values in dataset...')
154 | ##### Reduce memory usage and find mutual information score ####
155 | # try:
156 | # df_fit = reduce_mem_usage(df_fit)
157 | # except:
158 | # print('Reduce memory erroring. Continuing...')
159 | ##### Ready to perform fit and find mutual information score ####
160 | try:
161 | fs.fit(df_fit, df_target)
162 | except:
163 | print(' SelectKBest() function is erroring. Returning with all %s variables...' % len(numvars))
164 | return numvars
165 | try:
166 | #################################################################################
167 | ####### This is the main section where we use mutual info score to select vars
168 | #################################################################################
169 | mutual_info = dict(zip(corr_list, fs.scores_))
170 | #### The first variable in list has the highest correlation to the target variable ###
171 | sorted_by_mutual_info = [key for (key, val) in
172 | sorted(mutual_info.items(), key=lambda kv: kv[1], reverse=True)]
173 | ##### Now we select the final list of correlated variables ###########
174 | selected_corr_list = []
175 | #### You have to make multiple copies of this sorted list since it is iterated many times ####
176 | orig_sorted = copy.deepcopy(sorted_by_mutual_info)
177 | copy_sorted = copy.deepcopy(sorted_by_mutual_info)
178 | copy_pair = copy.deepcopy(corr_pair_dict)
179 | #### select each variable by the highest mutual info and see what vars are correlated to it
180 | for each_corr_name in copy_sorted:
181 | ### add the selected var to the selected_corr_list
182 | selected_corr_list.append(each_corr_name)
183 | for each_remove in copy_pair[each_corr_name]:
184 | #### Now remove each variable that is highly correlated to the selected variable
185 | if each_remove in copy_sorted:
186 | copy_sorted.remove(each_remove)
187 | ##### Now we combine the uncorrelated list to the selected correlated list above
188 | rem_col_list = left_subtract(numvars, corr_list)
189 | final_list = rem_col_list + selected_corr_list
190 | removed_cols = left_subtract(numvars, final_list)
191 | except Exception as e:
192 | print(' SULOV Method crashing due to %s' % e)
193 | #### Dropping highly correlated Features fast using simple linear correlation ###
194 | removed_cols = remove_highly_correlated_vars_fast(df, corr_limit)
195 | final_list = left_subtract(numvars, removed_cols)
196 | if len(removed_cols) > 0:
197 | print(' Removing (%d) highly correlated variables:' % (len(removed_cols)))
198 | if len(removed_cols) <= 30:
199 | print(' %s' % removed_cols)
200 | if len(final_list) <= 30:
201 | print(' Following (%d) vars selected: %s' % (len(final_list), final_list))
202 | ############## D R A W C O R R E L A T I O N N E T W O R K ##################
203 | selected = copy.deepcopy(final_list)
204 | try:
205 | #### Now start building the graph ###################
206 | gf = nx.Graph()
207 | ### the mutual info score gives the size of the bubble ###
208 | multiplier = 2100
209 | for each in orig_sorted:
210 | gf.add_node(each, size=int(max(1, mutual_info[each] * multiplier)))
211 | ######### This is where you calculate the size of each node to draw
212 | sizes = [mutual_info[x] * multiplier for x in list(gf.nodes())]
213 | #### The sizes of the bubbles for each node is determined by its mutual information score value
214 | corr = df_fit.corr()
215 | high_corr = corr[abs(corr) > corr_limit]
216 | ## high_corr is the dataframe of a few variables that are highly correlated to each other
217 | combos = combinations(corr_list, 2)
218 | ### this gives the strength of correlation between 2 nodes ##
219 | multiplier = 20
220 | for (var1, var2) in combos:
221 | if np.isnan(high_corr.loc[var1, var2]):
222 | pass
223 | else:
224 | gf.add_edge(var1, var2, weight=multiplier * high_corr.loc[var1, var2])
225 | ######## Now start building the networkx graph ##########################
226 | widths = nx.get_edge_attributes(gf, 'weight')
227 | nodelist = gf.nodes()
228 | cols = 5
229 | height_size = 5
230 | width_size = 15
231 | rows = int(len(corr_list) / cols)
232 | if rows < 1:
233 | rows = 1
234 | plt.figure(figsize=(width_size, min(20, height_size * rows)))
235 | pos = nx.shell_layout(gf)
236 | nx.draw_networkx_nodes(gf, pos,
237 | nodelist=nodelist,
238 | node_size=sizes,
239 | node_color='blue',
240 | alpha=0.5)
241 | nx.draw_networkx_edges(gf, pos,
242 | edgelist=widths.keys(),
243 | width=list(widths.values()),
244 | edge_color='lightblue',
245 | alpha=0.6)
246 | pos_higher = {}
247 | x_off = 0.04 # offset on the x axis
248 | y_off = 0.04 # offset on the y axis
249 | for k, v in pos.items():
250 | pos_higher[k] = (v[0] + x_off, v[1] + y_off)
251 | if len(selected) == 0:
252 | nx.draw_networkx_labels(gf, pos=pos_higher,
253 | labels=dict(zip(nodelist, nodelist)),
254 | font_color='black')
255 | else:
256 | nx.draw_networkx_labels(gf, pos=pos_higher,
257 | labels=dict(zip(nodelist,
258 | [x + ' (selected)' if x in selected else x + ' (removed)' for x
259 | in nodelist])),
260 | font_color='black')
261 | plt.box(True)
262 | plt.title("""In SULOV, we repeatedly remove features with lower mutual info scores among highly correlated pairs (see figure),
263 | SULOV selects the feature with higher mutual info score related to target when choosing between a pair. """,
264 | fontsize=10)
265 | plt.suptitle('How SULOV Method Works by Removing Highly Correlated Features', fontsize=20, y=1.03)
266 | red_patch = mpatches.Patch(color='blue', label='Bigger circle denotes higher mutual info score with target')
267 | blue_patch = mpatches.Patch(color='lightblue',
268 | label='Thicker line denotes higher correlation between two variables')
269 | plt.legend(handles=[red_patch, blue_patch], loc='best')
270 | plt.show()
271 | ##### N E T W O R K D I A G R A M C O M P L E T E #################
272 | return final_list + remvars
273 | except Exception as e:
274 | print(' Networkx library visualization crashing due to %s' % e)
275 | print('Continuing with SULOV. %d features selected' % len(final_list))
276 | return final_list + remvars
277 | ###################################################################################
278 |
--------------------------------------------------------------------------------
/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/AutoViML/Auto_ViML/0761412417b6dac0683425c372fd7e4f8942d815/logo.png
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | numpy<2
2 | pandas>=2.0
3 | ipython
4 | scipy==1.11.4
5 | matplotlib>3.7.4
6 | beautifulsoup4
7 | emoji
8 | ipython
9 | jupyter
10 | seaborn
11 | catboost>=1.2.7
12 | textblob
13 | nltk
14 | regex
15 | xgboost>=0.82,<=1.7.6
16 | vaderSentiment
17 | imbalanced-learn>=0.10.1
18 | shap>=0.36.0
19 | scikit-learn>=0.24,<=1.5.2
20 | lightgbm>=3.0.0
21 | networkx>=3.1
22 | pandas-dq>=1.29
23 |
--------------------------------------------------------------------------------
/setup.cfg:
--------------------------------------------------------------------------------
1 | [egg_info]
2 | tag_build =
3 | tag_date = 0
4 |
5 |
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | import setuptools
2 |
3 | with open("README.md", "r", encoding="utf-8") as fh:
4 | long_description = fh.read()
5 |
6 | setuptools.setup(
7 | name="autoviml",
8 | version="0.2.02",
9 | author="Ram Seshadri",
10 | # author_email="author@example.com",
11 | description="Automatically Build Variant Interpretable ML models fast - now with CatBoost!",
12 | long_description=long_description,
13 | long_description_content_type="text/markdown",
14 | license='Apache License 2.0',
15 | url="https://github.com/AutoViML/Auto_ViML",
16 | packages=setuptools.find_packages(exclude=("tests",)),
17 | install_requires=[
18 | "numpy<2",
19 | "pandas>=2.0",
20 | "scipy==1.11.4",
21 | "xlrd",
22 | "matplotlib>3.7.4",
23 | "beautifulsoup4",
24 | "emoji",
25 | "ipython",
26 | "jupyter",
27 | "seaborn",
28 | "catboost>=1.2.7",
29 | "textblob",
30 | "nltk",
31 | "regex",
32 | "xgboost>=0.82,<=1.7.6",
33 | "vaderSentiment",
34 | "imbalanced-learn>=0.10.1",
35 | "shap>=0.36.0",
36 | "scikit-learn>=0.24,<=1.5.2",
37 | "lightgbm>=3.0.0",
38 | "networkx>=3.1",
39 | "pandas-dq>=1.29",
40 | ],
41 | classifiers=[
42 | "Programming Language :: Python :: 3",
43 | "Operating System :: OS Independent",
44 | ],
45 | )
46 |
--------------------------------------------------------------------------------
/sulov_xgboost.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/AutoViML/Auto_ViML/0761412417b6dac0683425c372fd7e4f8942d815/sulov_xgboost.png
--------------------------------------------------------------------------------