├── .gitignore ├── 01_the_machine_learning_landscape.pdf ├── 02_end_to_end ├── future_encoders.py └── housing.ipynb ├── 04_training_linear_models.ipynb ├── 05_SVM ├── .gitignore ├── 05_SVM.ipynb └── assets │ ├── .DS_Store │ ├── 1.png │ ├── 2.png │ ├── 3.png │ ├── 5.png │ ├── 6.png │ ├── 7.png │ ├── 8.png │ ├── all.png │ ├── large_margin_classification_plot.png │ ├── npp.png │ ├── output_45_0.png │ ├── sel.png │ └── vapnik.png ├── 08_dimensionality_reduction.ipynb ├── 10_Introduction_to_neural_network.pdf ├── 11_DNN ├── 11_deep_learning.ipynb ├── 11_deep_learning_exercises.ipynb ├── Ha_optimizer.png ├── N_momentum.png ├── dropout.png ├── images │ └── deep │ │ ├── README │ │ ├── elu_plot.png │ │ ├── leaky_relu_plot.png │ │ ├── selu_plot.png │ │ └── sigmoid_saturation_plot.png ├── learning_curve.png ├── mnist.png ├── momentum.png ├── optimizer.png ├── selu.png ├── sig_01.jpeg ├── sig_02.jpeg ├── sig_03.jpeg ├── sig_04.jpeg ├── softmax.png ├── tensorflow_graph_in_jupyter.py ├── transfer.png └── unsuperv.png ├── 12_distributed ├── 12_distributed_tensorflow.ipynb ├── fig12-10.png ├── fig12-11.png ├── fig12-12.png ├── fig12-13.png ├── fig12-14.png ├── fig12-15.png ├── fig12-16.png ├── fig12-17.png ├── fig12-18.png ├── fig12-5.png ├── fig12-6.png ├── fig12-7.png ├── fig12-8.png ├── fig12-9.png ├── multigpu_cnn.ipynb └── multigpu_cnn_cluster.ipynb ├── 13_CNN.ipynb ├── 13_cnn_architecture.pdf ├── 14_RNN ├── 14_recurrent_neural_networks.ipynb ├── images │ ├── ann │ │ ├── README │ │ ├── activation_functions_plot.png │ │ └── perceptron_iris_plot.png │ ├── autoencoders │ │ ├── README │ │ ├── extracted_features_plot.png │ │ ├── generated_digits_plot.png │ │ ├── linear_autoencoder_pca_plot.png │ │ ├── reconstruction_plot.png │ │ └── sparsity_loss_plot.png │ ├── classification │ │ ├── README │ │ ├── cleaned_digit_example_plot.png │ │ ├── confusion_matrix_errors_plot.png │ │ ├── confusion_matrix_plot.png │ │ ├── error_analysis_digits_plot.png │ │ ├── more_digits_plot.png │ │ ├── noisy_digit_example_plot.png │ │ ├── precision_recall_vs_threshold_plot.png │ │ ├── precision_vs_recall_plot.png │ │ ├── roc_curve_comparison_plot.png │ │ ├── roc_curve_plot.png │ │ └── some_digit_plot.png │ ├── cnn │ │ ├── README │ │ ├── china_horizontal.png │ │ ├── china_max_pool.png │ │ ├── china_original.png │ │ ├── china_original_small.png │ │ ├── china_vertical.png │ │ └── test_image.png │ ├── decision_trees │ │ ├── README │ │ ├── decision_tree_decision_boundaries_plot.png │ │ ├── decision_tree_instability_plot.png │ │ ├── iris_tree.dot │ │ ├── iris_tree.png │ │ ├── min_samples_leaf_plot.png │ │ ├── regression_tree.dot │ │ ├── regression_tree.png │ │ ├── sensitivity_to_rotation_plot.png │ │ ├── tree_regression_plot.png │ │ └── tree_regression_regularization_plot.png │ ├── deep │ │ ├── README │ │ ├── elu_plot.png │ │ ├── leaky_relu_plot.png │ │ ├── selu_plot.png │ │ └── sigmoid_saturation_plot.png │ ├── dim_reduction │ │ ├── README │ │ ├── dataset_2d_plot.png │ │ ├── dataset_3d_plot.png │ │ ├── kernel_pca_plot.png │ │ ├── lle_unrolling_plot.png │ │ ├── manifold_decision_boundary_plot1.png │ │ ├── manifold_decision_boundary_plot2.png │ │ ├── manifold_decision_boundary_plot3.png │ │ ├── manifold_decision_boundary_plot4.png │ │ ├── mnist_compression_plot.png │ │ ├── other_dim_reduction_plot.png │ │ ├── pca_best_projection.png │ │ ├── preimage_plot.png │ │ ├── squished_swiss_roll_plot.png │ │ └── swiss_roll_plot.png │ ├── distributed │ │ └── README │ ├── end_to_end_project │ │ ├── README │ │ ├── attribute_histogram_plots.png │ │ ├── bad_visualization_plot.png │ │ ├── better_visualization_plot.png │ │ ├── california.png │ │ ├── california_housing_prices_plot.png │ │ ├── housing_prices_scatterplot.png │ │ ├── income_category_hist.png │ │ ├── income_vs_house_value_scatterplot.png │ │ └── scatter_matrix_plot.png │ ├── ensembles │ │ ├── README │ │ ├── boosting_plot.png │ │ ├── decision_tree_without_and_with_bagging_plot.png │ │ ├── early_stopping_gbrt_plot.png │ │ ├── gbrt_learning_rate_plot.png │ │ ├── gradient_boosting_plot.png │ │ ├── law_of_large_numbers_plot.png │ │ └── mnist_feature_importance_plot.png │ ├── fundamentals │ │ ├── README │ │ ├── best_fit_model_plot.png │ │ ├── cyprus_prediction_plot.png │ │ ├── money_happy_scatterplot.png │ │ ├── overfitting_model_plot.png │ │ ├── representative_training_data_scatterplot.png │ │ ├── ridge_model_plot.png │ │ └── tweaking_model_params_plot.png │ ├── rl │ │ ├── MsPacman.png │ │ ├── README │ │ ├── cart_pole_plot.png │ │ └── preprocessing_plot.png │ ├── rnn │ │ ├── README │ │ ├── creative_sequence_plot.png │ │ ├── time_series_plot.png │ │ └── time_series_pred_plot.png │ ├── svm │ │ ├── README │ │ ├── higher_dimensions_plot.png │ │ ├── hinge_plot.png │ │ ├── iris_3D_plot.png │ │ ├── kernel_method_plot.png │ │ ├── large_margin_classification_plot.png │ │ ├── moons_kernelized_polynomial_svc_plot.png │ │ ├── moons_polynomial_svc_plot.png │ │ ├── moons_rbf_svc_plot.png │ │ ├── regularization_plot.png │ │ ├── sensitivity_to_feature_scales_plot.png │ │ ├── sensitivity_to_outliers_plot.png │ │ ├── small_w_large_margin_plot.png │ │ ├── svm_regression_plot.png │ │ └── svm_with_polynomial_kernel_plot.png │ ├── tensorflow │ │ └── README │ └── training_linear_models │ │ ├── README │ │ ├── early_stopping_plot.png │ │ ├── generated_data_plot.png │ │ ├── gradient_descent_paths_plot.png │ │ ├── gradient_descent_plot.png │ │ ├── high_degree_polynomials_plot.png │ │ ├── lasso_regression_plot.png │ │ ├── lasso_vs_ridge_plot.png │ │ ├── learning_curves_plot.png │ │ ├── linear_model_predictions.png │ │ ├── logistic_function_plot.png │ │ ├── logistic_regression_contour_plot.png │ │ ├── logistic_regression_plot.png │ │ ├── quadratic_data_plot.png │ │ ├── quadratic_predictions_plot.png │ │ ├── ridge_regression_plot.png │ │ ├── sgd_plot.png │ │ ├── softmax_regression_contour_plot.png │ │ └── underfitting_learning_curves_plot.png └── tensorflow_graph_in_jupyter.py ├── 6 결정트리.pptx └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Created by https://www.gitignore.io/api/macos,python,windows,jupyternotebook 3 | 4 | ### JupyterNotebook ### 5 | .ipynb_checkpoints 6 | */.ipynb_checkpoints/* 7 | 8 | # Remove previous ipynb_checkpoints 9 | # git rm -r .ipynb_checkpoints/ 10 | # 11 | 12 | ### macOS ### 13 | # General 14 | .DS_Store 15 | .AppleDouble 16 | .LSOverride 17 | 18 | # Icon must end with two \r 19 | Icon 20 | 21 | # Thumbnails 22 | ._* 23 | 24 | # Files that might appear in the root of a volume 25 | .DocumentRevisions-V100 26 | .fseventsd 27 | .Spotlight-V100 28 | .TemporaryItems 29 | .Trashes 30 | .VolumeIcon.icns 31 | .com.apple.timemachine.donotpresent 32 | 33 | # Directories potentially created on remote AFP share 34 | .AppleDB 35 | .AppleDesktop 36 | Network Trash Folder 37 | Temporary Items 38 | .apdisk 39 | 40 | ### Python ### 41 | # Byte-compiled / optimized / DLL files 42 | __pycache__/ 43 | *.py[cod] 44 | *$py.class 45 | 46 | # C extensions 47 | *.so 48 | 49 | # Distribution / packaging 50 | .Python 51 | build/ 52 | develop-eggs/ 53 | dist/ 54 | downloads/ 55 | eggs/ 56 | .eggs/ 57 | lib/ 58 | lib64/ 59 | parts/ 60 | sdist/ 61 | var/ 62 | wheels/ 63 | *.egg-info/ 64 | .installed.cfg 65 | *.egg 66 | MANIFEST 67 | 68 | # PyInstaller 69 | # Usually these files are written by a python script from a template 70 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 71 | *.manifest 72 | *.spec 73 | 74 | # Installer logs 75 | pip-log.txt 76 | pip-delete-this-directory.txt 77 | 78 | # Unit test / coverage reports 79 | htmlcov/ 80 | .tox/ 81 | .coverage 82 | .coverage.* 83 | .cache 84 | nosetests.xml 85 | coverage.xml 86 | *.cover 87 | .hypothesis/ 88 | .pytest_cache/ 89 | 90 | # Translations 91 | *.mo 92 | *.pot 93 | 94 | # Django stuff: 95 | *.log 96 | local_settings.py 97 | db.sqlite3 98 | 99 | # Flask stuff: 100 | instance/ 101 | .webassets-cache 102 | 103 | # Scrapy stuff: 104 | .scrapy 105 | 106 | # Sphinx documentation 107 | docs/_build/ 108 | 109 | # PyBuilder 110 | target/ 111 | 112 | # Jupyter Notebook 113 | 114 | # pyenv 115 | .python-version 116 | 117 | # celery beat schedule file 118 | celerybeat-schedule 119 | 120 | # SageMath parsed files 121 | *.sage.py 122 | 123 | # Environments 124 | .env 125 | .venv 126 | env/ 127 | venv/ 128 | ENV/ 129 | env.bak/ 130 | venv.bak/ 131 | 132 | # Spyder project settings 133 | .spyderproject 134 | .spyproject 135 | 136 | # Rope project settings 137 | .ropeproject 138 | 139 | # mkdocs documentation 140 | /site 141 | 142 | # mypy 143 | .mypy_cache/ 144 | 145 | ### Windows ### 146 | # Windows thumbnail cache files 147 | Thumbs.db 148 | ehthumbs.db 149 | ehthumbs_vista.db 150 | 151 | # Dump file 152 | *.stackdump 153 | 154 | # Folder config file 155 | [Dd]esktop.ini 156 | 157 | # Recycle Bin used on file shares 158 | $RECYCLE.BIN/ 159 | 160 | # Windows Installer files 161 | *.cab 162 | *.msi 163 | *.msix 164 | *.msm 165 | *.msp 166 | 167 | # Windows shortcuts 168 | *.lnk 169 | 170 | 171 | # End of https://www.gitignore.io/api/macos,python,windows,jupyternotebook 172 | 173 | # Addition Rules 174 | *.tgz 175 | *-ubyte 176 | -------------------------------------------------------------------------------- /01_the_machine_learning_landscape.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/01_the_machine_learning_landscape.pdf -------------------------------------------------------------------------------- /02_end_to_end/future_encoders.py: -------------------------------------------------------------------------------- 1 | # Authors: Andreas Mueller 2 | # Joris Van den Bossche 3 | # License: BSD 3 clause 4 | 5 | from __future__ import division 6 | 7 | import numbers 8 | import warnings 9 | 10 | import numpy as np 11 | from scipy import sparse 12 | 13 | from sklearn.base import BaseEstimator, TransformerMixin 14 | from sklearn.externals import six 15 | from sklearn.utils import check_array 16 | from sklearn.utils.validation import check_is_fitted, FLOAT_DTYPES 17 | from sklearn.preprocessing.label import LabelEncoder 18 | 19 | 20 | BOUNDS_THRESHOLD = 1e-7 21 | 22 | 23 | zip = six.moves.zip 24 | map = six.moves.map 25 | range = six.moves.range 26 | 27 | __all__ = [ 28 | 'OneHotEncoder', 29 | 'OrdinalEncoder' 30 | ] 31 | 32 | 33 | def _argmax(arr_or_spmatrix, axis=None): 34 | return arr_or_spmatrix.argmax(axis=axis) 35 | 36 | 37 | def _handle_zeros_in_scale(scale, copy=True): 38 | ''' Makes sure that whenever scale is zero, we handle it correctly. 39 | 40 | This happens in most scalers when we have constant features.''' 41 | 42 | # if we are fitting on 1D arrays, scale might be a scalar 43 | if np.isscalar(scale): 44 | if scale == .0: 45 | scale = 1. 46 | return scale 47 | elif isinstance(scale, np.ndarray): 48 | if copy: 49 | # New array to avoid side-effects 50 | scale = scale.copy() 51 | scale[scale == 0.0] = 1.0 52 | return scale 53 | 54 | 55 | def _transform_selected(X, transform, selected="all", copy=True): 56 | """Apply a transform function to portion of selected features 57 | 58 | Parameters 59 | ---------- 60 | X : {array-like, sparse matrix}, shape [n_samples, n_features] 61 | Dense array or sparse matrix. 62 | 63 | transform : callable 64 | A callable transform(X) -> X_transformed 65 | 66 | copy : boolean, optional 67 | Copy X even if it could be avoided. 68 | 69 | selected: "all" or array of indices or mask 70 | Specify which features to apply the transform to. 71 | 72 | Returns 73 | ------- 74 | X : array or sparse matrix, shape=(n_samples, n_features_new) 75 | """ 76 | X = check_array(X, accept_sparse='csc', copy=copy, dtype=FLOAT_DTYPES) 77 | 78 | if isinstance(selected, six.string_types) and selected == "all": 79 | return transform(X) 80 | 81 | if len(selected) == 0: 82 | return X 83 | 84 | n_features = X.shape[1] 85 | ind = np.arange(n_features) 86 | sel = np.zeros(n_features, dtype=bool) 87 | sel[np.asarray(selected)] = True 88 | not_sel = np.logical_not(sel) 89 | n_selected = np.sum(sel) 90 | 91 | if n_selected == 0: 92 | # No features selected. 93 | return X 94 | elif n_selected == n_features: 95 | # All features selected. 96 | return transform(X) 97 | else: 98 | X_sel = transform(X[:, ind[sel]]) 99 | X_not_sel = X[:, ind[not_sel]] 100 | 101 | if sparse.issparse(X_sel) or sparse.issparse(X_not_sel): 102 | return sparse.hstack((X_sel, X_not_sel)) 103 | else: 104 | return np.hstack((X_sel, X_not_sel)) 105 | 106 | 107 | class _BaseEncoder(BaseEstimator, TransformerMixin): 108 | """ 109 | Base class for encoders that includes the code to categorize and 110 | transform the input features. 111 | 112 | """ 113 | 114 | def _fit(self, X, handle_unknown='error'): 115 | 116 | X_temp = check_array(X, dtype=None) 117 | if not hasattr(X, 'dtype') and np.issubdtype(X_temp.dtype, np.str_): 118 | X = check_array(X, dtype=np.object) 119 | else: 120 | X = X_temp 121 | 122 | n_samples, n_features = X.shape 123 | 124 | if self.categories != 'auto': 125 | for cats in self.categories: 126 | if not np.all(np.sort(cats) == np.array(cats)): 127 | raise ValueError("Unsorted categories are not yet " 128 | "supported") 129 | if len(self.categories) != n_features: 130 | raise ValueError("Shape mismatch: if n_values is an array," 131 | " it has to be of shape (n_features,).") 132 | 133 | self._label_encoders_ = [LabelEncoder() for _ in range(n_features)] 134 | 135 | for i in range(n_features): 136 | le = self._label_encoders_[i] 137 | Xi = X[:, i] 138 | if self.categories == 'auto': 139 | le.fit(Xi) 140 | else: 141 | if handle_unknown == 'error': 142 | valid_mask = np.in1d(Xi, self.categories[i]) 143 | if not np.all(valid_mask): 144 | diff = np.unique(Xi[~valid_mask]) 145 | msg = ("Found unknown categories {0} in column {1}" 146 | " during fit".format(diff, i)) 147 | raise ValueError(msg) 148 | le.classes_ = np.array(self.categories[i]) 149 | 150 | self.categories_ = [le.classes_ for le in self._label_encoders_] 151 | 152 | def _transform(self, X, handle_unknown='error'): 153 | 154 | X_temp = check_array(X, dtype=None) 155 | if not hasattr(X, 'dtype') and np.issubdtype(X_temp.dtype, np.str_): 156 | X = check_array(X, dtype=np.object) 157 | else: 158 | X = X_temp 159 | 160 | _, n_features = X.shape 161 | X_int = np.zeros_like(X, dtype=np.int) 162 | X_mask = np.ones_like(X, dtype=np.bool) 163 | 164 | for i in range(n_features): 165 | Xi = X[:, i] 166 | valid_mask = np.in1d(Xi, self.categories_[i]) 167 | 168 | if not np.all(valid_mask): 169 | if handle_unknown == 'error': 170 | diff = np.unique(X[~valid_mask, i]) 171 | msg = ("Found unknown categories {0} in column {1}" 172 | " during transform".format(diff, i)) 173 | raise ValueError(msg) 174 | else: 175 | # Set the problematic rows to an acceptable value and 176 | # continue `The rows are marked `X_mask` and will be 177 | # removed later. 178 | X_mask[:, i] = valid_mask 179 | Xi = Xi.copy() 180 | Xi[~valid_mask] = self.categories_[i][0] 181 | X_int[:, i] = self._label_encoders_[i].transform(Xi) 182 | 183 | return X_int, X_mask 184 | 185 | 186 | WARNING_MSG = ( 187 | "The handling of integer data will change in the future. Currently, the " 188 | "categories are determined based on the range [0, max(values)], while " 189 | "in the future they will be determined based on the unique values.\n" 190 | "If you want the future behaviour, you can specify \"categories='auto'\"." 191 | ) 192 | 193 | 194 | class OneHotEncoder(_BaseEncoder): 195 | """Encode categorical integer features as a one-hot numeric array. 196 | 197 | The input to this transformer should be an array-like of integers or 198 | strings, denoting the values taken on by categorical (discrete) features. 199 | The features are encoded using a one-hot (aka 'one-of-K' or 'dummy') 200 | encoding scheme. This creates a binary column for each category and 201 | returns a sparse matrix or dense array. 202 | 203 | By default, the encoder derives the categories based on the unique values 204 | in each feature. Alternatively, you can also specify the `categories` 205 | manually. 206 | The OneHotEncoder previously assumed that the input features take on 207 | values in the range [0, max(values)). This behaviour is deprecated. 208 | 209 | This encoding is needed for feeding categorical data to many scikit-learn 210 | estimators, notably linear models and SVMs with the standard kernels. 211 | 212 | Note: a one-hot encoding of y labels should use a LabelBinarizer 213 | instead. 214 | 215 | Read more in the :ref:`User Guide `. 216 | 217 | Parameters 218 | ---------- 219 | categories : 'auto' or a list of lists/arrays of values. 220 | Categories (unique values) per feature: 221 | 222 | - 'auto' : Determine categories automatically from the training data. 223 | - list : ``categories[i]`` holds the categories expected in the ith 224 | column. The passed categories must be sorted and should not mix 225 | strings and numeric values. 226 | 227 | The used categories can be found in the ``categories_`` attribute. 228 | 229 | sparse : boolean, default=True 230 | Will return sparse matrix if set True else will return an array. 231 | 232 | dtype : number type, default=np.float 233 | Desired dtype of output. 234 | 235 | handle_unknown : 'error' (default) or 'ignore' 236 | Whether to raise an error or ignore if a unknown categorical feature is 237 | present during transform (default is to raise). When this parameter 238 | is set to 'ignore' and an unknown category is encountered during 239 | transform, the resulting one-hot encoded columns for this feature 240 | will be all zeros. In the inverse transform, an unknown category 241 | will be denoted as None. 242 | 243 | n_values : 'auto', int or array of ints 244 | Number of values per feature. 245 | 246 | - 'auto' : determine value range from training data. 247 | - int : number of categorical values per feature. 248 | Each feature value should be in ``range(n_values)`` 249 | - array : ``n_values[i]`` is the number of categorical values in 250 | ``X[:, i]``. Each feature value should be 251 | in ``range(n_values[i])`` 252 | 253 | .. deprecated:: 0.20 254 | The `n_values` keyword is deprecated and will be removed in 0.22. 255 | Use `categories` instead. 256 | 257 | categorical_features : "all" or array of indices or mask 258 | Specify what features are treated as categorical. 259 | 260 | - 'all' (default): All features are treated as categorical. 261 | - array of indices: Array of categorical feature indices. 262 | - mask: Array of length n_features and with dtype=bool. 263 | 264 | Non-categorical features are always stacked to the right of the matrix. 265 | 266 | .. deprecated:: 0.20 267 | The `categorical_features` keyword is deprecated and will be 268 | removed in 0.22. 269 | 270 | Attributes 271 | ---------- 272 | categories_ : list of arrays 273 | The categories of each feature determined during fitting 274 | (in order corresponding with output of ``transform``). 275 | 276 | active_features_ : array 277 | Indices for active features, meaning values that actually occur 278 | in the training set. Only available when n_values is ``'auto'``. 279 | 280 | .. deprecated:: 0.20 281 | 282 | feature_indices_ : array of shape (n_features,) 283 | Indices to feature ranges. 284 | Feature ``i`` in the original data is mapped to features 285 | from ``feature_indices_[i]`` to ``feature_indices_[i+1]`` 286 | (and then potentially masked by `active_features_` afterwards) 287 | 288 | .. deprecated:: 0.20 289 | 290 | n_values_ : array of shape (n_features,) 291 | Maximum number of values per feature. 292 | 293 | .. deprecated:: 0.20 294 | 295 | Examples 296 | -------- 297 | Given a dataset with two features, we let the encoder find the unique 298 | values per feature and transform the data to a binary one-hot encoding. 299 | 300 | >>> from sklearn.preprocessing import OneHotEncoder 301 | >>> enc = OneHotEncoder(handle_unknown='ignore') 302 | >>> X = [['Male', 1], ['Female', 3], ['Female', 2]] 303 | >>> enc.fit(X) 304 | ... # doctest: +ELLIPSIS 305 | OneHotEncoder(categories='auto', dtype=<... 'numpy.float64'>, 306 | handle_unknown='ignore', sparse=True) 307 | 308 | >>> enc.categories_ 309 | [array(['Female', 'Male'], dtype=object), array([1, 2, 3], dtype=object)] 310 | >>> enc.transform([['Female', 1], ['Male', 4]]).toarray() 311 | array([[ 1., 0., 1., 0., 0.], 312 | [ 0., 1., 0., 0., 0.]]) 313 | >>> enc.inverse_transform([[0, 1, 1, 0, 0], [0, 0, 0, 1, 0]]) 314 | array([['Male', 1], 315 | [None, 2]], dtype=object) 316 | 317 | See also 318 | -------- 319 | sklearn.preprocessing.OrdinalEncoder : performs an ordinal (integer) 320 | encoding of the categorical features. 321 | sklearn.feature_extraction.DictVectorizer : performs a one-hot encoding of 322 | dictionary items (also handles string-valued features). 323 | sklearn.feature_extraction.FeatureHasher : performs an approximate one-hot 324 | encoding of dictionary items or strings. 325 | sklearn.preprocessing.LabelBinarizer : binarizes labels in a one-vs-all 326 | fashion. 327 | sklearn.preprocessing.MultiLabelBinarizer : transforms between iterable of 328 | iterables and a multilabel format, e.g. a (samples x classes) binary 329 | matrix indicating the presence of a class label. 330 | """ 331 | 332 | def __init__(self, n_values=None, categorical_features=None, 333 | categories=None, sparse=True, dtype=np.float64, 334 | handle_unknown='error'): 335 | self._categories = categories 336 | if categories is None: 337 | self.categories = 'auto' 338 | else: 339 | self.categories = categories 340 | self.sparse = sparse 341 | self.dtype = dtype 342 | self.handle_unknown = handle_unknown 343 | 344 | if n_values is not None: 345 | pass 346 | # warnings.warn("Deprecated", DeprecationWarning) 347 | else: 348 | n_values = "auto" 349 | self._deprecated_n_values = n_values 350 | 351 | if categorical_features is not None: 352 | pass 353 | # warnings.warn("Deprecated", DeprecationWarning) 354 | else: 355 | categorical_features = "all" 356 | self._deprecated_categorical_features = categorical_features 357 | 358 | # Deprecated keywords 359 | 360 | @property 361 | def n_values(self): 362 | warnings.warn("The 'n_values' parameter is deprecated.", 363 | DeprecationWarning) 364 | return self._deprecated_n_values 365 | 366 | @n_values.setter 367 | def n_values(self, value): 368 | warnings.warn("The 'n_values' parameter is deprecated.", 369 | DeprecationWarning) 370 | self._deprecated_n_values = value 371 | 372 | @property 373 | def categorical_features(self): 374 | warnings.warn("The 'categorical_features' parameter is deprecated.", 375 | DeprecationWarning) 376 | return self._deprecated_categorical_features 377 | 378 | @categorical_features.setter 379 | def categorical_features(self, value): 380 | warnings.warn("The 'categorical_features' parameter is deprecated.", 381 | DeprecationWarning) 382 | self._deprecated_categorical_features = value 383 | 384 | # Deprecated attributes 385 | 386 | @property 387 | def active_features_(self): 388 | check_is_fitted(self, 'categories_') 389 | warnings.warn("The 'active_features_' attribute is deprecated.", 390 | DeprecationWarning) 391 | return self._active_features_ 392 | 393 | @property 394 | def feature_indices_(self): 395 | check_is_fitted(self, 'categories_') 396 | warnings.warn("The 'feature_indices_' attribute is deprecated.", 397 | DeprecationWarning) 398 | return self._feature_indices_ 399 | 400 | @property 401 | def n_values_(self): 402 | check_is_fitted(self, 'categories_') 403 | warnings.warn("The 'n_values_' attribute is deprecated.", 404 | DeprecationWarning) 405 | return self._n_values_ 406 | 407 | def _handle_deprecations(self, X): 408 | 409 | user_set_categories = False 410 | 411 | if self._categories is not None: 412 | self._legacy_mode = False 413 | user_set_categories = True 414 | 415 | elif self._deprecated_n_values != 'auto': 416 | msg = ( 417 | "Passing 'n_values' is deprecated and will be removed in a " 418 | "future release. You can use the 'categories' keyword instead." 419 | " 'n_values=n' corresponds to 'n_values=[range(n)]'.") 420 | warnings.warn(msg, DeprecationWarning) 421 | 422 | # we internally translate this to the correct categories 423 | # and don't use legacy mode 424 | X = check_array(X, dtype=np.int) 425 | 426 | if isinstance(self._deprecated_n_values, numbers.Integral): 427 | n_features = X.shape[1] 428 | self.categories = [ 429 | list(range(self._deprecated_n_values)) 430 | for _ in range(n_features)] 431 | n_values = np.empty(n_features, dtype=np.int) 432 | n_values.fill(self._deprecated_n_values) 433 | else: 434 | try: 435 | n_values = np.asarray(self._deprecated_n_values, dtype=int) 436 | self.categories = [list(range(i)) 437 | for i in self._deprecated_n_values] 438 | except (ValueError, TypeError): 439 | raise TypeError( 440 | "Wrong type for parameter `n_values`. Expected 'auto'," 441 | " int or array of ints, got %r".format(type(X))) 442 | 443 | self._n_values_ = n_values 444 | n_values = np.hstack([[0], n_values]) 445 | indices = np.cumsum(n_values) 446 | self._feature_indices_ = indices 447 | 448 | self._legacy_mode = False 449 | 450 | else: # n_values = 'auto' 451 | if self.handle_unknown == 'ignore': 452 | # no change in behaviour, no need to raise deprecation warning 453 | self._legacy_mode = False 454 | else: 455 | 456 | # check if we have integer or categorical input 457 | try: 458 | X = check_array(X, dtype=np.int) 459 | except ValueError: 460 | self._legacy_mode = False 461 | else: 462 | warnings.warn(WARNING_MSG, DeprecationWarning) 463 | self._legacy_mode = True 464 | 465 | if (not isinstance(self._deprecated_categorical_features, 466 | six.string_types) 467 | or (isinstance(self._deprecated_categorical_features, 468 | six.string_types) 469 | and self._deprecated_categorical_features != 'all')): 470 | if user_set_categories: 471 | raise ValueError( 472 | "The 'categorical_features' keyword is deprecated, and " 473 | "cannot be used together with specifying 'categories'.") 474 | warnings.warn("The 'categorical_features' keyword is deprecated.", 475 | DeprecationWarning) 476 | self._legacy_mode = True 477 | 478 | def fit(self, X, y=None): 479 | """Fit OneHotEncoder to X. 480 | 481 | Parameters 482 | ---------- 483 | X : array-like, shape [n_samples, n_feature] 484 | The data to determine the categories of each feature. 485 | 486 | Returns 487 | ------- 488 | self 489 | """ 490 | if self.handle_unknown not in ['error', 'ignore']: 491 | template = ("handle_unknown should be either 'error' or " 492 | "'ignore', got %s") 493 | raise ValueError(template % self.handle_unknown) 494 | 495 | self._handle_deprecations(X) 496 | 497 | if self._legacy_mode: 498 | # TODO not with _transform_selected ?? 499 | self._legacy_fit_transform(X) 500 | return self 501 | else: 502 | self._fit(X, handle_unknown=self.handle_unknown) 503 | return self 504 | 505 | def _legacy_fit_transform(self, X): 506 | """Assumes X contains only categorical features.""" 507 | self_n_values = self._deprecated_n_values 508 | dtype = getattr(X, 'dtype', None) 509 | X = check_array(X, dtype=np.int) 510 | if np.any(X < 0): 511 | raise ValueError("X needs to contain only non-negative integers.") 512 | n_samples, n_features = X.shape 513 | if (isinstance(self_n_values, six.string_types) and 514 | self_n_values == 'auto'): 515 | n_values = np.max(X, axis=0) + 1 516 | elif isinstance(self_n_values, numbers.Integral): 517 | if (np.max(X, axis=0) >= self_n_values).any(): 518 | raise ValueError("Feature out of bounds for n_values=%d" 519 | % self_n_values) 520 | n_values = np.empty(n_features, dtype=np.int) 521 | n_values.fill(self_n_values) 522 | else: 523 | try: 524 | n_values = np.asarray(self_n_values, dtype=int) 525 | except (ValueError, TypeError): 526 | raise TypeError("Wrong type for parameter `n_values`. Expected" 527 | " 'auto', int or array of ints, got %r" 528 | % type(X)) 529 | if n_values.ndim < 1 or n_values.shape[0] != X.shape[1]: 530 | raise ValueError("Shape mismatch: if n_values is an array," 531 | " it has to be of shape (n_features,).") 532 | 533 | self._n_values_ = n_values 534 | self.categories_ = [np.arange(n_val - 1, dtype=dtype) 535 | for n_val in n_values] 536 | n_values = np.hstack([[0], n_values]) 537 | indices = np.cumsum(n_values) 538 | self._feature_indices_ = indices 539 | 540 | column_indices = (X + indices[:-1]).ravel() 541 | row_indices = np.repeat(np.arange(n_samples, dtype=np.int32), 542 | n_features) 543 | data = np.ones(n_samples * n_features) 544 | out = sparse.coo_matrix((data, (row_indices, column_indices)), 545 | shape=(n_samples, indices[-1]), 546 | dtype=self.dtype).tocsr() 547 | 548 | if (isinstance(self_n_values, six.string_types) and 549 | self_n_values == 'auto'): 550 | mask = np.array(out.sum(axis=0)).ravel() != 0 551 | active_features = np.where(mask)[0] 552 | out = out[:, active_features] 553 | self._active_features_ = active_features 554 | 555 | self.categories_ = [ 556 | np.unique(X[:, i]).astype(dtype) if dtype else np.unique(X[:, i]) 557 | for i in range(n_features)] 558 | #import pdb; pdb.set_trace() 559 | 560 | return out if self.sparse else out.toarray() 561 | 562 | def fit_transform(self, X, y=None): 563 | """Fit OneHotEncoder to X, then transform X. 564 | 565 | Equivalent to self.fit(X).transform(X), but more convenient and more 566 | efficient. See fit for the parameters, transform for the return value. 567 | 568 | Parameters 569 | ---------- 570 | X : array-like, shape [n_samples, n_feature] 571 | Input array of type int. 572 | """ 573 | if self.handle_unknown not in ['error', 'ignore']: 574 | template = ("handle_unknown should be either 'error' or " 575 | "'ignore', got %s") 576 | raise ValueError(template % self.handle_unknown) 577 | 578 | self._handle_deprecations(X) 579 | 580 | if self._legacy_mode: 581 | return _transform_selected(X, self._legacy_fit_transform, 582 | self._deprecated_categorical_features, 583 | copy=True) 584 | else: 585 | return self.fit(X).transform(X) 586 | 587 | def _legacy_transform(self, X): 588 | """Assumes X contains only categorical features.""" 589 | self_n_values = self._deprecated_n_values 590 | X = check_array(X, dtype=np.int) 591 | if np.any(X < 0): 592 | raise ValueError("X needs to contain only non-negative integers.") 593 | n_samples, n_features = X.shape 594 | 595 | indices = self._feature_indices_ 596 | if n_features != indices.shape[0] - 1: 597 | raise ValueError("X has different shape than during fitting." 598 | " Expected %d, got %d." 599 | % (indices.shape[0] - 1, n_features)) 600 | 601 | # We use only those categorical features of X that are known using fit. 602 | # i.e lesser than n_values_ using mask. 603 | # This means, if self.handle_unknown is "ignore", the row_indices and 604 | # col_indices corresponding to the unknown categorical feature are 605 | # ignored. 606 | mask = (X < self._n_values_).ravel() 607 | if np.any(~mask): 608 | if self.handle_unknown not in ['error', 'ignore']: 609 | raise ValueError("handle_unknown should be either error or " 610 | "unknown got %s" % self.handle_unknown) 611 | if self.handle_unknown == 'error': 612 | raise ValueError("unknown categorical feature present %s " 613 | "during transform." % X.ravel()[~mask]) 614 | 615 | column_indices = (X + indices[:-1]).ravel()[mask] 616 | row_indices = np.repeat(np.arange(n_samples, dtype=np.int32), 617 | n_features)[mask] 618 | data = np.ones(np.sum(mask)) 619 | out = sparse.coo_matrix((data, (row_indices, column_indices)), 620 | shape=(n_samples, indices[-1]), 621 | dtype=self.dtype).tocsr() 622 | if (isinstance(self_n_values, six.string_types) and 623 | self_n_values == 'auto'): 624 | out = out[:, self._active_features_] 625 | 626 | return out if self.sparse else out.toarray() 627 | 628 | def _transform_new(self, X): 629 | """New implementation assuming categorical input""" 630 | X_temp = check_array(X, dtype=None) 631 | if not hasattr(X, 'dtype') and np.issubdtype(X_temp.dtype, np.str_): 632 | X = check_array(X, dtype=np.object) 633 | else: 634 | X = X_temp 635 | 636 | n_samples, n_features = X.shape 637 | 638 | X_int, X_mask = self._transform(X, handle_unknown=self.handle_unknown) 639 | 640 | mask = X_mask.ravel() 641 | n_values = [cats.shape[0] for cats in self.categories_] 642 | n_values = np.array([0] + n_values) 643 | feature_indices = np.cumsum(n_values) 644 | 645 | indices = (X_int + feature_indices[:-1]).ravel()[mask] 646 | indptr = X_mask.sum(axis=1).cumsum() 647 | indptr = np.insert(indptr, 0, 0) 648 | data = np.ones(n_samples * n_features)[mask] 649 | 650 | out = sparse.csr_matrix((data, indices, indptr), 651 | shape=(n_samples, feature_indices[-1]), 652 | dtype=self.dtype) 653 | if not self.sparse: 654 | return out.toarray() 655 | else: 656 | return out 657 | 658 | def transform(self, X): 659 | """Transform X using one-hot encoding. 660 | 661 | Parameters 662 | ---------- 663 | X : array-like, shape [n_samples, n_features] 664 | The data to encode. 665 | 666 | Returns 667 | ------- 668 | X_out : sparse matrix if sparse=True else a 2-d array 669 | Transformed input. 670 | """ 671 | if not self._legacy_mode: 672 | return self._transform_new(X) 673 | else: 674 | return _transform_selected(X, self._legacy_transform, 675 | self._deprecated_categorical_features, 676 | copy=True) 677 | 678 | def inverse_transform(self, X): 679 | """Convert back the data to the original representation. 680 | 681 | In case unknown categories are encountered (all zero's in the 682 | one-hot encoding), ``None`` is used to represent this category. 683 | 684 | Parameters 685 | ---------- 686 | X : array-like or sparse matrix, shape [n_samples, n_encoded_features] 687 | The transformed data. 688 | 689 | Returns 690 | ------- 691 | X_tr : array-like, shape [n_samples, n_features] 692 | Inverse transformed array. 693 | 694 | """ 695 | # if self._legacy_mode: 696 | # raise ValueError("only supported for categorical features") 697 | 698 | check_is_fitted(self, 'categories_') 699 | X = check_array(X, accept_sparse='csr') 700 | 701 | n_samples, _ = X.shape 702 | n_features = len(self.categories_) 703 | n_transformed_features = sum([len(cats) for cats in self.categories_]) 704 | 705 | # validate shape of passed X 706 | msg = ("Shape of the passed X data is not correct. Expected {0} " 707 | "columns, got {1}.") 708 | if X.shape[1] != n_transformed_features: 709 | raise ValueError(msg.format(n_transformed_features, X.shape[1])) 710 | 711 | # create resulting array of appropriate dtype 712 | dt = np.find_common_type([cat.dtype for cat in self.categories_], []) 713 | X_tr = np.empty((n_samples, n_features), dtype=dt) 714 | 715 | j = 0 716 | found_unknown = {} 717 | 718 | for i in range(n_features): 719 | n_categories = len(self.categories_[i]) 720 | sub = X[:, j:j + n_categories] 721 | 722 | # for sparse X argmax returns 2D matrix, ensure 1D array 723 | labels = np.asarray(_argmax(sub, axis=1)).flatten() 724 | X_tr[:, i] = self.categories_[i][labels] 725 | 726 | if self.handle_unknown == 'ignore': 727 | # ignored unknown categories: we have a row of all zero's 728 | unknown = np.asarray(sub.sum(axis=1) == 0).flatten() 729 | if unknown.any(): 730 | found_unknown[i] = unknown 731 | 732 | j += n_categories 733 | 734 | # if ignored are found: potentially need to upcast result to 735 | # insert None values 736 | if found_unknown: 737 | if X_tr.dtype != object: 738 | X_tr = X_tr.astype(object) 739 | 740 | for idx, mask in found_unknown.items(): 741 | X_tr[mask, idx] = None 742 | 743 | return X_tr 744 | 745 | 746 | class OrdinalEncoder(_BaseEncoder): 747 | """Encode categorical features as an integer array. 748 | 749 | The input to this transformer should be an array-like of integers or 750 | strings, denoting the values taken on by categorical (discrete) features. 751 | The features are converted to ordinal integers. This results in 752 | a single column of integers (0 to n_categories - 1) per feature. 753 | 754 | Read more in the :ref:`User Guide `. 755 | 756 | Parameters 757 | ---------- 758 | categories : 'auto' or a list of lists/arrays of values. 759 | Categories (unique values) per feature: 760 | 761 | - 'auto' : Determine categories automatically from the training data. 762 | - list : ``categories[i]`` holds the categories expected in the ith 763 | column. The passed categories must be sorted and should not mix 764 | strings and numeric values. 765 | 766 | The used categories can be found in the ``categories_`` attribute. 767 | 768 | dtype : number type, default np.float64 769 | Desired dtype of output. 770 | 771 | Attributes 772 | ---------- 773 | categories_ : list of arrays 774 | The categories of each feature determined during fitting 775 | (in order corresponding with output of ``transform``). 776 | 777 | Examples 778 | -------- 779 | Given a dataset with two features, we let the encoder find the unique 780 | values per feature and transform the data to a binary one-hot encoding. 781 | 782 | >>> from sklearn.preprocessing import OrdinalEncoder 783 | >>> enc = OrdinalEncoder() 784 | >>> X = [['Male', 1], ['Female', 3], ['Female', 2]] 785 | >>> enc.fit(X) 786 | ... # doctest: +ELLIPSIS 787 | OrdinalEncoder(categories='auto', dtype=<... 'numpy.float64'>) 788 | >>> enc.categories_ 789 | [array(['Female', 'Male'], dtype=object), array([1, 2, 3], dtype=object)] 790 | >>> enc.transform([['Female', 3], ['Male', 1]]) 791 | array([[ 0., 2.], 792 | [ 1., 0.]]) 793 | 794 | >>> enc.inverse_transform([[1, 0], [0, 1]]) 795 | array([['Male', 1], 796 | ['Female', 2]], dtype=object) 797 | 798 | See also 799 | -------- 800 | sklearn.preprocessing.OneHotEncoder : performs a one-hot encoding of 801 | categorical features. 802 | sklearn.preprocessing.LabelEncoder : encodes target labels with values 803 | between 0 and n_classes-1. 804 | sklearn.feature_extraction.DictVectorizer : performs a one-hot encoding of 805 | dictionary items (also handles string-valued features). 806 | sklearn.feature_extraction.FeatureHasher : performs an approximate one-hot 807 | encoding of dictionary items or strings. 808 | """ 809 | 810 | def __init__(self, categories='auto', dtype=np.float64): 811 | self.categories = categories 812 | self.dtype = dtype 813 | 814 | def fit(self, X, y=None): 815 | """Fit the OrdinalEncoder to X. 816 | 817 | Parameters 818 | ---------- 819 | X : array-like, shape [n_samples, n_features] 820 | The data to determine the categories of each feature. 821 | 822 | Returns 823 | ------- 824 | self 825 | 826 | """ 827 | self._fit(X) 828 | 829 | return self 830 | 831 | def transform(self, X): 832 | """Transform X to ordinal codes. 833 | 834 | Parameters 835 | ---------- 836 | X : array-like, shape [n_samples, n_features] 837 | The data to encode. 838 | 839 | Returns 840 | ------- 841 | X_out : sparse matrix or a 2-d array 842 | Transformed input. 843 | 844 | """ 845 | X_int, _ = self._transform(X) 846 | return X_int.astype(self.dtype, copy=False) 847 | 848 | def inverse_transform(self, X): 849 | """Convert back the data to the original representation. 850 | 851 | Parameters 852 | ---------- 853 | X : array-like or sparse matrix, shape [n_samples, n_encoded_features] 854 | The transformed data. 855 | 856 | Returns 857 | ------- 858 | X_tr : array-like, shape [n_samples, n_features] 859 | Inverse transformed array. 860 | 861 | """ 862 | check_is_fitted(self, 'categories_') 863 | X = check_array(X, accept_sparse='csr') 864 | 865 | n_samples, _ = X.shape 866 | n_features = len(self.categories_) 867 | 868 | # validate shape of passed X 869 | msg = ("Shape of the passed X data is not correct. Expected {0} " 870 | "columns, got {1}.") 871 | if X.shape[1] != n_features: 872 | raise ValueError(msg.format(n_features, X.shape[1])) 873 | 874 | # create resulting array of appropriate dtype 875 | dt = np.find_common_type([cat.dtype for cat in self.categories_], []) 876 | X_tr = np.empty((n_samples, n_features), dtype=dt) 877 | 878 | for i in range(n_features): 879 | labels = X[:, i].astype('int64') 880 | X_tr[:, i] = self.categories_[i][labels] 881 | 882 | return X_tr 883 | -------------------------------------------------------------------------------- /05_SVM/.gitignore: -------------------------------------------------------------------------------- 1 | 2 | # Created by https://www.gitignore.io/api/macos,python,windows,jupyternotebook 3 | 4 | ### JupyterNotebook ### 5 | .ipynb_checkpoints 6 | */.ipynb_checkpoints/* 7 | 8 | # Remove previous ipynb_checkpoints 9 | # git rm -r .ipynb_checkpoints/ 10 | # 11 | 12 | ### macOS ### 13 | # General 14 | .DS_Store 15 | .AppleDouble 16 | .LSOverride 17 | 18 | # Icon must end with two \r 19 | Icon 20 | 21 | # Thumbnails 22 | ._* 23 | 24 | # Files that might appear in the root of a volume 25 | .DocumentRevisions-V100 26 | .fseventsd 27 | .Spotlight-V100 28 | .TemporaryItems 29 | .Trashes 30 | .VolumeIcon.icns 31 | .com.apple.timemachine.donotpresent 32 | 33 | # Directories potentially created on remote AFP share 34 | .AppleDB 35 | .AppleDesktop 36 | Network Trash Folder 37 | Temporary Items 38 | .apdisk 39 | 40 | ### Python ### 41 | # Byte-compiled / optimized / DLL files 42 | __pycache__/ 43 | *.py[cod] 44 | *$py.class 45 | 46 | # C extensions 47 | *.so 48 | 49 | # Distribution / packaging 50 | .Python 51 | build/ 52 | develop-eggs/ 53 | dist/ 54 | downloads/ 55 | eggs/ 56 | .eggs/ 57 | lib/ 58 | lib64/ 59 | parts/ 60 | sdist/ 61 | var/ 62 | wheels/ 63 | *.egg-info/ 64 | .installed.cfg 65 | *.egg 66 | MANIFEST 67 | 68 | # PyInstaller 69 | # Usually these files are written by a python script from a template 70 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 71 | *.manifest 72 | *.spec 73 | 74 | # Installer logs 75 | pip-log.txt 76 | pip-delete-this-directory.txt 77 | 78 | # Unit test / coverage reports 79 | htmlcov/ 80 | .tox/ 81 | .coverage 82 | .coverage.* 83 | .cache 84 | nosetests.xml 85 | coverage.xml 86 | *.cover 87 | .hypothesis/ 88 | .pytest_cache/ 89 | 90 | # Translations 91 | *.mo 92 | *.pot 93 | 94 | # Django stuff: 95 | *.log 96 | local_settings.py 97 | db.sqlite3 98 | 99 | # Flask stuff: 100 | instance/ 101 | .webassets-cache 102 | 103 | # Scrapy stuff: 104 | .scrapy 105 | 106 | # Sphinx documentation 107 | docs/_build/ 108 | 109 | # PyBuilder 110 | target/ 111 | 112 | # Jupyter Notebook 113 | 114 | # pyenv 115 | .python-version 116 | 117 | # celery beat schedule file 118 | celerybeat-schedule 119 | 120 | # SageMath parsed files 121 | *.sage.py 122 | 123 | # Environments 124 | .env 125 | .venv 126 | env/ 127 | venv/ 128 | ENV/ 129 | env.bak/ 130 | venv.bak/ 131 | 132 | # Spyder project settings 133 | .spyderproject 134 | .spyproject 135 | 136 | # Rope project settings 137 | .ropeproject 138 | 139 | # mkdocs documentation 140 | /site 141 | 142 | # mypy 143 | .mypy_cache/ 144 | 145 | ### Windows ### 146 | # Windows thumbnail cache files 147 | Thumbs.db 148 | ehthumbs.db 149 | ehthumbs_vista.db 150 | 151 | # Dump file 152 | *.stackdump 153 | 154 | # Folder config file 155 | [Dd]esktop.ini 156 | 157 | # Recycle Bin used on file shares 158 | $RECYCLE.BIN/ 159 | 160 | # Windows Installer files 161 | *.cab 162 | *.msi 163 | *.msix 164 | *.msm 165 | *.msp 166 | 167 | # Windows shortcuts 168 | *.lnk 169 | 170 | 171 | # End of https://www.gitignore.io/api/macos,python,windows,jupyternotebook 172 | 173 | # Addition Rules 174 | *.tgz 175 | *-ubyte 176 | -------------------------------------------------------------------------------- /05_SVM/assets/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/.DS_Store -------------------------------------------------------------------------------- /05_SVM/assets/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/1.png -------------------------------------------------------------------------------- /05_SVM/assets/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/2.png -------------------------------------------------------------------------------- /05_SVM/assets/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/3.png -------------------------------------------------------------------------------- /05_SVM/assets/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/5.png -------------------------------------------------------------------------------- /05_SVM/assets/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/6.png -------------------------------------------------------------------------------- /05_SVM/assets/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/7.png -------------------------------------------------------------------------------- /05_SVM/assets/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/8.png -------------------------------------------------------------------------------- /05_SVM/assets/all.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/all.png -------------------------------------------------------------------------------- /05_SVM/assets/large_margin_classification_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/large_margin_classification_plot.png -------------------------------------------------------------------------------- /05_SVM/assets/npp.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/npp.png -------------------------------------------------------------------------------- /05_SVM/assets/output_45_0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/output_45_0.png -------------------------------------------------------------------------------- /05_SVM/assets/sel.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/sel.png -------------------------------------------------------------------------------- /05_SVM/assets/vapnik.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/05_SVM/assets/vapnik.png -------------------------------------------------------------------------------- /10_Introduction_to_neural_network.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/10_Introduction_to_neural_network.pdf -------------------------------------------------------------------------------- /11_DNN/Ha_optimizer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/Ha_optimizer.png -------------------------------------------------------------------------------- /11_DNN/N_momentum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/N_momentum.png -------------------------------------------------------------------------------- /11_DNN/dropout.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/dropout.png -------------------------------------------------------------------------------- /11_DNN/images/deep/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /11_DNN/images/deep/elu_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/images/deep/elu_plot.png -------------------------------------------------------------------------------- /11_DNN/images/deep/leaky_relu_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/images/deep/leaky_relu_plot.png -------------------------------------------------------------------------------- /11_DNN/images/deep/selu_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/images/deep/selu_plot.png -------------------------------------------------------------------------------- /11_DNN/images/deep/sigmoid_saturation_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/images/deep/sigmoid_saturation_plot.png -------------------------------------------------------------------------------- /11_DNN/learning_curve.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/learning_curve.png -------------------------------------------------------------------------------- /11_DNN/mnist.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/mnist.png -------------------------------------------------------------------------------- /11_DNN/momentum.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/momentum.png -------------------------------------------------------------------------------- /11_DNN/optimizer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/optimizer.png -------------------------------------------------------------------------------- /11_DNN/selu.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/selu.png -------------------------------------------------------------------------------- /11_DNN/sig_01.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/sig_01.jpeg -------------------------------------------------------------------------------- /11_DNN/sig_02.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/sig_02.jpeg -------------------------------------------------------------------------------- /11_DNN/sig_03.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/sig_03.jpeg -------------------------------------------------------------------------------- /11_DNN/sig_04.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/sig_04.jpeg -------------------------------------------------------------------------------- /11_DNN/softmax.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/softmax.png -------------------------------------------------------------------------------- /11_DNN/tensorflow_graph_in_jupyter.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import, division, print_function, unicode_literals 2 | 3 | # This module defines the show_graph() function to visualize a TensorFlow graph within Jupyter. 4 | 5 | # As far as I can tell, this code was originally written by Alex Mordvintsev at: 6 | # https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb 7 | 8 | # The original code only worked on Chrome (because of the use of , but the version below 9 | # uses Polyfill (copied from this StackOverflow answer: https://stackoverflow.com/a/41463991/38626) 10 | # so that it can work on other browsers as well. 11 | 12 | import numpy as np 13 | import tensorflow as tf 14 | from IPython.display import clear_output, Image, display, HTML 15 | 16 | def strip_consts(graph_def, max_const_size=32): 17 | """Strip large constant values from graph_def.""" 18 | strip_def = tf.GraphDef() 19 | for n0 in graph_def.node: 20 | n = strip_def.node.add() 21 | n.MergeFrom(n0) 22 | if n.op == 'Const': 23 | tensor = n.attr['value'].tensor 24 | size = len(tensor.tensor_content) 25 | if size > max_const_size: 26 | tensor.tensor_content = b""%size 27 | return strip_def 28 | 29 | def show_graph(graph_def, max_const_size=32): 30 | """Visualize TensorFlow graph.""" 31 | if hasattr(graph_def, 'as_graph_def'): 32 | graph_def = graph_def.as_graph_def() 33 | strip_def = strip_consts(graph_def, max_const_size=max_const_size) 34 | code = """ 35 | 36 | 41 | 42 |
43 | 44 |
45 | """.format(data=repr(str(strip_def)), id='graph'+str(np.random.rand())) 46 | 47 | iframe = """ 48 | 49 | """.format(code.replace('"', '"')) 50 | display(HTML(iframe)) 51 | -------------------------------------------------------------------------------- /11_DNN/transfer.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/transfer.png -------------------------------------------------------------------------------- /11_DNN/unsuperv.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/11_DNN/unsuperv.png -------------------------------------------------------------------------------- /12_distributed/12_distributed_tensorflow.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "#%load_ext watermark\n", 10 | "#%watermark -v -p numpy,sklearn,scipy,matplotlib,tensorflow" 11 | ] 12 | }, 13 | { 14 | "cell_type": "markdown", 15 | "metadata": {}, 16 | "source": [ 17 | "***12장 – 분산 텐서플로***\n" 18 | ] 19 | }, 20 | { 21 | "cell_type": "markdown", 22 | "metadata": {}, 23 | "source": [ 24 | "목적\n", 25 | "- 대규모 NN 을 여러 장치에 병렬 / 분산 실행하여 수행 시간을 절약한다.\n", 26 | "\n", 27 | "Tensorflow 분산 처리의 장점\n", 28 | "- 계산 그래프의 여러 장치 / 머신 분할방법 제어 가능\n", 29 | "- 다양한 방식의 연산 병렬화 및 동기화 가능\n", 30 | "\n", 31 | "병렬 수행의 실사례\n", 32 | "- 신경망 병렬 수행\n", 33 | "- 모델 세밀 튜닝을 위해 큰 하이퍼파라미터 공간을 탐색\n", 34 | "- 대규모 NN 의 앙상블 수행" 35 | ] 36 | }, 37 | { 38 | "cell_type": "markdown", 39 | "metadata": {}, 40 | "source": [ 41 | "**12.1 단일 머신의 다중 장치**\n", 42 | "\n", 43 | "- 단일 머신에 GPU 추가\n", 44 | "- 다중 머신의 경우 네트워크 비용으로 인해 더 비효율적일 수 있음" 45 | ] 46 | }, 47 | { 48 | "cell_type": "markdown", 49 | "metadata": {}, 50 | "source": [ 51 | "*12.1.1 설치*\n", 52 | "\n", 53 | "- 그래픽 카드 호환성 확인 : https://developer.nvidia.com/cuda-gpus\n", 54 | "- CUDA : gpu 를 통한 명시적 computing 을 가능하게 하는 library\n", 55 | "- cuDNN : DNN 을 위한 기초적 GPU 가속 library\n", 56 | " - activation function, normalize, forward / back propagation, pooling 등\n", 57 | "- CUDA, cuDNN 바이너리 다운로드시 nvidia 개발자 계정이 필요 (https://developer.nvidia.com/)\n", 58 | "\n", 59 | "\n", 60 | "* 본 실습에서는 gcp 를 활용" 61 | ] 62 | }, 63 | { 64 | "cell_type": "code", 65 | "execution_count": 2, 66 | "metadata": {}, 67 | "outputs": [ 68 | { 69 | "name": "stdout", 70 | "output_type": "stream", 71 | "text": [ 72 | "Tue Jul 31 07:48:45 2018 \r\n", 73 | "+-----------------------------------------------------------------------------+\r\n", 74 | "| NVIDIA-SMI 396.37 Driver Version: 396.37 |\r\n", 75 | "|-------------------------------+----------------------+----------------------+\r\n", 76 | "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\r\n", 77 | "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\r\n", 78 | "|===============================+======================+======================|\r\n", 79 | "| 0 Tesla K80 Off | 00000000:00:04.0 Off | 0 |\r\n", 80 | "| N/A 73C P0 88W / 149W | 11MiB / 11441MiB | 0% Default |\r\n", 81 | "+-------------------------------+----------------------+----------------------+\r\n", 82 | "| 1 Tesla K80 Off | 00000000:00:05.0 Off | 0 |\r\n", 83 | "| N/A 74C P0 87W / 149W | 11MiB / 11441MiB | 9% Default |\r\n", 84 | "+-------------------------------+----------------------+----------------------+\r\n", 85 | " \r\n", 86 | "+-----------------------------------------------------------------------------+\r\n", 87 | "| Processes: GPU Memory |\r\n", 88 | "| GPU PID Type Process name Usage |\r\n", 89 | "|=============================================================================|\r\n", 90 | "| No running processes found |\r\n", 91 | "+-----------------------------------------------------------------------------+\r\n" 92 | ] 93 | } 94 | ], 95 | "source": [ 96 | "!nvidia-smi" 97 | ] 98 | }, 99 | { 100 | "cell_type": "markdown", 101 | "metadata": {}, 102 | "source": [ 103 | "- pyenv, anaconda 등 가상 환경을 사용하고 잇다면 적절한 환경으로 활성화" 104 | ] 105 | }, 106 | { 107 | "cell_type": "markdown", 108 | "metadata": {}, 109 | "source": [ 110 | "# 설정" 111 | ] 112 | }, 113 | { 114 | "cell_type": "markdown", 115 | "metadata": {}, 116 | "source": [ 117 | "파이썬 2와 3을 모두 지원합니다. 공통 모듈을 임포트하고 맷플롯립 그림이 노트북 안에 포함되도록 설정하고 생성한 그림을 저장하기 위한 함수를 준비합니다:" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": 3, 123 | "metadata": {}, 124 | "outputs": [], 125 | "source": [ 126 | "# 파이썬 2와 파이썬 3 지원\n", 127 | "from __future__ import division, print_function, unicode_literals\n", 128 | "\n", 129 | "# 공통\n", 130 | "import numpy as np\n", 131 | "import os\n", 132 | "\n", 133 | "# 일관된 출력을 위해 유사난수 초기화\n", 134 | "def reset_graph(seed=42):\n", 135 | " tf.reset_default_graph()\n", 136 | " tf.set_random_seed(seed)\n", 137 | " np.random.seed(seed)\n", 138 | "\n", 139 | "# 맷플롯립 설정\n", 140 | "%matplotlib inline\n", 141 | "import matplotlib\n", 142 | "import matplotlib.pyplot as plt\n", 143 | "plt.rcParams['axes.labelsize'] = 14\n", 144 | "plt.rcParams['xtick.labelsize'] = 12\n", 145 | "plt.rcParams['ytick.labelsize'] = 12\n", 146 | "\n", 147 | "# 그림을 저장할 폴더\n", 148 | "PROJECT_ROOT_DIR = \".\"\n", 149 | "CHAPTER_ID = \"distributed\"\n", 150 | "\n", 151 | "def save_fig(fig_id, tight_layout=True):\n", 152 | " path = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID, fig_id + \".png\")\n", 153 | " if tight_layout:\n", 154 | " plt.tight_layout()\n", 155 | " plt.savefig(path, format='png', dpi=300)" 156 | ] 157 | }, 158 | { 159 | "cell_type": "markdown", 160 | "metadata": {}, 161 | "source": [ 162 | "# 로컬 서버" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 4, 168 | "metadata": {}, 169 | "outputs": [ 170 | { 171 | "name": "stderr", 172 | "output_type": "stream", 173 | "text": [ 174 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 175 | " return f(*args, **kwds)\n", 176 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 177 | " return f(*args, **kwds)\n", 178 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 179 | " return f(*args, **kwds)\n" 180 | ] 181 | } 182 | ], 183 | "source": [ 184 | "import tensorflow as tf" 185 | ] 186 | }, 187 | { 188 | "cell_type": "code", 189 | "execution_count": 5, 190 | "metadata": {}, 191 | "outputs": [], 192 | "source": [ 193 | "c = tf.constant(\"Hello distributed TensorFlow!\")\n", 194 | "server = tf.train.Server.create_local_server()" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": 6, 200 | "metadata": {}, 201 | "outputs": [ 202 | { 203 | "name": "stdout", 204 | "output_type": "stream", 205 | "text": [ 206 | "b'Hello distributed TensorFlow!'\n" 207 | ] 208 | } 209 | ], 210 | "source": [ 211 | "with tf.Session(server.target) as sess:\n", 212 | " print(sess.run(c))" 213 | ] 214 | }, 215 | { 216 | "cell_type": "markdown", 217 | "metadata": {}, 218 | "source": [ 219 | "- gpu 초기화 및 정보 로그가 나온다면 성공\n", 220 | "\n", 221 | "[I 12:02:07.853 NotebookApp] Adapting to protocol v5.1 for kernel 11e846df-9d53-4d1a-8be6-0622be66099f\n", 222 | "2018-07-25 12:02:43.034875: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions tha\n", 223 | "t this TensorFlow binary was not compiled to use: AVX2 FMA\n", 224 | "2018-07-25 12:02:43.140845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read f\n", 225 | "rom SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 226 | "2018-07-25 12:02:43.141396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties\n", 227 | ": \n", 228 | "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n", 229 | "pciBusID: 0000:00:04.0\n", 230 | "totalMemory: 11.17GiB freeMemory: 11.10GiB\n", 231 | "2018-07-25 12:02:43.222979: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read f\n", 232 | "rom SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n", 233 | "2018-07-25 12:02:43.223631: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 1 with properties\n", 234 | ": \n", 235 | "name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235\n", 236 | "pciBusID: 0000:00:05.0\n", 237 | "totalMemory: 11.17GiB freeMemory: 11.10GiB\n", 238 | "2018-07-25 12:02:43.223833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0,\n", 239 | " 1\n", 240 | "2018-07-25 12:02:43.774320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecu\n", 241 | "tor with strength 1 edge matrix:\n", 242 | "2018-07-25 12:02:43.774378: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0 1 \n", 243 | "2018-07-25 12:02:43.774403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N Y \n", 244 | "2018-07-25 12:02:43.774408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 1: Y N \n", 245 | "2018-07-25 12:02:43.774839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/jo\n", 246 | "b:local/replica:0/task:0/device:GPU:0 with 10761 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id\n", 247 | ": 0000:00:04.0, compute capability: 3.7)\n", 248 | "2018-07-25 12:02:43.957038: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/jo\n", 249 | "b:local/replica:0/task:0/device:GPU:1 with 10761 MB memory) -> physical GPU (device: 1, name: Tesla K80, pci bus id\n", 250 | ": 0000:00:05.0, compute capability: 3.7)\n", 251 | "2018-07-25 12:02:44.140339: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCa\n", 252 | "che for job local -> {0 -> localhost:33667}\n", 253 | "2018-07-25 12:02:44.141491: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:334] Started server with t\n", 254 | "arget: grpc://localhost:33667\n", 255 | "2018-07-25 12:02:44.152483: I tensorflow/core/distributed_runtime/master_session.cc:1150] Start master session 6e81\n", 256 | "0a0c2f63410d with config: " 257 | ] 258 | }, 259 | { 260 | "cell_type": "markdown", 261 | "metadata": {}, 262 | "source": [ 263 | "*12.1.2 GPU RAM 관리*\n", 264 | "\n", 265 | "- 단일 GPU 에 여러 프로그램 수행시 경우에 따라 OOM 발생할 수 있음" 266 | ] 267 | }, 268 | { 269 | "cell_type": "markdown", 270 | "metadata": {}, 271 | "source": [ 272 | "1. 별도 GPU 에 각각 수행" 273 | ] 274 | }, 275 | { 276 | "cell_type": "code", 277 | "execution_count": 7, 278 | "metadata": {}, 279 | "outputs": [], 280 | "source": [ 281 | "#CUDA_VISIBLE_DEVICES=0,1 python3 code.py - 복수 device ID 지정시 multi gpu 로 구동되는지 확인\n", 282 | "#CUDA_VISIBLE_DEVICES=\"\" 의 경우 CPU 로 수행됨\n", 283 | "\n", 284 | "# https://stackoverflow.com/questions/37893755/tensorflow-set-cuda-visible-devices-within-jupyter\n", 285 | "# http://dongjinlee.tistory.com/entry/%EC%84%A0%ED%83%9D%ED%95%9C-GPU%EC%97%90%EB%A7%8C-%EB%A9%94%EB%AA%A8%EB%A6%AC-%ED%95%A0%EB%8B%B9%ED%95%98%EB%8A%94-%EB%B0%A9%EB%B2%95\n", 286 | "#import os\n", 287 | "#os.environ[\"CUDA_DEVICE_ORDER\"]=\"PCI_BUS_ID\" # see issue #152\n", 288 | "#os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"\n", 289 | "# do some tensor" 290 | ] 291 | }, 292 | { 293 | "cell_type": "markdown", 294 | "metadata": {}, 295 | "source": [ 296 | "2. GPU 메모리 일부만을 사용하도록 강제" 297 | ] 298 | }, 299 | { 300 | "cell_type": "code", 301 | "execution_count": 8, 302 | "metadata": {}, 303 | "outputs": [], 304 | "source": [ 305 | "config = tf.ConfigProto()\n", 306 | "config.gpu_options.per_process_gpu_memory_fraction = 0.4\n", 307 | "session = tf.Session(config=config)" 308 | ] 309 | }, 310 | { 311 | "cell_type": "code", 312 | "execution_count": 9, 313 | "metadata": {}, 314 | "outputs": [ 315 | { 316 | "name": "stdout", 317 | "output_type": "stream", 318 | "text": [ 319 | "Tue Jul 31 07:48:48 2018 \r\n", 320 | "+-----------------------------------------------------------------------------+\r\n", 321 | "| NVIDIA-SMI 396.37 Driver Version: 396.37 |\r\n", 322 | "|-------------------------------+----------------------+----------------------+\r\n", 323 | "| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\r\n", 324 | "| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\r\n", 325 | "|===============================+======================+======================|\r\n", 326 | "| 0 Tesla K80 Off | 00000000:00:04.0 Off | 0 |\r\n", 327 | "| N/A 73C P0 80W / 149W | 10876MiB / 11441MiB | 0% Default |\r\n", 328 | "+-------------------------------+----------------------+----------------------+\r\n", 329 | "| 1 Tesla K80 Off | 00000000:00:05.0 Off | 0 |\r\n", 330 | "| N/A 74C P0 76W / 149W | 10876MiB / 11441MiB | 0% Default |\r\n", 331 | "+-------------------------------+----------------------+----------------------+\r\n", 332 | " \r\n", 333 | "+-----------------------------------------------------------------------------+\r\n", 334 | "| Processes: GPU Memory |\r\n", 335 | "| GPU PID Type Process name Usage |\r\n", 336 | "|=============================================================================|\r\n", 337 | "| 0 3741 C /usr/bin/python 10863MiB |\r\n", 338 | "| 1 3741 C /usr/bin/python 10863MiB |\r\n", 339 | "+-----------------------------------------------------------------------------+\r\n" 340 | ] 341 | } 342 | ], 343 | "source": [ 344 | "!nvidia-smi" 345 | ] 346 | }, 347 | { 348 | "cell_type": "markdown", 349 | "metadata": {}, 350 | "source": [ 351 | "3. 필요할 때 메모리를 확장 할당하는 옵션 활용\n", 352 | "- 비활용 메모리 반납에 시간이 소요되므로(memory fragmentation 을 피하기 위해) 효용성이 낮음" 353 | ] 354 | }, 355 | { 356 | "cell_type": "code", 357 | "execution_count": 10, 358 | "metadata": {}, 359 | "outputs": [], 360 | "source": [ 361 | "config = tf.ConfigProto();\n", 362 | "config.gpu_options.allow_growth = True\n", 363 | "session = tf.Session(config=config)" 364 | ] 365 | }, 366 | { 367 | "cell_type": "markdown", 368 | "metadata": {}, 369 | "source": [ 370 | "*12.1.3 장치에 연산 배치하기*\n", 371 | "\n", 372 | "dynamic placer\n", 373 | "- 사용자 지정 배치 규칙에 비해 좋은 효율성을 보이고 있지 않다고 함.\n", 374 | "- G사 내부용으로 현재 오픈소스화 되어 잇지 않음." 375 | ] 376 | }, 377 | { 378 | "cell_type": "markdown", 379 | "metadata": {}, 380 | "source": [ 381 | "1) simple placer\n", 382 | "* computation graph 수행시 device 에 배치되지 않는 node 를 평가할 때 사용\n", 383 | "\n", 384 | "\n", 385 | "분배 규칙\n", 386 | "- node 가 이미 배치되어 있는 장비는 해당 node를 그대로 둠\n", 387 | "- 사용자가 node 를 어떤 장치에 할당햇다면 placer 가 node 를 배치\n", 388 | "- GPU #0 이 기본, GPU 가 소진되면 CPU 로 전환" 389 | ] 390 | }, 391 | { 392 | "cell_type": "markdown", 393 | "metadata": {}, 394 | "source": [ 395 | "2) 배치 로깅\n", 396 | "\n", 397 | "- placer 가 node 를 배치할 시점에 메세지를 기록하는 옵션" 398 | ] 399 | }, 400 | { 401 | "cell_type": "code", 402 | "execution_count": 11, 403 | "metadata": {}, 404 | "outputs": [], 405 | "source": [ 406 | "with tf.device(\"/cpu:0\"):\n", 407 | " a = tf.Variable(3.0)\n", 408 | " b = tf.constant(4.0)\n", 409 | "\n", 410 | "c = a * b\n", 411 | "\n", 412 | "config = tf.ConfigProto()\n", 413 | "config.log_device_placement = True\n", 414 | "sess = tf.Session(config=config)" 415 | ] 416 | }, 417 | { 418 | "cell_type": "code", 419 | "execution_count": 12, 420 | "metadata": {}, 421 | "outputs": [], 422 | "source": [ 423 | "a.initializer.run(session=sess)" 424 | ] 425 | }, 426 | { 427 | "cell_type": "markdown", 428 | "metadata": {}, 429 | "source": [ 430 | "Variable: (VariableV2): /job:localhost/replica:0/task:0/device:CPU:\n", 431 | "0\n", 432 | "2018-07-30 21:38:01.248833: I tensorflow/core/common_runtime/placer.cc:886] Variable: (VariableV2)/job:l\n", 433 | "ocalhost/replica:0/task:0/device:CPU:0\n", 434 | "Variable/Assign: (Assign): /job:localhost/replica:0/task:0/device:CPU:0\n", 435 | "2018-07-30 21:38:01.248857: I tensorflow/core/common_runtime/placer.cc:886] Variable/Assign: (Assign)/jo\n", 436 | "b:localhost/replica:0/task:0/device:CPU:0\n", 437 | "Variable/read: (Identity): /job:localhost/replica:0/task:0/device:CPU:0\n", 438 | "2018-07-30 21:38:01.248876: I tensorflow/core/common_runtime/placer.cc:886] Variable/read: (Identity)/jo\n", 439 | "b:localhost/replica:0/task:0/device:CPU:0\n", 440 | "mul: (Mul): /job:localhost/replica:0/task:0/device:GPU:0\n", 441 | "2018-07-30 21:38:01.248892: I tensorflow/core/common_runtime/placer.cc:886] mul: (Mul)/job:localhost/rep\n", 442 | "lica:0/task:0/device:GPU:0\n", 443 | "Const: (Const): /job:localhost/replica:0/task:0/device:CPU:0\n", 444 | "2018-07-30 21:38:01.248903: I tensorflow/core/common_runtime/placer.cc:886] Const: (Const)/job:localhost\n", 445 | "/replica:0/task:0/device:CPU:0\n", 446 | "Variable/initial_value: (Const): /job:localhost/replica:0/task:0/device:CPU:0\n", 447 | "2018-07-30 21:38:01.248915: I tensorflow/core/common_runtime/placer.cc:886] Variable/initial_value: (Con\n", 448 | "st)/job:localhost/replica:0/task:0/device:CPU:0\n", 449 | "Const_1: (Const): /job:localhost/replica:0/task:0/device:CPU:0\n", 450 | "2018-07-30 21:38:01.248947: I tensorflow/core/common_runtime/placer.cc:886] Const_1: (Const)/job:localho\n", 451 | "st/replica:0/task:0/device:CPU:0" 452 | ] 453 | }, 454 | { 455 | "cell_type": "code", 456 | "execution_count": 13, 457 | "metadata": {}, 458 | "outputs": [ 459 | { 460 | "data": { 461 | "text/plain": [ 462 | "12.0" 463 | ] 464 | }, 465 | "execution_count": 13, 466 | "metadata": {}, 467 | "output_type": "execute_result" 468 | } 469 | ], 470 | "source": [ 471 | "sess.run(c)" 472 | ] 473 | }, 474 | { 475 | "cell_type": "markdown", 476 | "metadata": {}, 477 | "source": [ 478 | "3) 동적 배치 함수\n", 479 | "\n", 480 | "- tf.device 함수 파라미터로 조건 함수를 넣을 수 있음" 481 | ] 482 | }, 483 | { 484 | "cell_type": "code", 485 | "execution_count": 14, 486 | "metadata": {}, 487 | "outputs": [], 488 | "source": [ 489 | "reset_graph()\n", 490 | "\n", 491 | "def variables_on_cpu(op):\n", 492 | " if(op.type == \"Variable\"):\n", 493 | " return \"/cpu:0\"\n", 494 | " else:\n", 495 | " return \"/gpu:0\"\n", 496 | "\n", 497 | "with tf.device(variables_on_cpu):\n", 498 | " a = tf.Variable(3.0)\n", 499 | " b = tf.constant(4.0)\n", 500 | " c = a * b" 501 | ] 502 | }, 503 | { 504 | "cell_type": "markdown", 505 | "metadata": {}, 506 | "source": [ 507 | "4) 연산과 커널\n", 508 | "\n", 509 | "- CPU, GPU 장치별로 지원가능한 연산이 구분되어 잇음\n", 510 | "- 커널 : 장치에 맞는 구현\n", 511 | "- 예) integer Variable 은 GPU 에서 지원하지 않음 - 효율성 문제로 강제 미지원\n", 512 | "- 하시 예시의 경우, dtype=tf.float32 로 명시하지 않으면 숫자 notation 을 통해 정수로 임의판단함" 513 | ] 514 | }, 515 | { 516 | "cell_type": "code", 517 | "execution_count": 15, 518 | "metadata": {}, 519 | "outputs": [ 520 | { 521 | "name": "stdout", 522 | "output_type": "stream", 523 | "text": [ 524 | "Cannot assign a device for operation 'Variable': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.\n", 525 | "Colocation Debug Info:\n", 526 | "Colocation group had the following types and devices: \n", 527 | "Assign: CPU \n", 528 | "Identity: GPU CPU \n", 529 | "VariableV2: CPU \n", 530 | "\n", 531 | "Colocation members and user-requested devices:\n", 532 | " Variable (VariableV2) /device:GPU:0\n", 533 | " Variable/Assign (Assign) /device:GPU:0\n", 534 | " Variable/read (Identity) /device:GPU:0\n", 535 | "\n", 536 | "Registered kernels:\n", 537 | " device='CPU'\n", 538 | " device='GPU'; dtype in [DT_INT64]\n", 539 | " device='GPU'; dtype in [DT_DOUBLE]\n", 540 | " device='GPU'; dtype in [DT_FLOAT]\n", 541 | " device='GPU'; dtype in [DT_HALF]\n", 542 | "\n", 543 | "\t [[Node: Variable = VariableV2[container=\"\", dtype=DT_INT32, shape=[], shared_name=\"\", _device=\"/device:GPU:0\"]()]]\n", 544 | "\n", 545 | "Caused by op 'Variable', defined at:\n", 546 | " File \"/usr/lib/python3.5/runpy.py\", line 184, in _run_module_as_main\n", 547 | " \"__main__\", mod_spec)\n", 548 | " File \"/usr/lib/python3.5/runpy.py\", line 85, in _run_code\n", 549 | " exec(code, run_globals)\n", 550 | " File \"/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py\", line 16, in \n", 551 | " app.launch_new_instance()\n", 552 | " File \"/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py\", line 658, in launch_instance\n", 553 | " app.start()\n", 554 | " File \"/usr/local/lib/python3.5/dist-packages/ipykernel/kernelapp.py\", line 486, in start\n", 555 | " self.io_loop.start()\n", 556 | " File \"/usr/local/lib/python3.5/dist-packages/tornado/platform/asyncio.py\", line 132, in start\n", 557 | " self.asyncio_loop.run_forever()\n", 558 | " File \"/usr/lib/python3.5/asyncio/base_events.py\", line 345, in run_forever\n", 559 | " self._run_once()\n", 560 | " File \"/usr/lib/python3.5/asyncio/base_events.py\", line 1312, in _run_once\n", 561 | " handle._run()\n", 562 | " File \"/usr/lib/python3.5/asyncio/events.py\", line 125, in _run\n", 563 | " self._callback(*self._args)\n", 564 | " File \"/usr/local/lib/python3.5/dist-packages/tornado/ioloop.py\", line 758, in _run_callback\n", 565 | " ret = callback()\n", 566 | " File \"/usr/local/lib/python3.5/dist-packages/tornado/stack_context.py\", line 300, in null_wrapper\n", 567 | " return fn(*args, **kwargs)\n", 568 | " File \"/usr/local/lib/python3.5/dist-packages/zmq/eventloop/zmqstream.py\", line 536, in \n", 569 | " self.io_loop.add_callback(lambda : self._handle_events(self.socket, 0))\n", 570 | " File \"/usr/local/lib/python3.5/dist-packages/zmq/eventloop/zmqstream.py\", line 450, in _handle_events\n", 571 | " self._handle_recv()\n", 572 | " File \"/usr/local/lib/python3.5/dist-packages/zmq/eventloop/zmqstream.py\", line 480, in _handle_recv\n", 573 | " self._run_callback(callback, msg)\n", 574 | " File \"/usr/local/lib/python3.5/dist-packages/zmq/eventloop/zmqstream.py\", line 432, in _run_callback\n", 575 | " callback(*args, **kwargs)\n", 576 | " File \"/usr/local/lib/python3.5/dist-packages/tornado/stack_context.py\", line 300, in null_wrapper\n", 577 | " return fn(*args, **kwargs)\n", 578 | " File \"/usr/local/lib/python3.5/dist-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\n", 579 | " return self.dispatch_shell(stream, msg)\n", 580 | " File \"/usr/local/lib/python3.5/dist-packages/ipykernel/kernelbase.py\", line 233, in dispatch_shell\n", 581 | " handler(stream, idents, msg)\n", 582 | " File \"/usr/local/lib/python3.5/dist-packages/ipykernel/kernelbase.py\", line 399, in execute_request\n", 583 | " user_expressions, allow_stdin)\n", 584 | " File \"/usr/local/lib/python3.5/dist-packages/ipykernel/ipkernel.py\", line 208, in do_execute\n", 585 | " res = shell.run_cell(code, store_history=store_history, silent=silent)\n", 586 | " File \"/usr/local/lib/python3.5/dist-packages/ipykernel/zmqshell.py\", line 537, in run_cell\n", 587 | " return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\n", 588 | " File \"/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py\", line 2662, in run_cell\n", 589 | " raw_cell, store_history, silent, shell_futures)\n", 590 | " File \"/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py\", line 2785, in _run_cell\n", 591 | " interactivity=interactivity, compiler=compiler, result=result)\n", 592 | " File \"/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py\", line 2903, in run_ast_nodes\n", 593 | " if self.run_code(code, result):\n", 594 | " File \"/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py\", line 2963, in run_code\n", 595 | " exec(code_obj, self.user_global_ns, self.user_ns)\n", 596 | " File \"\", line 4, in \n", 597 | " a = tf.Variable(3)\n", 598 | " File \"/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py\", line 259, in __init__\n", 599 | " constraint=constraint)\n", 600 | " File \"/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py\", line 396, in _init_from_args\n", 601 | " name=name)\n", 602 | " File \"/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py\", line 73, in variable_op_v2\n", 603 | " shared_name=shared_name)\n", 604 | " File \"/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py\", line 1255, in variable_v2\n", 605 | " shared_name=shared_name, name=name)\n", 606 | " File \"/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py\", line 787, in _apply_op_helper\n", 607 | " op_def=op_def)\n", 608 | " File \"/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py\", line 3414, in create_op\n", 609 | " op_def=op_def)\n", 610 | " File \"/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py\", line 1740, in __init__\n", 611 | " self._traceback = self._graph._extract_stack() # pylint: disable=protected-access\n", 612 | "\n", 613 | "InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'Variable': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.\n", 614 | "Colocation Debug Info:\n", 615 | "Colocation group had the following types and devices: \n", 616 | "Assign: CPU \n", 617 | "Identity: GPU CPU \n", 618 | "VariableV2: CPU \n", 619 | "\n", 620 | "Colocation members and user-requested devices:\n", 621 | " Variable (VariableV2) /device:GPU:0\n", 622 | " Variable/Assign (Assign) /device:GPU:0\n", 623 | " Variable/read (Identity) /device:GPU:0\n", 624 | "\n", 625 | "Registered kernels:\n", 626 | " device='CPU'\n", 627 | " device='GPU'; dtype in [DT_INT64]\n", 628 | " device='GPU'; dtype in [DT_DOUBLE]\n", 629 | " device='GPU'; dtype in [DT_FLOAT]\n", 630 | " device='GPU'; dtype in [DT_HALF]\n", 631 | "\n", 632 | "\t [[Node: Variable = VariableV2[container=\"\", dtype=DT_INT32, shape=[], shared_name=\"\", _device=\"/device:GPU:0\"]()]]\n", 633 | "\n" 634 | ] 635 | } 636 | ], 637 | "source": [ 638 | "reset_graph()\n", 639 | "\n", 640 | "with tf.device(\"/gpu:0\"):\n", 641 | " a = tf.Variable(3) \n", 642 | " #a = tf.Variable(3, dtype=tf.float32) \n", 643 | "\n", 644 | "with tf.Session() as sess:\n", 645 | " try:\n", 646 | " sess.run(a.initializer) # error\n", 647 | " except Exception as e:\n", 648 | " print(e)" 649 | ] 650 | }, 651 | { 652 | "cell_type": "markdown", 653 | "metadata": {}, 654 | "source": [ 655 | "5) 간접 배치\n", 656 | "\n", 657 | "- 해당 커널이 없는 device 에 할당된 연산을, 해당 커널을 가진 임의 device 에 할당하는 옵션\n", 658 | "- 묵시적 할당" 659 | ] 660 | }, 661 | { 662 | "cell_type": "code", 663 | "execution_count": 16, 664 | "metadata": {}, 665 | "outputs": [], 666 | "source": [ 667 | "reset_graph()\n", 668 | "\n", 669 | "with tf.device(\"/gpu:0\"):\n", 670 | " a = tf.Variable(3)\n", 671 | " \n", 672 | "config = tf.ConfigProto()\n", 673 | "config.log_device_placement = True\n", 674 | "config.allow_soft_placement = True\n", 675 | "session = tf.Session(config=config)\n", 676 | "session.run(a.initializer) # /cpu:0" 677 | ] 678 | }, 679 | { 680 | "cell_type": "markdown", 681 | "metadata": {}, 682 | "source": [ 683 | "Variable: (VariableV2): /job:localhost/replica:0/task:0/device:CPU:0\n", 684 | "2018-07-30 21:39:03.847717: I tensorflow/core/common_runtime/placer.cc:886] Variable: (VariableV2)/job:localhost/replica:0/task:0/device:CPU:0\n", 685 | "Variable/Assign: (Assign): /job:localhost/replica:0/task:0/device:CPU:0\n", 686 | "2018-07-30 21:39:03.847749: I tensorflow/core/common_runtime/placer.cc:886] Variable/Assign: (Assign)/job:localhost/replica:0/task:0/device:CPU:0\n", 687 | "Variable/read: (Identity): /job:localhost/replica:0/task:0/device:CPU:0\n", 688 | "2018-07-30 21:39:03.847778: I tensorflow/core/common_runtime/placer.cc:886] Variable/read: (Identity)/job:localhost/replica:0/task:0/device:CPU:0\n", 689 | "Variable/initial_value: (Const): /job:localhost/replica:0/task:0/device:GPU:0\n", 690 | "2018-07-30 21:39:03.847799: I tensorflow/core/common_runtime/placer.cc:886] Variable/initial_value: (Const)/job:localhost/replica:0/task:0/device:GPU:0" 691 | ] 692 | }, 693 | { 694 | "cell_type": "markdown", 695 | "metadata": {}, 696 | "source": [ 697 | "*12.1.4 병렬 실행*\n", 698 | "\n", 699 | "CPU\n", 700 | "\n", 701 | "1) queueing\n", 702 | "- nn graph 실행시, 먼저 평가할 연산을 찾은 뒤 관련 연산들의 의존도를 측정\n", 703 | "- 의존성이 전혀 없는 node 들을 할당된 장치의 evaluation queue 에 추가\n", 704 | "- 하나의 연산이 평가되면 다른 모든 연산의 dependency counter 가 감소\n", 705 | "- 그 후 dependency counter 가 0 이 되는 연산이 추가로 장치의 evaluation queue 에 추가\n", 706 | "- 필요한 모든 Node 가 평가되면 output 을 return\n", 707 | "\n", 708 | "2) pooling\n", 709 | "- cpu 의 evaluation queue 에 있는 연산은 inter-op thread pool 로 이동\n", 710 | "- 각 상황에 맞도록 병렬 처리 \n", 711 | " - multi-core hardware 를 이용\n", 712 | " - multithread cpu 커널 \n", 713 | " - 여러개의 부분연산으로 분리하여 다른 evaluation queue 에 배치\n", 714 | " - 상기 evaluation queue 에 배치된 연산은 (모든 multithread cpu 커널이 공유하는)intra-op thread pool 로 이동\n", 715 | "\n", 716 | "- inter / intra-op thread pool 의 thread 수는 옵션으로 조정가능 - default 0 (모든 코어 사용)\n", 717 | " - 현재 CPU 특정 코어 지정이 불가하기 때문에, 옵션값을 CPU 코어 수 보다 적에 부여해야함\n", 718 | "\n", 719 | "GPU\n", 720 | "- GPU 상의 evaluation queue 연산들은 순서대로 평가됨\n", 721 | "- 대다수의 연산에 대한 CUDA / cuDNN 기반 multithread GPU 커널 존재\n", 722 | "\n" 723 | ] 724 | }, 725 | { 726 | "cell_type": "markdown", 727 | "metadata": {}, 728 | "source": [ 729 | "" 730 | ] 731 | }, 732 | { 733 | "cell_type": "markdown", 734 | "metadata": {}, 735 | "source": [ 736 | "*12.1.5 제어 의존성*\n", 737 | "\n", 738 | "- 의존 연산이 모두 수행되엇음에도, 효율을 위해 evaluation 을 가급적 delay 할 경우\n", 739 | "-- 다량의 메모리 점유, 다수의 external I/O 발생 등\n", 740 | "- 다른 연산을 병렬 처리 하며 순차적 실행\n", 741 | "\n", 742 | "\n" 743 | ] 744 | }, 745 | { 746 | "cell_type": "code", 747 | "execution_count": 17, 748 | "metadata": {}, 749 | "outputs": [], 750 | "source": [ 751 | "a = tf.constant(1.0)\n", 752 | "b = a + 2.0\n", 753 | "\n", 754 | "with tf.control_dependencies([a,b]):\n", 755 | " x = tf.constant(3.0)\n", 756 | " y = tf.constant(4.0)\n", 757 | " \n", 758 | "z = x + y # z 도 a, b 의 evaluation 을 기다리는 의존성이 발생" 759 | ] 760 | }, 761 | { 762 | "cell_type": "markdown", 763 | "metadata": {}, 764 | "source": [ 765 | "**12.2 다중 머신의 다중 장치**\n", 766 | "\n", 767 | "- task : 하나 이상의 텐서플로 서버로 구성\n", 768 | "- job : 각기 이름이 부여된 task group\n", 769 | "- cluster : task 라고 불리는 하나 이상의 텐서플로 서버로 구성\n" 770 | ] 771 | }, 772 | { 773 | "cell_type": "markdown", 774 | "metadata": {}, 775 | "source": [ 776 | "" 777 | ] 778 | }, 779 | { 780 | "cell_type": "markdown", 781 | "metadata": {}, 782 | "source": [ 783 | "# 클러스터" 784 | ] 785 | }, 786 | { 787 | "cell_type": "code", 788 | "execution_count": 18, 789 | "metadata": {}, 790 | "outputs": [], 791 | "source": [ 792 | "# import tensorflow as tf\n", 793 | "# cluster_spec = tf.train.ClusterSpec({\n", 794 | "# \"ps\": [\n", 795 | "# \"machine-a-host:2221\", # /job:ps/task:0\n", 796 | "# ],\n", 797 | "# \"worker\": [\n", 798 | "# \"machine-a-host:2222\", # /job:worker/task:0 \n", 799 | "# \"machine-b-host:2222\", # /job:worker/task:1\n", 800 | "# ]})\n", 801 | "\n", 802 | "# # 동일 머신에서 여러 task 수행은 가능하나 비추천. 각 GPU RAM 점유를 수동으로 조정해줘야함\n", 803 | "# task_ps0 = tf.train.Server(cluster_spec, job_name=\"ps\", task_index=0)\n", 804 | "# task_ps1 = tf.train.Server(cluster_spec, job_name=\"ps\", task_index=1)\n", 805 | "# ...\n", 806 | "\n", 807 | "# #server.join() # 원격 클러스터를 호출하여 수행된 서버가 종료될 때까지 기다리도록 메인 스레드를 블록(명시적 서버 종료 방법은 없음)" 808 | ] 809 | }, 810 | { 811 | "cell_type": "code", 812 | "execution_count": 19, 813 | "metadata": {}, 814 | "outputs": [], 815 | "source": [ 816 | "# 클러스터에 사용할 원격 서버의 ip 기재\n", 817 | "# 한 장비(ip)에 port 는 겹치면 정상 동작하지 않는다.\n", 818 | "ps_task0_ip = \"10.138.0.4:3333\"\n", 819 | "ps_task1_ip = \"10.138.0.4:3334\"\n", 820 | "worker_task0_ip = \"10.138.0.2:2221\"\n", 821 | "worker_task1_ip = \"10.138.0.4:2222\"" 822 | ] 823 | }, 824 | { 825 | "cell_type": "code", 826 | "execution_count": 20, 827 | "metadata": {}, 828 | "outputs": [], 829 | "source": [ 830 | "cluster_mgpu = tf.train.ClusterSpec({\n", 831 | " \"ps\": [\n", 832 | " ps_task0_ip, # /job:ps/task:0\n", 833 | " ps_task1_ip, # /job:ps/task:1\n", 834 | " ],\n", 835 | " \"worker\": [\n", 836 | " worker_task0_ip, # /job:worker/task:0\n", 837 | " worker_task1_ip, # /job:worker/task:1\n", 838 | " ]}\n", 839 | ")\n", 840 | "\n", 841 | "task_ps0 = tf.train.Server(cluster_mgpu, job_name=\"ps\", task_index=0)\n", 842 | "task_ps1 = tf.train.Server(cluster_mgpu, job_name=\"ps\", task_index=1)\n", 843 | "\n", 844 | "task_worker0 = tf.train.Server(cluster_mgpu, job_name=\"worker\", task_index=0)\n", 845 | "task_worker1 = tf.train.Server(cluster_mgpu, job_name=\"worker\", task_index=1)" 846 | ] 847 | }, 848 | { 849 | "cell_type": "markdown", 850 | "metadata": {}, 851 | "source": [ 852 | "*12.2.1 세션 열기*\n", 853 | "\n", 854 | "- 각 클러스터에서 tf.train.Server 로 클러스터를 띄운 후, 한 머신의 프로세스의 클라이언트에서 다른 모든 서버에 대해 세션을 열 수 있음" 855 | ] 856 | }, 857 | { 858 | "cell_type": "markdown", 859 | "metadata": {}, 860 | "source": [ 861 | "*12.2.2 마스터와 워커 서비스*\n", 862 | "\n", 863 | "- Client / Server 는 gRPC 로 통신함\n", 864 | " - 적절한 통신 포트를 방화벽에서 열어주어야 함\n", 865 | "\n", 866 | "- 텐서플로 서버는 기본적으로 마스터, 워커 서비스를 제공함\n", 867 | " - 마스터 : 클라이언트가 세션을 열고 그래플르 실행할 수 잇게 해줌\n", 868 | " - 워커 : 하나의 서버에서 graph 실행을 담당하는 RPC 서비스\n", 869 | " \n", 870 | "- 유연성 제공\n", 871 | "- 1 Client 가 n Server 에 접속하는 각각 session 오픈 가능\n", 872 | "- task 마다 1 client 실행\n", 873 | "- 1 client 로 여러 task 제어\n", 874 | "\n" 875 | ] 876 | }, 877 | { 878 | "cell_type": "markdown", 879 | "metadata": {}, 880 | "source": [ 881 | "*12.2.3 여러 태스크에 연산 할당하기*\n", 882 | "\n", 883 | "- job 이름, task 번호, 장치 유형/번호 지정하여 연산 할당 가능\n", 884 | "- 장치 유형/번호가 지정되지 않으면 해당 task 의 기본 장치 사용\n" 885 | ] 886 | }, 887 | { 888 | "cell_type": "code", 889 | "execution_count": 21, 890 | "metadata": {}, 891 | "outputs": [ 892 | { 893 | "name": "stdout", 894 | "output_type": "stream", 895 | "text": [ 896 | "4\n" 897 | ] 898 | } 899 | ], 900 | "source": [ 901 | "reset_graph()\n", 902 | "\n", 903 | "with tf.device(\"/job:ps\"):\n", 904 | " a = tf.Variable(1, name=\"a\")\n", 905 | " d = tf.Variable(3.0, name=\"d\")\n", 906 | "\n", 907 | "with tf.device(\"/job:ps/task:1\"):\n", 908 | " e = tf.Variable(5.1, name=\"e\")\n", 909 | "\n", 910 | "with tf.device(\"/job:worker/task:0/gpu:0\"):\n", 911 | " b = a + 2\n", 912 | "\n", 913 | "with tf.device(\"/job:worker/task:1/gpu:1\"):\n", 914 | " c = a + b\n", 915 | "\n", 916 | "\n", 917 | "config = tf.ConfigProto()\n", 918 | "config.log_device_placement = True\n", 919 | " \n", 920 | "# 해당 원격 텐서플로 서버에 대한 세션을 오픈, c 를 evaluate 하라는 명령을 전달\n", 921 | "# 해당 외부 장비의 기본장치(GPU) 에 배치 후 수행, 결과 반환\n", 922 | "#with tf.Session(target=\"grpc://10.138.0.2:2221\", config=config) as sess: \n", 923 | "with tf.Session(target=\"grpc://\" + worker_task0_ip, config=config) as sess: \n", 924 | " sess.run(a.initializer)\n", 925 | " print(c.eval())" 926 | ] 927 | }, 928 | { 929 | "cell_type": "markdown", 930 | "metadata": {}, 931 | "source": [ 932 | "*12.2.4 여러 대의 파라미터 서버에 변수를 나누어 분산하기*\n", 933 | "\n", 934 | "- 다량의 파라미터가 있는 대규모 모델의 경우, 서버 한 대로 IO가 몰리지 않게 여러 서버에 분산해둠\n", 935 | "- 별도 명시설정없이 모든 task 에 round-robin 할당해주는 방법 제공\n", 936 | "- 대체로 파라미터 서버는 파라미터 저장 / 송수신 용도로 사용되고 무거운 연산을 수행하지 않게 구성" 937 | ] 938 | }, 939 | { 940 | "cell_type": "code", 941 | "execution_count": 22, 942 | "metadata": {}, 943 | "outputs": [], 944 | "source": [ 945 | "reset_graph()\n", 946 | "\n", 947 | "with tf.device(tf.train.replica_device_setter( # cluster=cluster_spec 처럼 클러스터 명세를 전달해도 ps_tasks 를 찾아서 이용함\n", 948 | " ps_tasks=2,\n", 949 | " ps_device=\"/job:ps\",\n", 950 | " worker_device=\"/job:worker\")):\n", 951 | " v1 = tf.Variable(1.0, name=\"v1\") # /job:ps/task:0 (defaults to /cpu:0) 에 할당\n", 952 | " v2 = tf.Variable(2.0, name=\"v2\") # /job:ps/task:1 (defaults to /cpu:0) 에 할당\n", 953 | " v3 = tf.Variable(3.0, name=\"v3\") # /job:ps/task:0 (defaults to /cpu:0) 에 할당\n", 954 | " s = v1 + v2 # /job:worker (defaults to task:0/cpu:0) 에 할당\n", 955 | " with tf.device(\"/task:1\"):\n", 956 | " p1 = 2 * s # /job:worker/task:1 (defaults to /cpu:0) 에 할당\n", 957 | " with tf.device(\"/cpu:0\"):\n", 958 | " p2 = 3 * s # /job:worker/task:1/cpu:0 에 할당\n", 959 | "\n", 960 | "config = tf.ConfigProto()\n", 961 | "config.log_device_placement = True\n", 962 | "\n", 963 | "with tf.Session(\"grpc://\" + worker_task0_ip, config=config) as sess:\n", 964 | " v1.initializer.run()" 965 | ] 966 | }, 967 | { 968 | "cell_type": "markdown", 969 | "metadata": {}, 970 | "source": [ 971 | "*12.2.5 리소스 컨테이너를 사용해 여러 세션에서 상태 공유하기*\n", 972 | "\n", 973 | "- local session 은 서로의 상태를 공유할 수 없음\n", 974 | "- distributed session 의 경우 variable 의 상태를 클러스터 내부의 resource container 로 관리함\n", 975 | " - 특정 클라이언트 세션 한 곳에서 새 변수를 생성하면 동일 클러스터 내의 다른 세션에서도 자동으로 사용 가능\n", 976 | " - 리소스 컨테이너는 master task 에서 관리 되는듯\n" 977 | ] 978 | }, 979 | { 980 | "cell_type": "markdown", 981 | "metadata": {}, 982 | "source": [ 983 | "" 984 | ] 985 | }, 986 | { 987 | "cell_type": "code", 988 | "execution_count": 23, 989 | "metadata": {}, 990 | "outputs": [], 991 | "source": [ 992 | "config = tf.ConfigProto()\n", 993 | "\n", 994 | "with tf.container(\"scope01\"):\n", 995 | " x = tf.Variable(0.0, name=\"x\")\n", 996 | " increment_x = tf.assign(x, x+1)\n", 997 | "\n", 998 | "with tf.container(\"scope02\"):\n", 999 | " x2 = tf.Variable(0.0, name=\"x\")\n", 1000 | " increment_x2 = tf.assign(x2, x2+1)" 1001 | ] 1002 | }, 1003 | { 1004 | "cell_type": "code", 1005 | "execution_count": 24, 1006 | "metadata": {}, 1007 | "outputs": [ 1008 | { 1009 | "name": "stdout", 1010 | "output_type": "stream", 1011 | "text": [ 1012 | "1.0\n", 1013 | "2.0\n" 1014 | ] 1015 | } 1016 | ], 1017 | "source": [ 1018 | "#tf.Session.reset(\"grpc://10.138.0.2:2221\", [\"scope01\"])\n", 1019 | "#tf.Session.reset(\"grpc://10.138.0.2:2221\", [\"scope02\"])\n", 1020 | "\n", 1021 | "with tf.Session(\"grpc://\" + worker_task0_ip, config=config) as sess:\n", 1022 | " with tf.container(\"scope01\"):\n", 1023 | " sess.run(x.initializer)\n", 1024 | " sess.run(increment_x)\n", 1025 | " print(x.eval())\n", 1026 | " with tf.container(\"scope02\"):\n", 1027 | " sess.run(x2.initializer)\n", 1028 | " sess.run(increment_x2)\n", 1029 | " sess.run(increment_x2)\n", 1030 | " print(x2.eval())\n", 1031 | " " 1032 | ] 1033 | }, 1034 | { 1035 | "cell_type": "markdown", 1036 | "metadata": {}, 1037 | "source": [ 1038 | "*12.2.6 텐서플로 큐를 사용한 비동기 통신*\n", 1039 | "\n", 1040 | "- 비동기 데이터 교환을 가능하게 함\n", 1041 | " - 데이터 로더 -> 큐 -> 데이터 학습\n", 1042 | "\n", 1043 | "- placeholder 를 사용해 클러스터에 데이터 주입" 1044 | ] 1045 | }, 1046 | { 1047 | "cell_type": "markdown", 1048 | "metadata": {}, 1049 | "source": [ 1050 | "" 1051 | ] 1052 | }, 1053 | { 1054 | "cell_type": "markdown", 1055 | "metadata": {}, 1056 | "source": [ 1057 | "- FIFOQueue\n", 1058 | " - tuple 지원\n", 1059 | "- RandomShuffleQueue\n", 1060 | "- PaddingFIFOQueue" 1061 | ] 1062 | }, 1063 | { 1064 | "cell_type": "code", 1065 | "execution_count": 25, 1066 | "metadata": {}, 1067 | "outputs": [], 1068 | "source": [ 1069 | "import tensorflow as tf\n", 1070 | "config = tf.ConfigProto()\n", 1071 | "\n", 1072 | "# tuple\n", 1073 | "q = tf.FIFOQueue(capacity=10, dtypes=[tf.float32], shapes=[[2]], name=\"q\", shared_name=\"shared_q\")\n", 1074 | "\n", 1075 | "training_instance = tf.placeholder(tf.float32, shape=(2))\n", 1076 | "enqueue = q.enqueue([training_instance]) \n", 1077 | "\n", 1078 | "training_instances = tf.placeholder(tf.float32, shape=(None, 2))\n", 1079 | "enqueue_many = q.enqueue_many([training_instances]) \n", 1080 | "\n", 1081 | "with tf.Session(\"grpc://\" + worker_task0_ip, config=config) as sess:\n", 1082 | " #sess.run(enqueue, feed_dict={training_instance: [1., 2.]})\n", 1083 | " sess.run(enqueue_many, feed_dict={training_instances: [[1., 2.], [3., 4.]]})\n", 1084 | " \n", 1085 | " " 1086 | ] 1087 | }, 1088 | { 1089 | "cell_type": "code", 1090 | "execution_count": 26, 1091 | "metadata": {}, 1092 | "outputs": [ 1093 | { 1094 | "name": "stdout", 1095 | "output_type": "stream", 1096 | "text": [ 1097 | "[[1. 2.]\n", 1098 | " [3. 4.]]\n" 1099 | ] 1100 | } 1101 | ], 1102 | "source": [ 1103 | "config = tf.ConfigProto()\n", 1104 | "\n", 1105 | "# tuple\n", 1106 | "q = tf.FIFOQueue(capacity=10, dtypes=[tf.float32], shapes=[[2]], name=\"q\", shared_name=\"shared_q\")\n", 1107 | "\n", 1108 | "dequeue = q.dequeue() \n", 1109 | "\n", 1110 | "batch_size=2\n", 1111 | "dequeue_many = q.dequeue_many(batch_size) \n", 1112 | "\n", 1113 | "with tf.Session(\"grpc://\" + worker_task0_ip, config=config) as sess:\n", 1114 | " #sess.run(dequeue)\n", 1115 | " print(sess.run(dequeue_many))\n", 1116 | " \n", 1117 | " " 1118 | ] 1119 | }, 1120 | { 1121 | "cell_type": "code", 1122 | "execution_count": 27, 1123 | "metadata": {}, 1124 | "outputs": [ 1125 | { 1126 | "name": "stdout", 1127 | "output_type": "stream", 1128 | "text": [ 1129 | "2.0\n", 1130 | "6.0\n", 1131 | "3.0\n", 1132 | "4.0\n", 1133 | "dequeue 타임 아웃\n" 1134 | ] 1135 | } 1136 | ], 1137 | "source": [ 1138 | "reset_graph()\n", 1139 | "\n", 1140 | "q = tf.FIFOQueue(capacity=10, dtypes=[tf.float32], shapes=[()])\n", 1141 | "v = tf.placeholder(tf.float32)\n", 1142 | "enqueue = q.enqueue([v])\n", 1143 | "dequeue = q.dequeue()\n", 1144 | "output = dequeue + 1\n", 1145 | "\n", 1146 | "config = tf.ConfigProto()\n", 1147 | "config.operation_timeout_in_ms = 1000\n", 1148 | "\n", 1149 | "with tf.Session(config=config) as sess:\n", 1150 | " sess.run(enqueue, feed_dict={v: 1.0})\n", 1151 | " sess.run(enqueue, feed_dict={v: 2.0})\n", 1152 | " sess.run(enqueue, feed_dict={v: 3.0})\n", 1153 | " print(sess.run(output))\n", 1154 | " print(sess.run(output, feed_dict={dequeue: 5}))\n", 1155 | " print(sess.run(output))\n", 1156 | " print(sess.run(output))\n", 1157 | " try:\n", 1158 | " print(sess.run(output))\n", 1159 | " except tf.errors.DeadlineExceededError as ex:\n", 1160 | " print(\"dequeue 타임 아웃\")" 1161 | ] 1162 | }, 1163 | { 1164 | "cell_type": "markdown", 1165 | "metadata": {}, 1166 | "source": [ 1167 | "*12.2.7 그래프에서 직접 데이터 로드하기*\n", 1168 | "\n", 1169 | "- 여러번 전송이 일어나서 대규모 환경에서는 비효율적\n", 1170 | " - 파일IO > 클라이언트 > 마스터 태스크 > 데이터를 필요로하는 다른 태스크\n", 1171 | "- 여러 클라이언트가 동시레 한 자원(데이터) 을 사용할때" 1172 | ] 1173 | }, 1174 | { 1175 | "cell_type": "markdown", 1176 | "metadata": {}, 1177 | "source": [ 1178 | "1) 데이터를 변수에 프리로드하기\n", 1179 | "- 훈련 데이터를 한번에 로드하여 변수에 할당(메모리 크기에 맞으면)\n", 1180 | " - 클라이언트 > 클러스터 1회 전송\n" 1181 | ] 1182 | }, 1183 | { 1184 | "cell_type": "markdown", 1185 | "metadata": {}, 1186 | "source": [ 1187 | "2) 그래프에서 직접 훈련 데이터 읽기\n", 1188 | "- 훈련 데이터가 메모리 크기에 안 맞다면 reader 활용\n", 1189 | "- 파일 시스템에서 직접 데이터를 읽음(클라이언트를 통하지 않고 가능)\n", 1190 | "- 지원 형식 : CSV, fixed length binary record, TFRecords\n", 1191 | "\n" 1192 | ] 1193 | }, 1194 | { 1195 | "cell_type": "markdown", 1196 | "metadata": {}, 1197 | "source": [ 1198 | "" 1199 | ] 1200 | }, 1201 | { 1202 | "cell_type": "code", 1203 | "execution_count": 28, 1204 | "metadata": {}, 1205 | "outputs": [ 1206 | { 1207 | "name": "stdout", 1208 | "output_type": "stream", 1209 | "text": [ 1210 | "더 이상 읽을 파일이 없습니다\n", 1211 | "[array([[ 4., 5.],\n", 1212 | " [17., 81.]], dtype=float32), array([1, 0], dtype=int32)]\n", 1213 | "[array([[14., 51.],\n", 1214 | " [ 1., -1.]], dtype=float32), array([1, 0], dtype=int32)]\n", 1215 | "[array([[ 1., -1.],\n", 1216 | " [ 7., 8.]], dtype=float32), array([0, 0], dtype=int32)]\n", 1217 | "더 이상 훈련 샘플이 없습니다\n" 1218 | ] 1219 | } 1220 | ], 1221 | "source": [ 1222 | "reset_graph()\n", 1223 | "\n", 1224 | "test_csv = open(\"my_test.csv\", \"w\")\n", 1225 | "test_csv.write(\"x1, x2 , target\\n\")\n", 1226 | "test_csv.write(\"1.,, 0\\n\")\n", 1227 | "test_csv.write(\"4., 5. , 1\\n\")\n", 1228 | "test_csv.write(\"7., 8. , 0\\n\")\n", 1229 | "test_csv.close()\n", 1230 | "\n", 1231 | "test_csv2 = open(\"my_test2.csv\", \"w\")\n", 1232 | "test_csv2.write(\"x1, x2 , target\\n\")\n", 1233 | "test_csv2.write(\"1.,, 0\\n\")\n", 1234 | "test_csv2.write(\"14., 51. , 1\\n\")\n", 1235 | "test_csv2.write(\"17., 81. , 0\\n\")\n", 1236 | "test_csv2.close()\n", 1237 | "\n", 1238 | "filename_queue = tf.FIFOQueue(capacity=10, dtypes=[tf.string], shapes=[()])\n", 1239 | "filename = tf.placeholder(tf.string)\n", 1240 | "enqueue_filename = filename_queue.enqueue([filename])\n", 1241 | "close_filename_queue = filename_queue.close()\n", 1242 | "\n", 1243 | "reader = tf.TextLineReader(skip_header_lines=1)\n", 1244 | "key, value = reader.read(filename_queue)\n", 1245 | "\n", 1246 | "x1, x2, target = tf.decode_csv(value, record_defaults=[[-1.], [-1.], [-1]])\n", 1247 | "features = tf.stack([x1, x2])\n", 1248 | "\n", 1249 | "instance_queue = tf.RandomShuffleQueue(\n", 1250 | " capacity=10, min_after_dequeue=2,\n", 1251 | " dtypes=[tf.float32, tf.int32], shapes=[[2],[]],\n", 1252 | " name=\"instance_q\", shared_name=\"shared_instance_q\")\n", 1253 | "enqueue_instance = instance_queue.enqueue([features, target])\n", 1254 | "close_instance_queue = instance_queue.close()\n", 1255 | "\n", 1256 | "minibatch_instances, minibatch_targets = instance_queue.dequeue_up_to(2)\n", 1257 | "\n", 1258 | "with tf.Session() as sess:\n", 1259 | " sess.run(enqueue_filename, feed_dict={filename: \"my_test.csv\"})\n", 1260 | " sess.run(enqueue_filename, feed_dict={filename: \"my_test2.csv\"})\n", 1261 | " sess.run(close_filename_queue)\n", 1262 | " try:\n", 1263 | " while True:\n", 1264 | " sess.run(enqueue_instance)\n", 1265 | " except tf.errors.OutOfRangeError as ex:\n", 1266 | " print(\"더 이상 읽을 파일이 없습니다\")\n", 1267 | " sess.run(close_instance_queue)\n", 1268 | " try:\n", 1269 | " while True:\n", 1270 | " print(sess.run([minibatch_instances, minibatch_targets]))\n", 1271 | " except tf.errors.OutOfRangeError as ex:\n", 1272 | " print(\"더 이상 훈련 샘플이 없습니다\")" 1273 | ] 1274 | }, 1275 | { 1276 | "cell_type": "markdown", 1277 | "metadata": {}, 1278 | "source": [ 1279 | "# 리더 (Reader) - 예전 방법" 1280 | ] 1281 | }, 1282 | { 1283 | "cell_type": "code", 1284 | "execution_count": 29, 1285 | "metadata": {}, 1286 | "outputs": [ 1287 | { 1288 | "name": "stdout", 1289 | "output_type": "stream", 1290 | "text": [ 1291 | "[1.0, 6, 44]\n" 1292 | ] 1293 | } 1294 | ], 1295 | "source": [ 1296 | "reset_graph()\n", 1297 | "\n", 1298 | "default1 = tf.constant([5.])\n", 1299 | "default2 = tf.constant([6])\n", 1300 | "default3 = tf.constant([7])\n", 1301 | "dec = tf.decode_csv(tf.constant(\"1.,,44\"),\n", 1302 | " record_defaults=[default1, default2, default3])\n", 1303 | "with tf.Session() as sess:\n", 1304 | " print(sess.run(dec))" 1305 | ] 1306 | }, 1307 | { 1308 | "cell_type": "code", 1309 | "execution_count": 30, 1310 | "metadata": {}, 1311 | "outputs": [], 1312 | "source": [ 1313 | "#coord = tf.train.Coordinator()\n", 1314 | "#threads = tf.train.start_queue_runners(coord=coord)\n", 1315 | "#filename_queue = tf.train.string_input_producer([\"test.csv\"])\n", 1316 | "#coord.request_stop()\n", 1317 | "#coord.join(threads)" 1318 | ] 1319 | }, 1320 | { 1321 | "cell_type": "markdown", 1322 | "metadata": {}, 1323 | "source": [ 1324 | "3) Coordinator 와 QueueRunner 를 사용하는 멀티스레드 리더\n", 1325 | "\n", 1326 | "- 여러 스레드가 여러 리더로 여러 파일을 동시에 read 할 목적\n", 1327 | "- 스레드 구현 및 제어 관리를 신경쓰지 않고 사용 가능" 1328 | ] 1329 | }, 1330 | { 1331 | "cell_type": "markdown", 1332 | "metadata": {}, 1333 | "source": [ 1334 | "" 1335 | ] 1336 | }, 1337 | { 1338 | "cell_type": "code", 1339 | "execution_count": 31, 1340 | "metadata": {}, 1341 | "outputs": [ 1342 | { 1343 | "name": "stdout", 1344 | "output_type": "stream", 1345 | "text": [ 1346 | "[array([[ 7., 8.],\n", 1347 | " [ 1., -1.]], dtype=float32), array([0, 0], dtype=int32)]\n", 1348 | "[array([[17., 81.],\n", 1349 | " [ 1., -1.]], dtype=float32), array([0, 0], dtype=int32)]\n", 1350 | "[array([[14., 51.],\n", 1351 | " [ 4., 5.]], dtype=float32), array([1, 1], dtype=int32)]\n", 1352 | "더 이상 훈련 샘플이 없습니다\n" 1353 | ] 1354 | } 1355 | ], 1356 | "source": [ 1357 | "reset_graph()\n", 1358 | "\n", 1359 | "filename_queue = tf.FIFOQueue(capacity=10, dtypes=[tf.string], shapes=[()])\n", 1360 | "filename = tf.placeholder(tf.string)\n", 1361 | "enqueue_filename = filename_queue.enqueue([filename])\n", 1362 | "close_filename_queue = filename_queue.close()\n", 1363 | "\n", 1364 | "reader = tf.TextLineReader(skip_header_lines=1)\n", 1365 | "key, value = reader.read(filename_queue)\n", 1366 | "\n", 1367 | "x1, x2, target = tf.decode_csv(value, record_defaults=[[-1.], [-1.], [-1]])\n", 1368 | "features = tf.stack([x1, x2])\n", 1369 | "\n", 1370 | "instance_queue = tf.RandomShuffleQueue(\n", 1371 | " capacity=10, min_after_dequeue=2,\n", 1372 | " dtypes=[tf.float32, tf.int32], shapes=[[2],[]],\n", 1373 | " name=\"instance_q\", shared_name=\"shared_instance_q\")\n", 1374 | "enqueue_instance = instance_queue.enqueue([features, target])\n", 1375 | "close_instance_queue = instance_queue.close()\n", 1376 | "\n", 1377 | "minibatch_instances, minibatch_targets = instance_queue.dequeue_up_to(2)\n", 1378 | "\n", 1379 | "n_threads = 5\n", 1380 | "queue_runner = tf.train.QueueRunner(instance_queue, [enqueue_instance] * n_threads)\n", 1381 | "coord = tf.train.Coordinator()\n", 1382 | "\n", 1383 | "with tf.Session() as sess:\n", 1384 | " sess.run(enqueue_filename, feed_dict={filename: \"my_test.csv\"})\n", 1385 | " sess.run(enqueue_filename, feed_dict={filename: \"my_test2.csv\"}) \n", 1386 | " sess.run(close_filename_queue)\n", 1387 | " enqueue_threads = queue_runner.create_threads(sess, coord=coord, start=True)\n", 1388 | " try:\n", 1389 | " while True:\n", 1390 | " print(sess.run([minibatch_instances, minibatch_targets]))\n", 1391 | " except tf.errors.OutOfRangeError as ex:\n", 1392 | " print(\"더 이상 훈련 샘플이 없습니다\")" 1393 | ] 1394 | }, 1395 | { 1396 | "cell_type": "code", 1397 | "execution_count": 32, 1398 | "metadata": {}, 1399 | "outputs": [ 1400 | { 1401 | "name": "stdout", 1402 | "output_type": "stream", 1403 | "text": [ 1404 | "[array([[ 1., -1.],\n", 1405 | " [ 4., 5.]], dtype=float32), array([0, 1], dtype=int32)]\n", 1406 | "[array([[14., 51.],\n", 1407 | " [ 1., -1.]], dtype=float32), array([1, 0], dtype=int32)]\n", 1408 | "[array([[17., 81.],\n", 1409 | " [ 7., 8.]], dtype=float32), array([0, 0], dtype=int32)]\n", 1410 | "더 이상 훈련 샘플이 없습니다\n" 1411 | ] 1412 | } 1413 | ], 1414 | "source": [ 1415 | "reset_graph()\n", 1416 | "\n", 1417 | "def read_and_push_instance(filename_queue, instance_queue):\n", 1418 | " reader = tf.TextLineReader(skip_header_lines=1)\n", 1419 | " key, value = reader.read(filename_queue)\n", 1420 | " x1, x2, target = tf.decode_csv(value, record_defaults=[[-1.], [-1.], [-1]])\n", 1421 | " features = tf.stack([x1, x2])\n", 1422 | " enqueue_instance = instance_queue.enqueue([features, target])\n", 1423 | " return enqueue_instance\n", 1424 | "\n", 1425 | "filename_queue = tf.FIFOQueue(capacity=10, dtypes=[tf.string], shapes=[()])\n", 1426 | "filename = tf.placeholder(tf.string)\n", 1427 | "enqueue_filename = filename_queue.enqueue([filename])\n", 1428 | "close_filename_queue = filename_queue.close()\n", 1429 | "\n", 1430 | "instance_queue = tf.RandomShuffleQueue(\n", 1431 | " capacity=10, min_after_dequeue=2,\n", 1432 | " dtypes=[tf.float32, tf.int32], shapes=[[2],[]],\n", 1433 | " name=\"instance_q\", shared_name=\"shared_instance_q\")\n", 1434 | "\n", 1435 | "minibatch_instances, minibatch_targets = instance_queue.dequeue_up_to(2)\n", 1436 | "\n", 1437 | "read_and_enqueue_ops = [read_and_push_instance(filename_queue, instance_queue) for i in range(5)]\n", 1438 | "queue_runner = tf.train.QueueRunner(instance_queue, read_and_enqueue_ops)\n", 1439 | "\n", 1440 | "with tf.Session() as sess:\n", 1441 | " sess.run(enqueue_filename, feed_dict={filename: \"my_test.csv\"})\n", 1442 | " sess.run(enqueue_filename, feed_dict={filename: \"my_test2.csv\"})\n", 1443 | " sess.run(close_filename_queue)\n", 1444 | " coord = tf.train.Coordinator()\n", 1445 | " enqueue_threads = queue_runner.create_threads(sess, coord=coord, start=True)\n", 1446 | " try:\n", 1447 | " while True:\n", 1448 | " print(sess.run([minibatch_instances, minibatch_targets]))\n", 1449 | " except tf.errors.OutOfRangeError as ex:\n", 1450 | " print(\"더 이상 훈련 샘플이 없습니다\")" 1451 | ] 1452 | }, 1453 | { 1454 | "cell_type": "markdown", 1455 | "metadata": {}, 1456 | "source": [ 1457 | "# Data API" 1458 | ] 1459 | }, 1460 | { 1461 | "cell_type": "markdown", 1462 | "metadata": {}, 1463 | "source": [ 1464 | "텐서플로 1.4에서 소개된 Data API를 사용하면 손쉽게 데이터를 효율적으로 읽을 수 있습니다." 1465 | ] 1466 | }, 1467 | { 1468 | "cell_type": "code", 1469 | "execution_count": 33, 1470 | "metadata": {}, 1471 | "outputs": [], 1472 | "source": [ 1473 | "tf.reset_default_graph()" 1474 | ] 1475 | }, 1476 | { 1477 | "cell_type": "markdown", 1478 | "metadata": {}, 1479 | "source": [ 1480 | "0에서 9까지 정수를 세 번 반복한 간단한 데이터셋을 일곱 개씩 배치로 만들어 시작해 보죠:" 1481 | ] 1482 | }, 1483 | { 1484 | "cell_type": "code", 1485 | "execution_count": 34, 1486 | "metadata": {}, 1487 | "outputs": [], 1488 | "source": [ 1489 | "dataset = tf.data.Dataset.from_tensor_slices(np.arange(10))\n", 1490 | "dataset = dataset.repeat(3).batch(7)" 1491 | ] 1492 | }, 1493 | { 1494 | "cell_type": "markdown", 1495 | "metadata": {}, 1496 | "source": [ 1497 | "첫 번째 줄은 0에서 9까지 정수를 담은 데이터셋을 만듭니다. 두 번째 줄은 이 데이터셋의 원소를 세 번 반복하고 일곱 개씩 담은 새로운 데이터셋을 만듭니다. 위에서 볼 수 있듯이 원본 데이터셋에서 여러 변환 메서드를 연결하여 호출하여 적용했습니다." 1498 | ] 1499 | }, 1500 | { 1501 | "cell_type": "markdown", 1502 | "metadata": {}, 1503 | "source": [ 1504 | "그다음, 데이터셋을 한 번 순회하는 원-샷-이터레이터(one-shot-iterator)를 만들고, 다음 원소를 지칭하는 텐서를 얻기 위해 `get_next()` 메서드를 호출합니다." 1505 | ] 1506 | }, 1507 | { 1508 | "cell_type": "code", 1509 | "execution_count": 35, 1510 | "metadata": {}, 1511 | "outputs": [], 1512 | "source": [ 1513 | "iterator = dataset.make_one_shot_iterator()\n", 1514 | "next_element = iterator.get_next()" 1515 | ] 1516 | }, 1517 | { 1518 | "cell_type": "markdown", 1519 | "metadata": {}, 1520 | "source": [ 1521 | "`next_element`를 반복적으로 평가해서 데이터셋을 순회해 보죠. 원소가 별로 없기 때문에 `OutOfRangeError`가 발생합니다:" 1522 | ] 1523 | }, 1524 | { 1525 | "cell_type": "code", 1526 | "execution_count": 36, 1527 | "metadata": {}, 1528 | "outputs": [ 1529 | { 1530 | "name": "stdout", 1531 | "output_type": "stream", 1532 | "text": [ 1533 | "[0 1 2 3 4 5 6]\n", 1534 | "[7 8 9 0 1 2 3]\n", 1535 | "[4 5 6 7 8 9 0]\n", 1536 | "[1 2 3 4 5 6 7]\n", 1537 | "[8 9]\n", 1538 | "완료\n" 1539 | ] 1540 | } 1541 | ], 1542 | "source": [ 1543 | "with tf.Session() as sess:\n", 1544 | " try:\n", 1545 | " while True:\n", 1546 | " print(next_element.eval())\n", 1547 | " except tf.errors.OutOfRangeError:\n", 1548 | " print(\"완료\")" 1549 | ] 1550 | }, 1551 | { 1552 | "cell_type": "markdown", 1553 | "metadata": {}, 1554 | "source": [ 1555 | "좋네요! 잘 작동합니다." 1556 | ] 1557 | }, 1558 | { 1559 | "cell_type": "markdown", 1560 | "metadata": {}, 1561 | "source": [ 1562 | "늘 그렇듯이 텐서는 그래프를 실행(`sess.run()`)할 때마다 한 번만 평가된다는 것을 기억하세요. `next_element`에 의존하는 텐서를 여러개 평가하더라도 한 번만 평가됩니다. 또한 `next_element`를 동시에 두 번 실행해도 마찬가지입니다:" 1563 | ] 1564 | }, 1565 | { 1566 | "cell_type": "code", 1567 | "execution_count": 37, 1568 | "metadata": {}, 1569 | "outputs": [ 1570 | { 1571 | "name": "stdout", 1572 | "output_type": "stream", 1573 | "text": [ 1574 | "[array([0, 1, 2, 3, 4, 5, 6]), array([0, 1, 2, 3, 4, 5, 6])]\n", 1575 | "[array([7, 8, 9, 0, 1, 2, 3]), array([7, 8, 9, 0, 1, 2, 3])]\n", 1576 | "[array([4, 5, 6, 7, 8, 9, 0]), array([4, 5, 6, 7, 8, 9, 0])]\n", 1577 | "[array([1, 2, 3, 4, 5, 6, 7]), array([1, 2, 3, 4, 5, 6, 7])]\n", 1578 | "[array([8, 9]), array([8, 9])]\n", 1579 | "완료\n" 1580 | ] 1581 | } 1582 | ], 1583 | "source": [ 1584 | "with tf.Session() as sess:\n", 1585 | " try:\n", 1586 | " while True:\n", 1587 | " print(sess.run([next_element, next_element]))\n", 1588 | " except tf.errors.OutOfRangeError:\n", 1589 | " print(\"완료\")" 1590 | ] 1591 | }, 1592 | { 1593 | "cell_type": "markdown", 1594 | "metadata": {}, 1595 | "source": [ 1596 | "`interleave()` 메서드는 강력하지만 처음에는 이해하기 좀 어렵습니다. 예제를 통해 이해하는 것이 가장 좋습니다:" 1597 | ] 1598 | }, 1599 | { 1600 | "cell_type": "code", 1601 | "execution_count": 38, 1602 | "metadata": {}, 1603 | "outputs": [], 1604 | "source": [ 1605 | "tf.reset_default_graph()" 1606 | ] 1607 | }, 1608 | { 1609 | "cell_type": "code", 1610 | "execution_count": 39, 1611 | "metadata": {}, 1612 | "outputs": [], 1613 | "source": [ 1614 | "dataset = tf.data.Dataset.from_tensor_slices(np.arange(10))\n", 1615 | "dataset = dataset.repeat(3).batch(7)\n", 1616 | "dataset = dataset.interleave(\n", 1617 | " lambda v: tf.data.Dataset.from_tensor_slices(v),\n", 1618 | " cycle_length=3,\n", 1619 | " block_length=2)\n", 1620 | "iterator = dataset.make_one_shot_iterator()\n", 1621 | "next_element = iterator.get_next()" 1622 | ] 1623 | }, 1624 | { 1625 | "cell_type": "code", 1626 | "execution_count": 40, 1627 | "metadata": {}, 1628 | "outputs": [ 1629 | { 1630 | "name": "stdout", 1631 | "output_type": "stream", 1632 | "text": [ 1633 | "0,1,7,8,4,5,2,3,9,0,6,7,4,5,1,2,8,9,6,3,0,1,2,8,9,3,4,5,6,7,완료\n" 1634 | ] 1635 | } 1636 | ], 1637 | "source": [ 1638 | "with tf.Session() as sess:\n", 1639 | " try:\n", 1640 | " while True:\n", 1641 | " print(next_element.eval(), end=\",\")\n", 1642 | " except tf.errors.OutOfRangeError:\n", 1643 | " print(\"완료\")" 1644 | ] 1645 | }, 1646 | { 1647 | "cell_type": "markdown", 1648 | "metadata": {}, 1649 | "source": [ 1650 | "`cycle_length=3`이므로 새로운 데이터셋은 이전 데이터셋에서 세 개의 원소를 추출합니다. 즉 `[0,1,2,3,4,5,6]`, `[7,8,9,0,1,2,3]`, `[4,5,6,7,8,9,0]` 입니다. 그다음 원소마다 하나의 데이터셋을 만들기 위해 람다(lambda) 함수를 호출합니다. `Dataset.from_tensor_slices()`를 사용했기 때문에 각 데이터셋은 차례대로 원소를 반환합니다. 다음 이 세 개의 데이터셋에서 각각 두 개의 아이템(`block_length=2`이므로)을 추출합니다. 세 개의 데이터셋의 아이템이 모두 소진될 때까지 반복됩니다. 즉 0,1 (첫 번째에서), 7,8 (두 번째에서), 4,5 (세 번째에서), 2,3 (첫 번째에서), 9,0 (두 번째에서) 등과 같은 식으로 8,9 (세 번째에서), 6 (첫 번째에서), 3 (두 번째에서), 0 (세 번째에서)까지 진행됩니다. 그다음에 원본 데이터셋에서 다음 번 세 개의 원소를 추출하려고 합니다. 하지만 두 개만 남아 있습니다. `[1,2,3,4,5,6,7]`와 `[8,9]` 입니다. 다시 이 원소로부터 데이터셋을 만들고 이 데이텃세의 아이템이 모두 소진될 때까지 두 개의 아이템을 추출합니다. 1,2 (첫 번째에서), 8,9 (두 번째에서), 3,4 (첫 번째에서), 5,6 (첫 번째에서), 7 (첫 번째에서)가 됩니다. 배열의 길이가 다르기 때문에 마지막에는 교대로 배치되지 않았습니다." 1651 | ] 1652 | }, 1653 | { 1654 | "cell_type": "markdown", 1655 | "metadata": {}, 1656 | "source": [ 1657 | "# 리더 (Reader) - 새로운 방법" 1658 | ] 1659 | }, 1660 | { 1661 | "cell_type": "markdown", 1662 | "metadata": {}, 1663 | "source": [ 1664 | "`from_tensor_slices()`나 `from_tensor()`를 기반으로 한 원본 데이터셋을 사용하는 대신 리더 데이터셋을 사용할 수 있습니다. 복잡한 일들을 대부분 대신 처리해 줍니다(예를 들면, 스레드):" 1665 | ] 1666 | }, 1667 | { 1668 | "cell_type": "code", 1669 | "execution_count": 41, 1670 | "metadata": {}, 1671 | "outputs": [], 1672 | "source": [ 1673 | "tf.reset_default_graph()" 1674 | ] 1675 | }, 1676 | { 1677 | "cell_type": "code", 1678 | "execution_count": 42, 1679 | "metadata": {}, 1680 | "outputs": [], 1681 | "source": [ 1682 | "filenames = [\"my_test.csv\"]" 1683 | ] 1684 | }, 1685 | { 1686 | "cell_type": "code", 1687 | "execution_count": 43, 1688 | "metadata": {}, 1689 | "outputs": [], 1690 | "source": [ 1691 | "dataset = tf.data.TextLineDataset(filenames)" 1692 | ] 1693 | }, 1694 | { 1695 | "cell_type": "markdown", 1696 | "metadata": {}, 1697 | "source": [ 1698 | "각 줄을 어떻게 디코드해야 하는지는 알려 주어야 합니다:" 1699 | ] 1700 | }, 1701 | { 1702 | "cell_type": "code", 1703 | "execution_count": 44, 1704 | "metadata": {}, 1705 | "outputs": [], 1706 | "source": [ 1707 | "def decode_csv_line(line):\n", 1708 | " x1, x2, y = tf.decode_csv(\n", 1709 | " line, record_defaults=[[-1.], [-1.], [-1.]])\n", 1710 | " X = tf.stack([x1, x2])\n", 1711 | " return X, y" 1712 | ] 1713 | }, 1714 | { 1715 | "cell_type": "markdown", 1716 | "metadata": {}, 1717 | "source": [ 1718 | "그다음, 이 디코딩 함수를 `map()`을 사용하여 데이터셋에 있는 각 원소에 적용할 수 있습니다:" 1719 | ] 1720 | }, 1721 | { 1722 | "cell_type": "code", 1723 | "execution_count": 45, 1724 | "metadata": {}, 1725 | "outputs": [], 1726 | "source": [ 1727 | "dataset = dataset.skip(1).map(decode_csv_line)" 1728 | ] 1729 | }, 1730 | { 1731 | "cell_type": "markdown", 1732 | "metadata": {}, 1733 | "source": [ 1734 | "마지막으로 원-샷-이터레이터를 만들어 보죠:" 1735 | ] 1736 | }, 1737 | { 1738 | "cell_type": "code", 1739 | "execution_count": 46, 1740 | "metadata": {}, 1741 | "outputs": [], 1742 | "source": [ 1743 | "it = dataset.make_one_shot_iterator()\n", 1744 | "X, y = it.get_next()" 1745 | ] 1746 | }, 1747 | { 1748 | "cell_type": "code", 1749 | "execution_count": 47, 1750 | "metadata": {}, 1751 | "outputs": [ 1752 | { 1753 | "name": "stdout", 1754 | "output_type": "stream", 1755 | "text": [ 1756 | "[ 1. -1.] 0.0\n", 1757 | "[4. 5.] 1.0\n", 1758 | "[7. 8.] 0.0\n", 1759 | "완료\n" 1760 | ] 1761 | } 1762 | ], 1763 | "source": [ 1764 | "with tf.Session() as sess:\n", 1765 | " try:\n", 1766 | " while True:\n", 1767 | " X_val, y_val = sess.run([X, y])\n", 1768 | " print(X_val, y_val)\n", 1769 | " except tf.errors.OutOfRangeError as ex:\n", 1770 | " print(\"완료\")" 1771 | ] 1772 | }, 1773 | { 1774 | "cell_type": "markdown", 1775 | "metadata": {}, 1776 | "source": [ 1777 | "**12.3 텐서플로 클러스터에서 신경망 병렬화하기**\n", 1778 | "\n" 1779 | ] 1780 | }, 1781 | { 1782 | "cell_type": "markdown", 1783 | "metadata": {}, 1784 | "source": [ 1785 | "*12.3.1 장치마나 하나의 신경망*\n", 1786 | "\n", 1787 | "- 가장 단순함, 직관적\n", 1788 | "- 여러 하이퍼파라미터를 돌려보며 튜닝하는 목적\n", 1789 | "- realtime prediction 전용 환경으로도 좋음(gpu 를 증가시켜 scale-out 가능)\n", 1790 | "\n", 1791 | "- tensorflow serving\n", 1792 | "\n" 1793 | ] 1794 | }, 1795 | { 1796 | "cell_type": "markdown", 1797 | "metadata": {}, 1798 | "source": [ 1799 | "" 1800 | ] 1801 | }, 1802 | { 1803 | "cell_type": "markdown", 1804 | "metadata": {}, 1805 | "source": [ 1806 | "*12.3.2 그래프 내 복제와 그래프 간 복제*\n", 1807 | "\n", 1808 | "- 여러 신경망을 분산배치하여 앙상블을 구성할 수 잇음\n", 1809 | "- 모든 신경망들의 prediction 을 모아서 앙상블의 prediction 이 가능\n", 1810 | "- 앙상블 구성방법\n", 1811 | " - 그래프 내 복제 : 구현이 간단함\n", 1812 | "\n", 1813 | "\n", 1814 | "\n", 1815 | "\n" 1816 | ] 1817 | }, 1818 | { 1819 | "cell_type": "markdown", 1820 | "metadata": {}, 1821 | "source": [ 1822 | "" 1823 | ] 1824 | }, 1825 | { 1826 | "cell_type": "markdown", 1827 | "metadata": {}, 1828 | "source": [ 1829 | " - 그래프 간 복제 : 구현이 복잡하지만, queue 통신 기반이기 때문에 앙상블의 안정성을 좀 더 보장해줌" 1830 | ] 1831 | }, 1832 | { 1833 | "cell_type": "markdown", 1834 | "metadata": {}, 1835 | "source": [ 1836 | "" 1837 | ] 1838 | }, 1839 | { 1840 | "cell_type": "markdown", 1841 | "metadata": {}, 1842 | "source": [ 1843 | "*12.3.3 모델 병렬화*\n", 1844 | "\n", 1845 | "- 모델을 여러 부분으로 나누어 각 부분을 다른 장치에서 실행\n", 1846 | "- 신경망 모델 구조에 따라 효율성과 구현 난이도가 갈림\n", 1847 | "\n" 1848 | ] 1849 | }, 1850 | { 1851 | "cell_type": "markdown", 1852 | "metadata": {}, 1853 | "source": [ 1854 | "1) Fully connected" 1855 | ] 1856 | }, 1857 | { 1858 | "cell_type": "markdown", 1859 | "metadata": {}, 1860 | "source": [ 1861 | "" 1862 | ] 1863 | }, 1864 | { 1865 | "cell_type": "markdown", 1866 | "metadata": {}, 1867 | "source": [ 1868 | "2) Partialy connected" 1869 | ] 1870 | }, 1871 | { 1872 | "cell_type": "markdown", 1873 | "metadata": {}, 1874 | "source": [ 1875 | "" 1876 | ] 1877 | }, 1878 | { 1879 | "cell_type": "markdown", 1880 | "metadata": {}, 1881 | "source": [ 1882 | "3) Deep RNN" 1883 | ] 1884 | }, 1885 | { 1886 | "cell_type": "markdown", 1887 | "metadata": {}, 1888 | "source": [ 1889 | "" 1890 | ] 1891 | }, 1892 | { 1893 | "cell_type": "markdown", 1894 | "metadata": {}, 1895 | "source": [ 1896 | "*12.3.4 데이터 병렬화*\n", 1897 | "\n", 1898 | "- 각 장치에 모델 복제 후 각각 다른 데이터(미니배치) 를 사용\n", 1899 | "- 훈련 후 발생하는 gradient 를 취합 후 모델 파라미터 업데이트\n", 1900 | " - 동기 / 비동기" 1901 | ] 1902 | }, 1903 | { 1904 | "cell_type": "markdown", 1905 | "metadata": {}, 1906 | "source": [ 1907 | "" 1908 | ] 1909 | }, 1910 | { 1911 | "cell_type": "markdown", 1912 | "metadata": {}, 1913 | "source": [ 1914 | "1) 동기\n", 1915 | "\n", 1916 | "- 모든 gradient 가 계산될때까지 대기한 후 평균을 계산\n", 1917 | "- 거의 같은 시점에 모든 복제모델에 파라미터가 업데이트 \n", 1918 | "\n", 1919 | "2) 비동기\n", 1920 | "\n", 1921 | "- gradient 가 계산될 시점에 그때그때 모델 파라미터 업데이트\n", 1922 | "- 훈련학습이 빨리되는 모델이 더 파라미터를 자주 업데이트하게 됨\n", 1923 | "- stale gradient\n", 1924 | " - gradient 가 계산된 후 그 값으로 파라미터가 업데이트 되기 전에 다른 모델이 파라미터를 업데이트 한다면?" 1925 | ] 1926 | }, 1927 | { 1928 | "cell_type": "markdown", 1929 | "metadata": {}, 1930 | "source": [ 1931 | "" 1932 | ] 1933 | }, 1934 | { 1935 | "cell_type": "markdown", 1936 | "metadata": {}, 1937 | "source": [ 1938 | "3) 대역폭 포화\n", 1939 | "\n", 1940 | "- gpu ram i/o 시간 > gpu 연산 계산시간(분할된)\n", 1941 | " - 네트워크나 버스로?\n", 1942 | "- 모델이 규모가 작고 데이터 량이 많다면 1 gpu 1 machine 이 더 낫다고함\n", 1943 | "\n", 1944 | "- 대역폭 감소 방안\n", 1945 | " - 적은 수의 머신에 GPU 를 모아서 네트워크 통신 최소화\n", 1946 | " - 여러 대의 파라미터 서버에 파라미터 분산\n", 1947 | " - 모델 파라미터 정밀도 조정 (float32 -> float16)\n", 1948 | " \n", 1949 | "4) 텐서플로 구현 \n", 1950 | "\n", 1951 | "- 그래프 내 복제 / 그래프 간 복제, 동기 / 비동기 등 방법 선택\n", 1952 | "\n", 1953 | "\n", 1954 | "1. 그래프 내 복제 + 동기 업데이트\n", 1955 | " - 모든 복제 모델을 담는 하나의 큰 그래프를 만들고 gradient 수집 후 optimizer 에 전달\n", 1956 | "\n", 1957 | "2. 그래프 내 복제 + 비동기 업데이트\n", 1958 | " - (모든 복제 모델을 담는 하나의 큰 그래프를 만들고) 복제 모델마다 하나의 optimizer 를 만들고 개별 스레드 마다 각 복제본의 optimizer 반복 수행\n", 1959 | " \n", 1960 | "3. 그래프 간 복제 + 비동기 업데이트 \n", 1961 | " - 독립적 모델들이 여러 개 실행되어 각기 훈련됨. 모델 파라미터는 리소스 컨테이너를 활용해 다른 모델과 공유됨.\n", 1962 | " \n", 1963 | "4. 그래프 간 복제 + 동기 업데이트 \n", 1964 | " - 3과 같으나 각 모델은 동일한 옵티마이져(SyncReplicasOptimizer) 사용. 각 모델들의 gradient 들이 큐로 보내지고, \n", 1965 | " \n", 1966 | " chief(복제 모델 중 1) 가 gradient 를 취합 / 적용 한 후 다른 복제 모델들에게 다음 step 진행 개시를 token queue 를 통해 전달\n" 1967 | ] 1968 | }, 1969 | { 1970 | "cell_type": "markdown", 1971 | "metadata": { 1972 | "collapsed": true 1973 | }, 1974 | "source": [ 1975 | "# 연습문제 해답" 1976 | ] 1977 | }, 1978 | { 1979 | "cell_type": "markdown", 1980 | "metadata": {}, 1981 | "source": [ 1982 | "https://github.com/ageron/handson-ml/issues/187" 1983 | ] 1984 | }, 1985 | { 1986 | "cell_type": "markdown", 1987 | "metadata": {}, 1988 | "source": [ 1989 | "1. 텐서플로 프로그램 수행시 CUDA_ERROR_OUT_OF_MEMORY 오류가 발생하는 원인 및 해결책은 무엇인가?" 1990 | ] 1991 | }, 1992 | { 1993 | "cell_type": "markdown", 1994 | "metadata": {}, 1995 | "source": [ 1996 | "2. (장치) 연산 할당 / 연산 배치 의 차이점." 1997 | ] 1998 | }, 1999 | { 2000 | "cell_type": "markdown", 2001 | "metadata": {}, 2002 | "source": [ 2003 | "3. GPU 버젼 텐서플로의 기본 배치로 실행하면 모든 연산이 GPU #0 에 배치되는가?" 2004 | ] 2005 | }, 2006 | { 2007 | "cell_type": "markdown", 2008 | "metadata": {}, 2009 | "source": [ 2010 | "4. \"/gpu:0\" 에 할당된 Variable 을 \"/gpu:1\" 상의 연산이 사용 할 수 있는가?\n", 2011 | "\n", 2012 | " 또는 \"/cpu:0\" 상의 연산이 사용 할 수 있는가? 다른 서버 장치에 할당된 연산은?\n" 2013 | ] 2014 | }, 2015 | { 2016 | "cell_type": "markdown", 2017 | "metadata": {}, 2018 | "source": [ 2019 | "5. 같은 장치에 배치된 두 연산이 동시에 수행될 수 있는가?" 2020 | ] 2021 | }, 2022 | { 2023 | "cell_type": "markdown", 2024 | "metadata": {}, 2025 | "source": [ 2026 | "6. 제어 의존성이 무엇인가? 언제 사용되는가?" 2027 | ] 2028 | }, 2029 | { 2030 | "cell_type": "markdown", 2031 | "metadata": {}, 2032 | "source": [ 2033 | "7. 클러스터로 며칠간 DNN 을 훈련한 후 Saver 로 모델을 저장하지 않앗다면 모델은 날아간 것인가?" 2034 | ] 2035 | }, 2036 | { 2037 | "cell_type": "markdown", 2038 | "metadata": {}, 2039 | "source": [ 2040 | "8. 클러스터에서 몇 개의 DNN 을 각기 다른 hparam 으로 훈련해 보라.\n", 2041 | " cross validation 또는 validation set 을 통해 가장 좋은 모델 세 개를 선택하라." 2042 | ] 2043 | }, 2044 | { 2045 | "cell_type": "markdown", 2046 | "metadata": {}, 2047 | "source": [ 2048 | "9. 상기 선택된 모델 세 개로 앙상블을 구성하라. 개별 DNN 보다 성능이 더 나은가?" 2049 | ] 2050 | }, 2051 | { 2052 | "cell_type": "markdown", 2053 | "metadata": {}, 2054 | "source": [ 2055 | "10. 그래프 간 복제, 비동기 업데이트를 통해 DNN 을 훈련 시켜보라. 동기 업데이트를 사용해 다시 훈련해보라.\n", 2056 | " 성능 및 훈련 소요 시간을 비교해보라.\n", 2057 | " DNN 수직 분할 후 훈련시켰을 때도 추가로 비교해 보라." 2058 | ] 2059 | }, 2060 | { 2061 | "cell_type": "code", 2062 | "execution_count": null, 2063 | "metadata": {}, 2064 | "outputs": [], 2065 | "source": [] 2066 | } 2067 | ], 2068 | "metadata": { 2069 | "kernelspec": { 2070 | "display_name": "Python 3", 2071 | "language": "python", 2072 | "name": "python3" 2073 | }, 2074 | "language_info": { 2075 | "codemirror_mode": { 2076 | "name": "ipython", 2077 | "version": 3 2078 | }, 2079 | "file_extension": ".py", 2080 | "mimetype": "text/x-python", 2081 | "name": "python", 2082 | "nbconvert_exporter": "python", 2083 | "pygments_lexer": "ipython3", 2084 | "version": "3.5.5" 2085 | }, 2086 | "nav_menu": {}, 2087 | "toc": { 2088 | "base_numbering": 1, 2089 | "nav_menu": {}, 2090 | "number_sections": true, 2091 | "sideBar": true, 2092 | "skip_h1_title": false, 2093 | "title_cell": "Table of Contents", 2094 | "title_sidebar": "Contents", 2095 | "toc_cell": false, 2096 | "toc_position": {}, 2097 | "toc_section_display": "block", 2098 | "toc_window_display": false 2099 | }, 2100 | "varInspector": { 2101 | "cols": { 2102 | "lenName": 16, 2103 | "lenType": 16, 2104 | "lenVar": 40 2105 | }, 2106 | "kernels_config": { 2107 | "python": { 2108 | "delete_cmd_postfix": "", 2109 | "delete_cmd_prefix": "del ", 2110 | "library": "var_list.py", 2111 | "varRefreshCmd": "print(var_dic_list())" 2112 | }, 2113 | "r": { 2114 | "delete_cmd_postfix": ") ", 2115 | "delete_cmd_prefix": "rm(", 2116 | "library": "var_list.r", 2117 | "varRefreshCmd": "cat(var_dic_list()) " 2118 | } 2119 | }, 2120 | "types_to_exclude": [ 2121 | "module", 2122 | "function", 2123 | "builtin_function_or_method", 2124 | "instance", 2125 | "_Feature" 2126 | ], 2127 | "window_display": false 2128 | } 2129 | }, 2130 | "nbformat": 4, 2131 | "nbformat_minor": 1 2132 | } 2133 | -------------------------------------------------------------------------------- /12_distributed/fig12-10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-10.png -------------------------------------------------------------------------------- /12_distributed/fig12-11.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-11.png -------------------------------------------------------------------------------- /12_distributed/fig12-12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-12.png -------------------------------------------------------------------------------- /12_distributed/fig12-13.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-13.png -------------------------------------------------------------------------------- /12_distributed/fig12-14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-14.png -------------------------------------------------------------------------------- /12_distributed/fig12-15.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-15.png -------------------------------------------------------------------------------- /12_distributed/fig12-16.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-16.png -------------------------------------------------------------------------------- /12_distributed/fig12-17.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-17.png -------------------------------------------------------------------------------- /12_distributed/fig12-18.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-18.png -------------------------------------------------------------------------------- /12_distributed/fig12-5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-5.png -------------------------------------------------------------------------------- /12_distributed/fig12-6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-6.png -------------------------------------------------------------------------------- /12_distributed/fig12-7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-7.png -------------------------------------------------------------------------------- /12_distributed/fig12-8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-8.png -------------------------------------------------------------------------------- /12_distributed/fig12-9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/12_distributed/fig12-9.png -------------------------------------------------------------------------------- /12_distributed/multigpu_cnn.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Multi-GPU Training Example\n", 8 | "\n", 9 | "Train a convolutional neural network on multiple GPU with TensorFlow.\n", 10 | "\n", 11 | "This example is using TensorFlow layers, see 'convolutional_network_raw' example\n", 12 | "for a raw TensorFlow implementation with variables.\n", 13 | "\n", 14 | "- Author: Aymeric Damien\n", 15 | "- Project: https://github.com/aymericdamien/TensorFlow-Examples/" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "## Training with multiple GPU cards\n", 23 | "\n", 24 | "In this example, we are using data parallelism to split the training accross multiple GPUs. Each GPU has a full replica of the neural network model, and the weights (i.e. variables) are updated synchronously by waiting that each GPU process its batch of data.\n", 25 | "\n", 26 | "First, each GPU process a distinct batch of data and compute the corresponding gradients, then, all gradients are accumulated in the CPU and averaged. The model weights are finally updated with the gradients averaged, and the new model weights are sent back to each GPU, to repeat the training process.\n", 27 | "\n", 28 | "\"Parallelism\"\n", 29 | "\n", 30 | "## MNIST Dataset Overview\n", 31 | "\n", 32 | "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n", 33 | "\n", 34 | "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n", 35 | "\n", 36 | "More info: http://yann.lecun.com/exdb/mnist/" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 1, 42 | "metadata": {}, 43 | "outputs": [ 44 | { 45 | "name": "stderr", 46 | "output_type": "stream", 47 | "text": [ 48 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 49 | " return f(*args, **kwds)\n", 50 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 51 | " return f(*args, **kwds)\n", 52 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 53 | " return f(*args, **kwds)\n", 54 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 55 | " return f(*args, **kwds)\n", 56 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 57 | " return f(*args, **kwds)\n", 58 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 59 | " return f(*args, **kwds)\n" 60 | ] 61 | }, 62 | { 63 | "name": "stdout", 64 | "output_type": "stream", 65 | "text": [ 66 | "WARNING:tensorflow:From :9: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 67 | "Instructions for updating:\n", 68 | "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n", 69 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n", 70 | "Instructions for updating:\n", 71 | "Please write your own downloading logic.\n", 72 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 73 | "Instructions for updating:\n", 74 | "Please use tf.data to implement this functionality.\n", 75 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 76 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 77 | "Instructions for updating:\n", 78 | "Please use tf.data to implement this functionality.\n", 79 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 80 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 81 | "Instructions for updating:\n", 82 | "Please use tf.one_hot on tensors.\n", 83 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 84 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n", 85 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 86 | "Instructions for updating:\n", 87 | "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n" 88 | ] 89 | } 90 | ], 91 | "source": [ 92 | "from __future__ import print_function\n", 93 | "\n", 94 | "import numpy as np\n", 95 | "import tensorflow as tf\n", 96 | "import time\n", 97 | "\n", 98 | "# Import MNIST data\n", 99 | "from tensorflow.examples.tutorials.mnist import input_data\n", 100 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 101 | "\n", 102 | "# Parameters\n", 103 | "num_gpus = 2\n", 104 | "num_steps = 200\n", 105 | "learning_rate = 0.001\n", 106 | "batch_size = 1024\n", 107 | "display_step = 10\n", 108 | "\n", 109 | "# Network Parameters\n", 110 | "num_input = 784 # MNIST data input (img shape: 28*28)\n", 111 | "num_classes = 10 # MNIST total classes (0-9 digits)\n", 112 | "dropout = 0.75 # Dropout, probability to keep units" 113 | ] 114 | }, 115 | { 116 | "cell_type": "code", 117 | "execution_count": 2, 118 | "metadata": {}, 119 | "outputs": [], 120 | "source": [ 121 | "# Build a convolutional neural network\n", 122 | "def conv_net(x, n_classes, dropout, reuse, is_training):\n", 123 | " # Define a scope for reusing the variables\n", 124 | " with tf.variable_scope('ConvNet', reuse=reuse):\n", 125 | " # MNIST data input is a 1-D vector of 784 features (28*28 pixels)\n", 126 | " # Reshape to match picture format [Height x Width x Channel]\n", 127 | " # Tensor input become 4-D: [Batch Size, Height, Width, Channel]\n", 128 | " x = tf.reshape(x, shape=[-1, 28, 28, 1])\n", 129 | "\n", 130 | " # Convolution Layer with 64 filters and a kernel size of 5\n", 131 | " x = tf.layers.conv2d(x, 64, 5, activation=tf.nn.relu)\n", 132 | " # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n", 133 | " x = tf.layers.max_pooling2d(x, 2, 2)\n", 134 | "\n", 135 | " # Convolution Layer with 256 filters and a kernel size of 5\n", 136 | " x = tf.layers.conv2d(x, 256, 3, activation=tf.nn.relu)\n", 137 | " # Convolution Layer with 512 filters and a kernel size of 5\n", 138 | " x = tf.layers.conv2d(x, 512, 3, activation=tf.nn.relu)\n", 139 | " # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n", 140 | " x = tf.layers.max_pooling2d(x, 2, 2)\n", 141 | "\n", 142 | " # Flatten the data to a 1-D vector for the fully connected layer\n", 143 | " x = tf.contrib.layers.flatten(x)\n", 144 | "\n", 145 | " # Fully connected layer (in contrib folder for now)\n", 146 | " x = tf.layers.dense(x, 2048)\n", 147 | " # Apply Dropout (if is_training is False, dropout is not applied)\n", 148 | " x = tf.layers.dropout(x, rate=dropout, training=is_training)\n", 149 | "\n", 150 | " # Fully connected layer (in contrib folder for now)\n", 151 | " x = tf.layers.dense(x, 1024)\n", 152 | " # Apply Dropout (if is_training is False, dropout is not applied)\n", 153 | " x = tf.layers.dropout(x, rate=dropout, training=is_training)\n", 154 | "\n", 155 | " # Output layer, class prediction\n", 156 | " out = tf.layers.dense(x, n_classes)\n", 157 | " # Because 'softmax_cross_entropy_with_logits' loss already apply\n", 158 | " # softmax, we only apply softmax to testing network\n", 159 | " out = tf.nn.softmax(out) if not is_training else out\n", 160 | "\n", 161 | " return out" 162 | ] 163 | }, 164 | { 165 | "cell_type": "code", 166 | "execution_count": 3, 167 | "metadata": {}, 168 | "outputs": [], 169 | "source": [ 170 | "# Build the function to average the gradients\n", 171 | "def average_gradients(tower_grads):\n", 172 | " average_grads = []\n", 173 | " for grad_and_vars in zip(*tower_grads):\n", 174 | " # Note that each grad_and_vars looks like the following:\n", 175 | " # ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))\n", 176 | " grads = []\n", 177 | " for g, _ in grad_and_vars:\n", 178 | " # Add 0 dimension to the gradients to represent the tower.\n", 179 | " expanded_g = tf.expand_dims(g, 0)\n", 180 | "\n", 181 | " # Append on a 'tower' dimension which we will average over below.\n", 182 | " grads.append(expanded_g)\n", 183 | "\n", 184 | " # Average over the 'tower' dimension.\n", 185 | " grad = tf.concat(grads, 0)\n", 186 | " grad = tf.reduce_mean(grad, 0)\n", 187 | "\n", 188 | " # Keep in mind that the Variables are redundant because they are shared\n", 189 | " # across towers. So .. we will just return the first tower's pointer to\n", 190 | " # the Variable.\n", 191 | " v = grad_and_vars[0][1]\n", 192 | " grad_and_var = (grad, v)\n", 193 | " average_grads.append(grad_and_var)\n", 194 | " return average_grads" 195 | ] 196 | }, 197 | { 198 | "cell_type": "code", 199 | "execution_count": 4, 200 | "metadata": { 201 | "scrolled": false 202 | }, 203 | "outputs": [ 204 | { 205 | "name": "stdout", 206 | "output_type": "stream", 207 | "text": [ 208 | "WARNING:tensorflow:From :36: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n", 209 | "Instructions for updating:\n", 210 | "\n", 211 | "Future major versions of TensorFlow will allow gradients to flow\n", 212 | "into the labels input on backprop by default.\n", 213 | "\n", 214 | "See @{tf.nn.softmax_cross_entropy_with_logits_v2}.\n", 215 | "\n", 216 | "Step 1: Minibatch Loss= 2.1074, Training Accuracy= 0.372, 120 Examples/sec\n", 217 | "Step 10: Minibatch Loss= 0.4523, Training Accuracy= 0.881, 7477 Examples/sec\n", 218 | "Step 20: Minibatch Loss= 0.2486, Training Accuracy= 0.949, 7544 Examples/sec\n", 219 | "Step 30: Minibatch Loss= 0.1060, Training Accuracy= 0.969, 7457 Examples/sec\n", 220 | "Step 40: Minibatch Loss= 0.0915, Training Accuracy= 0.976, 7355 Examples/sec\n", 221 | "Step 50: Minibatch Loss= 0.0777, Training Accuracy= 0.987, 7514 Examples/sec\n", 222 | "Step 60: Minibatch Loss= 0.0652, Training Accuracy= 0.985, 7636 Examples/sec\n", 223 | "Step 70: Minibatch Loss= 0.0752, Training Accuracy= 0.983, 7363 Examples/sec\n", 224 | "Step 80: Minibatch Loss= 0.0345, Training Accuracy= 0.992, 7264 Examples/sec\n", 225 | "Step 90: Minibatch Loss= 0.0474, Training Accuracy= 0.990, 7459 Examples/sec\n", 226 | "Step 100: Minibatch Loss= 0.0445, Training Accuracy= 0.994, 7540 Examples/sec\n", 227 | "Step 110: Minibatch Loss= 0.0265, Training Accuracy= 0.995, 7348 Examples/sec\n", 228 | "Step 120: Minibatch Loss= 0.0234, Training Accuracy= 0.992, 7302 Examples/sec\n", 229 | "Step 130: Minibatch Loss= 0.0275, Training Accuracy= 0.989, 7393 Examples/sec\n", 230 | "Step 140: Minibatch Loss= 0.0203, Training Accuracy= 0.995, 7446 Examples/sec\n", 231 | "Step 150: Minibatch Loss= 0.0138, Training Accuracy= 0.993, 7291 Examples/sec\n", 232 | "Step 160: Minibatch Loss= 0.0214, Training Accuracy= 0.996, 7288 Examples/sec\n", 233 | "Step 170: Minibatch Loss= 0.0091, Training Accuracy= 0.997, 7488 Examples/sec\n", 234 | "Step 180: Minibatch Loss= 0.0166, Training Accuracy= 0.997, 7434 Examples/sec\n", 235 | "Step 190: Minibatch Loss= 0.0064, Training Accuracy= 0.996, 7300 Examples/sec\n", 236 | "Step 200: Minibatch Loss= 0.0112, Training Accuracy= 0.995, 7360 Examples/sec\n", 237 | "Optimization Finished!\n", 238 | "Testing Accuracy: 0.99265385\n", 239 | "79.26985669136047\n" 240 | ] 241 | } 242 | ], 243 | "source": [ 244 | "import time\n", 245 | "start = time.time()\n", 246 | "\n", 247 | "# Place all ops on CPU by default\n", 248 | "with tf.device('/cpu:0'):\n", 249 | " tower_grads = []\n", 250 | " reuse_vars = False\n", 251 | "\n", 252 | " # tf Graph input\n", 253 | " X = tf.placeholder(tf.float32, [None, num_input])\n", 254 | " Y = tf.placeholder(tf.float32, [None, num_classes])\n", 255 | "\n", 256 | " #X = tf.placeholder(tf.float16, [None, num_input])\n", 257 | " #Y = tf.placeholder(tf.float16, [None, num_classes])\n", 258 | " \n", 259 | " # Loop over all GPUs and construct their own computation graph\n", 260 | " for i in range(num_gpus):\n", 261 | " with tf.device('/gpu:%d' % i):\n", 262 | "\n", 263 | " # Split data between GPUs\n", 264 | " _x = X[i * batch_size: (i+1) * batch_size]\n", 265 | " _y = Y[i * batch_size: (i+1) * batch_size]\n", 266 | "\n", 267 | " # Because Dropout have different behavior at training and prediction time, we\n", 268 | " # need to create 2 distinct computation graphs that share the same weights.\n", 269 | "\n", 270 | " # Create a graph for training\n", 271 | " logits_train = conv_net(_x, num_classes, dropout,\n", 272 | " reuse=reuse_vars, is_training=True)\n", 273 | " # Create another graph for testing that reuse the same weights\n", 274 | " logits_test = conv_net(_x, num_classes, dropout,\n", 275 | " reuse=True, is_training=False)\n", 276 | "\n", 277 | " # Define loss and optimizer (with train logits, for dropout to take effect)\n", 278 | " loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n", 279 | " logits=logits_train, labels=_y))\n", 280 | " \n", 281 | " optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n", 282 | " \n", 283 | " # https://stackoverflow.com/questions/42064941/tensorflow-float16-support-is-broken\n", 284 | " #optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, epsilon=1e-5) # for float16\n", 285 | " \n", 286 | " grads = optimizer.compute_gradients(loss_op)\n", 287 | "\n", 288 | " # Only first GPU compute accuracy\n", 289 | " if i == 0:\n", 290 | " # Evaluate model (with test logits, for dropout to be disabled)\n", 291 | " correct_pred = tf.equal(tf.argmax(logits_test, 1), tf.argmax(_y, 1))\n", 292 | " accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n", 293 | "\n", 294 | " reuse_vars = True\n", 295 | " tower_grads.append(grads)\n", 296 | "\n", 297 | " tower_grads = average_gradients(tower_grads)\n", 298 | " train_op = optimizer.apply_gradients(tower_grads)\n", 299 | "\n", 300 | " # Initializing the variables\n", 301 | " init = tf.global_variables_initializer()\n", 302 | "\n", 303 | " # Launch the graph\n", 304 | " with tf.Session() as sess:\n", 305 | " sess.run(init)\n", 306 | " step = 1\n", 307 | " # Keep training until reach max iterations\n", 308 | " for step in range(1, num_steps + 1):\n", 309 | " # Get a batch for each GPU\n", 310 | " batch_x, batch_y = mnist.train.next_batch(batch_size * num_gpus)\n", 311 | " # Run optimization op (backprop)\n", 312 | " ts = time.time()\n", 313 | " sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n", 314 | " te = time.time() - ts\n", 315 | " if step % display_step == 0 or step == 1:\n", 316 | " # Calculate batch loss and accuracy\n", 317 | " loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n", 318 | " Y: batch_y})\n", 319 | " print(\"Step \" + str(step) + \": Minibatch Loss= \" + \\\n", 320 | " \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n", 321 | " \"{:.3f}\".format(acc) + \", %i Examples/sec\" % int(len(batch_x)/te))\n", 322 | " step += 1\n", 323 | " print(\"Optimization Finished!\")\n", 324 | "\n", 325 | " # Calculate accuracy for 1000 mnist test images\n", 326 | " print(\"Testing Accuracy:\", \\\n", 327 | " np.mean([sess.run(accuracy, feed_dict={X: mnist.test.images[i:i+batch_size],\n", 328 | " Y: mnist.test.labels[i:i+batch_size]}) for i in range(0, len(mnist.test.images), batch_size)]))\n", 329 | " \n", 330 | " \n", 331 | "end = time.time()\n", 332 | "print(end-start)" 333 | ] 334 | }, 335 | { 336 | "cell_type": "markdown", 337 | "metadata": {}, 338 | "source": [ 339 | "32float \n", 340 | "\n", 341 | "Testing Accuracy: 0.9929468\n", 342 | "64.05570006370544\n", 343 | "\n", 344 | "16float \n", 345 | "\n", 346 | "Testing Accuracy: 0.9882075\n", 347 | "76.64730191230774\n", 348 | "\n" 349 | ] 350 | }, 351 | { 352 | "cell_type": "code", 353 | "execution_count": null, 354 | "metadata": {}, 355 | "outputs": [], 356 | "source": [] 357 | } 358 | ], 359 | "metadata": { 360 | "kernelspec": { 361 | "display_name": "Python 3", 362 | "language": "python", 363 | "name": "python3" 364 | }, 365 | "language_info": { 366 | "codemirror_mode": { 367 | "name": "ipython", 368 | "version": 3 369 | }, 370 | "file_extension": ".py", 371 | "mimetype": "text/x-python", 372 | "name": "python", 373 | "nbconvert_exporter": "python", 374 | "pygments_lexer": "ipython3", 375 | "version": "3.5.2" 376 | } 377 | }, 378 | "nbformat": 4, 379 | "nbformat_minor": 1 380 | } 381 | -------------------------------------------------------------------------------- /12_distributed/multigpu_cnn_cluster.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Multi-GPU Training Example\n", 8 | "\n", 9 | "Train a convolutional neural network on multiple GPU with TensorFlow.\n", 10 | "\n", 11 | "This example is using TensorFlow layers, see 'convolutional_network_raw' example\n", 12 | "for a raw TensorFlow implementation with variables.\n", 13 | "\n", 14 | "- Author: Aymeric Damien\n", 15 | "- Project: https://github.com/aymericdamien/TensorFlow-Examples/" 16 | ] 17 | }, 18 | { 19 | "cell_type": "markdown", 20 | "metadata": {}, 21 | "source": [ 22 | "## Training with multiple GPU cards\n", 23 | "\n", 24 | "In this example, we are using data parallelism to split the training accross multiple GPUs. Each GPU has a full replica of the neural network model, and the weights (i.e. variables) are updated synchronously by waiting that each GPU process its batch of data.\n", 25 | "\n", 26 | "First, each GPU process a distinct batch of data and compute the corresponding gradients, then, all gradients are accumulated in the CPU and averaged. The model weights are finally updated with the gradients averaged, and the new model weights are sent back to each GPU, to repeat the training process.\n", 27 | "\n", 28 | "\"Parallelism\"\n", 29 | "\n", 30 | "## MNIST Dataset Overview\n", 31 | "\n", 32 | "This example is using MNIST handwritten digits. The dataset contains 60,000 examples for training and 10,000 examples for testing. The digits have been size-normalized and centered in a fixed-size image (28x28 pixels) with values from 0 to 1. For simplicity, each image has been flatten and converted to a 1-D numpy array of 784 features (28*28).\n", 33 | "\n", 34 | "![MNIST Dataset](http://neuralnetworksanddeeplearning.com/images/mnist_100_digits.png)\n", 35 | "\n", 36 | "More info: http://yann.lecun.com/exdb/mnist/" 37 | ] 38 | }, 39 | { 40 | "cell_type": "code", 41 | "execution_count": 1, 42 | "metadata": {}, 43 | "outputs": [ 44 | { 45 | "name": "stderr", 46 | "output_type": "stream", 47 | "text": [ 48 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 49 | " return f(*args, **kwds)\n", 50 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 51 | " return f(*args, **kwds)\n", 52 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 53 | " return f(*args, **kwds)\n", 54 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 55 | " return f(*args, **kwds)\n", 56 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 57 | " return f(*args, **kwds)\n", 58 | "/usr/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n", 59 | " return f(*args, **kwds)\n" 60 | ] 61 | }, 62 | { 63 | "name": "stdout", 64 | "output_type": "stream", 65 | "text": [ 66 | "WARNING:tensorflow:From :9: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 67 | "Instructions for updating:\n", 68 | "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n", 69 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.\n", 70 | "Instructions for updating:\n", 71 | "Please write your own downloading logic.\n", 72 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 73 | "Instructions for updating:\n", 74 | "Please use tf.data to implement this functionality.\n", 75 | "Extracting /tmp/data/train-images-idx3-ubyte.gz\n", 76 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 77 | "Instructions for updating:\n", 78 | "Please use tf.data to implement this functionality.\n", 79 | "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n", 80 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 81 | "Instructions for updating:\n", 82 | "Please use tf.one_hot on tensors.\n", 83 | "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n", 84 | "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n", 85 | "WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.\n", 86 | "Instructions for updating:\n", 87 | "Please use alternatives such as official/mnist/dataset.py from tensorflow/models.\n" 88 | ] 89 | } 90 | ], 91 | "source": [ 92 | "from __future__ import print_function\n", 93 | "\n", 94 | "import numpy as np\n", 95 | "import tensorflow as tf\n", 96 | "import time\n", 97 | "\n", 98 | "# Import MNIST data\n", 99 | "from tensorflow.examples.tutorials.mnist import input_data\n", 100 | "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n", 101 | "\n", 102 | "# Parameters\n", 103 | "num_workers = 2 # # of workers\n", 104 | "num_gpus = 2\n", 105 | "num_steps = 200\n", 106 | "learning_rate = 0.001\n", 107 | "batch_size = 1024\n", 108 | "display_step = 10\n", 109 | "\n", 110 | "# Network Parameters\n", 111 | "num_input = 784 # MNIST data input (img shape: 28*28)\n", 112 | "num_classes = 10 # MNIST total classes (0-9 digits)\n", 113 | "dropout = 0.75 # Dropout, probability to keep units" 114 | ] 115 | }, 116 | { 117 | "cell_type": "code", 118 | "execution_count": 2, 119 | "metadata": {}, 120 | "outputs": [], 121 | "source": [ 122 | "# Build a convolutional neural network\n", 123 | "def conv_net(x, n_classes, dropout, reuse, is_training):\n", 124 | " # Define a scope for reusing the variables\n", 125 | " with tf.variable_scope('ConvNet', reuse=reuse):\n", 126 | " # MNIST data input is a 1-D vector of 784 features (28*28 pixels)\n", 127 | " # Reshape to match picture format [Height x Width x Channel]\n", 128 | " # Tensor input become 4-D: [Batch Size, Height, Width, Channel]\n", 129 | " x = tf.reshape(x, shape=[-1, 28, 28, 1])\n", 130 | "\n", 131 | " # Convolution Layer with 64 filters and a kernel size of 5\n", 132 | " x = tf.layers.conv2d(x, 64, 5, activation=tf.nn.relu)\n", 133 | " # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n", 134 | " x = tf.layers.max_pooling2d(x, 2, 2)\n", 135 | "\n", 136 | " # Convolution Layer with 256 filters and a kernel size of 5\n", 137 | " x = tf.layers.conv2d(x, 256, 3, activation=tf.nn.relu)\n", 138 | " # Convolution Layer with 512 filters and a kernel size of 5\n", 139 | " x = tf.layers.conv2d(x, 512, 3, activation=tf.nn.relu)\n", 140 | " # Max Pooling (down-sampling) with strides of 2 and kernel size of 2\n", 141 | " x = tf.layers.max_pooling2d(x, 2, 2)\n", 142 | "\n", 143 | " # Flatten the data to a 1-D vector for the fully connected layer\n", 144 | " x = tf.contrib.layers.flatten(x)\n", 145 | "\n", 146 | " # Fully connected layer (in contrib folder for now)\n", 147 | " x = tf.layers.dense(x, 2048)\n", 148 | " # Apply Dropout (if is_training is False, dropout is not applied)\n", 149 | " x = tf.layers.dropout(x, rate=dropout, training=is_training)\n", 150 | "\n", 151 | " # Fully connected layer (in contrib folder for now)\n", 152 | " x = tf.layers.dense(x, 1024)\n", 153 | " # Apply Dropout (if is_training is False, dropout is not applied)\n", 154 | " x = tf.layers.dropout(x, rate=dropout, training=is_training)\n", 155 | "\n", 156 | " # Output layer, class prediction\n", 157 | " out = tf.layers.dense(x, n_classes)\n", 158 | " # Because 'softmax_cross_entropy_with_logits' loss already apply\n", 159 | " # softmax, we only apply softmax to testing network\n", 160 | " out = tf.nn.softmax(out) if not is_training else out\n", 161 | "\n", 162 | " return out" 163 | ] 164 | }, 165 | { 166 | "cell_type": "code", 167 | "execution_count": 3, 168 | "metadata": {}, 169 | "outputs": [], 170 | "source": [ 171 | "# Build the function to average the gradients\n", 172 | "def average_gradients(tower_grads):\n", 173 | " average_grads = []\n", 174 | " for grad_and_vars in zip(*tower_grads):\n", 175 | " # Note that each grad_and_vars looks like the following:\n", 176 | " # ((grad0_gpu0, var0_gpu0), ... , (grad0_gpuN, var0_gpuN))\n", 177 | " grads = []\n", 178 | " for g, _ in grad_and_vars:\n", 179 | " # Add 0 dimension to the gradients to represent the tower.\n", 180 | " expanded_g = tf.expand_dims(g, 0)\n", 181 | "\n", 182 | " # Append on a 'tower' dimension which we will average over below.\n", 183 | " grads.append(expanded_g)\n", 184 | "\n", 185 | " # Average over the 'tower' dimension.\n", 186 | " grad = tf.concat(grads, 0)\n", 187 | " grad = tf.reduce_mean(grad, 0)\n", 188 | "\n", 189 | " # Keep in mind that the Variables are redundant because they are shared\n", 190 | " # across towers. So .. we will just return the first tower's pointer to\n", 191 | " # the Variable.\n", 192 | " v = grad_and_vars[0][1]\n", 193 | " grad_and_var = (grad, v)\n", 194 | " average_grads.append(grad_and_var)\n", 195 | " return average_grads" 196 | ] 197 | }, 198 | { 199 | "cell_type": "code", 200 | "execution_count": 4, 201 | "metadata": {}, 202 | "outputs": [], 203 | "source": [ 204 | "cluster_mgpu = tf.train.ClusterSpec({\n", 205 | "# \"ps\": [\n", 206 | "# \"10.138.0.4:3333\", # /job:ps/task:0\n", 207 | "# \"10.138.0.4:3334\", # /job:ps/task:1\n", 208 | "# ],\n", 209 | " \"worker\": [\n", 210 | " \"10.138.0.2:2221\", # /job:worker/task:0\n", 211 | " \"10.138.0.4:2222\", # /job:worker/task:1\n", 212 | " ]}\n", 213 | ")\n", 214 | "\n", 215 | "# task_ps0 = tf.train.Server(cluster_mgpu, job_name=\"ps\", task_index=0)\n", 216 | "# task_ps1 = tf.train.Server(cluster_mgpu, job_name=\"ps\", task_index=1)\n", 217 | "\n", 218 | "task_worker0 = tf.train.Server(cluster_mgpu, job_name=\"worker\", task_index=0)\n", 219 | "task_worker1 = tf.train.Server(cluster_mgpu, job_name=\"worker\", task_index=1)" 220 | ] 221 | }, 222 | { 223 | "cell_type": "code", 224 | "execution_count": 5, 225 | "metadata": {}, 226 | "outputs": [], 227 | "source": [ 228 | "#reset_graph()\n", 229 | "\n", 230 | "# with tf.device(tf.train.replica_device_setter( # cluster=cluster_spec 처럼 클러스터 명세를 전달해도 ps_tasks 를 찾아서 이용함\n", 231 | "# ps_tasks=2,\n", 232 | "# ps_device=\"/job:ps\",\n", 233 | "# worker_device=\"/job:worker\")):\n", 234 | "# v1 = tf.Variable(1.0, name=\"v1\") # /job:ps/task:0 (defaults to /cpu:0) 에 할당\n", 235 | "# v2 = tf.Variable(2.0, name=\"v2\") # /job:ps/task:1 (defaults to /cpu:0) 에 할당\n", 236 | "# v3 = tf.Variable(3.0, name=\"v3\") # /job:ps/task:0 (defaults to /cpu:0) 에 할당\n", 237 | "# s = v1 + v2 # /job:worker (defaults to task:0/cpu:0) 에 할당\n", 238 | "# with tf.device(\"/task:1\"):\n", 239 | "# p1 = 2 * s # /job:worker/task:1 (defaults to /cpu:0) 에 할당\n", 240 | "# with tf.device(\"/cpu:0\"):\n", 241 | "# p2 = 3 * s # /job:worker/task:1/cpu:0 에 할당\n", 242 | "\n", 243 | "# config = tf.ConfigProto()\n", 244 | "# config.log_device_placement = True\n", 245 | "\n", 246 | "# with tf.Session(\"grpc://10.138.0.4:2222\", config=config) as sess:\n", 247 | "# v1.initializer.run()" 248 | ] 249 | }, 250 | { 251 | "cell_type": "code", 252 | "execution_count": 5, 253 | "metadata": { 254 | "scrolled": false 255 | }, 256 | "outputs": [ 257 | { 258 | "name": "stdout", 259 | "output_type": "stream", 260 | "text": [ 261 | "WARNING:tensorflow:From :42: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.\n", 262 | "Instructions for updating:\n", 263 | "\n", 264 | "Future major versions of TensorFlow will allow gradients to flow\n", 265 | "into the labels input on backprop by default.\n", 266 | "\n", 267 | "See @{tf.nn.softmax_cross_entropy_with_logits_v2}.\n", 268 | "\n", 269 | "Step 1: Minibatch Loss= 2.0642, Training Accuracy= 0.454, 661 Examples/sec\n", 270 | "Step 10: Minibatch Loss= 0.4398, Training Accuracy= 0.904, 6627 Examples/sec\n", 271 | "Step 20: Minibatch Loss= 0.1351, Training Accuracy= 0.969, 6779 Examples/sec\n", 272 | "Step 30: Minibatch Loss= 0.1104, Training Accuracy= 0.977, 6917 Examples/sec\n", 273 | "Step 40: Minibatch Loss= 0.0621, Training Accuracy= 0.981, 6767 Examples/sec\n", 274 | "Step 50: Minibatch Loss= 0.0475, Training Accuracy= 0.988, 6572 Examples/sec\n", 275 | "Step 60: Minibatch Loss= 0.0269, Training Accuracy= 0.989, 6277 Examples/sec\n", 276 | "Step 70: Minibatch Loss= 0.0346, Training Accuracy= 0.995, 6705 Examples/sec\n", 277 | "Step 80: Minibatch Loss= 0.0259, Training Accuracy= 0.993, 6883 Examples/sec\n", 278 | "Step 90: Minibatch Loss= 0.0418, Training Accuracy= 0.995, 6734 Examples/sec\n", 279 | "Step 100: Minibatch Loss= 0.0193, Training Accuracy= 0.993, 6173 Examples/sec\n", 280 | "Step 110: Minibatch Loss= 0.0155, Training Accuracy= 0.999, 6650 Examples/sec\n", 281 | "Step 120: Minibatch Loss= 0.0201, Training Accuracy= 0.997, 6851 Examples/sec\n", 282 | "Step 130: Minibatch Loss= 0.0086, Training Accuracy= 0.996, 6521 Examples/sec\n", 283 | "Step 140: Minibatch Loss= 0.0123, Training Accuracy= 1.000, 6764 Examples/sec\n", 284 | "Step 150: Minibatch Loss= 0.0113, Training Accuracy= 0.998, 6889 Examples/sec\n", 285 | "Step 160: Minibatch Loss= 0.0085, Training Accuracy= 0.995, 6859 Examples/sec\n", 286 | "Step 170: Minibatch Loss= 0.0092, Training Accuracy= 0.998, 6706 Examples/sec\n", 287 | "Step 180: Minibatch Loss= 0.0133, Training Accuracy= 0.996, 6792 Examples/sec\n", 288 | "Step 190: Minibatch Loss= 0.0130, Training Accuracy= 0.999, 6941 Examples/sec\n", 289 | "Step 200: Minibatch Loss= 0.0041, Training Accuracy= 1.000, 7142 Examples/sec\n", 290 | "Optimization Finished!\n", 291 | "Testing Accuracy: 0.994314\n", 292 | "142.5129895210266\n" 293 | ] 294 | } 295 | ], 296 | "source": [ 297 | "import time\n", 298 | "start = time.time()\n", 299 | "\n", 300 | "\n", 301 | "with tf.device(tf.train.replica_device_setter( # cluster=cluster_spec 처럼 클러스터 명세를 전달해도 ps_tasks 를 찾아서 이용함\n", 302 | " #ps_tasks=2,\n", 303 | " #ps_device=\"/job:ps\",\n", 304 | " worker_device=\"/job:worker\")):\n", 305 | " tower_grads = []\n", 306 | " reuse_vars = False\n", 307 | "\n", 308 | " # tf Graph input\n", 309 | " X = tf.placeholder(tf.float32, [None, num_input])\n", 310 | " Y = tf.placeholder(tf.float32, [None, num_classes])\n", 311 | "\n", 312 | " #X = tf.placeholder(tf.float16, [None, num_input])\n", 313 | " #Y = tf.placeholder(tf.float16, [None, num_classes])\n", 314 | " \n", 315 | " # Loop over all GPUs and construct their own computation graph\n", 316 | " for i in range(num_gpus):\n", 317 | " for j in range(num_workers):\n", 318 | " with tf.device('/task:%d/gpu:%d' % (j, i)):\n", 319 | " \n", 320 | " idx = i * num_gpus + j\n", 321 | "\n", 322 | " # Split data between GPUs\n", 323 | " _x = X[idx * batch_size: (idx+1) * batch_size]\n", 324 | " _y = Y[idx * batch_size: (idx+1) * batch_size]\n", 325 | " \n", 326 | " #_x = X[idx * int(batch_size / num_workers): (idx+1) * int(batch_size / num_workers)]\n", 327 | " #_y = Y[idx * int(batch_size / num_workers): (idx+1) * int(batch_size / num_workers)]\n", 328 | "\n", 329 | " # Because Dropout have different behavior at training and prediction time, we\n", 330 | " # need to create 2 distinct computation graphs that share the same weights.\n", 331 | "\n", 332 | " # Create a graph for training\n", 333 | " logits_train = conv_net(_x, num_classes, dropout,\n", 334 | " reuse=reuse_vars, is_training=True)\n", 335 | " # Create another graph for testing that reuse the same weights\n", 336 | " logits_test = conv_net(_x, num_classes, dropout,\n", 337 | " reuse=True, is_training=False)\n", 338 | "\n", 339 | " # Define loss and optimizer (with train logits, for dropout to take effect)\n", 340 | " loss_op = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(\n", 341 | " logits=logits_train, labels=_y))\n", 342 | "\n", 343 | " optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n", 344 | "\n", 345 | " # https://stackoverflow.com/questions/42064941/tensorflow-float16-support-is-broken\n", 346 | " #optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, epsilon=1e-5) # for float16\n", 347 | "\n", 348 | " grads = optimizer.compute_gradients(loss_op)\n", 349 | "\n", 350 | " # Only first GPU compute accuracy\n", 351 | " if j == 0 and i == 0:\n", 352 | " # Evaluate model (with test logits, for dropout to be disabled)\n", 353 | " correct_pred = tf.equal(tf.argmax(logits_test, 1), tf.argmax(_y, 1))\n", 354 | " accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n", 355 | "\n", 356 | " reuse_vars = True\n", 357 | " tower_grads.append(grads)\n", 358 | "\n", 359 | " tower_grads = average_gradients(tower_grads)\n", 360 | " train_op = optimizer.apply_gradients(tower_grads)\n", 361 | "\n", 362 | " # Initializing the variables\n", 363 | " init = tf.global_variables_initializer()\n", 364 | "\n", 365 | " \n", 366 | "config = tf.ConfigProto()\n", 367 | "config.log_device_placement = True\n", 368 | "with tf.Session(\"grpc://10.138.0.2:2221\", config=config) as sess:\n", 369 | " sess.run(init)\n", 370 | " step = 1\n", 371 | " # Keep training until reach max iterations\n", 372 | " for step in range(1, num_steps + 1):\n", 373 | " # Get a batch for each GPU\n", 374 | " batch_x, batch_y = mnist.train.next_batch(batch_size * num_gpus * num_workers)\n", 375 | " #batch_x, batch_y = mnist.train.next_batch(batch_size * num_gpus)\n", 376 | " # Run optimization op (backprop)\n", 377 | " ts = time.time()\n", 378 | " sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})\n", 379 | " te = time.time() - ts\n", 380 | " if step % display_step == 0 or step == 1:\n", 381 | " # Calculate batch loss and accuracy\n", 382 | " loss, acc = sess.run([loss_op, accuracy], feed_dict={X: batch_x,\n", 383 | " Y: batch_y})\n", 384 | " print(\"Step \" + str(step) + \": Minibatch Loss= \" + \\\n", 385 | " \"{:.4f}\".format(loss) + \", Training Accuracy= \" + \\\n", 386 | " \"{:.3f}\".format(acc) + \", %i Examples/sec\" % int(len(batch_x)/te))\n", 387 | " step += 1\n", 388 | " print(\"Optimization Finished!\")\n", 389 | "\n", 390 | " # Calculate accuracy for 1000 mnist test images\n", 391 | " print(\"Testing Accuracy:\", \\\n", 392 | " np.mean([sess.run(accuracy, feed_dict={X: mnist.test.images[i:i+batch_size],\n", 393 | " Y: mnist.test.labels[i:i+batch_size]}) for i in range(0, len(mnist.test.images), batch_size)]))\n", 394 | " \n", 395 | " \n", 396 | "end = time.time()\n", 397 | "print(end-start)" 398 | ] 399 | }, 400 | { 401 | "cell_type": "markdown", 402 | "metadata": {}, 403 | "source": [ 404 | "1. single 2gpu \n", 405 | "\n", 406 | "Testing Accuracy: 0.9929468, 64.05570006370544 sec\n", 407 | "\n", 408 | "2. 2gpu * 2machine, 2x batch quantity\n", 409 | "\n", 410 | "Testing Accuracy: 0.99408084, 173.84509682655334 sec\n", 411 | "\n", 412 | "3. 2gpu * 2machine, no param server, 2x batch quantity\n", 413 | "\n", 414 | "Testing Accuracy: 0.994314, 142.5129895210266 sec\n", 415 | "\n", 416 | "4. 2gpu * 2machine, no param server, same batch quantity\n", 417 | "\n", 418 | "Testing Accuracy: 0.9941406, 119.99846577644348 sec" 419 | ] 420 | }, 421 | { 422 | "cell_type": "code", 423 | "execution_count": null, 424 | "metadata": {}, 425 | "outputs": [], 426 | "source": [] 427 | } 428 | ], 429 | "metadata": { 430 | "kernelspec": { 431 | "display_name": "Python 3", 432 | "language": "python", 433 | "name": "python3" 434 | }, 435 | "language_info": { 436 | "codemirror_mode": { 437 | "name": "ipython", 438 | "version": 3 439 | }, 440 | "file_extension": ".py", 441 | "mimetype": "text/x-python", 442 | "name": "python", 443 | "nbconvert_exporter": "python", 444 | "pygments_lexer": "ipython3", 445 | "version": "3.5.2" 446 | } 447 | }, 448 | "nbformat": 4, 449 | "nbformat_minor": 1 450 | } 451 | -------------------------------------------------------------------------------- /13_cnn_architecture.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/13_cnn_architecture.pdf -------------------------------------------------------------------------------- /14_RNN/images/ann/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/ann/activation_functions_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ann/activation_functions_plot.png -------------------------------------------------------------------------------- /14_RNN/images/ann/perceptron_iris_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ann/perceptron_iris_plot.png -------------------------------------------------------------------------------- /14_RNN/images/autoencoders/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/autoencoders/extracted_features_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/autoencoders/extracted_features_plot.png -------------------------------------------------------------------------------- /14_RNN/images/autoencoders/generated_digits_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/autoencoders/generated_digits_plot.png -------------------------------------------------------------------------------- /14_RNN/images/autoencoders/linear_autoencoder_pca_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/autoencoders/linear_autoencoder_pca_plot.png -------------------------------------------------------------------------------- /14_RNN/images/autoencoders/reconstruction_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/autoencoders/reconstruction_plot.png -------------------------------------------------------------------------------- /14_RNN/images/autoencoders/sparsity_loss_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/autoencoders/sparsity_loss_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/classification/cleaned_digit_example_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/cleaned_digit_example_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/confusion_matrix_errors_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/confusion_matrix_errors_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/confusion_matrix_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/confusion_matrix_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/error_analysis_digits_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/error_analysis_digits_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/more_digits_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/more_digits_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/noisy_digit_example_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/noisy_digit_example_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/precision_recall_vs_threshold_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/precision_recall_vs_threshold_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/precision_vs_recall_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/precision_vs_recall_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/roc_curve_comparison_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/roc_curve_comparison_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/roc_curve_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/roc_curve_plot.png -------------------------------------------------------------------------------- /14_RNN/images/classification/some_digit_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/classification/some_digit_plot.png -------------------------------------------------------------------------------- /14_RNN/images/cnn/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/cnn/china_horizontal.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/cnn/china_horizontal.png -------------------------------------------------------------------------------- /14_RNN/images/cnn/china_max_pool.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/cnn/china_max_pool.png -------------------------------------------------------------------------------- /14_RNN/images/cnn/china_original.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/cnn/china_original.png -------------------------------------------------------------------------------- /14_RNN/images/cnn/china_original_small.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/cnn/china_original_small.png -------------------------------------------------------------------------------- /14_RNN/images/cnn/china_vertical.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/cnn/china_vertical.png -------------------------------------------------------------------------------- /14_RNN/images/cnn/test_image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/cnn/test_image.png -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/decision_tree_decision_boundaries_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/decision_trees/decision_tree_decision_boundaries_plot.png -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/decision_tree_instability_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/decision_trees/decision_tree_instability_plot.png -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/iris_tree.dot: -------------------------------------------------------------------------------- 1 | digraph Tree { 2 | node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ; 3 | edge [fontname=helvetica] ; 4 | 0 [label="꽃잎 길이 (cm) <= 2.45\ngini = 0.667\nsamples = 150\nvalue = [50, 50, 50]\nclass = setosa", fillcolor="#e5813900"] ; 5 | 1 [label="gini = 0.0\nsamples = 50\nvalue = [50, 0, 0]\nclass = setosa", fillcolor="#e58139ff"] ; 6 | 0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ; 7 | 2 [label="꽃잎 너비 (cm) <= 1.75\ngini = 0.5\nsamples = 100\nvalue = [0, 50, 50]\nclass = versicolor", fillcolor="#39e58100"] ; 8 | 0 -> 2 [labeldistance=2.5, labelangle=-45, headlabel="False"] ; 9 | 3 [label="gini = 0.168\nsamples = 54\nvalue = [0, 49, 5]\nclass = versicolor", fillcolor="#39e581e5"] ; 10 | 2 -> 3 ; 11 | 4 [label="gini = 0.043\nsamples = 46\nvalue = [0, 1, 45]\nclass = virginica", fillcolor="#8139e5f9"] ; 12 | 2 -> 4 ; 13 | } -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/iris_tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/decision_trees/iris_tree.png -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/min_samples_leaf_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/decision_trees/min_samples_leaf_plot.png -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/regression_tree.dot: -------------------------------------------------------------------------------- 1 | digraph Tree { 2 | node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ; 3 | edge [fontname=helvetica] ; 4 | 0 [label="x1 <= 0.197\nmse = 0.098\nsamples = 200\nvalue = 0.354", fillcolor="#e5813953"] ; 5 | 1 [label="x1 <= 0.092\nmse = 0.038\nsamples = 44\nvalue = 0.689", fillcolor="#e58139c7"] ; 6 | 0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ; 7 | 2 [label="mse = 0.018\nsamples = 20\nvalue = 0.854", fillcolor="#e58139ff"] ; 8 | 1 -> 2 ; 9 | 3 [label="mse = 0.013\nsamples = 24\nvalue = 0.552", fillcolor="#e5813998"] ; 10 | 1 -> 3 ; 11 | 4 [label="x1 <= 0.772\nmse = 0.074\nsamples = 156\nvalue = 0.259", fillcolor="#e5813933"] ; 12 | 0 -> 4 [labeldistance=2.5, labelangle=-45, headlabel="False"] ; 13 | 5 [label="mse = 0.015\nsamples = 110\nvalue = 0.111", fillcolor="#e5813900"] ; 14 | 4 -> 5 ; 15 | 6 [label="mse = 0.036\nsamples = 46\nvalue = 0.615", fillcolor="#e58139ad"] ; 16 | 4 -> 6 ; 17 | } -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/regression_tree.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/decision_trees/regression_tree.png -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/sensitivity_to_rotation_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/decision_trees/sensitivity_to_rotation_plot.png -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/tree_regression_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/decision_trees/tree_regression_plot.png -------------------------------------------------------------------------------- /14_RNN/images/decision_trees/tree_regression_regularization_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/decision_trees/tree_regression_regularization_plot.png -------------------------------------------------------------------------------- /14_RNN/images/deep/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/deep/elu_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/deep/elu_plot.png -------------------------------------------------------------------------------- /14_RNN/images/deep/leaky_relu_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/deep/leaky_relu_plot.png -------------------------------------------------------------------------------- /14_RNN/images/deep/selu_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/deep/selu_plot.png -------------------------------------------------------------------------------- /14_RNN/images/deep/sigmoid_saturation_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/deep/sigmoid_saturation_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/dataset_2d_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/dataset_2d_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/dataset_3d_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/dataset_3d_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/kernel_pca_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/kernel_pca_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/lle_unrolling_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/lle_unrolling_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/manifold_decision_boundary_plot1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/manifold_decision_boundary_plot1.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/manifold_decision_boundary_plot2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/manifold_decision_boundary_plot2.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/manifold_decision_boundary_plot3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/manifold_decision_boundary_plot3.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/manifold_decision_boundary_plot4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/manifold_decision_boundary_plot4.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/mnist_compression_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/mnist_compression_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/other_dim_reduction_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/other_dim_reduction_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/pca_best_projection.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/pca_best_projection.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/preimage_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/preimage_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/squished_swiss_roll_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/squished_swiss_roll_plot.png -------------------------------------------------------------------------------- /14_RNN/images/dim_reduction/swiss_roll_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/dim_reduction/swiss_roll_plot.png -------------------------------------------------------------------------------- /14_RNN/images/distributed/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/attribute_histogram_plots.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/attribute_histogram_plots.png -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/bad_visualization_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/bad_visualization_plot.png -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/better_visualization_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/better_visualization_plot.png -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/california.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/california.png -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/california_housing_prices_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/california_housing_prices_plot.png -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/housing_prices_scatterplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/housing_prices_scatterplot.png -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/income_category_hist.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/income_category_hist.png -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/income_vs_house_value_scatterplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/income_vs_house_value_scatterplot.png -------------------------------------------------------------------------------- /14_RNN/images/end_to_end_project/scatter_matrix_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/end_to_end_project/scatter_matrix_plot.png -------------------------------------------------------------------------------- /14_RNN/images/ensembles/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/ensembles/boosting_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ensembles/boosting_plot.png -------------------------------------------------------------------------------- /14_RNN/images/ensembles/decision_tree_without_and_with_bagging_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ensembles/decision_tree_without_and_with_bagging_plot.png -------------------------------------------------------------------------------- /14_RNN/images/ensembles/early_stopping_gbrt_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ensembles/early_stopping_gbrt_plot.png -------------------------------------------------------------------------------- /14_RNN/images/ensembles/gbrt_learning_rate_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ensembles/gbrt_learning_rate_plot.png -------------------------------------------------------------------------------- /14_RNN/images/ensembles/gradient_boosting_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ensembles/gradient_boosting_plot.png -------------------------------------------------------------------------------- /14_RNN/images/ensembles/law_of_large_numbers_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ensembles/law_of_large_numbers_plot.png -------------------------------------------------------------------------------- /14_RNN/images/ensembles/mnist_feature_importance_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/ensembles/mnist_feature_importance_plot.png -------------------------------------------------------------------------------- /14_RNN/images/fundamentals/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/fundamentals/best_fit_model_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/fundamentals/best_fit_model_plot.png -------------------------------------------------------------------------------- /14_RNN/images/fundamentals/cyprus_prediction_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/fundamentals/cyprus_prediction_plot.png -------------------------------------------------------------------------------- /14_RNN/images/fundamentals/money_happy_scatterplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/fundamentals/money_happy_scatterplot.png -------------------------------------------------------------------------------- /14_RNN/images/fundamentals/overfitting_model_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/fundamentals/overfitting_model_plot.png -------------------------------------------------------------------------------- /14_RNN/images/fundamentals/representative_training_data_scatterplot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/fundamentals/representative_training_data_scatterplot.png -------------------------------------------------------------------------------- /14_RNN/images/fundamentals/ridge_model_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/fundamentals/ridge_model_plot.png -------------------------------------------------------------------------------- /14_RNN/images/fundamentals/tweaking_model_params_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/fundamentals/tweaking_model_params_plot.png -------------------------------------------------------------------------------- /14_RNN/images/rl/MsPacman.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/rl/MsPacman.png -------------------------------------------------------------------------------- /14_RNN/images/rl/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/rl/cart_pole_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/rl/cart_pole_plot.png -------------------------------------------------------------------------------- /14_RNN/images/rl/preprocessing_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/rl/preprocessing_plot.png -------------------------------------------------------------------------------- /14_RNN/images/rnn/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/rnn/creative_sequence_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/rnn/creative_sequence_plot.png -------------------------------------------------------------------------------- /14_RNN/images/rnn/time_series_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/rnn/time_series_plot.png -------------------------------------------------------------------------------- /14_RNN/images/rnn/time_series_pred_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/rnn/time_series_pred_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/svm/higher_dimensions_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/higher_dimensions_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/hinge_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/hinge_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/iris_3D_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/iris_3D_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/kernel_method_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/kernel_method_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/large_margin_classification_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/large_margin_classification_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/moons_kernelized_polynomial_svc_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/moons_kernelized_polynomial_svc_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/moons_polynomial_svc_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/moons_polynomial_svc_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/moons_rbf_svc_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/moons_rbf_svc_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/regularization_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/regularization_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/sensitivity_to_feature_scales_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/sensitivity_to_feature_scales_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/sensitivity_to_outliers_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/sensitivity_to_outliers_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/small_w_large_margin_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/small_w_large_margin_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/svm_regression_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/svm_regression_plot.png -------------------------------------------------------------------------------- /14_RNN/images/svm/svm_with_polynomial_kernel_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/svm/svm_with_polynomial_kernel_plot.png -------------------------------------------------------------------------------- /14_RNN/images/tensorflow/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/README: -------------------------------------------------------------------------------- 1 | Images generated by the notebooks 2 | -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/early_stopping_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/early_stopping_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/generated_data_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/generated_data_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/gradient_descent_paths_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/gradient_descent_paths_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/gradient_descent_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/gradient_descent_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/high_degree_polynomials_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/high_degree_polynomials_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/lasso_regression_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/lasso_regression_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/lasso_vs_ridge_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/lasso_vs_ridge_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/learning_curves_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/learning_curves_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/linear_model_predictions.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/linear_model_predictions.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/logistic_function_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/logistic_function_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/logistic_regression_contour_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/logistic_regression_contour_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/logistic_regression_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/logistic_regression_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/quadratic_data_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/quadratic_data_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/quadratic_predictions_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/quadratic_predictions_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/ridge_regression_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/ridge_regression_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/sgd_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/sgd_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/softmax_regression_contour_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/softmax_regression_contour_plot.png -------------------------------------------------------------------------------- /14_RNN/images/training_linear_models/underfitting_learning_curves_plot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/14_RNN/images/training_linear_models/underfitting_learning_curves_plot.png -------------------------------------------------------------------------------- /14_RNN/tensorflow_graph_in_jupyter.py: -------------------------------------------------------------------------------- 1 | from __future__ import absolute_import, division, print_function, unicode_literals 2 | 3 | # This module defines the show_graph() function to visualize a TensorFlow graph within Jupyter. 4 | 5 | # As far as I can tell, this code was originally written by Alex Mordvintsev at: 6 | # https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb 7 | 8 | # The original code only worked on Chrome (because of the use of , but the version below 9 | # uses Polyfill (copied from this StackOverflow answer: https://stackoverflow.com/a/41463991/38626) 10 | # so that it can work on other browsers as well. 11 | 12 | import numpy as np 13 | import tensorflow as tf 14 | from IPython.display import clear_output, Image, display, HTML 15 | 16 | def strip_consts(graph_def, max_const_size=32): 17 | """Strip large constant values from graph_def.""" 18 | strip_def = tf.GraphDef() 19 | for n0 in graph_def.node: 20 | n = strip_def.node.add() 21 | n.MergeFrom(n0) 22 | if n.op == 'Const': 23 | tensor = n.attr['value'].tensor 24 | size = len(tensor.tensor_content) 25 | if size > max_const_size: 26 | tensor.tensor_content = b""%size 27 | return strip_def 28 | 29 | def show_graph(graph_def, max_const_size=32): 30 | """Visualize TensorFlow graph.""" 31 | if hasattr(graph_def, 'as_graph_def'): 32 | graph_def = graph_def.as_graph_def() 33 | strip_def = strip_consts(graph_def, max_const_size=max_const_size) 34 | code = """ 35 | 36 | 41 | 42 |
43 | 44 |
45 | """.format(data=repr(str(strip_def)), id='graph'+str(np.random.rand())) 46 | 47 | iframe = """ 48 | 49 | """.format(code.replace('"', '"')) 50 | display(HTML(iframe)) 51 | -------------------------------------------------------------------------------- /6 결정트리.pptx: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/machinelearning-pangyo/Hands-On-MachineLearning/a85fa6d79987bee1485a329ef219bd4b1e9ea14a/6 결정트리.pptx -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Hands-On-MachineLearning 2 | 3 | * 내용 소개 TBD 4 | 5 | ## 교재 6 | 7 | * 핸즈온 머신러닝(사이킷런과 텐서플로를 활용한 머신러닝, 딥러닝 실무), 저자 Aurélien Géron, 옮긴이 박해선 8 | * [저자 code GitHub](https://github.com/ageron/handson-ml), [옮긴이 code GitHub](https://github.com/rickiepark/handson-ml) 9 | * [교재 errata](https://tensorflow.blog/%ED%95%B8%EC%A6%88%EC%98%A8-%EB%A8%B8%EC%8B%A0%EB%9F%AC%EB%8B%9D/) 10 | 11 | ## 커리큘럼 12 | 13 | | 날짜 | 주제 | 발표자 | 발표자료 | 14 | |:---:|-----|:----:|:------:| 15 | | 2018.05.24(목) |1. 한눈에 보는 머신 러닝 | 박종민 | [[pdf]](https://github.com/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/01_the_machine_learning_landscape.pdf), [[link]](https://goo.gl/6qFTEB) | 16 | | 2018.05.29(화) |2. 머신러닝 프로젝트 처음부터 끝까지 | 박성재 | [[colab]](https://colab.research.google.com/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/02_end_to_end/housing.ipynb), [[nbviewer]](http://nbviewer.jupyter.org/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/02_end_to_end/housing.ipynb) | 17 | | 2018.06.05(화) |3. 분류 | 이규영 | | 18 | | 2018.06.12(화) |4. 모델 훈련 | 박종민 | [[colab]](https://colab.research.google.com/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/04_training_linear_models.ipynb), [[nbviewer]](http://nbviewer.jupyter.org/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/04_training_linear_models.ipynb) | 19 | | 2018.06.28(목) |5. SVM | 손주형 | [[colab]](https://colab.research.google.com/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/05_SVM/05_SVM.ipynb), [[nbviewer]](http://nbviewer.jupyter.org/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/05_SVM/05_SVM.ipynb) | 20 | | 2018.06.19(화) |6. 결정트리 | 이규영 | | 21 | | 2018.06.19(화) |7. 앙상블 & 랜덤 포레스트 | 이규영 | | 22 | | 2018.07.03(화) |8. 차원축소 | 송치성 | [[colab]](https://colab.research.google.com/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/08_dimensionality_reduction.ipynb), [[nbviewer]](https://nbviewer.jupyter.org/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/08_dimensionality_reduction.ipynb) | 23 | | 2018.07.10(화) |9. 텐서플로 시작하기 | 조희주 | | 24 | | 2018.07.17(화) |10. 인공 신경망 | 전용진 | [[pdf]](https://github.com/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/10_Introduction_to_neural_network.pdf), [[link]](https://drive.google.com/open?id=1Z4vQ48zAkZhNJDTCEtkTe6YsYbr54YCI)| 25 | | 2018.07.24(화) |11. DNN 훈련 | 송영숙 | [[colab]](https://colab.research.google.com/drive/1OhdZAQMMP2XE3J1gpH0WjSGe82-W4iG_), [[nbviewer]](http://nbviewer.jupyter.org/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/11_DNN/11_deep_learning.ipynb) 26 | | 2018.07.31(화) |12. 다중 머신을 위한 분산 텐서플로 | 김대영 | [[colab]](https://colab.research.google.com/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/12_distributed/12_distributed_tensorflow.ipynb), [[nbviewer]](https://nbviewer.jupyter.org/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/12_distributed/12_distributed_tensorflow.ipynb) | 27 | | 2018.08.21(화) |13. CNN | 신형준, 김준호 | [[colab]](https://colab.research.google.com/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/13_CNN.ipynb), [[nbviewer]](https://nbviewer.jupyter.org/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/13_CNN.ipynb), [[pdf]](https://github.com/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/13_cnn_architecture.pdf), [[ResNet_tf]](https://github.com/taki0112/ResNet-Tensorflow) | 28 | | 2018.08.21(화) |14. RNN | 김소윤 | [[colab]](https://github.com/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/14_RNN/14_recurrent_neural_networks.ipynb), [[nbviewer]](https://nbviewer.jupyter.org/github/machinelearning-pangyo/Hands-On-MachineLearning/blob/master/14_RNN/14_recurrent_neural_networks.ipynb) | 29 | | 2018.08.28(화) |15. Autoencoder | 백현기 | | 30 | | 2018.09.04(화) |16. 강화학습 | | | 31 | --------------------------------------------------------------------------------