├── AutoGIS ├── 00-Useful-Library-and-Resource.ipynb ├── 01-motivation.ipynb └── 02-Geometric-Objects-Spatial-Data-Model.ipynb ├── Google_Cloud_Platform ├── 01. Overview.ipynb ├── 02. BigQuery.ipynb ├── 03. Datalab.ipynb ├── 04. bigtable.ipynb ├── 05. 구글 클라우드 SDK 여러 계정 설정.ipynb ├── 06. CloudML.ipynb ├── 07.Apache-Beam.md ├── 07.Dataflow │ ├── apache-beam-tutorial.py │ └── bigquery_to_dataflow.py ├── End-to-End-Machine-Learning-with-TF-on-GCP │ ├── .ipynb_checkpoints │ │ ├── BigQueryML-checkpoint.ipynb │ │ ├── lab04-preprocessing-using-dataflow-checkpoint.ipynb │ │ ├── lab05-AI-Platform(CloudML)-checkpoint.ipynb │ │ └── lab06-deploy-and-predict-with-model-checkpoint.ipynb │ ├── BigQueryML.ipynb │ ├── babyweight │ │ ├── PKG-INFO │ │ ├── setup.cfg │ │ ├── setup.py │ │ └── trainer │ │ │ ├── model.py │ │ │ └── task.py │ ├── lab04-preprocessing-using-dataflow.ipynb │ ├── lab05-AI-Platform(CloudML).ipynb │ ├── lab06-deploy-and-predict-with-model.ipynb │ └── lab07-App-Engine-app-to-serve-ML-predictions │ │ ├── README.md │ │ ├── application │ │ ├── README.md │ │ ├── app.yaml │ │ ├── appengine_config.py │ │ ├── docs │ │ │ └── img │ │ │ │ ├── cloud-shell.png │ │ │ │ └── screenshot.png │ │ ├── main.py │ │ ├── requirements.txt │ │ └── templates │ │ │ ├── form.html │ │ │ └── index.html │ │ ├── deploy.sh │ │ ├── exampledata.csv.gz │ │ ├── pipeline │ │ ├── pom.xml │ │ └── src │ │ │ └── main │ │ │ └── java │ │ │ └── com │ │ │ └── google │ │ │ └── cloud │ │ │ └── training │ │ │ └── mlongcp │ │ │ ├── AddPrediction.java │ │ │ ├── Baby.java │ │ │ ├── BabyPred.java │ │ │ ├── BabyweightMLService.java │ │ │ ├── InputOutput.java │ │ │ ├── PubSubBigQuery.java │ │ │ └── TextInputOutput.java │ │ ├── python_streaming.ipynb │ │ ├── run_dataflow.sh │ │ ├── run_once.sh │ │ └── run_ontext.sh └── img │ ├── Google Cloud Platform Products chart.png │ ├── config_1.png │ ├── config_2.png │ ├── config_3.png │ ├── config_4.png │ └── google_cloud_mlengine_01.png ├── Image-Processing ├── 01. load image.ipynb ├── 02. mp4 to jpg.ipynb ├── README.md ├── face_emotion.ipynb ├── image_handling.ipynb ├── image_viewer.py ├── main.py ├── mouse_callback.py ├── opencv.py ├── opencv_refactoring.py ├── test.jpeg ├── test1.jpg ├── test2.png └── xception_model.py ├── Julia ├── Julia Basic.ipynb ├── data.csv ├── data.txt ├── julia-with-or.ipynb ├── julia_tutorial.ipynb ├── optimization_example.jl ├── plot1.png ├── plot2.png ├── plot3.png └── results1.txt ├── LICENSE ├── Linux ├── 1장.리눅스-쉘과-명령어-기초.md ├── 2장.쉘-스크립트-맛보기.md ├── Crontab, Screen.md ├── function.sh ├── readonly 파일 저장.md ├── scp.ipynb ├── setting.md ├── shell_parameter.sh ├── test.sh ├── while.sh └── 압축.md ├── Network ├── README.md └── Summary.ipynb ├── README.md ├── SQL_performance_tuning ├── 00. SQL 실행 순서.ipynb └── 01. Index.ipynb ├── Tableau ├── 01. Tableau 기초.ipynb ├── 02. Searchbar 만들기 (Input 상자).ipynb ├── 03. Retention .ipynb ├── 04. 테마별 상위 5위.ipynb └── 05. 값이 비었을 경우 0으로 설정하기.ipynb ├── Tensorflow-Extended └── TFDV(data validation) example.ipynb ├── Tensorflow ├── Dataset.ipynb ├── Estimator(Custom).ipynb ├── High-Level-API-Eager-Execution.ipynb └── mnist.ipynb ├── Time-Series └── 01. Time-Series-Basic.ipynb ├── deep_learning_from_scratch ├── Lecture note.ipynb ├── Lecture note_Chapter 5 ~.ipynb ├── dataset │ ├── __init__.py │ ├── __pycache__ │ │ ├── __init__.cpython-36.pyc │ │ └── mnist.cpython-36.pyc │ ├── lena.png │ ├── lena_gray.png │ ├── mnist.py │ ├── t10k-images-idx3-ubyte.gz │ ├── t10k-labels-idx1-ubyte.gz │ ├── train-images-idx3-ubyte.gz │ └── train-labels-idx1-ubyte.gz ├── deep_convnet_params.pkl ├── params.pkl └── sample_weight.pkl ├── docker ├── README.md └── docker-compose.yml ├── hands-on-machine-learning └── 09.tensorflow.ipynb ├── information_graphics ├── 1장. 기본 원칙.ipynb ├── 2장. 똑똑하게 차트 그리기.ipynb └── flowchartmod.jpg ├── mac ├── README.md └── intellij.md ├── nlp ├── 1. Namuwiki json dump into plain text.ipynb ├── 2. Pos Tagging - Creating corpus.ipynb ├── 3. Word2vec - Gensim.ipynb └── Namuwiki-word2vec.ipynb ├── python ├── Bayesian-Optimization(skopt).ipynb ├── Downloading a picture via urllib and python.ipynb ├── Effective Python.ipynb ├── Face-Point-Detection.ipynb ├── Multiprocessing-Multithread.ipynb ├── Multiprocessing-pool.ipynb ├── No module named 'MySQLdb' error.ipynb ├── Select Multiple value in list.ipynb ├── Slack bot ( Slacker ).ipynb ├── Sliding-window-hold.ipynb ├── aiohttp-example.py ├── airflow.ipynb ├── asyncio-coroutine.py ├── asyncio-helloy.py ├── asyncio-sleep.py ├── austin_bikeshare_20191101.csv ├── bayesian-optimization.ipynb ├── cotyledon-daemon.py ├── cotyledon-producer-consumer.py ├── cotyledon-reconfig-process.py ├── crawling-google-play.ipynb ├── crawling │ ├── advanced_link_crawler.py │ ├── diskcache.py │ ├── downloader.py │ ├── redis_cache.py │ └── throttle.py ├── dask.ipynb ├── effective-python │ ├── better-way-29.py │ ├── better-way-30.py │ ├── better-way-31.py │ ├── better-way-32.py │ ├── better-way-36.py │ ├── better-way-56-utils_test.py │ ├── better-way-58-profile.py │ └── utils.py ├── fire_module.py ├── flask │ ├── README.md │ ├── ch1 │ │ ├── app │ │ │ ├── __init__.py │ │ │ ├── __pycache__ │ │ │ │ ├── __init__.cpython-36.pyc │ │ │ │ ├── controllers.cpython-36.pyc │ │ │ │ ├── database.cpython-36.pyc │ │ │ │ └── models.cpython-36.pyc │ │ │ ├── controllers.py │ │ │ ├── database.py │ │ │ ├── models.py │ │ │ ├── static │ │ │ │ ├── css │ │ │ │ │ ├── bootstrap-theme.css │ │ │ │ │ ├── bootstrap-theme.css.map │ │ │ │ │ ├── bootstrap-theme.min.css │ │ │ │ │ ├── bootstrap-theme.min.css.map │ │ │ │ │ ├── bootstrap.css │ │ │ │ │ ├── bootstrap.css.map │ │ │ │ │ ├── bootstrap.min.css │ │ │ │ │ └── bootstrap.min.css.map │ │ │ │ ├── fonts │ │ │ │ │ ├── glyphicons-halflings-regular.eot │ │ │ │ │ ├── glyphicons-halflings-regular.svg │ │ │ │ │ ├── glyphicons-halflings-regular.ttf │ │ │ │ │ ├── glyphicons-halflings-regular.woff │ │ │ │ │ └── glyphicons-halflings-regular.woff2 │ │ │ │ └── js │ │ │ │ │ ├── bootstrap.js │ │ │ │ │ ├── bootstrap.min.js │ │ │ │ │ └── npm.js │ │ │ └── templates │ │ │ │ ├── index.html │ │ │ │ ├── login.html │ │ │ │ └── profile.html │ │ └── run.py │ ├── hello.py │ ├── minitier │ │ ├── README.md │ │ └── app.py │ └── templates │ │ └── login_form.html ├── futures-process-worker.py ├── futures-threads-worker.py ├── futurist-check-and-reject.py ├── futurist-periodics.py ├── futurist-threads-worker.py ├── generator, yield, iterable, iterator.ipynb ├── hide input code in jupyter notebook.ipynb ├── image processing.ipynb ├── image │ ├── mytable.png │ ├── retrica.png │ └── retrica_1.png ├── json key value switching.ipynb ├── jupyter notebook TIP.ipynb ├── jupyter-notebook-in-jekyll.ipynb ├── keras_tutorial.ipynb ├── list of list flatten.ipynb ├── list(string type) to list(list) - using eval.ipynb ├── logging.ipynb ├── machinelearing-with-flask │ ├── README.md │ ├── app.py │ ├── build_model.py │ ├── lib │ │ ├── data │ │ │ ├── sampleSubmission.csv │ │ │ ├── test.tsv │ │ │ └── train.tsv │ │ └── models │ │ │ ├── SentimentClassifier.pkl │ │ │ └── TFIDFVectorizer.pkl │ ├── model.py │ ├── requirements.txt │ └── util.py ├── make UUID.ipynb ├── matplolib.ipynb ├── matplotlib style custom ( retina style ).ipynb ├── or-tools(solver)-tutorial.ipynb ├── pandas dataframe ( using image ).ipynb ├── pandas multiple filtering by row.ipynb ├── pandas 병렬처리.ipynb ├── pandas-datetime.ipynb ├── pandas.ipynb ├── papermill_example.ipynb ├── papermill_example_output.ipynb ├── papermill_execute.ipynb ├── property.ipynb ├── prophet-tutorial.ipynb ├── pydeck-example.ipynb ├── pyecharts.html ├── pyecharts.ipynb ├── pyspark.ipynb ├── reverse-geocoding.ipynb ├── sacred │ ├── actual_use │ │ ├── experiments │ │ │ ├── random_forest.py │ │ │ └── svc.py │ │ └── main.py │ ├── capture_functions.py │ ├── custom_logger.py │ ├── first_config.py │ ├── ingredient.py │ ├── my_command.py │ └── randomness.py ├── sacred_tutorial(experiment-support).ipynb ├── scheduler.py ├── selenium ( install chrome driver with mac ).ipynb ├── simulator.ipynb ├── spreadsheet_web_auth.ipynb ├── static method - @classmethod, @staticmethod.ipynb ├── tdd │ ├── application │ │ ├── app.py │ │ ├── requirements.txt │ │ └── test │ │ │ └── system │ │ │ ├── base_test.py │ │ │ └── test_home.py │ └── blog │ │ ├── app.py │ │ ├── blog.py │ │ ├── post.py │ │ └── tests │ │ ├── integration │ │ └── blog_test.py │ │ ├── system │ │ └── app_test.py │ │ └── unit │ │ ├── blog_test.py │ │ └── post_test.py ├── tensorflow-1.ipynb ├── tensorflow-2.ipynb ├── virtualenv setting.ipynb ├── virtualenvwrapper.ipynb ├── visualization(cufflinks).ipynb └── xlsxWriter.ipynb ├── pytorch ├── 00. pytorch_summary.ipynb ├── 01. pytorch_basic.ipynb ├── 02. linear regression.ipynb ├── 03.logistic-regression.ipynb ├── 04.feedforward_neural_network.ipynb ├── 05.CNN.ipynb ├── 06.deep-residual-net.ipynb ├── GAN.ipynb ├── Learning PyTorch with Examples.ipynb ├── NLP-1.ipynb ├── Transfer-Learning.ipynb ├── Tutorial.ipynb └── week3.ipynb ├── reinforcement_learning ├── All for deeplearing │ ├── Lab 02. Playing OpenAI GYM Games.py │ ├── Lab 03. Dummy Q-learning(table).py │ ├── Lab 04. Q-learning (exploit&exploration and discounted reward).py │ ├── Lab 05-1. Q-learning on Nondeterministic Worlds.py │ └── Lab 05-2. Q-learning on Nondeterministic Worlds.py ├── img │ ├── circle.png │ ├── down.png │ ├── left.png │ ├── rectangle.png │ ├── right.png │ ├── triangle.png │ └── up.png └── textbook summary │ ├── 01. 강화학습의 개요.ipynb │ ├── 02. 강화학습 기초(1) - MDP와 벨만 방정식.ipynb │ ├── 03. policy_iteration.py │ ├── 03. 강화학습 기초(2) - 그리드월드와 다이나믹 프로그래밍.ipynb │ └── environment.py ├── scala ├── 01.object.ipynb ├── 02.class.ipynb ├── 03.trait.ipynb ├── 04.Generic.ipynb ├── 05.Collection-Library.ipynb ├── 06.Collection-Library(2).ipynb └── 07.Type Class.ipynb └── scikit-learn ├── Chapter 1. Introduction.ipynb └── Chapter 2. Supervised Learning.ipynb /AutoGIS/01-motivation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "### 우리의 목표\n", 8 | "- 다양한 파일 형식에서 다른 공간 형식으로 공간 데이터 읽기 / 쓰기\n", 9 | "- 다른 Projection을 다루기\n", 10 | "- 다양한 기하 연산 및 지오 코딩 수행\n", 11 | "- 다른 criteria(영역)에 따라 데이터를 재 분류\n", 12 | "- 공간 쿼리 수행\n", 13 | "- 간단한 공간 분석 수행\n", 14 | "- 다음과 같이 데이터를 시각화하고 (대화 형) 지도 생성" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "### Python에서 GIS를 하는 이유\n", 22 | "- 모두 무료\n", 23 | "- 다른 geporocessing 작동 방식을 더 깊이 배움\n", 24 | "- 파이썬은 매우 효율적이고 유연함\n", 25 | "- 파이썬 오픈소스를 지원\n", 26 | "- PostGIS와 함께 GeoDjango를 백엔드로 사용하는 등 다양한 것을 결합 가능" 27 | ] 28 | }, 29 | { 30 | "cell_type": "markdown", 31 | "metadata": {}, 32 | "source": [ 33 | "### 데이터 분석 및 시각화 :\n", 34 | "- Numpy -> 파이썬으로 과학 컴퓨팅을위한 기본 패키지\n", 35 | "- Pandas -> 고성능, 사용하기 쉬운 데이터 구조 및 데이터 분석 도구\n", 36 | "- Scipy -> 신호 처리, 최적화 및 통계를 포함한 수치 알고리즘 및 도메인 별 도구 상자 모음\n", 37 | "- Matplotlib -> 파이썬 용 기본 플로팅 라이브러리\n", 38 | "- Bokeh -> 대화식 시각화 웹 (지도)\n", 39 | "- Plotly -> 웹용 대화 형 시각화 (지도) (교육용으로 상업용 없음)\n", 40 | "\n", 41 | "### GIS \n", 42 | "- GDAL -> 벡터 및 래스터 데이터 형식을 처리하기위한 기본 패키지 (많은 모듈이 GDAL에 의존적임). 래스터 처리에 사용됩니다\n", 43 | "- Geopandas -> Python에서 지형 공간 데이터로 작업하는 것이 더 쉽게 이루어졌으며 pandas의 기능과 shapely 기능을 결합했습니다\n", 44 | "- Shapely -> 평면 기하학적 객체의 조작 및 분석을위한 Python 패키지 (널리 배치 된 GEOS를 기반으로 함)\n", 45 | "- Fiona -> 공간 데이터 읽기 및 쓰기 (Geopandas의 대안)\n", 46 | "- Pyproj -> 지도 제작 변형 및 측지 계산을 수행합니다 (PROJ.4 기준)\n", 47 | "- Pysal -> 파이썬으로 작성된 공간 분석 함수의 라이브러리\n", 48 | "- Geopy -> 지오 코딩 라이브러리 <-> 주소를 좌표로 지정\n", 49 | "- Contextily -> (정적) 맵 시각화를위한 배경 기본 맵 추가\n", 50 | "- GeoViews -> 대화식 웹지도\n", 51 | "- Geoplot -> Python을 위한 고차원 지형 공간 데이터 시각화 라이브러리\n", 52 | "- Dash -> Dash는 분석 웹 응용 프로그램을 작성하기위한 Python 프레임 워크입니다\n", 53 | "- OSMnx -> 거리 네트워크 용 Python. OpenStreetMap에서 거리 네트워크 검색, 구성, 분석 및 시각화\n", 54 | "- Networkx -> Python의 네트워크 분석 및 라우팅 (예 : Dijkstra 및 A * 알고리즘)\n", 55 | "- Cartopy -> 데이터 분석 및 시각화를 위해 쉽게 지도를 그릴 수 있음\n", 56 | "- Scipy.spatial -> 공간 알고리즘 및 데이터 구조\n", 57 | "- Rtree -> 빠른 공간 검색을 위한 Python의 공간 색인화\n", 58 | "- Rasterio -> Python을위한 깨끗하고 빠르며 지형적인 래스터 I / O\n", 59 | "- RSGISLib -> Python 용 원격 감지 및 GIS 소프트웨어 라이브러리" 60 | ] 61 | }, 62 | { 63 | "cell_type": "markdown", 64 | "metadata": {}, 65 | "source": [ 66 | "### Lesson Overview\n", 67 | "- Geometric Objects\n", 68 | " - Shapely 사용\n", 69 | " Point\n", 70 | " LineString\n", 71 | " Polygon\n", 72 | " Geometry collections\n" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": null, 78 | "metadata": {}, 79 | "outputs": [], 80 | "source": [] 81 | } 82 | ], 83 | "metadata": { 84 | "kernelspec": { 85 | "display_name": "Python 3", 86 | "language": "python", 87 | "name": "python3" 88 | }, 89 | "language_info": { 90 | "codemirror_mode": { 91 | "name": "ipython", 92 | "version": 3 93 | }, 94 | "file_extension": ".py", 95 | "mimetype": "text/x-python", 96 | "name": "python", 97 | "nbconvert_exporter": "python", 98 | "pygments_lexer": "ipython3", 99 | "version": "3.7.3" 100 | }, 101 | "varInspector": { 102 | "cols": { 103 | "lenName": 16, 104 | "lenType": 16, 105 | "lenVar": 40 106 | }, 107 | "kernels_config": { 108 | "python": { 109 | "delete_cmd_postfix": "", 110 | "delete_cmd_prefix": "del ", 111 | "library": "var_list.py", 112 | "varRefreshCmd": "print(var_dic_list())" 113 | }, 114 | "r": { 115 | "delete_cmd_postfix": ") ", 116 | "delete_cmd_prefix": "rm(", 117 | "library": "var_list.r", 118 | "varRefreshCmd": "cat(var_dic_list()) " 119 | } 120 | }, 121 | "types_to_exclude": [ 122 | "module", 123 | "function", 124 | "builtin_function_or_method", 125 | "instance", 126 | "_Feature" 127 | ], 128 | "window_display": false 129 | } 130 | }, 131 | "nbformat": 4, 132 | "nbformat_minor": 2 133 | } 134 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/01. Overview.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Google Cloud Platform" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "- 구글 클라우드는 구글의 [데이터센터](https://www.google.com/about/datacenters/) 인프라를 기반으로 컴퓨트, 스토리지, 네트워킹, 머신러닝 등의 서비스를 제공하는 글로벌 클라우드\n", 15 | "- Regions : Central US, Western Europe, East Asia 존재하며 세부적으로 zones으로 구성\n", 16 | "\n", 17 | "![regions-zones](https://cloud.google.com/docs/images/overview/regions-zones.svg)\n", 18 | "\n", 19 | "\n", 20 | "### 구글 클라우드만의 특징\n", 21 | "- 빅데이터와 머신러닝 서비스\n", 22 | "- 구글 전용 네트워크를 이용한 글로벌 커버리지\n", 23 | "- 저렴한 가격 모델\n", 24 | "\n", 25 | "\n" 26 | ] 27 | }, 28 | { 29 | "cell_type": "markdown", 30 | "metadata": {}, 31 | "source": [ 32 | "# Google Cloud Platform Services\n", 33 | "\n", 34 | "![Google Cloud Platform Products](https://cloud.google.com/solutions/images/iot-overview-architecture.png)\n", 35 | "\n", 36 | "## 1. Big data services\n", 37 | "### Data Analysis\n", 38 | "### [BigQuery](https://cloud.google.com/bigquery/what-is-bigquery)\n", 39 | "- 대규모 저장 및 분석 플랫폼으로 일종의 데이터 웨어하우스, 문법은 SQL와 유사\n", 40 | "- 속도에 최적화되어 있음\n", 41 | "\n", 42 | "[빅쿼리 시작하기](https://cloud.google.com/bigquery/quickstart-web-ui)\n", 43 | "\n", 44 | "[Kubernetes, Redis, BigQuery를 이용한 실시간 분석](https://cloud.google.com/solutions/real-time/kubernetes-redis-bigquery)\n", 45 | "\n", 46 | "\n", 47 | "### Batch and streaming data processing\n", 48 | "### [Cloud Dataflow](https://cloud.google.com/dataflow/what-is-google-cloud-dataflow)\n", 49 | "- 아파치 스파크, 플링크와 같이 실시간 스트리밍 분석 및 배치 분석을 지원\n", 50 | "- 수집한 데이터를 변환하거나 여러 데이터 소스와 저장소간의 연결(라우팅)을 담당\n", 51 | "- ETL(extract - transform - load) 업무에도 좋음\n", 52 | "\n", 53 | "### Asynchronous messaging\n", 54 | "### [Cloud Pub/Sub](https://cloud.google.com/pubsub/docs)\n", 55 | "- 비동기(asyn) 메세징 서비스\n", 56 | "- 카프카와 같은 대규모 큐잉 시스템으로 데이터를 대규모로 수집\n", 57 | "\n", 58 | "### ETC\n", 59 | "### Cloud Datalab\n", 60 | "- Jupyter notebook의 클라우드 버전\n", 61 | "- 웹 기반의 저작 도구\n", 62 | "\n", 63 | "### Dataproc\n", 64 | "- 하둡 / 스파크의 매니지드 서비스로 클러스터를 90초안에 배포\n", 65 | "\n", 66 | "### Data Studio\n", 67 | "- SQL / 빅쿼리 등의 데이터 소스를 기반으로 시각적인 보고서를 생성해주는 제품\n", 68 | "\n" 69 | ] 70 | }, 71 | { 72 | "cell_type": "markdown", 73 | "metadata": {}, 74 | "source": [ 75 | "## [Python on Google Cloud](https://cloud.google.com/python/)" 76 | ] 77 | }, 78 | { 79 | "cell_type": "markdown", 80 | "metadata": {}, 81 | "source": [ 82 | "## 영어 단어\n", 83 | "- underlying : (외부로 드러나지 않는) 근본적인\n", 84 | " " 85 | ] 86 | }, 87 | { 88 | "cell_type": "code", 89 | "execution_count": null, 90 | "metadata": { 91 | "collapsed": true 92 | }, 93 | "outputs": [], 94 | "source": [] 95 | } 96 | ], 97 | "metadata": { 98 | "kernelspec": { 99 | "display_name": "Python 3", 100 | "language": "python", 101 | "name": "python3" 102 | }, 103 | "language_info": { 104 | "codemirror_mode": { 105 | "name": "ipython", 106 | "version": 3 107 | }, 108 | "file_extension": ".py", 109 | "mimetype": "text/x-python", 110 | "name": "python", 111 | "nbconvert_exporter": "python", 112 | "pygments_lexer": "ipython3", 113 | "version": "3.5.2" 114 | } 115 | }, 116 | "nbformat": 4, 117 | "nbformat_minor": 0 118 | } 119 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/04. bigtable.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# bigtable\n" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "bigtable : A high performance NoSQL database service\n", 15 | "\n", 16 | "- consistent low latency and high throughput ( 대기시간은 낮게 일관되며, 처리량은 많음 )\n", 17 | "- Simple & Integrated : Hadoop + Google Cloud Platform + Dataflow + Dataproc\n", 18 | "\n", 19 | "\n", 20 | "## Bigtable Features\n", 21 | "- High Performance\n", 22 | "- Security & Permissions\n", 23 | "- Low Latency Storage ( 일관된 낮은 대기시간 )\n", 24 | "- Global Availability\n", 25 | "- Fully Managed \n", 26 | "- Redundant Autoscaling Storage ( 중복 내부저장 스토리지 )\n", 27 | "- Scaling\n", 28 | "- HBase Compatible\n" 29 | ] 30 | }, 31 | { 32 | "cell_type": "code", 33 | "execution_count": null, 34 | "metadata": { 35 | "collapsed": true 36 | }, 37 | "outputs": [], 38 | "source": [] 39 | } 40 | ], 41 | "metadata": { 42 | "kernelspec": { 43 | "display_name": "Python 3", 44 | "language": "python", 45 | "name": "python3" 46 | }, 47 | "language_info": { 48 | "codemirror_mode": { 49 | "name": "ipython", 50 | "version": 3 51 | }, 52 | "file_extension": ".py", 53 | "mimetype": "text/x-python", 54 | "name": "python", 55 | "nbconvert_exporter": "python", 56 | "pygments_lexer": "ipython3", 57 | "version": "3.6.0" 58 | } 59 | }, 60 | "nbformat": 4, 61 | "nbformat_minor": 2 62 | } 63 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/07.Apache-Beam.md: -------------------------------------------------------------------------------- 1 | # Apache Beam 2 | - Dataflow를 사용하기 위해 Apache Beam에 대해 기록 3 | - [공식 문서](https://beam.apache.org/documentation/) 4 | 5 | - [프로그래밍 가이드](https://beam.apache.org/documentation/programming-guide/) 6 | - [실행 모델](https://beam.apache.org/documentation/execution-model/) 7 | - [추가 자료](https://beam.apache.org/documentation/resources/) 8 | 9 | 10 | 11 | ## 파이프라인 기본 요소 12 | - [파이프라인 설계](https://beam.apache.org/documentation/pipelines/design-your-pipeline/) 13 | - [파이프라인 생성](https://beam.apache.org/documentation/pipelines/create-your-pipeline/) 14 | - [파이프라인 테스트](https://beam.apache.org/documentation/pipelines/test-your-pipeline/) 15 | 16 | 17 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/babyweight/PKG-INFO: -------------------------------------------------------------------------------- 1 | Metadata-Version: 1.0 2 | Name: trainer 3 | Version: 0.1 4 | Summary: Baby Weight Example 5 | Home-page: UNKNOWN 6 | Author: Google 7 | Author-email: lak@google.com 8 | License: Apache 9 | Description: UNKNOWN 10 | Platform: UNKNOWN 11 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/babyweight/setup.cfg: -------------------------------------------------------------------------------- 1 | [egg_info] 2 | tag_build = 3 | tag_date = 0 4 | tag_svn_revision = 0 5 | 6 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/babyweight/setup.py: -------------------------------------------------------------------------------- 1 | # Copyright 2017 Google Inc. All Rights Reserved. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from setuptools import find_packages 16 | from setuptools import setup 17 | 18 | REQUIRED_PACKAGES = [ 19 | ] 20 | 21 | setup( 22 | name='babyweight', 23 | version='0.1', 24 | author = 'V Lakshmanan', 25 | author_email = 'lak@cloud.google.com', 26 | install_requires=REQUIRED_PACKAGES, 27 | packages=find_packages(), 28 | include_package_data=True, 29 | description='Baby Weight prediction in Cloud ML', 30 | requires=[] 31 | ) 32 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/babyweight/trainer/task.py: -------------------------------------------------------------------------------- 1 | import argparse 2 | import json 3 | import os 4 | 5 | from . import model 6 | 7 | import tensorflow as tf 8 | 9 | if __name__ == '__main__': 10 | parser = argparse.ArgumentParser() 11 | parser.add_argument( 12 | '--bucket', 13 | help = 'GCS path to data. We assume that data is in gs://BUCKET/babyweight/preproc/', 14 | required = True 15 | ) 16 | parser.add_argument( 17 | '--output_dir', 18 | help = 'GCS location to write checkpoints and export models', 19 | required = True 20 | ) 21 | parser.add_argument( 22 | '--batch_size', 23 | help = 'Number of examples to compute gradient over.', 24 | type = int, 25 | default = 512 26 | ) 27 | parser.add_argument( 28 | '--job-dir', 29 | help = 'this model ignores this field, but it is required by gcloud', 30 | default = 'junk' 31 | ) 32 | parser.add_argument( 33 | '--nnsize', 34 | help = 'Hidden layer sizes to use for DNN feature columns -- provide space-separated layers', 35 | nargs = '+', 36 | type = int, 37 | default=[128, 32, 4] 38 | ) 39 | parser.add_argument( 40 | '--nembeds', 41 | help = 'Embedding size of a cross of n key real-valued parameters', 42 | type = int, 43 | default = 3 44 | ) 45 | 46 | ## TODO 1: add the new arguments here 47 | parser.add_argument( 48 | '--train_examples', 49 | help = 'Number of examples (in thousands) to run the training job over. If this is more than actual # of examples available, it cycles through them. So specifying 1000 here when you have only 100k examples makes this 10 epochs.', 50 | type = int, 51 | default = 5000 52 | ) 53 | parser.add_argument( 54 | '--pattern', 55 | help = 'Specify a pattern that has to be in input files. For example 00001-of will process only one shard', 56 | default = 'of' 57 | ) 58 | parser.add_argument( 59 | '--eval_steps', 60 | help = 'Positive number of steps for which to evaluate model. Default to None, which means to evaluate until input_fn raises an end-of-input exception', 61 | type = int, 62 | default = None 63 | ) 64 | 65 | ## parse all arguments 66 | args = parser.parse_args() 67 | arguments = args.__dict__ 68 | 69 | # unused args provided by service 70 | arguments.pop('job_dir', None) 71 | arguments.pop('job-dir', None) 72 | 73 | ## assign the arguments to the model variables 74 | output_dir = arguments.pop('output_dir') 75 | model.BUCKET = arguments.pop('bucket') 76 | model.BATCH_SIZE = arguments.pop('batch_size') 77 | model.TRAIN_STEPS = (arguments.pop('train_examples') * 1000) / model.BATCH_SIZE 78 | model.EVAL_STEPS = arguments.pop('eval_steps') 79 | print ("Will train for {} steps using batch_size={}".format(model.TRAIN_STEPS, model.BATCH_SIZE)) 80 | model.PATTERN = arguments.pop('pattern') 81 | model.NEMBEDS= arguments.pop('nembeds') 82 | model.NNSIZE = arguments.pop('nnsize') 83 | print ("Will use DNN size of {}".format(model.NNSIZE)) 84 | 85 | # Append trial_id to path if we are doing hptuning 86 | # This code can be removed if you are not using hyperparameter tuning 87 | output_dir = os.path.join( 88 | output_dir, 89 | json.loads( 90 | os.environ.get('TF_CONFIG', '{}') 91 | ).get('task', {}).get('trial', '') 92 | ) 93 | 94 | # Run the training job 95 | model.train_and_evaluate(output_dir) -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/README.md: -------------------------------------------------------------------------------- 1 | ## To deploy and try out the AppEngine application: 2 | 3 | From CloudShell: 4 | * ./deploy.sh 5 | * Visit https://.appspot.com/ 6 | * Use the form to send a request to the service 7 | 8 | ## To deploy and try out the Dataflow service: 9 | * ./run_dataflow.sh 10 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/README.md: -------------------------------------------------------------------------------- 1 | # Baby Weight Prediction Example 2 | 3 | Disclaimer: This is not an official Google product. 4 | 5 | This is an example application demonstrating how the deployed model 6 | can be used to make online predictions from web applications. 7 | 8 | ## Products 9 | - [Cloud Machine Learning Engine][1] 10 | - [App Engine][2] 11 | 12 | ## Language 13 | - [Python][3] 14 | 15 | [1]: https://cloud.google.com/ml-engine/ 16 | [2]: https://cloud.google.com/appengine/docs 17 | [3]: https://python.org 18 | 19 | ## Prerequisites 20 | You are supposed to have deployed the trained model for online predictions 21 | by following the instruction in the Cloud Datalab [notebook][4]. You need to 22 | use the same project to run this application. 23 | 24 | [4]: https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/blogs/babyweight/babyweight.ipynb 25 | 26 | ## Do this first 27 | In this section you will start your [Google Cloud Shell][6] and clone the 28 | application code repository to it. 29 | 30 | 1. [Open the Cloud Console][7] 31 | 32 | 2. Click the Google Cloud Shell icon in the top-right and wait for your shell 33 | to open: 34 | 35 | ![](docs/img/cloud-shell.png) 36 | 37 | 3. Clone the lab repository in your cloud shell, then `cd` into that dir: 38 | 39 | ```shell 40 | $ git clone https://github.com/GoogleCloudPlatform/training-data-analyst 41 | Cloning into 'training-data-analyst'... 42 | ... 43 | 44 | $ cd training-data-analyst/blogs/babyweight/application 45 | ``` 46 | 47 | [6]: https://cloud.google.com/cloud-shell/docs/ 48 | [7]: https://console.cloud.google.com/ 49 | 50 | ## Customize the model name 51 | 52 | If you have deployed the model with the name different from 'babyweight', 53 | open `app.yaml` with a text editor and replace the model name accordingly. 54 | 55 | ```yaml 56 | env_variables: 57 | MODEL_NAME: 'babyweight' 58 | ``` 59 | 60 | ## Deploy the application 61 | 62 | ```shell 63 | $ pip install -r requirements.txt -t lib 64 | $ gcloud app create 65 | $ gcloud app deploy 66 | ``` 67 | 68 | By executing these commands on the Cloud Shell, the project id is automatically 69 | applied to the application and the application URL will be 70 | `https://[project id].appspot.com`. 71 | 72 | Now you can access the URL from your browser to use the web application 73 | to predict a baby's weight. 74 | 75 | ![](docs/img/screenshot.png) 76 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/app.yaml: -------------------------------------------------------------------------------- 1 | runtime: python27 2 | api_version: 1 3 | threadsafe: true 4 | 5 | handlers: 6 | - url: / 7 | script: main.app 8 | - url: /.* 9 | script: main.app 10 | login: required 11 | 12 | env_variables: 13 | MODEL_NAME: 'babyweight' 14 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/appengine_config.py: -------------------------------------------------------------------------------- 1 | # Copyright 2016 Google Inc. 2 | # 3 | # Licensed under the Apache License, Version 2.0 (the "License"); 4 | # you may not use this file except in compliance with the License. 5 | # You may obtain a copy of the License at 6 | # 7 | # http://www.apache.org/licenses/LICENSE-2.0 8 | # 9 | # Unless required by applicable law or agreed to in writing, software 10 | # distributed under the License is distributed on an "AS IS" BASIS, 11 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | # See the License for the specific language governing permissions and 13 | # limitations under the License. 14 | 15 | from google.appengine.ext import vendor 16 | vendor.add('lib') 17 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/docs/img/cloud-shell.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/docs/img/cloud-shell.png -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/docs/img/screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/docs/img/screenshot.png -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/main.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | # Copyright 2017 Google Inc. 4 | # 5 | # Licensed under the Apache License, Version 2.0 (the "License"); 6 | # you may not use this file except in compliance with the License. 7 | # You may obtain a copy of the License at 8 | # 9 | # http://www.apache.org/licenses/LICENSE-2.0 10 | # 11 | # Unless required by applicable law or agreed to in writing, software 12 | # distributed under the License is distributed on an "AS IS" BASIS, 13 | # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 14 | # See the License for the specific language governing permissions and 15 | # limitations under the License. 16 | 17 | 18 | import json 19 | import os 20 | 21 | from flask import Flask 22 | from flask import jsonify 23 | from flask import render_template 24 | from flask import request 25 | from flask import url_for 26 | from googleapiclient import discovery 27 | from oauth2client.client import GoogleCredentials 28 | 29 | from google.appengine.api import app_identity 30 | 31 | 32 | credentials = GoogleCredentials.get_application_default() 33 | api = discovery.build('ml', 'v1', credentials=credentials) 34 | project = app_identity.get_application_id() 35 | model_name = os.getenv('MODEL_NAME', 'babyweight') 36 | version_name = os.getenv('VERSION_NAME', 'ml_on_gcp') 37 | 38 | 39 | app = Flask(__name__) 40 | 41 | 42 | def get_prediction(features): 43 | input_data = {'instances': [features]} 44 | parent = 'projects/%s/models/%s/versions/%s' % (project, model_name, version_name) 45 | prediction = api.projects().predict(body=input_data, name=parent).execute() 46 | return prediction['predictions'][0]['predictions'][0] 47 | 48 | 49 | @app.route('/') 50 | def index(): 51 | return render_template('index.html') 52 | 53 | 54 | @app.route('/form') 55 | def input_form(): 56 | return render_template('form.html') 57 | 58 | 59 | @app.route('/api/predict', methods=['POST']) 60 | def predict(): 61 | def gender2str(val): 62 | genders = {'unknown': 'Unknown', 'male': 'True', 'female': 'False'} 63 | return genders[val] 64 | 65 | def plurality2str(val): 66 | pluralities = {'1': 'Single(1)', '2': 'Twins(2)', '3': 'Triplets(3)'} 67 | if features['is_male'] == 'Unknown' and int(val) > 1: 68 | return 'Multiple(2+)' 69 | return pluralities[val] 70 | 71 | data = json.loads(request.data.decode()) 72 | mandatory_items = ['baby_gender', 'mother_age', 73 | 'plurality', 'gestation_weeks'] 74 | for item in mandatory_items: 75 | if item not in data.keys(): 76 | return jsonify({'result': 'Set all items.'}) 77 | 78 | features = {} 79 | features['key'] = 'nokey' 80 | features['is_male'] = gender2str(data['baby_gender']) 81 | features['mother_age'] = float(data['mother_age']) 82 | features['plurality'] = plurality2str(data['plurality']) 83 | features['gestation_weeks'] = float(data['gestation_weeks']) 84 | 85 | prediction = get_prediction(features) 86 | return jsonify({'result': '{:.2f} lbs.'.format(prediction)}) 87 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/requirements.txt: -------------------------------------------------------------------------------- 1 | Flask==0.12.1 2 | google-api-python-client==1.6.2 3 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/application/templates/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | Welcome 6 | 7 | 8 |

You need to sign in to your Google Account to use this service.

9 | Start the baby weight prediction example 10 | 11 | 12 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/deploy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | cd application 4 | pip install -r requirements.txt -t lib 5 | gcloud app create 6 | gcloud app deploy 7 | 8 | PROJECT=$(gcloud config get-value project) 9 | echo "Visit https://PROJECT-ID.appspot.com/ e.g. https://${PROJECT}.appspot.com" 10 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/exampledata.csv.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/exampledata.csv.gz -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/pipeline/pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 17 | 20 | 4.0.0 21 | 22 | com.google.cloud.training.mlongcp.babyweight 23 | pipeline 24 | [1.0.0,2.0.0] 25 | 26 | 27 | UTF-8 28 | 29 | 30 | 31 | 32 | ossrh.snapshots 33 | Sonatype OSS Repository Hosting 34 | https://oss.sonatype.org/content/repositories/snapshots/ 35 | 36 | false 37 | 38 | 39 | true 40 | 41 | 42 | 43 | 44 | jar 45 | 46 | 47 | 48 | 49 | org.apache.maven.plugins 50 | maven-compiler-plugin 51 | 3.5.1 52 | 53 | 1.8 54 | 1.8 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | org.codehaus.mojo 63 | exec-maven-plugin 64 | 1.4.0 65 | 66 | false 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | com.google.cloud.dataflow 76 | google-cloud-dataflow-java-sdk-all 77 | [2.0.0-beta2,3.0.0) 78 | 79 | 80 | 81 | 82 | com.google.apis 83 | google-api-services-discovery 84 | v1-rev75-1.25.0 85 | 86 | 87 | 88 | 89 | org.slf4j 90 | slf4j-api 91 | 1.7.14 92 | 93 | 94 | org.slf4j 95 | slf4j-jdk14 96 | 1.7.14 97 | 98 | 99 | 100 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/pipeline/src/main/java/com/google/cloud/training/mlongcp/AddPrediction.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016 Google Inc. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package com.google.cloud.training.mlongcp; 18 | 19 | import org.apache.beam.runners.dataflow.DataflowRunner; 20 | import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions; 21 | import org.apache.beam.sdk.Pipeline; 22 | import org.apache.beam.sdk.PipelineResult; 23 | import org.apache.beam.sdk.options.Default; 24 | import org.apache.beam.sdk.options.Description; 25 | import org.apache.beam.sdk.options.PipelineOptionsFactory; 26 | import org.apache.beam.sdk.transforms.windowing.FixedWindows; 27 | import org.apache.beam.sdk.transforms.windowing.Window; 28 | import org.apache.beam.sdk.values.PCollection; 29 | import org.joda.time.Duration; 30 | 31 | /** 32 | * Adds prediction to incoming information. 33 | * 34 | * @author vlakshmanan 35 | * 36 | */ 37 | public class AddPrediction { 38 | // private static final Logger LOG = LoggerFactory.getLogger(AddPrediction.class); 39 | 40 | public static interface MyOptions extends DataflowPipelineOptions { 41 | @Description("Output dir or bqdataset.table") 42 | @Default.String("../output/") 43 | String getOutput(); 44 | 45 | void setOutput(String s); 46 | 47 | 48 | @Description("Input file or pubsub-topic") 49 | @Default.String("../exampledata.csv.gz") 50 | String getInput(); 51 | 52 | void setInput(String s); 53 | 54 | 55 | @Description("If real-time, it will read incoming flight info from Pub/Sub") 56 | @Default.Boolean(false) 57 | boolean isRealtime(); 58 | void setRealtime(boolean r); 59 | } 60 | 61 | public static void main(String[] args) { 62 | MyOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(MyOptions.class); 63 | if (options.isRealtime()) { 64 | options.setStreaming(true); 65 | options.setRunner(DataflowRunner.class); 66 | } 67 | 68 | 69 | Pipeline p = Pipeline.create(options); 70 | 71 | InputOutput io; 72 | if (options.isRealtime()) { 73 | // in real-time, we read from PubSub and write to BigQuery 74 | io = new PubSubBigQuery(); 75 | } else { 76 | io = new TextInputOutput(); 77 | } 78 | 79 | PCollection babies = io // 80 | .readInstances(p, options) // 81 | .apply(Window. into(FixedWindows.of(Duration.standardSeconds(20))).withAllowedLateness(Duration.standardSeconds(10)).discardingFiredPanes()); 82 | 83 | io.writePredictions(babies, options); 84 | 85 | PipelineResult result = p.run(); 86 | if (!options.isRealtime()) { 87 | result.waitUntilFinish(); 88 | } 89 | } 90 | } 91 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/pipeline/src/main/java/com/google/cloud/training/mlongcp/Baby.java: -------------------------------------------------------------------------------- 1 | /* 2 | * Copyright (C) 2016 Google Inc. 3 | * 4 | * Licensed under the Apache License, Version 2.0 (the "License"); you may not 5 | * use this file except in compliance with the License. You may obtain a copy of 6 | * the License at 7 | * 8 | * http://www.apache.org/licenses/LICENSE-2.0 9 | * 10 | * Unless required by applicable law or agreed to in writing, software 11 | * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT 12 | * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the 13 | * License for the specific language governing permissions and limitations under 14 | * the License. 15 | */ 16 | 17 | package com.google.cloud.training.mlongcp; 18 | 19 | import java.util.Arrays; 20 | 21 | import org.apache.beam.sdk.coders.AvroCoder; 22 | import org.apache.beam.sdk.coders.DefaultCoder; 23 | 24 | /** 25 | * @author vlakshmanan 26 | * 27 | */ 28 | @DefaultCoder(AvroCoder.class) 29 | public class Baby { 30 | public enum INPUTCOLS { 31 | weight_pounds,is_male,mother_age,plurality,gestation_weeks,key; 32 | } 33 | 34 | private String[] fields; 35 | 36 | public static Baby fromCsv(String line) { 37 | Baby f = new Baby(); 38 | f.fields = line.split(","); 39 | if (f.fields.length == INPUTCOLS.values().length) { 40 | return f; 41 | } 42 | return null; // malformed 43 | } 44 | 45 | public String[] getFields() { 46 | return fields; 47 | } 48 | 49 | public String getField(INPUTCOLS col) { 50 | return fields[col.ordinal()]; 51 | } 52 | 53 | public float getFieldAsFloat(INPUTCOLS col) { 54 | return Float.parseFloat(fields[col.ordinal()]); 55 | } 56 | 57 | public int getFieldAsInt(INPUTCOLS col) { 58 | return Integer.parseInt(fields[col.ordinal()]); 59 | } 60 | 61 | public float getFieldAsFloat(INPUTCOLS col, float defaultValue) { 62 | String s = fields[col.ordinal()]; 63 | if (s.length() > 0) { 64 | return Float.parseFloat(s); 65 | } else { 66 | return defaultValue; 67 | } 68 | } 69 | 70 | public Baby newCopy() { 71 | Baby f = new Baby(); 72 | f.fields = Arrays.copyOf(this.fields, this.fields.length); 73 | return f; 74 | } 75 | 76 | } 77 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/pipeline/src/main/java/com/google/cloud/training/mlongcp/BabyPred.java: -------------------------------------------------------------------------------- 1 | package com.google.cloud.training.mlongcp; 2 | 3 | import org.apache.beam.sdk.coders.AvroCoder; 4 | import org.apache.beam.sdk.coders.DefaultCoder; 5 | 6 | @DefaultCoder(AvroCoder.class) 7 | public class BabyPred { 8 | Baby flight; 9 | double predictedWeight; 10 | 11 | BabyPred(){} 12 | BabyPred(Baby f, double pred) { 13 | this.flight = f; 14 | this.predictedWeight = pred; 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/pipeline/src/main/java/com/google/cloud/training/mlongcp/InputOutput.java: -------------------------------------------------------------------------------- 1 | package com.google.cloud.training.mlongcp; 2 | 3 | import java.io.Serializable; 4 | 5 | import org.apache.beam.sdk.Pipeline; 6 | import org.apache.beam.sdk.transforms.DoFn; 7 | import org.apache.beam.sdk.transforms.GroupByKey; 8 | import org.apache.beam.sdk.transforms.ParDo; 9 | import org.apache.beam.sdk.values.KV; 10 | import org.apache.beam.sdk.values.PCollection; 11 | import org.slf4j.Logger; 12 | import org.slf4j.LoggerFactory; 13 | 14 | import com.google.cloud.training.mlongcp.AddPrediction.MyOptions; 15 | 16 | @SuppressWarnings("serial") 17 | public abstract class InputOutput implements Serializable { 18 | protected static final Logger LOG = LoggerFactory.getLogger(InputOutput.class); 19 | public abstract PCollection readInstances(Pipeline p, MyOptions options); 20 | public abstract void writePredictions(PCollection instances, MyOptions options); 21 | 22 | private static class CreateBatch extends DoFn> { 23 | private static final int NUM_BATCHES = 2; 24 | @ProcessElement 25 | public void processElement(ProcessContext c) throws Exception { 26 | Baby f = c.element(); 27 | String key = " " + (System.identityHashCode(f) % NUM_BATCHES); 28 | c.output(KV.of(key, f)); 29 | } 30 | } 31 | 32 | public static PCollection addPredictionInBatches(PCollection instances) { 33 | return instances // 34 | .apply("CreateKeys", ParDo.of(new CreateBatch())) // 35 | .apply("BatchByKey", GroupByKey. create()) // within window 36 | .apply("Inference", ParDo.of(new DoFn>, BabyPred>() { 37 | @ProcessElement 38 | public void processElement(ProcessContext c) throws Exception { 39 | Iterable instances = c.element().getValue(); 40 | 41 | // do ml inference for wheelsoff events, but as batch 42 | double[] result = BabyweightMLService.mock_batchPredict(instances); 43 | int resultno = 0; 44 | for (Baby f : instances) { 45 | double ontime = result[resultno++]; 46 | c.output(new BabyPred(f, ontime)); 47 | } 48 | } 49 | })); 50 | } 51 | 52 | 53 | 54 | @SuppressWarnings("unused") 55 | private static PCollection addPredictionOneByOne(PCollection instances) { 56 | return instances // 57 | .apply("Inference", ParDo.of(new DoFn() { 58 | @ProcessElement 59 | public void processElement(ProcessContext c) throws Exception { 60 | Baby f = c.element(); 61 | double predwt = BabyweightMLService.predict(f, -5.0); 62 | c.output(new BabyPred(f, predwt)); 63 | } 64 | })); 65 | } 66 | } 67 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/pipeline/src/main/java/com/google/cloud/training/mlongcp/TextInputOutput.java: -------------------------------------------------------------------------------- 1 | package com.google.cloud.training.mlongcp; 2 | 3 | import java.text.DecimalFormat; 4 | 5 | import org.apache.beam.sdk.Pipeline; 6 | import org.apache.beam.sdk.io.TextIO; 7 | import org.apache.beam.sdk.transforms.DoFn; 8 | import org.apache.beam.sdk.transforms.ParDo; 9 | import org.apache.beam.sdk.values.PCollection; 10 | 11 | import com.google.cloud.training.mlongcp.AddPrediction.MyOptions; 12 | 13 | @SuppressWarnings("serial") 14 | public class TextInputOutput extends InputOutput { 15 | @Override 16 | public PCollection readInstances(Pipeline p, MyOptions options) { 17 | String inputFile = options.getInput(); 18 | LOG.info("Reading data from " + inputFile); 19 | 20 | PCollection babies = p // 21 | .apply("ReadLines", TextIO.read().from(inputFile)) // 22 | .apply("Parse", ParDo.of(new DoFn() { 23 | @ProcessElement 24 | public void processElement(ProcessContext c) throws Exception { 25 | String line = c.element(); 26 | Baby f = Baby.fromCsv(line); 27 | if (f != null) { 28 | c.output(f); 29 | } 30 | } 31 | })); 32 | return babies; 33 | } 34 | 35 | @Override 36 | public void writePredictions(PCollection instances, MyOptions options) { 37 | try { 38 | PCollection prds = addPredictionInBatches(instances); 39 | PCollection lines = predToCsv(prds); 40 | lines.apply("Write", TextIO.write().to(options.getOutput() + "flightPreds").withSuffix(".csv")); 41 | } catch (Throwable t) { 42 | LOG.warn("Inference failed", t); 43 | } 44 | } 45 | 46 | private PCollection predToCsv(PCollection preds) { 47 | return preds.apply("pred->csv", ParDo.of(new DoFn() { 48 | @ProcessElement 49 | public void processElement(ProcessContext c) throws Exception { 50 | BabyPred pred = c.element(); 51 | String csv = String.join(",", pred.flight.getFields()); 52 | if (pred.predictedWeight >= 0) { 53 | csv = csv + "," + new DecimalFormat("0.00").format(pred.predictedWeight); 54 | } else { 55 | csv = csv + ","; // empty string -> null 56 | } 57 | c.output(csv); 58 | }})) // 59 | ; 60 | } 61 | } 62 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/run_dataflow.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | PROJECT_ID=$(gcloud config get-value project) 4 | 5 | cd pipeline 6 | bq mk babyweight 7 | bq rm -f babyweight.predictions 8 | 9 | mvn compile exec:java \ 10 | -Dexec.mainClass=com.google.cloud.training.mlongcp.AddPrediction \ 11 | -Dexec.args="--realtime --input=babies --output=babyweight.predictions --project=$PROJECT_ID" 12 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/run_once.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | cd pipeline 4 | rm -rf ../output 5 | 6 | PROJECT_ID=$(gcloud config get-value project) 7 | 8 | mvn compile exec:java \ 9 | -Dexec.mainClass=com.google.cloud.training.mlongcp.BabyweightMLService \ 10 | -Dexec.args="$PROJECT_ID" 11 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/End-to-End-Machine-Learning-with-TF-on-GCP/lab07-App-Engine-app-to-serve-ML-predictions/run_ontext.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | PROJECTID=$(gcloud config get-value project) 4 | 5 | cd pipeline 6 | rm -rf ../output 7 | mvn compile exec:java \ 8 | -Dexec.mainClass=com.google.cloud.training.mlongcp.AddPrediction \ 9 | -Dexec.args="--input=../*.csv.gz --output=../output/ --project=$PROJECTID" 10 | 11 | 12 | -------------------------------------------------------------------------------- /Google_Cloud_Platform/img/Google Cloud Platform Products chart.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/img/Google Cloud Platform Products chart.png -------------------------------------------------------------------------------- /Google_Cloud_Platform/img/config_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/img/config_1.png -------------------------------------------------------------------------------- /Google_Cloud_Platform/img/config_2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/img/config_2.png -------------------------------------------------------------------------------- /Google_Cloud_Platform/img/config_3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/img/config_3.png -------------------------------------------------------------------------------- /Google_Cloud_Platform/img/config_4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/img/config_4.png -------------------------------------------------------------------------------- /Google_Cloud_Platform/img/google_cloud_mlengine_01.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Google_Cloud_Platform/img/google_cloud_mlengine_01.png -------------------------------------------------------------------------------- /Image-Processing/README.md: -------------------------------------------------------------------------------- 1 | # 과제 1. Image Viewer 만들기 2 | 3 | - 파일 : opencv_refactoring.py 4 | 5 | ### 요청 기능 및 구현 결과 6 | 1. Image load & search in directory(move to next image) - Done 7 | 2. Image value read & view (RGB or gray) - Done 8 | 3. auto window size, editing window size - Done 그러나 현재 코드에선 auto로 고정 9 | 4. zooming + moving - 구현 시도했으나 계속 안되는.. 10 | 5. bbox size, position check - bbox을 구현했으나 crop과 다른 점을 못느껴 crop으로 대체했는데 차이점이 있다면 알려주실 수 있을까요!! 11 | 12 | ### 실행 방법 13 | ``` 14 | python opencv_refactoring.py 15 | ``` 16 | 17 | ### 기능 설명 18 | - 사용자가 현재 폴더의 이미지를 선택해 불러오기 19 | - 불러온 이미지에서 키보드의 좌 우 방향키를 통해 이전, 다음 이미지 불러오기 20 | - 제일 좌측 혹은 제일 우측 이미지의 경우 계속 돌아가도록 설정 21 | - ESC 버튼을 누를 경우 종료 22 | - r 버튼을 누를 경우 현재 이미지 reset 23 | - 마우스 드래그를 통해 rectangle을 만든 후 c를 누를 경우 crop 24 | - rectangle이 존재하지 않을 경우 콘솔에 'There is a no rectangle' 출력 25 | - g 버튼을 누를 경우 현재 보이는 이미지가 gray로 출력 26 | - 방향키를 이용해 다른 이미지로 전환했을 경우에도 현재 보이는 이미지가 gray 되도록 설정 27 | - gray된 사진에서 g를 다시 누르면 color로 변환 28 | 29 | 30 | -------------------------------------------------------------------------------- /Image-Processing/image_viewer.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import cv2 3 | import os 4 | import numpy as np 5 | 6 | def show_type(style): 7 | if style == 'color': 8 | return 1 9 | elif style == 'gray': 10 | return 0 11 | else: 12 | return -1 13 | 14 | 15 | def window_size(size_flag): 16 | if size_flag == 'auto': 17 | return cv2.WINDOW_AUTOSIZE 18 | elif size_flag == 'normal': 19 | return cv2.WINDOW_NORMAL 20 | elif size_flag == 'full': 21 | return cv2.WINDOW_FULLSCREEN 22 | 23 | class ImageViewer: 24 | refPt = [] 25 | cropping = False 26 | 27 | def __init__(self, args): 28 | self.args = args 29 | 30 | def show_image(self, img): 31 | flag = show_type(self.args.show_flag) 32 | image = cv2.imread(img, flag) 33 | size_flag = window_size(self.args.size_flag) 34 | cv2.namedWindow('image', int(size_flag)) 35 | cv2.imshow('image', image) 36 | cv2.waitKey(0) 37 | cv2.destroyAllWindows() 38 | 39 | def crop_image(self, img): 40 | global image 41 | 42 | def click_crop(event, x, y, flags, param): 43 | global refPt, cropping 44 | 45 | if event == cv2.EVENT_LBUTTONDOWN: 46 | refPt = [(x, y)] 47 | cropping = True 48 | 49 | elif event == cv2.EVENT_LBUTTONUP: 50 | refPt.append((x, y)) 51 | cropping = False 52 | 53 | cv2.rectangle(image, refPt[0], refPt[1], (0, 255, 0), 2) 54 | cv2.imshow("image", image) 55 | 56 | image = cv2.imread(img) 57 | clone = image.copy() 58 | cv2.namedWindow("image") 59 | cv2.setMouseCallback("image", click_crop) 60 | 61 | while True: 62 | cv2.imshow('image', image) 63 | key = cv2.waitKey(1) & 0xFF 64 | 65 | if key == ord("r"): 66 | image = clone.copy() 67 | 68 | elif key == ord("c"): 69 | break 70 | 71 | if len(refPt) == 2: 72 | crop = clone[refPt[0][1]:refPt[1][1], refPt[0][0]:refPt[1][0]] 73 | cv2.imshow("crop", crop) 74 | cv2.waitKey(0) 75 | cv2.destroyAllWindows() 76 | 77 | 78 | 79 | -------------------------------------------------------------------------------- /Image-Processing/main.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import cv2 3 | import numpy as np 4 | import os 5 | import argparse 6 | import sys 7 | 8 | import image_viewer 9 | 10 | print("python version : {version}".format(version=sys.version)) 11 | 12 | 13 | parser = argparse.ArgumentParser() 14 | parser.add_argument("--folder_path", help="folder path", default="./") 15 | parser.add_argument("--show_flag", help="'color' or 'gray'", default='color', type=str) 16 | parser.add_argument("--size_flag", help="window_size : auto, normal, full", default='auto', type=str) 17 | parser.add_argument("--crop_mode", help="crop_mode : 'True' or 'False' ", default=False, type=bool) 18 | 19 | args = parser.parse_args() 20 | 21 | image_extension = ('jpg', 'jpeg', 'png') 22 | current_folder = args.folder_path 23 | image_file = [i for i in os.listdir(current_folder) if i.endswith(image_extension) == True] 24 | print("current folder path : {current_folder}\nimage 개수 : {len_image}\nimage file : {image_file}".format( 25 | current_folder=current_folder, len_image=len(image_file), image_file=image_file 26 | )) 27 | 28 | a = raw_input("몇번째 이미지를 보여드릴까요?\n") 29 | try: 30 | selected_image = image_file[int(a) - 1] 31 | print(selected_image) 32 | except IndexError: 33 | print("1부터 {n}까지의 숫자를 입력해주세요".format(n=len(image_file))) 34 | finally: 35 | if int(a) <= 0: 36 | print("양수를 입력해주세요") 37 | 38 | 39 | def main(selected_image): 40 | if args.crop_mode == True: 41 | op = image_viewer.ImageViewer(args) 42 | op.crop_image(selected_image) 43 | 44 | else: 45 | op = image_viewer.ImageViewer(args) 46 | op.show_image(selected_image) 47 | 48 | if __name__ == '__main__': 49 | main(selected_image) -------------------------------------------------------------------------------- /Image-Processing/mouse_callback.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | import cv2 3 | import numpy as np 4 | 5 | drawing = False #Mouse가 클릭된 상태 확인용 6 | mode = True # True이면 사각형, false면 원 7 | ix, iy = -1, -1 8 | 9 | 10 | # Mouse Callback함수 11 | # def draw_circle(event, x, y, flags, param): 12 | # global ix, iy, drawing, mode 13 | # 14 | # if event == cv2.EVENT_LBUTTONDOWN: #마우스를 누른 상태 15 | # drawing = True 16 | # ix, iy = x, y 17 | # 18 | # elif event == cv2.EVENT_MOUSEMOVE: # 마우스 이동 19 | # if drawing == True: 20 | # if mode == True: 21 | # cv2.rectangle(img, (ix, iy), (x, y), (255, 0, 0), -1) 22 | # else: 23 | # cv2.circle(img, (x,y), 5, (0,255,0), -1) 24 | # 25 | # elif event == cv2.EVENT_LBUTTONUP: 26 | # drawing = False # 마우스를 때면 상태 변경 27 | # if mode == True: 28 | # cv2.rectangle(img, (ix, iy), (x, y), (255, 0, 0), -1) 29 | # else: 30 | # cv2.circle(img, (x,y), 5, (0, 255, 0), -1) 31 | box = [] 32 | def zoom_event(event, x, y, flags, param): 33 | global ix, iy, zoom 34 | 35 | 36 | if event == cv2.EVENT_LBUTTONDOWN: 37 | zoom = True 38 | print('start mouse, x=', x,'y=', y) 39 | start_box = [x, y] 40 | box.append(start_box) 41 | 42 | elif event == cv2.EVENT_LBUTTONUP: 43 | print('end mouse. x=', x, 'y=', y) 44 | zoom = False 45 | end_box = [x, y] 46 | box.append(end_box) 47 | print(box) 48 | 49 | img = np.zeros((512, 512, 3), np.uint8) 50 | cv2.namedWindow('image') 51 | cv2.setMouseCallback('image', zoom_event) 52 | 53 | while True: 54 | cv2.imshow('image', img) 55 | k = cv2.waitKey(1) & 0xFF 56 | if k == ord('m'): 57 | mode = not mode 58 | elif k == 27: 59 | break 60 | 61 | cv2.destroyAllWindows() 62 | 63 | # import numpy as np 64 | # import cv2 65 | # 66 | # boxes = [] 67 | # 68 | # def on_mouse(event, x, y, flags, params): 69 | # if event == cv2.EVENT_LBUTTONDOWN: 70 | # print 'Start Mouse Position: '+str(x)+', '+str(y) 71 | # sbox = [x, y] 72 | # boxes.append(sbox) 73 | # 74 | # elif event == cv2.EVENT_LBUTTONUP: 75 | # print 'End Mouse Position: '+str(x)+', '+str(y) 76 | # ebox = [x, y] 77 | # boxes.append(ebox) 78 | # print(boxes) 79 | # crop = img[boxes[-2][1]:boxes[-1][1], boxes[-2][0]:boxes[-1][0]] 80 | # 81 | # cv2.imshow('crop', crop) 82 | # 83 | # 84 | # count = 0 85 | # while(1): 86 | # count += 1 87 | # img = cv2.imread('test.jpeg',0) 88 | # img = cv2.blur(img, (3,3)) 89 | # 90 | # cv2.namedWindow('real image') 91 | # cv2.setMouseCallback('real image', on_mouse, 0) 92 | # cv2.imshow('real image', img) 93 | # if count < 50: 94 | # if cv2.waitKey(33) == 27: 95 | # cv2.destroyAllWindows() 96 | # break 97 | # elif count >= 50: 98 | # if cv2.waitKey(0) == 27: 99 | # cv2.destroyAllWindows() 100 | # break 101 | # count = 0 -------------------------------------------------------------------------------- /Image-Processing/test.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Image-Processing/test.jpeg -------------------------------------------------------------------------------- /Image-Processing/test1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Image-Processing/test1.jpg -------------------------------------------------------------------------------- /Image-Processing/test2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Image-Processing/test2.png -------------------------------------------------------------------------------- /Image-Processing/xception_model.py: -------------------------------------------------------------------------------- 1 | from keras.models import Model 2 | from keras.layers import Input, Conv2D, MaxPooling2D, Activation, Dropout 3 | from keras.layers import BatchNormalization, Dense, GlobalAveragePooling2D 4 | from keras.layers import SeparableConv2D 5 | # from keras.layers import Flatten 6 | from keras import layers 7 | from keras.regularizers import l2 8 | 9 | def XCEPTION(input_shape=None, l2_regularization=0.01): 10 | img_size = 48 11 | num_classes = 7 12 | 13 | # base 14 | if input_shape is not None: 15 | img_input = Input(input_shape) 16 | else: 17 | img_input = Input((img_size, img_size, 3)) 18 | x = Conv2D(32, (3, 3), strides=(2, 2), use_bias=False)(img_input) 19 | x = BatchNormalization(name='block1_conv1_bn')(x) 20 | x = Activation('relu', name='block1_conv1_act')(x) 21 | x = Conv2D(64, (3, 3), use_bias=False)(x) 22 | x = BatchNormalization(name='block1_conv2_bn')(x) 23 | x = Activation('relu', name='block1_conv2_act')(x) 24 | 25 | residual = Conv2D(128, (1, 1), strides=(2, 2), 26 | padding='same', use_bias=False)(x) 27 | residual = BatchNormalization()(residual) 28 | 29 | x = SeparableConv2D(128, (3, 3), padding='same', use_bias=False)(x) 30 | x = BatchNormalization(name='block2_sepconv1_bn')(x) 31 | x = Activation('relu', name='block2_sepconv2_act')(x) 32 | x = SeparableConv2D(128, (3, 3), padding='same', use_bias=False)(x) 33 | x = BatchNormalization(name='block2_sepconv2_bn')(x) 34 | 35 | x = MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x) 36 | x = layers.add([x, residual]) 37 | 38 | residual = Conv2D(256, (1, 1), strides=(2, 2), 39 | padding='same', use_bias=False)(x) 40 | residual = BatchNormalization()(residual) 41 | 42 | x = Activation('relu', name='block3_sepconv1_act')(x) 43 | x = SeparableConv2D(256, (3, 3), padding='same', use_bias=False)(x) 44 | x = BatchNormalization(name='block3_sepconv1_bn')(x) 45 | x = Activation('relu', name='block3_sepconv2_act')(x) 46 | x = SeparableConv2D(256, (3, 3), padding='same', use_bias=False)(x) 47 | x = BatchNormalization(name='block3_sepconv2_bn')(x) 48 | 49 | x = MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x) 50 | x = layers.add([x, residual]) 51 | x = Conv2D(num_classes, (3, 3), 52 | #kernel_regularizer=regularization, 53 | padding='same')(x) 54 | x = GlobalAveragePooling2D()(x) 55 | output = Activation('softmax',name='predictions')(x) 56 | 57 | model = Model(img_input, output) 58 | return model -------------------------------------------------------------------------------- /Julia/data.csv: -------------------------------------------------------------------------------- 1 | start node, end node link length 2 | 1,2,2, 3 | 1,3,4.5 4 | 2,3,6 5 | 2,4,3 6 | 3,4,5 7 | -------------------------------------------------------------------------------- /Julia/data.txt: -------------------------------------------------------------------------------- 1 | This is the first line 2 | this is the second 3 | thie is the third -------------------------------------------------------------------------------- /Julia/optimization_example.jl: -------------------------------------------------------------------------------- 1 | using JuMP, GLPK 2 | m = Model(with_optimizer(GLPK.Optimizer)) 3 | 4 | @variable(m, 0 <= x <= 2 ) 5 | @variable(m, 0 <= y <= 30 ) 6 | 7 | @objective(m, Max, 5x + 3*y ) 8 | 9 | @constraint(m, 1x + 5y <= 3.0 ) 10 | 11 | JuMP.optimize!(m) 12 | println("Objective value: ", JuMP.objective_value(m)) 13 | println("x = ", JuMP.value(x)) 14 | println("y = ", JuMP.value(y)) 15 | -------------------------------------------------------------------------------- /Julia/plot1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Julia/plot1.png -------------------------------------------------------------------------------- /Julia/plot2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Julia/plot2.png -------------------------------------------------------------------------------- /Julia/plot3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/Julia/plot3.png -------------------------------------------------------------------------------- /Julia/results1.txt: -------------------------------------------------------------------------------- 1 | Magic Johnsonis a basketball player 2 | Michael jordan is also a basketball palyer 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2017 Seong Yun Byeon 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Linux/Crontab, Screen.md: -------------------------------------------------------------------------------- 1 | # Linux 2 | 3 | ### Screen 4 | ```{r, engine='bash', code_block_name} ... 5 | Screen 을 띄워놓고 작업 진행! 6 | - screen --help : 도움말 7 | - screen -dmS name : name의 스크린을 생성 8 | - screen -list : 현재 스크린을 보여줌 9 | - screen -r xxxx.kyle(name) -> 접속!! 그 후 exit를 입력하면 세션이 나가짐 10 | - 세션에서 나가지 않고 화면전환은 control + a + d 를 누르면 됨..! 11 | ``` 12 | 13 | ### Crontab 14 | ```{r, engine='bash', code_block_name} ... 15 | Crontab : 예약 작업을 수행 16 | /etc/crontab에 입력 17 | 18 | 19 | * * * * * 수행할 명령어 20 | ┬ ┬ ┬ ┬ ┬ 21 | │ │ │ │ │ 22 | │ │ │ │ │ 23 | │ │ │ │ └───────── 요일 (0 - 6) (0 =일요일) 24 | │ │ │ └────────── 월 (1 - 12) 25 | │ │ └─────────── 일 (1 - 31) 26 | │ └──────────── 시 (0 - 23) 27 | └───────────── 분 (0 - 59) 28 | 29 | 30 | 0 8 * * 1-5 /root/weekday.sh : → 평일(월요일~금요일) 08:00 weekday.sh을 실행 31 | 32 | 33 | - *.sh 파일을 실행하라고 하는 경우 env를 입력하고 나오는 환경변수를 추가해야 실행됨 34 | - sudo service cron restart 35 | ``` -------------------------------------------------------------------------------- /Linux/function.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | lister(){ 3 | echo "현재 디렉터리는 'pwd'입니다" 4 | echo "현재 디렉터리 파일은 아래와 같습니다" 5 | ls 6 | } 7 | lister 8 | -------------------------------------------------------------------------------- /Linux/readonly 파일 저장.md: -------------------------------------------------------------------------------- 1 | # readonly 파일 저장하기 2 | - E45: 'readonly' option is set (add ! to override) 에러가 나오는 경우 3 | 4 | ### 해결 방법 5 | - 1) sudo vi filename 6 | - 2) 저장할 때, :wq 대신에 :w! 를 사용 7 | 8 | - 9 | -------------------------------------------------------------------------------- /Linux/scp.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# scp" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "1. 사용방법\n", 15 | " 1) 원격 서버 → 로컬 서버로 파일 전송\n", 16 | " # scp [옵션] [계정명]@[원격지IP주소]:[원본 경로 및 파일] [전송받을 위치]\n", 17 | "\n", 18 | " 예제1) IP 111.222.333.444 서버의 abc라는 계정으로 /home/abc/index.html 파일을 로컬서버 /home/me/ 디렉토리에 전송 받기\n", 19 | " # scp abc@111.222.333.444:/home/abc/index.html /home/me/\n", 20 | " \n", 21 | " 2) 로컬 서버 → 원격 서버로 파일 전송\n", 22 | " # scp [옵션] [원본 경로 및 파일] [계정명]@[원격지IP주소]:[전송할 경로]\n", 23 | "\n", 24 | " 예제2) 로컬서버 /home/me/wow.html 파일을 IP 111.222.333.444 서버의 /home/abc/ 디렉토리에 전송 하기\n", 25 | " # scp /home/me/wow.html abc@111.222.333.444:/home/abc/\n", 26 | " \n", 27 | " 3) ssh포트를 기본 22번으로 사용하고 있지 않는 서버로의 전송\n", 28 | "\n", 29 | " 예제3) 2222번인 SSH포트를 사용한다면 아래와 같이 –P 옵션과 포트번호를 넣어준다.\n", 30 | " # scp –P 2222 abc@111.222.333.444:/home/abc/index.html /home/me/\n", 31 | " # scp –P 2222 /home/me/wow.html abc@111.222.333.444:/home/abc/\n", 32 | " \n", 33 | " [주의사항]\n", 34 | " 옵션중에 –P와 –p가 있으니 대/소문자 확인을 하여야 한다.\n", 35 | " -P : 포트번호를 지정함\n", 36 | " -p : 원본파일 수정/사용시간 및 권한을 유지함\n", 37 | " -r : 하위 디렉토리 및 파일 모두 복사함" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "scp -i ~~~~~ ubuntu@ ~~ :/ , 전송받을 위치" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": { 51 | "collapsed": true 52 | }, 53 | "outputs": [], 54 | "source": [] 55 | } 56 | ], 57 | "metadata": { 58 | "kernelspec": { 59 | "display_name": "Python 3", 60 | "language": "python", 61 | "name": "python3" 62 | }, 63 | "language_info": { 64 | "codemirror_mode": { 65 | "name": "ipython", 66 | "version": 3 67 | }, 68 | "file_extension": ".py", 69 | "mimetype": "text/x-python", 70 | "name": "python", 71 | "nbconvert_exporter": "python", 72 | "pygments_lexer": "ipython3", 73 | "version": "3.6.1" 74 | } 75 | }, 76 | "nbformat": 4, 77 | "nbformat_minor": 2 78 | } 79 | -------------------------------------------------------------------------------- /Linux/setting.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | # setting 4 | - setting은 보통 .bashrc / .bashprofile / .zshrc 등에서 진행 5 | - bashrc는 bash이 실행될 때마다 수행 6 | - .bash_profile은 bash이 login shell로 쓰일 때(즉 처음 login할 때)에 수행. 7 | 8 | - child shell은 부모로부터 환경변수를 이어받으니 .bashrc에서 따로 PATH를 설정해 줄 필요는 없지만, GUI 환경에서 새 terminal을 여는 경우에는 login shell로 처리되지 않으므로 주의가 필요 9 | 10 | [참고](http://dogfeet.github.io/articles/2012/bash-profile.html) 11 | 12 | ## syslog 13 | 로그파일 종류 14 | 15 | /var/log 디렉토리에 위치 16 | 17 | - message : 시스템 운영에 대한 전반적인 기록 18 | - boot.log : 부팅될 때 출력되는 메세지 기록 19 | - cron : cron 기록 20 | 21 | 22 | ## 소유자 변경 (chown) 23 | ~~~ 24 | chown [변경할 소유자] [변경할 파일] 25 | ~~~ 26 | - -------------------------------------------------------------------------------- /Linux/shell_parameter.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | POS1="$1" 4 | POS2="$2" 5 | POS3="$3" 6 | 7 | echo "첫 위치 파라미터: $1" 8 | echo "두번째 파라미터 : $2" 9 | echo "세번째 파라미터 : $3" 10 | 11 | echo "아규먼트 위치 파라미터 총 개수 : $#" 12 | echo "아규먼트의 내용들 : $@" 13 | 14 | echo "쉘 프로세스 ID : $$" 15 | -------------------------------------------------------------------------------- /Linux/test.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | if [ -f file ] 4 | then 5 | echo "file exists" 6 | else 7 | echo "file not found" 8 | fi 9 | 10 | if [ -d file ] 11 | then 12 | echo "file is a directory" 13 | else 14 | echo "file is not a directory" 15 | fi 16 | 17 | if [ -s file ] 18 | then 19 | echo "file is not of zero length" 20 | else 21 | echo "file size is zero length" 22 | fi 23 | 24 | if [ -r file -a -w file ] 25 | then 26 | echo "file is readable and writable" 27 | else 28 | echo "file is not read/write" 29 | fi 30 | -------------------------------------------------------------------------------- /Linux/while.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | number=0 4 | while [ "$number" -lt 10 ] 5 | do 6 | echo "$number" 7 | number='expr $number + 1' 8 | done 9 | echo "script complete" 10 | -------------------------------------------------------------------------------- /Linux/압축.md: -------------------------------------------------------------------------------- 1 | # 압축 2 | 3 | 4 | ### tar.gz 5 | #### 압축하기 6 | - tar -cvzf [압축된 파일 이름] [압축할 파일이나 폴더명] 7 | - tar -cvzf xxx.tar.gz * : 해당 경로의 모든 파일을 xxx.tar.gz로 압축 8 | - tar -cvzf xxx.tar.gz * --ignore-failed-read : 권한이 없는 파일은 패스하고 해당 경로의 모든 파일을 xxx.tar.gz로 압축 9 | 10 | #### 옵션값 11 | - x : 묶음을 해제 12 | - c : 파일을 묶음 13 | - v : 묶음/해제 과정을 화면에 표시 14 | - z : gunzip을 사용 15 | - f : 파일 이름을 지정 16 | - p : 권한(permission)을 원본과 동일하게 유지 17 | 18 | #### 압축풀기 19 | - tar -xvzf [압축 해제할 압축 아카이브 이름] 20 | 21 | - 22 | 23 | 24 | ### zip 25 | - unzip 패키지 설치 26 | > apt-get install unzip 27 | 28 | - unzip happy.zip -d ./target : ./target 폴더에 happy.zip 압축을 풀기 29 | - zip test.zip ./* : 현재 폴더의 모든 것을 압축 30 | - zip -r test.zip ./* : test.zip으로 압축하고 현재 폴더의 모든 것과 현재 폴더의 하위 폴더들도 모두 압축 31 | -------------------------------------------------------------------------------- /Network/README.md: -------------------------------------------------------------------------------- 1 | # Network 2 | 3 | 4 | - [하루 3분 네트워크 교실](http://www.yes24.com/24/Goods/30670329?Acode=101) 을 보면서 정리한 폴더입니다 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## Today I Learned 2 | - [![HitCount](http://hits.dwyl.io/zzsza/TIL.svg)](http://hits.dwyl.io/zzsza/TIL) 3 | - 시작일 : 2017년 2월 13일 4 | 5 | ## 작성규칙 6 | - 언어나 기술명으로 폴더를 만든다(root에 문서를 만들지 않는다) 7 | - python의 경우 ipynb으로 생성, python이 아닌 다른 언어라면 md파일 권장 8 | - 파일명은 영어로 9 | 10 | ## Folder Structure 11 | ``` 12 | ├── Google_Cloud_Platform 13 | ├── Image-Processing 14 | ├── Linux 15 | ├── SQL_performance_tuning 16 | ├── Tableau 17 | ├── Tensorflow 18 | ├── deep_learning_from_scratch 19 | ├── docker 20 | ├── hands-on-machine-learning 21 | ├── information_graphics 22 | ├── mac 23 | ├── network 24 | ├── nlp 25 | ├── python 26 | ├── pytorch 27 | ├── reinforcement_learning 28 | ├── scala 29 | └── scikit-learn 30 | ``` 31 | 32 | ## Reference 33 | [유림님 TIL](https://github.com/milooy/TIL#today-i-learned) 34 | -------------------------------------------------------------------------------- /SQL_performance_tuning/00. SQL 실행 순서.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## SELECT 실행 순서 \n", 8 | "\n", 9 | " \n", 10 | "### -문법 순서-\n", 11 | " \n", 12 | "SELECT - 1\n", 13 | "\n", 14 | "FROM - 2\n", 15 | "\n", 16 | "WHERE - 3\n", 17 | "\n", 18 | "GROUP BY - 4\n", 19 | "\n", 20 | "HAVING - 5\n", 21 | "\n", 22 | "ORDER BY - 6\n", 23 | "\n", 24 | "\n", 25 | "### -실행 순서-\n", 26 | " \n", 27 | "FROM - 1\n", 28 | "\n", 29 | "WHERE - 2\n", 30 | "\n", 31 | "GROUP BY - 3\n", 32 | "\n", 33 | "HAVING - 4\n", 34 | "\n", 35 | "SELECT - 5\n", 36 | "\n", 37 | "ORDER BY - 6\n", 38 | " \n", 39 | " \n", 40 | "해당 데이터가 있는 곳을 찾아가서 (FROM)\n", 41 | "\n", 42 | "조건에 맞는 데이터만 가져와서 (WHERE)\n", 43 | "\n", 44 | "원하는 데이터로 집계 (GROUP BY)\n", 45 | "\n", 46 | "집계한 데이터에서 조건에 맞는 것만 (HAVING)\n", 47 | "\n", 48 | "뽑아내서 (SELECT)\n", 49 | "\n", 50 | "정렬 (ORDER BY)\n", 51 | " \n", 52 | "실행순서는 문법, 권한 검사 순서이면서 Alias 등록 순서" 53 | ] 54 | }, 55 | { 56 | "cell_type": "markdown", 57 | "metadata": {}, 58 | "source": [ 59 | " \n", 60 | "별칭(Alias)\n", 61 | "\n", 62 | "FROM 절에서 테이블에 Alias를 사용했다면 (FROM Table1 AS T1)\n", 63 | "\n", 64 | "SELECT, ORDER BY 절에서 사용할 수 있고 (SELECT T1.Col1, ORDER BY T1.Col1)\n", 65 | "\n", 66 | "SELECT 절에서 컬럼에 Alias를 사용했다면 (SELECT T1.Col1 AS a)\n", 67 | "\n", 68 | "ORDER BY 절에서 사용할 수 있다. (ORDER BY AS a)\n", 69 | " \n", 70 | "ORDER BY절에 T1.a가 안되는 것으로 보아 a는 T1.col1을 대신!" 71 | ] 72 | } 73 | ], 74 | "metadata": { 75 | "kernelspec": { 76 | "display_name": "Python 3", 77 | "language": "python", 78 | "name": "python3" 79 | }, 80 | "language_info": { 81 | "codemirror_mode": { 82 | "name": "ipython", 83 | "version": 3 84 | }, 85 | "file_extension": ".py", 86 | "mimetype": "text/x-python", 87 | "name": "python", 88 | "nbconvert_exporter": "python", 89 | "pygments_lexer": "ipython3", 90 | "version": "3.6.1" 91 | }, 92 | "latex_envs": { 93 | "LaTeX_envs_menu_present": true, 94 | "autocomplete": true, 95 | "bibliofile": "biblio.bib", 96 | "cite_by": "apalike", 97 | "current_citInitial": 1, 98 | "eqLabelWithNumbers": true, 99 | "eqNumInitial": 1, 100 | "hotkeys": { 101 | "equation": "Ctrl-E", 102 | "itemize": "Ctrl-I" 103 | }, 104 | "labels_anchors": false, 105 | "latex_user_defs": false, 106 | "report_style_numbering": false, 107 | "user_envs_cfg": false 108 | } 109 | }, 110 | "nbformat": 4, 111 | "nbformat_minor": 2 112 | } 113 | -------------------------------------------------------------------------------- /Tableau/01. Tableau 기초.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "** [Tableau 기초 동영상](https://www.tableau.com/ko-kr/learn/tutorials/)을 보다 필요한 것을 작성한 문서입니다 **" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "# 드릴다운 및 계층\n", 15 | "- 여러 연도의 매출 합계를 검토하는 경우 드릴다운하여 각 연도의 모든 월 매출을 볼 수 있습니다. 또는 모든 월의 매출 합계를 검토하는 경우 드릴업하여 각 연도의 매출을 볼 수 있습니다\n", 16 | "- ex) 국가별 특정 이벤트의 합 -> (드릴 다운) 국가별 특정 지역 이벤트의 합\n", 17 | "- 드릴다운 생성 : 사용할 필드를 끌고 다른 범주에 놓으면 생성됨 -> 이런 과정이 있다면 뷰의 알약 모양에 +, - 버튼이 생김\n", 18 | "- 드릴다운 제거 : 드릴다운에 있던 필드를 밖으로 끌던가, 우측 마우스를 누르고 제거 클릭\n", 19 | "\n", 20 | "- 날짜 데이터의 경우 자동으로 연/분기/월로 나눔\n", 21 | "\n", 22 | "[드릴다운 관련 문서](http://onlinehelp.tableau.com/current/pro/desktop/ko-kr/help.htm#buildmanual_multidimensional_drilldown.html)" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": {}, 28 | "source": [ 29 | "# 정렬\n", 30 | "- 퀵정렬 : 축이 있는 그래프의 경우 설정되어 있음\n", 31 | "- 범례에서 위로 올리면 올라감\n", 32 | "- 그냥 일단 클릭해서 위로 올라가는 경우가 많음\n", 33 | "- 디테일한 정렬이 필요할 시엔 알약 버튼을 누르고 정렬을 클릭 -> 새 창에서 설정\n", 34 | "- 알약의 순서에 따라 다양하게 정렬할 수 있음\n", 35 | "- 연속형 자료는 퀵정렬은 존재하지만 알약에선 정렬할 수 없음\n", 36 | "- 차원형 자료는 정렬할 수 있음" 37 | ] 38 | }, 39 | { 40 | "cell_type": "markdown", 41 | "metadata": {}, 42 | "source": [ 43 | "# 그룹화\n", 44 | "- 'FL' 이랑 '플로리다'랑 같은 경우엔 어떻게 할까?\n", 45 | "- 값이 작은 친구들은 그냥 '기타'로 처리하고 싶은 경우엔 어떻게 할까?\n", 46 | "- Control을 누르고 머리글을 2개 누르고 우측 클릭 -> 합치기\n", 47 | "\n", 48 | "- 차원 -> 우측 클릭 -> 만들기 -> 그룹\n", 49 | "\n", 50 | "#### 수익이 10만 미만인 그룹을 기타로 만들고 싶다면?\n", 51 | "- 마우스 우클릭 -> 계산된 필드 만들기 -> if sum([profit]) > 100000 then 'true' else 'false' END\n", 52 | "\n", 53 | "- 마우스 우클릭 -> 매개변수 만들기 -> 매개변수 우클릭 후 파라미터 컨트롤 Show! -> 100000 대신 변수명 입력!\n", 54 | "\n", 55 | "#### 구간차원 (10대, 20대, 30대)\n", 56 | "- 마우스 우클릭 -> 구간 차원(bin) 만들기" 57 | ] 58 | }, 59 | { 60 | "cell_type": "markdown", 61 | "metadata": {}, 62 | "source": [ 63 | "# 집합\n", 64 | "- 필터링을 한 것과 같은 효과\n", 65 | "- 그룹과도 미묘하게 비슷함" 66 | ] 67 | }, 68 | { 69 | "cell_type": "markdown", 70 | "metadata": {}, 71 | "source": [ 72 | "# 추세선\n", 73 | "- 분석 -> 모델 : 추세선\n", 74 | "- 추세선 옵션에서 선형 / 로그 / 지수 / 다항식 차수, 신뢰구간 등을 설정할 수 있음" 75 | ] 76 | }, 77 | { 78 | "cell_type": "markdown", 79 | "metadata": { 80 | "collapsed": true 81 | }, 82 | "source": [ 83 | "# 데이터 통합의 기술\n", 84 | "- Union : 동일한 형식 및 구조로 저장된 데이터가 포함된 테이블을 행을 기준으로 결합 (수동 : 그냥 끌고오기, 자동 : **와일드카드** 기능 사용)\n", 85 | "- Join : 공통 필드를 가지고 데이터를 연산 시켜서 동일한 데이터 베이스에 있는 두 개의 서로 다른 테이블을 결합. 열을 추가해 수평적으로 확장되는 가상 테이블을 생성\n", 86 | "- 교차 데이터베이스 조인 : 타 데이터베이스간 Join\n", 87 | "- 데이터 블렌딩 : 데이터를 가공 후, 하는 방식으로 Join을 사용할 수 없는 경우 사용" 88 | ] 89 | }, 90 | { 91 | "cell_type": "code", 92 | "execution_count": null, 93 | "metadata": { 94 | "collapsed": true 95 | }, 96 | "outputs": [], 97 | "source": [] 98 | } 99 | ], 100 | "metadata": { 101 | "kernelspec": { 102 | "display_name": "Python 3", 103 | "language": "python", 104 | "name": "python3" 105 | }, 106 | "language_info": { 107 | "codemirror_mode": { 108 | "name": "ipython", 109 | "version": 3 110 | }, 111 | "file_extension": ".py", 112 | "mimetype": "text/x-python", 113 | "name": "python", 114 | "nbconvert_exporter": "python", 115 | "pygments_lexer": "ipython3", 116 | "version": "3.6.1" 117 | }, 118 | "latex_envs": { 119 | "LaTeX_envs_menu_present": true, 120 | "autocomplete": true, 121 | "bibliofile": "biblio.bib", 122 | "cite_by": "apalike", 123 | "current_citInitial": 1, 124 | "eqLabelWithNumbers": true, 125 | "eqNumInitial": 1, 126 | "hotkeys": { 127 | "equation": "Ctrl-E", 128 | "itemize": "Ctrl-I" 129 | }, 130 | "labels_anchors": false, 131 | "latex_user_defs": false, 132 | "report_style_numbering": false, 133 | "user_envs_cfg": false 134 | } 135 | }, 136 | "nbformat": 4, 137 | "nbformat_minor": 2 138 | } 139 | -------------------------------------------------------------------------------- /Tableau/02. Searchbar 만들기 (Input 상자).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 02. Searchbar 만들기 (Input 상자)\n", 8 | "## 1. 매개변수 ( parameter ) 만들기\n", 9 | "- 이름 설정, 현재 값은 공란으로 만들기\n", 10 | "- Search User Input을 생성했습니다\n", 11 | "\n", 12 | "## 2. 계산된 필드 만들기\n", 13 | "- 계산된 필드의 이름을 설정하고, contains([user_id], [Search User Input]) 이런식으로 함수를 만듭니다\n", 14 | "- 만약 동시에 여러 조건을 걸고 싶다면 아래와 같이\n", 15 | "~~~\n", 16 | "contains([user_id], [Search User Input])\n", 17 | "or\n", 18 | "contains([user_name], [Search User Input])\n", 19 | "or\n", 20 | "contains([user_fullname], [Search User Input])\n", 21 | "~~~\n", 22 | "- 만약 모두 소문자로 바꾸고 진행할거라면 각 컬럼 ([]로 되어있는 것들) 앞에 lower 함수로 감싸주면 됩니다\n", 23 | "\n", 24 | "## 3. 보고싶은 그래프 혹은 테이블 생성하기\n", 25 | "- 이건 각자 다를 것 같아 패스-\n", 26 | "- 저는 일자별 event를 count했습니다\n", 27 | "\n", 28 | "## 4. 필터에 계산된 필드 추가\n", 29 | "- 계산된 필드를 필터에 추가합니다\n", 30 | "- 이 때, 필터의 값이 참(True)인 값만 체크합니다\n", 31 | "\n", 32 | "## 5. 분석 - 매개변수를 클릭해 필터를 활성화합니다\n", 33 | "- 워크시트라면 우측에 위치하고, 대시보드라면 위치 조정을 자유롭게 할 수 있습니다\n", 34 | "\n", 35 | "## 6. Input 박스에 user_id, name, fullname을 입력해 비교해봅니다" 36 | ] 37 | } 38 | ], 39 | "metadata": { 40 | "kernelspec": { 41 | "display_name": "Python 3", 42 | "language": "python", 43 | "name": "python3" 44 | }, 45 | "language_info": { 46 | "codemirror_mode": { 47 | "name": "ipython", 48 | "version": 3 49 | }, 50 | "file_extension": ".py", 51 | "mimetype": "text/x-python", 52 | "name": "python", 53 | "nbconvert_exporter": "python", 54 | "pygments_lexer": "ipython3", 55 | "version": "3.6.1" 56 | }, 57 | "latex_envs": { 58 | "LaTeX_envs_menu_present": true, 59 | "autocomplete": true, 60 | "bibliofile": "biblio.bib", 61 | "cite_by": "apalike", 62 | "current_citInitial": 1, 63 | "eqLabelWithNumbers": true, 64 | "eqNumInitial": 1, 65 | "hotkeys": { 66 | "equation": "Ctrl-E", 67 | "itemize": "Ctrl-I" 68 | }, 69 | "labels_anchors": false, 70 | "latex_user_defs": false, 71 | "report_style_numbering": false, 72 | "user_envs_cfg": false 73 | }, 74 | "varInspector": { 75 | "cols": { 76 | "lenName": 16, 77 | "lenType": 16, 78 | "lenVar": 40 79 | }, 80 | "kernels_config": { 81 | "python": { 82 | "delete_cmd_postfix": "", 83 | "delete_cmd_prefix": "del ", 84 | "library": "var_list.py", 85 | "varRefreshCmd": "print(var_dic_list())" 86 | }, 87 | "r": { 88 | "delete_cmd_postfix": ") ", 89 | "delete_cmd_prefix": "rm(", 90 | "library": "var_list.r", 91 | "varRefreshCmd": "cat(var_dic_list()) " 92 | } 93 | }, 94 | "types_to_exclude": [ 95 | "module", 96 | "function", 97 | "builtin_function_or_method", 98 | "instance", 99 | "_Feature" 100 | ], 101 | "window_display": false 102 | } 103 | }, 104 | "nbformat": 4, 105 | "nbformat_minor": 2 106 | } 107 | -------------------------------------------------------------------------------- /Tableau/04. 테마별 상위 5위.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 테마별 상위 5위" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "1. 계산된 필드 생성\n", 15 | "2. Index() 추가\n", 16 | "3. Edit Table Calculation -> Restarting Every : 해당 컬럼 추가, Sort order : custom\n", 17 | "4. Filter에 Index 추가해서 제한을 걸면 끝" 18 | ] 19 | } 20 | ], 21 | "metadata": { 22 | "kernelspec": { 23 | "display_name": "Python 3", 24 | "language": "python", 25 | "name": "python3" 26 | }, 27 | "language_info": { 28 | "codemirror_mode": { 29 | "name": "ipython", 30 | "version": 3 31 | }, 32 | "file_extension": ".py", 33 | "mimetype": "text/x-python", 34 | "name": "python", 35 | "nbconvert_exporter": "python", 36 | "pygments_lexer": "ipython3", 37 | "version": "3.6.3" 38 | }, 39 | "latex_envs": { 40 | "LaTeX_envs_menu_present": true, 41 | "autocomplete": true, 42 | "bibliofile": "biblio.bib", 43 | "cite_by": "apalike", 44 | "current_citInitial": 1, 45 | "eqLabelWithNumbers": true, 46 | "eqNumInitial": 1, 47 | "hotkeys": { 48 | "equation": "Ctrl-E", 49 | "itemize": "Ctrl-I" 50 | }, 51 | "labels_anchors": false, 52 | "latex_user_defs": false, 53 | "report_style_numbering": false, 54 | "user_envs_cfg": false 55 | }, 56 | "varInspector": { 57 | "cols": { 58 | "lenName": 16, 59 | "lenType": 16, 60 | "lenVar": 40 61 | }, 62 | "kernels_config": { 63 | "python": { 64 | "delete_cmd_postfix": "", 65 | "delete_cmd_prefix": "del ", 66 | "library": "var_list.py", 67 | "varRefreshCmd": "print(var_dic_list())" 68 | }, 69 | "r": { 70 | "delete_cmd_postfix": ") ", 71 | "delete_cmd_prefix": "rm(", 72 | "library": "var_list.r", 73 | "varRefreshCmd": "cat(var_dic_list()) " 74 | } 75 | }, 76 | "types_to_exclude": [ 77 | "module", 78 | "function", 79 | "builtin_function_or_method", 80 | "instance", 81 | "_Feature" 82 | ], 83 | "window_display": false 84 | } 85 | }, 86 | "nbformat": 4, 87 | "nbformat_minor": 2 88 | } 89 | -------------------------------------------------------------------------------- /Tableau/05. 값이 비었을 경우 0으로 설정하기.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "- 0으로 설정할 값을 우클릭 - 서식 - 패널 - 특수 값에 숨기기(라인 연결) 및 숨기기(라인 끊기) 등 설정\n" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "- 만약 date가 안짤려 보이면, date를 불연속형으로 바꾸고 정확한 날짜에 선택한 후, 위에 프로세스를 진행하면 정상적으로 반영됨" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [] 23 | } 24 | ], 25 | "metadata": { 26 | "kernelspec": { 27 | "display_name": "Python 3", 28 | "language": "python", 29 | "name": "python3" 30 | }, 31 | "language_info": { 32 | "codemirror_mode": { 33 | "name": "ipython", 34 | "version": 3 35 | }, 36 | "file_extension": ".py", 37 | "mimetype": "text/x-python", 38 | "name": "python", 39 | "nbconvert_exporter": "python", 40 | "pygments_lexer": "ipython3", 41 | "version": "3.7.0" 42 | } 43 | }, 44 | "nbformat": 4, 45 | "nbformat_minor": 2 46 | } 47 | -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/__init__.py: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/__init__.py -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/__pycache__/__init__.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/__pycache__/__init__.cpython-36.pyc -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/__pycache__/mnist.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/__pycache__/mnist.cpython-36.pyc -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/lena.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/lena.png -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/lena_gray.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/lena_gray.png -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/mnist.py: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | try: 3 | import urllib.request 4 | except ImportError: 5 | raise ImportError('You should use Python 3.x') 6 | import os.path 7 | import gzip 8 | import pickle 9 | import os 10 | import numpy as np 11 | 12 | 13 | url_base = 'http://yann.lecun.com/exdb/mnist/' 14 | key_file = { 15 | 'train_img':'train-images-idx3-ubyte.gz', 16 | 'train_label':'train-labels-idx1-ubyte.gz', 17 | 'test_img':'t10k-images-idx3-ubyte.gz', 18 | 'test_label':'t10k-labels-idx1-ubyte.gz' 19 | } 20 | 21 | dataset_dir = os.path.dirname(os.path.abspath(__file__)) 22 | save_file = dataset_dir + "/mnist.pkl" 23 | 24 | train_num = 60000 25 | test_num = 10000 26 | img_dim = (1, 28, 28) 27 | img_size = 784 28 | 29 | 30 | def _download(file_name): 31 | file_path = dataset_dir + "/" + file_name 32 | 33 | if os.path.exists(file_path): 34 | return 35 | 36 | print("Downloading " + file_name + " ... ") 37 | urllib.request.urlretrieve(url_base + file_name, file_path) 38 | print("Done") 39 | 40 | def download_mnist(): 41 | for v in key_file.values(): 42 | _download(v) 43 | 44 | def _load_label(file_name): 45 | file_path = dataset_dir + "/" + file_name 46 | 47 | print("Converting " + file_name + " to NumPy Array ...") 48 | with gzip.open(file_path, 'rb') as f: 49 | labels = np.frombuffer(f.read(), np.uint8, offset=8) 50 | print("Done") 51 | 52 | return labels 53 | 54 | def _load_img(file_name): 55 | file_path = dataset_dir + "/" + file_name 56 | 57 | print("Converting " + file_name + " to NumPy Array ...") 58 | with gzip.open(file_path, 'rb') as f: 59 | data = np.frombuffer(f.read(), np.uint8, offset=16) 60 | data = data.reshape(-1, img_size) 61 | print("Done") 62 | 63 | return data 64 | 65 | def _convert_numpy(): 66 | dataset = {} 67 | dataset['train_img'] = _load_img(key_file['train_img']) 68 | dataset['train_label'] = _load_label(key_file['train_label']) 69 | dataset['test_img'] = _load_img(key_file['test_img']) 70 | dataset['test_label'] = _load_label(key_file['test_label']) 71 | 72 | return dataset 73 | 74 | def init_mnist(): 75 | download_mnist() 76 | dataset = _convert_numpy() 77 | print("Creating pickle file ...") 78 | with open(save_file, 'wb') as f: 79 | pickle.dump(dataset, f, -1) 80 | print("Done!") 81 | 82 | def _change_ont_hot_label(X): 83 | T = np.zeros((X.size, 10)) 84 | for idx, row in enumerate(T): 85 | row[X[idx]] = 1 86 | 87 | return T 88 | 89 | 90 | def load_mnist(normalize=True, flatten=True, one_hot_label=False): 91 | """MNIST 데이터셋 읽기 92 | 93 | Parameters 94 | ---------- 95 | normalize : 이미지의 픽셀 값을 0.0~1.0 사이의 값으로 정규화할지 정한다. 96 | one_hot_label : 97 | one_hot_label이 True면、레이블을 원-핫(one-hot) 배열로 돌려준다. 98 | one-hot 배열은 예를 들어 [0,0,1,0,0,0,0,0,0,0]처럼 한 원소만 1인 배열이다. 99 | flatten : 입력 이미지를 1차원 배열로 만들지를 정한다. 100 | 101 | Returns 102 | ------- 103 | (훈련 이미지, 훈련 레이블), (시험 이미지, 시험 레이블) 104 | """ 105 | if not os.path.exists(save_file): 106 | init_mnist() 107 | 108 | with open(save_file, 'rb') as f: 109 | dataset = pickle.load(f) 110 | 111 | if normalize: 112 | for key in ('train_img', 'test_img'): 113 | dataset[key] = dataset[key].astype(np.float32) 114 | dataset[key] /= 255.0 115 | 116 | if one_hot_label: 117 | dataset['train_label'] = _change_ont_hot_label(dataset['train_label']) 118 | dataset['test_label'] = _change_ont_hot_label(dataset['test_label']) 119 | 120 | if not flatten: 121 | for key in ('train_img', 'test_img'): 122 | dataset[key] = dataset[key].reshape(-1, 1, 28, 28) 123 | 124 | return (dataset['train_img'], dataset['train_label']), (dataset['test_img'], dataset['test_label']) 125 | 126 | 127 | if __name__ == '__main__': 128 | init_mnist() 129 | -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/t10k-images-idx3-ubyte.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/t10k-images-idx3-ubyte.gz -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/t10k-labels-idx1-ubyte.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/t10k-labels-idx1-ubyte.gz -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/train-images-idx3-ubyte.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/train-images-idx3-ubyte.gz -------------------------------------------------------------------------------- /deep_learning_from_scratch/dataset/train-labels-idx1-ubyte.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/dataset/train-labels-idx1-ubyte.gz -------------------------------------------------------------------------------- /deep_learning_from_scratch/deep_convnet_params.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/deep_convnet_params.pkl -------------------------------------------------------------------------------- /deep_learning_from_scratch/params.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/params.pkl -------------------------------------------------------------------------------- /deep_learning_from_scratch/sample_weight.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/deep_learning_from_scratch/sample_weight.pkl -------------------------------------------------------------------------------- /docker/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.1' 2 | 3 | services: 4 | db: 5 | image: mysql:5.6 6 | container_name: mysql5.6 7 | ports: 8 | - "3306:3306" 9 | restart: always 10 | environment: 11 | - MYSQL_USER=test 12 | - MYSQL_PASSWORD=test 13 | - MYSQL_ROOT_PASSWORD=root1234 14 | - MYSQL_DATABASE=test 15 | volumes: 16 | - ./data:/var/lib/mysql 17 | -------------------------------------------------------------------------------- /information_graphics/2장. 똑똑하게 차트 그리기.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 2장. 똑똑하게 차트 그리기\n" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## 선\n", 15 | "### 높이와 굵기\n", 16 | "- 선 아래의 음영 처리는 반드시 차트의 기준선이 0인 경우에만 허용! 0에서 시작하지 않는 추세선 아래에 색상을 넣으면 데이터가 잘림\n", 17 | "- 기울기가 너무 평평하면 메세지가 분명하지 않다. 그렇다고 기울기를 너무 과장하면 메시지도 과장된다. 적절한 판단이 중요\n", 18 | "- **적절한 높이는 차트의 2/3 영역을 차지!**\n", 19 | "- 선이 너무 얇아도 문제, 굵어도 문제..! 적절한 굵기를 선정해야함\n", 20 | "- 시작값을 0으로 잡는 것이 좋으나, 데이터 크기가 크다면 적절한 위치에서 끊는 것도 좋다. 단, 그리드선을 일관되게 유지할 것\n", 21 | "- 차트 하나에 선은 최대 4개(3개 이하를 권장)\n", 22 | "\n", 23 | "\n", 24 | "\n", 25 | "### 이중축 척도\n", 26 | "- 성질이 다른 것을 섞지마라 : (연관성이 없는) 변수를 겹쳐놓으면 차트가 더 혼란스러워짐. ex) 매출액 - 시장지수 \n", 27 | "- (연관성이 있는) 변수를 같이 보면 데이터의 움직임을 볼 수 있다. ex) 매출액 - 시장점유율\n", 28 | "- 두개 이상의 차트를 같은 공간에 병렬로 배치할 경우엔 y축 범위를 동일하게 해야 편향된 이해를 피할 수 있다\n", 29 | "\n", 30 | "### 숫자 맞춤과 항목 배열\n", 31 | "- 정수는 왼쪽 정렬하지 않는다 -> 오른쪽 정렬\n", 32 | "- 단 크기가 작은 숫자는 가운데 정렬도 허용(1자리수)\n", 33 | "- 소수는 왼쪽이나 오른쪽으로 정렬하지 않는다 -> 소수점 기준으로 정렬\n", 34 | "- 논리의 순서에 따라 배열 (알파벳 순서, 데이터 크기 순서 등)\n", 35 | "- 백분율, 달러같은 단위는 첫번쨰 항목에서 1번만 표시한다\n", 36 | "\n" 37 | ] 38 | } 39 | ], 40 | "metadata": { 41 | "kernelspec": { 42 | "display_name": "Python 3", 43 | "language": "python", 44 | "name": "python3" 45 | }, 46 | "language_info": { 47 | "codemirror_mode": { 48 | "name": "ipython", 49 | "version": 3 50 | }, 51 | "file_extension": ".py", 52 | "mimetype": "text/x-python", 53 | "name": "python", 54 | "nbconvert_exporter": "python", 55 | "pygments_lexer": "ipython3", 56 | "version": "3.6.0" 57 | } 58 | }, 59 | "nbformat": 4, 60 | "nbformat_minor": 2 61 | } 62 | -------------------------------------------------------------------------------- /information_graphics/flowchartmod.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/information_graphics/flowchartmod.jpg -------------------------------------------------------------------------------- /mac/README.md: -------------------------------------------------------------------------------- 1 | # 맥북 기본 설정 2 | 3 | - 화면 분할 기능 : [Spectacle](https://www.spectacleapp.com/) 다운로드 후 손쉬운 사용에서 권한주면 사용 가능 [참고](http://macnews.tistory.com/3198) 4 | - Alfred : alt + space로 검색 및 유틸리티 열기 가능 5 | - karabiner : 한영전환키를 우측 command로 설정 가능! [참고](http://macnews.tistory.com/5043) 6 | - 맥북 화면보호기 [링크](https://plus.google.com/featuredphotos) 7 | - 키보드-입력소스- Caps Lock키로 abc 입력소스 전환 해제하면 좌측 caps lock은 본래의 기능을 사용하게 됨 8 | 9 | - 확인되지 않은 개발자가 배포한 어플을 실행하고 싶은 경우, 보안 및 개인 정보 보호쪽에서 확인없이 열기를 체크하거나 터미널에서 아래와 같이 입력한 후 설치! (다시 키고싶다면 -enable) 10 | ~~~ 11 | sudo spctl --master-disable 12 | ~~~ 13 | - 그 후, app market에서 update 14 | 15 | ### zsh setting 16 | - zsh 설치 [진한이형 블로그](http://hjh5488.tistory.com/2) 17 | - 폰트가 깨지는 경우 [meslo](https://github.com/powerline/fonts/blob/master/Meslo/Meslo%20LG%20M%20DZ%20Regular%20for%20Powerline.otf) 설치 후 설정 18 | - ```open .``` 입력하면 finder 열림 19 | 20 | ### Database tool 21 | - [Sequel Pro](https://www.sequelpro.com/) : MySQL 22 | - [PSequel](http://www.psequel.com/) : PostgreSQL용이라고 하는데 아직 많이 쓰는진 모르겠음 23 | 24 | ### iterm2 단축키 25 | - command + d : 우측에 새 화면 추가 26 | - command + shift + d : 하단에 새 화면 추가 27 | - command + t : 새 창 추가 28 | - command + [] : 화면 이동 29 | - command + w : 화면 닫기 30 | - command + q : iterm 종료 31 | 32 | ### markdown 편집기 33 | - macdown, typora 34 | 35 | ### Vim 명령어 36 | - [블로그 참고](https://zzsza.github.io/development/2018/07/20/vim-tips/) 37 | 38 | ### Vim 명령모드시 자동 영문전환 39 | ``` 40 | brew install cmake 41 | git clone https://github.com/vovkasm/input-source-switcher.git 42 | cd input-source-switcher 43 | mkdir build && cd build 44 | cmake .. 45 | make 46 | make install 47 | ``` 48 | 49 | - ```.vimrc```에 설정 추가 50 | 51 | ``` 52 | if filereadable('/usr/local/lib/libInputSourceSwitcher.dylib') 53 | autocmd InsertLeave * call libcall('/usr/local/lib/libInputSourceSwitcher.dylib', 'Xkb_Switch_setXkbLayout', 'com.apple.keylayout.ABC') 54 | endif 55 | ``` -------------------------------------------------------------------------------- /mac/intellij.md: -------------------------------------------------------------------------------- 1 | 인텔리제이, Intellij 2 | 3 | # Toolbox 소개 및 프로젝트 생성하기 4 | - 인텔리J를 직접 다운로드하는 것보다 Toolbox App을 설치하고 거기서 인텔리J를 사용하는 것을 추천 5 | - Toolbox는 설정을 앱 안에서 해결할 수 있음 => maximum heap size 6 | - Update to Release(EAP라는 프리뷰때 알람을 할지) 7 | 8 | # 메인메소드 생성하고 실행하기 9 | - Find Action : Command shift A 10 | - plugin : presentation assistant 11 | 12 | 흔히 사용하는 메서드 저장된 것이 있음 => 라이브 템플릿 / 코드 템플릿 13 | 14 | RUN : 포커스 실행(control shift R), 15 | RUN ( control R ) 이전 실행 다시 실행 16 | 17 | 18 | # 라인 수정하기 19 | 한줄 드래그 -> 복사 -> 붙여넣기 과정을 단축키 하나로 해결 20 | 21 | - 아래에 동일한 내용 복사 : command D (포커스를 해당 라인에 두고!, 엑셀 기능과 유사하네) 22 | - 해당 줄 내용 삭제 : command + delete 23 | - 라인 합치기 : 하단의 문자열이 합쳐짐(위로 올라옴), control shift j, SQL Query를 만들 때 자주 사용함 24 | - 라인 옮기기 25 | - option shift 위아래 : 문법에 상관없이 이동 26 | - command shift 위아래 : 이 메소드를 벗어날 수 없음. 파이썬에선 메소드 안이 아니어도 될듯? 27 | - 좌우로 요소 이동 : option shift command 좌우 (html에서 자주 쓰일듯) 28 | 29 | # 코드 즉시보기 30 | - 인자값 즉시 보기 : command p 클래스에 들어가서 인자값이 무엇인지 찾기 31 | - 선언부 바로가기 : option space (난 alfred 때문에 안보임) 32 | - docs 미리보기 : f1 (pycharm은 새 창이 뜨네..) 33 | 34 | # 포커스 에디터 35 | - 단어별 이동 : option + 화살표 36 | - 이동하며 선택 : option + shift + 화살표 37 | - 라인의 제일 마지막으로 이동 : function + 좌우(command 좌우랑 무슨 차이가 있지?) 38 | - page up/down : function + 위아래 39 | 40 | # 포커스 특수키 41 | - Extend selection : option + 위아래 ( 파이참은 작동 안함 ) 42 | - 포커스 이전으로 바꾸기 : command + [] 43 | - 멀티 포커스 : option 2번(누르고 있는 상태에서) + 방향키 44 | - 오류 포커스 : F2 45 | 46 | 47 | # 검색 텍스트 48 | - 단어 찾기 : command f 49 | - 단어 수정 : command r 50 | - 프로젝트에서 찾기 : shift command f 51 | - 프로젝트에서 수정 : shift command r 52 | - 찾기에서 정규 표현식 사용 가능 53 | 54 | 55 | - 최근 열었던 파일 목록 : command e 56 | 57 | # 자동 완성 58 | - 스마트 자동 완성 : control shift space 59 | 60 | 61 | # Live Template 62 | - 포커스에 있는 모든 축약어 나오기 : control j 63 | - command shift a : live template(preference)에 추가 가능 64 | 65 | # 리팩토링 - Extract 66 | - option command v : value 67 | - option command p : 파라미터 68 | - option command m : method 69 | 70 | # 리팩토링 기타 71 | - 이름 변경 : shift f6 72 | - 타입 변경 : shift command f6 73 | - import 정리 : command option o 74 | - optimize import on 75 | 76 | 77 | # 디버깅 78 | - 디버깅 실행 : command shift d 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | -------------------------------------------------------------------------------- /python/Downloading a picture via urllib and python.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Dowloading a picture via urllib and python\n", 8 | "- python2 는 urllib 아래에 urlretrieve가 존재\n", 9 | "- python3은 urllib.requrest 아래에 urlretrieve가 존재" 10 | ] 11 | }, 12 | { 13 | "cell_type": "code", 14 | "execution_count": 2, 15 | "metadata": { 16 | "ExecuteTime": { 17 | "end_time": "2017-10-30T11:44:37.512465Z", 18 | "start_time": "2017-10-30T11:44:37.507106Z" 19 | } 20 | }, 21 | "outputs": [], 22 | "source": [ 23 | "import sys\n", 24 | "\n", 25 | "if sys.version_info[0] >= 3:\n", 26 | " from urllib.request import urlretrieve\n", 27 | "else:\n", 28 | " from urllib import urlretrieve\n", 29 | "\n", 30 | "\n", 31 | "urlretrieve(\"http://www-scf.usc.edu/~chiso/oldspice/m-b1-hello.jpg\")\n", 32 | "# jpg가 아닌 mp3 파일 등도 다운 가능" 33 | ] 34 | } 35 | ], 36 | "metadata": { 37 | "kernelspec": { 38 | "display_name": "Python 3", 39 | "language": "python", 40 | "name": "python3" 41 | }, 42 | "language_info": { 43 | "codemirror_mode": { 44 | "name": "ipython", 45 | "version": 3 46 | }, 47 | "file_extension": ".py", 48 | "mimetype": "text/x-python", 49 | "name": "python", 50 | "nbconvert_exporter": "python", 51 | "pygments_lexer": "ipython3", 52 | "version": "3.6.1" 53 | }, 54 | "latex_envs": { 55 | "LaTeX_envs_menu_present": true, 56 | "autocomplete": true, 57 | "bibliofile": "biblio.bib", 58 | "cite_by": "apalike", 59 | "current_citInitial": 1, 60 | "eqLabelWithNumbers": true, 61 | "eqNumInitial": 1, 62 | "hotkeys": { 63 | "equation": "Ctrl-E", 64 | "itemize": "Ctrl-I" 65 | }, 66 | "labels_anchors": false, 67 | "latex_user_defs": false, 68 | "report_style_numbering": false, 69 | "user_envs_cfg": false 70 | }, 71 | "varInspector": { 72 | "cols": { 73 | "lenName": 16, 74 | "lenType": 16, 75 | "lenVar": 40 76 | }, 77 | "kernels_config": { 78 | "python": { 79 | "delete_cmd_postfix": "", 80 | "delete_cmd_prefix": "del ", 81 | "library": "var_list.py", 82 | "varRefreshCmd": "print(var_dic_list())" 83 | }, 84 | "r": { 85 | "delete_cmd_postfix": ") ", 86 | "delete_cmd_prefix": "rm(", 87 | "library": "var_list.r", 88 | "varRefreshCmd": "cat(var_dic_list()) " 89 | } 90 | }, 91 | "types_to_exclude": [ 92 | "module", 93 | "function", 94 | "builtin_function_or_method", 95 | "instance", 96 | "_Feature" 97 | ], 98 | "window_display": false 99 | } 100 | }, 101 | "nbformat": 4, 102 | "nbformat_minor": 2 103 | } 104 | -------------------------------------------------------------------------------- /python/Effective Python.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# Effective Python\n" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Better way 3\n", 15 | "\n", 16 | "- 파이썬 3 문자 시퀀스 - bytes (raw 8비트) / str (유니코드 문자)\n", 17 | "- 파이썬 2 문자 시퀀스 - str (raw 8비트) / unicode (유니코드 문자)\n", 18 | "\n", 19 | "유니코드 문자 -> 바이너리 데이터(raw 8비트)로 표현하는 방법은 많음. 일반적으로 UTF-8\n", 20 | "\n", 21 | "- 파이썬 3의 str 인스턴스와 파이썬 2의 unicode 인스턴스는 연관된 바이너리 인코딩이 없음\n", 22 | "- 유니코드 문자를 바이너리 데이터로 변환하려면 encode 메서드를 사용, 바이너리를 유니코드 문자로 변환할 경우 decode 메서드 사용\n", 23 | "\n" 24 | ] 25 | }, 26 | { 27 | "cell_type": "markdown", 28 | "metadata": { 29 | "collapsed": true 30 | }, 31 | "source": [ 32 | "# better way 45\n", 33 | "- 지역 시간은 time이 아닌 datetime으로 표현\n", 34 | "- UTC (협정 세계시) : 시간대에 의존하지 않는 표준 시간 표현.\n", 35 | "\n", 36 | "### time\n", 37 | "- 내장 모듈 time의 localtime 함수는 호스트 컴퓨터의 시간대로 변환\n", 38 | "\n", 39 | "### datetime\n", 40 | "- 한 지역의 시간을 다른 지역 시간으로 신뢰성 있게 변경ㅎ나, tzinfo 클래스와 관련 메서드를 이용한 시간대 변환 기능만 제공\n", 41 | "\n", 42 | "### pytz\n", 43 | "- 모든 시간대에 대한 정의를 담은 전체 데이터베이스를 포함-!\n", 44 | "- 지역 시간 -> utc -> 지역 시간" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": { 51 | "collapsed": true 52 | }, 53 | "outputs": [], 54 | "source": [] 55 | } 56 | ], 57 | "metadata": { 58 | "kernelspec": { 59 | "display_name": "Python 3", 60 | "language": "python", 61 | "name": "python3" 62 | }, 63 | "language_info": { 64 | "codemirror_mode": { 65 | "name": "ipython", 66 | "version": 3 67 | }, 68 | "file_extension": ".py", 69 | "mimetype": "text/x-python", 70 | "name": "python", 71 | "nbconvert_exporter": "python", 72 | "pygments_lexer": "ipython3", 73 | "version": "3.6.1" 74 | } 75 | }, 76 | "nbformat": 4, 77 | "nbformat_minor": 2 78 | } 79 | -------------------------------------------------------------------------------- /python/No module named 'MySQLdb' error.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## No module named 'MySQLdb' error\n", 8 | "\n", 9 | "- 이 에러는 python2에서 사용하던 코드를 python3에서 사용하는 경우 생기는 에러\n", 10 | "\n", 11 | "\n", 12 | "## 해결방법\n", 13 | "- pip3 install pymysql\n", 14 | "\n" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 1, 20 | "metadata": { 21 | "collapsed": true 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "try:\n", 26 | " import pymysql\n", 27 | " pymysql.install_as_MySQLdb()\n", 28 | "except ImportError:\n", 29 | " pass" 30 | ] 31 | }, 32 | { 33 | "cell_type": "code", 34 | "execution_count": null, 35 | "metadata": { 36 | "collapsed": true 37 | }, 38 | "outputs": [], 39 | "source": [ 40 | "Base.metadata.create_all(create_engine('This://'))\n" 41 | ] 42 | }, 43 | { 44 | "cell_type": "markdown", 45 | "metadata": {}, 46 | "source": [ 47 | "- This에 'mysql+pymysql'을 추가해주면 정상적으로 작동" 48 | ] 49 | } 50 | ], 51 | "metadata": { 52 | "kernelspec": { 53 | "display_name": "Python 3", 54 | "language": "python", 55 | "name": "python3" 56 | }, 57 | "language_info": { 58 | "codemirror_mode": { 59 | "name": "ipython", 60 | "version": 3 61 | }, 62 | "file_extension": ".py", 63 | "mimetype": "text/x-python", 64 | "name": "python", 65 | "nbconvert_exporter": "python", 66 | "pygments_lexer": "ipython3", 67 | "version": "3.6.1" 68 | } 69 | }, 70 | "nbformat": 4, 71 | "nbformat_minor": 2 72 | } 73 | -------------------------------------------------------------------------------- /python/Select Multiple value in list.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 6, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "# value별로 값을 다르게 주고싶어서 진행\n", 12 | "\n", 13 | "hash_tags = ['코딩','개발자','빅데이터','머신러닝','딥러닝','coding','developer','bigdata','machinelearning','deeplearning']\n", 14 | "most_like_tags = ['코딩','개발자','빅데이터']\n" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": 5, 20 | "metadata": {}, 21 | "outputs": [], 22 | "source": [ 23 | "for hash_tag in hash_tags:\n", 24 | " if any(e in hash_tag for e in most_like_tags):\n", 25 | " count_number = 100\n", 26 | " else:\n", 27 | " count_number = 20" 28 | ] 29 | }, 30 | { 31 | "cell_type": "markdown", 32 | "metadata": {}, 33 | "source": [ 34 | "any(iterable)\n", 35 | "- Return True if bool(x) is True for any x in the iterable." 36 | ] 37 | } 38 | ], 39 | "metadata": { 40 | "kernelspec": { 41 | "display_name": "Python 3", 42 | "language": "python", 43 | "name": "python3" 44 | }, 45 | "language_info": { 46 | "codemirror_mode": { 47 | "name": "ipython", 48 | "version": 3 49 | }, 50 | "file_extension": ".py", 51 | "mimetype": "text/x-python", 52 | "name": "python", 53 | "nbconvert_exporter": "python", 54 | "pygments_lexer": "ipython3", 55 | "version": "3.6.1" 56 | }, 57 | "latex_envs": { 58 | "LaTeX_envs_menu_present": true, 59 | "autocomplete": true, 60 | "bibliofile": "biblio.bib", 61 | "cite_by": "apalike", 62 | "current_citInitial": 1, 63 | "eqLabelWithNumbers": true, 64 | "eqNumInitial": 1, 65 | "hotkeys": { 66 | "equation": "Ctrl-E", 67 | "itemize": "Ctrl-I" 68 | }, 69 | "labels_anchors": false, 70 | "latex_user_defs": false, 71 | "report_style_numbering": false, 72 | "user_envs_cfg": false 73 | } 74 | }, 75 | "nbformat": 4, 76 | "nbformat_minor": 2 77 | } 78 | -------------------------------------------------------------------------------- /python/Slack bot ( Slacker ).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "# pip3 install slacker\n", 12 | "from slacker import Slacker" 13 | ] 14 | }, 15 | { 16 | "cell_type": "markdown", 17 | "metadata": {}, 18 | "source": [ 19 | "[slack api 사이트](https://api.slack.com/) -> 'start building' -> Create app -> OAuth & Permissions에 있는 Bot User OAuth Access Token 값을 아래 token에 넣으면 됩니다" 20 | ] 21 | }, 22 | { 23 | "cell_type": "code", 24 | "execution_count": null, 25 | "metadata": { 26 | "collapsed": true 27 | }, 28 | "outputs": [], 29 | "source": [ 30 | "token = 'xoxo-your-token'\n", 31 | "slack = Slacker(token)\n", 32 | "slack.chat.post_message(channel='채널명 혹은 채널 id', text='message test', as_user=False)" 33 | ] 34 | }, 35 | { 36 | "cell_type": "code", 37 | "execution_count": null, 38 | "metadata": { 39 | "collapsed": true 40 | }, 41 | "outputs": [], 42 | "source": [ 43 | "# Channel 확인 : id값 확인 가능\n", 44 | "slack.channels.list().body\n" 45 | ] 46 | }, 47 | { 48 | "cell_type": "code", 49 | "execution_count": null, 50 | "metadata": { 51 | "collapsed": true 52 | }, 53 | "outputs": [], 54 | "source": [ 55 | "# Group 확인 : id값 확인 가능\n", 56 | "slack.groups.list().body\n" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": { 63 | "collapsed": true 64 | }, 65 | "outputs": [], 66 | "source": [] 67 | } 68 | ], 69 | "metadata": { 70 | "kernelspec": { 71 | "display_name": "Python 3", 72 | "language": "python", 73 | "name": "python3" 74 | }, 75 | "language_info": { 76 | "codemirror_mode": { 77 | "name": "ipython", 78 | "version": 3 79 | }, 80 | "file_extension": ".py", 81 | "mimetype": "text/x-python", 82 | "name": "python", 83 | "nbconvert_exporter": "python", 84 | "pygments_lexer": "ipython3", 85 | "version": "3.6.1" 86 | } 87 | }, 88 | "nbformat": 4, 89 | "nbformat_minor": 2 90 | } 91 | -------------------------------------------------------------------------------- /python/aiohttp-example.py: -------------------------------------------------------------------------------- 1 | import aiohttp 2 | import asyncio 3 | 4 | async def get(url): 5 | async with aiohttp.ClientSession() as session: 6 | async with session.get(url) as response: 7 | return response 8 | 9 | loop = asyncio.get_event_loop() 10 | coroutines = [get("http://example.com") for _ in range(8)] 11 | results = loop.run_until_complete(asyncio.gather(*coroutines)) 12 | 13 | print(f"Results: {results}") -------------------------------------------------------------------------------- /python/asyncio-coroutine.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import time 3 | 4 | async def add_42(number): 5 | print("Adding 42") 6 | return 42 + number 7 | 8 | 9 | async def hello_world(): 10 | print("Hello world!") 11 | result = await add_42(23) 12 | time.sleep(5) 13 | return result 14 | 15 | 16 | event_loop = asyncio.get_event_loop() 17 | 18 | try: 19 | result = event_loop.run_until_complete(hello_world()) 20 | print(result) 21 | finally: 22 | event_loop.close() -------------------------------------------------------------------------------- /python/asyncio-helloy.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | 3 | 4 | async def hello_world(): 5 | print("Hello World") 6 | return 42 7 | 8 | 9 | hello_world_coroutine = hello_world() 10 | 11 | print(hello_world_coroutine) 12 | 13 | event_loop = asyncio.get_event_loop() 14 | try: 15 | print("entering event loop") 16 | result = event_loop.run_until_complete(hello_world_coroutine) 17 | print(result) 18 | finally: 19 | event_loop.close() -------------------------------------------------------------------------------- /python/asyncio-sleep.py: -------------------------------------------------------------------------------- 1 | import asyncio 2 | import time 3 | 4 | 5 | async def hello_world(): 6 | print("hello world!") 7 | 8 | 9 | async def hello_python(): 10 | print("hello python!") 11 | await asyncio.sleep(0.1) 12 | 13 | event_loop = asyncio.get_event_loop() 14 | try: 15 | result = event_loop.run_until_complete(asyncio.gather( 16 | hello_world(), 17 | hello_python(), 18 | )) 19 | print(result) 20 | finally: 21 | event_loop.close() 22 | 23 | # asyncio.sleep은 time.sleep의 비동기식 구현 24 | # asynci.gather는 await 키워드를 한번만 사용해 코루틴 여러개ㅡㄹ 기다릴 때 사용 -------------------------------------------------------------------------------- /python/cotyledon-daemon.py: -------------------------------------------------------------------------------- 1 | import threading 2 | import time 3 | 4 | import cotyledon 5 | 6 | 7 | class PrinterSerivce(cotyledon.Service): 8 | name = "printer" 9 | 10 | def __init__(self, worker_id): 11 | super(PrinterSerivce, self).__init__(worker_id) 12 | self._shutdown = threading.Event() 13 | 14 | def run(self): 15 | while not self._shutdown.is_set(): 16 | print("Doing stuff") 17 | time.sleep(1) 18 | 19 | def terminate(self): 20 | self._shutdonw.set() 21 | 22 | 23 | # manager 생성 24 | manager = cotyledon.ServiceManager() 25 | # PrinterService 2개를 실행하기 위해 추가 26 | manager.add(PrinterSerivce, 2) 27 | manager.run() 28 | 29 | -------------------------------------------------------------------------------- /python/cotyledon-producer-consumer.py: -------------------------------------------------------------------------------- 1 | import multiprocessing 2 | import time 3 | import cotyledon 4 | 5 | 6 | class Manager(cotyledon.ServiceManager): 7 | def __init__(self): 8 | super(Manager, self).__init__() 9 | queue = multiprocessing.Manager().Queue() 10 | self.add(ProducerService, args=(queue, )) 11 | self.add(PrinterService, args=(queue, ), workers=2) 12 | 13 | 14 | class ProducerService(cotyledon.Service): 15 | def __init__(self, worker_id, queue): 16 | super(ProducerService, self).__init__(worker_id) 17 | self.queue = queue 18 | 19 | def run(self): 20 | i = 0 21 | while True: 22 | self.queue.put(i) 23 | i += 1 24 | time.sleep(1) 25 | 26 | 27 | class PrinterService(cotyledon.Service): 28 | name = "printer" 29 | 30 | def __init__(self, worker_id, queue): 31 | super(PrinterService, self).__init__(worker_id) 32 | self.queue = queue 33 | 34 | def run(self): 35 | while True: 36 | job = self.queue.get(block=True) 37 | print(f"I am Worker: {self.worker_id} PID: {self.pid} and I print {job}") 38 | 39 | 40 | Manager().run() -------------------------------------------------------------------------------- /python/cotyledon-reconfig-process.py: -------------------------------------------------------------------------------- 1 | import multiprocessing 2 | import time 3 | 4 | import cotyledon 5 | 6 | 7 | class Manager(cotyledon.ServiceManager): 8 | def __init__(self): 9 | super(Manager, self).__init__() 10 | queue = multiprocessing.Manager().Queue() 11 | self.add(ProducerService, args=(queue,)) 12 | self.printer = self.add(PrinterService, args=(queue,), workers=2) 13 | self.register_hooks(on_reload=self.reload) 14 | 15 | def reload(self): 16 | print("Reloading") 17 | self.reconfigure(self.printer, 5) 18 | 19 | 20 | class ProducerService(cotyledon.Service): 21 | def __init__(self, worker_id, queue): 22 | super(ProducerService, self).__init__(worker_id) 23 | self.queue = queue 24 | 25 | def run(self): 26 | i = 0 27 | while True: 28 | self.queue.put(i) 29 | i += 1 30 | time.sleep(1) 31 | 32 | 33 | class PrinterService(cotyledon.Service): 34 | name = "printer" 35 | 36 | def __init__(self, worker_id, queue): 37 | super(PrinterService, self).__init__(worker_id) 38 | self.queue = queue 39 | 40 | def run(self): 41 | while True: 42 | job = self.queue.get(block=True) 43 | print(f"I am Worker: {self.worker_id} PID: {self.pid} and I print {job}") 44 | 45 | 46 | Manager().run() -------------------------------------------------------------------------------- /python/crawling/advanced_link_crawler.py: -------------------------------------------------------------------------------- 1 | import re 2 | from urllib import robotparser 3 | from urllib.parse import urljoin 4 | from downloader import Downloader 5 | 6 | 7 | def get_robots_parser(robots_url): 8 | rp = robotparser.RobotFileParser() 9 | rp.set_url(robots_url) 10 | rp.read() 11 | return rp 12 | 13 | 14 | def get_links(html): 15 | webpage_regex = re.compile("""]+href=["'](.*?)["']""", re.IGNORECASE) 16 | return webpage_regex.findall(html) 17 | 18 | 19 | def link_crawler(start_url, link_regex, robots_url=None, user_agent='wswp', 20 | proxies=None, delay=3, max_depth=4, num_retries=2, cache={}, scraper_callback=None): 21 | crawl_queue = [start_url] 22 | seen = {} 23 | if not robots_url: 24 | robots_url = '{}/robots.txt'.format(start_url) 25 | rp = get_robots_parser(robots_url) 26 | D = Downloader(delay=delay, user_agent=user_agent, proxies=proxies, cache=cache) 27 | while crawl_queue: 28 | url = crawl_queue.pop() 29 | if rp.can_fetch(user_agent, url): 30 | depth = seen.get(url, 0) 31 | if depth == max_depth: 32 | print('Skipping %s due to depth' % url) 33 | continue 34 | html = D(url, num_retries=num_retries) 35 | if not html: 36 | continue 37 | if scraper_callback: 38 | links = scraper_callback(url, html) or [] 39 | else: 40 | links = [] 41 | for link in get_links(html) + links: 42 | if re.match(link_regex, link): 43 | abs_link = urljoin(start_url, link) 44 | if abs_link not in seen: 45 | seen[abs_link] = depth + 1 46 | crawl_queue.append(abs_link) 47 | else: 48 | print('Blocked by robots.txt:', url) 49 | -------------------------------------------------------------------------------- /python/crawling/diskcache.py: -------------------------------------------------------------------------------- 1 | import os 2 | import json 3 | import re 4 | import zlib 5 | 6 | from datetime import datetime, timedelta 7 | from urllib.parse import urlsplit 8 | 9 | 10 | class DiskCache: 11 | def __init__(self, cache_dir='../data/cache', max_len=255, compress=True, 12 | encoding='utf-8', expires=timedelta(days=30)): 13 | self.cache_dir = cache_dir 14 | self.max_len = max_len 15 | self.compress = compress 16 | self.encoding = encoding 17 | self.expires = expires 18 | 19 | def url_to_path(self, url): 20 | components = urlsplit(url) 21 | path = components.path 22 | if not path.endswith('/'): 23 | path += '/index.html' 24 | elif path.endswith('/'): 25 | path += 'index.html' 26 | filename = components.netloc + path + components.query 27 | filename = re.sub(r'[^/0-9a-zA-Z\-.,;_ ]', '_', filename) 28 | filename = '/'.join(seg[:self.max_len] for seg in filename.split('/')) 29 | return os.path.join(self.cache_dir, filename) 30 | 31 | def __getitem__(self, url): 32 | path = self.url_to_path(url) 33 | if os.path.exists(path): 34 | mode = ('rb' if self.compress else 'r') 35 | with open(path, mode) as fp: 36 | if self.compress: 37 | data = zlib.decompress(fp.read()).decode(self.encoding) 38 | data = json.loads(data) 39 | else: 40 | data = json.load(fp) 41 | exp_date = data.get('expires') 42 | if exp_date and datetime.strptime(exp_date, '%Y-%m-%dT%H:%M:%S') <= datetime.utcnow(): 43 | print('Cache expired!', exp_date) 44 | raise KeyError(url + ' has expired.') 45 | return data 46 | else: 47 | raise KeyError(url + ' does not exist') 48 | 49 | def __setitem__(self, url, result): 50 | path = self.url_to_path(url) 51 | folder = os.path.dirname(path) 52 | if not os.path.exists(folder): 53 | os.makedirs(folder) 54 | mode = ('wb' if self.compress else 'w') 55 | result['expires'] = (datetime.utcnow() + self.expires).isoformat( 56 | timespec='seconds') 57 | with open(path, mode) as fp: 58 | if self.compress: 59 | data = bytes(json.dumps(result), self.encoding) 60 | fp.write(zlib.compress(data)) 61 | else: 62 | json.dump(result, fp) 63 | -------------------------------------------------------------------------------- /python/crawling/downloader.py: -------------------------------------------------------------------------------- 1 | from random import choice 2 | import requests 3 | 4 | from throttle import Throttle 5 | 6 | 7 | class Downloader: 8 | def __init__(self, delay=5, user_agent='wswp', proxies=None, cache={}, 9 | timeout=60): 10 | self.throttle = Throttle(delay) 11 | self.user_agent = user_agent 12 | self.proxies = proxies 13 | self.cache = cache 14 | self.num_retries = None # we will set this per request 15 | self.timeout = timeout 16 | 17 | def __call__(self, url, num_retries=2): 18 | self.num_retries = num_retries 19 | try: 20 | result = self.cache[url] 21 | print('Loaded from cache:', url) 22 | except KeyError: 23 | result = None 24 | if result and self.num_retries and 500 <= result['code'] < 600: 25 | result = None 26 | if result is None: 27 | self.throttle.wait(url) 28 | proxies = choice(self.proxies) if self.proxies else None 29 | headers = {'User-Agent': self.user_agent} 30 | result = self.download(url, headers, proxies) 31 | self.cache[url] = result 32 | return result['html'] 33 | 34 | def download(self, url, headers, proxies): 35 | print('Downloading:', url) 36 | try: 37 | resp = requests.get(url, headers=headers, proxies=proxies, 38 | timeout=self.timeout) 39 | html = resp.text 40 | if resp.status_code >= 400: 41 | print('Download error:', resp.text) 42 | html = None 43 | if self.num_retries and 500 <= resp.status_code < 600: 44 | self.num_retries -= 1 45 | return self.download(url, headers, proxies) 46 | except requests.exceptions.RequestException as e: 47 | print('Download error:', e) 48 | return {'html': None, 'code': 500} 49 | return {'html': html, 'code': resp.status_code} 50 | -------------------------------------------------------------------------------- /python/crawling/redis_cache.py: -------------------------------------------------------------------------------- 1 | import json 2 | import zlib 3 | from datetime import datetime, timedelta 4 | from redis import StrictRedis 5 | 6 | 7 | class RedisCache: 8 | """ RedisCache helps store urls and their responses to Redis 9 | Initialization components: 10 | client: a Redis client connected to the key-value database for 11 | the webcrawling cache (if not set, a localhost:6379 12 | default connection is used). 13 | expires (datetime.timedelta): timedelta when content will expire 14 | (default: 30 days ago) 15 | encoding (str): character encoding for serialization 16 | compress (bool): boolean indicating whether compression with zlib should be used 17 | """ 18 | def __init__(self, client=None, expires=timedelta(days=30), encoding='utf-8', compress=True): 19 | self.client = (StrictRedis(host='localhost', port=6379, db=0) 20 | if client is None else client) 21 | self.expires = expires 22 | self.encoding = encoding 23 | self.compress = compress 24 | 25 | def __getitem__(self, url): 26 | """Load data from Redis for given URL""" 27 | record = self.client.get(url) 28 | if record: 29 | if self.compress: 30 | record = zlib.decompress(record) 31 | return json.loads(record.decode(self.encoding)) 32 | else: 33 | # URL has not yet been cached 34 | raise KeyError(url + ' does not exist') 35 | 36 | def __setitem__(self, url, result): 37 | """Save data to Redis for given url""" 38 | data = bytes(json.dumps(result), self.encoding) 39 | if self.compress: 40 | data = zlib.compress(data) 41 | self.client.setex(url, self.expires, data) 42 | -------------------------------------------------------------------------------- /python/crawling/throttle.py: -------------------------------------------------------------------------------- 1 | from urllib.parse import urlparse 2 | import time 3 | 4 | 5 | class Throttle: 6 | def __init__(self, delay): 7 | self.delay = delay 8 | self.domains = {} 9 | 10 | def wait(self, url): 11 | domain = urlparse(url).netloc 12 | last_accessed = self.domains.get(domain) 13 | 14 | if self.delay > 0 and last_accessed is not None: 15 | sleep_secs = self.delay - (time.time() - last_accessed) 16 | if sleep_secs > 0: 17 | time.sleep(sleep_secs) 18 | self.domains[domain] = time.time() 19 | -------------------------------------------------------------------------------- /python/effective-python/better-way-29.py: -------------------------------------------------------------------------------- 1 | class Resistor: 2 | def __init__(self, ohms): 3 | self.ohms = ohms 4 | self.voltage = 0 5 | self.current = 0 6 | 7 | 8 | class VoltageResistance(Resistor): 9 | def __init__(self, ohms): 10 | super().__init__(ohms) 11 | self._voltage = 0 12 | print("init") 13 | 14 | @property 15 | def voltage(self): 16 | print("property") 17 | return self._voltage 18 | 19 | @voltage.setter 20 | def voltage(self, voltage): 21 | print("setter") 22 | self._voltage = voltage 23 | self.current = self._voltage / self.ohms 24 | 25 | 26 | class BoundedResistance(Resistor): 27 | def __init__(self, ohms): 28 | super().__init__(ohms) 29 | 30 | @property 31 | def ohms(self): 32 | return self._ohms 33 | 34 | @ohms.setter 35 | def ohms(self, ohms): 36 | if ohms <= 0: 37 | raise ValueError("{:4f} ohms must be > 0".format(ohms)) 38 | self._ohms = ohms 39 | 40 | 41 | class FixedResistance(Resistor): 42 | @property 43 | def ohms(self): 44 | return self._ohms 45 | 46 | @ohms.setter 47 | def ohms(self, ohms): 48 | if hasattr(self, "_ohms"): 49 | raise AttributeError("Can't set attribute") 50 | self._ohms = 1 51 | 52 | 53 | 54 | 55 | # r1 = Registor(50e3) 56 | # r1.ohms = 10e3 57 | # print("before r1 is {}".format(r1.ohms)) 58 | # r1.ohms += 5e3 59 | # print("after r1 is {}".format(r1.ohms)) 60 | 61 | # 62 | # r2 = VoltageResistance(1e3) 63 | # print(1e3) 64 | # print("Before : {} amps".format(r2.current)) 65 | # r2.voltage = 10 66 | # print("After : {} amps".format(r2.current)) 67 | 68 | # r3 = BoundedResistance(1e3) 69 | # r3.ohms = 0 70 | # BoundedResistance(-5) 71 | 72 | r4 = FixedResistance(1e3) 73 | print(r4.ohms) 74 | r4.ohms = 2e3 75 | -------------------------------------------------------------------------------- /python/effective-python/better-way-30.py: -------------------------------------------------------------------------------- 1 | from datetime import timedelta 2 | from datetime import datetime 3 | 4 | 5 | class Bucket: 6 | def __init__(self, period): 7 | self.period_delta = timedelta(seconds=period) 8 | self.reset_time = datetime.now() 9 | self.max_quota = 0 10 | self.quota_consumed= 0 11 | 12 | def __repr__(self): 13 | return "Bucket(max_quota={max_quota}, quota_consumed={quota_consumed}".format( 14 | max_quota=self.max_quota, quota_consumed=self.quota_consumed) 15 | 16 | @property 17 | def quota(self): 18 | return self.max_quota - self.quota_consumed 19 | 20 | @quota.setter 21 | def quota(self, amount): 22 | delta = self.max_quota - amount 23 | if amount == 0: 24 | self.quota_consumed = 0 25 | self.max_quota = 0 26 | elif delta < 0: 27 | assert self.quota_consumed == 0 28 | self.max_quota = amount 29 | else: 30 | assert self.max_quota >= self.quota_consumed 31 | self.quota_consumed += delta 32 | 33 | 34 | def fill(bucket, amount): 35 | now = datetime.now() 36 | if now - bucket.reset_time > bucket.period_delta: 37 | bucket.quota = 0 38 | bucket.reset_time = now 39 | bucket.quota += amount 40 | 41 | 42 | def deduct(bucket, amount): 43 | now = datetime.now() 44 | if now - bucket.reset_time > bucket.period_delta: 45 | return False 46 | if bucket.quota - amount < 0: 47 | return False 48 | bucket.quota -= amount 49 | return True 50 | 51 | # 52 | # bucket = Bucket(60) 53 | # fill(bucket, 100) 54 | # print(bucket) 55 | # 56 | # if deduct(bucket, 99): 57 | # print("Had 99 quota") 58 | # else: 59 | # print("Not enough for 99 quota") 60 | # print(bucket) 61 | 62 | 63 | bucket = Bucket(60) 64 | print("Initial", bucket) 65 | fill(bucket, 100) 66 | print("Filled", bucket) 67 | 68 | if deduct(bucket, 99): 69 | print("Had 99 quota") 70 | else: 71 | print("Not enough for 99 quota") 72 | 73 | print("Now", bucket) 74 | 75 | if deduct(bucket, 3): 76 | print("Had 3 quota") 77 | else: 78 | print("Not enough for 3 quota") 79 | 80 | 81 | print("Still", bucket) 82 | 83 | 84 | 85 | -------------------------------------------------------------------------------- /python/effective-python/better-way-31.py: -------------------------------------------------------------------------------- 1 | from weakref import WeakKeyDictionary 2 | 3 | class Homework: 4 | def __init__(self): 5 | self._grade = 0 6 | 7 | @property 8 | def grade(self): 9 | return self._grade 10 | 11 | @grade.setter 12 | def grade(self, value): 13 | if not (0 <= value <= 100): 14 | raise ValueError("Grade must be between 0 and 100") 15 | self._grade = value 16 | 17 | 18 | galileo = Homework() 19 | galileo.grade = 95 20 | 21 | # 22 | # class Exam(): 23 | # def __init__(self): 24 | # self._writing_grade = 0 25 | # self._math_grade = 0 26 | # 27 | # @staticmethod 28 | # def _check_grade(value): 29 | # if not (0 <= value <= 100): 30 | # raise ValueError("Grade must be between 0 and 100") 31 | # 32 | # @property 33 | # def writing_grade(self): 34 | # return self._writing_grade 35 | # 36 | # @writing_grade.setter 37 | # def writing_grade(self, value): 38 | # self._check_grade(value) 39 | # self._writing_grade = value 40 | # 41 | # @property 42 | # def math_grade(self): 43 | # return self._math_grade 44 | # 45 | # @math_grade.setter 46 | # def math_grade(self, value): 47 | # self._check_grade(value) 48 | # self._math_grade = value 49 | # 50 | 51 | class Grade: 52 | def __init__(self): 53 | self._values = WeakKeyDictionary() 54 | 55 | def __get__(self, instance, instance_type): 56 | if instance is None: return self 57 | return self._values.get(instance, 0) 58 | 59 | def __set__(self, instance, value): 60 | if not (0 <= value <= 100): 61 | raise ValueError("Grade must be between 0 and 100") 62 | self._values[instance] = value 63 | 64 | 65 | class Exam: 66 | math_grade = Grade() 67 | writing_grade = Grade() 68 | science_grade = Grade() 69 | 70 | first_exam = Exam() 71 | first_exam.writing_grade = 82 72 | second_exam = Exam() 73 | second_exam.writing_grade = 75 74 | print("first ", first_exam.writing_grade, "is right") 75 | print("second ", second_exam.writing_grade, "is right") -------------------------------------------------------------------------------- /python/effective-python/better-way-32.py: -------------------------------------------------------------------------------- 1 | class LazyDB: 2 | def __init__(self): 3 | self.exists = 5 4 | 5 | def __getattr__(self, name): 6 | value = "Value for {name}".format(name=name) 7 | setattr(self, name, value) 8 | return value 9 | 10 | 11 | data = LazyDB() 12 | print("Before: ", data.__dict__) 13 | print("foo: ", data.foo) 14 | print("After ", data.__dict__) 15 | 16 | 17 | class LoggingLazyDB(LazyDB): 18 | def __getattr__(self, name): 19 | print("Called __getattr__({name})".format(name=name)) 20 | return super().__getattr__(name) 21 | 22 | 23 | data = LoggingLazyDB() 24 | print("exists: ", data.exists) 25 | print("foo: ", data.foo) 26 | print("foo: ", data.foo) 27 | print("="*30) 28 | 29 | 30 | class ValidatingDB: 31 | def __init__(self): 32 | self.exists = 5 33 | 34 | def __getattribute__(self, name): 35 | print("Called __getattributed__ ({name})".format(name=name)) 36 | try: 37 | return super().__getattribute__(name) 38 | except AttributeError: 39 | value = "Value for {name}".format(name=name) 40 | setattr(self, name, value) 41 | return value 42 | 43 | 44 | data2 = ValidatingDB() 45 | print("exists: ", data2.exists) 46 | print("foo: ", data2.foo) 47 | print("foo: ", data2.foo) -------------------------------------------------------------------------------- /python/effective-python/better-way-36.py: -------------------------------------------------------------------------------- 1 | import subprocess 2 | from time import time 3 | import os 4 | 5 | # proc = subprocess.Popen( 6 | # ['echo', 'Hello from the child!"'], 7 | # stdout=subprocess.PIPE) 8 | # out, err = proc.communicate() 9 | # print(out.decode("utf-8")) 10 | 11 | # proc = subprocess.Popen(["sleep", "0.3"]) 12 | # while proc.poll() is None: 13 | # print("Working...") 14 | # 15 | # print("Exit status', proc.poll()") 16 | 17 | 18 | def run_sleep(period): 19 | proc = subprocess.Popen(["sleep", str(period)]) 20 | return proc 21 | 22 | 23 | start = time() 24 | procs = [] 25 | for _ in range(10): 26 | proc = run_sleep(0.1) 27 | procs.append(proc) 28 | 29 | 30 | for proc in procs: 31 | proc.communicate() 32 | end = time() 33 | 34 | print("Finished in {:3f} seconds".format(end-start)) 35 | 36 | #======== 37 | 38 | proc = run_sleep(10) 39 | try: 40 | proc.communicate(timeout=0.1) 41 | except subprocess.TimeoutExpired: 42 | proc.terminate() 43 | proc.wait() 44 | 45 | 46 | print("Exit status", proc.poll()) -------------------------------------------------------------------------------- /python/effective-python/better-way-56-utils_test.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase, main 2 | from .utils import to_str 3 | 4 | 5 | class UtilsTestCase(TestCase): 6 | def test_to_str_bytes(self): 7 | self.assertEqual("hello", to_str(b"hello")) 8 | 9 | def test_to_str_str(self): 10 | self.assertEqual("hello", to_str("hello")) 11 | 12 | def to_str_bad(self): 13 | self.assertRaises(TypeError, to_str, object()) 14 | 15 | 16 | if __name__ == '__main__': 17 | main() -------------------------------------------------------------------------------- /python/effective-python/better-way-58-profile.py: -------------------------------------------------------------------------------- 1 | from cProfile import Profile 2 | from pstats import Stats 3 | 4 | def insertion_sort(data): 5 | result = [] 6 | for value in data: 7 | insert_value(result, value) 8 | return result 9 | 10 | def insert_value(array, value): 11 | for i, existing in enumerate(array): 12 | if existing > value: 13 | array.insert(i, value) 14 | return 15 | array.append(value) 16 | 17 | 18 | from random import randint 19 | 20 | max_size = 10**4 21 | data = [randint(0, max_size) for _ in range(max_size)] 22 | test = lambda: insertion_sort(data) 23 | 24 | 25 | profiler = Profile() 26 | profiler.runcall(test) 27 | 28 | stats = Stats(profiler) 29 | stats.strip_dirs() 30 | stats.sort_stats('cumulative') 31 | stats.print_stats() 32 | 33 | # ncalls : 호출 횟수, tottime : 함수 실행되는 소비한 초 단위, tottime percall : tottime을 ncalss로 나눈 값 34 | 35 | # 20003 function calls in 1.413 seconds 36 | # 37 | # Ordered by: cumulative time 38 | # 39 | # ncalls tottime percall cumtime percall filename:lineno(function) 40 | # 1 0.000 0.000 1.413 1.413 better-way-58-profile.py:22() 41 | # 1 0.002 0.002 1.413 1.413 better-way-58-profile.py:4(insertion_sort) 42 | # 10000 1.392 0.000 1.410 0.000 better-way-58-profile.py:10(insert_value) 43 | # 9991 0.018 0.000 0.018 0.000 {method 'insert' of 'list' objects} 44 | # 9 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects} 45 | # 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 46 | -------------------------------------------------------------------------------- /python/effective-python/utils.py: -------------------------------------------------------------------------------- 1 | def to_str(data): 2 | if isinstance(data, str): 3 | return data 4 | elif isinstance(data, bytes): 5 | return data.decode("utf-8") 6 | else: 7 | raise TypeError("must supply str or bytes, found : {}".format(data)) 8 | 9 | -------------------------------------------------------------------------------- /python/fire_module.py: -------------------------------------------------------------------------------- 1 | import fire 2 | 3 | class Calculator: 4 | def add(self, x, y): 5 | return x + y 6 | 7 | def multiply(self, x, y): 8 | return x * y 9 | 10 | if __name__ == '__main__': 11 | calculator = Calculator() 12 | fire.Fire(calculator) 13 | -------------------------------------------------------------------------------- /python/flask/README.md: -------------------------------------------------------------------------------- 1 | # Flask 2 | 3 | @app.route('/') : 데코레이터로 라우팅 4 | def main(): 5 | return 리턴값 6 | 7 | -------------------------------------------------------------------------------- /python/flask/ch1/app/__init__.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | 3 | app = Flask(__name__, template_folder="templates") 4 | 5 | # app을 어디로 두느냐에 따라 다른데 flask는 app안에 templates와 static을 쌓아두는 경우가 많음 ( 처음에 app안에 안넣어서 오류뿜뿜 6 | # best practice를 검색해볼것 7 | 8 | from app import controllers -------------------------------------------------------------------------------- /python/flask/ch1/app/__pycache__/__init__.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/flask/ch1/app/__pycache__/__init__.cpython-36.pyc -------------------------------------------------------------------------------- /python/flask/ch1/app/__pycache__/controllers.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/flask/ch1/app/__pycache__/controllers.cpython-36.pyc -------------------------------------------------------------------------------- /python/flask/ch1/app/__pycache__/database.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/flask/ch1/app/__pycache__/database.cpython-36.pyc -------------------------------------------------------------------------------- /python/flask/ch1/app/__pycache__/models.cpython-36.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/flask/ch1/app/__pycache__/models.cpython-36.pyc -------------------------------------------------------------------------------- /python/flask/ch1/app/controllers.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | from app.database import init_db, db_session 3 | from flask import Flask, request, session, redirect, url_for, abort, render_template, flash 4 | from app.models import User, Post 5 | # 이 경우 app.을 붙여줘야 경로를 인식함 6 | from app import app 7 | 8 | 9 | # 메인페이지 10 | @app.route('/') 11 | @app.route('/index') 12 | def index(): 13 | users = User.query.all() 14 | return render_template('index.html', users=users) 15 | 16 | 17 | # 프로필 18 | @app.route('/profile/') 19 | def profile(userId): 20 | # 사용자 정보 21 | user = User.query.filter_by(id=userId).first() 22 | # app.logger.debug(user.image) 23 | x 24 | # 컨텐츠 리스트 25 | posts = Post.query.filter(Post.userid == User.id).order_by(Post.id.desc()).all(); 26 | # app.logger.debug(len(posts)) 27 | 28 | return render_template('profile.html', user=user, posts=posts, userId=userId) 29 | 30 | 31 | # 로그인 화면 32 | @app.route('/login-form') 33 | def loginForm(): 34 | return render_template('login.html') 35 | 36 | 37 | # 로그인 38 | @app.route('/login', methods=['POST']) 39 | def login(): 40 | if request.method == 'POST': 41 | email = request.form['email'] 42 | password = request.form['password'] 43 | user = User.query.filter_by(email=email).filter_by(password=password).first() 44 | # app.logger.debug(request.form['email']) 45 | # app.logger.debug(user.name) 46 | if (user): 47 | session['logged_in'] = True 48 | session['id'] = user.id 49 | session['email'] = user.email 50 | session['name'] = user.name 51 | session['image'] = user.image 52 | return redirect("/profile/" + str(user.id)) 53 | else: 54 | return '로그인 정보가 맞지 않습니다.' 55 | 56 | else: 57 | return '잘못된 접근' 58 | 59 | 60 | # 로그아웃 61 | @app.route('/logout') 62 | def logout(): 63 | session['logged_in'] = False 64 | session.clear() 65 | return redirect(url_for('index')) 66 | 67 | 68 | # 컨텐츠 등록 69 | @app.route('/add/', methods=['POST']) 70 | def add(userId): 71 | if request.method == 'POST': 72 | if (session['logged_in']): 73 | # app.logger.debug(request.form['contents']) 74 | # app.logger.debug(request.form['userId']) 75 | post = Post(request.form['contents'], userId, session['id']) 76 | db_session.add(post) 77 | db_session.commit() 78 | return redirect("/profile/" + str(userId)) 79 | else: 80 | return '로그인해주세요' 81 | else: 82 | return '잘못된 접근' 83 | 84 | 85 | # 컨텐츠 삭제 86 | @app.route('/delete//') 87 | def delete(userId, postId): 88 | if (session['logged_in']): 89 | Post.query.filter_by(id=postId).delete() 90 | return redirect("/profile/" + str(userId)) 91 | else: 92 | return '잘못된 접근' -------------------------------------------------------------------------------- /python/flask/ch1/app/database.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | from sqlalchemy import create_engine 4 | from sqlalchemy.orm import scoped_session, sessionmaker 5 | from sqlalchemy.ext.declarative import declarative_base 6 | 7 | engine = create_engine('mysql://abcd:1234@127.0.0.1/abcd?charset=utf8', convert_unicode=True) 8 | db_session = scoped_session(sessionmaker(autocommit=False, 9 | autoflush=False, 10 | bind=engine)) 11 | Base = declarative_base() 12 | Base.query = db_session.query_property() 13 | 14 | def init_db(): 15 | # import all modules here that might define models so that 16 | # they will be registered properly on the metadata. Otherwise 17 | # you will have to import them first before calling init_db() 18 | import models 19 | Base.metadata.create_all(bind=engine) -------------------------------------------------------------------------------- /python/flask/ch1/app/models.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | import datetime 3 | from sqlalchemy import Column, Integer, String, Text, DateTime, ForeignKey 4 | from sqlalchemy.orm import sessionmaker, relationship 5 | from app.database import Base 6 | # 이 경우 경로를 위해 app.을 붙여줘야합니다 7 | 8 | # users 테이블에 맵핑 9 | class User(Base): 10 | __tablename__ = 'users' 11 | id = Column(Integer, primary_key=True) 12 | name = Column(String(50), unique=False) 13 | password = Column(String(50), unique=False, nullable=False) 14 | email = Column(String(120), unique=True, nullable=False) 15 | image = Column(String(200), unique=False) 16 | company = Column(String(100), unique=False) 17 | gender = Column(String(50), unique=False) 18 | location = Column(String(100), unique=False) 19 | tel = Column(String(100), unique=False) 20 | description = Column(String(500), unique=False) 21 | 22 | posts = relationship('Post', backref='user', lazy='dynamic') 23 | 24 | def __init__(self, name=None, password=None, email=None, image=None, 25 | company=None, gender=None, location=None, tel=None, description=None): 26 | self.name = name 27 | self.password = password 28 | self.email = email 29 | self.image = image 30 | self.company = company 31 | self.gender = gender 32 | self.location = location 33 | self.tel = tel 34 | self.description = description 35 | 36 | # Representation 37 | def __repr__(self): 38 | return '' % (self.name) 39 | 40 | 41 | # posts 테이블에 맵핑 42 | class Post(Base): 43 | __tablename__ = 'posts' 44 | id = Column(Integer, primary_key=True) 45 | contents = Column(Text, unique=False) 46 | userid = Column(Integer, nullable=False) 47 | writer = Column(Integer, ForeignKey('users.id'), nullable=False) 48 | date = Column(DateTime, default=datetime.datetime.utcnow, unique=False) # 자동으로 들어간다. 49 | 50 | users = relationship('User', backref='post', lazy='joined') 51 | 52 | def __init__(self, contents=None, userid=None, writer=None): 53 | self.contents = contents 54 | self.userid = userid 55 | self.writer = writer 56 | 57 | def __repr__(self): 58 | return '' % (self.contents, self.userid) -------------------------------------------------------------------------------- /python/flask/ch1/app/static/fonts/glyphicons-halflings-regular.eot: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/flask/ch1/app/static/fonts/glyphicons-halflings-regular.eot -------------------------------------------------------------------------------- /python/flask/ch1/app/static/fonts/glyphicons-halflings-regular.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/flask/ch1/app/static/fonts/glyphicons-halflings-regular.ttf -------------------------------------------------------------------------------- /python/flask/ch1/app/static/fonts/glyphicons-halflings-regular.woff: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/flask/ch1/app/static/fonts/glyphicons-halflings-regular.woff -------------------------------------------------------------------------------- /python/flask/ch1/app/static/fonts/glyphicons-halflings-regular.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/flask/ch1/app/static/fonts/glyphicons-halflings-regular.woff2 -------------------------------------------------------------------------------- /python/flask/ch1/app/static/js/npm.js: -------------------------------------------------------------------------------- 1 | // This file is autogenerated via the `commonjs` Grunt task. You can require() this file in a CommonJS environment. 2 | require('../../js/transition.js') 3 | require('../../js/alert.js') 4 | require('../../js/button.js') 5 | require('../../js/carousel.js') 6 | require('../../js/collapse.js') 7 | require('../../js/dropdown.js') 8 | require('../../js/modal.js') 9 | require('../../js/tooltip.js') 10 | require('../../js/popover.js') 11 | require('../../js/scrollspy.js') 12 | require('../../js/tab.js') 13 | require('../../js/affix.js') -------------------------------------------------------------------------------- /python/flask/ch1/app/templates/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 인덱스 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 |
20 |

등록된 사용자

21 | {% for item in users %} 22 | {{item.name}} 23 |
24 | {% endfor %} 25 |
26 | 27 | 28 | 29 | 30 | 31 | 32 | -------------------------------------------------------------------------------- /python/flask/ch1/app/templates/login.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 로그인 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 |
20 | 21 | 27 | 28 |
29 | 30 | 31 | 32 | 33 | 34 | 35 | -------------------------------------------------------------------------------- /python/flask/ch1/run.py: -------------------------------------------------------------------------------- 1 | #!flask/bin/python 2 | # -*- coding: utf-8 -*- 3 | from app import app 4 | 5 | app.secret_key = 'abcd_key' 6 | app.config['SQLALCHEMY_ECHO'] = True 7 | 8 | if __name__ == '__main__': 9 | app.run(host='0.0.0.0', port=5000, debug=True) -------------------------------------------------------------------------------- /python/flask/hello.py: -------------------------------------------------------------------------------- 1 | from flask import Flask, request, url_for, redirect, render_template, session 2 | 3 | app = Flask(__name__) 4 | 5 | 6 | @app.route('/') # app.route 데코레이터로 위치를 말한다 7 | def index(): 8 | return 'index page' 9 | # return redirect(url_for('login')) 으로하면 재연결 10 | 11 | @app.route('/user/') 12 | def show_post(username): 13 | return 'user %s' % username 14 | 15 | @app.route('/login_form') 16 | def login_form(): 17 | return render_template('login_form.html') 18 | 19 | 20 | @app.route('/login', methods=['POST']) 21 | def login(): 22 | if request.method == 'POST': 23 | if ( request.form['username'] == 'jamie' and request.form['password'] == '1234'): 24 | session['logged_in'] = True 25 | session['username'] = request.form['username'] 26 | return request.form['username'] + " 님 환영합니다" 27 | else: 28 | return '로그인 정보가 맞지 않습니다' 29 | else: 30 | return '잘못된 접근' 31 | 32 | # @app.route('/get_test', methods = ['GET']) # 실제 서비스에선 절대 get방식으로 로그인을 하지 않아요!! post로 해야함 33 | # def get_test(): 34 | # if request.method == 'GET': 35 | # if (request.args.get('username') == 'jamie' and request.args.get('password') == '1234'): 36 | # # http://127.0.0.1:5000/get_test?username=jamie&password=1234 로 접속하면 jamie님 환영합니다라고 나 37 | # return request.args.get('username') + " 님 환영합니다" 38 | # else: 39 | # return '로그인 정보가 맞지 않습니다' 40 | # else: 41 | # return '잘못된 접근' 42 | 43 | 44 | @app.route('/template') 45 | @app.route('/template/') 46 | def template_test(tempid=None): 47 | sports = ['야구', '축구', '농구'] 48 | return render_template('template.html', tempid=tempid, sports=sports) 49 | 50 | @app.route('/logout') 51 | def logout(): 52 | session['logged_in'] = False 53 | session.pop('username', None) 54 | return redirect(url_for('index')) 55 | 56 | app.secret_key = 'sample_secret_key' 57 | 58 | 59 | if __name__ == '__main__': 60 | app.run(debug=True) 61 | 62 | -------------------------------------------------------------------------------- /python/flask/minitier/app.py: -------------------------------------------------------------------------------- 1 | from flask import Flask, jsonify, request 2 | from flask.json import JSONEncoder 3 | 4 | 5 | class CustomJSONEncoder(JSONEncoder): 6 | def default(self, obj): 7 | if isinstance(obj, set): 8 | return list(obj) 9 | return JSONEncoder.default(self, obj) 10 | 11 | 12 | app = Flask(__name__) 13 | app.users = {} 14 | app.id_count = 1 15 | app.tweets = [] 16 | app.json_encoder = CustomJSONEncoder 17 | 18 | 19 | @app.route("/ping", methods=["GET"]) 20 | def ping(): 21 | return "pong" 22 | 23 | 24 | @app.route("/sign-up", methods=["POST"]) 25 | def sign_up(): 26 | new_user = request.json 27 | new_user["id"] = app.id_count 28 | app.users[app.id_count] = new_user 29 | app.id_count = app.id_count + 1 30 | 31 | return jsonify(new_user) 32 | 33 | 34 | @app.route("/tweet", methods=["POST"]) 35 | def tweet(): 36 | payload = request.json 37 | user_id = int(payload["id"]) 38 | tweet = payload["tweet"] 39 | 40 | if user_id not in app.users: 41 | return "사용자가 존재하지 않습니다", 400 42 | 43 | if len(tweet) > 300: 44 | return "300자를 초과했습니다", 400 45 | 46 | user_id = int(payload["id"]) 47 | 48 | app.tweets.append({ 49 | "user_id": user_id, 50 | "tweet": tweet 51 | }) 52 | 53 | return "", 200 54 | 55 | 56 | @app.route("/follow", methods=["POST"]) 57 | def follow(): 58 | payload = request.json 59 | user_id = int(payload["id"]) 60 | user_id_to_follow = int(payload["follow"]) 61 | 62 | if user_id not in app.users or user_id_to_follow not in app.users: 63 | return "사용자가 존재하지 않습니다", 400 64 | 65 | user = app.users[user_id] 66 | user.setdefault("follow", set()).add(user_id_to_follow) 67 | 68 | return jsonify(user) 69 | 70 | 71 | @app.route("/unfollow", methods=["POST"]) 72 | def unfollow(): 73 | payload = request.json 74 | user_id = int(payload["id"]) 75 | user_id_to_follow = int(payload["unfollow"]) 76 | 77 | if user_id not in app.users or user_id_to_follow not in app.users: 78 | return "사용자가 존재하지 않습니다", 400 79 | 80 | user = app.users[user_id] 81 | user.setdefault("follow", set()).discard(user_id_to_follow) 82 | 83 | return jsonify(user) 84 | 85 | 86 | @app.route("/timeline/", methods=["GET"]) 87 | def timeline(user_id): 88 | if user_id not in app.users: 89 | return "사용자가 존재하지 않습니다", 400 90 | 91 | follow_list = app.users[user_id].get("follow", set()) 92 | follow_list.add(user_id) 93 | timeline = [tweet for tweet in app.tweets if tweet["user_id"] in follow_list] 94 | 95 | return jsonify({ 96 | "user_id": user_id, 97 | "timeline": timeline 98 | }) -------------------------------------------------------------------------------- /python/flask/templates/login_form.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | login form 6 | 7 | 8 |
9 | 10 |

로그인 정보를 입력해주세요

11 | 12 | 13 | 14 | 15 |
16 | 17 | -------------------------------------------------------------------------------- /python/futures-process-worker.py: -------------------------------------------------------------------------------- 1 | from concurrent import futures 2 | import random 3 | 4 | 5 | def compute(): 6 | return sum( 7 | [random.randint(1, 100) for i in range(1000000)]) 8 | 9 | 10 | with futures.ProcessPoolExecutor() as executor: 11 | futures = [executor.submit(compute) for _ in range(8)] 12 | 13 | results = [f.result() for f in futures] 14 | 15 | print(f"Results: {results}") 16 | 17 | # 내부적으로 multiprocessing.cpu_count 함수를 호출해 사용할 워커를 결정 18 | # 따라서 별도의 max_workers를 설정할 필요 없음 19 | # class ProcessPoolExecutor(_base.Executor): 20 | # def __init__(self, max_workers=None): 21 | # ... 22 | # if max_workers is None: 23 | # self._max_workers = multiprocessing.cpu_count() 24 | # else: 25 | # self._max_workers = max_workers 26 | -------------------------------------------------------------------------------- /python/futures-threads-worker.py: -------------------------------------------------------------------------------- 1 | # 사용 방법 2 | # 실행자(executor) 선택 3 | # 실행자 : 스케줄링과 비동기 작업을 실행할 책임, 엔진 4 | # 스레드 기반, 프로세스 기반 5 | # Future 객체 : 해당 작업이 미래에 언젠가 완료될 것이라는 일종의 약속 6 | 7 | from concurrent import futures 8 | import random 9 | 10 | 11 | def compute(): 12 | return sum( 13 | [random.randint(1, 100) for i in range(1000000)]) 14 | 15 | 16 | with futures.ThreadPoolExecutor(max_workers=8) as executor: 17 | futures = [executor.submit(compute) for _ in range(8)] 18 | 19 | results = [f.result() for f in futures] 20 | 21 | print(f"Results: {results}") 22 | 23 | # 함수형 프로그래밍 형태 24 | # Future 객체는 done()이나 add_done_callback(fn) 등의 함수도 제공 25 | -------------------------------------------------------------------------------- /python/futurist-check-and-reject.py: -------------------------------------------------------------------------------- 1 | import futurist 2 | from futurist import waiters, rejection 3 | import random 4 | 5 | 6 | def compute(): 7 | return sum( 8 | [random.randint(1, 100) for i in range(10000)] 9 | ) 10 | 11 | 12 | with futurist.ThreadPoolExecutor( 13 | max_workers=8, 14 | check_and_reject=rejection.reject_when_reached(2)) as executor: 15 | futures = [executor.submit(compute) for _ in range(8)] 16 | print(executor.statistics) 17 | 18 | 19 | results = [f.result() for f in futures] 20 | print(executor.statistics) 21 | 22 | print(f"Results: {results}") -------------------------------------------------------------------------------- /python/futurist-periodics.py: -------------------------------------------------------------------------------- 1 | import time 2 | 3 | from futurist import periodics 4 | 5 | 6 | @periodics.periodic(1) 7 | def every_one(started_at): 8 | print(f"1: {time.time() - started_at}") 9 | 10 | 11 | w = periodics.PeriodicWorker([ 12 | (every_one, (time.time(), ), {}), 13 | ]) 14 | 15 | 16 | @periodics.periodic(4) 17 | def print_stats(): 18 | print(f"stats {list(w.iter_watchers())}") 19 | 20 | 21 | w.add(print_stats) 22 | w.start() -------------------------------------------------------------------------------- /python/futurist-threads-worker.py: -------------------------------------------------------------------------------- 1 | import futurist 2 | from futurist import waiters, rejection 3 | import random 4 | 5 | 6 | def compute(): 7 | return sum( 8 | [random.randint(1, 100) for i in range(10000)] 9 | ) 10 | 11 | 12 | with futurist.ThreadPoolExecutor(max_workers=8) as executor: 13 | futures = [executor.submit(compute) for _ in range(8)] 14 | print(executor.statistics) 15 | 16 | 17 | results = waiters.wait_for_all(futures) 18 | print(executor.statistics) 19 | 20 | print(f"Results: {[r.result() for r in results.done]}") -------------------------------------------------------------------------------- /python/hide input code in jupyter notebook.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# hide input code in jupyter notebook" 8 | ] 9 | }, 10 | { 11 | "cell_type": "code", 12 | "execution_count": 2, 13 | "metadata": { 14 | "collapsed": true 15 | }, 16 | "outputs": [], 17 | "source": [ 18 | "from IPython.display import HTML" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [ 29 | "HTML('''\n", 41 | "The raw code for this IPython notebook is by default hidden for easier reading.\n", 42 | "To toggle on/off the raw code, click here.''')" 43 | ] 44 | }, 45 | { 46 | "cell_type": "markdown", 47 | "metadata": {}, 48 | "source": [ 49 | "- 위 코드를 입력하면 toggle 이 on/off\n", 50 | "\n", 51 | "- 터미널에서 jupyter nbconvert ~~~~ --to html을 치면 html파일로 떨어짐" 52 | ] 53 | }, 54 | { 55 | "cell_type": "code", 56 | "execution_count": null, 57 | "metadata": { 58 | "collapsed": true 59 | }, 60 | "outputs": [], 61 | "source": [] 62 | } 63 | ], 64 | "metadata": { 65 | "kernelspec": { 66 | "display_name": "Python 3", 67 | "language": "python", 68 | "name": "python3" 69 | }, 70 | "language_info": { 71 | "codemirror_mode": { 72 | "name": "ipython", 73 | "version": 3 74 | }, 75 | "file_extension": ".py", 76 | "mimetype": "text/x-python", 77 | "name": "python", 78 | "nbconvert_exporter": "python", 79 | "pygments_lexer": "ipython3", 80 | "version": "3.6.1" 81 | }, 82 | "latex_envs": { 83 | "LaTeX_envs_menu_present": true, 84 | "autocomplete": true, 85 | "bibliofile": "biblio.bib", 86 | "cite_by": "apalike", 87 | "current_citInitial": 1, 88 | "eqLabelWithNumbers": true, 89 | "eqNumInitial": 1, 90 | "hotkeys": { 91 | "equation": "Ctrl-E", 92 | "itemize": "Ctrl-I" 93 | }, 94 | "labels_anchors": false, 95 | "latex_user_defs": false, 96 | "report_style_numbering": false, 97 | "user_envs_cfg": false 98 | } 99 | }, 100 | "nbformat": 4, 101 | "nbformat_minor": 2 102 | } 103 | -------------------------------------------------------------------------------- /python/image/mytable.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/image/mytable.png -------------------------------------------------------------------------------- /python/image/retrica.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/image/retrica.png -------------------------------------------------------------------------------- /python/image/retrica_1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/image/retrica_1.png -------------------------------------------------------------------------------- /python/json key value switching.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "### 여러 json 파일을 순차적으로 읽으면서 json key, value switching\n", 8 | "- 조건 : value에 list로 담겨져 있음\n", 9 | "\n", 10 | "\n", 11 | "\n", 12 | "#### 내용\n", 13 | "- dict.get('key')로 value값을 가지고 올 수 있음\n", 14 | "- dict.keys()로 key값들을 가지고 올 수 있음" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": { 21 | "collapsed": true 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "import pandas as pd" 26 | ] 27 | }, 28 | { 29 | "cell_type": "code", 30 | "execution_count": 1, 31 | "metadata": { 32 | "collapsed": true 33 | }, 34 | "outputs": [], 35 | "source": [ 36 | "sample_dict1 = {'0':['a','b','c'], '1':['d','e']}\n", 37 | "sample_dict2 = {'0':['k'], '2':['z']}" 38 | ] 39 | }, 40 | { 41 | "cell_type": "code", 42 | "execution_count": 3, 43 | "metadata": {}, 44 | "outputs": [ 45 | { 46 | "data": { 47 | "text/plain": [ 48 | "['a', 'b', 'c']" 49 | ] 50 | }, 51 | "execution_count": 3, 52 | "metadata": {}, 53 | "output_type": "execute_result" 54 | } 55 | ], 56 | "source": [ 57 | "sample_dict1.get('0')" 58 | ] 59 | }, 60 | { 61 | "cell_type": "code", 62 | "execution_count": 4, 63 | "metadata": {}, 64 | "outputs": [ 65 | { 66 | "data": { 67 | "text/plain": [ 68 | "['k']" 69 | ] 70 | }, 71 | "execution_count": 4, 72 | "metadata": {}, 73 | "output_type": "execute_result" 74 | } 75 | ], 76 | "source": [ 77 | "sample_dict2.get('0')" 78 | ] 79 | }, 80 | { 81 | "cell_type": "code", 82 | "execution_count": 5, 83 | "metadata": {}, 84 | "outputs": [ 85 | { 86 | "data": { 87 | "text/plain": [ 88 | "['a', 'b', 'c', 'k']" 89 | ] 90 | }, 91 | "execution_count": 5, 92 | "metadata": {}, 93 | "output_type": "execute_result" 94 | } 95 | ], 96 | "source": [ 97 | "sample_dict1.get('0') + sample_dict2.get('0')" 98 | ] 99 | }, 100 | { 101 | "cell_type": "code", 102 | "execution_count": 6, 103 | "metadata": {}, 104 | "outputs": [ 105 | { 106 | "data": { 107 | "text/plain": [ 108 | "dict_keys(['0', '1'])" 109 | ] 110 | }, 111 | "execution_count": 6, 112 | "metadata": {}, 113 | "output_type": "execute_result" 114 | } 115 | ], 116 | "source": [ 117 | "sample_dict1.keys()" 118 | ] 119 | }, 120 | { 121 | "cell_type": "code", 122 | "execution_count": null, 123 | "metadata": { 124 | "collapsed": true 125 | }, 126 | "outputs": [], 127 | "source": [ 128 | "\n", 129 | "test_dict = {}\n", 130 | "dir_path = '/path/'\n", 131 | "for i in os.listdir(dir_path):\n", 132 | " with open(i) as json_data:\n", 133 | " data = json.load(json_data)\n", 134 | " for j in data.keys():\n", 135 | " try:\n", 136 | " if test_dict[j]:\n", 137 | " pass\n", 138 | " except:\n", 139 | " test_dict.setdefault(j,[])\n", 140 | " test_dict[j] = test_dict.get(j) + data.get(j)" 141 | ] 142 | }, 143 | { 144 | "cell_type": "code", 145 | "execution_count": null, 146 | "metadata": { 147 | "collapsed": true 148 | }, 149 | "outputs": [], 150 | "source": [ 151 | "df = pd.DataFrame(dict([ (k,pd.Series(v)) for k,v in test_dict.items() ]))" 152 | ] 153 | } 154 | ], 155 | "metadata": { 156 | "kernelspec": { 157 | "display_name": "Python 3", 158 | "language": "python", 159 | "name": "python3" 160 | }, 161 | "language_info": { 162 | "codemirror_mode": { 163 | "name": "ipython", 164 | "version": 3 165 | }, 166 | "file_extension": ".py", 167 | "mimetype": "text/x-python", 168 | "name": "python", 169 | "nbconvert_exporter": "python", 170 | "pygments_lexer": "ipython3", 171 | "version": "3.6.1" 172 | } 173 | }, 174 | "nbformat": 4, 175 | "nbformat_minor": 2 176 | } 177 | -------------------------------------------------------------------------------- /python/list(string type) to list(list) - using eval.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": null, 6 | "metadata": { 7 | "collapsed": true 8 | }, 9 | "outputs": [], 10 | "source": [ 11 | "#eval\n", 12 | "보이기엔 []인데\n", 13 | "\n", 14 | "type(df_0611_change['test'][0]) 을 하면 str이라고 나오는 경우\n" 15 | ] 16 | }, 17 | { 18 | "cell_type": "code", 19 | "execution_count": null, 20 | "metadata": { 21 | "collapsed": true 22 | }, 23 | "outputs": [], 24 | "source": [ 25 | "df_0611_change['test'] = df_0611_change['test'].apply(lambda x: eval(x))" 26 | ] 27 | } 28 | ], 29 | "metadata": { 30 | "kernelspec": { 31 | "display_name": "Python 3", 32 | "language": "python", 33 | "name": "python3" 34 | }, 35 | "language_info": { 36 | "codemirror_mode": { 37 | "name": "ipython", 38 | "version": 3 39 | }, 40 | "file_extension": ".py", 41 | "mimetype": "text/x-python", 42 | "name": "python", 43 | "nbconvert_exporter": "python", 44 | "pygments_lexer": "ipython3", 45 | "version": "3.6.1" 46 | } 47 | }, 48 | "nbformat": 4, 49 | "nbformat_minor": 2 50 | } 51 | -------------------------------------------------------------------------------- /python/machinelearing-with-flask/README.md: -------------------------------------------------------------------------------- 1 | ## Machinelearning with Flask 2 | 3 | ### Virtualenv 설정 4 | 1. ```virtualenv env``` 5 | 2. ```source env/bin/activate``` 6 | 3. ```pip3 install -r requirements.txt``` 7 | 8 | ### Testing API 9 | 1. ```python3 app.py``` 10 | 2. ```curl -X GET http://127.0.0.1:5000/prediction -d query='that movie was boring'``` 11 | 12 | 13 | ### File Structure 14 | ``` 15 | ├── README.md 16 | ├── app.py : Flask API application 17 | ├── build_model.py : imports the class object from `model.py` and initiates a new model, trains the model, and pickle 18 | ├── env : virtualenv 19 | ├── lib 20 | │   ├── __init__.py 21 | │   ├── data : directory that contains the data files from Kaggle 22 | │   └── models : directory that contains the pickled model files 23 | ├── model.py : class object for classifier 24 | ├── requirements.txt 25 | └── util.py : helper functions for `model.py` 26 | ``` 27 | 28 | 29 | ### 맥에서 matplotlib import시 오류 30 | ``` 31 | RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are using (Ana)Conda please install python.app and replace the use of 'python' with 'pythonw'. See 'Working with Matplotlib on OSX' in the Matplotlib FAQ for more information. 32 | ``` 33 | 34 | - 위 메세지가 나타날 경우 2가지로 해결 가능 35 | - 1) 스크립트에서 backend 추가 36 | 37 | ``` 38 | import matplotlib 39 | matplotlib.use('TkAgg') 40 | import matplotlib.pyplot as plt 41 | ``` 42 | 43 | - 2) 터미널에서(작업 폴더에서) 44 | 45 | ``` 46 | echo "backend: Tkagg" >> ~/.matplotlib/matplotlibrc 47 | ``` -------------------------------------------------------------------------------- /python/machinelearing-with-flask/app.py: -------------------------------------------------------------------------------- 1 | from flask import Flask 2 | from flask_restful import reqparse, abort, Api, Resource 3 | import pickle 4 | import numpy as np 5 | from model import NLPModel 6 | 7 | app = Flask(__name__) 8 | api = Api(app) 9 | 10 | model = NLPModel() 11 | 12 | clf_path = 'lib/models/SentimentClassifier.pkl' 13 | with open(clf_path, 'rb') as f: 14 | model.clf = pickle.load(f) 15 | 16 | vec_path = 'lib/models/TFIDFVectorizer.pkl' 17 | with open(vec_path, 'rb') as f: 18 | model.vectorizer = pickle.load(f) 19 | 20 | # argument parsing 21 | parser = reqparse.RequestParser() 22 | parser.add_argument('query') 23 | 24 | @app.route('/') 25 | def main(): 26 | return "Main Page\nIf you use curl, using 'curl -X GET http://127.0.0.1:5000/prediction -d query='that movie was boring''" 27 | 28 | class PredictSentiment(Resource): 29 | def get(self): 30 | # use parser and find the user's query 31 | args = parser.parse_args() 32 | user_query = args['query'] 33 | 34 | # vectorize the user's query and make a prediction 35 | uq_vectorized = model.vectorizer_transform(np.array([user_query])) 36 | prediction = model.predict(uq_vectorized) 37 | pred_proba = model.predict_proba(uq_vectorized) 38 | 39 | # Output either 'Negative' or 'Positive' along with the score 40 | if prediction == 0: 41 | pred_text = 'Negative' 42 | else: 43 | pred_text = 'Positive' 44 | 45 | # round the predict proba value and set to new variable 46 | confidence = round(pred_proba[0], 3) 47 | 48 | # create JSON object 49 | output = {'prediction': pred_text, 'confidence': confidence} 50 | 51 | return output 52 | 53 | 54 | # Setup the Api resource routing here 55 | # Route the URL to the resource 56 | api.add_resource(PredictSentiment, '/prediction') 57 | 58 | 59 | if __name__ == '__main__': 60 | app.run(debug=True) 61 | -------------------------------------------------------------------------------- /python/machinelearing-with-flask/build_model.py: -------------------------------------------------------------------------------- 1 | from model import NLPModel 2 | import pandas as pd 3 | # import numpy as np 4 | # import matplotlib.pyplot as plt 5 | from sklearn.model_selection import train_test_split 6 | 7 | 8 | def build_model(): 9 | model = NLPModel() 10 | 11 | # filename = os.path.join( 12 | # os.path.dirname(__file__), 'chalicelib', 'all/train.tsv') 13 | with open('../sentiment_data/train.tsv') as f: 14 | data = pd.read_csv(f, sep='\t') 15 | 16 | pos_neg = data[(data['Sentiment'] == 0) | (data['Sentiment'] == 4)] 17 | 18 | pos_neg['Binary'] = pos_neg.apply( 19 | lambda x: 0 if x['Sentiment'] == 0 else 1, axis=1) 20 | 21 | model.vectorizer_fit(pos_neg.loc[:, 'Phrase']) 22 | print('Vectorizer fit complete') 23 | 24 | X = model.vectorizer_transform(pos_neg.loc[:, 'Phrase']) 25 | print('Vectorizer transform complete') 26 | y = pos_neg.loc[:, 'Binary'] 27 | 28 | X_train, X_test, y_train, y_test = train_test_split(X, y) 29 | 30 | model.train(X_train, y_train) 31 | print('Model training complete') 32 | 33 | model.pickle_clf() 34 | model.pickle_vectorizer() 35 | 36 | model.plot_roc(X_test, y_test) 37 | 38 | 39 | if __name__ == "__main__": 40 | build_model() 41 | -------------------------------------------------------------------------------- /python/machinelearing-with-flask/lib/models/SentimentClassifier.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/machinelearing-with-flask/lib/models/SentimentClassifier.pkl -------------------------------------------------------------------------------- /python/machinelearing-with-flask/lib/models/TFIDFVectorizer.pkl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/python/machinelearing-with-flask/lib/models/TFIDFVectorizer.pkl -------------------------------------------------------------------------------- /python/machinelearing-with-flask/model.py: -------------------------------------------------------------------------------- 1 | # ML imports 2 | from sklearn.naive_bayes import MultinomialNB 3 | # from sklearn.naive_bayes import BernoulliNB 4 | from sklearn.feature_extraction.text import TfidfVectorizer 5 | 6 | # from sklearn.ensemble import RandomForestClassifier 7 | import pickle 8 | 9 | from util import plot_roc 10 | # spacy_tok 11 | 12 | 13 | class NLPModel(object): 14 | 15 | def __init__(self): 16 | """Simple NLP 17 | Attributes: 18 | clf: sklearn classifier model 19 | vectorizor: TFIDF vectorizer or similar 20 | """ 21 | self.clf = MultinomialNB() 22 | # self.vectorizer = TfidfVectorizer(tokenizer=spacy_tok) 23 | self.vectorizer = TfidfVectorizer() 24 | 25 | def vectorizer_fit(self, X): 26 | """Fits a TFIDF vectorizer to the text 27 | """ 28 | self.vectorizer.fit(X) 29 | 30 | def vectorizer_transform(self, X): 31 | """Transform the text data to a sparse TFIDF matrix 32 | """ 33 | X_transformed = self.vectorizer.transform(X) 34 | return X_transformed 35 | 36 | def train(self, X, y): 37 | """Trains the classifier to associate the label with the sparse matrix 38 | """ 39 | # X_train, X_test, y_train, y_test = train_test_split(X, y) 40 | self.clf.fit(X, y) 41 | 42 | def predict_proba(self, X): 43 | """Returns probability for the binary class '1' in a numpy array 44 | """ 45 | y_proba = self.clf.predict_proba(X) 46 | return y_proba[:, 1] 47 | 48 | def predict(self, X): 49 | """Returns the predicted class in an array 50 | """ 51 | y_pred = self.clf.predict(X) 52 | return y_pred 53 | 54 | def pickle_vectorizer(self, path='chalicelib/models/TFIDFVectorizer.pkl'): 55 | """Saves the trained vectorizer for future use. 56 | """ 57 | with open(path, 'wb') as f: 58 | pickle.dump(self.vectorizer, f) 59 | print("Pickled vectorizer at {}".format(path)) 60 | 61 | def pickle_clf(self, path='chalicelib/models/SentimentClassifier.pkl'): 62 | """Saves the trained classifier for future use. 63 | """ 64 | with open(path, 'wb') as f: 65 | pickle.dump(self.clf, f) 66 | print("Pickled classifier at {}".format(path)) 67 | 68 | def plot_roc(self, X, y, size_x, size_y): 69 | """Plot the ROC curve for X_test and y_test. 70 | """ 71 | plot_roc(self.clf, X, y, size_x, size_y) 72 | -------------------------------------------------------------------------------- /python/machinelearing-with-flask/requirements.txt: -------------------------------------------------------------------------------- 1 | chalice 2 | matplotlib 3 | sklearn 4 | numpy 5 | scipy 6 | pandas 7 | flask 8 | flask_restful 9 | -------------------------------------------------------------------------------- /python/machinelearing-with-flask/util.py: -------------------------------------------------------------------------------- 1 | # import spacys 2 | from sklearn.metrics import roc_curve, auc 3 | import matplotlib.pyplot as plt 4 | 5 | # nlp = spacy.load('en') 6 | 7 | 8 | # def spacy_tok(text, lemmatize=False): 9 | # doc = nlp(text) 10 | # if lemmatize: 11 | # tokens = [tok.lemma_ for tok in doc] 12 | # else: 13 | # tokens = [tok.text for tok in doc] 14 | # return tokens 15 | 16 | 17 | def plot_roc(model, x_columns, y_true, size_x=12, size_y=12): 18 | """Returns a ROC plot 19 | 20 | Forked from Matt Drury. 21 | """ 22 | 23 | y_pred = model.predict_proba(x_columns) 24 | 25 | fpr, tpr, threshold = roc_curve(y_true, y_pred[:, 1]) 26 | area_under_curve = auc(fpr, tpr) 27 | 28 | # method I: plt 29 | fig, ax = plt.subplots(figsize=(size_x, size_y)) 30 | model_name = str(type(model)).split('.')[-1].strip(">\'") 31 | plt.title(f'{model_name} ROC') 32 | ax.plot(fpr, tpr, 'k', label='AUC = %0.3f' % area_under_curve) 33 | 34 | ax.legend(loc='lower right') 35 | ax.plot([0, 1], [0, 1], 'r--') 36 | plt.xlim([0, 1]) 37 | plt.ylim([0, 1]) 38 | plt.ylabel('True Positive Rate') 39 | plt.xlabel('False Positive Rate') 40 | plt.show() 41 | -------------------------------------------------------------------------------- /python/make UUID.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# make UUID\n", 8 | "\n", 9 | "- UUID : 기본적으로 어떤 개체(데이터)를 고유하게 식별하는 데 사용되는 16바이트(128비트) 길이의 숫자로 이 숫자는 32개의 16진수로 구성되며, 5개의 그룹으로 표시되고 각 그룹은 하이픈으로 구분! \n", 10 | "\n", 11 | "- URI : 통합 자원 식별자 (Uniform Resource Identifier)" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": 1, 17 | "metadata": { 18 | "collapsed": true 19 | }, 20 | "outputs": [], 21 | "source": [ 22 | "import uuid" 23 | ] 24 | }, 25 | { 26 | "cell_type": "code", 27 | "execution_count": 2, 28 | "metadata": {}, 29 | "outputs": [ 30 | { 31 | "name": "stdout", 32 | "output_type": "stream", 33 | "text": [ 34 | "303a53bd-fd90-403d-b154-9459294832d6\n" 35 | ] 36 | } 37 | ], 38 | "source": [ 39 | "print(uuid.uuid4())" 40 | ] 41 | }, 42 | { 43 | "cell_type": "code", 44 | "execution_count": 4, 45 | "metadata": {}, 46 | "outputs": [ 47 | { 48 | "name": "stdout", 49 | "output_type": "stream", 50 | "text": [ 51 | "0ddff47e-29fe-48\n" 52 | ] 53 | } 54 | ], 55 | "source": [ 56 | "print(str(uuid.uuid4())[:16])" 57 | ] 58 | }, 59 | { 60 | "cell_type": "code", 61 | "execution_count": null, 62 | "metadata": { 63 | "collapsed": true 64 | }, 65 | "outputs": [], 66 | "source": [ 67 | "def uuid1(node=None, clock_seq=None):\n", 68 | "\"\"\"호스트 ID와 현재 시간을 기준으로 uuid를 생성\"\"\"" 69 | ] 70 | }, 71 | { 72 | "cell_type": "code", 73 | "execution_count": null, 74 | "metadata": { 75 | "collapsed": true 76 | }, 77 | "outputs": [], 78 | "source": [ 79 | "def uuid3(namespace, name):\n", 80 | "\"\"\"namespace의 md5 hash와 name을 가지고 uuid를 생성\"\"\"" 81 | ] 82 | }, 83 | { 84 | "cell_type": "code", 85 | "execution_count": null, 86 | "metadata": { 87 | "collapsed": true 88 | }, 89 | "outputs": [], 90 | "source": [ 91 | "def uuid4():\n", 92 | "\"\"\"랜덤으로 uuid를 생성\"\"\"" 93 | ] 94 | }, 95 | { 96 | "cell_type": "code", 97 | "execution_count": 9, 98 | "metadata": {}, 99 | "outputs": [], 100 | "source": [ 101 | "def uuid5(namespace, name):\n", 102 | "\"\"\"namespace의 SHA-1 hash와 name을 가지고 uuid를 생성\"\"\"" 103 | ] 104 | }, 105 | { 106 | "cell_type": "code", 107 | "execution_count": null, 108 | "metadata": { 109 | "collapsed": true 110 | }, 111 | "outputs": [], 112 | "source": [] 113 | } 114 | ], 115 | "metadata": { 116 | "kernelspec": { 117 | "display_name": "Python 3", 118 | "language": "python", 119 | "name": "python3" 120 | }, 121 | "language_info": { 122 | "codemirror_mode": { 123 | "name": "ipython", 124 | "version": 3 125 | }, 126 | "file_extension": ".py", 127 | "mimetype": "text/x-python", 128 | "name": "python", 129 | "nbconvert_exporter": "python", 130 | "pygments_lexer": "ipython3", 131 | "version": "3.6.1" 132 | }, 133 | "latex_envs": { 134 | "LaTeX_envs_menu_present": true, 135 | "autocomplete": true, 136 | "bibliofile": "biblio.bib", 137 | "cite_by": "apalike", 138 | "current_citInitial": 1, 139 | "eqLabelWithNumbers": true, 140 | "eqNumInitial": 1, 141 | "hotkeys": { 142 | "equation": "Ctrl-E", 143 | "itemize": "Ctrl-I" 144 | }, 145 | "labels_anchors": false, 146 | "latex_user_defs": false, 147 | "report_style_numbering": false, 148 | "user_envs_cfg": false 149 | } 150 | }, 151 | "nbformat": 4, 152 | "nbformat_minor": 2 153 | } 154 | -------------------------------------------------------------------------------- /python/sacred/actual_use/experiments/random_forest.py: -------------------------------------------------------------------------------- 1 | from sklearn import datasets, model_selection 2 | from sklearn.ensemble import RandomForestClassifier 3 | from sacred import Experiment 4 | from sacred.observers import FileStorageObserver 5 | from sklearn import metrics 6 | 7 | ex = Experiment('rf') 8 | ex.observers.append(FileStorageObserver.create('my_runs')) 9 | 10 | 11 | @ex.config 12 | def cfg(): 13 | n_estimators=100 14 | seed = 42 15 | 16 | 17 | @ex.capture 18 | def get_model(n_estimators): 19 | return RandomForestClassifier(n_estimators) 20 | 21 | 22 | @ex.automain 23 | def run(_log): 24 | X, y = datasets.load_breast_cancer(return_X_y=True) 25 | _log.info("[INFO] Now split dataset in RF") 26 | X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2) 27 | clf = get_model() 28 | clf.fit(X_train, y_train) 29 | y_pred = clf.predict(X_test) 30 | _log.info("[INFO] End Predict!") 31 | 32 | return metrics.accuracy_score(y_test, y_pred) 33 | 34 | -------------------------------------------------------------------------------- /python/sacred/actual_use/experiments/svc.py: -------------------------------------------------------------------------------- 1 | from sklearn import svm, datasets, model_selection 2 | from sacred import Experiment 3 | from sacred.observers import FileStorageObserver 4 | 5 | ex = Experiment('svc') 6 | ex.observers.append(FileStorageObserver.create('my_runs')) 7 | 8 | 9 | @ex.config 10 | def cfg(): 11 | C = 1.0 12 | gamma = 0.7 13 | kernel = "rbf" 14 | seed = 42 15 | 16 | 17 | @ex.capture 18 | def get_model(C, gamma, kernel): 19 | return svm.SVC(C=C, kernel=kernel, gamma=gamma) 20 | 21 | 22 | @ex.automain 23 | def run(_log): 24 | X, y = datasets.load_breast_cancer(return_X_y=True) 25 | _log.info("[INFO] Now split dataset") 26 | X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=0.2) 27 | clf = get_model() 28 | clf.fit(X_train, y_train) 29 | _log.info("[INFO] End Fit!") 30 | return clf.score(X_test, y_test) 31 | 32 | -------------------------------------------------------------------------------- /python/sacred/actual_use/main.py: -------------------------------------------------------------------------------- 1 | from experiments.random_forest import ex as rf_ex 2 | from experiments.svc import ex as svc_ex 3 | 4 | rf_run = rf_ex.run() 5 | print(rf_run.config) 6 | print(rf_run.result) 7 | print(rf_run.info.keys()) 8 | print("New experiment run!") 9 | rf_run2 = rf_ex.run(config_updates={"n_estimators": 10}) 10 | print(rf_run2.config) 11 | print(rf_run2.result) 12 | 13 | svc_run = svc_ex.run() 14 | print(svc_run.config) 15 | print(svc_run.result) 16 | -------------------------------------------------------------------------------- /python/sacred/capture_functions.py: -------------------------------------------------------------------------------- 1 | from sacred import Experiment 2 | ex = Experiment('my_experiment') 3 | 4 | 5 | @ex.config 6 | def my_config(): 7 | foo = 42 8 | bar = 'baz' 9 | 10 | 11 | @ex.capture 12 | def some_function(a, foo, bar=10): 13 | print(a, foo, bar) 14 | 15 | 16 | @ex.automain 17 | def my_main(): 18 | some_function(1, 2, 3) # 1 2 3 19 | some_function(1) # 1 42 'baz' 20 | some_function(1, bar=12) # 1 42 12 21 | some_function() # TypeError : missing value for 'a' 22 | -------------------------------------------------------------------------------- /python/sacred/custom_logger.py: -------------------------------------------------------------------------------- 1 | import logging 2 | from sacred import Experiment 3 | 4 | ex = Experiment('log_example') 5 | 6 | # set up a custom logger 7 | logger = logging.getLogger('mylogger') 8 | logger.handlers = [] 9 | ch = logging.StreamHandler() 10 | formatter = logging.Formatter('[%(levelname).1s] %(name)s >> "%(message)s"') 11 | ch.setFormatter(formatter) 12 | logger.addHandler(ch) 13 | logger.setLevel('INFO') 14 | 15 | ex.logger = logger 16 | 17 | @ex.config 18 | def my_config(): 19 | recipient = "world" 20 | message = "Hello %s!" % recipient 21 | 22 | @ex.capture 23 | def some_function(_log): 24 | _log.info('Custom Info message!') 25 | 26 | @ex.automain 27 | def my_main(message): 28 | some_function() 29 | print(message) -------------------------------------------------------------------------------- /python/sacred/first_config.py: -------------------------------------------------------------------------------- 1 | from sacred import Experiment 2 | from sacred.observers import FileStorageObserver 3 | ex = Experiment('hello_config') 4 | ex.observers.append(FileStorageObserver.create('my_runs')) 5 | 6 | @ex.config 7 | def my_config(): 8 | recipient = "world" 9 | message = "Hello %s!" % recipient 10 | 11 | @ex.capture 12 | def some_function(_log): 13 | _log.info('Custom Info message!') 14 | 15 | @ex.automain 16 | def my_main(message): 17 | some_function() 18 | print(message) 19 | -------------------------------------------------------------------------------- /python/sacred/ingredient.py: -------------------------------------------------------------------------------- 1 | from sacred import Ingredient, Experiment 2 | 3 | # ================== Dataset Ingredient ======================================= 4 | # could be in a separate file 5 | 6 | data_ingredient = Ingredient('dataset') 7 | 8 | 9 | @data_ingredient.config 10 | def cfg1(): 11 | filename = 'my_dataset.npy' # dataset filename 12 | normalize = True # normalize dataset 13 | 14 | 15 | @data_ingredient.capture 16 | def load_data(filename, normalize): 17 | print("loading dataset from '{}'".format(filename)) 18 | if normalize: 19 | print("normalizing dataset") 20 | return 1 21 | return 42 22 | 23 | 24 | @data_ingredient.command 25 | def stats(filename, foo=12): 26 | print('Statistics for dataset "{}":'.format(filename)) 27 | print('mean = 42.23') 28 | print('foo=', foo) 29 | 30 | 31 | # ================== Experiment =============================================== 32 | 33 | @data_ingredient.config 34 | def cfg2(): 35 | filename = 'foo.npy' 36 | 37 | # add the Ingredient while creating the experiment 38 | ex = Experiment('my_experiment', ingredients=[data_ingredient]) 39 | 40 | @ex.config 41 | def cfg3(): 42 | a = 12 43 | b = 42 44 | 45 | @ex.named_config 46 | def fbb(): 47 | a = 22 48 | dataset = {"filename": "AwwwJiss.py"} 49 | 50 | @ex.automain 51 | def run(): 52 | data = load_data() # just use the function 53 | print('data={}'.format(data)) -------------------------------------------------------------------------------- /python/sacred/my_command.py: -------------------------------------------------------------------------------- 1 | from sacred import Experiment 2 | 3 | ex = Experiment('my_commands') 4 | 5 | 6 | @ex.config 7 | def cfg(): 8 | name = 'kyle' 9 | 10 | 11 | @ex.command 12 | def greet(name): 13 | print('Hello {}! Nice to greet you!'.format(name)) 14 | 15 | 16 | @ex.command 17 | def shout(): 18 | print('WHAZZZUUUUUUUUUUP!!!????') 19 | 20 | 21 | @ex.automain 22 | def main(): 23 | print('This is just the main command. Try greet or shout.') -------------------------------------------------------------------------------- /python/sacred/randomness.py: -------------------------------------------------------------------------------- 1 | from sacred import Experiment 2 | 3 | ex = Experiment('randomness') 4 | 5 | 6 | @ex.config 7 | def cfg(): 8 | reverse = False 9 | numbers = 1 10 | 11 | 12 | @ex.capture 13 | def do_random_stuff(numbers, _rnd): 14 | print([_rnd.randint(1, 100) for _ in range(numbers)]) 15 | 16 | 17 | @ex.capture 18 | def do_more_random_stuff(_seed): 19 | print(_seed) 20 | 21 | 22 | @ex.automain 23 | def run(reverse): 24 | if reverse: 25 | do_more_random_stuff() 26 | do_random_stuff() 27 | do_random_stuff() 28 | else: 29 | do_random_stuff() 30 | do_random_stuff() 31 | do_more_random_stuff() 32 | 33 | do_random_stuff() -------------------------------------------------------------------------------- /python/scheduler.py: -------------------------------------------------------------------------------- 1 | from apscheduler.jobstores.base import JobLookupError 2 | from apscheduler.schedulers.background import BackgroundScheduler 3 | import time 4 | 5 | 6 | class Scheduler: 7 | def __init__(self): 8 | self.sched = BackgroundScheduler() 9 | self.sched.start() 10 | self.job_id = '' 11 | 12 | def __del__(self): 13 | self.shutdown() 14 | 15 | def shutdown(self): 16 | self.sched.shutdown() 17 | 18 | def kill_scheduler(self, job_id): 19 | try: 20 | self.sched.remove_job(job_id) 21 | except JobLookupError as err: 22 | print("fail to stop Scheduler: {err}".format(err=err)) 23 | return 24 | 25 | def hello(self, type, job_id): 26 | print("%s Scheduler process_id[%s] : %d" % (type, job_id, time.localtime().tm_sec)) 27 | 28 | def scheduler(self, type, job_id): 29 | print("{type} Scheduler Start".format(type=type)) 30 | if type == 'interval': 31 | self.sched.add_job(self.hello, type, seconds=10, id=job_id, args=(type, job_id)) 32 | elif type == 'cron': 33 | self.sched.add_job(self.hello, type, day_of_week='mon-fri', 34 | hour='0-23', second='*/2', 35 | id=job_id, args=(type, job_id)) 36 | 37 | 38 | if __name__ == '__main__': 39 | scheduler = Scheduler() 40 | scheduler.scheduler('cron', "1") 41 | scheduler.scheduler('interval', "2") 42 | 43 | count = 0 44 | while True: 45 | ''' 46 | count 제한할 경우 아래와 같이 사용 47 | ''' 48 | print("Running main process") 49 | time.sleep(1) 50 | count += 1 51 | if count == 10: 52 | scheduler.kill_scheduler("1") 53 | print("Kill cron Scheduler") 54 | elif count == 15: 55 | scheduler.kill_scheduler("2") 56 | print("Kill interval Scheduler") -------------------------------------------------------------------------------- /python/selenium ( install chrome driver with mac ).ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": { 6 | "deletable": true, 7 | "editable": true 8 | }, 9 | "source": [ 10 | "# selenium path 설정 ( chrome driver )\n", 11 | "\n", 12 | "- [chorome driver 다운](https://sites.google.com/a/chromium.org/chromedriver/downloads)\n", 13 | "- 다운로드한 파일을 finder에서 더블클릭하면 파일이 나옴\n", 14 | "- 그 후 terminal을 열어서 echo $PATH 입력\n", 15 | "- 환경변수 폴더가 나오는데, /usr/local/bin/으로 파일을 옮기면 됩니다!\n", 16 | "~~~\n", 17 | "cd Downloads\n", 18 | "cp chromedriver /usr/local/bin/\n", 19 | "cd /usr/local/bin/\n", 20 | "~~~\n", 21 | "\n", 22 | "- 확인해보면 이제 크롬으로도 셀레니움 사용 가능!" 23 | ] 24 | }, 25 | { 26 | "cell_type": "markdown", 27 | "metadata": { 28 | "deletable": true, 29 | "editable": true 30 | }, 31 | "source": [ 32 | "- phantomjs의 경우도 다운받아서 bin파일에 넣는 방법도 있고, 터미널에서 아래와 같이 입력해도됨\n", 33 | "~~~\n", 34 | "brew install phantomjs\n", 35 | "~~~" 36 | ] 37 | }, 38 | { 39 | "cell_type": "markdown", 40 | "metadata": { 41 | "deletable": true, 42 | "editable": true 43 | }, 44 | "source": [ 45 | "## 최근 selenium을 사용하는 경우, 파이어폭스는 geckodriver를 설치해야함\n", 46 | "\n", 47 | "[gecko driver](https://github.com/mozilla/geckodriver/releases)\n", 48 | "\n", 49 | "다운로드 후, /usr/local/bin/으로 옮기면 됩니다" 50 | ] 51 | }, 52 | { 53 | "cell_type": "markdown", 54 | "metadata": { 55 | "collapsed": true, 56 | "deletable": true, 57 | "editable": true 58 | }, 59 | "source": [ 60 | "# 윈도우\n", 61 | "드라이버를 다운로드한 후, c:\\windows\\system32에 복붙하면 됩니다!!!" 62 | ] 63 | }, 64 | { 65 | "cell_type": "code", 66 | "execution_count": null, 67 | "metadata": { 68 | "collapsed": true 69 | }, 70 | "outputs": [], 71 | "source": [] 72 | } 73 | ], 74 | "metadata": { 75 | "kernelspec": { 76 | "display_name": "Python 3", 77 | "language": "python", 78 | "name": "python3" 79 | }, 80 | "language_info": { 81 | "codemirror_mode": { 82 | "name": "ipython", 83 | "version": 3 84 | }, 85 | "file_extension": ".py", 86 | "mimetype": "text/x-python", 87 | "name": "python", 88 | "nbconvert_exporter": "python", 89 | "pygments_lexer": "ipython3", 90 | "version": "3.6.0" 91 | } 92 | }, 93 | "nbformat": 4, 94 | "nbformat_minor": 2 95 | } 96 | -------------------------------------------------------------------------------- /python/tdd/application/app.py: -------------------------------------------------------------------------------- 1 | from flask import Flask, jsonify 2 | 3 | app = Flask(__name__) 4 | 5 | 6 | @app.route('/') 7 | def home(): 8 | return jsonify({'message': 'Hello, world!'}) 9 | 10 | 11 | if __name__ == '__main__': 12 | app.run() 13 | 14 | -------------------------------------------------------------------------------- /python/tdd/application/requirements.txt: -------------------------------------------------------------------------------- 1 | Flask -------------------------------------------------------------------------------- /python/tdd/application/test/system/base_test.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | from app import app 3 | 4 | 5 | class BaseTest(TestCase): 6 | def setUp(self): 7 | app.testing = True 8 | self.app = app.test_client -------------------------------------------------------------------------------- /python/tdd/application/test/system/test_home.py: -------------------------------------------------------------------------------- 1 | import json 2 | from application.test.system.base_test import BaseTest 3 | 4 | 5 | class TestHome(BaseTest): 6 | def test_home(self): 7 | with self.app() as c: 8 | resp = c.get('/') 9 | 10 | self.assertEqual(resp.status_code, 200) 11 | self.assertEqual( 12 | json.loads(resp.get_data()), 13 | {'message': 'Hello, world!'} 14 | ) -------------------------------------------------------------------------------- /python/tdd/blog/app.py: -------------------------------------------------------------------------------- 1 | from blog import Blog 2 | 3 | MENU_PROMPT = 'Enter "c" to create a blog, "1" to list blogs, "r" to read one, "p" to create a post, or "q" to quit : ' 4 | POST_TEMPLATE = ''' 5 | --- {} --- 6 | 7 | {} 8 | 9 | ''' 10 | 11 | blogs = dict() # blog_name : Blog object 12 | 13 | 14 | def menu(): 15 | # show the user the available blogs 16 | # choice 17 | # do something 18 | # eventually exit 19 | 20 | print_blogs() 21 | selection = input(MENU_PROMPT) 22 | while selection != 'q': 23 | if selection == 'c': 24 | ask_create_blog() 25 | elif selection == '1': 26 | print_blogs() 27 | elif selection == 'r': 28 | ask_read_blog() 29 | elif selection == 'p': 30 | ask_create_post() 31 | selection = input(MENU_PROMPT) 32 | 33 | 34 | def print_blogs(): 35 | for key, blog in blogs.items(): 36 | print('- {}'.format(blog)) 37 | 38 | 39 | def ask_create_blog(): 40 | title = input('Enter your blog title: ') 41 | author = input('Enter your name: ') 42 | 43 | blogs[title] = Blog(title, author) 44 | 45 | 46 | def ask_read_blog(): 47 | title = input('Enter the blog title you want to read: ') 48 | print_posts(blogs[title]) 49 | 50 | 51 | def print_posts(blog): 52 | for post in blog.posts: 53 | print_post(post) 54 | 55 | 56 | def print_post(post): 57 | print(POST_TEMPLATE.format(post.title, post.content)) 58 | 59 | 60 | def ask_create_post(): 61 | blog_name = input('Enter the blog title you want to write a post in: ') 62 | title = input('Enter your post title: ') 63 | content = input('Enter your post content: ') 64 | 65 | blogs[blog_name].create_post(title, content) 66 | 67 | 68 | 69 | 70 | -------------------------------------------------------------------------------- /python/tdd/blog/blog.py: -------------------------------------------------------------------------------- 1 | from post import Post 2 | 3 | 4 | class Blog: 5 | def __init__(self, title, author): 6 | self.title = title 7 | self.author = author 8 | self.posts = [] 9 | 10 | def __repr__(self): 11 | return '{} by {} ({} post{})'.format(self.title, 12 | self.author, 13 | len(self.posts), 14 | 's' if len(self.posts) != 1 else '') 15 | 16 | def create_post(self, title, content): 17 | self.posts.append(Post(title, content)) 18 | 19 | def json(self): 20 | return { 21 | 'title': self.title, 22 | 'author': self.author, 23 | 'posts': [post.json()for post in self.posts] 24 | } 25 | 26 | -------------------------------------------------------------------------------- /python/tdd/blog/post.py: -------------------------------------------------------------------------------- 1 | class Post: 2 | def __init__(self, title, content): 3 | self.title = title 4 | self.content = content 5 | 6 | def json(self): 7 | return { 8 | 'title': self.title, 9 | 'content': self.content 10 | } 11 | 12 | -------------------------------------------------------------------------------- /python/tdd/blog/tests/integration/blog_test.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | from blog import Blog 3 | 4 | 5 | class BlogTest(TestCase): 6 | def test_create_post_in_blog(self): 7 | b = Blog('Test', 'Test Author') 8 | b.create_post('Test Post', 'Test Content') 9 | 10 | self.assertEqual(len(b.posts), 1) 11 | self.assertEqual(b.posts[0].title, 'Test Post') 12 | self.assertEqual(b.posts[0].content, 'Test Content') 13 | 14 | def test_json_no_posts(self): 15 | b = Blog('Test', 'Test Author') 16 | expected = {'title': 'Test', 'author': 'Test Author', 'posts': []} 17 | 18 | self.assertDictEqual(expected, b.json()) 19 | 20 | def test_json(self): 21 | b = Blog('Test', 'Test Author') 22 | b.create_post('Test Post', 'Test Content') 23 | 24 | expected = { 25 | 'title': 'Test', 26 | 'author': 'Test Author', 27 | 'posts': [ 28 | { 29 | 'title': 'Test Post', 30 | 'content': 'Test Content' 31 | } 32 | ] 33 | } 34 | 35 | self.assertDictEqual(expected, b.json()) -------------------------------------------------------------------------------- /python/tdd/blog/tests/system/app_test.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | from unittest.mock import patch 3 | import app 4 | from blog import Blog 5 | from post import Post 6 | 7 | 8 | class AppTest(TestCase): 9 | def setUp(self): 10 | blog = Blog('Test', 'Test Author') 11 | app.blogs = {'Test': blog} 12 | 13 | def test_menu_prints_blogs(self): 14 | with patch('builtins.print') as mocked_print: 15 | with patch('builtins.input', return_value='q'): 16 | app.menu() 17 | 18 | mocked_print.assert_called_with('- Test by Test Author (0 posts)') 19 | 20 | def test_menu_prints_prompt(self): 21 | with patch('builtins.input', return_value='q') as mocked_input: 22 | app.menu() 23 | 24 | mocked_input.assert_called_with(app.MENU_PROMPT) 25 | 26 | def test_menu_calls_create_blog(self): 27 | with patch('builtins.input') as mocked_input: 28 | mocked_input.side_effect = ('c', 'Test Two', 'Test Author Two', 'q') 29 | app.menu() 30 | 31 | self.assertIsNotNone(app.blogs['Test Two']) 32 | 33 | def test_menu_calls_print_blogs(self): 34 | with patch('builtins.input') as mocked_input: 35 | with patch('app.print_blogs') as mocked_print_blogs: 36 | mocked_input.side_effect = ('l', 'q') 37 | app.menu() 38 | 39 | mocked_print_blogs.assert_called() 40 | 41 | def test_menu_calls_ask_read_blogs(self): 42 | with patch('builtins.input') as mocked_input: 43 | with patch('app.ask_read_blog') as mocked_ask_read_blog: 44 | mocked_input.side_effect = ('r', 'Test', 'q') 45 | app.menu() 46 | 47 | mocked_ask_read_blog.assert_called() 48 | 49 | def test_menu_calls_ask_create_post(self): 50 | with patch('builtins.input') as mocked_input: 51 | with patch('app.ask_create_post') as mocked_ask_create_post: 52 | mocked_input.side_effect = ('p', 'Test', 'New Post', 'New Content', 'q') 53 | app.menu() 54 | 55 | mocked_ask_create_post.assert_called() 56 | 57 | def test_print_blogs(self): 58 | with patch('builtins.print') as mocked_print: 59 | app.print_blogs() 60 | mocked_print.assert_called_with('- Test by Test Author (0 posts)') 61 | 62 | def test_ask_create_blog(self): 63 | with patch('builtins.input') as mocked_input: 64 | mocked_input.side_effect = ('Test', 'Author') 65 | 66 | app.ask_create_blog() 67 | 68 | self.assertIsNotNone(app.blogs.get('Test')) 69 | self.assertEqual(app.blogs.get('Test').title, 'Test') 70 | self.assertEqual(app.blogs.get('Test').author, 'Author') 71 | 72 | def test_ask_read_blog(self): 73 | with patch('builtins.input', return_value='Test'): 74 | with patch('app.print_posts') as mocked_print_posts: 75 | app.ask_read_blog() 76 | 77 | mocked_print_posts.assert_called_with(app.blogs['Test']) 78 | 79 | def test_print_posts(self): 80 | blog = app.blogs['Test'] 81 | blog.create_post('Post title', 'Post content') 82 | 83 | with patch('app.print_post') as mocked_print_post: 84 | app.print_posts(blog) 85 | 86 | mocked_print_post.assert_called_with(blog.posts[0]) 87 | 88 | def test_print_post(self): 89 | post = Post('Post title', 'Post content') 90 | expected_print = """ 91 | --- Post title --- 92 | 93 | Post content 94 | 95 | """ 96 | 97 | with patch('builtins.print') as mocked_print: 98 | app.print_post(post) 99 | 100 | mocked_print.assert_called_with(expected_print) 101 | 102 | def test_ask_create_post(self): 103 | blog = app.blogs['Test'] 104 | with patch('builtins.input') as mocked_input: 105 | mocked_input.side_effect = ('Test', 'Test Title', 'Test Content') 106 | 107 | app.ask_create_post() 108 | 109 | self.assertEqual(blog.posts[0].title, 'Test Title') 110 | self.assertEqual(blog.posts[0].content, 'Test Content') 111 | # mock : 의존성이 있는 것들을 실제로 실행시키지 말고 호출 여부, 인터페이스만 확인! -------------------------------------------------------------------------------- /python/tdd/blog/tests/unit/blog_test.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | from blog import Blog 3 | 4 | 5 | class BlogTest(TestCase): 6 | def test_create_blog(self): 7 | b = Blog('Test', 'Test Author') 8 | 9 | self.assertEqual('Test', b.title) 10 | self.assertEqual('Test Author', b.author) 11 | self.assertListEqual([], b.posts) 12 | self.assertEqual(0, len(b.posts)) 13 | 14 | def test_repr(self): 15 | b = Blog('Test', 'Test Author') 16 | b2 = Blog('My Day', 'Rolf') 17 | 18 | self.assertEqual(b.__repr__(), 'Test by Test Author (0 posts)') 19 | self.assertEqual(b2.__repr__(), 'My Day by Rolf (0 posts)') 20 | 21 | def test_repr_multiple_posts(self): 22 | b = Blog('Test', 'Test Author') 23 | b.posts = ['test'] 24 | b2 = Blog('My Day', 'Rolf') 25 | b2.posts = ['test', 'another'] 26 | 27 | self.assertEqual(b.__repr__(), 'Test by Test Author (1 post)') 28 | self.assertEqual(b2.__repr__(), 'My Day by Rolf (2 posts)') 29 | 30 | -------------------------------------------------------------------------------- /python/tdd/blog/tests/unit/post_test.py: -------------------------------------------------------------------------------- 1 | from unittest import TestCase 2 | from post import Post 3 | 4 | 5 | class PostTest(TestCase): 6 | def test_create_post(self): 7 | p = Post('Test', 'Test Content') 8 | 9 | self.assertEqual('Test', p.title) 10 | self.assertEqual('Test Content', p.content) 11 | 12 | def test_json(self): 13 | p = Post('Test', 'Test Content') 14 | expected = {'title': 'Test', 'content': 'Test Content'} 15 | 16 | self.assertDictEqual(expected, p.json()) 17 | 18 | -------------------------------------------------------------------------------- /python/virtualenv setting.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "## virtualenv install\n", 8 | "> pip3 install virtualenv\n", 9 | "\n", 10 | "## make virtualenv\n", 11 | "> virtualenv --python=python3(version) env(이름)\n", 12 | "\n", 13 | "\n", 14 | "## activate\n", 15 | "> source env/bin/activate\n", 16 | "\n", 17 | "## install library\n", 18 | "> pip3 install pandas\n", 19 | "\n", 20 | "## current environment to requirements.txt\n", 21 | "> pip3 freeze > requirements.txt\n", 22 | "\n", 23 | "## install from requirements.txt\n", 24 | "> pip install -r requirements.txt" 25 | ] 26 | }, 27 | { 28 | "cell_type": "code", 29 | "execution_count": null, 30 | "metadata": { 31 | "collapsed": true 32 | }, 33 | "outputs": [], 34 | "source": [] 35 | } 36 | ], 37 | "metadata": { 38 | "kernelspec": { 39 | "display_name": "Python 3", 40 | "language": "python", 41 | "name": "python3" 42 | }, 43 | "language_info": { 44 | "codemirror_mode": { 45 | "name": "ipython", 46 | "version": 3 47 | }, 48 | "file_extension": ".py", 49 | "mimetype": "text/x-python", 50 | "name": "python", 51 | "nbconvert_exporter": "python", 52 | "pygments_lexer": "ipython3", 53 | "version": "3.6.1" 54 | }, 55 | "latex_envs": { 56 | "LaTeX_envs_menu_present": true, 57 | "autocomplete": true, 58 | "bibliofile": "biblio.bib", 59 | "cite_by": "apalike", 60 | "current_citInitial": 1, 61 | "eqLabelWithNumbers": true, 62 | "eqNumInitial": 1, 63 | "hotkeys": { 64 | "equation": "Ctrl-E", 65 | "itemize": "Ctrl-I" 66 | }, 67 | "labels_anchors": false, 68 | "latex_user_defs": false, 69 | "report_style_numbering": false, 70 | "user_envs_cfg": false 71 | } 72 | }, 73 | "nbformat": 4, 74 | "nbformat_minor": 2 75 | } 76 | -------------------------------------------------------------------------------- /python/virtualenvwrapper.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# virtualenvwrapper" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "- virtualenv는 많은 사람들이 사용하고 있을텐데, 매번 해당 폴더에 들어가서 source env/bin/activate 해야되는데 그 경로를 기억하기 어려운 경우가 존재\n", 15 | "- 이럴 경우 virtualenvwrapper를 사용하면 편함!\n", 16 | "~~~\n", 17 | "pip install virtualenvwrapper\n", 18 | "cd ~\n", 19 | "mkdir ~/.virtualenvs\n", 20 | "~~~" 21 | ] 22 | }, 23 | { 24 | "cell_type": "markdown", 25 | "metadata": {}, 26 | "source": [ 27 | "그 후, .bashrc / .zshrc에 아래 코드를 입력\n", 28 | "~~~\n", 29 | "vi ~/.bashrc\n", 30 | "or\n", 31 | "vi ~/.zshrc\n", 32 | "\n", 33 | "\n", 34 | "export WORKON_HOME=~/.virtualenvs\n", 35 | "export VIRTUALENVWRAPPER_PYTHON='$(python3 혹은 python 경로)' \n", 36 | "source /usr/local/bin/virtualenvwrapper.sh\n", 37 | "~~~\n", 38 | "- python3 혹은 python 경로는 which python3 / python (또는 python2)를 입력해서 나온 곳을 넣어주면 된다\n", 39 | "- virtualenvwrapper.sh 경로로 에러가 난다면 which virtualenvwrapper.sh로 경로 찾은 후 넣어두면 된다" 40 | ] 41 | }, 42 | { 43 | "cell_type": "markdown", 44 | "metadata": {}, 45 | "source": [ 46 | "## 명령어\n", 47 | "### 가상환경 만들기\n", 48 | "~~~\n", 49 | "mkvirtualenv 가상환경이름\n", 50 | "~~~" 51 | ] 52 | }, 53 | { 54 | "cell_type": "markdown", 55 | "metadata": {}, 56 | "source": [ 57 | "### 가상환경 지우기\n", 58 | "~~~\n", 59 | "rmvirtualenv 가상환경이름\n", 60 | "~~~\n", 61 | "\n", 62 | "### 가상환경 복사하기\n", 63 | "~~~\n", 64 | "cpvirtualenv old환경이름 new환경이름\n", 65 | "~~~\n", 66 | "\n", 67 | "### 가상환경 진입\n", 68 | "~~~\n", 69 | "workon 가상환경이름\n", 70 | "~~~\n", 71 | "\n", 72 | "### 가상환경 빠져나오기\n", 73 | "~~~\n", 74 | "deactivate\n", 75 | "~~~\n", 76 | "\n", 77 | "- 가상환경 이름을 바꾸고 싶다면, cp(복사) 후, 기존의 것을 rm하는 방식을 사용하면 편함" 78 | ] 79 | } 80 | ], 81 | "metadata": { 82 | "kernelspec": { 83 | "display_name": "Python 3", 84 | "language": "python", 85 | "name": "python3" 86 | }, 87 | "language_info": { 88 | "codemirror_mode": { 89 | "name": "ipython", 90 | "version": 3 91 | }, 92 | "file_extension": ".py", 93 | "mimetype": "text/x-python", 94 | "name": "python", 95 | "nbconvert_exporter": "python", 96 | "pygments_lexer": "ipython3", 97 | "version": "3.6.3" 98 | }, 99 | "latex_envs": { 100 | "LaTeX_envs_menu_present": true, 101 | "autocomplete": true, 102 | "bibliofile": "biblio.bib", 103 | "cite_by": "apalike", 104 | "current_citInitial": 1, 105 | "eqLabelWithNumbers": true, 106 | "eqNumInitial": 1, 107 | "hotkeys": { 108 | "equation": "Ctrl-E", 109 | "itemize": "Ctrl-I" 110 | }, 111 | "labels_anchors": false, 112 | "latex_user_defs": false, 113 | "report_style_numbering": false, 114 | "user_envs_cfg": false 115 | }, 116 | "varInspector": { 117 | "cols": { 118 | "lenName": 16, 119 | "lenType": 16, 120 | "lenVar": 40 121 | }, 122 | "kernels_config": { 123 | "python": { 124 | "delete_cmd_postfix": "", 125 | "delete_cmd_prefix": "del ", 126 | "library": "var_list.py", 127 | "varRefreshCmd": "print(var_dic_list())" 128 | }, 129 | "r": { 130 | "delete_cmd_postfix": ") ", 131 | "delete_cmd_prefix": "rm(", 132 | "library": "var_list.r", 133 | "varRefreshCmd": "cat(var_dic_list()) " 134 | } 135 | }, 136 | "types_to_exclude": [ 137 | "module", 138 | "function", 139 | "builtin_function_or_method", 140 | "instance", 141 | "_Feature" 142 | ], 143 | "window_display": false 144 | } 145 | }, 146 | "nbformat": 4, 147 | "nbformat_minor": 2 148 | } 149 | -------------------------------------------------------------------------------- /reinforcement_learning/All for deeplearing/Lab 02. Playing OpenAI GYM Games.py: -------------------------------------------------------------------------------- 1 | import gym 2 | from gym.envs.registration import register 3 | import sys, tty, termios 4 | 5 | class _Getch: 6 | def __call__(self): 7 | fd = sys.stdin.fileno() 8 | old_settings = termios.tcgetattr(fd) 9 | try: 10 | tty.setraw(sys.stdin.fileno()) 11 | ch = sys.stdin.read(3) 12 | finally: 13 | termios.tcsetattr(fd, termios.TCSADRAIN, old_settings) 14 | return ch 15 | 16 | inkey = _Getch() 17 | 18 | LEFT = 0 19 | DOWN = 1 20 | RIGHT = 2 21 | UP = 3 22 | 23 | arrow_keys = {'\x1b[A': UP, '\x1b[B': DOWN, '\x1b[C' : RIGHT, '\x1b[D' : LEFT} 24 | 25 | register(id = 'FrozenLake-v3', entry_point='gym.envs.toy_text:FrozenLakeEnv', 26 | kwargs={'map_name': '4x4', 'is_slippery':False}) 27 | 28 | env = gym.make('FrozenLake-v3') 29 | env.render() 30 | 31 | while True: 32 | key = inkey() 33 | if key not in arrow_keys.keys(): 34 | print("Game aborted!") 35 | break 36 | 37 | action = arrow_keys[key] 38 | state, reward, done, info = env.step(action) 39 | env.render() 40 | print("State: ", state, "Action: ", action, "Reward: ", reward, "Info: ", info) 41 | 42 | if done: 43 | print("Finished with reward", reward) 44 | break -------------------------------------------------------------------------------- /reinforcement_learning/All for deeplearing/Lab 03. Dummy Q-learning(table).py: -------------------------------------------------------------------------------- 1 | import gym 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | from gym.envs.registration import register 5 | import random 6 | 7 | def rargmax(vector): 8 | ''' 9 | :param vector: vector value 10 | :return: argmax that chooses randomly among eligible maxmimum indices 11 | ''' 12 | m = np.amax(vector) 13 | indices = np.nonzero(vector == m)[0] 14 | return random.choice(indices) 15 | 16 | register( 17 | id = 'FrozenLake-v3', 18 | entry_point = 'gym.envs.toy_text:FrozenLakeEnv', 19 | kwargs = {'map_name' : '4x4', 'is_slippery' : False} 20 | ) 21 | env = gym.make("FrozenLake-v3") 22 | 23 | Q = np.zeros([env.observation_space.n, env.action_space.n]) 24 | 25 | num_episodes = 2000 26 | 27 | rList = [] 28 | for i in range(num_episodes): 29 | state = env.reset() 30 | rAll = 0 31 | done = False 32 | 33 | while not done: 34 | action = rargmax(Q[state, :]) 35 | new_state, reward, done, _ = env.step(action) 36 | Q[state, action] = reward + np.max(Q[new_state, :]) 37 | rAll += reward 38 | state = new_state 39 | 40 | rList.append(rAll) 41 | print("Success rate: " + str(sum(rList)/num_episodes)) 42 | print("Final Q-Table Values") 43 | print("Left Donw Right Up") 44 | print(Q) 45 | plt.bar(range(len(rList)), rList, color='blue') 46 | plt.show() -------------------------------------------------------------------------------- /reinforcement_learning/All for deeplearing/Lab 04. Q-learning (exploit&exploration and discounted reward).py: -------------------------------------------------------------------------------- 1 | import gym 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | from gym.envs.registration import register 5 | 6 | 7 | register( 8 | id = 'FrozenLake-v3', 9 | entry_point = 'gym.envs.toy_text:FrozenLakeEnv', 10 | kwargs = {'map_name' : '4x4', 'is_slippery' : False} 11 | ) 12 | env = gym.make("FrozenLake-v3") 13 | 14 | Q = np.zeros([env.observation_space.n, env.action_space.n]) 15 | dis = 0.99 16 | num_episodes = 2000 17 | 18 | rList = [] 19 | for i in range(num_episodes): 20 | state = env.reset() 21 | rAll = 0 22 | done = False 23 | 24 | # Q-learning with noise and discount 25 | while not done: 26 | action = np.argmax(Q[state,:] + np.random.randn(1, env.action_space.n) / (i + 1)) 27 | new_state, reward, done, _ = env.step(action) 28 | Q[state, action] = reward + dis * np.max(Q[new_state, :]) 29 | rAll += reward 30 | state = new_state 31 | 32 | # e-greedy 33 | # e = 1. / (( i // 100) + 1) 34 | # while not done: 35 | # if np.random.rand(1) < e: 36 | # action = env.action_space.sample() 37 | # else: 38 | # action = np.argmax(Q[state, :]) 39 | # 40 | # new_state, reward, done, _ = env.step(action) 41 | # Q[state, action] = reward + dis * np.max(Q[new_state, :]) 42 | # rAll += reward 43 | # state = new_state 44 | # 45 | rList.append(rAll) 46 | 47 | 48 | 49 | print("Success rate: " + str(sum(rList)/num_episodes)) 50 | print("Final Q-Table Values") 51 | print("Left Donw Right Up") 52 | print(Q) 53 | plt.bar(range(len(rList)), rList, color='blue') 54 | plt.show() -------------------------------------------------------------------------------- /reinforcement_learning/All for deeplearing/Lab 05-1. Q-learning on Nondeterministic Worlds.py: -------------------------------------------------------------------------------- 1 | import gym 2 | import readchar 3 | 4 | LEFT = 0 5 | DOWN = 1 6 | RIGHT = 2 7 | UP = 3 8 | 9 | arrow_keys = {'\x1b[A': UP, '\x1b[B': DOWN, '\x1b[C' : RIGHT, '\x1b[D' : LEFT} 10 | 11 | env = gym.make("FrozenLake-v0") 12 | env.reset() 13 | env.render() 14 | 15 | while True: 16 | key = readchar.readkey() 17 | if key not in arrow_keys.keys(): 18 | print("Game aborted!") 19 | break 20 | 21 | action = arrow_keys[key] 22 | state, reward, done, info = env.step(action) 23 | env.render() 24 | print("State: ", state, "Action: ", action, "Reward: ", reward, "Info: ", info) 25 | 26 | if done: 27 | print("Finished with reward", reward) 28 | break -------------------------------------------------------------------------------- /reinforcement_learning/All for deeplearing/Lab 05-2. Q-learning on Nondeterministic Worlds.py: -------------------------------------------------------------------------------- 1 | import gym 2 | import numpy as np 3 | import matplotlib.pyplot as plt 4 | 5 | 6 | env = gym.make("FrozenLake-v0") 7 | 8 | Q = np.zeros([env.observation_space.n, env.action_space.n]) 9 | learning_rate = 0.85 10 | dis = 0.99 11 | num_episodes = 2000 12 | 13 | rList = [] 14 | for i in range(num_episodes): 15 | state = env.reset() 16 | rAll = 0 17 | done = False 18 | 19 | while not done: 20 | action = np.argmax(Q[state,:] + np.random.randn(1, env.action_space.n) / (i + 1)) 21 | new_state, reward, done, _ = env.step(action) 22 | Q[state, action] = (1-learning_rate) * Q[state, action] + learning_rate * (reward + dis * np.max(Q[new_state, :])) 23 | rAll += reward 24 | state = new_state 25 | 26 | rList.append(rAll) 27 | 28 | 29 | 30 | print("Success rate: " + str(sum(rList)/num_episodes)) 31 | print("Final Q-Table Values") 32 | print("Left Donw Right Up") 33 | print(Q) 34 | plt.bar(range(len(rList)), rList, color='blue') 35 | plt.show() -------------------------------------------------------------------------------- /reinforcement_learning/img/circle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/reinforcement_learning/img/circle.png -------------------------------------------------------------------------------- /reinforcement_learning/img/down.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/reinforcement_learning/img/down.png -------------------------------------------------------------------------------- /reinforcement_learning/img/left.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/reinforcement_learning/img/left.png -------------------------------------------------------------------------------- /reinforcement_learning/img/rectangle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/reinforcement_learning/img/rectangle.png -------------------------------------------------------------------------------- /reinforcement_learning/img/right.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/reinforcement_learning/img/right.png -------------------------------------------------------------------------------- /reinforcement_learning/img/triangle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/reinforcement_learning/img/triangle.png -------------------------------------------------------------------------------- /reinforcement_learning/img/up.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/zzsza/TIL/8e623ebcbeca0f3fe1acbc2900f992a3c462a50b/reinforcement_learning/img/up.png -------------------------------------------------------------------------------- /reinforcement_learning/textbook summary/03. policy_iteration.py: -------------------------------------------------------------------------------- 1 | import random 2 | from environment import GraphicDisplay, Env 3 | 4 | class PolicyIteration: 5 | def __init__(self, env): 6 | self.env = env 7 | self.value_table = [[0.00] * env.width for _ in range(env.height)] 8 | self.policy_table = [[0.25] * env.width for _ in range(env.height)] 9 | self.policy_table[2][2] = [] # 마침 상태 10 | self.discount_factor = 0.9 # 감가율 11 | 12 | # 정책 평가 13 | def policy_evaluation(self): 14 | next_value_table = [[0.00] * self.env.width for _ in range(self.env.height)] 15 | 16 | # 모든 상태에서 벨만 기대방정식 계산 17 | for state in self.env.get_all_states(): 18 | value = 0.0 19 | if state == [2, 2]: 20 | next_value_table[state[0]][state[1]] = 0.0 21 | continue 22 | 23 | # 벨만 기대방정식 24 | for action in self.env.possible_actions: 25 | next_state = self.env.state_after_action(state, action)action) 26 | reward = self.env.get_reward(state, action) 27 | next_value = self.get_value(next_state) 28 | value += (self.get_policy(state)[action] * (reward + self.discount_factor * next_value)) 29 | 30 | next_value_table[state[0]][state[1]] = round(value, 2) 31 | 32 | self.value_table = next_value_table 33 | 34 | # 정책 발전 35 | def policy_improvement(self): 36 | next_policy = self.policy_table 37 | 38 | for state in self.env.get_all_states(): 39 | 40 | if state == [2, 2]: 41 | continue 42 | 43 | value = -999999 44 | max_index = [] 45 | result = [0.0, 0.0, 0.0, 0.0] 46 | 47 | # 모든 행동에 대해 [보상 + (감가율 * 다음 상태 가치 함수)] 계산 48 | for index, action in enumerate(self.env.possible_actions): 49 | next_state = self.env.state_after_action(state, action) 50 | reward = self.env.get_reward(state, action) 51 | next_value = self.get_value(next_state) 52 | temp = reward + self.discount_factor * next_value 53 | 54 | 55 | # 받을 보상이 최대인 행동의 인덱스 추출 56 | if temp == value: 57 | max_index.append(index) 58 | elif temp > value: 59 | value = temp 60 | max_index.clear() 61 | max_index.append(index) 62 | 63 | prob = 1 / len(max_index) 64 | 65 | for index in max_index: 66 | result[index] = prob 67 | 68 | next_policy[state[0]][state[1]] = result 69 | 70 | self.policy_table = next_policy 71 | 72 | # 특정 상태에서 정책에 따른 행동 73 | def get_action(self, state): 74 | random_pick = random.randrange(100) / 100 75 | policy = self.get_policy(state) 76 | policy_sum = 0.0 77 | 78 | for index, value in enumerate(policy): 79 | policy_sum += value 80 | if random_pick < policy_sum: 81 | return index 82 | 83 | def get_policy(self, state): 84 | if state == [2, 2]: 85 | return 0.0 86 | return self.policy_table[state[0]][state[1]] 87 | 88 | def get_value(self, state): 89 | return round(self.value_table[state[0]][state[1]], 2) 90 | 91 | if __name__ == "__main__": 92 | env = Env() 93 | policy_iteration = PolicyIteration(env) 94 | grid_world = GraphicDisplay(policy_iteration) 95 | grid_world.mainloop() -------------------------------------------------------------------------------- /reinforcement_learning/textbook summary/03. 강화학습 기초(2) - 그리드월드와 다이나믹 프로그래밍.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 03. 강화학습 기초(2) - 그리드월드와 다이나믹 프로그래밍" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "\n", 15 | "- 다이나믹 프로그래밍 : 작은 문제가 큰 문제 안에 중첩돼 있는 경우에 작은 문제의 답을 다른 작은 문제에서 이용함으로써 효율적으로 계산하는 방법\n", 16 | "\n", 17 | "- 정책 이터레이션 : DP로 벨만 기대 방정식을 푸는 것\n", 18 | "- 가치 이터레이션 : 벨만 최적 방정식을 이용해 문제를 푸는 것" 19 | ] 20 | }, 21 | { 22 | "cell_type": "code", 23 | "execution_count": null, 24 | "metadata": { 25 | "collapsed": true 26 | }, 27 | "outputs": [], 28 | "source": [] 29 | } 30 | ], 31 | "metadata": { 32 | "kernelspec": { 33 | "display_name": "Python 3", 34 | "language": "python", 35 | "name": "python3" 36 | }, 37 | "language_info": { 38 | "codemirror_mode": { 39 | "name": "ipython", 40 | "version": 3 41 | }, 42 | "file_extension": ".py", 43 | "mimetype": "text/x-python", 44 | "name": "python", 45 | "nbconvert_exporter": "python", 46 | "pygments_lexer": "ipython3", 47 | "version": "3.6.1" 48 | }, 49 | "latex_envs": { 50 | "LaTeX_envs_menu_present": true, 51 | "autocomplete": true, 52 | "bibliofile": "biblio.bib", 53 | "cite_by": "apalike", 54 | "current_citInitial": 1, 55 | "eqLabelWithNumbers": true, 56 | "eqNumInitial": 1, 57 | "hotkeys": { 58 | "equation": "Ctrl-E", 59 | "itemize": "Ctrl-I" 60 | }, 61 | "labels_anchors": false, 62 | "latex_user_defs": false, 63 | "report_style_numbering": false, 64 | "user_envs_cfg": false 65 | } 66 | }, 67 | "nbformat": 4, 68 | "nbformat_minor": 2 69 | } 70 | --------------------------------------------------------------------------------