├── .gitignore ├── 1_Schedule ├── 20161102.md ├── 20161109.md ├── 20161116.md ├── 20161123.md ├── 20161130.md ├── 20161207.md ├── 20161214.md ├── 20161221.md ├── 20161228.md ├── 20170104.md ├── 20170111.md ├── 20170118.md ├── 20170125.md ├── 20170201.md ├── 20170316.md ├── 20170329.md ├── 20170405.md ├── 20180221.md ├── 20180307.md ├── 20180314.md └── 20180321.md ├── 2_LabMaterials ├── README.md └── kycho_lec │ ├── lecture_slides │ └── test.txt │ └── practice_code │ └── Seq2Seq │ └── s2s.ipynb ├── 3_Paper ├── 20170104_GloVe_EunSook.pdf ├── GloVe- Global Vectors for Word Representation.pdf ├── GloVe-Global Vectors forWord Representation.ipynb ├── Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees-deepmind-nlp2.pdf ├── README.md ├── Sequence-To-Sequence.pdf └── textrankalgorithm-141001134151-phpapp02.pdf ├── 4_Source ├── .DS_Store ├── 04_deeppavlov_chitchat.ipynb ├── README.md ├── SimpleDS-master_by_junchangwook │ ├── .DS_Store │ ├── LICENSE │ ├── README.md │ ├── README.txt │ ├── bin │ │ └── simpleDS │ │ │ ├── interaction │ │ │ ├── SimpleActions.class │ │ │ ├── SimpleInteractionPolicy.class │ │ │ └── SimpleUserSimulator.class │ │ │ ├── learning │ │ │ ├── SimpleAgent$1.class │ │ │ ├── SimpleAgent.class │ │ │ ├── SimpleClassifier.class │ │ │ └── SimpleEnvironment.class │ │ │ ├── main │ │ │ └── SimpleDS.class │ │ │ ├── networking │ │ │ ├── SimpleServer.class │ │ │ ├── SimpleSocketHandler$MessageHandler.class │ │ │ ├── SimpleSocketHandler.class │ │ │ ├── SimpleSocketServer.class │ │ │ └── SimpleWebServer.class │ │ │ └── util │ │ │ ├── ConfigParser.class │ │ │ ├── IOUtil.class │ │ │ ├── KeyValuePair.class │ │ │ ├── Logger.class │ │ │ ├── StringUtil.class │ │ │ └── Vocabulary.class │ ├── build.xml │ ├── config.txt │ ├── doc │ │ ├── How2UseSimpleDS.txt │ │ └── hc-iwsds2016.pdf │ ├── lib │ │ ├── jetty │ │ │ ├── jetty-client-9.0.5.v20130815.jar │ │ │ ├── jetty-http-9.0.5.v20130815.jar │ │ │ ├── jetty-io-9.0.5.v20130815.jar │ │ │ ├── jetty-security-9.0.5.v20130815.jar │ │ │ ├── jetty-server-9.0.5.v20130815.jar │ │ │ ├── jetty-servlet-9.0.5.v20130815.jar │ │ │ ├── jetty-servlets-9.0.5.v20130815.jar │ │ │ ├── jetty-util-9.0.5.v20130815.jar │ │ │ ├── jetty-webapp-9.0.5.v20130815.jar │ │ │ └── servlet-api-3.0.jar │ │ ├── websocket │ │ │ ├── websocket-api-9.0.5.v20130815.jar │ │ │ ├── websocket-client-9.0.5.v20130815.jar │ │ │ ├── websocket-common-9.0.5.v20130815.jar │ │ │ ├── websocket-server-9.0.5.v20130815.jar │ │ │ └── websocket-servlet-9.0.5.v20130815.jar │ │ └── weka.jar │ ├── models │ │ └── demonstrations.arff │ ├── resources │ │ ├── english │ │ │ ├── SlotValues.txt │ │ │ ├── SysResponses.txt │ │ │ └── UsrResponses.txt │ │ ├── german │ │ │ ├── SlotValues.txt │ │ │ ├── SysResponses.txt │ │ │ └── UsrResponses.txt │ │ └── spanish │ │ │ ├── SlotValues.txt │ │ │ ├── SysResponses.txt │ │ │ └── UsrResponses.txt │ ├── results │ │ ├── english │ │ │ ├── simpleds-output.png │ │ │ ├── simpleds-output.txt │ │ │ └── simpleds-policy.json │ │ ├── german │ │ │ ├── simpleds-output.png │ │ │ ├── simpleds-output.txt │ │ │ └── simpleds-policy.json │ │ └── spanish │ │ │ ├── simpleds-output.png │ │ │ ├── simpleds-output.txt │ │ │ └── simpleds-policy.json │ ├── screenshots │ │ └── Screenshot-SimpleDS.png │ ├── scripts │ │ ├── plotdata.m │ │ └── run.sh │ ├── src │ │ └── simpleDS │ │ │ ├── interaction │ │ │ ├── SimpleActions.java │ │ │ ├── SimpleInteractionPolicy.java │ │ │ └── SimpleUserSimulator.java │ │ │ ├── learning │ │ │ ├── SimpleAgent.java │ │ │ ├── SimpleClassifier.java │ │ │ └── SimpleEnvironment.java │ │ │ ├── main │ │ │ └── SimpleDS.java │ │ │ ├── networking │ │ │ ├── SimpleServer.java │ │ │ ├── SimpleSocketHandler.java │ │ │ ├── SimpleSocketServer.java │ │ │ └── SimpleWebServer.java │ │ │ └── util │ │ │ ├── ConfigParser.java │ │ │ ├── IOUtil.java │ │ │ ├── KeyValuePair.java │ │ │ ├── Logger.java │ │ │ ├── StringUtil.java │ │ │ └── Vocabulary.java │ └── web │ │ ├── .svn │ │ └── entries │ │ ├── convnet │ │ ├── convnet.js │ │ ├── deepqlearn.js │ │ └── util.js │ │ ├── main │ │ └── runclient.js │ │ ├── node_modules │ │ └── ws │ │ │ ├── .npmignore │ │ │ ├── .travis.yml │ │ │ ├── Makefile │ │ │ ├── README.md │ │ │ ├── index.js │ │ │ ├── lib │ │ │ ├── BufferPool.js │ │ │ ├── BufferUtil.fallback.js │ │ │ ├── BufferUtil.js │ │ │ ├── ErrorCodes.js │ │ │ ├── Extensions.js │ │ │ ├── PerMessageDeflate.js │ │ │ ├── Receiver.hixie.js │ │ │ ├── Receiver.js │ │ │ ├── Sender.hixie.js │ │ │ ├── Sender.js │ │ │ ├── Validation.fallback.js │ │ │ ├── Validation.js │ │ │ ├── WebSocket.js │ │ │ ├── WebSocketServer.js │ │ │ └── browser.js │ │ │ ├── node_modules │ │ │ ├── options │ │ │ │ ├── .npmignore │ │ │ │ ├── Makefile │ │ │ │ ├── README.md │ │ │ │ ├── lib │ │ │ │ │ └── options.js │ │ │ │ └── package.json │ │ │ └── ultron │ │ │ │ ├── .npmignore │ │ │ │ ├── .travis.yml │ │ │ │ ├── LICENSE │ │ │ │ ├── README.md │ │ │ │ ├── index.js │ │ │ │ ├── package.json │ │ │ │ └── test.js │ │ │ └── package.json │ │ └── util │ │ ├── FileSaver.js │ │ └── WebSocket.js ├── Telegram_ChatBot_by_yangsunhee │ ├── .DS_Store │ └── examples │ │ ├── LICENSE.txt │ │ ├── README.md │ │ ├── conversationbot.png │ │ ├── conversationbot.py │ │ ├── conversationbot2.png │ │ ├── conversationbot2.py │ │ ├── echobot.py │ │ ├── echobot2.py │ │ ├── inlinebot.py │ │ ├── inlinekeyboard.py │ │ ├── timerbot.py │ │ ├── ts.py │ │ └── w2v.py ├── kaggle_insincere_competition │ ├── Data_Prepro.ipynb │ └── Modeling.ipynb ├── lec5_ngram_example_code │ ├── ngram-model.ipynb │ └── ratings_train.txt ├── ngram-master_by_kimchanghoon │ ├── .travis.yml │ ├── LICENSE │ ├── README.md │ ├── corpus.data │ ├── ngram.py │ ├── smoothing.pdf │ └── tests.py ├── nmt_with_attention.ipynb ├── seq2seq_chatbot_by_ohyeontak │ ├── .DS_Store │ ├── README.md │ ├── __pycache__ │ │ ├── config.cpython-35.pyc │ │ ├── dialog.cpython-35.pyc │ │ └── model.cpython-35.pyc │ ├── chat.py │ ├── config.py │ ├── dialog.py │ ├── logs │ │ └── events.out.tfevents.1484699399.mpr │ ├── model.py │ └── train.py ├── test.py └── transformer_JH.ipynb ├── 5_Idea └── README.md ├── 6_dataset └── data_sites ├── 7.YandexSchool_NLPCourse ├── README.md ├── assets │ └── week01 │ │ ├── yandex_embedding_007.png │ │ ├── yandex_embedding_012.png │ │ ├── yandex_embedding_013.png │ │ ├── yandex_embedding_015.png │ │ ├── yandex_embedding_018.png │ │ ├── yandex_embedding_022.png │ │ ├── yandex_embedding_025.png │ │ ├── yandex_embedding_029.png │ │ ├── yandex_embedding_030.png │ │ ├── yandex_embedding_031.png │ │ ├── yandex_embedding_033.png │ │ ├── yandex_embedding_035.png │ │ ├── yandex_embedding_039.png │ │ ├── yandex_embedding_040.png │ │ ├── yandex_embedding_044.png │ │ ├── yandex_embedding_045.png │ │ ├── yandex_embedding_047.png │ │ ├── yandex_embedding_048.png │ │ ├── yandex_embedding_049.png │ │ ├── yandex_embedding_052.png │ │ ├── yandex_embedding_054.png │ │ ├── yandex_embedding_056.png │ │ ├── yandex_embedding_057.png │ │ ├── yandex_embedding_058.png │ │ ├── yandex_embedding_059.png │ │ ├── yandex_embedding_060.png │ │ ├── yandex_embedding_061.png │ │ ├── yandex_embedding_062.png │ │ ├── yandex_embedding_066.png │ │ ├── yandex_embedding_067.png │ │ ├── yandex_embedding_068.png │ │ ├── yandex_embedding_069.png │ │ ├── yandex_embedding_070.png │ │ ├── yandex_embedding_071.png │ │ ├── yandex_embedding_072.png │ │ ├── yandex_embedding_073.png │ │ ├── yandex_embedding_074.png │ │ ├── yandex_embedding_075.png │ │ ├── yandex_embedding_076.png │ │ ├── yandex_embedding_077.png │ │ ├── yandex_embedding_079.png │ │ ├── yandex_embedding_080.png │ │ ├── yandex_embedding_081.png │ │ ├── yandex_embedding_082.png │ │ ├── yandex_embedding_084.png │ │ ├── yandex_embedding_085.png │ │ ├── yandex_embedding_087.png │ │ ├── yandex_embedding_089.png │ │ ├── yandex_embedding_090.png │ │ ├── yandex_embedding_092.png │ │ ├── yandex_embedding_093.png │ │ └── yandex_embedding_094.png ├── week01_embeddings-lecture_slides.md ├── week01_embeddings-seminar.ipynb └── word2vec.py ├── Quora_insincere ├── .gitignore ├── CNN_with_RNN.ipynb ├── README.md ├── RNN_only.ipynb ├── jupyter_examples │ ├── Data_Prepro.ipynb │ ├── Modeling.ipynb │ ├── data_preprocessing.py │ ├── dataset │ │ ├── test.csv.zip │ │ └── train.csv.zip │ ├── lstm_kernel_shin.ipynb │ └── lstm_kernel_simple.ipynb └── lstm.py ├── README.md └── requirements.txt /.gitignore: -------------------------------------------------------------------------------- 1 | /2_NLP_Study 2 | .DS_Store 3 | .ipynb_checkpoints/ 4 | data/ 5 | sh/ 6 | results/ 7 | checkpoints/ 8 | 9 | # Byte-compiled / optimized / DLL files 10 | __pycache__/ 11 | *.py[cod] 12 | *$py.class 13 | 14 | # C extensions 15 | *.so 16 | 17 | # Distribution / packaging 18 | .Python 19 | build/ 20 | develop-eggs/ 21 | dist/ 22 | downloads/ 23 | eggs/ 24 | .eggs/ 25 | lib/ 26 | lib64/ 27 | parts/ 28 | sdist/ 29 | var/ 30 | wheels/ 31 | *.egg-info/ 32 | .installed.cfg 33 | *.egg 34 | MANIFEST 35 | 36 | # PyInstaller 37 | # Usually these files are written by a python script from a template 38 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 39 | *.manifest 40 | *.spec 41 | 42 | # Installer logs 43 | pip-log.txt 44 | pip-delete-this-directory.txt 45 | 46 | # Unit test / coverage reports 47 | htmlcov/ 48 | .tox/ 49 | .coverage 50 | .coverage.* 51 | .cache 52 | nosetests.xml 53 | coverage.xml 54 | *.cover 55 | .hypothesis/ 56 | .pytest_cache/ 57 | 58 | # Translations 59 | *.mo 60 | *.pot 61 | 62 | # Django stuff: 63 | *.log 64 | local_settings.py 65 | db.sqlite3 66 | 67 | # Flask stuff: 68 | instance/ 69 | .webassets-cache 70 | 71 | # Scrapy stuff: 72 | .scrapy 73 | 74 | # Sphinx documentation 75 | docs/_build/ 76 | 77 | # PyBuilder 78 | target/ 79 | 80 | # Jupyter Notebook 81 | .ipynb_checkpoints 82 | 83 | # pyenv 84 | .python-version 85 | 86 | # celery beat schedule file 87 | celerybeat-schedule 88 | 89 | # SageMath parsed files 90 | *.sage.py 91 | 92 | # Environments 93 | .env 94 | .venv 95 | env/ 96 | venv/ 97 | ENV/ 98 | env.bak/ 99 | venv.bak/ 100 | 101 | # Spyder project settings 102 | .spyderproject 103 | .spyproject 104 | 105 | # Rope project settings 106 | .ropeproject 107 | 108 | # mkdocs documentation 109 | /site 110 | 111 | # mypy 112 | .mypy_cache/ 113 | 114 | .vscode 115 | .ipynb_checkpoints 116 | 117 | *.voc 118 | checkPoint 119 | *.log 120 | 121 | OLD/ -------------------------------------------------------------------------------- /1_Schedule/20161102.md: -------------------------------------------------------------------------------- 1 | # 2016.11.02 2 | 3 | ## 프로젝트 이름 : '챗러닝(chat learning)' 챗봇과 딥러닝을 이용한 신개념 영어학습 서비스(앱) 4 | 5 | << LoadMap >> 6 | 7 | 1. 자연어 학습 8 | 2. 서비스화 9 | 3. 비즈니스 모델 생성 10 | 4. 상용화 11 | 12 | 13 | << 학습 >> 14 | 15 | 1)강의 16 | - 코세라 미시간 강의 : week3부터 보기 시작 17 | https://www.coursera.org/learn/natural-language-processing 18 | 19 | 2) 논문 20 | - 자연어 처리 관련된 CS224d 관련 논문 위주로 학습 21 | http://cs224d.stanford.edu/ 22 | 23 | - 11/9 : 서기호님 Sequence to Sequence Learning with Neural Networks 발표 24 | http://cs224d.stanford.edu/papers/seq2seq.pdf 25 | 26 | 3) 소스 리뷰 27 | - 자연어 처리, RNN, LSTM 관련 소스 리뷰 진행 28 | - 11/9 : 전창욱님 JavaScript기반 ChatBot 소스 리뷰 29 | 30 | - 그 다음 LSTM 관련 소스 준비 31 | 32 | 기호님 : 33 | - 워털루 대학에서 하는 50개 논문 사이트 공유 예정 34 | - 서울대 유투브 강의 : variational autoencoder - 곽동현님 35 | https://www.youtube.com/channel/UCph4USRd-ADh4WNBfFkK_qg 36 | 37 | << 매주 해야 할 리스트 >> 38 | 1. 논문1개 39 | 2. 강의1개 40 | 3. 소스1개 41 | 42 | << 서비스 화를 위한 고민 사항 >> 43 | - 앱을 만들 것 구체화, 필요 기술 리스트업 44 | 45 | << 서비스 구축 >> 46 | 47 | Idea_1) 48 | 영어 공부를 하는데 많이 느낀다. 49 | 50 | 채팅 형태로 영어공부의 피드백을 줘서 학습시키면 어떨까 하는 아이디어에서 시작해서 51 | 52 | 내가 좋아하는 영화, 미드의 대본 스크립트를 따와서 영화속 문장을 챗봇이 말해주고 사용자는 다음 답을 추측해서 답변을한다. 53 | 54 | 그리고 실제 영화의 정답과 내가 대답한 문장을 비교해서 알려준다. 55 | 이후, 문법이나 문장의 응용 보완을 추가적으로 제공한다. 56 | 57 | 서비스 구축의 형태는 안드로이드 어플, PC웹 등으로. 58 | 수익은 사용자가 대답후에 얻고자 하는 정보의 따라서 59 | - 무료화 : 정답의 후보들을 보여주는 것 60 | - 유료화 : 원어민 검증, 응용 문장 학습 등. 61 | 62 | << 기타 토의 >> 63 | 64 | 다음사전 65 | 안드로이드 바로 task에 띄어져 잇는 상태에서 꾹 누르면 그 앱을 보여줌 66 | 67 | 서머셋 모험의 달과 6펜스 68 | http://blog.daum.net/thomas999/17187710 69 | 70 | 최성준 : 구운몽 소설을 새로운 글을 만듦 71 | 72 | << 오늘 진행 한 사항>> 73 | 1. 성진님 Word2Vec 기초 설명 74 | 2. 수진님 코사인 유사도 설명 75 | -------------------------------------------------------------------------------- /1_Schedule/20161109.md: -------------------------------------------------------------------------------- 1 | 2 | << 11/9 Wen >> 3 | 4 | 1. NLTK : 은숙님 5 | - 소개 6 | - 사용법 7 | - 제공하는 기능들 8 | - 다음 주에 예제 돌려본 것 공유! 9 | http://www.nltk.org/ 10 | https://www.lucypark.kr/courses/2015-dm/text-mining.html 11 | https://www.datascienceschool.net/view-notebook/118731eec74b4ad3bdd2f89bab077e1b/ 12 | 13 | 2. Word2Vec : 창욱님 14 | - python join함수 : 문장을 join하고 space로 spilt 15 | - tf.nn.nce_loss : softmax와 비슷, 코사인 유사도 대신 사용 , 좀더 알아보기! 16 | - tf.nn.embedding_lookup 좀더 알아보기! 17 | - cbow : context > word 18 | - skip-gram : word > context 19 | - 배경 지식 : tf, if 20 | https://github.com/tensorflow/tensorflow/blob/r0.11/tensorflow/examples/tutorials/word2vec/word2vec_basic.py 21 | http://khanrc.tistory.com/entry/TensorFlow-7-word2vec-Implementation 22 | 23 | 3. 성진님 24 | - 최규민님 발표자료 공유 25 | - 아프리카TV, 추천 서비스 하고있음. 26 | http://www.slideshare.net/ssuser2fe594/2015-py-con-word2vec 27 | - 넷플릭스 : 코사인 유사도 28 | - 왓챠 : 사용자 별점대로 군집화 29 | http://www.slideshare.net/ssuser2fe594/deview2014-live-broadcasting 30 | 31 | 4. 논문 - 기호님 32 | - Sequence to Sequence Learning for NLP 33 | https://www.tensorflow.org/versions/r0.11/tutorials/seq2seq/index.html 34 | - Greedy : 탐욕적으로 가장 최대 좋은것만 취함, Multi Armed Bandit Algorithm 35 | - beam search : 여러가지 주어지는 거 36 | - AutoReply 37 | - SmartReplay 38 | - embedding 내부 알고리즘 원리! 알아야함! 39 | - 다음주에 Seq to Seq 코드리뷰 발표 40 | - word2vec에 입력값을 넣으면 뭐가 나올까? 41 | 42 | 5. Coursera : 성진님 43 | - 강의 설명 44 | 45 | 6. Service 회의 46 | 47 | << 다음주 >> 48 | 1. 기호님 : Seq to Seq 나머지, 소스코드 - 1시간 49 | 2. 광재님 : 서비스 - 30분 50 | 3. 성진님 : Coursera - 30분 51 | 4. 은숙님: NLTK 마무리 - 25분 52 | 53 | << Question >> 54 | 1. Word로 학습을 시키면 잘못된 문장이 나올 수도 있으니, 문장 단위로 학습을 시켜서 하는게 좋지 않을까? 55 | - 문장 자체를 하나의 단어 처럼 학습을 하게 되면, 중복해서 3문장들을 사용하지 않지 않나 56 | - Sentence to Vector 57 | - Doc to Vector : 논문, report 58 | - Semantic Similarlity 59 | 60 | << NLP의 대표적인 예들 >> 61 | - 번역 62 | - summary : 장문의 영어지문을 요약해줌 63 | - word2vec 상용 사례 : 뉴욕타임즈에서 음식 레시피 만들어줌, 문장을 기고해줌 64 | - IBM : chief wathson, 재료의 화학적 성분까지 파악해서 음식 추천해줌 ex) 김치와 어울리는 음식 65 | - 강화학습 챗봇 문장 생성 : 유명작가의 소설들을 학습 시켜서, 문장을 생성 시킴 66 | 67 | << 기타 >> 68 | - Text가 이미지보다 시간이 오래걸림 69 | - factorial 연산하게 될것임! 70 | - 원리 이해 Application 71 | -------------------------------------------------------------------------------- /1_Schedule/20161116.md: -------------------------------------------------------------------------------- 1 | << 11/16 Wen >> 2 | 3 | 1. 새로오신 분 : 사람인, 텍스트 분석 - 오연택님 4 | 2. 영어 데이터 관련된 소스들은 많음, step-by-step tutorials는 찾기 어려울것 5 | 3. 자연어 처리의 종류에는 분류, 의도검색, 추천 키워드 등이 있음 6 | 4. python 샘플 코드 돌려보기 7 | - DIBYY English : 단어를 찾으면 유투브 동영상을 찾아줌, 클립으로 짤라줌 (소장님) 8 | - TED 자막 크롤링 가능 (소장님) 9 | 10 | << 서비스 - 광재님 >> 11 | 국내에서 잘나가는 인강 어플 12 | 1. 시원스쿨 13 | 2. 아나두 14 | 3. 스피킹 맥스 15 | 16 | 국내만 해도 5조 시장이 점유되어 있음, 1만 7천개 정도가 있음. 17 | 암기법 - 뇌새김 - 연상기억법 - 빅데이터 -- 빅보카 18 | - 영어는 '습관' 만드는 요소는 두 가지 1) 재미 2) 반복 19 | 20 | 우리가 그러면... 챗봇을 가지고 어떻게 차별화를 할 것인지? 21 | - 사람들은 누구나 로망을 가지고 있음. 22 | -- TED 강의, Google Alert, 원문 23 | - 내가 원하는 컨텐츠를 가지고 하면 재미가 생긴다! 24 | - seamless service : 반복 25 | -- mobile : moving(voice), standing 26 | -- stationary : 장소 고정 27 | - 챗봇 서비스를 가지고 반복 학습을 하자! 28 | 29 | LoadMap으로는 30 | 1. 서비스 시나리오 선정 및 구현 31 | 2. 페이스북 광고 32 | 3. open Platform 형태 33 | 34 | == Title == 35 | -- 영어의 로망 -- 36 | 37 | 현재 학습시장 38 | - 오누이 앱 : 수학앱에서 시작해서 39 | - 사용자들이 원하는 컨텐츠를 업로드하면 가공해서 학습가능한 데이터로 만들어주는 것 40 | - 챗봇의 강점 : 서로 인터랙션 하는 것 같이 하는 학습법 41 | 자연어가 어려운게 모호성과 목표성이 없다는 것 42 | 43 | - 영화, 미드 매니아층에게는 딱 매칭 시킬 수 있도록 44 | - 미소년 챗봇 45 | 46 | 1. 학습 컨텐츠 47 | 2. 캐릭터 이입 48 | 3. 교정 49 | 50 | 여기에서 원어민과 사용자 간의 발음 음파의 차를 비교해서 발음 교정 해 주는것 51 | 다수의 70%를 만족시키지 말고 1명의 100%를 만족시키자! 52 | 린스타트업 방식으로 해보자! 53 | 챗봇형태로 어떻게 제공해 줄지 54 | 55 | << Learning - 성진님 >> 56 | 1. 자연어 기초 Coursera 57 | 2. 관련 소스 58 | 3. 논문 - seq to seq를 통한 machine translation 59 | 60 | Text Rank Algorithm 61 | 딥러닝을 통해서 62 | 필링크 63 | 64 | 1. Dimensionality Reduction 65 | - eigenVector, eigenValue 66 | - IR 67 | - 차원 축소를 해도 그 의미가 사라지지 않도록 하는 방법론! 68 | - PCA : 주성분 분석 69 | - 다차원에서 유클리디안으로 거리를 구해도 쓸모가 없음. 70 | 71 | 2. Parse Tagging 72 | - Top Down, Bottom Down 73 | - 자연어 처리의 네 가지 74 | 1. 형태소 75 | 2. 구문 76 | 3. 중의성 분석 77 | 4. 화형 분석 78 | ==>> Classical 한 방법임 79 | 이것들이 딥러닝으로 가면 데이터들이 vector 공간에 위치되고 wordEmbedding으로 학습이 되기 떄문에 형태소 분석이 의미가 없어지게됨. 80 | 영어는 중의성 문제까지 한글은 구문에서 81 | 82 | 3. 3강 까지는 몸풀기 같음 83 | 수학관련 내용 자세히 볼 필요가 있음. 84 | 85 | 4. 4, 5강 쯤에 프로그래밍 과제있음 3개정도 잇음 86 | 87 | 미드 스크립트를 가져와서 88 | 페이스북 ChatBot 형태로 89 | 90 | << 창욱님 >> 91 | 웨이블릿 개념 소개 92 | 93 | Word2Vec negative sampling 알아보기! 94 | 95 | << 기호님 >> 96 | 논문 정리 : Seq To Seq 97 | 98 | 언어 처리에서 가장 중요하게 생각하는 요소는 99 | - 좌,우 context 정보 100 | - Frequency 101 | 102 | 자연어 처리는 UnSupervised Learning 103 | 비슷한 문맥에서는 유사한 단어가 많이 나온다. 104 | 105 | Word2Vec에 들어가는 것은 106 | 1. 단어에 대한 주변 단어 107 | 2. Frequency 108 | 3. Window 109 | 110 | << 다음주 >> 111 | 1. 기호님 논문 나머지, 소스 112 | 2. 창욱님 SeqGan 113 | 3. 은숙님 NLTK 소스 + 자연어 처리 Python소스 114 | 115 | 구운몽 자연어 처리 예제 소스 성진님이 소스 전달 예정 116 | -------------------------------------------------------------------------------- /1_Schedule/20161123.md: -------------------------------------------------------------------------------- 1 | << 11/23 Wen >> 2 | 3 | 1. 은숙님 - NLTK 설치 및 간단 함수 소개 4 | http://blog.il-q.com/entry/%ED%8C%8C%EC%9D%B4%EC%8D%AC%EA%B3%BC-%ED%95%A8%EA%BB%98%ED%95%98%EB%8A%94-%EC%9E%90%EC%97%B0%EC%96%B4-%EC%B2%98%EB%A6%AC-NLTK 5 | 6 | 2. 기호님 - 논문 / 소스 7 | 8 | Chris Olah Blog 9 | http://distill.pub/2016/augmented-rnns/ 10 | 11 | http://cs224d.stanford.edu/lecture_notes/LectureNotes5.pdf 12 | 13 | GRU LSTM 14 | http://aikorea.org/blog/rnn-tutorial-4/ 15 | 16 | Peeking into the neural network architecture used for Google's Neural Machine Translation 17 | http://smerity.com/articles/2016/google_nmt_arch.html 18 | 19 | 피아노 - 사람의 음성 단어 20 | 21 | 바이두 - 음성인식 세계 최고 22 | - Deep Speech To 23 | - CTC 24 | 25 | End-To-End Deep Learning 26 | 27 | 데미스 하사비스 28 | http://it.donga.com/23894/ 29 | 30 | Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation 31 | https://arxiv.org/abs/1609.08144 32 | 33 | http://www.bayareadlschool.org/schedule 34 | 35 | http://distill.pub/2016/augmented-rnns/#citation 36 | 37 | hidden layer안에서 일어나는 처리는 알지 못한다. 38 | 요즘 어떤 사람이 안에 처리에 대해서 분석하려는 시도가 있음 39 | 40 | 3. 창욱님 - Google SyntaxNet / 챗봇 어떻게 만드는지? 41 | - 구문 분석에 NLTK를 사용함! 42 | - 자바스크립트 챗봇 소스 다시 분석 예정 43 | 44 | 4. DNC 45 | - 인간도 장기 기억에서 키워드만 뽑아서 재 해석 함. 46 | - 인간의 패턴이 그렇게 다양하지 않음 ex)혈액형 성격 분석 , 점 보는 것. 47 | - 비서가 사람의 감정 상태를 분석해서 친구, 엄마처럼 대화를 시도해줌. 48 | - 사람은 처음이라는 것을 강렬하게 기억함. 49 | -------------------------------------------------------------------------------- /1_Schedule/20161130.md: -------------------------------------------------------------------------------- 1 | << 11/30 Wen >> 2 | 3 | 1. 백병인님 간단 논문 소개 4 | Word Ordering Without Syntax 5 | https://arxiv.org/pdf/1604.08633v2.pdf 6 | 7 | 2. 오연택님 8 | - 부산대 맞춤법 검사기. 우리말 배움터 9 | http://urimal.cs.pusan.ac.kr/urimal_new/ 10 | 코난 테크놀로지 - 코난 검색엔진 11 | http://www.konantech.com/ 12 | - 사전의 상태에 따라 형태소 분석의 성능이 좌우됨. 13 | 사전의 데이터 저장 형태는 로컬의 메모리에 Stack구조로 저장됨.(1GB도 안됨) 14 | 어미/조사 사전 15 | 신조어는 DB형태 16 | 17 | 3줄 요약 18 | https://summariz3.herokuapp.com/ 19 | 20 | 한글판 썸리 21 | http://www.mediatoday.co.kr/?mod=news&act=articleView&idxno=110676 22 | 23 | TextRank 24 | http://www.slideshare.net/andrewkoo/textrank-algorithm 25 | 관련 논문 26 | https://web.eecs.umich.edu/~mihalcea/papers/mihalcea.emnlp04.pdf 27 | 참고-구글의 페이지랭크 28 | http://www.emh.co.kr/content.pl?google_pagerank_citation_ranking 29 | 30 | 한글 및 한국어 정보 학술 학회 31 | https://sites.google.com/site/2016hclt/nonmunbalpyo-iljeong 32 | 33 | << PLAN >> 34 | 35 | 1. CS224d NLP 36 | http://cs224d.stanford.edu/ 37 | https://www.youtube.com/watch?v=l6K-MFgIEjc&index=7&list=PLlJy-eBtNFt4CSVWYqscHDdP58M3zFHIG 38 | -30분 39 | 40 | 1-1. Coursera 보강 41 | + 패스트 캠퍼스 텍스트 마이닝(135만원) 42 | -30분 43 | 44 | 2. 논문 - 2주에 한번 45 | = 챗봇에 관련된 논문 위주 46 | -30분 47 | 48 | 3. 소스코드 - 2주에 한 49 | -30분 50 | 51 | 챗봇에 필요한 기술 52 | - 형태소 분석 53 | 54 | ==> 기초를 먼저 쌓고 서비스 런칭을 위한 기술 시작을 하자! 55 | 56 | - 2월에는 챗봇을 위한 소규모 팀 분업화 57 | - 4월에는 서비스 런칭을 위한 챗봇 개발을 시작하기로 하자! 58 | 59 | 챗봇을 만드려면 결국엔, 하이브리드로 가야됨. 60 | 딥마인드 논문 섭렵. 61 | 62 | 찬우님 수학, 이론 기초 63 | 연택님 부록 64 | 65 | << 다음주 12/7>> 66 | 1. 224d - 성진님 67 | 2. 소스 코드 - 창욱님 68 | 3. Coursera(10분) - 성진님 69 | 4. 구문 분석 / TextRank - 연택님 70 | 71 | << 다다음주 12/14>> 72 | 1. 224d - 은숙님 73 | 2. 이론 / 논문 - 금재님 74 | 3. 수학, 이론LSTM에 소스 기초 / 소스 코드 - 찬우님 75 | 76 | << 12/21 >> 77 | 1. 224d - 연택님 78 | 2. 소스 - 창욱님 79 | 3. Coursera + FastCamp - 성진님 80 | 81 | << 12/28 >> 82 | 1. 224d - 병인님 83 | 2. 이론 / 논문 - 광재님 84 | 3. Coursera + FastCamp - 성진님 85 | -------------------------------------------------------------------------------- /1_Schedule/20161207.md: -------------------------------------------------------------------------------- 1 | << 12/7 Wed >> 2 | 3 | 1. 오연택님 4 | - 한국어 의존문법 개념 발표 5 | - Google에서 공개한 SyntaxNet 과 비슷한 기능을 가지는 개념인 것 같음. 6 | - 형태소 분석 이후 입력 문장에 대해서 의존소 / 지배소로 분류하여 문장의 의미를 파악함. 7 | - 나올 수 있는 문장의 경우의 수 리스트를 추출함. 8 | - Rule Based 9 | - 목적은 주어, 서술어, 목적어를 찾겠다. 10 | - 자료는 '/2_NLP_Study' 11 | 12 | 2. SimpleDS - 전창욱님 13 | 어떻게하면 구축하는지? / 학습방법 14 | - 환경 구축 방법 15 | - 소스 Agent(Server, JS) / Environment(Client, Java) 16 | - 다음시간에 계속. 17 | 18 | 3. 신성진님 19 | https://www.microsoft.com/cognitive-services 20 | IBM 왓슷 : Rule Based , 학습을 시킨다는게 별로 없음. 미국에 있는 한국 개발자가 한글을 개발함. 21 | SK C&C 22 | - 다음주에 벡터연산관련해서 연관어 찾는 개념 공개해주시기로함. 23 | 24 | - 토론 ~~ 25 | Word2Vec은 전처리 과정인가? 네트워크 과정인가? 26 | 1) Word2Vec은 Word를 컴퓨터가 이해할 수 있게 Vector로 수치화 한 이후 코사인 유사도를 이용하여 단어 간의 연관성을 찾아내는 작업. 27 | 2) 단어 간의 연관성을 찾기 위해 RNN or LSTM 모델에 넣어서 학습 시킴. 요즘에는 LSTM Attention Memory 이런 것도 사용함. 28 | (DNC에 Attention / LSTM.. 여러가지가 다들어 있음. => 논문 발표 예정) 29 | == Word Embedding 방법 중에 하나가 word2vec 30 | embedding은 뭐라고 정의할 수 있나? : 수치화 31 | -------------------------------------------------------------------------------- /1_Schedule/20161214.md: -------------------------------------------------------------------------------- 1 | << 12/14 Wed >> 2 | 3 | 1. 전창욱님 - simpleDS 두 번째 발표 4 | - 강화학습을 기본으로 학습시키는 방법론 5 | - 나이브 베이지 방법으로 함. 6 | - 마스터 알고리즘 책 소개, 연결주의 나이브 베이즈 7 | 8 | - 아마존 에코, api.api : 사용자가 질문한 내역을 다시한번 되 물어본다 9 | - 일단 소스는 keep 해 두고 기본기를 익힌 다음 활용할 부분이 생기면 활용하자. 10 | 11 | 2. 신성진님 - Coursera 6~7장 12 | - UX : 수치화 시킨다. : 추후에 공유 예정 13 | 1) Probabilites 14 | - 기본 통계 확률 방법론 15 | 2) Bayes Theorem 16 | 1종오류 : 참이 아닌데 거짓이라 하는거 17 | 2종오류 : 참인데 참이라고 안하는거 18 | 검정력 19 | 20 | 3) Language Modeling 21 | n-gram 22 | HMM : 자세히 뭔지? 다음주에 소개! 23 | 확률이 0일때, 라플라스 모델링을 이용해서 1로 만들어 버림. 모든 경우의 수에 다 더해서 확률값이 나올 수 있게함 24 | 25 | 3. 강은숙 - CS224d Lecture2 Word vector 26 | - Word2Vec 이 나오기 이전에 컴퓨터가 이해 할 수 있게 언어를 모델링 하는 방법들이 존재함. 27 | 1) Word tree 28 | 2) one-hot-Vector 29 | 3) Co-occurence matrix 30 | 4) Low dimension 으로 처음부터 학습 31 | - SVD(특이값 분해) : 차원 축소를 하기 위해서 벡터의 고유값을 추출하여 벡터(단어)간의 관계를 찾아내어 차원을 줄임. 32 | - Main Idea : Word2Vec! 33 | - Word2Vec 으로 가기 전에 논문들이 존재함. 34 | - CBOW 보단 Skip-gram 이 성능이 더 좋음. 35 | - skip-gram 의 object function은 중심단어 w t가 주어졌을때, 그 주변 단어 w t+j가 나올 확률을 log를 취해(Maximum likelihood방법을 위해) 36 | Sum한 후 시간 T까지 Sum 한후 1/T로 나눔 37 | - Softmax function으로 확률을 구함. 38 | 39 | 40 | -- Word2Vec에 대한 기본적인 Source도 살펴 보는 것이 좋겟다. 41 | -------------------------------------------------------------------------------- /1_Schedule/20161221.md: -------------------------------------------------------------------------------- 1 | << 12/21 Wed >> 2 | 3 | [이찬우님] 4 | 자연어 처리를 이전에 어떻게 했는가? 5 | 1. Morkov Model 6 | - p(wt+1 | wt) : 단어가 많아지면 계산 코스트가 높아짐. 7 | Markov 조건, 현재단어를 추론할때 바로 앞단어 만을 바라보고 결정한다. 8 | MC : 앞으로 연쇄적으로 연결하여 추론함 9 | Markov Hidden 10 | Rule 기반 11 | 데이터가 많고, 손보기가 쉬어서 성능이 잘 나옴 12 | 13 | 2. RL 14 | - Markov에서 발전된 모델 15 | - Value function, reward 튜닝 잘해주면 좋음 16 | 17 | 3. DNN 18 | 19 | AutoEncoder, LikeLihood (사전, 사후 확률) 20 | 21 | 4. word2Vec, wordEmbedding 22 | 23 | [이금재님, DNC] 24 | - 자료 공유. 25 | -------------------------------------------------------------------------------- /1_Schedule/20161228.md: -------------------------------------------------------------------------------- 1 | << 12/28 Wed >> 2 | 3 | 1. 추광재님 4 | - 대화 잘의 응답 시스템 5 | 6 | 딥러닝과 정보검색을 결합한 질의 응답 시스템 7 | 챗봇 형태 8 | 9 | 질의 응답 : 1회성 질의. 10 | 1) QA 시스템 : 지도 학습, rule based 11 | 2) 지능형 검색 : 12 | ==> 지식 검색이 이 계통에 속함. 13 | 14 | 기존에는 질문에서 핵심 키워드를 뽑아서 파악 15 | 논문에서는 의미 분석 16 | RDF : Resource Description Framework 17 | 18 | 술부-논항구조 : 19 | 20 | DBpedia : 지식베이스에서 유명 21 | 22 | IBM 왓슷 23 | 24 | 지식베이스 25 | 26 | SPARQL : SQL같이 쿼리 날리는 형태 27 | - 서울 열린데이터 광장도 이 형태로 되어있음. 28 | 29 | 자연언어 질의를 기계가 읽을수 있는 쿼리로 변경하는 방법 30 | 1) 정보추출 : DB에 키워드가 없으면 분석이 안됨. 31 | 2) 의미분석 : DB의 형태에 상관없이 파악하기 위해. 32 | 33 | Frame 매칭 34 | - 버클리팀이 함. 35 | 36 | Deep Learning 으로 어떻게 하지? 37 | 1) Word2Vec 38 | 2) utterance - Encoding 39 | 3) RNN - LSTM ------ Seq-To-Seq 40 | 4) Dialog ------ Seq-To-Seq 41 | -------------------------------------------------------------------------------- /1_Schedule/20170104.md: -------------------------------------------------------------------------------- 1 | << 20170104 / Wed>> 2 | 3 | 1. CS224d Lecture 4 : Word Window Classification and Neural Networks [백병인님] 4 | - 다시 Deep Learning의 기초가 시작됨. 5 | - 끝에 학생들이 한 프로젝트가 보석 6 | - Loss function - Cross Entropy함수에 대해 수학적으로 이해시키려고 함. 7 | 8 | - 지도학습을 수학적으로 풀어보자! 9 | Classification 10 | logistic regression 11 | cross Entropy 12 | logistic으로 바라본 딥러닝, 자연어, 윈도우 13 | 그러나, logistic은 아님 딥러닝을 쓰자! 14 | logistic - 딥러닝 관련있다 15 | 백프로파게이션 다음시간에~ 16 | 오늘은, 딥러닝 에듀반 첫 시간인거 같음 ㅎㅎ 17 | - Cross Entropy에 대한 자세한 설명! 18 | http://blog.acronym.co.kr/433 19 | 20 | linear boundary는 21 | 22 | regularlization : 모델의 수를 떨쳐서 평균화, 오버피팅을 막기 위해 일부러 데이터를 훼손 시키는 것 23 | - L2 regularlization 24 | normalization : 분포를 정규화 25 | 26 | 2. Glove 논문 - 강은숙 27 | - Glove To Vec은 Word2Vec에 비해 Co-Occurence Matrix를 행렬 인수분해 방법을 통해 0이 아닌 요소만 학습 시킴으로써 속도가 빠르다. 28 | - 하지만, 일정 시점이 넘어가는 면에 있어서는 (논문에서 xmax=100으로 설정한 그래프) 100 이상이면 100으로 퉁 설정하기 때문에 29 | 계속해서 Linear하게 빈도수를 체크하는 Word2Vec에 비해 단점이 존재한다. 30 | 31 | 3. Glove Source - 전수진 32 | - Python을 이용해서 build Co-Occurence Matrix를 구축하는 소스 설명 33 | - window Size를 설정해 놓고, Vocabulary Size만큼 Iteration을 돌리면서 34 | Search 하는 word를 기준으로 주변 단어들 간의 distance의 값을 1/distance 의 값을 표에 채운다. 35 | - 남은 소스에 대해서는 다음시간에 계속~ 36 | -------------------------------------------------------------------------------- /1_Schedule/20170111.md: -------------------------------------------------------------------------------- 1 | 1. Lecture5. Project Information + Neural Networks & Backprop (백병인님) 2 | - Backpropagation 얘기 3 | - coursera : 4 | - cs231n : 실용적, 예제, 수학적 의미, 필드에서 사용할 수 있는, 가장 좋은 것 같음 5 | Backproppagation : http://cs231n.github.io/optimization-2/#staged 6 | - cs224d : 코세라보단 불친절 7 | - Lecture 3 Note 보면 도움됨. 8 | http://cs224d.stanford.edu/lecture_notes/notes3.pdf 9 | 10 | 11 | 2. Glove Source - 2 (전수진님) 12 | - AdaGrad : Sparse한 Weight의 업데이트에 좋은 optimizer / SGD에서 Learning Rate를 제거하는 방법 13 | https://xcorr.net/2014/01/23/adagrad-eliminating-learning-rates-in-stochastic-gradient-descent/ 14 | - Learning Rate를 학습시키자는 논문 : https://arxiv.org/abs/1606.04474 15 | 16 | 3. Neural Variational Inference for Text Processing 소스 및 논문 소개 (전창욱님) 17 | - 해당 논문은 베이지안 추론을 활용해서 wiki 백과에 있는 데이터를 기준으로 Q이 input 되었을때, Answer를 추출해 주는 논문 인것 같음. 18 | - 해당 소스는 김태훈님이 자체로 짠 소스로 논문에서 나온 Question을 넣어도 Answer가 안나오는 것으로 보아 분석 안하는게 좋아보임.. 19 | 20 | 4. 김진원님 21 | 분배되는 전력의 패턴을 보고 이것을 모으면 엄청 빅데이터가 되니 22 | 패턴을 분석해서 휴일/평일/밤.. 등등 상업/가정 등등의 이런 것들을 23 | 이상징후를 파악해서 서비스 해주는 것 24 | 25 | 한전이나 발전소를 지으려면 돈이 많이 드니 전기를 효율적으로 사용할 수 있게끔 26 | 전력을 모아놧다 peak 시간대에 사용하게끔 해서 분배해주는 것 27 | -------------------------------------------------------------------------------- /1_Schedule/20170118.md: -------------------------------------------------------------------------------- 1 | << 20170118 / Wed>> 2 | 3 | 1. 신성진님 Various 4 | (1) Bi-Directional 5 | BIDAF : Standford Q&A Dataset(SQuAD) 6 | - paragraph안에서 질문을 던져서 대답을 얻음 7 | - 추론 8 | 학습속도가 좋음. 9 | 10 | 연관검색어 11 | 12 | BI-DIRECTIONAL ATTENTION FLOW FOR MACHINE COMPREHENSION 13 | https://arxiv.org/pdf/1611.01603v3.pdf 14 | 15 | SQuAD 16 | https://rajpurkar.github.io/SQuAD-explorer/ 17 | 18 | 구글 데이터셋 19 | https://research.google.com/research-outreach.html#/research-outreach/research-datasets 20 | 21 | 2. one-shot learning Intro - 이금재님 22 | - small dataset으로 학습 23 | - Like VGG(Learning Model) <- 이미지에 대해서 학습되어 있는데 추가적인 이미지가 있는지의 기능 24 | VGG가 없으면 one-shot이 안됨 25 | - word vector의 논리와 비슷함 26 | - labled되지 않은 새로운 데이터가 입력되었을때 기존의 학습된 모델에서 가장 가까운 값을 결과로 줌. 27 | - 기존 모델을 backpro해서 모델을 변형하거나 그러지 않는 다는 점. 28 | - DNC에서 외부 메모리에 들어있는 컨텐츠 검색에서 입력과 가장 비슷한 데이터의 결과를 얻을 때 사용함 그래서 기존 100개 있다면 101번쨰를 만듦 29 | 30 | - unlabled되어 있는 곳에 넣어 놓고 나중에 labled함 31 | 32 | 3. 텍스트 마이닝 33 | noise canceling 34 | tokenizing 35 | part-of speech tagging (POS : ontology) 36 | filtering : stopwords(등장하지 않는 단어) removal 37 | Term vector representation 38 | Transformation tf into tfidf 39 | Applying algorithm 40 | 41 | - SyntaxNet 한글 지원! 42 | - 구문 분석 : 모호성이 해결됨 43 | - SyntaxNet LSTM보다 100배 빠르다 44 | - 구문 분석 + 어순 파악 45 | 46 | KoNLP - WordToVec처럼 키워드를 뽑는 역할 소스 리뷰 47 | 48 | - 성진님 자료는 개인적으로 보고 공유는 자제! 49 | 50 | 4. CS224d 6강은 강은숙님 51 | Neural Tips and Tricks + Recurrent Neural Networks 52 | - NLL(Negative Log Likelyhood) , Cross Entropy loss function 차이점 이후에 공유!d 53 | -------------------------------------------------------------------------------- /1_Schedule/20170125.md: -------------------------------------------------------------------------------- 1 | << 20170125 / Wed>> 2 | 3 | 1. Machine Network for one shot learning - 이금재님 4 | - CNN이 seq2seq 역할을 함 5 | - CNN에서 사진을 거치면서 사진의 사이즈는 작아진다. 마지막에 벡터로 쭉 나열되고 그 안에서 순서는 상관없음. 여러장을 거치면서 대표 이미지가 나오는것 이것으로 classifier 6 | - 여기에서는 classifier만 띄어낸 상태의 CNN 7 | - 학습과 입력이미지의 CNN은 같은 모델을 사용함. 8 | 9 | - CNN -> KNN 10 | CNN으로 feature(특징)을 뽑아서 언어처리 가능 - sequence 개념이 나옴 11 | 여러장의 이미지 sequence (순서의 개념이 들어간 것) 12 | 13 | - 일반 KNN 14 | : 매트릭 공간에 임베딩되어 두 점의 사이의 거리를 구함. 15 | - 가중 KNN 16 | : weight를 더해줌. 17 | 18 | 각각의 요소들이 패러다임에 묶일 수 있음. 19 | - metric learning : 거리로 학습 = KNN 20 | - 외부 메모리 Augmented 21 | VGG를 이용함 : 외부메모리로 봄. 22 | CNN으로 학습된 모든 메모리는 외부메모리라고 볼 수 잇음 23 | - LSTM 두 가지 쓰임 24 | 일반 25 | BI-DIRECTIONAL : 양방향에서 하기 때문에 순서는 중요하지 않고 26 | 27 | LSTM 을 넣은 이유는? 28 | Reference 논문 : Set to Set 읽고 보면 one shot learning에 대한 이해가 더 잘됨. 29 | 활용 분야 : 필체인식, 언어 작업 30 | 31 | 페이페이리 동영상 : 아이가 기계보다 동물을 더 잘 맞춘다. (아이는 아는 것이 많지 않기 떄문에) 32 | https://www.ted.com/talks/fei_fei_li_how_we_re_teaching_computers_to_understand_pictures?language=ko 33 | 34 | Attention Kernel 35 | - softmax , cosine similarlity 36 | - attention도 학습되는 것 - 가중치를 주기 위함 37 | 38 | 가중 KNN 39 | : Data들이 겹쳐서 분포 될때 이것을 해결하려면 다시 feature부터 뽑고 해야하는데 이것을 피하기 위해서 40 | + Attention 가중치를 줘서 분리해서 이웃 사이의 거리를 확실히 구분되게끔 해서 입력 이미지를 넣었을때 가까운 이미지를 구하기 위함. 41 | one shot learning 에서는 Model을 수정하지 않음. 42 | 43 | 2. 신성진님 44 | 나라면 이런 영어 챗봇을 만들고 싶다. 45 | 브레인 스토밍 해보기 46 | 47 | 3. CS224d - TensorFlow - 김진원님 48 | - Tensorflow의 대한 기본 개념 49 | - 함수 설명 50 | - Initial Variable 함수 변경됨. 51 | - Linear Regreesion 52 | - TensorBoard - API들이 신규로 다 바뀜 53 | -------------------------------------------------------------------------------- /1_Schedule/20170201.md: -------------------------------------------------------------------------------- 1 | << 20170201 / Wed>> 2 | 3 | 1. CS224d RNN - 강은숙 4 | - Vanishing Gradient의 문제점 및 해결방안 5 | - 강의와 더불어서 Lecture note도 같이 보는게 좋음. 6 | 7 | 2. NLU, NLG 개념 설명 - 신성진 8 | 3. FastCam 단어 추출 - 신성진 9 | 10 | << Service 회의 >> 11 | - 영어 교육 챗봇 12 | 1. 한글 처리 지양 13 | 2. character sensitive 14 | 3. 영화의 대본을 학습해서 내가 실제 아이언맨의 자비스와 이야기 나누듯이 15 | 4. 사용자가 대답한 것을 교정해서 교정본을 제공해줌 16 | 5. 딥러닝으로 무언가를 만들어보고 싶다 17 | 6. 하이브리드 딥러닝 = Rule Based + Deep Learning 18 | 7. Seamless 한 서비스 19 | 8. 음성으로 구어체를 문어체로 변경해주는 기술 20 | - voice data 수집 21 | 22 | 카테고리를 정하고 23 | 샘플하고 24 | 앱으로 만드는 것으로 25 | 26 | 27 | 기사 28 | - 영어교육 무료 모바일앱 http://www.bloter.net/archives/251050 29 | - 스마트폰으로 쓰기 좋은 앱 https://chattingcat.com/posts/3apps-for-english 30 | - 영어학습자를 위한 앱 http://www.fluentu.com/english/blog/ko/%EC%98%81%EC%96%B4-%ED%95%99%EC%8A%B5%EC%9E%90-esl-%ED%95%99%EC%83%9D%EB%93%A4-%EC%B5%9C%EA%B3%A0%EC%9D%98-%EC%95%B1/ 31 | - 영어학습 앱 야금야금 http://www.hankyung.com/news/app/newsview.php?aid=2014090336321 32 | - 어린이 교육 http://monthly.appstory.co.kr/apps3984 33 | -------------------------------------------------------------------------------- /1_Schedule/20170316.md: -------------------------------------------------------------------------------- 1 | << 20170316 / Wed>> 2 | 3 | 태성님 - ‘Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees‘ 4 | 5 | https://tensorflow.blog/tag/differentiable-boundary-tree/ 6 | 7 | 1. KNN 8 | 2. Boundary Tree 9 | 3. Differentiable Boundary Trees 10 | 11 | 찬우님 - Word2Vec 12 | 1. 학습 원리 13 | 2. Corpus Data에 따라서 Co-Occurance Matrix가 생성되고 서로 간의 유사도가 측정되는 것임 14 | https://www.tensorflow.org/versions/r0.11/tutorials/word2vec/ 15 | -------------------------------------------------------------------------------- /1_Schedule/20170329.md: -------------------------------------------------------------------------------- 1 | <<20170329 Wed>> 2 | 3 | 번역 4 | 5명 = 300 5 | 구글번역 6 | - IT 단어들이나 매끄럽지 않음 7 | 사람보다 더 번역이 잘되는 도메인을 찾는 것 8 | 9 | 영어를학습 시키고 거기에 결과인 벡터공간을 10 | 프랑스어에 적용시키면 11 | 의미로 매칭을 해서 12 | 13 | 2. 태성님 사업 계획 공유 14 | - 사람이 번역 하는 것을 기계로 번역 해서 더 효율적으로 할 수 있게 하는 것 15 | 16 | 구글 북스 합법으로 통과됨 17 | 18 | 인공지능은 데이터를 어떻게 수집하냐가 관건 인것 같음 19 | 20 | Bi-Directinal RNN 21 | https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/bidirectional_rnn.py 22 | https://github.com/allenai/bi-att-flow 23 | 24 | https://arxiv.org/pdf/1703.01898.pdf 25 | 26 | TensorFlow API Change 27 | https://tensorflow.blog/2016/12/22/tensorflow-api-changes/comment-page-1/ 28 | 29 | TensorFlow Math API 30 | http://visionprogrammer.tistory.com/19 31 | 32 | Word2Vec 33 | https://jybaek.gitbooks.io/ml/content/tensorflow/word2vec.html 34 | 35 | LG CNS ChatBot 36 | http://blog.lgcns.com/1126 37 | 38 | A Convolutional Neural Network for Modelling Sentences - Book 39 | https://books.google.co.kr/books?id=5MaiDQAAQBAJ&pg=PA797&dq=A+Convolutional+Neural+Network+for+Modelling+Sentences&hl=ko&sa=X&redir_esc=y#v=onepage&q=A%20Convolutional%20Neural%20Network%20for%20Modelling%20SentencesA%20Convolutional%20Neural%20Network%20for%20Modelling%20Sentences&f=false 40 | 41 | Google Books 42 | http://www.bloter.net/archives/169856 43 | 44 | Bloter 45 | http://www.bloter.net/archives/275445 46 | 47 | tf.train.SequenceExample 48 | https://www.tensorflow.org/api_docs/python/tf/train/SequenceExample 49 | 50 | tf.parse_single_sequence_example(serialized, context_features=None, sequence_features=None, example_name=None, name=None) 51 | https://www.tensorflow.org/api_docs/python/tf/parse_single_sequence_example 52 | 53 | tf.contrib.learn Quickstart 54 | https://www.tensorflow.org/get_started/tflearn 55 | 56 | tf.train.batch(tensors, batch_size, num_threads=1, capacity=32, enqueue_many=False, shapes=None, dynamic_pad=False, allow_smaller_final_batch=False, shared_name=None, name=None) 57 | https://www.tensorflow.org/api_docs/python/tf/train/batch 58 | 59 | tf.train.range_input_producer(limit, num_epochs=None, shuffle=True, seed=None, capacity=32, shared_name=None, name=None) 60 | https://www.tensorflow.org/api_docs/python/tf/train/range_input_producer 61 | 62 | Input pipeline 63 | https://www.tensorflow.org/versions/r0.11/api_docs/python/io_ops/input_pipeline 64 | 65 | tf.nn.embedding_lookup(params, ids, partition_strategy='mod', name=None, validate_indices=True, max_norm=None) 66 | https://www.tensorflow.org/api_docs/python/tf/nn/embedding_lookup 67 | 68 | tf.nn.dynamic_rnn(cell, inputs, sequence_length=None, initial_state=None, dtype=None, parallel_iterations=None, swap_memory=False, time_major=False, scope=None) 69 | https://www.tensorflow.org/api_docs/python/tf/nn/dynamic_rnn 70 | 71 | tf.sign(x, name=None) 72 | https://www.tensorflow.org/api_docs/python/tf/sign 73 | -------------------------------------------------------------------------------- /1_Schedule/20170405.md: -------------------------------------------------------------------------------- 1 | << 20170405 / Wed >> 2 | 3 | 최신 논문 동향 : SE-GAN (Speech Enhanced GAN) 4 | https://arxiv.org/pdf/1703.09452.pdf 5 | 6 | CS224d의 새로운 강의 CS224n 7 | 강의 동영상 : https://www.youtube.com/watch?v=OQQ-W_63UgQ&list=PL3FW7Lu3i5Jsnh1rnUwq_TcylNr7EkRe6&index=1 8 | 자료 : http://web.stanford.edu/class/cs224n/syllabus.html 9 | 10 | Deep Mind Oxford NLP 11 | 강의 동영상 : https://www.youtube.com/watch?v=RP3tZFcC2e8&list=PL613dYIGMXoZBtZhbyiBqb0QtgK6oJbpm 12 | 자료 : https://github.com/oxford-cs-deepnlp-2017/lectures 13 | https://tensorflow.blog/2017/02/08/dl-for-nlp-by-oxford-deepmind/ 14 | 15 | 오연택님 ChatBot 소스 16 | Seq-To-Seq : https://www.tensorflow.org/tutorials/seq2seq 17 | ChatBot Source Reference : https://github.com/golbin/TensorFlow-Tutorials/tree/master/06%20-%20RNN%2C%20ChatBot/04%20-%20ChatBot 18 | -------------------------------------------------------------------------------- /1_Schedule/20180221.md: -------------------------------------------------------------------------------- 1 | #DeepNLP 20180221 2 | 3 | Preprocessing 부분 4 | 1. Open Data 수집 완료 5 | wiki dataset https://github.com/tscheepers/Wikipedia-Summary-Dataset 6 | 7 | CMU data set http://www.cs.cmu.edu/~dbamman/booksummaries.html 8 | 9 | 2. 학습데이터 중 논문 title을 추가로 수집한다.(EDA요청) 10 | 11 | 12 | EDA 부분 13 | 1. NLTK 사용 . 14 | 15 | 2. 동사 따로 명사 따로 뽑아 본다. - 제목도 데이터로 필요 할거 같다. (Preprocessing 요청) 16 | 17 | 3. 탐색적 데이터 분석의 목적 탐색적 데이터 분석의 목적은 데이터를 이해하는 것이다. 최대한 빨리 높이는 길은 한정된 시간에 가능하면 많은 질문을 생성해 내고, 탐색과정을 거쳐 최대한 빨리 좋은 질문을 생성해 내는 것에 대한 정의 인지. 18 | 19 | 4. Text Rank기법을 이용한 핵심 어구 추출 및 텍스트 요약 링크 참조 : http://bab2min.tistory.com/552 20 | 21 | 5. 다음주 시각화 예정 및 사용한 pandas 설명 22 | 23 | 24 | Modeling 부분 25 | 1. amazon reviews 데이터로 two-layered bidirectional RNN with LSTMs 구현 참조 26 | https://github.com/Currie32/Text-Summarization-with-Amazon-Reviews 27 | 28 | Research 부 29 | 1. Attention Survey: https://arxiv.org/pdf/1601.06823.pdf (~3월말) 30 | - 2.3까지 공통 31 | - Rinhye: Item-wise 32 | - Donghyun: Location-wise 33 | - Hyeseon: Soft+Hard Attention 34 | 35 | 2. Convolutional Seq2Seq Learning: https://arxiv.org/pdf/1705.03122.pdf (3월중순) 36 | 3. Convolutional Seq2Seq Learning source code: https://github.com/facebookresearch/fairseq-py 37 | -------------------------------------------------------------------------------- /1_Schedule/20180307.md: -------------------------------------------------------------------------------- 1 | #DeepNLP 20180307 2 | 3 | Common 4 | 5 | git config 설정 6 |   7 | https://git-scm.com/book/ko/v1/%EC%8B%9C%EC%9E%91%ED%95%98%EA%B8%B0-Git-%EC%B5%9C%EC%B4%88-%EC%84%A4%EC%A0%95 8 | 9 | Research 부분 10 | 11 | seq2seq 발표 자료 (최태균님) 12 | 13 | https://www.dropbox.com/s/rknoxzan9h2608d/Conv_Seq2Seq.pdf?dl=0 14 | 15 | 1. Convolutional seq2seq learning paper understanding 16 |   17 | - learned how Convolution Blocks are working in decode Module 18 | 19 | - understood mechanism of Gated Linear Unit 20 | 21 | - need to study how these modules are specifically implemented 22 |   23 | 24 | Preprocessing 부분 25 | 1. data 변경 -> data_title ( abstract, title ) 26 | 2. stop word 제거 27 | 3. dic 구성 28 | 4. word2id 구성 : Convert all the tokens in the data into 29 | their corresponding index in the vocabulary. 30 | 31 | 32 | EDA 부분 33 | 34 | - EDA 1차 Pandas를 통해서 Q&A data frame 구성 35 | - EDA 2차 View 확인하고 word2vec을 통해서 Embadding 처리하고 t-sne로 시각화 처리 36 | - EDA 3차 Pos 처리하고 명사 동사 갯수 처리 37 | 38 | NLTK 39 | - Pos Tag : http://sens.tistory.com/454 40 |   41 | 해야 할일 42 | 43 |   - Doc2Vec으로 Title, Abstract의 관계 확인 (창욱님) 44 | 45 |   - 가장 적절한 문장 길이를 도출 한다. (현님) 46 | 47 |   - BM25로 빈도수 체크 확인 (지은님) 48 | 49 | 50 | Modeling 부분 51 | 52 |  https://arxiv.org/pdf/1704.04368.pdf  토론 53 | -------------------------------------------------------------------------------- /1_Schedule/20180314.md: -------------------------------------------------------------------------------- 1 | 2 | #DeepNLP 20180314 3 | 4 | Pointer network 논문 5 | 6 | https://arxiv.org/pdf/1704.04368.pdf 7 | 8 | 9 | https://arxiv.org/pdf/1604.00788.pdf 10 | 11 | 12 | https://arxiv.org/pdf/1506.03134.pdf 13 | 14 | 15 | TSP Algorism 검토 16 | 17 | 18 | http://www.qukihub.com/post/traveling-salesperson-problem-tsp 19 | 20 | 21 | 모델링 부분 공유 (open source link) 22 | 23 | 24 | https://github.com/abisee/pointer-generator/blob/master/model.py 25 | 26 | 27 | -------------------------------------------------------------------------------- /1_Schedule/20180321.md: -------------------------------------------------------------------------------- 1 | #DeepNLP 20180322 2 | 3 | Convolutional Seq2Seq Model 4 | Taekyoon Choi, Rinhye Kim, Hyeseon Ko 5 | 6 | Motivation 7 | - The dominant approach in Seq2Seq was RNN-based Model with Attention Mechanism 8 | - But...the problem of RNN-based model is SLOW computation speed because it can’t be parallelized. 9 | 10 | 태균 연구원님과 혜선 연구원님의 열정적인 설명이 들어간 자료를 이미지로 공유해 드립니다. (설명이 추가 되어 있어서 더 도움이 될지 더 어려울지는 모르겠네요 .) 11 | 12 | * 현재 NLP 실무에서 사용 되는 기술 및 상용화에 대해서 깊은 설명 및 토론 있었습니다. 13 | 자료는 공개 할 수 없어서 비공개 합니다. 14 | -------------------------------------------------------------------------------- /2_LabMaterials/README.md: -------------------------------------------------------------------------------- 1 | # Lecture 2 | -------------------------------------------------------------------------------- /2_LabMaterials/kycho_lec/lecture_slides/test.txt: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/2_LabMaterials/kycho_lec/lecture_slides/test.txt -------------------------------------------------------------------------------- /2_LabMaterials/kycho_lec/practice_code/Seq2Seq/s2s.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "code", 5 | "execution_count": 1, 6 | "metadata": {}, 7 | "outputs": [], 8 | "source": [ 9 | "# https://github.com/ematvey/tensorflow-seq2seq-tutorials/blob/master/3-seq2seq-native-new.ipynb\n", 10 | "\n", 11 | "# https://gist.github.com/ilblackdragon/c92066d9d38b236a21d5a7b729a10f12" 12 | ] 13 | }, 14 | { 15 | "cell_type": "code", 16 | "execution_count": null, 17 | "metadata": {}, 18 | "outputs": [], 19 | "source": [] 20 | } 21 | ], 22 | "metadata": { 23 | "kernelspec": { 24 | "display_name": "Python 3", 25 | "language": "python", 26 | "name": "python3" 27 | }, 28 | "language_info": { 29 | "codemirror_mode": { 30 | "name": "ipython", 31 | "version": 3 32 | }, 33 | "file_extension": ".py", 34 | "mimetype": "text/x-python", 35 | "name": "python", 36 | "nbconvert_exporter": "python", 37 | "pygments_lexer": "ipython3", 38 | "version": "3.6.4" 39 | } 40 | }, 41 | "nbformat": 4, 42 | "nbformat_minor": 2 43 | } 44 | -------------------------------------------------------------------------------- /3_Paper/20170104_GloVe_EunSook.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/3_Paper/20170104_GloVe_EunSook.pdf -------------------------------------------------------------------------------- /3_Paper/GloVe- Global Vectors for Word Representation.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/3_Paper/GloVe- Global Vectors for Word Representation.pdf -------------------------------------------------------------------------------- /3_Paper/GloVe-Global Vectors forWord Representation.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# GloVe: Global Vectors forWord Representation" 8 | ] 9 | }, 10 | { 11 | "cell_type": "markdown", 12 | "metadata": {}, 13 | "source": [ 14 | "## Abstract" 15 | ] 16 | }, 17 | { 18 | "cell_type": "markdown", 19 | "metadata": {}, 20 | "source": [ 21 | "The result is a new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods.\n", 22 | "\n", 23 | "규칙성에 필요한 모델 특성을 분석하여 단어 벡터로 나타 내기 위해\n", 24 | "글로벌 행렬 인수 분해 및 로컬 컨텍스트 창 방법을 결합한 새로운 글로벌 로그 선형 회귀 모델을 사용했다.\n", 25 | "\n", 26 | "### What is this?\n", 27 | "\n", 28 | "1. global matrix factorization\n", 29 | "2. local context window methods\n", 30 | "\n", 31 | "\n", 32 | "\n", 33 | "\n", 34 | "\n", 35 | "Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word cooccurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus\n", 36 | "\n", 37 | "우리 모델은 전체 희소 행렬 또는 대형 코퍼스의 개별 컨텍스트 창보다는 단어 - 단어 공분산 행렬에서 0이 아닌 요소에 대해서만 학습함으로써 통계 정보를 효율적으로 활용합니다" 38 | ] 39 | }, 40 | { 41 | "cell_type": "markdown", 42 | "metadata": {}, 43 | "source": [ 44 | "## The GloVe Model\n", 45 | "\n", 46 | "We use our insights to construct a new model for word representation which we call GloVe, for Global Vectors, because the global corpus statistics are captured directly by the model.\n", 47 | "\n", 48 | "전역 통찰력 통계는 모델에 의해 직접 포착되기 때문에 Global Vectors에 대해 GloVe라고 부르는 단어 표현을위한 새로운 모델을 만들기 위해 우리의 통찰력을 사용합니다." 49 | ] 50 | } 51 | ], 52 | "metadata": { 53 | "anaconda-cloud": {}, 54 | "kernelspec": { 55 | "display_name": "Python [conda root]", 56 | "language": "python", 57 | "name": "conda-root-py" 58 | }, 59 | "language_info": { 60 | "codemirror_mode": { 61 | "name": "ipython", 62 | "version": 2 63 | }, 64 | "file_extension": ".py", 65 | "mimetype": "text/x-python", 66 | "name": "python", 67 | "nbconvert_exporter": "python", 68 | "pygments_lexer": "ipython2", 69 | "version": "2.7.12" 70 | } 71 | }, 72 | "nbformat": 4, 73 | "nbformat_minor": 1 74 | } 75 | -------------------------------------------------------------------------------- /3_Paper/Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees-deepmind-nlp2.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/3_Paper/Learning Deep Nearest Neighbor Representations Using Differentiable Boundary Trees-deepmind-nlp2.pdf -------------------------------------------------------------------------------- /3_Paper/README.md: -------------------------------------------------------------------------------- 1 | # Chat_Learning 2 | 3 | 2016/11/9 Wednesday 4 | 5 | 1. Sequence to sequence learning with neural networks (짧음) 6 | http://cs224d.stanford.edu/papers/seq2seq.pdf 7 | 8 | 2. A Neural Conversational Model (짧음) 9 | http://cs224d.stanford.edu/papers/ancm.pdf 10 | 11 | 3. Grammar as a Foreign Language 12 | https://arxiv.org/pdf/1412.7449.pdf 13 | 14 | 4. NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE (나중에 기회될때 발표하겠습니다. 위 논문들이랑 연관되어 있어서 공유합니다.) 15 | https://arxiv.org/pdf/1409.0473.pdf 16 | 17 | 18 | 19 | 20 | 21 | -------------------------------------------------------------------------------- /3_Paper/Sequence-To-Sequence.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/3_Paper/Sequence-To-Sequence.pdf -------------------------------------------------------------------------------- /3_Paper/textrankalgorithm-141001134151-phpapp02.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/3_Paper/textrankalgorithm-141001134151-phpapp02.pdf -------------------------------------------------------------------------------- /4_Source/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/.DS_Store -------------------------------------------------------------------------------- /4_Source/README.md: -------------------------------------------------------------------------------- 1 | # Chat_Learning 2 | 2016년 11월 2일 수요일 3 | - 강화학습 챗봇 (자바 스크립트) https://github.com/cuayahuitl/SimpleDS (전창욱) 4 | - wildml deep learning for chatbots part 1 (blog) http://www.wildml.com/2016/04/deep-learning-for-chatbots-part-1-introduction/ (서기호) 5 | - wildml deep learning for chatbots part 2 (blog) http://www.wildml.com/2016/07/deep-learning-for-chatbots-2-retrieval-based-model-tensorflow/ (서기호) 6 | - wildml retrieved based bots (python) https://github.com/dennybritz/chatbot-retrieval/ (서기호) 7 | - 서울대 딥러닝 강의 Deep Learning on Dialogue Systems (video) https://www.youtube.com/watch?v=HLPfmnQDCzw&index=8&list=PLzWH6Ydh35ggVGbBh48TNs635gv2nxkFI (서기호) 8 | - Python Korea News Jam - for data collection (slide) http://www.slideshare.net/koorukuroo/20160813-pycon2016apac (서기호) 9 | - Github Awesome Datasets (github) https://github.com/caesar0301/awesome-public-datasets#natural-language (서기호) 10 | - Stanford BeautifulSoup Tutorial (python) http://web.stanford.edu/~zlotnick/TextAsData/Web_Scraping_with_Beautiful_Soup.html (서기호) 11 | 12 | 13 | 2017년 1월 4일 수요일 14 | - Glove Source on Python Reference http://www.foldl.me/2014/glove-python/ 15 | - Source https://github.com/hans/glove.py 16 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/.DS_Store -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/interaction/SimpleActions.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/interaction/SimpleActions.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/interaction/SimpleInteractionPolicy.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/interaction/SimpleInteractionPolicy.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/interaction/SimpleUserSimulator.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/interaction/SimpleUserSimulator.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/learning/SimpleAgent$1.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/learning/SimpleAgent$1.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/learning/SimpleAgent.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/learning/SimpleAgent.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/learning/SimpleClassifier.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/learning/SimpleClassifier.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/learning/SimpleEnvironment.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/learning/SimpleEnvironment.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/main/SimpleDS.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/main/SimpleDS.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleServer.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleServer.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleSocketHandler$MessageHandler.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleSocketHandler$MessageHandler.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleSocketHandler.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleSocketHandler.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleSocketServer.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleSocketServer.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleWebServer.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/networking/SimpleWebServer.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/ConfigParser.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/ConfigParser.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/IOUtil.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/IOUtil.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/KeyValuePair.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/KeyValuePair.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/Logger.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/Logger.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/StringUtil.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/StringUtil.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/Vocabulary.class: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/bin/simpleDS/util/Vocabulary.class -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/build.xml: -------------------------------------------------------------------------------- 1 | 2 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/config.txt: -------------------------------------------------------------------------------- 1 | Dialogues=1 2 | Verbose=true 3 | Language=english 4 | SysResponses=resources/%Language%/SysResponses.txt 5 | UsrResponses=resources/%Language%/UsrResponses.txt 6 | SlotValues=resources/%Language%/SlotValues.txt 7 | DemonstrationsPath=data/%Language% 8 | DemonstrationsFile=models/demonstrations.arff 9 | MinimumProbability=0.001 10 | SlotsToConfirm=3 11 | NoiseLevel=0 12 | OutputPath=results 13 | 14 | AddressPort=ws://localhost:8082/simpleds 15 | SavingFrequency=2500 16 | 17 | LearningSteps=20000 18 | ExperienceSize=10000 19 | BurningSteps=100 20 | DiscountFactor=0.7 21 | MinimumEpsilon=0.001 22 | BatchSize=32 23 | 24 | TypedInputSupport=false 25 | AndroidSupport=false 26 | SocketServerPort=2015 27 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/doc/How2UseSimpleDS.txt: -------------------------------------------------------------------------------- 1 | --------------------------------------------- 2 | How to apply SimpleDS to interactive systems? 3 | --------------------------------------------- 4 | 5 | SimpleDS can be applied to interactive system with/without verbal interaction. 6 | It has been applied in an intact way to strategic dialogue management, which 7 | confirms its potential in terms of applicability across domains. This system 8 | runs using a client-server architecture, where the 'server' is your system 9 | and the 'client' is the Deep Reinforcement Learner. The following steps aim 10 | to help you in integrating SimpleDS with your interactive system. 11 | 12 | (1) Instantiate the class "simpleDS/learning/SimpleAgent", which is parame- 13 | terised by a string containing the number of state features and the 14 | number of actions (separated by a comma). This step will treat your 15 | system as a server. See example in simpleDS.main.SimpleDS.java 16 | 17 | (2) Create your own learning environment including states (vector of real 18 | numbers), actions (vector of N integers) and rewards (vector of N integers 19 | or single real number) and communicate them to the client via the method 20 | simpleAgent.sendMessage(). See example in SimpleDS.getSystemAction_key(), 21 | which will return the action choosen by the learning agent. 22 | 23 | (3) Copy the file 'config.txt' and the folder 'web' (and all its contents) 24 | to a directory accessible by your system -- both at the same level. For 25 | example, at the same level of your source directory. 26 | 27 | (4) Edit the file 'config.txt' and change the parameters of your interest. 28 | For example: learning steps, experience size, batch size. The results 29 | after training are stored in the folder results/english/*.* 30 | (N.B. Please make your that such folders exist in your computer) 31 | 32 | (5) Edit the file web/main/runClient.js, where you can modify the agent's 33 | configurations, e.g. network nodes and layers. 34 | 35 | (6) Run the server and client as described in the file README.txt 36 | 37 | N.B. If your system does not use verbal interaction (e.g. word-based features), 38 | you can ignore the language related parameters in the file 'config.txt'. They 39 | roughly correspond to the first 11 parameters in the config file. 40 | 41 | If you have questions or comments please send email to h.cuayahuitl@gmail.com 42 | 43 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/doc/hc-iwsds2016.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/doc/hc-iwsds2016.pdf -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-client-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-client-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-http-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-http-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-io-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-io-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-security-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-security-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-server-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-server-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-servlet-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-servlet-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-servlets-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-servlets-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-util-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-util-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-webapp-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/jetty-webapp-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/jetty/servlet-api-3.0.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/jetty/servlet-api-3.0.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-api-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-api-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-client-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-client-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-common-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-common-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-server-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-server-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-servlet-9.0.5.v20130815.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/websocket/websocket-servlet-9.0.5.v20130815.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/lib/weka.jar: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/lib/weka.jar -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/english/SlotValues.txt: -------------------------------------------------------------------------------- 1 | $food:"italian|indian|mexican|chinese|japanese" 2 | $price:"cheap|reasonably priced|expensive" 3 | $area:"north|south|east|west|centre" 4 | $more:"no" 5 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/english/SysResponses.txt: -------------------------------------------------------------------------------- 1 | Salutation(greeting):"Hello!" 2 | Request(hmihy):"How can I help you?" 3 | Request(food):"What type of food would you like?" 4 | Request(price):"What price range are you looking for?" 5 | Request(area):"What area are you looking for?" 6 | Request(food,price):"What type of food and price range are you looking for?" 7 | Request(food,area):"What type of food and area are you looking for?" 8 | Request(price,area):"What price range and area are you looking for?" 9 | Request(food,price,area):"What type of food, price range, and area are you looking for?" 10 | AskFor(more):"Anything else?" 11 | 12 | Apology(food):"Sorry, I didn't get the food type." 13 | Apology(price):"Sorry, I didn't get the price range." 14 | Apology(area):"Sorry, I didn't get the area." 15 | Apology(food,price):"Sorry, I didn't get the food and price." 16 | Apology(food,area):"Sorry, I didn't get the food and area." 17 | Apology(price,area):"Sorry, I didn't get the price and area." 18 | Apology(food,price,area):"Sorry, I didn't get the food, price and area." 19 | 20 | ExpConfirm(food=$food):"Did you say $food food?" 21 | ExpConfirm(price=$price):"Did you say $price?" 22 | ExpConfirm(area=$area):"Did you say in the $area?" 23 | ExpConfirm(food=$food,price=$price):"Did you say $price $food food?" 24 | ExpConfirm(food=$food,area=$area):"Did you say $food food in the $area?" 25 | ExpConfirm(price=$price,area=$area):"Did you say $price food in the $area?" 26 | ExpConfirm(food=$food,price=$price,area=$area):"Did you say $price $food food in the $area?" 27 | 28 | ImpConfirm(food=$food):"Okay, $food food." 29 | ImpConfirm(price=$price):"Okay, $price." 30 | ImpConfirm(area=$area):"Okay, in the $area." 31 | ImpConfirm(food=$food,price=$price):"Okay, $price $food food." 32 | ImpConfirm(food=$food,area=$area):"Okay, $food food in the $area." 33 | ImpConfirm(price=$price,area=$area):"Okay, $price food in the $area." 34 | ImpConfirm(food=$food,price=$price,area=$area):"Okay, $price $food food in the $area." 35 | 36 | Retrieve(info):"Let me see." 37 | Provide(unknown):"I'm very sorry, I didn't find a match with those preferences." 38 | Provide(known):"Restaurant X is an excellent choice. It is located in Y." 39 | Salutation(closing):"Okay, talk to you soon. Bye!" 40 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/english/UsrResponses.txt: -------------------------------------------------------------------------------- 1 | AskFor(more):Confirm(more=$more) 2 | Request(hmihy):Provide(food=$food,price=$price,area=$area)|Provide(food=$food,price=$price)|Provide(price=$price,area=$area) 3 | Request(food):Provide(food=$food) 4 | Request(price):Provide(price=$price) 5 | Request(area):Provide(area=$area) 6 | Request(food,price):Provide(food=$food,price=$price)|Provide(food=$food)|Provide(price=$price) 7 | Request(food,area):Provide(food=$food,area=$area)|Provide(food=$food)|Provide(area=$area) 8 | Request(price,area):Provide(price=$price,area=$area)|Provide(price=$price)|Provide(area=$area) 9 | Request(food,price,area):Provide(food=$food,price=$price,area=$area) 10 | Apology(food):Provide(food=$food) 11 | Apology(price):Provide(price=$price) 12 | Apology(area):Provide(area=$area) 13 | Apology(food,price):Provide(food=$food,price=$price)|Provide(food=$food)|Provide(price=$price) 14 | Apology(food,area):Provide(food=$food,area=$area)|Provide(food=$food)|Provide(area=$area) 15 | Apology(price,area):Provide(price=$price,area=$area)|Provide(price=$price)|Provide(area=$area) 16 | Apology(food,price,area):Provide(food=$food,price=$price,area=$area)|Provide(food=$food,price=$price)|Provide(price=$price,area=$area) 17 | ExpConfirm(food=$food):Confirm($yesno) 18 | ExpConfirm(price=$price):Confirm($yesno) 19 | ExpConfirm(area=$area):Confirm($yesno) 20 | ExpConfirm(food=$food,price=$price):Confirm($yesno) 21 | ExpConfirm(food=$food,area=$area):Confirm($yesno) 22 | ExpConfirm(price=$price,area=$area):Confirm($yesno) 23 | ExpConfirm(food=$food,price=$price,area=$area):Confirm($yesno) 24 | 25 | Provide(food=$food):"I am desperate for $food food"|"I am looking for $food food"|"I would like $food food"|"$food food"|"$food" 26 | Provide(price=$price):"$price food"|"Something $price"|"$price" 27 | Provide(area=$area):"In the $area"|"Around the $area"|"the $area"|"$area" 28 | Provide(food=$food,price=$price):"I need $price $food food"|"I would like $price $food food"|"$price $food food" 29 | Provide(food=$food,area=$area):"I would like $food food in the $area"|"I would love $food food in the $area"|"$food food in the $area" 30 | Provide(price=$price,area=$area):"I want $price food in the $area"|"I am looking for $price food in the $area"|"$price food in the $area" 31 | Provide(food=$food,price=$price,area=$area):"I am looking for $price $food food in the $area of town"|"We are looking for $price $food food in the $area"|"$price $food food in the $area of town"|"$price $food food in the $area of the city"|"$price $food food in the $area" 32 | Confirm(more=$more):"no" 33 | Confirm(yes):"Yes I did"|"Yes please"|"I did" 34 | Confirm(no):"No I didn't"|"No thanks"|"Nope" 35 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/german/SlotValues.txt: -------------------------------------------------------------------------------- 1 | $food:"italienisches|indisches|mexikanisches|chinesisches|japanisches" 2 | $price:"billig|mit akzeptablem preis|teuer" 3 | $area:"norden|sueden|osten|westen|zentrum" 4 | $more:"nein" 5 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/german/SysResponses.txt: -------------------------------------------------------------------------------- 1 | Salutation(greeting):"Hallo!" 2 | Request(hmihy):"Wie kann ich ihnen helfen?" 3 | Request(food):"Was möchten sie essen?" 4 | Request(price):"An welche Preisrichtung haben sie gedacht?" 5 | Request(area):"In welcher Gegend moechten sie essen?" 6 | Request(food,price):"Nach welcher Essensrichtung und Preisklasse suchen sie?" 7 | Request(food,area):"Nach welcher Essensrichtung und nach welcher Gegend suchen sie?" 8 | Request(price,area):"An welche Preisrichtung und Gegend haben sie gedacht?" 9 | Request(food,price,area):"Nach welcher Essensrichtung, Preisrichtung und Gegend suchen sie?" 10 | AskFor(more):"Sonst noch etwas?" 11 | 12 | Apology(food):"Entschuldigung, ich habe die Essensrichtung nicht verstanden." 13 | Apology(price):"Entschuldigung, ich habe die Preisrichtung nicht verstanden." 14 | Apology(area):"Entschuldigung, ich habe die Gegend nicht verstanden." 15 | Apology(food,price):"Entschuldigung, ich habe die Essensrichtung und Preisrichtung nicht verstanden." 16 | Apology(food,area):"Entschuldigung, ich habe die Essensrichtung und Gegend nicht verstanden." 17 | Apology(price,area):"Entschuldigung, ich habe die Preisrichtung und Gegend nicht verstanden." 18 | Apology(food,price,area):"Entschuldigung, ich habe die Essensrichtung, Preisrichtung und Gegend nicht verstanden." 19 | 20 | ExpConfirm(food=$food):"Haben Sie gesagt, sie moechten $food essen?" 21 | ExpConfirm(price=$price):"Haben Sie gesagt $price?" 22 | ExpConfirm(area=$area):"Haben Sie gesagt sie möchten in der $area essen?" 23 | ExpConfirm(food=$food,price=$price):"Haben Sie gesagt, sie möchten $price $food essen?" 24 | ExpConfirm(food=$food,area=$area):"Haben Sie gesagt, sie möchten $food essen im $area?" 25 | ExpConfirm(price=$price,area=$area):"Haben Sie gesagt, si möchten $price essen im $area?" 26 | ExpConfirm(food=$food,price=$price,area=$area):"Haben Sie gesagt, sie moechten $price $food essen im $area?" 27 | 28 | ImpConfirm(food=$food):"Okay, $food essen." 29 | ImpConfirm(price=$price):"Okay, $price." 30 | ImpConfirm(area=$area):"Okay, in $area." 31 | ImpConfirm(food=$food,price=$price):"Okay, $price $food essen." 32 | ImpConfirm(food=$food,area=$area):"Okay, $food essen im $area." 33 | ImpConfirm(price=$price,area=$area):"Okay, $price essen im $area." 34 | ImpConfirm(food=$food,price=$price,area=$area):"Okay, $price $food essen im $area." 35 | 36 | Retrieve(info):"Lassen sie mich einmal nachsehen." 37 | Provide(unknown):"Es tut mir leid, ich habe keine Ergebnisse für ihre Wünsche gefunden." 38 | Provide(known):"Restaurant X ist eine ausgezeichnete Wahl. Es ist in Y." 39 | Salutation(closing):"Okay, auf Wiedersehen. Tschüss!" 40 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/german/UsrResponses.txt: -------------------------------------------------------------------------------- 1 | AskFor(more):Confirm(more=$more) 2 | Request(hmihy):Provide(food=$food,price=$price,area=$area)|Provide(food=$food,price=$price)|Provide(price=$price,area=$area) 3 | Request(food):Provide(food=$food) 4 | Request(price):Provide(price=$price) 5 | Request(area):Provide(area=$area) 6 | Request(food,price):Provide(food=$food,price=$price)|Provide(food=$food)|Provide(price=$price) 7 | Request(food,area):Provide(food=$food,area=$area)|Provide(food=$food)|Provide(area=$area) 8 | Request(price,area):Provide(price=$price,area=$area)|Provide(price=$price)|Provide(area=$area) 9 | Request(food,price,area):Provide(food=$food,price=$price,area=$area) 10 | Apology(food):Provide(food=$food) 11 | Apology(price):Provide(price=$price) 12 | Apology(area):Provide(area=$area) 13 | Apology(food,price):Provide(food=$food,price=$price)|Provide(food=$food)|Provide(price=$price) 14 | Apology(food,area):Provide(food=$food,area=$area)|Provide(food=$food)|Provide(area=$area) 15 | Apology(price,area):Provide(price=$price,area=$area)|Provide(price=$price)|Provide(area=$area) 16 | Apology(food,price,area):Provide(food=$food,price=$price,area=$area)|Provide(food=$food,price=$price)|Provide(price=$price,area=$area) 17 | ExpConfirm(food=$food):Confirm($yesno) 18 | ExpConfirm(price=$price):Confirm($yesno) 19 | ExpConfirm(area=$area):Confirm($yesno) 20 | ExpConfirm(food=$food,price=$price):Confirm($yesno) 21 | ExpConfirm(food=$food,area=$area):Confirm($yesno) 22 | ExpConfirm(price=$price,area=$area):Confirm($yesno) 23 | ExpConfirm(food=$food,price=$price,area=$area):Confirm($yesno) 24 | 25 | Provide(food=$food):"Ich moechte gerne $food essen"|"Ich suche nach $food essen"|"Ich hätte gern $food essen"|"$food essen"|"$food" 26 | Provide(price=$price):"$price essen"|"Etwas $price"|"$price" 27 | Provide(area=$area):"In $area"|"In der Naehe von $area"|"im $area"|"$area" 28 | Provide(food=$food,price=$price):"Ich brauche $price $food essen"|"Ich haette gerne $price $food essen"|"$price $food essen" 29 | Provide(food=$food,area=$area):"Ich haette gerne $food essen im $area"|"Am liebsten moechte ich $food essen in $area"|"$food essen im $area" 30 | Provide(price=$price,area=$area):"Ich will $price essen im $area"|"Ich suche nach $price essen im $area"|"$price essen im $area" 31 | Provide(food=$food,price=$price,area=$area):"I suche nach $price $food essen in der $area Gegend"|"Wir haetten gerne $price $food essen im $area"|"$price $food essen in der $area Gegend"|"$price $food essen in der $area Gegend"|"$price $food essen im $area" 32 | Confirm(more=$more):"Nein" 33 | Confirm(yes):"Ja genau"|"Ja bitte"|"Das stimmt" 34 | Confirm(no):"Nein habe ich nicht"|"Nein danke"|"Noe" 35 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/spanish/SlotValues.txt: -------------------------------------------------------------------------------- 1 | $food:"italiana|hindu|mexicana|china|japonesa" 2 | $price:"barato|razonable|costoso" 3 | $area:"norte|sur|este|oeste|centro" 4 | $more:"no" 5 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/spanish/SysResponses.txt: -------------------------------------------------------------------------------- 1 | Salutation(greeting):"Hola!" 2 | Request(hmihy):"En que te puedo ayudar?" 3 | Request(food):"Que tipo de comida te gustaría?" 4 | Request(price):"Que rango de precio estas buscando?" 5 | Request(area):"Que area de la ciudad te gustaría?" 6 | Request(food,price):"Que tipo de comida y rango de precio?" 7 | Request(food,area):"Que tipo de comida y area estas buscando?" 8 | Request(price,area):"Que rango de precio y area estas buscando?" 9 | Request(food,price,area):"Que tipo de comida, rango de precio y area estas buscando?" 10 | AskFor(more):"Algo más?" 11 | 12 | Apology(food):"Disculpa, que tipo de comida?" 13 | Apology(price):"Disculpa, en que rango de precio?" 14 | Apology(area):"Disculpa, en que area?" 15 | Apology(food,price):"Disculpa, que tipo de comida y rango de precio?" 16 | Apology(food,area):"Disculpa, que tipo de comida y area? 17 | Apology(price,area):"Disculpa, que rango de precio y area?" 18 | Apology(food,price,area):"Disculpa, que tipo de comida, rango de precio y area?" 19 | 20 | ExpConfirm(food=$food):"Dijiste comida $food?" 21 | ExpConfirm(price=$price):"Dijiste de precio $price?" 22 | ExpConfirm(area=$area):"Dijiste en el area $area?" 23 | ExpConfirm(food=$food,price=$price):"Dijiste comida $food de precio $price?" 24 | ExpConfirm(food=$food,area=$area):"Dijiste comida $food en el $area?" 25 | ExpConfirm(price=$price,area=$area):"Dijiste de precio $price en el $area?" 26 | ExpConfirm(food=$food,price=$price,area=$area):"Dijiste comida $food de precio $price en el $area?" 27 | 28 | ImpConfirm(food=$food):"Bien, comida $food." 29 | ImpConfirm(price=$price):"Bien, de precio $price." 30 | ImpConfirm(area=$area):"Bien, en el $area." 31 | ImpConfirm(food=$food,price=$price):"Bien, comida $food de precio $price." 32 | ImpConfirm(food=$food,area=$area):"Bien, comida $food en el $area." 33 | ImpConfirm(price=$price,area=$area):"Bien, de precio $price en el $area." 34 | ImpConfirm(food=$food,price=$price,area=$area):"Bien, comida $food de precio $price en el $area." 35 | 36 | Retrieve(info):"Dejame ver." 37 | Provide(unknown):"Disculpame, no encontré ningún restaurante con esas preferencias." 38 | Provide(known):"El restaurante X es una excelente opción. Esta localizado en Y." 39 | Salutation(closing):"Muy bien. Hasta la próxima!" 40 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/resources/spanish/UsrResponses.txt: -------------------------------------------------------------------------------- 1 | AskFor(more):Confirm(more=$more) 2 | Request(hmihy):Provide(food=$food,price=$price,area=$area)|Provide(food=$food,price=$price)|Provide(price=$price,area=$area) 3 | Request(food):Provide(food=$food) 4 | Request(price):Provide(price=$price) 5 | Request(area):Provide(area=$area) 6 | Request(food,price):Provide(food=$food,price=$price)|Provide(food=$food)|Provide(price=$price) 7 | Request(food,area):Provide(food=$food,area=$area)|Provide(food=$food)|Provide(area=$area) 8 | Request(price,area):Provide(price=$price,area=$area)|Provide(price=$price)|Provide(area=$area) 9 | Request(food,price,area):Provide(food=$food,price=$price,area=$area) 10 | Apology(food):Provide(food=$food) 11 | Apology(price):Provide(price=$price) 12 | Apology(area):Provide(area=$area) 13 | Apology(food,price):Provide(food=$food,price=$price)|Provide(food=$food)|Provide(price=$price) 14 | Apology(food,area):Provide(food=$food,area=$area)|Provide(food=$food)|Provide(area=$area) 15 | Apology(price,area):Provide(price=$price,area=$area)|Provide(price=$price)|Provide(area=$area) 16 | Apology(food,price,area):Provide(food=$food,price=$price,area=$area)|Provide(food=$food,price=$price)|Provide(price=$price,area=$area) 17 | ExpConfirm(food=$food):Confirm($yesno) 18 | ExpConfirm(price=$price):Confirm($yesno) 19 | ExpConfirm(area=$area):Confirm($yesno) 20 | ExpConfirm(food=$food,price=$price):Confirm($yesno) 21 | ExpConfirm(food=$food,area=$area):Confirm($yesno) 22 | ExpConfirm(price=$price,area=$area):Confirm($yesno) 23 | ExpConfirm(food=$food,price=$price,area=$area):Confirm($yesno) 24 | 25 | Provide(food=$food):"Estoy desesperado por comida $food"|"Estoy buscando comida $food"|"Me gustaría comer comida $food"|"Comida $food"|"$food" 26 | Provide(price=$price):"De precio $price"|"Algo $price"|"$price" 27 | Provide(area=$area):"En el $area"|"Alrededor del $area"|"El $area"|"$area" 28 | Provide(food=$food,price=$price):"Quiero comida $food de precio $price"|"Me gustaría comer comida $food de precio $price"|"Comida $food de precio $price" 29 | Provide(food=$food,area=$area):"Me gustaría comer comida $food en el $area"|"Me encantaría comida $food en el $area"|"Comida $food en el $area" 30 | Provide(price=$price,area=$area):"Quiero comida de precio $price en el $area"|"Busco comida de precio $price en el $area"|"Comida de precio $price en el $area" 31 | Provide(food=$food,price=$price,area=$area):"Busco comida $food de precio $price en el $area de la ciudad"|"Estoy buscando comida $food de precio $price en el $area de la ciudad"|"Estamos buscando comida $food de precio $price en el $area"|"Comida $food de precio $price en el $area de la ciudad"|"Comida $food de precio $price en el $area" 32 | Confirm(more=$more):"no" 33 | Confirm(yes):"Si"|"Si por favor"|"Asi es" 34 | Confirm(no):"No"|"No gracias"|"Nunca" 35 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/results/english/simpleds-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/results/english/simpleds-output.png -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/results/german/simpleds-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/results/german/simpleds-output.png -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/results/spanish/simpleds-output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/results/spanish/simpleds-output.png -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/screenshots/Screenshot-SimpleDS.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/SimpleDS-master_by_junchangwook/screenshots/Screenshot-SimpleDS.png -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/scripts/plotdata.m: -------------------------------------------------------------------------------- 1 | #################################################################################### 2 | # This script plots learning curves of SimpleDS agents 3 | # Heriberto Cuayahuitl 4 | #################################################################################### 5 | 6 | arg_list = argv (); 7 | if (nargin < 1) 8 | printf ("Usage: sampleplot.m InputDataFile.out [OutputFile.png]"); 9 | exit 10 | else 11 | inputfile = arg_list{1}; 12 | endif 13 | 14 | fid=fopen(inputfile); 15 | line=0; 16 | data=zeros(3,6); 17 | line_i=1; 18 | while (-1 ~= (line=fgetl(fid))) 19 | data(line_i++,:)=str2num(line); 20 | end 21 | fclose(fid); 22 | 23 | x = data(:,1); 24 | y = data(:,2); 25 | yy = data(:,6); 26 | [h,h1,h2] = plotyy(x, y, x, yy); 27 | set(h, "fontsize", 12, "linewidth", 2.0); 28 | set(h1, "LineStyle", "-", "linewidth", 2.0); 29 | set(h2, "LineStyle", "--", "linewidth", 2.0); 30 | grid on; 31 | xlabel('Learning Steps (no. experiences)'); 32 | ylabel(h(1), 'Average Reward'); 33 | ylabel(h(2), 'Learning Time (in hours)'); 34 | 35 | pause 36 | 37 | if (nargin == 2) 38 | outputfile = arg_list{2}; 39 | saveas(1, outputfile); 40 | endif 41 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/scripts/run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | #################################################################################### 4 | # This script runs the SimpleDS system for training dialogue agents 5 | # Heriberto Cuayahuitl 6 | #################################################################################### 7 | 8 | stty cols 120 9 | 10 | if [ $1 = "train" ] ; then 11 | xterm -geometry 120x50 -T "SimpleDS.Server" -hold -e "ant SimpleDS" & 12 | sleep 7 13 | cd web/main 14 | xterm -geometry 80x20 -T "SimpleDS.Client" -hold -e "nodejs runclient.js train" 15 | #node runclient.js train 16 | cd ../../ 17 | 18 | elif [ $1 = "test" ] ; then 19 | xterm -geometry 120x50 -T "SimpleDS.Client" -hold -e "ant SimpleDS" & 20 | sleep 7 21 | cd web/main 22 | xterm -geometry 80x20 -T "SimpleDS.Client" -hold -e "nodejs runclient.js test" 23 | #node runclient.js test 24 | cd ../../ 25 | 26 | else 27 | echo "usage: run.sh (train | test)" 28 | fi 29 | 30 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/interaction/SimpleUserSimulator.java: -------------------------------------------------------------------------------- 1 | /* @description This class implements a rule-based user simulator for training SimpleDS agents. 2 | * It randomises over alternative choices of user responses given the last system action. 3 | * 4 | * @history 2.Nov.2015 Beta version 5 | * 6 | * @author Heriberto Cuayahuitl 7 | */ 8 | 9 | package simpleDS.interaction; 10 | 11 | import java.util.ArrayList; 12 | import java.util.HashMap; 13 | 14 | import simpleDS.util.IOUtil; 15 | import simpleDS.util.Logger; 16 | import simpleDS.util.StringUtil; 17 | 18 | public class SimpleUserSimulator { 19 | private HashMap actions; 20 | private HashMap slots; 21 | private HashMap usrGoal; 22 | private boolean verbose = false; 23 | 24 | public SimpleUserSimulator(HashMap configurations) { 25 | verbose = configurations.get("Verbose").equals("true") ? true : false; 26 | String usrResponsesFile = configurations.get("UsrResponses"); 27 | String slotsFile = configurations.get("SlotValues"); 28 | loadUserResponses(usrResponsesFile, slotsFile); 29 | } 30 | 31 | private void loadUserResponses(String usrResponsesFile, String slotsFile) { 32 | actions = new HashMap(); 33 | IOUtil.readHashMap(usrResponsesFile, actions, ":"); 34 | IOUtil.printHashMap(actions, "USER ACTIONS"); 35 | 36 | slots = new HashMap(); 37 | IOUtil.readHashMap(slotsFile, slots, ":"); 38 | IOUtil.printHashMap(slots, "SLOTS"); 39 | } 40 | 41 | public void resetUserGoal() { 42 | usrGoal = new HashMap(); 43 | for (String slot : slots.keySet()) { 44 | String values = slots.get(slot); 45 | values = values.replace("\"", " "); 46 | values = values.trim(); 47 | ArrayList list = StringUtil.getArrayListFromString(values, "|"); 48 | int randomIndex = (int) Math.floor(Math.random()*list.size()); 49 | String randomValue = list.get(randomIndex); 50 | usrGoal.put(slot, randomValue); 51 | } 52 | 53 | /*if (verbose) { 54 | Logger.debug("SimpleUserSimulator", "resetUserGoal", "usrGoal="+usrGoal); 55 | }*/ 56 | } 57 | 58 | public String getAction(String action_sys_key, String action_sys_val) { 59 | String action = actions.get(action_sys_key); 60 | 61 | if (action == null) { 62 | return ""; // silence 63 | 64 | } else if (action.equals("Confirm($yesno)")) { 65 | action = validConfirmation(action_sys_val) ? "Confirm(yes)" : "Confirm(no)"; 66 | return action; 67 | 68 | } else { 69 | String options = actions.get(action_sys_key); 70 | String option = StringUtil.getRandomisedTemplate(options); 71 | return option; 72 | } 73 | } 74 | 75 | public boolean validConfirmation(String dialAct) { 76 | String pairs = dialAct.substring(dialAct.indexOf("(")+1, dialAct.indexOf(")")); 77 | ArrayList list = StringUtil.getArrayListFromString(pairs, ","); 78 | 79 | for (String pair : list) { 80 | String slot = pair.substring(0, pair.indexOf("=")); 81 | String value_sys = pair.substring(pair.indexOf("=")+1); 82 | String value_usr = usrGoal.get("$"+slot); 83 | if (value_usr == null || !value_usr.equals(value_sys)) { 84 | return false; 85 | } 86 | } 87 | 88 | return true; 89 | } 90 | 91 | public String getAction_Unfolded(String action_usr) { 92 | return StringUtil.getExpandedDialAct(action_usr, usrGoal); 93 | } 94 | 95 | public String getResponse(String action_usr) { 96 | String rand_action_usr = StringUtil.getRandomisedTemplate(action_usr); 97 | String templates = actions.get(rand_action_usr); 98 | String template = StringUtil.getRandomisedTemplate(templates); 99 | String response = StringUtil.getExpandedTemplate(template, usrGoal); 100 | 101 | return response; 102 | } 103 | 104 | public void updateUserGoal(String response_asr) { 105 | if (response_asr == null || response_asr.equals("")) return; 106 | 107 | for (String slot : slots.keySet()) { 108 | String slotValues = slots.get(slot); 109 | 110 | for (String value : StringUtil.getArrayListFromString(slotValues, "|\"")) { 111 | 112 | if (isSlotValue_Valid(value, response_asr)) { 113 | usrGoal.put(slot, value); 114 | } 115 | } 116 | } 117 | } 118 | 119 | public boolean isSlotValue_Valid(String value, String response_asr) { 120 | for (String token : StringUtil.getArrayListFromString(value, " ")) { 121 | if (response_asr.indexOf(token) == -1) { 122 | return false; 123 | } 124 | } 125 | 126 | return true; 127 | } 128 | 129 | public HashMap getUsrGoal() { 130 | return usrGoal; 131 | } 132 | } 133 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/learning/SimpleAgent.java: -------------------------------------------------------------------------------- 1 | /* @description This class implements a WebServer for communicating with ConvNetJS. 2 | * 3 | * @history 2.Nov.2015 Beta version 4 | * 5 | * @author Heriberto Cuayahuitl 6 | */ 7 | 8 | package simpleDS.learning; 9 | 10 | import java.util.ArrayList; 11 | 12 | import simpleDS.networking.SimpleWebServer; 13 | import simpleDS.networking.SimpleSocketHandler; 14 | import simpleDS.util.Logger; 15 | import simpleDS.util.StringUtil; 16 | 17 | public class SimpleAgent extends Thread { 18 | private SimpleSocketHandler handler = null; 19 | private SimpleWebServer deepServer = null; 20 | private String lastDeepAction = null; 21 | private String numInputOutputs = null; 22 | public String executionMode = null; 23 | public String dialogues = null; 24 | public String verbose = null; 25 | public boolean connected = false; 26 | 27 | public SimpleAgent(String numInputOutputs) { 28 | this.numInputOutputs = numInputOutputs; 29 | 30 | try { 31 | Thread thread = new Thread("SERVER"); 32 | thread.start(); 33 | 34 | synchronized(thread) { 35 | try{ 36 | Logger.debug(this.getClass().getName(), "SimpleAgent", "Waiting for deepServer to complete..."); 37 | thread.wait(); 38 | 39 | } catch(InterruptedException e) { 40 | e.printStackTrace(); 41 | } 42 | } 43 | 44 | } catch (Exception e) { 45 | e.printStackTrace(); 46 | } 47 | } 48 | 49 | public void run() { 50 | synchronized(this){ 51 | Logger.debug(this.getClass().getName(), "run", "Trying to connect..."); 52 | handler = new SimpleSocketHandler(); 53 | handler.addMessageHandler(new SimpleSocketHandler.MessageHandler() { 54 | 55 | public void handleMessage(String action) { 56 | if (action.startsWith("Hello Server")) { 57 | connected = true; 58 | action = null; 59 | sendMessage(numInputOutputs); 60 | 61 | } else if (action.startsWith("params")) { 62 | ArrayList list = StringUtil.getArrayListFromString(action, "=,"); 63 | for (int i=0; iHeriberto Cuayahuitl 7 | */ 8 | 9 | package simpleDS.learning; 10 | 11 | import java.util.ArrayList; 12 | import java.util.HashMap; 13 | 14 | import simpleDS.util.Logger; 15 | import weka.classifiers.bayes.NaiveBayes; 16 | import weka.core.Attribute; 17 | import weka.core.Instance; 18 | import weka.core.Instances; 19 | import weka.core.converters.ConverterUtils.DataSource; 20 | 21 | public class SimpleClassifier { 22 | private Instances instances; 23 | private NaiveBayes classifier; 24 | private double minimumProbability; 25 | 26 | public SimpleClassifier(String trainingFile, String minimumProbability) { 27 | this.minimumProbability = Double.parseDouble(minimumProbability); 28 | loadData(trainingFile); 29 | trainModel(); 30 | } 31 | 32 | public void loadData(String trainingFile) { 33 | try { 34 | Logger.debug(this.getClass().getName(), "loadData", "Reading "+trainingFile); 35 | instances = DataSource.read(trainingFile); 36 | instances.setClassIndex(instances.numAttributes() - 1); 37 | Logger.debug(this.getClass().getName(), "loadData", "Data loaded!"); 38 | 39 | } catch (Exception e) { 40 | e.printStackTrace(); 41 | } 42 | } 43 | 44 | public void trainModel() { 45 | try { 46 | classifier = new NaiveBayes(); 47 | classifier.buildClassifier(instances); 48 | Logger.debug(this.getClass().getName(), "trainModel", "Model created!"); 49 | 50 | } catch (Exception e) { 51 | e.printStackTrace(); 52 | } 53 | } 54 | 55 | public Instance createInstance(HashMap evidence) { 56 | Instance firstInstance = (Instance) instances.firstInstance().copy(); 57 | 58 | for (int i=0; i evidence, String label, ArrayList actions) { 75 | String output = ""; 76 | 77 | try { 78 | Instance firstInstance = createInstance(evidence); 79 | double[] dist = classifier.distributionForInstance(firstInstance); 80 | 81 | for (int i=0; i evidence, String label) { 99 | double bestScore = 0; 100 | String bestLabel = ""; 101 | 102 | try { 103 | Instance firstInstance = createInstance(evidence); 104 | Attribute attribute = firstInstance.attribute(instances.numAttributes()-1); 105 | double[] dist = classifier.distributionForInstance(firstInstance); 106 | 107 | for (int i=0; ibestScore) { 111 | bestScore = dist[i]; 112 | bestLabel = attValue; 113 | } 114 | } 115 | 116 | } catch (Exception e) { 117 | e.printStackTrace(); 118 | System.exit(0); 119 | } 120 | 121 | return bestLabel; 122 | } 123 | 124 | public String getMostProbableActions(HashMap evidence, String label) { 125 | String output = ""; 126 | 127 | try { 128 | Instance firstInstance = createInstance(evidence); 129 | Attribute attribute = firstInstance.attribute(instances.numAttributes()-1); 130 | double[] dist = classifier.distributionForInstance(firstInstance); 131 | 132 | for (int i=0; ithis.minimumProbability) { 135 | output += (output.equals("")) ? attValue : ","+attValue; 136 | } 137 | } 138 | 139 | } catch (Exception e) { 140 | e.printStackTrace(); 141 | System.exit(0); 142 | } 143 | 144 | return output; 145 | } 146 | } -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/networking/SimpleServer.java: -------------------------------------------------------------------------------- 1 | /* @description This creates a WebServer object for client-server communication. 2 | * 3 | * @history 2.Nov.2015 Beta version 4 | * 5 | * @author Heriberto Cuayahuitl 6 | */ 7 | package simpleDS.networking; 8 | 9 | import org.eclipse.jetty.server.Server; 10 | 11 | public class SimpleServer implements Runnable { 12 | private SimpleSocketHandler handler; 13 | 14 | public SimpleServer(SimpleSocketHandler handler) { 15 | this.handler = handler; 16 | run(); 17 | } 18 | 19 | public void run() { 20 | try { 21 | Server server = new Server(8082); 22 | server.setHandler(handler); 23 | server.start(); 24 | 25 | } catch (Exception e) { 26 | e.printStackTrace(); 27 | } 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/networking/SimpleSocketHandler.java: -------------------------------------------------------------------------------- 1 | /* @description This class implements a WebSocket handler for client-server communication. 2 | * 3 | * @history 2.Nov.2015 Beta version 4 | * 5 | * @author Heriberto Cuayahuitl 6 | */ 7 | 8 | package simpleDS.networking; 9 | 10 | import org.eclipse.jetty.websocket.api.Session; 11 | import org.eclipse.jetty.websocket.api.annotations.OnWebSocketClose; 12 | import org.eclipse.jetty.websocket.api.annotations.OnWebSocketConnect; 13 | import org.eclipse.jetty.websocket.api.annotations.OnWebSocketError; 14 | import org.eclipse.jetty.websocket.api.annotations.OnWebSocketMessage; 15 | import org.eclipse.jetty.websocket.api.annotations.WebSocket; 16 | import org.eclipse.jetty.websocket.server.WebSocketHandler; 17 | import org.eclipse.jetty.websocket.servlet.WebSocketServletFactory; 18 | 19 | import simpleDS.util.Logger; 20 | 21 | @WebSocket 22 | public class SimpleSocketHandler extends WebSocketHandler { 23 | private static Session session = null; 24 | private static MessageHandler messageHandler; 25 | public boolean connected = false; 26 | 27 | @OnWebSocketClose 28 | public void onClose(int statusCode, String reason) { 29 | Logger.debug(this.getClass().getName(), "onClose", "statusCode=" + statusCode + ", reason=" + reason); 30 | } 31 | 32 | @OnWebSocketError 33 | public void onError(Throwable t) { 34 | Logger.error(this.getClass().getName(), "onError", "Error: " + t.getMessage()); 35 | } 36 | 37 | @OnWebSocketConnect 38 | public void onConnect(Session session) { 39 | Logger.debug(this.getClass().getName(), "onConnect", "Connect: " + session.getRemoteAddress().getAddress()); 40 | try { 41 | this.session = session; 42 | 43 | } catch (Exception e) { 44 | e.printStackTrace(); 45 | } 46 | } 47 | 48 | @SuppressWarnings("static-access") 49 | @OnWebSocketMessage 50 | public void onMessage(String msg) { 51 | try { 52 | if (this.messageHandler != null) { 53 | this.messageHandler.handleMessage(msg); 54 | } 55 | 56 | } catch (Exception e) { 57 | e.printStackTrace(); 58 | } 59 | } 60 | 61 | @SuppressWarnings("static-access") 62 | public void sendMessage(String msg) { 63 | try { 64 | if (this.session != null) { 65 | session.getRemote().sendString(msg); 66 | 67 | } else { 68 | Logger.debug(this.getClass().getName(), "sendMessage", "Undelivered message ... session is NULL"); 69 | } 70 | } catch (Exception e) { 71 | e.printStackTrace(); 72 | } 73 | } 74 | 75 | @Override 76 | public void configure(WebSocketServletFactory factory) { 77 | factory.register(this.getClass()); 78 | } 79 | 80 | @SuppressWarnings("static-access") 81 | public void addMessageHandler(MessageHandler msgHandler) { 82 | this.messageHandler = msgHandler; 83 | } 84 | 85 | public static interface MessageHandler { 86 | public void handleMessage(String features); 87 | } 88 | } -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/networking/SimpleSocketServer.java: -------------------------------------------------------------------------------- 1 | /* @description This class creates a SocketServer object for client-server communication. 2 | * It has been used to integrate with the Google Speech Recogniser for 3 | * testing realistic human-machine speech-based interactions. 4 | * 5 | * @history 5.Nov.2015 Beta version 6 | * 7 | * @author Heriberto Cuayahuitl 8 | */ 9 | 10 | package simpleDS.networking; 11 | 12 | import java.net.*; 13 | import java.io.*; 14 | 15 | import simpleDS.util.Logger; 16 | 17 | public class SimpleSocketServer { 18 | private ServerSocket serverSocket; 19 | private Socket clientSocket; 20 | private DataOutputStream writer; 21 | private DataInputStream reader; 22 | private int port; 23 | 24 | public SimpleSocketServer(String port) { 25 | this.port = Integer.parseInt(port); 26 | } 27 | 28 | public void createServer() { 29 | try { 30 | serverSocket = new ServerSocket(port); 31 | String helpMsg = "Please click 'Connect' from your Android App!"; 32 | Logger.debug(this.getClass().getName(), "createServer", "Listening on port: " + port + " ... " + helpMsg); 33 | clientSocket = serverSocket.accept(); 34 | 35 | } catch(Exception e) { 36 | Logger.debug(this.getClass().getName(), "createServer", "Couldn't listen on port: " + port); 37 | e.printStackTrace(); 38 | } 39 | } 40 | 41 | public String listen() { 42 | try { 43 | reader = new DataInputStream(clientSocket.getInputStream()); 44 | return reader.readUTF(); 45 | 46 | } catch(Exception e) { 47 | Logger.debug(this.getClass().getName(), "listen", "Connection lost"); 48 | } 49 | 50 | return "null"; 51 | } 52 | 53 | public void send(String msg) { 54 | try { 55 | if (clientSocket.isConnected() && msg != null && !msg.equals("")) { 56 | writer = new DataOutputStream(clientSocket.getOutputStream()); 57 | writer.writeUTF(msg); 58 | } 59 | 60 | } catch(Exception e) { 61 | Logger.debug(this.getClass().getName(), "send", "Connection lost"); 62 | } 63 | } 64 | 65 | public boolean isClientConnected() { 66 | return (clientSocket.isConnected()) ? true : false; 67 | } 68 | 69 | public void acceptClient() { 70 | try { 71 | clientSocket.close(); 72 | clientSocket = serverSocket.accept(); 73 | 74 | } catch(Exception e) { 75 | Logger.debug(this.getClass().getName(), "acceptClient()", "Couldn't accept client on port: " + port); 76 | e.printStackTrace(); 77 | } 78 | } 79 | 80 | public void closeServer() { 81 | try { 82 | serverSocket.close(); 83 | clientSocket.close(); 84 | 85 | } catch(Exception e) { 86 | Logger.debug(this.getClass().getName(), "closeServer", "Couldn't close on port: " + port); 87 | e.printStackTrace(); 88 | } 89 | } 90 | } 91 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/networking/SimpleWebServer.java: -------------------------------------------------------------------------------- 1 | /* @description This creates a WebServer object for client-server communication. 2 | * 3 | * @history 2.Nov.2015 Beta version 4 | * 5 | * @author Heriberto Cuayahuitl 6 | */ 7 | 8 | package simpleDS.networking; 9 | 10 | import org.eclipse.jetty.server.Server; 11 | 12 | public class SimpleWebServer implements Runnable { 13 | private SimpleSocketHandler handler; 14 | 15 | public SimpleWebServer(SimpleSocketHandler handler) { 16 | this.handler = handler; 17 | run(); 18 | } 19 | 20 | public void run() { 21 | try { 22 | Server server = new Server(8082); 23 | server.setHandler(handler); 24 | server.start(); 25 | 26 | } catch (Exception e) { 27 | e.printStackTrace(); 28 | } 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/util/KeyValuePair.java: -------------------------------------------------------------------------------- 1 | /* @description This class is used to instantiate objects as key-value pairs. 2 | * 3 | * @history 2.Nov.2015 Beta version 4 | * 5 | * @author Heriberto Cuayahuitl 6 | */ 7 | 8 | 9 | package simpleDS.util; 10 | 11 | 12 | public class KeyValuePair { 13 | private String key; 14 | private String value; 15 | 16 | public KeyValuePair(String key, String value) { 17 | this.key = key; 18 | this.value = value; 19 | } 20 | 21 | public String getKey() { 22 | return key; 23 | } 24 | 25 | public String getValue() { 26 | return value; 27 | } 28 | 29 | public String get() { 30 | return key + ":" + value; 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/util/Logger.java: -------------------------------------------------------------------------------- 1 | /* @description This class implements reusable methods for logging information. 2 | * 3 | * @history 2.Nov.2015 Beta version 4 | * 5 | * @author Heriberto Cuayahuitl 6 | */ 7 | 8 | package simpleDS.util; 9 | 10 | 11 | public class Logger { 12 | 13 | public static void debug(String className, String method, String message) { 14 | System.out.println("[DEB] " + className + "." + method + "(): " + message); 15 | } 16 | 17 | public static void info(String className, String method, String message) { 18 | System.out.println("[INF] " + className + "." + method + "(): " + message); 19 | } 20 | 21 | public static void warning(String className, String method, String message) { 22 | System.out.println("[WAR] ------------------------------------------------------------------"); 23 | System.out.println("[WAR] " + className + "." + method + "(): " + message); 24 | System.out.println("[WAR] ------------------------------------------------------------------"); 25 | } 26 | 27 | public static void error(String className, String method, String message) { 28 | System.out.println("[ERR] ******************************************************************"); 29 | System.out.println("[ERR] " + className + "." + method + "(): " + message); 30 | System.out.println("[ERR] ******************************************************************"); 31 | System.exit(0); 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/src/simpleDS/util/Vocabulary.java: -------------------------------------------------------------------------------- 1 | /* @description This class extracts and maintains a lexicon from system+user template responses. 2 | * 3 | * @history 2.Nov.2015 Beta version 4 | * 5 | * @author Heriberto Cuayahuitl 6 | */ 7 | 8 | package simpleDS.util; 9 | 10 | import java.util.ArrayList; 11 | import java.util.Collections; 12 | import java.util.HashMap; 13 | 14 | public class Vocabulary { 15 | private ArrayList words = new ArrayList(); 16 | 17 | public Vocabulary(String sysResponsesFile, String usrResponsesFile, String slotsFile) { 18 | extractWordSequences(sysResponsesFile, ":"); 19 | extractWordSequences(usrResponsesFile, ":"); 20 | extractWordSequences(slotsFile, ":"); 21 | Collections.sort(words); 22 | IOUtil.printArrayList(words, "VOCABULARY"); 23 | } 24 | 25 | private void extractWordSequences(String fileToRead, String separator) { 26 | HashMap map = new HashMap(); 27 | IOUtil.readHashMap(fileToRead, map, separator); 28 | 29 | for (String key : map.keySet()) { 30 | String sequence = map.get(key); 31 | 32 | if (sequence.indexOf("(")>0 && sequence.indexOf(")")>0) { 33 | continue; 34 | 35 | } else { 36 | for (String item : extractWordsFromSequence(sequence)) { 37 | if (!words.contains(item)) { 38 | words.add(item); 39 | } 40 | } 41 | } 42 | } 43 | } 44 | 45 | public ArrayList extractWordsFromSequence(String sequence) { 46 | ArrayList output = new ArrayList(); 47 | 48 | if (sequence == null) return output; 49 | else sequence = sequence.toLowerCase(); 50 | 51 | ArrayList list = StringUtil.getArrayListFromString(sequence, " \".,?!|"); 52 | for (String item : list) { 53 | if (!output.contains(item) && !item.startsWith("$")) { 54 | output.add(item); 55 | } 56 | } 57 | 58 | return output; 59 | } 60 | 61 | public ArrayList getWordList() { 62 | return words; 63 | } 64 | 65 | public int getVocabularySize() { 66 | return words.size(); 67 | } 68 | } -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/.svn/entries: -------------------------------------------------------------------------------- 1 | 10 2 | 3 | dir 4 | 353 5 | svn://settlers.inf.ed.ac.uk:9880/STACSettlers/web 6 | svn://settlers.inf.ed.ac.uk:9880/STACSettlers 7 | 8 | 9 | 10 | 2012-11-02T13:07:40.963528Z 11 | 2 12 | settlers 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 75838ac5-1bb7-4d88-bd01-9f19308f4a4f 28 | 29 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/convnet/util.js: -------------------------------------------------------------------------------- 1 | 2 | // contains various utility functions 3 | var cnnutil = (function(exports){ 4 | 5 | // a window stores _size_ number of values 6 | // and returns averages. Useful for keeping running 7 | // track of validation or training accuracy during SGD 8 | var Window = function(size, minsize) { 9 | this.v = []; 10 | this.size = typeof(size)==='undefined' ? 100 : size; 11 | this.minsize = typeof(minsize)==='undefined' ? 20 : minsize; 12 | this.sum = 0; 13 | } 14 | Window.prototype = { 15 | add: function(x) { 16 | this.v.push(x); 17 | this.sum += x; 18 | if(this.v.length>this.size) { 19 | var xold = this.v.shift(); 20 | this.sum -= xold; 21 | } 22 | }, 23 | get_average: function() { 24 | if(this.v.length < this.minsize) return -1; 25 | else return this.sum/this.v.length; 26 | }, 27 | reset: function(x) { 28 | this.v = []; 29 | this.sum = 0; 30 | } 31 | } 32 | 33 | // returns min, max and indeces of an array 34 | var maxmin = function(w) { 35 | if(w.length === 0) { return {}; } // ... ;s 36 | 37 | var maxv = w[0]; 38 | var minv = w[0]; 39 | var maxi = 0; 40 | var mini = 0; 41 | for(var i=1;i maxv) { maxv = w[i]; maxi = i; } 43 | if(w[i] < minv) { minv = w[i]; mini = i; } 44 | } 45 | return {maxi: maxi, maxv: maxv, mini: mini, minv: minv, dv:maxv-minv}; 46 | } 47 | 48 | // returns string representation of float 49 | // but truncated to length of d digits 50 | var f2t = function(x, d) { 51 | if(typeof(d)==='undefined') { var d = 5; } 52 | var dd = 1.0 * Math.pow(10, d); 53 | return '' + Math.floor(x*dd)/dd; 54 | } 55 | 56 | exports = exports || {}; 57 | exports.Window = Window; 58 | exports.maxmin = maxmin; 59 | exports.f2t = f2t; 60 | return exports; 61 | 62 | })(typeof module != 'undefined' && module.exports); // add exports to module.exports if in node.js 63 | 64 | 65 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/.npmignore: -------------------------------------------------------------------------------- 1 | npm-debug.log 2 | node_modules 3 | .*.swp 4 | .lock-* 5 | build 6 | 7 | bench 8 | doc 9 | examples 10 | test 11 | 12 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/.travis.yml: -------------------------------------------------------------------------------- 1 | language: node_js 2 | sudo: false 3 | npm_args: --ws:native 4 | node_js: 5 | - "iojs-v3" 6 | - "iojs-v2" 7 | - "iojs-v1" 8 | - "0.12" 9 | - "0.11" 10 | - "0.10" 11 | - "0.9" 12 | - "0.8" 13 | addons: 14 | apt: 15 | sources: 16 | - ubuntu-toolchain-r-test 17 | packages: 18 | - gcc-4.9 19 | - g++-4.9 20 | before_install: 21 | - export CC="gcc-4.9" CXX="g++-4.9" 22 | - "if [[ $(node --version) == v0.8.* ]]; then npm install -g npm@2.1.18; fi" 23 | matrix: 24 | fast_finish: true 25 | allow_failures: 26 | - node_js: "0.11" 27 | - node_js: "0.9" 28 | - node_js: "0.8" 29 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/Makefile: -------------------------------------------------------------------------------- 1 | ALL_TESTS = $(shell find test/ -name '*.test.js') 2 | ALL_INTEGRATION = $(shell find test/ -name '*.integration.js') 3 | 4 | all: 5 | node-gyp configure build 6 | 7 | clean: 8 | node-gyp clean 9 | 10 | run-tests: 11 | @./node_modules/.bin/mocha \ 12 | -t 5000 \ 13 | -s 2400 \ 14 | $(TESTFLAGS) \ 15 | $(TESTS) 16 | 17 | run-integrationtests: 18 | @./node_modules/.bin/mocha \ 19 | -t 5000 \ 20 | -s 6000 \ 21 | $(TESTFLAGS) \ 22 | $(TESTS) 23 | 24 | test: 25 | @$(MAKE) NODE_TLS_REJECT_UNAUTHORIZED=0 NODE_PATH=lib TESTS="$(ALL_TESTS)" run-tests 26 | 27 | integrationtest: 28 | @$(MAKE) NODE_TLS_REJECT_UNAUTHORIZED=0 NODE_PATH=lib TESTS="$(ALL_INTEGRATION)" run-integrationtests 29 | 30 | benchmark: 31 | @node bench/sender.benchmark.js 32 | @node bench/parser.benchmark.js 33 | 34 | autobahn: 35 | @NODE_PATH=lib node test/autobahn.js 36 | 37 | autobahn-server: 38 | @NODE_PATH=lib node test/autobahn-server.js 39 | 40 | .PHONY: test 41 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/index.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | /*! 4 | * ws: a node.js websocket client 5 | * Copyright(c) 2011 Einar Otto Stangvik 6 | * MIT Licensed 7 | */ 8 | 9 | var WS = module.exports = require('./lib/WebSocket'); 10 | 11 | WS.Server = require('./lib/WebSocketServer'); 12 | WS.Sender = require('./lib/Sender'); 13 | WS.Receiver = require('./lib/Receiver'); 14 | 15 | /** 16 | * Create a new WebSocket server. 17 | * 18 | * @param {Object} options Server options 19 | * @param {Function} fn Optional connection listener. 20 | * @returns {WS.Server} 21 | * @api public 22 | */ 23 | WS.createServer = function createServer(options, fn) { 24 | var server = new WS.Server(options); 25 | 26 | if (typeof fn === 'function') { 27 | server.on('connection', fn); 28 | } 29 | 30 | return server; 31 | }; 32 | 33 | /** 34 | * Create a new WebSocket connection. 35 | * 36 | * @param {String} address The URL/address we need to connect to. 37 | * @param {Function} fn Open listener. 38 | * @returns {WS} 39 | * @api public 40 | */ 41 | WS.connect = WS.createConnection = function connect(address, fn) { 42 | var client = new WS(address); 43 | 44 | if (typeof fn === 'function') { 45 | client.on('open', fn); 46 | } 47 | 48 | return client; 49 | }; 50 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/BufferPool.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * ws: a node.js websocket client 3 | * Copyright(c) 2011 Einar Otto Stangvik 4 | * MIT Licensed 5 | */ 6 | 7 | var util = require('util'); 8 | 9 | function BufferPool(initialSize, growStrategy, shrinkStrategy) { 10 | if (this instanceof BufferPool === false) { 11 | throw new TypeError("Classes can't be function-called"); 12 | } 13 | 14 | if (typeof initialSize === 'function') { 15 | shrinkStrategy = growStrategy; 16 | growStrategy = initialSize; 17 | initialSize = 0; 18 | } 19 | else if (typeof initialSize === 'undefined') { 20 | initialSize = 0; 21 | } 22 | this._growStrategy = (growStrategy || function(db, size) { 23 | return db.used + size; 24 | }).bind(null, this); 25 | this._shrinkStrategy = (shrinkStrategy || function(db) { 26 | return initialSize; 27 | }).bind(null, this); 28 | this._buffer = initialSize ? new Buffer(initialSize) : null; 29 | this._offset = 0; 30 | this._used = 0; 31 | this._changeFactor = 0; 32 | this.__defineGetter__('size', function(){ 33 | return this._buffer == null ? 0 : this._buffer.length; 34 | }); 35 | this.__defineGetter__('used', function(){ 36 | return this._used; 37 | }); 38 | } 39 | 40 | BufferPool.prototype.get = function(length) { 41 | if (this._buffer == null || this._offset + length > this._buffer.length) { 42 | var newBuf = new Buffer(this._growStrategy(length)); 43 | this._buffer = newBuf; 44 | this._offset = 0; 45 | } 46 | this._used += length; 47 | var buf = this._buffer.slice(this._offset, this._offset + length); 48 | this._offset += length; 49 | return buf; 50 | } 51 | 52 | BufferPool.prototype.reset = function(forceNewBuffer) { 53 | var len = this._shrinkStrategy(); 54 | if (len < this.size) this._changeFactor -= 1; 55 | if (forceNewBuffer || this._changeFactor < -2) { 56 | this._changeFactor = 0; 57 | this._buffer = len ? new Buffer(len) : null; 58 | } 59 | this._offset = 0; 60 | this._used = 0; 61 | } 62 | 63 | module.exports = BufferPool; 64 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/BufferUtil.fallback.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * ws: a node.js websocket client 3 | * Copyright(c) 2011 Einar Otto Stangvik 4 | * MIT Licensed 5 | */ 6 | 7 | module.exports.BufferUtil = { 8 | merge: function(mergedBuffer, buffers) { 9 | var offset = 0; 10 | for (var i = 0, l = buffers.length; i < l; ++i) { 11 | var buf = buffers[i]; 12 | buf.copy(mergedBuffer, offset); 13 | offset += buf.length; 14 | } 15 | }, 16 | mask: function(source, mask, output, offset, length) { 17 | var maskNum = mask.readUInt32LE(0, true); 18 | var i = 0; 19 | for (; i < length - 3; i += 4) { 20 | var num = maskNum ^ source.readUInt32LE(i, true); 21 | if (num < 0) num = 4294967296 + num; 22 | output.writeUInt32LE(num, offset + i, true); 23 | } 24 | switch (length % 4) { 25 | case 3: output[offset + i + 2] = source[i + 2] ^ mask[2]; 26 | case 2: output[offset + i + 1] = source[i + 1] ^ mask[1]; 27 | case 1: output[offset + i] = source[i] ^ mask[0]; 28 | case 0:; 29 | } 30 | }, 31 | unmask: function(data, mask) { 32 | var maskNum = mask.readUInt32LE(0, true); 33 | var length = data.length; 34 | var i = 0; 35 | for (; i < length - 3; i += 4) { 36 | var num = maskNum ^ data.readUInt32LE(i, true); 37 | if (num < 0) num = 4294967296 + num; 38 | data.writeUInt32LE(num, i, true); 39 | } 40 | switch (length % 4) { 41 | case 3: data[i + 2] = data[i + 2] ^ mask[2]; 42 | case 2: data[i + 1] = data[i + 1] ^ mask[1]; 43 | case 1: data[i] = data[i] ^ mask[0]; 44 | case 0:; 45 | } 46 | } 47 | } 48 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/BufferUtil.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | /*! 4 | * ws: a node.js websocket client 5 | * Copyright(c) 2011 Einar Otto Stangvik 6 | * MIT Licensed 7 | */ 8 | 9 | try { 10 | module.exports = require('bufferutil'); 11 | } catch (e) { 12 | module.exports = require('./BufferUtil.fallback'); 13 | } 14 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/ErrorCodes.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * ws: a node.js websocket client 3 | * Copyright(c) 2011 Einar Otto Stangvik 4 | * MIT Licensed 5 | */ 6 | 7 | module.exports = { 8 | isValidErrorCode: function(code) { 9 | return (code >= 1000 && code <= 1011 && code != 1004 && code != 1005 && code != 1006) || 10 | (code >= 3000 && code <= 4999); 11 | }, 12 | 1000: 'normal', 13 | 1001: 'going away', 14 | 1002: 'protocol error', 15 | 1003: 'unsupported data', 16 | 1004: 'reserved', 17 | 1005: 'reserved for extensions', 18 | 1006: 'reserved for extensions', 19 | 1007: 'inconsistent or invalid data', 20 | 1008: 'policy violation', 21 | 1009: 'message too big', 22 | 1010: 'extension handshake missing', 23 | 1011: 'an unexpected condition prevented the request from being fulfilled', 24 | }; -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/Extensions.js: -------------------------------------------------------------------------------- 1 | 2 | var util = require('util'); 3 | 4 | /** 5 | * Module exports. 6 | */ 7 | 8 | exports.parse = parse; 9 | exports.format = format; 10 | 11 | /** 12 | * Parse extensions header value 13 | */ 14 | 15 | function parse(value) { 16 | value = value || ''; 17 | 18 | var extensions = {}; 19 | 20 | value.split(',').forEach(function(v) { 21 | var params = v.split(';'); 22 | var token = params.shift().trim(); 23 | var paramsList = extensions[token] = extensions[token] || []; 24 | var parsedParams = {}; 25 | 26 | params.forEach(function(param) { 27 | var parts = param.trim().split('='); 28 | var key = parts[0]; 29 | var value = parts[1]; 30 | if (typeof value === 'undefined') { 31 | value = true; 32 | } else { 33 | // unquote value 34 | if (value[0] === '"') { 35 | value = value.slice(1); 36 | } 37 | if (value[value.length - 1] === '"') { 38 | value = value.slice(0, value.length - 1); 39 | } 40 | } 41 | (parsedParams[key] = parsedParams[key] || []).push(value); 42 | }); 43 | 44 | paramsList.push(parsedParams); 45 | }); 46 | 47 | return extensions; 48 | } 49 | 50 | /** 51 | * Format extensions header value 52 | */ 53 | 54 | function format(value) { 55 | return Object.keys(value).map(function(token) { 56 | var paramsList = value[token]; 57 | if (!util.isArray(paramsList)) { 58 | paramsList = [paramsList]; 59 | } 60 | return paramsList.map(function(params) { 61 | return [token].concat(Object.keys(params).map(function(k) { 62 | var p = params[k]; 63 | if (!util.isArray(p)) p = [p]; 64 | return p.map(function(v) { 65 | return v === true ? k : k + '=' + v; 66 | }).join('; '); 67 | })).join('; '); 68 | }).join(', '); 69 | }).join(', '); 70 | } 71 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/Sender.hixie.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * ws: a node.js websocket client 3 | * Copyright(c) 2011 Einar Otto Stangvik 4 | * MIT Licensed 5 | */ 6 | 7 | var events = require('events') 8 | , util = require('util') 9 | , EventEmitter = events.EventEmitter; 10 | 11 | /** 12 | * Hixie Sender implementation 13 | */ 14 | 15 | function Sender(socket) { 16 | if (this instanceof Sender === false) { 17 | throw new TypeError("Classes can't be function-called"); 18 | } 19 | 20 | events.EventEmitter.call(this); 21 | 22 | this.socket = socket; 23 | this.continuationFrame = false; 24 | this.isClosed = false; 25 | } 26 | 27 | module.exports = Sender; 28 | 29 | /** 30 | * Inherits from EventEmitter. 31 | */ 32 | 33 | util.inherits(Sender, events.EventEmitter); 34 | 35 | /** 36 | * Frames and writes data. 37 | * 38 | * @api public 39 | */ 40 | 41 | Sender.prototype.send = function(data, options, cb) { 42 | if (this.isClosed) return; 43 | 44 | var isString = typeof data == 'string' 45 | , length = isString ? Buffer.byteLength(data) : data.length 46 | , lengthbytes = (length > 127) ? 2 : 1 // assume less than 2**14 bytes 47 | , writeStartMarker = this.continuationFrame == false 48 | , writeEndMarker = !options || !(typeof options.fin != 'undefined' && !options.fin) 49 | , buffer = new Buffer((writeStartMarker ? ((options && options.binary) ? (1 + lengthbytes) : 1) : 0) + length + ((writeEndMarker && !(options && options.binary)) ? 1 : 0)) 50 | , offset = writeStartMarker ? 1 : 0; 51 | 52 | if (writeStartMarker) { 53 | if (options && options.binary) { 54 | buffer.write('\x80', 'binary'); 55 | // assume length less than 2**14 bytes 56 | if (lengthbytes > 1) 57 | buffer.write(String.fromCharCode(128+length/128), offset++, 'binary'); 58 | buffer.write(String.fromCharCode(length&0x7f), offset++, 'binary'); 59 | } else 60 | buffer.write('\x00', 'binary'); 61 | } 62 | 63 | if (isString) buffer.write(data, offset, 'utf8'); 64 | else data.copy(buffer, offset, 0); 65 | 66 | if (writeEndMarker) { 67 | if (options && options.binary) { 68 | // sending binary, not writing end marker 69 | } else 70 | buffer.write('\xff', offset + length, 'binary'); 71 | this.continuationFrame = false; 72 | } 73 | else this.continuationFrame = true; 74 | 75 | try { 76 | this.socket.write(buffer, 'binary', cb); 77 | } catch (e) { 78 | this.error(e.toString()); 79 | } 80 | }; 81 | 82 | /** 83 | * Sends a close instruction to the remote party. 84 | * 85 | * @api public 86 | */ 87 | 88 | Sender.prototype.close = function(code, data, mask, cb) { 89 | if (this.isClosed) return; 90 | this.isClosed = true; 91 | try { 92 | if (this.continuationFrame) this.socket.write(new Buffer([0xff], 'binary')); 93 | this.socket.write(new Buffer([0xff, 0x00]), 'binary', cb); 94 | } catch (e) { 95 | this.error(e.toString()); 96 | } 97 | }; 98 | 99 | /** 100 | * Sends a ping message to the remote party. Not available for hixie. 101 | * 102 | * @api public 103 | */ 104 | 105 | Sender.prototype.ping = function(data, options) {}; 106 | 107 | /** 108 | * Sends a pong message to the remote party. Not available for hixie. 109 | * 110 | * @api public 111 | */ 112 | 113 | Sender.prototype.pong = function(data, options) {}; 114 | 115 | /** 116 | * Handles an error 117 | * 118 | * @api private 119 | */ 120 | 121 | Sender.prototype.error = function (reason) { 122 | this.emit('error', reason); 123 | return this; 124 | }; 125 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/Validation.fallback.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * ws: a node.js websocket client 3 | * Copyright(c) 2011 Einar Otto Stangvik 4 | * MIT Licensed 5 | */ 6 | 7 | module.exports.Validation = { 8 | isValidUTF8: function(buffer) { 9 | return true; 10 | } 11 | }; 12 | 13 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/Validation.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | /*! 4 | * ws: a node.js websocket client 5 | * Copyright(c) 2011 Einar Otto Stangvik 6 | * MIT Licensed 7 | */ 8 | 9 | try { 10 | module.exports = require('utf-8-validate'); 11 | } catch (e) { 12 | module.exports = require('./Validation.fallback'); 13 | } 14 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/lib/browser.js: -------------------------------------------------------------------------------- 1 | 2 | /** 3 | * Module dependencies. 4 | */ 5 | 6 | var global = (function() { return this; })(); 7 | 8 | /** 9 | * WebSocket constructor. 10 | */ 11 | 12 | var WebSocket = global.WebSocket || global.MozWebSocket; 13 | 14 | /** 15 | * Module exports. 16 | */ 17 | 18 | module.exports = WebSocket ? ws : null; 19 | 20 | /** 21 | * WebSocket constructor. 22 | * 23 | * The third `opts` options object gets ignored in web browsers, since it's 24 | * non-standard, and throws a TypeError if passed to the constructor. 25 | * See: https://github.com/einaros/ws/issues/227 26 | * 27 | * @param {String} uri 28 | * @param {Array} protocols (optional) 29 | * @param {Object) opts (optional) 30 | * @api public 31 | */ 32 | 33 | function ws(uri, protocols, opts) { 34 | var instance; 35 | if (protocols) { 36 | instance = new WebSocket(uri, protocols); 37 | } else { 38 | instance = new WebSocket(uri); 39 | } 40 | return instance; 41 | } 42 | 43 | if (WebSocket) ws.prototype = WebSocket.prototype; 44 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/options/.npmignore: -------------------------------------------------------------------------------- 1 | npm-debug.log 2 | node_modules 3 | .*.swp 4 | .lock-* 5 | build/ 6 | 7 | test 8 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/options/Makefile: -------------------------------------------------------------------------------- 1 | ALL_TESTS = $(shell find test/ -name '*.test.js') 2 | 3 | run-tests: 4 | @./node_modules/.bin/mocha \ 5 | -t 2000 \ 6 | $(TESTFLAGS) \ 7 | $(TESTS) 8 | 9 | test: 10 | @$(MAKE) NODE_PATH=lib TESTS="$(ALL_TESTS)" run-tests 11 | 12 | .PHONY: test 13 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/options/README.md: -------------------------------------------------------------------------------- 1 | # options.js # 2 | 3 | A very light-weight in-code option parsers for node.js. 4 | 5 | ## Usage ## 6 | 7 | ``` js 8 | var Options = require("options"); 9 | 10 | // Create an Options object 11 | function foo(options) { 12 | var default_options = { 13 | foo : "bar" 14 | }; 15 | 16 | // Create an option object with default value 17 | var opts = new Options(default_options); 18 | 19 | // Merge options 20 | opts = opts.merge(options); 21 | 22 | // Reset to default value 23 | opts.reset(); 24 | 25 | // Copy selected attributes out 26 | var seled_att = opts.copy("foo"); 27 | 28 | // Read json options from a file. 29 | opts.read("options.file"); // Sync 30 | opts.read("options.file", function(err){ // Async 31 | if(err){ // If error occurs 32 | console.log("File error."); 33 | }else{ 34 | // No error 35 | } 36 | }); 37 | 38 | // Attributes defined or not 39 | opts.isDefinedAndNonNull("foobar"); 40 | opts.isDefined("foobar"); 41 | } 42 | 43 | ``` 44 | 45 | 46 | ## License ## 47 | 48 | (The MIT License) 49 | 50 | Copyright (c) 2012 Einar Otto Stangvik <einaros@gmail.com> 51 | 52 | Permission is hereby granted, free of charge, to any person obtaining 53 | a copy of this software and associated documentation files (the 54 | 'Software'), to deal in the Software without restriction, including 55 | without limitation the rights to use, copy, modify, merge, publish, 56 | distribute, sublicense, and/or sell copies of the Software, and to 57 | permit persons to whom the Software is furnished to do so, subject to 58 | the following conditions: 59 | 60 | The above copyright notice and this permission notice shall be 61 | included in all copies or substantial portions of the Software. 62 | 63 | THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, 64 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 65 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 66 | IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY 67 | CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 68 | TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 69 | SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 70 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/options/lib/options.js: -------------------------------------------------------------------------------- 1 | /*! 2 | * Copyright(c) 2011 Einar Otto Stangvik 3 | * MIT Licensed 4 | */ 5 | 6 | var fs = require('fs'); 7 | 8 | function Options(defaults) { 9 | var internalValues = {}; 10 | var values = this.value = {}; 11 | Object.keys(defaults).forEach(function(key) { 12 | internalValues[key] = defaults[key]; 13 | Object.defineProperty(values, key, { 14 | get: function() { return internalValues[key]; }, 15 | configurable: false, 16 | enumerable: true 17 | }); 18 | }); 19 | this.reset = function() { 20 | Object.keys(defaults).forEach(function(key) { 21 | internalValues[key] = defaults[key]; 22 | }); 23 | return this; 24 | }; 25 | this.merge = function(options, required) { 26 | options = options || {}; 27 | if (Object.prototype.toString.call(required) === '[object Array]') { 28 | var missing = []; 29 | for (var i = 0, l = required.length; i < l; ++i) { 30 | var key = required[i]; 31 | if (!(key in options)) { 32 | missing.push(key); 33 | } 34 | } 35 | if (missing.length > 0) { 36 | if (missing.length > 1) { 37 | throw new Error('options ' + 38 | missing.slice(0, missing.length - 1).join(', ') + ' and ' + 39 | missing[missing.length - 1] + ' must be defined'); 40 | } 41 | else throw new Error('option ' + missing[0] + ' must be defined'); 42 | } 43 | } 44 | Object.keys(options).forEach(function(key) { 45 | if (key in internalValues) { 46 | internalValues[key] = options[key]; 47 | } 48 | }); 49 | return this; 50 | }; 51 | this.copy = function(keys) { 52 | var obj = {}; 53 | Object.keys(defaults).forEach(function(key) { 54 | if (keys.indexOf(key) !== -1) { 55 | obj[key] = values[key]; 56 | } 57 | }); 58 | return obj; 59 | }; 60 | this.read = function(filename, cb) { 61 | if (typeof cb == 'function') { 62 | var self = this; 63 | fs.readFile(filename, function(error, data) { 64 | if (error) return cb(error); 65 | var conf = JSON.parse(data); 66 | self.merge(conf); 67 | cb(); 68 | }); 69 | } 70 | else { 71 | var conf = JSON.parse(fs.readFileSync(filename)); 72 | this.merge(conf); 73 | } 74 | return this; 75 | }; 76 | this.isDefined = function(key) { 77 | return typeof values[key] != 'undefined'; 78 | }; 79 | this.isDefinedAndNonNull = function(key) { 80 | return typeof values[key] != 'undefined' && values[key] !== null; 81 | }; 82 | Object.freeze(values); 83 | Object.freeze(this); 84 | } 85 | 86 | module.exports = Options; 87 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/options/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "author": { 3 | "name": "Einar Otto Stangvik", 4 | "email": "einaros@gmail.com", 5 | "url": "http://2x.io" 6 | }, 7 | "name": "options", 8 | "description": "A very light-weight in-code option parsers for node.js.", 9 | "version": "0.0.6", 10 | "repository": { 11 | "type": "git", 12 | "url": "git://github.com/einaros/options.js.git" 13 | }, 14 | "main": "lib/options", 15 | "scripts": { 16 | "test": "make test" 17 | }, 18 | "engines": { 19 | "node": ">=0.4.0" 20 | }, 21 | "dependencies": {}, 22 | "devDependencies": { 23 | "mocha": "latest" 24 | }, 25 | "gitHead": "ff53d0a092c897cb95964232a96fe17da65c11af", 26 | "bugs": { 27 | "url": "https://github.com/einaros/options.js/issues" 28 | }, 29 | "homepage": "https://github.com/einaros/options.js", 30 | "_id": "options@0.0.6", 31 | "_shasum": "ec22d312806bb53e731773e7cdaefcf1c643128f", 32 | "_from": "options@>=0.0.5", 33 | "_npmVersion": "1.4.21", 34 | "_npmUser": { 35 | "name": "einaros", 36 | "email": "einaros@gmail.com" 37 | }, 38 | "maintainers": [ 39 | { 40 | "name": "einaros", 41 | "email": "einaros@gmail.com" 42 | } 43 | ], 44 | "dist": { 45 | "shasum": "ec22d312806bb53e731773e7cdaefcf1c643128f", 46 | "tarball": "http://registry.npmjs.org/options/-/options-0.0.6.tgz" 47 | }, 48 | "directories": {}, 49 | "_resolved": "https://registry.npmjs.org/options/-/options-0.0.6.tgz", 50 | "readme": "ERROR: No README data found!" 51 | } 52 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/ultron/.npmignore: -------------------------------------------------------------------------------- 1 | node_modules 2 | coverage 3 | .tern-port 4 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/ultron/.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: false 2 | language: node_js 3 | node_js: 4 | - "0.12" 5 | - "0.10" 6 | - "0.8" 7 | - "iojs" 8 | before_install: 9 | - 'if [ "${TRAVIS_NODE_VERSION}" == "0.8" ]; then npm install -g npm@2.11.1; fi' 10 | script: 11 | - "npm run test-travis" 12 | after_script: 13 | - "npm install coveralls@2.11.x && cat coverage/lcov.info | coveralls" 14 | matrix: 15 | fast_finish: true 16 | notifications: 17 | irc: 18 | channels: 19 | - "irc.freenode.org#unshift" 20 | on_success: change 21 | on_failure: change 22 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/ultron/LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Unshift.io, Arnout Kazemier, the Contributors. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/ultron/README.md: -------------------------------------------------------------------------------- 1 | # Ultron 2 | 3 | [![Made by unshift](https://img.shields.io/badge/made%20by-unshift-00ffcc.svg?style=flat-square)](http://unshift.io)[![Version npm](http://img.shields.io/npm/v/ultron.svg?style=flat-square)](http://browsenpm.org/package/ultron)[![Build Status](http://img.shields.io/travis/unshiftio/ultron/master.svg?style=flat-square)](https://travis-ci.org/unshiftio/ultron)[![Dependencies](https://img.shields.io/david/unshiftio/ultron.svg?style=flat-square)](https://david-dm.org/unshiftio/ultron)[![Coverage Status](http://img.shields.io/coveralls/unshiftio/ultron/master.svg?style=flat-square)](https://coveralls.io/r/unshiftio/ultron?branch=master)[![IRC channel](http://img.shields.io/badge/IRC-irc.freenode.net%23unshift-00a8ff.svg?style=flat-square)](http://webchat.freenode.net/?channels=unshift) 4 | 5 | Ultron is a high-intelligence robot. It gathers intelligence so it can start 6 | improving upon his rudimentary design. It will learn your event emitting 7 | patterns and find ways to exterminate them. Allowing you to remove only the 8 | event emitters that **you** assigned and not the ones that your users or 9 | developers assigned. This can prevent race conditions, memory leaks and even file 10 | descriptor leaks from ever happening as you won't remove clean up processes. 11 | 12 | ## Installation 13 | 14 | The module is designed to be used in browsers using browserify and in Node.js. 15 | You can install the module through the public npm registry by running the 16 | following command in CLI: 17 | 18 | ``` 19 | npm install --save ultron 20 | ``` 21 | 22 | ## Usage 23 | 24 | In all examples we assume that you've required the library as following: 25 | 26 | ```js 27 | 'use strict'; 28 | 29 | var Ultron = require('ultron'); 30 | ``` 31 | 32 | Now that we've required the library we can construct our first `Ultron` instance. 33 | The constructor requires one argument which should be the `EventEmitter` 34 | instance that we need to operate upon. This can be the `EventEmitter` module 35 | that ships with Node.js or `EventEmitter3` or anything else as long as it 36 | follow the same API and internal structure as these 2. So with that in mind we 37 | can create the instance: 38 | 39 | ```js 40 | // 41 | // For the sake of this example we're going to construct an empty EventEmitter 42 | // 43 | var EventEmitter = require('events').EventEmitter; // or require('eventmitter3'); 44 | var events = new EventEmitter(); 45 | 46 | var ultron = new Ultron(events); 47 | ``` 48 | 49 | You can now use the following API's from the Ultron instance: 50 | 51 | ### Ultron.on 52 | 53 | Register a new event listener for the given event. It follows the exact same API 54 | as `EventEmitter.on` but it will return itself instead of returning the 55 | EventEmitter instance. If you are using EventEmitter3 it also supports the 56 | context param: 57 | 58 | ```js 59 | ultron.on('event-name', handler, { custom: 'function context' }); 60 | ``` 61 | 62 | ### Ultron.once 63 | 64 | Exactly the same as the [Ultron.on](#ultronon) but it only allows the execution 65 | once. 66 | 67 | ### Ultron.remove 68 | 69 | This is where all the magic happens and the safe removal starts. This function 70 | accepts different argument styles: 71 | 72 | - No arguments, assume that all events need to be removed so it will work as 73 | `removeAllListeners()` API. 74 | - 1 argument, when it's a string it will be split on ` ` and `,` to create a 75 | list of events that need to be cleared. 76 | - Multiple arguments, we assume that they are all names of events that need to 77 | be cleared. 78 | 79 | ```js 80 | ultron.remove('foo, bar baz'); // Removes foo, bar and baz. 81 | ultron.remove('foo', 'bar', 'baz'); // Removes foo, bar and baz. 82 | ultron.remove(); // Removes everything. 83 | ``` 84 | 85 | If you just want to remove a single event listener using a function reference 86 | you can still use the EventEmitter's `removeListener(event, fn)` API: 87 | 88 | ```js 89 | function foo() {} 90 | 91 | ulton.on('foo', foo); 92 | events.removeListener('foo', foo); 93 | ``` 94 | 95 | ## License 96 | 97 | MIT 98 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/ultron/index.js: -------------------------------------------------------------------------------- 1 | 'use strict'; 2 | 3 | var has = Object.prototype.hasOwnProperty; 4 | 5 | /** 6 | * An auto incrementing id which we can use to create "unique" Ultron instances 7 | * so we can track the event emitters that are added through the Ultron 8 | * interface. 9 | * 10 | * @type {Number} 11 | * @private 12 | */ 13 | var id = 0; 14 | 15 | /** 16 | * Ultron is high-intelligence robot. It gathers intelligence so it can start improving 17 | * upon his rudimentary design. It will learn from your EventEmitting patterns 18 | * and exterminate them. 19 | * 20 | * @constructor 21 | * @param {EventEmitter} ee EventEmitter instance we need to wrap. 22 | * @api public 23 | */ 24 | function Ultron(ee) { 25 | if (!(this instanceof Ultron)) return new Ultron(ee); 26 | 27 | this.id = id++; 28 | this.ee = ee; 29 | } 30 | 31 | /** 32 | * Register a new EventListener for the given event. 33 | * 34 | * @param {String} event Name of the event. 35 | * @param {Functon} fn Callback function. 36 | * @param {Mixed} context The context of the function. 37 | * @returns {Ultron} 38 | * @api public 39 | */ 40 | Ultron.prototype.on = function on(event, fn, context) { 41 | fn.__ultron = this.id; 42 | this.ee.on(event, fn, context); 43 | 44 | return this; 45 | }; 46 | /** 47 | * Add an EventListener that's only called once. 48 | * 49 | * @param {String} event Name of the event. 50 | * @param {Function} fn Callback function. 51 | * @param {Mixed} context The context of the function. 52 | * @returns {Ultron} 53 | * @api public 54 | */ 55 | Ultron.prototype.once = function once(event, fn, context) { 56 | fn.__ultron = this.id; 57 | this.ee.once(event, fn, context); 58 | 59 | return this; 60 | }; 61 | 62 | /** 63 | * Remove the listeners we assigned for the given event. 64 | * 65 | * @returns {Ultron} 66 | * @api public 67 | */ 68 | Ultron.prototype.remove = function remove() { 69 | var args = arguments 70 | , event; 71 | 72 | // 73 | // When no event names are provided we assume that we need to clear all the 74 | // events that were assigned through us. 75 | // 76 | if (args.length === 1 && 'string' === typeof args[0]) { 77 | args = args[0].split(/[, ]+/); 78 | } else if (!args.length) { 79 | args = []; 80 | 81 | for (event in this.ee._events) { 82 | if (has.call(this.ee._events, event)) args.push(event); 83 | } 84 | } 85 | 86 | for (var i = 0; i < args.length; i++) { 87 | var listeners = this.ee.listeners(args[i]); 88 | 89 | for (var j = 0; j < listeners.length; j++) { 90 | event = listeners[j]; 91 | 92 | // 93 | // Once listeners have a `listener` property that stores the real listener 94 | // in the EventEmitter that ships with Node.js. 95 | // 96 | if (event.listener) { 97 | if (event.listener.__ultron !== this.id) continue; 98 | delete event.listener.__ultron; 99 | } else { 100 | if (event.__ultron !== this.id) continue; 101 | delete event.__ultron; 102 | } 103 | 104 | this.ee.removeListener(args[i], event); 105 | } 106 | } 107 | 108 | return this; 109 | }; 110 | 111 | /** 112 | * Destroy the Ultron instance, remove all listeners and release all references. 113 | * 114 | * @returns {Boolean} 115 | * @api public 116 | */ 117 | Ultron.prototype.destroy = function destroy() { 118 | if (!this.ee) return false; 119 | 120 | this.remove(); 121 | this.ee = null; 122 | 123 | return true; 124 | }; 125 | 126 | // 127 | // Expose the module. 128 | // 129 | module.exports = Ultron; 130 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/node_modules/ultron/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "ultron", 3 | "version": "1.0.2", 4 | "description": "Ultron is high-intelligence robot. It gathers intel so it can start improving upon his rudimentary design", 5 | "main": "index.js", 6 | "scripts": { 7 | "100%": "istanbul check-coverage --statements 100 --functions 100 --lines 100 --branches 100", 8 | "test": "mocha test.js", 9 | "watch": "mocha --watch test.js", 10 | "coverage": "istanbul cover ./node_modules/.bin/_mocha -- test.js", 11 | "test-travis": "istanbul cover node_modules/.bin/_mocha --report lcovonly -- test.js" 12 | }, 13 | "repository": { 14 | "type": "git", 15 | "url": "git+https://github.com/unshiftio/ultron.git" 16 | }, 17 | "keywords": [ 18 | "Ultron", 19 | "robot", 20 | "gather", 21 | "intelligence", 22 | "event", 23 | "events", 24 | "eventemitter", 25 | "emitter", 26 | "cleanup" 27 | ], 28 | "author": { 29 | "name": "Arnout Kazemier" 30 | }, 31 | "license": "MIT", 32 | "devDependencies": { 33 | "assume": "1.2.x", 34 | "eventemitter3": "1.1.x", 35 | "istanbul": "0.3.x", 36 | "mocha": "2.2.x", 37 | "pre-commit": "1.0.x" 38 | }, 39 | "bugs": { 40 | "url": "https://github.com/unshiftio/ultron/issues" 41 | }, 42 | "homepage": "https://github.com/unshiftio/ultron", 43 | "gitHead": "a10482ae98a09120821545456c90c6d60d540f7c", 44 | "_id": "ultron@1.0.2", 45 | "_shasum": "ace116ab557cd197386a4e88f4685378c8b2e4fa", 46 | "_from": "ultron@>=1.0.0 <1.1.0", 47 | "_npmVersion": "2.9.1", 48 | "_nodeVersion": "0.12.3", 49 | "_npmUser": { 50 | "name": "3rdeden", 51 | "email": "npm@3rd-Eden.com" 52 | }, 53 | "maintainers": [ 54 | { 55 | "name": "unshift", 56 | "email": "npm@unshift.io" 57 | }, 58 | { 59 | "name": "v1", 60 | "email": "info@3rd-Eden.com" 61 | }, 62 | { 63 | "name": "3rdeden", 64 | "email": "npm@3rd-Eden.com" 65 | } 66 | ], 67 | "dist": { 68 | "shasum": "ace116ab557cd197386a4e88f4685378c8b2e4fa", 69 | "tarball": "http://registry.npmjs.org/ultron/-/ultron-1.0.2.tgz" 70 | }, 71 | "directories": {}, 72 | "_resolved": "https://registry.npmjs.org/ultron/-/ultron-1.0.2.tgz", 73 | "readme": "ERROR: No README data found!" 74 | } 75 | -------------------------------------------------------------------------------- /4_Source/SimpleDS-master_by_junchangwook/web/node_modules/ws/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "author": { 3 | "name": "Einar Otto Stangvik", 4 | "email": "einaros@gmail.com", 5 | "url": "http://2x.io" 6 | }, 7 | "name": "ws", 8 | "description": "simple to use, blazing fast and thoroughly tested websocket client, server and console for node.js, up-to-date against RFC-6455", 9 | "version": "0.8.0", 10 | "license": "MIT", 11 | "keywords": [ 12 | "Hixie", 13 | "HyBi", 14 | "Push", 15 | "RFC-6455", 16 | "WebSocket", 17 | "WebSockets", 18 | "real-time" 19 | ], 20 | "repository": { 21 | "type": "git", 22 | "url": "git://github.com/websockets/ws.git" 23 | }, 24 | "scripts": { 25 | "test": "make test" 26 | }, 27 | "dependencies": { 28 | "options": ">=0.0.5", 29 | "ultron": "1.0.x", 30 | "bufferutil": "1.2.x", 31 | "utf-8-validate": "1.2.x" 32 | }, 33 | "optionalDependencies": { 34 | "bufferutil": "1.2.x", 35 | "utf-8-validate": "1.2.x" 36 | }, 37 | "devDependencies": { 38 | "ansi": "0.3.x", 39 | "benchmark": "0.3.x", 40 | "expect.js": "0.3.x", 41 | "mocha": "2.2.x", 42 | "should": "4.3.x", 43 | "tinycolor": "0.0.x" 44 | }, 45 | "browser": "./lib/browser.js", 46 | "component": { 47 | "scripts": { 48 | "ws/index.js": "./lib/browser.js" 49 | } 50 | }, 51 | "gypfile": true, 52 | "gitHead": "21383fdcacdf47eb96775e51748e51b258e07dd4", 53 | "bugs": { 54 | "url": "https://github.com/websockets/ws/issues" 55 | }, 56 | "homepage": "https://github.com/websockets/ws#readme", 57 | "_id": "ws@0.8.0", 58 | "_shasum": "ac60ebad312121d01e16cc3383d7ec67ad0f0f1f", 59 | "_from": "ws@*", 60 | "_npmVersion": "2.9.1", 61 | "_nodeVersion": "0.12.3", 62 | "_npmUser": { 63 | "name": "3rdeden", 64 | "email": "npm@3rd-Eden.com" 65 | }, 66 | "maintainers": [ 67 | { 68 | "name": "einaros", 69 | "email": "einaros@gmail.com" 70 | }, 71 | { 72 | "name": "v1", 73 | "email": "info@3rd-Eden.com" 74 | }, 75 | { 76 | "name": "3rdeden", 77 | "email": "npm@3rd-Eden.com" 78 | } 79 | ], 80 | "dist": { 81 | "shasum": "ac60ebad312121d01e16cc3383d7ec67ad0f0f1f", 82 | "tarball": "http://registry.npmjs.org/ws/-/ws-0.8.0.tgz" 83 | }, 84 | "directories": {}, 85 | "_resolved": "https://registry.npmjs.org/ws/-/ws-0.8.0.tgz", 86 | "readme": "ERROR: No README data found!" 87 | } 88 | -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/Telegram_ChatBot_by_yangsunhee/.DS_Store -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/README.md: -------------------------------------------------------------------------------- 1 | # Examples 2 | 3 | The examples in this folder are small bots meant to show you how a bot that is written with `python-telegram-bot` looks like. Some bots focus on one specific aspect of the Telegram Bot API while others focus on one of the mechanics of this library. Except for the `echobot.py` example, they all use the high-level framework this library provides with the `telegram.ext` submodule. 4 | 5 | All examples are licensed under the [CC0 License](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/LICENSE.txt) and are therefore fully dedicated to the public domain. You can use them as the base for your own bots without worrying about copyrights. 6 | 7 | ### [`echobot2.py`](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/echobot2.py) 8 | This is probably the base for most of the bots made with `python-telegram-bot`. It simply replies to each text message with a message that contains the same text. 9 | 10 | ### [`timerbot.py`](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/timerbot.py) 11 | This bot uses the [`JobQueue`](https://python-telegram-bot.readthedocs.io/en/latest/telegram.ext.jobqueue.html) class to send timed messages. The user sets a timer by using `/set` command with a specific time, for example `/set 30`. The bot then sets up a job to send a message to that user after 30 seconds. The user can also cancel the timer by sending `/unset`. To learn more about the `JobQueue`, read [this wiki article](https://github.com/python-telegram-bot/python-telegram-bot/wiki/Extensions-%E2%80%93-JobQueue). 12 | 13 | ### [`conversationbot.py`](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/conversationbot.py) 14 | A common task for a bot is to ask information from the user. In v5.0 of this library, we introduced the [`ConversationHandler`](https://python-telegram-bot.readthedocs.io/en/latest/telegram.ext.conversationhandler.html) for that exact purpose. This example uses it to retrieve user-information in a conversation-like style. To get a better understanding, take a look at the [state diagram](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/conversationbot.png). 15 | 16 | ### [`conversationbot2.py`](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/conversationbot2.py) 17 | A more complex example of a bot that uses the `ConversationHandler`. It is also more confusing. Good thing there is a [fancy state diagram](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/conversationbot2.png) for this one, too! 18 | 19 | ### [`inlinekeyboard.py`](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/inlinekeyboard.py) 20 | This example sheds some light on inline keyboards, callback queries and message editing. 21 | 22 | ### [`inlinebot.py`](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/inlinebot.py) 23 | A basic example of an [inline bot](https://core.telegram.org/bots/inline). Don't forget to enable inline mode with [@BotFather](https://telegram.me/BotFather). 24 | 25 | ## Pure API 26 | The [`echobot.py`](https://github.com/python-telegram-bot/python-telegram-bot/blob/master/examples/echobot.py) example uses only the pure, "bare-metal" API wrapper. 27 | -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/conversationbot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/Telegram_ChatBot_by_yangsunhee/examples/conversationbot.png -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/conversationbot2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/Telegram_ChatBot_by_yangsunhee/examples/conversationbot2.png -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/echobot.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # Simple Bot to reply to Telegram messages. This is built on the API wrapper, see 5 | # echobot2.py to see the same example built on the telegram.ext bot framework. 6 | # This program is dedicated to the public domain under the CC0 license. 7 | import logging 8 | import telegram 9 | from telegram.error import NetworkError, Unauthorized 10 | from time import sleep 11 | import tensorflow as tf 12 | 13 | update_id = None 14 | 15 | def main(): 16 | global update_id 17 | # Telegram Bot Authorization Token 18 | bot = telegram.Bot('358456821:AAFkeA5G0dlZPNavaB16YSgD_VNndE4pkPA') 19 | 20 | # get the first pending update_id, this is so we can skip over it in case 21 | # we get an "Unauthorized" exception. 22 | try: 23 | update_id = bot.getUpdates()[0].update_id 24 | except IndexError: 25 | update_id = None 26 | 27 | logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s') 28 | 29 | while True: 30 | try: 31 | echo(bot) 32 | except NetworkError: 33 | sleep(1) 34 | except Unauthorized: 35 | # The user has removed or blocked the bot. 36 | update_id += 1 37 | 38 | 39 | def echo(bot): 40 | global update_id 41 | # Request updates after the last update_id 42 | for update in bot.getUpdates(offset=update_id, timeout=10): 43 | # chat_id is required to reply to any message 44 | chat_id = update.message.chat_id 45 | update_id = update.update_id + 1 46 | 47 | if update.message: # your bot can receive updates without messages 48 | # Reply to the message 49 | update.message.reply_text(update.message.text) 50 | 51 | 52 | if __name__ == '__main__': 53 | main() 54 | -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/echobot2.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # Simple Bot to reply to Telegram messages 5 | # This program is dedicated to the public domain under the CC0 license. 6 | """ 7 | This Bot uses the Updater class to handle the bot. 8 | 9 | First, a few handler functions are defined. Then, those functions are passed to 10 | the Dispatcher and registered at their respective places. 11 | Then, the bot is started and runs until we press Ctrl-C on the command line. 12 | 13 | Usage: 14 | Basic Echobot example, repeats messages. 15 | Press Ctrl-C on the command line or send a signal to the process to stop the 16 | bot. 17 | """ 18 | 19 | from telegram.ext import Updater, CommandHandler, MessageHandler, Filters 20 | import logging 21 | 22 | # Enable logging 23 | logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', 24 | level=logging.INFO) 25 | 26 | logger = logging.getLogger(__name__) 27 | 28 | 29 | # Define a few command handlers. These usually take the two arguments bot and 30 | # update. Error handlers also receive the raised TelegramError object in error. 31 | def start(bot, update): 32 | update.message.reply_text('Hi!') 33 | 34 | 35 | def help(bot, update): 36 | update.message.reply_text('Help!') 37 | 38 | 39 | def echo(bot, update): 40 | update.message.reply_text(update.message.text) 41 | 42 | 43 | def error(bot, update, error): 44 | logger.warn('Update "%s" caused error "%s"' % (update, error)) 45 | 46 | 47 | def main(): 48 | # Create the EventHandler and pass it your bot's token. 49 | updater = Updater("TOKEN") 50 | 51 | # Get the dispatcher to register handlers 52 | dp = updater.dispatcher 53 | 54 | # on different commands - answer in Telegram 55 | dp.add_handler(CommandHandler("start", start)) 56 | dp.add_handler(CommandHandler("help", help)) 57 | 58 | # on noncommand i.e message - echo the message on Telegram 59 | dp.add_handler(MessageHandler(Filters.text, echo)) 60 | 61 | # log all errors 62 | dp.add_error_handler(error) 63 | 64 | # Start the Bot 65 | updater.start_polling() 66 | 67 | # Run the bot until you press Ctrl-C or the process receives SIGINT, 68 | # SIGTERM or SIGABRT. This should be used most of the time, since 69 | # start_polling() is non-blocking and will stop the bot gracefully. 70 | updater.idle() 71 | 72 | 73 | if __name__ == '__main__': 74 | main() 75 | -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/inlinebot.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # Simple Bot to reply to Telegram messages 5 | # This program is dedicated to the public domain under the CC0 license. 6 | """ 7 | This Bot uses the Updater class to handle the bot. 8 | 9 | First, a few handler functions are defined. Then, those functions are passed to 10 | the Dispatcher and registered at their respective places. 11 | Then, the bot is started and runs until we press Ctrl-C on the command line. 12 | 13 | Usage: 14 | Basic inline bot example. Applies different text transformations. 15 | Press Ctrl-C on the command line or send a signal to the process to stop the 16 | bot. 17 | """ 18 | from uuid import uuid4 19 | 20 | import re 21 | 22 | from telegram import InlineQueryResultArticle, ParseMode, \ 23 | InputTextMessageContent 24 | from telegram.ext import Updater, InlineQueryHandler, CommandHandler 25 | import logging 26 | 27 | # Enable logging 28 | logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', 29 | level=logging.INFO) 30 | 31 | logger = logging.getLogger(__name__) 32 | 33 | 34 | # Define a few command handlers. These usually take the two arguments bot and 35 | # update. Error handlers also receive the raised TelegramError object in error. 36 | def start(bot, update): 37 | update.message.reply_text('Hi!') 38 | 39 | 40 | def help(bot, update): 41 | update.message.reply_text('Help!') 42 | 43 | 44 | def escape_markdown(text): 45 | """Helper function to escape telegram markup symbols""" 46 | escape_chars = '\*_`\[' 47 | return re.sub(r'([%s])' % escape_chars, r'\\\1', text) 48 | 49 | 50 | def inlinequery(bot, update): 51 | query = update.inline_query.query 52 | results = list() 53 | 54 | results.append(InlineQueryResultArticle(id=uuid4(), 55 | title="Caps", 56 | input_message_content=InputTextMessageContent( 57 | query.upper()))) 58 | 59 | results.append(InlineQueryResultArticle(id=uuid4(), 60 | title="Bold", 61 | input_message_content=InputTextMessageContent( 62 | "*%s*" % escape_markdown(query), 63 | parse_mode=ParseMode.MARKDOWN))) 64 | 65 | results.append(InlineQueryResultArticle(id=uuid4(), 66 | title="Italic", 67 | input_message_content=InputTextMessageContent( 68 | "_%s_" % escape_markdown(query), 69 | parse_mode=ParseMode.MARKDOWN))) 70 | 71 | update.inline_query.answer(results) 72 | 73 | 74 | def error(bot, update, error): 75 | logger.warn('Update "%s" caused error "%s"' % (update, error)) 76 | 77 | 78 | def main(): 79 | # Create the Updater and pass it your bot's token. 80 | updater = Updater("TOKEN") 81 | 82 | # Get the dispatcher to register handlers 83 | dp = updater.dispatcher 84 | 85 | # on different commands - answer in Telegram 86 | dp.add_handler(CommandHandler("start", start)) 87 | dp.add_handler(CommandHandler("help", help)) 88 | 89 | # on noncommand i.e message - echo the message on Telegram 90 | dp.add_handler(InlineQueryHandler(inlinequery)) 91 | 92 | # log all errors 93 | dp.add_error_handler(error) 94 | 95 | # Start the Bot 96 | updater.start_polling() 97 | 98 | # Block until the user presses Ctrl-C or the process receives SIGINT, 99 | # SIGTERM or SIGABRT. This should be used most of the time, since 100 | # start_polling() is non-blocking and will stop the bot gracefully. 101 | updater.idle() 102 | 103 | 104 | if __name__ == '__main__': 105 | main() 106 | -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/inlinekeyboard.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # Basic example for a bot that uses inline keyboards. 5 | # This program is dedicated to the public domain under the CC0 license. 6 | 7 | import logging 8 | from telegram import InlineKeyboardButton, InlineKeyboardMarkup 9 | from telegram.ext import Updater, CommandHandler, CallbackQueryHandler 10 | 11 | logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', 12 | level=logging.INFO) 13 | 14 | 15 | def start(bot, update): 16 | keyboard = [[InlineKeyboardButton("Option 1", callback_data='1'), 17 | InlineKeyboardButton("Option 2", callback_data='2')], 18 | 19 | [InlineKeyboardButton("Option 3", callback_data='3')]] 20 | 21 | reply_markup = InlineKeyboardMarkup(keyboard) 22 | 23 | update.message.reply_text('Please choose:', reply_markup=reply_markup) 24 | 25 | 26 | def button(bot, update): 27 | query = update.callback_query 28 | 29 | bot.editMessageText(text="Selected option: %s" % query.data, 30 | chat_id=query.message.chat_id, 31 | message_id=query.message.message_id) 32 | 33 | 34 | def help(bot, update): 35 | update.message.reply_text("Use /start to test this bot.") 36 | 37 | 38 | def error(bot, update, error): 39 | logging.warning('Update "%s" caused error "%s"' % (update, error)) 40 | 41 | 42 | # Create the Updater and pass it your bot's token. 43 | updater = Updater("TOKEN") 44 | 45 | updater.dispatcher.add_handler(CommandHandler('start', start)) 46 | updater.dispatcher.add_handler(CallbackQueryHandler(button)) 47 | updater.dispatcher.add_handler(CommandHandler('help', help)) 48 | updater.dispatcher.add_error_handler(error) 49 | 50 | # Start the Bot 51 | updater.start_polling() 52 | 53 | # Run the bot until the user presses Ctrl-C or the process receives SIGINT, 54 | # SIGTERM or SIGABRT 55 | updater.idle() 56 | -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/timerbot.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # Simple Bot to send timed Telegram messages 5 | # This program is dedicated to the public domain under the CC0 license. 6 | """ 7 | This Bot uses the Updater class to handle the bot and the JobQueue to send 8 | timed messages. 9 | 10 | First, a few handler functions are defined. Then, those functions are passed to 11 | the Dispatcher and registered at their respective places. 12 | Then, the bot is started and runs until we press Ctrl-C on the command line. 13 | 14 | Usage: 15 | Basic Alarm Bot example, sends a message after a set time. 16 | Press Ctrl-C on the command line or send a signal to the process to stop the 17 | bot. 18 | """ 19 | 20 | from telegram.ext import Updater, CommandHandler, Job 21 | import logging 22 | 23 | # Enable logging 24 | logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', 25 | level=logging.INFO) 26 | 27 | logger = logging.getLogger(__name__) 28 | 29 | 30 | # Define a few command handlers. These usually take the two arguments bot and 31 | # update. Error handlers also receive the raised TelegramError object in error. 32 | def start(bot, update): 33 | update.message.reply_text('Hi! Use /set to set a timer') 34 | 35 | 36 | def alarm(bot, job): 37 | """Function to send the alarm message""" 38 | bot.sendMessage(job.context, text='Beep!') 39 | 40 | 41 | def set(bot, update, args, job_queue, chat_data): 42 | """Adds a job to the queue""" 43 | chat_id = update.message.chat_id 44 | try: 45 | # args[0] should contain the time for the timer in seconds 46 | due = int(args[0]) 47 | if due < 0: 48 | update.message.reply_text('Sorry we can not go back to future!') 49 | return 50 | 51 | # Add job to queue 52 | job = Job(alarm, due, repeat=False, context=chat_id) 53 | chat_data['job'] = job 54 | job_queue.put(job) 55 | 56 | update.message.reply_text('Timer successfully set!') 57 | 58 | except (IndexError, ValueError): 59 | update.message.reply_text('Usage: /set ') 60 | 61 | 62 | def unset(bot, update, chat_data): 63 | """Removes the job if the user changed their mind""" 64 | 65 | if 'job' not in chat_data: 66 | update.message.reply_text('You have no active timer') 67 | return 68 | 69 | job = chat_data['job'] 70 | job.schedule_removal() 71 | del chat_data['job'] 72 | 73 | update.message.reply_text('Timer successfully unset!') 74 | 75 | 76 | def error(bot, update, error): 77 | logger.warning('Update "%s" caused error "%s"' % (update, error)) 78 | 79 | 80 | def main(): 81 | updater = Updater("TOKEN") 82 | 83 | # Get the dispatcher to register handlers 84 | dp = updater.dispatcher 85 | 86 | # on different commands - answer in Telegram 87 | dp.add_handler(CommandHandler("start", start)) 88 | dp.add_handler(CommandHandler("help", start)) 89 | dp.add_handler(CommandHandler("set", set, 90 | pass_args=True, 91 | pass_job_queue=True, 92 | pass_chat_data=True)) 93 | dp.add_handler(CommandHandler("unset", unset, pass_chat_data=True)) 94 | 95 | # log all errors 96 | dp.add_error_handler(error) 97 | 98 | # Start the Bot 99 | updater.start_polling() 100 | 101 | # Block until you press Ctrl-C or the process receives SIGINT, SIGTERM or 102 | # SIGABRT. This should be used most of the time, since start_polling() is 103 | # non-blocking and will stop the bot gracefully. 104 | updater.idle() 105 | 106 | 107 | if __name__ == '__main__': 108 | main() 109 | -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/ts.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # Simple Bot to reply to Telegram messages. This is built on the API wrapper, see 5 | # echobot2.py to see the same example built on the telegram.ext bot framework. 6 | # This program is dedicated to the public domain under the CC0 license. 7 | import logging 8 | import telegram 9 | from telegram.error import NetworkError, Unauthorized 10 | from time import sleep 11 | import tensorflow as tf 12 | import numpy as np 13 | 14 | update_id = None 15 | 16 | def main(): 17 | global update_id 18 | # Telegram Bot Authorization Token 19 | bot = telegram.Bot('358456821:AAFkeA5G0dlZPNavaB16YSgD_VNndE4pkPA') 20 | 21 | # get the first pending update_id, this is so we can skip over it in case 22 | # we get an "Unauthorized" exception. 23 | try: 24 | update_id = bot.getUpdates()[0].update_id 25 | except IndexError: 26 | update_id = None 27 | 28 | logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s') 29 | 30 | while True: 31 | try: 32 | echo(bot) 33 | except NetworkError: 34 | sleep(1) 35 | except Unauthorized: 36 | # The user has removed or blocked the bot. 37 | update_id += 1 38 | 39 | 40 | def echo(bot): 41 | global update_id 42 | # Request updates after the last update_id 43 | for update in bot.getUpdates(offset=update_id, timeout=10): 44 | # chat_id is required to reply to any message 45 | chat_id = update.message.chat_id 46 | update_id = update.update_id + 1 47 | 48 | if update.message: # your bot can receive updates without messages 49 | hello = tf.constant('Hello, TensorFlow!') 50 | sess = tf.Session() 51 | result = sess.run(hello) 52 | re = [] 53 | re.append('jeromwolf') 54 | 55 | 56 | # Reply to the message 57 | update.message.reply_text(update.message.text + ' '.join(re)) 58 | 59 | 60 | if __name__ == '__main__': 61 | main() 62 | -------------------------------------------------------------------------------- /4_Source/Telegram_ChatBot_by_yangsunhee/examples/w2v.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | # -*- coding: utf-8 -*- 3 | # 4 | # Simple Bot to reply to Telegram messages. This is built on the API wrapper, see 5 | # echobot2.py to see the same example built on the telegram.ext bot framework. 6 | # This program is dedicated to the public domain under the CC0 license. 7 | import logging 8 | import telegram 9 | from telegram.error import NetworkError, Unauthorized 10 | from time import sleep 11 | import tensorflow as tf 12 | import numpy as np 13 | from gensim.models import Word2Vec 14 | 15 | raw_corpus_fname = 'norm_2016-10-24_article_all.txt' # Fill your corpus file 16 | 17 | class Word2VecCorpus: 18 | def __init__(self, fname): 19 | self.fname = fname 20 | def __iter__(self): 21 | with open(self.fname, encoding='utf-8') as f: 22 | for line in f: 23 | line = line.strip().replace('\n', '') 24 | if not line: 25 | continue 26 | yield line.split() 27 | 28 | word2vec_corpus = Word2VecCorpus(raw_corpus_fname) 29 | word2vec_model = Word2Vec(word2vec_corpus, size=150, min_count=10) 30 | 31 | 32 | update_id = None 33 | 34 | def main(): 35 | global update_id 36 | # Telegram Bot Authorization Token 37 | bot = telegram.Bot('358456821:AAFkeA5G0dlZPNavaB16YSgD_VNndE4pkPA') 38 | 39 | # get the first pending update_id, this is so we can skip over it in case 40 | # we get an "Unauthorized" exception. 41 | try: 42 | update_id = bot.getUpdates()[0].update_id 43 | except IndexError: 44 | update_id = None 45 | 46 | logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s') 47 | 48 | while True: 49 | try: 50 | echo(bot) 51 | except NetworkError: 52 | sleep(1) 53 | except KeyError: 54 | sleep(1) 55 | # update.message.reply_text("sorry!! not in vocabulary") 56 | except Unauthorized: 57 | # The user has removed or blocked the bot. 58 | update_id += 1 59 | 60 | 61 | def echo(bot): 62 | global update_id 63 | # Request updates after the last update_id 64 | for update in bot.getUpdates(offset=update_id, timeout=10): 65 | # chat_id is required to reply to any message 66 | chat_id = update.message.chat_id 67 | update_id = update.update_id + 1 68 | 69 | # update.message.reply_text("검색단어를 입력하세요? ex) 오바마,박근혜(코퍼스가 신문사설입니다.)") 70 | 71 | if update.message: # your bot can receive updates without messages 72 | values = ',\n'.join(str(v) for v in word2vec_model.most_similar(update.message.text)) 73 | 74 | # Reply to the message 75 | update.message.reply_text(values) 76 | update.message.reply_text("검색단어를 입력하세요? ex)오바마, 박근혜(코퍼스가 신문내용입니다.)") 77 | 78 | 79 | if __name__ == '__main__': 80 | main() 81 | -------------------------------------------------------------------------------- /4_Source/ngram-master_by_kimchanghoon/.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "2.7" 4 | - "3.2" 5 | - "3.3" 6 | - "3.4" 7 | - "3.5" 8 | - "3.5-dev" # 3.5 development branch 9 | - "nightly" # currently points to 3.6-dev 10 | # command to install dependencies 11 | # install: "pip install -r requirements.txt" 12 | # command to run tests 13 | script: nosetests -------------------------------------------------------------------------------- /4_Source/ngram-master_by_kimchanghoon/LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Jaideep Bhoosreddy 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /4_Source/ngram-master_by_kimchanghoon/README.md: -------------------------------------------------------------------------------- 1 | [![Build Status](https://travis-ci.org/jbhoosreddy/ngram.svg?branch=master)](https://travis-ci.org/jbhoosreddy/ngram) 2 | 3 | Ngram 4 | ===== 5 | 6 | A software which creates n-Gram (1-5) Maximum Likelihood Probabilistic Language Model with Laplace Add-1 smoothing and stores it in hash-able dictionary form. 7 | 8 | class nGram 9 | | A program which creates n-Gram (1-5) Maximum Likelihood Probabilistic Language Model with Laplace Add-1 smoothing 10 | | and stores it in hash-able dictionary form. 11 | | n: number of bigrams (supports up to 5) 12 | | corpus_file: relative path to the corpus file. 13 | | cache: saves computed values if True 14 | | 15 | | 16 | | Usage: 17 | | >>> ng = nGram(n=5, corpus_file=None, cache=False) 18 | | >>> print(ng.sentence_probability(sentence='hold your horses', n=2, form='log')) 19 | | >>> -18.655540764 20 | | 21 | | Methods defined here: 22 | | 23 | | __init__(self, n=1, corpus_file=None, cache=False) 24 | | Constructor method which loads the corpus from file and creates ngrams based on imput parameters. 25 | | 26 | | create_bigram(self, cache) 27 | | Method to create Bigram Model for words loaded from corpus. 28 | | 29 | | create_pentigram(self, cache) 30 | | Method to create Pentigram Model for words loaded from corpus. 31 | | 32 | | create_quadrigram(self, cache) 33 | | Method to create Quadrigram Model for words loaded from corpus. 34 | | 35 | | create_trigram(self, cache) 36 | | Method to create Trigram Model for words loaded from corpus. 37 | | 38 | | create_unigram(self, cache) 39 | | Method to create Unigram Model for words loaded from corpus. 40 | | 41 | | load_corpus(self, file_name) 42 | | Method to load external file which contains raw corpus. 43 | | 44 | | probability(self, word, words='', n=1) 45 | | Method to calculate the Maximum Likelihood Probability of n-Grams on the basis of various parameters. 46 | | 47 | | sentence_probability(self, sentence, n=1, form='antilog') 48 | | Method to calculate cumulative n-gram Maximum Likelihood Probability of a phrase or sentence. 49 | -------------------------------------------------------------------------------- /4_Source/ngram-master_by_kimchanghoon/smoothing.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/ngram-master_by_kimchanghoon/smoothing.pdf -------------------------------------------------------------------------------- /4_Source/ngram-master_by_kimchanghoon/tests.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from ngram import nGram 3 | ng = nGram(n=5, corpus_file=None, cache=False) 4 | 5 | 6 | class TestNgram(unittest.TestCase): 7 | def test_uni_log(self): 8 | probability = ng.sentence_probability(sentence='hold your horses', n=1, form='log') 9 | self.assertAlmostEqual(probability, -24.9337710989) 10 | 11 | def test_uni_antilog(self): 12 | probability = ng.sentence_probability(sentence='hold your horses', n=1, form='antilog') 13 | self.assertAlmostEqual(probability, 1.48388689281e-11) 14 | 15 | def test_bi_log(self): 16 | probability = ng.sentence_probability(sentence='hold your horses', n=2, form='log') 17 | self.assertAlmostEqual(probability, -18.655540764) 18 | 19 | def test_bi_antilog(self): 20 | probability = ng.sentence_probability(sentence='hold your horses', n=2, form='antilog') 21 | self.assertAlmostEqual(probability, 7.90681521418e-09) 22 | 23 | def test_tri_log(self): 24 | probability = ng.sentence_probability(sentence='hold your horses', n=3, form='log') 25 | self.assertAlmostEqual(probability, -11.3066636125) 26 | 27 | def test_tri_antilog(self): 28 | probability = ng.sentence_probability(sentence='hold your horses', n=3, form='antilog') 29 | self.assertAlmostEqual(probability, 1.22907499816e-05) 30 | 31 | def test_quadri_log(self): 32 | probability = ng.sentence_probability(sentence='hold your horses', n=4, form='log') 33 | self.assertAlmostEqual(probability, 0) 34 | 35 | def test_quadri_antilog(self): 36 | probability = ng.sentence_probability(sentence='hold your horses', n=4, form='antilog') 37 | self.assertAlmostEqual(probability, 1) 38 | 39 | def test_penti_log(self): 40 | probability = ng.sentence_probability(sentence='hold your horses', n=5, form='log') 41 | self.assertAlmostEqual(probability, 0) 42 | 43 | def test_penti_antilog(self): 44 | probability = ng.sentence_probability(sentence='hold your horses', n=5, form='antilog') 45 | self.assertAlmostEqual(probability, 1) 46 | 47 | 48 | if "__name__" == "__main__": 49 | unittest.main() -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/seq2seq_chatbot_by_ohyeontak/.DS_Store -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/README.md: -------------------------------------------------------------------------------- 1 | # 챗봇 2 | 3 | - 간단하게 만든 Seq2Seq 모델을 이용하여 챗봇을 만들어봅니다. 4 | - 소설 어린왕자에서 어린왕자와 여우의 25번의 대화가 포함되어 있습니다. 5 | - 위 대화로 간단히 학습시킨 모델이 있어 채팅해보기를 바로 사용해볼 수 있습니다. (data/chat.log 를 봐주세요) 6 | - Seq2Seq 모델 자체는 크게 변경하지 않았으며, 모델을 이용하기 위해 단어 임베딩을 만들고, 이를 이용하는 전처리/후처리 코드가 많이 추가되었습니다. 7 | 8 | ### 사용방법 9 | 10 | 기본적으로 다음 데이터를 사용합니다. config.py 파일에서 추가 옵션들을 확인해보세요. 11 | 12 | - data/chat.log : 대화 데이터 13 | - data/chat.voc : 어휘 데이터 14 | 15 | ### 채팅해보기 16 | 17 | ``` 18 | python chat.py 19 | ``` 20 | 21 | 22 | 23 | ### 학습시키기 24 | 25 | ``` 26 | python train.py --train 27 | ``` 28 | 29 | 학습된 모델이 있으면 새로 생성하지 않고 추가학습을 합니다. 30 | 31 | 다음과 같이 텐서보드를 통해 cost 를 확인할 수 있습니다. 32 | 33 | ``` 34 | tensorboard --logdir=./logs 35 | ``` 36 | 37 | ### 테스트 해 보기 38 | 39 | 모델이 작동하는지만 봅니다. 정확도는 신뢰할 수 없습니다. 40 | 41 | ``` 42 | python train.py --test 43 | ``` 44 | 45 | ### 어휘 데이터 생성하기 46 | 47 | 토크나이저를 바꿨거나 새로운 데이터를 이용하기를 원하는 경우, 다음과 같은 방법으로 먼저 어휘 데이터를 생성해야 합니다. 48 | 49 | 기본 토크나이저는 공백과 특수문자로 구분합니다. 50 | 51 | ``` 52 | python dialog.py --voc_build 53 | ``` -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/__pycache__/config.cpython-35.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/seq2seq_chatbot_by_ohyeontak/__pycache__/config.cpython-35.pyc -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/__pycache__/dialog.cpython-35.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/seq2seq_chatbot_by_ohyeontak/__pycache__/dialog.cpython-35.pyc -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/__pycache__/model.cpython-35.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/seq2seq_chatbot_by_ohyeontak/__pycache__/model.cpython-35.pyc -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/chat.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import tensorflow as tf 4 | import numpy as np 5 | import math 6 | import sys 7 | 8 | from config import FLAGS 9 | from model import Seq2Seq 10 | from dialog import Dialog 11 | 12 | 13 | class ChatBot: 14 | 15 | def __init__(self, voc_path, train_dir): 16 | self.dialog = Dialog() 17 | self.dialog.load_vocab(voc_path) 18 | 19 | self.model = Seq2Seq(self.dialog.vocab_size) 20 | 21 | self.sess = tf.Session() 22 | ckpt = tf.train.get_checkpoint_state(train_dir) 23 | self.model.saver.restore(self.sess, ckpt.model_checkpoint_path) 24 | 25 | def run(self): 26 | sys.stdout.write("> ") 27 | sys.stdout.flush() 28 | line = sys.stdin.readline() 29 | 30 | while line: 31 | print (self.get_replay(line.strip())) 32 | 33 | sys.stdout.write("\n> ") 34 | sys.stdout.flush() 35 | 36 | line = sys.stdin.readline() 37 | 38 | def decode(self, enc_input, dec_input): 39 | if type(dec_input) is np.ndarray: 40 | dec_input = dec_input.tolist() 41 | 42 | # TODO: 구글처럼 시퀀스 사이즈에 따라 적당한 버킷을 사용하도록 만들어서 사용하도록 43 | input_len = int(math.ceil((len(enc_input) + 1) * 1.5)) 44 | 45 | enc_input, dec_input, _ = self.dialog.transform(enc_input, dec_input, 46 | input_len, 47 | FLAGS.max_decode_len) 48 | 49 | return self.model.predict(self.sess, [enc_input], [dec_input]) 50 | 51 | def get_replay(self, msg): 52 | enc_input = self.dialog.tokenizer(msg) 53 | enc_input = self.dialog.tokens_to_ids(enc_input) 54 | dec_input = [] 55 | 56 | # TODO: 구글처럼 Seq2Seq2 모델 안의 RNN 셀을 생성하는 부분에 넣을것 57 | # 입력값에 따라 디코더셀의 상태를 순차적으로 구성하도록 함 58 | # 여기서는 최종 출력값을 사용하여 점진적으로 시퀀스를 만드는 방식을 사용 59 | # 다만 상황에 따라서는 이런 방식이 더 유연할 수도 있을 듯 60 | curr_seq = 0 61 | for i in range(FLAGS.max_decode_len): 62 | outputs = self.decode(enc_input, dec_input) 63 | if self.dialog.is_eos(outputs[0][curr_seq]): 64 | break 65 | elif self.dialog.is_defined(outputs[0][curr_seq]) is not True: 66 | dec_input.append(outputs[0][curr_seq]) 67 | curr_seq += 1 68 | 69 | reply = self.dialog.decode([dec_input], True) 70 | 71 | return reply 72 | 73 | 74 | def main(_): 75 | print ("깨어나는 중 입니다. 잠시만 기다려주세요...\n") 76 | 77 | chatbot = ChatBot(FLAGS.voc_path, FLAGS.train_dir) 78 | chatbot.run() 79 | 80 | if __name__ == "__main__": 81 | tf.app.run() 82 | -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/config.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import tensorflow as tf 4 | 5 | 6 | tf.app.flags.DEFINE_string("train_dir", "./model", "학습한 신경망을 저장할 폴더") 7 | 8 | tf.app.flags.DEFINE_string("log_dir", "./logs", "로그를 저장할 폴더") 9 | tf.app.flags.DEFINE_string("ckpt_name", "conversation.ckpt", "체크포인트 파일명") 10 | 11 | tf.app.flags.DEFINE_boolean("train", False, "학습을 진행합니다.") 12 | tf.app.flags.DEFINE_boolean("test", True, "테스트를 합니다.") 13 | tf.app.flags.DEFINE_boolean("data_loop", True, "작은 데이터셋을 실험해보기 위해 사용합니다.") 14 | tf.app.flags.DEFINE_integer("batch_size", 100, "미니 배치 크기") 15 | tf.app.flags.DEFINE_integer("epoch", 1000, "총 학습 반복 횟수") 16 | 17 | tf.app.flags.DEFINE_string("data_path", "./data/chat.log", "대화 파일 위치") 18 | tf.app.flags.DEFINE_string("voc_path", "./data/chat.voc", "어휘 사전 파일 위치") 19 | 20 | 21 | tf.app.flags.DEFINE_boolean("voc_test", False, "어휘 사전을 테스트합니다.") 22 | tf.app.flags.DEFINE_boolean("voc_build", False, "주어진 대화 파일을 이용해 어휘 사전을 작성합니다.") 23 | 24 | tf.app.flags.DEFINE_integer("max_decode_len", 20, "최대 디코더 셀 크기 = 최대 답변 크기.") 25 | 26 | 27 | FLAGS = tf.app.flags.FLAGS 28 | -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/logs/events.out.tfevents.1484699399.mpr: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/4_Source/seq2seq_chatbot_by_ohyeontak/logs/events.out.tfevents.1484699399.mpr -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/model.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # 자세한 설명은 상위 폴더의 03 - Seq2Seq.py 등에서 찾으실 수 있습니다. 3 | 4 | import tensorflow as tf 5 | 6 | 7 | # Seq2Seq 기본 클래스 8 | class Seq2Seq: 9 | logits = None 10 | outputs = None 11 | cost = None 12 | train_op = None 13 | 14 | def __init__(self, vocab_size, n_hidden=128, n_layers=3): 15 | self.learning_late = 0.001 16 | 17 | self.vocab_size = vocab_size 18 | self.n_hidden = n_hidden 19 | self.n_layers = n_layers 20 | 21 | self.enc_input = tf.placeholder(tf.float32, [None, None, self.vocab_size]) 22 | self.dec_input = tf.placeholder(tf.float32, [None, None, self.vocab_size]) 23 | self.targets = tf.placeholder(tf.int64, [None, None]) 24 | 25 | self.weights = tf.Variable(tf.ones([self.n_hidden, self.vocab_size]), name="weights") 26 | self.bias = tf.Variable(tf.zeros([self.vocab_size]), name="bias") 27 | self.global_step = tf.Variable(0, trainable=False, name="global_step") 28 | 29 | self.build_model() 30 | 31 | self.saver = tf.train.Saver(tf.global_variables()) 32 | 33 | # model 생성 34 | def build_model(self): 35 | # enc_input, dec_input 대해서 transpose를 수행 36 | self.enc_input = tf.transpose(self.enc_input, [1, 0, 2]) 37 | self.dec_input = tf.transpose(self.dec_input, [1, 0, 2]) 38 | 39 | # Cell 생성 후 저장 40 | enc_cell, dec_cell = self.build_cells() 41 | 42 | # 모든 Tensorflow의 RNN 함수들은 Cell을 인자로 받음 43 | with tf.variable_scope('encode'): 44 | outputs, enc_states = tf.nn.dynamic_rnn(enc_cell, self.enc_input, dtype=tf.float32) 45 | with tf.variable_scope('decode'): 46 | outputs, dec_states = tf.nn.dynamic_rnn(dec_cell, self.dec_input, dtype=tf.float32) 47 | 48 | self.logits, self.cost, self.train_op = self.build_ops(outputs, self.targets) 49 | 50 | self.outputs = tf.argmax(self.logits, 2) 51 | 52 | # cell 생성 53 | def build_cells(self, output_keep_prob=0.5): 54 | # tensorflow 1.0.1버전 수정 (오연택) 55 | 56 | # LSTM cell 57 | enc_cell = tf.contrib.rnn.BasicLSTMCell(self.n_hidden); 58 | 59 | # RNN cell의 입력과 출력 연결에 대해 dropout 기능을 추가해주는 wrapper 60 | # 다중 레이어와 과적합 방지를 위한 Dropout 기법을 사용 61 | enc_cell = tf.contrib.rnn.DropoutWrapper(enc_cell, output_keep_prob=output_keep_prob) 62 | 63 | # 여러개의 RNN cell을 연결하여, Multi-layer cell로 구성해주는 wrapper입니다. 64 | enc_cell = tf.contrib.rnn.MultiRNNCell([enc_cell] * self.n_layers) 65 | 66 | # decoding cell 67 | dec_cell = tf.contrib.rnn.BasicRNNCell(self.n_hidden) 68 | dec_cell = tf.contrib.rnn.DropoutWrapper(dec_cell, output_keep_prob=output_keep_prob) 69 | dec_cell = tf.contrib.rnn.MultiRNNCell([dec_cell] * self.n_layers) 70 | 71 | # 생성된 cell retrun 72 | return enc_cell, dec_cell 73 | 74 | # logits, loss, optimizer 정의 75 | def build_ops(self, outputs, targets): 76 | time_steps = tf.shape(outputs)[1] 77 | outputs = tf.reshape(outputs, [-1, self.n_hidden]) 78 | 79 | # logits 는 one-hot 인코딩을 사용합니다. 80 | logits = tf.matmul(outputs, self.weights) + self.bias 81 | logits = tf.reshape(logits, [-1, time_steps, self.vocab_size]) 82 | 83 | ## Define loss and optimizer 84 | # Tensorflow 1.0.0에서는 파라미터 순서가 변경 85 | cost = tf.reduce_mean( 86 | tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, 87 | labels=targets)) # Softmax loss 88 | # AdamOptimizer 89 | train_op = tf.train.AdamOptimizer(learning_rate=self.learning_late).minimize(cost, global_step=self.global_step) 90 | 91 | tf.summary.scalar('cost', cost) 92 | 93 | return logits, cost, train_op 94 | 95 | # train 함수 96 | def train(self, session, enc_input, dec_input, targets): 97 | return session.run([self.train_op, self.cost], 98 | feed_dict={self.enc_input: enc_input, 99 | self.dec_input: dec_input, 100 | self.targets: targets}) 101 | 102 | def predict(self, session, enc_input, dec_input): 103 | return session.run(self.outputs, 104 | feed_dict={self.enc_input: enc_input, 105 | self.dec_input: dec_input}) 106 | 107 | def write_logs(self, session, writer, enc_input, dec_input, targets): 108 | merged = tf.summary.merge_all() 109 | 110 | summary = session.run(merged, feed_dict={self.enc_input: enc_input, 111 | self.dec_input: dec_input, 112 | self.targets: targets}) 113 | 114 | writer.add_summary(summary, self.global_step.eval()) 115 | -------------------------------------------------------------------------------- /4_Source/seq2seq_chatbot_by_ohyeontak/train.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | import tensorflow as tf 4 | import random 5 | import math 6 | import os 7 | 8 | from config import FLAGS 9 | from model import Seq2Seq 10 | from dialog import Dialog 11 | 12 | 13 | def train(dialog, batch_size=100, epoch=100): 14 | model = Seq2Seq(dialog.vocab_size) 15 | 16 | with tf.Session() as sess: 17 | 18 | # 학습된 모델이 저장된 경로 체크 19 | ckpt = tf.train.get_checkpoint_state(FLAGS.train_dir) 20 | 21 | if ckpt and tf.train.checkpoint_exists(ckpt.model_checkpoint_path): 22 | print("다음 파일에서 모델을 읽는 중 입니다..", ckpt.model_checkpoint_path) 23 | model.saver.restore(sess, ckpt.model_checkpoint_path) 24 | else: 25 | print("새로운 모델을 생성하는 중 입니다.") 26 | sess.run(tf.global_variables_initializer()) 27 | 28 | # 로그를 저장 29 | writer = tf.summary.FileWriter(FLAGS.log_dir, sess.graph) 30 | 31 | # 전체 batch size 결정 32 | total_batch = int(math.ceil(len(dialog.examples) / float(batch_size))) 33 | 34 | # total step 35 | print(total_batch * epoch) 36 | 37 | # 신경망 모델 학습 38 | for step in range(total_batch * epoch): 39 | enc_input, dec_input, targets = dialog.next_batch(batch_size) 40 | 41 | # model 학습 42 | _, loss = model.train(sess, enc_input, dec_input, targets) 43 | 44 | # log 출력 45 | if (step + 1) % 100 == 0: 46 | model.write_logs(sess, writer, enc_input, dec_input, targets) 47 | print('Step:', '%06d' % model.global_step.eval(), \ 48 | 'cost =', '{:.6f}'.format(loss)) 49 | 50 | checkpoint_path = os.path.join(FLAGS.train_dir, FLAGS.ckpt_name) 51 | model.saver.save(sess, checkpoint_path, global_step=model.global_step) 52 | 53 | print('최적화 완료!') 54 | 55 | 56 | def main(_): 57 | dialog = Dialog() 58 | 59 | dialog.load_vocab(FLAGS.voc_path) # 어절 사전 파일 위치 60 | dialog.load_dialogue(FLAGS.data_path) # 대화 스크립트 파일 위치 61 | train(dialog, batch_size=FLAGS.batch_size, epoch=FLAGS.epoch) # 학습 시작 62 | 63 | 64 | if __name__ == "__main__": 65 | tf.app.run() 66 | -------------------------------------------------------------------------------- /4_Source/test.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | 3 | from __future__ import unicode_literals 4 | 5 | import unittest 6 | 7 | from textrankr import TextRank 8 | 9 | 10 | class TestReordered(unittest.TestCase): 11 | 12 | def setUp(self): 13 | self.text = "트위터, \"정보당국에 데이터 분석자료 팔지 않겠다\". 트위터가 수많은 트윗을 분석해 정보를 판매하는 서비스를 미국 정보당국에는 제공하지 않기로 했다. 월스트리트저널은 미국 정보당국 관계자 등을 인용해 데이터마이너(Dataminer)가 정보당국에 대한 서비스는 중단하기로 했다고 9일(현지시간) 보도했다. 트위터가 5% 지분을 가진 데이터마이너는 소셜미디어상 자료를 분석해 고객이 의사결정을 하도록 정보를 제공하는 기업이다. 트위터에 올라오는 트윗에 실시간으로 접근해 분석한 자료를 고객에게 팔 수 있는 독점권을 갖고 있다. 정보당국은 이 회사로부터 구매한 자료로 테러나 정치적 불안정 등과 관련된 정보를 획득했다. 이 회사가 정보당국에 서비스를 판매하지 않기로 한 것은 트위터의 결정인 것으로 알려졌다. 데이터마이너 경영진은 최근 “트위터가 정보당국에 서비스하는 것을 원치 않는다”고 밝혔다고 이 신문은 전했다. 트위터도 성명을 내고 “정보당국 감시용으로 데이터를 팔지 않는 것은 트위터의 오래된 정책”이라며 “트위터 자료는 대체로 공개적이고 미국 정부도 다른 사용자처럼 공개된 어카운트를 살펴볼 수 있다”고 해명했다. 그러나 이는 이 회사가 2년 동안 정보당국에 서비스를 제공해 온 데 대해서는 타당한 설명이 되지 않는다. 트위터의 이번 결정은 미국의 정보기술(IT)기업과 정보당국 간 갈등의 연장 선상에서 이뤄진 것으로 여겨지고 있다. IT기업은 이용자 프라이버시에 무게 중심을 두는 데 비해 정보당국은 공공안전을 우선시해 차이가 있었다. 특히 애플은 캘리포니아 주 샌버너디노 총격범의 아이폰에 저장된 정보를 보겠다며 데이터 잠금장치 해제를 요구하는 미 연방수사국(FBI)과 소송까지 진행했다. 정보당국 고위 관계자도 “트위터가 정보당국과 너무 가까워 보이는 것을 우려하는 것 같다”고 말했다. 데이터마이너는 금융기관이나, 언론사 등 정보당국을 제외한 고객에 대한 서비스는 계속할 계획이다." 14 | 15 | def test_ranked(self): 16 | textrank = TextRank(self.text) 17 | print(textrank.summarize()) 18 | 19 | 20 | if __name__ == '__main__': 21 | unittest.main() -------------------------------------------------------------------------------- /5_Idea/README.md: -------------------------------------------------------------------------------- 1 | # Chat_Learning 2 | 3 | 1. 웨이블릿 : https://ko.wikipedia.org/wiki/%EC%9B%A8%EC%9D%B4%EB%B8%94%EB%A6%BF 4 | -------------------------------------------------------------------------------- /6_dataset/data_sites: -------------------------------------------------------------------------------- 1 | Premade Datasets 2 | 1. http://research.microsoft.com/en-us/um/redmond/projects/mctest/index.html 3 | MCTest is a freely available set of 660 stories and associated questions intended for research on the machine comprehension of text. 4 | 2. http://www.gutenberg.org/wiki/Gutenberg:Offline_Catalogs 5 | Gutenberge has a lot of books 6 | 3. https://catalog.ldc.upenn.edu/LDC2006T13 7 | Web 1T 5-gram Version 1, contributed by Google Inc., contains English word n-grams and their observed frequency counts. 8 | The length of the n-grams ranges from unigrams (single words) to five-grams. 9 | This data is expected to be useful for statistical language modeling, e.g., for machine translation or speech recognition, etc. 10 | 4. http://www.iesl.cs.umass.edu/data 11 | A lot of datasets 12 | 5. http://webdatacommons.org/webtables/ 13 | A subset of the HTML tables on the Web contains relational data which can be useful for various applications. 14 | The Web Data Commons project has extracted two large corpora of relational Web tables from the Common Crawl and offers them for public download. 15 | This page provides an overview of the corpora as well as their use cases. 16 | 6. http://statmt.org/ngrams/ 17 | Unpruend Unpruned 5-gram counts and language models trained on 9 billion web pages -- Large amounts of raw data in many languages 18 | 7. https://en.wikipedia.org/wiki/Wikipedia:Database_download 19 | Wikipedia Database Download 20 | 8. https://aws.amazon.com/ko/datasets/google-books-ngrams/ 21 | A data set containing Google Books n-gram corpora. 22 | 9. https://aws.amazon.com/ko/public-datasets/common-crawl/ 23 | The Common Crawl corpus includes web crawl data collected over 8 years. 24 | Common Crawl offers the largest, most comprehensive, open repository of web crawl data on the cloud. 25 | 10. http://commoncrawl.org/the-data/tutorials/ 26 | 착한 아이들 ㅋㅋ 27 | 11. https://wikireverse.org/data 28 | The full dataset of 36 million links can be downloaded as a torrent. 29 | The download is a tarball containing 4 tab-delimited files. 30 | The data is 1.1 GB when compressed and 5.4 GB when extracted. 31 | 12. https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html 32 | This corpus contains a large metadata-rich collection of fictional conversations extracted from raw movie scripts. 33 | 13. https://www.uow.edu.au/~dlee/corpora.htm 34 | several dozens of english corpus 35 | 14. http://research.google.com/research-outreach.html#/research-outreach/research-datasets 36 | Google Datasets 37 | 15. http://www.cs.cornell.edu/home/llee/data/ 38 | Collection of Cornell Datasets 39 | 16. https://github.com/rkadlec/ubuntu-ranking-dataset-creator 40 | Ubuntu Dialogue Datasets 41 | 17. http://ebiquity.umbc.edu/resource/html/id/351 42 | The UMBC webBase corpus (http://ebiq.org/r/351) is a dataset containing a collection of English paragraphs with over three billion words 43 | processed from the February 2007 crawl from the Stanford WebBase project (http://bit.ly/WebBase). Compressed, it is about 13GB in size. 44 | 45 | 46 | 47 | Movie Subtitles Datasets (BE AWARE OF COPYRIGHTS!!!) 48 | 49 | http://www.opensubtitles.org/en/search 50 | https://subscene.com/ 51 | http://www.moviesubtitles.org/ 52 | http://www.divxsubtitles.net/ 53 | http://www.subs4free.com/ 54 | 55 | https://videoconverter.iskysoft.com/video-tips/download-subtitles.html (15 Best Subtitle Software and Top 10 Subtitle Download Sites) 56 | 57 | 58 | 59 | Q&A Datasets 60 | https://www.researchgate.net/post/What_are_the_datasets_available_for_question_answering_system 61 | https://archive.org/details/stackexchange 62 | https://rajpurkar.github.io/SQuAD-explorer/ 63 | https://www.quora.com/Datasets-How-can-I-get-corpus-of-a-question-answering-website-like-Quora-or-Yahoo-Answers-or-Stack-Overflow-for-analyzing-answer-quality 64 | http://jmcauley.ucsd.edu/data/amazon/qa/ 65 | 66 | 67 | A lot of Datasets 68 | https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/ 69 | https://github.com/caesar0301/awesome-public-datasets#natural-language 70 | 71 | 72 | Miscellaneous 73 | https://github.com/deepmind/rc-data 74 | http://u.cs.biu.ac.il/~koppel/BlogCorpus.htm 75 | http://wiki.dbpedia.org/Downloads2015-10 76 | https://aws.amazon.com/ko/datasets/google-books-ngrams/ 77 | 78 | 79 | -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/README.md: -------------------------------------------------------------------------------- 1 | # YandexSchool_NLPCourse 2 | 3 | -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_007.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_007.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_012.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_012.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_013.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_013.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_015.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_015.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_018.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_018.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_022.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_022.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_025.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_025.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_029.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_029.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_030.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_030.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_031.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_031.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_033.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_033.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_035.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_035.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_039.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_039.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_040.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_040.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_044.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_044.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_045.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_045.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_047.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_047.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_048.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_048.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_049.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_049.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_052.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_052.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_054.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_054.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_056.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_056.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_057.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_057.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_058.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_058.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_059.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_059.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_060.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_060.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_061.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_061.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_062.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_062.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_066.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_066.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_067.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_067.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_068.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_068.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_069.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_069.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_070.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_070.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_071.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_071.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_072.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_072.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_073.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_073.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_074.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_074.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_075.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_075.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_076.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_076.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_077.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_077.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_079.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_079.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_080.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_080.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_081.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_081.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_082.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_082.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_084.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_084.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_085.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_085.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_087.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_087.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_089.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_089.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_090.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_090.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_092.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_092.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_093.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_093.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_094.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/7.YandexSchool_NLPCourse/assets/week01/yandex_embedding_094.png -------------------------------------------------------------------------------- /7.YandexSchool_NLPCourse/word2vec.py: -------------------------------------------------------------------------------- 1 | import tensorflow as tf 2 | import numpy as np 3 | 4 | class Word2Vec(tf.estimator.Estimator): 5 | """ 6 | Wrod Embedding 방법 중 기본이 되는 Word2Vec을 확인하기 위한 Class 7 | """ 8 | def __init__(self, n_input, n_embed, n_sampl, learning_rate=0.1): 9 | """ 10 | 설명: 초기 생성 함수 11 | Parameters: 12 | - n_input: 입력 Vocabulary size 13 | - n_embed: embedding 사이즈 (hidden size) 14 | - n_sampl: sample size ??? 15 | """ 16 | self.n_input = n_input 17 | self.n_embed = n_embed 18 | self.n_sampl = min(n_embed, n_sampl) 19 | self.learning_rate = learning_rate 20 | 21 | self.global_step = tf.Variable(0, trainable=False, name="global_step") 22 | self._build_network() 23 | 24 | def _build_network(self): 25 | """ 26 | 설명: Tensorflow Graph 생성 27 | """ 28 | self.inputs = tf.placeholder(tf.int32, shape=[None]) 29 | self.labels = tf.placeholder(tf.int32, shape=[None, 1]) 30 | 31 | self.embeddings = tf.Variable(tf.random_uniform([self.n_input, self.n_embed], -1.0, 1.0)) 32 | 33 | self.selected_embed = tf.nn.embedding_lookup(self.embeddings, self.inputs) 34 | 35 | nce_weights = tf.Variable(tf.random_uniform([self.n_input, self.n_embed], -1.0, 1.0)) 36 | nce_biases = tf.Variable(tf.zeros([self.n_input])) 37 | 38 | self.loss = tf.reduce_mean(tf.nn.nce_loss(nce_weights, nce_biases, self.labels, self.selected_embed, self.n_sampl, self.n_input)) 39 | self.optimizer = tf.train.AdamOptimizer(self.learning_rate).minimize(self.loss, global_step=self.global_step) 40 | 41 | def train(self, sess, inputs, labels): 42 | """ 43 | 설명: 학습 44 | Parameters: 45 | - sess: Tensorflow Session 46 | - inputs: 입력값 47 | - labels: 출력값 48 | Return: 49 | - otimizer: 학습결과 50 | - loss: 오차 51 | """ 52 | return sess.run([self.optimizer, self.loss], feed_dict={self.inputs: inputs, self.labels: labels}) 53 | 54 | def select(self, sess, inputs): 55 | """ 56 | 설명: 단어 선택 57 | Parameters: 58 | - sess: Tensorflow Session 59 | - inputs: 입력값 60 | Return: embedding 백터 값 61 | """ 62 | return sess.run(self.selected_embed, feed_dict={self.inputs: inputs}) 63 | 64 | -------------------------------------------------------------------------------- /Quora_insincere/.gitignore: -------------------------------------------------------------------------------- 1 | /2_NLP_Study 2 | .DS_Store 3 | .ipynb_checkpoints/ 4 | data_in/ 5 | sh/ 6 | input/ 7 | 8 | # Byte-compiled / optimized / DLL files 9 | __pycache__/ 10 | *.py[cod] 11 | *$py.class 12 | 13 | # C extensions 14 | *.so 15 | 16 | # Distribution / packaging 17 | .Python 18 | build/ 19 | develop-eggs/ 20 | dist/ 21 | downloads/ 22 | eggs/ 23 | .eggs/ 24 | lib/ 25 | lib64/ 26 | parts/ 27 | sdist/ 28 | var/ 29 | wheels/ 30 | *.egg-info/ 31 | .installed.cfg 32 | *.egg 33 | MANIFEST 34 | 35 | # PyInstaller 36 | # Usually these files are written by a python script from a template 37 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 38 | *.manifest 39 | *.spec 40 | 41 | # Installer logs 42 | pip-log.txt 43 | pip-delete-this-directory.txt 44 | 45 | # Unit test / coverage reports 46 | htmlcov/ 47 | .tox/ 48 | .coverage 49 | .coverage.* 50 | .cache 51 | nosetests.xml 52 | coverage.xml 53 | *.cover 54 | .hypothesis/ 55 | .pytest_cache/ 56 | 57 | # Translations 58 | *.mo 59 | *.pot 60 | 61 | # Django stuff: 62 | *.log 63 | local_settings.py 64 | db.sqlite3 65 | 66 | # Flask stuff: 67 | instance/ 68 | .webassets-cache 69 | 70 | # Scrapy stuff: 71 | .scrapy 72 | 73 | # Sphinx documentation 74 | docs/_build/ 75 | 76 | # PyBuilder 77 | target/ 78 | 79 | # Jupyter Notebook 80 | .ipynb_checkpoints 81 | 82 | # pyenv 83 | .python-version 84 | 85 | # celery beat schedule file 86 | celerybeat-schedule 87 | 88 | # SageMath parsed files 89 | *.sage.py 90 | 91 | # Environments 92 | .env 93 | .venv 94 | env/ 95 | venv/ 96 | ENV/ 97 | env.bak/ 98 | venv.bak/ 99 | 100 | # Spyder project settings 101 | .spyderproject 102 | .spyproject 103 | 104 | # Rope project settings 105 | .ropeproject 106 | 107 | # mkdocs documentation 108 | /site 109 | 110 | # mypy 111 | .mypy_cache/ 112 | 113 | .vscode 114 | .ipynb_checkpoints 115 | 116 | *.voc 117 | checkPoint 118 | *.log 119 | 120 | OLD/ -------------------------------------------------------------------------------- /Quora_insincere/README.md: -------------------------------------------------------------------------------- 1 | Kaggle 2 | 3 | https://www.kaggle.com/c/quora-insincere-questions-classification/data -------------------------------------------------------------------------------- /Quora_insincere/jupyter_examples/dataset/test.csv.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/Quora_insincere/jupyter_examples/dataset/test.csv.zip -------------------------------------------------------------------------------- /Quora_insincere/jupyter_examples/dataset/train.csv.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/modulabs/DeepNLP/d45b1ff791ad13cbda3b15317bc34eb7e64c1a7a/Quora_insincere/jupyter_examples/dataset/train.csv.zip -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Chat_Learning 2 | * **2019년 DeepNLP Repository**: https://github.com/modudeepnlp/DeepNLP2019 3 | * 스케쥴: https://docs.google.com/spreadsheets/d/1-m9TveaMZ54EVI-ikGYcp1orD1-GsIwXZbaZDi0oW4c/edit#gid=1282839535 4 | 5 | * 구글드라이브 저장소: https://drive.google.com/drive/folders/13fitbwqw_r-IjNJuSk-tMjKETjyIuWFd?usp=sharing 6 | ## 매주 수요일 8:00 ~ 10:30. 강남 모두의 연구소 캠퍼스 7 | 8 | # Tensorflow로 시작하는 Kaggle 자연어처리 (On Progress) 9 | Git Book: https://nlp.gitbook.io/book/ 10 | DeepNLP 위키북스 GitHub : https://github.com/NLP-kr/tensorflow-ml-nlp 11 | DeepNLP 2019 GitHub : https://github.com/NLP-kr/DeepNLP2019 12 | DeepNLP 논문 구현 : https://github.com/NLP-kr/paper-implement.git 13 | 14 | ### 자세한 사항은 모두연 딥 NLP반에 Contact: modu@modulabs.co.kr### 15 | 16 | 17 | # 스터디 Member 자료 공유 18 | 신성진 깃헙 : https://github.com/aiscientist 19 | 전창욱 깃헙 : https://github.com/changwookjun 20 | 조중현 개인 블로그 : https://reniew.github.io/ 21 | 22 | 23 | 24 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | padnas 2 | sklearn 3 | tensorflow 4 | matplotlib 5 | --------------------------------------------------------------------------------