├── .github └── ISSUE_TEMPLATE │ └── -----.md └── README.md /.github/ISSUE_TEMPLATE/-----.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: 논문 분석 3 | about: "\b공유하는 논문에 대한 요약 정보" 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | # Keywords 11 | 12 | # TL;DR 13 | 14 | # Abstract 15 | 16 | # Paper link 17 | 18 | # Presentation link 19 | 20 | # video link 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # [스터디 종료] 거꾸로 읽는 self-supervised-learning 시즌2: Contrastive learning on NLP 2 | 3 | - [마감] [신청 Google form (8/6까지)](https://docs.google.com/forms/d/1kHKbNFMOPAo5ZxirlgoTr4L4eu1AeeEXIergfTyiLyg) 4 | - 거꾸로 읽는 SSL 이번에는 NLP 분야로 넘어 왔습니다! :) 5 | - 이번에는 Contrastive Learning이 적용된 NLP논문에 중하여 역사적으로 의미가 있었던 논문을 살펴봅니다. 6 | - 해당 논문에서 제시하는 메소드의 특징 그리고 역사적으로 평가되는 이유에 대해서 즐겁게 토론하는 시간을 가집니다. 7 | - [스터디 종료] 2022.10.29 8 | 9 | ## 기간 10 | - 8/13 ~ 10/29 (11주간, 추석주간 제외) 11 | 12 | ## 발표 논문 및 순서 13 | > [google sheet](https://docs.google.com/spreadsheets/d/1P-pACgU9G0xq6M9Gufad-3tLUBavSMyUL0NIdd6TVH8/edit#gid=360530169) 14 | 15 | #### In-batch negative training 16 | | | paper title | 논문 발표 | YouTube 17 | -- | -- | -- | -- 18 | 1 주차 | [DPR: Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906) | [김강민](https://github.com/) | [video](https://youtu.be/GE2Qzq1Xj6c) 19 | 20 | #### Positive mining 21 | | | paper title | 논문 발표 | YouTube 22 | -- | -- | -- | -- 23 | 2 주차 | [ICT: Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://arxiv.org/abs/1906.00300) | [조성국](https://github.com/seongkukcho) | [video](https://youtu.be/MypoV0xAn18) 24 | 3 주차 | [DAPT/TAPT: Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://aclanthology.org/2020.acl-main.740/) | [김동규](https://github.com/Dien-ES) | [video](https://youtu.be/2W_2vHamLYo) 25 | 4 주차 | [Condenser: a Pre-training Architecture for Dense Retrieval](https://aclanthology.org/2021.emnlp-main.75.pdf)| [김희은](https://github.com/hekim3434) | [video](https://youtu.be/TLG6sJcsJxA) 26 | 5 주차 | [Contriever: Towards Unsupervised Dense Information Retrieval with Contrastive Learning](https://arxiv.org/abs/2112.09118) | [박이삭](https://github.com/tentoto) | [video](https://youtu.be/5r7iSQVK4xg) 27 | 28 | #### Hard negative mining 29 | | | paper title | 논문 발표 | YouTube 30 | -- | -- | -- | -- 31 | 6 주차 | [ANCE: Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://arxiv.org/abs/2007.00808)| [김기범](https://github.com/isingmodel) | [video](https://youtu.be/ZF9Q-jdb_rE) 32 | 7 주차 | [coCondensor: Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://arxiv.org/abs/2108.05540) | [이유경](https://github.com/yukyunglee) | [video](https://youtu.be/G7Yxz0_QkWk) 33 | 8 주차 | [Multi-stage Training with Improved Negative Contrast for Neural Passage Retrieval](https://aclanthology.org/2021.emnlp-main.492/) | [이영수](https://github.com/haconedu) | [video](https://youtu.be/QgeOPfWj9fs) 34 | 35 | 36 | #### Contrastive learning with model scaling 37 | | | paper title | 논문 발표 | YouTube 38 | -- | -- | -- | -- 39 | 9 주차 | [Text and Code Embeddings by Contrastive Pre-Training](https://arxiv.org/abs/2201.10005) | [이동형](https://github.com/abidan-korea) | [video](https://youtu.be/aPcddRRDn2U) 40 | 10 주차 | wrap-up | [강재욱](https://github.com/jwkanggist) | [video](https://youtu.be/2q8krnv87wY) 41 | 42 | ## 관련 링크 43 | - [거꾸로 읽는 self-supervised-learning 파트1 repo](https://github.com/jwkanggist/self-supervised-learning-narratives-1) 44 | - [거꾸로 읽는 SSL 유튭 채널](https://www.youtube.com/channel/UCTwcUmKhqeBhG0rQHkPVP6Q) 45 | --------------------------------------------------------------------------------