├── MANIFEST.in ├── mescrappy ├── scrappy │ ├── __init__.py │ ├── __pycache__ │ │ ├── __init__.cpython-37.pyc │ │ └── scrapper.cpython-37.pyc │ └── scrapper.py ├── __pycache__ │ ├── __main__.cpython-37.pyc │ └── scrapper.cpython-37.pyc ├── __main__.py └── test.py ├── mescrappy.egg-info ├── top_level.txt ├── dependency_links.txt ├── SOURCES.txt └── PKG-INFO ├── requirements.txt ├── dist ├── mescrappy-1.0.0.tar.gz └── mescrappy-1.0.0-py3-none-any.whl ├── Annotation 2021-08-17 204528.png ├── pyproject.toml ├── .gitignore ├── setup.cfg ├── LICENCE └── README.md /MANIFEST.in: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /mescrappy/scrappy/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /mescrappy.egg-info/top_level.txt: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /mescrappy.egg-info/dependency_links.txt: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | autopep8==1.5.7 2 | pycodestyle==2.7.0 3 | selenium==3.141.0 4 | toml==0.10.2 5 | urllib3==1.26.6 6 | -------------------------------------------------------------------------------- /dist/mescrappy-1.0.0.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MerlinEmris/youtube_srapping_with_python/HEAD/dist/mescrappy-1.0.0.tar.gz -------------------------------------------------------------------------------- /Annotation 2021-08-17 204528.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MerlinEmris/youtube_srapping_with_python/HEAD/Annotation 2021-08-17 204528.png -------------------------------------------------------------------------------- /dist/mescrappy-1.0.0-py3-none-any.whl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MerlinEmris/youtube_srapping_with_python/HEAD/dist/mescrappy-1.0.0-py3-none-any.whl -------------------------------------------------------------------------------- /mescrappy/__pycache__/__main__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MerlinEmris/youtube_srapping_with_python/HEAD/mescrappy/__pycache__/__main__.cpython-37.pyc -------------------------------------------------------------------------------- /mescrappy/__pycache__/scrapper.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MerlinEmris/youtube_srapping_with_python/HEAD/mescrappy/__pycache__/scrapper.cpython-37.pyc -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [build-system] 2 | requires = [ 3 | "selenium==3.141.0", 4 | "setuptools>=54", 5 | "wheel" 6 | ] 7 | build-backend = "setuptools.build_meta" -------------------------------------------------------------------------------- /mescrappy/scrappy/__pycache__/__init__.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MerlinEmris/youtube_srapping_with_python/HEAD/mescrappy/scrappy/__pycache__/__init__.cpython-37.pyc -------------------------------------------------------------------------------- /mescrappy/scrappy/__pycache__/scrapper.cpython-37.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/MerlinEmris/youtube_srapping_with_python/HEAD/mescrappy/scrappy/__pycache__/scrapper.cpython-37.pyc -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Default ignored files 2 | /.idea 3 | /shelf/ 4 | /workspace.xml 5 | # Datasource local storage ignored files 6 | /dataSources/ 7 | /dataSources.local.xml 8 | # Editor-based HTTP Client requests 9 | /httpRequests/ 10 | -------------------------------------------------------------------------------- /mescrappy.egg-info/SOURCES.txt: -------------------------------------------------------------------------------- 1 | LICENCE 2 | MANIFEST.in 3 | README.md 4 | pyproject.toml 5 | setup.cfg 6 | mescrappy.egg-info/PKG-INFO 7 | mescrappy.egg-info/SOURCES.txt 8 | mescrappy.egg-info/dependency_links.txt 9 | mescrappy.egg-info/top_level.txt -------------------------------------------------------------------------------- /mescrappy/__main__.py: -------------------------------------------------------------------------------- 1 | import sys 2 | from scrappy.scrapper import youtube_video_data_scrapper as yvds 3 | 4 | 5 | if __name__ == "__main__": 6 | url = sys.argv[1] 7 | driver = sys.argv[2] 8 | print(url, driver) 9 | # url = 'https://www.youtube.com/watch?v=rMO7APyBiMI' 10 | # driver = r"C:\Users\ME\projects\for_github\chromedriver_win32\chromedriver.exe" 11 | print(yvds(url, driver)) 12 | -------------------------------------------------------------------------------- /mescrappy/test.py: -------------------------------------------------------------------------------- 1 | import unittest 2 | from scrappy.scrapper import youtube_video_data_scrapper 3 | 4 | 5 | class TestScrapper(unittest.TestCase): 6 | def test_scrap(self): 7 | url = 'https://www.youtube.com/watch?v=TFMnICdHiyM' 8 | driver = r"C:\Users\ME\projects\for_github\chromedriver_win32\chromedriver.exe" 9 | self.assertAlmostEqual( 10 | youtube_video_data_scrapper(url=url, driver=driver)['title'], "Xiaomi Mi 11 Lite vs Samsung A52: SIMILAR BUT ONLY ONE WINNER! Let's Find Out!") 11 | 12 | 13 | if __name__ == '__main__': 14 | unittest.main() 15 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | [metadata] 2 | name = mescrappy 3 | version = 1.0.0 4 | author = Merdan Chariyarov 5 | author_email = merdanchariyarov@gmail.com 6 | description = python selenium youtube scrapper 7 | long_description = file: README.md 8 | long_description_content_type = text/markdown 9 | url = https://github.com/MerlinEmris/youtube_srapping_with_python 10 | classifiers = 11 | Programming Language :: Python :: 3 12 | License :: OSI Approved :: MIT License 13 | Operating System :: OS Independent 14 | 15 | [options] 16 | packages = find: 17 | python_requires = >=3.7 18 | include_package_data = True -------------------------------------------------------------------------------- /LICENCE: -------------------------------------------------------------------------------- 1 | 2 | MIT License 3 | 4 | Copyright (c) 2021 Merdan Chariyarov 5 | 6 | Permission is hereby granted, free of charge, to any person obtaining a copy 7 | of this software and associated documentation files (the "Software"), to deal 8 | in the Software without restriction, including without limitation the rights 9 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 10 | copies of the Software, and to permit persons to whom the Software is 11 | furnished to do so, subject to the following conditions: 12 | 13 | The above copyright notice and this permission notice shall be included in all 14 | copies or substantial portions of the Software. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 17 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 18 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 19 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 20 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 21 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 22 | SOFTWARE. -------------------------------------------------------------------------------- /mescrappy/scrappy/scrapper.py: -------------------------------------------------------------------------------- 1 | from selenium import webdriver 2 | from selenium.webdriver.common.by import By 3 | from selenium.webdriver.support.ui import WebDriverWait 4 | from selenium.webdriver.support import expected_conditions as EC 5 | 6 | 7 | def youtube_video_data_scrapper(url: str, driver: str) -> dict: 8 | f""" 9 | youtube video data scrapper 10 | :param url: video url starting with https:// 11 | :param driver:path to browser driver for selenium 12 | :return: dict[str: str] 13 | """ 14 | chrome_path = driver 15 | driver = webdriver.Chrome(chrome_path) 16 | driver.get(url) 17 | 18 | title = driver.find_element_by_xpath( 19 | '/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[5]/div[1]/div/div[6]/div[2]/ytd-video-primary-info-renderer/div/h1/yt-formatted-string').text 20 | description = driver.find_element_by_xpath( 21 | '//*[@id="description"]/yt-formatted-string').text 22 | owner = driver.find_element_by_xpath( 23 | '/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[5]/div[1]/div/div[7]/div[2]/ytd-video-secondary-info-renderer/div/div/ytd-video-owner-renderer/div[1]/ytd-channel-name/div/div/yt-formatted-string/a').text 24 | views = driver.find_element_by_xpath( 25 | '/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[5]/div[1]/div/div[6]/div[2]/ytd-video-primary-info-renderer/div/div/div[1]/div[1]/ytd-video-view-count-renderer/span[1]').text.split('views')[0].strip().strip(',') 26 | date = driver.find_element_by_xpath( 27 | '//*[@id="info-strings"]/yt-formatted-string').text 28 | like = driver.find_element_by_xpath( 29 | '/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[5]/div[1]/div/div[6]/div[2]/ytd-video-primary-info-renderer/div/div/div[3]/div/ytd-menu-renderer/div/ytd-toggle-button-renderer[1]/a/yt-formatted-string').text 30 | dislike = driver.find_element_by_xpath( 31 | '/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[5]/div[1]/div/div[6]/div[2]/ytd-video-primary-info-renderer/div/div/div[3]/div/ytd-menu-renderer/div/ytd-toggle-button-renderer[2]/a/yt-formatted-string').text 32 | 33 | data = { 34 | 'title': title, 35 | "description": description, 36 | 'owner': owner, 37 | 'date': date, 38 | "views": views, 39 | 'like': like, 40 | 'dislike': dislike 41 | } 42 | driver.close() 43 | return data 44 | -------------------------------------------------------------------------------- /mescrappy.egg-info/PKG-INFO: -------------------------------------------------------------------------------- 1 | Metadata-Version: 2.1 2 | Name: mescrappy 3 | Version: 1.0.0 4 | Summary: python selenium youtube scrapper 5 | Home-page: https://github.com/MerlinEmris/youtube_srapping_with_python 6 | Author: Merdan Chariyarov 7 | Author-email: merdanchariyarov@gmail.com 8 | License: MIT 9 | Platform: Independent 10 | Classifier: Programming Language :: Python :: 3 11 | Classifier: License :: OSI Approved :: MIT License 12 | Classifier: Operating System :: OS Independent 13 | Requires-Python: >=3.7 14 | Description-Content-Type: text/markdown 15 | License-File: LICENCE 16 | 17 | # mescrappy - Python + Selenium **Youtube** scrapper 18 | 19 | 20 |
21 |

22 | 23 | Logo 24 | 25 | 26 |

Youtube Srapping With Python (Selenium)

27 | 28 | 40 |

41 | 42 | 43 | 44 |
45 |

Table of Contents

 46 |
    47 |
  1. 48 | About The Project 49 | 52 |
    2. 53 |
  2. 54 | Getting Started 55 | 58 |
    3. 59 |
  3. Usage
    4. 60 |
  4. Roadmap
    5. 61 |
  5. Contributing
    6. 62 |
  6. License
    7. 63 |
  7. Contact
    8. 64 |
  8. Acknowledgements
    9. 65 |
66 |
67 | 68 | 69 | 70 | ## About The Project 71 | 72 | 73 | 74 | 75 | 76 | ### Built With 77 | 78 | - [python3]() 79 | - [selenium]() 80 | 81 | 82 | ## Getting Started 83 | 84 | ### Installation 85 | 86 | 1. To use this package first install it: [package](https://test.pypi.org/project/mescrappy) 87 | 88 | `pip install mescrappy` 89 | 90 | 2. import srapper to your project 91 | 92 | `from mescrappy.srappy import scrapper` 93 | `scrapper.youtube_video_data_scrapper(url='', driver='')` 94 | 95 | ## Usage 96 | 97 | You can use this package to scrap youtube video's: 98 | 99 | 1. title 100 | 2. view count 101 | 3. description 102 | 4. like 103 | 5. dislike 104 | 6. post date 105 | 7. owner 106 | 107 | 108 | 109 | ## Roadmap 110 | 111 | See the [open issues](https://github.com/MerlinEmris/youtube_srapping_with_python/issues) for a list of proposed features (and known issues). 112 | 113 | 114 | 115 | 116 | 117 | ## License 118 | 119 | Distributed under the MIT License. See `LICENSE` for more information. 120 | 121 | 122 | 123 | ## Contact 124 | 125 | Merdan Chariyarov - [@merlinemris96](https://twitter.com/merlinemris96) - merdanchariyarov@gmail.com 126 | 127 | Project Link: [https://github.com/MerlinEmris/youtube_srapping_with_python](https://github.com/MerlinEmris/youtube_srapping_with_python) 128 | 129 | 130 | 131 | ## Acknowledgements 132 | 133 | - [https://towardsdatascience.com/how-to-package-your-python-code-df5a7739ab2e](https://towardsdatascience.com/how-to-package-your-python-code-df5a7739ab2e) 134 | 135 | 136 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # mescrappy - Python + Selenium **Youtube** scraper 2 | 3 | 4 |
5 |

6 | 7 | image 8 | 9 | 10 |

Youtube Sraping With Python (Selenium)

11 | 12 | 24 |

25 | 26 | 27 | 28 |
29 |

Table of Contents

 30 |
    31 |
  1. 32 | About The Project 33 | 36 |
    2. 37 |
  2. 38 | Getting Started 39 | 42 |
    3. 43 |
  3. Usage
    4. 44 |
  4. Roadmap
    5. 45 |
  5. Test
    6. 46 |
  6. License
    7. 47 |
  7. Contact
    8. 48 |
  8. Acknowledgements
    9. 49 |
50 |
51 | 52 | 53 | 54 | ## About The Project 55 | 56 | 57 | 58 | 59 | 60 | ### Built With 61 | 62 | - [python3](https://www.python.org/) 63 | - [selenium](https://selenium-python.readthedocs.io/) 64 | - [chrome driver](https://chromedriver.chromium.org/downloads) 65 | 66 | 67 | ## Getting Started 68 | 69 | ### Installation 70 | 71 | 1. To use this package first install it: [package](https://test.pypi.org/project/mescrappy) 72 | 73 | `pip install mescrappy` 74 | 75 | 2. import sraper to your project 76 | 77 | `from mescrappy.srappy import scrapper` 78 | `scrapper.youtube_video_data_scrapper(url='', driver='')` 79 | 80 | You can also run it directly using terminal: 81 | 82 | `python mescrappy "https://www.youtube.com/watch?v=rMO7APyBiMI" "C:\Users\ME\projects\for_github\chromedriver_win32\chromedriver.exe"` 83 | 84 | 1'st argument is video url and 2'nd argument is browser driver location 85 | 86 | ## Usage 87 | 88 | You can use this package to scrap youtube video's: 89 | 90 | 1. title 91 | 2. view count 92 | 3. description 93 | 4. like 94 | 5. dislike 95 | 6. post date 96 | 7. owner 97 | 98 | EX: 99 | 100 | ` 101 | python mescrappy "https://www.youtube.com/watch?v=rMO7APyBiMI" "C:\Users\ME\projects\for_github\chromedriver_win32\chromedriver.exe" 102 | 103 | { 104 | 105 | 'title': 'Şeker (Cover)', 106 | 107 | 'description': 'Aýdymyň sazy : Soprano - Скорее забудь (ft. Ka-Re)\n\nLyrics:\n\nÖňler diýýärdim Aşgabat\nGerek däl başga zat\nIndi bolsa maşgalam\nGerek däl Aşgabat\n\nYanyñda oýanmak iñ uly keýp\nBaşymyz aýlanýar gerek däl meý,\nSeret! günem eýýäm günorta\nOýanmaly hiç, bulambujar otag\nBize name?! bolsun bulambujar dünýä,\nSamsyk pikirlem, şonda-da gülyäñ\n\nÝüregiñ aşagynda ýürejik urýar\nKime meñzeýärkä?, pikirlenip durýan,\nBilýäňmi näme?!, dakjak adyna Emin,\nMeñzesin ejesine, bolmasyn kemi\n\nGowy hasiyetleñ hemmesi geçmez belki,\nGeçse bolýa ejesiniñ göz reñki,\nYaşyl, öñem diýýärdim men yaşyla aşyk,\nMillet pula diýýärmikam diýýärdi, daşda...\n\nÇalyşman sizi hiç kime, \nGaraşyañ maňa giç gelsem\nUgradýañ meni ir gitsem,\nÝeke söýgimiñ biri sen,\nSöýülýänçämem diri men.\n\nDiñle! Sen meni söýýärdiñ hiçkimkäm,\nDurmuşyma geleliñ bäri men hiçkim däl.\nIşlisyraýanmy ýa çyndanam işim kän\nJaň edeýin, ýa söýýan diýmäne giçmikä?\n\nGaty seýrek hem söýmek hem söýülmek\nYüregiñ ýarysy garaşyp duranda öýüňde,\nGaýdasyñ gelýär oturan ýeriñden\nAýdasyñ gelýär oglanlara, gelenok diliñden,\nÝaýdanyp durýan, çykañok pikirden\nAýlanyp durýaň sen bedenimiñ içinde\nSaýlanyp durýar, gözelligiñ maña aýan,\nBerildi sadalyk, berildi haýa\n\nBu dünýäň oýnuny oýnasam\nMenemä ölmerin sen söýmäni goýmasañ\nGark bolman men söýgimiz meni boýlasa,\nPikire batýan sen barada oýlansam\n\nNädip saýladyñ meni hemem näme üçin,\nErbet hasiýetlem köp kemçiliklem näçe,\nBilyän, söygi şeýlerak bolaýýar nätjek\nÝüregiñ päkje özüne gatyrak çek meni\n\nJemi dört ýyldan gowrak bile eýýäm\nÖzümden gaçsamam ýene öýümize gelýän,\nHawa söýan sözi ýönekey söz däl\nSeniň ýaly tapylmaz başlasamam gözläp\n\nDaşymyzda göz kän\nÖzümede dözmän, seni bermen özgä\nYnan, muny ýazamda-da tolgunýan,\nAýlanýan ýaly depesinde tolkunlañ\n \nGerek däl bize hiçkim,\nÝyldyzlary sanamana äkideýin giçlik\nBolaly biraz işsiz\nUklap galaly asmany ýapynyp üstümize\nBir zat bor galany\n\nArzuw etmegem gowy zat\nDuýýan özümi sana zar, ýöne men azat\nGoşulmasyn duýgularymyza gazap\nAýyrmasyn biribar, degmesin nazar..\n\nDiñle! Sen meni söýýardiñ hiçkimkäm,\nDurmuşyma geleliñ bäri men hiçkim dal.\nMenema ölmerin bu söýgimiz dirika,\nDañama atdy söýýän diýmäne irmika\n\nSen men söýgülim, joram, hakyky dostum\nSen maňa hossar ýekelikden dolsam\nMeni ýumşadyañ aladalar üstüme gonsa\nSöýýän sözi ýaly gerek ýene on söz\n\nHarplar ýerini alýar, setirler dolýar\nPikirlerimi saña ýetiren bolsam bolýar,\nBar iki göz - diňe seni görýän\nBar diýjegim iki söz - Seni söýýän.\n\nBar diýjek bolanym - Seni söýýän!\nDiňe sen hakyky meni görýäň...', 108 | 109 | 'owner': 'Dali Dade', 110 | 111 | 'date': 'Premiered Jul 1, 2021', 112 | 113 | 'views': '29,392', 114 | 115 | 'like': '888', 116 | 117 | 'dislike': '5' 118 | 119 | }` 120 | 121 | 122 | 123 | ## Roadmap 124 | 125 | See the [open issues](https://github.com/MerlinEmris/youtube_srapping_with_python/issues) for a list of proposed features (and known issues). 126 | 127 | 128 | 129 | ## Test 130 | 131 | for testing : 132 | 133 | `python mescrappy\test.py` 134 | 135 | 136 | 137 | ## License 138 | 139 | Distributed under the MIT License. See `LICENSE` for more information. 140 | 141 | 142 | 143 | ## Contact 144 | 145 | Merdan Chariyarov - [@merlinemris96](https://twitter.com/merlinemris96) - merdanchariyarov@gmail.com 146 | 147 | Project Link: [https://github.com/MerlinEmris/youtube_srapping_with_python](https://github.com/MerlinEmris/youtube_srapping_with_python) 148 | 149 | 150 | 151 | ## Acknowledgements 152 | 153 | - [https://towardsdatascience.com/how-to-package-your-python-code-df5a7739ab2e](https://towardsdatascience.com/how-to-package-your-python-code-df5a7739ab2e) 154 | --------------------------------------------------------------------------------