├── .DS_Store ├── readme.md ├── requirements.txt └── scripts ├── .gitignore ├── Old_scripts ├── files │ ├── Disney Stock.csv │ ├── Facebook Stock.csv │ ├── Netflix stock.csv │ ├── Tesla Stock.csv │ ├── Uber Stock.csv │ └── geckodriver.log ├── paa-q&A.py ├── paa.py └── readme.md ├── netflix.csv └── paa.py /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sundios/people-also-ask/e4d33a429ffc36902c8c5769015832f5d3bb3962/.DS_Store -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | 2 | # People Also Ask Google Scraper 3 | 4 | PPA scraper is a Python script that, based on a query scrapes Google people also ask questions. 5 | 6 | If you want to get Free People Also Ask queries please visit --> [https://www.kwrds.ai/](https://www.kwrds.ai/) 7 | 8 | 9 | Buy Me A Coffee 10 | 11 | 12 | ## Installation 13 | 14 | To install and run this script, we need to install the requirements.txt and the driver. Since we are using Chrome, we need to download the latest Chrome driver. 15 | 16 | 17 | ### Requirements 18 | 19 | To install requirements.txt, please run the following on the terminal. 20 | 21 | ```bash 22 | pip install requirements.txt 23 | ``` 24 | 25 | ### Chrome driver 26 | 27 | #### 1.- Step one (Check Chrome Version) 28 | 29 | Check which Chrome version you are using. 30 | 31 | you can do it via the terminal using the following command: 32 | 33 | ```bash 34 | /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version 35 | ``` 36 | 37 | or you can simply go into Chrome and click on Chrome> About Google Chrome. you should see something like: 38 | 39 | ``` 40 | Google Chrome is up to date. 41 | Version 89.0.4389.90 (Official Build) (x86_64) 42 | ``` 43 | 44 | #### 2.- Step two (Download Chrome Driver) 45 | 46 | Download the correct version of Chrome driver. 47 | 48 | Go to https://chromedriver.chromium.org/downloads 49 | 50 | And click on your chrome version. 51 | 52 | E.g: 'Chrome version 89, please download ChromeDriver 89.0.4389.23' 53 | 54 | 55 | #### 3.- Step three (Chrome driver permissions) 56 | 57 | If you are on MacOS Catalina do the following: 58 | 59 | Navigate to the path where your Chromedriver file is located and run 60 | 61 | ```bash 62 | xattr -d com.apple.quarantine chromedriver 63 | 64 | ``` 65 | 66 | #### 4.- Step four (chromedriver path) 67 | 68 | You can do either solution: 69 | 70 | a.- Move chrome driver to /usr/local/bin/ 71 | 72 | Make sure you are on the folder chromedriver is and run. 73 | 74 | ```bash 75 | sudo mv chromedriver /usr/local/bin/ 76 | ``` 77 | 78 | b.- Update the chrome driver path on the Python script. 79 | 80 | On line 35, update `executable_path` with your Chrome driver path 81 | 82 | ```python 83 | executable_path='add_path_here' 84 | ``` 85 | 86 | 87 | ## Usage 88 | 89 | To run the script, we go to the terminal and ensure we are in the correct folder. We type: 90 | 91 | 92 | ```bash 93 | python3 paa.py 94 | ``` 95 | 96 | Once you run the script it will ask you to provide 3 inputs that are necessary. 97 | 98 | 1. **Add Query:** Query that you want to get the answers from. 99 | 100 | **E.g** 101 | 102 | ``` 103 | netflix stock 104 | ``` 105 | 106 | 2. **How many questions do you want to click?:** The more clicks you add the more question you get. Usually, 1 click adds 2 or 3 questions and by default, Google provides 4 questions. So if we add 5 as clicks, then we would get around 15 questions. 107 | 108 | 3. **Please select your language:** I added only 2 languages, English (en) and Spanish (es). Make sure that the language is in lowercase. 109 | 110 | ## Result 111 | 112 | Once you run the script, it will output a df with all the questions that found for the query you requested. The output should look like this: 113 | 114 | **English E.g** 115 | ``` 116 | Questions 117 | 0 Is Netflix a buy or sell? 118 | 1 Why is Netflix stock dropping? 119 | 2 Does Netflix stock pay a dividend? 120 | 3 Is Netflix overvalued? 121 | 4 What stock is best to buy right now? 122 | 5 Is Walmart a good stock to buy? 123 | 6 Is Netflix going broke? 124 | 7 What stocks are up today? 125 | 8 What is the most expensive stock in the world? 126 | 9 What stock pays the best dividend? 127 | 10 Do Amazon pay dividends? 128 | 11 Does Apple stock pay a dividend? 129 | 12 Can you invest in Netflix? 130 | 13 What is the future of Netflix? 131 | ``` 132 | 133 | **Spanish E.g** 134 | ``` 135 | Questions 136 | 0 ¿Cuánto cuesta Netflix al mes? 137 | 1 ¿Cuánto cuesta Netflix al año en España? 138 | 2 ¿Cómo iniciar una sesión de Netflix? 139 | 3 ¿Cuánto cuesta una cuenta en Netflix Venezuela? 140 | 4 ¿Cuánto cuesta Netflix en México 2020? 141 | 5 ¿Qué ofrece Netflix basico? 142 | 6 ¿Cuánto vale Netflix en Colombia 2020? 143 | 7 ¿Cuánto vale Netflix 2020? 144 | 8 ¿Cuánto está Netflix Perú 2020? 145 | 9 ¿Dónde pongo mi código de Netflix? 146 | 10 ¿Cómo hago para tener el mes gratis de Netflix? 147 | 11 ¿Por qué no puedo acceder a Netflix? 148 | ``` 149 | 150 | 151 | ## Issues & Contributing 152 | Pull requests are welcome. If you have any issues please open an issue to see if I can help. 153 | 154 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | selenium==3.141.0 -------------------------------------------------------------------------------- /scripts/.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | -------------------------------------------------------------------------------- /scripts/Old_scripts/files/Disney Stock.csv: -------------------------------------------------------------------------------- 1 | 0,1,2,3,4,5 2 | Is Disney a good stock?,"Disney stock has slid, demand-wise in the past few years, but its conservatively-run business model and steady earnings make it a favorite of value investors who know a stable company when they see one.",,Pros and Cons to Buying Walt Disney Co (DIS) Stock | Stock Market ...,money.usnews.com › stock-market-news › articles › walt-...,Search for: Is Disney a good stock? 3 | Is Disney a buy or sell?,"Disney shares are a buy if they fall to this level, technical analyst says. ... Atlantic Equities also cut earnings estimates but upgraded the stock to overweight after its sharp sell-off.Apr 3, 2020",,"Disney shares are a buy if they fall to this level, technical analyst says",www.cnbc.com › 2020/04/03 › disney-shares-are-a-buy-i...,Search for: Is Disney a buy or sell? 4 | Why is Disney stock dropping?,"Shares of Walt Disney Co. DIS, +4.85% are down 3.1 % in Monday trading after UBS analyst John Hodulik downgraded the stock to neutral from buy, writing that the company is in ""the eye of the storm"" amid the COVID-19 pandemic.Apr 20, 2020",,Disney stock falls after UBS downgrade - MarketWatch,www.marketwatch.com › story › disney-stock-falls-after-...,Search for: Why is Disney stock dropping? 5 | What will Disney stock be worth in 10 years?,"Its current stock price is hovering around $130 a share. And if you invested in the company 10 years ago, you would have made a profit: A $1,000 investment on April 15, 2009, would be worth more than $7,600 as of April 15, 2019, a total return over 660%, according to CNBC calculations.Apr 16, 2019",,"What a $1,000 investment in Disney 10 years ago would be worth now",www.cnbc.com › 2019/04/15 › what-a-1000-dollar-invest...,Search for: What will Disney stock be worth in 10 years? 6 | Is Disney stock a buy today?,,,,, 7 | Does Disney pay dividends?,,,,, 8 | Is it good time to buy Disney stock?,,,,, 9 | Is Disney stock overpriced?,,,,, 10 | Why is Tesla's stock so high?,,,,, 11 | Is Disney stock a buy?,,,,, 12 | What stock is best to invest in right now?,,,,, 13 | Which stocks took the biggest hit today?,,,,, 14 | Are there any benefits to owning Disney stock?,,,,, 15 | ,,,,, 16 | ,,,,, 17 | -------------------------------------------------------------------------------- /scripts/Old_scripts/files/Facebook Stock.csv: -------------------------------------------------------------------------------- 1 | 0,1,2,3,4,5,6,7,8,9,10,11 2 | Will Facebook stock recover?,"For Facebook, the company's stock continues to rise. After climbing over 50% last year, Facebook's stock is already up over 5% so far in 2020. ... The stock largely recovered in the first half of 2019, but plunged again nearly 10% in May, amid calls for the company's breakup.Jan 16, 2020",,Why Facebook's Stock Seems Indestructible - Forbes,www.forbes.com › sites › forbeswealthteam › 2020/01/16,Search for: Will Facebook stock recover?,,,,,, 3 | Why is Facebook stock dropping?,"Facebook Stock Is Dropping Because Analysts Are Realizing Just How Tough the Ad Business Is. As the depth of the global advertising market decline becomes clearer, the Street is slashing estimates for ad-driven internet stocks. Among those seeing earnings forecasts slashed: Facebook.Apr 21, 2020",,Facebook Stock Slides as Analysts Realize Just How Tough Ad ...,www.barrons.com › articles › facebook-stock-slides-as-an...,Search for: Why is Facebook stock dropping?,,,,,, 4 | Is Facebook a buy stock?,"The social media mammoth strives to use its networks to bring the world closer together. , you can buy any dollar amount of Facebook stock, or any other fund or stock you know on Stash. Add a custom amount.",...,Historical stock performance.,Last 12 months -5.36%,Last 3 years 5.45%,Last 5 years 14.96%,Since inception 23.29%,,Buy shares of Facebook (FB) stock - Stash,www.stashinvest.com › investments › stocks › facebook-fb,Search for: Is Facebook a buy stock? 5 | Is Facebook a good stock to buy 2020?,FB Stock Is Not A Buy,"Going into 2020, Wall Street analysts were mostly bullish, betting on stronger Facebook earnings growth. They think this FANG stock still has a long growth runway and provides value to advertisers. Tasty margins might even mean a dividend in Facebook's not-so-distant future.Apr 15, 2020",,Is This FANG Stock A Good Buy? - Investor's Business Daily,www.investors.com › fb-stock-buy-now-facebook-stock,Search for: Is Facebook a good stock to buy 2020?,,,,, 6 | Is FB dying?,,,,,,,,,,, 7 | Can you invest in TikTok?,,,,,,,,,,, 8 | Is Facebook growing or dying?,,,,,,,,,,, 9 | Is Disney stock a buy?,,,,,,,,,,, 10 | Is Facebook stock a good buy?,,,,,,,,,,, 11 | Is now a good time to buy Facebook stock?,,,,,,,,,,, 12 | Can I invest 50 dollars in stocks?,,,,,,,,,,, 13 | Is Facebook a dying platform?,,,,,,,,,,, 14 | Is FB a buy or sell?,,,,,,,,,,, 15 | ,,,,,,,,,,, 16 | ,,,,,,,,,,, 17 | -------------------------------------------------------------------------------- /scripts/Old_scripts/files/Netflix stock.csv: -------------------------------------------------------------------------------- 1 | 0,1,2,3,4,5,6 2 | Is Netflix stock a buy?,Outlook For The Stock In The Current Market Environment,"When all is said and done, Netflix Inc stock turned a corner in 2020. The handle that got tacked on to the 2019 base is really a base in its own right. And the relative strength line is soaring. The S&P 500 fell 35% from its February highs to its March lows.May 1, 2020",,Why Netflix Is A Buy Right Now! - Yahoo Finance,finance.yahoo.com › why-netflix-buy-now-140541024,Search for: Is Netflix stock a buy? 3 | Why is the Netflix stock going down?,"Netflix stock is down, by about 3%, in midday trading Monday because the streaming wars are heating up. ... “While Netflix should remain the dominant global SVOD provider, its 2019 results so far suggest increasing elasticity and/or a rising cost of customer acquisition,” wrote Hargreaves.Dec 2, 2019",,Netflix Stock Is Down Because Streaming Competition Is Up - Barron's,www.barrons.com › articles › netflix-stock-streaming-co...,Search for: Why is the Netflix stock going down?, 4 | What was Netflix original stock price?,"Key Takeaways. Netflix went public on May 23, 2002, with an initial public offering (IPO) price of $15 per share.",,If You Invested Right After Netflix's IPO (NFLX) - Investopedia,www.investopedia.com › articles › active-trading › if-you...,Search for: What was Netflix original stock price?, 5 | Is Netflix stock too expensive?,"Netflix stock is still expensive at a price-to-earnings ratio of around 100, meaning any unanticipated slowdown in subscriber growth or profits could send the stock reeling. So far, early data shows Netflix has seen ""little to no impact"" from the Disney+ launch, according to Credit Suisse.Dec 1, 2019",,Better Buy: Netflix vs. Amazon | The Motley Fool,www.fool.com › investing › 2019/12/01 › better-buy-am...,Search for: Is Netflix stock too expensive?, 6 | What is the best stock to buy now?,,,,,, 7 | Will Netflix stock go up again?,,,,,, 8 | Is now a good time to buy Apple stocks?,,,,,, 9 | Will Netflix stock bounce back?,,,,,, 10 | Is Netflix going out of business?,,,,,, 11 | Why is Netflix stock so high?,,,,,, 12 | What if I bought Netflix stock?,,,,,, 13 | What is the most expensive stock in the world?,,,,,, 14 | Is Amazon still a good buy?,,,,,, 15 | How much would it cost to buy Netflix?,,,,,, 16 | ,,,,,, 17 | -------------------------------------------------------------------------------- /scripts/Old_scripts/files/Tesla Stock.csv: -------------------------------------------------------------------------------- 1 | 0,1,2,3,4,5 2 | Is Tesla good stock to buy?,"Tesla stock has spent about 30% of the time over the past decade trading above the average analyst price target. ... Buying in May 2019 was a good idea too: Tesla shares are up about 130% from their 52-week low. The S&P 500 is down about 14% over the same span.Mar 20, 2020",,This Might Be the Best Time to Buy Tesla Stock Ever - Barron's,www.barrons.com › articles › this-might-be-the-best-time-...,Search for: Is Tesla good stock to buy? 3 | Why is Tesla stock dropping?,"Shares of Tesla dropped as much as 12% Friday before closing down 10.3% after CEO Elon Musk tweeted that the company's shares are priced “too high.” The company just days before reported a strong quarter, sending shares higher.May 1, 2020",,"Tesla CEO Elon Musk says stock price is 'too high,' shares fall",www.cnbc.com › 2020/05/01 › tesla-ceo-elon-musk-says-...,Search for: Why is Tesla stock dropping? 4 | Is Tesla stock a buy now?,"Tesla is looking like a good deal to Jefferies. The firm on Monday upgraded shares of the automaker to ""buy"" from ""hold"" while lowering its price target to $650 from $800. That implies a 35% upside from where shares closed at $480 per share on Friday.Apr 6, 2020",,"Jefferies says now is the time to buy Tesla, which it says could surge ...",markets.businessinsider.com › news › stocks › tesla-stock-...,Search for: Is Tesla stock a buy now? 5 | Why is Tesla's stock so high?,"Tesla shares, which had closed Thursday at $781.88, an 82% gain for the year, were down more than 9% in midday trading Friday after the Twitter messages. ... Musk's ambitious growth plans for the electric car maker has sent shares soaring this year even amid fears of a prolonged recession from the coronavirus pandemic.May 1, 2020",,Elon Musk Tweeted That Tesla's Stock Was Too High. The Market ...,www.wsj.com › articles › tesla-stock-falls-after-ceo-tweets...,Search for: Why is Tesla's stock so high? 6 | Is Tesla overvalued?,,,,, 7 | What will Tesla stock be worth in 5 years?,,,,, 8 | Is Tesla stock going to keep dropping?,,,,, 9 | Will Tesla hit 1000?,,,,, 10 | Is Tesla stock a buy?,,,,, 11 | Is Tesla stock expected to rise?,,,,, 12 | Will Tesla pay dividends?,,,,, 13 | Is Tesla a good stock to buy 2020?,,,,, 14 | Is Tesla going to fail?,,,,, 15 | Is Tesla stock a bubble?,,,,, 16 | ,,,,, 17 | -------------------------------------------------------------------------------- /scripts/Old_scripts/files/Uber Stock.csv: -------------------------------------------------------------------------------- 1 | 0,1,2,3,4,5 2 | Is Uber a good investment?,"Uber is a quintessential high-risk, high-reward tech stock. However, the long-run upside is significant enough to make the investment worthwhile.Mar 5, 2020",,Here's Why Uber Stock Is a Buy and Hold for the Long-Term Investor ...,www.fool.com › investing › 2020/03/05 › heres-why-ube...,Search for: Is Uber a good investment? 3 | Is Uber a good stock to buy in 2020?,"With market demand expanding and Uber coming up with creative ways to survive regulatory barriers, it's easy to see why Uber is a hot stock in 2020. There will always be enhanced risks when investing in IPOs, but for now, Uber appears ready to deliver enhanced rewards, too.Jan 22, 2020",,3 Reasons Uber Stock Is Winning in 2020 | Nasdaq,www.nasdaq.com › articles › 3-reasons-uber-stock-is-win...,Search for: Is Uber a good stock to buy in 2020? 4 | How can I buy stock in Uber?,"For most people, the broker's website platform is the easiest and most user-friendly option. Open an order ticket on your broker's website, enter the number of shares you want to purchase and Uber's stock symbol — conveniently, UBER. That will bring up detailed information about the stock and its current price.",,How to Buy Uber Stock - NerdWallet,www.nerdwallet.com › blog › investing › how-to-buy-ub...,Search for: How can I buy stock in Uber? 5 | Why is uber stock up today?,"Shares of Uber and Lyft skyrocketed Thursday after Uber said in an investor call that it has plenty of cash to get through the coronavirus crisis and is seeing growth in other areas of the business as rides fall dramatically as people stay home. Uber stock was up as much as 40% Thursday following the call.Mar 19, 2020",,Uber stock pops after CEO says it has cash to get through coronavirus,www.cnbc.com › 2020/03/19 › uber-stock-pops-after-say...,Search for: Why is uber stock up today? 6 | Why is uber losing money?,,,,, 7 | Why is uber stock so low?,,,,, 8 | Is Uber going broke?,,,,, 9 | What is best stock to buy right now?,,,,, 10 | Is Uber stock a buy or sell?,,,,, 11 | Will Uber ever be profitable?,,,,, 12 | Should I buy pre IPO stock?,,,,, 13 | Why is uber going public?,,,,, 14 | Is Tesla a good buy?,,,,, 15 | ,,,,, 16 | ,,,,, 17 | -------------------------------------------------------------------------------- /scripts/Old_scripts/paa-q&A.py: -------------------------------------------------------------------------------- 1 | from selenium import webdriver 2 | from selenium.webdriver.common.action_chains import ActionChains 3 | from selenium.webdriver.common.keys import Keys 4 | from selenium.webdriver.firefox.options import Options 5 | 6 | from tqdm import tqdm 7 | from time import sleep 8 | 9 | import itertools 10 | import threading 11 | import time 12 | import sys 13 | import csv 14 | import pandas as pd 15 | 16 | # query = "uber eats" Not being used 17 | 18 | clicks = 4 19 | clicks = int(clicks) # parse string into an integer 20 | 21 | 22 | 23 | #Search Query and get results 24 | 25 | options = Options() 26 | options.headless = True 27 | 28 | def search(query,clicks): 29 | with webdriver.Firefox(options=options) as driver: 30 | driver.get("https://www.google.com?hl=en") 31 | driver.find_element_by_xpath("//input[@aria-label='Search']").send_keys(query) 32 | driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div[1]/div[3]/center/input[1]") 33 | searchbtn = driver.find_elements_by_xpath("//input[@aria-label='Google Search']") 34 | searchbtn[-1].click() 35 | 36 | #Questions with answers. Have to clean a little bit. 37 | 38 | paa = driver.find_elements_by_css_selector('div.related-question-pair') 39 | for i in range(clicks): 40 | try: 41 | paa[i].click() 42 | paa = driver.find_elements_by_css_selector('div.related-question-pair') 43 | except IndexError as k: 44 | print('There are no questions to Click! Index is out of Range. Please add another Keyword that contains questions') 45 | pass 46 | list_paa = [] 47 | for j in paa: 48 | p = format(j.text) 49 | p = p.splitlines() 50 | 51 | print(p) 52 | list_paa.append(p) 53 | 54 | 55 | df = pd.DataFrame(list_paa) 56 | 57 | df.to_csv( query + '.csv', index=False) 58 | 59 | 60 | 61 | """ File where we will save all keywords we want to run. Make sure the file is .xlsx and follows always the same format""" 62 | df = pd.read_excel (r'keywords.xlsx') 63 | 64 | 65 | import time 66 | from random import randint 67 | 68 | for i in df['Keywords']: 69 | search(i,clicks) 70 | 71 | #Random sleep so that we dont get Google Captcha 72 | sleep = randint(1,20) 73 | print('Sleep time is', sleep) 74 | 75 | #Counter to show in terminal 76 | for k in range(sleep): 77 | sys.stdout.write(str(k)+' ') 78 | sys.stdout.flush() 79 | time.sleep(1) 80 | 81 | """ Function that puts together all the CSV files into one big master file 'Biglist.csv'""" 82 | def csv_concat(): 83 | import os, glob 84 | import pandas as pd 85 | 86 | #Files that we want to run to get totals 87 | files = sorted(glob.glob('*.csv')) 88 | 89 | #opening all files and selecting 4 top rows and onoy 5 columns 90 | big_df = [] 91 | for f in files: 92 | #try & Except so that we can keep going through the loop even if file has no columns. 93 | try: 94 | df = pd.read_csv(f,index_col=False) 95 | print(df) 96 | d = df.iloc[0:4,0:6] 97 | big_df.append(d) 98 | except: 99 | continue 100 | 101 | #merging all into one DF 102 | merged = pd.concat(big_df) 103 | 104 | merged.columns = ['Question','Answer','Answer 2', 105 | 'Title of page','URL','Search for question'] 106 | 107 | 108 | #Exporting file into CSV 109 | merged.to_csv('Biglist.csv', index=False) 110 | 111 | csv_concat() 112 | 113 | 114 | -------------------------------------------------------------------------------- /scripts/Old_scripts/paa.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | Created on Thu Apr 30 23:28:11 2020 5 | 6 | @author: konradburchardt 7 | """ 8 | 9 | 10 | from selenium import webdriver 11 | from selenium.webdriver.common.action_chains import ActionChains 12 | from selenium.webdriver.common.keys import Keys 13 | from selenium.webdriver.firefox.options import Options 14 | import pandas as pd 15 | from random import randint 16 | import time 17 | import sys 18 | 19 | 20 | # Variables that user has to input 21 | 22 | query = input("Add your query: ") 23 | clicks = input("How many questions do you want to click? : ") 24 | lang = input("Please select your languange. (For english type en. For spanish type es. For French type fr) : ") 25 | clicks = int(clicks) # parse string into an integer 26 | 27 | 28 | #headless option 29 | options = Options() 30 | options.headless = True 31 | 32 | """ Search function. It opens Google, adds query in search box clicks search. Then it looks for question box and clicks N times 33 | each of the questions and prints them out. The more question it clicks the more answers we get""" 34 | 35 | def search(query,clicks,lang): 36 | with webdriver.Firefox(options=options) as driver: 37 | driver.get("https://www.google.com?hl=" + lang) 38 | 39 | #English 40 | if lang == "en": 41 | driver.find_element_by_xpath("//input[@aria-label='Search']").send_keys(query) 42 | driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div[1]/div[3]/center/input[1]") 43 | searchbtn = driver.find_elements_by_xpath("//input[@aria-label='Google Search']") 44 | searchbtn[-1].click() 45 | 46 | #printing query 47 | print("Query is: ", query) 48 | 49 | #clicking questions 50 | clickingKW(clicks,driver) 51 | 52 | #Spanish 53 | if lang == "es": 54 | driver.find_element_by_xpath("//input[@aria-label='Buscar']").send_keys(query) 55 | driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div[1]/div[3]/center/input[1]") 56 | searchbtn = driver.find_elements_by_xpath("//input[@aria-label='Buscar con Google']") 57 | searchbtn[-1].click() 58 | 59 | #printing query 60 | print("Palabra clave es: ", query) 61 | 62 | #clicking questions 63 | clickingKW(clicks,driver) 64 | 65 | #French 66 | if lang == "fr": 67 | driver.find_element_by_xpath("//input[@aria-label='Rech.']").send_keys(query) 68 | driver.find_elements_by_xpath("/html/body/div/div[3]/form/div[2]/div[1]/div[3]/center/input[1]") 69 | searchbtn = driver.find_elements_by_xpath("//input[@aria-label='Recherche Google']") 70 | searchbtn[-1].click() 71 | 72 | #Printing Query 73 | print("la requête est: ", query) 74 | 75 | #clicking questions 76 | clickingKW(clicks,driver) 77 | 78 | """Function that clicks N Questions from here we need to figure out how to click each question and save parent question and childrens""" 79 | 80 | def clickingKW(clicks,driver): 81 | paa = driver.find_elements_by_xpath("//span/following-sibling::div[contains(@class,'match-mod-horizontal-padding')]") 82 | #Its range because clicks is int. 83 | for i in range(clicks): 84 | print(i) 85 | try: 86 | paa[i].click() 87 | paa = driver.find_elements_by_xpath("//span/following-sibling::div[contains(@class,'match-mod-horizontal-padding')]") 88 | for j in paa: 89 | print(format(j.text)) 90 | except IndexError as k: 91 | raise Exception('There are no questions to Click! Index is out of Range. Please add another Keyword that contains questions') 92 | 93 | 94 | """ File where we will save all keywords we want to run. Make sure the file is .xlsx and follows always the same format""" 95 | df = pd.read_excel (r'keywords.xlsx') 96 | 97 | print('Your list of Keywords is:\n',df) 98 | 99 | search(query,clicks,lang) 100 | 101 | """ Loops that runs all the keywords from our File. This has a Random sleep time between 1 and 20 seconds 102 | so that google doesnt give us a captcha if we do a high number of keywords""" 103 | 104 | # for i in df['Keywords']: 105 | # search(i,clicks,lang) 106 | 107 | # #Random sleep so that we dont get Google Captcha 108 | # sleep = randint(1,20) 109 | # print('Sleep time is', sleep) 110 | 111 | # #Counter to show in terminal 112 | # for k in range(sleep): 113 | # sys.stdout.write(str(k)+' ') 114 | # sys.stdout.flush() 115 | # time.sleep(1) 116 | -------------------------------------------------------------------------------- /scripts/Old_scripts/readme.md: -------------------------------------------------------------------------------- 1 | 2 | # People Also Ask Google Scraper 3 | 4 | PPA scraper is a Python script that based on a query it scrapes Google people also ask questions. 5 | You can see this script in action on https://simpletools.io 6 | 7 | 8 | Note: *This is a work in Progress* 9 | 10 | ## Installation 11 | 12 | To install and run this script we need to install our requirements.txt. Since we are using Firefox we need to download the latest geckodriver driver. 13 | 14 | 15 | ### Requirements 16 | 17 | To install requirements.txt please run the following on the terminal. 18 | 19 | ```bash 20 | pip install requirements.txt 21 | ``` 22 | 23 | ### Geckodriver 24 | 25 | You can find it the latest firefox driver [here](https://github.com/mozilla/geckodriver/releases). 26 | 27 | Once we download the driver you can save it in the same folder than the script is /people-also-ask/ and we can do two things to make sure it runs: 28 | 29 | 1. Add executable_path arg in webdriver: 30 | 31 | ```bash 32 | 33 | with webdriver.Firefox(executable_path='/path/to/geckodriver') as driver: 34 | 35 | ``` 36 | 37 | 2.The second way its to add folder that contains geckodriver using export (only folder, not geckodriver). We go to the terminal and type: 38 | 39 | ```bash 40 | 41 | $ export PATH=$PATH:/path/to/ 42 | 43 | ``` 44 | 45 | 3.- (**RECOMENNDED**) Make geckodriver available system wide, open up your Terminal App and perform the following command: 46 | 47 | ``` 48 | brew install geckodriver 49 | ``` 50 | 51 | ## Usage 52 | 53 | Once we install the requirements and the driver, we can go ahead and run our script. There are two scripts that have similar outputs. 54 | 55 | - **paa.py** : Outputs only questions. 56 | 57 | - **paa-q&A.py** : Outputs questions & answers. *Still work in progress out put is very messy* 58 | 59 | 60 | To run the script we go on the terminal and make sure we are in the correct folder. We type: 61 | 62 | 63 | ```bash 64 | python3 paa.py 65 | ``` 66 | 67 | Once you run the script it will ask you to provide 3 inputs that are necessary. 68 | 69 | 1. **Add Query:** Query that you want to get the answers from. 70 | 71 | **E.g** 72 | 73 | ``` 74 | netflix stock 75 | ``` 76 | 77 | 2. **How many questions do you want to click?:** The more clicks you add the more question you get. Usually one clicks adds 2 or 3 questions and by default Google provides 4 questions. So if we add 5 as clicks then we would get around 15 questions. 78 | 79 | 3. **Please select your languange:** I added only 3 languages, English (en) , Spanish (es) and French (fr). Make sure that the language is in lowcaps. 80 | 81 | ## Result 82 | 83 | Once you get allthe parameters the script will go and get all the questions for you. Then your output will consist of the question index that its being clicked, and the questions. 84 | 85 | **E.g** 86 | ``` 87 | 0 88 | Is Netflix a good stock to buy? 89 | Why is the Netflix stock going down? 90 | Will Netflix stock go up? 91 | What was Netflix original stock price? 92 | 93 | 94 | 1 95 | Is Netflix a good stock to buy? 96 | Why is the Netflix stock going down? 97 | Will Netflix stock go up? 98 | What was Netflix original stock price? 99 | Should I buy Tesla stock now? 100 | What is the best stock to buy right now? 101 | 102 | 2 103 | Is Netflix a good stock to buy? 104 | Why is the Netflix stock going down? 105 | Will Netflix stock go up? 106 | What was Netflix original stock price? 107 | Should I buy Tesla stock now? 108 | What is the best stock to buy right now? 109 | Is Disney stock a buy? 110 | Is Netflix a buy or sell? 111 | ``` 112 | 113 | ## Issues & Contributing 114 | Pull requests are welcome. If you have any issues please open an issue to see if I can help. 115 | 116 | -------------------------------------------------------------------------------- /scripts/netflix.csv: -------------------------------------------------------------------------------- 1 | Questions 2 | How do you get Netflix for free? 3 | How much does it cost for Netflix for a month? 4 | Is Netflix free now? 5 | How much is Netflix free? 6 | What is the cheapest way to get Netflix? 7 | How much is Netflix 2022 monthly? 8 | What are the 3 Netflix plans? 9 | Are Netflix prices going up 2022? 10 | Which Netflix plan is the best? 11 | Is Netflix free with Amazon Prime? 12 | Is Netflix free on Roku? 13 | Do I have to pay for Netflix on a Smart TV? 14 | -------------------------------------------------------------------------------- /scripts/paa.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | """ 4 | @author: konradburchardt 5 | """ 6 | 7 | import undetected_chromedriver as uc 8 | import pandas as pd 9 | import time 10 | import random 11 | from selenium.webdriver.common.by import By 12 | from selenium.webdriver.common.keys import Keys 13 | 14 | def get_stealth_driver(): 15 | """Set up an undetected Chrome WebDriver to avoid Google bot detection.""" 16 | 17 | options = uc.ChromeOptions() 18 | options.headless = False # Set to True for headless mode 19 | options.add_argument("--disable-blink-features=AutomationControlled") # Prevent bot detection 20 | options.add_argument("--disable-popup-blocking") 21 | 22 | driver = uc.Chrome(options=options) 23 | 24 | print("✅ Stealth Chrome Driver Ready") 25 | return driver 26 | 27 | def search(query, clicks, lang): 28 | """ 29 | Opens Google, enters a query, clicks PAA questions, and extracts answers. 30 | Parameters: 31 | query (str): The search query 32 | clicks (int): Number of PAA questions to click 33 | lang (str): Language code ('en' or 'es') 34 | """ 35 | 36 | driver = get_stealth_driver() 37 | 38 | try: 39 | # Step 1: Open Google 40 | google_url = f"https://www.google.com?hl={lang}" 41 | print(f"🔎 Opening Google: {google_url}") 42 | driver.get(google_url) 43 | time.sleep(random.uniform(5, 8)) # Human-like delay 44 | 45 | # Step 2: Find search bar and enter query 46 | search_box = driver.find_element(By.NAME, "q") 47 | search_box.send_keys(query) 48 | search_box.send_keys(Keys.RETURN) 49 | 50 | # Wait for results to load 51 | time.sleep(random.uniform(6, 10)) 52 | 53 | # Click and extract PAA questions and answers 54 | extract_paa_questions(driver, clicks) 55 | 56 | except Exception as e: 57 | print(f"❌ Error: {str(e)}") 58 | 59 | finally: 60 | driver.quit() 61 | 62 | def extract_paa_questions(driver, clicks): 63 | """ 64 | Clicks 'People Also Ask' (PAA) questions in order (top-to-bottom) 65 | and extracts all expanded questions. 66 | Parameters: 67 | driver: Selenium WebDriver instance 68 | clicks (int): Number of PAA questions to click 69 | """ 70 | try: 71 | paa_data = set() # Store unique questions 72 | clicked_count = 0 # Track how many questions were clicked 73 | 74 | while clicked_count < clicks: 75 | # Get all PAA questions currently available 76 | paa_questions = driver.find_elements(By.XPATH, "//div[contains(@jsname,'N760b')]//div[contains(@jsname,'tJHJj')]") 77 | 78 | if clicked_count >= len(paa_questions): 79 | print("⚠️ No more new questions available to click.") 80 | break # Stop if we run out of questions 81 | 82 | # Click the next available question (top-to-bottom) 83 | try: 84 | question = paa_questions[clicked_count] # Select the question in order 85 | question_text = question.text.strip() 86 | 87 | if question_text and question_text not in paa_data: 88 | print(f"🔹 Clicking question #{clicked_count+1}: {question_text}") 89 | driver.execute_script("arguments[0].click();", question) 90 | time.sleep(random.uniform(2, 3)) # Delay to simulate human behavior 91 | 92 | paa_data.add(question_text) # Store question to avoid duplicates 93 | clicked_count += 1 # Increment click count 94 | 95 | # Wait and refresh the list after clicking 96 | time.sleep(2) 97 | 98 | except Exception as e: 99 | print(f"⚠️ Skipping question due to error: {str(e)}") 100 | break # Stop clicking if an error occurs 101 | 102 | # Step 3: Extract all expanded questions (including new ones added) 103 | time.sleep(3) # Allow new questions to load 104 | all_visible_questions = driver.find_elements(By.XPATH, "//div[contains(@jsname,'N760b')]//div[contains(@jsname,'tJHJj')]") 105 | 106 | for q in all_visible_questions: 107 | q_text = q.text.strip() 108 | if q_text: 109 | paa_data.add(q_text) 110 | 111 | # Convert to DataFrame and save results 112 | df = pd.DataFrame(sorted(paa_data), columns=["Questions"]) 113 | df.to_csv("paa_results.csv", index=False, encoding="utf-8") 114 | print("\n✅ Saved results to 'paa_results.csv'") 115 | print(df) 116 | 117 | except Exception as e: 118 | print(f"❌ Error clicking PAA questions: {str(e)}") 119 | 120 | # Get user input 121 | query = input("Add your query: ") 122 | clicks = int(input("How many questions do you want to click?: ")) 123 | lang = input("Please select your language (en/es): ") 124 | 125 | # Run search 126 | search(query, clicks, lang) 127 | --------------------------------------------------------------------------------