├── CONTRIBUTING.md └── README.md /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contribution Guidelines 2 | 3 | Thank you for considering contributing to our projects! We appreciate your time and effort and are excited to collaborate with you. 4 | 5 | ## How to Contribute 6 | 7 | ### 1. Reporting Issues 8 | If you find a bug or have a feature request, feel free to create a new issue with a clear and descriptive title. 9 | 10 | ### 2. Fork the Repository 11 | Before you start working on a new feature or bug fix, fork the repository and create a new branch for your work. Make sure your branch name is descriptive, such as `feature/add-authentication` or `bugfix/fix-login-error`. 12 | 13 | ### 3. Make Your Changes 14 | Ensure your changes follow the project's coding standards and best practices. Test your code thoroughly before submitting it. 15 | 16 | ### 4. Submit a Pull Request 17 | Once your changes are ready, submit a pull request (PR) to the main branch of the repository. In your PR description, explain what changes you made and why. If your PR addresses an open issue, please link to the issue. 18 | 19 | ### 5. Code Review 20 | Your PR will be reviewed by one of the project maintainers. Please be responsive to any feedback and make necessary adjustments to your code. 21 | 22 | ### 6. Join the Conversation 23 | We encourage you to join our [Discord server](https://discord.gg/GbxmdGhZjq) to discuss your ideas, get help, and connect with other contributors. It's a great place to collaborate and learn from others in the community. 24 | 25 | ## Coding Standards 26 | 27 | - Write clear, concise, and descriptive commit messages. 28 | - Ensure your code is well-documented, especially for complex functions or modules. 29 | - Include unit tests for any new functionality. 30 | 31 | ## Getting Help 32 | 33 | If you need help or have any questions about contributing, don't hesitate to ask in our [Discord server](https://discord.gg/GbxmdGhZjq). 34 | 35 | Thank you for helping us improve our projects! 36 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Welcome to Oxylabs on GitHub 2 | 3 | [![Oxylabs promo code](https://raw.githubusercontent.com/oxylabs/product-integrations/refs/heads/master/Affiliate-Universal-1090x275.png)](https://oxylabs.io/pages/gitoxy?utm_source=877&utm_medium=affiliate&groupid=877&utm_content=oxylabs-readme-github&transaction_id=102f49063ab94276ae8f116d224b67) 4 | 5 | [![](https://dcbadge.vercel.app/api/server/eWsVUJrnG5)](https://discord.gg/Pds3gBmKMH) 6 | 7 | We provide premium proxy services and web scraping solutions, trusted by developers worldwide. Our GitHub organisation offers tools, SDKs, and tutorials. Explore our repositories' collections: 8 | 9 | ## 🔧 Open-Source projects 10 | 11 | Open-source frameworks designed to facilitate your web scraping projecs. 12 | 13 | - **[OxyParser](https://github.com/oxylabs/OxyParser):** Oxy®Parser simplifies HTML parsing by leveraging LLMs and Pydantic models. Define your HTML structure with Pydantic, and let OxyParser automatically transform HTML into structured data models. 14 | - **[OxyMouse](https://github.com/oxylabs/OxyMouse):** Generates realistic mouse movement paths using advanced algorithms. 15 | - **[Web Scraper API Scheduler](https://github.com/oxylabs/Oxylabs-Web-Scraper-API-Scheduler/):** Quickly start and test Scraper API payloads with basic programming knowledge. 16 | 17 | 18 | ## 📦 SDKs and integrations 19 | 20 | SDKs and code samples to help you integrate Oxylabs' solutions into your projects. 21 | 22 | - **[oxylabs-sdk-go](https://github.com/oxylabs/oxylabs-sdk-go):** Go SDK for Oxylabs' Scraper API. 23 | - **[oxylabs-sdk-python](https://github.com/oxylabs/oxylabs-sdk-python):** Python SDK for working with Oxylabs' API. 24 | - **[product-integrations](https://github.com/oxylabs/product-integrations):** Oxylabs product integration tutorials. 25 | - **[proxy-integrations](https://github.com/oxylabs/proxy-integrations):** Proxy integration tutorials. 26 | - **[selenium-proxy-integration-python](https://github.com/oxylabs/selenium-proxy-integration-python):** Residential Proxies' integration with Selenium in Python. 27 | - **[selenium-proxy-integration-java](https://github.com/oxylabs/selenium-proxy-integration-java):** Residential Proxies' integration with Selenium using Java. 28 | - **[puppeteer-proxy-integration-js](https://github.com/oxylabs/puppeteer-proxy-integration-js):** Proxies' integration with Puppeteer using JavaScript. 29 | - **[playwright-proxy-integration-js](https://github.com/oxylabs/playwright-proxy-integration-js):** Residential and Datacenter proxies' integration with Playwright using JavaScript. 30 | - **[mechanicalsoup-proxy-integration](https://github.com/oxylabs/mechanicalsoup-proxy-integration):** Python tutorial for integrating Residential Proxies with MechanicalSoup. 31 | - **[aiohttp-proxy-integration](https://github.com/oxylabs/aiohttp-proxy-integration):** Python tutorial for integrating Residential Proxies with AIOHTTP. 32 | - **[proxy-integration-with-parsehub](https://github.com/oxylabs/proxy-integration-with-parsehub):** Proxies' integration with ParseHub. 33 | - **[integration-python-requests](https://github.com/oxylabs/integration-python-requests):** Residential and Datacenter Proxies' integration with Python Requests. 34 | - **[integration-with-node-fetch](https://github.com/oxylabs/proxies-with-node-fetch):** Node-Fetch integration with proxies. 35 | 36 | ## 📚 Tutorials and guides 37 | 38 | Tutorials and guides to help you get started with web scraping (organized by language). 39 | 40 | ### Python 41 | 42 | - **[web-scraping-selenium-python](https://github.com/oxylabs/web-scraping-selenium-python):** Web Scraping with Python Selenium for beginners. 43 | - **[seo-monitoring](https://github.com/oxylabs/seo-monitoring):** SEO Monitoring System with Python, Celery, and a SERP Scraper API (now a part of Web Scraper API). 44 | - **[scrapy-web-scraping](https://github.com/oxylabs/scrapy-web-scraping):** Tutorial for web scraping with Scrapy. 45 | - **[python-cache-tutorial](https://github.com/oxylabs/python-cache-tutorial):** Tutorial to caching web scraping scripts in Python. 46 | - **[playwright-web-scraping](https://github.com/oxylabs/playwright-web-scraping):** Web scraping using Playwright headless browser. 47 | - **[how-to-make-web-scraping-faster](https://github.com/oxylabs/how-to-make-web-scraping-faster):** Tips to make web scraping faster with Python. 48 | - **[automate-competitors-benchmark-analysis](https://github.com/oxylabs/automate-competitors-benchmark-analysis):** Tutorial to automating competitors’ & benchmark analysis. 49 | - **[asynchronous-web-scraping-python](https://github.com/oxylabs/asynchronous-web-scraping-python):** A comparison of asynchronous and synchronous web scraping methods. 50 | - **[Python-Web-Scraping-Tutorial](https://github.com/oxylabs/Python-Web-Scraping-Tutorial):** Guide to Python web scraping. 51 | - **[oxycon2021-active-fingerprinting-pyppeteer](https://github.com/oxylabs/oxycon2021-active-fingerprinting-pyppeteer):** Tutorial for active fingerprinting with Pyppeteer. 52 | - **[best-buy-price-tracker](https://github.com/oxylabs/best-buy-price-tracker):** Scalable Best Buy price tracker tutorial. 53 | - **[walmart-price-tracker](https://github.com/oxylabs/walmart-price-tracker):** Walmart price tracker tutorial. 54 | 55 | ### JavaScript 56 | 57 | - **[puppeteer-tutorial](https://github.com/oxylabs/puppeteer-tutorial):** Web scraping with Puppeteer tutorial. 58 | - **[crawlee-web-scraping-tutorial](https://github.com/oxylabs/crawlee-web-scraping-tutorial):** Web scraping using Crawlee tutorial. 59 | 60 | ### Ruby 61 | 62 | - **[webscraping-with-ruby](https://github.com/oxylabs/webscraping-with-ruby):** Web scraping with Ruby. 63 | 64 | ### R 65 | 66 | - **[web-scraping-r](https://github.com/oxylabs/web-scraping-r):** Web scraping with R. 67 | 68 | ### PHP 69 | 70 | - **[web-scraping-php](https://github.com/oxylabs/web-scraping-php):** Web scraping with PHP. 71 | - **[guzzle-proxy](https://github.com/oxylabs/guzzle-proxy):** Proxies' setup in Guzzle for PHP. 72 | 73 | ### PowerShell 74 | 75 | - **[web-scraping-powershell](https://github.com/oxylabs/web-scraping-powershell):** Tutorial on using PowerShell and PowerHTML for web scraping. 76 | 77 | ## 🛠️ Google scraping solutions 78 | 79 | Solutions and tools for scraping Google pages. 80 | 81 | - **[google-scraper](https://github.com/oxylabs/scrape-google-python):** Scrape Google data with Python. 82 | - **[google-images-scraper](https://github.com/oxylabs/how-to-scrape-google-images):** Scrape Google images in real time. 83 | - **[google-shopping-scraper](https://github.com/oxylabs/scrape-google-shopping):** Get real-time data from Google Shopping. 84 | - **[google-maps-scraper](https://github.com/oxylabs/google-maps-scraper):** Scrape Google Maps for data from various Google Maps listings. 85 | - **[google-trends-scraper](https://github.com/oxylabs/how-to-scrape-google-trends):** Scrape Google Trends data and make a result comparison using Python. 86 | - **[google-flights-scraper](https://github.com/oxylabs/how-to-scrape-google-flights):** Scrape flight data from Google Flights and analyze flight prices with Python. 87 | - **[google-news-scraper](https://github.com/oxylabs/google-news-scraper):** Use Google News API to get the latest global news. 88 | - **[google-maps-scraper](https://github.com/oxylabs/google-maps-scraper):** Scrape Google Maps for data from various Google Maps listings. 89 | - **[google-play-scraper](https://github.com/oxylabs/google-play-scraper):** Scrape Google Play and collect public details about applications, and more. 90 | - **[google-scholar-scraper](https://github.com/oxylabs/how-to-scrape-google-scholar):** Extract titles, authors, and citations from Google Scholar using Python. 91 | - **[google-jobs-scraper](https://github.com/oxylabs/how-to-scrape-google-jobs):** Build your own Google Jobs scraper with Python. 92 | 93 | ## 🛍️ E-commerce scraping tutorials 94 | 95 | Specialized scrapers for popular e-commerce platforms. 96 | 97 | - **[quick-start-guide](https://github.com/oxylabs/quick-start-guide/blob/main/Guides/E-Commerce%20Websites.md):** A tutorial on using Oxylabs' Web Scraper API for E-Commerce targets. 98 | - **[amazon-asin-scraper](https://github.com/oxylabs/amazon-asin-scraper):** Extract product ASINs from Amazon. 99 | - **[bypass-amazon-captcha](https://github.com/oxylabs/how-to-bypass-amazon-captcha):** Bypass CAPTCHA when extracting public data from Amazon. 100 | - **[amazon-review-scraper](https://github.com/oxylabs/amazon-review-scraper):** Extract Amazonreview data. 101 | - **[ebay-scraper](https://github.com/oxylabs/ebay-scraper):** Extract product details from eBay. 102 | - **[alibaba-scraper](https://github.com/oxylabs/alibaba-scraper):** Extract data from Alibaba for e-commerce analysis. 103 | - **[walmart-scraper](https://github.com/oxylabs/walmart-scraper):** Scrape product information from Walmart. 104 | - **[wayfair-scraper](https://github.com/oxylabs/wayfair-scraper):** Extract titles, prices, reviews Wayfair. 105 | - **[zara-scraper](https://github.com/oxylabs/zara-scraper):** Extract data from Zara's online store. 106 | - **[redbubble-scraper](https://github.com/oxylabs/redbubble-scraper):** Extract data from the Redbubble platform. 107 | - **[aliexpress-scraper](https://github.com/oxylabs/aliexpress-scraper):** Extract prices, product details, and ratings. 108 | - **[priceline-scraper](https://github.com/oxylabs/priceline-scraper):** Get Priceline data. 109 | - **[shein-scraper](https://github.com/oxylabs/shein-scraper):** Extract product data from Shein. 110 | - **[macys-scraper](https://github.com/oxylabs/macys-scraper):** Extract data from the Macy's ecommerce platform. 111 | - **[opensea-scraper](https://github.com/oxylabs/opensea-scraper):** Get NFT's data. 112 | - **[azada-scraper](https://github.com/oxylabs/lazada-scraper):** Get product data from the Lazada e-commerce platform. 113 | - **[etsy-scraper](https://github.com/oxylabs/etsy-scraper):** Extract data from Etsy. 114 | - **[adidas-scraper](https://github.com/oxylabs/adidas-scraper):** Extract data from Adidas' online platforms. 115 | - **[1688-scraper](https://github.com/oxylabs/1688-scraper):** Extract data from the 1688 e-commerce platform. 116 | - **[xbox-scraper](https://github.com/oxylabs/xbox-scraper):** Extract game data from Xbox. 117 | - **[biglots-scraper](https://github.com/oxylabs/biglots-scraper):** Get data from Biglots's retailers' websites. 118 | - **[depop-scraper](https://github.com/oxylabs/depop-scraper):** Extract Depop's online shopping data. 119 | - **[gamestop-scraper](https://github.com/oxylabs/gamestop-scraper):** Extract data from Gamestop's site. 120 | - **[worten-scraper](https://github.com/oxylabs/worten-scraper):** Extract product data from Worten. 121 | - **[digikey-scraper](https://github.com/oxylabs/digikey-scraper):** Extract electronic component data from the Digikey. 122 | - **[bunnings-scraper](https://github.com/oxylabs/bunnings-scraper):** Extract data from Bunnings' website. 123 | 124 | ## 🌐 Other websites 125 | 126 | Specialized scrapers for various websites. 127 | 128 | - **[indeed-scraper](https://github.com/oxylabs/how-to-scrape-indeed):** Extract job postings from Indeed. 129 | - **[redfin-scraper](https://github.com/oxylabs/scraping-real-estate-data-with-python):** Extract real estate data from Redfin. 130 | - **[youtube-scraper](https://github.com/oxylabs/youtube-scraper):** Extract public YouTube data. 131 | - **[expedia-scraper](https://github.com/oxylabs/expedia-scraper):** Extract data from the Expedia website. 132 | - **[trip-scraper](https://github.com/oxylabs/trip-scraper):** Extract travel data from various sources globally. 133 | - **[trivago-scraper](https://github.com/oxylabs/trivago-scraper):** Get real-time data from Trivago. 134 | - **[craigslist-scraper](https://github.com/oxylabs/craigslist-scraper):** Gather public Craigslist data. 135 | 136 | ## 📝 Data parsing and processing 137 | 138 | Guides and tools for parsing and processing scraped data. 139 | 140 | - **[parse-html-pyquery](https://github.com/oxylabs/parse-html-pyquery):** Tutorial for parsing HTML with PyQuery in Python. 141 | - **[beautiful-soup-parsing-tutorial](https://github.com/oxylabs/beautiful-soup-parsing-tutorial):** Tutorial for using Beautiful Soup to parse data in Python. 142 | - **[pandas-read-html](https://github.com/oxylabs/pandas-read-html):** Tutorial for reading HTML tables with pandas. 143 | - **[date-parser-tutorial](https://github.com/oxylabs/Date-Parser-Tutorial):** Tutorial for parsing dates in Python. 144 | - **[parse-xml-in-python](https://github.com/oxylabs/how-to-parse-xml-in-python):** Tutorial for parsing XML data in Python. 145 | - **[price-parsing-tutorial](https://github.com/oxylabs/Price-Parsing-Tutorial):** Tutorial for parsing prices in Python. 146 | - **[python-parse-json](https://github.com/oxylabs/python-parse-json):** Tutorial for parsing JSON data with Python. 147 | - **[custom-parser-instructions](https://github.com/oxylabs/custom-parser-instructions):** Tutorial for writing parsing instructions with Oxylabs' Custom Parser. 148 | 149 | ## 🔒 Proxy solutions 150 | 151 | Guides and tools for setting up and using proxies. 152 | 153 | - **[proxy-manager-app](https://github.com/oxylabs/proxy-manager-app):** Tutorial for proxy manager app. 154 | - **[proxy-chrome-extension](https://github.com/oxylabs/proxy-chrome-extension):** Tutorial for Proxy Chrome extension. 155 | - **[curl-with-proxy](https://github.com/oxylabs/curl-with-proxy):** A tutorial on using cURL command with proxies. 156 | - **[http-headers-with-curl](https://github.com/oxylabs/http-headers-with-curl):** Guide to sending and receiving HTTP headers using cURL. 157 | - **[rotating-proxies-javascript](https://github.com/oxylabs/rotating-proxies-javascript):** Tutorial for rotating proxies with JavaScript. 158 | - **[rotating-proxies-python](https://github.com/oxylabs/Rotating-Proxies-With-Python):** Tutorial for rotating proxies with Python. 159 | - **[wget-proxy](https://github.com/oxylabs/wget-proxy):** Tutorial for using wget command with proxies. 160 | - **[tamilyogi-proxy](https://github.com/oxylabs/tamilyogi-proxy):** Tutorial for TamilYogi proxies. 161 | - **[datacenter-proxies](https://github.com/oxylabs/datacenter-proxies):** Tutorial for Oxylabs' Datacenter proxies. 162 | 163 | ## 🗨️ Community contributions 164 | 165 | We welcome contributions from the community. If you would like to contribute to our projects, please check out our [Contribution Guidelines](https://github.com/oxylabs/oxylabs-readme/blob/main/CONTRIBUTING.md) for more information on how to get involved. 166 | 167 | Additionally, feel free to join our [Discord server](https://discord.gg/GbxmdGhZjq) to connect with other developers, share ideas, and stay updated on the latest developments. 168 | 169 | 170 | ## 💬 Get in Touch 171 | 172 | If you have any questions or need further assistance, feel free to [contact us](mailto:hello@oxylabs.io) or visit our [website](https://www.oxylabs.io) for more information. 173 | 174 | --------------------------------------------------------------------------------