├── .github └── workflows │ └── python-publish.yml ├── .gitignore ├── LICENSE ├── README.md ├── XAgent ├── __init__.py ├── __pycache__ │ ├── __init__.cpython-312.pyc │ ├── __init__.cpython-39.pyc │ ├── agent.cpython-312.pyc │ ├── agent.cpython-39.pyc │ ├── github.cpython-39.pyc │ ├── twitter.cpython-39.pyc │ ├── x.cpython-312.pyc │ └── x.cpython-39.pyc ├── agent.py └── x.py ├── build └── lib │ └── xagent │ ├── __init__.py │ ├── agent.py │ └── x.py ├── dist ├── x_web_crawler-0.1.4-py3-none-any.whl └── x_web_crawler-0.1.4.tar.gz ├── requirements.txt ├── setup.py └── x_web_crawler.egg-info └── PKG-INFO /.github/workflows/python-publish.yml: -------------------------------------------------------------------------------- 1 | # This workflow will upload a Python Package using Twine when a release is created 2 | # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries 3 | 4 | # This workflow uses actions that are not certified by GitHub. 5 | # They are provided by a third-party and are governed by 6 | # separate terms of service, privacy policy, and support 7 | # documentation. 8 | 9 | name: Upload Python Package 10 | 11 | on: 12 | release: 13 | types: [published] 14 | 15 | permissions: 16 | contents: read 17 | 18 | jobs: 19 | deploy: 20 | runs-on: ubuntu-latest 21 | 22 | steps: 23 | - uses: actions/checkout@v4 24 | - name: Set up Python 25 | uses: actions/setup-python@v3 26 | with: 27 | python-version: '3.x' 28 | - name: Install dependencies 29 | run: | 30 | python -m pip install --upgrade pip 31 | pip install build 32 | - name: Build package 33 | run: python -m build 34 | - name: Publish package 35 | uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29 36 | with: 37 | user: __token__ 38 | password: ${{ secrets.PYPI_API_TOKEN }} 39 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # gitignore 2 | test.py 3 | test.ipynb 4 | # txt files 5 | *.txt -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2024 Jacob Somer 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | 2 | # 🚀 x-web-crawler 3 | 4 | x-web-crawler is a Python package for automating interactions on X. 5 | 6 | ## Installation 7 | 8 | You can install x-web-crawler using [pip](https://pypi.org/project/x-web-crawler/): 9 | 10 | ```bash 11 | pip install x-web-crawler 12 | ``` 13 | 14 | ## Usage 15 | 16 | Here’s an example of how to use x-web-crawler to automate actions on Twitter (X) and GitHub: 17 | 18 | ```python 19 | from xagent import XAgent 20 | 21 | def main(): 22 | driver_path = "YOUR_DRIVER_PATH" 23 | profile_path = "YOUR_PROFILE_PATH" 24 | twitter_username = "YOUR_TWITTER_USERNAME" # works best with your X handle and not email 25 | twitter_password = "YOUR_TWITTER_PASSWORD" 26 | 27 | agent = XAgent( 28 | driver_path=driver_path, 29 | profile_path=profile_path, 30 | x_username=twitter_username, 31 | x_password=twitter_password, 32 | ) 33 | 34 | try: 35 | agent.like_x_posts(duration=300) 36 | agent.follow_x_users(["https://x.com/jacob_somer_"], duration=300) 37 | finally: 38 | agent.close() 39 | 40 | if __name__ == "__main__": 41 | main() 42 | ``` 43 | 44 | ## Examples 45 | 46 | ### Automating Twitter (X) Actions 47 | 48 | Here's how to automate actions on Twitter (X) using the Chrome browser: 49 | 50 | ```python 51 | from xagent import XAgent 52 | 53 | driver_path = "DRIVER_PATH" 54 | profile_path = "PROFILE_PATH" 55 | x_username = "YOUR_TWITTER_USERNAME" 56 | x_password = "YOUR_TWITTER_PASSWORD" 57 | 58 | agent = XAgent( 59 | driver_path=driver_path, 60 | profile_path=profile_path, 61 | x_username=x_username, 62 | x_password=x_password, 63 | ) 64 | 65 | try: 66 | # Like posts on your feed for 5 minutes 67 | agent.like_x_posts(duration=300) 68 | 69 | # Follow specific users 70 | agent.follow_x_users(["https://x.com/jacob_somer_"], duration=300) 71 | finally: 72 | agent.close() 73 | ``` 74 | 75 | ### Using Microsoft Edge Browser 76 | 77 | To automate actions using the Edge browser, modify the driver and profile paths: 78 | 79 | ```python 80 | from xagent.agent import XAgent 81 | 82 | edge_driver_path = "/Users//edgedriver_mac64_m1/msedgedriver" 83 | edge_profile_path = "/Users//Library/Application Support/Microsoft Edge/User Data" 84 | 85 | agent = XAgent( 86 | driver_path=edge_driver_path, 87 | profile_path=edge_profile_path, 88 | browser="edge" 89 | ) 90 | 91 | try: 92 | # Like posts on your feed for 5 minutes 93 | agent.like_x_posts(duration=300) 94 | 95 | # Follow specific users 96 | agent.follow_x_users(["https://x.com/jacob_somer_"], duration=300) 97 | finally: 98 | agent.close() 99 | ``` 100 | 101 | ### XAgent Methods 102 | 103 | #### `__init__(self, driver_path, profile_path, x_username=None, x_password=None, browser="chrome")` 104 | 105 | Initializes the XAgent. 106 | 107 | - `driver_path`: Path to the ChromeDriver executable. Download ChromeDriver from [here](https://googlechromelabs.github.io/chrome-for-testing/). 108 | - `profile_path`: Path to the user profile directory for Chrome. To find this, type "chrome://version" into your Chrome browser's address bar, and look for the "Profile Path" variable. 109 | - `x_username` (optional): Twitter (X) username for authentication. 110 | - `x_password` (optional): Twitter (X) password for authentication. 111 | - `browser`: The browser to use, either "chrome" or "edge". 112 | 113 | #### `like_x_posts(duration=300)` 114 | 115 | 👍 Likes posts on the user's feed for the specified duration. 116 | 117 | #### `follow_x_users(users, duration=300)` 118 | 119 | 👥 Follows the specified Twitter (X) users for the specified duration. 120 | 121 | #### `get_x_followers(username)` 122 | 123 | 📈 Gets the followers of a specified Twitter (X) user. 124 | 125 | #### `get_x_following(username)` 126 | 127 | 📊 Gets the users that a specified Twitter (X) user is following. 128 | 129 | #### `unfollow_x_users(users)` 130 | 131 | 🚫 Unfollows the specified Twitter (X) users. 132 | 133 | #### `unfollow_x_users_alternative(user, users)` 134 | 135 | 🔄 Unfollows the specified Twitter (X) users using an alternative method. 136 | 137 | #### `get_x_handles(query, num_handles=10)` 138 | 139 | 🔍 Gets the handles of users based on a query. 140 | 141 | ## Running Tests 142 | 143 | To run the tests for this package, use the `unittest` framework: 144 | 145 | ```bash 146 | python -m unittest discover tests 147 | ``` 148 | 149 | ## License 150 | 151 | This project is licensed under the MIT License - see the LICENSE file for details. 152 | -------------------------------------------------------------------------------- /XAgent/__init__.py: -------------------------------------------------------------------------------- 1 | from .agent import XAgent 2 | 3 | __all__ = ['XAgent'] -------------------------------------------------------------------------------- /XAgent/__pycache__/__init__.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/XAgent/__pycache__/__init__.cpython-312.pyc -------------------------------------------------------------------------------- /XAgent/__pycache__/__init__.cpython-39.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/XAgent/__pycache__/__init__.cpython-39.pyc -------------------------------------------------------------------------------- /XAgent/__pycache__/agent.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/XAgent/__pycache__/agent.cpython-312.pyc -------------------------------------------------------------------------------- /XAgent/__pycache__/agent.cpython-39.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/XAgent/__pycache__/agent.cpython-39.pyc -------------------------------------------------------------------------------- /XAgent/__pycache__/github.cpython-39.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/XAgent/__pycache__/github.cpython-39.pyc -------------------------------------------------------------------------------- /XAgent/__pycache__/twitter.cpython-39.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/XAgent/__pycache__/twitter.cpython-39.pyc -------------------------------------------------------------------------------- /XAgent/__pycache__/x.cpython-312.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/XAgent/__pycache__/x.cpython-312.pyc -------------------------------------------------------------------------------- /XAgent/__pycache__/x.cpython-39.pyc: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/XAgent/__pycache__/x.cpython-39.pyc -------------------------------------------------------------------------------- /XAgent/agent.py: -------------------------------------------------------------------------------- 1 | from selenium import webdriver 2 | from .x import X 3 | 4 | class XAgent: 5 | def __init__(self, driver_path, profile_path, x_username=None, x_password=None, browser="chrome"): 6 | self.driver_path = driver_path 7 | self.profile_path = profile_path 8 | self.browser = browser 9 | self.driver = self.create_driver() 10 | self.x_agent = X(self.driver) 11 | self.x_username = x_username 12 | self.x_password = x_password 13 | self.login_to_x() 14 | 15 | def set_common_options(self, options): 16 | if self.profile_path: 17 | options.add_argument(f"user-data-dir={self.profile_path}") 18 | options.add_experimental_option("excludeSwitches", ["enable-automation"]) 19 | options.add_argument("--remote-debugging-port=9222") 20 | 21 | options.add_argument("--disable-dev-shm-using") 22 | options.add_argument("--disable-extensions") 23 | options.add_argument("--disable-gpu") 24 | options.add_argument("start-maximized") 25 | options.add_argument("disable-infobars") 26 | options.add_argument(f"window-size={1920//2},1080") 27 | options.add_experimental_option("prefs", { 28 | "credentials_enable_service": False, 29 | "profile.password_manager_enabled": False, 30 | }) 31 | return options 32 | 33 | def create_driver(self): 34 | if self.browser == "chrome": 35 | chrome_options = webdriver.ChromeOptions() 36 | chrome_options = self.set_common_options(chrome_options) 37 | driver = webdriver.Chrome(options=chrome_options) 38 | return driver 39 | elif self.browser == "edge": 40 | edge_options = webdriver.EdgeOptions() 41 | edge_options = self.set_common_options(edge_options) 42 | edge_service = webdriver.EdgeService(executable_path=self.driver_path) 43 | driver = webdriver.Edge(service=edge_service, options=edge_options) 44 | return driver 45 | 46 | """ 47 | The like_x_posts method will like x posts on the user's feed for a specified amount of time. 48 | """ 49 | def like_x_posts(self, duration=300): 50 | self.x_agent.like_x_posts_on_feed(duration) 51 | 52 | """ 53 | Gets the followers of a user. 54 | """ 55 | def get_x_followers(self, username): 56 | ret = self.x_agent.get_followers(username) 57 | return ret 58 | 59 | """ 60 | Gets the following of a user. 61 | """ 62 | def get_x_following(self, username): 63 | ret = self.x_agent.get_following(username) 64 | return ret 65 | 66 | """ 67 | Given a list of users, unfollows them. 68 | """ 69 | def unfollow_x_users(self, users): 70 | self.x_agent.unfollow_users(users) 71 | 72 | """ 73 | Given a list of users and the current user, unfollows them. Uses an alternative method. 74 | """ 75 | def unfollow_x_users_alternative(self, user, users): 76 | self.x_agent.unfollow_users_alternative(user, users) 77 | 78 | """ 79 | Given a list of users, follows them. 80 | """ 81 | def follow_x_users(self, users, duration=300): 82 | self.x_agent.follow_users(users, duration) 83 | 84 | """ 85 | Given a query, gets the handles of the users. 86 | """ 87 | def get_x_handles(self, query, num_handles=10): 88 | ret = self.x_agent.get_handles(query, num_handles) 89 | return ret 90 | 91 | def login_to_x(self): 92 | self.x_agent.login(self.x_username, self.x_password) 93 | 94 | def close(self): 95 | self.driver.quit() 96 | -------------------------------------------------------------------------------- /XAgent/x.py: -------------------------------------------------------------------------------- 1 | from selenium.webdriver.common.by import By # type: ignore 2 | from selenium.webdriver.support.ui import WebDriverWait # type: ignore 3 | from selenium.webdriver.support import expected_conditions as EC # type: ignore 4 | from selenium.common.exceptions import StaleElementReferenceException # type: ignore 5 | from selenium.webdriver import ActionChains 6 | import time 7 | 8 | class X: 9 | def __init__(self, driver): 10 | self.driver = driver 11 | self.likes = set() 12 | self.followed = set() 13 | 14 | def is_ad(self, x_post): 15 | try: 16 | x_post.find_element(By.XPATH, ".//span[text()='Ad']") 17 | return True 18 | except Exception: 19 | return False 20 | 21 | def like_x_post(self, x_post): 22 | 23 | try: 24 | x_post_id = x_post.find_element(By.XPATH, ".//a[contains(@href, '/status/')]").get_attribute("href").split("/")[-1] 25 | print(f"x_post ID: {x_post_id}") 26 | if x_post_id and x_post_id not in self.likes: 27 | like_button = x_post.find_element(By.XPATH, ".//button[contains(@aria-label, 'Likes. Like')]") 28 | if like_button.is_enabled() and like_button.is_displayed(): 29 | like_button.click() 30 | print(f"Liked x_post: {x_post_id}") 31 | self.likes.add(x_post_id) 32 | time.sleep(1) # Short delay after each like action 33 | return True 34 | return False 35 | except Exception as e: 36 | print(f"Failed to like x_post: {e}") 37 | return False 38 | 39 | def like_x_posts_on_feed(self, run_time=300): 40 | self.driver.get("https://x.com/home") 41 | start_time = time.time() 42 | while time.time() - start_time < run_time: 43 | time.sleep(2) 44 | try: 45 | x_posts = WebDriverWait(self.driver, 10).until( 46 | EC.visibility_of_all_elements_located((By.XPATH, "//article[@data-testid='x_post']")) 47 | ) 48 | 49 | non_ad_x_posts = [x_post for x_post in x_posts if not self.is_ad(x_post)] 50 | 51 | for x_post in non_ad_x_posts: 52 | self.like_x_post(x_post) 53 | 54 | except Exception as e: 55 | print(f"Error during x_post processing: {e}") 56 | 57 | try: 58 | self.driver.execute_script("window.scrollBy(0,1000)") 59 | except Exception as e: 60 | print(f"Failed to scroll: {e}") 61 | time.sleep(3) 62 | 63 | 64 | def follow_users(self, users, duration=300): 65 | start_time = time.time() 66 | for user in users: 67 | if time.time() - start_time > duration: 68 | break 69 | user = user.replace("https://x.com/", "") 70 | user = user.replace("https://twitter.com/", "") 71 | self.driver.get(f"https://x.com/{user}") 72 | try: 73 | follow_button = WebDriverWait(self.driver, 10).until( 74 | EC.element_to_be_clickable((By.XPATH, "//span[contains(text(), 'Follow')]")) 75 | ) 76 | follow_button.click() 77 | time.sleep(2) 78 | except Exception as e: 79 | print(f"Error following user {user}: {e}") 80 | 81 | def unfollow_users(self, users): 82 | for user in users: 83 | user = user.replace("https://x.com/", "") 84 | user = user.replace("https://twitter.com/", "") 85 | self.driver.get(f"https://x.com/{user}") 86 | try: 87 | unfollow_button = WebDriverWait(self.driver, 10).until( 88 | EC.element_to_be_clickable((By.XPATH, "//span[contains(text(), 'Following')]")) 89 | ) 90 | unfollow_button.click() 91 | time.sleep(2) 92 | confirm_unfollow_button = WebDriverWait(self.driver, 10).until( 93 | EC.element_to_be_clickable((By.XPATH, "//button[contains(@style,'border-color: rgba(0, 0, 0, 0)')]")) 94 | ) 95 | confirm_unfollow_button.click() 96 | time.sleep(2) 97 | except Exception as e: 98 | print(f"Error unfollowing user {user}: {e}") 99 | 100 | def unfollow_users_alternative(self, user, users): 101 | self.driver.get(f"https://x.com/{user}/following") 102 | unfollowed = [] 103 | last_height = self.driver.execute_script("return document.body.scrollHeight") 104 | 105 | while True: 106 | try: 107 | WebDriverWait(self.driver, 10).until( 108 | EC.visibility_of_all_elements_located((By.XPATH, "//button[contains(@data-testid, 'UserCell')]")) 109 | ) 110 | current_following = {} 111 | for elem in self.driver.find_elements(By.XPATH, "//button[contains(@data-testid, 'UserCell')]"): 112 | try: 113 | username = elem.find_element(By.XPATH, ".//span[starts-with(text(), '@')]").text 114 | unfollow_button = elem.find_element(By.XPATH, f".//button[contains(@aria-label, 'Following {username}')]") 115 | current_following[username] = unfollow_button 116 | except Exception as e: 117 | print(f"Error getting following: {e}") 118 | users_to_unfollow = {f: current_following[f] for f in current_following if f in users} 119 | if not users_to_unfollow: 120 | # Scroll down to the bottom of the page 121 | self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 122 | time.sleep(2) 123 | new_height = self.driver.execute_script("return document.body.scrollHeight") 124 | if new_height == last_height: 125 | break 126 | last_height = new_height 127 | else: 128 | for key in users_to_unfollow: 129 | try: 130 | unfollow_button = WebDriverWait(self.driver, 10).until( 131 | EC.element_to_be_clickable((By.XPATH, f".//button[contains(@aria-label, 'Following {key}')]")) 132 | ) 133 | 134 | 135 | self.driver.execute_script("arguments[0].scrollIntoView(true);", unfollow_button) 136 | # scroll up a bit to make sure the unfollow button is visible 137 | self.driver.execute_script("window.scrollBy(0, -100);") 138 | mouse = ActionChains(self.driver) 139 | mouse.move_to_element(unfollow_button).perform() 140 | 141 | 142 | attempt = 0 143 | while attempt < 3: 144 | try: 145 | unfollow_button.click() 146 | break 147 | except Exception as e: 148 | print(f"Click failed on attempt {attempt+1}: {str(e)}") 149 | time.sleep(1) 150 | attempt += 1 151 | 152 | confirm_unfollow_button = WebDriverWait(self.driver, 10).until( 153 | EC.element_to_be_clickable((By.XPATH, "//button[@data-testid='confirmationSheetConfirm']")) 154 | ) 155 | confirm_unfollow_button.click() 156 | unfollowed.append(key) 157 | print(f"Successfully unfollowed {key}") 158 | except Exception as e: 159 | print(f"Failed to unfollow {key}: {str(e)}") 160 | 161 | except StaleElementReferenceException: 162 | print("Encountered a stale element, retrying...") 163 | time.sleep(1) 164 | 165 | 166 | 167 | def get_followers(self, user): 168 | time.sleep(2) 169 | self.driver.get(f"https://x.com/{user}") 170 | time.sleep(2) 171 | try: 172 | number_of_followers = self.driver.find_element(By.XPATH, f"//a[contains(@href, '/{user}/verified_followers')]").text.split()[0] 173 | print(f"Number of followers: {number_of_followers}") 174 | except Exception as e: 175 | print(f"Error retrieving follower count: {e}") 176 | return {} 177 | 178 | self.driver.get(f"https://x.com/{user}/followers") 179 | followers = {} 180 | last_height = self.driver.execute_script("return document.body.scrollHeight") 181 | 182 | while True: 183 | try: 184 | WebDriverWait(self.driver, 10).until( 185 | EC.visibility_of_all_elements_located((By.XPATH, "//button[contains(@data-testid, 'UserCell')]")) 186 | ) 187 | 188 | current_followers = {} 189 | for elem in self.driver.find_elements(By.XPATH, "//button[contains(@data-testid, 'UserCell')]"): 190 | try: 191 | username = elem.find_element(By.XPATH, ".//span[starts-with(text(), '@')]").text 192 | all_text = elem.text 193 | current_followers[username] = all_text 194 | except Exception as e: 195 | print(f"Error getting follower: {e}") 196 | 197 | new_followers = {f: current_followers[f] for f in current_followers if f not in followers} 198 | if not new_followers: 199 | # Scroll down to the bottom of the page 200 | self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 201 | time.sleep(2) 202 | new_height = self.driver.execute_script("return document.body.scrollHeight") 203 | if new_height == last_height: 204 | break 205 | last_height = new_height 206 | else: 207 | for key in new_followers: 208 | followers[key] = new_followers[key] 209 | if len(followers) >= int(number_of_followers.replace(',', '')): 210 | break 211 | except StaleElementReferenceException: 212 | print("Encountered a stale element, retrying...") 213 | time.sleep(1) 214 | except Exception as e: 215 | print(f"Error during scrolling/loading: {e}") 216 | break 217 | return followers 218 | 219 | 220 | 221 | def get_following(self, user): 222 | self.driver.get(f"https://x.com/{user}") 223 | time.sleep(2) 224 | try: 225 | number_of_following = self.driver.find_element(By.XPATH, f"//a[contains(@href, '/{user}/following')]").text.split()[0] 226 | print(f"Number of following: {number_of_following}") 227 | except Exception as e: 228 | print(f"Error retrieving following count: {e}") 229 | return {} 230 | 231 | self.driver.get(f"https://x.com/{user}/following") 232 | following = {} 233 | last_height = self.driver.execute_script("return document.body.scrollHeight") 234 | 235 | while True: 236 | try: 237 | WebDriverWait(self.driver, 10).until( 238 | EC.visibility_of_all_elements_located((By.XPATH, "//button[contains(@data-testid, 'UserCell')]")) 239 | ) 240 | 241 | current_following = {} 242 | for elem in self.driver.find_elements(By.XPATH, "//button[contains(@data-testid, 'UserCell')]"): 243 | try: 244 | username = elem.find_element(By.XPATH, ".//span[starts-with(text(), '@')]").text 245 | all_text = elem.text 246 | current_following[username] = all_text 247 | except Exception as e: 248 | print(f"Error getting following: {e}") 249 | new_following = {f: current_following[f] for f in current_following if f not in following} 250 | if not new_following: 251 | # Scroll down to the bottom of the page 252 | self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 253 | time.sleep(2) 254 | new_height = self.driver.execute_script("return document.body.scrollHeight") 255 | if new_height == last_height: 256 | break 257 | last_height = new_height 258 | else: 259 | for key in new_following: 260 | following[key] = new_following[key] 261 | if len(following) >= int(number_of_following.replace(',', '')): 262 | break 263 | except StaleElementReferenceException: 264 | print("Encountered a stale element, retrying...") 265 | time.sleep(1) 266 | except Exception as e: 267 | print(f"Error during scrolling/loading: {e}") 268 | break 269 | 270 | return following 271 | 272 | # create a method that gets all the handles for a specific search query 273 | #ex. https://x.com/search?q="@PKU1898"&f=user 274 | def get_handles(self, query, num_handles=10): 275 | self.driver.get(f'https://x.com/search?q={query}&f=user') 276 | time.sleep(2) 277 | handles = [] 278 | last_height = self.driver.execute_script("return document.body.scrollHeight") 279 | 280 | while len(handles) < num_handles: 281 | try: 282 | WebDriverWait(self.driver, 10).until( 283 | EC.visibility_of_all_elements_located((By.XPATH, "//span[starts-with(text(), '@')]")) 284 | ) 285 | current_handles = [elem.text for elem in self.driver.find_elements(By.XPATH, "//span[starts-with(text(), '@')]")] 286 | new_handles = [f for f in current_handles if f not in handles] 287 | 288 | if not new_handles: 289 | # Scroll down to the bottom of the page 290 | self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 291 | time.sleep(2) 292 | new_height = self.driver.execute_script("return document.body.scrollHeight") 293 | if new_height == last_height: 294 | break 295 | last_height = new_height 296 | else: 297 | handles.extend(new_handles) 298 | print(f"Added {len(new_handles)} new handles.") 299 | except StaleElementReferenceException: 300 | print("Encountered a stale element, retrying...") 301 | time.sleep(1) 302 | except Exception as e: 303 | print(f"Error during scrolling/loading: {e}") 304 | break 305 | 306 | return handles[:num_handles] 307 | 308 | def login(self , username, password): 309 | self.driver.get("https://x.com/login") 310 | time.sleep(2) 311 | if self.driver.current_url in ["https://x.com/login", "https://x.com/i/flow/login"]: 312 | try: 313 | self.driver.get("https://x.com/i/flow/login") 314 | print("Logging in...") 315 | username_input = WebDriverWait(self.driver, 10).until( 316 | EC.element_to_be_clickable((By.XPATH, "//input[@autocomplete='username']")) 317 | ) 318 | print("Found username input.") 319 | username_input.send_keys(username) 320 | next_button = WebDriverWait(self.driver, 10).until( 321 | EC.element_to_be_clickable((By.XPATH, "//span[contains(text(), 'Next')]")) 322 | ) 323 | next_button.click() 324 | time.sleep(2) 325 | 326 | password_input = WebDriverWait(self.driver, 10).until( 327 | EC.element_to_be_clickable((By.XPATH, "//input[@autocomplete='current-password']")) 328 | ) 329 | password_input.send_keys(password) 330 | # click on the next button 331 | login_button = WebDriverWait(self.driver, 10).until( 332 | EC.element_to_be_clickable((By.XPATH, "//span[contains(text(), 'Log in')]")) 333 | ) 334 | 335 | login_button.click() 336 | time.sleep(2) 337 | print("Successfully logged in.") 338 | except Exception as e: 339 | print(f"Error logging in: {e}") 340 | else: 341 | print("User already logged in.") -------------------------------------------------------------------------------- /build/lib/xagent/__init__.py: -------------------------------------------------------------------------------- 1 | from .agent import XAgent 2 | 3 | __all__ = ['XAgent'] -------------------------------------------------------------------------------- /build/lib/xagent/agent.py: -------------------------------------------------------------------------------- 1 | from selenium import webdriver 2 | from .x import X 3 | 4 | class XAgent: 5 | def __init__(self, driver_path, profile_path, x_username=None, x_password=None, browser="chrome"): 6 | self.driver_path = driver_path 7 | self.profile_path = profile_path 8 | self.browser = browser 9 | self.driver = self.create_driver() 10 | self.x_agent = X(self.driver) 11 | self.x_username = x_username 12 | self.x_password = x_password 13 | self.login_to_x() 14 | 15 | def set_common_options(self, options): 16 | if self.profile_path: 17 | options.add_argument(f"user-data-dir={self.profile_path}") 18 | options.add_experimental_option("excludeSwitches", ["enable-automation"]) 19 | options.add_argument("--remote-debugging-port=9222") 20 | 21 | options.add_argument("--disable-dev-shm-using") 22 | options.add_argument("--disable-extensions") 23 | options.add_argument("--disable-gpu") 24 | options.add_argument("start-maximized") 25 | options.add_argument("disable-infobars") 26 | options.add_argument(f"window-size={1920//2},1080") 27 | options.add_experimental_option("prefs", { 28 | "credentials_enable_service": False, 29 | "profile.password_manager_enabled": False, 30 | }) 31 | return options 32 | 33 | def create_driver(self): 34 | if self.browser == "chrome": 35 | chrome_options = webdriver.ChromeOptions() 36 | chrome_options = self.set_common_options(chrome_options) 37 | driver = webdriver.Chrome(options=chrome_options) 38 | return driver 39 | elif self.browser == "edge": 40 | edge_options = webdriver.EdgeOptions() 41 | edge_options = self.set_common_options(edge_options) 42 | edge_service = webdriver.EdgeService(executable_path=self.driver_path) 43 | driver = webdriver.Edge(service=edge_service, options=edge_options) 44 | return driver 45 | 46 | """ 47 | The like_x_posts method will like x posts on the user's feed for a specified amount of time. 48 | """ 49 | def like_x_posts(self, duration=300): 50 | self.x_agent.like_x_posts_on_feed(duration) 51 | 52 | """ 53 | Gets the followers of a user. 54 | """ 55 | def get_x_followers(self, username): 56 | ret = self.x_agent.get_followers(username) 57 | return ret 58 | 59 | """ 60 | Gets the following of a user. 61 | """ 62 | def get_x_following(self, username): 63 | ret = self.x_agent.get_following(username) 64 | return ret 65 | 66 | """ 67 | Given a list of users, unfollows them. 68 | """ 69 | def unfollow_x_users(self, users): 70 | self.x_agent.unfollow_users(users) 71 | 72 | """ 73 | Given a list of users and the current user, unfollows them. Uses an alternative method. 74 | """ 75 | def unfollow_x_users_alternative(self, user, users): 76 | self.x_agent.unfollow_users_alternative(user, users) 77 | 78 | """ 79 | Given a list of users, follows them. 80 | """ 81 | def follow_x_users(self, users, duration=300): 82 | self.x_agent.follow_users(users, duration) 83 | 84 | """ 85 | Given a query, gets the handles of the users. 86 | """ 87 | def get_x_handles(self, query, num_handles=10): 88 | ret = self.x_agent.get_handles(query, num_handles) 89 | return ret 90 | 91 | def login_to_x(self): 92 | self.x_agent.login(self.x_username, self.x_password) 93 | 94 | def close(self): 95 | self.driver.quit() 96 | -------------------------------------------------------------------------------- /build/lib/xagent/x.py: -------------------------------------------------------------------------------- 1 | from selenium.webdriver.common.by import By # type: ignore 2 | from selenium.webdriver.support.ui import WebDriverWait # type: ignore 3 | from selenium.webdriver.support import expected_conditions as EC # type: ignore 4 | from selenium.common.exceptions import StaleElementReferenceException # type: ignore 5 | from selenium.webdriver import ActionChains 6 | import time 7 | 8 | class X: 9 | def __init__(self, driver): 10 | self.driver = driver 11 | self.likes = set() 12 | self.followed = set() 13 | 14 | def is_ad(self, x_post): 15 | try: 16 | x_post.find_element(By.XPATH, ".//span[text()='Ad']") 17 | return True 18 | except Exception: 19 | return False 20 | 21 | def like_x_post(self, x_post): 22 | 23 | try: 24 | x_post_id = x_post.find_element(By.XPATH, ".//a[contains(@href, '/status/')]").get_attribute("href").split("/")[-1] 25 | print(f"x_post ID: {x_post_id}") 26 | if x_post_id and x_post_id not in self.likes: 27 | like_button = x_post.find_element(By.XPATH, ".//button[contains(@aria-label, 'Likes. Like')]") 28 | if like_button.is_enabled() and like_button.is_displayed(): 29 | like_button.click() 30 | print(f"Liked x_post: {x_post_id}") 31 | self.likes.add(x_post_id) 32 | time.sleep(1) # Short delay after each like action 33 | return True 34 | return False 35 | except Exception as e: 36 | print(f"Failed to like x_post: {e}") 37 | return False 38 | 39 | def like_x_posts_on_feed(self, run_time=300): 40 | self.driver.get("https://x.com/home") 41 | start_time = time.time() 42 | while time.time() - start_time < run_time: 43 | time.sleep(2) 44 | try: 45 | x_posts = WebDriverWait(self.driver, 10).until( 46 | EC.visibility_of_all_elements_located((By.XPATH, "//article[@data-testid='x_post']")) 47 | ) 48 | 49 | non_ad_x_posts = [x_post for x_post in x_posts if not self.is_ad(x_post)] 50 | 51 | for x_post in non_ad_x_posts: 52 | self.like_x_post(x_post) 53 | 54 | except Exception as e: 55 | print(f"Error during x_post processing: {e}") 56 | 57 | try: 58 | self.driver.execute_script("window.scrollBy(0,1000)") 59 | except Exception as e: 60 | print(f"Failed to scroll: {e}") 61 | time.sleep(3) 62 | 63 | 64 | def follow_users(self, users, duration=300): 65 | start_time = time.time() 66 | for user in users: 67 | if time.time() - start_time > duration: 68 | break 69 | user = user.replace("https://x.com/", "") 70 | user = user.replace("https://twitter.com/", "") 71 | self.driver.get(f"https://x.com/{user}") 72 | try: 73 | follow_button = WebDriverWait(self.driver, 10).until( 74 | EC.element_to_be_clickable((By.XPATH, "//span[contains(text(), 'Follow')]")) 75 | ) 76 | follow_button.click() 77 | time.sleep(2) 78 | except Exception as e: 79 | print(f"Error following user {user}: {e}") 80 | 81 | def unfollow_users(self, users): 82 | for user in users: 83 | user = user.replace("https://x.com/", "") 84 | user = user.replace("https://twitter.com/", "") 85 | self.driver.get(f"https://x.com/{user}") 86 | try: 87 | unfollow_button = WebDriverWait(self.driver, 10).until( 88 | EC.element_to_be_clickable((By.XPATH, "//span[contains(text(), 'Following')]")) 89 | ) 90 | unfollow_button.click() 91 | time.sleep(2) 92 | confirm_unfollow_button = WebDriverWait(self.driver, 10).until( 93 | EC.element_to_be_clickable((By.XPATH, "//button[contains(@style,'border-color: rgba(0, 0, 0, 0)')]")) 94 | ) 95 | confirm_unfollow_button.click() 96 | time.sleep(2) 97 | except Exception as e: 98 | print(f"Error unfollowing user {user}: {e}") 99 | 100 | def unfollow_users_alternative(self, user, users): 101 | self.driver.get(f"https://x.com/{user}/following") 102 | unfollowed = [] 103 | last_height = self.driver.execute_script("return document.body.scrollHeight") 104 | 105 | while True: 106 | try: 107 | WebDriverWait(self.driver, 10).until( 108 | EC.visibility_of_all_elements_located((By.XPATH, "//button[contains(@data-testid, 'UserCell')]")) 109 | ) 110 | current_following = {} 111 | for elem in self.driver.find_elements(By.XPATH, "//button[contains(@data-testid, 'UserCell')]"): 112 | try: 113 | username = elem.find_element(By.XPATH, ".//span[starts-with(text(), '@')]").text 114 | unfollow_button = elem.find_element(By.XPATH, f".//button[contains(@aria-label, 'Following {username}')]") 115 | current_following[username] = unfollow_button 116 | except Exception as e: 117 | print(f"Error getting following: {e}") 118 | users_to_unfollow = {f: current_following[f] for f in current_following if f in users} 119 | if not users_to_unfollow: 120 | # Scroll down to the bottom of the page 121 | self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 122 | time.sleep(2) 123 | new_height = self.driver.execute_script("return document.body.scrollHeight") 124 | if new_height == last_height: 125 | break 126 | last_height = new_height 127 | else: 128 | for key in users_to_unfollow: 129 | try: 130 | unfollow_button = WebDriverWait(self.driver, 10).until( 131 | EC.element_to_be_clickable((By.XPATH, f".//button[contains(@aria-label, 'Following {key}')]")) 132 | ) 133 | 134 | 135 | self.driver.execute_script("arguments[0].scrollIntoView(true);", unfollow_button) 136 | # scroll up a bit to make sure the unfollow button is visible 137 | self.driver.execute_script("window.scrollBy(0, -100);") 138 | mouse = ActionChains(self.driver) 139 | mouse.move_to_element(unfollow_button).perform() 140 | 141 | 142 | attempt = 0 143 | while attempt < 3: 144 | try: 145 | unfollow_button.click() 146 | break 147 | except Exception as e: 148 | print(f"Click failed on attempt {attempt+1}: {str(e)}") 149 | time.sleep(1) 150 | attempt += 1 151 | 152 | confirm_unfollow_button = WebDriverWait(self.driver, 10).until( 153 | EC.element_to_be_clickable((By.XPATH, "//button[@data-testid='confirmationSheetConfirm']")) 154 | ) 155 | confirm_unfollow_button.click() 156 | unfollowed.append(key) 157 | print(f"Successfully unfollowed {key}") 158 | except Exception as e: 159 | print(f"Failed to unfollow {key}: {str(e)}") 160 | 161 | except StaleElementReferenceException: 162 | print("Encountered a stale element, retrying...") 163 | time.sleep(1) 164 | 165 | 166 | 167 | def get_followers(self, user): 168 | time.sleep(2) 169 | self.driver.get(f"https://x.com/{user}") 170 | time.sleep(2) 171 | try: 172 | number_of_followers = self.driver.find_element(By.XPATH, f"//a[contains(@href, '/{user}/verified_followers')]").text.split()[0] 173 | print(f"Number of followers: {number_of_followers}") 174 | except Exception as e: 175 | print(f"Error retrieving follower count: {e}") 176 | return {} 177 | 178 | self.driver.get(f"https://x.com/{user}/followers") 179 | followers = {} 180 | last_height = self.driver.execute_script("return document.body.scrollHeight") 181 | 182 | while True: 183 | try: 184 | WebDriverWait(self.driver, 10).until( 185 | EC.visibility_of_all_elements_located((By.XPATH, "//button[contains(@data-testid, 'UserCell')]")) 186 | ) 187 | 188 | current_followers = {} 189 | for elem in self.driver.find_elements(By.XPATH, "//button[contains(@data-testid, 'UserCell')]"): 190 | try: 191 | username = elem.find_element(By.XPATH, ".//span[starts-with(text(), '@')]").text 192 | all_text = elem.text 193 | current_followers[username] = all_text 194 | except Exception as e: 195 | print(f"Error getting follower: {e}") 196 | 197 | new_followers = {f: current_followers[f] for f in current_followers if f not in followers} 198 | if not new_followers: 199 | # Scroll down to the bottom of the page 200 | self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 201 | time.sleep(2) 202 | new_height = self.driver.execute_script("return document.body.scrollHeight") 203 | if new_height == last_height: 204 | break 205 | last_height = new_height 206 | else: 207 | for key in new_followers: 208 | followers[key] = new_followers[key] 209 | if len(followers) >= int(number_of_followers.replace(',', '')): 210 | break 211 | except StaleElementReferenceException: 212 | print("Encountered a stale element, retrying...") 213 | time.sleep(1) 214 | except Exception as e: 215 | print(f"Error during scrolling/loading: {e}") 216 | break 217 | return followers 218 | 219 | 220 | 221 | def get_following(self, user): 222 | self.driver.get(f"https://x.com/{user}") 223 | time.sleep(2) 224 | try: 225 | number_of_following = self.driver.find_element(By.XPATH, f"//a[contains(@href, '/{user}/following')]").text.split()[0] 226 | print(f"Number of following: {number_of_following}") 227 | except Exception as e: 228 | print(f"Error retrieving following count: {e}") 229 | return {} 230 | 231 | self.driver.get(f"https://x.com/{user}/following") 232 | following = {} 233 | last_height = self.driver.execute_script("return document.body.scrollHeight") 234 | 235 | while True: 236 | try: 237 | WebDriverWait(self.driver, 10).until( 238 | EC.visibility_of_all_elements_located((By.XPATH, "//button[contains(@data-testid, 'UserCell')]")) 239 | ) 240 | 241 | current_following = {} 242 | for elem in self.driver.find_elements(By.XPATH, "//button[contains(@data-testid, 'UserCell')]"): 243 | try: 244 | username = elem.find_element(By.XPATH, ".//span[starts-with(text(), '@')]").text 245 | all_text = elem.text 246 | current_following[username] = all_text 247 | except Exception as e: 248 | print(f"Error getting following: {e}") 249 | new_following = {f: current_following[f] for f in current_following if f not in following} 250 | if not new_following: 251 | # Scroll down to the bottom of the page 252 | self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 253 | time.sleep(2) 254 | new_height = self.driver.execute_script("return document.body.scrollHeight") 255 | if new_height == last_height: 256 | break 257 | last_height = new_height 258 | else: 259 | for key in new_following: 260 | following[key] = new_following[key] 261 | if len(following) >= int(number_of_following.replace(',', '')): 262 | break 263 | except StaleElementReferenceException: 264 | print("Encountered a stale element, retrying...") 265 | time.sleep(1) 266 | except Exception as e: 267 | print(f"Error during scrolling/loading: {e}") 268 | break 269 | 270 | return following 271 | 272 | # create a method that gets all the handles for a specific search query 273 | #ex. https://x.com/search?q="@PKU1898"&f=user 274 | def get_handles(self, query, num_handles=10): 275 | self.driver.get(f'https://x.com/search?q={query}&f=user') 276 | time.sleep(2) 277 | handles = [] 278 | last_height = self.driver.execute_script("return document.body.scrollHeight") 279 | 280 | while len(handles) < num_handles: 281 | try: 282 | WebDriverWait(self.driver, 10).until( 283 | EC.visibility_of_all_elements_located((By.XPATH, "//span[starts-with(text(), '@')]")) 284 | ) 285 | current_handles = [elem.text for elem in self.driver.find_elements(By.XPATH, "//span[starts-with(text(), '@')]")] 286 | new_handles = [f for f in current_handles if f not in handles] 287 | 288 | if not new_handles: 289 | # Scroll down to the bottom of the page 290 | self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") 291 | time.sleep(2) 292 | new_height = self.driver.execute_script("return document.body.scrollHeight") 293 | if new_height == last_height: 294 | break 295 | last_height = new_height 296 | else: 297 | handles.extend(new_handles) 298 | print(f"Added {len(new_handles)} new handles.") 299 | except StaleElementReferenceException: 300 | print("Encountered a stale element, retrying...") 301 | time.sleep(1) 302 | except Exception as e: 303 | print(f"Error during scrolling/loading: {e}") 304 | break 305 | 306 | return handles[:num_handles] 307 | 308 | def login(self , username, password): 309 | self.driver.get("https://x.com/login") 310 | time.sleep(2) 311 | if self.driver.current_url in ["https://x.com/login", "https://x.com/i/flow/login"]: 312 | try: 313 | self.driver.get("https://x.com/i/flow/login") 314 | print("Logging in...") 315 | username_input = WebDriverWait(self.driver, 10).until( 316 | EC.element_to_be_clickable((By.XPATH, "//input[@autocomplete='username']")) 317 | ) 318 | print("Found username input.") 319 | username_input.send_keys(username) 320 | next_button = WebDriverWait(self.driver, 10).until( 321 | EC.element_to_be_clickable((By.XPATH, "//span[contains(text(), 'Next')]")) 322 | ) 323 | next_button.click() 324 | time.sleep(2) 325 | 326 | password_input = WebDriverWait(self.driver, 10).until( 327 | EC.element_to_be_clickable((By.XPATH, "//input[@autocomplete='current-password']")) 328 | ) 329 | password_input.send_keys(password) 330 | # click on the next button 331 | login_button = WebDriverWait(self.driver, 10).until( 332 | EC.element_to_be_clickable((By.XPATH, "//span[contains(text(), 'Log in')]")) 333 | ) 334 | 335 | login_button.click() 336 | time.sleep(2) 337 | print("Successfully logged in.") 338 | except Exception as e: 339 | print(f"Error logging in: {e}") 340 | else: 341 | print("User already logged in.") -------------------------------------------------------------------------------- /dist/x_web_crawler-0.1.4-py3-none-any.whl: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/dist/x_web_crawler-0.1.4-py3-none-any.whl -------------------------------------------------------------------------------- /dist/x_web_crawler-0.1.4.tar.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jacobsomer/x-web-crawler/6399e6f0ffaa4beff88606b8d81d3cc22dccd4dc/dist/x_web_crawler-0.1.4.tar.gz -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | selenium -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | from setuptools import setup, find_packages 2 | 3 | setup( 4 | name='x-web-crawler', 5 | version='0.1.4', 6 | packages=find_packages(), 7 | install_requires=[ 8 | 'selenium', 9 | ], 10 | entry_points={ 11 | 'console_scripts': [ 12 | 'x_automation=autofollow.x:main', 13 | 'github_automation=autofollow.github:main', 14 | ], 15 | }, 16 | author='Jacob Somer', 17 | author_email='somerjacob@gmail.com', 18 | description='A web crawler for x.com', 19 | long_description=open('README.md').read(), 20 | long_description_content_type='text/markdown', 21 | url='https://github.com/jacobsomer/autofollow', 22 | classifiers=[ 23 | 'Programming Language :: Python :: 3', 24 | 'License :: OSI Approved :: MIT License', 25 | 'Operating System :: OS Independent', 26 | ], 27 | python_requires='>=3.6', 28 | ) 29 | -------------------------------------------------------------------------------- /x_web_crawler.egg-info/PKG-INFO: -------------------------------------------------------------------------------- 1 | Metadata-Version: 2.1 2 | Name: x-web-crawler 3 | Version: 0.1.4 4 | Summary: A web crawler for x.com 5 | Home-page: https://github.com/jacobsomer/autofollow 6 | Author: Jacob Somer 7 | Author-email: somerjacob@gmail.com 8 | Classifier: Programming Language :: Python :: 3 9 | Classifier: License :: OSI Approved :: MIT License 10 | Classifier: Operating System :: OS Independent 11 | Requires-Python: >=3.6 12 | Description-Content-Type: text/markdown 13 | License-File: LICENSE 14 | Requires-Dist: selenium 15 | 16 | 17 | # 🚀 x-web-crawler 18 | 19 | x-web-crawler is a Python package for automating interactions on social media platforms like Twitter (X) and GitHub. 20 | 21 | ## Installation 22 | 23 | You can install x-web-crawler using [pip](https://pypi.org/project/x-web-crawler/): 24 | 25 | ```bash 26 | pip install x-web-crawler 27 | ``` 28 | 29 | ## Usage 30 | 31 | Here’s an example of how to use x-web-crawler to automate actions on Twitter (X) and GitHub: 32 | 33 | ```python 34 | from xagent import XAgent 35 | 36 | def main(): 37 | driver_path = "YOUR_DRIVER_PATH" 38 | profile_path = "YOUR_PROFILE_PATH" 39 | twitter_username = "YOUR_TWITTER_USERNAME" # works best with your twitter handle and not email 40 | twitter_password = "YOUR_TWITTER_PASSWORD" 41 | url = "https://github.com/orgs/Azure/people" 42 | 43 | agent = XAgent( 44 | driver_path=driver_path, 45 | profile_path=profile_path, 46 | x_username=twitter_username, 47 | x_password=twitter_password, 48 | ) 49 | 50 | try: 51 | agent.like_x_posts(duration=300) 52 | agent.follow_x_users(["https://x.com/jacob_somer_"], duration=300) 53 | finally: 54 | agent.close() 55 | 56 | if __name__ == "__main__": 57 | main() 58 | ``` 59 | 60 | ## Examples 61 | 62 | ### Automating Twitter (X) Actions 63 | 64 | Here's how to automate actions on Twitter (X) using the Chrome browser: 65 | 66 | ```python 67 | from xagent import XAgent 68 | 69 | driver_path = "/Users/jacobsomer/Documents/side_prod/salesBook/chromedriver-mac-arm64/chromedriver" 70 | profile_path = "/Users/jacobsomer/Library/Application Support/Google/Chrome/chromeProfile" 71 | x_username = "YOUR_TWITTER_USERNAME" 72 | x_password = "YOUR_TWITTER_PASSWORD" 73 | 74 | agent = XAgent( 75 | driver_path=driver_path, 76 | profile_path=profile_path, 77 | x_username=x_username, 78 | x_password=x_password, 79 | ) 80 | 81 | try: 82 | # Like posts on your feed for 5 minutes 83 | agent.like_x_posts(duration=300) 84 | 85 | # Follow specific users 86 | agent.follow_x_users(["https://x.com/jacob_somer_"], duration=300) 87 | finally: 88 | agent.close() 89 | ``` 90 | 91 | ### Using Microsoft Edge Browser 92 | 93 | To automate actions using the Edge browser, modify the driver and profile paths: 94 | 95 | ```python 96 | from xagent.agent import XAgent 97 | 98 | edge_driver_path = "/Users/jacobsomer/Documents/side_prod/salesBook/edgedriver_mac64_m1 (1)/msedgedriver" 99 | edge_profile_path = "/Users/jacobsomer/Library/Application Support/Microsoft Edge/User Data" 100 | 101 | agent = XAgent( 102 | driver_path=edge_driver_path, 103 | profile_path=edge_profile_path, 104 | browser="edge" 105 | ) 106 | 107 | try: 108 | # Like posts on your feed for 5 minutes 109 | agent.like_x_posts(duration=300) 110 | 111 | # Follow specific users 112 | agent.follow_x_users(["https://x.com/jacob_somer_"], duration=300) 113 | finally: 114 | agent.close() 115 | ``` 116 | 117 | ### XAgent Methods 118 | 119 | #### `__init__(self, driver_path, profile_path, x_username=None, x_password=None, browser="chrome")` 120 | 121 | Initializes the XAgent. 122 | 123 | - `driver_path`: Path to the ChromeDriver executable. Download ChromeDriver from [here](https://googlechromelabs.github.io/chrome-for-testing/). 124 | - `profile_path`: Path to the user profile directory for Chrome. To find this, type "chrome://version" into your Chrome browser's address bar, and look for the "Profile Path" variable. 125 | - `x_username` (optional): Twitter (X) username for authentication. 126 | - `x_password` (optional): Twitter (X) password for authentication. 127 | - `browser`: The browser to use, either "chrome" or "edge". 128 | 129 | #### `like_x_posts(duration=300)` 130 | 131 | 👍 Likes posts on the user's feed for the specified duration. 132 | 133 | #### `follow_x_users(users, duration=300)` 134 | 135 | 👥 Follows the specified Twitter (X) users for the specified duration. 136 | 137 | #### `get_x_followers(username)` 138 | 139 | 📈 Gets the followers of a specified Twitter (X) user. 140 | 141 | #### `get_x_following(username)` 142 | 143 | 📊 Gets the users that a specified Twitter (X) user is following. 144 | 145 | #### `unfollow_x_users(users)` 146 | 147 | 🚫 Unfollows the specified Twitter (X) users. 148 | 149 | #### `unfollow_x_users_alternative(user, users)` 150 | 151 | 🔄 Unfollows the specified Twitter (X) users using an alternative method. 152 | 153 | #### `get_x_handles(query, num_handles=10)` 154 | 155 | 🔍 Gets the handles of users based on a query. 156 | 157 | ## Running Tests 158 | 159 | To run the tests for this package, use the `unittest` framework: 160 | 161 | ```bash 162 | python -m unittest discover tests 163 | ``` 164 | 165 | ## License 166 | 167 | This project is licensed under the MIT License - see the LICENSE file for details. 168 | --------------------------------------------------------------------------------