├── .gitignore ├── README.md ├── config-sample.json ├── message_example.png ├── requirements.txt ├── runner.py └── upwork_job_feed_notifier.py /.gitignore: -------------------------------------------------------------------------------- 1 | 2 | venv/ 3 | .idea/ 4 | 5 | config.json 6 | processed_jobs.json 7 | 8 | cron_out.txt 9 | *.log 10 | 11 | upwork_jobs.db 12 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Upwork Job Feed Notifier 2 | This Python script fetches new job postings from one or more RSS feeds on Upwork and sends notifications to a Telegram chat with relevant job details. 3 | 4 | 5 | ## Usage 6 | 1. Clone the repository to your local machine. 7 | 2. Install the required libraries (`feedparser`, `requests`, `tzlocal`, and `bs4`) using pip. 8 | 3. Rename the `config-sample.json` file to `config.json` and update the contents with your own values. 9 | 10 | The `config.json` file contains the following settings: 11 | - `tgBotToken`: Your Telegram bot token. 12 | - `chat_id`: The chat ID of the Telegram chat to send notifications to. 13 | - `feed_url`: A list of URLs for the Upwork RSS feeds to monitor. 14 | 4. Run the script using python `upwork_job_feed_notifier.py`. 15 | 16 | ## Configuration 17 | The `config.json` file contains the following settings: 18 | 19 | - `tgBotToken`: Your Telegram bot token. 20 |

21 |

22 | How to get Telegram bot token 23 | 24 | 1. Open Telegram and search for the "BotFather" bot. 25 | 2. Start a chat with the BotFather by clicking on the "Start" button. 26 | 3. Type `/newbot` and follow the instructions to create a new bot. 27 | 4. Choose a name for your bot and a username that ends with "bot". 28 | 5. BotFather will provide you with a unique Token for your bot. 29 | 6. Save the Token in a safe place, as you will need it to communicate with your bot. 30 | 31 |
32 |

33 | - `chat_id`: The chat ID of the Telegram chat to send notifications to. 34 |

35 |

36 | How to get chat ID 37 | 38 | 1. Start a chat with your bot. 39 | 2. Send any message to your bot. 40 | 3. Open the following URL in your browser, replacing YOUR_BOT_TOKEN with the actual token for your bot: 41 | ```bash 42 | https://api.telegram.org/botYOUR_BOT_TOKEN/getUpdates 43 | ``` 44 | 4. Look for the `"chat":{"id":` value in the response. This is your chat ID. 45 | 46 |
47 |

48 | - `feed_url`: A list of URLs for the Upwork RSS feeds to monitor. 49 |

50 |

51 | How to feed URL 52 | 53 | 1. Log in to your Upwork account. 54 | 2. Click on the "Find Work" tab in the top navigation menu. 55 | 3. Select the category you're interested in, and then select the subcategory. 56 | 4. Click on the "RSS" icon on the right side of the page. 57 | 5. Copy the URL in your browser's address bar. This is the RSS feed URL for that category/subcategory. 58 |
59 |

60 | 61 | # Setup script to run every interval 62 | 63 | ## Use runner.py 64 | 65 | > Just run the script using python3 runner.py and it will ask to set the interval. Just type in the values or leave it blank to set it to run every hour. 66 | 67 |
68 | Old Method 69 |

Set up a cron job to run this script

70 | 1. Open your terminal and type crontab -e to open the crontab file in your default editor. 71 | 2. Add a new line to the crontab file to specify when you want the script to run. For example, if you want the script to run every hour at minute 50, add the following line: 72 | ```bash 73 | 50 * * * * /usr/bin/python3 /path/to/script/upwork_job_feed_notifier.py 74 | ``` 75 | Replace /path/to/script.py with the actual path to your script file. 76 | 3. Save and close the crontab file. 77 | 4. The cron daemon will automatically start the script at the specified time. You can check the system log to verify that the script is running by typing `tail -f /var/log/syslog` in your terminal. If you see a message that says `CRON[xxx]: (username) CMD (/usr/bin/python3 /path/to/script/upwork_job_feed_notifier.py)`, it means the script is running. 78 |
79 | 80 | ## Screenshot of the notifiacation 81 | ![Telegram message](message_example.png) -------------------------------------------------------------------------------- /config-sample.json: -------------------------------------------------------------------------------- 1 | { 2 | "tgBotToken": "111111111:Your_Telegram_token", 3 | "feed_url": [ 4 | "https://www.upwork.com/ab/feed/jobs/rss?q=...", 5 | "https://www.upwork.com/ab/feed/jobs/rss?q=..." 6 | ], 7 | "chat_id": "1234567" 8 | } 9 | -------------------------------------------------------------------------------- /message_example.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/akashrchandran/Upwork-Notifier-Bot/288597af044aba20d042073e39f7864572a0eab0/message_example.png -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | bs4 2 | feedparser 3 | requests 4 | tzlocal==4.2 5 | APScheduler -------------------------------------------------------------------------------- /runner.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | from apscheduler.schedulers.blocking import BlockingScheduler 3 | import subprocess 4 | import sys 5 | import os 6 | 7 | python_command = sys.executable 8 | script_to_run = "upwork_job_feed_notifier.py" 9 | 10 | 11 | # function to invoke the main script 12 | def cron_process(): 13 | print("checking for new notifications...") 14 | subprocess.Popen([python_command, script_to_run]) 15 | 16 | 17 | logo = """ 18 | ooooo ooo oooo 19 | `888' `8' `888 20 | 888 8 oo.ooooo. oooo oooo ooo .ooooo. oooo d8b 888 oooo 21 | 888 8 888' `88b `88. `88. .8' d88' `88b `888""8P 888 .8P' 22 | 888 8 888 888 `88..]88..8' 888 888 888 888888. 23 | `88. .8' 888 888 `888'`888' 888 888 888 888 `88b. 24 | `YbodP' 888bod8P' `8' `8' `Y8bod8P' d888b o888o o888o 25 | 888 26 | o888o 27 | ~ Notifier 28 | """ 29 | print(logo, end="\n\n") 30 | 31 | if not os.path.exists("config.json"): 32 | print("Config File Not Found, Please read README.MD") 33 | exit() 34 | print("Enter the invterval to run the script: ") 35 | print("Leave blank to run the script every hour") 36 | 37 | 38 | # if user leave it blank assgin 1 to hours 39 | hours = int(input("Hours: ") or 1) 40 | 41 | # if user leave it blank assgin 0 to minutes and seconds 42 | minutes = int(input("Minutes: ") or 0) 43 | seconds = int(input("Seconds: ") or 0) 44 | 45 | # initialize the scheduler 46 | scheduler = BlockingScheduler(timezone="asia/kolkata") 47 | 48 | # add the job to the scheduler 49 | scheduler.add_job( 50 | cron_process, "interval", hours=hours, minutes=minutes, seconds=seconds 51 | ) 52 | 53 | # start the scheduler 54 | scheduler.start() 55 | -------------------------------------------------------------------------------- /upwork_job_feed_notifier.py: -------------------------------------------------------------------------------- 1 | import json 2 | import logging 3 | import os 4 | import sqlite3 5 | from datetime import datetime 6 | from re import findall 7 | 8 | import feedparser 9 | import requests 10 | import tzlocal 11 | from bs4 import BeautifulSoup 12 | 13 | # Get the absolute path of the script file 14 | script_dir = os.path.dirname(os.path.abspath(__file__)) 15 | processed_jobs_file = os.path.join(script_dir, 'processed_jobs.json') 16 | configs_file = os.path.join(script_dir, 'config.json') 17 | 18 | logging.basicConfig( 19 | format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', 20 | level=logging.INFO, 21 | filename=f'{script_dir}/upwork_scraper.log' 22 | ) 23 | logger = logging.getLogger(__name__) 24 | 25 | # Load configurations from config.json file 26 | logger.debug('Reading config file') 27 | with open(configs_file, 'r') as f: 28 | config = json.load(f) 29 | 30 | logger.debug('Creating connection to the SQLite database') 31 | # Create a connection to the SQLite database 32 | db_file = os.path.join(script_dir, 'upwork_jobs.db') 33 | conn = sqlite3.connect(db_file) 34 | cursor = conn.cursor() 35 | 36 | # Create the jobs table if it doesn't exist 37 | cursor.execute(''' 38 | CREATE TABLE IF NOT EXISTS jobs ( 39 | job_id TEXT PRIMARY KEY, 40 | title TEXT, 41 | category TEXT, 42 | rate TEXT, 43 | summary TEXT, 44 | link TEXT, 45 | posted_on TEXT, 46 | country TEXT, 47 | skills TEXT 48 | ); 49 | ''') 50 | 51 | tgBotToken = config['tgBotToken'] 52 | feed_urls = config['feed_url'] 53 | bot_url = f'https://api.telegram.org/bot{tgBotToken}/' 54 | chat_id = config['chat_id'] 55 | 56 | # Set your local timezone 57 | local_tz = tzlocal.get_localzone() 58 | 59 | # Fetch new jobs from the RSS feed and send notifications to Telegram 60 | logger.debug('Fetching jobs from the RSS feed') 61 | for feed_url in feed_urls: 62 | # Parse the RSS feed 63 | feed = feedparser.parse(feed_url) 64 | 65 | # Loop through the entries in the feed (most recent first) 66 | for entry in reversed(feed.entries): 67 | # Check if this job has already been processed 68 | job_id = findall(r'(?<=_)%([a-zA-Z0-9]+)', entry.link)[0] 69 | cursor.execute('SELECT * FROM jobs WHERE job_id = ?', (job_id,)) 70 | if cursor.fetchone(): 71 | continue 72 | logger.debug('New job was found') 73 | 74 | # Convert the published time to your local timezone 75 | published_time = datetime.strptime(entry.published, '%a, %d %b %Y %H:%M:%S %z') 76 | published_time = published_time.astimezone(local_tz) 77 | 78 | # Calculate the relative time since the job was published 79 | now = datetime.now(local_tz) 80 | relative_time = now - published_time 81 | if relative_time.days > 1: 82 | relative_time = f"{relative_time.days} days" 83 | else: 84 | total_seconds = relative_time.total_seconds() 85 | hours = int(total_seconds // 3600) 86 | minutes = int((total_seconds % 3600) // 60) 87 | relative_time = f"{hours}h {minutes}m" 88 | 89 | posted_on = f'{relative_time} ago ({published_time.strftime("%Y-%m-%d %H:%M")})' 90 | 91 | # Parse the RSS entry 92 | soup = BeautifulSoup(entry.content[0]['value'], 'html.parser') 93 | 94 | # Get payment type 95 | budget = soup.find('b', string='Budget') 96 | hourly_rate = soup.find('b', string='Hourly Range') 97 | try: 98 | rate = budget.find_next_sibling(string=True) if budget else hourly_rate.find_next_sibling(string=True) 99 | rate = rate.replace(":", "").replace("\n", "").strip() 100 | rate = ( 101 | f'Budget {rate}' 102 | if budget 103 | else f'Hourly {rate}' 104 | if hourly_rate 105 | else 'N/A' 106 | ) 107 | except Exception as e: 108 | logger.debug(f'Rate is not available for {entry.link.strip()}: {e}') 109 | rate = 'N/A' 110 | 111 | # Get job category 112 | category = soup.find('b', string='Category').find_next_sibling(string=True).replace(":", "").strip().replace(" ", "_").replace("-", "_").replace("/", "_").replace("&", "and") 113 | 114 | # Get customer country 115 | try: 116 | country = soup.find('b', string='Country').find_next_sibling(string=True).replace(":", "").strip() 117 | except Exception as e: 118 | country='N/A' 119 | # Get required skill and format them as hashtags 120 | try: 121 | skills = soup.find('b', string='Skills').find_next_sibling(string=True).replace(":", "").strip() 122 | except Exception as e: 123 | skills='N/A' 124 | try: 125 | skills_hashtags = " ".join(["#" + word.strip().replace(" ", "_").replace("-", "_").replace("/", "_").replace("&", "and") for word in skills.split(", ")[:10]]).strip() 126 | except Exception as e: 127 | skills_hashtags = "N/A" 128 | # Get the 1st sentence of the summary 129 | summary = (entry.summary.split('.')[0] + ".").replace("
", "\n").replace("
", "\n").replace("
", "\n").replace("
", "\n").replace('\n\n', '\n') 130 | 131 | # Build the message to send to Telegram 132 | message = f'{entry.title.replace(" - Upwork", "")}' \ 133 | f'\n#{category}' \ 134 | f'\n💲 {rate}' \ 135 | f'\n\n📄 {summary}' \ 136 | f'\n🔗 {entry.link.strip()}' \ 137 | f'\n\n🕑 {posted_on}' \ 138 | f'\n🌍 {country}' \ 139 | f'\n\n{skills_hashtags}' 140 | 141 | # Send the message to Telegram 142 | payload = {'chat_id': chat_id, 'text': message, 'parse_mode': 'HTML', 'disable_web_page_preview': True} 143 | try: 144 | r = requests.post(f'{bot_url}sendMessage', json=payload) 145 | r.raise_for_status() 146 | logger.debug('Message sent successfully') 147 | except requests.exceptions.RequestException as e: 148 | logger.error(f"Error sending message to Telegram: {e}") 149 | continue 150 | # Add the job ID to the list of processed jobs 151 | logger.debug(f'Saving job {job_id} to db') 152 | cursor.execute("INSERT INTO jobs VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", 153 | (job_id, entry.title, category, rate, summary, entry.link, entry.published, country, skills)) 154 | 155 | # Save the processed jobs to db 156 | conn.commit() 157 | conn.close() 158 | --------------------------------------------------------------------------------