├── .gitignore
├── README.md
├── config-sample.json
├── message_example.png
├── requirements.txt
├── runner.py
└── upwork_job_feed_notifier.py
/.gitignore:
--------------------------------------------------------------------------------
1 |
2 | venv/
3 | .idea/
4 |
5 | config.json
6 | processed_jobs.json
7 |
8 | cron_out.txt
9 | *.log
10 |
11 | upwork_jobs.db
12 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 | # Upwork Job Feed Notifier
2 | This Python script fetches new job postings from one or more RSS feeds on Upwork and sends notifications to a Telegram chat with relevant job details.
3 |
4 |
5 | ## Usage
6 | 1. Clone the repository to your local machine.
7 | 2. Install the required libraries (`feedparser`, `requests`, `tzlocal`, and `bs4`) using pip.
8 | 3. Rename the `config-sample.json` file to `config.json` and update the contents with your own values.
9 |
10 | The `config.json` file contains the following settings:
11 | - `tgBotToken`: Your Telegram bot token.
12 | - `chat_id`: The chat ID of the Telegram chat to send notifications to.
13 | - `feed_url`: A list of URLs for the Upwork RSS feeds to monitor.
14 | 4. Run the script using python `upwork_job_feed_notifier.py`.
15 |
16 | ## Configuration
17 | The `config.json` file contains the following settings:
18 |
19 | - `tgBotToken`: Your Telegram bot token.
20 |
21 |
22 | How to get Telegram bot token
23 |
24 | 1. Open Telegram and search for the "BotFather" bot.
25 | 2. Start a chat with the BotFather by clicking on the "Start" button.
26 | 3. Type `/newbot` and follow the instructions to create a new bot.
27 | 4. Choose a name for your bot and a username that ends with "bot".
28 | 5. BotFather will provide you with a unique Token for your bot.
29 | 6. Save the Token in a safe place, as you will need it to communicate with your bot.
30 |
31 |
32 |
33 | - `chat_id`: The chat ID of the Telegram chat to send notifications to.
34 |
35 |
36 | How to get chat ID
37 |
38 | 1. Start a chat with your bot.
39 | 2. Send any message to your bot.
40 | 3. Open the following URL in your browser, replacing YOUR_BOT_TOKEN with the actual token for your bot:
41 | ```bash
42 | https://api.telegram.org/botYOUR_BOT_TOKEN/getUpdates
43 | ```
44 | 4. Look for the `"chat":{"id":` value in the response. This is your chat ID.
45 |
46 |
47 |
48 | - `feed_url`: A list of URLs for the Upwork RSS feeds to monitor.
49 |
50 |
51 | How to feed URL
52 |
53 | 1. Log in to your Upwork account.
54 | 2. Click on the "Find Work" tab in the top navigation menu.
55 | 3. Select the category you're interested in, and then select the subcategory.
56 | 4. Click on the "RSS" icon on the right side of the page.
57 | 5. Copy the URL in your browser's address bar. This is the RSS feed URL for that category/subcategory.
58 |
59 |
60 |
61 | # Setup script to run every interval
62 |
63 | ## Use runner.py
64 |
65 | > Just run the script using python3 runner.py and it will ask to set the interval. Just type in the values or leave it blank to set it to run every hour.
66 |
67 |
68 | Old Method
69 | Set up a cron job to run this script
70 | 1. Open your terminal and type crontab -e to open the crontab file in your default editor.
71 | 2. Add a new line to the crontab file to specify when you want the script to run. For example, if you want the script to run every hour at minute 50, add the following line:
72 | ```bash
73 | 50 * * * * /usr/bin/python3 /path/to/script/upwork_job_feed_notifier.py
74 | ```
75 | Replace /path/to/script.py with the actual path to your script file.
76 | 3. Save and close the crontab file.
77 | 4. The cron daemon will automatically start the script at the specified time. You can check the system log to verify that the script is running by typing `tail -f /var/log/syslog` in your terminal. If you see a message that says `CRON[xxx]: (username) CMD (/usr/bin/python3 /path/to/script/upwork_job_feed_notifier.py)`, it means the script is running.
78 |
79 |
80 | ## Screenshot of the notifiacation
81 | 
--------------------------------------------------------------------------------
/config-sample.json:
--------------------------------------------------------------------------------
1 | {
2 | "tgBotToken": "111111111:Your_Telegram_token",
3 | "feed_url": [
4 | "https://www.upwork.com/ab/feed/jobs/rss?q=...",
5 | "https://www.upwork.com/ab/feed/jobs/rss?q=..."
6 | ],
7 | "chat_id": "1234567"
8 | }
9 |
--------------------------------------------------------------------------------
/message_example.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/akashrchandran/Upwork-Notifier-Bot/288597af044aba20d042073e39f7864572a0eab0/message_example.png
--------------------------------------------------------------------------------
/requirements.txt:
--------------------------------------------------------------------------------
1 | bs4
2 | feedparser
3 | requests
4 | tzlocal==4.2
5 | APScheduler
--------------------------------------------------------------------------------
/runner.py:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env python3
2 | from apscheduler.schedulers.blocking import BlockingScheduler
3 | import subprocess
4 | import sys
5 | import os
6 |
7 | python_command = sys.executable
8 | script_to_run = "upwork_job_feed_notifier.py"
9 |
10 |
11 | # function to invoke the main script
12 | def cron_process():
13 | print("checking for new notifications...")
14 | subprocess.Popen([python_command, script_to_run])
15 |
16 |
17 | logo = """
18 | ooooo ooo oooo
19 | `888' `8' `888
20 | 888 8 oo.ooooo. oooo oooo ooo .ooooo. oooo d8b 888 oooo
21 | 888 8 888' `88b `88. `88. .8' d88' `88b `888""8P 888 .8P'
22 | 888 8 888 888 `88..]88..8' 888 888 888 888888.
23 | `88. .8' 888 888 `888'`888' 888 888 888 888 `88b.
24 | `YbodP' 888bod8P' `8' `8' `Y8bod8P' d888b o888o o888o
25 | 888
26 | o888o
27 | ~ Notifier
28 | """
29 | print(logo, end="\n\n")
30 |
31 | if not os.path.exists("config.json"):
32 | print("Config File Not Found, Please read README.MD")
33 | exit()
34 | print("Enter the invterval to run the script: ")
35 | print("Leave blank to run the script every hour")
36 |
37 |
38 | # if user leave it blank assgin 1 to hours
39 | hours = int(input("Hours: ") or 1)
40 |
41 | # if user leave it blank assgin 0 to minutes and seconds
42 | minutes = int(input("Minutes: ") or 0)
43 | seconds = int(input("Seconds: ") or 0)
44 |
45 | # initialize the scheduler
46 | scheduler = BlockingScheduler(timezone="asia/kolkata")
47 |
48 | # add the job to the scheduler
49 | scheduler.add_job(
50 | cron_process, "interval", hours=hours, minutes=minutes, seconds=seconds
51 | )
52 |
53 | # start the scheduler
54 | scheduler.start()
55 |
--------------------------------------------------------------------------------
/upwork_job_feed_notifier.py:
--------------------------------------------------------------------------------
1 | import json
2 | import logging
3 | import os
4 | import sqlite3
5 | from datetime import datetime
6 | from re import findall
7 |
8 | import feedparser
9 | import requests
10 | import tzlocal
11 | from bs4 import BeautifulSoup
12 |
13 | # Get the absolute path of the script file
14 | script_dir = os.path.dirname(os.path.abspath(__file__))
15 | processed_jobs_file = os.path.join(script_dir, 'processed_jobs.json')
16 | configs_file = os.path.join(script_dir, 'config.json')
17 |
18 | logging.basicConfig(
19 | format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
20 | level=logging.INFO,
21 | filename=f'{script_dir}/upwork_scraper.log'
22 | )
23 | logger = logging.getLogger(__name__)
24 |
25 | # Load configurations from config.json file
26 | logger.debug('Reading config file')
27 | with open(configs_file, 'r') as f:
28 | config = json.load(f)
29 |
30 | logger.debug('Creating connection to the SQLite database')
31 | # Create a connection to the SQLite database
32 | db_file = os.path.join(script_dir, 'upwork_jobs.db')
33 | conn = sqlite3.connect(db_file)
34 | cursor = conn.cursor()
35 |
36 | # Create the jobs table if it doesn't exist
37 | cursor.execute('''
38 | CREATE TABLE IF NOT EXISTS jobs (
39 | job_id TEXT PRIMARY KEY,
40 | title TEXT,
41 | category TEXT,
42 | rate TEXT,
43 | summary TEXT,
44 | link TEXT,
45 | posted_on TEXT,
46 | country TEXT,
47 | skills TEXT
48 | );
49 | ''')
50 |
51 | tgBotToken = config['tgBotToken']
52 | feed_urls = config['feed_url']
53 | bot_url = f'https://api.telegram.org/bot{tgBotToken}/'
54 | chat_id = config['chat_id']
55 |
56 | # Set your local timezone
57 | local_tz = tzlocal.get_localzone()
58 |
59 | # Fetch new jobs from the RSS feed and send notifications to Telegram
60 | logger.debug('Fetching jobs from the RSS feed')
61 | for feed_url in feed_urls:
62 | # Parse the RSS feed
63 | feed = feedparser.parse(feed_url)
64 |
65 | # Loop through the entries in the feed (most recent first)
66 | for entry in reversed(feed.entries):
67 | # Check if this job has already been processed
68 | job_id = findall(r'(?<=_)%([a-zA-Z0-9]+)', entry.link)[0]
69 | cursor.execute('SELECT * FROM jobs WHERE job_id = ?', (job_id,))
70 | if cursor.fetchone():
71 | continue
72 | logger.debug('New job was found')
73 |
74 | # Convert the published time to your local timezone
75 | published_time = datetime.strptime(entry.published, '%a, %d %b %Y %H:%M:%S %z')
76 | published_time = published_time.astimezone(local_tz)
77 |
78 | # Calculate the relative time since the job was published
79 | now = datetime.now(local_tz)
80 | relative_time = now - published_time
81 | if relative_time.days > 1:
82 | relative_time = f"{relative_time.days} days"
83 | else:
84 | total_seconds = relative_time.total_seconds()
85 | hours = int(total_seconds // 3600)
86 | minutes = int((total_seconds % 3600) // 60)
87 | relative_time = f"{hours}h {minutes}m"
88 |
89 | posted_on = f'{relative_time} ago ({published_time.strftime("%Y-%m-%d %H:%M")})'
90 |
91 | # Parse the RSS entry
92 | soup = BeautifulSoup(entry.content[0]['value'], 'html.parser')
93 |
94 | # Get payment type
95 | budget = soup.find('b', string='Budget')
96 | hourly_rate = soup.find('b', string='Hourly Range')
97 | try:
98 | rate = budget.find_next_sibling(string=True) if budget else hourly_rate.find_next_sibling(string=True)
99 | rate = rate.replace(":", "").replace("\n", "").strip()
100 | rate = (
101 | f'Budget {rate}'
102 | if budget
103 | else f'Hourly {rate}'
104 | if hourly_rate
105 | else 'N/A'
106 | )
107 | except Exception as e:
108 | logger.debug(f'Rate is not available for {entry.link.strip()}: {e}')
109 | rate = 'N/A'
110 |
111 | # Get job category
112 | category = soup.find('b', string='Category').find_next_sibling(string=True).replace(":", "").strip().replace(" ", "_").replace("-", "_").replace("/", "_").replace("&", "and")
113 |
114 | # Get customer country
115 | try:
116 | country = soup.find('b', string='Country').find_next_sibling(string=True).replace(":", "").strip()
117 | except Exception as e:
118 | country='N/A'
119 | # Get required skill and format them as hashtags
120 | try:
121 | skills = soup.find('b', string='Skills').find_next_sibling(string=True).replace(":", "").strip()
122 | except Exception as e:
123 | skills='N/A'
124 | try:
125 | skills_hashtags = " ".join(["#" + word.strip().replace(" ", "_").replace("-", "_").replace("/", "_").replace("&", "and") for word in skills.split(", ")[:10]]).strip()
126 | except Exception as e:
127 | skills_hashtags = "N/A"
128 | # Get the 1st sentence of the summary
129 | summary = (entry.summary.split('.')[0] + ".").replace("
", "\n").replace("
", "\n").replace("
", "\n").replace("
", "\n").replace('\n\n', '\n')
130 |
131 | # Build the message to send to Telegram
132 | message = f'{entry.title.replace(" - Upwork", "")}' \
133 | f'\n#{category}' \
134 | f'\n💲 {rate}' \
135 | f'\n\n📄 {summary}' \
136 | f'\n🔗 {entry.link.strip()}' \
137 | f'\n\n🕑 {posted_on}' \
138 | f'\n🌍 {country}' \
139 | f'\n\n{skills_hashtags}'
140 |
141 | # Send the message to Telegram
142 | payload = {'chat_id': chat_id, 'text': message, 'parse_mode': 'HTML', 'disable_web_page_preview': True}
143 | try:
144 | r = requests.post(f'{bot_url}sendMessage', json=payload)
145 | r.raise_for_status()
146 | logger.debug('Message sent successfully')
147 | except requests.exceptions.RequestException as e:
148 | logger.error(f"Error sending message to Telegram: {e}")
149 | continue
150 | # Add the job ID to the list of processed jobs
151 | logger.debug(f'Saving job {job_id} to db')
152 | cursor.execute("INSERT INTO jobs VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
153 | (job_id, entry.title, category, rate, summary, entry.link, entry.published, country, skills))
154 |
155 | # Save the processed jobs to db
156 | conn.commit()
157 | conn.close()
158 |
--------------------------------------------------------------------------------