├── .example-env
├── .gitignore
├── README.md
├── ta-helper-trigger.py
└── ta-helper.py


/.example-env:
--------------------------------------------------------------------------------
 1 | # Logging Level - valid levels: CRITICAL,ERROR,WARNING,INFO,DEBUG,NOTSET
 2 | # Normally left at INFO.  Set to DEBUG for more logging.
 3 | LOGLEVEL="INFO"
 4 | 
 5 | # URL for your Tube Archivist server, including port.
 6 | TA_SERVER = "http://192.168.1.11:8000"
 7 | 
 8 | # TA API token obtained from:
 9 | # <TA_SERVER>/settings/#integrations
10 | TA_TOKEN = "c0ff142e1c336e9f43be560b5c942d61e7e7c7fb"
11 | 
12 | # The host path to the TA Docker cache channel folder.
13 | # Just drop "/cache" from your docker's config value.
14 | # If you do not have access to the cache folder, leave it empty: "".
15 | TA_CACHE = "/home/me/dockers/YouTube"
16 | 
17 | # Folder where TA stores its videos with Channel/Title ID's
18 | TA_MEDIA_FOLDER = "/home/me/Videos/YouTube"
19 | 
20 | # Folder where this script will put human readable symlinks to TA's
21 | # obfuscated videos, as well as per video NFO files for media managers.
22 | TARGET_FOLDER = "/home/me/Videos/YT-Subs"
23 | 
24 | # "True" for enable, "False" for disable
25 | NOTIFICATIONS_ENABLED = "True"
26 | 
27 | # Mail info for sending notifications
28 | MAIL_USER="me@x.com"
29 | # Can use 1 or multiple destination emails seperated by ','
30 | MAIL_RECIPIENTS="me@x.com,you@x.com"
31 | 
32 | # Whether this script should generate media NFO files
33 | # "True" for enable, "False" for disable
34 | GENERATE_NFO = "True"
35 | 
36 | # Instruction to tell apprise how to notify.  Read all of the options
37 | # here: https://pypi.org/project/apprise/
38 | APPRISE_LINK = "mailto://<username>:<password>@gmail.com"
39 | 
40 | # Stop processing channel once an already indexed video is reached
41 | QUICK = "True"
42 | 
43 | # Set this to the port you'd like to be notified on.
44 | # Make sure you have no conflicts.
45 | # Change your apprise links in TA settings to match:
46 | # json://<IP/hostname where TA helper will run>:<PORT>/tahelper-trigger
47 | # For example: json://192.168.1.11:8001/ta-helper-trigger
48 | APPRISE_TRIGGER_PORT=8001
49 | 
50 | # Set this path to point to your ta-helper.py script
51 | TA_HELPER_SCRIPT="/home/me/projects/ta-helper/ta-helper.py"
52 | 
53 | # TA can be configured to delete watched videos.  If a video is deleted the
54 | # symbolic link to it in the TARGET_FOLDER becomes bad.  The bad symlinks can
55 | # be used to trigger resource cleanup of deleted videos.  So if the symlink
56 | # to video "x.mp4" becomes bad then we should delete the x.NFO file and x.vtt
57 | # and x.mp4 symlinks.
58 | CLEANUP_DELETED_VIDEOS = "False"
59 | 


--------------------------------------------------------------------------------
/.gitignore:
--------------------------------------------------------------------------------
1 | .env
2 | sendEmail_service_key.json
3 | *.log
4 | __pycache__
5 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | #######################################################
 2 | # Project No Longer Active
 3 | #######################################################
 4 | 
 5 | # ta-helper
 6 | 
 7 | Python script to post process Tube Archivist products, generate human readable folders and files, support NFO files and new per video notifications using apprise library.  Various functions can be enabled/disabled.  Nothing is touched in your Tube Archivist media folders.  The image below shows your original obfuscated naming on the left and new, human readable folders on the right.
 8 | 
 9 | ![Screenshot from 2023-08-12 16-56-04](https://github.com/RoninTech/ta-helper/assets/226861/4cf31133-8d40-4a93-b363-cf8f26054f25)
10 | 
11 | Here is how your NFO file supporting media manager (Kodi, Emby, Jellyfin etc.) can look after you add your new YouTube videos folder:
12 | 
13 | ![screenshot00000](https://github.com/RoninTech/ta-helper/assets/226861/b2625c9f-c600-43ac-9b72-cdacc9f6ea7f)
14 | 
15 | ![screenshot00002](https://github.com/RoninTech/ta-helper/assets/226861/ad2a539a-3b84-4045-9c98-4e78886ae3db)
16 | 
17 | ## Using It
18 | 
19 | Copy .example-env to .env in the same folder as the script and edit to put in your own settings. .example-env has detailed comments that explain how to use each setting.
20 | 
21 | ## What exactly does it do?
22 | 
23 | It iterates through the Tube Archivist video folders and does several things:
24 | 
25 | 1. Creates a mirror set of folders via symbolic links with the actual channel names and video titles for human readability.
26 | 2. If GENERATE_NFO is set to "True" it will generate .nfo files for each new channel and/or video which allows media managers such as Kodi, Emby, Jellyfin etc. to show meta info.
27 | 3. If TA_CACHE is not "" it can generate symbolic links to subtitles, poster, cover and banner jpg's inside TA cache for media managers.
28 | 4. If NOTIFICATIONS_ENABLED is set to "True" in your .env, apprise will be used to notify of new videos using the apprise URL you provide, also in .env, based off the [apprise documentation](https://github.com/caronc/apprise/wiki). Here i san example of an apprise link to send notification via Gmail: "mailto://<username>:<password>@gmail.com"
29 | 5. If CLEANUP_DELETED_VIDEOS is set to "True" any broken symlinks or hanging nfo files (nfo file with no corresponding video) will be deleted from TARGET_FOLDER.  So if TA is configured to delete watched videos, this will clean-up any leftovers.
30 | 
31 | **NOTE:** When apprise is setup to send emails via gmail, each notification takes approx 3s on a Raspberry Pi4.  So if you are doing an initial run on a large library, temporarily setting NOTIFICATIONS_ENABLED to "False" will save a lot of time.
32 | 
33 | ## Triggering ta-helper to run:
34 | 
35 | To kickstart the ta-helper when TA adds new videos to the archive we have 3 options:
36 | 
37 | 1. Manually run the script when you know new videos have been added via "python ta-helper.py"
38 |    
39 | 2. Use a cron job to periodically run the script, say every half an hour.  Using crontab -e to add a cron job.  This would trigger it every 30 minutes: 30 * * * * python /home/me/projects/ta-helper/ta-helper.py
40 | 
41 | 3. Use apprise notifications from TA server as event driven triggers to run the script only when needed.  More efficient than cron polling described in (2) above. We use apprise via the Settings page.  Under "Start Download" set the apprise notification URL to "json://<IP/Hostname>:<PORT>/ta-helper-trigger".  For example I use: "json://192.168.1.11:8001/ta-helper-trigger".  **NOTE**: Remove the quotes before copying into TA settings.  To process these notifications and trigger the script, simply run "python ta-helper-trigger.py" on the machine where your TA archive is.  Make sure the PORT value in ta-helper-trigger.py matches the PORT you used in the TA apprise link mentioned above.  I added the following line (without quotes) as a cron job (using crontab -e): "@reboot python /home/me/projects/ta-helper/ta-helper-trigger.py > /home/me/Desktop/ta-trigger.log".  Change these paths to match your own config.
42 | 
43 | Most up to date version of script can be found on GitHub, [here](https://github.com/RoninTech/ta-helper).
44 | 
45 | Script originally created by user GordonFreeman on the [Tube Archivist discord server](https://www.tubearchivist.com/discord).
46 | 


--------------------------------------------------------------------------------
/ta-helper-trigger.py:
--------------------------------------------------------------------------------
 1 | from dotenv import load_dotenv
 2 | from flask import Flask, request, Response
 3 | import os
 4 | import subprocess
 5 | 
 6 | # Pull configuration details from .env file.
 7 | load_dotenv()
 8 | TA_HELPER_SCRIPT = os.environ.get("TA_HELPER_SCRIPT")
 9 | APPRISE_TRIGGER_PORT = os.environ.get("APPRISE_TRIGGER_PORT")
10 | 
11 | app = Flask(__name__)
12 | @app.route('/ta-helper-trigger', methods=['POST'])
13 | 
14 | def return_response():
15 |     print(request.json);
16 |     result = Response(status=200)
17 | 
18 |     # Kickstart ta-helper script as there have been changes.
19 |     print("TA has made changes to the video archive, invoking helper script.")
20 |     
21 |     # Use Popen so we immediately return and sending apprise doesn't time out.
22 |     subprocess.Popen(["python", TA_HELPER_SCRIPT])
23 |     
24 |     ## Do something with the request.json data.
25 |     return result
26 | 
27 | if __name__ == "__main__": app.run(host="0.0.0.0", port=APPRISE_TRIGGER_PORT)
28 | 


--------------------------------------------------------------------------------
/ta-helper.py:
--------------------------------------------------------------------------------
  1 | import apprise
  2 | from distutils.util import strtobool
  3 | from dotenv import load_dotenv
  4 | import html2text
  5 | import logging
  6 | import os
  7 | import requests
  8 | import re
  9 | import sys
 10 | import time
 11 | 
 12 | # Configure logging.
 13 | logger = logging.getLogger(__name__)
 14 | handler = logging.StreamHandler()
 15 | formatter = logging.Formatter(fmt='%(asctime)s %(filename)s:%(lineno)s %(levelname)-8s %(message)s',
 16 |                               datefmt='%d-%b-%y %H:%M:%S')
 17 | handler.setFormatter(formatter)
 18 | logger.addHandler(handler)
 19 | 
 20 | # Pull configuration details from .env file.
 21 | load_dotenv()
 22 | NOTIFICATIONS_ENABLED = bool(strtobool(os.environ.get("NOTIFICATIONS_ENABLED", 'False')))
 23 | GENERATE_NFO = bool(strtobool(os.environ.get("GENERATE_NFO", 'False')))
 24 | FROMADDR = str(os.environ.get("MAIL_USER"))
 25 | RECIPIENTS = str(os.environ.get("MAIL_RECIPIENTS"))
 26 | RECIPIENTS = RECIPIENTS.split(',')
 27 | TA_MEDIA_FOLDER = str(os.environ.get("TA_MEDIA_FOLDER"))
 28 | TA_SERVER = str(os.environ.get("TA_SERVER"))
 29 | TA_TOKEN = str(os.environ.get("TA_TOKEN"))
 30 | TA_CACHE = str(os.environ.get("TA_CACHE"))
 31 | TARGET_FOLDER = str(os.environ.get("TARGET_FOLDER"))
 32 | APPRISE_LINK = str(os.environ.get("APPRISE_LINK"))
 33 | QUICK = bool(strtobool(os.environ.get("QUICK", 'True')))
 34 | CLEANUP_DELETED_VIDEOS = bool(strtobool(str(os.environ.get("CLEANUP_DELETED_VIDEOS"))))
 35 | 
 36 | logger.setLevel(os.environ.get("LOGLEVEL", "INFO"))
 37 | 
 38 | def setup_new_channel_resources(chan_name, chan_data):
 39 |     logger.info("New Channel %s, setup resources.", chan_name)
 40 |     if TA_CACHE == "":
 41 |         logger.info("No TA_CACHE available so cannot setup symlinks to cache files.")
 42 |     else:
 43 |         # Link the channel logo from TA docker cache into target folder for media managers
 44 |         # and file explorers.  Provide cover.jpg, poster.jpg and banner.jpg symlinks.
 45 |         channel_thumb_path = TA_CACHE + chan_data['channel_thumb_url']
 46 |         logger.info("Symlink cache %s thumb to poster, cover and folder.jpg files.", channel_thumb_path)
 47 |         os.symlink(channel_thumb_path, TARGET_FOLDER + "/" + chan_name + "/" + "poster.jpg")
 48 |         os.symlink(channel_thumb_path, TARGET_FOLDER + "/" + chan_name + "/" + "cover.jpg")
 49 |         os.symlink(channel_thumb_path, TARGET_FOLDER + "/" + chan_name + "/" + "folder.jpg")
 50 |         channel_banner_path = TA_CACHE + chan_data['channel_banner_url']
 51 |         os.symlink(channel_banner_path, TARGET_FOLDER + "/" + chan_name + "/" + "banner.jpg")
 52 |     
 53 |     # Generate tvshow.nfo for media managers, no TA_CACHE required.
 54 |     logger.info("Generating %s", TARGET_FOLDER + "/" + chan_name + "/" + "tvshow.nfo")
 55 |     f= open(TARGET_FOLDER + "/" + chan_name + "/" + "tvshow.nfo","w+")
 56 |     f.write('<?xml version="1.0" encoding="UTF-8" standalone="yes"?>' + '\n'
 57 |             '<tvshow>' + '\n\t' + '<title>' +
 58 |             chan_data['channel_name'] + "</title>\n\t" +
 59 |             "<showtitle>" + chan_data['channel_name'] + "</showtitle>\n\t" +
 60 |             "<uniqueid>" + chan_data['channel_id'] + "</uniqueid>\n\t" +
 61 |             "<plot>" + chan_data['channel_description'] + "</plot>\n\t" +
 62 |             "<premiered>" + chan_data['channel_last_refresh'] + "</premiered>\n</episodedetails>")
 63 |     f.close()
 64 | 
 65 | def generate_new_video_nfo(chan_name, title, video_meta_data):
 66 |     logger.info("Generating NFO file and subtitle symlink for %s video: %s", video_meta_data['channel']['channel_name'], video_meta_data['title'])
 67 |     # TA has added a new video.  Create a symlink to subtitles and an NFO file for media managers.
 68 |     video_basename = os.path.splitext(video_meta_data['media_url'])[0]
 69 |     os.symlink(TA_MEDIA_FOLDER + video_basename + ".en.vtt", TARGET_FOLDER + "/" + chan_name + "/" + title.replace(".mp4",".en.vtt"))
 70 |     title = title.replace('.mp4','.nfo')
 71 |     f= open(TARGET_FOLDER + "/" + chan_name + "/" + title,"w+")
 72 |     f.write('<?xml version="1.0" ?>\n<episodedetails>\n\t' +
 73 |         "<title>" + video_meta_data['title'] + "</title>\n\t" +
 74 |         "<showtitle>" + video_meta_data['channel']['channel_name'] + "</showtitle>\n\t" +
 75 |         "<uniqueid>" + video_meta_data['youtube_id'] + "</uniqueid>\n\t" +
 76 |         "<plot>" + video_meta_data['description'] + "</plot>\n\t" +
 77 |         "<premiered>" + video_meta_data['published'] + "</premiered>\n</episodedetails>")
 78 |     f.close()
 79 | 
 80 | def notify(video_meta_data):
 81 | 
 82 |     # Send a notification via apprise library.
 83 |     logger.info("Sending new %s video notification: %s", video_meta_data['channel']['channel_name'],
 84 |                 video_meta_data['title'])
 85 | 
 86 |     email_body = '<!DOCTYPE PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” '
 87 |     email_body += '“https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”>' + '\n'
 88 |     email_body += '<html xmlns="http://www.w3.org/1999/xhtml">' + '\n'
 89 |     email_body += '<head>' + '\n\t'
 90 |     email_body += '<title>' + video_meta_data['title'] + '</title>' + '\n'
 91 |     email_body += '</head>' + '\n'
 92 |     email_body += '<body>'
 93 | 
 94 |     video_url = TA_SERVER + "/video/" + video_meta_data['youtube_id']
 95 |     email_body += "\n\n<b>Video Title:</b> " + video_meta_data['title']  + "<br>" + '\n'
 96 |     email_body += "\n<b>Video Date:</b> " + video_meta_data['published'] + "<br>" + '\n'
 97 |     email_body += "\n<b>Video Views:</b> " + str(video_meta_data['stats']['view_count']) + "<br>" + '\n'
 98 |     email_body += "\n<b>Video Likes:</b> " + str(video_meta_data['stats']['like_count']) + "<br>" + '\n\n'
 99 |     email_body += "\n<b>Video Link:</b> <a href=\"" + video_url + "\">" + video_url + "</a><br>" + '\n'
100 |     email_body += "\n<b>Video Description:</b>\n\n<pre>" + video_meta_data['description'] + '</pre></br>\n\n'
101 |     email_body += '\n</body>\n</html>'
102 | 
103 |     # Dump for local debug viewing
104 |     pretty_text = html2text.HTML2Text()
105 |     pretty_text.ignore_links = True
106 |     pretty_text.body_width = 200
107 |     logger.debug(pretty_text.handle(email_body))
108 |     logger.debug(email_body)
109 | 
110 |     video_title = "[TA] New video from " + video_meta_data['channel']['channel_name']
111 | 
112 |     apobj = apprise.Apprise()
113 |     apobj.add(APPRISE_LINK)
114 |     apobj.notify(body=email_body,title=video_title)
115 | 
116 | def cleanup_after_deleted_videos():
117 |     logger.info("Check for broken symlinks and nfo files without videos in our target folder.")
118 |     broken = []
119 |     for root, dirs, files in os.walk(TARGET_FOLDER):
120 |         if root.startswith('./.git'):
121 |             # Ignore the .git directory.
122 |             continue
123 |         for filename in files:
124 |             path = os.path.join(root,filename)
125 |             file_info = os.path.splitext(path)
126 |             # Check if the file is a video's nfo file
127 |             if not filename == "tvshow.nfo" and file_info[1] == ".nfo" :
128 |                 # Check if there is a corresponding video file and if not, delete the nfo file.
129 |                 expected_video = path.replace('.nfo','.mp4')
130 |                 if not os.path.exists(expected_video):
131 |                     logger.info("Found hanging .nfo file: %s", path)
132 |                     # Queue the hanging nfo file for deletion.
133 |                     broken.append(path)
134 |             elif os.path.islink(path):
135 |                 # We've found a symlink.
136 |                 target_path = os.readlink(path)
137 |                 # Resolve relative symlinks
138 |                 if not os.path.isabs(target_path):
139 |                     target_path = os.path.join(os.path.dirname(path),target_path)     
140 |                 if not os.path.exists(target_path):
141 |                     # The symlink is broken.
142 |                     broken.append(path)
143 |             else:
144 |                 # If it's not a symlink or hanging nfo file, we're not interested.
145 |                 logger.debug("No need to clean-up  %s", path)
146 |                 continue
147 |         for dir in dirs:
148 |             logger.debug("Found channel folder: %s", dir)
149 | 
150 |     if broken == []:
151 |         logger.info("No deleted videos found, no cleanup required.")
152 |     else:
153 |         logger.info('%d Broken symlinks found...', len(broken))
154 |         for link in broken:
155 |             logger.info("Deleting file: %s", link )
156 |             # Here we need to delete the NFO file and video and subtitle symlinks
157 |             # associated with the deleted video.
158 |             os.remove(link)
159 |             # TBD Also check TA if channel target folder should be deleted?
160 | 
161 | def urlify(s):
162 |     s = re.sub(r"[^\w\s]", '', s)
163 |     s = re.sub(r"\s+", '-', s)
164 |     return s
165 | 
166 | os.makedirs(TARGET_FOLDER, exist_ok = True)
167 | url = TA_SERVER + '/api/channel/'
168 | headers = {'Authorization': 'Token ' + TA_TOKEN}
169 | req = requests.get(url, headers=headers)
170 | if req and req.status_code == 200:
171 |     channels_json = req.json() 
172 |     channels_data = channels_json['data']
173 | else :
174 |     logger.info("No Channels in TA, exiting")
175 |     # Bail from program as we have no channels in TA.
176 |     sys.exit()
177 | 
178 | while channels_json['paginate']['last_page']:
179 |     channels_json = requests.get(url, headers=headers, params={'page': channels_json['paginate']['current_page'] + 1}).json()
180 |     channels_data.extend(channels_json['data'])
181 |                      
182 | for channel in channels_data:
183 |     chan_name = urlify(channel['channel_name'])
184 |     # some channels have False as channel_description, which later on gives a type error
185 |     if not channel['channel_description']:
186 |         channel['channel_description'] = ""
187 |     description = channel['channel_description']
188 |     logger.debug("Video Description: " + description)
189 |     logger.debug("Channel Name: " + chan_name)
190 |     if(len(chan_name) < 1): chan_name = channel['channel_id']
191 |     chan_url = url+channel['channel_id']+"/video/"
192 |     try:
193 |         os.makedirs(TARGET_FOLDER + "/" + chan_name, exist_ok = False)
194 |         setup_new_channel_resources(chan_name, channel)
195 |     except OSError as error:
196 |         logger.debug("We already have %s channel folder", chan_name)
197 | 
198 |     logger.debug("Channel URL: " + chan_url)
199 |     chan_videos = requests.get(chan_url, headers=headers)
200 |     chan_videos_json = chan_videos.json() if chan_videos and chan_videos.status_code == 200 else None
201 | 
202 |     if chan_videos_json is not None:
203 |         chan_videos_data = chan_videos_json['data']
204 |         while chan_videos_json['paginate']['last_page']:
205 |             chan_videos_json = requests.get(chan_url, headers=headers, params={'page': chan_videos_json['paginate']['current_page'] + 1}).json()
206 |             chan_videos_data.extend(chan_videos_json['data'])
207 | 
208 |         for video in chan_videos_data:
209 |             video['media_url'] = video['media_url'].replace('/media','')
210 |             logger.debug(video['published'] + "_" + video['youtube_id'] + "_" + urlify(video['title']) + ", " + video['media_url'])
211 |             title=video['published'] + "_" + video['youtube_id'] + "_" + urlify(video['title']) + ".mp4"
212 |             try:
213 |                 os.symlink(TA_MEDIA_FOLDER + video['media_url'], TARGET_FOLDER + "/" + chan_name + "/" + title)
214 |                 # Getting here means a new video.
215 |                 logger.info("Processing new video from %s: %s", chan_name, title)
216 |                 if NOTIFICATIONS_ENABLED:
217 |                     notify(video)
218 |                 else:
219 |                     logger.debug("Notification not sent for %s new video: %s as NOTIFICATIONS_ENABLED is set to False in .env settings.", chan_name, title)
220 |                 if GENERATE_NFO:
221 |                     generate_new_video_nfo(chan_name, title, video)
222 |                 else:
223 |                     logger.debug("Not generating NFO files for %s new video: %s as GENERATE_NFO is et to False in .env settings.", chan_name, title)
224 |             except FileExistsError:
225 |                 # This means we already had processed the video, completely normal.
226 |                 logger.debug("Symlink exists for " + title)
227 |                 if(QUICK):
228 |                     time.sleep(.5)
229 |                     break;
230 | 
231 | # If enabled, check for deleted video and if found cleanup video NFO file and video and subtitle symlinks.
232 | if CLEANUP_DELETED_VIDEOS:
233 |     cleanup_after_deleted_videos()
234 | 


--------------------------------------------------------------------------------