├── README.md ├── cnb_tvbox_tools.py ├── requirements.txt └── tvbox_tools.py /README.md: -------------------------------------------------------------------------------- 1 | ## [tvbox_tools](https://hub.docker.com/r/2011820123/tvbox) 2 | 写这个工具的主要原因是网上各种接口重复率和失效率极高。几个多仓接口能有成百上千个线路，实际上不重复、可用的线路只有那么几十个，实在是过于冗余了。所以做了这个整理工具，把接口中所有线路进行去重和格式化，json下载保存为同名txt文件，jar文件保存到jar目录下，最后输出个all.json(包含所有历史下载线路接口)和{target}.json(本次下载线路接口，默认tvbox.json)文件用于配置app，看起来比较简洁，方便修改维护。 3 | 4 | ## 功能概述 5 | - 支持多仓、单仓、线路接口的私有化(json和对应的jar文件下载到本地，经过格式化后推送到自己的git仓库) 6 | - 支持js动态渲染数据的接口 7 | - 移除失效线路 8 | - 移除名称中的emoj表情 9 | - 根据hash和文件大小去重线路 10 | - 为文件链接自动使用加速（支持多种加速） 11 | - 默认使用 https://githubfast.com 加速clone、push(push文件过大会限制) 12 | 13 | - 新增`cnb_tvbox_tools.py`，专用于将在线接口私有化到`https://cnb.cool/`，并支持将site中的文件(外挂jar、api中的py或drpy2.min.js、ext中的json)一同私有化。效果参考： 14 | - https://cnb.cool/fish2018/duanju/-/git/raw/main/tvbox.json 15 | - https://cnb.cool/fish2018/test/-/git/raw/main/tvbox.json 16 | 17 | ## 使用方法： 18 | 19 | #### 参数选项 20 | docker run时使用-e选项通过环境变量传参 21 | 22 | - [ * ] username or u 指定用户名 23 | - [ * ] token [github.com中设置token](https://github.com/settings/tokens) 24 | - [ * ] url 指定要下载的源，多个url使用英文逗号分隔，`?&signame=`指定单线路名 25 | - repo 指定你的代码仓库名，默认tvbox 26 | - target 指定你想保存的json文件名，默认tvbox.json 27 | - num 多仓时可以指定下载前num个仓库源 28 | - timeout http请求超时，默认3s 29 | - signame url是单个线路时可以指定线路名(jar同名)，不指定随机生成 30 | - jar_suffix 指定spider字段jar包保存后缀名，默认`jar`，一些CDN禁止'jar'后缀，可以修改为`txt`、`json`、`js`、`css`、`html` 31 | - mirror 指定镜像cdn加速，默认mirror=1 32 | - gh1类型 https://raw.githubusercontent.com/fish2018/tvbox/master/all.json => https://xxxx/gh/fish2018/tvbox/all.json 33 | - mirror=1 https://ghp.ci/https://raw.githubusercontent.com 34 | - mirror=2 https://gitdl.cn/https://raw.githubusercontent.com 35 | - mirror=3 https://ghproxy.net/https://raw.githubusercontent.com 36 | - mirror=4 https://github.moeyy.xyz/https://raw.githubusercontent.com 37 | - mirror=5 https://gh-proxy.com/https://raw.githubusercontent.com 38 | - mirror=6 https://ghproxy.cc/https://raw.githubusercontent.com 39 | - mirror=7 https://raw.yzuu.cf 可加速clone、push速度非常快(限制低于50M) 40 | - mirror=8 https://raw.nuaa.cf 41 | - mirror=9 https://raw.kkgithub.com 42 | - mirror=10 https://gh.con.sh/https://raw.githubusercontent.com 43 | - mirror=11 https://gh.llkk.cc/https://raw.githubusercontent.com 44 | - mirror=12 https://gh.ddlc.top/https://raw.githubusercontent.com 45 | - mirror=13 https://gh-proxy.llyke.com/https://raw.githubusercontent.com 46 | - gh2类型(缓存不能及时更新，禁止缓存jar后缀，建议txt、json、js、css、html) https://raw.githubusercontent.com/fish2018/tvbox/master/all.json => https://xxxx/fish2018/tvbox/master/all.json 47 | - mirror=21 https://fastly.jsdelivr.net 48 | - mirror=22 https://jsd.onmicrosoft.cn 49 | - mirror=23 https://gcore.jsdelivr.net 50 | - mirror=24 https://cdn.jsdmirror.com 51 | - mirror=25 https://cdn.jsdmirror.cn 52 | - mirror=26 https://jsd.proxy.aks.moe 53 | - mirror=27 https://jsdelivr.b-cdn.net 54 | - mirror=28 https://jsdelivr.pai233.top 55 | 56 | #### Docker执行示例: 57 | Docker镜像`2011820123/tvbox`，也可以使用代理拉取镜像`dockerproxy.com/2011820123/tvbox:latest`
58 | 首先在github.com上创建自己的代码仓库，推荐命名'tvbox'，其他仓库名需要指定参数repo
59 | 支持多url下载，英文逗号`,`分隔多个url，`?&signame={name}`指定单线路名，不指定会生成随机名，{target}.json以最后一个url为准。
60 | 61 | ```bash 62 | docker run --rm -e username=xxx -e token=xxx -e url='http://肥猫.com?&signame=肥猫,http://www.饭太硬.com/tv/?&signame=饭太硬' 2011820123/tvbox 63 | ``` 64 | 65 | 演示： 66 | 67 | ``` 68 | docker run --rm -e repo=ol -e mirror=2 -e jar_suffix=css -e token=XXX -e username=fish2018 -e num=1 -e url='https://www.iyouhun.com/tv/0' 2011820123/tvbox 69 | 70 | >>> 71 | 72 | 开始克隆：git clone https://githubfast.com/fish2018/ol.git 73 | --------- 开始私有化在线接口 ---------- 74 | 当前url: https://www.iyouhun.com/tv/0 75 | 【多仓】 🌹游魂主仓库🌹.json: https://xn--s6wu47g.u.nxog.top/nxog/ou1.php?b=游魂 76 | 开始下载【线路】游魂家庭1: https://xn--s6wu47g.u.nxog.top/m/111.php?ou=公众号欧歌app&mz=index&jar=index&123&b=游魂 77 | 开始下载【线路】游魂云盘2: https://xn--s6wu47g.u.nxog.top/m/111.php?ou=公众号欧歌app&mz=all&jar=all&b=游魂 78 | 开始下载【线路】游魂学习3: https://xn--s6wu47g.u.nxog.top/m/111.php?ou=公众号欧歌app&mz=a3&jar=a3&b=游魂 79 | 开始下载【线路】下面游魂收集网络: https://xn--s6wu47g.u.nxog.top/m/111.php?ou=公众号欧歌app&mz=index&jar=index&321&b=游魂 80 | 开始下载【线路】饭太硬: http://py.nxog.top/?ou=http://www.饭太硬.com/tv/ 81 | 开始下载【线路】OK: http://py.nxog.top/?ou=http://ok321.top/ok 82 | 开始下载【线路】盒子迷: http://py.nxog.top/?ou=https://盒子迷.top/禁止贩卖 83 | 开始下载【线路】D佬: https://download.kstore.space/download/2883/nzk/nzk0722.json 84 | 开始下载【线路】PG: https://gh.con.sh/https://raw.githubusercontent.com/ouhaibo1980/tvbox/master/pg/jsm.json 85 | 开始下载【线路】肥猫: http://py.nxog.top/?ou=http://肥猫.com 86 | 开始下载【线路】小米: http://py.nxog.top/?ou=http://www.mpanso.com/%E5%B0%8F%E7%B1%B3/DEMO.json 87 | 开始下载【线路】放牛: http://py.nxog.top/?ou=http://tvbox.xn--4kq62z5rby2qupq9ub.top 88 | 开始下载【线路】小马: https://szyyds.cn/tv/x.json 89 | 开始下载【线路】天天开心: http://ttkx.live:55/天天开心 90 | 开始下载【线路】摸鱼: http://我不是.摸鱼儿.top 91 | 开始下载【线路】老刘备: https://raw.liucn.cc/box/m.json 92 | 开始下载【线路】香雅情: https://gh.con.sh/https://raw.githubusercontent.com/xyq254245/xyqonlinerule/main/XYQTVBox.json 93 | 开始下载【线路】俊佬: http://home.jundie.top:81/top98.json 94 | 开始下载【线路】月光: https://gh.con.sh/https://raw.githubusercontent.com/guot55/yg/main/max.json 95 | 开始下载【线路】巧技: http://cdn.qiaoji8.com/tvbox.json 96 | 开始下载【线路】荷城茶秀: https://gh.con.sh/https://raw.githubusercontent.com/HeChengChaXiu/tvbox/main/hccx.json 97 | 开始下载【线路】云星日记: http://itvbox.cc/云星日记 98 | 开始下载【线路】吾爱: http://52pan.top:81/api/v3/file/get/174964/%E5%90%BE%E7%88%B1%E8%AF%84%E6%B5%8B.m3u?sign=rPssLoffquDXszCARt6UNF8MobSa1FA27XomzOluJBY%3D%3A0 99 | 开始下载【线路】南风: https://gh.con.sh/https://raw.githubusercontent.com/yoursmile66/TVBox/main/XC.json 100 | 开始下载【线路】2游魂收集不分排名: https://xn--s6wu47g.u.nxog.top/m/333.php?ou=公众号欧歌app&mz=all&jar=all&b=游魂 101 | 开始写入单仓🌹游魂主仓库🌹.json 102 | 开始写入tvbox.json 103 | 开始写入all.json 104 | --------- 完成私有化在线接口 ---------- 105 | 开始推送：git push https://githubfast.com/fish2018/ol.git 106 | 耗时: 176.29488706588745 秒 107 | 108 | #################影视仓APP配置接口######################## 109 | 110 | https://gitdl.cn/https://raw.githubusercontent.com/fish2018/ol/main/all.json 111 | https://gitdl.cn/https://raw.githubusercontent.com/fish2018/ol/main/tvbox.json 112 | 113 | ``` 114 | 115 | 116 | ## 更新说明 117 | - V2.5版本 新增三个gh1代理；设置jar_suffix后会自动把历史的jar后缀批量成新的后缀；兼容证书失效的接口 118 | - V2.4版本 mirror=1 https://mirror.ghproxy.com变更为https://ghp.ci；增加mirror=10 119 | - V2.3版本 更新大量cdn支持；默认使用githubfast.com加速clone和push，失败切换hub.yzuu.cf 120 | - V2.2版本 支持通过jar_suffix参数修改jar包后缀 121 | - V2.1版本 支持多种镜像加速，通过mirror={num}指定；当mirror<4时自动设置/etc/hosts加速github.com，解决运行docker的本地网络不能访问github 122 | - V2.0版本 修复指定target生成指定`{target}`.json；支持多url下载，英文逗号分隔多个url，`?&signame={name}`指定单线路名，不指定会生成随机名。例子：url = 'http://肥猫.com?&signame=肥猫,http://www.饭太硬.com/tv/?&signame=饭太硬' 123 | - V1.9版本 移除多线程下载接口；已下载接口不重复下载；支持js动态渲染数据的接口；增加根据文件大小去重线路；单线路下载不指定signame(单线路名)时会生成一个"{随机字符串}.txt"；兼容主分支main/master 124 | - V1.8版本 移除agit.ai支持；all.json线路排序； 125 | - V1.7版本 优化git clone速度，仓库重置提交记录计数(始终commit 1，使仓库存储占用小，下载速度快) 126 | - V1.6版本 不规范json兼容优化，http请求timeout默认3s，优化移除emoji表情 127 | - V1.5版本 bug修复，github.com支持优化 128 | - V1.4版本 bug修复，jar下载失败，不会创建0字节jar文件，保留原jar链接 129 | - V1.3版本 支持github.com 130 | - V1.2版本 支持jar本地化 131 | - V1.1版本 bug修复，仅支持agit.ai，不支持jar本地化 132 | - V1.0版本 支持单线路、单仓、多仓下载，输出：{target}(默认tvbox.json)，和url填写的源内容一致；all.json是仓库中所有下载的历史线路总和，并且去重了内容相同的线路 133 | 134 | 135 | -------------------------------------------------------------------------------- /cnb_tvbox_tools.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # !/usr/bin/env python3 3 | from requests_html import HTMLSession 4 | import pprint 5 | import random 6 | import string 7 | import time 8 | import hashlib 9 | import json 10 | import git # gitpython 11 | import re 12 | import base64 13 | import requests 14 | import asyncio 15 | import aiohttp 16 | from requests.adapters import HTTPAdapter, Retry 17 | import os 18 | import subprocess 19 | import ssl 20 | import shutil 21 | from pathlib import Path 22 | from urllib.parse import urlparse, parse_qs, urljoin 23 | import commentjson 24 | ssl._create_default_https_context = ssl._create_unverified_context 25 | import urllib3 26 | from urllib3.exceptions import InsecureRequestWarning 27 | urllib3.disable_warnings(InsecureRequestWarning) 28 | 29 | global pipes 30 | pipes = set() 31 | 32 | class GetSrc: 33 | def __init__(self, username=None, token=None, url=None, repo=None, num=10, target=None, timeout=3, signame=None, mirror=None, jar_suffix=None, site_down=True): 34 | self.jar_suffix = jar_suffix if jar_suffix else 'jar' 35 | self.site_down = site_down # 是否下载site里的文件到本地 36 | self.mirror = int(str(mirror).strip()) if mirror else 1 37 | self.mirror_proxy = 'https://ghp.ci/https://raw.githubusercontent.com' 38 | self.num = int(num) 39 | self.sep = os.path.sep 40 | self.username = username 41 | self.token = token 42 | self.timeout=timeout 43 | self.url = url.replace(' ','').replace('，',',') if url else url 44 | self.repo = repo if repo else 'tvbox' 45 | self.target = f'{target.split(".json")[0]}.json' if target else 'tvbox.json' 46 | self.headers = {"user-agent": "okhttp/3.15 Html5Plus/1.0 (Immersed/23.92157)"} 47 | self.s = requests.Session() 48 | self.signame = signame 49 | retries = Retry(total=3, backoff_factor=1) 50 | self.s.mount('http://', HTTPAdapter(max_retries=retries)) 51 | self.s.mount('https://', HTTPAdapter(max_retries=retries)) 52 | self.size_tolerance = 15 # 线路文件大小误差在15以内认为是同一个 53 | self.main_branch = 'main' 54 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 55 | self.cnb_slot = f'https://cnb.cool/{self.username}/{self.repo}/-/git/raw/{self.main_branch}' 56 | 57 | self.registry = 'cnb.cool' 58 | self.gitusername = 'cnb' 59 | self.repo = repo 60 | self.token = token 61 | self.branch = 'main' 62 | 63 | self.gh1 = [ 64 | 'https://ghp.ci/https://raw.githubusercontent.com', 65 | 'https://gh.xxooo.cf/https://raw.githubusercontent.com', 66 | 'https://ghproxy.net/https://raw.githubusercontent.com', 67 | 'https://github.moeyy.xyz/https://raw.githubusercontent.com', 68 | 'https://gh-proxy.com/https://raw.githubusercontent.com', 69 | 'https://ghproxy.cc/https://raw.githubusercontent.com', 70 | 'https://raw.yzuu.cf', 71 | 'https://raw.nuaa.cf', 72 | 'https://raw.kkgithub.com', 73 | 'https://mirror.ghproxy.com/https://raw.githubusercontent.com', 74 | 'https://gh.llkk.cc/https://raw.githubusercontent.com', 75 | 'https://gh.ddlc.top/https://raw.githubusercontent.com', 76 | 'https://gh-proxy.llyke.com/https://raw.githubusercontent.com', 77 | 'https://slink.ltd', 78 | 'https://cors.zme.ink', 79 | 'https://git.886.be' 80 | ] 81 | self.gh2 = [ 82 | "https://fastly.jsdelivr.net/gh", 83 | "https://jsd.onmicrosoft.cn/gh", 84 | "https://gcore.jsdelivr.net/gh", 85 | "https://cdn.jsdmirror.com/gh", 86 | "https://cdn.jsdmirror.cn/gh", 87 | "https://jsd.proxy.aks.moe/gh", 88 | "https://jsdelivr.b-cdn.net/gh", 89 | "https://jsdelivr.pai233.top/gh" 90 | ] 91 | 92 | # 定义 drpy2 文件列表 93 | self.drpy2 = False 94 | self.drpy2_files = [ 95 | "cat.js", "crypto-js.js", "drpy2.min.js", "http.js", "jquery.min.js", 96 | "jsencrypt.js", "log.js", "pako.min.js", "similarity.js", "uri.min.js", 97 | "cheerio.min.js", "deep.parse.js", "gbk.js", "jinja.js", "json5.js", 98 | "node-rsa.js", "script.js", "spider.js", "模板.js", "quark.min.js" 99 | ] 100 | 101 | async def download_drpy2_files(self): 102 | """ 103 | 异步下载 drpy2 文件到 self.repo/api/drpy2 104 | """ 105 | # 创建 drpy2 目录 106 | api_drpy2_dir = os.path.join(self.repo, "api/drpy2") 107 | if not os.path.exists(api_drpy2_dir): 108 | os.makedirs(api_drpy2_dir) 109 | async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(ssl=False), 110 | timeout=aiohttp.ClientTimeout(total=60, connect=15) 111 | ) as session: 112 | tasks = [] 113 | for filename in self.drpy2_files: 114 | local_path = os.path.join(api_drpy2_dir, filename) 115 | if os.path.exists(local_path): 116 | # print(f"文件已存在，跳过: {local_path}") 117 | continue 118 | json_url = f"https://github.moeyy.xyz/https://raw.githubusercontent.com/fish2018/lib/main/js/dr_py/{filename}" 119 | 120 | async def download_task(json_url=json_url, local_path=local_path, filename=filename): 121 | retries = 3 122 | for attempt in range(retries): 123 | try: 124 | async with session.get(json_url) as response: 125 | response.raise_for_status() 126 | content = await response.read() 127 | with open(local_path, "wb") as f: 128 | f.write(content) 129 | # print(f"下载成功: {filename}") 130 | return True 131 | except Exception as e: 132 | # print(f"下载 {json_url} 失败 (尝试 {attempt + 1}/{retries}): {e}") 133 | if attempt < retries - 1: 134 | await asyncio.sleep(1) 135 | else: 136 | print(f"下载 {json_url} 最终失败") 137 | return False 138 | 139 | tasks.append(download_task()) 140 | 141 | if tasks: 142 | # print(f"开始下载 {len(tasks)} 个 drpy2 文件") 143 | await asyncio.gather(*tasks, return_exceptions=True) 144 | else: 145 | pass 146 | # print("所有 drpy2 文件已存在，无需下载") 147 | def file_hash(self, filepath): 148 | with open(filepath, 'rb') as f: 149 | file_contents = f.read() 150 | return hashlib.sha256(file_contents).hexdigest() 151 | def remove_duplicates(self, folder_path): 152 | folder_path = Path(folder_path) 153 | jar_folder = f'{folder_path}/jar' 154 | excludes = {'.json', '.git', 'jar', '.idea', 'ext', '.DS_Store', '.md'} 155 | files_info = {} 156 | 157 | # 把jar目录下所有文件后缀都改成新的self.jar_suffix 158 | self.rename_jar_suffix(jar_folder) 159 | 160 | # 存储文件名、大小和哈希值 161 | for file_path in folder_path.iterdir(): 162 | if file_path.is_file() and file_path.suffix not in excludes: 163 | file_size = file_path.stat().st_size 164 | file_hash = self.file_hash(file_path) 165 | files_info[file_path.name] = {'path': str(file_path), 'size': file_size, 'hash': file_hash} 166 | 167 | # 保留的文件列表 168 | keep_files = [] 169 | # 按文件大小排序，然后按顺序处理 170 | for file_name, info in sorted(files_info.items(), key=lambda item: item[1]['size']): 171 | if not keep_files or abs(info['size'] - files_info[keep_files[-1]]['size']) > self.size_tolerance: 172 | keep_files.append(file_name) 173 | # 删除jar目录下除了{self.jar_suffix}的文件 174 | # self.remove_all_except_jar(jar_folder) 175 | else: 176 | # 如果当前文件大小在容忍范围内，删除当前文件和对应的jar文件 177 | os.remove(info['path']) 178 | self.remove_jar_file(jar_folder, file_name.replace('.txt', f'{self.jar_suffix}')) 179 | 180 | keep_files.sort() 181 | return keep_files 182 | def rename_jar_suffix(self,jar_folder): 183 | # 遍历目录中的所有文件和子目录 184 | for root, dirs, files in os.walk(jar_folder): 185 | for file in files: 186 | # 构造完整的文件路径 187 | old_file = os.path.join(root, file) 188 | # 构造新的文件名，去除原有的后缀，加上 self.jar_suffix 189 | new_file = os.path.join(root, os.path.splitext(file)[0] + f'.{self.jar_suffix}') 190 | # 重命名文件 191 | os.rename(old_file, new_file) 192 | # print(f"文件已重命名: {old_file} -> {new_file}") 193 | def remove_all_except_jar(self, jar_folder): 194 | # 列出文件夹中的所有文件 195 | for file_name in os.listdir(jar_folder): 196 | # 构建完整的文件路径 197 | full_path = os.path.join(jar_folder, file_name) 198 | # 检查是否为文件 199 | if os.path.isfile(full_path): 200 | # 获取文件的扩展名 201 | _, file_extension = os.path.splitext(file_name) 202 | # 如果扩展名不是self.jar_suffix，则删除文件 203 | if file_extension != f'.{self.jar_suffix}': 204 | self.remove_jar_file(jar_folder, file_name) 205 | def remove_jar_file(self, jar_folder, file_name): 206 | # 构建jar文件的路径 207 | jar_file_path = os.path.join(jar_folder, file_name) 208 | # 如果jar文件存在，则删除它 209 | if os.path.isfile(jar_file_path): 210 | os.remove(jar_file_path) 211 | def remove_emojis(self, text): 212 | emoji_pattern = re.compile("[" 213 | u"\U0001F600-\U0001F64F" # emoticons 214 | u"\U0001F300-\U0001F5FF" # symbols & pictographs 215 | u"\U0001F680-\U0001F6FF" # transport & map symbols 216 | u"\U0001F1E0-\U0001F1FF" # flags (iOS) 217 | "\U00002500-\U00002BEF" # chinese char 218 | "\U00010000-\U0010ffff" 219 | "\u200d" # zero width joiner 220 | "\u20E3" # combining enclosing keycap 221 | "\ufe0f" # VARIATION SELECTOR-16 222 | "]+", flags=re.UNICODE) 223 | text = text.replace('/', '_').replace('多多', '').replace('┃', '').replace('线路', '').replace('匚','').strip() 224 | return emoji_pattern.sub('', text) 225 | def json_compatible(self, str): 226 | # 兼容错误json 227 | # res = str.replace(' ', '').replace("'",'"').replace('//"', '"').replace('//{', '{').replace('key:', '"key":').replace('name:', '"name":').replace('type:', '"type":').replace('api:','"api":').replace('searchable:', '"searchable":').replace('quickSearch:', '"quickSearch":').replace('filterable:','"filterable":').strip() 228 | res = str.replace('//"', '"').replace('//{', '{').replace('key:', '"key":').replace('name:', '"name":').replace('type:', '"type":').replace('api:','"api":').replace('searchable:', '"searchable":').replace('quickSearch:', '"quickSearch":').replace('filterable:','"filterable":').strip() 229 | return res 230 | def ghproxy(self, str): 231 | u = 'https://github.moeyy.xyz/' 232 | res = str.replace('https://ghproxy.net/', u).replace('https://ghproxy.com/', u).replace('https://gh-proxy.com/',u).replace('https://mirror.ghproxy.com/',u).replace('https://gh.xxooo.cf/',u).replace('https://ghp.ci/',u).replace('https://gitdl.cn/',u) 233 | return res 234 | def set_hosts(self): 235 | # 设置github.com的加速hosts 236 | try: 237 | response = requests.get('https://hosts.gitcdn.top/hosts.json') 238 | if response.status_code == 200: 239 | hosts_data = response.json() 240 | # 遍历JSON数据，找到"github.com"对应的IP 241 | github_ip = None 242 | for entry in hosts_data: 243 | if entry[1] == "github.com": 244 | github_ip = entry[0] 245 | break 246 | if github_ip: 247 | # 读取现有的/etc/hosts文件 248 | with open('/etc/hosts', 'r+') as file: 249 | hosts_content = file.read() 250 | # 检查是否已经存在对应的IP 251 | if github_ip not in hosts_content: 252 | # 将新的IP添加到文件末尾 253 | file.write(f'\n{github_ip} github.com') 254 | print(f'IP address {github_ip} for github.com has been added to /etc/hosts.') 255 | else: 256 | print(f'IP address for github.com is already in /etc/hosts.') 257 | else: 258 | print('No IP found for github.com in the provided data.') 259 | else: 260 | print('Failed to retrieve data from https://hosts.gitcdn.top/hosts.json') 261 | except Exception as e: 262 | pass 263 | def picparse(self, url): 264 | r = self.s.get(url, headers=self.headers, timeout=self.timeout, verify=False) 265 | pattern = r'([A-Za-z0-9+/]+={0,2})' 266 | matches = re.findall(pattern, r.text) 267 | decoded_data = base64.b64decode(matches[-1]) 268 | text = decoded_data.decode('utf-8') 269 | return text 270 | 271 | async def js_render(self, url): 272 | # 获取 JS 渲染页面源代码 273 | timeout = self.timeout * 4 274 | if timeout > 15: 275 | timeout = 15 276 | browser_args = [ 277 | '--no-sandbox', 278 | '--disable-dev-shm-usage', 279 | '--disable-gpu', 280 | '--disable-software-rasterizer', 281 | '--disable-setuid-sandbox', 282 | ] 283 | from requests_html import AsyncHTMLSession 284 | 285 | session = AsyncHTMLSession(browser_args=browser_args) 286 | try: 287 | r = await session.get( 288 | f'http://lige.unaux.com/?url={url}', 289 | headers=self.headers, 290 | timeout=timeout, 291 | verify=False, 292 | ) 293 | # 等待页面加载完成并渲染 JavaScript 294 | await r.html.arender(timeout=timeout) 295 | return r.html 296 | finally: 297 | await session.close() 298 | async def site_file_down(self, files, url): 299 | """ 300 | 异步函数，用于同时下载和更新 ext、jar 和 api 文件。 301 | 302 | 参数: 303 | files: 包含一个或两个文件路径的列表，例如 ['api.json', 'output.json'] 304 | url: 基础 URL，用于构造完整的下载 URL 305 | """ 306 | # 设置 ext、jar 和 api 的保存目录 307 | ext_dir = f"{self.repo}/ext" 308 | jar_dir = f"{self.repo}/jar" 309 | api_dir = f"{self.repo}/api" 310 | api_drpy2_dir = f"{self.repo}/api/drpy2" 311 | for directory in [ext_dir, jar_dir, api_dir, api_drpy2_dir]: 312 | if not os.path.exists(directory): 313 | os.makedirs(directory) 314 | 315 | # 获取文件路径并读取 api.json 316 | file = files[0] 317 | file2 = files[1] if len(files) > 1 else '' 318 | 319 | with open(file, 'r', encoding='utf-8') as f: 320 | try: 321 | api_data = commentjson.load(f) 322 | sites = api_data["sites"] 323 | print(f"总站点数: {len(sites)}") 324 | except Exception as e: 325 | # print(f"解析 {file} 失败: {e}") 326 | return 327 | 328 | # 使用 aiohttp 创建会话并收集下载任务 329 | async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(ssl=False), 330 | timeout=aiohttp.ClientTimeout(total=60, connect=15) 331 | ) as session: 332 | tasks = [] 333 | for site in sites: 334 | for field in ["ext", "jar", "api"]: 335 | repo_dir_name = field 336 | if field in site: 337 | value = site[field] 338 | if isinstance(value, str): 339 | clean_value = value.split(';')[0].rstrip('?') 340 | if field == "ext": 341 | if not clean_value.endswith((".js", ".txt", ".json")): 342 | continue 343 | elif field == "api": 344 | if os.path.basename(clean_value).lower() in ["drpy2.min.js","quark.min.js"]: 345 | self.drpy2 = True 346 | # 替换 api 字段 347 | site[field] = f"{self.cnb_slot}/{api_drpy2_dir}/{os.path.basename(clean_value).lower()}" 348 | continue 349 | if not clean_value.endswith(".py"): 350 | continue 351 | 352 | # 默认下载逻辑（ext、jar 和 api 的 .py 文件） 353 | filename = os.path.basename(clean_value) 354 | if './' in value: 355 | path = os.path.dirname(url) 356 | json_url = value.replace('./', f'{path}/') 357 | else: 358 | json_url = urljoin(url, value) 359 | local_path = os.path.join(f"{self.repo}/{repo_dir_name}", filename) 360 | 361 | async def download_task(site=site, json_url=json_url, local_path=local_path, 362 | filename=filename, field=field, repo_dir_name=repo_dir_name): 363 | retries = 3 364 | for attempt in range(retries): 365 | try: 366 | async with session.get(json_url) as response: 367 | response.raise_for_status() 368 | if os.path.exists(local_path): 369 | site[field] = f'{self.cnb_slot}/{repo_dir_name}/{filename}' 370 | return True 371 | content = await response.read() 372 | with open(local_path, "wb") as f: 373 | f.write(content) 374 | site[field] = f'{self.cnb_slot}/{repo_dir_name}/{filename}' 375 | return True 376 | except Exception as e: 377 | # print(f"下载 {json_url} 失败 (尝试 {attempt + 1}/{retries}): {e}") 378 | if attempt < retries - 1: 379 | await asyncio.sleep(1) 380 | else: 381 | # print(f"下载 {json_url} 最终失败") 382 | return False 383 | 384 | tasks.append(download_task()) 385 | 386 | if tasks: 387 | # print(f"总下载任务数: {len(tasks)}") 388 | await asyncio.gather(*tasks, return_exceptions=True) 389 | else: 390 | pass 391 | # print("没有找到符合条件的 ext、jar 或 api 文件需要下载") 392 | 393 | # 将更新后的数据写回文件 394 | with open(file, 'w', encoding='utf-8') as f: 395 | json.dump(api_data, f, indent=4, ensure_ascii=False) 396 | 397 | if file2 and os.path.basename(file2): 398 | with open(file2, 'w', encoding='utf-8') as f: 399 | json.dump(api_data, f, indent=4, ensure_ascii=False) 400 | def get_jar(self, name, url, text): 401 | if not os.path.exists(f'{self.repo}/jar'): 402 | os.makedirs(f'{self.repo}/jar') 403 | name = f'{name}.{self.jar_suffix}' 404 | pattern = r'\"spider\":(\s)?\"([^,]+)\"' 405 | matches = re.search(pattern, text) 406 | try: 407 | jar = matches.group(2).replace('./', f'{url}/').split(';')[0] 408 | jar = jar.split('"spider":"')[-1] 409 | if name==f'{self.repo}.{self.jar_suffix}': 410 | name = f"{jar.split('/')[-1]}" 411 | print('jar地址: ', jar) 412 | timeout = self.timeout * 4 413 | if timeout > 15: 414 | timeout = 15 415 | r = self.s.get(jar, timeout=timeout, verify=False) 416 | if r.status_code != 200: 417 | raise f'【jar下载失败】{name} jar地址: {jar} status_code:{r.status_code}' 418 | with open(f'{self.repo}/jar/{name}', 'wb') as f: 419 | f.write(r.content) 420 | jar = f'{self.cnb_slot}/jar/{name}' 421 | text = text.replace(matches.group(2), jar) 422 | except Exception as e: 423 | print(f'【jar下载失败】{name} jar地址: {jar} error:{e}') 424 | return text 425 | async def download(self, url, name, filename, cang=True): 426 | file_list = [] 427 | for root, dirs, files in os.walk(self.repo): 428 | for file in files: 429 | file_list.append(file) 430 | if filename in file_list: 431 | print(f'{filename}：已经存在，无需重复下载') 432 | return 433 | if 'agit.ai' in url: 434 | print(f'下载异常：agit.ai失效') 435 | return 436 | # 下载单线路 437 | item = {} 438 | try: 439 | path = os.path.dirname(url) 440 | r = self.s.get(url, headers=self.headers, allow_redirects=True, timeout=self.timeout, verify=False) 441 | if r.status_code == 200: 442 | print("开始下载【线路】{}: {}".format(name, url)) 443 | if 'searchable' not in r.text: 444 | r = self.js_render(url) 445 | if not r.text: 446 | r = self.picparse(url) 447 | if 'searchable' not in r: 448 | raise 449 | r = self.get_jar(name, url, r) 450 | with open(f'{self.repo}{self.sep}{filename}', 'w+', encoding='utf-8') as f: 451 | f.write(r) 452 | return 453 | if 'searchable' not in r.text: 454 | raise 455 | with open(f'{self.repo}{self.sep}{filename}', 'w+', encoding='utf-8') as f: 456 | try: 457 | if r.content.decode('utf8').startswith(u'\ufeff'): 458 | str = r.content.decode('utf8').encode('utf-8')[3:].decode('utf-8') 459 | else: 460 | str = r.content.decode('utf-8').replace('./', f'{path}/') 461 | except: 462 | str = r.text 463 | finally: 464 | r = self.ghproxy(str.replace('./', f'{path}/')) 465 | 466 | r = self.get_jar(name, url, r) 467 | f.write(r) 468 | pipes.add(name) 469 | try: 470 | if self.site_down: 471 | await self.site_file_down([f'{self.repo}{self.sep}{filename}'], url) 472 | except Exception as e: 473 | print(f'下载ext中的json失败: {e}') 474 | except Exception as e: 475 | print(f"【线路】{name}: {url} 下载错误：{e}") 476 | # 单仓时写入item 477 | if os.path.exists(f'{self.repo}{self.sep}{filename}') and cang: 478 | item['name'] = name 479 | item['url'] = f'{self.cnb_slot}/{filename}' 480 | items.append(item) 481 | async def down(self, data, s_name): 482 | ''' 483 | 下载单仓 484 | ''' 485 | newJson = {} 486 | global items 487 | items = [] 488 | urls = data.get("urls") if data.get("urls") else data.get("sites") 489 | for u in urls: 490 | name = u.get("name").strip() 491 | name = self.remove_emojis(name) 492 | url = u.get("url") 493 | url = self.ghproxy(url) 494 | filename = '{}.txt'.format(name) 495 | if name in pipes: 496 | print(f"【线路】{name} 已存在，无需重复下载") 497 | continue 498 | await self.download(url, name, filename) 499 | newJson['urls'] = items 500 | newJson = pprint.pformat(newJson, width=200) 501 | print(f'开始写入单仓{s_name}') 502 | with open(f'{self.repo}{self.sep}{s_name}', 'w+', encoding='utf-8') as f: 503 | content = str(newJson).replace("'", '"') 504 | f.write(json.loads(json.dumps(content, indent=4, ensure_ascii=False))) 505 | def all(self): 506 | # 整合所有文件到all.json 507 | newJson = {} 508 | items = [] 509 | files = self.remove_duplicates(self.repo) 510 | for file in files: 511 | item = {} 512 | item['name'] = file.split('.txt')[0] 513 | item['url'] = f'{self.cnb_slot}/{file}' 514 | items.append(item) 515 | newJson['urls'] = items 516 | newJson = pprint.pformat(newJson, width=200) 517 | print(f'开始写入all.json') 518 | with open(f'{self.repo}{self.sep}all.json', 'w+', encoding='utf-8') as f: 519 | content = str(newJson).replace("'", '"') 520 | f.write(json.loads(json.dumps(content, indent=4, ensure_ascii=False))) 521 | async def batch_handle_online_interface(self): 522 | # 下载线路，处理多url场景 523 | print(f'--------- 开始私有化在线接口 ----------') 524 | urls = self.url.split(',') 525 | for url in urls: 526 | # 解析URL 527 | parsed_url = urlparse(url) 528 | # 获取查询参数 529 | query_params = parse_qs(parsed_url.query) 530 | # 提取'signame'参数的值 531 | signame_value = query_params.get('signame', [''])[0] 532 | item = url.split('?&signame=') 533 | self.url = item[0] 534 | self.signame = signame_value if signame_value else None 535 | print(f'当前url: {self.url}') 536 | await self.storeHouse() 537 | await self.clean_directories() 538 | 539 | async def clean_directories(self): 540 | # Step 1: 删除 api/drpy2 目录（如果 self.drpy2 为假） 541 | if not self.drpy2: 542 | drpy2_path = f"{self.repo}/api/drpy2" 543 | if os.path.exists(drpy2_path): 544 | await asyncio.to_thread(shutil.rmtree, drpy2_path) 545 | 546 | # Step 2: 检查并删除空的 api 和 ext 目录 547 | directories = [f"{self.repo}/api", f"{self.repo}/ext"] 548 | for dir_path in directories: 549 | if os.path.exists(dir_path) and os.path.isdir(dir_path): 550 | if not os.listdir(dir_path): # 目录为空 551 | await asyncio.to_thread(shutil.rmtree, dir_path) 552 | def git_clone(self): 553 | self.domain = f'https://{self.gitusername}:{self.token}@{self.registry}/{self.username}/{self.repo}.git' 554 | if os.path.exists(self.repo): 555 | subprocess.call(['rm', '-rf', self.repo]) 556 | try: 557 | print(f'开始克隆：git clone https://{self.registry}/{self.username}/{self.repo}.git') 558 | git.Repo.clone_from(self.domain, to_path=self.repo, depth=1) 559 | except Exception as e: 560 | print(222222, e) 561 | def get_local_repo(self): 562 | # 打开本地仓库，读取仓库信息 563 | repo = git.Repo(self.repo) 564 | config_writer = repo.config_writer() 565 | config_writer.set_value('user', 'name', self.username) 566 | config_writer.set_value('user', 'email', self.username) 567 | # 设置 http.postBuffer 568 | config_writer.set_value('http', 'postBuffer', '73400320') 569 | config_writer.release() 570 | # 获取远程仓库的引用 571 | remote = repo.remote(name='origin') 572 | # 获取远程分支列表 573 | remote_branches = remote.refs 574 | # 遍历远程分支，查找主分支 575 | for branch in remote_branches: 576 | if branch.name == 'origin/master' or branch.name == 'origin/main': 577 | self.branch = branch.name.split('/')[-1] 578 | break 579 | # print(f"仓库{self.repo} 主分支为: {self.main_branch}") 580 | return repo 581 | def reset_commit(self,repo): 582 | # 重置commit 583 | try: 584 | os.chdir(self.repo) 585 | # print('开始清理git',os.getcwd()) 586 | repo.git.checkout('--orphan', 'tmp_branch') 587 | repo.git.add(A=True) 588 | repo.git.commit(m="update") 589 | repo.git.execute(['git', 'branch', '-D', self.branch]) 590 | repo.git.execute(['git', 'branch', '-m', self.branch]) 591 | repo.git.execute(['git', 'push', '-f', 'origin', self.branch]) 592 | except Exception as e: 593 | print('git清理异常', e) 594 | def git_push(self, repo): 595 | # 推送并重置commit计数 596 | # 推送 597 | print(f'开始推送：git push https://{self.registry}/{self.username}/{self.repo}.git') 598 | try: 599 | repo.git.add(A=True) 600 | repo.git.commit(m="update") 601 | repo.git.push() 602 | self.reset_commit(repo) 603 | except Exception as e: 604 | try: 605 | repo.git.execute(['git', 'push', '--set-upstream', 'origin', self.branch]) 606 | self.reset_commit(repo) 607 | except Exception as e: 608 | print('git推送异常', e) 609 | async def storeHouse(self): 610 | ''' 611 | 生成多仓json文件 612 | ''' 613 | await self.download_drpy2_files() 614 | 615 | newJson = {} 616 | items = [] 617 | 618 | # 解析最初链接 619 | try: 620 | res = self.s.get(self.url, headers=self.headers, verify=False, timeout=self.timeout).content.decode('utf8') 621 | if '404 Not Found' in res: 622 | print(f'{self.url} 请求异常') 623 | return 624 | except Exception as e: 625 | if 'Read timed out' in str(e) or 'nodename nor servname provided, or not known' in str(e): 626 | print(f'{self.url} 请求异常：{e}') 627 | return 628 | html = await self.js_render(self.url) 629 | res = html.text.replace(' ', '').replace("'", '"') 630 | if 'Read timed out' in str(e) or 'nodename nor servname provided, or not known' in str(e): 631 | print(f'{self.url} 请求异常：{e}') 632 | return 633 | if not res: 634 | res = self.picparse(self.url).replace(' ', '').replace("'", '"') 635 | 636 | # 线路 637 | if 'searchable' in str(res): 638 | filename = self.signame + '.txt' if self.signame else f"{''.join(random.choices(string.ascii_letters + string.digits, k=10))}.txt" 639 | path = os.path.dirname(self.url) 640 | print("【线路】 {}: {}".format(self.repo, self.url)) 641 | try: 642 | with open(f'{self.repo}{self.sep}{filename}', 'w+', encoding='utf-8') as f, open( 643 | f'{self.repo}{self.sep}{self.target}', 'w+', encoding='utf-8') as f2: 644 | r = self.ghproxy(res.replace('./', f'{path}/')) 645 | r = self.get_jar(filename.split('.txt')[0], url, r) 646 | # json容错处理 647 | r = self.json_compatible(r) 648 | f.write(r) 649 | f2.write(r) 650 | except Exception as e: 651 | print(333333333, e) 652 | try: 653 | if self.site_down: 654 | await self.site_file_down([f'{self.repo}{self.sep}{filename}',f'{self.repo}{self.sep}{self.target}'], self.url) 655 | except Exception as e: 656 | pass 657 | return 658 | 659 | # json容错处理 660 | res = self.json_compatible(res) 661 | # 移除注释 662 | datas = '' 663 | for d in res.splitlines(): 664 | if d.find(" //") != -1 or d.find("// ") != -1 or d.find(",//") != -1 or d.startswith("//"): 665 | d = d.split(" //", maxsplit=1)[0] 666 | d = d.split("// ", maxsplit=1)[0] 667 | d = d.split(",//", maxsplit=1)[0] 668 | d = d.split("//", maxsplit=1)[0] 669 | datas = '\n'.join([datas, d]) 670 | # 容错处理，便于json解析 671 | datas = datas.replace('\n', '') 672 | res = datas.replace(' ', '').replace("'", '"').replace('\n', '') 673 | if datas.startswith(u'\ufeff'): 674 | try: 675 | res = datas.encode('utf-8')[3:].decode('utf-8').replace(' ', '').replace("'", '"').replace('\n', '') 676 | except Exception as e: 677 | res = datas.encode('utf-8')[4:].decode('utf-8').replace(' ', '').replace("'", '"').replace('\n', '') 678 | 679 | # 多仓 680 | elif 'storeHouse' in datas: 681 | res = json.loads(str(res)) 682 | srcs = res.get("storeHouse") if res.get("storeHouse") else None 683 | if srcs: 684 | i = 1 685 | for s in srcs: 686 | if i > self.num: 687 | break 688 | i += 1 689 | item = {} 690 | s_name = s.get("sourceName") 691 | s_name = self.remove_emojis(s_name) 692 | s_name = f'{s_name}.json' 693 | s_url = s.get("sourceUrl") 694 | print("【多仓】 {}: {}".format(s_name, s_url)) 695 | try: 696 | if self.s.get(s_url, headers=self.headers).status_code >= 400: 697 | continue 698 | except Exception as e: 699 | print('地址无法响应: ',e) 700 | continue 701 | try: 702 | if self.s.get(s_url, headers=self.headers).content.decode('utf8').lstrip().startswith(u'\ufeff'): 703 | data = self.s.get(s_url, headers=self.headers).content.decode('utf-8')[1:] 704 | else: 705 | data = self.s.get(s_url, headers=self.headers).content.decode('utf-8') 706 | except Exception as e: 707 | try: 708 | data = self.s.get(s_url, headers=self.headers).content.decode('utf8') 709 | data = data.encode('utf-8').decode('utf-8') 710 | except Exception as e: 711 | continue 712 | datas = '' 713 | for d in data.splitlines(): 714 | if d.find(" //") != -1 or d.find("// ") != -1 or d.find(",//") != -1 or d.startswith("//"): 715 | d = d.split(" //", maxsplit=1)[0] 716 | d = d.split("// ", maxsplit=1)[0] 717 | d = d.split(",//", maxsplit=1)[0] 718 | d = d.split("//", maxsplit=1)[0] 719 | datas = '\n'.join([datas, d]) 720 | 721 | try: 722 | if datas.lstrip().startswith(u'\ufeff'): 723 | datas = datas.encode('utf-8')[1:] 724 | await self.down(json.loads(datas), s_name) 725 | except Exception as e: 726 | try: 727 | data = self.s.get(s_url, headers=self.headers).text 728 | except Exception as e: 729 | continue 730 | datas = '' 731 | for d in data.splitlines(): 732 | datas += d.replace('\n', '').replace(' ', '').strip() 733 | datas = datas.encode('utf-8') 734 | if 'DOCTYPEhtml' in str(datas): 735 | continue 736 | datas = re.sub(r'^(.*?)\{', '{', datas.decode('utf-8'), flags=re.DOTALL | re.MULTILINE) 737 | await self.down(json.loads(datas), s_name) 738 | item['sourceName'] = s_name.split('.json')[0] 739 | item['sourceUrl'] = f'{self.cnb_slot}/{s_name}' 740 | items.append(item) 741 | newJson["storeHouse"] = items 742 | newJson = pprint.pformat(newJson, width=200) 743 | with open(f'{self.repo}{self.sep}{self.target}', 'w+', encoding='utf-8') as f: 744 | print(f"开始写入{self.target}") 745 | f.write(json.dumps(json.loads(str(newJson).replace("'", '"')), sort_keys=True, indent=4, ensure_ascii=False)) 746 | # 单仓 747 | else: 748 | try: 749 | res = json.loads(str(res)) 750 | except Exception as e: 751 | if 'domainnameisinvalid' in res: 752 | print(f'该域名无效，请提供正常可用接口') 753 | return 754 | html = await self.js_render(self.url) 755 | res = html.text.replace(' ', '').replace("'", '"') 756 | if not res: 757 | res = self.picparse(self.url).replace(' ', '').replace("'", '"') 758 | try: 759 | res = json.loads(str(res)) 760 | except Exception as e: 761 | # print(111111, e, res) 762 | pass 763 | s_name = self.target 764 | s_url = self.url 765 | print("【单仓】 {}: {}".format(s_name, s_url)) 766 | try: 767 | await self.down(res, s_name) 768 | except Exception as e: 769 | if 'searchable' in str(res): 770 | filename = self.signame + '.txt' if self.signame else f"{''.join(random.choices(string.ascii_letters + string.digits, k=10))}.txt" 771 | print("【线路】 {}: {}".format(filename, self.url)) 772 | try: 773 | await self.download(self.url, filename.split('.txt')[0], filename, cang=False) 774 | except Exception as e: 775 | print('下载异常', e) 776 | def replace_urls_gh1(self, content): 777 | # 适用于https://ghp.ci/https://raw.githubusercontent.com|https://raw.yzuu.cf类型 778 | # 使用正则表达式查找并替换链接 779 | def replace_match(match): 780 | username = match.group(2) 781 | repo_name = match.group(3) 782 | path = match.group(4) 783 | # 构建新的URL 784 | new_url = f"{self.mirror_proxy}/{username}/{repo_name}/{self.main_branch}" 785 | if path: 786 | new_url += path 787 | return new_url 788 | return self.pattern.sub(replace_match, content) 789 | def replace_urls_gh2(self, content): 790 | # 适用于https://gcore/jsdelivr.net/gh类型 791 | # 替换函数 792 | def replace_match(match): 793 | return f'{self.mirror_proxy}/{match.group(2)}/{match.group(3)}{match.group(5)}' 794 | # 执行替换 795 | return self.pattern.sub(replace_match, content) 796 | def mirror_proxy2new(self): 797 | # 把文本文件中所有镜像代理路径替换掉 798 | if self.mirror < 20: 799 | # gh1 适用于https://ghp.ci/https://raw.githubusercontent.com|https://raw.yzuu.cf类型 800 | # 自动转换提供的域名列表为正则表达式所需的格式 801 | patterns = [re.escape(proxy) for proxy in self.gh2] 802 | # 组合成一个正则表达式 803 | self.pattern = re.compile(r'({})/([a-zA-Z0-9_-]+)/([a-zA-Z0-9_-]+)(/.*)?'.format('|'.join(patterns))) 804 | # 遍历文件夹中的所有文件 805 | for root, dirs, files in os.walk(self.repo): 806 | for file in files: 807 | if file.endswith('.txt') or file.endswith('.json'): 808 | file_path = os.path.join(root, file) 809 | with open(file_path, 'r', encoding='utf-8') as f: 810 | content = f.read() 811 | # 替换文件中的URL 812 | new_content = self.replace_urls_gh1(content) 813 | for i in self.gh1: 814 | new_content = new_content.replace(i,self.mirror_proxy) 815 | with open(file_path, 'w', encoding='utf-8') as f: 816 | f.write(new_content) 817 | elif self.mirror > 20: 818 | # gh2适用于https://gcore.jsdelivr.net/gh类型 819 | if self.jar_suffix not in ['html','js','css','json','txt']: 820 | return 821 | # 自动转换提供的域名列表为正则表达式所需的格式 822 | patterns = [re.escape(proxy) for proxy in self.gh1] 823 | # 组合成一个正则表达式 824 | self.pattern = re.compile(r'({})/(.+?)/(.+?)/(master|main)(/|/.*)'.format('|'.join(patterns))) 825 | # 遍历文件夹中的所有文件 826 | for filename in os.listdir(self.repo): 827 | file_path = os.path.join(self.repo, filename) 828 | # 检查是否是文件而不是文件夹 829 | if os.path.isfile(file_path) and (filename.endswith('.txt') or filename.endswith('.json')): 830 | # 读取文件内容 831 | with open(file_path, 'r', encoding='utf-8') as file: 832 | content = file.read() 833 | # 替换文件中的URL 834 | new_content = self.replace_urls_gh2(content) 835 | for i in self.gh2: 836 | new_content = new_content.replace(i,self.mirror_proxy) 837 | # 写回文件 838 | with open(file_path, 'w', encoding='utf-8') as f: 839 | f.write(new_content) 840 | def mirror_init(self): 841 | # gh1 842 | if self.mirror == 1: 843 | self.mirror_proxy = 'https://ghp.ci/https://raw.githubusercontent.com' 844 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 845 | elif self.mirror == 2: 846 | self.mirror_proxy = 'https://gh.xxooo.cf/https://raw.githubusercontent.com' 847 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 848 | elif self.mirror == 3: 849 | self.mirror_proxy = 'https://ghproxy.net/https://raw.githubusercontent.com' 850 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 851 | elif self.mirror == 4: 852 | self.mirror_proxy = 'https://github.moeyy.xyz/https://raw.githubusercontent.com' 853 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 854 | elif self.mirror == 5: 855 | self.mirror_proxy = 'https://gh-proxy.com/https://raw.githubusercontent.com' 856 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 857 | elif self.mirror == 6: 858 | self.mirror_proxy = 'https://ghproxy.cc/https://raw.githubusercontent.com' 859 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 860 | elif self.mirror == 7: 861 | self.mirror_proxy = 'https://raw.yzuu.cf' 862 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 863 | # self.registry = 'hub.yzuu.cf' 864 | elif self.mirror == 8: 865 | self.mirror_proxy = 'https://raw.nuaa.cf' 866 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 867 | elif self.mirror == 9: 868 | self.mirror_proxy = 'https://raw.kkgithub.com' 869 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 870 | elif self.mirror == 10: 871 | self.mirror_proxy = 'https://gh.con.sh/https://raw.githubusercontent.com' 872 | elif self.mirror == 11: 873 | self.mirror_proxy = 'https://gh.llkk.cc/https://raw.githubusercontent.com' 874 | elif self.mirror == 12: 875 | self.mirror_proxy = 'https://gh.ddlc.top/https://raw.githubusercontent.com' 876 | elif self.mirror == 13: 877 | self.mirror_proxy = 'https://gh-proxy.llyke.com/https://raw.githubusercontent.com' 878 | 879 | # gh2 880 | elif self.mirror == 21: 881 | self.mirror_proxy = "https://fastly.jsdelivr.net/gh" 882 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 883 | elif self.mirror == 22: 884 | self.mirror_proxy = "https://jsd.onmicrosoft.cn/gh" 885 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 886 | elif self.mirror == 23: 887 | self.mirror_proxy = "https://gcore.jsdelivr.net/gh" 888 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 889 | elif self.mirror == 24: 890 | self.mirror_proxy = "https://cdn.jsdmirror.com/gh" 891 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 892 | elif self.mirror == 25: 893 | self.mirror_proxy = "https://cdn.jsdmirror.cn/gh" 894 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 895 | elif self.mirror == 26: 896 | self.mirror_proxy = "https://jsd.proxy.aks.moe/gh" 897 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 898 | elif self.mirror == 27: 899 | self.mirror_proxy = "https://jsdelivr.b-cdn.net/gh" 900 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 901 | elif self.mirror == 28: 902 | self.mirror_proxy = "https://jsdelivr.pai233.top/gh" 903 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 904 | 905 | def run(self): 906 | start_time = time.time() 907 | self.git_clone() 908 | asyncio.run(self.batch_handle_online_interface()) 909 | repo = self.get_local_repo() 910 | self.all() 911 | self.mirror_proxy2new() 912 | self.git_push(repo) 913 | end_time = time.time() 914 | print(f'耗时: {end_time - start_time} 秒\n\n#################影视仓APP配置接口########################\n\n{self.cnb_slot}/all.json\n{self.cnb_slot}/{self.target}') 915 | 916 | 917 | if __name__ == '__main__': 918 | token = 'xxx' 919 | username = 'fish2018' 920 | repo = 'test' 921 | # url = 'https://github.moeyy.xyz/https://raw.githubusercontent.com/wwb521/live/main/video.json?signame=18' 922 | # url = 'https://github.moeyy.xyz/https://raw.githubusercontent.com/supermeguo/BoxRes/main/Myuse/catcr.json?signame=v18' 923 | # url = 'http://box.ufuzi.com/tv/qq/%E7%9F%AD%E5%89%A7%E9%A2%91%E9%81%93/api.json?signame=duanju' 924 | # url = 'https://肥猫.com?signame=肥猫' 925 | url = 'https://tvbox.catvod.com/xs/api.json?signame=xs' 926 | site_down = True # 将site中的文件下载本地化 927 | GetSrc(username=username, token=token, url=url, repo=repo, mirror=4, num=10, site_down=site_down).run() 928 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | GitPython==3.1.43 2 | requests==2.27.1 3 | requests-html==0.10.0 4 | lxml_html_clean==0.2.1 5 | pyppeteer==2.0.0 6 | -------------------------------------------------------------------------------- /tvbox_tools.py: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8 -*- 2 | # !/usr/bin/env python3 3 | from requests_html import HTMLSession 4 | import pprint 5 | import random 6 | import string 7 | import time 8 | import hashlib 9 | import json 10 | import git # gitpython 11 | import re 12 | import base64 13 | import requests 14 | from requests.adapters import HTTPAdapter, Retry 15 | import os 16 | import subprocess 17 | import ssl 18 | from pathlib import Path 19 | ssl._create_default_https_context = ssl._create_unverified_context 20 | import urllib3 21 | from urllib3.exceptions import InsecureRequestWarning 22 | urllib3.disable_warnings(InsecureRequestWarning) 23 | 24 | global pipes 25 | pipes = set() 26 | 27 | class GetSrc: 28 | def __init__(self, username=None, token=None, url=None, repo=None, num=10, target=None, timeout=3, signame=None, mirror=None, jar_suffix=None): 29 | self.jar_suffix = jar_suffix if jar_suffix else 'jar' 30 | self.mirror = int(str(mirror).strip()) if mirror else 1 31 | self.registry = 'github.com' 32 | self.mirror_proxy = 'https://ghp.ci/https://raw.githubusercontent.com' 33 | self.num = int(num) 34 | self.sep = os.path.sep 35 | self.username = username 36 | self.token = token 37 | self.timeout=timeout 38 | self.url = url 39 | self.repo = repo if repo else 'tvbox' 40 | self.target = f'{target.split(".json")[0]}.json' if target else 'tvbox.json' 41 | self.headers = {"user-agent": "okhttp/3.15 Html5Plus/1.0 (Immersed/23.92157)"} 42 | self.s = requests.Session() 43 | self.signame = signame 44 | retries = Retry(total=3, backoff_factor=1) 45 | self.s.mount('http://', HTTPAdapter(max_retries=retries)) 46 | self.s.mount('https://', HTTPAdapter(max_retries=retries)) 47 | self.size_tolerance = 15 # 线路文件大小误差在15以内认为是同一个 48 | self.main_branch = 'main' 49 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 50 | self.gh1 = [ 51 | 'https://ghp.ci/https://raw.githubusercontent.com', 52 | 'https://gitdl.cn/https://raw.githubusercontent.com', 53 | 'https://ghproxy.net/https://raw.githubusercontent.com', 54 | 'https://github.moeyy.xyz/https://raw.githubusercontent.com', 55 | 'https://gh-proxy.com/https://raw.githubusercontent.com', 56 | 'https://ghproxy.cc/https://raw.githubusercontent.com', 57 | 'https://raw.yzuu.cf', 58 | 'https://raw.nuaa.cf', 59 | 'https://raw.kkgithub.com', 60 | 'https://mirror.ghproxy.com/https://raw.githubusercontent.com', 61 | 'https://gh.llkk.cc/https://raw.githubusercontent.com', 62 | 'https://gh.ddlc.top/https://raw.githubusercontent.com', 63 | 'https://gh-proxy.llyke.com/https://raw.githubusercontent.com', 64 | 'https://slink.ltd', 65 | 'https://cors.zme.ink', 66 | 'https://git.886.be' 67 | ] 68 | self.gh2 = [ 69 | "https://fastly.jsdelivr.net/gh", 70 | "https://jsd.onmicrosoft.cn/gh", 71 | "https://gcore.jsdelivr.net/gh", 72 | "https://cdn.jsdmirror.com/gh", 73 | "https://cdn.jsdmirror.cn/gh", 74 | "https://jsd.proxy.aks.moe/gh", 75 | "https://jsdelivr.b-cdn.net/gh", 76 | "https://jsdelivr.pai233.top/gh" 77 | ] 78 | def file_hash(self, filepath): 79 | with open(filepath, 'rb') as f: 80 | file_contents = f.read() 81 | return hashlib.sha256(file_contents).hexdigest() 82 | def remove_duplicates(self, folder_path): 83 | folder_path = Path(folder_path) 84 | jar_folder = f'{folder_path}/jar' 85 | excludes = {'.json', '.git', 'jar', '.idea', 'ext', '.DS_Store', '.md'} 86 | files_info = {} 87 | 88 | # 把jar目录下所有文件后缀都改成新的self.jar_suffix 89 | self.rename_jar_suffix(jar_folder) 90 | 91 | # 存储文件名、大小和哈希值 92 | for file_path in folder_path.iterdir(): 93 | if file_path.is_file() and file_path.suffix not in excludes: 94 | file_size = file_path.stat().st_size 95 | file_hash = self.file_hash(file_path) 96 | files_info[file_path.name] = {'path': str(file_path), 'size': file_size, 'hash': file_hash} 97 | 98 | # 保留的文件列表 99 | keep_files = [] 100 | # 按文件大小排序，然后按顺序处理 101 | for file_name, info in sorted(files_info.items(), key=lambda item: item[1]['size']): 102 | if not keep_files or abs(info['size'] - files_info[keep_files[-1]]['size']) > self.size_tolerance: 103 | keep_files.append(file_name) 104 | # 删除jar目录下除了{self.jar_suffix}的文件 105 | self.remove_all_except_jar(jar_folder) 106 | else: 107 | # 如果当前文件大小在容忍范围内，删除当前文件和对应的jar文件 108 | os.remove(info['path']) 109 | self.remove_jar_file(jar_folder, file_name.replace('.txt', f'{self.jar_suffix}')) 110 | 111 | keep_files.sort() 112 | return keep_files 113 | def rename_jar_suffix(self,jar_folder): 114 | # 遍历目录中的所有文件和子目录 115 | for root, dirs, files in os.walk(jar_folder): 116 | for file in files: 117 | # 构造完整的文件路径 118 | old_file = os.path.join(root, file) 119 | # 构造新的文件名，去除原有的后缀，加上 self.jar_suffix 120 | new_file = os.path.join(root, os.path.splitext(file)[0] + f'.{self.jar_suffix}') 121 | # 重命名文件 122 | os.rename(old_file, new_file) 123 | # print(f"文件已重命名: {old_file} -> {new_file}") 124 | def remove_all_except_jar(self, jar_folder): 125 | # 列出文件夹中的所有文件 126 | for file_name in os.listdir(jar_folder): 127 | # 构建完整的文件路径 128 | full_path = os.path.join(jar_folder, file_name) 129 | # 检查是否为文件 130 | if os.path.isfile(full_path): 131 | # 获取文件的扩展名 132 | _, file_extension = os.path.splitext(file_name) 133 | # 如果扩展名不是self.jar_suffix，则删除文件 134 | if file_extension != f'.{self.jar_suffix}': 135 | self.remove_jar_file(jar_folder, file_name) 136 | def remove_jar_file(self, jar_folder, file_name): 137 | # 构建jar文件的路径 138 | jar_file_path = os.path.join(jar_folder, file_name) 139 | # 如果jar文件存在，则删除它 140 | if os.path.isfile(jar_file_path): 141 | os.remove(jar_file_path) 142 | def remove_emojis(self, text): 143 | emoji_pattern = re.compile("[" 144 | u"\U0001F600-\U0001F64F" # emoticons 145 | u"\U0001F300-\U0001F5FF" # symbols & pictographs 146 | u"\U0001F680-\U0001F6FF" # transport & map symbols 147 | u"\U0001F1E0-\U0001F1FF" # flags (iOS) 148 | "\U00002500-\U00002BEF" # chinese char 149 | "\U00010000-\U0010ffff" 150 | "\u200d" # zero width joiner 151 | "\u20E3" # combining enclosing keycap 152 | "\ufe0f" # VARIATION SELECTOR-16 153 | "]+", flags=re.UNICODE) 154 | text = text.replace('/', '_').replace('多多', '').replace('┃', '').replace('线路', '').replace('匚','').strip() 155 | return emoji_pattern.sub('', text) 156 | def json_compatible(self, str): 157 | # 兼容错误json 158 | res = str.replace(' ', '').replace("'",'"').replace('key:', '"key":').replace('name:', '"name":').replace('type:', '"type":').replace('api:','"api":').replace('searchable:', '"searchable":').replace('quickSearch:', '"quickSearch":').replace('filterable:','"filterable":').strip() 159 | return res 160 | def ghproxy(self, str): 161 | u = 'https://ghp.ci/' 162 | res = str.replace('https://ghproxy.net/', u).replace('https://ghproxy.com/', u).replace('https://gh-proxy.com/',u).replace('https://mirror.ghproxy.com/',u) 163 | return res 164 | def set_hosts(self): 165 | # 设置github.com的加速hosts 166 | try: 167 | response = requests.get('https://hosts.gitcdn.top/hosts.json') 168 | if response.status_code == 200: 169 | hosts_data = response.json() 170 | # 遍历JSON数据，找到"github.com"对应的IP 171 | github_ip = None 172 | for entry in hosts_data: 173 | if entry[1] == "github.com": 174 | github_ip = entry[0] 175 | break 176 | if github_ip: 177 | # 读取现有的/etc/hosts文件 178 | with open('/etc/hosts', 'r+') as file: 179 | hosts_content = file.read() 180 | # 检查是否已经存在对应的IP 181 | if github_ip not in hosts_content: 182 | # 将新的IP添加到文件末尾 183 | file.write(f'\n{github_ip} github.com') 184 | print(f'IP address {github_ip} for github.com has been added to /etc/hosts.') 185 | else: 186 | print(f'IP address for github.com is already in /etc/hosts.') 187 | else: 188 | print('No IP found for github.com in the provided data.') 189 | else: 190 | print('Failed to retrieve data from https://hosts.gitcdn.top/hosts.json') 191 | except Exception as e: 192 | pass 193 | def picparse(self, url): 194 | r = self.s.get(url, headers=self.headers, timeout=self.timeout, verify=False) 195 | pattern = r'([A-Za-z0-9+/]+={0,2})' 196 | matches = re.findall(pattern, r.text) 197 | decoded_data = base64.b64decode(matches[-1]) 198 | text = decoded_data.decode('utf-8') 199 | return text 200 | def js_render(self, url): 201 | # 获取js渲染页面源代码 202 | timeout = self.timeout * 4 203 | if timeout > 15: 204 | timeout = 15 205 | browser_args = ['--no-sandbox', '--disable-dev-shm-usage', '--disable-gpu', '--disable-software-rasterizer','--disable-setuid-sandbox'] 206 | session = HTMLSession(browser_args=browser_args) 207 | r = session.get(f'http://lige.unaux.com/?url={url}', headers=self.headers, timeout=timeout, verify=False) 208 | # 等待页面加载完成，Requests-HTML 会自动等待 JavaScript 执行完成 209 | r.html.render(timeout=timeout) 210 | # print('解密结果:',r.html.text) 211 | return r.html 212 | def get_jar(self, name, url, text): 213 | name = f'{name}.{self.jar_suffix}' 214 | pattern = r'\"spider\":(\s)?\"([^,]+)\"' 215 | matches = re.search(pattern, text) 216 | try: 217 | jar = matches.group(2).replace('./', f'{url}/').split(';')[0] 218 | jar = jar.split('"spider":"')[-1] 219 | if name==f'{self.repo}.{self.jar_suffix}': 220 | name = f"{jar.split('/')[-1]}" 221 | # print('jar地址: ', jar) 222 | timeout = self.timeout * 4 223 | if timeout > 15: 224 | timeout = 15 225 | r = self.s.get(jar, timeout=timeout) 226 | with open(f'{self.repo}/jar/{name}', 'wb') as f: 227 | f.write(r.content) 228 | jar = f'{self.slot}/jar/{name}' 229 | print(111111,jar) 230 | text = text.replace(matches.group(2), jar) 231 | except Exception as e: 232 | print(f'【jar下载失败】{name} jar地址: {jar} error:{e}') 233 | return text 234 | def download(self, url, name, filename, cang=True): 235 | # 下载单线路 236 | item = {} 237 | try: 238 | path = os.path.dirname(url) 239 | r = self.s.get(url, headers=self.headers, allow_redirects=True, timeout=self.timeout, verify=False) 240 | if r.status_code == 200: 241 | print("开始下载【线路】{}: {}".format(name, url)) 242 | if 'searchable' not in r.text: 243 | r = self.js_render(url) 244 | if not r.text: 245 | r = self.picparse(url) 246 | if 'searchable' not in r: 247 | raise 248 | r = self.get_jar(name, url, r) 249 | with open(f'{self.repo}{self.sep}{filename}', 'w+', encoding='utf-8') as f: 250 | f.write(r) 251 | return 252 | if 'searchable' not in r.text: 253 | raise 254 | with open(f'{self.repo}{self.sep}{filename}', 'w+', encoding='utf-8') as f: 255 | try: 256 | if r.content.decode('utf8').startswith(u'\ufeff'): 257 | str = r.content.decode('utf8').encode('utf-8')[3:].decode('utf-8') 258 | else: 259 | str = r.content.decode('utf-8').replace('./', f'{path}/') 260 | except: 261 | str = r.text 262 | finally: 263 | r = self.ghproxy(str.replace('./', f'{path}/')) 264 | 265 | r = self.get_jar(name, url, r) 266 | f.write(r) 267 | pipes.add(name) 268 | 269 | except Exception as e: 270 | print(f"【线路】{name}: {url} 下载错误：{e}") 271 | # 单仓时写入item 272 | if os.path.exists(f'{self.repo}{self.sep}{filename}') and cang: 273 | item['name'] = name 274 | item['url'] = f'{self.slot}/{filename}' 275 | items.append(item) 276 | def down(self, data, s_name): 277 | ''' 278 | 下载单仓 279 | ''' 280 | newJson = {} 281 | global items 282 | items = [] 283 | urls = data.get("urls") if data.get("urls") else data.get("sites") 284 | for u in urls: 285 | name = u.get("name").strip() 286 | name = self.remove_emojis(name) 287 | url = u.get("url") 288 | url = self.ghproxy(url) 289 | filename = '{}.txt'.format(name) 290 | if name in pipes: 291 | print(f"【线路】{name} 已存在，无需重复下载") 292 | continue 293 | self.download(url, name, filename) 294 | newJson['urls'] = items 295 | newJson = pprint.pformat(newJson, width=200) 296 | print(f'开始写入单仓{s_name}') 297 | with open(f'{self.repo}{self.sep}{s_name}', 'w+', encoding='utf-8') as f: 298 | content = str(newJson).replace("'", '"') 299 | f.write(json.loads(json.dumps(content, indent=4, ensure_ascii=False))) 300 | def all(self): 301 | # 整合所有文件到all.json 302 | newJson = {} 303 | items = [] 304 | files = self.remove_duplicates(self.repo) 305 | for file in files: 306 | item = {} 307 | item['name'] = file.split('.txt')[0] 308 | item['url'] = f'{self.slot}/{file}' 309 | items.append(item) 310 | newJson['urls'] = items 311 | newJson = pprint.pformat(newJson, width=200) 312 | print(f'开始写入all.json') 313 | with open(f'{self.repo}{self.sep}all.json', 'w+', encoding='utf-8') as f: 314 | content = str(newJson).replace("'", '"') 315 | f.write(json.loads(json.dumps(content, indent=4, ensure_ascii=False))) 316 | def batch_handle_online_interface(self): 317 | # 下载线路，处理多url场景 318 | print(f'--------- 开始私有化在线接口 ----------') 319 | urls = self.url.split(',') 320 | for url in urls: 321 | item = url.split('?&signame=') 322 | self.url = item[0] 323 | self.signame = item[1] if len(item) > 1 else None 324 | print(f'当前url: {self.url}') 325 | self.storeHouse() 326 | def git_clone(self): 327 | # self.registry = 'githubfast.com' 328 | # self.registry = 'hub.yzuu.cf' 329 | print(f'开始克隆：git clone https://github.com/{self.username}/{self.repo}.git') 330 | self.domain = f'https://{self.token}@{self.registry}/{self.username}/{self.repo}.git' 331 | if os.path.exists(self.repo): 332 | subprocess.call(['rm', '-rf', self.repo]) 333 | try: 334 | repo = git.Repo.clone_from(self.domain, to_path=self.repo, depth=1) 335 | [os.makedirs(d, exist_ok=True) for d in [f'{self.repo}/jar']] 336 | self.get_local_repo() 337 | except Exception as e: 338 | try: 339 | self.registry = 'gitdl.cn' 340 | self.domain = f'https://{self.token}@{self.registry}/https://github.com/{self.username}/{self.repo}.git' 341 | if os.path.exists(self.repo): 342 | subprocess.call(['rm', '-rf', self.repo]) 343 | repo = git.Repo.clone_from(self.domain, to_path=self.repo, depth=1) 344 | [os.makedirs(d, exist_ok=True) for d in [f'{self.repo}/jar']] 345 | self.get_local_repo() 346 | except Exception as e: 347 | print(222222, e) 348 | def get_local_repo(self): 349 | # 打开本地仓库，读取仓库信息 350 | repo = git.Repo(self.repo) 351 | config_writer = repo.config_writer() 352 | config_writer.set_value('user', 'name', self.username) 353 | config_writer.set_value('user', 'email', self.username) 354 | # 设置 http.postBuffer 355 | config_writer.set_value('http', 'postBuffer', '524288000') 356 | config_writer.release() 357 | 358 | # 获取远程仓库的引用 359 | remote = repo.remote(name='origin') 360 | # 获取远程分支列表 361 | remote_branches = remote.refs 362 | # 遍历远程分支，查找主分支 363 | for branch in remote_branches: 364 | if branch.name == 'origin/master' or branch.name == 'origin/main': 365 | self.main_branch = branch.name.split('/')[-1] 366 | break 367 | print(f"仓库{self.repo} 主分支为: {self.main_branch}") 368 | self.mirror_init() 369 | return repo 370 | def git_push(self,repo): 371 | # 推送并重置commit计数 372 | # 推送 373 | print(f'--------- 完成私有化在线接口 ----------\n开始推送：git push https://{self.registry}/{self.username}/{self.repo}.git') 374 | try: 375 | repo.git.add(A=True) 376 | repo.git.commit(m="update") 377 | repo.git.push() 378 | except Exception as e: 379 | # print('git推送异常', e) 380 | # 打开本地仓库，读取仓库信息 381 | config_writer = repo.config_writer() 382 | self.registry = 'github.com' 383 | config_writer.set_value('remote "origin"', 'url', f'https://{self.token}@{self.registry}/{self.username}/{self.repo}.git') 384 | config_writer.release() 385 | # 重置commit 386 | try: 387 | os.chdir(self.repo) 388 | # print('开始清理git',os.getcwd()) 389 | repo.git.checkout('--orphan', 'tmp_branch') 390 | repo.git.add(A=True) 391 | repo.git.commit(m="update") 392 | repo.git.execute(['git', 'branch', '-D', self.main_branch]) 393 | repo.git.execute(['git', 'branch', '-m', self.main_branch]) 394 | repo.git.execute(['git', 'push', '-f', 'origin', self.main_branch]) 395 | except Exception as e: 396 | print('git清理异常', e) 397 | def storeHouse(self): 398 | ''' 399 | 生成多仓json文件 400 | ''' 401 | newJson = {} 402 | items = [] 403 | # 解析最初链接 404 | try: 405 | res = self.s.get(self.url, headers=self.headers, verify=False).content.decode('utf8') 406 | except Exception as e: 407 | res = self.js_render(self.url).text.replace(' ', '').replace("'", '"') 408 | if not res: 409 | res = self.picparse(self.url).replace(' ', '').replace("'", '"') 410 | # 线路 411 | if 'searchable' in str(res): 412 | filename = self.signame + '.txt' if self.signame else f"{''.join(random.choices(string.ascii_letters + string.digits, k=10))}.txt" 413 | path = os.path.dirname(self.url) 414 | print("【线路】 {}: {}".format(self.repo, self.url)) 415 | try: 416 | with open(f'{self.repo}{self.sep}{filename}', 'w+', encoding='utf-8') as f, open( 417 | f'{self.repo}{self.sep}{self.target}', 'w+', encoding='utf-8') as f2: 418 | r = self.ghproxy(res.replace('./', f'{path}/')) 419 | r = self.get_jar(filename.split('.txt')[0], url, r) 420 | f.write(r) 421 | f2.write(r) 422 | except Exception as e: 423 | print(333333333, e) 424 | return 425 | 426 | # json容错处理 427 | res = self.json_compatible(res) 428 | # 移除注释 429 | datas = '' 430 | for d in res.splitlines(): 431 | if d.find(" //") != -1 or d.find("// ") != -1 or d.find(",//") != -1 or d.startswith("//"): 432 | d = d.split(" //", maxsplit=1)[0] 433 | d = d.split("// ", maxsplit=1)[0] 434 | d = d.split(",//", maxsplit=1)[0] 435 | d = d.split("//", maxsplit=1)[0] 436 | datas = '\n'.join([datas, d]) 437 | # 容错处理，便于json解析 438 | datas = datas.replace('\n', '') 439 | res = datas.replace(' ', '').replace("'", '"').replace('\n', '') 440 | if datas.startswith(u'\ufeff'): 441 | try: 442 | res = datas.encode('utf-8')[3:].decode('utf-8').replace(' ', '').replace("'", '"').replace('\n', '') 443 | except Exception as e: 444 | res = datas.encode('utf-8')[4:].decode('utf-8').replace(' ', '').replace("'", '"').replace('\n', '') 445 | 446 | # 多仓 447 | elif 'storeHouse' in datas: 448 | res = json.loads(str(res)) 449 | srcs = res.get("storeHouse") if res.get("storeHouse") else None 450 | if srcs: 451 | i = 1 452 | for s in srcs: 453 | if i > self.num: 454 | break 455 | i += 1 456 | item = {} 457 | s_name = s.get("sourceName") 458 | s_name = self.remove_emojis(s_name) 459 | s_name = f'{s_name}.json' 460 | s_url = s.get("sourceUrl") 461 | print("【多仓】 {}: {}".format(s_name, s_url)) 462 | try: 463 | if self.s.get(s_url, headers=self.headers).status_code >= 400: 464 | continue 465 | except Exception as e: 466 | print('地址无法响应: ',e) 467 | continue 468 | try: 469 | if self.s.get(s_url, headers=self.headers).content.decode('utf8').lstrip().startswith(u'\ufeff'): 470 | data = self.s.get(s_url, headers=self.headers).content.decode('utf-8')[1:] 471 | else: 472 | data = self.s.get(s_url, headers=self.headers).content.decode('utf-8') 473 | except Exception as e: 474 | try: 475 | data = self.s.get(s_url, headers=self.headers).content.decode('utf8') 476 | data = data.encode('utf-8').decode('utf-8') 477 | except Exception as e: 478 | continue 479 | datas = '' 480 | for d in data.splitlines(): 481 | if d.find(" //") != -1 or d.find("// ") != -1 or d.find(",//") != -1 or d.startswith("//"): 482 | d = d.split(" //", maxsplit=1)[0] 483 | d = d.split("// ", maxsplit=1)[0] 484 | d = d.split(",//", maxsplit=1)[0] 485 | d = d.split("//", maxsplit=1)[0] 486 | datas = '\n'.join([datas, d]) 487 | 488 | try: 489 | if datas.lstrip().startswith(u'\ufeff'): 490 | datas = datas.encode('utf-8')[1:] 491 | self.down(json.loads(datas), s_name) 492 | except Exception as e: 493 | try: 494 | data = self.s.get(s_url, headers=self.headers).text 495 | except Exception as e: 496 | continue 497 | datas = '' 498 | for d in data.splitlines(): 499 | datas += d.replace('\n', '').replace(' ', '').strip() 500 | datas = datas.encode('utf-8') 501 | if 'DOCTYPEhtml' in str(datas): 502 | continue 503 | datas = re.sub(r'^(.*?)\{', '{', datas.decode('utf-8'), flags=re.DOTALL | re.MULTILINE) 504 | self.down(json.loads(datas), s_name) 505 | item['sourceName'] = s_name.split('.json')[0] 506 | item['sourceUrl'] = f'{self.slot}/{s_name}' 507 | items.append(item) 508 | newJson["storeHouse"] = items 509 | newJson = pprint.pformat(newJson, width=200) 510 | with open(f'{self.repo}{self.sep}{self.target}', 'w+', encoding='utf-8') as f: 511 | print(f"开始写入{self.target}") 512 | f.write(json.dumps(json.loads(str(newJson).replace("'", '"')), sort_keys=True, indent=4, ensure_ascii=False)) 513 | # 单仓 514 | else: 515 | try: 516 | res = json.loads(str(res)) 517 | except Exception as e: 518 | res = self.js_render(self.url).text.replace(' ', '').replace("'", '"') 519 | if not res: 520 | res = self.picparse(self.url).replace(' ', '').replace("'", '"') 521 | try: 522 | res = json.loads(str(res)) 523 | except Exception as e: 524 | # print(111111, e, res) 525 | pass 526 | s_name = self.target 527 | s_url = self.url 528 | print("【单仓】 {}: {}".format(s_name, s_url)) 529 | try: 530 | self.down(res, s_name) 531 | except Exception as e: 532 | if 'searchable' in str(res): 533 | filename = self.signame + '.txt' if self.signame else f"{''.join(random.choices(string.ascii_letters + string.digits, k=10))}.txt" 534 | print("【线路】 {}: {}".format(filename, self.url)) 535 | try: 536 | self.download(self.url, filename.split('.txt')[0], filename, cang=False) 537 | except Exception as e: 538 | print('下载异常', e) 539 | def replace_urls_gh1(self, content): 540 | # 适用于https://ghp.ci/https://raw.githubusercontent.com|https://raw.yzuu.cf类型 541 | # 使用正则表达式查找并替换链接 542 | def replace_match(match): 543 | username = match.group(2) 544 | repo_name = match.group(3) 545 | path = match.group(4) 546 | # 构建新的URL 547 | new_url = f"{self.mirror_proxy}/{username}/{repo_name}/{self.main_branch}" 548 | if path: 549 | new_url += path 550 | return new_url 551 | return self.pattern.sub(replace_match, content) 552 | def replace_urls_gh2(self, content): 553 | # 适用于https://gcore/jsdelivr.net/gh类型 554 | # 替换函数 555 | def replace_match(match): 556 | return f'{self.mirror_proxy}/{match.group(2)}/{match.group(3)}{match.group(5)}' 557 | # 执行替换 558 | return self.pattern.sub(replace_match, content) 559 | def mirror_proxy2new(self): 560 | # 把文本文件中所有镜像代理路径替换掉 561 | if self.mirror < 20: 562 | # gh1 适用于https://ghp.ci/https://raw.githubusercontent.com|https://raw.yzuu.cf类型 563 | # 自动转换提供的域名列表为正则表达式所需的格式 564 | patterns = [re.escape(proxy) for proxy in self.gh2] 565 | # 组合成一个正则表达式 566 | self.pattern = re.compile(r'({})/([a-zA-Z0-9_-]+)/([a-zA-Z0-9_-]+)(/.*)?'.format('|'.join(patterns))) 567 | # 遍历文件夹中的所有文件 568 | for root, dirs, files in os.walk(self.repo): 569 | for file in files: 570 | if file.endswith('.txt') or file.endswith('.json'): 571 | file_path = os.path.join(root, file) 572 | with open(file_path, 'r', encoding='utf-8') as f: 573 | content = f.read() 574 | # 替换文件中的URL 575 | new_content = self.replace_urls_gh1(content) 576 | for i in self.gh1: 577 | new_content = new_content.replace(i,self.mirror_proxy) 578 | with open(file_path, 'w', encoding='utf-8') as f: 579 | f.write(new_content) 580 | elif self.mirror > 20: 581 | # gh2适用于https://gcore.jsdelivr.net/gh类型 582 | if self.jar_suffix not in ['html','js','css','json','txt']: 583 | return 584 | # 自动转换提供的域名列表为正则表达式所需的格式 585 | patterns = [re.escape(proxy) for proxy in self.gh1] 586 | # 组合成一个正则表达式 587 | self.pattern = re.compile(r'({})/(.+?)/(.+?)/(master|main)(/|/.*)'.format('|'.join(patterns))) 588 | # 遍历文件夹中的所有文件 589 | for filename in os.listdir(self.repo): 590 | file_path = os.path.join(self.repo, filename) 591 | # 检查是否是文件而不是文件夹 592 | if os.path.isfile(file_path) and (filename.endswith('.txt') or filename.endswith('.json')): 593 | # 读取文件内容 594 | with open(file_path, 'r', encoding='utf-8') as file: 595 | content = file.read() 596 | # 替换文件中的URL 597 | new_content = self.replace_urls_gh2(content) 598 | for i in self.gh2: 599 | new_content = new_content.replace(i,self.mirror_proxy) 600 | # 写回文件 601 | with open(file_path, 'w', encoding='utf-8') as f: 602 | f.write(new_content) 603 | def mirror_init(self): 604 | # gh1 605 | if self.mirror == 1: 606 | self.mirror_proxy = 'https://ghp.ci/https://raw.githubusercontent.com' 607 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 608 | elif self.mirror == 2: 609 | self.mirror_proxy = 'https://gitdl.cn/https://raw.githubusercontent.com' 610 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 611 | elif self.mirror == 3: 612 | self.mirror_proxy = 'https://ghproxy.net/https://raw.githubusercontent.com' 613 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 614 | elif self.mirror == 4: 615 | self.mirror_proxy = 'https://github.moeyy.xyz/https://raw.githubusercontent.com' 616 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 617 | elif self.mirror == 5: 618 | self.mirror_proxy = 'https://gh-proxy.com/https://raw.githubusercontent.com' 619 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 620 | elif self.mirror == 6: 621 | self.mirror_proxy = 'https://ghproxy.cc/https://raw.githubusercontent.com' 622 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 623 | elif self.mirror == 7: 624 | self.mirror_proxy = 'https://raw.yzuu.cf' 625 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 626 | # self.registry = 'hub.yzuu.cf' 627 | elif self.mirror == 8: 628 | self.mirror_proxy = 'https://raw.nuaa.cf' 629 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 630 | elif self.mirror == 9: 631 | self.mirror_proxy = 'https://raw.kkgithub.com' 632 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}/{self.main_branch}' 633 | elif self.mirror == 10: 634 | self.mirror_proxy = 'https://gh.con.sh/https://raw.githubusercontent.com' 635 | elif self.mirror == 11: 636 | self.mirror_proxy = 'https://gh.llkk.cc/https://raw.githubusercontent.com' 637 | elif self.mirror == 12: 638 | self.mirror_proxy = 'https://gh.ddlc.top/https://raw.githubusercontent.com' 639 | elif self.mirror == 13: 640 | self.mirror_proxy = 'https://gh-proxy.llyke.com/https://raw.githubusercontent.com' 641 | 642 | # gh2 643 | elif self.mirror == 21: 644 | self.mirror_proxy = "https://fastly.jsdelivr.net/gh" 645 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 646 | elif self.mirror == 22: 647 | self.mirror_proxy = "https://jsd.onmicrosoft.cn/gh" 648 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 649 | elif self.mirror == 23: 650 | self.mirror_proxy = "https://gcore.jsdelivr.net/gh" 651 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 652 | elif self.mirror == 24: 653 | self.mirror_proxy = "https://cdn.jsdmirror.com/gh" 654 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 655 | elif self.mirror == 25: 656 | self.mirror_proxy = "https://cdn.jsdmirror.cn/gh" 657 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 658 | elif self.mirror == 26: 659 | self.mirror_proxy = "https://jsd.proxy.aks.moe/gh" 660 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 661 | elif self.mirror == 27: 662 | self.mirror_proxy = "https://jsdelivr.b-cdn.net/gh" 663 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 664 | elif self.mirror == 28: 665 | self.mirror_proxy = "https://jsdelivr.pai233.top/gh" 666 | self.slot = f'{self.mirror_proxy}/{self.username}/{self.repo}' 667 | 668 | def run(self): 669 | start_time = time.time() 670 | self.set_hosts() 671 | self.mirror_init() 672 | self.git_clone() 673 | self.batch_handle_online_interface() 674 | repo = self.get_local_repo() 675 | self.all() 676 | self.mirror_proxy2new() 677 | self.git_push(repo) 678 | end_time = time.time() 679 | print(f'耗时: {end_time - start_time} 秒\n\n#################影视仓APP配置接口########################\n\n{self.slot}/all.json\n{self.slot}/{self.target}') 680 | 681 | 682 | if __name__ == '__main__': 683 | token = os.getenv('token') 684 | username = os.getenv('username') if os.getenv('username') else os.getenv('u') 685 | repo = os.getenv('repo') 686 | signame = os.getenv('signame') 687 | target = os.getenv('target') 688 | num = os.getenv('num') if os.getenv('num') else 10 689 | url = os.getenv('url') 690 | timeout = os.getenv('timeout') if os.getenv('timeout') else 3 691 | mirror = os.getenv('mirror') 692 | jar_suffix = os.getenv('jar_suffix') 693 | GetSrc(username=username, token=token, url=url, repo=repo, num=num, target=target, timeout=timeout, signame=signame, mirror=mirror, jar_suffix=jar_suffix).run() 694 | --------------------------------------------------------------------------------