├── Telegram ├── __init__.py ├── .env.sample ├── main.py ├── win11_tgram.svg ├── TelegramFUSE.py └── fuse_impl.py ├── requirements.txt └── README.md /Telegram/__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | telethon 2 | python-dotenv 3 | cryptography 4 | cryptg 5 | pyfuse3 6 | cachetools -------------------------------------------------------------------------------- /Telegram/.env.sample: -------------------------------------------------------------------------------- 1 | APP_ID=YOUR_TELEGRAM_APP_ID 2 | APP_HASH=YOUR_TELEGRAM_APP_HASH 3 | CHANNEL_LINK=YOUR_TELEGRAM_CHANNEL_LINK 4 | SESSION_NAME=user-session -------------------------------------------------------------------------------- /Telegram/main.py: -------------------------------------------------------------------------------- 1 | import os 2 | from dotenv import load_dotenv 3 | from fuse_impl import runFs 4 | from TelegramFUSE import TelegramFileClient 5 | 6 | def init(): 7 | load_dotenv() 8 | 9 | api_id = os.getenv("APP_ID") 10 | api_hash = os.getenv("APP_HASH") 11 | channel_link = os.getenv("CHANNEL_LINK") 12 | session_name = os.getenv("SESSION_NAME") 13 | 14 | client = TelegramFileClient(session_name, api_id, api_hash, channel_link) 15 | runFs(client) 16 | 17 | if __name__ == "__main__": 18 | init() 19 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Keep In Mind 2 | Though Telegram allows file uploads, it is not intended to be used as cloud storage. Your files could be lost at any time. Don't rely on this project (or any similar ones) for storing important files on Telegram. Storing large amounts of files on this **could result in Telegram deleting your files or banning you, proceed at your own risk**. 3 | 4 | # TelegramFS 5 | A [FUSE](https://en.wikipedia.org/wiki/Filesystem_in_Userspace) program that stores files on Telegram. 6 | 7 | Though I demonstrated Discord in the video too, I haven't included the code here. While I believe that storing your OWN files on Discord does NOT violate TOS, I think that spreading the code to do so might. Idk I'm trying to not actually get banned :) 8 | 9 | ## Usage and How to Run 10 | ### Requirements 11 | - Linux. Sorry Windows gamers, probably WSL would work 12 | - Python - I used 3.10.12 13 | - libfuse3 14 | - A non-negligible amount of memory, if you plan to upload large files. I ran this on a system 15 | with 16GB RAM and that was usually enough, but I had occasional crashes uploading lots of large files in a row. 16 | 17 | 18 | ### To Run 19 | Before running this, I recommend creating a virtual environment in Python. 20 | 21 | - (optional) Create a venv with `python -m venv ` 22 | - Rename `.env.sample` to `.env` and fill in the variables with appropriate values. Here's a description of these values and where you can get them from: 23 | - `APP_ID` and `APP_HASH`: You can get these from https://my.telegram.org/myapp. **AGAIN, storing large amounts of files could get you banned. So be careful and take precautions if you care about losing your account.** 24 | - `CHANNEL_LINK`: the link to your Telegram channel. 25 | - `SESSION_NAME`: this can be whatever you want, just the name that will be used for the file storing details of your Telegram session. 26 | - Run `pip install -r requirements.txt`. 27 | - This might fail to get pyfuse3. If it does, you may be missing some requirements. On my system (Ubuntu 20.04), running this command to install some exta packages worked for me: 28 | `sudo apt install meson cmake fuse3 libfuse3-dev libglib2.0-dev pkg-config`. You can also install from these directions: http://www.rath.org/pyfuse3-docs/install.html 29 | - Enable the `user_allow_other` option in `/etc/fuse.conf`. 30 | - Run `python main.py ` for instance, `python3 main.py ./telegramfs` will mount at the directory `telegramfs`` in the current working dir. The directory you are mounting must exist. 31 | 32 | ## Known Issues 33 | Uploading large files to Telegram (more than ~3GB) may result in degraded performance or the system 34 | killing the process for using too much memory. This is probably my fault. I found the behavior of memory management in Python is a bit strange, even calling gc.collect() after clearing the buffer doesn't always seem to work. It could also be an issue with the LRU cache I implemented... I don't have the patience to wait 20 minutes for it to crash and then debug, especially because the vast majority of my files are pretty small. 35 | 36 | Error handling is somewhat lacking. If Telegram uploads fail, you'll probably see message in console, but it won't retry. Worst case, you can delete whatever file you were trying to upload and try again. 37 | 38 | # TO UNMOUNT, IF SOMETHING BREAKS 39 | `fusermount -u ` 40 | -------------------------------------------------------------------------------- /Telegram/win11_tgram.svg: -------------------------------------------------------------------------------- 1 | 68 | -------------------------------------------------------------------------------- /Telegram/TelegramFUSE.py: -------------------------------------------------------------------------------- 1 | import logging 2 | logging.basicConfig(filename='example.log', encoding='utf-8', level=logging.DEBUG) 3 | from telethon import TelegramClient, events, sync, utils 4 | from dotenv import load_dotenv 5 | import os 6 | from io import BytesIO 7 | from cryptography.fernet import Fernet 8 | from collections import defaultdict 9 | from cachetools import LRUCache 10 | import gc 11 | 12 | load_dotenv() 13 | 14 | FILE_MAX_SIZE_BYTES = int(2 * 1e9) # 2GB 15 | 16 | # Real LRU cache implementation very cool 17 | CACHE_MAXSIZE = 5e9 # 5GB 18 | def getsizeofelt(val): 19 | try: 20 | s = len(val) 21 | return s 22 | except: 23 | return 1 24 | 25 | def progress_cb(sent_bytes, total): 26 | percentTotal = int(sent_bytes/total * 100) 27 | if percentTotal % 5 == 0: 28 | print(f"Progress: {percentTotal}%...") 29 | 30 | class TelegramFileClient(): 31 | def __init__(self, session_name, api_id, api_hash, channel_link): 32 | self.client = TelegramClient(session_name, api_id, api_hash) 33 | self.client.start() 34 | self.channel_entity = self.client.get_entity(channel_link) 35 | # key to use for encryption, if not set, not encrypted. 36 | self.encryption_key = os.getenv("ENCRYPTION_KEY") 37 | self.cached_files = LRUCache(CACHE_MAXSIZE, getsizeof=getsizeofelt) 38 | self.fname_to_msgs = defaultdict(tuple) 39 | 40 | print("USING ENCRYPTION: ", self.encryption_key != None) 41 | 42 | def upload_file(self, bytesio, fh, file_name=None): 43 | # invalidate cache as soon as we upload file 44 | if fh in self.cached_files: 45 | self.cached_files.pop(fh) 46 | print("CLEANED UP ", gc.collect()) 47 | 48 | # file_bytes is bytesio obj 49 | file_bytes = bytesio.read() 50 | 51 | if self.encryption_key != None: 52 | print("ENCRYPTING") 53 | f = Fernet(bytes(self.encryption_key, 'utf-8')) 54 | file_bytes = f.encrypt(file_bytes) 55 | 56 | chunks = [] 57 | file_len = len(file_bytes) 58 | 59 | if isinstance(file_len, float): 60 | print("UH OH FILEBYTES LENGTH IS FLOAT", file_len) 61 | if isinstance(FILE_MAX_SIZE_BYTES, float): 62 | print("UH OH FILE_MAX_SIZE_BYTES IS FLOAT", FILE_MAX_SIZE_BYTES) 63 | 64 | if file_len > FILE_MAX_SIZE_BYTES: 65 | # Calculate the number of chunks needed 66 | num_chunks = (len(file_bytes) + FILE_MAX_SIZE_BYTES) // FILE_MAX_SIZE_BYTES 67 | 68 | if isinstance(num_chunks, float): 69 | print("UH OH num_chunks IS FLOAT", num_chunks) 70 | 71 | # Split the file into chunks 72 | for i in range(num_chunks): 73 | start = i * FILE_MAX_SIZE_BYTES 74 | end = (i + 1) * FILE_MAX_SIZE_BYTES 75 | chunk = file_bytes[start:end] 76 | chunks.append(chunk) 77 | else: 78 | # File is within the size limit, no need to split 79 | chunks = [file_bytes] 80 | 81 | upload_results = [] 82 | 83 | fname=file_name 84 | 85 | i = 0 86 | for c in chunks: 87 | fname = f"{file_name}_part{i}.txt" # convert everything to text. tgram is weird about some formats 88 | f = self.client.upload_file(c, file_name=fname, part_size_kb=512, progress_callback=progress_cb) 89 | result = self.client.send_file(self.channel_entity, f) 90 | upload_results.append(result) 91 | i += 1 92 | 93 | self.fname_to_msgs[file_name] = tuple([m.id for m in upload_results]) 94 | print(f"CACHED FILE! NEW SIZE: {self.cached_files.currsize}; maxsize: {self.cached_files.maxsize}") 95 | return upload_results 96 | 97 | def get_cached_file(self, fh): 98 | if fh in self.cached_files and self.cached_files[fh] != bytearray(b''): 99 | print("CACHE HIT") 100 | return self.cached_files[fh] 101 | return None 102 | 103 | # download entire file from telegram 104 | def download_file(self, fh, msgIds): 105 | if fh in self.cached_files and self.cached_files[fh] != bytearray(b''): 106 | print("CACHE HIT in download") 107 | return self.cached_files[fh] 108 | msgs = self.get_messages(msgIds) 109 | buf = BytesIO() 110 | for m in msgs: 111 | buf.write(self.download_message(m)) # error handling WHO?? 112 | numBytes = buf.getbuffer().nbytes 113 | print(f"Downloaded file is size {numBytes}") 114 | buf.seek(0) 115 | readBytes = buf.read() 116 | 117 | if self.encryption_key != None: 118 | print("DECRYPTING") 119 | f = Fernet(bytes(self.encryption_key, 'utf-8')) 120 | readBytes = f.decrypt(readBytes) 121 | 122 | barr = bytearray(readBytes) 123 | # add to cache 124 | self.cached_files[fh] = barr 125 | return barr 126 | 127 | def get_messages(self, ids): 128 | result = self.client.get_messages(self.channel_entity, ids=ids) 129 | return result 130 | 131 | def download_message(self, msg): 132 | result = msg.download_media(bytes) 133 | return result 134 | 135 | def delete_messages(self, ids): 136 | return self.client.delete_messages(self.channel_entity, message_ids=ids) -------------------------------------------------------------------------------- /Telegram/fuse_impl.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # -*- coding: utf-8 -*- 3 | ''' 4 | Based on the tmpfs.py example from pyfuse3: https://github.com/libfuse/pyfuse3/blob/master/examples/tmpfs.py 5 | 6 | A mountable filesystem that stores data on Telegram. Maintains a sqlite db on-disk to keep track of stuff 7 | like filename, which Telegram message IDs are associated with a file, etc. 8 | 9 | Permission is hereby granted, free of charge, to any person obtaining a copy of 10 | this software and associated documentation files (the "Software"), to deal in 11 | the Software without restriction, including without limitation the rights to 12 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 13 | the Software, and to permit persons to whom the Software is furnished to do so. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | ''' 22 | 23 | import os 24 | import sys 25 | 26 | # If we are running from the pyfuse3 source directory, try 27 | # to load the module from there first. 28 | basedir = os.path.abspath(os.path.join(os.path.dirname(sys.argv[0]), '..')) 29 | if (os.path.exists(os.path.join(basedir, 'setup.py')) and 30 | os.path.exists(os.path.join(basedir, 'src', 'pyfuse3', '__init__.pyx'))): 31 | sys.path.insert(0, os.path.join(basedir, 'src')) 32 | 33 | import pyfuse3 34 | import errno 35 | import stat 36 | from time import time 37 | import sqlite3 38 | import logging 39 | from collections import defaultdict 40 | from pyfuse3 import FUSEError 41 | from argparse import ArgumentParser 42 | import trio 43 | from io import BytesIO 44 | import gc 45 | import atexit 46 | 47 | try: 48 | import faulthandler 49 | except ImportError: 50 | pass 51 | else: 52 | faulthandler.enable() 53 | 54 | log = logging.getLogger() 55 | 56 | class Operations(pyfuse3.Operations): 57 | enable_writeback_cache = True 58 | 59 | def __init__(self, client): 60 | super(Operations, self).__init__() 61 | self.db = sqlite3.connect('telegram.db') 62 | self.db.text_factory = str 63 | self.db.row_factory = sqlite3.Row 64 | self.cursor = self.db.cursor() 65 | self.inode_open_count = defaultdict(int) 66 | self.client = client 67 | # buffer data for writes. BYTEARRAY ONLY. I really wanted this to be a dictionary to support multiple files 68 | # at once, but I had crazy memory management issues with dictionaries (even cachetools, even though we clear entries) 69 | # so just a buffer. 70 | self.write_buffer = bytearray(b'') 71 | try: 72 | # Check if inodes table exists 73 | self.cursor.execute(f"SELECT name FROM sqlite_master WHERE type='table' AND name=?", ("inodes",)) 74 | table_exists = self.cursor.fetchone() is not None 75 | 76 | # Create if not 77 | if not table_exists: 78 | print("creating tables") 79 | self.init_tables() 80 | 81 | except Exception as e: 82 | print(f"Error checking if table exists: {e}") 83 | 84 | # debug. shows contents of tables 85 | def load_tables(self): 86 | self.cursor.execute("SELECT * FROM inodes;") 87 | rows = self.cursor.fetchall() 88 | print(f"GOT {len(rows)} inodes") 89 | print(" id guid gid mode mtime_ns atime_ns ctime_ns target size rdev data") 90 | for row in rows: 91 | print("Row: ", " ".join([str(r) for r in row])) 92 | 93 | def init_tables(self): 94 | '''Initialize file system tables''' 95 | 96 | self.cursor.execute(""" 97 | CREATE TABLE inodes ( 98 | id INTEGER PRIMARY KEY, 99 | uid INT NOT NULL, 100 | gid INT NOT NULL, 101 | mode INT NOT NULL, 102 | mtime_ns INT NOT NULL, 103 | atime_ns INT NOT NULL, 104 | ctime_ns INT NOT NULL, 105 | target BLOB(256) , 106 | size INT NOT NULL DEFAULT 0, 107 | rdev INT NOT NULL DEFAULT 0, 108 | data BLOB 109 | ) 110 | """) 111 | 112 | # store telegram msgid associated with this inode 113 | self.cursor.execute(""" 114 | CREATE TABLE telegram_messages ( 115 | id INTEGER PRIMARY KEY, 116 | inode INT NOT NULL REFERENCES inodes(id) 117 | ) 118 | """) 119 | 120 | self.cursor.execute(""" 121 | CREATE TABLE contents ( 122 | rowid INTEGER PRIMARY KEY AUTOINCREMENT, 123 | name BLOB(256) NOT NULL, 124 | inode INT NOT NULL REFERENCES inodes(id), 125 | parent_inode INT NOT NULL REFERENCES inodes(id), 126 | 127 | UNIQUE (name, parent_inode) 128 | )""") 129 | 130 | # Insert root directory 131 | now_ns = int(time() * 1e9) 132 | self.cursor.execute("INSERT INTO inodes (id,mode,uid,gid,mtime_ns,atime_ns,ctime_ns) " 133 | "VALUES (?,?,?,?,?,?,?)", 134 | (pyfuse3.ROOT_INODE, stat.S_IFDIR | stat.S_IRUSR | stat.S_IWUSR 135 | | stat.S_IXUSR | stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH 136 | | stat.S_IXOTH, os.getuid(), os.getgid(), now_ns, now_ns, now_ns)) 137 | self.cursor.execute("INSERT INTO contents (name, parent_inode, inode) VALUES (?,?,?)", 138 | (b'..', pyfuse3.ROOT_INODE, pyfuse3.ROOT_INODE)) 139 | 140 | self.db.commit() 141 | 142 | 143 | def get_row(self, *a, **kw): 144 | self.cursor.execute(*a, **kw) 145 | try: 146 | row = next(self.cursor) 147 | except StopIteration: 148 | raise NoSuchRowError() 149 | try: 150 | next(self.cursor) 151 | except StopIteration: 152 | pass 153 | else: 154 | raise NoUniqueValueError() 155 | 156 | return row 157 | 158 | def get_rows(self, *a, **kw): 159 | self.cursor.execute(*a, **kw) 160 | rows = self.cursor.fetchall() 161 | 162 | if not rows: 163 | raise NoSuchRowError() 164 | 165 | return rows 166 | 167 | async def lookup(self, inode_p, name, ctx=None): 168 | if name == '.': 169 | inode = inode_p 170 | elif name == '..': 171 | inode = self.get_row("SELECT * FROM contents WHERE inode=?", 172 | (inode_p,))['parent_inode'] 173 | else: 174 | try: 175 | inode = self.get_row("SELECT * FROM contents WHERE name=? AND parent_inode=?", 176 | (name, inode_p))['inode'] 177 | except NoSuchRowError: 178 | raise(pyfuse3.FUSEError(errno.ENOENT)) 179 | 180 | return await self.getattr(inode, ctx) 181 | 182 | 183 | async def getattr(self, inode, ctx=None): 184 | try: 185 | row = self.get_row("SELECT * FROM inodes WHERE id=?", (inode,)) 186 | except NoSuchRowError: 187 | raise(pyfuse3.FUSEError(errno.ENOENT)) 188 | 189 | entry = pyfuse3.EntryAttributes() 190 | entry.st_ino = inode 191 | entry.generation = 0 192 | entry.entry_timeout = 300 193 | entry.attr_timeout = 300 194 | entry.st_mode = row['mode'] 195 | entry.st_nlink = self.get_row("SELECT COUNT(inode) FROM contents WHERE inode=?", 196 | (inode,))[0] 197 | entry.st_uid = row['uid'] 198 | entry.st_gid = row['gid'] 199 | entry.st_rdev = row['rdev'] 200 | entry.st_size = row['size'] 201 | 202 | entry.st_blksize = 512 203 | entry.st_blocks = 1 204 | entry.st_atime_ns = row['atime_ns'] 205 | entry.st_mtime_ns = row['mtime_ns'] 206 | entry.st_ctime_ns = row['ctime_ns'] 207 | 208 | return entry 209 | 210 | async def readlink(self, inode, ctx): 211 | return self.get_row('SELECT * FROM inodes WHERE id=?', (inode,))['target'] 212 | 213 | async def opendir(self, inode, ctx): 214 | return inode 215 | 216 | async def readdir(self, inode, off, token): 217 | if off == 0: 218 | off = -1 219 | 220 | cursor2 = self.db.cursor() 221 | cursor2.execute("SELECT * FROM contents WHERE parent_inode=? " 222 | 'AND rowid > ? ORDER BY rowid', (inode, off)) 223 | 224 | for row in cursor2: 225 | pyfuse3.readdir_reply( 226 | token, row['name'], await self.getattr(row['inode']), row['rowid']) 227 | 228 | async def unlink(self, inode_p, name,ctx): 229 | entry = await self.lookup(inode_p, name) 230 | 231 | if stat.S_ISDIR(entry.st_mode): 232 | raise pyfuse3.FUSEError(errno.EISDIR) 233 | 234 | self._remove(inode_p, name, entry) 235 | 236 | async def rmdir(self, inode_p, name, ctx): 237 | entry = await self.lookup(inode_p, name) 238 | 239 | if not stat.S_ISDIR(entry.st_mode): 240 | raise pyfuse3.FUSEError(errno.ENOTDIR) 241 | 242 | self._remove(inode_p, name, entry) 243 | 244 | def _remove(self, inode_p, name, entry): 245 | if self.get_row("SELECT COUNT(inode) FROM contents WHERE parent_inode=?", 246 | (entry.st_ino,))[0] > 0: 247 | raise pyfuse3.FUSEError(errno.ENOTEMPTY) 248 | 249 | if entry.st_nlink == 1 and entry.st_ino not in self.inode_open_count: 250 | # delete inode from db 251 | self.cursor.execute("DELETE FROM inodes WHERE id=?", (entry.st_ino,)) 252 | 253 | # delete message from Telegram 254 | self.delete_msgs_for_inode(entry.st_ino) 255 | 256 | # delete from contents 257 | self.cursor.execute("DELETE FROM contents WHERE name=? AND parent_inode=?", 258 | (name, inode_p)) 259 | 260 | # Delete from telegram_messages 261 | self.cursor.execute("DELETE FROM telegram_messages where inode=?", (entry.st_ino,)) 262 | 263 | if entry.st_nlink > 1 and entry.st_ino not in self.inode_open_count: 264 | # delete from contents 265 | self.cursor.execute("DELETE FROM contents WHERE name=? AND parent_inode=?", 266 | (name, inode_p)) 267 | 268 | self.db.commit() 269 | 270 | async def symlink(self, inode_p, name, target, ctx): 271 | mode = (stat.S_IFLNK | stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR | 272 | stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP | 273 | stat.S_IROTH | stat.S_IWOTH | stat.S_IXOTH) 274 | return await self._create(inode_p, name, mode, ctx, target=target) 275 | 276 | async def rename(self, inode_p_old, name_old, inode_p_new, name_new, 277 | flags, ctx): 278 | if flags != 0: 279 | raise FUSEError(errno.EINVAL) 280 | 281 | entry_old = await self.lookup(inode_p_old, name_old) 282 | 283 | try: 284 | entry_new = await self.lookup(inode_p_new, name_new) 285 | except pyfuse3.FUSEError as exc: 286 | if exc.errno != errno.ENOENT: 287 | raise 288 | target_exists = False 289 | else: 290 | target_exists = True 291 | 292 | if target_exists: 293 | self._replace(inode_p_old, name_old, inode_p_new, name_new, 294 | entry_old, entry_new) 295 | else: 296 | self.cursor.execute("UPDATE contents SET name=?, parent_inode=? WHERE name=? " 297 | "AND parent_inode=?", (name_new, inode_p_new, 298 | name_old, inode_p_old)) 299 | self.db.commit() 300 | 301 | def delete_msgs_for_inode(self, fh): 302 | # delete from Telegram 303 | rows = self.cursor.execute("SELECT id FROM telegram_messages WHERE inode = ?", (fh,)) 304 | ids = [r[0] for r in rows] 305 | self.client.delete_messages(ids) 306 | 307 | def _replace(self, inode_p_old, name_old, inode_p_new, name_new, 308 | entry_old, entry_new): 309 | 310 | if self.get_row("SELECT COUNT(inode) FROM contents WHERE parent_inode=?", 311 | (entry_new.st_ino,))[0] > 0: 312 | raise pyfuse3.FUSEError(errno.ENOTEMPTY) 313 | 314 | self.cursor.execute("UPDATE contents SET inode=? WHERE name=? AND parent_inode=?", 315 | (entry_old.st_ino, name_new, inode_p_new)) 316 | self.db.execute('DELETE FROM contents WHERE name=? AND parent_inode=?', 317 | (name_old, inode_p_old)) 318 | 319 | if entry_new.st_nlink == 1 and entry_new.st_ino not in self.inode_open_count: 320 | self.cursor.execute("DELETE FROM inodes WHERE id=?", (entry_new.st_ino,)) 321 | self.delete_msgs_for_inode(entry_new.st_ino) 322 | 323 | self.db.commit() 324 | 325 | 326 | async def link(self, inode, new_inode_p, new_name, ctx): 327 | entry_p = await self.getattr(new_inode_p) 328 | if entry_p.st_nlink == 0: 329 | log.warning('Attempted to create entry %s with unlinked parent %d', 330 | new_name, new_inode_p) 331 | raise FUSEError(errno.EINVAL) 332 | 333 | self.cursor.execute("INSERT INTO contents (name, inode, parent_inode) VALUES(?,?,?)", 334 | (new_name, inode, new_inode_p)) 335 | self.db.commit() 336 | return await self.getattr(inode) 337 | 338 | async def setattr(self, inode, attr, fields, fh, ctx): 339 | 340 | if fields.update_size: 341 | # get data from telegram 342 | data = await self.get_telegram_data(fh) 343 | if data is None: 344 | data = b'' 345 | if len(data) < attr.st_size: 346 | data = data + b'\0' * (attr.st_size - len(data)) 347 | else: 348 | data = data[:attr.st_size] 349 | self.cursor.execute('UPDATE inodes SET size=? WHERE id=?', 350 | (attr.st_size, inode)) 351 | if fields.update_mode: 352 | self.cursor.execute('UPDATE inodes SET mode=? WHERE id=?', 353 | (attr.st_mode, inode)) 354 | 355 | if fields.update_uid: 356 | self.cursor.execute('UPDATE inodes SET uid=? WHERE id=?', 357 | (attr.st_uid, inode)) 358 | 359 | if fields.update_gid: 360 | self.cursor.execute('UPDATE inodes SET gid=? WHERE id=?', 361 | (attr.st_gid, inode)) 362 | 363 | if fields.update_atime: 364 | self.cursor.execute('UPDATE inodes SET atime_ns=? WHERE id=?', 365 | (attr.st_atime_ns, inode)) 366 | 367 | if fields.update_mtime: 368 | self.cursor.execute('UPDATE inodes SET mtime_ns=? WHERE id=?', 369 | (attr.st_mtime_ns, inode)) 370 | 371 | if fields.update_ctime: 372 | self.cursor.execute('UPDATE inodes SET ctime_ns=? WHERE id=?', 373 | (attr.st_ctime_ns, inode)) 374 | else: 375 | self.cursor.execute('UPDATE inodes SET ctime_ns=? WHERE id=?', 376 | (int(time()*1e9), inode)) 377 | 378 | self.db.commit() 379 | return await self.getattr(inode) 380 | 381 | async def mknod(self, inode_p, name, mode, rdev, ctx): 382 | return await self._create(inode_p, name, mode, ctx, rdev=rdev) 383 | 384 | async def mkdir(self, inode_p, name, mode, ctx): 385 | return await self._create(inode_p, name, mode, ctx) 386 | 387 | async def statfs(self, ctx): 388 | stat_ = pyfuse3.StatvfsData() 389 | 390 | stat_.f_bsize = 512 391 | stat_.f_frsize = 512 392 | 393 | size = self.get_row('SELECT SUM(size) FROM inodes')[0] 394 | stat_.f_blocks = size // stat_.f_frsize 395 | stat_.f_bfree = max(size // stat_.f_frsize, 1024) 396 | stat_.f_bavail = stat_.f_bfree 397 | 398 | inodes = self.get_row('SELECT COUNT(id) FROM inodes')[0] 399 | stat_.f_files = inodes 400 | stat_.f_ffree = max(inodes , 100) 401 | stat_.f_favail = stat_.f_ffree 402 | 403 | return stat_ 404 | 405 | async def open(self, inode, flags, ctx): 406 | # Yeah, unused arguments 407 | #pylint: disable=W0613 408 | self.inode_open_count[inode] += 1 409 | 410 | # Use inodes as a file handles 411 | return pyfuse3.FileInfo(fh=inode) 412 | 413 | async def access(self, inode, mode, ctx): 414 | # Yeah, could be a function and has unused arguments 415 | #pylint: disable=R0201,W0613 416 | return True 417 | 418 | async def create(self, inode_parent, name, mode, flags, ctx): 419 | 420 | #pylint: disable=W0612 421 | entry = await self._create(inode_parent, name, mode, ctx) 422 | self.inode_open_count[entry.st_ino] += 1 423 | return (pyfuse3.FileInfo(fh=entry.st_ino), entry) 424 | 425 | async def _create(self, inode_p, name, mode, ctx, rdev=0, target=None): 426 | if (await self.getattr(inode_p)).st_nlink == 0: 427 | log.warning('Attempted to create entry %s with unlinked parent %d', 428 | name, inode_p) 429 | raise FUSEError(errno.EINVAL) 430 | 431 | now_ns = int(time() * 1e9) 432 | self.cursor.execute('INSERT INTO inodes (uid, gid, mode, mtime_ns, atime_ns, ' 433 | 'ctime_ns, target, rdev) VALUES(?, ?, ?, ?, ?, ?, ?, ?)', 434 | (ctx.uid, ctx.gid, mode, now_ns, now_ns, now_ns, target, rdev)) 435 | 436 | inode = self.cursor.lastrowid 437 | self.db.execute("INSERT INTO contents(name, inode, parent_inode) VALUES(?,?,?)", 438 | (name, inode, inode_p)) 439 | 440 | self.db.commit() 441 | return await self.getattr(inode) 442 | 443 | # helper function to get all data for a file from telegram 444 | async def get_telegram_data(self, fh): 445 | filebuf = self.client.get_cached_file(fh) 446 | if filebuf != None: 447 | return filebuf 448 | # CHECK if we have ANY messages for this inode 449 | try: 450 | # get telegram messages for inode 451 | msgIds = self.get_rows('SELECT * FROM telegram_messages WHERE inode=?', (fh,)) 452 | ids = [r[0] for r in msgIds] 453 | 454 | # FOR EACH message, call telegram API and get contents 455 | filebuf = self.client.download_file(fh, ids) 456 | return filebuf 457 | # if no rows, return empty bytes immediately 458 | except Exception as e: 459 | print("EXCEPTION: ", e) 460 | return bytearray(b'') 461 | 462 | 463 | async def read(self, fh, offset, length): 464 | row = self.get_row('SELECT * FROM inodes WHERE id=?', (fh,)) 465 | 466 | # if no data, don't query telegram 467 | if row is None: 468 | return b'' 469 | 470 | telegram_data = await self.get_telegram_data(fh) 471 | return telegram_data[offset:offset+length] 472 | 473 | # buffer in memory first... 474 | async def write(self, fh, offset, buf): 475 | # get data if exists 476 | result_bytes = self.write_buffer 477 | 478 | # if we are not already writing, try to get data from telegram 479 | if result_bytes == bytearray(b''): 480 | row = self.get_row('SELECT * FROM inodes WHERE id=?', (fh,)) 481 | if row != None: 482 | result_bytes = await self.get_telegram_data(fh) 483 | self.write_buffer = result_bytes 484 | if offset == len(result_bytes): 485 | self.write_buffer += buf 486 | else: 487 | self.write_buffer = result_bytes[:offset] + buf + result_bytes[offset+len(buf):] # this is kind of slow 488 | 489 | return len(buf) 490 | 491 | async def close(self, fh): 492 | pass 493 | 494 | async def fsync(self, fh): 495 | pass 496 | 497 | async def release(self, fh): 498 | self.inode_open_count[fh] -= 1 499 | # THIS is where we write data for real! 500 | # IF we have un-written data in the buffer for this fh, yeet it to discord/tgram. 501 | if len(self.write_buffer) > 0: 502 | data = self.write_buffer 503 | self.write_buffer = bytearray(b'') 504 | # self.write_buffer.pop(fh) 505 | print("CLEANED UP ", gc.collect()) 506 | filename = "" 507 | 508 | fname = self.get_row("SELECT name FROM contents WHERE inode=?", (fh,)) 509 | if fname != None: 510 | name = fname[0] 511 | filename = name.decode() 512 | # write data back to telegram 513 | fileBytes = BytesIO(data) 514 | telegram_msgs = self.client.upload_file(fileBytes, fh, filename) 515 | 516 | # clear existing from telegram 517 | self.delete_msgs_for_inode(fh) 518 | 519 | # clear any existing messages from DB 520 | self.cursor.execute("DELETE FROM telegram_messages WHERE inode = ?", (fh,)) 521 | 522 | # add new message ids back to DB 523 | for msg in telegram_msgs: 524 | self.cursor.execute("INSERT INTO telegram_messages (id, inode) VALUES (?, ?)", (msg.id, fh,)) 525 | 526 | # update inodes 527 | self.cursor.execute('UPDATE inodes SET size=? WHERE id=?', 528 | (len(data), fh)) 529 | self.db.commit() 530 | 531 | if self.inode_open_count[fh] == 0: 532 | del self.inode_open_count[fh] 533 | if (await self.getattr(fh)).st_nlink == 0: 534 | self.cursor.execute("DELETE FROM inodes WHERE id=?", (fh,)) 535 | self.db.commit() 536 | 537 | class NoUniqueValueError(Exception): 538 | def __str__(self): 539 | return 'Query generated more than 1 result row' 540 | 541 | 542 | class NoSuchRowError(Exception): 543 | def __str__(self): 544 | return 'Query produced 0 result rows' 545 | 546 | def init_logging(debug=False): 547 | formatter = logging.Formatter('%(asctime)s.%(msecs)03d %(threadName)s: ' 548 | '[%(name)s] %(message)s', datefmt="%Y-%m-%d %H:%M:%S") 549 | handler = logging.StreamHandler() 550 | handler.setFormatter(formatter) 551 | root_logger = logging.getLogger() 552 | if debug: 553 | handler.setLevel(logging.DEBUG) 554 | root_logger.setLevel(logging.DEBUG) 555 | else: 556 | handler.setLevel(logging.INFO) 557 | root_logger.setLevel(logging.INFO) 558 | root_logger.addHandler(handler) 559 | 560 | def parse_args(): 561 | '''Parse command line''' 562 | 563 | parser = ArgumentParser() 564 | 565 | parser.add_argument('mountpoint', type=str, 566 | help='Where to mount the file system', default="./telegramfs") 567 | parser.add_argument('--debug', action='store_true', default=False, 568 | help='Enable debugging output') 569 | parser.add_argument('--debug-fuse', action='store_true', default=False, 570 | help='Enable FUSE debugging output') 571 | 572 | return parser.parse_args() 573 | 574 | def runFs(client): 575 | try: 576 | options = parse_args() 577 | init_logging(options.debug) 578 | operations = Operations(client) 579 | 580 | fuse_options = set(pyfuse3.default_options) 581 | fuse_options.add('fsname=telegram_fuse') 582 | fuse_options.add("allow_other") 583 | fuse_options.discard('default_permissions') 584 | if options.debug_fuse: 585 | fuse_options.add('debug') 586 | 587 | pyfuse3.init(operations, options.mountpoint, fuse_options) 588 | 589 | # close db and unmount fs when program is closed 590 | def cleanup(): 591 | print("RUNNING CLEANUP") 592 | operations.cursor.close() 593 | operations.db.close() 594 | pyfuse3.close(unmount=True) 595 | 596 | atexit.register(cleanup) 597 | trio.run(pyfuse3.main) 598 | 599 | except: 600 | raise 601 | 602 | 603 | --------------------------------------------------------------------------------