├── LICENSE ├── README.md ├── calibre.py ├── formats.txt └── upload.bash /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 Geremia 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # LibgenBulkUpload 2 | bulk upload files to [https://library.bz/main/upload/](https://library.bz/main/upload/) 3 | 4 | ## Usage 5 | ```bash 6 | ./upload.bash file_to_upload title language_code main_or_fiction 7 | ``` 8 | ### Specifying additional metadata 9 | 10 | Example 11 | ```bash 12 | isbn=9783868386066 authors='Feser, Edward' ./upload.bash Immortal_Souls.pdf 'Immortal Souls: A Treatise on Human Nature' eng main 13 | ``` 14 | 15 | ### Batch with GNU `parallel` 16 | 17 | This [GNU `parallel`](https://www.gnu.org/software/parallel/) command uploads (one-at-a-time) all the PDFs in the current directory, assigns their Libgen metadata author to the PDF filename (sans suffix), and sets the language to English: 18 | ```bash 19 | parallel -j1 ./upload.bash {} {.} eng main ::: *.pdf 20 | ``` 21 | 22 | ### Metadata 23 | 24 | The following metadata can be optionally passed as an environment variable to the script: 25 | 26 | - `asin`, `authors`, `bookmarks`, `city`, `cleaned`, `colored`, `cover`, `ddc`, `description`, `doi`, `dpi`, `dpi_select`, `edition`, `file_commentary`, `file_source`, `file_source_issue`, `gb_id`, `isbn`, `issn`, `language`, `language_options`, `lbc`, `lcc`, `metadata_query`, `metadata_source`, `ol_id`, `page_orientation`, `pages`, `paginated`, `periodical`, `publisher`, `scan`, `series`, `searchable`, `tags`, `title`, `toc`, `topic`, `udc`, `volume`, `year` 27 | 28 | ## Calibre 29 | 30 | If you [export a CSV catalog from Calibre](https://manual.calibre-ebook.com/gui.html#catalogs), you can use it with `calibre.py` to feed in the metadata for bulk book uploads, automatically. 31 | 32 | Your Calibre catalog CSV file can have the following headings (in any order), including the custom ones (which begin with '#'): 33 | 34 | - `#amazon`, `#doi`, `#google`, `#issn`, `#lcn`, `#pubyear`, `authors`, `comments`, `isbn`, `languages`, `publisher`, `series`, `series_index`, `tags`, `title` 35 | 36 | The specified "files to upload" directory is expected to contain symlinks to the files in your Calibre library. 37 | 38 | To setup the symlinks, execute a command like this in your upload directory: 39 | ```bash 40 | for i in `cat ../formats.txt`; 41 | do 42 | find ~/Calibre\ Library/ -type f -iname "*.$i" -print0 | \ 43 | xargs -0 -I{} sh -c 'filename=`basename "{}"`; 44 | random_part=$(mktemp -u XXXXXX); 45 | suffix=".${filename##*.}"; 46 | newname="${filename%.*}-${random_part}${suffix}"; 47 | ln -sv "{}" "$newname";' 48 | done 49 | ``` 50 | 51 | `calibre.py` is a similar script to the old `upload.py` Selenium script. 52 | -------------------------------------------------------------------------------- /calibre.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python3 -u 2 | 3 | import csv 4 | import os 5 | import pathlib as pl 6 | import re 7 | import subprocess as sp 8 | import sys 9 | import urllib.parse as up 10 | 11 | if len(sys.argv) != 5: 12 | print("""4 args required: 13 | relative path of directory of 14 | (1) files to upload 15 | (2) uploaded files 16 | (3) rejected files 17 | plus: (4) Calibre catalog csv file""") 18 | exit(1) 19 | 20 | upload_dir = './'+sys.argv[1]+'/' 21 | uploaded_dir = './'+sys.argv[2]+'/' 22 | rejects_dir = './'+sys.argv[3]+'/' 23 | 24 | print("Specified directories:") 25 | for i in [upload_dir, uploaded_dir, rejects_dir]: 26 | print(i) 27 | if not os.path.isdir(i): 28 | os.mkdir(i) 29 | print() 30 | 31 | def sortKey(filename): 32 | return os.path.getsize(upload_dir+filename) 33 | 34 | files = os.listdir(upload_dir) 35 | files = sorted(files, key=sortKey) 36 | if len(files) == 0: 37 | print("No books to upload.") 38 | sys.exit(1) 39 | 40 | # Process Calibre catalog CSV 41 | nested_list = [] 42 | catalog = sys.argv[4] 43 | with open(catalog, mode='r', newline='') as file: 44 | csv_reader = csv.reader(file) 45 | for row in csv_reader: 46 | nested_list.append(row) 47 | 48 | book_catalog = [] 49 | with open('Formats.csv', mode='r', newline='', encoding='utf-8-sig') as file: 50 | csv_reader = csv.reader(file) 51 | for row in csv_reader: 52 | book_catalog.append(row) 53 | header = book_catalog[0] 54 | 55 | # in Calibre catalog export 56 | calibre_metadata = [ 57 | '#amazon', 58 | '#doi', 59 | '#google', 60 | '#issn', 61 | '#lcn', 62 | '#pubyear', 63 | 'authors', 64 | 'comments', 65 | 'isbn', 66 | 'languages', 67 | 'publisher', 68 | 'series', 69 | 'series_index', 70 | 'tags', 71 | 'title' 72 | ] 73 | 74 | # check that at least 'id' header exists 75 | if 'id' not in header: 76 | print("Calibre catalog CSV file must have an 'id' column.") 77 | sys.exit(1) 78 | 79 | # corresponding Libgen metadata 80 | libgen_metadata = [ 81 | 'asin', 82 | 'doi', 83 | 'gb_id', 84 | 'issn', 85 | 'lcc', 86 | 'year', 87 | 'authors', 88 | 'description', 89 | 'isbn', 90 | 'language', 91 | 'publisher', 92 | 'series', 93 | 'volume', 94 | 'tags', 95 | 'title' 96 | ] 97 | 98 | # Upload books 99 | for f in files: 100 | upload_dir_f = upload_dir + f 101 | pat = pl.Path(upload_dir_f).absolute() 102 | if pat.is_symlink(): 103 | pat = pat.resolve() 104 | parent = pat.parent.parts[-1] 105 | 106 | try: 107 | id = re.findall('[0-9]+', parent)[-1] 108 | except: 109 | print("No valid Calibre id found for '" + f 110 | + "' Be sure your uploads directory contains only symlinks to the files in your Calibre library.") 111 | sys.exit(1) 112 | 113 | try: 114 | book_entry = [item for item in book_catalog if item[header.index('id')] == str(id)][0] 115 | except: 116 | print(id, "wasn't found in Calibre catalog CSV file '" + catalog + "'. Skipping.") 117 | continue 118 | 119 | env = {} 120 | for i in range(len(calibre_metadata)): 121 | try: 122 | idx = header.index(calibre_metadata[i]) 123 | except: 124 | continue 125 | val = up.quote(book_entry[idx]) 126 | if val: 127 | env[libgen_metadata[i]] = val 128 | 129 | title = up.quote(book_entry[header.index('title')]) 130 | lang = up.quote(book_entry[header.index('languages')]) 131 | process = sp.Popen(['./upload.bash', upload_dir_f, title, lang, 'main'], env=env) 132 | process.communicate() 133 | rc = process.returncode 134 | if rc == 0: # success 135 | print("Moving '" + f + "' to '" + uploaded_dir + "'.") 136 | os.rename(upload_dir_f, uploaded_dir + f) 137 | else: # failure 138 | print("Moving '" + f + "' to '" + rejects_dir + "'.") 139 | os.rename(upload_dir_f, rejects_dir + f) 140 | 141 | -------------------------------------------------------------------------------- /formats.txt: -------------------------------------------------------------------------------- 1 | 7z 2 | azw 3 | azw3 4 | azw4 5 | cb7 6 | cbr 7 | cbz 8 | chm 9 | djvu 10 | doc 11 | docx 12 | epub 13 | fb2 14 | mobi 15 | pdf 16 | rar 17 | rtf 18 | zip 19 | -------------------------------------------------------------------------------- /upload.bash: -------------------------------------------------------------------------------- 1 | #!/usr/bin/bash 2 | 3 | file=${1:?File to upload. Allowed formats: PDF, DJVU, EPUB, MOBI, AZW, AZW3, AZW4, FB2, CHM, RTF, DOC, DOCX, ZIP, RAR, 7Z, CBZ, CBR, CB7.} 4 | title=${2:?Title} 5 | language=${3:?Language} 6 | mainOrFiction=${4:?Upload to \'main\' or \'fiction\'?} 7 | md5sum=`md5sum -z "$file" | sed 's/ .*//'` 8 | upload_url="https://library.bz/$mainOrFiction/upload/" 9 | data_entry_url="https://library.bz/$mainOrFiction/uploads/new/$md5sum" 10 | done_url="https://library.bz/$mainOrFiction/uploads/$md5sum" 11 | 12 | # All optional and required variables 13 | vars=( "metadata_source" "metadata_query" "title" "volume" "authors" "language" "language_options" "edition" "series" "pages" "year" "publisher" "city" "periodical" "isbn" "issn" "doi" "gb_id" "asin" "ol_id" "ddc" "lcc" "udc" "lbc" "topic" "tags" "cover" "description" "toc" "scan" "dpi" "dpi_select" "sfearchable" "paginated" "page_orientation" "colored" "cleaned" "bookmarks" "file_source" "file_source_issue" "file_commentary" ) 14 | 15 | # Initialize default values 16 | metadata_source=${metadata_source:-local} 17 | cover=${cover:-$md5sum-g.jpg} 18 | sfearchable=${sfearchable:-1} 19 | 20 | # POST only non-null values 21 | data_raw='' 22 | pretty_data_str='' 23 | for var_name in "${vars[@]}"; do 24 | value="${!var_name}" 25 | if [ -n "$value" ]; then 26 | data_raw+="$var_name=$value&" 27 | pretty_data_str+="\t$var_name: $value\n" 28 | fi 29 | done 30 | data_raw="${data_raw%'&'}" 31 | pretty_data_str="${pretty_data_str%'\n'}" 32 | 33 | echo -e "Uploading '$file' to $upload_url…" 34 | curl "$upload_url" -X POST \ 35 | -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0' \ 36 | -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' \ 37 | -H 'Accept-Language: en-US,en;q=0.5' \ 38 | -H 'Accept-Encoding: gzip, deflate, br, zstd' \ 39 | -H 'Content-Type: multipart/form-data' \ 40 | -H 'Origin: https://library.bz' \ 41 | -H 'Authorization: Basic Z2VuZXNpczp1cGxvYWQ=' \ 42 | -H 'Connection: keep-alive' \ 43 | -H 'Referer: https://library.bz/main/upload/' \ 44 | -H 'Upgrade-Insecure-Requests: 1' \ 45 | -H 'Sec-Fetch-Dest: document' \ 46 | -H 'Sec-Fetch-Mode: navigate' \ 47 | -H 'Sec-Fetch-Site: same-origin' \ 48 | -H 'Sec-Fetch-User: ?1' \ 49 | -H 'Priority: u=1' \ 50 | -F "file=@\"$file\"" &> /dev/null 51 | 52 | echo -e "Entering data\n$pretty_data_str\nat $data_entry_url…" 53 | curl -L "$data_entry_url" --compressed -X POST \ 54 | -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0' \ 55 | -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8' \ 56 | -H 'Accept-Language: en-US,en;q=0.5' \ 57 | -H 'Accept-Encoding: gzip, deflate, br, zstd' \ 58 | -H 'Content-Type: application/x-www-form-urlencoded' \ 59 | -H 'Origin: https://library.bz' \ 60 | -H 'Authorization: Basic Z2VuZXNpczp1cGxvYWQ=' \ 61 | -H 'Connection: keep-alive' \ 62 | -H "Referer: $data_entry_url" \ 63 | -H 'Upgrade-Insecure-Requests: 1' \ 64 | -H 'Sec-Fetch-Dest: document' \ 65 | -H 'Sec-Fetch-Mode: navigate' \ 66 | -H 'Sec-Fetch-Site: same-origin' \ 67 | -H 'Sec-Fetch-User: ?1' \ 68 | -H 'Priority: u=1' --data-raw "$data_raw" 2>&1 \ 69 | | grep -zo "The record has been successfully saved." && echo " $done_url" && exit 0 70 | 71 | echo -e "Problem with upload (or already uploaded)." && exit 1 72 | 73 | --------------------------------------------------------------------------------