├── .github └── CODEOWNERS ├── .gitignore ├── LICENSE ├── README.md ├── howtogetthedirectlinks.md ├── images ├── anonfiles.png ├── batchlinks_logo.png ├── cancel.jpg ├── checkconsole1.jpg ├── checkconsole2.jpg ├── civit1.jpg ├── civit2.jpg ├── civitnew1.jpg ├── civitnew2.jpg ├── discord1.jpg ├── discord2.jpg ├── downloaded.jpg ├── drive1.png ├── drive2.png ├── dropbox1.png ├── dropbox2.png ├── example.jpg ├── ext_installer.jpg ├── github.jpg ├── githubdirect1.png ├── githubdirect2.png ├── githubdirect3.png ├── githubrelease1.png ├── githubrelease2.png ├── gradiooff1.jpg ├── gradiooff2.jpg ├── huggingface1.jpg ├── huggingface2.jpg ├── logging.jpg ├── mediafire.png ├── mega1.jpg ├── mega2.jpg ├── pastebin.png ├── pixeldrain.png ├── progress.jpg ├── python.png ├── queue.jpg ├── rename.jpg ├── sdless.png ├── shell1.jpg └── shell2.jpg ├── install.py ├── javascript └── notification.js ├── misc └── markydynamictohtml.py ├── notification.mp3 ├── regardingthisrepo.md ├── releasenotes.md ├── scripts ├── a_gradiocheck.py └── batchlinks-downloader.py ├── sdless-macos.sh ├── sdless-windows-debug.bat ├── sdless-windows.bat └── version.txt /.github/CODEOWNERS: -------------------------------------------------------------------------------- 1 | # Global rule: 2 | * @etherealxx 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.bak 2 | *.backup 3 | imageratio.* 4 | *.lnk 5 | tempCodeRunnerFile.python 6 | .DS_Store 7 | gradiovenv/ 8 | 9 | # Byte-compiled / optimized / DLL files 10 | __pycache__/ 11 | *.py[cod] 12 | *$py.class 13 | 14 | # C extensions 15 | *.so 16 | 17 | # Distribution / packaging 18 | .Python 19 | build/ 20 | develop-eggs/ 21 | dist/ 22 | downloads/ 23 | eggs/ 24 | .eggs/ 25 | lib/ 26 | lib64/ 27 | parts/ 28 | sdist/ 29 | var/ 30 | wheels/ 31 | share/python-wheels/ 32 | *.egg-info/ 33 | .installed.cfg 34 | *.egg 35 | MANIFEST 36 | 37 | # PyInstaller 38 | # Usually these files are written by a python script from a template 39 | # before PyInstaller builds the exe, so as to inject date/other infos into it. 40 | *.manifest 41 | *.spec 42 | 43 | # Installer logs 44 | pip-log.txt 45 | pip-delete-this-directory.txt 46 | 47 | # Unit test / coverage reports 48 | htmlcov/ 49 | .tox/ 50 | .nox/ 51 | .coverage 52 | .coverage.* 53 | .cache 54 | nosetests.xml 55 | coverage.xml 56 | *.cover 57 | *.py,cover 58 | .hypothesis/ 59 | .pytest_cache/ 60 | cover/ 61 | 62 | # Translations 63 | *.mo 64 | *.pot 65 | 66 | # Django stuff: 67 | *.log 68 | local_settings.py 69 | db.sqlite3 70 | db.sqlite3-journal 71 | 72 | # Flask stuff: 73 | instance/ 74 | .webassets-cache 75 | 76 | # Scrapy stuff: 77 | .scrapy 78 | 79 | # Sphinx documentation 80 | docs/_build/ 81 | 82 | # PyBuilder 83 | .pybuilder/ 84 | target/ 85 | 86 | # Jupyter Notebook 87 | .ipynb_checkpoints 88 | 89 | # IPython 90 | profile_default/ 91 | ipython_config.py 92 | 93 | # pyenv 94 | # For a library or package, you might want to ignore these files since the code is 95 | # intended to run in multiple environments; otherwise, check them in: 96 | # .python-version 97 | 98 | # pipenv 99 | # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. 100 | # However, in case of collaboration, if having platform-specific dependencies or dependencies 101 | # having no cross-platform support, pipenv may install dependencies that don't work, or not 102 | # install all needed dependencies. 103 | #Pipfile.lock 104 | 105 | # poetry 106 | # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. 107 | # This is especially recommended for binary packages to ensure reproducibility, and is more 108 | # commonly ignored for libraries. 109 | # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control 110 | #poetry.lock 111 | 112 | # pdm 113 | # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. 114 | #pdm.lock 115 | # pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it 116 | # in version control. 117 | # https://pdm.fming.dev/#use-with-ide 118 | .pdm.toml 119 | 120 | # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm 121 | __pypackages__/ 122 | 123 | # Celery stuff 124 | celerybeat-schedule 125 | celerybeat.pid 126 | 127 | # SageMath parsed files 128 | *.sage.py 129 | 130 | # Environments 131 | .env 132 | .venv 133 | env/ 134 | venv/ 135 | ENV/ 136 | env.bak/ 137 | venv.bak/ 138 | 139 | # Spyder project settings 140 | .spyderproject 141 | .spyproject 142 | 143 | # Rope project settings 144 | .ropeproject 145 | 146 | # mkdocs documentation 147 | /site 148 | 149 | # mypy 150 | .mypy_cache/ 151 | .dmypy.json 152 | dmypy.json 153 | 154 | # Pyre type checker 155 | .pyre/ 156 | 157 | # pytype static type analyzer 158 | .pytype/ 159 | 160 | # Cython debug symbols 161 | cython_debug/ 162 | 163 | # PyCharm 164 | # JetBrains specific template is maintained in a separate JetBrains.gitignore that can 165 | # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore 166 | # and can be added to the global gitignore or merged into this file. For a more nuclear 167 | # option (not recommended) you can uncomment the following to ignore the entire idea folder. 168 | #.idea/ 169 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Etherealxx 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 |
2 | 3 |
4 | 5 | Logo 6 | 7 | 8 |

BatchLinks Downloader

9 | 10 |

11 | Batch-downloading models and stuff in stable-diffusion-webui colab made simple and fast. 12 |
13 | 14 |
15 | Report Bug 16 | · 17 | Request Feature 18 |

19 | 20 |
21 | 22 | 23 | 58 | 59 | 60 |
61 | Table of Contents 62 |
    63 |
  1. Installation
    2. 64 |
  2. About
    3. 65 |
  3. Example
    4. 66 |
  4. Syntax
    5. 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 |
  5. Additional Syntax
    6. 75 | 76 | 77 | 78 | 79 | 80 |
  6. Gradio Queue
    7. 81 | 82 | 83 | 84 | 85 |
  7. Other Features
    8. 86 | 87 | 88 | 89 | 90 | 91 |
  8. Release Notes
    9. 92 |
  9. Roadmap
    10. 93 |
  10. Known Bugs
    11. 94 | 95 |
  11. Contributing
    12. 96 |
  12. Contact
    13. 97 |
  13. Acknowledgments
    14. 98 | 99 |
100 |
101 | 102 | [does this repo still maintained?](https://github.com/etherealxx/batchlinks-webui/blob/main/regardingthisrepo.md) 103 | 104 | ## Installation 105 | 106 | Copy this line into your colab installation cell. Or into a new cell if you already launched the webui. 107 | 108 | ``` 109 | !git clone https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webui 110 | ``` 111 | 112 | or, you can copy the url of this repo and install it via webui and restart the UI.
113 | Logo
114 | (If `gradio no interface is running` or `bad gateway` shows up when restarting the UI, that means you need to restart the cell anyway 😅) 115 | 116 | 123 | 124 | Using `--gradio-queue` on the launch.py argument is highly recommended, as it enables this extension to show download progress bar on the UI and a cancel button. The option itself has no negative effect on the webui. [Read more here.](https://github.com/etherealxx/batchlinks-webui#gradio-queue)
125 | Logo
126 | 127 | You can also run this extension in [SDless mode](https://github.com/etherealxx/batchlinks-webui#sdless-mode) btw. 128 | 129 | While it's not recommended to use this extension on your local installation, you can use this extension on Windows. [More here.](https://github.com/etherealxx/batchlinks-webui#local-installation-support) 130 | 131 | ## About 132 | 133 | This extension will streamline your downloads on your [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui) colab session. Paste the links you need to download (or you can upload a txt file containing the links), use the hashstag syntax to choose the download location (see below), and hit the `Download All` button to download them! 134 | 135 | ## Example 136 | 137 | Look at this example
138 | Logo
139 | 140 | ``` 141 | #model 142 | https://huggingface.co/WarriorMama777/OrangeMixs/resolve/main/Models/AbyssOrangeMix3/AOM3_orangemixs.safetensors 143 | #vae 144 | https://huggingface.co/hakurei/waifu-diffusion-v1-4/resolve/main/vae/kl-f8-anime2.ckpt 145 | #embed 146 | https://huggingface.co/datasets/Nerfgun3/bad_prompt/resolve/main/bad_prompt_version2.pt 147 | https://huggingface.co/etherealxx/whoseisthis/resolve/main/bad_artist.pt 148 | #lora 149 | https://civitai.com/api/download/models/8840 ##1-mb-lora-trained-in-5-mins-that-does-the-same-thing-as-25-gb-model-but-better 150 | https://civitai.com/api/download/models/6891 ##sans undertale 151 | https://mega.nz/file/gAxTWBAI#uL7EZay-OND5G6ELJlGfUNG0s7Q4TynZKqFdvs0v0tc ##agent8-finetune 152 | https://mega.nz/file/oU43FSTY#vwAfsAb7RKJ4xtsSP7uzrKpWhh1y8BdpIBFurwsVP2o ##agent8-dreambooth 153 | #hypernet 154 | https://cdn.discordapp.com/attachments/1070489470127841381/1070489471964954684/MomopocoV3.pt 155 | ``` 156 | 157 | This piece of lines will be read from top to bottom. Every hashtag, it will change the current output directory to said directory (see below). So what this example do is it will download `AOM3` model to the model folder, then it will download the vae and put it to the Vae folder. Next it will download two embed, `bad prompt` and `bad artist`. Next it will download several LoRAs from CivitAI and MEGA, and put it to the Lora folder. Lastly, it changes the directory to hypernet directory, then 158 | 159 | You can also copy that example and paste it to a `.txt` file to use later. You can load a `.txt` file containing that piece of lines directly from the UI. 160 | 161 | When the items is downloading, you can inspect the running code on the colab cell, or just take a coffee and chill☕. If you activate logging, you can inspect the download progress from the UI, more [here](https://github.com/etherealxx/batchlinks-webui#logging) 162 | 163 | When the download is complete, the downloaded file will be listed
164 | Logo
165 | 166 | ## Syntax 167 | 168 | #### Hashtag 169 | - Hashtag means change current output directory to this directory. `#model` means every links below this hashtag, will be downloaded to _/content/stable-diffusion-webui/models/Stable-diffusion_, until it hits another hashtag, which will change the output directory again. See [below](https://github.com/etherealxx/batchlinks-webui#valid-hashtags) for valid hashtags. 170 | 171 | Note: If you use some colab that purposefully uses `sd-webui-additional-networks` extension to load Lora, use `#addnetlora` instead of `#lora`. It will download the lora to where it supposed to be. 172 | 173 | #### Links 174 | - Links are the main things you wants to be downloaded. Current supported links are from: 175 | - Huggingface (https://huggingface.co/) 176 | - MEGA (https://mega.nz/) {using `mega-cmd`} 177 | - CivitAI (https://civitai.com/) 178 | - Discord attachments (https://cdn.discordapp.com/attachments/) 179 | - catbox (https://files.catbox.moe) 180 | - Github (https://github.com or https://raw.githubusercontent.com) 181 | - Google Drive (https://drive.google.com) {using `gdown`} 182 | - Pixeldrain (https://pixeldrain.com/u/) 183 | - Mediafire (https://www.mediafire.com/file) {using `mediafire-dl`} 184 | - anonfiles (https://anonfiles.com) 185 | 186 | Every links other than that will be ignored. Keep in mind the only supported links are direct download links (see [here](https://github.com/etherealxx/batchlinks-downloader/blob/main/howtogetthedirectlinks.md)). For Huggingface, Civitai (model page link method), Discord attachments, catbox, pixeldrain, and anonfiles, there will be four different method of downloading offered (see [below](https://github.com/etherealxx/batchlinks-webui#huggingfaces-download-method)). For MEGA, it will use `mega-cmd` to download. For Github, if, the link is a raw file, it will download the file. Else, it will use `git clone`, useful to clone extension repo into the webui extension folder. 187 | 188 | More about CivitAI download method [here](https://github.com/etherealxx/batchlinks-webui#civitais-download-method). 189 | 190 | #### Double Hashtag 191 | - Double hashtag means comment. You can put double hashtag in the same line of the link and it will be ignored (keep in mind to put the link first then the double hashtag) 192 | 193 | #### Others 194 | - Other texts will be ignored. 195 | 196 | ### Valid Hashtags 197 | 198 | `#model`, `#models`, `#checkpoint`, or `#checkpoints` will put the downloaded file to _/content/stable-diffusion-webui/models/Stable-diffusion_ 199 | 200 | `#embedding`, `#embeddings`, `#embed`, `#embeds`, `#textualinversion`, or `#ti` will put the downloaded file to _/content/stable-diffusion-webui/embeddings_ 201 | 202 | `#vae` or `#vaes` will put the downloaded file to _/content/stable-diffusion-webui/models/VAE_ 203 | 204 | `#hypernetwork`, `#hypernetworks`, `#hypernet`, `#hypernets`, `#hynet`, or `#hynets` will put the downloaded file to _/content/stable-diffusion-webui/models/hypernetworks_ 205 | 206 | `#lora` or `#loras` will put the downloaded file to _/content/stable-diffusion-webui/models/Lora_ 207 | 208 | `#addnetlora`, `#loraaddnet`, `#additionalnetworks`, or `#addnet` will put the downloaded file to _/content/stable-diffusion-webui/extensions/sd-webui-additional-networks/models/lora_ 209 | 210 | `#aestheticembedding` or `#aestheticembed` will put the downloaded file to _content/stable-diffusion-webui/extensions/stable-diffusion-webui-aesthetic-gradients/aesthetic_embeddings_ 211 | 212 | `#controlnet` or `#cnet` will put the downloaded file to _/content/stable-diffusion-webui/extensions/sd-webui-controlnet/models_ 213 | 214 | `#upscale` or `#upscaler` will put the downloaded file to _/content/stable-diffusion-webui/models/ESRGAN_ 215 | 216 | `#lycoris`, `#locon`, or `#loha` will put the downloaded file to _/content/stable-diffusion-webui/extensions/sd-webui-additional-networks/models/lora/lycoris_ 217 | 218 | `#altmodel` or `#altmodels` will put the downloaded file to the path you choose when using `--ckptdir` argument on `launch.py` line. If you didn't use that argument, this hashtag will points to the same directory as `#model` 219 | 220 | _Lycoris/Locon/Loha_ will works just fine if you use `#addnetlora` instead, as long as you have both [addnet extension](https://github.com/kohya-ss/sd-webui-additional-networks) and [locon extension](https://github.com/KohakuBlueleaf/a1111-sd-webui-locon) installed. It's unclear if it's can be loaded by native auto1111 lora. (Tell me if you know more about this.) 221 | 222 | Github links (if it does not contain `/raw/` or `/release/download/` in it) doesn't need hashtag. It will always considered as webui extension, and the repository will be cloned to _/content/stable-diffusion-webui/extensions/(reponame)_ 223 | 224 | ### How to get the direct links (Important!) 225 | 226 | See [here](https://github.com/etherealxx/batchlinks-downloader/blob/main/howtogetthedirectlinks.md) 227 | 228 | ### Huggingface's download method 229 | 230 | So there's four supported method: `gdown`, `wget`, `curl` and `aria2`. Use whatever, really. The difference between them are actually little. Myself love using `gdown` since the output is cleaner than the others. `aria2` has the fastest download speed though. 231 | 232 | 253 | 254 | ## Additional Syntax 255 | ### Rename Downloaded Files 256 | Using `>` symbol, you can rename files. Take this for example
257 | Log
258 | Mostly catbox file has random name, by using `>` symbol after the link, you can type the desired name on the right. (Don't forget the extension) 259 | 260 | ### Running Shell Commands 261 | You can run shell commands by using `!` in front of the command you want, just like in google colab cells. Then press the `Download All!` button. (Sure, it doesn't download anything, but, well😅)
262 | Log 263 | Log
264 | You can run many lines at once too! 265 | 266 | ### Extract Everyting (`@extract`) 267 | You can use `@extract` to extract every `*.7z`, `*.rar`, and `*.zip` on current directory). 268 | ``` 269 | #lora 270 | 271 | @extract 272 | #embed 273 | ``` 274 | This will extract everything on Lora folder, since the code runs from top to bottom, and when the `@extract` executes, the current directory is still on Lora folder. 275 | 276 | ### Custom Hashtag Path (`@new`) 277 | You can use `@new ` to make a new usable hashtag and assign a directory path to it.
For example, you type this and press the `Download All!` button: 278 | ``` 279 | @new #private /content/stable-diffusion-webui/outputs 280 | #private 281 | 282 | ``` 283 | This will make a new hashtag `#private`, make it points to _/content/stable-diffusion-webui/outputs_, set `#private` as current directory, then download whatever into it. You can check if the new hashtag is assigned successfully by looking at the hashtag table on the right/bottom of the screen. 284 | 285 | ### Custom Direct Link Download with `aria2` (`@aria2`) 286 | If there are other direct links that isn't supported by Batchlinks, you can use `@aria` or `@aria2` custom command to download it. The main syntax is: 287 | ``` 288 | @aria2 {link} {path/hashtag} > {filename} 289 | ``` 290 | You can change `@aria2` with `@aria`, both works. Both the `path/hashtag` and `filename` is optional. This command was made to avoid hassle writing this: 291 | ``` 292 | aria2c --summary-interval=1 --console-log-level=error -c -x 16 -s 16 -k 1M {link} -d {path} -o {filename} 293 | ``` 294 | There are several ways to use it. Here i use a [20MB download test file](http://212.183.159.230/20MB.zip) as an example. 295 | ``` 296 | @aria2 http://212.183.159.230/20MB.zip 297 | ``` 298 | ☝️ This will download the file into the current directory (where the hashtags points to). So if before that line you use `#vae`, it will be downloaded to the vae directory. If there are no hashtags before, it'll be sent to the default `#model` directory. The filename will be the last part of the link (`512MB.zip`). 299 | ``` 300 | @aria2 http://212.183.159.230/20MB.zip #vae 301 | ``` 302 | ☝️ This will download the file into where `#vae` points to (which is the vae directory). You can use custom hashtag here as well. 303 | ``` 304 | @aria2 http://212.183.159.230/20MB.zip /content/stable-diffusion-webui 305 | ``` 306 | ☝️ This will download the file into the written path, which is _/content/stable-diffusion-webui_. Make sure it is a folder path, not a file path. 307 | ``` 308 | @aria2 http://212.183.159.230/20MB.zip /content/stable-diffusion-webui > 20megabytes.zip 309 | ``` 310 | ☝️ This will download the file into the written path and rename the file into `20megabytes.zip`. Remember to make sure to write the extension too. 311 | 312 | ## Gradio Queue 313 | 314 | If you use --gradio-queue argument on `launch.py`, some feature will be activated. 315 | 316 | ### Logging 317 | 318 | 319 | Enable logging by pressing the `Turn On Logging` radio button, and wait till `Logging activated` shows up and the box is blinking with orange border. Logging will tell you what are you actually downloading right now on the webui. After your download session is completed, it's recommended to turn back off the feature.
320 | Log
321 | 322 | ### Cancel 323 | Pressing `cancel` button while download in progress will stops the current session. Useful when at one time the download speeds is too slow. If you're currently downloading a single item, that item will be cancelled, but the other downloaded one will remain intact.
324 | Log
325 | 326 | ### Progress Bar 327 | There will be an additional progress bar that tells you the current activities.
328 | Log
329 | 330 | ### Changes if `--gradio-queue` is off 331 | Feature listed above will dissapear, and your only option for download is just `aria2` (speed is priority).
Note that when you pressed the `Download All!` button, **nothing will shows up on the UI**. You need to check the colab console.
332 | checkoff
333 | Another thing to note is your download session will always be cutted every 70 seconds (to prevent hangs/desync).
334 | off
335 | Don't worry, you can continue your session by pressing the `Resume Download` button. It will refresh the links with the one you haven't downloaded yet, then pressing `Download All!` will download the remaining links.
336 | off
337 | 338 | ## Other Features 339 | 340 | ### Notification 341 | 342 | This extension will play sound effect when the download process is completed. It's somewhat buggy at the moment though. 343 | 344 | ### Copy From Pastebin 345 | 346 | You can put your links on pastebin (write it the way you wrote on Batchlinks), put the pastebin link (https://pastebin.com/xxxx) on the link input, and press the 'Copy from Pastebin' button. It will copy all the the links on the pastebin to the UI. Useful to avoid hassle copy and pasting a bunch of text while on mobile.
347 | off
348 | 349 | ### SDless mode 350 | This mode will run the extension without the need of stable-diffusion-webui. Good for my own purpose of debugging it😁.
351 | off
352 | 353 | Copy-paste this on a new colab cell then run it: 354 | ``` 355 | from IPython.display import clear_output 356 | !pip3 show virtualenv >/dev/null || pip install -q virtualenv 357 | ![ -d gradiovenv ] || virtualenv gradiovenv 358 | !git clone https://github.com/etherealxx/batchlinks-webui \ 359 | /content/stable-diffusion-webui/extensions/batchlinks-webui 360 | pip3 show gradio >/dev/null || pip install -q gradio==3.16.2; \ 361 | pip3 show tqdm >/dev/null || pip3 install -q tqdm 362 | clear_output(wait=True) 363 | !source gradiovenv/bin/activate; \ 364 | python /content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py 365 | ``` 366 | or here's the quicker version without venv: 367 | ``` 368 | !pip install gradio==3.16.2 369 | !git clone https://github.com/etherealxx/batchlinks-webui /content/stable-diffusion-webui/extensions/batchlinks-webui 370 | !python /content/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py 371 | ``` 372 | 373 | For Windows, run the sdless-windows.bat. Make sure you have `python3` and `gradio` package version 3.16.2 or above. It won't use venv. 374 | 375 | This one for MacOS. Tested on Mojave. Create a shell script and give execute access to it. 376 | ``` 377 | #!/usr/bin/env bash 378 | pip3 show virtualenv >/dev/null || pip3 install virtualenv 379 | [ -d gradiovenv ] || virtualenv gradiovenv 380 | git clone https://github.com/etherealxx/batchlinks-webui \ 381 | $HOME/Downloads/stable-diffusion-webui/extensions/batchlinks-webui 382 | source gradiovenv/bin/activate; \ 383 | pip3 show gradio >/dev/null || pip3 install gradio==3.16.2; \ 384 | pip3 show tqdm >/dev/null || pip3 install -q tqdm 385 | source gradiovenv/bin/activate; \ 386 | python3 $HOME/Downloads/stable-diffusion-webui/extensions/batchlinks-webui/scripts/batchlinks-downloader.py 387 | ``` 388 | 389 | ### Local Installation Support 390 | 391 | This extension was tested to work on Windows 11. But ultimately, the number one priority for me is colab use, so there's might be bugs on Windows 11 one. Maybe also works on Debian-based linux (but you better inspect the source code first). 392 | On Windows, this extension will install [MEGAcmd](https://github.com/meganz/MEGAcmd) for MEGA file download, [wget-windows](https://github.com/webfolderio/wget-windows) for download using `wget`, [aria2-static-builds](https://github.com/q3aql/aria2-static-builds) for download using `aria2`, [7zr and 7z](https://www.7-zip.org) for extracting `aria2` installation, and for `@extract` command. 393 | MacOS is just partially supported. 394 | 395 | Also remember, in order for this extension to work properly, you need make sure that you tick that `add Python 3.10 to PATH` checkbox when you installed Python (Which is already recommended when installing stable-diffusion-webui)
396 | python
397 | 398 | ### Debug Mode (Developer only) 399 | 400 | By manually switching `globaldebug = False` to `True` on the source code, or using `--debug` argument when running the sdless mode, the Debug Mode will be activated. It shows every `print` calls i use to track variables. It will also tracks every files on the models (etc.) paths into a `snapshot.txt` file (if there isn't one). 401 | 402 | There are some batchlinks syntax features that only available on debug mode (putting it on the textbox and click `Download All!`): 403 | 404 | `@debugresetdownload` - This command will check every model (etc.) directory, and remove every file that isn't on `snapshot.txt`. Useful for me to removes file quickly when testing. **Be careful using this on your local installation.** 405 | 406 | `@debugresetdownload` - This command is (supposed to) download a single link with every method available (gdown, curl, wget, aria2), but now i rarely use this command, and haven't updated since. Might be buggy. 407 | 408 | ## Release Notes 409 | Moved [here](https://github.com/etherealxx/batchlinks-webui/blob/main/releasenotes.md). Check it out for latest features & bug fixes. 410 | 411 | ## Roadmap 412 | 413 | - [ ] Add checker for downloaded models (so that it won't download again after the model is downloaded) 414 | - [ ] Different UI for mobile 415 | - [ ] Gradio progress bar 416 | - [ ] Integrating refresh button (`create_refresh_button`) to refresh models etc. all at once 417 | - [ ] Logo change 418 | - [ ] Moving most of the content of this `Readme.md` to Wiki instead 419 | - [ ] Other download sites (s-ul.eu, gitgud, bunkr.ru, icedrive) 420 | - [ ] Separate 'download' and 'run command' function with a subtab 421 | - [ ] Support download MEGA and Google Drive folder 422 | - [ ] Using `yield` instead of gradio's `every` 423 | - [ ] Youtube video (Tutorial) 424 | - [ ] (Crawler) Download every sd-webui related file from a huggingface repo 425 | - [ ] (Crawler) Download every model from a civitai user page 426 |
_ 427 | - [x] aria2 for huggingface download method 428 | - [x] Cleaning the code from unnecesarry comments 429 | - [x] Completed download will use the webui's notification.mp3 430 | - [x] Support customizable hashtag from the UI 431 | - [x] Supports Windows local installation 432 | - [x] UI overhaul 433 | - [x] Using threading/subprocess instead of os.system to download files 434 | - [x] (Windows) wget & aria2 support 435 | 436 | ## Known Bugs 437 | 438 | - Progress bar (the yellow bar) doesn't progress as expected 439 | - Sometimes notification sound doesn't play when downloading same file twice in a row 440 | - Sometimes notification sound shows up when starting download, instead of when the download process is completed. 441 | - There's still a chance that the UI of non `--gradio-queue` session and/or onedotsix freezes after a download session 442 | - Windows: The delay between file is downloaded and the output shows is pretty long, and even sometimes the notification comes at the wrong time. 443 |
444 | 445 | ### Known Fixed Bugs 446 | 447 |
448 | 👈Fixed Bugs 449 |
    450 |
      451 |
    • Links that has bracket in it needs to be 'escaped' (For example, Baka-DiffusionV1(Fp16).safetensors must be typed Baka-DiffusionV1\(Fp16\).safetensors)
      ☝️Fixed in v2.0.0 452 |
    • The delay between file is downloaded and the output shows is really long (1min+) on camenduru's v1.6 colab (Gradio related?)
      ☝️Seems like fixed in v1.1.0 453 |
    • File downloaded from MEGA will not listed on the output, as it use different download method. There is some delay between the transfare() function complete until it writes the file. I don't know how long the delay is.
      ☝️Fixed in v1.1.0 454 |
    • Sometimes colab cannot be shut down with a single click on the stop button. Hitting the button several times will raise a KeyboardInterrupt and forcely stopping the cell.
      ☝️Seems like fixed somewhere on v3.0.0 and after 455 |
    456 |
457 |
458 | 459 | ## Contributing 460 | 461 | 462 | 463 | [I no longer actively maintain this project](https://github.com/etherealxx/batchlinks-webui/blob/main/regardingthisrepo.md). if you have a suggestion or code-fixing that would make this better, please notify me in the issue tab, fork the repo and create a pull request.
464 | But still, a star on this project would be nice. Thanks again. 465 | 466 | 475 | 476 | ## Contact 477 | 478 | My Email - gwathon3@gmail.com
479 | My Youtube - [MJ Devlog](https://www.youtube.com/@mjdevlog) 480 | 481 | ## Acknowledgments 482 | 483 | - [Camenduru's Webui Huggingface](https://github.com/camenduru/stable-diffusion-webui-huggingface) - I use his extension as my base (my gradio skill sucks T.T) 484 | - [SD Civitai Browser](https://github.com/Vetchems/sd-civitai-browser) - Civit download script 485 | - [Mega-to-Google-Drive](https://github.com/menukaonline/Mega-to-Google-Drive) - MEGA download script 486 | - [MEGAcmd](https://github.com/meganz/MEGAcmd) 487 | - [mediafire-dl](https://github.com/Juvenal-Yescas/mediafire-dl) 488 | - [Pixeldrain downloader](https://github.com/FayasNoushad/Pixeldrain) - Inspiration for Pixeldrain download method 489 | - [wget-windows](https://github.com/webfolderio/wget-windows) 490 | - [aria2-static-builds](https://github.com/q3aql/aria2-static-builds) 491 | - [7zr, 7z](https://www.7-zip.org) 492 | -------------------------------------------------------------------------------- /howtogetthedirectlinks.md: -------------------------------------------------------------------------------- 1 | # How to get the direct links 2 | 3 | Jump to: [Huggingface](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#huggingface) - [Civit AI](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#civit-ai) - [MEGA](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#mega) - [Discord](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#discord-attachments) - [Github (Extension)](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#webui-extension-github-repo-link) - [Github (Raw File)](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#raw-file-from-github) - [Github (Release File)](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#release-file-from-github) - [Google Drive](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#google-drive) - [Pixeldrain](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#pixeldrain) - - [Mediafire](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#mediafire) - [Anonfiles](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#anonfiles) - [Dropbox](https://github.com/etherealxx/batchlinks-webui/blob/main/howtogetthedirectlinks.md#dropbox) 4 | 5 | ## Huggingface 6 | 7 | Locate the model page, click on `files` tab.

8 | hg1

9 | choose the model you want, **right click** the arrow button beside the LFS logo, then click `copy link address`.

10 | hg2

11 | 12 | ## Civit AI 13 | 14 | There are two ways to download links from Civit. The model page link method, and the direct link method. The model page link method **will automatically** choose the directory of the saved model without even using hashtag, **except** if you want to put Lora into `#addnetlora` directory, you can use that hashtag. You still need to define the output directory when using the direct link method. 15 | 16 | ### Model Page Link Method 17 | 18 | Locate the desired model on the main page of Civitai, **right click** the model thumbnail, then click `copy link address`.

19 | cv

20 | You can also click the thumbnail first then copy the link.

21 | cv

22 | The advantage of this method is you will also get the model preview images that can shows up on the latest version of [stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui). 23 | 24 | ### Direct Link Method 25 | 26 | Locate the model page, **right click** the huge blue button, then click `copy link address`.

27 | cv

28 | With this method, you can also choose model variation, by clicking the dropdown icon on the blue button, then **right click** the desired model an.

29 | cv

30 | 31 | ## MEGA 32 | 33 | Locate the model you wanted, **right click** the item, click `copy link address`.

34 | mg1

35 | Click the huge green copy button.

36 | mg2

37 | 38 | ## Discord attachments 39 | 40 | Locate the file you wanted, **right click** the download icon, then click `get link`.

41 | mg1

42 | Then click `copy link`.

43 | mg2

44 | 45 | ## webui Extension (Github repo link) 46 | 47 | Locate the extension's repository, then copy the repository link.

48 | mg1

49 | 50 | Extension **will automatically** cloned/downloaded on `extensions` directory of webui without the need of hashtag. 51 | 52 | ## Raw file from Github 53 | 54 | Locate the file's repository, then click the desired file.

55 | mg1

56 | Then **right click** the text that says `raw` **or** `download`, and click `copy link address`

57 | mg1

58 | mg1

59 | 60 | ## Release file from Github 61 | 62 | Locate the file's repository, then click release section.

63 | mg1

64 | Then **right click** the desired assets (the blue text), and click `copy link address`

65 | mg1

66 | 67 | ## Google Drive 68 | 69 | Whether you own the file or not, make sure the file is public. If you owns the file, right click on the file on Google Drive, then click `get link` (or `share`).

70 | mg1

71 | Then change the access to `anyone with thelink`, and copy the link by clicking the button

72 | mg1

73 | 74 | These are some valid google drive link patterns that works: 75 | ``` 76 | https://drive.google.com/file/d/1-UbwFtSzKTQkvjdPiKLCejH0Ztq9UJFO/view 77 | https://drive.google.com/file/d/1-UbwFtSzKTQkvjdPiKLCejH0Ztq9UJFO/view?usp=share_link 78 | https://drive.google.com/u/0/uc?id=1-UbwFtSzKTQkvjdPiKLCejH0Ztq9UJFO&export=download 79 | https://drive.google.com/u/0/uc?id=1-UbwFtSzKTQkvjdPiKLCejH0Ztq9UJFO&export=download&confirm=t 80 | ``` 81 | ## Pixeldrain 82 | 83 | Just click the `copy link` button, or just copy the link.

84 | mg1

85 | 86 | ## Anonfiles 87 | 88 | **Don't** right click the download button. Instead, copy the link on the url bar

89 | mg1

90 | 91 | ## Mediafire 92 | 93 | When you arrived on this page, click the chain link icon to copy link, or just copy the link above

94 | mg1

95 | 96 | Don't forget that when being pasted, the link must starts with `https://www.mediafire.com/file/` (notice the triple W) 97 | 98 | ## Dropbox 99 | 100 | When the item is yours, go to the homepage, and click copy link, the link will be copied to your clipboard

101 | mg1

102 | 103 | Or, if you got into this screen, just copy the link above, with the link ends with either `?dl=0` or `?dl=1`

104 | mg1

-------------------------------------------------------------------------------- /images/anonfiles.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/anonfiles.png -------------------------------------------------------------------------------- /images/batchlinks_logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/batchlinks_logo.png -------------------------------------------------------------------------------- /images/cancel.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/cancel.jpg -------------------------------------------------------------------------------- /images/checkconsole1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/checkconsole1.jpg -------------------------------------------------------------------------------- /images/checkconsole2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/checkconsole2.jpg -------------------------------------------------------------------------------- /images/civit1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/civit1.jpg -------------------------------------------------------------------------------- /images/civit2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/civit2.jpg -------------------------------------------------------------------------------- /images/civitnew1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/civitnew1.jpg -------------------------------------------------------------------------------- /images/civitnew2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/civitnew2.jpg -------------------------------------------------------------------------------- /images/discord1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/discord1.jpg -------------------------------------------------------------------------------- /images/discord2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/discord2.jpg -------------------------------------------------------------------------------- /images/downloaded.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/downloaded.jpg -------------------------------------------------------------------------------- /images/drive1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/drive1.png -------------------------------------------------------------------------------- /images/drive2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/drive2.png -------------------------------------------------------------------------------- /images/dropbox1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/dropbox1.png -------------------------------------------------------------------------------- /images/dropbox2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/dropbox2.png -------------------------------------------------------------------------------- /images/example.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/example.jpg -------------------------------------------------------------------------------- /images/ext_installer.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/ext_installer.jpg -------------------------------------------------------------------------------- /images/github.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/github.jpg -------------------------------------------------------------------------------- /images/githubdirect1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/githubdirect1.png -------------------------------------------------------------------------------- /images/githubdirect2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/githubdirect2.png -------------------------------------------------------------------------------- /images/githubdirect3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/githubdirect3.png -------------------------------------------------------------------------------- /images/githubrelease1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/githubrelease1.png -------------------------------------------------------------------------------- /images/githubrelease2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/githubrelease2.png -------------------------------------------------------------------------------- /images/gradiooff1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/gradiooff1.jpg -------------------------------------------------------------------------------- /images/gradiooff2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/gradiooff2.jpg -------------------------------------------------------------------------------- /images/huggingface1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/huggingface1.jpg -------------------------------------------------------------------------------- /images/huggingface2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/huggingface2.jpg -------------------------------------------------------------------------------- /images/logging.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/logging.jpg -------------------------------------------------------------------------------- /images/mediafire.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/mediafire.png -------------------------------------------------------------------------------- /images/mega1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/mega1.jpg -------------------------------------------------------------------------------- /images/mega2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/mega2.jpg -------------------------------------------------------------------------------- /images/pastebin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/pastebin.png -------------------------------------------------------------------------------- /images/pixeldrain.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/pixeldrain.png -------------------------------------------------------------------------------- /images/progress.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/progress.jpg -------------------------------------------------------------------------------- /images/python.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/python.png -------------------------------------------------------------------------------- /images/queue.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/queue.jpg -------------------------------------------------------------------------------- /images/rename.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/rename.jpg -------------------------------------------------------------------------------- /images/sdless.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/sdless.png -------------------------------------------------------------------------------- /images/shell1.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/shell1.jpg -------------------------------------------------------------------------------- /images/shell2.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/images/shell2.jpg -------------------------------------------------------------------------------- /install.py: -------------------------------------------------------------------------------- 1 | import launch 2 | import platform 3 | 4 | if platform.system() == "Windows": 5 | if not launch.is_installed("gdown"): 6 | launch.run_pip("install gdown", "requirements for Batchlinks Download extension") 7 | 8 | if not launch.is_installed("wget"): 9 | launch.run_pip("install wget", "requirements for Batchlinks Download extension") -------------------------------------------------------------------------------- /javascript/notification.js: -------------------------------------------------------------------------------- 1 | let prevTextAreaValue = ''; 2 | let isTextAreaVisible = true; 3 | 4 | onUiUpdate(function() { 5 | const textArea = gradioApp().querySelectorAll('#tab_batchlinks textarea')?.[2]; 6 | const textAreaValue = textArea ? textArea.value : ''; 7 | 8 | if (isTextAreaVisible && textAreaValue.startsWith('All done!') && prevTextAreaValue !== textAreaValue) { 9 | gradioApp().querySelector('#finish_audio audio')?.play(); 10 | } 11 | 12 | if (!textArea) { 13 | isTextAreaVisible = false; 14 | } else if (textArea.style.display !== 'none') { 15 | isTextAreaVisible = true; 16 | } 17 | 18 | prevTextAreaValue = textAreaValue; 19 | }); 20 | -------------------------------------------------------------------------------- /misc/markydynamictohtml.py: -------------------------------------------------------------------------------- 1 | #tableofcontentmaker 2 | intro = """ 3 |
4 | Table of Contents 5 |
    6 | """ 7 | 8 | outro = """ 9 | 10 |
11 |
12 | """ 13 | 14 | tableofcontents = """ 15 | - [Installation](#installation) 16 | - [About](#about) 17 | - [Example](#example) 18 | - [Syntax](#syntax) 19 | - [Hashtag](#hashtag) 20 | - [Links](#links) 21 | - [Double Hashtag](#double-hashtag) 22 | - [Others](#others) 23 | - [Valid Hashtags](#valid-hashtags) 24 | - [How to get the direct links (Important!)](#how-to-get-the-direct-links-important) 25 | - [Huggingface's download method](#huggingfaces-download-method) 26 | - [Additional Syntax](#additional-syntax) 27 | - [Rename Downloaded Files](#rename-downloaded-files) 28 | - [Running Shell Commands](#running-shell-commands) 29 | - [Extract Everyting (`@extract`)](#extract-everyting-extract) 30 | - [Custom Hashtag Path (`@new`)](#custom-hashtag-path-new) 31 | - [Custom Direct Link Download with `aria2` (`@aria2`)](#custom-direct-link-download-with-aria2-aria2) 32 | - [Gradio Queue](#gradio-queue) 33 | - [Logging](#logging) 34 | - [Cancel](#cancel) 35 | - [Progress Bar](#progress-bar) 36 | - [Changes if `--gradio-queue` is off](#changes-if---gradio-queue-is-off) 37 | - [Other Features](#other-features) 38 | - [Notification](#notification) 39 | - [Copy From Pastebin](#copy-from-pastebin) 40 | - [SDless mode](#sdless-mode) 41 | - [Local Installation Support](#local-installation-support) 42 | - [Debug Mode (Developer only)](#debug-mode-developer-only) 43 | - [Release Notes](#release-notes) 44 | - [Roadmap](#roadmap) 45 | - [Known Bugs](#known-bugs) 46 | - [Known Fixed Bugs](#known-fixed-bugs) 47 | - [Contributing](#contributing) 48 | - [Contact](#contact) 49 | - [Acknowledgments](#acknowledgments) 50 | """ 51 | 52 | def mlineprint(toprint): 53 | if toprint[-1] == '\n': 54 | toprint = toprint[:-1] 55 | print(toprint.replace('\n', '', 1)) 56 | 57 | 58 | def writehtmllist(space, namehead, hashtaghead): 59 | spaces = "" 60 | for _ in range(space): 61 | spaces += " " 62 | if initialspace >= 2: 63 | return spaces + f'' 64 | else: 65 | return spaces + f'
  • {namehead}
    • ' 66 | 67 | if __name__ == "__main__": 68 | rows = tableofcontents.replace('\n', '', 1).splitlines()#split('\n') # split into a list of rows 69 | mlineprint(intro) 70 | for row in rows: 71 | if not row == '': 72 | initialspace = 0 73 | for char in row: 74 | if char == ' ': 75 | initialspace += 1 76 | else: 77 | break 78 | extractedrow = row.split("[") 79 | cuttedrow = extractedrow[1].partition("](") 80 | headingname = cuttedrow[0] 81 | headinghashtag = cuttedrow[2].rstrip(")") 82 | print(writehtmllist(initialspace+4, headingname, headinghashtag)) 83 | # print(cuttedrow[0]) 84 | # print(cuttedrow[2].rstrip(")")) 85 | # hashtagrow = row.partit 86 | mlineprint(outro) 87 | -------------------------------------------------------------------------------- /notification.mp3: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/etherealxx/batchlinks-webui/d44bbb5e2a043f2eed80c3945c0f2c676e41d0e5/notification.mp3 -------------------------------------------------------------------------------- /regardingthisrepo.md: -------------------------------------------------------------------------------- 1 | #### Dear beloved Batchlinks user, 2 | 3 | I want to share with you that I have lost interest in this project/repo. It was a time when I was fully invested in it, spending countless hours fixing bugs, adding new features, and learning Python/Markdown. I remember the excitement of refreshing the notification page on Github to find a new bug report or feature request. It was an addicting experience, and I enjoyed it more than my college studies. 4 | 5 | But unfortunately, things have changed, and my mood and interests shift rapidly. I have been into different things throughout my life, such as game development, graphic design, video editing, music creation, gaming, and more. Even though I loved generating images with AI during the early days of stable diffusion, I rarely do it now, since my focus has shifted into this project. 6 | 7 | Despite my change in interests, I want to emphasize that I don't want to abandon this project. I have made many friends through this project, and it holds a special place in my heart. However, I just can't continue to give it the same level of attention and dedication that it deserves. I might still do some small bug fixes, but for adding features, maybe not. 8 | 9 | I will miss those days of working on this project, but I'm excited to explore new interests in gaming, video editing, and many others. I want to thank camenduru, machiavel, 7743, NUROISEA, NoCrypt, and many other people who supported me on various Discord servers, github issues, and many other places. Also thanks to everyone who has contributed to this project, whether by reporting bug, requesting features, pull request, and many others. And lastly, thank YOU for using this tool. It might just helps you a little, but every single user means A LOT to me. 10 | 11 | I hope this continues to thrive and helps people in the future.💖 12 | 13 | Best regards,
    14 | MJ -------------------------------------------------------------------------------- /releasenotes.md: -------------------------------------------------------------------------------- 1 | # Release Notes 2 | 3 | ### Latest Patch: v3.2.1 4 | - Changed "pruned" option to "fp precision" due to CivitAI API changes 5 | - Fixed bug: No matter when the download process is cancelled or not, CivitAI download will give print out notification that the download process is successful 6 | - Fixed bug: Preventing the UI errored out when storage is full and trying to convert jpg image while downloading civitai stuff (PIL.UnidentifiedImageError) 7 | - UI overhaul 8 | 9 | #### Patch: v3.2.0a 10 | - Added support for [vladmandic's fork of automatic1111's webui](https://github.com/vladmandic/automatic) 11 | 12 | ## Latest Release: v3.2.0 13 | - Bug fixed when downloading with aria2 on Windows 14 | - Bug fixed on gradio update checker on sdless batch file for windows 15 | - CivitAi error prevention (when the website is down) 16 | - Cleaned the code of the hashtag system 17 | - `Copy from Pastebin` feature 18 | - Custom aria command with `@aria` 19 | - Making sure custom hashtag doesn't start with number 20 | - New calmer notification sound 21 | - Readme page cleanup 22 | - Some help link changed to link into wiki instead 23 | - UI fix to adapt with latest gradio (v3.23.0) 24 | - Various bug fixes related to CivitAI download 25 | 26 | #### Release v3.1.1b 27 | - Hotfix: Fixed CivitAI link refuses to be downloaded properly. Seems like CivitAI changes its API. 28 | - Fixed several bugs on Windows, especially SDless 29 | 30 | #### Release v3.1.1a 31 | - Hotfix: Fixed CivitAI 'model type chooser' bug and fixed `@extract` bug on colab 32 | 33 | #### Release v3.1.1 34 | - Added some fix in case CivitAI website is down 35 | - Added message when user pressed "Download All!" but the textbox is empty 36 | - Fixed bug where Lycoris folder always shows on downloaded files the first time user download something 37 | 38 | #### Release v3.1.0a 39 | - Hotfix: Indented block on line 1497 fix 40 | 41 | ### Release v3.1.0 42 | - New hashtag: `#altmodel`, when you use `--ckptdir` argument on `launch.py` line, this hashtag will points to that directory. Otherwise, it'll point to the same directory as `#model` 43 | - New hashtag: `#lycoris`, change current save directory to _/content/stable-diffusion-webui/extensions/sd-webui-additional-networks/models/lora/lycoris_
    (Side note: _Lycoris/Locon/Loha_ will works just fine if you use `#addnetlora` instead, as long as you have both [addnet extension](https://github.com/kohya-ss/sd-webui-additional-networks) and [locon extension](https://github.com/KohakuBlueleaf/a1111-sd-webui-locon) installed) 44 | - New hashtag: `#upscaler`, change current save directory to _/content/stable-diffusion-webui/models/ESRGAN_ (This was added few commits ago, but i forgot to write it on the release notes) 45 | 46 | Fixes: 47 | - Fixed bug when installing wget/aria2 on Windows 48 | - Fixed sdless scripts so that it also installs `tqdm` 49 | - Fixed bug where custom paths doesn't put CivitAi download (Model Path method) when custom hashtag is used 50 | - Ongoing downloads will correctly stops when Cancel button is pressed on Windows 51 | - CivitAi (Model Path method) now will not download training dataset by accident 52 | 53 | #### Release v3.0.2 54 | - Hotfix: removed a comment that messed up the non queue mode. Now the extension works again without `--gradio-queue`. 55 | 56 | #### Release v3.0.1 57 | - Hotfix: using `urllib.request` instead of `curl -sI` to get the model name on CivitAI direct link method, since it's more reliable (and the curl method always fails somehow). The `requests` method is returned as a fallback. 58 | 59 | ### Release v3.0.0 60 | - Added `@extract` syntax 61 | - (Almost) Full Windows support 62 | - Auto-download config file if available when downloading from CivitAI (SD 2.0+) 63 | - Auto-renaming for downloading ckpt/safetensors and pruned model from CivitAI using direct link method 64 | - CivitAI direct link now use `curl` to get the filename, and use the chosen download method (from the four) to download. Huge download speed boost. `requests` is no longer needed. 65 | - Supports download from Anonfiles, Dropbox, Google Drive, Mediafire, Pixeldrain 66 | - Supports download from Github (raw and release files) 67 | - Supports for custom hashtags with `@new` syntax 68 | - Supports for SDless mode (read more [here](https://github.com/etherealxx/batchlinks-webui#sdless-mode)) 69 | - UI overhaul: 70 | - Now there's a table that shows where does the hashtags points into 71 | - Option to stretch the UI, if your monitor is small, or using colab on mobile 72 | - Option to hide help text 73 | - Option to choose preferred CivitAI models. This will works if you download the model via model page link (https://civitai.com/models/) 74 | - Upload txt now use a little button instead of covering half of the screen 75 | 76 | Fixes: 77 | - CivitAI `model page link` no longer randomly download the first model on the json list. 78 | - Most of Windows bugs 79 | - Renaming problem when using CivitAI model page link method 80 | - Warning message when CivitAI download isn't possible (server down) 81 | 82 | ### Older Release 83 |
    84 | 👈v2.0.0 - v2.1.1 85 |
      86 |

      Release v2.1.1

      87 | Partial Windows support is back 88 | Changes: 89 |
        90 |
      • wget disabled on windows currently, until it fixed 91 |
      92 | Fixes: 93 |
        94 |
      • gdown & curl bug fixed 95 |
      • utf-8 as default encoding for queue checker (fix bug in Windows) 96 |
      97 |

      Release v2.1.0

      98 | Features: 99 |
        100 |
      • Supports renaming downloaded file with > (for example: https://files.catbox.moe/uarze8.safetensors > neurosama.safetensors) 101 |
      • Supports extension usage without --gradio-queue (ported from onedotsix) 102 |
      • Supports running shell command from the UI with ! (for example: type !pip freeze, then hit the Download all! button and see the colab console) 103 |
      • Progress bar for --gradio-queue 104 |
      105 | Changes: 106 |
        107 |
      • aria2 as the only download method when using without --gradio-queue 108 |
      • Download session will be cut every 80 seconds on when using without --gradio-queue (just like onedotsix) 109 |
      • Debug stopwatch (decorator) won't run automatically when globaldebug = True, must be uncommented manually (it disrupt the progress bar) 110 |
      • Dropped support for webui based on Gradio 3.9 (update your installation, or use onedotsix instead) 111 |
      • UI tweak (Smaller font size) 112 |
      113 |

      Release v2.0.0

      114 | Features: 115 |
        116 |
      • aria2 as download method. 117 |
      • Cancel button for cancelling download process (--gradio-queue required) 118 |
      • Detection if a CivitAI links no longer exist 119 |
      • New hashtags: #textualinversion, #ti, #aestheticembedding, #aestheticembed, #controlnet, and #cnet 120 |
      • Toggle logging on/off 121 |
      • shlex.quote to properly quote links (Thanks @rti7743!) 122 |
      • Supports cloning webui extensions 123 |
      • Supports download from catbox.moe 124 |
      • Supports download from CivitAI model links (Thanks @rti7743!) 125 |
      • Supports download from Github (repository and raw files) 126 |
      • Supports for aesthetic gradients, controlnet model, and extensions path. 127 |
      • UI font scaled down 128 |
      • Uses subprocess.Popen instead of os.system 129 |
        _ 130 |
      • Debug snapshot
        131 | When globaldebug = True, the moment this extension launch, it saves the current state of the webui on various location (into snapshot.txt), and when you type #debugresetdownloads on the textbox, it will compare the current state and the last saved state, and removes every new file/folder. This will be useful for debugging and testing. 132 |
      • Debug every download method
        133 | When globaldebug = True and you type #debugevery method on the textbox, every link that has 4 different method of download (Huggingface etc.) will be downloaded with every method, regardless of the radio button choice. The result is 4 file being downloaded. 134 |
      • Debug stopwatch
        135 | When globaldebug = True, it will give an output for how long a single download session lasts 136 |
      137 |
    138 |
    -------------------------------------------------------------------------------- /scripts/a_gradiocheck.py: -------------------------------------------------------------------------------- 1 | from modules.shared import cmd_opts 2 | import os, inspect, re #, sys, shlex 3 | 4 | script_dir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) 5 | batchlinks_dir = os.path.join(script_dir, "batchlinks-downloader.py") 6 | 7 | # remove the progress bar if queue off 8 | with open(batchlinks_dir, 'r', encoding='utf-8') as f: 9 | contents = f.read() 10 | 11 | gradio_queue_on = False 12 | 13 | try: 14 | if cmd_opts.gradio_queue: #automatic1111 15 | gradio_queue_on = True 16 | except AttributeError: 17 | if not cmd_opts.disable_queue: #vladmandic 18 | gradio_queue_on = True 19 | 20 | if gradio_queue_on: 21 | #os.system(f"sed -i 's/def run(command, choosedowner):/def run(command, choosedowner, progress=gr.Progress()):/g' {batchlinks_dir}") 22 | # os.system(f"sed -i '/^#progress/s/^#//' {batchlinks_dir}") 23 | new_contents = re.sub(r'^(\s*)#progress\(', r'\1progress(', contents, flags=re.MULTILINE) 24 | contents = re.sub(r'civitvae\):', 'civitvae, progress=gr.Progress()):', new_contents, flags=re.MULTILINE) 25 | 26 | with open(batchlinks_dir, 'w', encoding='utf-8') as f: 27 | f.write(contents) 28 | 29 | else: 30 | # os.system(f"sed -i 's/def run(command, choosedowner, progress=gr.Progress()):/def run(command, choosedowner):/g' {batchlinks_dir}") 31 | # os.system(f"sed -i '/^progress(/ s/^/#/' {batchlinks_dir}") 32 | new_contents = re.sub(r'^(\s*)progress\(', r'\1#progress(', contents, flags=re.MULTILINE) 33 | contents = re.sub(r'civitvae,\s*progress=gr\.Progress\(\)\):$', 'civitvae):', new_contents, flags=re.MULTILINE) 34 | 35 | with open(batchlinks_dir, 'w', encoding='utf-8') as f: 36 | f.write(contents) -------------------------------------------------------------------------------- /scripts/batchlinks-downloader.py: -------------------------------------------------------------------------------- 1 | #github.com/etherealxx 2 | import os 3 | import time 4 | import gradio as gr 5 | import urllib.request, subprocess, contextlib #these handle mega.nz 6 | import http.client 7 | import requests #this handle civit 8 | import re 9 | from tqdm import tqdm 10 | #from IPython.display import display, clear_output 11 | import pathlib 12 | import inspect 13 | import platform 14 | import shlex 15 | import signal 16 | import sys 17 | sdless = False 18 | try: 19 | from modules import script_callbacks #,scripts 20 | from modules.paths import models_path, script_path #, data_path 21 | from modules.shared import cmd_opts #check for gradio queue 22 | except ImportError: #sdless 23 | if platform.system() == "Windows": 24 | userprofile = os.environ['USERPROFILE'] 25 | downloadpath = os.path.join(userprofile, "Downloads") 26 | script_path = os.path.join(downloadpath, "stable-diffusion-webui") 27 | elif platform.system() == "Darwin": 28 | userhome = os.environ['HOME'] 29 | downloadpath = os.path.join(userhome, "Downloads") 30 | script_path = os.path.join(downloadpath, "stable-diffusion-webui") 31 | else: 32 | script_path = '/content/stable-diffusion-webui' 33 | models_path = os.path.join(script_path, 'models') 34 | gradio_queue = True 35 | ckpt_dir = None 36 | import sys 37 | import types 38 | module = types.ModuleType('cmd_opts') 39 | module.gradio_queue = gradio_queue 40 | module.ckpt_dir = ckpt_dir 41 | sys.modules['cmd_opts'] = module 42 | import cmd_opts 43 | sdless = True 44 | 45 | script_dir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) 46 | extension_dir = os.path.abspath(os.path.join(script_dir, "../")) 47 | #Version checking{ 48 | version_dir = os.path.join(extension_dir, "version.txt") 49 | with open(version_dir, 'r', encoding='utf-8') as file: 50 | curverall = file.readlines() 51 | currentversion = curverall[0].strip() 52 | 53 | try: 54 | versionurl = "https://raw.githubusercontent.com/etherealxx/batchlinks-webui/main/version.txt" 55 | versionresp = requests.get(versionurl) 56 | version_lines = versionresp.text.splitlines() 57 | latestversion = version_lines[0].strip() 58 | except requests.exceptions.RequestException: 59 | latestversion = '??' 60 | 61 | if latestversion != '??': 62 | if currentversion == latestversion: 63 | latestversiontext = "" 64 | else: 65 | latestversiontext = f"[Latest version: {latestversion}]" 66 | else: 67 | latestversiontext = "" 68 | 69 | currentverforlink = latestversion.replace('.', '') 70 | #} 71 | 72 | vladmandic = False 73 | try: 74 | global gradiostate 75 | if cmd_opts.gradio_queue: #automatic1111 76 | gradiostate = True 77 | else: 78 | gradiostate = False 79 | except AttributeError: 80 | try: 81 | if not cmd_opts.disable_queue: #vladmandic 82 | gradiostate = True 83 | else: 84 | gradiostate = False 85 | vladmandic = True 86 | except AttributeError: 87 | gradiostate = False #at this point just use onedotsix 88 | pass 89 | 90 | typechecker = [ 91 | "embedding", "embeddings", "embed", "embeds", "textualinversion", "ti", 92 | "model", "models", "checkpoint", "checkpoints", 93 | "vae", "vaes", 94 | "lora", "loras", 95 | "hypernetwork", "hypernetworks", "hypernet", "hypernets", "hynet", "hynets", 96 | "addnetlora", "loraaddnet", "additionalnetworks", "addnet", 97 | "aestheticembedding", "aestheticembed", 98 | "controlnet", "cnet", 99 | "extension", "extensions", "ext", #obsolete 100 | "upscaler", "upscale", 101 | "altmodel", "altmodels", 102 | "lycoris", "locon", "loha" 103 | ] 104 | 105 | typemain = [ 106 | "model", "altmodel", "vae", 107 | "embed", "hynet", "lora", 108 | "addnetlora", "aestheticembed", "cnet", 109 | "ext", "upscaler", "lycoris" 110 | ] 111 | countofdefaulthashtags = len(typemain) 112 | 113 | supportedlinks = [ 114 | "https://mega.nz", 115 | "https://huggingface.co", 116 | "https://civitai.com/api/download/models", 117 | "https://civitai.com/models/", 118 | "https://cdn.discordapp.com/attachments", 119 | "https://github.com", 120 | "https://raw.githubusercontent.com", 121 | "https://files.catbox.moe", 122 | "https://drive.google.com", 123 | "https://pixeldrain.com", 124 | "https://www.mediafire.com/file", 125 | "https://anonfiles.com", 126 | "https://www.dropbox.com/s" 127 | ] 128 | 129 | modelpath = os.path.join(script_path, "models/Stable-diffusion") 130 | embedpath = os.path.join(script_path, "embeddings") 131 | vaepath = os.path.join(script_path, "models/VAE") 132 | lorapath = os.path.join(script_path, "models/Lora") 133 | addnetlorapath = os.path.join(script_path, "extensions/sd-webui-additional-networks/models/lora") 134 | hynetpath = os.path.join(script_path, "models/hypernetworks") 135 | aestheticembedpath = os.path.join(script_path, "extensions/stable-diffusion-webui-aesthetic-gradients/aesthetic_embeddings") 136 | cnetpath = os.path.join(script_path, "extensions/sd-webui-controlnet/models") 137 | extpath = os.path.join(script_path, "extensions") #obsolete 138 | upscalerpath = os.path.join(script_path, "models/ESRGAN") 139 | lycorispath = os.path.join(addnetlorapath, "lycoris") 140 | 141 | if vladmandic: 142 | cnetpath = os.path.join(script_path, "models/ControlNet") 143 | lycorispath = os.path.join(script_path, "models/LyCORIS") 144 | 145 | if cmd_opts.ckpt_dir: 146 | altmodelpath = cmd_opts.ckpt_dir 147 | currentfolder = altmodelpath 148 | else: 149 | altmodelpath = modelpath 150 | currentfolder = modelpath 151 | 152 | if platform.system() == "Windows": 153 | for x in typemain: 154 | # exec(f"{x}path = os.path.normpath({x}path)") 155 | exec(f"{x}path = {x}path.replace('/', '\\\\')") 156 | #exec(f"print({x}path)") 157 | if os.path.exists(addnetlorapath) and not os.path.exists(lycorispath): 158 | os.makedirs(lycorispath, exist_ok=True) 159 | 160 | newlines = ['\n', '\r\n', '\r'] 161 | currentlink = '' 162 | finalwrite = [] 163 | currentcondition = '' 164 | currentsuboutput = '' 165 | processid = '' 166 | logging = False 167 | #currentiterfolder = '' 168 | prockilled = False 169 | currentfoldertrack = [] 170 | everyprocessid = [] 171 | remaininglinks = [] 172 | gdownupgraded = False 173 | mediafireinstalled = False 174 | prevciviterror = False 175 | # addedcustompath = [] 176 | # ariamode = False 177 | 178 | globaldebug = False #set this to true to activate every debug features 179 | if len(sys.argv) > 1: 180 | if sys.argv[1] == '--debug': 181 | globaldebug = True 182 | 183 | process = '' 184 | 185 | def stopwatch(func): #unused 186 | """ 187 | A function that acts as a stopwatch for another function. 188 | 189 | Args: 190 | func (function): The function to be timed. 191 | 192 | Returns: 193 | The return value of the timed function. 194 | """ 195 | def wrapper(*args, **kwargs): 196 | if not globaldebug: 197 | return func(*args, **kwargs) 198 | start_time = time.time() 199 | result = func(*args, **kwargs) 200 | end_time = time.time() 201 | elapsed_time = end_time - start_time 202 | minutes = int(elapsed_time // 60) 203 | seconds = elapsed_time % 60 204 | print(f"Time taken by {func.__name__}: {minutes:.0f}m {seconds:.2f}s") 205 | return result 206 | return wrapper 207 | 208 | #debuggingpurpose{ 209 | #Hello debuggers! This will track every files when the extension is launched, and 210 | #you can remove every downloaded files after with hashtag '@debugresetdownloads', for debugging purposes on colab 211 | #(Note: You need to fill the textbox with only a single line of @debugresetdownloads and nothing more) 212 | #There's also another feature, '#debugeverymethod', which will download a single link with all four possible methods available. 213 | import shutil 214 | snapshot = [] 215 | paths_to_scan = [] 216 | 217 | # take a snapshot of the directories 218 | def take_snapshot(): 219 | snapshotdir = os.path.join(script_path, 'snapshot.txt') 220 | global snapshot 221 | global paths_to_scan 222 | paths_to_scan = [] 223 | for x in typemain: 224 | exec(f"paths_to_scan.append({x}path)") 225 | if os.path.exists(snapshotdir): 226 | with open(snapshotdir, 'r') as f: 227 | snapshot = [line.strip() for line in f.readlines()] 228 | print("Batchlinks extension: snapshot already exist.") 229 | return 230 | else: 231 | snapshot = [] 232 | for path in paths_to_scan: 233 | if os.path.exists(path): 234 | pathtemp = os.listdir(path) 235 | for file in pathtemp: 236 | pathoffile = os.path.join(path, file) 237 | snapshot.append(pathoffile) 238 | with open(snapshotdir, 'w') as f: 239 | for item in snapshot: 240 | f.write(f"{item}\n") 241 | print("Batchlinks extension: snapshot taken.") 242 | 243 | # rewind everything to a fresh state 244 | def global_rewind(): 245 | global paths_to_scan 246 | global path 247 | global currentsuboutput 248 | removedall, removed_files, removed_dirs, new_snapshot = [], [], [], [] 249 | print('Debug rewind initiated...') 250 | print('') 251 | for path in paths_to_scan: 252 | if os.path.exists(path): 253 | pathtemp = os.listdir(path) 254 | for file in pathtemp: 255 | pathoffile = os.path.join(path, file) 256 | new_snapshot.append(pathoffile) 257 | toremoves = [x for x in new_snapshot if x not in snapshot] 258 | for fileordir in toremoves: 259 | if os.path.exists(fileordir): 260 | if os.path.isdir(fileordir): 261 | try: 262 | shutil.rmtree(fileordir) 263 | removed_dirs.append(fileordir) 264 | except PermissionError as e: 265 | print("Error: " + str(e)) 266 | else: 267 | os.remove(fileordir) 268 | removed_files.append(fileordir) 269 | if removed_files or removed_dirs: 270 | print("Removed files:") 271 | removedall.append("Removed files:") 272 | for file in removed_files: 273 | print(file) 274 | removedall.append(file) 275 | print("Removed directories:") 276 | removedall.append("Removed directories:") 277 | for dir in removed_dirs: 278 | print(dir) 279 | removedall.append(dir) 280 | print('rewind completed') 281 | print('') 282 | return removedall 283 | 284 | # Take a snapshot of the directories 285 | if globaldebug == True: 286 | take_snapshot() 287 | # } 288 | 289 | def printdebug(toprint): 290 | if globaldebug == True: 291 | print(toprint) 292 | 293 | def printvardebug(toprint): 294 | if globaldebug == True: 295 | import inspect 296 | caller = inspect.currentframe().f_back 297 | try: 298 | name = [k for k, v in caller.f_locals.items() if v is toprint][0] 299 | except IndexError: 300 | name = "idk" 301 | if not isinstance(toprint, str): 302 | toprint = str(toprint) 303 | print(f"debug {name}: {toprint}") 304 | 305 | def runwithsubprocess(rawcommand, folder=None, justrun=False, additionalcontext=''): #@note runwithsubprocess 306 | 307 | commandtorun = shlex.split(rawcommand) #construct_command(rawcommand) 308 | printdebug(f"raw command: {rawcommand}") 309 | printdebug(f"merged command: {commandtorun}") 310 | global prockilled 311 | global everyprocessid 312 | # if gradiostate == False and not rawcommand.startswith("aria"): 313 | # subprocess.run(commandtorun, stderr=subprocess.STDOUT, universal_newlines=True) 314 | # else: 315 | if folder != None: 316 | printdebug("debug folderforsavestate: " + str(folder)) 317 | savestate_folder(folder) 318 | printdebug("debug currentfoldertrack: " + str(currentfoldertrack)) 319 | 320 | global process 321 | if justrun: 322 | process = subprocess.Popen(rawcommand, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True, shell=True) 323 | else: 324 | process = subprocess.Popen(commandtorun, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True) 325 | global processid 326 | processid = process.pid 327 | everyprocessid.append(processid) 328 | printdebug("debug processid: " + str(processid)) 329 | 330 | ariacomplete = False 331 | global currentsuboutput 332 | while True: 333 | # Read the output from the process 334 | nextline = process.stdout.readline() 335 | if nextline == '' and process.poll() is not None: 336 | break 337 | # Check if the line contains progress information 338 | elif additionalcontext == 'aria2': 339 | if 'Download Results' in nextline: 340 | ariacomplete = True 341 | print('\n') 342 | print(nextline, end='') 343 | currentsuboutput = nextline 344 | else: 345 | if not ariacomplete: 346 | match = re.search(r'\[[^\]]+\]', nextline) 347 | if match: 348 | stripnext = match.group().strip() 349 | print("\r", end="") 350 | print(f"\r{stripnext}", end='') 351 | currentsuboutput = stripnext 352 | else: 353 | print(nextline, end='') 354 | currentsuboutput = nextline 355 | elif additionalcontext == '7z': 356 | sevenzmessage = [ 357 | "Extracting", "Everything", "ERROR" 358 | ] 359 | if nextline.strip().startswith(tuple(sevenzmessage)): 360 | print(nextline, end=' ') 361 | currentsuboutput = nextline 362 | else: 363 | stripnext = nextline.strip() 364 | print("\r", end="") 365 | print(f"\r{stripnext}", end='') 366 | currentsuboutput = stripnext 367 | else: 368 | if justrun: 369 | print(nextline, end='') 370 | else: 371 | if "%" in nextline.strip() or rawcommand.startswith("curl"): 372 | stripnext = nextline.strip() 373 | print("\r", end="") 374 | print(f"\r{stripnext}", end='') 375 | else: 376 | print(nextline, end='') 377 | if additionalcontext == 'mega': 378 | process.kill() 379 | if additionalcontext == 'mega': 380 | if nextline.strip().startswith("Download finished:"): 381 | _ = subprocess.getoutput("taskkill /f /t /im MEGAcmdServer.exe") 382 | printdebug("MEGA server killed") 383 | currentsuboutput = nextline 384 | 385 | 386 | process.wait() 387 | currentsuboutput = '' 388 | processid = '' 389 | if prockilled == True: 390 | printdebug("before rewind folder") 391 | printvardebug(folder) 392 | rewind_folder(folder) 393 | print('Operation Cancelled') 394 | print('') 395 | global currentcondition 396 | currentcondition = 'Operation Cancelled' 397 | return 398 | 399 | #these code below handle mega.nz 400 | def unbuffered(proc, stream='stdout'): 401 | stream = getattr(proc, stream) 402 | with contextlib.closing(stream): 403 | while prockilled == False: 404 | out = [] 405 | last = stream.read(1) 406 | # Don't loop forever 407 | if last == '' and proc.poll() is not None: 408 | break 409 | while last not in newlines: 410 | # Don't loop forever 411 | if last == '' and proc.poll() is not None: 412 | break 413 | out.append(last) 414 | last = stream.read(1) 415 | out = ''.join(out) 416 | yield out 417 | 418 | def transfare(todownload, folder, torename=''): 419 | #import codecs 420 | #decoder = codecs.getincrementaldecoder("UTF-8")() 421 | todownload_s = shlex.quote(todownload) 422 | folder_s = shlex.quote(folder) 423 | if platform.system() == "Windows": 424 | savestate_folder(folder) 425 | localappdata = os.environ['LOCALAPPDATA'] 426 | megagetloc = os.path.join(localappdata, "MEGAcmd", "mega-get.bat") 427 | global process 428 | process = runwithsubprocess(f"{shlex.quote(megagetloc)} {todownload_s} {folder_s}", folder, False, 'mega') 429 | else: 430 | savestate_folder(folder_s) 431 | cmd = ["mega-get", todownload_s, folder_s] 432 | proc = subprocess.Popen( 433 | cmd, 434 | stdout=subprocess.PIPE, 435 | stderr=subprocess.STDOUT, 436 | universal_newlines=True, 437 | ) 438 | global processid 439 | global everyprocessid 440 | processid = proc.pid 441 | everyprocessid.append(processid) 442 | 443 | global currentsuboutput 444 | for line in unbuffered(proc): 445 | if prockilled == False: 446 | if not line.startswith("Download"): 447 | currentsuboutput = line 448 | print(f"\r{line}", end="") 449 | else: 450 | print(f"\n{line}") 451 | _ = subprocess.getoutput("pkill -f \"mega-cmd-server\"") 452 | printdebug("MEGA server killed") 453 | 454 | else: 455 | currentsuboutput = '' 456 | print('Operation Cancelled') 457 | print('') 458 | global currentcondition 459 | currentcondition = 'Operation Cancelled' 460 | return 461 | currentsuboutput = '' 462 | if torename: 463 | listfilenew = os.listdir(folder) 464 | newerfoldertrack = [] 465 | for file in listfilenew: 466 | pathoffile = os.path.join(folder, file) 467 | newerfoldertrack.append(pathoffile) 468 | checkrename = [x for x in newerfoldertrack if x not in currentfoldertrack] 469 | if checkrename: 470 | # renamedfile = os.path.basename(checkrename[0]) 471 | # pathtorename = os.path.join(folder, renamedfile) 472 | os.rename(checkrename[0], os.path.join(folder, torename)) 473 | 474 | 475 | def installmega(): 476 | HOME = os.path.expanduser("~") 477 | ocr_file = pathlib.Path(f"{HOME}/.ipython/ocr.py") 478 | if not ocr_file.exists(): 479 | hCode = "https://raw.githubusercontent.com/biplobsd/" \ 480 | "OneClickRun/master/res/ocr.py" 481 | urllib.request.urlretrieve(hCode, str(ocr_file)) 482 | 483 | from importlib.util import module_from_spec, spec_from_file_location 484 | ocr_spec = spec_from_file_location("ocr", str(ocr_file)) 485 | ocr = module_from_spec(ocr_spec) 486 | ocr_spec.loader.exec_module(ocr) 487 | 488 | if not os.path.exists("/usr/bin/mega-cmd"): 489 | #ocr.loadingAn() 490 | print('Installing MEGA ...') 491 | print('') 492 | ocr.runSh('sudo apt-get -y update') 493 | ocr.runSh('sudo apt-get -y install libmms0 libc-ares2 libc6 libcrypto++6 libgcc1 libmediainfo0v5 libpcre3 libpcrecpp0v5 libssl1.1 libstdc++6 libzen0v5 zlib1g apt-transport-https') 494 | ocr.runSh('sudo curl -sL -o /var/cache/apt/archives/MEGAcmd.deb https://mega.nz/linux/MEGAsync/Debian_9.0/amd64/megacmd-Debian_9.0_amd64.deb', output=True) 495 | ocr.runSh('sudo dpkg -i /var/cache/apt/archives/MEGAcmd.deb', output=True) 496 | print('MEGA is installed.') 497 | print('') 498 | 499 | def installmegawin(): 500 | userprofile = os.environ['USERPROFILE'] 501 | localappdata = os.environ['LOCALAPPDATA'] 502 | megagetloc = os.path.join(localappdata, "MEGAcmd", "mega-get.bat") 503 | megacmdloc = os.path.join(userprofile, "Downloads", "MEGAcmdSetup64.exe") 504 | if not os.path.exists(megagetloc): 505 | print('Installing MEGA ...') 506 | print('') 507 | runwithsubprocess(f"curl -o {shlex.quote(megacmdloc)} https://mega.nz/MEGAcmdSetup64.exe") 508 | time.sleep(1) 509 | runwithsubprocess(f"{shlex.quote(megacmdloc)} /S") 510 | time.sleep(4) 511 | print('MEGA is installed.') 512 | print('') 513 | #clear_output() 514 | #these code above handle mega.nz 515 | 516 | def civitdown(url, folder, torename=''): 517 | filename = url.split('?')[0].rsplit('/', 1)[-1] + ".bdgh" 518 | pathtodown = os.path.join(folder, filename) 519 | max_retries = 5 520 | retry_delay = 10 521 | # url_s = quote(url) 522 | 523 | while prockilled == False: 524 | 525 | downloaded_size = 0 526 | headers = {} 527 | 528 | progress = tqdm(total=1000000000, unit="B", unit_scale=True, desc=f"Downloading {filename}. (will be renamed correctly after downloading)", initial=downloaded_size, leave=False) 529 | global currentsuboutput 530 | global currentcondition 531 | with open(pathtodown, "ab") as f: 532 | while prockilled == False: 533 | try: 534 | response = requests.get(url, headers=headers, stream=True) 535 | total_size = int(response.headers.get("Content-Length", 0)) 536 | # if total_size == 0: 537 | # total_size = downloaded_size 538 | # progress.total = total_size 539 | 540 | 541 | for chunk in response.iter_content(chunk_size=1024): 542 | if chunk and prockilled == False: 543 | f.write(chunk) 544 | progress.update(len(chunk)) 545 | currentsuboutput = str(progress) 546 | else: 547 | break 548 | 549 | downloaded_size = os.path.getsize(pathtodown) 550 | currentsuboutput = '' 551 | break 552 | except ConnectionError as e: 553 | max_retries -= 1 554 | 555 | if max_retries == 0: 556 | raise e 557 | 558 | time.sleep(retry_delay) 559 | 560 | progress.close() 561 | if prockilled == True: 562 | if os.path.exists(pathtodown): 563 | os.remove(pathtodown) 564 | print('Operation Cancelled') 565 | print('') 566 | currentcondition = 'Operation Cancelled' 567 | currentsuboutput = '' 568 | return "Operation Cancelled" 569 | 570 | if torename: 571 | actualfilename = torename 572 | else: 573 | try: 574 | actualfilename = response.headers['Content-Disposition'].split("filename=")[1].strip('"') 575 | except KeyError: 576 | print('Link Error') 577 | print('') 578 | break 579 | #%cd {folder} 580 | actualpath = os.path.join(folder, actualfilename) 581 | os.rename(pathtodown, actualpath) 582 | downloaded_size = os.path.getsize(actualpath) 583 | # Check if the download was successful 584 | if downloaded_size >= total_size: 585 | print(f"{actualfilename} successfully downloaded.") 586 | break 587 | else: 588 | print(f"Error: File download failed. Retrying...") 589 | 590 | def getcivitname(link, frommodeltypechooser=False): #@note getcivitname 591 | # searcher = "findstr" if platform.system() == "Windows" else "grep" 592 | printdebug("getcivitname:") 593 | printvardebug(link) 594 | global currentcondition 595 | tempcondition = currentcondition 596 | currentcondition = "Connecting to CivitAI..." 597 | try: 598 | req = urllib.request.Request(link, headers={'User-Agent': 'Mozilla/5.0'}) 599 | response = urllib.request.urlopen(req, timeout=7) 600 | contentdis = response.geturl() 601 | # contentdis = [line for line in subprocess.getoutput(f"curl -sI {link} | {searcher} -i content-disposition").splitlines() if line.startswith('location')][0] 602 | except Exception as e: 603 | errortype = str(type(e)) 604 | printdebug("Error: " + errortype) 605 | if 'TimeoutError' in errortype: 606 | print("Unable to connect to CivitAI in 7 seconds. Skipping the link...") 607 | currentcondition = tempcondition 608 | return 'batchlinksskip' 609 | else: 610 | if frommodeltypechooser: 611 | currentcondition = tempcondition 612 | return '' 613 | else: 614 | print("Cannot get the filename. Download will continue with the old slow method.\nReason:" + str(e)) 615 | currentcondition = tempcondition 616 | return 'batchlinksold' 617 | if contentdis == "https://civitai.com/": 618 | print('CivitAI website is currently down ツ') 619 | print('') 620 | currentcondition = tempcondition 621 | return 'batchlinksskip' 622 | cuttedcontent = contentdis.find('response-content-disposition=attachment%3B%20filename%3D%22') + 59 623 | # filename = str(contentdis[cuttedcontent:]).replace('%22&x-id=GetObject', '') #obsolete since 30-03-2023 624 | filename = str(contentdis[cuttedcontent:]).partition('%22&')[0] 625 | # filename = contentdis.split('filename=')[-1].split('%22')[1] #this also works i guess 626 | filename = civitmodeltypename(filename, link) 627 | currentcondition = tempcondition 628 | return filename 629 | 630 | def getcivitname2(link): 631 | searcher = "findstr" if platform.system() == "Windows" else "grep" 632 | try: 633 | contentdis = [line for line in subprocess.getoutput(f"curl -sI {link} | {searcher} -i content-disposition").splitlines() if line.startswith('location')][0] 634 | cuttedcontent = contentdis.find('response-content-disposition=attachment%3B%20filename%3D%22') + 59 635 | filename = str(contentdis[cuttedcontent:]).replace('%22&x-id=GetObject', '') 636 | if filename: 637 | return [0, link, filename] 638 | except: 639 | return [] 640 | 641 | def civitmodeltypename(name, filelink): 642 | nameonly, extension = os.path.splitext(name) 643 | if 'type=Pruned' in filelink: 644 | nameonly += "_pruned" 645 | 646 | if 'format=Safetensor' in filelink: 647 | extension = '.safetensors' 648 | elif 'format=PickleTensor' in filelink: 649 | extension = '.ckpt' 650 | 651 | nameoffile = nameonly + extension 652 | return nameoffile 653 | 654 | def checkcivitconfig(link): #check if the current civit link has a config file (v2.0+) @note checkcivitconfig 655 | def getrequest(link2): 656 | response = requests.get(link2) 657 | return response.status_code 658 | 659 | params = ['?type=Config','?type=Config&format=Other'] 660 | 661 | for param in params: 662 | response = getrequest(link + param) 663 | if response == 200: 664 | #print('The link exists') 665 | return link + param 666 | return '' 667 | 668 | def civitmodeltypechooser(modeljson, prunedmodel, torchortensor, linkandnames): 669 | 670 | # prunedorfull = ['?type=Model', '?type=Pruned%20Model'] 671 | # if prunedmodel == 'True': 672 | prunedorfull = ['&fp=fp32', '&fp=fp16'] 673 | if prunedmodel == 'fp16': 674 | prunedorfull.reverse() 675 | pickleorsafe = ['&format=SafeTensor', '&format=PickleTensor'] 676 | if torchortensor=='ckpt': 677 | pickleorsafe.reverse() 678 | 679 | defaultlinkurl = [link.get('downloadUrl') for link in modeljson['modelVersions'][0]['files'] if not '?type=' in link.get('downloadUrl')][0] 680 | 681 | indexlinkname = list() 682 | checklater = '' 683 | for index, (link, name) in enumerate(linkandnames.items()): 684 | if prunedorfull[0] in link: 685 | if pickleorsafe[0] in link: 686 | indexlinkname.extend([index, link, name]) 687 | break 688 | else: 689 | checklater = link 690 | continue 691 | else: 692 | continue 693 | 694 | if checklater and not indexlinkname: 695 | for index, (link, name) in enumerate(linkandnames.items()): 696 | if checklater == link: 697 | indexlinkname.extend([index, link, name]) 698 | checklater = '' 699 | break 700 | 701 | if not indexlinkname: 702 | indexlinkname = getcivitname(defaultlinkurl+prunedorfull[0]+pickleorsafe[0], True) #@note indexlinkname 703 | if indexlinkname == "batchlinksskip" or not indexlinkname: 704 | indexlinkname = list() 705 | 706 | if not indexlinkname: 707 | for index, (link, name) in enumerate(linkandnames.items()): 708 | if defaultlinkurl == link: 709 | indexlinkname.extend([index, link, name]) 710 | break 711 | 712 | return indexlinkname 713 | 714 | #thank you @rti7743 for this part { 715 | def civitdown2_get_json(url): 716 | import re 717 | m = re.search(r'https://civitai.com/models/(\d+)', url) 718 | model_id = m.group(1) 719 | 720 | api_url = "https://civitai.com/api/v1/models/" + model_id 721 | 722 | import requests 723 | txt = requests.get(api_url).text 724 | 725 | import json 726 | try: 727 | return json.loads(txt) 728 | except json.decoder.JSONDecodeError: 729 | return 'error' 730 | 731 | def civitdown2_get_save_directory(model_type, default_folder): 732 | printdebug("model_type: " + model_type) 733 | customtypemains = list(typemain[countofdefaulthashtags:]) 734 | printdebug("customtypemains: " + str(customtypemains)) 735 | for type in customtypemains: 736 | xpath = eval(type + "path") 737 | if default_folder == xpath: 738 | printdebug("customtype returned") 739 | return default_folder 740 | if model_type == "Checkpoint": 741 | if cmd_opts.ckpt_dir and default_folder == altmodelpath: 742 | return altmodelpath 743 | else: 744 | return modelpath 745 | elif model_type == "Hypernetwork": 746 | return hynetpath 747 | elif model_type == "TextualInversion": 748 | return embedpath 749 | elif model_type == "AestheticGradient": 750 | return aestheticembedpath 751 | elif model_type == "VAE": 752 | return vaepath 753 | elif model_type == "LORA": 754 | if default_folder == addnetlorapath: 755 | return addnetlorapath 756 | else: 757 | return lorapath 758 | elif model_type == "LoCon": 759 | return lycorispath 760 | else: 761 | printdebug("defaultfolder returned") 762 | return default_folder 763 | 764 | def civitdown2_convertimage(imagejpg_save_path, imagepng_save_path): 765 | from PIL import Image 766 | try: 767 | img = Image.open(imagejpg_save_path) 768 | img_resized = img.resize((img.width // 2, img.height // 2)) 769 | img_resized.save(imagepng_save_path) 770 | if os.path.exists(imagejpg_save_path): 771 | os.remove(imagejpg_save_path) 772 | except Exception as e: 773 | print("Civitai image convertion failed. Reason: " + str(e)) 774 | pass 775 | 776 | def civitdown2(url, folder, downloader, renamedfilename, isdebugevery, modeldefaulttype, isprunedmodel, isdownvae): #@note civitdown2 777 | def civitlinkandnamer(model): 778 | linkandname = dict() 779 | for i, link in enumerate(model['modelVersions'][0]['files']): 780 | name = link.get('name') 781 | url = link.get('downloadUrl') 782 | if name and url: 783 | linkandname[url] = name 784 | # print(i, url, name) 785 | else: 786 | pass 787 | return linkandname 788 | 789 | model = civitdown2_get_json(url) 790 | if model == 'error': 791 | print('Failed retrieving the model data.') 792 | print('CivitAI website might going down currently.') 793 | print('') 794 | return 795 | 796 | if model == {'error': 'Model not found'}: 797 | print('Model ' + url + ' is not available anymore') 798 | print('') 799 | return 800 | 801 | save_directory = civitdown2_get_save_directory(model['type'], folder) 802 | if not os.path.exists(save_directory): 803 | os.makedirs(save_directory, exist_ok=True) 804 | try: 805 | parameter = url.split("?")[-1] + "?" 806 | except: 807 | parameter = '' 808 | 809 | civitlinkandnames = civitlinkandnamer(model) 810 | 811 | if model['type'] == "Checkpoint": 812 | data_index, data_url, data_filename = civitmodeltypechooser(model, isprunedmodel, modeldefaulttype, civitlinkandnames) #@note 813 | data_filename = civitmodeltypename(data_filename, data_url) 814 | image_filename = model['modelVersions'][0]['files'][0]['name'] 815 | else: 816 | data_url = model['modelVersions'][0]['files'][0]['downloadUrl'] 817 | data_filename = model['modelVersions'][0]['files'][0]['name'] 818 | if "?type=Training%20Data" in data_url: 819 | data_url = data_url.partition("?")[0] 820 | wannabename = getcivitname(data_url) 821 | if wannabename: 822 | if wannabename != 'batchlinksskip': 823 | data_filename = wannabename 824 | else: 825 | return '' 826 | image_filename = data_filename 827 | 828 | if renamedfilename: 829 | data_filename = renamedfilename 830 | image_filename = data_filename 831 | 832 | image_url = model['modelVersions'][0]['images'][0]['url'] 833 | printvardebug(data_url) 834 | printvardebug(data_filename) 835 | printvardebug(image_url) 836 | if model['type'] == "TextualInversion": 837 | image_filename_jpg = pathlib.PurePath(image_filename).stem + ".preview.jpg" 838 | image_filename_png = pathlib.PurePath(image_filename).stem + ".preview.png" 839 | else: 840 | image_filename_jpg = pathlib.PurePath(image_filename).stem + ".jpg" 841 | image_filename_png = pathlib.PurePath(image_filename).stem + ".png" 842 | 843 | data_save_path = os.path.join(save_directory, data_filename) 844 | imagejpg_save_path = os.path.join(save_directory, image_filename_jpg) 845 | imagepng_save_path = os.path.join(save_directory, image_filename_png) 846 | 847 | currentmode = 'civit' 848 | if isdebugevery: 849 | currentmode = 'civitdebugevery' 850 | 851 | printdebug(f"debug download_url({data_url}, {data_save_path}, {downloader})") 852 | if prockilled == False: 853 | hfdown(data_url, data_save_path, downloader, currentmode) 854 | printdebug("normal download done, now check for the config") 855 | config_url = checkcivitconfig(data_url) 856 | vae_url, vae_name = '','' 857 | for link, name in civitlinkandnames.items(): 858 | if '?type=VAE' in link: 859 | vae_url, vae_name = link, name 860 | break 861 | if model['type'] == "Checkpoint": 862 | if prockilled == False and isdownvae and vae_url: 863 | namefile = os.path.splitext(data_filename)[0] 864 | vae_name = namefile + '.vae.pt' 865 | data_save_path = os.path.join(os.path.dirname(data_save_path), vae_name) 866 | hfdown(vae_url, data_save_path, downloader, currentmode) 867 | if prockilled == False and config_url: 868 | namefile = os.path.splitext(data_filename)[0] 869 | config_name = namefile + '.yaml' 870 | data_save_path = os.path.join(os.path.dirname(data_save_path), config_name) 871 | hfdown(config_url, data_save_path, downloader, currentmode) 872 | 873 | if prockilled == False: 874 | hfdown(image_url, imagejpg_save_path, downloader, currentmode) 875 | if os.path.exists(imagejpg_save_path): 876 | civitdown2_convertimage(imagejpg_save_path, imagepng_save_path) 877 | else: 878 | print(imagejpg_save_path + " doesn't exist") 879 | if prockilled == False: 880 | print(f"{data_save_path} successfully downloaded.") 881 | 882 | if isdebugevery: 883 | additionalname = '-' + downloader 884 | 885 | if model['type'] == "TextualInversion": 886 | image_filename_jpg = pathlib.PurePath(data_filename).stem + additionalname + ".preview.jpg" 887 | image_filename_png = pathlib.PurePath(data_filename).stem + additionalname + ".preview.png" 888 | else: 889 | image_filename_jpg = pathlib.PurePath(data_filename).stem + additionalname + ".jpg" 890 | image_filename_png = pathlib.PurePath(data_filename).stem + additionalname + ".png" 891 | 892 | imagejpg_save_path = os.path.join(save_directory, image_filename_jpg) 893 | imagepng_save_path = os.path.join(save_directory, image_filename_png) 894 | 895 | #} 896 | def mediadrivedown(todownload, folder, mode, torename=''): #@note mediadrivedown 897 | todownload_s = shlex.quote(todownload) 898 | if platform.system() == "Windows": 899 | folder_s = folder 900 | else: 901 | folder_s = shlex.quote(folder) 902 | prevcurdir = os.getcwd() 903 | os.chdir(folder) 904 | savestate_folder(folder_s) 905 | 906 | if mode=='gdrive': 907 | runwithsubprocess(f"gdown --fuzzy {todownload_s}", folder_s) 908 | elif mode=='mediafire': 909 | runwithsubprocess(f"mediafire-dl {todownload_s}", folder_s) 910 | 911 | os.chdir(prevcurdir) 912 | if torename: 913 | listfilenew = os.listdir(folder) 914 | newerfoldertrack = [] 915 | for file in listfilenew: 916 | pathoffile = os.path.join(folder, file) 917 | newerfoldertrack.append(pathoffile) 918 | checkrename = [x for x in newerfoldertrack if x not in currentfoldertrack] 919 | if checkrename: 920 | os.rename(checkrename[0], os.path.join(folder, torename)) 921 | 922 | def hfdown(todownload, folder, downloader, mode='default', torename=''): #@note hfdown 923 | global currentcondition 924 | global prevciviterror 925 | prevciviterror = False 926 | # global ariamode 927 | if mode=='civit' or mode=='civitdebugevery': 928 | filename = pathlib.Path(folder).name 929 | filename_s = shlex.quote(filename) 930 | filepath = folder 931 | filepath_s = shlex.quote(folder) 932 | todownload_s = todownload 933 | folder = os.path.dirname(filepath) 934 | # folder_s = pathlib.Path(folder).parent.resolve() 935 | folder_s = shlex.quote(folder) 936 | folder_win = folder 937 | else: 938 | if mode=='pixeldrain' or mode=='dropbox': 939 | filename = torename 940 | else: 941 | filename = todownload.rsplit('/', 1)[-1] 942 | filename_s = shlex.quote(filename) 943 | filepath = os.path.join(folder, filename) 944 | filepath_s = shlex.quote(filepath) 945 | todownload_s = shlex.quote(todownload) 946 | folder_s = shlex.quote(folder) 947 | folder_win = folder 948 | ariafilename_s = filename_s 949 | if torename: 950 | ariafilename_s = shlex.quote(torename) 951 | 952 | printvardebug(folder) 953 | printvardebug(todownload) 954 | printvardebug(filename) 955 | printvardebug(filename_s) 956 | printvardebug(filepath) 957 | printvardebug(filepath_s) 958 | printvardebug(todownload_s) 959 | printvardebug(folder_s) 960 | printvardebug(folder_win) 961 | #savestate_folder(folder_s) 962 | 963 | if platform.system() == "Windows": #@note windows downloader 964 | localappdata = os.environ['LOCALAPPDATA'] 965 | batchlinksinstallpath = os.path.join(localappdata, "batchlinks") 966 | printvardebug(batchlinksinstallpath) 967 | wgetpath = os.path.join(batchlinksinstallpath, "wget-gnutls-x64.exe") 968 | 969 | ariazippath = os.path.join(batchlinksinstallpath, "aria2-1.36.0-win-64bit-build2.7z") 970 | ariapath = os.path.join(batchlinksinstallpath, "aria2-1.36.0-win-64bit-build2", "aria2c.exe") 971 | os.makedirs(batchlinksinstallpath, exist_ok=True) 972 | if os.path.exists(batchlinksinstallpath): 973 | printdebug("batchlinksinstallpath created") 974 | if downloader=='gdown': 975 | global gdownupgraded 976 | if gdownupgraded == False: 977 | tempcondition = currentcondition 978 | currentcondition = "Upgrading gdown..." 979 | print('Upgrading gdown ...') 980 | print('') 981 | runwithsubprocess(f"pip3 install -q --upgrade --no-cache-dir gdown") 982 | print('gdown upgraded!') 983 | print('') 984 | currentcondition = tempcondition 985 | gdownupgraded = True 986 | try: 987 | runwithsubprocess(f"gdown {todownload_s} -O {filepath_s}", folder_win) 988 | except FileNotFoundError: 989 | import gdown 990 | gdown.download(todownload, filepath, quiet=False) 991 | elif downloader=='wget': 992 | #os.system("python -m wget -o " + os.path.join(folder, filename) + " " + todownload) 993 | # import wget 994 | # wget.download(todownload, filepath) 995 | # checkwget = subprocess.getoutput(wgetpath) 996 | # if "is not recognized" in checkwget: 997 | if not os.path.exists(wgetpath): 998 | tempcondition = currentcondition 999 | currentcondition = "Installing wget Windows..." 1000 | print('Installing wget Windows...') 1001 | print('') 1002 | wgetwinlink = "https://github.com/webfolderio/wget-windows/releases/download/v1.21.3.june.19.2022/wget-gnutls-x64.exe" 1003 | print(wgetwinlink) 1004 | runwithsubprocess("curl -Lo " + shlex.quote(wgetpath) + " " + wgetwinlink, batchlinksinstallpath) 1005 | print('wget Windows installed!') 1006 | print('') 1007 | currentcondition = tempcondition 1008 | runwithsubprocess(f"{shlex.quote(wgetpath)} -O {filepath_s} {todownload_s} --progress=bar:force", folder_win) 1009 | elif downloader=='curl': 1010 | runwithsubprocess(f"curl -Lo {filepath_s} {todownload_s}", folder_win) 1011 | elif downloader=='aria2': 1012 | # checkaria = subprocess.getoutput(ariapath) 1013 | # if "is not recognized" in checkaria or "cannot find the path" in checkaria: 1014 | if not os.path.exists(ariapath): 1015 | tempcondition = currentcondition 1016 | currentcondition = "Installing aria2 Windows..." 1017 | print('Installing aria2 Windows...') 1018 | print('') 1019 | ariaziplink = "https://github.com/q3aql/aria2-static-builds/releases/download/v1.36.0/aria2-1.36.0-win-64bit-build2.7z" 1020 | print(ariaziplink) 1021 | runwithsubprocess("curl -Lo " + shlex.quote(ariazippath) + " " + ariaziplink, batchlinksinstallpath) 1022 | sevenzpath = install7zWin() 1023 | runwithsubprocess(f"{shlex.quote(sevenzpath)} x {shlex.quote(ariazippath)} -o{shlex.quote(batchlinksinstallpath)}", batchlinksinstallpath, False, '7z') 1024 | print() 1025 | print('aria2 Windows installed!') 1026 | print('') 1027 | currentcondition = tempcondition 1028 | # ariamode = True 1029 | runwithsubprocess(f"{shlex.quote(ariapath)} --summary-interval=1 --console-log-level=error --check-certificate=false -c -x 16 -s 16 -k 1M {todownload_s} -d {folder_s} -o {ariafilename_s}", folder_win, False, 'aria2') 1030 | else: #non-windows 1031 | if downloader=='gdown': 1032 | printdebug(f"debug gdown {todownload_s} -O {filepath_s}") 1033 | runwithsubprocess(f"gdown {todownload_s} -O {filepath_s}", folder_s) 1034 | elif downloader=='wget': 1035 | runwithsubprocess(f"wget -O {filepath_s} {todownload_s} --progress=bar:force", folder_s) 1036 | elif downloader=='curl': 1037 | runwithsubprocess(f"curl -Lo {filepath_s} {todownload_s}", folder_s) 1038 | # curdir = os.getcwd() 1039 | # os.rename(os.path.join(curdir, filename), filepath) 1040 | elif downloader=='aria2': 1041 | ariachecker = "dpkg-query -W -f='${Status}' aria2" 1042 | checkaria = subprocess.getoutput(ariachecker) 1043 | if "no packages found matching aria2" in checkaria: 1044 | tempcondition = currentcondition 1045 | currentcondition = "Installing aria2..." 1046 | print('Installing aria2 ...') 1047 | print('') 1048 | runwithsubprocess(f"apt -y update -qq") 1049 | runwithsubprocess(f"apt -y install -qq aria2") 1050 | print('aria2 installed!') 1051 | print('') 1052 | currentcondition = tempcondition 1053 | # ariamode = True 1054 | runwithsubprocess(f"aria2c --summary-interval=1 --console-log-level=error -c -x 16 -s 16 -k 1M {todownload_s} -d {folder_s} -o {ariafilename_s}", folder_s, False, 'aria2') 1055 | printdebug("\nmode: " + mode) 1056 | if mode=='debugevery': 1057 | time.sleep(2) 1058 | try: 1059 | os.rename(filepath, os.path.join(folder, f"{os.path.splitext(filename)[0]}-{downloader}{os.path.splitext(filename)[1]}")) 1060 | printdebug("renamed to " + f"{os.path.splitext(filename)[0]}-{downloader}{os.path.splitext(filename)[1]}") 1061 | except FileNotFoundError or FileExistsError: 1062 | printdebug("rename failed somehow") 1063 | pass 1064 | elif mode=='civitdebugevery': 1065 | time.sleep(2) 1066 | printdebug("debug filename: " + str(filename)) 1067 | printdebug("debug filename_s: " + str(filename_s)) 1068 | printdebug("debug filepath: " + str(filepath_s)) 1069 | printdebug("debug todownload_s: " + str(todownload_s)) 1070 | printdebug("debug folder_s: " + str(folder_s)) 1071 | try: 1072 | os.rename(folder, os.path.join(folder_s, f"{os.path.splitext(filename)[0]}-{downloader}{os.path.splitext(filename)[1]}")) 1073 | printdebug("renamed to " + f"{os.path.splitext(filename)[0]}-{downloader}{os.path.splitext(filename)[1]}") 1074 | except FileNotFoundError or FileExistsError: 1075 | printdebug("rename failed somehow") 1076 | pass 1077 | if torename and prockilled == False and filename != torename and downloader != 'aria2': 1078 | def saferename(oldpath, newpath): 1079 | try: 1080 | os.rename(oldpath, newpath) 1081 | except Exception as e: 1082 | print("Rename failed. Reason: " + str(e)) 1083 | pass 1084 | 1085 | saferename(filepath, os.path.join(folder, shlex.quote(torename))) 1086 | if mode=='civit0' and prockilled == False: 1087 | # if torename: 1088 | # filepath = os.path.join(folder, torename) 1089 | try: 1090 | if os.path.exists(filepath): 1091 | if os.path.getsize(filepath) <= 5 * 1024: 1092 | try: 1093 | with open(filepath, "r", encoding="utf-8") as file: 1094 | if "We'll be right back | Civitai" in file.read(): 1095 | prevciviterror = True 1096 | print('CivitAI website is currently down ツ') 1097 | print('') 1098 | except Exception as e: 1099 | print("File size checking failed. Reason: " + str(e)) 1100 | pass 1101 | except Exception as e: 1102 | print("File size checking failed. Reason: " + str(e)) 1103 | pass 1104 | if mode=='dropbox' and prockilled == False: 1105 | if os.path.exists(filepath): 1106 | if os.path.getsize(filepath) <= 1024: 1107 | print('Dropbox filesize below 1kb') 1108 | print("There's a chance that the file got too much traffic and dropbox blocked access to it") 1109 | print('') 1110 | # if prockilled == True: 1111 | # #rewind_folder(folder_s) 1112 | # pass 1113 | 1114 | def install7zWin(): #@note install7z 1115 | #usage: sevenzpath = install7zWin() 1116 | global currentcondition 1117 | tempcondition = currentcondition 1118 | currentcondition = "Installing 7z..." 1119 | localappdata = os.environ['LOCALAPPDATA'] 1120 | batchlinksinstallpath = os.path.join(localappdata, "batchlinks") 1121 | sevenzrpath = os.path.join(batchlinksinstallpath, "7zr.exe") 1122 | sevenzrlink = "https://www.7-zip.org/a/7zr.exe" 1123 | svnzpacklink = "https://7-zip.org/a/7z2201-x64.exe" 1124 | svnzpackpath = os.path.join(batchlinksinstallpath, "7z2201-x64.exe") 1125 | svnzexecpath = os.path.join(batchlinksinstallpath, "7z.exe") 1126 | if not os.path.exists(sevenzrpath): 1127 | print(sevenzrlink) 1128 | runwithsubprocess("curl -Lo " + shlex.quote(sevenzrpath) + " " + sevenzrlink, batchlinksinstallpath) 1129 | if not os.path.exists(svnzpackpath): 1130 | print(svnzpacklink) 1131 | runwithsubprocess("curl -Lo " + shlex.quote(svnzpackpath) + " " + svnzpacklink, batchlinksinstallpath) 1132 | if not os.path.exists(svnzexecpath): 1133 | runwithsubprocess(f"{shlex.quote(sevenzrpath)} x {shlex.quote(svnzpackpath)} -p- -o{shlex.quote(batchlinksinstallpath)} -y -sdel -bb0", batchlinksinstallpath, False, '7z') 1134 | currentcondition = tempcondition 1135 | return svnzexecpath 1136 | 1137 | def savestate_folder(folder): 1138 | global currentfoldertrack 1139 | currentfoldertrack = [] 1140 | listfile = os.listdir(folder) 1141 | for file in listfile: 1142 | pathoffile = os.path.join(folder, file) 1143 | currentfoldertrack.append(pathoffile) 1144 | 1145 | def rewind_folder(folder): 1146 | listfilenew = os.listdir(folder) 1147 | newerfoldertrack = [] 1148 | for file in listfilenew: 1149 | pathoffile = os.path.join(folder, file) 1150 | newerfoldertrack.append(pathoffile) 1151 | toremove = [x for x in newerfoldertrack if x not in currentfoldertrack] 1152 | printdebug("\ndebug toremove: " + str(toremove)) 1153 | print() 1154 | for fileordir in toremove: 1155 | if os.path.exists(fileordir): 1156 | if os.path.isdir(fileordir): 1157 | shutil.rmtree(fileordir) 1158 | print("Removed incomplete download: " + fileordir) 1159 | else: 1160 | os.remove(fileordir) 1161 | print("Removed incomplete download: " + fileordir) 1162 | 1163 | def writeall(olddict, shellonly, custompaths=''): 1164 | newdict = trackall() 1165 | global finalwrite 1166 | finalwrite = [] 1167 | 1168 | finalwrite.append("All done!") 1169 | if custompaths: 1170 | finalwrite.append("⬇️Custom path added:⬇️") 1171 | for hashtag, thepath in custompaths.items(): 1172 | finalwrite.append(f"{hashtag} -> {thepath}") 1173 | finalwrite.append("Downloaded files: ") 1174 | 1175 | for oldtype, olddir in olddict.items(): 1176 | for newtype, newdir in newdict.items(): 1177 | if newtype == oldtype: 1178 | s = set(olddir) 1179 | trackcompare = [x for x in newdir if x not in s] 1180 | if len(trackcompare) > 0: 1181 | exec(f"finalwrite.append('⬇️' + {newtype}path + '⬇️')") 1182 | for item in trackcompare: 1183 | finalwrite.append(item) 1184 | if bool(remaininglinks): 1185 | finalwrite.append("(There are still some files that have not been downloaded. Click the 'Resume Download' button to load the links that haven't been downloaded.)") 1186 | printvardebug(finalwrite) 1187 | 1188 | if len(finalwrite) > 1 and "Downloaded files:" in finalwrite[1] and len(finalwrite) < 3: 1189 | finalwrite.append("(Not finding something here? Check the terminal/colab console)") 1190 | 1191 | finaloutput = list_to_text(finalwrite) 1192 | finalwrite = [] 1193 | if shellonly: 1194 | return "Commands executed successfully." 1195 | else: 1196 | return finaloutput 1197 | 1198 | def writepart(box, path): 1199 | global finalwrite 1200 | if len(box) > 0: 1201 | finalwrite.append("⬇️" + path + "⬇️") 1202 | for item in box: 1203 | finalwrite.append(item) 1204 | 1205 | def trackall(): 1206 | filesdict = dict() 1207 | for x in typemain: 1208 | if x == "altmodel" and altmodelpath == modelpath: 1209 | continue 1210 | exec(f"os.makedirs({x}path, exist_ok=True)") 1211 | exec(f"filesdict['{x}'] = os.listdir({x}path)") 1212 | return filesdict 1213 | 1214 | def currentfoldertohashtag(folder): 1215 | for x in typemain: 1216 | checkpath = str() 1217 | checkpath = eval(x+'path') 1218 | printdebug("checkpath: " + checkpath) 1219 | if str(folder).strip() == checkpath: 1220 | thehashtag = "#" + x 1221 | printdebug("thehashtag: " + thehashtag) 1222 | return thehashtag 1223 | return "#debug" 1224 | 1225 | def splitrename(linkcurrent): 1226 | renamecurrent = '' 1227 | if ">" in linkcurrent: 1228 | file_rename = linkcurrent.split(">") 1229 | file_rename = [file_rename.strip() for file_rename in file_rename] 1230 | linkcurrent = file_rename[0] 1231 | if file_rename[1]: 1232 | renamecurrent = file_rename[1] 1233 | return linkcurrent, renamecurrent 1234 | 1235 | def extractcurdir(currentdir): #@note extractcurdir 1236 | allfileshere = os.listdir(currentdir) 1237 | szfileall = [] 1238 | if platform.system() == "Windows": 1239 | sevenzpath = install7zWin() 1240 | global currentcondition 1241 | extensiontoextract = [".zip", ".rar", ".7z", ".tar"] 1242 | for filehere in allfileshere: 1243 | if filehere.endswith(tuple(extensiontoextract)): 1244 | szpath = os.path.join(currentdir, filehere) 1245 | szfileall.append(szpath) 1246 | for szfile in szfileall: 1247 | if prockilled == False: 1248 | # currentcondition = "Extracting" + os.path.basename(szfile) + "..." 1249 | if platform.system() == "Windows": 1250 | runwithsubprocess(f"{shlex.quote(sevenzpath)} x {shlex.quote(szfile)} -p- -o{shlex.quote(currentdir)} -y -sdel -bb0", currentdir, False, '7z') 1251 | else: 1252 | runwithsubprocess(f"7z x {shlex.quote(szfile)} -p- -o{shlex.quote(currentdir)} -y -sdel -bb0", currentdir, False, '7z') 1253 | 1254 | #@stopwatch #the decorator mess with the progress bar #@note run 1255 | def run(command, choosedowner, civitdefault, civitpruned, civitvae, progress=gr.Progress()): 1256 | progress(0.01, desc='') 1257 | global prockilled 1258 | prockilled = False 1259 | global everyprocessid 1260 | everyprocessid = [] 1261 | everymethod = False 1262 | global currentcondition 1263 | resumebuttonvisible = False 1264 | if command.strip().startswith('https://pastebin.com/') and command.strip().count('\n') == 0: 1265 | currentcondition = f'Done.' 1266 | if gradiostate == True: 1267 | return ["Use the 'Copy from Pastebin' button instead", gr.Dataframe.update(value=buildarrayofhashtags('bottom'))] #gr.Dataframe.update(value=buildarrayofhashtags('right')), 1268 | else: 1269 | return ["Use the 'Copy from Pastebin' button instead", gr.Dataframe.update(value=buildarrayofhashtags('bottom')), gr.Button.update(visible=resumebuttonvisible)] #gr.Dataframe.update(value=buildarrayofhashtags('right')), 1270 | 1271 | if command.strip() == '@debugresetdownloads' and snapshot != {} and globaldebug == True: 1272 | currentcondition = f'Removing downloaded files...' 1273 | removed_files = global_rewind() 1274 | texttowrite = ["⬇️Removed files⬇️"] 1275 | for item in removed_files: 1276 | texttowrite.append(item) 1277 | writefinal = list_to_text(texttowrite) 1278 | currentcondition = f'Removing done.' 1279 | if gradiostate == True: 1280 | return [writefinal, gr.Dataframe.update(value=buildarrayofhashtags('bottom'))] #gr.Dataframe.update(value=buildarrayofhashtags('right')), 1281 | else: 1282 | return [writefinal, gr.Dataframe.update(value=buildarrayofhashtags('bottom')), gr.Button.update(visible=resumebuttonvisible)] #gr.Dataframe.update(value=buildarrayofhashtags('right')), 1283 | 1284 | if not command.strip(): 1285 | currentcondition = "Logging activated." 1286 | texttowrite = ["The link box is empty."] 1287 | writefinal = list_to_text(texttowrite) 1288 | if gradiostate == True: 1289 | return [writefinal, gr.Dataframe.update(value=buildarrayofhashtags('bottom'))] #gr.Dataframe.update(value=buildarrayofhashtags('right')) 1290 | else: 1291 | return [writefinal, gr.Dataframe.update(value=buildarrayofhashtags('bottom')), gr.Button.update(visible=resumebuttonvisible)] #gr.Dataframe.update(value=buildarrayofhashtags('right')), 1292 | oldfilesdict = trackall() 1293 | if cmd_opts.ckpt_dir: 1294 | altmodelpath = cmd_opts.ckpt_dir 1295 | currentfolder = altmodelpath 1296 | else: 1297 | altmodelpath = os.path.join(models_path, "Stable-diffusion") 1298 | currentfolder = modelpath 1299 | currenthashtag = '#model' 1300 | os.makedirs(currentfolder, exist_ok=True) 1301 | currentcondition = 'Extracting links...' 1302 | links = extract_links(command) 1303 | isshell = True 1304 | for listpart in links: 1305 | if not listpart.startswith('!'): 1306 | isshell = False 1307 | break 1308 | printdebug("links: " + str(links)) 1309 | steps = float(0) 1310 | totalsteps = float(1) 1311 | 1312 | usemega = False 1313 | usegdrive = False 1314 | usemediafire = False 1315 | # global ariamode 1316 | global gdownupgraded 1317 | global mediafireinstalled 1318 | for item in links: 1319 | if not item.startswith('#'): 1320 | totalsteps += 1 1321 | if item.startswith('https://mega.nz'): 1322 | usemega = True 1323 | if item.startswith('https://drive.google.com'): 1324 | usegdrive = True 1325 | if item.startswith('https://www.mediafire.com/file'): 1326 | usemediafire = True 1327 | 1328 | #break 1329 | printdebug("totalsteps: " + str(totalsteps)) 1330 | if usemega == True: 1331 | currentcondition = 'Installing Mega...' 1332 | progress(0.01, desc='Installing Mega...') 1333 | if platform.system() == "Windows": 1334 | installmegawin() 1335 | else: 1336 | installmega() 1337 | if usegdrive == True and gdownupgraded == False: 1338 | tempcondition = currentcondition 1339 | currentcondition = "Upgrading gdown..." 1340 | print('Upgrading gdown ...') 1341 | print('') 1342 | runwithsubprocess(f"pip3 install -q --upgrade --no-cache-dir gdown") 1343 | print('gdown upgraded!') 1344 | print('') 1345 | currentcondition = tempcondition 1346 | gdownupgraded = True 1347 | if usemediafire == True and mediafireinstalled == False: 1348 | tempcondition = currentcondition 1349 | currentcondition = "Installing mediafire-dl..." 1350 | print('Installing mediafire-dl...') 1351 | print('') 1352 | runwithsubprocess(f"pip3 install git+https://github.com/Juvenal-Yescas/mediafire-dl") 1353 | print('mediafire-dl installed!') 1354 | print('') 1355 | currentcondition = tempcondition 1356 | mediafireinstalled = True 1357 | 1358 | print('BatchLinks Downloads starting...') 1359 | print('') 1360 | printdebug('prockilled: ' + str(prockilled)) 1361 | global remaininglinks 1362 | batchtime = time.time() 1363 | addedcustompath = dict() 1364 | downmethod = ['gdown', 'wget', 'curl', 'aria2'] 1365 | hfmethods = [ 1366 | "https://raw.githubusercontent.com", 1367 | "https://huggingface.co", 1368 | "https://cdn.discordapp.com/attachments" 1369 | ] 1370 | global typemain 1371 | global typechecker 1372 | for listpart in links: 1373 | if prockilled == False: 1374 | currenttorename = '' 1375 | printdebug("steps: " + str(steps)) 1376 | printdebug("total steps: " + str(totalsteps)) 1377 | printdebug("percentage: " + str(round(steps/totalsteps, 1))) 1378 | printdebug("currenttypemain: " + str(typemain)) 1379 | # ariamode = False 1380 | if gradiostate == False: 1381 | if time.time() - batchtime >= 70: 1382 | remaininglinks = links[links.index(listpart):] 1383 | printdebug("remaining links: " + str(remaininglinks)) 1384 | if bool(remaininglinks): 1385 | printdebug("currentfolder: " + currentfolder) 1386 | tophashtag = currentfoldertohashtag(currentfolder) 1387 | printdebug("tophashtag: " + tophashtag) 1388 | remaininglinks.insert(0, tophashtag) 1389 | printdebug("remaining links new: " + str(remaininglinks)) 1390 | print() 1391 | print('Runtime was stopped to prevent hangs.') 1392 | print("Check the UI and press 'Resume Download' to load the remaining links") 1393 | print("Then click 'Download All!' again") 1394 | print('') 1395 | print("These are some links that haven't been downloaded yet.👇") 1396 | printremains = list_to_text(remaininglinks) 1397 | print(printremains) 1398 | resumebuttonvisible = True 1399 | cancelrun() 1400 | break 1401 | 1402 | if listpart.startswith("https://mega.nz"): 1403 | currentlink, currenttorename = splitrename(listpart) 1404 | print('\n') 1405 | print(currentlink) 1406 | currentcondition = f'Downloading {currentlink}...' 1407 | progress(round(steps/totalsteps, 3), desc='Downloading from ' + os.path.basename(currentlink).split('#')[0] + f' into {currenthashtag}...') 1408 | transfare(currentlink, currentfolder, currenttorename) 1409 | steps += 1 1410 | 1411 | elif listpart.startswith(tuple(hfmethods)): 1412 | # elif listpart.startswith("https://huggingface.co") or listpart.startswith("https://raw.githubusercontent.com") or listpart.startswith("https://cdn.discordapp.com/attachments"): 1413 | currentlink, currenttorename = splitrename(listpart) 1414 | print('\n') 1415 | print(currentlink) 1416 | currentcondition = f'Downloading {currentlink}...' 1417 | progress(round(steps/totalsteps, 3), desc='Downloading ' + os.path.basename(currentlink) + f' into {currenthashtag}...') 1418 | if everymethod == False: 1419 | hfdown(currentlink, currentfolder, choosedowner, 'default', currenttorename) 1420 | else: 1421 | for xmethod in downmethod: 1422 | if prockilled == False: 1423 | hfdown(currentlink, currentfolder, xmethod, 'debugevery') 1424 | steps += 1 1425 | 1426 | elif listpart.startswith("https://www.dropbox.com/s"): #@note dropbox 1427 | currentlink, currenttorename = splitrename(listpart) 1428 | if currentlink.endswith("?dl=0"): 1429 | currentlink = currentlink.split("?dl=")[0] + "?dl=1" 1430 | realname = currentlink.split("/")[-1].split("?dl=")[0] 1431 | if currenttorename == '': 1432 | currenttorename = realname 1433 | print('\n') 1434 | print(currentlink) 1435 | currentcondition = f'Downloading {currentlink}...' 1436 | progress(round(steps/totalsteps, 3), desc='Downloading ' + currenttorename + f' into {currenthashtag}...') 1437 | hfdown(currentlink, currentfolder, choosedowner, 'dropbox', currenttorename) 1438 | steps += 1 1439 | 1440 | elif listpart.startswith("https://drive.google.com"): 1441 | currentlink, currenttorename = splitrename(listpart) 1442 | print('\n') 1443 | print(currentlink) 1444 | currentcondition = f'Downloading {currentlink}...' 1445 | match = re.search(r'\?id=([^\&]+)\&export|/d/([^\s/]+)/', currentlink) 1446 | if match: 1447 | if match.group(1): 1448 | extracted_string = match.group(1) 1449 | else: 1450 | extracted_string = match.group(2) 1451 | progress(round(steps/totalsteps, 3), desc='Downloading from ' + extracted_string + f' into {currenthashtag}...') 1452 | mediadrivedown(currentlink, currentfolder, 'gdrive', currenttorename) 1453 | steps += 1 1454 | 1455 | elif listpart.startswith("https://pixeldrain.com"): #@note pixeldrain 1456 | currentlink, currenttorename = splitrename(listpart) 1457 | if not currentlink.startswith("https://pixeldrain.com/api/file/"): 1458 | fileid = currentlink.split("/")[-1] 1459 | currentlink = f"https://pixeldrain.com/api/file/{fileid}" 1460 | currentcondition = f'Retrieving Pixeldrain link...' 1461 | searcher = "findstr" if platform.system() == "Windows" else "grep" 1462 | try: 1463 | filename = subprocess.getoutput(f"curl -sI {currentlink} | {searcher} -i Content-Disposition").split('filename="', 1)[1].rsplit('"', 1)[0] 1464 | except IndexError: 1465 | print("Something wrong while retrieving the Pixeldrain link") 1466 | continue 1467 | if currenttorename == '': 1468 | currenttorename = filename 1469 | print('\n') 1470 | print(currentlink) 1471 | currentcondition = f'Downloading {currentlink}...' 1472 | progress(round(steps/totalsteps, 3), desc='Downloading ' + currenttorename + f' into {currenthashtag}...') 1473 | # if everymethod == False: 1474 | hfdown(currentlink, currentfolder, choosedowner, 'pixeldrain', currenttorename) 1475 | # else: 1476 | # for xmethod in downmethod: 1477 | # if prockilled == False: 1478 | # hfdown(currentlink, currentfolder, xmethod, 'debugevery') 1479 | steps += 1 1480 | 1481 | elif listpart.startswith("https://www.mediafire.com/file"): #@note mediafire 1482 | currentlink, currenttorename = splitrename(listpart) 1483 | print('\n') 1484 | print(currentlink) 1485 | currentcondition = f'Downloading {currentlink}...' 1486 | progress(round(steps/totalsteps, 3), desc='Downloading ' + currentlink.split("/")[-2] + f' into {currenthashtag}...') 1487 | # if everymethod == False: 1488 | mediadrivedown(currentlink, currentfolder, 'mediafire', currenttorename) 1489 | # else: 1490 | # for xmethod in downmethod: 1491 | # if prockilled == False: 1492 | # hfdown(currentlink, currentfolder, xmethod, 'debugevery') 1493 | steps += 1 1494 | 1495 | elif listpart.startswith("https://anonfiles.com"): #@note anonfiles 1496 | currentlink, currenttorename = splitrename(listpart) 1497 | print('\n') 1498 | print(currentlink) 1499 | 1500 | currentcondition = f'Retrieving anonfiles link...' 1501 | # print(filename) 1502 | # Send HTTP request to the website and read the response 1503 | response = urllib.request.urlopen(currentlink) 1504 | html_content = response.read().decode('utf-8') 1505 | 1506 | # Find all links that contain "anonfiles" in them using regular expressions 1507 | download_links = re.findall(r'href=["\'](https?:\/\/.*?anonfiles.*?)["\']', html_content) 1508 | currentlink = max(download_links, key=len) 1509 | 1510 | currentcondition = f'Downloading {currentlink}...' 1511 | progress(round(steps/totalsteps, 3), desc='Downloading ' + currentlink.split("/")[-1] + f' into {currenthashtag}...') 1512 | if everymethod == False: 1513 | hfdown(currentlink, currentfolder, choosedowner, 'default', currenttorename) 1514 | else: 1515 | for xmethod in downmethod: 1516 | if prockilled == False: 1517 | hfdown(currentlink, currentfolder, xmethod, 'debugevery') 1518 | steps += 1 1519 | 1520 | elif listpart.startswith("https://files.catbox.moe"): 1521 | currentlink, currenttorename = splitrename(listpart) 1522 | print('\n') 1523 | print(currentlink) 1524 | try: 1525 | urllib.request.urlopen(currentlink) 1526 | except http.client.RemoteDisconnected: 1527 | print('Connection to ' + currentlink + ' failed.') 1528 | print("This colab session's server might doesn't have access to catbox") 1529 | print('') 1530 | continue 1531 | except urllib.error.URLError as e: 1532 | print('Connection to ' + currentlink + ' failed.') 1533 | print("This colab session's server might doesn't have access to catbox") 1534 | print('') 1535 | continue 1536 | print(currentlink) 1537 | currentcondition = f'Downloading {currentlink}...' 1538 | progress(round(steps/totalsteps, 3), desc='Downloading ' + os.path.basename(currentlink) + f' into {currenthashtag}...') 1539 | if everymethod == False: 1540 | hfdown(currentlink, currentfolder, choosedowner, 'default', currenttorename) 1541 | else: 1542 | for xmethod in downmethod: 1543 | if prockilled == False: 1544 | hfdown(currentlink, currentfolder, xmethod, 'debugevery') 1545 | steps += 1 1546 | 1547 | elif listpart.startswith("https://civitai.com/api/download/models/"): #@note civit direct 1548 | usenewmethod = True 1549 | currentlink, currenttorename = splitrename(listpart) 1550 | if currenttorename == '': 1551 | currenttorename = getcivitname(listpart) 1552 | if currenttorename == 'batchlinksold': 1553 | usenewmethod = False 1554 | elif currenttorename == 'batchlinksskip': 1555 | continue 1556 | # print("That CivitAI link no longer exist, or the server is currently down.") 1557 | # continue 1558 | 1559 | print('\n') 1560 | print(currentlink) 1561 | currentcondition = f'Downloading {currentlink}...' 1562 | if everymethod == False: 1563 | if usenewmethod: 1564 | progress(round(steps/totalsteps, 3), desc='Downloading ' + currenttorename + f' into {currenthashtag}...') 1565 | hfdown(currentlink, currentfolder, choosedowner, 'civit0', currenttorename) 1566 | else: 1567 | progress(round(steps/totalsteps, 3), desc='Downloading model number ' + os.path.basename(currentlink) + '...') 1568 | civitdown(currentlink, currentfolder, currenttorename) 1569 | if not prevciviterror: 1570 | configlink = checkcivitconfig(currentlink) 1571 | if not configlink=='': 1572 | namefile= os.path.splitext(currenttorename.split('?')[0])[0] 1573 | currenttorename = namefile + '.yaml' 1574 | hfdown(configlink, currentfolder, choosedowner, 'civit0', currenttorename) 1575 | else: 1576 | for xmethod in downmethod: 1577 | if prockilled == False: 1578 | hfdown(currentlink, currentfolder, xmethod, 'debugevery') 1579 | # civitdown(currentlink, currentfolder, currenttorename) 1580 | steps += 1 1581 | 1582 | elif listpart.startswith("https://github.com"): 1583 | currentlink, currenttorename = splitrename(listpart) 1584 | print('\n') 1585 | print(currentlink) 1586 | if '/raw/' in listpart or '/releases/download/' in listpart: 1587 | currentcondition = f'Downloading {currentlink}...' 1588 | progress(round(steps/totalsteps, 3), desc='Downloading ' + os.path.basename(currentlink) + f' into {currenthashtag}...') 1589 | if everymethod == False: 1590 | hfdown(currentlink, currentfolder, choosedowner, 'default', currenttorename) 1591 | else: 1592 | for xmethod in downmethod: 1593 | if prockilled == False: 1594 | hfdown(currentlink, currentfolder, xmethod, 'debugevery') 1595 | else: 1596 | splits = listpart.split('#')[0].split("/") 1597 | currentlink = "/".join(splits[:5]) 1598 | foldername = listpart.split('#')[0].rsplit('/', 1)[-1] 1599 | folderpath = os.path.join(extpath, foldername) 1600 | currentcondition = f'Cloning {currentlink}...' 1601 | progress(round(steps/totalsteps, 3), desc='Cloning from ' + currentlink.split('/', 3)[-1] + f' into #ext...') 1602 | if platform.system() == "Windows": 1603 | runwithsubprocess(f"git clone {currentlink} {shlex.quote(folderpath)}") 1604 | else: 1605 | runwithsubprocess(f"git clone {currentlink} {folderpath}") 1606 | steps += 1 1607 | 1608 | elif listpart.startswith("https://civitai.com/models/"): 1609 | currentlink, currenttorename = splitrename(listpart) 1610 | print('\n') 1611 | print(currentlink) 1612 | currentcondition = f'Downloading {currentlink}...' 1613 | #customtypemains = list 1614 | progress(round(steps/totalsteps, 3), desc='Downloading ' + os.path.basename(currentlink) + f'...') 1615 | for tag in tuple((typemain[countofdefaulthashtags:])): 1616 | if "#" + tag == currenthashtag and gradiostate: 1617 | progress(round(steps/totalsteps, 3), desc='Downloading ' + os.path.basename(currentlink) + f' into {currenthashtag}...') 1618 | pass 1619 | break 1620 | if everymethod == False: 1621 | civitdown2(currentlink, currentfolder, choosedowner, currenttorename, False, civitdefault, civitpruned, civitvae) 1622 | # else: 1623 | # for xmethod in downmethod: 1624 | # if prockilled == False: 1625 | # civitdown2(currentlink, currentfolder, xmethod, True) 1626 | steps += 1 1627 | 1628 | elif listpart.startswith("!"): 1629 | commandtorun = listpart[1:] 1630 | currentcondition = f'Running command: {commandtorun}' 1631 | progress(round(steps/totalsteps, 3), desc=currentcondition) 1632 | runwithsubprocess(commandtorun, None, True) 1633 | steps += 1 1634 | 1635 | elif listpart.startswith("@debugeverymethod") and globaldebug == True and gradiostate == True: 1636 | print('\n') 1637 | everymethod = True 1638 | print('DebugEveryMethod activated!') 1639 | print('One link will be downloaded with every possible download method.') 1640 | print('') 1641 | 1642 | elif listpart.startswith("@debugresetdownloads") and snapshot != {} and globaldebug == True: 1643 | print('\n') 1644 | currentcondition = f'Removing downloaded files...' 1645 | removed_files = global_rewind() 1646 | oldfilesdict = trackall() 1647 | texttowrite = ["⬇️Removed files⬇️"] 1648 | for item in removed_files: 1649 | texttowrite.append(item) 1650 | writefinal = list_to_text(texttowrite) 1651 | printdebug(str(writefinal)) 1652 | 1653 | elif listpart.startswith("@extract"): #@note hashtagextract 1654 | print('\n') 1655 | currentcondition = 'Extracting every archive file in current directory...' 1656 | print('Extracting every archive file in current directory...\n') 1657 | extractcurdir(currentfolder) 1658 | 1659 | elif listpart.startswith("@new"): #@note hashtagcustom 1660 | newcommand, newhashtag, newpath = shlex.split(listpart) 1661 | if newcommand and newhashtag and newpath: 1662 | if newhashtag.startswith("#"): 1663 | printdebug("making custom hashtag") 1664 | newtype = newhashtag[1:] 1665 | #global typemain #moved to top 1666 | #global typechecker #moved to top 1667 | if not newtype[0].isdigit(): 1668 | if not newtype in typemain and not newpath in typechecker: 1669 | try: 1670 | typemain.append(newtype) 1671 | printdebug('typemain: ' + str(typemain)) 1672 | typechecker.append(newtype) 1673 | printdebug('typechecker: ' + str(typechecker)) 1674 | newglobalpath = newtype + "path" 1675 | printdebug('newglobalpath: ' + newtype + "path") 1676 | newpath = os.path.abspath(os.path.normpath(newpath).rstrip(os.sep)) 1677 | os.makedirs(newpath, exist_ok=True) 1678 | globals()[newglobalpath] = newpath 1679 | printdebug("modelpath = " + eval(newglobalpath)) 1680 | oldfilesdict[newtype] = os.listdir(newpath) 1681 | # global addedcustompath 1682 | addedcustompath[newhashtag] = newpath 1683 | printdebug("addedcustompath: " + str(addedcustompath)) 1684 | print(f"New custom path added!\n{newhashtag} means {newpath}") 1685 | except Exception as e: 1686 | print("Adding custom path failed! Reason: " + str(e)) 1687 | else: 1688 | print("Adding custom path failed! Reason: Custom hashtag name cannot starts with a number") 1689 | 1690 | elif listpart.startswith("@aria") or listpart.startswith("@aria2"): #@note customaria 1691 | 1692 | # checkariacommand = ['##', '>'] 1693 | printvardebug(listpart) 1694 | if "##" in listpart: 1695 | listpart = listpart.split('##')[0] 1696 | if ":\\" in listpart: #windows path 1697 | listpart = listpart.replace('\\', '\\\\').replace('"', '\\"') 1698 | ariacmd = shlex.split(listpart) 1699 | printvardebug(ariacmd) 1700 | arialink, ariarename, ariapath = '', '', '' 1701 | renamedalready = False 1702 | custompath = False 1703 | # startcomment = False 1704 | if (ariacmd[0] == '@aria2' or ariacmd[0] == '@aria') and len(ariacmd) > 1: #two items. @aria and link 1705 | if ariacmd[1].startswith('http'): 1706 | arialink = ariacmd[1] 1707 | 1708 | if len(ariacmd) > 2: #three items. @aria, link, and path 1709 | if ariacmd[2].startswith(">") and len(ariacmd) > 3: #four items. @aria, link, '>', and rename (no custom path) 1710 | ariarename = shlex.quote(ariacmd[3]) 1711 | renamedalready = True 1712 | else: 1713 | if ariacmd[2].startswith('#'): #three items. @aria, link, hashtagpath 1714 | tobeariapath, _ = hashtagtopath(ariacmd[2]) 1715 | if tobeariapath: 1716 | ariapath = tobeariapath 1717 | else: #three items. @aria, link, custompath 1718 | # ariapath = shlex.quote(ariacmd[2]) 1719 | ariapath = ariacmd[2] 1720 | custompath = True #message for file downloaded to custom path will be added later 1721 | # if len(cmd) > 3: 1722 | # if cmd[3].startswith('>'): 1723 | # renaming = True 1724 | # else: 1725 | # renaming = False 1726 | if len(ariacmd) > 4 and ariacmd[3].startswith('>') and not renamedalready: #five items. @aria, link, path, '>', and rename 1727 | ariarename = shlex.quote(ariacmd[4]) 1728 | 1729 | if not ariapath: 1730 | ariapath = currentfolder 1731 | 1732 | if not ariarename: 1733 | ariarename = arialink.rsplit('/', 1)[-1] 1734 | 1735 | #finalcommand = f"aria2c --summary-interval=1 --console-log-level=error -c -x 16 -s 16 -k 1M {arialink} -d {ariapath} -o {ariarename}" 1736 | printvardebug(arialink) 1737 | printvardebug(ariapath) 1738 | printvardebug(ariarename) 1739 | if arialink and not prockilled: 1740 | try: 1741 | currentcondition = f'Downloading from {arialink} into {ariapath}...' 1742 | hfdown(arialink, ariapath, 'aria2', 'default', ariarename) 1743 | except Exception as e: 1744 | print(f"Custom aria download failed. Reason: {str(e)}") 1745 | 1746 | elif listpart.startswith("#") and listpart.endswith(tuple(typemain)): #tuple(typemain[countofdefaulthashtags:]) 1747 | try: 1748 | printdebug("one of typemain") 1749 | currenthashtag = listpart 1750 | currentfolder = eval(listpart[1:] + "path") 1751 | os.makedirs(currentfolder, exist_ok=True) 1752 | except Exception as e: 1753 | print(f"Cannot use hashtag: {str(e)}") 1754 | 1755 | else: 1756 | tobecurrentfolder, tobecurrenthashtag = hashtagtopath(listpart) 1757 | if tobecurrentfolder: 1758 | currentfolder = tobecurrentfolder 1759 | if tobecurrentfolder: 1760 | currenthashtag = tobecurrenthashtag 1761 | 1762 | else: 1763 | currentcondition = 'Operation cancelled' 1764 | return "Operation cancelled" 1765 | 1766 | currentcondition = 'Writing output...' 1767 | downloadedfiles = writeall(oldfilesdict, isshell, addedcustompath) 1768 | for tokill in everyprocessid: 1769 | try: 1770 | os.kill(tokill, signal.SIGTERM) 1771 | except ProcessLookupError: 1772 | pass 1773 | except PermissionError: 1774 | pass 1775 | except OSError: 1776 | pass 1777 | print() 1778 | print('BatchLinks Downloads finished!') 1779 | print('') 1780 | currentcondition = 'Done!' 1781 | printdebug(f"this should be the output:\n" + str(downloadedfiles)) 1782 | progress(1.00, desc='') 1783 | if gradiostate == True: 1784 | return [downloadedfiles, gr.Dataframe.update(value=buildarrayofhashtags('bottom'))] #@note dataframe #gr.Dataframe.update(value=buildarrayofhashtags('right')), was here 1785 | else: 1786 | return [downloadedfiles, gr.Dataframe.update(value=buildarrayofhashtags('bottom')), gr.Button.update(visible=resumebuttonvisible)] #gr.Dataframe.update(value=buildarrayofhashtags('right')), was here 1787 | 1788 | def hashtagtopath(thehashtag): 1789 | hashtagcurrent, foldercurrent = '','' 1790 | notbreaking = True 1791 | typemainlocal = [] 1792 | for x in typemain: 1793 | typemainlocal.append(x) 1794 | for y in typemainlocal: 1795 | if thehashtag[1:] == y: 1796 | printdebug("one of typemain, found on else") 1797 | hashtagcurrent = thehashtag 1798 | foldercurrent = eval(thehashtag[1:] + "path") 1799 | os.makedirs(foldercurrent, exist_ok=True) 1800 | notbreaking = False 1801 | for prefix in typechecker: 1802 | if thehashtag.startswith("#" + prefix) and notbreaking: 1803 | if prefix in ["embedding", "embeddings", "embed", "embeds","textualinversion", "ti"]: 1804 | hashtagcurrent = '#embed' 1805 | elif prefix in ["model", "models", "checkpoint", "checkpoints"]: 1806 | hashtagcurrent = '#model' 1807 | elif prefix in ["vae", "vaes"]: 1808 | hashtagcurrent = '#vae' 1809 | elif prefix in ["lora", "loras"]: 1810 | hashtagcurrent ='#lora' 1811 | elif prefix in ["hypernetwork", "hypernetworks", "hypernet", "hypernets", "hynet", "hynets",]: 1812 | hashtagcurrent = '#hynet' 1813 | elif prefix in ["addnetlora", "loraaddnet", "additionalnetworks", "addnet"]: 1814 | hashtagcurrent = '#addnetlora' 1815 | elif prefix in ["controlnet", "cnet"]: 1816 | hashtagcurrent = '#cnet' 1817 | elif prefix in ["extension", "extensions", "ext"]: 1818 | hashtagcurrent = '#ext' 1819 | elif prefix in ["aestheticembedding", "aestheticembed"]: 1820 | hashtagcurrent = '#aestheticembed' 1821 | elif prefix in ["upscaler", "upscale"]: 1822 | hashtagcurrent = '#upscaler' 1823 | elif prefix in ["altmodel", "altmodels"]: 1824 | hashtagcurrent = '#altmodel' 1825 | elif prefix in ["lycoris", "locon", "loha"]: 1826 | hashtagcurrent = '#lycoris' 1827 | try: 1828 | foldercurrent = eval(hashtagcurrent[1:] + 'path') 1829 | except Exception as e: 1830 | print(f"Cannot use hashtag: {e}") 1831 | continue 1832 | 1833 | os.makedirs(foldercurrent, exist_ok=True) 1834 | 1835 | return foldercurrent, hashtagcurrent 1836 | 1837 | wildcardcommand = [ 1838 | "@debugeverymethod", "@debugresetdownloads", 1839 | "@extract", "@aria", "@aria2", "@new" 1840 | ] 1841 | 1842 | def extract_links(string): 1843 | links = [] 1844 | lines = string.split('\n') 1845 | for line in lines: 1846 | line = line.split('##')[0].strip() 1847 | if line.startswith(tuple(supportedlinks)): 1848 | links.append(line) 1849 | elif line.startswith(tuple(wildcardcommand)): 1850 | links.append(line) 1851 | elif line.startswith("!"): 1852 | links.append(line.strip()) 1853 | elif line.startswith("#"): 1854 | links.append(line.strip()) 1855 | # else: 1856 | # for prefix in typechecker: 1857 | # if line.startswith("#" + prefix): 1858 | # links.append(line) 1859 | 1860 | #print(f"links: {links}") 1861 | return links 1862 | 1863 | def list_to_text(lst): 1864 | stripped_list = [item.strip(',').strip('\"') for item in lst] 1865 | return '\n'.join(stripped_list) 1866 | 1867 | def uploaded(textpath): 1868 | if not textpath is None: 1869 | print(textpath) 1870 | file_paths = textpath.name 1871 | print(file_paths) 1872 | links = [] 1873 | 1874 | with open(file_paths, 'r') as file: 1875 | for line in file: 1876 | if line.startswith(tuple(supportedlinks)): 1877 | links.append(line.strip()) 1878 | elif line.startswith("!"): 1879 | links.append(line.strip()) 1880 | elif line.startswith(tuple(wildcardcommand)): 1881 | links.append(line.strip()) 1882 | elif line.startswith("#"): 1883 | links.append(line.strip()) 1884 | # else: 1885 | # for prefix in typechecker: 1886 | # if line.startswith("#" + prefix): 1887 | # links.append(line.strip()) 1888 | 1889 | text = list_to_text(links) 1890 | return text 1891 | 1892 | count = 0 1893 | def keeplog(): 1894 | global currentcondition 1895 | global currentsuboutput 1896 | global logging 1897 | if logging == False: 1898 | currentcondition = "Logging activated." 1899 | currentsuboutput = '' 1900 | logging = True 1901 | return [currentcondition, gr.Button.update(visible=True), gr.Button.update(visible=False)] 1902 | if currentsuboutput == '': 1903 | return [currentcondition, gr.Button.update(visible=True), gr.Button.update(visible=False)] 1904 | else: 1905 | return [f"{currentcondition}\n{currentsuboutput}", gr.Button.update(visible=True), gr.Button.update(visible=False)] 1906 | 1907 | def offlog(): 1908 | global currentcondition 1909 | global currentsuboutput 1910 | global logging 1911 | if logging == True: 1912 | currentcondition = "Logging deactivated." 1913 | currentsuboutput = '' 1914 | logging = False 1915 | return [f"{currentcondition}", gr.Button.update(visible=False), gr.Button.update(visible=True)] 1916 | 1917 | def empty(): 1918 | return '' 1919 | 1920 | def cancelrun(): 1921 | global processid 1922 | global prockilled 1923 | printdebug("debug processid: " + str(processid)) 1924 | if not processid == '': 1925 | if platform.system() == "Windows": 1926 | global process 1927 | process.terminate() 1928 | _ = subprocess.getoutput("taskkill /F /T /PID" + str(process.pid)) 1929 | _ = subprocess.getoutput("taskkill /f /t /im MEGAcmdServer.exe") 1930 | else: 1931 | try: 1932 | os.kill(processid, signal.SIGTERM) 1933 | except ProcessLookupError: 1934 | pass 1935 | except PermissionError: 1936 | pass 1937 | except OSError: 1938 | pass 1939 | #os.killpg(os.getpgid(processid.pid), signal.SIGTERM) 1940 | prockilled = True 1941 | if prockilled == True and globaldebug == True: 1942 | print() 1943 | print("This should kill") 1944 | print() 1945 | return ["Operation Cancelled",gr.Dataframe.update(value=buildarrayofhashtags('bottom'))] 1946 | 1947 | def fillbox(): 1948 | global remaininglinks 1949 | if bool(remaininglinks): 1950 | text = list_to_text(remaininglinks) 1951 | remaininglinks = [] 1952 | return [text, 'Links updated!\nClick Download All! to download the rest of the links', gr.Button.update(visible=False)] 1953 | return ['', '', gr.Button.update(visible=False)] 1954 | 1955 | if gradiostate == True: 1956 | storedstatedownloader = "gdown" 1957 | else: 1958 | storedstatedownloader = "aria2" 1959 | storedstatemodeltype = "safetensors" 1960 | storedstateprecision = "fp16" 1961 | storedstatevae = True 1962 | 1963 | def grchangedown(downstate): 1964 | global storedstatedownloader 1965 | storedstatedownloader = downstate 1966 | printvardebug(storedstatedownloader) 1967 | 1968 | def grchangetype(typestate): 1969 | global storedstatemodeltype 1970 | storedstatemodeltype = typestate 1971 | printvardebug(storedstatemodeltype) 1972 | 1973 | def grchangefp(fpstate): 1974 | global storedstateprecision 1975 | storedstateprecision = fpstate 1976 | printvardebug(storedstateprecision) 1977 | 1978 | def grchangevae(vaestate): 1979 | global storedstatevae 1980 | storedstatevae = vaestate 1981 | printvardebug(storedstatevae) 1982 | 1983 | def stretchui(stretch): #outputs=[boxtohide, bottomlist] #outputs=[boxtohide, downbox, civitbox, choose_downloader, civit_default, civit_ispruned, civit_alsodownvae]) 1984 | # if stretch: 1985 | # return [gr.Box.update(visible=False), gr.Accordion.update(visible=False), gr.Accordion.update(visible=True)] 1986 | # else: 1987 | # return [gr.Box.update(visible=True), gr.Accordion.update(visible=True), gr.Accordion.update(visible=False)] 1988 | global storedstatedownloader 1989 | global storedstatemodeltype 1990 | global storedstateprecision 1991 | global storedstatevae 1992 | if stretch: 1993 | return [gr.Box.update(visible=False), gr.Row.update(visible=True), gr.Row.update(visible=True), gr.Radio.update(value=storedstatedownloader), gr.Radio.update(value=storedstatemodeltype), gr.Radio.update(value=storedstateprecision), gr.Checkbox.update(value=storedstatevae)] 1994 | else: 1995 | return [gr.Box.update(visible=True), gr.Row.update(visible=False), gr.Row.update(visible=False), gr.Radio.update(value=storedstatedownloader), gr.Radio.update(value=storedstatemodeltype), gr.Radio.update(value=storedstateprecision), gr.Checkbox.update(value=storedstatevae)] 1996 | 1997 | def hidehelp(hide): 1998 | if hide: 1999 | return [gr.Markdown.update(value=titletext), gr.Markdown.update(visible=False)] 2000 | else: 2001 | return [gr.Markdown.update(value=introductiontext), gr.Markdown.update(visible=True)] 2002 | 2003 | # countofdefaulthashtags = len(typemain) 2004 | def buildarrayofhashtags(rightorbottom): 2005 | printdebug(f"buildarray {rightorbottom} initiated!") 2006 | # defaultpathtime = True 2007 | def writingpath(j, path): 2008 | if path == modelpath: 2009 | return path + " (default path)" 2010 | elif path == altmodelpath: 2011 | return path 2012 | else: 2013 | if rightorbottom == 'right' and j < countofdefaulthashtags: 2014 | return path.replace(script_path, "~") 2015 | else: 2016 | return path 2017 | hashtagandpath = [] 2018 | for i, x in enumerate(typemain): 2019 | try: 2020 | xpath = eval(x+"path") 2021 | if cmd_opts.ckpt_dir: 2022 | hashtagandpath.append(["#"+x, writingpath(i, xpath)]) 2023 | elif x != "altmodel": 2024 | hashtagandpath.append(["#"+x, writingpath(i, xpath)]) 2025 | # defaultpathtime = False 2026 | except Exception as e: 2027 | print(str(e)) 2028 | return hashtagandpath 2029 | 2030 | def copyfrompastebin(boxwithlink): 2031 | pastebinlink = boxwithlink.strip() 2032 | if pastebinlink.startswith('https://pastebin.com/'): 2033 | if pastebinlink.count('\n') == 0: 2034 | global currentcondition 2035 | currentcondition = f'Gathering links from Pastebin..' 2036 | if not '/raw/' in pastebinlink: 2037 | pastebinlink = pastebinlink.replace("pastebin.com/", "pastebin.com/raw/") 2038 | 2039 | pbinresponse = requests.get(pastebinlink) 2040 | 2041 | if pbinresponse.status_code == 200: 2042 | pbintextlist = pbinresponse.text.splitlines() 2043 | links = [] 2044 | for line in pbintextlist: 2045 | if line.startswith(tuple(supportedlinks)): 2046 | links.append(line.strip()) 2047 | elif line.startswith("!"): 2048 | links.append(line.strip()) 2049 | elif line.startswith("@new"): 2050 | links.append(line.strip()) 2051 | elif line.startswith(tuple(wildcardcommand)): 2052 | links.append(line.strip()) 2053 | elif line.startswith("#"): 2054 | links.append(line.strip()) 2055 | currentcondition = f'Done.' 2056 | finallinks = list_to_text(links) 2057 | return [finallinks, "Pastebin links retrieved successfully."] 2058 | else: 2059 | currentcondition = f'Done.' 2060 | print("Error retrieving data from pastebin") 2061 | return [boxwithlink, "Error retrieving data from pastebin"] 2062 | else: 2063 | currentcondition = f'Done.' 2064 | return [boxwithlink, "Pastebin link must be the only one link on the textbox! (And only one pastebin link supported)"] 2065 | else: 2066 | currentcondition = f'Done.' 2067 | return [boxwithlink, "Pastebin link must be the first (and the only) link on the textbox!"] 2068 | 2069 | titletext = f"""

    ⬇️ Batchlinks Downloader ({currentversion}) {latestversiontext}

    """ 2070 | introductiontext = f""" 2071 | {titletext} 2072 |
    (what's new?)
    2073 |

    This tool will read the textbox and download every links from top to bottom one by one
    2074 | Put your links down below. Supported link: Huggingface, CivitAI, MEGA, Discord, Github, Catbox, Google Drive, Pixeldrain, Mediafire, Anonfiles, Dropbox
    2075 | Use hashtag to separate downloaded items based on their download location
    2076 | Valid hashtags: #embed, #model, #hypernet, #lora, #vae, #addnetlora, etc.
    2077 | (For colab that uses sd-webui-additional-networks extension to load LoRA, use #addnetlora instead)
    2078 | Use double hashtag (##) after links for comment. Useful to mark which links downloads what.
    2079 | Remember to always press the 🔄️ refresh button on the UI after downloading models etc. in order for them to show up on the list.

    2080 | """ 2081 | knowmoretext = f""" 2082 |

    Click these links for more:
    2083 | Readme Page
    2084 | Example
    2085 | Syntax
    2086 | Valid Hashtags
    2087 | Here's how you can get the direct links
    2088 | Report Bug

    2089 | """ 2090 | testboxplaceholder = f"""#model 2091 | 2092 | #vae 2093 | 2094 | #lora 2095 | 2096 | ##this is a comment, and these text is just an example 2097 | """ 2098 | 2099 | def on_ui_tabs(): 2100 | with gr.Blocks() as batchlinks: 2101 | with gr.Row(): 2102 | with gr.Column(scale=2): 2103 | introduction = gr.Markdown(introductiontext) 2104 | with gr.Column(scale=1): 2105 | with gr.Row(): 2106 | uistretcher = gr.Checkbox(value=False, label="Stretch UI", interactive=True) 2107 | helphider = gr.Checkbox(value=False, label="Hide Help", interactive=True) 2108 | knowmore = gr.Markdown(knowmoretext) 2109 | with gr.Group(): 2110 | command = gr.Textbox(label="Links", placeholder=testboxplaceholder, lines=5) 2111 | if gradiostate == True: 2112 | logbox = gr.Textbox(label="Log", interactive=False) 2113 | else: 2114 | if vladmandic: 2115 | logbox = gr.Textbox("(remove the --disable-queue args on launch.py to enable optional logging)", label="Log", interactive=False) 2116 | else: 2117 | logbox = gr.Textbox("(use --gradio-queue args on launch.py to enable optional logging)", label="Log", interactive=False) 2118 | 2119 | with gr.Row(): 2120 | with gr.Box(): 2121 | if gradiostate == True: 2122 | with gr.Column(): 2123 | # with gr.Row(): 2124 | # gr.Textbox(value=None, interactive=False, show_label=False) 2125 | btn_onlog = gr.Button("Turn On Logging", variant="primary", visible=True) 2126 | # with gr.Row(): 2127 | btn_offlog = gr.Button("Turn Off Logging", visible=False) 2128 | loggingon = btn_onlog.click(keeplog, outputs=[logbox, btn_offlog, btn_onlog], every=1) 2129 | btn_offlog.click(offlog, outputs=[logbox, btn_offlog, btn_onlog], cancels=[loggingon]) 2130 | # gr.Textbox(value=None, interactive=False, show_label=False) 2131 | # logging = gr.Radio(["Turn On Logging"], show_label=False) 2132 | # logging.change(keeplog, outputs=logbox, every=1) 2133 | out_text = gr.Textbox(label="Output") 2134 | else: 2135 | if vladmandic: 2136 | print("Batchlinks webui extension: (Optional) Remove the --disable-queue args to enable logging & cancel button on this extension") 2137 | else: 2138 | print("Batchlinks webui extension: (Optional) Use --gradio-queue args to enable logging & cancel button on this extension") 2139 | out_text = gr.Textbox("(If this text disappear, that means a download session is in progress.)", label="Output") 2140 | 2141 | # if platform.system() == "Windows": 2142 | # choose_downloader = gr.Radio(["gdown", "wget", "curl"], value="gdown", label="Download method") 2143 | # else: 2144 | 2145 | 2146 | with gr.Row(variant='panel', visible=False) as downbox: 2147 | 2148 | if gradiostate == True: 2149 | with gr.Column(scale=1): 2150 | choose_downloader2 = gr.Radio(["gdown", "wget", "curl", "aria2"], value="gdown", label="Download method") 2151 | else: 2152 | with gr.Column(scale=1): 2153 | choose_downloader2 = gr.Radio(["aria2"], value="aria2", label="Downloader") 2154 | 2155 | 2156 | with gr.Row(variant='panel', visible=False) as civitbox: 2157 | # with gr.Box(): 2158 | # with gr.Column(scale=1): 2159 | # with gr.Row(): 2160 | civit_default2 = gr.Radio(["ckpt", "safetensors"], value="safetensors", label="CivitAI Preferred Model Type", interactive=True) 2161 | civit_ispruned2 = gr.Radio(["fp16", "fp32"], value="fp16", label="Model Precision", interactive=True) 2162 | # with gr.Row(): 2163 | # civit_ispruned = gr.Checkbox(True, label="Pruned", interactive=True) 2164 | civit_alsodownvae2 = gr.Checkbox(True, label="Also Download VAE", interactive=True) 2165 | 2166 | # def passvaluebetweenradio(newval): 2167 | # return gr.Radio.update(value=newval) 2168 | 2169 | # def passvaluebetweencheckbox(newval): 2170 | # return gr.Checkbox.update(value=newval) 2171 | 2172 | # choose_downloader2.change(passvaluebetweenradio, inputs=choose_downloader2, outputs=choose_downloader) 2173 | # civit_default2.change(passvaluebetweenradio, inputs=civit_default2, outputs=civit_default) 2174 | # civit_ispruned2.change(passvaluebetweenradio, inputs=civit_ispruned2, outputs=civit_ispruned) 2175 | # civit_alsodownvae2.change(passvaluebetweencheckbox, inputs=civit_alsodownvae2, outputs=civit_alsodownvae) 2176 | 2177 | with gr.Row(): 2178 | if gradiostate == True: 2179 | with gr.Column(scale=1, min_width=100): 2180 | btn_run = gr.Button("Download All!", variant="primary") 2181 | # btn_upload = gr.UploadButton("Upload .txt", file_types="text") 2182 | # btn_upload.upload(uploaded, btn_upload, file_output) 2183 | with gr.Column(scale=1, min_width=100): 2184 | btn_cancel = gr.Button("Cancel") 2185 | with gr.Row(): 2186 | file_output = gr.UploadButton(file_types=['.txt'], label='Upload txt') 2187 | btn_pastebin = gr.Button("Copy from Pastebin") 2188 | 2189 | else: 2190 | btn_run = gr.Button("Download All!", variant="primary") 2191 | btn_resume = gr.Button("Resume Download", visible=False) 2192 | with gr.Column(scale=1, min_width=100): 2193 | with gr.Row(): 2194 | file_output = gr.UploadButton(file_types=['.txt'], label='Upload txt') 2195 | btn_pastebin = gr.Button("Copy from Pastebin") 2196 | # with gr.Column(scale=1, min_width=100): 2197 | # file_output = gr.UploadButton(file_types=['.txt'], label='Upload txt') 2198 | # copy_pastebin = gr.Button("Copy from Pastebin") 2199 | if gradiostate == False: 2200 | with gr.Row(): 2201 | gr.Markdown( 2202 | f""" 2203 |


    After clicking the Download All button, it's recommended to inspect the 2204 | colab console, as every information about the download progress is there.

    2205 | """) 2206 | 2207 | if gradiostate == False: 2208 | btn_resume.click(fillbox, None, outputs=[command, out_text, btn_resume]) 2209 | 2210 | # file_output = gr.File(file_types=['.txt'], label="you can upload a .txt file containing links here") 2211 | # file_output.change(uploaded, file_output, command) 2212 | 2213 | file_output.upload(uploaded, file_output, command) 2214 | with gr.Box(visible=True) as boxtohide: 2215 | with gr.Row(variant='panel'): 2216 | if gradiostate == True: 2217 | with gr.Column(scale=1): 2218 | choose_downloader = gr.Radio(["gdown", "wget", "curl", "aria2"], value=storedstatedownloader, label="Download method") 2219 | else: 2220 | with gr.Column(scale=1): 2221 | choose_downloader = gr.Radio(["aria2"], value=storedstatedownloader, label="Downloader") 2222 | with gr.Row(variant='panel'): 2223 | civit_default = gr.Radio(["ckpt", "safetensors"], value=storedstatemodeltype, label="CivitAI Preferred Model Type", interactive=True) 2224 | civit_ispruned = gr.Radio(["fp16", "fp32"], value=storedstateprecision, label="Model Precision", interactive=True) 2225 | # with gr.Row(): 2226 | # civit_ispruned = gr.Checkbox(True, label="Pruned", interactive=True) 2227 | civit_alsodownvae = gr.Checkbox(storedstatevae, label="Also Download VAE", interactive=True) 2228 | choose_downloader.change(grchangedown, inputs=choose_downloader) 2229 | civit_default.change(grchangetype, inputs=civit_default) 2230 | civit_ispruned.change(grchangefp, inputs=civit_ispruned) 2231 | civit_alsodownvae.change(grchangevae, inputs=civit_alsodownvae) 2232 | # with gr.Box(visible=True) as boxtohide: 2233 | # gr.Markdown(""" 2234 | #
    If you feel the UI is too cramped, click the Stretch UI button above.
    2235 | # """) 2236 | # with gr.Accordion("List of Every Hashtags and its Path", open=False, visible=True) as rightlist: 2237 | # righttable = gr.DataFrame( 2238 | # buildarrayofhashtags('right'), 2239 | # headers=["hashtag", "path"], 2240 | # datatype=["str", "str"], 2241 | # interactive=False 2242 | # ) 2243 | # sidetext = gr.Markdown(knowmoretext, visible=True) 2244 | def passvaluebetweenradio(newval): 2245 | return gr.Radio.update(value=newval) 2246 | 2247 | def passvaluebetweencheckbox(newval): 2248 | return gr.Checkbox.update(value=newval) 2249 | 2250 | choose_downloader2.change(passvaluebetweenradio, inputs=choose_downloader2, outputs=choose_downloader) 2251 | civit_default2.change(passvaluebetweenradio, inputs=civit_default2, outputs=civit_default) 2252 | civit_ispruned2.change(passvaluebetweenradio, inputs=civit_ispruned2, outputs=civit_ispruned) 2253 | civit_alsodownvae2.change(passvaluebetweencheckbox, inputs=civit_alsodownvae2, outputs=civit_alsodownvae) 2254 | finish_audio = gr.Audio(interactive=False, value=os.path.join(extension_dir, "notification.mp3"), elem_id="finish_audio", visible=False) 2255 | with gr.Accordion("List of Every Hashtags and its Path", open=False, visible=True) as bottomlist: 2256 | bottomtable = gr.DataFrame( 2257 | buildarrayofhashtags('bottom'), 2258 | headers=["hashtag", "path"], 2259 | datatype=["str", "str"], 2260 | interactive=False, 2261 | ) 2262 | gr.Markdown( 2263 | f""" 2264 |

    Made with ❤️ by etherealxx

    2265 | """) 2266 | helphider.change(hidehelp, helphider, outputs=[introduction, knowmore]) 2267 | # uistretcher.change(stretchui, uistretcher, outputs=[boxtohide, rightlist, bottomlist]) 2268 | # uistretcher.change(stretchui, uistretcher, outputs=boxtohide) 2269 | uistretcher.change(stretchui, uistretcher, outputs=[boxtohide, downbox, civitbox, choose_downloader2, civit_default2, civit_ispruned2, civit_alsodownvae2]) 2270 | #batchlinks.load(debug, output=debug_txt, every=1) 2271 | if gradiostate == True: 2272 | run_event = btn_run.click(run, inputs=[command, choose_downloader, civit_default, civit_ispruned, civit_alsodownvae], outputs=[out_text, bottomtable]) #righttable was on output 2273 | btn_cancel.click(cancelrun, None, outputs=[out_text, bottomtable], cancels=[run_event]) 2274 | else: 2275 | btn_run.click(run, inputs=[command, choose_downloader, civit_default, civit_ispruned, civit_alsodownvae], outputs=[out_text, bottomtable, btn_resume]) #righttable was on output 2276 | 2277 | btn_pastebin.click(copyfrompastebin, inputs=[command], outputs=[command, out_text]) 2278 | 2279 | if sdless: 2280 | if platform.system() == "Windows": 2281 | batchlinks.queue(64).launch(inbrowser=True) 2282 | else: 2283 | batchlinks.queue(64).launch(share=True) 2284 | else: 2285 | return (batchlinks, "Batchlinks Downloader", "batchlinks"), 2286 | if not sdless: 2287 | script_callbacks.on_ui_tabs(on_ui_tabs) 2288 | else: 2289 | on_ui_tabs() 2290 | -------------------------------------------------------------------------------- /sdless-macos.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | pip3 show virtualenv >/dev/null || pip3 install -q virtualenv 3 | [ -d gradiovenv ] || virtualenv gradiovenv 4 | source gradiovenv/bin/activate; \ 5 | pip3 show gradio >/dev/null || pip3 install -q gradio==3.16.2; \ 6 | pip3 show tqdm >/dev/null || pip3 install -q tqdm 7 | source gradiovenv/bin/activate; \ 8 | python3 scripts/batchlinks-downloader.py -------------------------------------------------------------------------------- /sdless-windows-debug.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | if exist gradioinstalled.bak goto beginning 3 | echo Checking gradio version... 4 | echo. 5 | python -c "import gradio" 2>nul && ( 6 | python -c "import gradio; assert gradio.__version__ == '3.16.2'" 2>nul || ( 7 | pip install gradio==3.16.2 8 | ) 9 | ) || ( 10 | pip install gradio==3.16.2 11 | ) 12 | pip show tqdm >nul || pip install tqdm 13 | pip freeze | findstr /i "\" >nul && pip freeze | findstr /i "\" >nul && (echo. > gradioinstalled.bak) 14 | echo. 15 | 16 | :beginning 17 | cd scripts 18 | echo ^> python -m batchlinks-downloader.py --debug 19 | echo. 20 | python -m batchlinks-downloader.py --debug 21 | echo. 22 | echo Close the terminal to stop the looping launch 23 | echo. 24 | pause 25 | echo. 26 | goto beginning -------------------------------------------------------------------------------- /sdless-windows.bat: -------------------------------------------------------------------------------- 1 | @echo off 2 | if exist gradioinstalled.bak goto beginning 3 | echo Checking gradio version... 4 | echo. 5 | python -c "import gradio" 2>nul && ( 6 | python -c "import gradio; assert gradio.__version__ == '3.16.2'" 2>nul || ( 7 | pip install gradio==3.16.2 8 | ) 9 | ) || ( 10 | pip install gradio==3.16.2 11 | ) 12 | pip show tqdm >nul || pip install tqdm 13 | pip freeze | findstr /i "\" >nul && pip freeze | findstr /i "\" >nul && (echo. > gradioinstalled.bak) 14 | echo. 15 | 16 | :beginning 17 | cd scripts 18 | echo ^> python -m batchlinks-downloader.py 19 | echo. 20 | python -m batchlinks-downloader.py 21 | echo. 22 | echo Close the terminal to stop the looping launch 23 | echo. 24 | pause 25 | echo. 26 | goto beginning -------------------------------------------------------------------------------- /version.txt: -------------------------------------------------------------------------------- 1 | v3.2.1 2 | (^current version, hopefully not the last^) 3 | 4 | Release notes: 5 | 6 | ## v3.2.1 7 | - Changed "pruned" option to "fp precision" due to CivitAI API changes 8 | - Fixed bug: No matter when the download process is cancelled or not, CivitAI download will give print out notification that the download process is successful 9 | - Fixed bug: Preventing the UI errored out when storage is full and trying to convert jpg image while downloading civitai stuff (PIL.UnidentifiedImageError) 10 | - UI overhaul 11 | 12 | ## v3.2.0a 13 | - Added support for vladmandic's fork of automatic1111's webui 14 | 15 | ## v3.2.0 16 | - Bug fixed when downloading with aria2 on Windows 17 | - Bug fixed on gradio update checker on sdless batch file for windows 18 | - CivitAi error prevention (when the website is down) 19 | - Cleaned the code of the hashtag system 20 | - `Copy from Pastebin` feature 21 | - Custom aria command with `@aria` 22 | - Making sure custom hashtag doesn't start with number 23 | - New calmer notification sound 24 | - Readme page cleanup 25 | - Some help link changed to link into wiki instead 26 | - UI fix to adapt with latest gradio (v3.23.0) 27 | - Various bug fixes related to CivitAI download 28 | 29 | ## v3.1.1b 30 | - Hotfix: Fixed CivitAI link refuses to be downloaded properly. Seems like CivitAI changes its API. 31 | - Fixed several bugs on Windows, especially SDless 32 | 33 | ## v3.1.1a 34 | - Hotfix: Fixed CivitAI 'model type chooser' bug and fixed `@extract` bug on colab 35 | 36 | ## v3.1.1 37 | - Added some fix and messages in case CivitAI website is down 38 | - Added message when user pressed "Download All!" but the textbox is empty 39 | - Fixed bug where Lycoris folder always shows on downloaded files the first time user download something 40 | 41 | ## v3.1.0a 42 | - Hotfix: Indented block on line 1497 fix 43 | 44 | ## v3.1.0 45 | - New hashtag: `#altmodel`, when you use `--ckptdir` argument on `launch.py` line, this hashtag will points to that directory. Otherwise, it'll point to the same directory as `#model` 46 | - New hashtag: `#lycoris`, change current save directory to _/content/stable-diffusion-webui/extensions/sd-webui-additional-networks/models/lora/lycoris_
    (Side note: _Lycoris/Locon/Loha_ will works just fine if you use `#addnetlora` instead, as long as you have both [addnet extension](https://github.com/kohya-ss/sd-webui-additional-networks) and [locon extension](https://github.com/KohakuBlueleaf/a1111-sd-webui-locon) installed) 47 | - New hashtag: `#upscaler`, change current save directory to _/content/stable-diffusion-webui/models/ESRGAN_ (This was added few commits ago, but i forgot to write it on the release notes) 48 | 49 | Fixes: 50 | - Fixed bug when installing wget/aria2 on Windows 51 | - Fixed bug where custom paths doesn't put CivitAi download (Model Path method) when custom hashtag is used 52 | - Ongoing downloads will correctly stops when Cancel button is pressed on Windows 53 | - CivitAi (Model Path method) now will not download training dataset by accident 54 | 55 | ## v3.0.2 56 | - Hotfix: removed a comment that messed up the non queue mode. Now the extension works again without `--gradio-queue`. 57 | 58 | ## v3.0.1 59 | - Hotfix: using `urllib.request` instead of `curl -sI` to get the model name on CivitAI direct link method, since it's more reliable (and the curl method always fails somehow). The `requests` method is returned as a fallback. 60 | 61 | ## v3.0.0 62 | - Added `@extract` syntax 63 | - Supports for custom hashtags with `@new` syntax 64 | - (Almost) Full Windows support 65 | - Auto-download config file if available when downloading from CivitAI (SD 2.0+) 66 | - Auto-renaming for downloading ckpt/safetensors and pruned model from CivitAI using direct link method 67 | - CivitAI direct link now use `curl` to get the filename, and use the chosen download method (from the four) to download. Huge download speed boost. `requests` is no longer needed. 68 | - Supports download from Anonfiles, Dropbox, Google Drive, Mediafire, Pixeldrain 69 | - Supports download from Github (raw and release files) 70 | - Supports for SDless mode (read more [here](https://github.com/etherealxx/batchlinks-webui#sdless-mode)) 71 | - UI overhaul: 72 | - Now there's a table that shows where does the hashtags points into 73 | - Option to stretch the UI, if your monitor is small, or using colab on mobile 74 | - Option to hide help text 75 | - Option to choose preferred CivitAI models. This will works if you download the model via model page link (https://civitai.com/models/) 76 | - Upload txt now use a little button instead of covering half of the screen 77 | 78 | Fixes: 79 | - CivitAI `model page link` no longer randomly download the first model on the json list. 80 | - Most of Windows bugs 81 | - Renaming problem when using CivitAI model page link method 82 | - Warning message when CivitAI download isn't possible (server down) 83 | 84 | ## v2.1.2 85 | - CivitAI direct link now use curl to get the filename, and use the chosen download method to download. Requests is no longer needed. 86 | 87 | ## v2.1.1 88 | - Partial Windows support is back 89 | 90 | Changes: 91 | - wget disabled on windows currently, until it fixed 92 | 93 | Fixes: 94 | - gdown & curl bug fixed 95 | - utf-8 as default encoding for queue checker (fix bug in Windows) 96 | 97 | ## v2.1.0 98 | Features: 99 | - Supports renaming downloaded file with `>` (for example: `https://files.catbox.moe/uarze8.safetensors > neurosama.safetensors`) 100 | - Supports extension usage without `--gradio-queue` (ported from [onedotsix](https://github.com/etherealxx/batchlinks-webui/tree/onedotsix)) 101 | - Supports running shell command from the UI with `!` (for example: type `!pip freeze`, then hit the `Download all!` button and see the colab console) 102 | - Progress bar for `--gradio-queue` 103 | 104 | Changes: 105 | - `aria2` as *the only* download method when using without `--gradio-queue` 106 | - Download session will be cut every 80 seconds on when using without `--gradio-queue` (just like [onedotsix](https://github.com/etherealxx/batchlinks-webui/tree/onedotsix)) 107 | - _Debug stopwatch (decorator)_ won't run automatically when `globaldebug = True`, must be uncommented manually (it disrupt the progress bar) 108 | - Dropped support for webui based on Gradio 3.9 (update your installation, or use [onedotsix](https://github.com/etherealxx/batchlinks-webui/tree/onedotsix) instead) 109 | - UI tweak (Smaller font size) 110 | 111 | ## v2.0.0 112 | Features: 113 | - `aria2` as download method. 114 | - Cancel button for cancelling download process (`--gradio-queue` required) 115 | - Debug snapshot.
    116 | When `global_debug = True`, the moment this extension launch, it saves the current state of the webui on various location (into `snapshot.txt`), and when you type `#debugresetdownloads` on the textbox, it will compare the current state and the last saved state, and removes every new file/folder. This will be useful for debugging and testing. 117 | - Debug every download method.
    118 | When `global_debug = True` and you type `#debugevery method` on the textbox, every link that has 4 different method of download (Huggingface etc.) will be downloaded with every method, regardless of the radio button choice. The result is 4 file being downloaded. 119 | - Detection if a CivitAI links no longer exist 120 | - New hashtags: `#textualinversion`, `#ti`, `#aestheticembedding`, `#aestheticembed`, `#controlnet`, and `#cnet` 121 | - Toggle logging on/off 122 | - `shlex.quote` to properly quote links (Thanks **[@rti7743](https://github.com/rti7743)**!) 123 | - Supports cloning webui extensions 124 | - Supports download from CivitAI model links (Thanks **[@rti7743](https://github.com/rti7743)**!) 125 | - Supports download from Github (repository and raw files) 126 | - Supports for aesthetic gradients, controlnet model, and extensions path. 127 | - UI font scaled down 128 | - Uses `subprocess.Popen` instead of `os.system` 129 | 130 | 131 | Older releases: 132 | 133 | - commit "alpha" is v0.1.0 (09b1d81806e5bfc4d1783ccf644359fabc372fb7) 134 | - commit "Civit support, i hope it works." is v0.2.0 (2420ea4dd670ec71481a2ed43ff22c7c1a959543) 135 | - commit "🪲Bug fix and 🖼️stuff" is v0.2.1 (c7a91b6e14ca6f35438e29ab7ad3d2e00d62c99b) 136 | - commit "Merge pull request #1 from etherealxx/gradioevery" is v1.0.0 (a89e50fdfb5008f703750cf0e7878fc6c25f9219) 137 | --------------------------------------------------------------------------------