├── .github ├── FUNDING.yml ├── ISSUE_TEMPLATE │ ├── bug_report.md │ └── feature_request.md ├── dependabot.yml └── workflows │ └── codeql.yml ├── CODE_OF_CONDUCT.md ├── CONTRIBUTING.md ├── LICENSE ├── Notebooks ├── Colab │ └── HuggingFace_Backup_2024_Colab.ipynb ├── Dataset Zipper │ └── Dataset_Zipper.ipynb ├── HuggingFace_Backup_Jupyter_2025_May_Update.ipynb └── Previous Notebooks │ ├── HuggingFace_Backup_Jupyter_2025_EXTRA.ipynb │ ├── HuggingFace_Backup_Jupyter_2025_Jan_Update.ipynb │ └── HuggingFace_Backup_Jupyter_2025_March_Update.ipynb ├── README.md └── SECURITY.md /.github/FUNDING.yml: -------------------------------------------------------------------------------- 1 | # These are supported funding model platforms 2 | 3 | github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2] 4 | patreon: earthndusk 5 | open_collective: # Replace with a single Open Collective username 6 | ko_fi: duskfallcrew 7 | tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel 8 | community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry 9 | liberapay: # Replace with a single Liberapay username 10 | issuehunt: # Replace with a single IssueHunt username 11 | otechie: # Replace with a single Otechie username 12 | lfx_crowdfunding: # Replace with a single LFX Crowdfunding project-name e.g., cloud-foundry 13 | custom: # Replace with up to 4 custom sponsorship URLs e.g., ['link1', 'link2'] 14 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | # 🤖 Dependabot Configuration - Your Friendly Update Assistant! 2 | version: 2 3 | updates: 4 | # 🐍 Python Packages (for our notebook magic!) 5 | - package-ecosystem: "pip" 6 | directory: "/" 7 | schedule: 8 | interval: "weekly" 9 | day: "monday" # Let's start the week fresh! 10 | time: "09:00" # Early bird gets the updates! 11 | open-pull-requests-limit: 5 12 | labels: 13 | - "dependencies" 14 | - "automated" 15 | commit-message: 16 | prefix: "📦" 17 | include: "scope" 18 | 19 | # 🎯 GitHub Actions (keeping our workflows sparkly!) 20 | - package-ecosystem: "github-actions" 21 | directory: "/" 22 | schedule: 23 | interval: "weekly" 24 | labels: 25 | - "automation" 26 | - "dependencies" 27 | -------------------------------------------------------------------------------- /.github/workflows/codeql.yml: -------------------------------------------------------------------------------- 1 | # For most projects, this workflow file will not need changing; you simply need 2 | # to commit it to your repository. 3 | # 4 | # You may wish to alter this file to override the set of languages analyzed, 5 | # or to provide custom queries or build logic. 6 | # 7 | # ******** NOTE ******** 8 | # We have attempted to detect the languages in your repository. Please check 9 | # the `language` matrix defined below to confirm you have the correct set of 10 | # supported CodeQL languages. 11 | # 12 | name: "CodeQL Advanced" 13 | 14 | on: 15 | push: 16 | branches: [ "main" ] 17 | pull_request: 18 | branches: [ "main" ] 19 | schedule: 20 | - cron: '41 12 * * 5' 21 | 22 | jobs: 23 | analyze: 24 | name: Analyze (${{ matrix.language }}) 25 | # Runner size impacts CodeQL analysis time. To learn more, please see: 26 | # - https://gh.io/recommended-hardware-resources-for-running-codeql 27 | # - https://gh.io/supported-runners-and-hardware-resources 28 | # - https://gh.io/using-larger-runners (GitHub.com only) 29 | # Consider using larger runners or machines with greater resources for possible analysis time improvements. 30 | runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }} 31 | permissions: 32 | # required for all workflows 33 | security-events: write 34 | 35 | # required to fetch internal or private CodeQL packs 36 | packages: read 37 | 38 | # only required for workflows in private repositories 39 | actions: read 40 | contents: read 41 | 42 | strategy: 43 | fail-fast: false 44 | matrix: 45 | include: 46 | # CodeQL supports the following values keywords for 'language': 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift' 47 | # Use `c-cpp` to analyze code written in C, C++ or both 48 | # Use 'java-kotlin' to analyze code written in Java, Kotlin or both 49 | # Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both 50 | # To learn more about changing the languages that are analyzed or customizing the build mode for your analysis, 51 | # see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning. 52 | # If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how 53 | # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages 54 | steps: 55 | - name: Checkout repository 56 | uses: actions/checkout@v4 57 | 58 | # Add any setup steps before running the `github/codeql-action/init` action. 59 | # This includes steps like installing compilers or runtimes (`actions/setup-node` 60 | # or others). This is typically only required for manual builds. 61 | # - name: Setup runtime (example) 62 | # uses: actions/setup-example@v1 63 | 64 | # Initializes the CodeQL tools for scanning. 65 | - name: Initialize CodeQL 66 | uses: github/codeql-action/init@v3 67 | with: 68 | languages: ${{ matrix.language }} 69 | build-mode: ${{ matrix.build-mode }} 70 | # If you wish to specify custom queries, you can do so here or in a config file. 71 | # By default, queries listed here will override any specified in a config file. 72 | # Prefix the list here with "+" to use these queries and those in the config file. 73 | 74 | # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs 75 | # queries: security-extended,security-and-quality 76 | 77 | # If the analyze step fails for one of the languages you are analyzing with 78 | # "We were unable to automatically build your code", modify the matrix above 79 | # to set the build mode to "manual" for that language. Then modify this step 80 | # to build your code. 81 | # ℹ️ Command-line programs to run using the OS shell. 82 | # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun 83 | - if: matrix.build-mode == 'manual' 84 | shell: bash 85 | run: | 86 | echo 'If you are using a "manual" build mode for one or more of the' \ 87 | 'languages you are analyzing, replace this with the commands to build' \ 88 | 'your code, for example:' 89 | echo ' make bootstrap' 90 | echo ' make release' 91 | exit 1 92 | 93 | - name: Perform CodeQL Analysis 94 | uses: github/codeql-action/analyze@v3 95 | with: 96 | category: "/language:${{matrix.language}}" 97 | -------------------------------------------------------------------------------- /CODE_OF_CONDUCT.md: -------------------------------------------------------------------------------- 1 | # Contributor Covenant Code of Conduct 2 | 3 | ## Our Pledge 4 | 5 | We as members, contributors, and leaders pledge to make participation in our 6 | community a harassment-free experience for everyone, regardless of age, body 7 | size, visible or invisible disability, ethnicity, sex characteristics, gender 8 | identity and expression, level of experience, education, socio-economic status, 9 | nationality, personal appearance, race, religion, or sexual identity 10 | and orientation. 11 | 12 | We pledge to act and interact in ways that contribute to an open, welcoming, 13 | diverse, inclusive, and healthy community. 14 | 15 | ## Our Standards 16 | 17 | Examples of behavior that contributes to a positive environment for our 18 | community include: 19 | 20 | * Demonstrating empathy and kindness toward other people 21 | * Being respectful of differing opinions, viewpoints, and experiences 22 | * Giving and gracefully accepting constructive feedback 23 | * Accepting responsibility and apologizing to those affected by our mistakes, 24 | and learning from the experience 25 | * Focusing on what is best not just for us as individuals, but for the 26 | overall community 27 | 28 | Examples of unacceptable behavior include: 29 | 30 | * The use of sexualized language or imagery, and sexual attention or 31 | advances of any kind 32 | * Trolling, insulting or derogatory comments, and personal or political attacks 33 | * Public or private harassment 34 | * Publishing others' private information, such as a physical or email 35 | address, without their explicit permission 36 | * Other conduct which could reasonably be considered inappropriate in a 37 | professional setting 38 | 39 | ## Enforcement Responsibilities 40 | 41 | Community leaders are responsible for clarifying and enforcing our standards of 42 | acceptable behavior and will take appropriate and fair corrective action in 43 | response to any behavior that they deem inappropriate, threatening, offensive, 44 | or harmful. 45 | 46 | Community leaders have the right and responsibility to remove, edit, or reject 47 | comments, commits, code, wiki edits, issues, and other contributions that are 48 | not aligned to this Code of Conduct, and will communicate reasons for moderation 49 | decisions when appropriate. 50 | 51 | ## Scope 52 | 53 | This Code of Conduct applies within all community spaces, and also applies when 54 | an individual is officially representing the community in public spaces. 55 | Examples of representing our community include using an official e-mail address, 56 | posting via an official social media account, or acting as an appointed 57 | representative at an online or offline event. 58 | 59 | ## Enforcement 60 | 61 | Instances of abusive, harassing, or otherwise unacceptable behavior may be 62 | reported to the community leaders responsible for enforcement at 63 | kieranxsomer@gmail.com. 64 | All complaints will be reviewed and investigated promptly and fairly. 65 | 66 | All community leaders are obligated to respect the privacy and security of the 67 | reporter of any incident. 68 | 69 | ## Enforcement Guidelines 70 | 71 | Community leaders will follow these Community Impact Guidelines in determining 72 | the consequences for any action they deem in violation of this Code of Conduct: 73 | 74 | ### 1. Correction 75 | 76 | **Community Impact**: Use of inappropriate language or other behavior deemed 77 | unprofessional or unwelcome in the community. 78 | 79 | **Consequence**: A private, written warning from community leaders, providing 80 | clarity around the nature of the violation and an explanation of why the 81 | behavior was inappropriate. A public apology may be requested. 82 | 83 | ### 2. Warning 84 | 85 | **Community Impact**: A violation through a single incident or series 86 | of actions. 87 | 88 | **Consequence**: A warning with consequences for continued behavior. No 89 | interaction with the people involved, including unsolicited interaction with 90 | those enforcing the Code of Conduct, for a specified period of time. This 91 | includes avoiding interactions in community spaces as well as external channels 92 | like social media. Violating these terms may lead to a temporary or 93 | permanent ban. 94 | 95 | ### 3. Temporary Ban 96 | 97 | **Community Impact**: A serious violation of community standards, including 98 | sustained inappropriate behavior. 99 | 100 | **Consequence**: A temporary ban from any sort of interaction or public 101 | communication with the community for a specified period of time. No public or 102 | private interaction with the people involved, including unsolicited interaction 103 | with those enforcing the Code of Conduct, is allowed during this period. 104 | Violating these terms may lead to a permanent ban. 105 | 106 | ### 4. Permanent Ban 107 | 108 | **Community Impact**: Demonstrating a pattern of violation of community 109 | standards, including sustained inappropriate behavior, harassment of an 110 | individual, or aggression toward or disparagement of classes of individuals. 111 | 112 | **Consequence**: A permanent ban from any sort of public interaction within 113 | the community. 114 | 115 | ## Attribution 116 | 117 | This Code of Conduct is adapted from the [Contributor Covenant][homepage], 118 | version 2.0, available at 119 | https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. 120 | 121 | Community Impact Guidelines were inspired by [Mozilla's code of conduct 122 | enforcement ladder](https://github.com/mozilla/diversity). 123 | 124 | [homepage]: https://www.contributor-covenant.org 125 | 126 | For answers to common questions about this code of conduct, see the FAQ at 127 | https://www.contributor-covenant.org/faq. Translations are available at 128 | https://www.contributor-covenant.org/translations. 129 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to [Project Name] 2 | 3 | Thank you for your interest in contributing to [Project Name]! We welcome contributions from everyone. This project primarily uses Python and Jupyter Notebooks for [briefly describe the project's purpose, e.g., data analysis, machine learning, scientific computing]. This document outlines how to get involved and help improve this project. 4 | 5 | ## Table of Contents 6 | 7 | * [Getting Started](#getting-started) 8 | * [Ways to Contribute](#ways-to-contribute) 9 | * [Issue Guidelines](#issue-guidelines) 10 | * [Pull Request Guidelines](#pull-request-guidelines) 11 | * [Development Setup](#development-setup) 12 | * [Using a Virtual Environment (Recommended)](#using-a-virtual-environment-recommended) 13 | * [Installing Dependencies](#installing-dependencies) 14 | * [Working with Jupyter Notebooks](#working-with-jupyter-notebooks) 15 | * [Coding Style](#coding-style) 16 | * [PEP 8](#pep-8) 17 | * [Docstrings](#docstrings) 18 | * [Notebook Best Practices](#notebook-best-practices) 19 | * [Commit Message Guidelines](#commit-message-guidelines) 20 | * [Community](#community) 21 | * [Code of Conduct](#code-of-conduct) 22 | * [License](#license) 23 | 24 | ## Getting Started 25 | 26 | * **Fork the Repository:** On GitHub, click the "Fork" button to create your own copy of the repository. 27 | * **Clone the Repository:** Clone the forked repository to your local machine: 28 | 29 | ```bash 30 | git clone https://github.com/YOUR_USERNAME/REPOSITORY_NAME.git 31 | ``` 32 | 33 | * **Create a Branch:** Create a new branch for your work, using a descriptive name: 34 | 35 | ```bash 36 | git checkout -b feature/my-new-feature 37 | ``` 38 | 39 | ## Ways to Contribute 40 | 41 | There are many ways to contribute: 42 | 43 | * **Reporting Bugs:** If you find a bug, please create a detailed issue (see [Issue Guidelines](#issue-guidelines)). 44 | * **Suggesting Enhancements:** Have an idea for an improvement? Create an issue to discuss it. 45 | * **Writing Code:** Implement new features, fix bugs, or improve existing code. Follow our [Pull Request Guidelines](#pull-request-guidelines) and [Coding Style](#coding-style). 46 | * **Improving Documentation:** Fix typos, clarify explanations, or add new documentation (in code, README, or separate doc files). 47 | * **Developing Tutorials/Examples:** Create Jupyter Notebooks that demonstrate how to use the project or showcase its features. 48 | * **Testing:** Help test the code to ensure quality and identify bugs. 49 | * **Answering Questions:** Assist other users in the issues or community forums. 50 | 51 | ## Issue Guidelines 52 | 53 | Before creating an issue, please search to see if a similar one already exists. When creating a new issue: 54 | 55 | * **Use a clear and descriptive title.** 56 | * **Provide a detailed description of the problem or suggestion.** 57 | * **Include steps to reproduce the bug (if applicable), including code snippets.** 58 | * **Specify your environment (OS, Python version, package versions).** 59 | * **Add relevant screenshots or error messages.** 60 | 61 | ## Pull Request Guidelines 62 | 63 | * **Keep pull requests small and focused.** 64 | * **Base your work on a feature branch.** 65 | * **Write clear commit messages** (see [Commit Message Guidelines](#commit-message-guidelines)). 66 | * **Provide a detailed description of your changes in the pull request.** 67 | * **Ensure your code adheres to our [Coding Style](#coding-style) (PEP 8, docstrings, notebook best practices).** 68 | * **Add tests** if applicable. 69 | * **Make sure all existing tests pass.** 70 | * **Be responsive to feedback.** 71 | * **Squash commits if necessary** before merging. 72 | 73 | ## Development Setup 74 | 75 | ### Using a Virtual Environment (Recommended) 76 | 77 | It's highly recommended to use a virtual environment to isolate project dependencies. You can create one using `venv` (built-in) or `conda` (if you're using Anaconda/Miniconda): 78 | 79 | **Using `venv`:** 80 | 81 | ```bash 82 | python3 -m venv .venv # Create a virtual environment named .venv 83 | source .venv/bin/activate # Activate the environment (macOS/Linux) 84 | .venv\Scripts\activate # Activate the environment (Windows) 85 | ``` 86 | ** Using conda: 87 | 88 | ```bash 89 | conda create -n myenv python=3.9 # Replace 3.9 with your desired Python version 90 | conda activate myenv 91 | ``` 92 | 93 | ### Installing Dependencies 94 | Install the project's dependencies using pip and the provided requirements.txt file: 95 | 96 | ```bash 97 | pip install -r requirements.txt 98 | ``` 99 | 100 | If you need to install additional packages for development, it is good practice to add them to a requirements-dev.txt file (or similar) and install them with pip install -r requirements-dev.txt. 101 | 102 | ## Working with Jupyter Notebooks 103 | 104 | Launch Jupyter: 105 | 106 | 107 | ```bash 108 | jupyter notebook 109 | ``` 110 | 111 | Or if you prefer Jupyter Lab: 112 | 113 | ```bash 114 | jupyter lab 115 | ``` 116 | 117 | - Kernel: Ensure that you're using the correct kernel (the virtual environment you created) within the notebook. 118 | 119 | - nbconvert: If you need to convert notebooks to other formats (e.g., Python scripts, HTML), you can use nbconvert: 120 | 121 | ```bash 122 | jupyter nbconvert --to script mynotebook.ipynb 123 | ``` 124 | - Clear Outputs before Committing: To prevent large diffs and potential merge conflicts arising from stored cell outputs, it's recommended to clear the outputs of all cells in your notebook before committing. You can do this manually within Jupyter by selecting "Kernel" -> "Restart & Clear Output" or use the following command in your terminal: 125 | 126 | ```bash 127 | jupyter nbconvert --ClearOutputPreprocessor.enabled=True --inplace mynotebook.ipynb 128 | ``` 129 | 130 | - Consider nbstripout: If you want to automate the clearing of notebook outputs, you can use the nbstripout tool. Install it using pip: 131 | ```bash 132 | pip install nbstripout 133 | ``` 134 | 135 | - Then, install it as a Git filter in your repository, which will automatically remove outputs whenever you commit a notebook: 136 | 137 | ```bash 138 | nbstripout --install 139 | ``` 140 | This will modify your repository's .gitattributes file to use nbstripout as a filter for .ipynb files. 141 | - Note: If your team members are working with nbstripout, they must also install the filter for it to work correctly on their end. 142 | 143 | ## Notebook Best Practices 144 | 145 | - Keep notebooks concise and focused on a specific task or analysis. 146 | 147 | 148 | - Use Markdown cells to provide clear explanations, context, and narrative. 149 | 150 | 151 | - Organize code into logical sections with headings. 152 | 153 | 154 | - Avoid long code cells; break them into smaller, more manageable chunks. 155 | 156 | 157 | - Use meaningful variable names. 158 | 159 | 160 | - Comment your code where necessary to explain complex logic. 161 | 162 | 163 | - Restart the kernel and run all cells before submitting a pull request to ensure reproducibility. 164 | 165 | 166 | - If your notebook generates large outputs or intermediate data files, consider excluding them from version control (using .gitignore). 167 | 168 | 169 | - Consider testing your notebooks automatically using tools like nbval. 170 | 171 | ## Commit Message Guidelines 172 | 173 | - Use the imperative mood (e.g., "Fix: Resolve issue with data loading"). 174 | 175 | 176 | - Keep the subject line short (under 50 characters). 177 | 178 | 179 | - Wrap the body at 72 characters. 180 | 181 | 182 | - Explain why the change was made, not just what was changed. 183 | 184 | 185 | ### Use a conventional commit format such as: 186 | 187 | 188 | - feat: for new features 189 | 190 | 191 | - fix: for bug fixes 192 | 193 | 194 | - docs: for documentation changes 195 | 196 | 197 | - style: for code style changes (e.g., formatting) 198 | 199 | 200 | - refactor: for code refactoring 201 | 202 | 203 | - test: for adding or updating tests 204 | 205 | 206 | - chore: for maintenance tasks (e.g., updating dependencies) 207 | 208 | Example: 209 | ```bash 210 | fix: Handle missing values in data processing 211 | The previous data processing pipeline did not correctly handle missing values, leading to inaccurate results. This commit adds a step to impute missing values using the mean of each column, ensuring that the analysis is more robust. 212 | ``` 213 | ### Community 214 | 215 | Join our community to discuss the project, ask questions, and get help: 216 | 217 | ADDING THIS LATER I GOTTA REFACTOR MOST OF THIS IN TIME ITS JUST A GUIDE 218 | 219 | ## License 220 | see the LICENSE file for details. 221 | 222 | **Important Notes for Python/Jupyter Projects:** 223 | 224 | * **Virtual Environments:** Emphasize the importance of virtual environments for dependency management. 225 | * **`requirements.txt`:** Make sure you have a `requirements.txt` file (and consider `requirements-dev.txt` for development-specific packages). 226 | * **PEP 8:** Reinforce the importance of following PEP 8 for Python code. 227 | * **Notebook Best Practices:** Provide specific guidance on using Jupyter Notebooks effectively within the project. 228 | * **Testing:** If you have tests (especially for notebooks), explain how to run them. 229 | * **Documentation:** If you use a documentation generator (like Sphinx), provide instructions on how to build the documentation. 230 | 231 | Remember to customize this template with your project's specific details, including links, tools, and community channels. Let me know if you have any further questions! 232 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2025 Ktiseos Nyx 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Notebooks/Colab/HuggingFace_Backup_2024_Colab.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "nbformat": 4, 3 | "nbformat_minor": 0, 4 | "metadata": { 5 | "colab": { 6 | "private_outputs": true, 7 | "provenance": [], 8 | "collapsed_sections": [ 9 | "IZ_JYwvBLrg-", 10 | "PNF2kdyeO3Dn" 11 | ] 12 | }, 13 | "kernelspec": { 14 | "name": "python3", 15 | "display_name": "Python 3" 16 | }, 17 | "language_info": { 18 | "name": "python" 19 | } 20 | }, 21 | "cells": [ 22 | { 23 | "cell_type": "markdown", 24 | "source": [ 25 | "# 🌈 HuggingFace Backup - Colab Edition\n", 26 | "\n", 27 | "Hey there, fellow adventurers in AI! Welcome to our delightfully unique corner of the universe! ✨\n", 28 | "\n", 29 | "## 💫 What's This All About?\n", 30 | "This is our cozy mashup of Nocrypt's notebook and everyday AI tools, but with our own special twist! Think of it as the Colab-flavored version of our Jupyter notebook - because sometimes you need your AI magic on the go!\n", 31 | "\n", 32 | "### 🎯 Quick Features:\n", 33 | "- Fast uploads to HuggingFace (because waiting is overrated!)\n", 34 | "- Focus on checkpoints (keeping it simple and sweet)\n", 35 | "- Built for Google Colab (our cloud-powered playground)\n", 36 | "- All the same magic as our Jupyter version, just with cloud sprinkles! ✨\n", 37 | "\n", 38 | "## 🔍 Lost & Found\n", 39 | "- Repo doing a disappearing act? [Find us on GitHub!](https://github.com/duskfallcrew/HuggingFace_Backup)\n", 40 | "- CivitAI friends: Updates drop faster than a cat chasing a laser!\n", 41 | "\n", 42 | "## ☕ Support the Magic\n", 43 | "[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/Z8Z8L4EO)\n", 44 | "\n", 45 | "## 🎮 Join Our Adventure!\n", 46 | "We're a system of 300+ alters, rocking life with DID, ADHD, Autism, and CPTSD. We believe AI can be a game-changer for mental health and creativity. Come explore with us!\n", 47 | "\n", 48 | "### 🌟 Find Us Here:\n", 49 | "- 🏠 [End Media](https://www.end-media.org/)\n", 50 | "- 🎮 [Discord](https://discord.gg/5t2kYxt7An)\n", 51 | "- 🤗 [HuggingFace](https://huggingface.co/EarthnDusk)\n", 52 | "- 🎵 [YouTube](https://www.youtube.com/channel/UCk7MGP7nrJz5awBSP75xmVw)\n", 53 | "- 🎪 [Reddit](https://www.reddit.com/r/earthndusk/)\n", 54 | "- 🎨 [DeviantArt](https://www.deviantart.com/diffusionai)\n", 55 | "\n", 56 | "### 🏴‍☠️ Special Thanks To:\n", 57 | "- [Pirate Diffusion](https://www.piratediffusion.com/)\n", 58 | "- [Yodayo](https://yodayo.com/)\n", 59 | "\n", 60 | "## 💝 Credits & Cool People\n", 61 | "Huge thanks to our code ancestors:\n", 62 | "- EVERYDREAM2 TRAINER\n", 63 | "- LINAQRUF\n", 64 | "- NOCRYPT [![](https://dcbadge.vercel.app/api/shield/442099748669751297?style=flat)](https://lookup.guru/442099748669751297)\n", 65 | "\n", 66 | "Want more AI adventures? Check out the original SD Colab:\n", 67 | "\n", 68 | " \"Open\n", 69 | "\n", 70 | "\n", 71 | "## 🛠️ Need Help?\n", 72 | "- Found a bug? Don't panic! Drop by our GitHub or Discord\n", 73 | "- Questions? We're here for you!\n", 74 | "- Want to help? Pull requests are like hugs - always welcome!\n", 75 | "\n", 76 | "## 🎯 Recent Updates:\n", 77 | "1. 🌟 Enhanced Colab compatibility\n", 78 | "2. 🔧 Streamlined file handling\n", 79 | "3. 🎨 Made everything prettier (because we can!)\n", 80 | "4. 📝 Clearer instructions (no more confusion!)\n", 81 | "5. ✨ Added extra sprinkles of magic\n", 82 | "\n", 83 | "Remember: We're not pro coders, we're creative problem solvers! Sometimes the best solutions come with a side of chaos and a sprinkle of adventure! 🌈✨\n", 84 | "\n", 85 | "Let's make some AI magic happen! Ready to dive in? 🚀" 86 | ], 87 | "metadata": { 88 | "id": "CxgFI41UGzRu" 89 | } 90 | }, 91 | { 92 | "cell_type": "markdown", 93 | "source": [ 94 | "# Google Drive Mount\n", 95 | "\n" 96 | ], 97 | "metadata": { 98 | "id": "IZ_JYwvBLrg-" 99 | } 100 | }, 101 | { 102 | "cell_type": "code", 103 | "source": [ 104 | "# @title 🚀 Connect Your Google Drive\n", 105 | "# @markdown ## Let's Get Your Storage Powers Activated! ✨\n", 106 | "\n", 107 | "# @markdown ### What's This All About?\n", 108 | "# @markdown Hey there, fellow creator! Before we dive into the fun stuff, let's talk storage!\n", 109 | "\n", 110 | "# @markdown #### 🎯 Your Options:\n", 111 | "# @markdown 1. **Google Drive** (Recommended!)\n", 112 | "# @markdown - Perfect for keeping your models safe\n", 113 | "# @markdown - Easy access to all your files\n", 114 | "# @markdown - Everything stays organized\n", 115 | "# @markdown\n", 116 | "# @markdown 2. **Alternative Routes** (For the adventurous!)\n", 117 | "# @markdown - Terminal wizardry (if you're comfortable with commands)\n", 118 | "# @markdown - Direct uploads to Colab\n", 119 | "# @markdown - Other cloud storage solutions\n", 120 | "# @markdown\n", 121 | "# @markdown #### 🌟 Pro Tips:\n", 122 | "# @markdown - Watch for the popup after clicking \"Connect\"\n", 123 | "# @markdown - You'll need to authorize access (don't worry, it's safe!)\n", 124 | "# @markdown - Your drive will appear as `/content/drive`\n", 125 | "\n", 126 | "from google.colab import drive\n", 127 | "from google.colab import output\n", 128 | "import os\n", 129 | "\n", 130 | "def mount_drive():\n", 131 | " if not os.path.exists('/content/drive'):\n", 132 | " print(\"🔮 Summoning your Google Drive...\")\n", 133 | " drive.mount('/content/drive')\n", 134 | " print(\"\\n✨ Drive mounted successfully! You're ready to rock!\")\n", 135 | " else:\n", 136 | " print(\"🎉 Your Drive is already connected and ready to go!\")\n", 137 | "\n", 138 | "# Enable our magical widgets\n", 139 | "output.enable_custom_widget_manager()\n", 140 | "\n", 141 | "# Let's get that drive connected!\n", 142 | "mount_drive()\n", 143 | "\n", 144 | "print(\"\"\"\n", 145 | "📚 Quick Navigation Guide:\n", 146 | "- Your Drive lives at: /content/drive\n", 147 | "- Main workspace: /content/drive/MyDrive\n", 148 | "- Look for the folder icon in the left sidebar\n", 149 | "\n", 150 | "Need to upload files?\n", 151 | "1. Find the folder icon on the left\n", 152 | "2. Navigate to your drive\n", 153 | "3. Drag and drop your files\n", 154 | "4. Magic happens! ✨\n", 155 | "\n", 156 | "Having troubles? No worries! You can:\n", 157 | "1. Try reconnecting (run this cell again)\n", 158 | "2. Use terminal commands if you're feeling brave\n", 159 | "3. Direct upload to Colab (look for the folder icon!)\n", 160 | "\n", 161 | "Remember: Your creative journey is valid, no matter which path you choose! 🌈\n", 162 | "\"\"\")\n" 163 | ], 164 | "metadata": { 165 | "id": "cGrPfIjkH-rQ", 166 | "cellView": "form" 167 | }, 168 | "execution_count": null, 169 | "outputs": [] 170 | }, 171 | { 172 | "cell_type": "markdown", 173 | "source": [ 174 | "# Installation & Requirements\n", 175 | "-----------------\n", 176 | "\n" 177 | ], 178 | "metadata": { 179 | "id": "J_I_G6_i0rIM" 180 | } 181 | }, 182 | { 183 | "cell_type": "code", 184 | "source": [ 185 | "# @title 🛠️ Installation Wizard\n", 186 | "# @markdown ## Let's get your workspace ready for some AI magic! ✨\n", 187 | "\n", 188 | "# @markdown ### What's in the Box?\n", 189 | "# @markdown - Hugging Face Hub (for all your model adventures)\n", 190 | "# @markdown - IPython Widgets (for a prettier interface)\n", 191 | "# @markdown - Essential file handling tools\n", 192 | "\n", 193 | "print(\"🔍 Starting the setup process...\")\n", 194 | "\n", 195 | "def check_and_install(package, install_name=None, verbose=False):\n", 196 | " \"\"\"Checks if a package is installed and installs it if needed.\"\"\"\n", 197 | " import importlib\n", 198 | " if install_name is None:\n", 199 | " install_name = package\n", 200 | " try:\n", 201 | " importlib.import_module(package)\n", 202 | " print(f\"✅ {package} is already installed.\")\n", 203 | " except ImportError:\n", 204 | " print(f\"📦 Installing {package}...\")\n", 205 | " if verbose:\n", 206 | " !pip install {install_name}\n", 207 | " else:\n", 208 | " !pip install -q {install_name}\n", 209 | " print(f\"✨ {package} installed successfully.\")\n", 210 | " except Exception as e:\n", 211 | " print(f\"❌ An error occurred during install of {package}: {e}\")\n", 212 | "\n", 213 | "# Packages needed for our project\n", 214 | "required_packages = {\n", 215 | " 'huggingface_hub': '--force-reinstall huggingface_hub', #Reinstalls if exists\n", 216 | " 'ipywidgets': 'ipywidgets',\n", 217 | " 'glob': 'glob', # Changed to glob, since glob2 is not essential here\n", 218 | " 'tqdm': 'tqdm' # Install tqdm\n", 219 | "}\n", 220 | "\n", 221 | "# Run installations\n", 222 | "for package, install_cmd in required_packages.items():\n", 223 | " check_and_install(package, install_cmd, verbose=True) #Turn verbose output on\n", 224 | "\n", 225 | "# Clean up output\n", 226 | "from IPython.display import clear_output\n", 227 | "clear_output()\n", 228 | "\n", 229 | "print(\"\"\"\n", 230 | "🎉 All set! Here's a summary of what was installed:\n", 231 | "\n", 232 | "🤗 Hugging Face Hub - Your portal to model magic\n", 233 | "🎨 IPython Widgets - To make things pretty\n", 234 | "📁 File Handling - Tools to keep things organized\n", 235 | "📊 Progress Bars - For better tracking of uploads\n", 236 | "\n", 237 | "Having issues? Here's a checklist:\n", 238 | "1. Try running this cell again\n", 239 | "2. Ensure your Colab runtime is not encountering errors\n", 240 | "3. If still wonky, drop by our Discord for help!\n", 241 | "\n", 242 | "Let's make some AI magic! 🚀\n", 243 | "\"\"\")" 244 | ], 245 | "metadata": { 246 | "cellView": "form", 247 | "id": "Nhi-k9j30ssE" 248 | }, 249 | "execution_count": null, 250 | "outputs": [] 251 | }, 252 | { 253 | "cell_type": "code", 254 | "source": [ 255 | "# @title 🔐 Connect to Hugging Face Hub - Enhanced\n", 256 | "# @markdown ## Let's get you logged into your Hugging Face account! ✨\n", 257 | "\n", 258 | "# @markdown ### Quick Guide:\n", 259 | "# @markdown 1. **New User?** Head over to [Hugging Face](https://huggingface.co/) and create a free account.\n", 260 | "# @markdown 2. **Get Your Token:** Visit your [tokens page](https://huggingface.co/settings/tokens).\n", 261 | "# @markdown 3. **Create a Token:** Click 'New Token' or copy an existing one (ensure it has `Write` access).\n", 262 | "# @markdown 4. **Paste Below:** Copy and paste your token into the field below.\n", 263 | "\n", 264 | "print(\"✨ Checking Hugging Face connection...\")\n", 265 | "\n", 266 | "# Check if hub_ok exists, set it if not\n", 267 | "try:\n", 268 | " hub_ok\n", 269 | "except NameError:\n", 270 | " print(\"📦 Installing huggingface_hub (if needed)...\")\n", 271 | " !pip install --force-reinstall -q huggingface_hub\n", 272 | " hub_ok = True\n", 273 | "\n", 274 | "from IPython.display import clear_output\n", 275 | "from huggingface_hub import login\n", 276 | "clear_output() #clear the output\n", 277 | "\n", 278 | "# @markdown ### 🔑 Your Hugging Face Token\n", 279 | "write_token = \"YOUR TOKEN HERE\" # @param {type:\"string\"}\n", 280 | "\n", 281 | "try:\n", 282 | " if write_token:\n", 283 | " print(\"🔑 Logging you in with your Hugging Face token...\")\n", 284 | " login(write_token, add_to_git_credential=True)\n", 285 | " print(\"🎉 Successfully logged in to Hugging Face! Your powers are activated!\")\n", 286 | " print(\"💫 You're ready to upload models and files.\")\n", 287 | " else:\n", 288 | " print(\"⚠️ No token was provided. Please paste your Hugging Face token above.\")\n", 289 | "except Exception as e:\n", 290 | " print(\"❌ Login failed. Please check the following:\")\n", 291 | " print(f\" 1. Ensure your token is correct and has 'Write' access.\")\n", 292 | " print(f\" 2. The error is: {e}\")\n", 293 | " print(\"🤔 If you're still having problems, visit our Discord for help!\")\n", 294 | " raise" 295 | ], 296 | "metadata": { 297 | "cellView": "form", 298 | "id": "1B8DZ_p8IcNh" 299 | }, 300 | "execution_count": null, 301 | "outputs": [] 302 | }, 303 | { 304 | "cell_type": "code", 305 | "source": [ 306 | "# @title 🏗️ Repository Setup Wizard - Enhanced\n", 307 | "# @markdown ### Let's get your Hugging Face repository ready! ✨\n", 308 | "\n", 309 | "from huggingface_hub.utils import validate_repo_id, HfHubHTTPError\n", 310 | "from huggingface_hub import HfApi\n", 311 | "import os\n", 312 | "\n", 313 | "# @markdown #### 🎨 Customize Your Repository\n", 314 | "repo_name = \"YOUR INFO HERE\" # @param {type:\"string\"}\n", 315 | "make_private = False # @param {type:\"boolean\"} {description:\"Keep your repo private?\"}\n", 316 | "clone_repo = True # @param {type:\"boolean\"} {description:\"Download repo to Colab?\"}\n", 317 | "\n", 318 | "print(\"🛠️ Setting up your Hugging Face repository...\")\n", 319 | "\n", 320 | "try:\n", 321 | " # Initialize the Hugging Face API\n", 322 | " api = HfApi()\n", 323 | " user = api.whoami()\n", 324 | " model_repo = f\"{user['name']}/{repo_name.strip()}\"\n", 325 | "\n", 326 | " # Validate the repository name\n", 327 | " print(f\"🔍 Validating repository name: '{model_repo}'\")\n", 328 | " validate_repo_id(model_repo)\n", 329 | "\n", 330 | " # Create or check the repository\n", 331 | " print(\"📦 Creating repository if it doesn't exist...\")\n", 332 | " api.create_repo(\n", 333 | " repo_id=model_repo,\n", 334 | " private=make_private,\n", 335 | " exist_ok=True\n", 336 | " )\n", 337 | " print(f\"🎉 Repository '{model_repo}' is ready for action!\")\n", 338 | "\n", 339 | " # Clone the repository if requested\n", 340 | " if clone_repo:\n", 341 | " print(\"\\n📦 Cloning your repository (with LFS)...\")\n", 342 | " clone_path = f\"/content/{repo_name.strip()}\"\n", 343 | "\n", 344 | " if os.path.exists(clone_path):\n", 345 | " print(f\"⚠️ Repository already exists at '{clone_path}'. Skipping clone step.\")\n", 346 | " else:\n", 347 | " !git lfs install --skip-smudge\n", 348 | " !export GIT_LFS_SKIP_SMUDGE=1\n", 349 | " !git clone https://huggingface.co/{model_repo} {clone_path}\n", 350 | " print(\"✨ Repository cloned successfully!\")\n", 351 | "\n", 352 | "except Exception as e:\n", 353 | " print(f\"❌ An error occurred during repository setup: {e}\")\n", 354 | " raise\n", 355 | "\n", 356 | "print(\"✅ Repository setup complete!\")" 357 | ], 358 | "metadata": { 359 | "id": "J851eLx6Ii3h", 360 | "cellView": "form" 361 | }, 362 | "execution_count": null, 363 | "outputs": [] 364 | }, 365 | { 366 | "cell_type": "code", 367 | "source": [ 368 | "# @title 🚀 Hugging Face File Uploader - Interactive\n", 369 | "# @markdown ## 🌟 Ready to Upload Your Files to Hugging Face?\n", 370 | "\n", 371 | "# @markdown ### Quick Guide:\n", 372 | "# @markdown 1. Fill in your Hugging Face `Organization/Username` and `Repository name`.\n", 373 | "# @markdown 2. Specify the directory to search in below.\n", 374 | "# @markdown 3. Select the files you want to upload using the file picker below.\n", 375 | "# @markdown 4. Click the `Upload` button to begin the upload process.\n", 376 | "\n", 377 | "# @markdown ### Pro Tips:\n", 378 | "# @markdown - If files are missing, ensure they are in the current directory.\n", 379 | "# @markdown - Use Ctrl/Cmd to select multiple files.\n", 380 | "# @markdown - Customize the commit message.\n", 381 | "\n", 382 | "import os\n", 383 | "import glob\n", 384 | "import time\n", 385 | "from pathlib import Path\n", 386 | "from huggingface_hub import HfApi\n", 387 | "from IPython.display import display, HTML, clear_output, Image\n", 388 | "from tqdm.notebook import tqdm\n", 389 | "from ipywidgets import *\n", 390 | "\n", 391 | "# @markdown ### 📚 Repository Details\n", 392 | "hfuser = \"\" # @param {type:\"string\"} {description: \"Your Hugging Face Username or Organization\"}\n", 393 | "hfrepo = \"\" # @param {type:\"string\"} {description: \"Your Hugging Face Repository Name\"}\n", 394 | "\n", 395 | "# @markdown ### 🗂️ File Settings\n", 396 | "file_type = \"mp3\" # @param [\"safetensors\", \"pt\", \"mp3\", \"pth\", \"onnx\", \"pb\", \"h5\", \"ckpt\", \"bin\", \"json\", \"yaml\", \"yml\", \"txt\", \"csv\", \"pkl\", \"png\", \"jpg\", \"jpeg\", \"webp\", \"gif\", \"zip\", \"tar\", \"gz\", \"mp3\", \"wav\", \"ogg\", \"mp4\", \"mov\", \"avi\", \"mkv\"]\n", 397 | "#sort_by = \"name\" # @param [\"name\", \"date\"] # Removed sort by as it causes issues with dynamically updating\n", 398 | "\n", 399 | "# @markdown ### 📁 File Location\n", 400 | "file_location = None # @param {type:\"string\"} {description: \"The directory to search for files\"}\n", 401 | "\n", 402 | "# @markdown ### 💭 Upload Settings\n", 403 | "commit_message = \"Upload with Earth & Dusk Huggingface 🤗 Backup\" # @param {type:\"string\"}\n", 404 | "create_pr = False # @param {type:\"boolean\"}\n", 405 | "clear_after = True # @param {type:\"boolean\"}\n", 406 | "\n", 407 | "# Initialize HfApi\n", 408 | "api = HfApi()\n", 409 | "\n", 410 | "def format_size(size):\n", 411 | " \"\"\"Formats a file size into a human-readable string.\"\"\"\n", 412 | " for unit in ['B', 'KB', 'MB', 'GB']:\n", 413 | " if size < 1024:\n", 414 | " return f\"{size:.2f} {unit}\"\n", 415 | " size /= 1024\n", 416 | " return f\"{size:.2f} TB\"\n", 417 | "\n", 418 | "def find_files(file_location, file_type):\n", 419 | " \"\"\"Finds files matching the selected file type in the given directory.\"\"\"\n", 420 | " try:\n", 421 | " files = glob.glob(os.path.join(file_location, f\"*.{file_type}\"))\n", 422 | " return files\n", 423 | " except Exception as e:\n", 424 | " print(f\"❌ Error finding files: {e}\")\n", 425 | " return []\n", 426 | "\n", 427 | "# Widget Setup\n", 428 | "all_ckpts = find_files(file_location, file_type)\n", 429 | "ckpt_picker = SelectMultiple(options=all_ckpts, layout=Layout(width=\"600px\", height=\"200px\"), description=\"Select File(s)\")\n", 430 | "upload_btn = Button(description='Upload')\n", 431 | "out = Output()\n", 432 | "\n", 433 | "def upload_ckpts(_):\n", 434 | " repo_id = f\"{hfuser}/{hfrepo}\"\n", 435 | " with out:\n", 436 | " if not ckpt_picker.value:\n", 437 | " print(\"Nothing selected for upload, make sure to click one of the files in the list, or verify there are files in the specified directory.\")\n", 438 | " return\n", 439 | " for ckpt in ckpt_picker.value:\n", 440 | " print(f\"Uploading to HF: huggingface.co/{repo_id}/{os.path.basename(ckpt)}\")\n", 441 | " size = os.path.getsize(ckpt)\n", 442 | " print(f\"📦 Uploading file: {ckpt} ({format_size(size)})\")\n", 443 | "\n", 444 | " try:\n", 445 | " start_time = time.time()\n", 446 | " response = api.upload_file(\n", 447 | " path_or_fileobj=ckpt,\n", 448 | " path_in_repo=os.path.basename(ckpt),\n", 449 | " repo_id=repo_id,\n", 450 | " repo_type=None,\n", 451 | " create_pr=create_pr,\n", 452 | " commit_message=commit_message\n", 453 | " )\n", 454 | " duration = time.time() - start_time\n", 455 | " print(f\"✅ Upload completed in {duration:.1f} seconds\")\n", 456 | " except Exception as e:\n", 457 | " print(f\"❌ Error uploading {ckpt}: {e}\")\n", 458 | " print(\"\\n✨ All uploads complete!\")\n", 459 | " if create_pr:\n", 460 | " print(\"🎉 Check your repository for the new Pull Request!\")\n", 461 | " else:\n", 462 | " print(\"🎉 Files have been uploaded directly to your repository!\")\n", 463 | " if clear_after:\n", 464 | " time.sleep(3)\n", 465 | " clear_output()\n", 466 | "\n", 467 | "upload_btn.on_click(upload_ckpts)\n", 468 | "\n", 469 | "# Create Text widgets for displaying values (read-only)\n", 470 | "hfuser_text = Text(value=hfuser, disabled=True)\n", 471 | "hfrepo_text = Text(value=hfrepo, disabled=True)\n", 472 | "file_location_text = Text(value=file_location, disabled=True)\n", 473 | "\n", 474 | "# Function to update the file list when location or type changes\n", 475 | "def update_file_list(change):\n", 476 | " global all_ckpts\n", 477 | " all_ckpts = find_files(file_location, file_type)\n", 478 | " ckpt_picker.options = all_ckpts\n", 479 | "\n", 480 | "# Function to handle changes in file location\n", 481 | "def on_file_location_change(change):\n", 482 | " global file_location\n", 483 | " file_location = change['new']\n", 484 | " file_location_text.value = file_location\n", 485 | " update_file_list(None)\n", 486 | "\n", 487 | "# Function to handle changes in file type\n", 488 | "def on_file_type_change(change):\n", 489 | " global file_type\n", 490 | " file_type = change['new']\n", 491 | " update_file_list(None)\n", 492 | "\n", 493 | "# Create widgets for interactive updates\n", 494 | "observe_file_location = Text(value=file_location)\n", 495 | "observe_file_location.observe(on_file_location_change, names='value')\n", 496 | "\n", 497 | "observe_file_type = Dropdown(options=[\"safetensors\", \"pt\", \"pth\", \"mp3\", \"onnx\", \"pb\", \"h5\", \"ckpt\", \"bin\", \"json\", \"yaml\", \"yml\", \"txt\", \"csv\", \"pkl\", \"png\", \"jpg\", \"jpeg\", \"webp\", \"gif\", \"zip\", \"tar\", \"gz\", \"mp3\", \"wav\", \"ogg\", \"mp4\", \"mov\", \"avi\", \"mkv\"], value=file_type)\n", 498 | "observe_file_type.observe(on_file_type_change, names='value')\n", 499 | "\n", 500 | "# Display widgets using GridspecLayout for better arrangement\n", 501 | "grid = GridspecLayout(8, 2, height='500px')\n", 502 | "\n", 503 | "grid[0, 0] = Label(\"HF User:\")\n", 504 | "grid[0, 1] = hfuser_text\n", 505 | "grid[1, 0] = Label(\"HF Repo:\")\n", 506 | "grid[1, 1] = hfrepo_text\n", 507 | "grid[2, 0] = Label(\"File Directory:\")\n", 508 | "grid[2, 1] = observe_file_location\n", 509 | "grid[3, 0] = Label(\"File Type:\")\n", 510 | "grid[3, 1] = observe_file_type\n", 511 | "grid[4, :] = ckpt_picker\n", 512 | "grid[5, :] = upload_btn\n", 513 | "grid[6:8, :] = out\n", 514 | "\n", 515 | "display(grid)" 516 | ], 517 | "metadata": { 518 | "id": "lucnlbQQCBZd", 519 | "cellView": "form" 520 | }, 521 | "execution_count": null, 522 | "outputs": [] 523 | } 524 | ] 525 | } -------------------------------------------------------------------------------- /Notebooks/Dataset Zipper/Dataset_Zipper.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "id": "298d71cb-766c-435f-9178-ab20133cb772", 6 | "metadata": {}, 7 | "source": [ 8 | "# 🚀 Zip File Creator and Downloader with ipywidgets\n", 9 | "\n", 10 | "This Jupyter Notebook provides an interactive way to create a zip archive of a selected folder and then download the resulting zip file. It uses `ipywidgets` for the user interface and standard Python libraries for file handling and zipping.\n", 11 | "\n", 12 | "## How it Works\n", 13 | "\n", 14 | "The notebook is built around a few key components:\n", 15 | "\n", 16 | "1. **Folder Selection:**\n", 17 | "\n", 18 | " * **Text Input:** A `Text` widget allows you to manually enter the path to the folder you want to zip.\n", 19 | " * **`ipyfilechooser` : A widget for file and folder choosing. \n", 20 | " * **Error Handling:** The code checks if the entered path is a valid directory. If not, it displays an error message in a dedicated `Output` widget.\n", 21 | "\n", 22 | "2. **Zip File Naming:**\n", 23 | "\n", 24 | " * **Text Input:** Another `Text` widget lets you specify the name of the zip file (without the `.zip` extension, which is added automatically).\n", 25 | " * **Input Validation:** The code ensures that you've entered a name. Leading/trailing whitespace is removed.\n", 26 | "\n", 27 | "3. **Zipping and Downloading:**\n", 28 | "\n", 29 | " * **\"Zip and Download\" Button:** Clicking this button triggers the main zipping and download process.\n", 30 | " * **Temporary Directory:** A temporary directory is created *within the notebook's working directory* to store the zip file *before* downloading. This is crucial to prevent the zip file from being included in the archive itself! This temporary directory is cleaned up afterward.\n", 31 | " * **`zipfile` Module:** The core zipping is handled by Python's built-in `zipfile` module:\n", 32 | " * `zipfile.ZipFile(..., 'w', zipfile.ZIP_DEFLATED)`: Creates a new zip file in write mode (`'w'`) with DEFLATE compression (to reduce file size).\n", 33 | " * `os.walk()`: Recursively walks through the selected folder, getting all files and subfolders.\n", 34 | " * `os.path.relpath()`: Calculates the *relative* path of each file within the folder. This is essential for maintaining the correct directory structure within the zip archive.\n", 35 | " * `zipf.write(file_path, relative_path)`: Adds each file to the zip archive, using the calculated relative path to preserve the folder structure.\n", 36 | " * **`IPython.display.FileLink`:** After the zip file is created, `FileLink` generates a clickable link in the notebook's output area. Clicking this link initiates the download.\n", 37 | " * **Error Handling:** The entire process is wrapped in a `try...except...finally` block. This ensures that:\n", 38 | " * Any errors during the zipping process (e.g., file access problems) are caught.\n", 39 | " * An error message is displayed to the user.\n", 40 | " * The temporary directory is *always* deleted, even if an error occurred. This prevents clutter.\n", 41 | " * **Output Widget:** Messages (success or error) are displayed in a dedicated `Output` widget (`output_area`). This provides feedback to the user. The output area is cleared before each operation.\n", 42 | "\n", 43 | "## How to Use\n", 44 | "\n", 45 | "1. **Install Dependencies:** Make sure you have `ipywidgets` and `jupyter` installed (`pip install ipywidgets jupyter`).\n", 46 | "2. **Install `ipyfilechooser` (Recommended):** `pip install ipyfilechooser`. Then, follow the instructions in the code comments to replace the \"Browse\" button workaround with the `ipyfilechooser` dialog.\n", 47 | "3. **Run the Notebook:** Open the `.ipynb` file in Jupyter Notebook or JupyterLab.\n", 48 | "4. **Select a Folder:** Either type the folder path into the \"Folder\" text box or use the \"Browse\" button (or the `ipyfilechooser` dialog if you've installed it).\n", 49 | "5. **Enter a Zip File Name:** Type the desired name for your zip file in the \"Zip Name\" box (don't include `.zip`).\n", 50 | "6. **Click \"Zip and Download\":** This will create the zip file and display a download link.\n", 51 | "7. **Download:** Click on the displayed link to download the zip file.\n", 52 | "\n", 53 | "## Key Improvements Over Simpler Approaches\n", 54 | "\n", 55 | "* **Robust Error Handling:** Handles various potential errors (invalid paths, file access issues) gracefully.\n", 56 | "* **Temporary Directory:** Prevents the zip file from being included in itself, a common problem.\n", 57 | "* **Relative Paths:** Preserves the directory structure within the zip archive correctly.\n", 58 | "* **Compression:** Uses DEFLATE compression to reduce the zip file size.\n", 59 | "* **Clean Download Link:** Provides a direct download link using `FileLink`.\n", 60 | "* **Organized UI:** Uses `HBox` and `VBox` for a better layout.\n", 61 | "* **Message Output:** Displays informative messages to the user.\n", 62 | "* **Clear Previous Output:** prevents old messages stacking up.\n", 63 | "* **Comprehensive Explanations:** Detailed comments and this markdown cell make the code easier to understand.\n", 64 | "\n", 65 | "This notebook provides a user-friendly and reliable way to create zip archives of folders directly within a Jupyter environment. The use of `ipywidgets` makes it interactive, while the robust error handling and temporary directory usage ensure a smooth and correct zipping process.\n", 66 | "\n", 67 | "**Community & Support:**\n", 68 | "\n", 69 | "* **GitHub:** [Ktiseos Nyx @ Github](https://github.com/Ktiseos-Nyx/HuggingFace_Backup) (for updates, bug reports, and contributions)\n", 70 | "* **Discord:**\n", 71 | " * [Ktiseos Nyx AI/ML Discord](https://discord.gg/HhBSvM9gBY)\n", 72 | " * [Earth & Dusk Media](https://discord.gg/5t2kYxt7An)" 73 | ] 74 | }, 75 | { 76 | "cell_type": "code", 77 | "execution_count": 3, 78 | "id": "c777f23b-1130-4990-a45a-a37a8e905e22", 79 | "metadata": { 80 | "scrolled": true 81 | }, 82 | "outputs": [ 83 | { 84 | "name": "stdout", 85 | "output_type": "stream", 86 | "text": [ 87 | "Requirement already satisfied: ipywidgets in /workspace/venv/lib/python3.10/site-packages (8.1.5)\n", 88 | "Requirement already satisfied: jupyter in /workspace/venv/lib/python3.10/site-packages (1.1.1)\n", 89 | "Requirement already satisfied: widgetsnbextension~=4.0.12 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets) (4.0.13)\n", 90 | "Requirement already satisfied: comm>=0.1.3 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets) (0.2.2)\n", 91 | "Requirement already satisfied: jupyterlab-widgets~=3.0.12 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets) (3.0.13)\n", 92 | "Requirement already satisfied: ipython>=6.1.0 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets) (8.28.0)\n", 93 | "Requirement already satisfied: traitlets>=4.3.1 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets) (5.14.3)\n", 94 | "Requirement already satisfied: jupyterlab in /workspace/venv/lib/python3.10/site-packages (from jupyter) (4.3.6)\n", 95 | "Requirement already satisfied: notebook in /workspace/venv/lib/python3.10/site-packages (from jupyter) (7.3.3)\n", 96 | "Requirement already satisfied: ipykernel in /workspace/venv/lib/python3.10/site-packages (from jupyter) (6.29.5)\n", 97 | "Requirement already satisfied: jupyter-console in /workspace/venv/lib/python3.10/site-packages (from jupyter) (6.6.3)\n", 98 | "Requirement already satisfied: nbconvert in /workspace/venv/lib/python3.10/site-packages (from jupyter) (7.16.4)\n", 99 | "Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.48)\n", 100 | "Requirement already satisfied: pygments>=2.4.0 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (2.18.0)\n", 101 | "Requirement already satisfied: pexpect>4.3 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (4.9.0)\n", 102 | "Requirement already satisfied: jedi>=0.16 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (0.19.1)\n", 103 | "Requirement already satisfied: matplotlib-inline in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.7)\n", 104 | "Requirement already satisfied: exceptiongroup in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (1.2.2)\n", 105 | "Requirement already satisfied: typing-extensions>=4.6 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (4.12.2)\n", 106 | "Requirement already satisfied: decorator in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)\n", 107 | "Requirement already satisfied: stack-data in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets) (0.6.3)\n", 108 | "Requirement already satisfied: psutil in /workspace/venv/lib/python3.10/site-packages (from ipykernel->jupyter) (5.9.5)\n", 109 | "Requirement already satisfied: debugpy>=1.6.5 in /workspace/venv/lib/python3.10/site-packages (from ipykernel->jupyter) (1.8.7)\n", 110 | "Requirement already satisfied: tornado>=6.1 in /workspace/venv/lib/python3.10/site-packages (from ipykernel->jupyter) (6.4.1)\n", 111 | "Requirement already satisfied: pyzmq>=24 in /workspace/venv/lib/python3.10/site-packages (from ipykernel->jupyter) (26.2.0)\n", 112 | "Requirement already satisfied: packaging in /workspace/venv/lib/python3.10/site-packages (from ipykernel->jupyter) (24.1)\n", 113 | "Requirement already satisfied: nest-asyncio in /workspace/venv/lib/python3.10/site-packages (from ipykernel->jupyter) (1.6.0)\n", 114 | "Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /workspace/venv/lib/python3.10/site-packages (from ipykernel->jupyter) (5.7.2)\n", 115 | "Requirement already satisfied: jupyter-client>=6.1.12 in /workspace/venv/lib/python3.10/site-packages (from ipykernel->jupyter) (8.6.3)\n", 116 | "Requirement already satisfied: jupyter-lsp>=2.0.0 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (2.2.5)\n", 117 | "Requirement already satisfied: notebook-shim>=0.2 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (0.2.4)\n", 118 | "Requirement already satisfied: httpx>=0.25.0 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (0.28.1)\n", 119 | "Requirement already satisfied: jinja2>=3.0.3 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (3.1.4)\n", 120 | "Requirement already satisfied: tomli>=1.2.2 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (2.0.2)\n", 121 | "Requirement already satisfied: setuptools>=40.8.0 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (69.5.1)\n", 122 | "Requirement already satisfied: jupyter-server<3,>=2.4.0 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (2.14.2)\n", 123 | "Requirement already satisfied: jupyterlab-server<3,>=2.27.1 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (2.27.3)\n", 124 | "Requirement already satisfied: async-lru>=1.0.0 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab->jupyter) (2.0.4)\n", 125 | "Requirement already satisfied: bleach!=5.0.0 in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (6.1.0)\n", 126 | "Requirement already satisfied: mistune<4,>=2.0.3 in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (3.0.2)\n", 127 | "Requirement already satisfied: jupyterlab-pygments in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (0.3.0)\n", 128 | "Requirement already satisfied: nbformat>=5.7 in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (5.10.4)\n", 129 | "Requirement already satisfied: nbclient>=0.5.0 in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (0.10.0)\n", 130 | "Requirement already satisfied: beautifulsoup4 in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (4.12.3)\n", 131 | "Requirement already satisfied: defusedxml in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (0.7.1)\n", 132 | "Requirement already satisfied: markupsafe>=2.0 in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (2.1.5)\n", 133 | "Requirement already satisfied: pandocfilters>=1.4.1 in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (1.5.1)\n", 134 | "Requirement already satisfied: tinycss2 in /workspace/venv/lib/python3.10/site-packages (from nbconvert->jupyter) (1.4.0)\n", 135 | "Requirement already satisfied: six>=1.9.0 in /workspace/venv/lib/python3.10/site-packages (from bleach!=5.0.0->nbconvert->jupyter) (1.16.0)\n", 136 | "Requirement already satisfied: webencodings in /workspace/venv/lib/python3.10/site-packages (from bleach!=5.0.0->nbconvert->jupyter) (0.5.1)\n", 137 | "Requirement already satisfied: anyio in /workspace/venv/lib/python3.10/site-packages (from httpx>=0.25.0->jupyterlab->jupyter) (3.7.1)\n", 138 | "Requirement already satisfied: certifi in /workspace/venv/lib/python3.10/site-packages (from httpx>=0.25.0->jupyterlab->jupyter) (2024.8.30)\n", 139 | "Requirement already satisfied: idna in /workspace/venv/lib/python3.10/site-packages (from httpx>=0.25.0->jupyterlab->jupyter) (3.10)\n", 140 | "Requirement already satisfied: httpcore==1.* in /workspace/venv/lib/python3.10/site-packages (from httpx>=0.25.0->jupyterlab->jupyter) (1.0.7)\n", 141 | "Requirement already satisfied: h11<0.15,>=0.13 in /workspace/venv/lib/python3.10/site-packages (from httpcore==1.*->httpx>=0.25.0->jupyterlab->jupyter) (0.14.0)\n", 142 | "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /workspace/venv/lib/python3.10/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.4)\n", 143 | "Requirement already satisfied: python-dateutil>=2.8.2 in /workspace/venv/lib/python3.10/site-packages (from jupyter-client>=6.1.12->ipykernel->jupyter) (2.9.0.post0)\n", 144 | "Requirement already satisfied: platformdirs>=2.5 in /workspace/venv/lib/python3.10/site-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel->jupyter) (4.3.6)\n", 145 | "Requirement already satisfied: prometheus-client>=0.9 in /workspace/venv/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (0.21.0)\n", 146 | "Requirement already satisfied: argon2-cffi>=21.1 in /workspace/venv/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (23.1.0)\n", 147 | "Requirement already satisfied: jupyter-events>=0.9.0 in /workspace/venv/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (0.10.0)\n", 148 | "Requirement already satisfied: send2trash>=1.8.2 in /workspace/venv/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (1.8.3)\n", 149 | "Requirement already satisfied: overrides>=5.0 in /workspace/venv/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (7.7.0)\n", 150 | "Requirement already satisfied: terminado>=0.8.3 in /workspace/venv/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (0.18.1)\n", 151 | "Requirement already satisfied: websocket-client>=1.7 in /workspace/venv/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (1.8.0)\n", 152 | "Requirement already satisfied: jupyter-server-terminals>=0.4.4 in /workspace/venv/lib/python3.10/site-packages (from jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (0.5.3)\n", 153 | "Requirement already satisfied: jsonschema>=4.18.0 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (4.23.0)\n", 154 | "Requirement already satisfied: babel>=2.10 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (2.16.0)\n", 155 | "Requirement already satisfied: requests>=2.31 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (2.32.3)\n", 156 | "Requirement already satisfied: json5>=0.9.0 in /workspace/venv/lib/python3.10/site-packages (from jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (0.9.25)\n", 157 | "Requirement already satisfied: fastjsonschema>=2.15 in /workspace/venv/lib/python3.10/site-packages (from nbformat>=5.7->nbconvert->jupyter) (2.20.0)\n", 158 | "Requirement already satisfied: ptyprocess>=0.5 in /workspace/venv/lib/python3.10/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)\n", 159 | "Requirement already satisfied: wcwidth in /workspace/venv/lib/python3.10/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=6.1.0->ipywidgets) (0.2.13)\n", 160 | "Requirement already satisfied: soupsieve>1.2 in /workspace/venv/lib/python3.10/site-packages (from beautifulsoup4->nbconvert->jupyter) (2.6)\n", 161 | "Requirement already satisfied: asttokens>=2.1.0 in /workspace/venv/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.4.1)\n", 162 | "Requirement already satisfied: executing>=1.2.0 in /workspace/venv/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.1.0)\n", 163 | "Requirement already satisfied: pure-eval in /workspace/venv/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.3)\n", 164 | "Requirement already satisfied: sniffio>=1.1 in /workspace/venv/lib/python3.10/site-packages (from anyio->httpx>=0.25.0->jupyterlab->jupyter) (1.3.1)\n", 165 | "Requirement already satisfied: argon2-cffi-bindings in /workspace/venv/lib/python3.10/site-packages (from argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (21.2.0)\n", 166 | "Requirement already satisfied: rpds-py>=0.7.1 in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (0.20.0)\n", 167 | "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (2024.10.1)\n", 168 | "Requirement already satisfied: attrs>=22.2.0 in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (24.2.0)\n", 169 | "Requirement already satisfied: referencing>=0.28.4 in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (0.35.1)\n", 170 | "Requirement already satisfied: rfc3986-validator>=0.1.1 in /workspace/venv/lib/python3.10/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (0.1.1)\n", 171 | "Requirement already satisfied: pyyaml>=5.3 in /workspace/venv/lib/python3.10/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (6.0.2)\n", 172 | "Requirement already satisfied: rfc3339-validator in /workspace/venv/lib/python3.10/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (0.1.4)\n", 173 | "Requirement already satisfied: python-json-logger>=2.0.4 in /workspace/venv/lib/python3.10/site-packages (from jupyter-events>=0.9.0->jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (2.0.7)\n", 174 | "Requirement already satisfied: urllib3<3,>=1.21.1 in /workspace/venv/lib/python3.10/site-packages (from requests>=2.31->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (2.2.3)\n", 175 | "Requirement already satisfied: charset-normalizer<4,>=2 in /workspace/venv/lib/python3.10/site-packages (from requests>=2.31->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (3.4.0)\n", 176 | "Requirement already satisfied: jsonpointer>1.13 in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (3.0.0)\n", 177 | "Requirement already satisfied: uri-template in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (1.3.0)\n", 178 | "Requirement already satisfied: fqdn in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (1.5.1)\n", 179 | "Requirement already satisfied: isoduration in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (20.11.0)\n", 180 | "Requirement already satisfied: webcolors>=24.6.0 in /workspace/venv/lib/python3.10/site-packages (from jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (24.8.0)\n", 181 | "Requirement already satisfied: cffi>=1.0.1 in /workspace/venv/lib/python3.10/site-packages (from argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (1.17.1)\n", 182 | "Requirement already satisfied: pycparser in /workspace/venv/lib/python3.10/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi>=21.1->jupyter-server<3,>=2.4.0->jupyterlab->jupyter) (2.22)\n", 183 | "Requirement already satisfied: arrow>=0.15.0 in /workspace/venv/lib/python3.10/site-packages (from isoduration->jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (1.3.0)\n", 184 | "Requirement already satisfied: types-python-dateutil>=2.8.10 in /workspace/venv/lib/python3.10/site-packages (from arrow>=0.15.0->isoduration->jsonschema>=4.18.0->jupyterlab-server<3,>=2.27.1->jupyterlab->jupyter) (2.9.0.20241003)\n", 185 | "Collecting ipyfilechooser\n", 186 | " Downloading ipyfilechooser-0.6.0-py3-none-any.whl (11 kB)\n", 187 | "Requirement already satisfied: ipywidgets in /workspace/venv/lib/python3.10/site-packages (from ipyfilechooser) (8.1.5)\n", 188 | "Requirement already satisfied: widgetsnbextension~=4.0.12 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets->ipyfilechooser) (4.0.13)\n", 189 | "Requirement already satisfied: traitlets>=4.3.1 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets->ipyfilechooser) (5.14.3)\n", 190 | "Requirement already satisfied: comm>=0.1.3 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets->ipyfilechooser) (0.2.2)\n", 191 | "Requirement already satisfied: jupyterlab-widgets~=3.0.12 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets->ipyfilechooser) (3.0.13)\n", 192 | "Requirement already satisfied: ipython>=6.1.0 in /workspace/venv/lib/python3.10/site-packages (from ipywidgets->ipyfilechooser) (8.28.0)\n", 193 | "Requirement already satisfied: typing-extensions>=4.6 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (4.12.2)\n", 194 | "Requirement already satisfied: pexpect>4.3 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (4.9.0)\n", 195 | "Requirement already satisfied: pygments>=2.4.0 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (2.18.0)\n", 196 | "Requirement already satisfied: stack-data in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (0.6.3)\n", 197 | "Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (3.0.48)\n", 198 | "Requirement already satisfied: matplotlib-inline in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (0.1.7)\n", 199 | "Requirement already satisfied: jedi>=0.16 in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (0.19.1)\n", 200 | "Requirement already satisfied: exceptiongroup in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (1.2.2)\n", 201 | "Requirement already satisfied: decorator in /workspace/venv/lib/python3.10/site-packages (from ipython>=6.1.0->ipywidgets->ipyfilechooser) (5.1.1)\n", 202 | "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /workspace/venv/lib/python3.10/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets->ipyfilechooser) (0.8.4)\n", 203 | "Requirement already satisfied: ptyprocess>=0.5 in /workspace/venv/lib/python3.10/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets->ipyfilechooser) (0.7.0)\n", 204 | "Requirement already satisfied: wcwidth in /workspace/venv/lib/python3.10/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=6.1.0->ipywidgets->ipyfilechooser) (0.2.13)\n", 205 | "Requirement already satisfied: pure-eval in /workspace/venv/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets->ipyfilechooser) (0.2.3)\n", 206 | "Requirement already satisfied: asttokens>=2.1.0 in /workspace/venv/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets->ipyfilechooser) (2.4.1)\n", 207 | "Requirement already satisfied: executing>=1.2.0 in /workspace/venv/lib/python3.10/site-packages (from stack-data->ipython>=6.1.0->ipywidgets->ipyfilechooser) (2.1.0)\n", 208 | "Requirement already satisfied: six>=1.12.0 in /workspace/venv/lib/python3.10/site-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets->ipyfilechooser) (1.16.0)\n", 209 | "Installing collected packages: ipyfilechooser\n", 210 | "Successfully installed ipyfilechooser-0.6.0\n" 211 | ] 212 | } 213 | ], 214 | "source": [ 215 | "!pip install ipywidgets jupyter\n", 216 | "!pip install ipyfilechooser" 217 | ] 218 | }, 219 | { 220 | "cell_type": "code", 221 | "execution_count": 6, 222 | "id": "64653b38-86f2-4aed-a9f7-fdbaed7c2248", 223 | "metadata": {}, 224 | "outputs": [ 225 | { 226 | "data": { 227 | "application/vnd.jupyter.widget-view+json": { 228 | "model_id": "733ce8d86ac3469893a821569acc8764", 229 | "version_major": 2, 230 | "version_minor": 0 231 | }, 232 | "text/plain": [ 233 | "VBox(children=(VBox(children=(FileChooser(path='/workspace/stable-diffusion-webui/outputs/txt2img-images', fil…" 234 | ] 235 | }, 236 | "metadata": {}, 237 | "output_type": "display_data" 238 | } 239 | ], 240 | "source": [ 241 | "import ipywidgets as widgets\n", 242 | "from IPython.display import display, FileLink\n", 243 | "import zipfile\n", 244 | "import os\n", 245 | "import shutil\n", 246 | "from ipywidgets import Layout\n", 247 | "from ipyfilechooser import FileChooser\n", 248 | "\n", 249 | "# --- 1. Folder Selection Widget (using ipyfilechooser) ---\n", 250 | "\n", 251 | "folder_chooser = FileChooser(os.getcwd())\n", 252 | "folder_chooser.title = \"Select a Folder\"\n", 253 | "\n", 254 | "folder_selector = widgets.Text(\n", 255 | " value='',\n", 256 | " placeholder='Enter folder path or select using the chooser',\n", 257 | " description='Folder:',\n", 258 | " disabled=False,\n", 259 | " layout=Layout(width='80%')\n", 260 | ")\n", 261 | "\n", 262 | "def on_folder_selected(chooser):\n", 263 | " if chooser.selected:\n", 264 | " folder_selector.value = chooser.selected\n", 265 | "\n", 266 | "folder_chooser.register_callback(on_folder_selected)\n", 267 | "\n", 268 | "folder_selection_vbox = widgets.VBox([folder_chooser, folder_selector])\n", 269 | "\n", 270 | "# --- 2. Zip File Name Widget ---\n", 271 | "\n", 272 | "zip_name_input = widgets.Text(\n", 273 | " value='my_archive',\n", 274 | " placeholder='Enter zip file name (without .zip)',\n", 275 | " description='Zip Name:',\n", 276 | " disabled=False,\n", 277 | " layout=Layout(width='50%')\n", 278 | ")\n", 279 | "\n", 280 | "# --- 3. Zip and Download Button ---\n", 281 | "\n", 282 | "zip_button = widgets.Button(description=\"Zip and Download\")\n", 283 | "output_area = widgets.Output()\n", 284 | "\n", 285 | "def zip_and_download(b):\n", 286 | " with output_area:\n", 287 | " output_area.clear_output()\n", 288 | " folder_path = folder_selector.value\n", 289 | " zip_file_name = zip_name_input.value.strip()\n", 290 | "\n", 291 | " if not folder_path:\n", 292 | " print(\"Please select a folder.\")\n", 293 | " return\n", 294 | " if not zip_file_name:\n", 295 | " print(\"Please enter a zip file name.\")\n", 296 | " return\n", 297 | " if not os.path.isdir(folder_path):\n", 298 | " print(\"Invalid folder path. Please provide a valid directory\")\n", 299 | " return\n", 300 | "\n", 301 | " zip_file_path = zip_file_name + '.zip'\n", 302 | " # Path in the *notebook's* directory (where the link will work)\n", 303 | " final_zip_path = os.path.join(os.getcwd(), zip_file_path)\n", 304 | "\n", 305 | " try:\n", 306 | " temp_dir = \"temp_zip_dir\"\n", 307 | " os.makedirs(temp_dir, exist_ok=True)\n", 308 | " temp_zip_path = os.path.join(temp_dir, zip_file_path)\n", 309 | "\n", 310 | " with zipfile.ZipFile(temp_zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:\n", 311 | " for root, _, files in os.walk(folder_path):\n", 312 | " for file in files:\n", 313 | " file_path = os.path.join(root, file)\n", 314 | " relative_path = os.path.relpath(file_path, folder_path)\n", 315 | " zipf.write(file_path, relative_path)\n", 316 | "\n", 317 | " # *** Copy the zip file to the notebook's directory ***\n", 318 | " shutil.copy2(temp_zip_path, final_zip_path)\n", 319 | "\n", 320 | " print(f\"Successfully created {zip_file_path}\")\n", 321 | " # *** Use the path in the notebook's directory for FileLink ***\n", 322 | " display(FileLink(final_zip_path))\n", 323 | "\n", 324 | " except Exception as e:\n", 325 | " print(f\"Error creating zip file: {e}\")\n", 326 | " finally:\n", 327 | " shutil.rmtree(temp_dir, ignore_errors=True)\n", 328 | "\n", 329 | "zip_button.on_click(zip_and_download)\n", 330 | "\n", 331 | "# --- 4. Display Widgets ---\n", 332 | "\n", 333 | "display(widgets.VBox([folder_selection_vbox, zip_name_input, zip_button, output_area]))" 334 | ] 335 | }, 336 | { 337 | "cell_type": "code", 338 | "execution_count": null, 339 | "id": "5eb32828-7005-4e20-a69c-f5c5cad7108d", 340 | "metadata": {}, 341 | "outputs": [], 342 | "source": [] 343 | } 344 | ], 345 | "metadata": { 346 | "kernelspec": { 347 | "display_name": "Python 3 (ipykernel)", 348 | "language": "python", 349 | "name": "python3" 350 | }, 351 | "language_info": { 352 | "codemirror_mode": { 353 | "name": "ipython", 354 | "version": 3 355 | }, 356 | "file_extension": ".py", 357 | "mimetype": "text/x-python", 358 | "name": "python", 359 | "nbconvert_exporter": "python", 360 | "pygments_lexer": "ipython3", 361 | "version": "3.10.12" 362 | } 363 | }, 364 | "nbformat": 4, 365 | "nbformat_minor": 5 366 | } 367 | -------------------------------------------------------------------------------- /Notebooks/Previous Notebooks/HuggingFace_Backup_Jupyter_2025_EXTRA.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 🚀 Welcome to the Hugging Face File Uploader!\n", 8 | "\n", 9 | "This Jupyter Notebook provides an easy-to-use interface for uploading files directly to your Hugging Face repositories.\n", 10 | "Now including a way to zip up files for you to port to huggingface! No more \"API LIMITS\"! \n", 11 | "\n", 12 | "**Getting Started (One-Time Setup):**\n", 13 | "\n", 14 | "1. **Install Dependencies:**\n", 15 | " * Run the first code cell in this notebook to install the required libraries (`huggingface_hub`, `ipywidgets`, etc.). You only need to do this *once* per environment.\n", 16 | "2. **Authenticate with Hugging Face:**\n", 17 | " * Run the `notebook_login()` cell (usually the second cell). This will prompt you for your Hugging Face API token.\n", 18 | " * Get your API token from your [Hugging Face settings](https://huggingface.co/settings/tokens) (make sure it has **write** access).\n", 19 | " * **🚨 Security:** After logging in, *never* share your notebook file (`.ipynb`) or commit it to a repository. The `notebook_login()` function handles your token securely, but sharing the notebook could still expose sensitive information.\n", 20 | "\n", 21 | "**Using the Uploader:**\n", 22 | "\n", 23 | "See the detailed instructions in the \"🚀 Using the Hugging Face File Uploader\" section below. The key steps are:\n", 24 | "\n", 25 | "1. **Enter Repository Details:** Provide your username/organization, repository name, and type.\n", 26 | "2. **Select a Directory:** Specify the folder containing your files.\n", 27 | "3. **Choose Files:** Select the files you want to upload.\n", 28 | "4. **Set Upload Options:** Choose whether to create a pull request and clear output.\n", 29 | "5. **Click \"⬆️ Upload\":** Start the upload process.\n", 30 | "\n", 31 | "**Using the Zipfile Cell**\n", 32 | "\n", 33 | "1. **Install Dependencies:** Make sure you have `ipywidgets` and `jupyter` installed (`pip install ipywidgets jupyter`).\n", 34 | "2. **Install `ipyfilechooser` (Recommended):** `pip install ipyfilechooser`. Then, follow the instructions in the code comments to replace the \"Browse\" button workaround with the `ipyfilechooser` dialog.\n", 35 | "3. **Run the Notebook:** Open the `.ipynb` file in Jupyter Notebook or JupyterLab.\n", 36 | "4. **Select a Folder:** Either type the folder path into the \"Folder\" text box or use the \"Browse\" button (or the `ipyfilechooser` dialog if you've installed it).\n", 37 | "5. **Enter a Zip File Name:** Type the desired name for your zip file in the \"Zip Name\" box (don't include `.zip`).\n", 38 | "6. **Click \"Zip and Download\":** This will create the zip file and display a download link.\n", 39 | "7. **Download:** Click on the displayed link to download the zip file.\n", 40 | "\n", 41 | "**Key Features & Notes:**\n", 42 | "\n", 43 | "* **Direct Uploads:** This uploader uses the Hugging Face API for direct file uploads. You don't need to use `git` commands directly for basic uploads.\n", 44 | "* **Interactive Interface:** The uploader provides a user-friendly interface within the Jupyter Notebook.\n", 45 | "* **Progress Tracking:** You'll see real-time progress updates during the upload.\n", 46 | "* **Error Handling:** Detailed error messages will help you troubleshoot any problems.\n", 47 | "* **Troubleshooting** If you encounter issues, double-check your credentials, repo name, write permissions, etc.\n", 48 | "* **Robust Error Handling:** Handles various potential errors (invalid paths, file access issues) gracefully.\n", 49 | "* **Temporary Directory:** Prevents the zip file from being included in itself, a common problem.\n", 50 | "* **Relative Paths:** Preserves the directory structure within the zip archive correctly.\n", 51 | "* **Compression:** Uses DEFLATE compression to reduce the zip file size.\n", 52 | "* **Clean Download Link:** Provides a direct download link using `FileLink`.\n", 53 | "* **Organized UI:** Uses `HBox` and `VBox` for a better layout.\n", 54 | "* **Message Output:** Displays informative messages to the user.\n", 55 | "* **Clear Previous Output:** prevents old messages stacking up.\n", 56 | "* **Comprehensive Explanations:** Detailed comments and this markdown cell make the code easier to understand.\n", 57 | "\n", 58 | "\n", 59 | "**Community & Support:**\n", 60 | "\n", 61 | "* **GitHub:** [Ktiseos Nyx @ Github](https://github.com/Ktiseos-Nyx/HuggingFace_Backup) (for updates, bug reports, and contributions)\n", 62 | "* **Discord:**\n", 63 | " * [Ktiseos Nyx AI/ML Discord](https://discord.gg/HhBSvM9gBY)\n", 64 | " * [Earth & Dusk Media](https://discord.gg/5t2kYxt7An)\n", 65 | "\n", 66 | "This uploader is designed to simplify the process of uploading files to Hugging Face. We hope you find it useful!" 67 | ] 68 | }, 69 | { 70 | "cell_type": "markdown", 71 | "metadata": {}, 72 | "source": [ 73 | "# Install Dependencies" 74 | ] 75 | }, 76 | { 77 | "cell_type": "code", 78 | "execution_count": 1, 79 | "metadata": {}, 80 | "outputs": [], 81 | "source": [ 82 | "!pip install -q huggingface_hub ipywidgets ipyfilechooser" 83 | ] 84 | }, 85 | { 86 | "cell_type": "markdown", 87 | "metadata": { 88 | "id": "Xs1mb1VKLuUW" 89 | }, 90 | "source": [ 91 | "# ✨ Connecting to Hugging Face: Authentication\n", 92 | "\n", 93 | "To upload files to Hugging Face, you need to authenticate with your Hugging Face API token. This token grants your notebook secure access to the Hugging Face Hub.\n", 94 | "\n", 95 | "**🔑 Get Your API Token**\n", 96 | "\n", 97 | "1. **Go to your Hugging Face settings:** [Hugging Face Access Tokens](https://huggingface.co/settings/tokens).\n", 98 | "2. **Find \"Access Tokens\":** Locate the section labeled \"Access Tokens\".\n", 99 | "3. **Create or Copy:**\n", 100 | " * If you don't have a token, create a new one with **write** access. *Crucially, you need write access to upload.*\n", 101 | " * If you have an existing token, copy it.\n", 102 | "\n", 103 | "**🔒 Login to Hugging Face in this Notebook**\n", 104 | "\n", 105 | "1. **Run the next cell:** Execute the code cell immediately below this one.\n", 106 | "2. **Paste your token:** A prompt will appear. Paste your copied API token into the input field. *You won't see any characters as you type or paste – this is for security.*\n", 107 | "3. **Press Enter:** This completes the login process.\n", 108 | "\n", 109 | "**🚨 Important Security Notes 🚨**\n", 110 | "\n", 111 | "* **Why you need to log in:** This login authorizes *this notebook environment* to interact with the Hugging Face Hub on your behalf. It's required for uploading.\n", 112 | "* **Session-specific:** The login is only valid for the *current notebook session*. If you restart the kernel or open the notebook later, you'll need to log in again. The token is *not* stored within the notebook itself, enhancing security.\n", 113 | "* **`notebook_login()` is secure:** The `notebook_login()` function from the `huggingface_hub` library handles the token securely. It does *not* store the token in plain text within the notebook.\n", 114 | "* **Do NOT share your notebook:** After logging in, **never** share your notebook file (`.ipynb`) with others or commit it to a public (or even private, but shared) repository. While the token isn't *directly* visible in the notebook file, the notebook's internal state might contain information that could be used to compromise your account. Treat the logged-in notebook as a sensitive document.\n", 115 | "* **Do not store the token in your code.**\n", 116 | "\n", 117 | "**Troubleshooting**\n", 118 | "\n", 119 | "* **Authentication Errors:** If you encounter authentication issues after entering your API token:\n", 120 | " * **Local Development:** If you are running Jupyter Notebook on your *own computer*, open your system terminal (Terminal on macOS, Command Prompt/PowerShell on Windows).\n", 121 | " * **Rented GPU / Cloud Environment:** If you are using a *rented GPU service* (Paperspace, RunPod, Google Colab, etc.), open a terminal *within your Jupyter environment*. This is usually done through a \"New\" -> \"Terminal\" option in the Jupyter file browser.\n", 122 | " * **Run this command (in the appropriate terminal):**\n", 123 | " ```bash\n", 124 | " git config --global credential.helper store\n", 125 | " ```\n", 126 | " This configures Git to store credentials, which can sometimes resolve conflicts. It does *not* store your Hugging Face token in plain text. It's a general Git configuration setting. If you are still concerned with security, you may remove this credential after.\n", 127 | "\n", 128 | "* **Still having trouble?** Consider refreshing your Hugging Face token (creating a new one and deleting the old one) on the [Hugging Face website](https://huggingface.co/settings/tokens), and then try logging in again with the new token. This is a good security practice if you suspect your token might have been compromised." 129 | ] 130 | }, 131 | { 132 | "cell_type": "code", 133 | "execution_count": 2, 134 | "metadata": {}, 135 | "outputs": [ 136 | { 137 | "data": { 138 | "application/vnd.jupyter.widget-view+json": { 139 | "model_id": "f6ba9021f2ef4d0e8cd933141e1f0178", 140 | "version_major": 2, 141 | "version_minor": 0 142 | }, 143 | "text/plain": [ 144 | "VBox(children=(HTML(value='
📚 Repository Details'), HBox(children=(Text(value='', description='Owner:', …" 351 | ] 352 | }, 353 | "metadata": {}, 354 | "output_type": "display_data" 355 | } 356 | ], 357 | "source": [ 358 | "import glob\n", 359 | "import os\n", 360 | "import time # Standard library imports\n", 361 | "from pathlib import Path\n", 362 | "\n", 363 | "from huggingface_hub import HfApi # Third-party imports\n", 364 | "from ipywidgets import (Text, Dropdown, Button, SelectMultiple, VBox, HBox,\n", 365 | " Output, Layout, Checkbox, HTML, Textarea, Label,\n", 366 | " FloatProgress)\n", 367 | "from IPython.display import display, clear_output\n", 368 | "\n", 369 | "# from tqdm import tqdm No longer needed\n", 370 | "\n", 371 | "class HuggingFaceUploader:\n", 372 | "\n", 373 | " def __init__(self):\n", 374 | " self.api = HfApi()\n", 375 | " self.file_types = [\n", 376 | " # AI Model Files 🤖\n", 377 | " ('SafeTensors', 'safetensors'),\n", 378 | " ('PyTorch Models', 'pt'),\n", 379 | " ('PyTorch Legacy', 'pth'),\n", 380 | " ('ONNX Models', 'onnx'),\n", 381 | " ('TensorFlow Models', 'pb'),\n", 382 | " ('Keras Models', 'h5'),\n", 383 | "\n", 384 | " # Checkpoint Files 🎯\n", 385 | " ('Checkpoints', 'ckpt'),\n", 386 | " ('Binary Files', 'bin'),\n", 387 | "\n", 388 | " # Config & Data Files 📝\n", 389 | " ('JSON Files', 'json'),\n", 390 | " ('YAML Files', 'yaml'),\n", 391 | " ('YAML Alt', 'yml'),\n", 392 | " ('Text Files', 'txt'),\n", 393 | " ('CSV Files', 'csv'),\n", 394 | " ('Pickle Files', 'pkl'),\n", 395 | "\n", 396 | " # Image Files 🎨\n", 397 | " ('PNG Images', 'png'),\n", 398 | " ('JPEG Images', 'jpg'),\n", 399 | " ('JPEG Alt', 'jpeg'),\n", 400 | " ('WebP Images', 'webp'),\n", 401 | " ('GIF Images', 'gif'),\n", 402 | "\n", 403 | " # Archive Files 📦\n", 404 | " ('ZIP Archives', 'zip'),\n", 405 | " ('TAR Files', 'tar'),\n", 406 | " ('GZ Archives', 'gz')\n", 407 | " ]\n", 408 | " self.current_directory = os.getcwd() # Set to the current working directory\n", 409 | " self.progress_bar = FloatProgress(value=0, min=0, max=100, description='Uploading:', bar_style='info')\n", 410 | " self._create_widgets()\n", 411 | " self._bind_events()\n", 412 | "\n", 413 | " def _create_widgets(self):\n", 414 | " # Repository info section\n", 415 | " self.repo_info = HTML(value=\"📚 Repository Details\")\n", 416 | " self.org_name = Text(\n", 417 | " placeholder='Organization or Username',\n", 418 | " description='Owner:',\n", 419 | " style={'description_width': 'initial'}\n", 420 | " )\n", 421 | " self.repo_name = Text(\n", 422 | " placeholder='Repository Name',\n", 423 | " description='Repo:',\n", 424 | " style={'description_width': 'initial'}\n", 425 | " )\n", 426 | " self.repo_type = Dropdown(\n", 427 | " options=['model', 'dataset', 'space'],\n", 428 | " value='model',\n", 429 | " description='Repo Type:',\n", 430 | " style={'description_width': 'initial'}\n", 431 | " )\n", 432 | " # Subfolder in Repo\n", 433 | " self.repo_folder = Text(\n", 434 | " placeholder='Optional Folder in Repo',\n", 435 | " description='Subfolder:',\n", 436 | " style={'description_width': 'initial'}\n", 437 | " )\n", 438 | " # File handling section\n", 439 | " self.file_section = HTML(value=\"🗂️ File Selection\")\n", 440 | " self.file_type = Dropdown(\n", 441 | " options=self.file_types,\n", 442 | " value='safetensors',\n", 443 | " description='File Type:',\n", 444 | " style={'description_width': 'initial'}\n", 445 | " )\n", 446 | " self.sort_by = Dropdown(\n", 447 | " options=['name', 'date'],\n", 448 | " value='name',\n", 449 | " description='Sort By:'\n", 450 | " )\n", 451 | " self.directory_label = Label(value=f\"Current Directory: {self.current_directory}\")\n", 452 | " self.directory_text = Text(\n", 453 | " value=self.current_directory,\n", 454 | " description=\"Path:\",\n", 455 | " style={'description_width': 'initial'},\n", 456 | " layout=Layout(width=\"400px\")\n", 457 | " )\n", 458 | " self.directory_update_btn = Button(\n", 459 | " description='🔄 Update Dir',\n", 460 | " button_style='info',\n", 461 | " tooltip='Refresh directory'\n", 462 | " )\n", 463 | "\n", 464 | " # Custom commit message\n", 465 | " self.commit_section = HTML(value=\"💭 Commit Details\")\n", 466 | " self.commit_msg = Textarea(\n", 467 | " value=\"Upload with Earth & Dusk Huggingface 🤗 Backup\",\n", 468 | " placeholder='Enter your commit message (optional)',\n", 469 | " description='Message:',\n", 470 | " layout=Layout(width='400px', height='60px')\n", 471 | " )\n", 472 | "\n", 473 | " # Upload options section\n", 474 | " self.upload_section = HTML(value=\"🚀 Upload Settings\")\n", 475 | " self.create_pr = Checkbox(\n", 476 | " value=False,\n", 477 | " description='Create Pull Request',\n", 478 | " indent=False,\n", 479 | " tooltip='When checked, creates a Pull Request instead of direct upload'\n", 480 | " )\n", 481 | " self.clear_after = Checkbox(\n", 482 | " value=True,\n", 483 | " description='Clear output after upload',\n", 484 | " indent=False,\n", 485 | " tooltip='Clears output area after successful upload'\n", 486 | " )\n", 487 | "\n", 488 | " # Action buttons with style\n", 489 | " self.update_btn = Button(\n", 490 | " description='🔄 Update Files',\n", 491 | " button_style='info',\n", 492 | " tooltip='Refresh file list'\n", 493 | " )\n", 494 | " self.upload_btn = Button(\n", 495 | " description='⬆️ Upload',\n", 496 | " button_style='success',\n", 497 | " tooltip='Start upload process',\n", 498 | " layout=Layout(width='auto', height='auto'),\n", 499 | " )\n", 500 | " self.clear_btn = Button(\n", 501 | " description='🧹 Clear Output',\n", 502 | " button_style='warning',\n", 503 | " tooltip='Clear output area'\n", 504 | " )\n", 505 | "\n", 506 | " # File selector and output\n", 507 | " self.ckpt_picker = SelectMultiple(\n", 508 | " options=[],\n", 509 | " layout=Layout(width=\"600px\", height=\"200px\")\n", 510 | " )\n", 511 | " self.out = Output(layout=Layout(padding='10px'))\n", 512 | "\n", 513 | " self.current_file_label = Label(value=\"Ready.\")\n", 514 | " self.progress_percent_label = Label(value=\"0%\")\n", 515 | "\n", 516 | " def _bind_events(self):\n", 517 | " self.update_btn.on_click(self._update_files)\n", 518 | " self.upload_btn.on_click(self._upload_ckpts)\n", 519 | " self.clear_btn.on_click(lambda _: self.out.clear_output())\n", 520 | " self.file_type.observe(self._update_files, names='value')\n", 521 | " self.directory_update_btn.on_click(self._update_directory)\n", 522 | "\n", 523 | " def _update_directory(self, _):\n", 524 | " new_dir = self.directory_text.value\n", 525 | " if os.path.isdir(new_dir):\n", 526 | " self.current_directory = new_dir\n", 527 | " self.directory_label.value = f\"Current Directory: {self.current_directory}\"\n", 528 | " self._update_files(None)\n", 529 | " else:\n", 530 | " with self.out:\n", 531 | " print(\"❌ Invalid Directory\")\n", 532 | "\n", 533 | " def _update_files(self, _):\n", 534 | " file_extension = self.file_type.value\n", 535 | " try:\n", 536 | " # Glob files based on file extension\n", 537 | " all_files = glob.glob(os.path.join(self.current_directory, f\"*.{file_extension}\"))\n", 538 | "\n", 539 | " # Filter out symlinks, old files and ignore patterns\n", 540 | " filtered_files = []\n", 541 | " for file_path in all_files:\n", 542 | " if os.path.islink(file_path):\n", 543 | " with self.out:\n", 544 | " print(f\"ℹ️ Skipping symlink: {file_path}\")\n", 545 | " continue\n", 546 | " if not os.path.isfile(file_path):\n", 547 | " with self.out:\n", 548 | " print(f\"ℹ️ Skipping non-file: {file_path}\")\n", 549 | " continue\n", 550 | "\n", 551 | " filtered_files.append(file_path)\n", 552 | "\n", 553 | " # Sort the files\n", 554 | " all_ckpts = sorted(\n", 555 | " filtered_files,\n", 556 | " key=os.path.getmtime if self.sort_by.value == 'date' else str\n", 557 | " )\n", 558 | "\n", 559 | " self.ckpt_picker.options = all_ckpts\n", 560 | "\n", 561 | " with self.out:\n", 562 | " print(f\"✨ Found {len(all_ckpts)} {file_extension} files in {self.current_directory}\")\n", 563 | "\n", 564 | " except Exception as e:\n", 565 | " with self.out:\n", 566 | " print(f\"❌ Error listing files: {str(e)}\")\n", 567 | "\n", 568 | " def _format_size(self, size):\n", 569 | " for unit in ['B', 'KB', 'MB', 'GB']:\n", 570 | " if size < 1024.0: # Use float for consistent division\n", 571 | " return f\"{size:.2f} {unit}\"\n", 572 | " size /= 1024.0\n", 573 | " return f\"{size:.2f} TB\"\n", 574 | "\n", 575 | " def _print_file_info(self, file_path, file_size, index, total):\n", 576 | " with self.out:\n", 577 | " print(f\"📦 File {index}/{total}: {file_path} ({self._format_size(file_size)})\")\n", 578 | "\n", 579 | " def _upload_ckpts(self, _):\n", 580 | " if not self.org_name.value or not self.repo_name.value:\n", 581 | " with self.out:\n", 582 | " print(\"❗ Please fill in both Organization/Username and Repository name\")\n", 583 | " return\n", 584 | "\n", 585 | " repo_id = f\"{self.org_name.value}/{self.repo_name.value}\"\n", 586 | " selected_files = self.ckpt_picker.value\n", 587 | " total_files = len(selected_files)\n", 588 | " repo_type = self.repo_type.value\n", 589 | " repo_folder = self.repo_folder.value.strip() # Get folder and strip whitespace\n", 590 | " current_directory = self.directory_text.value # Get the correct current directory\n", 591 | "\n", 592 | " with self.out:\n", 593 | " if not selected_files:\n", 594 | " print(\"📝 Nothing selected for upload. Please select files from the list.\")\n", 595 | " return\n", 596 | "\n", 597 | " print(f\"🎯 Starting upload to: huggingface.co/{repo_id}\")\n", 598 | "\n", 599 | " commit_msg = (self.commit_msg.value\n", 600 | " if self.commit_msg.value else\n", 601 | " \"Uploaded with Earth & Dusk Huggingface 🤗 Backup\")\n", 602 | "\n", 603 | " # Update current file label and progress (make them visible)\n", 604 | " file_label_container = VBox([\n", 605 | " HTML(value=\"Current Uploading File:\"),\n", 606 | " self.current_file_label\n", 607 | " ], layout=Layout(margin='10px 0 0 0'))\n", 608 | " display(file_label_container) # Display it\n", 609 | " progress_container = HBox([\n", 610 | " HTML(value=\"Progress:\"),\n", 611 | " self.progress_percent_label,\n", 612 | " self.progress_bar # Add progress bar here\n", 613 | " ], layout=Layout(align_items='center'))\n", 614 | " display(progress_container) # Display Progress\n", 615 | "\n", 616 | " for idx, ckpt in enumerate(selected_files, 1):\n", 617 | " try:\n", 618 | " self.current_file_label.value = f\"{ckpt}\" # Update the current file label\n", 619 | " file_size = os.path.getsize(ckpt)\n", 620 | " self._print_file_info(ckpt, file_size, idx, total_files)\n", 621 | " start_time = time.time()\n", 622 | "\n", 623 | " # Use os.path to ensure forward slashes for paths\n", 624 | " path_in_repo = os.path.basename(ckpt)\n", 625 | "\n", 626 | " # Handle folders by creating folder in the repo\n", 627 | " # Split the path\n", 628 | " path_parts = Path(ckpt).parts\n", 629 | " if len(path_parts) > 1:\n", 630 | " # Get only the folder names\n", 631 | " folder_path_parts = path_parts[len(Path(current_directory).parts):-1]\n", 632 | " # Generate the folder path in the repo\n", 633 | " if folder_path_parts:\n", 634 | " path_in_repo = os.path.join(*folder_path_parts, os.path.basename(ckpt))\n", 635 | "\n", 636 | " # Add the subfolder to the path\n", 637 | " if repo_folder:\n", 638 | " path_in_repo = os.path.join(repo_folder, path_in_repo)\n", 639 | "\n", 640 | " response = self.api.upload_file(\n", 641 | " path_or_fileobj=ckpt,\n", 642 | " path_in_repo=path_in_repo, # Use the path in repo\n", 643 | " repo_id=repo_id,\n", 644 | " repo_type=repo_type, # Use the repo type dropdown value here\n", 645 | " create_pr=self.create_pr.value,\n", 646 | " commit_message=commit_msg\n", 647 | " )\n", 648 | " duration = time.time() - start_time\n", 649 | " with self.out:\n", 650 | " print(f\"✅ Upload completed in {duration:.1f} seconds\")\n", 651 | " display(response)\n", 652 | "\n", 653 | " # Calculate and update progress percentage\n", 654 | " percentage = int((idx / total_files) * 100)\n", 655 | " self.progress_percent_label.value = f\"{percentage}%\"\n", 656 | " self.progress_bar.value = percentage # Update progress bar\n", 657 | "\n", 658 | " except Exception as e:\n", 659 | " with self.out:\n", 660 | " print(f\"❌ Error uploading {ckpt}: {type(e).__name__} - {str(e)}\") #Detailed error\n", 661 | " continue\n", 662 | "\n", 663 | " with self.out:\n", 664 | " print(\"\\n✨ All uploads completed! ✨\")\n", 665 | " if self.create_pr.value:\n", 666 | " print(\"🎉 Check your repository for the new Pull Request!\")\n", 667 | " else:\n", 668 | " print(\"🎉 Files have been uploaded directly to your repository!\")\n", 669 | "\n", 670 | " if self.clear_after.value:\n", 671 | " time.sleep(3)\n", 672 | " self.out.clear_output()\n", 673 | " # Hide the current file label and progress after upload:\n", 674 | " file_label_container.layout.visibility = 'hidden'\n", 675 | " progress_container.layout.visibility = 'hidden'\n", 676 | "\n", 677 | " def display(self):\n", 678 | " box = VBox([\n", 679 | " self.repo_info,\n", 680 | " HBox([self.org_name, self.repo_name, self.repo_type]),\n", 681 | " HBox([self.repo_folder]),\n", 682 | " self.file_section,\n", 683 | " HBox([self.file_type, self.sort_by]),\n", 684 | " HBox([self.directory_label, self.directory_text, self.directory_update_btn]),\n", 685 | " self.commit_section,\n", 686 | " self.commit_msg,\n", 687 | " self.upload_section,\n", 688 | " HBox([self.create_pr, self.clear_after]),\n", 689 | " self.update_btn,\n", 690 | " self.ckpt_picker,\n", 691 | " HBox([self.upload_btn, self.clear_btn], layout=Layout(align_items='center')),\n", 692 | " self.out, #Removed self.progress_text\n", 693 | " ])\n", 694 | " display(box)\n", 695 | "\n", 696 | "# Create and display the uploader - just one line to rule them all! ✨\n", 697 | "uploader = HuggingFaceUploader()\n", 698 | "uploader.display(); # Semicolon suppresses the output" 699 | ] 700 | }, 701 | { 702 | "cell_type": "code", 703 | "execution_count": null, 704 | "metadata": {}, 705 | "outputs": [], 706 | "source": [] 707 | } 708 | ], 709 | "metadata": { 710 | "colab": { 711 | "collapsed_sections": [ 712 | "IZ_JYwvBLrg-", 713 | "PNF2kdyeO3Dn" 714 | ], 715 | "private_outputs": true, 716 | "provenance": [] 717 | }, 718 | "kernelspec": { 719 | "display_name": "Python 3 (ipykernel)", 720 | "language": "python", 721 | "name": "python3" 722 | }, 723 | "language_info": { 724 | "codemirror_mode": { 725 | "name": "ipython", 726 | "version": 3 727 | }, 728 | "file_extension": ".py", 729 | "mimetype": "text/x-python", 730 | "name": "python", 731 | "nbconvert_exporter": "python", 732 | "pygments_lexer": "ipython3", 733 | "version": "3.10.12" 734 | } 735 | }, 736 | "nbformat": 4, 737 | "nbformat_minor": 4 738 | } 739 | -------------------------------------------------------------------------------- /Notebooks/Previous Notebooks/HuggingFace_Backup_Jupyter_2025_Jan_Update.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 🚀 Welcome to the Hugging Face Uploader Notebook!\n", 8 | "\n", 9 | "This notebook provides a user-friendly tool for uploading files directly to your Hugging Face repositories. Here's how to get started:\n", 10 | "\n", 11 | "**🔑 Initial Setup (One Time Only)**\n", 12 | "\n", 13 | "1. **Authenticate:**\n", 14 | " * Run the `notebook_login()` cell *once* to securely store your Hugging Face API token.\n", 15 | " * **Important:** For security, avoid sharing your notebook file or system state after you have run `notebook_login()`. Do not commit your notebook file to a shared repo, as this could expose your API token.\n", 16 | "\n", 17 | "**🗂️ Using the Uploader**\n", 18 | "\n", 19 | "1. **Repository Details:**\n", 20 | " * Enter your Hugging Face Organization or Username in the \"Owner\" field.\n", 21 | " * Enter your repository name in the \"Repo\" field.\n", 22 | "\n", 23 | "2. **Directory Selection:**\n", 24 | " * Enter a directory path where your files are located.\n", 25 | " * Select the 'Update Dir' button to set that path.\n", 26 | "\n", 27 | "3. **File Selection:**\n", 28 | " * Select the appropriate file type from the dropdown menu.\n", 29 | " * Select the files you want to upload from the list. You can sort them by name or date modified.\n", 30 | "\n", 31 | "4. **Commit Message (Optional):**\n", 32 | " * Add a commit message to your upload, or use the default message.\n", 33 | "\n", 34 | "5. **Upload Options:**\n", 35 | " * Choose whether to create a pull request or upload directly to the main branch.\n", 36 | " * Select whether to clear the output after a successful upload.\n", 37 | "\n", 38 | "6. **Start Upload:**\n", 39 | " * Click the \"⬆️ Upload\" button.\n", 40 | "\n", 41 | "**💡 Important Notes**\n", 42 | "\n", 43 | "* **Direct Uploads:** This uploader uses the Hugging Face API for direct file uploads, bypassing the need for traditional Git operations for core functionality.\n", 44 | "* **Git LFS:** Most users will not need to interact with Git or Git LFS. If you need to clone or push changes to a repository *outside this notebook*, you will require a separate Git credential setup (e.g., your operating system's credential manager or SSH keys). This is separate from your Hugging Face API token, and you should not share any git credentials in your notebook.\n", 45 | "* **Troubleshooting:** If you encounter any issues, please review the steps, or double-check that you have write access to the repository, and that your API token has the correct scope of access.\n", 46 | "\n", 47 | "**📣 Updates & Community**\n", 48 | "\n", 49 | "* This tool will continue to be updated.\n", 50 | "* For the latest patches, fixes, and community contributions, visit [https://github.com/duskfallcrew/HuggingFace_Backup](https://github.com/duskfallcrew/HuggingFace_Backup)\n", 51 | "\n", 52 | "We hope this notebook makes your Hugging Face uploads easier! If you have any questions or suggestions, please reach out." 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 1, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "!pip install -q huggingface_hub ipywidgets " 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "id": "Xs1mb1VKLuUW" 68 | }, 69 | "source": [ 70 | "# ✨ Connecting to Hugging Face: Authentication Required\n", 71 | "\n", 72 | "To upload files to Hugging Face using this notebook, you'll need to authenticate using your Hugging Face API token. This token acts as your secure access key to the Hugging Face Hub.\n", 73 | "\n", 74 | "**🔑 Obtaining Your API Token**\n", 75 | "\n", 76 | "1. **Go to Your Settings:** Navigate to your [Hugging Face account settings](https://huggingface.co/settings/tokens).\n", 77 | "2. **Access Tokens Section:** Find the \"Access Tokens\" section.\n", 78 | "3. **Create or Copy Token:**\n", 79 | " * If you don't have one already, create a new access token.\n", 80 | " * Copy your existing access token.\n", 81 | "\n", 82 | "**🔒 Logging In**\n", 83 | "\n", 84 | "1. **Run the Cell:** Execute the cell below this section.\n", 85 | "2. **Paste Token:** When prompted, paste your Hugging Face API token into the input field.\n", 86 | "3. **Confirm:** Press `Enter` to complete the login process. No characters will be displayed as you type your token.\n", 87 | " * **Security Note:** This ensures your token is kept private while pasting.\n", 88 | "\n", 89 | "**Important Notes**\n", 90 | "\n", 91 | "* **Purpose:** This login step authorizes your notebook environment to interact with the Hugging Face Hub. This is required for uploading your models and data.\n", 92 | "* **Session-Based:** You will need to perform this login procedure each time you restart the notebook or start a new session (the token is stored locally on your computer, but not in the notebook, and it doesn't persist between sessions).\n", 93 | "* **Security:** After you have logged in using `notebook_login()`, **do not** share your notebook or your system state with untrusted users, and do not commit your notebook to a shared repository, as this could compromise your API token.\n", 94 | "\n", 95 | "After completing this step, you'll be ready to use the upload functionality of this notebook." 96 | ] 97 | }, 98 | { 99 | "cell_type": "code", 100 | "execution_count": 2, 101 | "metadata": {}, 102 | "outputs": [ 103 | { 104 | "data": { 105 | "application/vnd.jupyter.widget-view+json": { 106 | "model_id": "27172be37dc847f899ec294cfbacb902", 107 | "version_major": 2, 108 | "version_minor": 0 109 | }, 110 | "text/plain": [ 111 | "VBox(children=(HTML(value='
📚 Repository Details'), HBox(children=(Text(value='', description='Owner:', …" 183 | ] 184 | }, 185 | "metadata": {}, 186 | "output_type": "display_data" 187 | } 188 | ], 189 | "source": [ 190 | "import glob\n", 191 | "import os\n", 192 | "from pathlib import Path\n", 193 | "from huggingface_hub import HfApi\n", 194 | "from ipywidgets import Text, Dropdown, Button, SelectMultiple, VBox, HBox, Output, Layout, Checkbox, HTML, Textarea, Label\n", 195 | "from IPython.display import display, clear_output\n", 196 | "import time\n", 197 | "#from tqdm import tqdm No longer needed\n", 198 | "\n", 199 | "class HuggingFaceUploader:\n", 200 | " def __init__(self):\n", 201 | " self.api = HfApi()\n", 202 | " self.file_types = [\n", 203 | " # AI Model Files 🤖\n", 204 | " ('SafeTensors', 'safetensors'),\n", 205 | " ('PyTorch Models', 'pt'),\n", 206 | " ('PyTorch Legacy', 'pth'),\n", 207 | " ('ONNX Models', 'onnx'),\n", 208 | " ('TensorFlow Models', 'pb'),\n", 209 | " ('Keras Models', 'h5'),\n", 210 | "\n", 211 | " # Checkpoint Files 🎯\n", 212 | " ('Checkpoints', 'ckpt'),\n", 213 | " ('Binary Files', 'bin'),\n", 214 | "\n", 215 | " # Config & Data Files 📝\n", 216 | " ('JSON Files', 'json'),\n", 217 | " ('YAML Files', 'yaml'),\n", 218 | " ('YAML Alt', 'yml'),\n", 219 | " ('Text Files', 'txt'),\n", 220 | " ('CSV Files', 'csv'),\n", 221 | " ('Pickle Files', 'pkl'),\n", 222 | "\n", 223 | " # Image Files 🎨\n", 224 | " ('PNG Images', 'png'),\n", 225 | " ('JPEG Images', 'jpg'),\n", 226 | " ('JPEG Alt', 'jpeg'),\n", 227 | " ('WebP Images', 'webp'),\n", 228 | " ('GIF Images', 'gif'),\n", 229 | "\n", 230 | " # Archive Files 📦\n", 231 | " ('ZIP Archives', 'zip'),\n", 232 | " ('TAR Files', 'tar'),\n", 233 | " ('GZ Archives', 'gz')\n", 234 | " ]\n", 235 | " self.current_directory = os.getcwd() # Set to the current working directory\n", 236 | " self._create_widgets()\n", 237 | " self._bind_events()\n", 238 | "\n", 239 | "\n", 240 | " def _create_widgets(self):\n", 241 | " # Repository info section\n", 242 | " self.repo_info = HTML(value=\"📚 Repository Details\")\n", 243 | " self.org_name = Text(\n", 244 | " placeholder='Organization or Username',\n", 245 | " description='Owner:',\n", 246 | " style={'description_width': 'initial'}\n", 247 | " )\n", 248 | " self.repo_name = Text(\n", 249 | " placeholder='Repository Name',\n", 250 | " description='Repo:',\n", 251 | " style={'description_width': 'initial'}\n", 252 | " )\n", 253 | " self.repo_type = Dropdown(\n", 254 | " options=['model', 'dataset', 'space'],\n", 255 | " value='model',\n", 256 | " description='Repo Type:',\n", 257 | " style={'description_width': 'initial'}\n", 258 | " )\n", 259 | " # Subfolder in Repo\n", 260 | " self.repo_folder = Text(\n", 261 | " placeholder='Optional Folder in Repo',\n", 262 | " description='Subfolder:',\n", 263 | " style={'description_width': 'initial'}\n", 264 | " )\n", 265 | " # File handling section\n", 266 | " self.file_section = HTML(value=\"🗂️ File Selection\")\n", 267 | " self.file_type = Dropdown(\n", 268 | " options=self.file_types,\n", 269 | " value='safetensors',\n", 270 | " description='File Type:',\n", 271 | " style={'description_width': 'initial'}\n", 272 | " )\n", 273 | " self.sort_by = Dropdown(\n", 274 | " options=['name', 'date'],\n", 275 | " value='name',\n", 276 | " description='Sort By:'\n", 277 | " )\n", 278 | " self.directory_label = Label(value=f\"Current Directory: {self.current_directory}\")\n", 279 | " self.directory_text = Text(\n", 280 | " value=self.current_directory,\n", 281 | " description=\"Path:\",\n", 282 | " style={'description_width': 'initial'},\n", 283 | " layout=Layout(width=\"400px\")\n", 284 | " )\n", 285 | " self.directory_update_btn = Button(\n", 286 | " description='🔄 Update Dir',\n", 287 | " button_style='info',\n", 288 | " tooltip='Refresh directory'\n", 289 | " )\n", 290 | "\n", 291 | "\n", 292 | " # Custom commit message\n", 293 | " self.commit_section = HTML(value=\"💭 Commit Details\")\n", 294 | " self.commit_msg = Textarea(\n", 295 | " value=\"Upload with Earth & Dusk Huggingface 🤗 Backup\",\n", 296 | " placeholder='Enter your commit message (optional)',\n", 297 | " description='Message:',\n", 298 | " layout=Layout(width='400px', height='60px')\n", 299 | " )\n", 300 | "\n", 301 | " # Upload options section\n", 302 | " self.upload_section = HTML(value=\"🚀 Upload Settings\")\n", 303 | " self.create_pr = Checkbox(\n", 304 | " value=False,\n", 305 | " description='Create Pull Request',\n", 306 | " indent=False,\n", 307 | " tooltip='When checked, creates a Pull Request instead of direct upload'\n", 308 | " )\n", 309 | " self.clear_after = Checkbox(\n", 310 | " value=True,\n", 311 | " description='Clear output after upload',\n", 312 | " indent=False,\n", 313 | " tooltip='Clears output area after successful upload'\n", 314 | " )\n", 315 | "\n", 316 | " # Progress tracking\n", 317 | " self.progress_text = HTML(value=\"\")\n", 318 | "\n", 319 | " # Action buttons with style\n", 320 | " self.update_btn = Button(\n", 321 | " description='🔄 Update Files',\n", 322 | " button_style='info',\n", 323 | " tooltip='Refresh file list'\n", 324 | " )\n", 325 | " self.upload_btn = Button(\n", 326 | " description='⬆️ Upload',\n", 327 | " button_style='success',\n", 328 | " tooltip='Start upload process',\n", 329 | " layout=Layout(width='auto', height='auto'),\n", 330 | " )\n", 331 | " self.clear_btn = Button(\n", 332 | " description='🧹 Clear Output',\n", 333 | " button_style='warning',\n", 334 | " tooltip='Clear output area'\n", 335 | " )\n", 336 | "\n", 337 | " # File selector and output\n", 338 | " self.ckpt_picker = SelectMultiple(\n", 339 | " options=[],\n", 340 | " layout=Layout(width=\"600px\", height=\"200px\")\n", 341 | " )\n", 342 | " self.out = Output(layout=Layout(padding='10px'))\n", 343 | " \n", 344 | " self.current_file_label = Label(value=\"Ready.\")\n", 345 | " self.progress_percent_label = Label(value=\"0%\")\n", 346 | "\n", 347 | " def _bind_events(self):\n", 348 | " self.update_btn.on_click(self._update_files)\n", 349 | " self.upload_btn.on_click(self._upload_ckpts)\n", 350 | " self.clear_btn.on_click(lambda _: self.out.clear_output())\n", 351 | " self.file_type.observe(self._update_files, names='value')\n", 352 | " self.directory_update_btn.on_click(self._update_directory)\n", 353 | "\n", 354 | " def _update_directory(self,_):\n", 355 | " new_dir = self.directory_text.value\n", 356 | " if os.path.isdir(new_dir):\n", 357 | " self.current_directory = new_dir\n", 358 | " self.directory_label.value = f\"Current Directory: {self.current_directory}\"\n", 359 | " self._update_files(None)\n", 360 | " else:\n", 361 | " with self.out:\n", 362 | " print(\"❌ Invalid Directory\")\n", 363 | "\n", 364 | " def _update_files(self, _):\n", 365 | " file_extension = self.file_type.value\n", 366 | " try:\n", 367 | " # Glob files based on file extension\n", 368 | " all_files = glob.glob(os.path.join(self.current_directory, f\"*.{file_extension}\"))\n", 369 | "\n", 370 | "\n", 371 | " # Filter out symlinks, old files and ignore patterns\n", 372 | " filtered_files = []\n", 373 | " for file_path in all_files:\n", 374 | " if os.path.islink(file_path):\n", 375 | " continue\n", 376 | " if not os.path.isfile(file_path):\n", 377 | " continue\n", 378 | "\n", 379 | " filtered_files.append(file_path)\n", 380 | " \n", 381 | " # Sort the files\n", 382 | " all_ckpts = sorted(\n", 383 | " filtered_files,\n", 384 | " key=os.path.getmtime if self.sort_by.value == 'date' else str\n", 385 | " )\n", 386 | "\n", 387 | " self.ckpt_picker.options = all_ckpts\n", 388 | "\n", 389 | " with self.out:\n", 390 | " print(f\"✨ Found {len(all_ckpts)} {file_extension} files in {self.current_directory}\")\n", 391 | "\n", 392 | " except Exception as e:\n", 393 | " with self.out:\n", 394 | " print(f\"❌ Error listing files: {str(e)}\")\n", 395 | "\n", 396 | "\n", 397 | " def _format_size(self, size):\n", 398 | " for unit in ['B', 'KB', 'MB', 'GB']:\n", 399 | " if size < 1024:\n", 400 | " return f\"{size:.2f} {unit}\"\n", 401 | " size /= 1024\n", 402 | " return f\"{size:.2f} TB\"\n", 403 | "\n", 404 | " def _print_file_info(self, file_path, file_size, index, total):\n", 405 | " with self.out:\n", 406 | " print(f\"📦 File {index}/{total}: {file_path} ({self._format_size(file_size)})\")\n", 407 | "\n", 408 | " def _upload_ckpts(self, _):\n", 409 | " if not self.org_name.value or not self.repo_name.value:\n", 410 | " with self.out:\n", 411 | " print(\"❗ Please fill in both Organization/Username and Repository name\")\n", 412 | " return\n", 413 | "\n", 414 | " repo_id = f\"{self.org_name.value}/{self.repo_name.value}\"\n", 415 | " selected_files = self.ckpt_picker.value\n", 416 | " total_files = len(selected_files)\n", 417 | " repo_type = self.repo_type.value\n", 418 | " repo_folder = self.repo_folder.value.strip() # Get folder and strip whitespace\n", 419 | " current_directory = self.directory_text.value #Get the correct current directory from the path text box\n", 420 | "\n", 421 | " with self.out:\n", 422 | " if not selected_files:\n", 423 | " print(\"📝 Nothing selected for upload. Please select files from the list.\")\n", 424 | " return\n", 425 | "\n", 426 | " print(f\"🎯 Starting upload to: huggingface.co/{repo_id}\")\n", 427 | "\n", 428 | " commit_msg = self.commit_msg.value if self.commit_msg.value else \"Uploaded with Earth & Dusk Huggingface 🤗 Backup\"\n", 429 | "\n", 430 | " # Update current file label and progress\n", 431 | " file_label_container = VBox([HTML(value=\"Current Uploading File:\"), self.current_file_label],\n", 432 | " layout=Layout(margin='10px 0 0 0'))\n", 433 | " display(file_label_container) #Display it\n", 434 | " progress_container = HBox([HTML(value=\"Progress:\"), self.progress_percent_label], layout=Layout(align_items='center'))\n", 435 | " display(progress_container) #Display Progress\n", 436 | " \n", 437 | " for idx, ckpt in enumerate(selected_files, 1):\n", 438 | " try:\n", 439 | " self.current_file_label.value = f\"{ckpt}\" # Update the current file label\n", 440 | " file_size = os.path.getsize(ckpt)\n", 441 | " self._print_file_info(ckpt, file_size, idx, total_files)\n", 442 | " start_time = time.time()\n", 443 | " \n", 444 | " # Use os.path to ensure forward slashes for paths\n", 445 | " path_in_repo = os.path.basename(ckpt)\n", 446 | "\n", 447 | " # Handle folders by creating folder in the repo\n", 448 | " \n", 449 | " # Split the path\n", 450 | " path_parts = Path(ckpt).parts\n", 451 | " if len(path_parts) > 1:\n", 452 | " # Get only the folder names\n", 453 | " folder_path_parts = path_parts[len(Path(current_directory).parts):-1]\n", 454 | " # Generate the folder path in the repo\n", 455 | " if folder_path_parts:\n", 456 | " path_in_repo = os.path.join(*folder_path_parts, os.path.basename(ckpt))\n", 457 | " \n", 458 | " #Add the subfolder to the path\n", 459 | " if repo_folder:\n", 460 | " path_in_repo = os.path.join(repo_folder, path_in_repo)\n", 461 | " \n", 462 | " response = self.api.upload_file(\n", 463 | " path_or_fileobj=ckpt,\n", 464 | " path_in_repo=path_in_repo, #Use the path in repo\n", 465 | " repo_id=repo_id,\n", 466 | " repo_type=repo_type, # Use the repo type dropdown value here\n", 467 | " create_pr=self.create_pr.value,\n", 468 | " commit_message=commit_msg\n", 469 | " )\n", 470 | " duration = time.time() - start_time\n", 471 | " with self.out:\n", 472 | " print(f\"✅ Upload completed in {duration:.1f} seconds\")\n", 473 | " display(response)\n", 474 | " \n", 475 | " # Calculate and update progress percentage\n", 476 | " percentage = int((idx / total_files) * 100)\n", 477 | " self.progress_percent_label.value = f\"{percentage}%\"\n", 478 | "\n", 479 | " except Exception as e:\n", 480 | " with self.out:\n", 481 | " print(f\"❌ Error uploading {ckpt}: {str(e)}\")\n", 482 | " continue\n", 483 | "\n", 484 | "\n", 485 | " with self.out:\n", 486 | " print(\"\\n✨ All uploads completed! ✨\")\n", 487 | " if self.create_pr.value:\n", 488 | " print(\"🎉 Check your repository for the new Pull Request!\")\n", 489 | " else:\n", 490 | " print(\"🎉 Files have been uploaded directly to your repository!\")\n", 491 | "\n", 492 | " if self.clear_after.value:\n", 493 | " time.sleep(3)\n", 494 | " self.out.clear_output()\n", 495 | " #Hide the current file label after upload:\n", 496 | " file_label_container.layout.visibility = 'hidden'\n", 497 | " progress_container.layout.visibility = 'hidden'\n", 498 | "\n", 499 | " def display(self):\n", 500 | " box = VBox([\n", 501 | " self.repo_info,\n", 502 | " HBox([self.org_name, self.repo_name, self.repo_type]),\n", 503 | " HBox([self.repo_folder]),\n", 504 | " self.file_section,\n", 505 | " HBox([self.file_type, self.sort_by]),\n", 506 | " HBox([self.directory_label, self.directory_text, self.directory_update_btn]),\n", 507 | " self.commit_section,\n", 508 | " self.commit_msg,\n", 509 | " self.upload_section,\n", 510 | " HBox([self.create_pr, self.clear_after]),\n", 511 | " self.update_btn,\n", 512 | " self.ckpt_picker,\n", 513 | " HBox([self.upload_btn, self.clear_btn], layout=Layout(align_items='center')),\n", 514 | " self.progress_text,\n", 515 | " self.out,\n", 516 | " ])\n", 517 | " display(box)\n", 518 | "\n", 519 | "# Create and display the uploader - just one line to rule them all! ✨\n", 520 | "uploader = HuggingFaceUploader()\n", 521 | "uploader.display()" 522 | ] 523 | }, 524 | { 525 | "cell_type": "code", 526 | "execution_count": null, 527 | "metadata": {}, 528 | "outputs": [], 529 | "source": [] 530 | } 531 | ], 532 | "metadata": { 533 | "colab": { 534 | "collapsed_sections": [ 535 | "IZ_JYwvBLrg-", 536 | "PNF2kdyeO3Dn" 537 | ], 538 | "private_outputs": true, 539 | "provenance": [] 540 | }, 541 | "kernelspec": { 542 | "display_name": "Python 3 (ipykernel)", 543 | "language": "python", 544 | "name": "python3" 545 | }, 546 | "language_info": { 547 | "codemirror_mode": { 548 | "name": "ipython", 549 | "version": 3 550 | }, 551 | "file_extension": ".py", 552 | "mimetype": "text/x-python", 553 | "name": "python", 554 | "nbconvert_exporter": "python", 555 | "pygments_lexer": "ipython3", 556 | "version": "3.10.12" 557 | } 558 | }, 559 | "nbformat": 4, 560 | "nbformat_minor": 4 561 | } 562 | -------------------------------------------------------------------------------- /Notebooks/Previous Notebooks/HuggingFace_Backup_Jupyter_2025_March_Update.ipynb: -------------------------------------------------------------------------------- 1 | { 2 | "cells": [ 3 | { 4 | "cell_type": "markdown", 5 | "metadata": {}, 6 | "source": [ 7 | "# 🚀 Welcome to the Hugging Face File Uploader!\n", 8 | "\n", 9 | "This Jupyter Notebook provides an easy-to-use interface for uploading files directly to your Hugging Face repositories.\n", 10 | "\n", 11 | "**Getting Started (One-Time Setup):**\n", 12 | "\n", 13 | "1. **Install Dependencies:**\n", 14 | " * Run the first code cell in this notebook to install the required libraries (`huggingface_hub`, `ipywidgets`, etc.). You only need to do this *once* per environment.\n", 15 | "2. **Authenticate with Hugging Face:**\n", 16 | " * Run the `notebook_login()` cell (usually the second cell). This will prompt you for your Hugging Face API token.\n", 17 | " * Get your API token from your [Hugging Face settings](https://huggingface.co/settings/tokens) (make sure it has **write** access).\n", 18 | " * **🚨 Security:** After logging in, *never* share your notebook file (`.ipynb`) or commit it to a repository. The `notebook_login()` function handles your token securely, but sharing the notebook could still expose sensitive information.\n", 19 | "\n", 20 | "**Using the Uploader:**\n", 21 | "\n", 22 | "See the detailed instructions in the \"🚀 Using the Hugging Face File Uploader\" section below. The key steps are:\n", 23 | "\n", 24 | "1. **Enter Repository Details:** Provide your username/organization, repository name, and type.\n", 25 | "2. **Select a Directory:** Specify the folder containing your files.\n", 26 | "3. **Choose Files:** Select the files you want to upload.\n", 27 | "4. **Set Upload Options:** Choose whether to create a pull request and clear output.\n", 28 | "5. **Click \"⬆️ Upload\":** Start the upload process.\n", 29 | "\n", 30 | "**Key Features & Notes:**\n", 31 | "\n", 32 | "* **Direct Uploads:** This uploader uses the Hugging Face API for direct file uploads. You don't need to use `git` commands directly for basic uploads.\n", 33 | "* **Interactive Interface:** The uploader provides a user-friendly interface within the Jupyter Notebook.\n", 34 | "* **Progress Tracking:** You'll see real-time progress updates during the upload.\n", 35 | "* **Error Handling:** Detailed error messages will help you troubleshoot any problems.\n", 36 | "* **Troubleshooting** If you encounter issues, double-check your credentials, repo name, write permissions, etc.\n", 37 | "\n", 38 | "**Community & Support:**\n", 39 | "\n", 40 | "* **GitHub:** [Ktiseos Nyx @ Github](https://github.com/Ktiseos-Nyx/HuggingFace_Backup) (for updates, bug reports, and contributions)\n", 41 | "* **Discord:**\n", 42 | " * [Ktiseos Nyx AI/ML Discord](https://discord.gg/HhBSvM9gBY)\n", 43 | " * [Earth & Dusk Media](https://discord.gg/5t2kYxt7An)\n", 44 | "\n", 45 | "This uploader is designed to simplify the process of uploading files to Hugging Face. We hope you find it useful!" 46 | ] 47 | }, 48 | { 49 | "cell_type": "markdown", 50 | "metadata": {}, 51 | "source": [ 52 | "# Install Dependencies" 53 | ] 54 | }, 55 | { 56 | "cell_type": "code", 57 | "execution_count": 1, 58 | "metadata": {}, 59 | "outputs": [], 60 | "source": [ 61 | "!pip install -q huggingface_hub ipywidgets " 62 | ] 63 | }, 64 | { 65 | "cell_type": "markdown", 66 | "metadata": { 67 | "id": "Xs1mb1VKLuUW" 68 | }, 69 | "source": [ 70 | "# ✨ Connecting to Hugging Face: Authentication\n", 71 | "\n", 72 | "To upload files to Hugging Face, you need to authenticate with your Hugging Face API token. This token grants your notebook secure access to the Hugging Face Hub.\n", 73 | "\n", 74 | "**🔑 Get Your API Token**\n", 75 | "\n", 76 | "1. **Go to your Hugging Face settings:** [Hugging Face Access Tokens](https://huggingface.co/settings/tokens).\n", 77 | "2. **Find \"Access Tokens\":** Locate the section labeled \"Access Tokens\".\n", 78 | "3. **Create or Copy:**\n", 79 | " * If you don't have a token, create a new one with **write** access. *Crucially, you need write access to upload.*\n", 80 | " * If you have an existing token, copy it.\n", 81 | "\n", 82 | "**🔒 Login to Hugging Face in this Notebook**\n", 83 | "\n", 84 | "1. **Run the next cell:** Execute the code cell immediately below this one.\n", 85 | "2. **Paste your token:** A prompt will appear. Paste your copied API token into the input field. *You won't see any characters as you type or paste – this is for security.*\n", 86 | "3. **Press Enter:** This completes the login process.\n", 87 | "\n", 88 | "**🚨 Important Security Notes 🚨**\n", 89 | "\n", 90 | "* **Why you need to log in:** This login authorizes *this notebook environment* to interact with the Hugging Face Hub on your behalf. It's required for uploading.\n", 91 | "* **Session-specific:** The login is only valid for the *current notebook session*. If you restart the kernel or open the notebook later, you'll need to log in again. The token is *not* stored within the notebook itself, enhancing security.\n", 92 | "* **`notebook_login()` is secure:** The `notebook_login()` function from the `huggingface_hub` library handles the token securely. It does *not* store the token in plain text within the notebook.\n", 93 | "* **Do NOT share your notebook:** After logging in, **never** share your notebook file (`.ipynb`) with others or commit it to a public (or even private, but shared) repository. While the token isn't *directly* visible in the notebook file, the notebook's internal state might contain information that could be used to compromise your account. Treat the logged-in notebook as a sensitive document.\n", 94 | "* **Do not store the token in your code.**\n", 95 | "\n", 96 | "**Troubleshooting**\n", 97 | "\n", 98 | "* **Authentication Errors:** If you encounter authentication issues after entering your API token:\n", 99 | " * **Local Development:** If you are running Jupyter Notebook on your *own computer*, open your system terminal (Terminal on macOS, Command Prompt/PowerShell on Windows).\n", 100 | " * **Rented GPU / Cloud Environment:** If you are using a *rented GPU service* (Paperspace, RunPod, Google Colab, etc.), open a terminal *within your Jupyter environment*. This is usually done through a \"New\" -> \"Terminal\" option in the Jupyter file browser.\n", 101 | " * **Run this command (in the appropriate terminal):**\n", 102 | " ```bash\n", 103 | " git config --global credential.helper store\n", 104 | " ```\n", 105 | " This configures Git to store credentials, which can sometimes resolve conflicts. It does *not* store your Hugging Face token in plain text. It's a general Git configuration setting. If you are still concerned with security, you may remove this credential after.\n", 106 | "\n", 107 | "* **Still having trouble?** Consider refreshing your Hugging Face token (creating a new one and deleting the old one) on the [Hugging Face website](https://huggingface.co/settings/tokens), and then try logging in again with the new token. This is a good security practice if you suspect your token might have been compromised." 108 | ] 109 | }, 110 | { 111 | "cell_type": "code", 112 | "execution_count": 2, 113 | "metadata": {}, 114 | "outputs": [ 115 | { 116 | "data": { 117 | "application/vnd.jupyter.widget-view+json": { 118 | "model_id": "27172be37dc847f899ec294cfbacb902", 119 | "version_major": 2, 120 | "version_minor": 0 121 | }, 122 | "text/plain": [ 123 | "VBox(children=(HTML(value='
📚 Repository Details'), HBox(children=(Text(value='', description='Owner:', …" 209 | ] 210 | }, 211 | "metadata": {}, 212 | "output_type": "display_data" 213 | } 214 | ], 215 | "source": [ 216 | "import glob\n", 217 | "import os\n", 218 | "import time # Standard library imports\n", 219 | "from pathlib import Path\n", 220 | "\n", 221 | "from huggingface_hub import HfApi # Third-party imports\n", 222 | "from ipywidgets import (Text, Dropdown, Button, SelectMultiple, VBox, HBox,\n", 223 | " Output, Layout, Checkbox, HTML, Textarea, Label,\n", 224 | " FloatProgress)\n", 225 | "from IPython.display import display, clear_output\n", 226 | "\n", 227 | "# from tqdm import tqdm No longer needed\n", 228 | "\n", 229 | "class HuggingFaceUploader:\n", 230 | "\n", 231 | " def __init__(self):\n", 232 | " self.api = HfApi()\n", 233 | " self.file_types = [\n", 234 | " # AI Model Files 🤖\n", 235 | " ('SafeTensors', 'safetensors'),\n", 236 | " ('PyTorch Models', 'pt'),\n", 237 | " ('PyTorch Legacy', 'pth'),\n", 238 | " ('ONNX Models', 'onnx'),\n", 239 | " ('TensorFlow Models', 'pb'),\n", 240 | " ('Keras Models', 'h5'),\n", 241 | "\n", 242 | " # Checkpoint Files 🎯\n", 243 | " ('Checkpoints', 'ckpt'),\n", 244 | " ('Binary Files', 'bin'),\n", 245 | "\n", 246 | " # Config & Data Files 📝\n", 247 | " ('JSON Files', 'json'),\n", 248 | " ('YAML Files', 'yaml'),\n", 249 | " ('YAML Alt', 'yml'),\n", 250 | " ('Text Files', 'txt'),\n", 251 | " ('CSV Files', 'csv'),\n", 252 | " ('Pickle Files', 'pkl'),\n", 253 | "\n", 254 | " # Image Files 🎨\n", 255 | " ('PNG Images', 'png'),\n", 256 | " ('JPEG Images', 'jpg'),\n", 257 | " ('JPEG Alt', 'jpeg'),\n", 258 | " ('WebP Images', 'webp'),\n", 259 | " ('GIF Images', 'gif'),\n", 260 | "\n", 261 | " # Archive Files 📦\n", 262 | " ('ZIP Archives', 'zip'),\n", 263 | " ('TAR Files', 'tar'),\n", 264 | " ('GZ Archives', 'gz')\n", 265 | " ]\n", 266 | " self.current_directory = os.getcwd() # Set to the current working directory\n", 267 | " self.progress_bar = FloatProgress(value=0, min=0, max=100, description='Uploading:', bar_style='info')\n", 268 | " self._create_widgets()\n", 269 | " self._bind_events()\n", 270 | "\n", 271 | " def _create_widgets(self):\n", 272 | " # Repository info section\n", 273 | " self.repo_info = HTML(value=\"📚 Repository Details\")\n", 274 | " self.org_name = Text(\n", 275 | " placeholder='Organization or Username',\n", 276 | " description='Owner:',\n", 277 | " style={'description_width': 'initial'}\n", 278 | " )\n", 279 | " self.repo_name = Text(\n", 280 | " placeholder='Repository Name',\n", 281 | " description='Repo:',\n", 282 | " style={'description_width': 'initial'}\n", 283 | " )\n", 284 | " self.repo_type = Dropdown(\n", 285 | " options=['model', 'dataset', 'space'],\n", 286 | " value='model',\n", 287 | " description='Repo Type:',\n", 288 | " style={'description_width': 'initial'}\n", 289 | " )\n", 290 | " # Subfolder in Repo\n", 291 | " self.repo_folder = Text(\n", 292 | " placeholder='Optional Folder in Repo',\n", 293 | " description='Subfolder:',\n", 294 | " style={'description_width': 'initial'}\n", 295 | " )\n", 296 | " # File handling section\n", 297 | " self.file_section = HTML(value=\"🗂️ File Selection\")\n", 298 | " self.file_type = Dropdown(\n", 299 | " options=self.file_types,\n", 300 | " value='safetensors',\n", 301 | " description='File Type:',\n", 302 | " style={'description_width': 'initial'}\n", 303 | " )\n", 304 | " self.sort_by = Dropdown(\n", 305 | " options=['name', 'date'],\n", 306 | " value='name',\n", 307 | " description='Sort By:'\n", 308 | " )\n", 309 | " self.directory_label = Label(value=f\"Current Directory: {self.current_directory}\")\n", 310 | " self.directory_text = Text(\n", 311 | " value=self.current_directory,\n", 312 | " description=\"Path:\",\n", 313 | " style={'description_width': 'initial'},\n", 314 | " layout=Layout(width=\"400px\")\n", 315 | " )\n", 316 | " self.directory_update_btn = Button(\n", 317 | " description='🔄 Update Dir',\n", 318 | " button_style='info',\n", 319 | " tooltip='Refresh directory'\n", 320 | " )\n", 321 | "\n", 322 | " # Custom commit message\n", 323 | " self.commit_section = HTML(value=\"💭 Commit Details\")\n", 324 | " self.commit_msg = Textarea(\n", 325 | " value=\"Upload with Earth & Dusk Huggingface 🤗 Backup\",\n", 326 | " placeholder='Enter your commit message (optional)',\n", 327 | " description='Message:',\n", 328 | " layout=Layout(width='400px', height='60px')\n", 329 | " )\n", 330 | "\n", 331 | " # Upload options section\n", 332 | " self.upload_section = HTML(value=\"🚀 Upload Settings\")\n", 333 | " self.create_pr = Checkbox(\n", 334 | " value=False,\n", 335 | " description='Create Pull Request',\n", 336 | " indent=False,\n", 337 | " tooltip='When checked, creates a Pull Request instead of direct upload'\n", 338 | " )\n", 339 | " self.clear_after = Checkbox(\n", 340 | " value=True,\n", 341 | " description='Clear output after upload',\n", 342 | " indent=False,\n", 343 | " tooltip='Clears output area after successful upload'\n", 344 | " )\n", 345 | "\n", 346 | " # Action buttons with style\n", 347 | " self.update_btn = Button(\n", 348 | " description='🔄 Update Files',\n", 349 | " button_style='info',\n", 350 | " tooltip='Refresh file list'\n", 351 | " )\n", 352 | " self.upload_btn = Button(\n", 353 | " description='⬆️ Upload',\n", 354 | " button_style='success',\n", 355 | " tooltip='Start upload process',\n", 356 | " layout=Layout(width='auto', height='auto'),\n", 357 | " )\n", 358 | " self.clear_btn = Button(\n", 359 | " description='🧹 Clear Output',\n", 360 | " button_style='warning',\n", 361 | " tooltip='Clear output area'\n", 362 | " )\n", 363 | "\n", 364 | " # File selector and output\n", 365 | " self.ckpt_picker = SelectMultiple(\n", 366 | " options=[],\n", 367 | " layout=Layout(width=\"600px\", height=\"200px\")\n", 368 | " )\n", 369 | " self.out = Output(layout=Layout(padding='10px'))\n", 370 | "\n", 371 | " self.current_file_label = Label(value=\"Ready.\")\n", 372 | " self.progress_percent_label = Label(value=\"0%\")\n", 373 | "\n", 374 | " def _bind_events(self):\n", 375 | " self.update_btn.on_click(self._update_files)\n", 376 | " self.upload_btn.on_click(self._upload_ckpts)\n", 377 | " self.clear_btn.on_click(lambda _: self.out.clear_output())\n", 378 | " self.file_type.observe(self._update_files, names='value')\n", 379 | " self.directory_update_btn.on_click(self._update_directory)\n", 380 | "\n", 381 | " def _update_directory(self, _):\n", 382 | " new_dir = self.directory_text.value\n", 383 | " if os.path.isdir(new_dir):\n", 384 | " self.current_directory = new_dir\n", 385 | " self.directory_label.value = f\"Current Directory: {self.current_directory}\"\n", 386 | " self._update_files(None)\n", 387 | " else:\n", 388 | " with self.out:\n", 389 | " print(\"❌ Invalid Directory\")\n", 390 | "\n", 391 | " def _update_files(self, _):\n", 392 | " file_extension = self.file_type.value\n", 393 | " try:\n", 394 | " # Glob files based on file extension\n", 395 | " all_files = glob.glob(os.path.join(self.current_directory, f\"*.{file_extension}\"))\n", 396 | "\n", 397 | " # Filter out symlinks, old files and ignore patterns\n", 398 | " filtered_files = []\n", 399 | " for file_path in all_files:\n", 400 | " if os.path.islink(file_path):\n", 401 | " with self.out:\n", 402 | " print(f\"ℹ️ Skipping symlink: {file_path}\")\n", 403 | " continue\n", 404 | " if not os.path.isfile(file_path):\n", 405 | " with self.out:\n", 406 | " print(f\"ℹ️ Skipping non-file: {file_path}\")\n", 407 | " continue\n", 408 | "\n", 409 | " filtered_files.append(file_path)\n", 410 | "\n", 411 | " # Sort the files\n", 412 | " all_ckpts = sorted(\n", 413 | " filtered_files,\n", 414 | " key=os.path.getmtime if self.sort_by.value == 'date' else str\n", 415 | " )\n", 416 | "\n", 417 | " self.ckpt_picker.options = all_ckpts\n", 418 | "\n", 419 | " with self.out:\n", 420 | " print(f\"✨ Found {len(all_ckpts)} {file_extension} files in {self.current_directory}\")\n", 421 | "\n", 422 | " except Exception as e:\n", 423 | " with self.out:\n", 424 | " print(f\"❌ Error listing files: {str(e)}\")\n", 425 | "\n", 426 | " def _format_size(self, size):\n", 427 | " for unit in ['B', 'KB', 'MB', 'GB']:\n", 428 | " if size < 1024.0: # Use float for consistent division\n", 429 | " return f\"{size:.2f} {unit}\"\n", 430 | " size /= 1024.0\n", 431 | " return f\"{size:.2f} TB\"\n", 432 | "\n", 433 | " def _print_file_info(self, file_path, file_size, index, total):\n", 434 | " with self.out:\n", 435 | " print(f\"📦 File {index}/{total}: {file_path} ({self._format_size(file_size)})\")\n", 436 | "\n", 437 | " def _upload_ckpts(self, _):\n", 438 | " if not self.org_name.value or not self.repo_name.value:\n", 439 | " with self.out:\n", 440 | " print(\"❗ Please fill in both Organization/Username and Repository name\")\n", 441 | " return\n", 442 | "\n", 443 | " repo_id = f\"{self.org_name.value}/{self.repo_name.value}\"\n", 444 | " selected_files = self.ckpt_picker.value\n", 445 | " total_files = len(selected_files)\n", 446 | " repo_type = self.repo_type.value\n", 447 | " repo_folder = self.repo_folder.value.strip() # Get folder and strip whitespace\n", 448 | " current_directory = self.directory_text.value # Get the correct current directory\n", 449 | "\n", 450 | " with self.out:\n", 451 | " if not selected_files:\n", 452 | " print(\"📝 Nothing selected for upload. Please select files from the list.\")\n", 453 | " return\n", 454 | "\n", 455 | " print(f\"🎯 Starting upload to: huggingface.co/{repo_id}\")\n", 456 | "\n", 457 | " commit_msg = (self.commit_msg.value\n", 458 | " if self.commit_msg.value else\n", 459 | " \"Uploaded with Earth & Dusk Huggingface 🤗 Backup\")\n", 460 | "\n", 461 | " # Update current file label and progress (make them visible)\n", 462 | " file_label_container = VBox([\n", 463 | " HTML(value=\"Current Uploading File:\"),\n", 464 | " self.current_file_label\n", 465 | " ], layout=Layout(margin='10px 0 0 0'))\n", 466 | " display(file_label_container) # Display it\n", 467 | " progress_container = HBox([\n", 468 | " HTML(value=\"Progress:\"),\n", 469 | " self.progress_percent_label,\n", 470 | " self.progress_bar # Add progress bar here\n", 471 | " ], layout=Layout(align_items='center'))\n", 472 | " display(progress_container) # Display Progress\n", 473 | "\n", 474 | " for idx, ckpt in enumerate(selected_files, 1):\n", 475 | " try:\n", 476 | " self.current_file_label.value = f\"{ckpt}\" # Update the current file label\n", 477 | " file_size = os.path.getsize(ckpt)\n", 478 | " self._print_file_info(ckpt, file_size, idx, total_files)\n", 479 | " start_time = time.time()\n", 480 | "\n", 481 | " # Use os.path to ensure forward slashes for paths\n", 482 | " path_in_repo = os.path.basename(ckpt)\n", 483 | "\n", 484 | " # Handle folders by creating folder in the repo\n", 485 | " # Split the path\n", 486 | " path_parts = Path(ckpt).parts\n", 487 | " if len(path_parts) > 1:\n", 488 | " # Get only the folder names\n", 489 | " folder_path_parts = path_parts[len(Path(current_directory).parts):-1]\n", 490 | " # Generate the folder path in the repo\n", 491 | " if folder_path_parts:\n", 492 | " path_in_repo = os.path.join(*folder_path_parts, os.path.basename(ckpt))\n", 493 | "\n", 494 | " # Add the subfolder to the path\n", 495 | " if repo_folder:\n", 496 | " path_in_repo = os.path.join(repo_folder, path_in_repo)\n", 497 | "\n", 498 | " response = self.api.upload_file(\n", 499 | " path_or_fileobj=ckpt,\n", 500 | " path_in_repo=path_in_repo, # Use the path in repo\n", 501 | " repo_id=repo_id,\n", 502 | " repo_type=repo_type, # Use the repo type dropdown value here\n", 503 | " create_pr=self.create_pr.value,\n", 504 | " commit_message=commit_msg\n", 505 | " )\n", 506 | " duration = time.time() - start_time\n", 507 | " with self.out:\n", 508 | " print(f\"✅ Upload completed in {duration:.1f} seconds\")\n", 509 | " display(response)\n", 510 | "\n", 511 | " # Calculate and update progress percentage\n", 512 | " percentage = int((idx / total_files) * 100)\n", 513 | " self.progress_percent_label.value = f\"{percentage}%\"\n", 514 | " self.progress_bar.value = percentage # Update progress bar\n", 515 | "\n", 516 | " except Exception as e:\n", 517 | " with self.out:\n", 518 | " print(f\"❌ Error uploading {ckpt}: {type(e).__name__} - {str(e)}\") #Detailed error\n", 519 | " continue\n", 520 | "\n", 521 | " with self.out:\n", 522 | " print(\"\\n✨ All uploads completed! ✨\")\n", 523 | " if self.create_pr.value:\n", 524 | " print(\"🎉 Check your repository for the new Pull Request!\")\n", 525 | " else:\n", 526 | " print(\"🎉 Files have been uploaded directly to your repository!\")\n", 527 | "\n", 528 | " if self.clear_after.value:\n", 529 | " time.sleep(3)\n", 530 | " self.out.clear_output()\n", 531 | " # Hide the current file label and progress after upload:\n", 532 | " file_label_container.layout.visibility = 'hidden'\n", 533 | " progress_container.layout.visibility = 'hidden'\n", 534 | "\n", 535 | " def display(self):\n", 536 | " box = VBox([\n", 537 | " self.repo_info,\n", 538 | " HBox([self.org_name, self.repo_name, self.repo_type]),\n", 539 | " HBox([self.repo_folder]),\n", 540 | " self.file_section,\n", 541 | " HBox([self.file_type, self.sort_by]),\n", 542 | " HBox([self.directory_label, self.directory_text, self.directory_update_btn]),\n", 543 | " self.commit_section,\n", 544 | " self.commit_msg,\n", 545 | " self.upload_section,\n", 546 | " HBox([self.create_pr, self.clear_after]),\n", 547 | " self.update_btn,\n", 548 | " self.ckpt_picker,\n", 549 | " HBox([self.upload_btn, self.clear_btn], layout=Layout(align_items='center')),\n", 550 | " self.out, #Removed self.progress_text\n", 551 | " ])\n", 552 | " display(box)\n", 553 | "\n", 554 | "# Create and display the uploader - just one line to rule them all! ✨\n", 555 | "uploader = HuggingFaceUploader()\n", 556 | "uploader.display(); # Semicolon suppresses the output" 557 | ] 558 | } 559 | ], 560 | "metadata": { 561 | "colab": { 562 | "collapsed_sections": [ 563 | "IZ_JYwvBLrg-", 564 | "PNF2kdyeO3Dn" 565 | ], 566 | "private_outputs": true, 567 | "provenance": [] 568 | }, 569 | "kernelspec": { 570 | "display_name": "Python 3 (ipykernel)", 571 | "language": "python", 572 | "name": "python3" 573 | }, 574 | "language_info": { 575 | "codemirror_mode": { 576 | "name": "ipython", 577 | "version": 3 578 | }, 579 | "file_extension": ".py", 580 | "mimetype": "text/x-python", 581 | "name": "python", 582 | "nbconvert_exporter": "python", 583 | "pygments_lexer": "ipython3", 584 | "version": "3.10.12" 585 | } 586 | }, 587 | "nbformat": 4, 588 | "nbformat_minor": 4 589 | } 590 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # 🚀 Doro Hugging Face Uploader: Streamline Your Model Sharing! 🚀 2 | 3 | [![Model on HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/model-on-hf-xl-dark.svg)](https://huggingface.co/Duskfallcrew/Huggingface_Backup) [![GitHub](https://img.shields.io/badge/github-%23121011.svg?style=for-the-badge&logo=github&logoColor=white)](https://github.com/duskfallcrew/HuggingFace_Backup) 4 | 5 | This tool provides a user-friendly way to upload files directly to your Hugging Face repositories using a Jupyter Notebook. Designed to streamline your workflow, it makes sharing your models, datasets, and spaces easier than ever! 6 | 7 | We are actively working on further development and security enhancements for the Jupyter Notebook edition. 8 | 9 | We are also actively struggling to update and keep track of the Google Colab version. 10 | 11 | > 2025 'EXTRA' is the version with the extra cell to zip up images or files before uploading to huggingface. March 2025 has minor adjustments to environment code. 12 | > Dataset Zipper is a standalone zip & download function notebook, which it's cell is in the 'extra' notebook as well. 13 | > The folder structure is much nicer now, and the May 2025 one NOW HAS added hf transfer for better serverside uploads 14 | 15 | --- 16 | 17 | ## 📝 Table of Contents 18 | 19 | * [Key Features](#key-features) 20 | * [Jupyter Notebook Edition: Interactive and User-Friendly](#-jupyter-notebook-edition-interactive-and-user-friendly) 21 | * [One-Time Setup for Jupyter Users](#one-time-setup-for-jupyter-users) 22 | * [Using the Uploader Widget in the Jupyter Notebook](#️-using-the-uploader-widget-in-the-jupyter-notebook) 23 | * [Important Notes for Jupyter Users](#💡-important-notes-for-jupyter-users) 24 | * [Updates & Community](#-updates--community) 25 | * [About Us](#-about-us) 26 | * [Let's Connect!](#-lets-connect) 27 | * [Support Our Adventures](#-support-our-adventures) 28 | * [Proudly Supported By](#-proudly-supported-by) 29 | * [Need Help?](#️-need-help) 30 | * [Credits & Origins](#-credits--origins) 31 | * [Changelog: Our Journey So Far](#-changelog-our-journey-so-far) 32 | 33 | --- 34 | 35 | ## Key Features 36 | 37 | | Feature | Badge/Link | Description | 38 | | :---------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------------------------------------------------------------------------- | 39 | | **Coded With Help From** | [![Google Gemini](https://img.shields.io/badge/google%20gemini-8E75B2?style=for-the-badge&logo=google%20gemini&logoColor=white)](https://gemini.google.com/) | Acknowledgment of AI assistance in development. | 40 | | **Language** | [![Python](https://img.shields.io/badge/python-3670A0?style=for-the-badge&logo=python&logoColor=ffdd54)](https://www.python.org/) | Developed primarily in Python. | 41 | | **Hugging Face Repository** | [![Model on HF](https://huggingface.co/datasets/huggingface/badges/resolve/main/model-on-hf-xl-dark.svg)](https://huggingface.co/Duskfallcrew/Huggingface_Backup) | Link to the Hugging Face repository for the tool. | 42 | | **GitHub Repository** | [![GitHub](https://img.shields.io/badge/github-%23121011.svg?style=for-the-badge&logo=github&logoColor=white)](https://github.com/duskfallcrew/HuggingFace_Backup) | Link to the GitHub repository for the tool. Always check here for the latest updates. | 43 | | **Open in Google Colab** | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/Ktiseos-Nyx/HuggingFace_Backup/blob/main/HuggingFace_Backup_2024_Colab.ipynb) | Directly open and run the Jupyter Notebook in Google Colab. | 44 | 45 | --- 46 | 47 | ## 💻 Jupyter Notebook Edition: Interactive and User-Friendly 48 | 49 | For an interactive and visually intuitive experience, use our Jupyter Notebook edition. 50 | 51 | 52 |
53 | 54 | ### One-Time Setup for Jupyter Users 55 | 56 | 1. **How To Get Your Huggingface API Token:** 57 | 58 | * Navigate to your [Hugging Face settings page](https://huggingface.co/settings/tokens). 59 | * Click "New token." 60 | * Provide a descriptive name for your token (e.g., "Uploader Token"). 61 | * Select the appropriate role and permissions. For uploading files, ensure it has "write" permissions. 62 | * Click "Generate token." 63 | * **Securely copy the generated API token.** This token is essential for authentication and will only be displayed once. If lost, you will need to generate a new one. 64 | 65 | 2. **Authentication:** After generating your API token, execute the `notebook_login()` cell *once* at the beginning of the notebook. This securely stores your Hugging Face API token for the session. 66 | 67 | * **Important Security Note:** To maintain the security of your API token, avoid sharing the notebook file or system state after running `notebook_login()`. **Do not commit the notebook to public repositories**, especially if using platforms like Google Colab. If you suspect your token has been compromised, generate a new one immediately. 68 | 69 | ### 🗂️ Using the Uploader Widget in the Jupyter Notebook 70 | 71 | 1. **Repository Details:** 72 | * **Owner:** Enter your Hugging Face Organization or Username. This is the name found in your repository URL. 73 | * **Repo:** Enter the name of the repository you wish to upload to. 74 | * Ensure both "Owner" and "Repo" names are accurate to prevent upload failures. 75 | * **Repo Type:** Select the repository type from the dropdown menu: "model," "dataset," or "space." 76 | * **Subfolder:** (Optional) To upload files to a specific subfolder within your repository, enter the subfolder name. If left blank, files will be uploaded to the repository root. 77 | 78 | 2. **Directory Selection:** 79 | * **Path:** Enter the full path to the local directory containing the files you want to upload. 80 | * **Update Dir:** Click '🔄 Update Dir' to set the path and refresh the file list based on the selected file type. 81 | * If no files appear, verify that the path and selected file type are correct. 82 | 83 | 3. **File Selection:** 84 | * **File Type:** Choose the appropriate file type from the dropdown (e.g., `safetensors`, `png`, `txt`). This will display all files of that type within the specified directory. 85 | * **File List:** Select the files for upload from the displayed list. Use the "Sort By" dropdown to organize files by name or modification date. 86 | 87 | 4. **Commit Message:** 88 | * **(Optional):** Add a specific message for the commit in the "Message" field. If left empty, a default message "Uploaded with Earth & Dusk Huggingface 🤗 Backup" will be used. 89 | 90 | 5. **Upload Options:** 91 | * **Create Pull Request:** Check this box to upload changes as a pull request. Unchecked, changes are directly committed to the main branch. 92 | * **Clear output after upload:** Select this option to clear the output area after a successful upload. 93 | 94 | 6. **Start Upload:** 95 | * Click '⬆️ Upload' to begin the upload process. 96 | * The current file being uploaded and the upload progress percentage will be displayed. 97 | 98 | ### 💡 Important Notes for Jupyter Users 99 | 100 | * **Direct Uploads:** This tool utilizes the Hugging Face API for direct file uploads, bypassing Git operations for core functionality. Command-line Git usage is not required for basic uploads. 101 | * **Git LFS:** Git LFS is not necessary for using this tool. Separate Git credentials are required for repository operations outside of this notebook (like cloning or pushing via Git) and should not be stored within the notebook. 102 | * **Subfolders:** The tool will create subfolders in your repository based on your local file structure or the specified "Subfolder." 103 | * **Troubleshooting:** If you encounter issues, re-verify all steps, ensure you have write access to the repository, and that your API token has the correct permissions. Double-check file paths and file types. 104 | 105 |
106 | 107 | ## 📣 Updates & Community 108 | 109 | * This tool is continuously updated and improved. 110 | * For the latest updates, fixes, and community contributions, please visit the [GitHub repository](https://github.com/duskfallcrew/HuggingFace_Backup). 111 | 112 | We hope this tool simplifies your Hugging Face uploads! For questions or suggestions, please reach out. 113 | 114 |
115 | 116 | ## 🌈 About Us 117 | 118 | We are a diverse system of 300+ alters, navigating life with DID, ADHD, Autism, and CPTSD. We believe in the positive potential of AI for mental health and creativity and are excited to explore this intersection. 119 | 120 | This project is managed by **Ktiseos Nyx**, the programming division of Earth & Dusk. 121 | 122 |
123 | 124 | ### 🤝 Let's Connect! 125 | 126 | | Platform | Link | 127 | | :---------------- | :------------------------------------------ | 128 | | **Website** | [End Media](https://www.end-media.org/) | 129 | | **Discord** | [Discord Community](https://discord.gg/5t2kYxt7An) | 130 | | **Hugging Face** | [HuggingFace Space](https://huggingface.co/EarthnDusk) | 131 | | **YouTube** | [YouTube Channel](https://www.youtube.com/channel/UCk7MGP7nrJz5awBSP75xmVw) | 132 | | **DeviantArt** | [DeviantArt Group](https://www.deviantart.com/diffusionai) | 133 | | **Subreddit** | [Subreddit](https://www.reddit.com/r/earthndusk/) | 134 | 135 |
136 | 137 | ### ☕ Support Our Adventures 138 | 139 | [![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/Z8Z8L4EO) 140 | 141 | You can support our work and future development through Ko-fi. 142 | 143 |
144 | 145 | ## 🏴‍☠️ Proudly Supported By 146 | 147 | * [Pirate Diffusion](https://www.piratediffusion.com/) 148 | * [Yodayo](https://yodayo.com/) 149 | 150 |
151 | 152 | ## 🛠️ Need Help? 153 | 154 | If you encounter a bug or need assistance, please reach out through: 155 | 156 | * GitHub Pull Requests & Bug Tracker 157 | * CivitAi Direct Messages/Comments 158 | * Earth & Dusk Discord 159 | 160 |
161 | 162 | ## 💝 Credits & Origins 163 | 164 | We extend our sincere gratitude to the original creators and contributors who laid the foundation for this project: 165 | 166 | * EVERYDREAM2 TRAINER: [https://github.com/victorchall/EveryDream2trainer](https://github.com/victorchall/EveryDream2trainer) 167 | * LINAQRUF 168 | * NOCRYPT: [![](https://dcbadge.vercel.app/api/shield/442099748669751297?style=flat)](https://lookup.guru/442099748669751297) 169 | 170 | Explore the original Stable Diffusion Colab notebook: 171 | 172 | [Open Original SD Colab](https://colab.research.google.com/drive/1wEa-tS10h4LlDykd87TF5zzpXIIQoCmq) 173 |
174 | 175 | ## 📝 Changelog: Our Journey So Far 176 | 177 | 1. **🔧 Enhanced with EveryDream2Trainer's Python widget integration.** 178 | 2. **🌟 Maintained and highlighted Nocrypt's valuable contributions.** 179 | 3. **📦 Integrated essential functionalities into a single Jupyter Notebook.** 180 | 4. **🤓 Improved file handling with proper file extensions (*.safetensors).** 181 | 5. **📝 Created user-friendly and accessible instructions.** 182 | 6. **🤖 Utilized GPT assistance to enhance clarity and readability.** 183 | 7. **🎨 Improved the visual presentation of the Jupyter Notebook edition.** 184 | 8. **🔄 Synchronized Colab and Jupyter Notebook versions for consistency.** 185 | 9. **🧹 Optimized dependencies by removing unnecessary transformers.** 186 | 10. **✨ Added advanced folder upload capabilities.** 187 | 11. **🔄 Refined and updated the Colab notebook for improved performance.** 188 | 12. **🚀 Implemented more concise widget features and restored Colab functionality.** 189 | 13. **💎 Underwent a Gemini-assisted overhaul for improved quality.** 190 | 14. **➕ Expanded supported file types in Colab, streamlined instructions, and introduced command-line functionality (Note: Command-line section removed as per request).** 191 | 192 | We acknowledge that we are not professional programmers, and appreciate community contributions. Pull requests are always welcome for improvements! 🎉 193 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | # 🛡️ Security Policy 2 | 3 | ## 💫 Supported Versions 4 | Hey community! Here's the current scoop on which versions we're actively supporting and keeping safe: 5 | 6 | | Version | Status | Notes | 7 | |---------|---------|-------| 8 | | 3.0.x | ✅ | Current stable release - All the shiny new features! | 9 | | 2.5.x | ✅ | Legacy support until 2024 | 10 | | 2.0.x | ❌ | Time to upgrade, friend! | 11 | | < 2.0 | ❌ | Here be dragons - please update! | 12 | 13 | ## 🌈 Version Features 14 | - **3.0.x**: Our latest and greatest! Full Colab & Jupyter support 15 | - **2.5.x**: The stable one everyone knows and loves 16 | - **2.0.x**: The OG version (but maybe time for an upgrade?) 17 | 18 | ## 🎯 Reporting Vulnerabilities 19 | 20 | Found something wonky? We've got your back! Here's how to let us know: 21 | 22 | 1. **Quick Issues**: 23 | - Drop by our [GitHub Issues](https://github.com/yourusername/repo/issues) 24 | - Tag it with 'security' for priority attention 25 | 26 | 2. **Sensitive Stuff**: 27 | - Message us on [Discord](https://discord.gg/5t2kYxt7An) 28 | - Or email us at [your-email] 29 | 30 | ### What to Expect: 31 | - Initial response within 48 hours (we're human!) 32 | - Updates every 3-5 days while we investigate 33 | - Credit for responsible disclosure (if you want!) 34 | 35 | ## 💝 A Note from Us 36 | Remember: Security is a community effort! We might not be security experts, but we care deeply about keeping our tools safe and accessible for everyone. Your help in spotting and reporting issues makes this space better for all of us! 37 | 38 | Questions? Concerns? Random thoughts about AI? We're here for all of it! 🌟 39 | --------------------------------------------------------------------------------