├── .gitattributes ├── CHANGELOG_BLORA_KOHYA.md ├── README.md ├── blora_slicer.bat ├── blora_slicer.py ├── blora_traits.json ├── lycoris_presets ├── bdora_content_style_clip.toml ├── blora_content_layout_style.toml ├── blora_content_style.toml ├── blora_content_style_clip.toml └── blora_layout_style.toml ├── sdxl_blora_classic.bat └── sdxl_blora_fast.bat /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /CHANGELOG_BLORA_KOHYA.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | All notable changes to this project will be documented in this file. 3 | 4 | ## 0.3.0 - 6 August 2024 5 | 6 | ### Added 7 | - New Lycoris preset `blora_content_style_clip.toml` which targets the first two layers of the text encoders (0 and 1) for training, applying the "content" and "style" separation idea of B-LoRA 8 | - New Lycoris preset `bdora_content_style_clip.toml`: same as above, but also enables DoRA weight composition 9 | - New B-LoRA trait `clip_2_content` for use with `blora_slicer.py`; slices layer 0 of the 2nd text encoder, which seems to effectively learn content while minimally impacting style 10 | - New `blora_slicer.py` setting `--not_combined`: Saves out individual LoRA files instead of packing all settings into one 11 | 12 | ### Changed 13 | - Set `use_bias_correction=True` in `sdxl_blora_fast.bat` while I run more tests to determine this option's impact on quality 14 | 15 | ## 0.2.0 - 12 July 2024 16 | 17 | ### Added 18 | - New preset `sdxl_blora_fast.bat`: Optimized training settings to dramatically improve convergence time, while (hopefully) maintaining quality - feedback would be appreciated 19 | 20 | ### Changed 21 | - Renamed preset `sdxl_blora_project.bat` to `sdxl_blora_classic.bat` and added 500 steps of warmup to match origin B-LoRA repo 22 | 23 | ## 0.1.0 - 1 July 2024 24 | 25 | ### Added 26 | - New Lycoris preset `blora_content_layout_style.toml`: targets additional blocks that allegedly correlate to image layout (which may further reduce overfitting, but this is not used in original B-LoRA) 27 | - New Lycoris preset `blora_layout_style.toml` 28 | - The B-LoRA Slicer will now load `blora_traits.json` for arbitrary block filters, you can easily add your own to this file 29 | 30 | ### Changed 31 | - The B-LoRA Slicer now uses `--loras`, `--traits`, and `--alphas` arguments so you can specify multiple arbitrary operations (i.e. no longer limited to just content and style) 32 | 33 | ## 0.0.1 - 30 June 2024 34 | 35 | ### Added 36 | - Initial release -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # B-LoRA for Kohya-SS 2 | 3 | This repository contains tools needed for training [B-LoRA](https://github.com/yardenfren1996/B-LoRA) files with [kohya-ss/sd-scripts](https://github.com/kohya-ss/sd-scripts). 4 | 5 | ## What is B-LoRA and why should I care? 6 | 7 | The B-LoRA method *"enables implicit style-content separation"* for SDXL LoRAs. In other words, if you wish to train a character, you can use this method to avoid picking up unwanted styles, colors or layouts - i.e. features that often creep into character LoRAs by mistake. 8 | 9 | B-LoRA's approach works quite well, and it deserves more recognition in the broader Stable Diffusion community. 10 | 11 | Combining it with sd-scripts gives you access to awesome features like aspect ratio bucketing. 12 | 13 | ## How does B-LoRA work? 14 | 15 | B-LoRA targets specific unet blocks that correlate surprisingly well to `content` and `style`. It seems best to learn these traits in tandem, and then use the provided `blora_slicer.py` on the final LoRA to preserve only the traits you want to keep. 16 | 17 | Additionally, the B-LoRA trainer has several non-standard options that I have specified in `sdxl_blora_classic.bat`. 18 | 19 | ## Setup and Use 20 | 21 | 1. Install the [Lycoris network](https://github.com/KohakuBlueleaf/LyCORIS). 22 | 2. Download this repo and place all its files into the root of your `kohya-ss` directory. 23 | 3. Make a copy of `sdxl_blora_classic.bat`, adjust it to your needs (in particular, the topmost variables and paths), then launch to begin training. 24 | 4. Run `blora_slicer.bat` on the resulting LoRA to filter out `content` or `style` blocks. This will produce a final, smaller LoRA that you can use in the WebUI or Comfy. 25 | 26 | Note that while these batch files are for Windows, it should be trivial to adapt them to Linux. 27 | 28 | If you're using the Kohya GUI instead of sd-scripts, the Lycoris features are pre-installed and you will need to plug in the `sdxl_blora_classic.bat` settings by hand. 29 | 30 | --- 31 | 32 | ## Notes on `sdxl_blora_classic.bat` 33 | 34 | - The B-LoRA method responds well to training a substantially higher number of dimensions than usual. On my GeForce 3090, I have set the rank to `1024` and observed great results for `content` without overfitting. This does increase training time and hardware requirements, however. For anime characters and other simple subjects, you can probably get away with a much lower rank. 35 | - Despite the use of the `prodigy` optimizer, B-LoRA training time is pretty slow, possibly due to options such as `use_bias_correction`. You can try the experimental `sdxl_blora_fast.bat` preset instead - it improves convergence time but may come at a slight hit to quality (more testing needed.) 36 | - As a point of reference, one of my character datasets contains 168 images and takes ~3 hours to train to 3000 steps. This yields incredibly good likeness without significant signs of overfitting. 37 | 38 | ## Notes on `blora_slicer.bat` 39 | 40 | This tool is adapted from B-LoRA's `inference.py`. 41 | 42 | You only need to set up one argument: 43 | 44 | - `--loras`: Path(s) to one or more `safetensors` files to extract the blocks from. Most likely, you want to set this to the combined B-LoRA you trained with kohya. 45 | 46 | Additional optional arguments are available: 47 | 48 | - `--traits`: A list of traits to filter from your LoRAs, in the same order as the LoRAs. Defaults to `content`. Check the included `blora_traits.json` for supported traits. 49 | - `--alphas`: A list of alpha values to scale the LoRAs, in the same order as the LoRAs. Defaults to `1.0`, or full strength. 50 | - `--output_path`: The save location of the sliced-up LoRA. Defaults to `model.safetensors` in the same directory as this script. 51 | - `--debug`: Print some diagnostic information to the console. 52 | 53 | ## Notes on Lycoris presets 54 | 55 | The Lycoris preset determines which unet blocks to train. 56 | 57 | - `blora_content_style.toml`: This preset matches the original B-LoRA method, training content and style blocks in tandem. 58 | - `blora_content_layout_style.toml`: In addition to content and style, this preset targets blocks that allegedly correlate to image layout. This can potentially improve learning/reduce potential for overfitting, but it has a couple drawbacks: 1) higher VRAM requirement, limiting the number of dimensions you can train. 2) In my initial testing, the layout blocks appear to contain some information that affects character likeness, or what I would classify as "content." 59 | 60 | --- 61 | 62 | ⭐ Feel free to give this repo a star if you found it helpful. 63 | 64 | Thank you to @slashedstar for [introducing the idea of using Lycoris to recreate the B-LoRA method.](https://github.com/kohya-ss/sd-scripts/issues/1215) -------------------------------------------------------------------------------- /blora_slicer.bat: -------------------------------------------------------------------------------- 1 | call ./venv/Scripts/activate 2 | python blora_slicer.py --loras C:/path/to/lora.safetensors --traits content --debug 3 | pause -------------------------------------------------------------------------------- /blora_slicer.py: -------------------------------------------------------------------------------- 1 | # Adapted from the original B-LoRA inference.py script 2 | 3 | import torch, argparse, json 4 | from safetensors.torch import save_file, load_file 5 | 6 | print("B-LoRA Slicer v0.3.0 by Therefore Games") 7 | 8 | print("Parsing arguments...") 9 | 10 | parser = argparse.ArgumentParser(description='Process some paths and parameters.') 11 | parser.add_argument("--loras", type=str, nargs="*", required=True, help="Path(s) to one or more LoRA safetensors.") 12 | parser.add_argument("--traits", type=str, nargs="*", default=["content"], help="A list of traits to filter from the LoRAs, in the same order as the LoRAs.") 13 | parser.add_argument("--alphas", type=float, nargs="*", default=[1.0], help="A list of alpha values to scale the LoRAs, in the same order as the LoRAs.") 14 | parser.add_argument("--output_path", type=str, default="model.safetensors", help="Path to new file") 15 | parser.add_argument("--debug", action="store_true", help="Debug mode") 16 | parser.add_argument("--not_combined", action="store_true", help="Save each LoRA separately") 17 | 18 | args = parser.parse_args() 19 | 20 | # If there are more traits than LoRAs, repeat the LoRAs for the remaining traits 21 | if len(args.traits) > len(args.loras): 22 | print("More traits than LoRAs detected. Repeating the last LoRA for the remaining traits.") 23 | args.loras += [args.loras[-1]] * (len(args.traits) - len(args.loras)) 24 | 25 | # If there are fewer traits than LoRAs, repeat "content" for the remaining LoRAs 26 | if len(args.traits) < len(args.loras): 27 | print("More LoRAs than traits detected. Targeting 'content' for the remaining LoRAs.") 28 | args.traits += ["content"] * (len(args.loras) - len(args.traits)) 29 | 30 | # If there are fewer alphas than LoRAs, repeat 1.0 for the remaining LoRAs 31 | if len(args.alphas) < len(args.loras): 32 | print("More LoRAs than alphas detected. Using alpha value 1.0 for the remaining LoRAs.") 33 | args.alphas += [1.0] * (len(args.loras) - len(args.alphas)) 34 | 35 | 36 | def is_belong_to_blocks(key, whitelist, blacklist): 37 | try: 38 | for block in whitelist: 39 | # print(f"Testing block: {block}") 40 | if block in key: 41 | if any(bad_block in key for bad_block in blacklist): 42 | continue 43 | if args.debug: 44 | print(f"VALID! Key: {key}") 45 | return True 46 | if args.debug: 47 | print(f"Key: {key}") 48 | return False 49 | except Exception as e: 50 | raise type(e)(f'failed to is_belong_to_block, due to: {e}') 51 | 52 | 53 | def filter_lora(state_dict, whitelist, blacklist): 54 | try: 55 | return {k: v for k, v in state_dict.items() if is_belong_to_blocks(k, whitelist, blacklist)} 56 | except Exception as e: 57 | raise type(e)(f'failed to filter_lora, due to: {e}') 58 | 59 | 60 | def scale_lora(state_dict, alpha): 61 | try: 62 | return {k: v * alpha for k, v in state_dict.items()} 63 | except Exception as e: 64 | raise type(e)(f'failed to scale_lora, due to: {e}') 65 | 66 | 67 | print("Loading unet block traits...") 68 | with open("blora_traits.json", "r") as f: 69 | traits = json.load(f) 70 | 71 | # For each LoRA, filter and scale the traits 72 | loras = [] 73 | for i, lora_path in enumerate(args.loras): 74 | print(f"Loading LoRA: {lora_path}") 75 | lora_sd = load_file(lora_path) 76 | lora = filter_lora(lora_sd, traits[args.traits[i]]["whitelist"], traits[args.traits[i]]["blacklist"]) 77 | lora = scale_lora(lora, args.alphas[i]) 78 | loras.append(lora) 79 | 80 | if args.not_combined: 81 | print("Saving LoRAs separately...") 82 | for i, lora in enumerate(loras): 83 | save_file(lora, f"{args.traits[i]}_{i}.safetensors") 84 | else: 85 | # Merge B-LoRAs 86 | res_lora = {} 87 | for lora in loras: 88 | res_lora = {**res_lora, **lora} 89 | 90 | print("Saving new model...") 91 | save_file(res_lora, args.output_path) 92 | -------------------------------------------------------------------------------- /blora_traits.json: -------------------------------------------------------------------------------- 1 | { 2 | "content": 3 | { 4 | "whitelist": ["lora_unet_output_blocks_0_1_"], 5 | "blacklist": ["ff_net", "proj_in", "proj_out", "alpha"] 6 | }, 7 | "layout": 8 | { 9 | "whitelist": ["lora_unet_input_blocks_8_1_"], 10 | "blacklist": ["ff_net", "proj_in", "proj_out", "alpha"] 11 | }, 12 | "style": 13 | { 14 | "whitelist": ["lora_unet_output_blocks_1_1_"], 15 | "blacklist": ["ff_net", "proj_in", "proj_out", "alpha"] 16 | }, 17 | "clip_2_content": { 18 | "whitelist": ["lora_te2_text_model_encoder_layers_0_"], 19 | "blacklist": ["ff_net", "proj_in", "proj_out", "alpha"] 20 | } 21 | } -------------------------------------------------------------------------------- /lycoris_presets/bdora_content_style_clip.toml: -------------------------------------------------------------------------------- 1 | dora_wd = true 2 | enable_conv = false 3 | 4 | # An example for use different algo/settings in "full" preset 5 | unet_target_module = [ 6 | ] 7 | unet_target_name = [ 8 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.0\\.1\\..*$", 9 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.1\\.1\\..*$" 10 | ] 11 | text_encoder_target_module = [ 12 | ] 13 | text_encoder_target_name = [ 14 | "text_model.encoder.layers.0", 15 | "^text_model\\.encoder\\.layers\\.1(?!\\d)(\\..*)?$" 16 | ] -------------------------------------------------------------------------------- /lycoris_presets/blora_content_layout_style.toml: -------------------------------------------------------------------------------- 1 | enable_conv = false 2 | 3 | # An example for use different algo/settings in "full" preset 4 | unet_target_module = [ 5 | ] 6 | unet_target_name = [ 7 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.0\\.1\\..*$", # Content 8 | "^(?!.*(ff\\.net|proj)).*input_blocks\\.8\\.1\\..*$", # Layout 9 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.1\\.1\\..*$" # Style 10 | ] 11 | text_encoder_target_module = [ 12 | "CLIPAttention", 13 | ] 14 | text_encoder_target_name = [ 15 | ] -------------------------------------------------------------------------------- /lycoris_presets/blora_content_style.toml: -------------------------------------------------------------------------------- 1 | enable_conv = false 2 | 3 | # An example for use different algo/settings in "full" preset 4 | unet_target_module = [ 5 | ] 6 | unet_target_name = [ 7 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.0\\.1\\..*$", # Content 8 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.1\\.1\\..*$" # Style 9 | ] 10 | text_encoder_target_module = [ 11 | "CLIPAttention", 12 | ] 13 | text_encoder_target_name = [ 14 | ] -------------------------------------------------------------------------------- /lycoris_presets/blora_content_style_clip.toml: -------------------------------------------------------------------------------- 1 | enable_conv = false 2 | 3 | # An example for use different algo/settings in "full" preset 4 | unet_target_module = [ 5 | ] 6 | unet_target_name = [ 7 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.0\\.1\\..*$", 8 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.1\\.1\\..*$" 9 | ] 10 | text_encoder_target_module = [ 11 | ] 12 | text_encoder_target_name = [ 13 | "text_model.encoder.layers.0", 14 | "^text_model\\.encoder\\.layers\\.1(?!\\d)(\\..*)?$" 15 | ] -------------------------------------------------------------------------------- /lycoris_presets/blora_layout_style.toml: -------------------------------------------------------------------------------- 1 | enable_conv = false 2 | 3 | # An example for use different algo/settings in "full" preset 4 | unet_target_module = [ 5 | ] 6 | unet_target_name = [ 7 | "^(?!.*(ff\\.net|proj)).*input_blocks\\.8\\.1\\..*$", # Layout 8 | "^(?!.*(ff\\.net|proj)).*output_blocks\\.1\\.1\\..*$" # Style 9 | ] 10 | text_encoder_target_module = [ 11 | "CLIPAttention", 12 | ] 13 | text_encoder_target_name = [ 14 | ] -------------------------------------------------------------------------------- /sdxl_blora_classic.bat: -------------------------------------------------------------------------------- 1 | set NAME="my_project" 2 | set REVISION="1" 3 | set PROJECT="%NAME%_%REVISION%" 4 | set RANK=1024 5 | call ./venv/Scripts/activate 6 | 7 | accelerate launch --num_cpu_threads_per_process 8 sdxl_train_network.py ^ 8 | --pretrained_model_name_or_path="C:/path/to/sdxl_checkpoint.safetensors" ^ 9 | --dataset_config="./_in/lora/sdxl/%NAME%/config.toml" ^ 10 | --output_dir="./_out/lora/sdxl/%PROJECT%" ^ 11 | --output_name="%PROJECT%" ^ 12 | --network_args="preset=C:/absolute/path/to/lycoris_presets/blora_content_style.toml" ^ 13 | --resolution="1024,1024" ^ 14 | --save_model_as="safetensors" ^ 15 | --network_module="lycoris.kohya" ^ 16 | --max_train_steps=3000 ^ 17 | --save_every_n_steps=1000 ^ 18 | --save_every_n_epochs=1 ^ 19 | --network_dim=%RANK% ^ 20 | --network_alpha=%RANK% ^ 21 | --no_half_vae ^ 22 | --gradient_checkpointing ^ 23 | --persistent_data_loader_workers ^ 24 | --enable_bucket ^ 25 | --random_crop ^ 26 | --bucket_reso_steps=32 ^ 27 | --min_bucket_reso=512 ^ 28 | --mem_eff_attn ^ 29 | --mixed_precision="fp16" ^ 30 | --caption_extension=".txt" ^ 31 | --lr_scheduler="cosine" ^ 32 | --lr_warmup_steps=500 ^ 33 | --network_train_unet_only ^ 34 | --prior_loss_weight=0 ^ 35 | --optimizer_type="prodigy" ^ 36 | --max_grad_norm=1.0 ^ 37 | --learning_rate=1.0 ^ 38 | --seed=0 ^ 39 | --optimizer_args weight_decay=1e-04 betas=(0.9,0.999) eps=1e-08 decouple=True use_bias_correction=True safeguard_warmup=True beta3=None 40 | 41 | pause -------------------------------------------------------------------------------- /sdxl_blora_fast.bat: -------------------------------------------------------------------------------- 1 | set NAME="my_project" 2 | set REVISION="1" 3 | set PROJECT="%NAME%_%REVISION%" 4 | set RANK=1024 5 | call ./venv/Scripts/activate 6 | 7 | accelerate launch --num_cpu_threads_per_process 8 sdxl_train_network.py ^ 8 | --pretrained_model_name_or_path="C:/path/to/sdxl_checkpoint.safetensors" ^ 9 | --dataset_config="./_in/lora/sdxl/%NAME%/config.toml" ^ 10 | --output_dir="./_out/lora/sdxl/%PROJECT%" ^ 11 | --output_name="%PROJECT%" ^ 12 | --network_args "preset=C:/absolute/path/to/lycoris_presets/bdora_content_style_clip.toml" dora_wd=True ^ 13 | --resolution="1024,1024" ^ 14 | --save_model_as="safetensors" ^ 15 | --network_module="lycoris.kohya" ^ 16 | --max_train_steps=5000 ^ 17 | --save_every_n_steps=1000 ^ 18 | --save_every_n_epochs=5 ^ 19 | --network_dim=%RANK% ^ 20 | --network_alpha=%RANK% ^ 21 | --no_half_vae ^ 22 | --gradient_checkpointing ^ 23 | --persistent_data_loader_workers ^ 24 | --enable_bucket ^ 25 | --random_crop ^ 26 | --bucket_reso_steps=32 ^ 27 | --min_bucket_reso=512 ^ 28 | --xformers ^ 29 | --mixed_precision="fp16" ^ 30 | --caption_extension=".txt" ^ 31 | --lr_scheduler="cosine" ^ 32 | --lr_warmup_steps=0 ^ 33 | --network_train_unet_only ^ 34 | --prior_loss_weight=0 ^ 35 | --optimizer_type="prodigy" ^ 36 | --max_grad_norm=1.0 ^ 37 | --learning_rate=1.0 ^ 38 | --seed=0 ^ 39 | --optimizer_args weight_decay=1e-04 betas=(0.9,0.9999) eps=1e-08 decouple=True use_bias_correction=True safeguard_warmup=True beta3=None 40 | 41 | pause --------------------------------------------------------------------------------