├── .gitattributes ├── .github └── workflows │ └── publish.yml ├── .gitignore ├── README.md ├── README_EN.md ├── __init__.py ├── nodes ├── image_generation.py ├── image_understanding.py └── model_loader.py ├── pyproject.toml ├── requirements.txt └── workflow └── ComfyUI Janus-Pro-workflow.png /.gitattributes: -------------------------------------------------------------------------------- 1 | # Auto detect text files and perform LF normalization 2 | * text=auto 3 | -------------------------------------------------------------------------------- /.github/workflows/publish.yml: -------------------------------------------------------------------------------- 1 | name: Publish to Comfy registry 2 | on: 3 | workflow_dispatch: 4 | push: 5 | branches: 6 | - main 7 | - master 8 | paths: 9 | - "pyproject.toml" 10 | 11 | permissions: 12 | issues: write 13 | 14 | jobs: 15 | publish-node: 16 | name: Publish Custom Node to registry 17 | runs-on: ubuntu-latest 18 | if: ${{ github.repository_owner == 'CY-CHENYUE' }} 19 | steps: 20 | - name: Check out code 21 | uses: actions/checkout@v4 22 | with: 23 | submodules: true 24 | - name: Publish Custom Node 25 | uses: Comfy-Org/publish-node-action@v1 26 | with: 27 | ## Add your own personal access token to your Github Repository secrets and reference it here. 28 | personal_access_token: ${{ secrets.REGISTRY_ACCESS_TOKEN }} 29 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Python 2 | __pycache__/ 3 | *.py[cod] 4 | *$py.class 5 | 6 | # Distribution / packaging 7 | dist/ 8 | build/ 9 | *.egg-info/ 10 | 11 | # Virtual Environment 12 | venv/ 13 | env/ 14 | .env/ 15 | 16 | # IDE 17 | .vscode/ 18 | .idea/ 19 | *.swp 20 | *.swo 21 | 22 | # OS 23 | .DS_Store 24 | Thumbs.db -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # ComfyUI-Janus-Pro 2 | 3 | [English](README_EN.md) | 简体中文 4 | 5 | ComfyUI 的 Janus-Pro 节点，一个统一的多模态理解和生成框架。 6 | 7 | ![alt text]() 8 | 9 | 10 | ## 安装方法 11 | 12 | ### 方法一：通过 ComfyUI Manager 安装（推荐） 13 | 1. 安装 [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager) 14 | 2. 在管理器中搜索 "Janus-Pro" 15 | 3. 点击安装 16 | 17 | ### 方法二：手动安装 18 | 1. 将此仓库克隆到你的 ComfyUI 的 custom_nodes 文件夹中： 19 | ```bash 20 | cd ComfyUI/custom_nodes 21 | git clone https://github.com/CY-CHENYUE/ComfyUI-Janus-Pro 22 | ``` 23 | 24 | 2. 安装所需依赖： 25 | 26 | Windows系统： 27 | ```bash 28 | # 如果你使用ComfyUI便携版 29 | cd ComfyUI-Janus-Pro 30 | ..\..\..\python_embeded\python.exe -m pip install -r requirements.txt 31 | 32 | # 如果你使用自己的Python环境 33 | cd ComfyUI-Janus-Pro 34 | path\to\your\python.exe -m pip install -r requirements.txt 35 | ``` 36 | 37 | Linux/Mac系统： 38 | ```bash 39 | # 使用ComfyUI的Python环境 40 | cd ComfyUI-Janus-Pro 41 | ../../python_embeded/bin/python -m pip install -r requirements.txt 42 | 43 | # 或者使用你的环境 44 | cd ComfyUI-Janus-Pro 45 | python -m pip install -r requirements.txt 46 | ``` 47 | 48 | 注意：如果你遇到安装问题： 49 | - 确保已安装 git 50 | - 尝试更新 pip：`python -m pip install --upgrade pip` 51 | - 如果你使用代理，确保 git 可以访问 GitHub 52 | - 确保使用的是与 ComfyUI 相同的 Python 环境 53 | 54 | 55 | ## 模型下载 56 | 57 | 将模型文件放在 `ComfyUI/models/Janus-Pro` 文件夹中： 58 | 1. 在你的 ComfyUI 的 models 目录下创建 `Janus-Pro` 文件夹 59 | 2. 从 Hugging Face 下载模型： 60 | - [Janus-Pro-1B](https://huggingface.co/deepseek-ai/Janus-Pro-1B) 61 | - [Janus-Pro-7B](https://huggingface.co/deepseek-ai/Janus-Pro-7B) 62 | 3. 将模型解压到各自的文件夹中： 63 | ``` 64 | ComfyUI/models/Janus-Pro/Janus-Pro-1B/ 65 | ComfyUI/models/Janus-Pro/Janus-Pro-7B/ 66 | ``` 67 | ## Star History 68 | 69 | [![Star History Chart](https://api.star-history.com/svg?repos=CY-CHENYUE/ComfyUI-Janus-Pro&type=Date)](https://star-history.com/#CY-CHENYUE/ComfyUI-Janus-Pro&Date) 70 | 71 | ## Contact Me 72 | 73 | - X (Twitter): [@cychenyue](https://x.com/cychenyue) 74 | - TikTok: [@cychenyue](https://www.tiktok.com/@cychenyue) 75 | - YouTube: [@CY-CHENYUE](https://www.youtube.com/@CY-CHENYUE) 76 | - BiliBili: [@CY-CHENYUE](https://space.bilibili.com/402808950) 77 | - 小红书: [@CY-CHENYUE](https://www.xiaohongshu.com/user/profile/6360e61f000000001f01bda0) 78 | 79 | --- 80 |
81 | 如果这个项目对你有帮助，请给它一个 Star ⭐️ 82 |
-------------------------------------------------------------------------------- /README_EN.md: -------------------------------------------------------------------------------- 1 | # ComfyUI-Janus-Pro 2 | 3 | English | [简体中文](README.md) 4 | 5 | ComfyUI nodes for Janus-Pro, a unified multimodal understanding and generation framework. 6 | 7 | ![alt text]() 8 | 9 | ## Installation 10 | 11 | ### Method 1: Install via ComfyUI Manager (Recommended) 12 | 1. Install [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager) 13 | 2. Search for "Janus-Pro" in the Manager 14 | 3. Click install 15 | 16 | ### Method 2: Manual Installation 17 | 1. Clone this repository to your ComfyUI's custom_nodes folder: 18 | ```bash 19 | cd ComfyUI/custom_nodes 20 | git clone https://github.com/CY-CHENYUE/ComfyUI-Janus-Pro 21 | ``` 22 | 23 | 2. Install the required dependencies: 24 | 25 | For Windows: 26 | ```bash 27 | # If you're using ComfyUI portable version 28 | cd ComfyUI-Janus-Pro 29 | ..\..\..\python_embeded\python.exe -m pip install -r requirements.txt 30 | 31 | # If you're using your own Python environment 32 | cd ComfyUI-Janus-Pro 33 | path\to\your\python.exe -m pip install -r requirements.txt 34 | ``` 35 | 36 | For Linux/Mac: 37 | ```bash 38 | # Using ComfyUI's Python environment 39 | cd ComfyUI-Janus-Pro 40 | ../../python_embeded/bin/python -m pip install -r requirements.txt 41 | 42 | # Or using your own environment 43 | cd ComfyUI-Janus-Pro 44 | python -m pip install -r requirements.txt 45 | ``` 46 | 47 | Note: If you encounter any installation issues: 48 | - Make sure you have git installed 49 | - Try updating pip: `python -m pip install --upgrade pip` 50 | - If you're behind a proxy, make sure git can access GitHub 51 | - Make sure you're using the same Python environment as ComfyUI 52 | 53 | 54 | ## Model Download 55 | 56 | Place your models in the `ComfyUI/models/Janus-Pro` folder: 57 | 1. Create a `Janus-Pro` folder in your ComfyUI's models directory 58 | 2. Download the models from Hugging Face: 59 | - [Janus-Pro-1B](https://huggingface.co/deepseek-ai/Janus-Pro-1B) 60 | - [Janus-Pro-7B](https://huggingface.co/deepseek-ai/Janus-Pro-7B) 61 | 3. Extract the models into their respective folders: 62 | ``` 63 | ComfyUI/models/Janus-Pro/Janus-Pro-1B/ 64 | ComfyUI/models/Janus-Pro/Janus-Pro-7B/ 65 | ``` 66 | 67 | ## Star History 68 | 69 | [![Star History Chart](https://api.star-history.com/svg?repos=CY-CHENYUE/ComfyUI-Janus-Pro&type=Date)](https://star-history.com/#CY-CHENYUE/ComfyUI-Janus-Pro&Date) 70 | 71 | ## Contact Me 72 | 73 | - X (Twitter): [@cychenyue](https://x.com/cychenyue) 74 | - TikTok: [@cychenyue](https://www.tiktok.com/@cychenyue) 75 | - YouTube: [@CY-CHENYUE](https://www.youtube.com/@CY-CHENYUE) 76 | - BiliBili: [@CY-CHENYUE](https://space.bilibili.com/402808950) 77 | - Xiaohongshu: [@CY-CHENYUE](https://www.xiaohongshu.com/user/profile/6360e61f000000001f01bda0) 78 | 79 | --- 80 |
81 | If you find this project helpful, please give it a Star ⭐️ 82 |
-------------------------------------------------------------------------------- /__init__.py: -------------------------------------------------------------------------------- 1 | from .nodes.model_loader import JanusModelLoader 2 | from .nodes.image_understanding import JanusImageUnderstanding 3 | from .nodes.image_generation import JanusImageGeneration 4 | 5 | NODE_CLASS_MAPPINGS = { 6 | "JanusModelLoader": JanusModelLoader, 7 | "JanusImageUnderstanding": JanusImageUnderstanding, 8 | "JanusImageGeneration": JanusImageGeneration, 9 | } 10 | 11 | NODE_DISPLAY_NAME_MAPPINGS = { 12 | "JanusModelLoader": "Janus Model Loader", 13 | "JanusImageUnderstanding": "Janus Image Understanding", 14 | "JanusImageGeneration": "Janus Image Generation", 15 | } 16 | 17 | __all__ = ['NODE_CLASS_MAPPINGS', 'NODE_DISPLAY_NAME_MAPPINGS'] -------------------------------------------------------------------------------- /nodes/image_generation.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import numpy as np 3 | from PIL import Image 4 | 5 | class JanusImageGeneration: 6 | @classmethod 7 | def INPUT_TYPES(s): 8 | return { 9 | "required": { 10 | "model": ("JANUS_MODEL",), 11 | "processor": ("JANUS_PROCESSOR",), 12 | "prompt": ("STRING", { 13 | "multiline": True, 14 | "default": "A beautiful photo of" 15 | }), 16 | "seed": ("INT", { 17 | "default": 666666666666666, 18 | "min": 0, 19 | "max": 0xffffffffffffffff 20 | }), 21 | "batch_size": ("INT", { 22 | "default": 1, 23 | "min": 1, 24 | "max": 16 25 | }), 26 | "cfg_weight": ("FLOAT", { 27 | "default": 5.0, 28 | "min": 1.0, 29 | "max": 10.0, 30 | "step": 0.5 31 | }), 32 | "temperature": ("FLOAT", { 33 | "default": 1.0, 34 | "min": 0.1, 35 | "max": 2.0, 36 | "step": 0.1 37 | }), 38 | "top_p": ("FLOAT", { 39 | "default": 0.95, 40 | "min": 0.0, 41 | "max": 1.0 42 | }), 43 | }, 44 | } 45 | 46 | RETURN_TYPES = ("IMAGE",) 47 | RETURN_NAMES = ("images",) 48 | FUNCTION = "generate_images" 49 | CATEGORY = "Janus-Pro" 50 | 51 | def generate_images(self, model, processor, prompt, seed, batch_size=1, temperature=1.0, cfg_weight=5.0, top_p=0.95): 52 | try: 53 | from janus.models import MultiModalityCausalLM 54 | except ImportError: 55 | raise ImportError("Please install Janus using 'pip install -r requirements.txt'") 56 | 57 | # 设置随机种子 58 | torch.manual_seed(seed) 59 | torch.cuda.manual_seed(seed) 60 | 61 | # 图像参数设置 62 | image_token_num = 576 # 24x24 patches 63 | img_size = 384 # 输出图像大小 64 | patch_size = 16 # 每个patch的大小 65 | parallel_size = batch_size 66 | 67 | # 准备对话格式 68 | conversation = [ 69 | { 70 | "role": "<|User|>", 71 | "content": prompt, 72 | }, 73 | {"role": "<|Assistant|>", "content": ""}, 74 | ] 75 | 76 | # 准备输入 77 | sft_format = processor.apply_sft_template_for_multi_turn_prompts( 78 | conversations=conversation, 79 | sft_format=processor.sft_format, 80 | system_prompt="", 81 | ) 82 | prompt = sft_format + processor.image_start_tag 83 | 84 | # 编码输入文本 85 | input_ids = processor.tokenizer.encode(prompt) 86 | input_ids = torch.LongTensor(input_ids) 87 | 88 | # 准备条件和无条件输入 89 | tokens = torch.zeros((parallel_size*2, len(input_ids)), dtype=torch.int).cuda() 90 | for i in range(parallel_size*2): 91 | tokens[i, :] = input_ids 92 | if i % 2 != 0: # 无条件输入 93 | tokens[i, 1:-1] = processor.pad_id 94 | 95 | # 获取文本嵌入 96 | inputs_embeds = model.language_model.get_input_embeddings()(tokens) 97 | 98 | # 生成图像tokens 99 | generated_tokens = torch.zeros((parallel_size, image_token_num), dtype=torch.int).cuda() 100 | outputs = None 101 | 102 | # 自回归生成 103 | for i in range(image_token_num): 104 | outputs = model.language_model.model( 105 | inputs_embeds=inputs_embeds, 106 | use_cache=True, 107 | past_key_values=outputs.past_key_values if i != 0 else None 108 | ) 109 | hidden_states = outputs.last_hidden_state 110 | 111 | # 获取logits并应用CFG 112 | logits = model.gen_head(hidden_states[:, -1, :]) 113 | logit_cond = logits[0::2, :] 114 | logit_uncond = logits[1::2, :] 115 | 116 | logits = logit_uncond + cfg_weight * (logit_cond-logit_uncond) 117 | probs = torch.softmax(logits / temperature, dim=-1) 118 | 119 | # 采样下一个token 120 | next_token = torch.multinomial(probs, num_samples=1) 121 | generated_tokens[:, i] = next_token.squeeze(dim=-1) 122 | 123 | # 准备下一步的输入 124 | next_token = torch.cat([next_token.unsqueeze(dim=1), next_token.unsqueeze(dim=1)], dim=1).view(-1) 125 | img_embeds = model.prepare_gen_img_embeds(next_token) 126 | inputs_embeds = img_embeds.unsqueeze(dim=1) 127 | 128 | # 解码生成的tokens为图像 129 | dec = model.gen_vision_model.decode_code( 130 | generated_tokens.to(dtype=torch.int), 131 | shape=[parallel_size, 8, img_size//patch_size, img_size//patch_size] 132 | ) 133 | 134 | # 转换为numpy进行处理 135 | dec = dec.to(torch.float32).cpu().numpy() 136 | 137 | # 确保是BCHW格式 138 | if dec.shape[1] != 3: 139 | dec = np.repeat(dec, 3, axis=1) 140 | 141 | # 从[-1,1]转换到[0,1] 142 | dec = (dec + 1) / 2 143 | 144 | # 确保值范围在[0,1]之间 145 | dec = np.clip(dec, 0, 1) 146 | 147 | # 转换为ComfyUI需要的格式 [B,C,H,W] -> [B,H,W,C] 148 | dec = np.transpose(dec, (0, 2, 3, 1)) 149 | 150 | # 转换为tensor 151 | images = torch.from_numpy(dec).float() 152 | 153 | # 打印详细的形状信息 154 | # print(f"Initial dec shape: {dec.shape}") 155 | # print(f"Final tensor: shape={images.shape}, dtype={images.dtype}, range=[{images.min():.3f}, {images.max():.3f}]") 156 | 157 | # 确保格式正确 158 | assert images.ndim == 4 and images.shape[-1] == 3, f"Unexpected shape: {images.shape}" 159 | 160 | return (images,) 161 | 162 | @classmethod 163 | def IS_CHANGED(cls, seed, **kwargs): 164 | return seed -------------------------------------------------------------------------------- /nodes/image_understanding.py: -------------------------------------------------------------------------------- 1 | import torch 2 | import numpy as np 3 | from PIL import Image 4 | 5 | class JanusImageUnderstanding: 6 | @classmethod 7 | def INPUT_TYPES(s): 8 | return { 9 | "required": { 10 | "model": ("JANUS_MODEL",), 11 | "processor": ("JANUS_PROCESSOR",), 12 | "image": ("IMAGE",), 13 | "question": ("STRING", { 14 | "multiline": True, 15 | "default": "Describe this image in detail." 16 | }), 17 | "seed": ("INT", { 18 | "default": 666666666666666, 19 | "min": 0, 20 | "max": 0xffffffffffffffff 21 | }), 22 | "temperature": ("FLOAT", { 23 | "default": 0.1, 24 | "min": 0.0, 25 | "max": 1.0 26 | }), 27 | "top_p": ("FLOAT", { 28 | "default": 0.95, 29 | "min": 0.0, 30 | "max": 1.0 31 | }), 32 | "max_new_tokens": ("INT", { 33 | "default": 512, 34 | "min": 1, 35 | "max": 2048 36 | }), 37 | }, 38 | } 39 | 40 | RETURN_TYPES = ("STRING",) 41 | RETURN_NAMES = ("text",) 42 | FUNCTION = "analyze_image" 43 | CATEGORY = "Janus-Pro" 44 | 45 | def analyze_image(self, model, processor, image, question, seed, temperature, top_p, max_new_tokens): 46 | try: 47 | from janus.models import MultiModalityCausalLM 48 | except ImportError: 49 | raise ImportError("Please install Janus using 'pip install -r requirements.txt'") 50 | 51 | # 设置随机种子 52 | torch.manual_seed(seed) 53 | torch.cuda.manual_seed(seed) 54 | 55 | # 打印初始图像信息 56 | # print(f"Initial image shape: {image.shape}") 57 | # print(f"Initial image type: {image.dtype}") 58 | # print(f"Initial image device: {image.device}") 59 | 60 | # ComfyUI中的图像格式是 BCHW (Batch, Channel, Height, Width) 61 | if len(image.shape) == 4: # BCHW format 62 | if image.shape[0] == 1: 63 | image = image.squeeze(0) # 移除batch维度，现在是 [H, W, C] 64 | 65 | # print(f"After squeeze shape: {image.shape}") 66 | 67 | # 确保值范围在[0,1]之间并转换为uint8 68 | image = (torch.clamp(image, 0, 1) * 255).cpu().numpy().astype(np.uint8) 69 | 70 | # print(f"Final numpy shape: {image.shape}") 71 | # print(f"Final numpy dtype: {image.dtype}") 72 | # print(f"Final value range: [{image.min()}, {image.max()}]") 73 | 74 | # 转换为PIL图像 75 | pil_image = Image.fromarray(image, mode='RGB') 76 | 77 | conversation = [ 78 | { 79 | "role": "<|User|>", 80 | "content": f"\n{question}", 81 | "images": [pil_image], 82 | }, 83 | {"role": "<|Assistant|>", "content": ""}, 84 | ] 85 | 86 | prepare_inputs = processor( 87 | conversations=conversation, 88 | images=[pil_image], 89 | force_batchify=True 90 | ).to(model.device) 91 | 92 | inputs_embeds = model.prepare_inputs_embeds(**prepare_inputs) 93 | 94 | outputs = model.language_model.generate( 95 | inputs_embeds=inputs_embeds, 96 | attention_mask=prepare_inputs.attention_mask, 97 | pad_token_id=processor.tokenizer.eos_token_id, 98 | bos_token_id=processor.tokenizer.bos_token_id, 99 | eos_token_id=processor.tokenizer.eos_token_id, 100 | max_new_tokens=max_new_tokens, 101 | do_sample=True, 102 | temperature=temperature, 103 | top_p=top_p, 104 | use_cache=True, 105 | ) 106 | 107 | answer = processor.tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True) 108 | 109 | return (answer,) 110 | 111 | @classmethod 112 | def IS_CHANGED(cls, seed, **kwargs): 113 | return seed -------------------------------------------------------------------------------- /nodes/model_loader.py: -------------------------------------------------------------------------------- 1 | import os 2 | 3 | class JanusModelLoader: 4 | def __init__(self): 5 | pass 6 | 7 | @classmethod 8 | def INPUT_TYPES(s): 9 | return { 10 | "required": { 11 | "model_name": (["deepseek-ai/Janus-Pro-1B", "deepseek-ai/Janus-Pro-7B"],), 12 | }, 13 | } 14 | 15 | RETURN_TYPES = ("JANUS_MODEL", "JANUS_PROCESSOR") 16 | RETURN_NAMES = ("model", "processor") 17 | FUNCTION = "load_model" 18 | CATEGORY = "Janus-Pro" 19 | 20 | def load_model(self, model_name): 21 | try: 22 | from janus.models import MultiModalityCausalLM, VLChatProcessor 23 | from transformers import AutoModelForCausalLM 24 | import torch 25 | except ImportError: 26 | raise ImportError("Please install Janus using 'pip install -r requirements.txt'") 27 | 28 | device = "cuda" if torch.cuda.is_available() else "cpu" 29 | 30 | try: 31 | dtype = torch.bfloat16 32 | torch.zeros(1, dtype=dtype, device=device) 33 | except RuntimeError: 34 | dtype = torch.float16 35 | 36 | # 获取ComfyUI根目录 37 | comfy_path = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(__file__)))) 38 | # 构建模型路径 39 | model_dir = os.path.join(comfy_path, 40 | "models", 41 | "Janus-Pro", 42 | os.path.basename(model_name)) 43 | if not os.path.exists(model_dir): 44 | raise ValueError(f"Local model not found at {model_dir}. Please download the model and place it in the ComfyUI/models/Janus-Pro folder.") 45 | 46 | vl_chat_processor = VLChatProcessor.from_pretrained(model_dir) 47 | 48 | vl_gpt = AutoModelForCausalLM.from_pretrained( 49 | model_dir, 50 | trust_remote_code=True 51 | ) 52 | 53 | vl_gpt = vl_gpt.to(dtype).to(device).eval() 54 | 55 | return (vl_gpt, vl_chat_processor) -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [project] 2 | name = "janus-pro" 3 | description = "ComfyUI nodes for Janus-Pro, a unified multimodal understanding and generation framework." 4 | version = "1.0.0" 5 | license = {file = "LICENSE"} 6 | dependencies = ["git+https://github.com/deepseek-ai/Janus.git"] 7 | 8 | [project.urls] 9 | Repository = "https://github.com/CY-CHENYUE/ComfyUI-Janus-Pro" 10 | # Used by Comfy Registry https://comfyregistry.org 11 | 12 | [tool.comfy] 13 | PublisherId = "cychenyue" 14 | DisplayName = "ComfyUI-Janus-Pro" 15 | Icon = "" 16 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | git+https://github.com/deepseek-ai/Janus.git -------------------------------------------------------------------------------- /workflow/ComfyUI Janus-Pro-workflow.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/CY-CHENYUE/ComfyUI-Janus-Pro/4400129e5c33664ae6e927162a39ba4116f44b8b/workflow/ComfyUI Janus-Pro-workflow.png --------------------------------------------------------------------------------