ComfyUI Hand Correction Workflow - HandRefiner
ฝัง
- เผยแพร่เมื่อ 22 ก.ค. 2024
- Comfyui work flow w/ HandRefiner, easy and convenient hand correction or hand fix.
HandRefiner Github: github.com/wenquanlu/HandRefiner
Controlnet inpaint depth hand model: huggingface.co/hr16/ControlNe...
comfyui_controlnet_aux: github.com/Fannovel16/comfyui...
Mesh Graphormer (FYI): github.com/microsoft/MeshGrap...
Workflow: drive.google.com/file/d/11Zde...
00:00 Introduction
00:34 Workflow
02:01 Examples
#comfyui #stablediffusion #HandRefiner #wrokflow #controlnet #inpaint
#aihandfix - ภาพยนตร์และแอนิเมชัน
great video - any advice on making a workflow to work with img2img ? I guess I miss some important step as the meshgraphormer just returns black instead of Zdepth but works fine with txt2img..
Thanks, just simply replace the first "preview bridge" with "load image" and mute the first "vae decode", that's all. I also found black sometimes if the hand shape distorted too much, and it also doesn't support well for anime picture.
@@JanRTstudio I tried your method for img2img but did not work. Could you share a workflow download for img2img including bulk multiple image fix hands?
@@bestof467 Sure, let me test and I will upload later
@@JanRTstudio Was this solved?
Getting runtime error "mat1 and mat2" shapes cannot be multiplied. Any idea as to what could be causing this?
Wonderful explainer video. This method works like a charm
Thank you!
Amazing node, BTW is there a way to fix "2 right hands" issue? I tried this node, fixing finger is fine, but it seems not able to detect 2 right hands.
Yes, both Handrefiner and MeshGraphormer got some limitations, depending on their training dataset, might need to fix it manually for this case. The way I am thinking is to paste a correct left hand depth pic to the image generated by meshgraphormer, and then load it to Apply Controlnet node.
Nice. Could you use Automatic 1111 to train a lora for monkeys hands using this as a base model? By "can you", I mean is it possible do you think?
I believe yes, A1111 not sure, but you can find training using python here: github.com/microsoft/MeshGraphormer/blob/main/docs/EXP.md
@@JanRTstudio - thanks
no problem! @@jeffg4686
@@JanRTstudio - Nice. I might have to take a trip to the zoo, or even do some gens with dalle or something.
Sounds good 😄@@jeffg4686
Great video! Is there a reason why you didn’t use the mask generated by the graphormer node for your inpainting - looks like you could cut out a couple of nodes
Yes, that's correct, I started with BBox & Sam detector together so missed that inpaint mask, yes it will save a lot of nodes if only using BBox. Thanks for pointing out.
@@JanRTstudio You can save yourself the BBox Detector completely, because the Graphormer itself detects and thus also generates an inpaint mask, you pass the mask to "Set Latent Noise Mask" and then it goes into the KSampler.
@@SaschaFuchs Yes, you are right, can save the detector part. Thanks!
Its unfortunatelly that crossing fingers is interpreted as single hand, on many angles, I wish they could fix this up. A traning should be made, probably, and some sign to detect when one hand is under the other, because it deforms really bad as if there is only one hand on person! Also gloves have been a problem to me. It can be better yet.
Exactly, they mentioned this limitation. Works well in general poses, but still need to fix manually when crossing, overlapping, partial hands, etc.
ありがとう！
no problem!😀
I got error like this
Error occurred when executing KSampler:
mat1 and mat2 shapes cannot be multiplied (154x2048 and 768x320)
File "D:\000AI\00ComfyUI\ComfyUI\execution.py", line 154, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI\execution.py", line 84, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI\execution.py", line 77, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI
odes.py", line 1299, in sample
return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI
odes.py", line 1269, in common_ksampler
samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 22, in informative_sample
raise e
File "D:\000AI\00ComfyUI\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample
return original_sample(*args, **kwargs) # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 218, in motion_sample
return orig_comfy_sample(model, noise, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI\comfy\sample.py", line 101, in sample
samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI\comfy\samplers.py", line 716, in sample
return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI\comfy\samplers.py", line 622, in sample
samples = sampler.sample(model_wrap, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\ComfyUI\comfy\samplers.py", line 561, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\000AI\00ComfyUI\python_embeded\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
Have you added AnimateDiff in? also check if you mix used checkpoint and lora, like 2.1 ckpt w/ 1.5 lora or SDXL
@@JanRTstudio I used your workflow only, but used SDXL model, and also disabled lora.
I see, currently the controlnet model (control_sd15_inpaint_depth_hand_fp16.safetensors) is 1.5, so mismatch here. Hope they will update SDXL model soon. @@RahulGupta1981
is there an sdxl version?
Not yet, because the controlnet model was trained on 1.5, but I did try XL + 1.5 and it works. I will update it later.
@@JanRTstudiowhat do you mean xl + sd15
@@JanRTstudio what do you mean 1.5+xl?
SDXL for generation, SD1.5 for hand fix @@___x__x_r___xa__x_____f______
@@JanRTstudio yup makes sense
It's a nice video but hard to follow when you are a beginner in comfyui. Maybe start from scratch or loading the default template and build-up with that ?
Good idea, I will consider it like adding as part 2. Thanks for the suggestions.
this does not work on resting hands only raised hands
This workflow CREATES and image, but in most of the cases, you'd like to load an EXISTING image to refine it 😐
yeah similar suggestion in other comments, I will test and upload a Img2Img workflow
Anyone know why i'm getting an error code returned when running the mesh graphormer? the error reads as follows:
WinError 206] The filename or extension is too long: 'C:\\Users\\Christian\\OneDrive\\Documents\\ComfyUI_windows_portable\\ComfyUI\\custom_nodes\\comfyui_controlnet_aux\\ckpts\\hr16/ControlNet-HandRefiner-pruned\\cache\\models--hr16--ControlNet-HandRefiner-pruned\\snapshots\\f0917f0595ecb7f6435f49e4b2b28f8dd68ab0cb'
Can you try to directly download the files "graphormer_hand_state_dict.bin" and "hrnetv2_w64_imagenet_pretrained.pth" from huggingface link in the above Description and put them into your "ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts\hr16\ControlNet-HandRefiner-pruned" folder?
@@JanRTstudio absolute hero, this has worked, thank you!
Great, you are welcome!@@DYYGITAL
You can also solve it by making the pathname shorter. For example, Reducing parts like "ComfyUI_windows_portable" to ComfyUI or placing ComfyUI straight into the C root folder because the problem is literally that the total pathname is too long.
I keep getting this error from Within Comfy UI
Error occurred when executing MeshGraphormer-DepthMapPreprocessor:
[Errno 2] No such file or directory: 'C:\\Users\\AShea\\Downloads\\ComfyUI_windows_portable\\ComfyUI\\custom_nodes\\comfyui_controlnet_aux\\ckpts\\hr16/ControlNet-HandRefiner-pruned\\cache\\models--hr16--ControlNet-HandRefiner-pruned\\blobs\\41ed675bcd1f4f4b62a49bad64901f08f8b67ed744b715da87738f926dae685c.incomplete'
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\ComfyUI\execution.py", line 154, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\ComfyUI\execution.py", line 84, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\ComfyUI\execution.py", line 77, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux
ode_wrappers\mesh_graphormer.py", line 83, in execute
from controlnet_aux.mesh_graphormer import MeshGraphormerDetector
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\__init__.py", line 5, in
from controlnet_aux.mesh_graphormer.pipeline import MeshGraphormerMediapipe, args
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\pipeline.py", line 54, in
hrnet_checkpoint=custom_hf_download("hr16/ControlNet-HandRefiner-pruned", 'hrnetv2_w64_imagenet_pretrained.pth')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\util.py", line 257, in custom_hf_download
model_path = hf_hub_download(repo_id=pretrained_model_or_path,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\utils\_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\file_download.py", line 1449, in hf_hub_download
with temp_file_manager() as temp_file:
File "contextlib.py", line 137, in __enter__
File "C:\Users\AShea\Downloads\ComfyUI_windows_portable\python_embeded\Lib\site-packages\huggingface_hub\file_download.py", line 1433, in _resumable_file_manager
with open(incomplete_path, "ab") as f:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Seems files not downloaded completely, can you directly download the files "graphormer_hand_state_dict.bin" and "hrnetv2_w64_imagenet_pretrained.pth" from huggingface link in the above Description and put them into your "ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts\hr16\ControlNet-HandRefiner-pruned" folder?
i finally figured that out late last night, thanks for your help@@JanRTstudio
@@JanRTstudio omg thank you
You are welcome! @@borkor458
I keep getting the below error despite having installed YACS already :
No module named 'yacs.config'
File "C:\Users\xpare\Desktop\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xpare\Desktop\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xpare\Desktop\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xpare\Desktop\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux
ode_wrappers\mesh_graphormer.py", line 66, in execute
from controlnet_aux.mesh_graphormer import MeshGraphormerDetector
File "C:\Users\xpare\Desktop\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\__init__.py", line 5, in
from controlnet_aux.mesh_graphormer.pipeline import MeshGraphormerMediapipe, args
File "C:\Users\xpare\Desktop\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux\mesh_graphormer\pipeline.py", line 12, in
from custom_mesh_graphormer.modeling.hrnet.config import config as hrnet_config
File "C:\Users\xpare\Desktop\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_mesh_graphormer\modeling\hrnet\config\__init__.py", line 7, in
from .default import _C as config
File "C:\Users\xpare\Desktop\ComfyUI_windows_portable\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_mesh_graphormer\modeling\hrnet\config\default.py", line 15, in
from yacs.config import CfgNode as CN