@christopher-beckham
thanks for the PR! would you be able to provide a script that currently fail but will pass with this PR? - that will really help!

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @yiyixuxu

Sure thing. The core of it is the following (taken from the example code here):

import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetPipeline, FluxControlNetModel
from diffusers import AutoencoderKL

if __name__ == '__main__':

    control_image = load_image(
        "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union-alpha/resolve/main/images/canny.jpg"
    ).resize((512,512))
    controlnet_conditioning_scale = 0.5
    control_mode = 0

    width, height = control_image.size

    base_model = 'black-forest-labs/FLUX.1-dev'
    controlnet_model = 'InstantX/FLUX.1-dev-Controlnet-Union'

    controlnet = FluxControlNetModel.from_pretrained(
        controlnet_model, torch_dtype=torch.bfloat16
    )
    vae = AutoencoderKL.from_pretrained(base_model, subfolder="vae", torch_dtype=torch.bfloat16)
    pipe = FluxControlNetPipeline.from_pretrained(
        base_model, vae=vae, controlnet=controlnet, torch_dtype=torch.bfloat16
    )
    print("to cuda...")
    pipe.to("cuda")

    prompt = 'A bohemian-style female travel blogger with sun-kissed skin and messy beach waves.'

    pipe.enable_model_cpu_offload()

    generator = torch.Generator(device="cuda").manual_seed(0)

This should fail pre-fix because we're passing in an int for the control mode:

    image = pipe(
        prompt, 
        num_images_per_prompt=2,
        control_image=control_image,
        control_mode=0,
        width=width,
        height=height,
        controlnet_conditioning_scale=controlnet_conditioning_scale,
        num_inference_steps=1, 
        guidance_scale=3.5,
        generator=generator
    ).images[0]

I don't have time to test the MultiControlNet just now (maybe tonight), but I would strongly presume that if you pass in a list of control modes and any of them are -ve then it should throw a cuda error (since you cannot index into an embedding layer with a -ve index). Example below (though this isn't for multi control net)

    image = pipe(
        prompt, 
        num_images_per_prompt=2,
        control_image=control_image,
        control_mode=-1,
        width=width,
        height=height,
        controlnet_conditioning_scale=controlnet_conditioning_scale,
        num_inference_steps=1, 
        guidance_scale=3.5,
        generator=generator
    ).images[0]

thanks @christopher-beckham
controlnet mode is only needed for union controlnet no? do we need to use MultiControlNet with union?
cc @asomoza here too

controlnet mode is only needed for union controlnet no? do we need to use MultiControlNet with union?

The original one worked like that but you pass a tensor with the modes and an image list, so you don't need the multicontrolnet option.

For what I know of the one made for Flux, it's not the same, it doesn't allow to pass multiple modes and images to the same controlnet so this one would need the multicontrolnet option if you want to use more than one condition to guide the generation.

controlnet mode is only needed for union controlnet no? do we need to use MultiControlNet with union?

yes. Though confusingly, one could do MultiControlNet where the controlnets passed are actually union controlnets. That seems to be what is assumed in the code because when you call the MultiControlNet you're able to pass in a control mode. For instance, if none of the controlnets in the MultiControlNet were of the "union" variety, you would just be doing controlnet_1(image) + ... + controlnet_N(image)

I think maybe is a good time for me to clarify my own understanding of what's going on since I am not 100% sure. Please correct me if I've misunderstood:

CN-Union

ControlNet-Union is specifically a type of controlnet which is trained to handle multiple input conditions due to the extra conditioning tensor which gets passed in when it is trained. At generation time, it basically allows one to pass in arbitrary input conditions as long as you tell the controlnet what it use via control_mode.

This means that you can do the following (written in pseudocode):

block_outs_1 = controlnet_union(control_image_1, mode_1)
pred_noise_1 = transformer(..., block_outs_1)
...
block_outs_2 = controlnet_union(control_image_N, mode_N)
pred_noise_2 = transformer(..., block_outs_2)

In the FluxPipeline, it should therefore be possible to batch this and do pipeline(prompt, control_images, controlnet_modes), i.e. control_images = [control_image_1, ..., control_image_N] and controlnet_modes = [mode_1, ..., mode_N]. (Let's just assume for this example num_images_per_prompt==1, because if it's > 1 then we need to do some extra work in the pipeline so that things still make sense.)

It should also be possible to do pipeline(control_images, mode_int), in which case it is assumed you want to generate from multiple control images but use the same mode. (What my PR proposed initially.)

MultiControlNet

MultiControlNet basically assumes a list of control nets, but -- perhaps confusingly -- these controlnets can also be union controlnets (in which case they each support controlnet_mode's, or they can just be regular uni-conditional controlnets and therefore require that controlnet_mode=None). But essentially the purpose is to let you do the following:

block_samples_1 = controlnet_1(control_image_1, mode_1, scale_1)
...
block_samples_N = controlnet_N(control_image_N, mode_N, scale_N)

block_samples_combined = block_samples_1 + ... + block_samples_N
pred_noise = transformer(..., block_samples_combined)

if any of the controlnet_i (i = 1...N) models are non-union, we expect mode_i==None, otherwise they may also be integers.

(Somewhat coincidentally, block_samples_combined is basically a "union" as well, but it carries a different semantic meaning to what is going on with union controlnets.)

In both cases

In both cases, it seems reasonable to assume that the number of control modes (i.e. length of controlnet_mode) should be == the number of control images. But in the case of MultiControlNet we can allow that the list of modes contain None if need be.

I will test all of these things out later today.

I've been testing the multi controlnet stuff out as well. Here's the code segment:

control_image = load_image(
    "https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union-alpha/resolve/main/images/canny.jpg"
).resize((512,512))

width, height = control_image.size

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_union = FluxControlNetModel.from_pretrained(
    'InstantX/FLUX.1-dev-Controlnet-Union', torch_dtype=torch.bfloat16
)
controlnet_depth = FluxControlNetModel.from_pretrained(
    "Shakker-Labs/FLUX.1-dev-ControlNet-Depth", torch_dtype=torch.bfloat16
)

multinet = FluxMultiControlNetModel([controlnet_union, controlnet_depth])
vae = AutoencoderKL.from_pretrained(base_model, subfolder="vae", torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(
    base_model, vae=vae, controlnet=multinet, torch_dtype=torch.bfloat16
)
#pipe.to("cuda")
pipe.enable_model_cpu_offload()

pipe(
    control_image=[control_image, control_image], 
    control_mode=[0, None], 
    controlnet_conditioning_scale=[1., 1.]
)

So I was looking at this again:

About the 'negative indices', for a MultiControlNet this shouldn't raise any issues -- for instance, if we have a multi controlnet where the first controlnet is union and the second is "regular" (see my above code), if we passed control_mode as [0, None] it would get internally converted to [0, -1] but for the latter controlnet this if statement wouldn't get executed, so it would not raise a device side assert:

However, it just makes me wonder why we have that code in the first place -- this wouldn't be a bug fix, rather just cleaning up code which may not ultimately be necessary. Thoughts?

Lastly, if we try and do this (just for a regular controlnet, not multi):

pipe(control_image=[control_image, control_image], control_mode=0, num_images_per_prompt=1)

We get an error like RuntimeError: shape '[1, 16, 16, 2, 16, 2]' is invalid for input of size 32768 in the call to this self._pack_latents here:

Basically, this code:

is not taking into account the fact that control_image can be a List[PIL.Image.Image] (at least according to the docstring). While it would be easy to hack up a fix for this, it gets rather complicated thinking about all the degrees of freedom here with regard to batching: apparently the prompt can be a list, the control image can be a list, and we also have num_images_per_prompt. Is the actual internal "batch size" meant to be the product of all these things?*

(Apparently it can be any one of torch.Tensor, PIL.Image.Image, np.ndarray, List[torch.Tensor], List[PIL.Image.Image], List[np.ndarray],List[List[torch.Tensor]], List[List[np.ndarray]] or List[List[PIL.Image.Image]])

It seems like a simple fix for now is to just enforce that len(prompts) == len(control_image).

Thoughts @yiyixuxu @asomoza ? Thanks.

thanks for your investigation!!
Yes I think we should start to have some validation, and make it clear how inputs format should be when we want to use multiple images + controlnet + num_images_per_prompt(currently very confusing!)
but since we are not doing any of these checks yet maybe we can do it in a separate PR, and probably across all our controllers

Thanks for your suggestions! I just made a commit incorporating your changes manually (not sure what the best practice is, i.e. whether I should have committed your changes in directly then modified them, but anyways the changes are made).

wrt to my changes, for multi-controlnet, the control mode must be passed as a list, even if it's a singleton. I didn't want to automagically convert it from int to a singleton list (like in the single controlnet case) since it seems for fitting to be explicit about it in the multi case.

Yes I think we should start to have some validation, and make it clear how inputs format should be when we want to use multiple images + controlnet + num_images_per_prompt(currently very confusing!) but since we are not doing any of these checks yet maybe we can do it in a separate PR, and probably across all our controllers

Yeah maybe for another PR we can explore this. When I looked at the SDXL controlnet class, it basically looks like the prompt is the "ground truth" for how batching should be handled -- that is, the control images variable (whether it be a List[PIL.Image] or torch.Tensor or np.ndarray) is checked to make sure its length (batch size) should be equivalent to the number of prompts. After that has been satisfied then you can then handle num_images_per_prompt to double/triple/whatever the effective batch size. A lot of that logic can be ported over here.

Lastly, I have some unit tests I used here:

https://github.com/christopher-beckham/diffusers-tests/blob/master/flux_controlnet/test_multicontrolnet.py

hi @christopher-beckham
I continued this PR on #9507 (it is branched out from your PR so all your commits are there and you're a co-authtor there already)
would be really appreciate if you can give a review!!

I made a slow test with 4 test cases here #9507 (comment)

I think we do not need to consider something like multinet = FluxMultiControlNetModel([controlnet_union, controlnet_depth]) because:

1. it is not very meaningful (union controlnet already has a mode for depth condition)
2. it is currently no checkpoint to make it possible to use it this way, the only regular controlnet checkpoint we have is canny, which does not have the single blocks so it cannot be used together with union at all

Thanks @yiyixuxu I'm gonna check it out shortly :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

closing this since PR is merged!