Fix the bug that `joint_attention_kwargs` is not passed to the FLUX's transformer attention processors #9517

HorizonWind2004 · 2024-09-24T14:55:04Z

Issue link is below:

yiyixuxu

thanks for the PR! I left a question

yiyixuxu · 2024-09-24T21:13:30Z

src/diffusers/models/transformers/transformer_flux.py

        encoder_hidden_states: torch.FloatTensor,
        temb: torch.FloatTensor,
        image_rotary_emb=None,
+        joint_attention_kwargs={},


can you explain what additional argument you need to pass down to flux attention processor?

Thank you for your recognition!

In our work, I am trying to integrate box and mask into the FLUX model and implement layout control (similar to what has been done in many works on SD1.4). This requires modifying the attention processor. I believe that the architecture of FLUX and other transformers can also be used to develop better layout control algorithms, so I believe these modifications will contribute to future training-free experiments on FLUX.

yiyixuxu

thanks! let's support this :)

yiyixuxu · 2024-09-25T23:36:37Z

src/diffusers/models/transformers/transformer_flux.py

        encoder_hidden_states: torch.FloatTensor,
        temb: torch.FloatTensor,
        image_rotary_emb=None,
+        joint_attention_kwargs={},


Suggested change

joint_attention_kwargs={},

joint_attention_kwargs=None,

yiyixuxu · 2024-09-25T23:39:56Z

src/diffusers/models/transformers/transformer_flux.py

        norm_hidden_states, gate = self.norm(hidden_states, emb=temb)
        mlp_hidden_states = self.act_mlp(self.proj_mlp(norm_hidden_states))
-
+        joint_attention_kwargs = joint_attention_kwargs if joint_attention_kwargs is not None else {}


should we pass this to attn too?

Yes! I think it will be useful for other trial!

fix a little bug

HorizonWind2004 · 2024-09-26T12:30:13Z

I fixed a bug and now it is okay XD.

HuggingFaceDocBuilderDev · 2024-09-27T11:07:02Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

a-r-r-o-w

thank you, looks good! could you run make style to fix the failing quality tests.

i have a question though: in this comment, you mention that your work uses this feature to modify something in, or pass additional arguments to, the attention processor. it is understandable why this would be useful, but we generally do not add features that are not testable or usable without a public implementation. is your work/method utilizing this change available for testing this? if it will be available in the near future, i think it might be best to postpone merging this PR after that

a-r-r-o-w · 2024-09-27T11:11:26Z

thanks! let's support this :)

oh, i just saw yiyi's comment about being okay with supporting this. in that case, please disregard my question above. let's fix the quality tests and we should be okay to merge

yiyixuxu · 2024-09-27T18:02:22Z

@a-r-r-o-w
I think, based on my undersanding, they just want to use a custom attention processor so they can do pipe.transformer.set_attn_processor(); they do not intend to add that to diffusers;
but I think if we do not allow passing that kwargs, it won't work with a custom attention processor taking additional arguments. o

HorizonWind2004 · 2024-10-06T20:51:23Z

@a-r-r-o-w I think, based on my undersanding, they just want to use a custom attention processor so they can do pipe.transformer.set_attn_processor(); they do not intend to add that to diffusers; but I think if we do not allow passing that kwargs, it won't work with a custom attention processor taking additional arguments. o

Yes, that's it! QWQ

HorizonWind2004 · 2024-10-06T20:53:29Z

@a-r-r-o-w
I sincerely apologize for my late response. I have been busy with another project and forgot to check GitHub. I will proceed with fixing the issues that caused the quality tests to fail right away. TAT

a-r-r-o-w · 2024-10-08T11:55:15Z

Hi @HorizonWind2004, looks good to me! Could you run make style in the diffusers root folder and push the changes? Happy to merge after that once @yiyixuxu approves too

HorizonWind2004 · 2024-10-08T12:08:32Z

Hi @HorizonWind2004, looks good to me! Could you run make style in the diffusers root folder and push the changes? Happy to merge after that once @yiyixuxu approves too

Yes! Now it is okay.

HorizonWind2004 · 2024-10-08T12:10:13Z

@a-r-r-o-w @yiyixuxu
Now I've run make style && make quality and push it! OVO

a-r-r-o-w

thanks!

yiyixuxu

thanks!

… transformer attention processors (huggingface#9517) * Update transformer_flux.py

… transformer attention processors (#9517) * Update transformer_flux.py

HorizonWind2004 added 2 commits September 24, 2024 22:50

Update transformer_flux.py

8a1d136

Update transformer_flux.py

b12ee92

HorizonWind2004 mentioned this pull request Sep 24, 2024

parameters joint_attention_kwargs doesn't be passed to FLUX's transformers model #9516

Closed

Update transformer_flux.py

9509764

yiyixuxu reviewed Sep 24, 2024

View reviewed changes

Merge branch 'main' into main

a89fd51

yiyixuxu reviewed Sep 25, 2024

View reviewed changes

yiyixuxu requested a review from a-r-r-o-w September 25, 2024 23:41

Update transformer_flux.py

5104098

fix a little bug

Merge branch 'main' into main

8b0be8d

a-r-r-o-w reviewed Sep 27, 2024

View reviewed changes

HorizonWind2004 added 2 commits October 8, 2024 19:52

Update transformer_flux.py

9ed5284

Merge branch 'main' into main

c9d624b

fix the quality

b6128a4

a-r-r-o-w approved these changes Oct 8, 2024

View reviewed changes

yiyixuxu approved these changes Oct 8, 2024

View reviewed changes

yiyixuxu merged commit acd6d2c into huggingface:main Oct 8, 2024
15 checks passed

leisuzz pushed a commit to leisuzz/diffusers that referenced this pull request Oct 11, 2024

Fix the bug that joint_attention_kwargs is not passed to the FLUX's…

900a1b5

… transformer attention processors (huggingface#9517) * Update transformer_flux.py

Matrix53 mentioned this pull request Oct 31, 2024

Support pass kwargs to sd3 custom attention processor #9818

Merged

6 tasks

sayakpaul pushed a commit that referenced this pull request Dec 23, 2024

Fix the bug that joint_attention_kwargs is not passed to the FLUX's…

752f5c2

… transformer attention processors (#9517) * Update transformer_flux.py

huanngzh mentioned this pull request Jan 5, 2025

Support pass kwargs to cogvideox custom attention processor #10456

Merged

6 tasks

piercus mentioned this pull request Jun 26, 2025

Flux: pass joint_attention_kwargs when using gradient_checkpointing #11814

Merged

Fix the bug that joint_attention_kwargs is not passed to the FLUX's transformer attention processors #9517

Fix the bug that joint_attention_kwargs is not passed to the FLUX's transformer attention processors #9517

Uh oh!

Conversation

HorizonWind2004 commented Sep 24, 2024

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

yiyixuxu Sep 24, 2024

Choose a reason for hiding this comment

Uh oh!

HorizonWind2004 Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

yiyixuxu Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

yiyixuxu Sep 25, 2024

Choose a reason for hiding this comment

Uh oh!

HorizonWind2004 Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

HorizonWind2004 commented Sep 26, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Sep 27, 2024

Uh oh!

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Uh oh!

a-r-r-o-w commented Sep 27, 2024

Uh oh!

yiyixuxu commented Sep 27, 2024

Uh oh!

HorizonWind2004 commented Oct 6, 2024

Uh oh!

HorizonWind2004 commented Oct 6, 2024

Uh oh!

a-r-r-o-w commented Oct 8, 2024

Uh oh!

HorizonWind2004 commented Oct 8, 2024

Uh oh!

HorizonWind2004 commented Oct 8, 2024

Uh oh!

a-r-r-o-w left a comment

Choose a reason for hiding this comment

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fix the bug that `joint_attention_kwargs` is not passed to the FLUX's transformer attention processors #9517

Fix the bug that `joint_attention_kwargs` is not passed to the FLUX's transformer attention processors #9517