KEMBAR78
Merge modular diffusers with main by a-r-r-o-w · Pull Request #11893 · huggingface/diffusers · GitHub
Skip to content

Conversation

@a-r-r-o-w
Copy link
Contributor

Merges main with modular-diffusers to resolve conflicts with First Block Cache.

I used the following code to verify that the skip layer changes work as expected.

import json
import torch
from diffusers import AutoGuidance, SkipLayerGuidance, ClassifierFreeGuidance, SmoothedEnergyGuidance, SmoothedEnergyGuidanceConfig, AdaptiveProjectedGuidance, PerturbedAttentionGuidance, ClassifierFreeZeroStarGuidance, TangentialClassifierFreeGuidance, LayerSkipConfig
from diffusers.modular_pipelines import SequentialPipelineBlocks, ComponentSpec
from diffusers.modular_pipelines.stable_diffusion_xl import TEXT2IMAGE_BLOCKS
from diffusers.utils.logging import set_verbosity_debug

set_verbosity_debug()

blocks = SequentialPipelineBlocks.from_blocks_dict(TEXT2IMAGE_BLOCKS)

modular_repo_id = "YiYiXu/modular-loader-t2i"
pipeline = blocks.init_pipeline(modular_repo_id)
pipeline.load_default_components(torch_dtype=torch.float16)
pipeline.to("cuda")

for guider_cls in [
    AutoGuidance,
    SkipLayerGuidance,
    ClassifierFreeGuidance,
    SmoothedEnergyGuidance,
    AdaptiveProjectedGuidance,
    PerturbedAttentionGuidance,
    ClassifierFreeZeroStarGuidance,
    TangentialClassifierFreeGuidance
]:
    print(f"Testing {guider_cls.__name__}...")
    
    kwargs = {"guidance_scale": 5.0}
    if guider_cls is AutoGuidance:
        kwargs.update({"auto_guidance_config": LayerSkipConfig(indices=[9, 10], fqn="mid_block.attentions.0.transformer_blocks", skip_attention=True, skip_ff=True, dropout=0.75)})
        kwargs.update({"stop": 0.6})
    elif guider_cls is SkipLayerGuidance:
        kwargs.update({"skip_layer_config": LayerSkipConfig(indices=[9, 10], fqn="mid_block.attentions.0.transformer_blocks", skip_attention=True, skip_ff=True)})
        kwargs.update({"skip_layer_guidance_stop": 0.4})
    elif guider_cls is SmoothedEnergyGuidance:
        kwargs.update({"seg_guidance_config": SmoothedEnergyGuidanceConfig(indices=[9, 10], fqn="mid_block.attentions.0.transformer_blocks")})
        kwargs.update({"seg_guidance_stop": 0.8})
    elif guider_cls is PerturbedAttentionGuidance:
        kwargs.update({"perturbed_guidance_config": LayerSkipConfig(indices=[9, 10], fqn="mid_block.attentions.0.transformer_blocks", skip_attention=False, skip_attention_scores=True, skip_ff=False)})
        kwargs.update({"perturbed_guidance_stop": 0.5})

    pipeline.update_components(
        guider=ComponentSpec(
            name="cfg",
            type_hint=guider_cls,
            config=kwargs,
            default_creation_method="from_config",
        )
    )
    pipeline.save_pretrained("test_modular_t2i_guider")
    pipeline.components["guider"].save_config("test_modular_t2i_guider/guider")
    
    with open("test_modular_t2i_guider/guider/guider_config.json", "r") as f:
        guider_config = json.load(f)
    guider_config["guidance_scale"] = 10.0
    with open("test_modular_t2i_guider/guider/guider_config.json", "w") as f:
        json.dump(guider_config, f, indent=2)
    
    guider_spec = ComponentSpec(
        name="cfg",
        type_hint=guider_cls,
        config=guider_cls.from_config("test_modular_t2i_guider/guider").config,
        default_creation_method="from_config",
    )
    pipeline.update_components(guider=guider_spec.create())
    print(pipeline.components["guider"])

    image = pipeline(prompt="Astronaut in a jungle, cold color palette, muted colors, detailed, 8k", num_inference_steps=20, output="images", generator=torch.Generator().manual_seed(42))[0]
    output_filename = f"output_guider_{guider_cls.__name__.lower()}.png"
    image.save(output_filename)

DN6 and others added 4 commits July 8, 2025 22:09
* update

* modify flux single blocks to make compatible with cache techniques (without too much model-specific intrusion code)

* remove debug logs

* update

* cache context for different batches of data

* fix hs residual bug for single return outputs; support ltx

* fix controlnet flux

* support flux, ltx i2v, ltx condition

* update

* update

* Update docs/source/en/api/cache.md

* Update src/diffusers/hooks/hooks.py

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* address review comments pt. 1

* address review comments pt. 2

* cache context refacotr; address review pt. 3

* address review comments

* metadata registration with decorators instead of centralized

* support cogvideox

* support mochi

* fix

* remove unused function

* remove central registry based on review

* update

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
@a-r-r-o-w a-r-r-o-w requested a review from yiyixuxu July 9, 2025 02:52
Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@yiyixuxu yiyixuxu merged commit de7cdf6 into modular-diffusers Jul 9, 2025
1 of 2 checks passed
@a-r-r-o-w a-r-r-o-w deleted the merge-modular-with-main branch July 9, 2025 04:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants