Hunyuan + Flux Sim SCHNELL Unchained

AbstractPhilaAbstractPhila
50
474
base model
v1512Comfy Workflow LoraTest

Details

Download Files

Model description

It's Schnell Simulacrum v1512 merged in a very specific way with Hunyuan.

Works with many Flux loras AND Hunyuan loras.

I know for certain that this is a confusingly derived model, and the inference is different on my ComfyUI. I cannot be certain as to why, and I do not understand as of right now WHY this is happening.

https://civitai.com/articles/11578/accidental-discovery-flux-loras-work-on-hunyuan

There is LITERALLY NO PRECEDENT TO THIS. So if you're expecting some sort of miracle or magical cure-all to merge Flux and Hunyuan this isn't going to happen.

However, there ARE multiple loras that have very obvious effects to outcome; especially those that train the CLIP_L.

I repeat, this is a BRAND NEW discovery that I made, and I almost never get attention like this from model releases. In this case I don't have all the answers.

Just know that I'm looking, and trying to FIND those answers for you all. That's the best I can do for now.

Consult the article for the ComfyUI flow chart that I used to merge the things. The outcome is essentially the same.

Combine the merged Hunyuan with OTHER HUNYUAN LORAS to specify more specific outcomes with the system; since this is highly unpredictable and unprecedented currently.

All the models are compacted.

To preface... I have no idea how to help you use this. Schnell Sim v1512 was trained heavily on plain English mixed with booru tagging and has full burned pathways for anime, 3d, and realistic. It supports negative prompting and runs optimally between cfg 3.5 and 6; but this isn't Flux, it has different rules, and yet the model works. I'll be working towards a reason in the coming weeks as to why, but as of right this minute I have no idea why.

I'll need to analyze the call chain and the block loading to see if Comfy's guidance system is somehow introducing T5 into the mix... I actually have about 50 things I need to check.

/////

Good luck...

/////

I STRONGLY advise the use of loras until I actually train this thing properly.

Prompt it mixed with actions and Schnell Simulacrum tagging.

Contents:

  • Hunyuan base model BF16

    • I chose the BF16 because it's the most responsive to the bf16 Schnell Sim.

    • In it's current state is forced to be bf16 until I release a proper checkpoint loader node with quantification attached to each of the subsets.

  • Simulacrum Schnell v1512 BF16 LINK

    • There is a full series of articles and notes on this model; which includes tagging, structures, prompt use, and careful prompt planning.

  • CLIP_24_L_OMEGA BF16

    • This is trained with over 38 million samples at this point. Try not to stub your toe, it can probably identify the bandaid you're looking for.

  • LLAMA FP8

    • This is more conveniently small and moderately fast.

    • It shouldn't impact performance too much

  • Hunyuan BF16 VAE

    • This is the only one that seems to compact correctly and actually yield the correct valuations based on the system.

Merging Help:

  • ComfyUI:

    • Checkpoint Save Node -> connect all the endpoints of the chain here.

  • Lora Loader:

    • ****

    • We are using this to load the CLIP from the Flux model.

    • If the Flux model does not have a CLIP it will likely not work; but it may.

    • ****

    • The standard Lora loader needs to be used to merge the clip correctly with the CLIP_L omega 24. Hook the model and the clip to it, only hook the OUTPUT from the clip, but not the output from the model.

    • Strength:

      • Model: 0

      • Clip: 1.0

  • Hunyuan Lora Loader:

    • ****

    • We ARE using the models from these now.

    • ****

    • This loads loras by single blocks or double blocks. Loading Flux loras should be loaded by SINGLE BLOCK, at less than 1.0; 1.0 will superimpose TOO MUCH POWER for most loras, and the majority of loras are literally BURNT TO A CRISP so they cannot be used. Lucky for us, Sim Schnell is like a perfectly cooked pot roast and is not burnt at all. It's seared to perfection on low heat over a period of weeks.

    • Strength:

      • Double Blocks: 0.2

      • Single Blocks: 0.8

  • Run both chains to the Checkpoint Save;

    • Model OUT from Hunyuan Lora Model chain

    • Clip OUT from Flux Lora Loader chain

    • VAE OUT from the VAE Load node

  • It will take more than 90 gigs of ram if set to CPU mode TO COMPACT THE MODEL, so bare with it; it takes time to convert a model. I think it took nearly 15 minutes.

You may now use your merged model; which generates faster, and will utilize many optimizations that the system provides.

You may be able to load it in many different and unique ways, or compact your own using a similar process.

This is GROUND WORK; laying foundation for the potential to build something truly great.

My version of Schnell is protected under a modified Apache 2.0 with the stipulation where small businesses, companies, and research firms may profit from this; but larger corporations and for-profit larger research groups cannot directly profit from this without monetary compensation.

For Hunyuan's case, I yield the license to Hunyuan for the merge as this isn't my base model. All rights are reserved by the original model creators and there will be no fight if a cease and desist is issued.

I will make a full apache protected finetune eventually, just not today my friends. Not today.

I am an independent researcher stationed in the USA and will respect the licensing and rights of the model owners based on local USA law and the implications therein.

Images made by this model

No Images Found.