AnimateDiff SparseCtrl RGB does not work as expected
#9 508 ouverte le 23 sept. 2024
Métriques du dépôt
- Stars
- (22 190 stars)
- Métriques de merge PR
- (Merge moyen 13j 1h) (96 PRs mergées en 30 j)
Description
Relevant comments are this and this.
AnimateDiff SparseCtrl RGB does not work similar to other implementations and cannot replicate their outputs. This makes me believe that there is something incorrect with our SparseControlNet or MotionAdapter implementation.
When comparing the results of the original/Comfy implementation to Diffusers implementation, one can notice that if an image is used with an unrelated prompt, the Diffusers implementation ignores the image and just follows the prompt whereas the other implementations try to incorporate both.
Since the original and Comfy implementations produce this behaviour consistently, this seems more like a problem with Diffusers implementation. However, I've not been able to spot differences in implementation just by comparing the code visually. I also tried matching outputs layerwise and it seemed to be alright (although I didn't investigate this as deeply as I should have due to other priorities).
If someone from the community actively following/using the AnimateDiff implementations can help determine the cause of this bug, it would be really awesome and helpful.