Hlky did a great job the k-diffuser schedulers, and we now have them running comparibly to Compvis, but they need another pass to be great.
- They all share a lot of common code, so that should be factored into a common base class
- They all need converting to 0.4.1-style "All PyTorch all the time" schedulers
- We need more tests to make sure we're behaving the same as Compvis (although this seems like a common problem, as I often see differences even between DDIM or K_LMS in Diffusers versus Compvis, and FP16 + XFormers only makes it worse).
Hlky did a great job the k-diffuser schedulers, and we now have them running comparibly to Compvis, but they need another pass to be great.