[Tests] Speed up fast pipelines part II (#7521) #25

Job	Run time
Flax TPU Tests	1s
PyTorch Compile CUDA tests	1s
PEFT CUDA Tests	1s
ONNX CUDA Tests	1s
Setup Torch Pipelines CUDA Slow Tests Matrix	22s
Torch CUDA Tests (models)	1s
Examples PyTorch CUDA tests on Ubuntu	1s
Torch CUDA Tests (schedulers)	1s
PyTorch xformers CUDA tests	1s
Torch CUDA Tests (lora)	1s
Torch CUDA Tests (others)	1s
Torch Pipelines CUDA Slow Tests (kandinsky2_2)	1s
Torch Pipelines CUDA Slow Tests (ip_adapters)	1s
Torch Pipelines CUDA Slow Tests (stable_diffusion_adapter)	1s
Torch Pipelines CUDA Slow Tests (wuerstchen)	1s
Torch Pipelines CUDA Slow Tests (deepfloyd_if)	1s
Torch Pipelines CUDA Slow Tests (utils)	1s
Torch Pipelines CUDA Slow Tests (stable_diffusion)	1s
Torch Pipelines CUDA Slow Tests (stable_diffusion_xl)	1s
Torch Pipelines CUDA Slow Tests (controlnet)	1s
Torch Pipelines CUDA Slow Tests (kandinsky)	1s
Torch Pipelines CUDA Slow Tests (text_to_video_synthesis)	1s
Torch Pipelines CUDA Slow Tests (stable_diffusion_2)	1s
	44s

Provide feedback