Abstract

We present the Hourglass Diffusion Transformer (HDiT), an image generative model that exhibits linear scaling with pixel count, supporting training at high-resolution (e.g. 1024×1024) directly in pixel-space. Building on the Transformer architecture, which is known to scale to billions of parameters, it bridges the gap between the efficiency of convolutional U-Nets and the scalability of Transformers. HDiT trains successfully without typical high-resolution training techniques such as multiscale architectures, latent autoencoders or self-conditioning. We demonstrate that HDiT performs competitively with existing models on ImageNet 2562, and sets a new state-of-the-art for diffusion models on FFHQ-10242.

Paper: arxiv.org/abs/2401.11605

Code: github.com/crowsonkb/k-diffusion

Project Page: …github.io/hourglass-diffusion-transformers/