This is such a natural extension to LLMs. I’m shocked it hasn’t been tried before.<p>When I ask a diffusion model to generate a chessboard, I’d expect the pieces to be placed randomly. We are getting closer to image generators that not only know what chess pieces look like but also where to place them.

You can talk to the authors directly on alphaXiv! <a href="https:&#x2F;&#x2F;www.alphaxiv.org&#x2F;abs&#x2F;2408.11039v1" rel="nofollow">https:&#x2F;&#x2F;www.alphaxiv.org&#x2F;abs&#x2F;2408.11039v1</a>

Stupid question: is their 7B model available? Is there public inference code that we could run? Or do they not usually release them along with these kinds of papers?

Hmm. I wonder if this is similar to Diffusion Transformers?

Would such a model be able to give more accurate description of images as well?

HN

Transfusion: Predict the next token and diffuse images with one multimodal model

Comments (10)