Diffusion models: Everything you need to know

This is possible because the model will have access to the real images during training and will be able to learn the right parameters by applying such noise to the image iteratively until it reaches complete noise and is unrecognizable. Then, when we are satisfied with the noise we get from all images, meaning that they are similar and generate noise from a similar distribution, we are ready to use our model in reverse and feed it similar noise in the reverse order to expect an image similar to the ones used during training.

Text-to-Image models like DALLE or stable diffusion are really cool and allow us to generate fantastic pictures with a simple text input. But would it be even cooler to give them a picture of you and ask it to turn into a painting? Imagine being able to send any picture of an object, person, or even your cat, and ask the model to transform it into another style like turning yourself into a cyborg or into your preferred artistic style or adding it to a new scene.

Basically, how cool would it be to have a version of DALLE we can use to photoshop our pictures instead of having random generations? Having a personalized DALLE, while making it much more simple to control the generation as “an image is worth a thousand words”. It would be like having a DALLE model that is just as personalized and addictive as the TikTok algorithm.

Well, this is what researchers from Tel Aviv University and NVIDIA worked on. They developed an approach for conditioning text-to-image models, like stable diffusion we just reviewed, with a few images to represent any object or concept through the words you will send along your images. Transforming the object of your input images into whatever you want! Learn more in the article or the video below!

Meta AI published a model for video generation called make-a-video. In a single sentence: Make-A-Video generates videos from text. It’s not only able to generate videos, but it’s also the new state-of-the-art method, producing higher quality and more coherent videos than ever before!

You can see this model as a stable diffusion model for videos. Surely the next step after being able to generate images. This is all information you must’ve seen already on a news website or just by reading the title, but what you don’t know yet is what it is exactly and how they could adapt such a model to videos, adding a time dimension to images. Learn more in the article or the video below!

We’ve seen models able to take a sentence and generate images. Then, other approaches to manipulate the generated images by learning specific concepts like an object or particular style.

Fast forwards a few more weeks and Meta published the Make-A-Video model we just learned about, which allows you to generate a short video also from a text sentence. The results aren’t perfect yet, but the progress we’ve made in the field since last year is just incredible.

Now, we make another step forward.

Here’s DreamFusion, a new Google Research model that can understand a sentence enough to generate a 3D model of it. You can see this as a DALLE or Stable Diffusion but in 3D. 

It basically uses two models I already covered: NeRFs and one of the text-to-image models. In their case, it is the Imagen model, but any would do, like stable diffusion. Learn more about Dreamfusion in the article or in the video below!

Despite having some choice words for OpenAI regarding their bias problems with CLIP, I think they’ve done a great job at mitigating bias at many points early on with DALL•E 2, and it seems to be working well. I especially appreciate the reweighting of the images when the team realized there was an imbalance reflected in the results. It often appears like filtering negative data is the ultimate bias mitigator, but OpenAI has proved you can (and should) go multiple steps further to reduce harm.

My greatest hope for DALL•E 2 is that its bias mitigation treatment sets a precedent for the level of care that should be shown in many other models, even when they’re not massive, widely used meme sources. Do the right thing even when no one is watching!