Diffusion models learn to undo noise. During training, real samples are progressively corrupted with Gaussian noise; the network learns to predict that noise. At generation time it starts from pure noise and denoises step by step, steered by your prompt, until an image emerges.
How it works
- Forward process. Noise is added to a training image over many timesteps until nothing recognisable remains.
- Reverse process. A network — historically a U-Net, increasingly a diffusion transformer — is trained to estimate the noise at each step so it can be subtracted.
- Latent space. Doing all of this on raw pixels is expensive, so most systems compress images with an autoencoder first and diffuse in that smaller latent space.
- Conditioning. Your prompt is encoded by a text model and injected via cross-attention; classifier-free guidance controls how strictly the result follows it.
- Sampling. Schedulers such as DDIM, DPM++ or flow-matching solvers reach a good sample in tens of steps instead of hundreds; distilled models manage a handful.
Controls and extensions
- Image-to-image and inpainting — start from an existing image, or restrict changes to a masked region.
- ControlNet and adapters — condition on pose, depth, edges or layout to keep composition under control.
- LoRA fine-tunes — teach a base model a specific style, product or character from a few dozen images.
- Upscaling and refiners — generate at moderate resolution, then enlarge with a model trained for detail.
- Video and audio — the same recipe extends across time with temporal attention, giving text-to-video and text-to-music systems.
Practical notes
- Guidance scale is a trade-off: too low ignores the prompt, too high produces oversaturated, brittle images.
- Fix the random seed to make results reproducible while you iterate on wording.
- Provenance matters — C2PA content credentials and watermarking are becoming the norm for generated media.