How Diffusion Models Work

Learn a family of methods, not every image generator

Imagine a collection of simple cup drawings. We want a system that can generate another plausible cup, including variations absent from a short list of examples. Diffusion modelling offers one approach: learn how data looks at different noise levels, then use that learned behaviour to guide a noisy starting point toward a sample. The result is generated, not a hidden photograph uncovered from a particular patch of static.

This guide explains the denoising diffusion probabilistic model, or DDPM, as a useful starting model. Later systems can change the prediction target, sampling method, representation and architecture. Not every image generator is a diffusion model, and a product's ability to create attractive images does not reveal its proprietary implementation.

Keep three activities separate: adding noise creates training examples; training adjusts model parameters; sampling uses the trained model to generate outputs. Confusing these activities leads to claims such as “more generation steps train the model better”. During ordinary sampling, the model's learned weights are not being updated.

Create noisy versions with a known process

In the standard DDPM setup, a forward process gradually mixes data with Gaussian noise according to a schedule. Early noise levels retain more of the original signal; later levels retain less. Training can construct a sample at a chosen level directly, rather than physically saving every intermediate image. The chosen level tells the model how corrupted its input is.

A common expression is x_t = sqrt(a) × x_0 + sqrt(1 − a) × noise. Here x_0 is the clean data, x_t is its noisy version, and a is the cumulative fraction controlling signal strength at that level. The square roots scale the signal and noise terms. This is arithmetic on model data, not simply adding a visual blur filter to a photograph.

The next block uses two numbers and a fixed noise vector so the calculation is reproducible. Real training draws noise samples; this example deliberately avoids randomness. Values are abstract normalised coordinates and need not stay between zero and one. They are not an actual two-pixel image or a working model.

const clean = [1, -1];
const noise = [0, 2]; // Fixed teaching values, not sampled Gaussian noise.
function mix(a) {
  return clean.map((value, i) =>
    Math.sqrt(a) * value + Math.sqrt(1 - a) * noise[i]);
}
function close(actual, expected) {
  if (!Number.isFinite(actual) || !Number.isFinite(expected) ||
      Math.abs(actual - expected) > 1e-12) throw new Error('Mismatch');
}
for (const [actual, expected] of [[NaN, 0], [0, NaN],
  [Infinity, Infinity], [-Infinity, 0], [0, Infinity]]) {
  let rejected = false;
  try { close(actual, expected); } catch { rejected = true; }
  if (!rejected) throw new Error('Accepted a non-finite comparison');
}
mix(1).forEach((value, i) => close(value, clean[i]));
mix(0).forEach((value, i) => close(value, noise[i]));
const noisy = mix(0.25);
close(noisy[0], 0.5);
close(noisy[1], Math.sqrt(3) - 0.5);
// An oracle calculation: we know the exact noise only because we created it.
const recovered = noisy.map((value, i) =>
  (value - Math.sqrt(0.75) * noise[i]) / Math.sqrt(0.25));
recovered.forEach((value, i) => close(value, clean[i]));
console.log('Forward-noise checks passed; no model was trained.');

Train a prediction at many noise levels

For a simplified noise-prediction training step, select a clean example, choose a timestep, sample noise and construct the noisy input. Give the noisy input and timestep to a neural network. Compare its predicted noise with the noise that was actually added, then adjust the weights to reduce the error. Repeating this across many examples and levels teaches a denoising relationship.

The training code knows the injected noise because it created the noisy example. The network must infer it from the corrupted input and the conditioning it receives. Our little calculation recovers the clean vector using an oracle: it already knows the exact noise. That algebra demonstrates the target relationship but does not solve the learning problem.

Different formulations can predict noise, clean data or another related quantity. They require compatible training objectives and sampling equations. It is therefore misleading to swap a prediction target into a sampler without checking its documented assumptions. Low training error also does not prove broad image quality or eliminate memorisation and bias in the training data.

Use a model and scheduler to generate

At generation time, a basic diffusion sampler begins with noise. At each selected timestep, the trained network estimates the quantity its formulation expects. A scheduler or sampler uses that estimate and the noise schedule to compute the next sample. Some methods add a controlled random term during the process. The output becomes the input to the next step.

The network supplies a learned prediction; the scheduler supplies the update rule. Neither is interchangeable with the other. Hugging Face's documented DDPM example exposes these as separate components, which helps explain why changing a sampling method can change speed and output while using the same model weights. Always use a compatible model and scheduler configuration.

The number of inference steps is not the number of training updates. Fewer sampling steps can reduce work but may change fidelity, and some methods are designed for particular step ranges. More steps are not a universal guarantee of better results. Compare representative outputs and record the actual settings instead of treating a large number as a quality certificate.

Guide generation with information

An unconditional model generates from its learned distribution without a requested description. A conditional model also receives information such as a class label, text representation or reference image. In our cup example, a description of a blue ceramic cup can influence the denoising predictions. It does not give the sampler a perfect three-dimensional cup model or enforce every geometric constraint.

Some systems use classifier-free guidance, combining predictions with and without conditioning to strengthen the prompt's influence. That is a particular method, not a required part of all diffusion systems. The effect and useful setting range depend on the model and sampler. Stronger guidance can also distort outputs; treat it as a parameter to evaluate, not a scale of intelligence.

A seed controls pseudorandom choices in supported implementations. Holding it fixed can help compare a prompt or setting change, but reproducibility also depends on model weights, software, scheduler and execution details. Changing the seed explores another sample; it does not repair missing knowledge or make a factual diagram trustworthy.

Generate in a learned representation

Pixel-space diffusion works with a representation shaped like the image's pixel data. Latent diffusion first uses an encoder to map an image to a smaller learned representation. Diffusion operates there, and a decoder turns the final latent into an image. “Latent” means that the representation is internal; its coordinates are not necessarily recognisable miniature pixels.

The latent diffusion paper demonstrates this approach as a way to reduce the computational burden of working directly with full-resolution images. Compression is a tradeoff: the encoder and decoder can limit which fine details survive. An excellent denoising prediction cannot reconstruct information the representation cannot express faithfully. Text, small structures and exact geometry still need output review.

Do not infer a specific encoder, U-Net, transformer or training dataset from a product's interface. The published latent diffusion system is an example with documented components. Other systems can use different representations and architectures while sharing broad ideas about conditioning or iterative generation.

Connect the mechanism to a useful experiment

If you have a documented diffusion tool, use an ordinary fictional object such as a cup and record the model, sampler, prompt, dimensions, seed if available and step settings. Change one supported setting, keep the rest fixed where possible and compare shape, prompt adherence and unwanted artifacts. Repeat with several examples before drawing a general conclusion.

Use references and datasets you are authorised to process; training and uploading inference inputs are different data-handling decisions. Generated samples can reproduce biases or resemble training examples, so a mechanism description is not a privacy guarantee. Avoid using a plausible generated image as evidence of a real object or event.

Without a generation tool, the local calculation still lets you explain the forward process and its limits. Success means you can identify what is known during training, what must be predicted during sampling and why the arithmetic example is not a denoiser. That conceptual distinction transfers to more advanced diffusion implementations.

Sources & further reading

Primary references for the concepts in this guide. Examples and exercises are illustrative; verify provider-specific behaviour against the version you use.

• Ho, Jain and Abbeel: Denoising Diffusion Probabilistic Models
• Rombach et al.: High-Resolution Image Synthesis with Latent Diffusion Models
• Hugging Face: Pipelines, models and schedulers
• Ho and Salimans: Classifier-Free Diffusion Guidance
• Hugging Face: Reproducibility