Types of Generative AI Models

Types of generative AI models are the main architecture families used to create new content: transformers, diffusion models, generative adversarial networks (GANs), variational autoencoders (VAEs) and flow-based models. Each family learns the distribution of its training data differently, which determines its output quality, generation speed and the kind of data it suits most.

• Transformers: They generate sequences one token at a time and power most text and code models.
• Diffusion models: They create images, audio or video by removing noise from a random starting point over many steps.
• GANs: They train two competing networks, a generator and a discriminator, until generated samples look realistic.
• VAEs: They compress data into a compact latent space and generate new samples by decoding points from that space.
• Flow-based models: They use reversible transformations, so they can calculate the exact probability of any sample.
• Combined designs: Many production systems join several families, such as a VAE, a transformer and a diffusion model in one image pipeline.

For example, a developer product might use a transformer to write API docs and test code, and a diffusion model to create illustrations for the same documentation.

Overview of Types of Generative AI Models

Transformer Models

• Mechanism: The transformer architecture uses attention, which lets each token weigh every earlier token when predicting the next one.
• Autoregressive generation: Output is produced token by token, so long responses take proportionally longer to generate.
• Scale: Large transformer models trained on text and code are known as large language models.

Example: a code assistant completes def parse_invoice( with arguments and a docstring.

Diffusion Models

• Mechanism: During training, noise is added to images step by step, and the network learns to predict and remove that noise.
• Generation: Sampling starts from pure noise and applies many denoising steps, guided by an encoded text prompt.
• Trade-off: Output quality is high, but the repeated steps make generation slower than a single forward pass.

Example: a docs team generates a clean architecture illustration from the prompt "three services behind an API gateway".

Generative Adversarial Networks (GANs)

• Mechanism: The generator creates samples, and the discriminator learns to separate generated samples from real ones.
• Strength: A trained generator produces an output in one forward pass, which makes inference fast.
• Weakness: Training can be unstable, and mode collapse, where the generator produces only a few similar outputs, is common.

Example: a design team upscales low-resolution product screenshots for high-density displays.

Variational Autoencoders (VAEs)

• Mechanism: An encoder maps each input to a probability distribution in a latent space, and a decoder reconstructs data from sampled points.
• Smooth latent space: Nearby latent points decode to similar outputs, which allows controlled variation and interpolation.
• Role today: VAEs often act as the compression stage inside latent diffusion systems rather than as standalone generators.

Example: a monitoring team flags unusual telemetry traces that the VAE reconstructs poorly.

Flow-Based Models

• Mechanism: A sequence of invertible functions maps simple random noise to data and back again.
• Exact likelihood: Because each step is reversible, the model can compute the precise probability of any input.
• Current use: Related techniques, such as flow matching, appear in some recent image and audio generators.

Example: a research team estimates how likely a new audio sample is under the training distribution.

Combining Model Types

• Latent diffusion: A VAE compresses images, a diffusion network denoises in that compressed space, and a text encoder conditions the result on the prompt.
• Diffusion transformers: Some newer image and video models replace the older convolutional denoiser with a transformer.
• Multimodal models: Transformers that accept text and images together can describe screenshots or answer questions about diagrams.

Objective Functions of GANs, VAEs and Diffusion Models

Each model family optimises a different training objective, the quantity that training either maximises or minimises. The objectives of GANs, VAEs and diffusion models explain much of their practical behaviour, from unstable adversarial training to comparatively slow diffusion sampling.

GAN Loss Function

The GAN loss function describes a two-player minimax game, in which the discriminator $D$ attempts to maximise the value $V$ and the generator $G$ attempts to minimise it:

• $D(x)$: the discriminator's estimated probability, between 0 and 1, that a sample $x$ is genuine.
• $G(z)$: a synthetic sample that the generator creates from random input noise $z$.
• $p_{\text{data}},p_z$: the probability distributions of the real training data and of the input noise.
• $\mathbb{E}$: the expected value, which is an average calculated over many samples.

In other words, the discriminator wants scores near 1 for genuine samples and near 0 for generated ones, while the generator wants its samples classified as genuine. For one real UI screenshot scored 0.9 and one generated screenshot scored 0.2, the value is $V=\log 0.9+\log 0.8\approx -0.105-0.223=-0.329$. When the generator becomes good enough that the discriminator outputs 0.5 for both samples, the value falls to $2\log 0.5\approx -1.386$, which equals $-\log 4$, the theoretical value of the game at equilibrium.

VAE Evidence Lower Bound

A variational autoencoder maximises the evidence lower bound (ELBO), a quantity that can never exceed the log-probability of the training data:

• $q_{\varphi }(z\mid x)$: the encoder, which produces a probability distribution over latent codes for a given input.
• $p_{\theta }(x\mid z)$: the decoder, which measures how accurately the input is reconstructed from a latent code.
• $p(z)$: the prior distribution over latent codes, usually a standard normal distribution.
• $D_{\mathrm{KL}}$: the Kullback-Leibler divergence, which measures how different one probability distribution is from another.

The first term rewards accurate reconstruction of the input. The second term penalises latent codes that drift away from the prior, which keeps the latent space smooth and continuous enough for sampling new outputs.

Diffusion Model Math

Diffusion model math begins with a fixed forward process that adds a small amount of Gaussian noise at every step $t$:

Because consecutive Gaussian steps combine into a single Gaussian, any step can be calculated directly from the clean data $x_0$:

• $x_0,x_t$: the original clean data, such as an image, and its noisy version after $t$ steps.
• ${\beta }_t$: the variance of the noise added at step $t$, determined in advance by a noise schedule.
• ${\bar{\alpha }}_t$: the proportion of the original signal's variance that survives after $t$ steps.
• $ϵ$: standard Gaussian noise, where $I$ denotes the identity matrix.

The neural network is trained to predict the added noise from the noisy input and the step number, and generation then reverses the process one denoising step at a time.

Worked example. With a constant noise variance of 0.1, the cumulative products are ${\bar{\alpha }}_1=0.9$, ${\bar{\alpha }}_2=0.9\times 0.9=0.81$ and ${\bar{\alpha }}_3=0.729$. After three steps, $x_3=\sqrt{0.729}{x}_0+\sqrt{0.271}ϵ\approx 0.854x_0+0.521ϵ$, so the original pixel still contributes about 85 percent of its value. The program below calculates both GAN values and a complete 1,000-step linear schedule, with the variance rising from 0.0001 to 0.02, for one pixel value of 1.0 and one fixed noise value of -0.8.

import math

# GAN value V(D, G) for one real sample and one generated sample.
def gan_value(d_real, d_fake):
    return math.log(d_real) + math.log(1 - d_fake)

print(f"confident D (0.9 real, 0.2 fake): V = {gan_value(0.9, 0.2):.3f}")
print(f"fooled D    (0.5 real, 0.5 fake): V = {gan_value(0.5, 0.5):.3f}")

# Diffusion forward process with the linear schedule from the DDPM paper:
# beta rises from 0.0001 to 0.02 over T = 1000 steps.
T = 1000
betas = [1e-4 + (0.02 - 1e-4) * (t - 1) / (T - 1) for t in range(1, T + 1)]
x0, eps = 1.0, -0.8  # one pixel value and one fixed noise draw, so output repeats

alpha_bar = 1.0
print(f"\n{'t':>5}{'alpha_bar':>12}{'signal':>8}{'noise':>7}{'x_t':>8}")
for t, beta in enumerate(betas, start=1):
    alpha_bar *= 1 - beta
    if t in (1, 100, 250, 500, 750, 1000):
        signal, noise = math.sqrt(alpha_bar), math.sqrt(1 - alpha_bar)
        x_t = signal * x0 + noise * eps
        print(f"{t:>5}{alpha_bar:>12.5f}{signal:>8.3f}{noise:>7.3f}{x_t:>8.3f}")

confident D (0.9 real, 0.2 fake): V = -0.329
fooled D    (0.5 real, 0.5 fake): V = -1.386

    t   alpha_bar  signal  noise     x_t
    1     0.99990   1.000  0.010   0.992
  100     0.89702   0.947  0.321   0.690
  250     0.52409   0.724  0.690   0.172
  500     0.07859   0.280  0.960  -0.488
  750     0.00335   0.058  0.998  -0.741
 1000     0.00004   0.006  1.000  -0.794

• Signal fades gradually: Signal and noise contribute equally near step 260, and by step 1,000 the signal coefficient has fallen to 0.006.
• Sampling starts from noise: Because the final step is almost pure noise, generation can begin from random noise and reverse the process.
• GANs aim for equilibrium: The ideal outcome is a discriminator that can no longer distinguish genuine from generated samples, which is the $-1.386$ case.

In practice, each reverse diffusion step requires one call to the network, so the number of steps determines the generation cost, which is why faster samplers use far fewer steps than the 1,000 used during training.

How to Choose

• Text or code output: Choose a transformer model, usually through an existing LLM API or an open-weight model.
• High-quality images or video: Choose a diffusion model, accepting slower generation in exchange for quality.
• Low-latency image generation: Consider a GAN when real-time output matters more than variety.
• Compact representations or anomaly detection: Choose a VAE, which provides a structured latent space.
• Exact probabilities: Choose a flow-based model when likelihood values are required.

The training process shared by these families is described in how generative AI works, and industry uses are listed in applications of generative AI. For the definition, see what is generative AI.