The History of Generative AI

By Victor Hoang · Updated Oct 24, 2026

On January 23, 1913, in St. Petersburg, the mathematician Andrey Markov lectured to the Imperial Academy of Sciences. He counted vowels and consonants in Pushkin's Eugene Onegin. He wasn't trying to write a poem.

Generative AI is software that produces new text, images, audio, music, video or code instead of only labeling or predicting. Its precursors reach back centuries, to rule-driven combination and dice games. The modern field took shape between 2013 and 2022, with variational autoencoders, generative adversarial networks, the transformer and diffusion models. It reached the public on November 30, 2022, when ChatGPT launched. The throughline: each era replaced a human-written rule with something the machine learned. Each jump in quality came from scale and a better general-purpose design, not from a new goal. The goal, a machine that makes new things, is the oldest part of the story.

Markov was proving a point about probability. But the idea he used, that what comes next depends on what came just before, is the seed of every text generator built since. For the wider field around this story, see The History of AI and the AI timeline; for the definition, see what generative AI is.

Timeline at a Glance

The Combination Age (c. 1300-1956)

Rules, dice and statistics recombine what people wrote

Start with the Majorcan philosopher Ramon Llull, who spent decades from 1274 refining a set of rotating figures. His Ars was, in the words of the Stanford Encyclopedia of Philosophy, a combining logical system meant to reveal the truth. It started from 16 terms, each paired with the other 15, and its later Demonstrative Figure was built from six circles with a mechanism that lets them rotate. Llull wrote the Ars brevis in 1308. Keep precursor and concept apart: nothing was learned and nothing was new. The wheels produced combinations, and a human still had to reason over them, as the same entry notes. This was a generator in the narrow sense that a deck of cards is one.

Chance took over the cutting in the 1700s. A musical dice game published in Berlin in 1792 asked a player to build a 16-measure waltz from a pool of 176 numbered measures, rolling two dice to pick each one. The publisher credited Mozart, who had died the year before, though most scholars reject the claim. The machine here is a table. Every measure was written by a person, and the dice only chose the order.

Then came the first statistics of text. In 1913 Markov counted vowels and consonants in the first 20,000 letters of Eugene Onegin and showed that letters depend on one another. He was analyzing a poem, not writing one. Thirty-five years later Claude Shannon turned the same logic around. In "A Mathematical Theory of Communication," printed in the Bell System Technical Journal in 1948, he built a series of approximations to English using a 27-symbol alphabet of 26 letters and a space. His second-order word approximation, where word transition probabilities are correct but nothing else is, begins "THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER." He noted that the resemblance to English grew at each step. That is a language model in outline, produced by tables rather than by a trained network.

What this era solved: a way to make output from structure. What it left open: every rule, measure and probability table was supplied by a person, so the machine could only recombine what it was handed. For the parallel story of conversation programs, see the first AI chatbots.

For teams today: a template with a randomizer is not generative AI, and a Markov-style autocomplete is not a modern model. When a vendor says "generative," ask what the system learned from data and what a person still wrote by hand.

The Computer Composers (1957-1985)

Programmers write the rules and the machine writes the piece

In 1957 a computer program produced the score for a string quartet. Lejaren Hiller, a postdoctoral associate, and Leonard Isaacson, a graduate student, both in Fred Wall's research group at the University of Illinois, adapted their ILLIAC work on polymer conformations to build rules for composing music. The result, the Illiac Suite, is described there as "generally acknowledged as the first music composed by a computer." Its four movements are called Experiments, and their methods were published as a book in 1959. Hand-coded rules and random numbers did the work. Nothing in it was trained.

Visual art followed the same path. The painter Harold Cohen worked with his drawing program AARON from 1973 until his death in April 2016, and the Computer History Museum calls it one of the longest-running, continually maintained AI systems in history. Robotic turtles and later arms put its drawings on paper, and color came later. In a 1995 interview the museum published, AARON is described as an expert system that decides on drawing elements and their attributes. This is the symbolic tradition: knowledge written by a person, executed by a machine.

The era shows both the promise and the limit. The outputs could be striking, but each new style needed a new programmer or artist to encode it. The same ceiling that stopped rule-based AI everywhere stopped generation, which is part of the story told in the history of symbolic AI.

For teams today: hand-built generation scales with the skill of its author. If your content engine depends on one person's templates, the engine is that person.

The Learned Generators (1985-2014)

Networks stop following rules and start modeling the data

The turn came when researchers asked a network to model the data itself. In 1985, David Ackley and Geoffrey Hinton at Carnegie-Mellon and Terrence Sejnowski at Johns Hopkins published "A Learning Algorithm for Boltzmann Machines" in Cognitive Science, a method based on statistical mechanics. The abstract frames it as a way to search for solutions and to change connection strengths so a network absorbs knowledge about a task. The generative use came through the years: when Hinton shared the 2024 Nobel Prize in Physics, the Academy's release said the Boltzmann machine can be used to classify images or create new examples of the type of pattern on which it was trained. The wider arc of the field is in the history of neural networks and the history of deep learning.

Then, between December 2013 and June 2014, two designs arrived. Diederik Kingma and Max Welling posted "Auto-Encoding Variational Bayes" on December 20, 2013. It described a way to train models with continuous hidden variables using a reparameterized estimator that standard gradient methods can optimize, and it became known as the variational autoencoder, or VAE. On June 10, 2014, Ian Goodfellow, Yoshua Bengio and six co-authors posted "Generative Adversarial Networks". Two models train together: a generator and a discriminator that judges whether a sample is real or generated, framed as a minimax two-player game. When both are multilayer perceptrons, the system trains with backpropagation, with no Markov chains or unrolled approximate inference needed.

The era solved the hand-coding problem for images. Sample quality was the new problem. GANs were hard to train, and the models were far from the sentences and pictures people actually wanted. These were also the years deep learning won vision, and generation was riding the same wave.

For teams today: the dividing line between a script and a model is whether it learned from your data. That also means your data decides what it can produce, and any bias or gap in it carries through.

The Scale Years (2015-2021)

Attention and noise removal become general-purpose engines

Two ideas that would run the next decade appeared quietly. In March 2015, Jascha Sohl-Dickstein and colleagues posted "Deep Unsupervised Learning using Nonequilibrium Thermodynamics". The method adds noise to data step by step, then learns the reverse process that recovers structure, and the abstract says it is inspired by non-equilibrium statistical physics. Audio went first on the neural path: in September 2016 Aaron van den Oord and eight colleagues posted WaveNet, a generative model of raw audio, where listeners rated its text-to-speech as more natural than the best parametric and concatenative systems in English and Mandarin.

In June 2017 eight researchers posted "Attention Is All You Need", proposing the transformer, a design built on attention alone without recurrence or convolutions. It reached 28.4 BLEU on English-to-German translation. A separate sibling article covers the language-model line in depth; here the point is that one design turned out to fit many media.

OpenAI's research line shows how fast text generation moved. The first GPT paper, "Improving Language Understanding by Generative Pre-Training," pre-trained a language model on unlabeled text and then fine-tuned it, improving the state of the art on 9 of 12 tasks. The follow-up, "Language Models are Unsupervised Multitask Learners," described a 1.5-billion-parameter transformer and said its samples contain coherent paragraphs of text. On May 28, 2020, "Language Models are Few-Shot Learners" introduced GPT-3, an autoregressive model with 175 billion parameters, ten times more than any previous non-sparse language model. The text line has its own story, told in the history of large language models.

Images joined in 2020 and 2021. On June 19, 2020, Jonathan Ho, Ajay Jain and Pieter Abbeel posted "Denoising Diffusion Probabilistic Models", reporting a CIFAR-10 FID score of 3.17 and sample quality on 256x256 LSUN comparable to ProgressiveGAN. On February 24, 2021, Aditya Ramesh and colleagues posted "Zero-Shot Text-to-Image Generation," a transformer that models text and image tokens as a single stream of data; the system became known as DALL-E. Code followed on July 7, 2021, when "Evaluating Large Language Models Trained on Code" described Codex, a GPT model fine-tuned on public GitHub code. It solved 28.8% of HumanEval problems, against 0% for GPT-3, and a production version powers GitHub Copilot.

The era solved quality and generality. It created cost: training needed vast data and compute, and the best models sat inside a few labs. How much a given model had memorized, and who owned the training data, were questions nobody had settled.

For teams today: the same few designs now sit under writing, design, voice and coding tools. When you compare vendors, compare data, evaluation and workflow fit, because the underlying architecture is converging.

The Public Release (2022-present)

Open weights and a chat box put generation in everyone's hands

The research was public. In 2022 the product was. On April 13, OpenAI researchers posted a two-stage image model that uses CLIP embeddings with a diffusion decoder, the paper behind DALL-E 2. On March 4, InstructGPT had shown that fine-tuning with human feedback made a 1.3-billion-parameter model preferred by human raters over the 175-billion-parameter GPT-3 on their prompt set, with improvements in truthfulness and fewer toxic outputs. That was the step that made a language model feel like an assistant.

Then came two launches. The latent diffusion paper by Robin Rombach, Björn Ommer and colleagues, first posted December 20, 2021 and presented at CVPR 2022, ran diffusion in the latent space of a pretrained autoencoder to cut cost. On August 22, Stability AI put the Stable Diffusion weights, model card and code on Hugging Face under the Creative ML OpenRAIL-M license, with a stated memory use of about 6.9 GB of VRAM. A model that anyone could download changed who could build with image generation.

On November 30, 2022, OpenAI released ChatGPT as a free preview, based on a GPT-3.5 model. According to OpenAI's chief executive Sam Altman, as reported there, it reached 1 million users within five days. That is Altman's statement, not an independent count. The shift for working teams is covered in AI moving from research labs to work, and the question of what separates prediction from creation is explained in predictive vs generative AI.

The next months widened the range. Make-A-Video, posted September 29, 2022 by Meta researchers, translated text-to-image progress into text-to-video without paired text-video data. ReAct, posted October 6, 2022, had language models produce reasoning steps and actions in an interleaved way, a pattern behind AI agents. MusicLM, posted January 26, 2023, generated music at 24 kHz from text descriptions and stayed consistent over several minutes. And the GPT-4 technical report, posted March 15, 2023, described a multimodal model that accepts image and text inputs and produces text outputs.

This era solved access and range, and it opened the problems teams live with now: accuracy, ownership of training data, safety of open weights, and where a person must stay in the loop. The release of Stable Diffusion shows both sides, since the same page announcing the open license also notes a safety classifier included by default.

For teams today: treat generative tools as drafts from a fast, uneven colleague. Decide which outputs need human review before they leave the building, and write that rule down.

Key People

Key Facts

• Markov's analysis covered the first 20,000 letters of Eugene Onegin and was presented on January 23, 1913.
• Shannon's 1948 paper printed computed approximations to English built on a 27-symbol alphabet.
• GANs were posted on June 10, 2014 and use two models in a minimax game.
• GPT-3 has 175 billion parameters, ten times more than any previous non-sparse language model.
• Codex solved 28.8% of HumanEval problems, while GPT-3 solved 0%.
• ChatGPT launched November 30, 2022; Sam Altman said it reached 1 million users in five days.

The Pattern Behind the History

Across five eras, the same ladder repeats. We call it the Rule-to-Release Ladder. It's our synthesis of the history above, not a research finding.

1. People write the rules; the machine recombines. Llull's wheels, the dice game and the Illiac Suite all stayed inside what a person supplied.
2. Statistics replace hand-written rules. Markov and Shannon showed that sequences carry structure that a table can capture.
3. Learning replaces the tables. The Boltzmann machine, the VAE and the GAN let networks model data directly.
4. A general design plus scale unifies the media. The transformer and diffusion carried text, images, audio and code with the same few ideas.
5. Access makes it mainstream. Open weights and a free chat preview turned research into a daily tool in 2022.

What It Teaches Teams Today

• Ask what the system learned and what it was given. Rule-based generation and learned generation fail in different ways.
• Expect the architecture to change. GANs led in 2014 and diffusion led by 2020; keep your data, prompts and evaluation sets portable between models.
• Give people a way to try it. The 2022 breakthrough was access: a free preview and downloadable weights, not a new idea.
• Keep a human check on outputs. From Llull's wheels to ChatGPT, someone still had to judge whether the combination made sense.
• Read the claims by source. A company's own user count is a claim from the company; treat it that way in planning.

Generative AI looks new because the interface is new. Underneath, it's a long line from wheels and dice to tables to learned models. Ask of any tool: what did a person write, what did the machine learn, and who checks the result? For where this fits in the longer record, see the history of machine learning and the history of deep learning.