The History of Deep Learning

By Victor Hoang · Updated Oct 23, 2026

In the late 1950s, Frank Rosenblatt was writing up a hypothetical machine at the Cornell Aeronautical Laboratory. He called it the perceptron. The Office of Naval Research supported the work.

Deep learning is the use of neural networks with many layers that learn their own features from data. Its story starts in 1943, when Warren McCulloch and Walter Pitts described a logical model of the nerve cell. It became a commercial force in 2012, when a deep network won the ImageNet image-recognition contest by a wide margin. It was honored with a Nobel Prize in Physics in 2024. The throughline: the core ideas are decades older than the results. What changed was never the idea alone but the data, the hardware and the people who kept working while the field looked away.

The longer story of the idea itself, back to the neuron doctrine, is told in the history of neural networks. Fifty-four years later, in 2012, a network descended from that idea would train for under a week on two gaming graphics cards and beat every other entry in the world's biggest image contest. The gap between those two scenes is the story below. For the wider field, see The History of AI and the AI timeline; for the definition, see what deep learning is.

Timeline at a Glance

The Neuron Models (1943-1958)

Brain science and logic meet before the computers exist

In December 1943, the Bulletin of Mathematical Biophysics printed a 19-page paper. Warren McCulloch and Walter Pitts argued that, because nerve cells fire in an "all-or-none" way, neural events can be treated by means of propositional logic, and that for any logical expression meeting certain conditions one can find a net that behaves as it describes. Nothing in it learned. It was a claim about what networks of simple units could compute.

Learning arrived in the 1950s. Rosenblatt, writing from the Cornell Aeronautical Laboratory under an Office of Naval Research contract, set out a theory of a hypothetical nervous system or machine called a perceptron in Psychological Review in 1958. His paper takes sides in a debate about memory: whether the brain stores coded copies of what it senses, or whether information lives in connections that change with experience, an idea he links to Donald Hebb's "cell assembly." That second view is the ancestor of every trained network, because the thing a model learns is a set of connection strengths.

Keep precursor and concept apart. McCulloch and Pitts built a calculus, and Rosenblatt built a learning model. Neither had many layers, and neither had a method to train them. "Deep learning" as a name and as a practice arrived decades later. What this era supplied was the central picture: simple units, weighted links, behavior that comes from the pattern of weights.

For teams today: the idea behind your newest model is probably older than your company. Ask what changed, data, compute or method, before you credit a vendor with a breakthrough.

The Lean Years (1960s-1985)

A discouraging book, and quiet progress nobody could fund

Deep networks existed earlier than most people think. Alexey Ivakhnenko and colleagues grew networks layer by layer using a method of regression analysis, published from 1965. Jürgen Schmidhuber's survey of the field says these networks were perhaps the first deep learning systems of the multilayer perceptron type, and that a 1971 paper described one with 8 layers. Treat the "first" as Schmidhuber's judgment; he is a participant in this history, and says so himself in his preface.

Then came 1969. Marvin Minsky and Seymour Papert's book on the limits of simple single-layer perceptrons, Schmidhuber writes, discouraged some researchers from further studying neural networks, which he finds surprising given the multilayer work that already existed. It fit the mood of an AI field that was moving toward rules and symbols, and soon into the first AI winter (see the history of symbolic AI and the history of expert systems).

The pieces kept arriving anyway. The method that would train deep networks, the chain rule applied efficiently backward through a network, was described in a 1970 master's thesis by Seppo Linnainmaa without reference to neural networks, and Paul Werbos described a neural-network application in 1981, per the same survey. In Tokyo, Kunihiko Fukushima at the NHK Broadcasting Science Research Laboratories published the neocognitron in 1980, a network with alternating layers of cells modeled on the visual system proposed by Hubel and Wiesel. It learned without a teacher, and Schmidhuber notes Fukushima did not set weights by supervised backpropagation. The convolutional design was there. The efficient training was not yet joined to it.

This era solved little in practice, and it created the field's unfinished business: the methods for depth and for training were known separately, and no one had yet shown them together on a problem anyone would pay for.

For teams today: a promising method can sit idle for a decade because one piece is missing. When you evaluate emerging technology, list what's still missing, and set a date to look again.

The Backpropagation Years (1986-1997)

Training hidden layers works, and the first useful products appear

In the autumn of 1986, Nature published a four-page paper (received May 1, issue dated October 9) by David Rumelhart, Geoffrey Hinton and Ronald Williams. It described back-propagation, a procedure that repeatedly adjusts connection weights to shrink the difference between actual and desired output. Its abstract makes the claim that mattered: internal "hidden" units come to represent important features of the task, which distinguishes the method from earlier, simpler procedures such as the perceptron-convergence procedure. Schmidhuber credits the paper with popularizing backpropagation for neural networks by showing useful internal representations emerge, while crediting earlier work. Both are true: the paper did not invent the math, and it changed what researchers believed networks could do.

The first real application followed within three years. At AT&T Bell Laboratories in Holmdel, New Jersey, Yann LeCun and six colleagues trained a back-propagation network to read handwritten digits from U.S. Postal Service zip codes. Their 1989 paper reports a 1% error rate with about a 9% reject rate and states its point plainly: large back-propagation networks can handle real image recognition without a complex, hand-engineered preprocessing stage. The architecture was highly constrained and designed for the task, a lesson in itself. Profiles of the people involved are in our guides to Yann LeCun's leadership.

Sequences were the other front. Recurrent networks struggled because error signals decayed as they traveled back through time. In 1997, Sepp Hochreiter and Schmidhuber introduced Long Short-Term Memory, which could bridge time lags of more than 1,000 discrete time steps through gated units that control a constant flow of error.

The era solved training and proved the value on narrow jobs. It created a ceiling: networks beyond a few layers were hard to train, data was scarce, and computers were slow. Competing methods were often easier to use, and attention drifted.

For teams today: a technology can work in production and still lose the mindshare contest. Judge on results for your task, and don't read a field's mood as a verdict on its method.

The Quiet Years (1998-2008)

A small group keeps the faith, and layers learn one at a time

For about a decade, neural networks were a minority pursuit, while kernel methods and other statistical tools led the wider history of machine learning. ACM's account of the 2018 Turing Award puts it directly: by the early 2000s, LeCun, Hinton and Yoshua Bengio were among a small group who remained committed to this approach, and their efforts to rekindle interest in neural networks were initially met with skepticism. This is the stretch the history books tend to skip, because little was published that looked like a victory.

The turning move came in 2006. Hinton, Simon Osindero and Yee-Whye Teh published "A Fast Learning Algorithm for Deep Belief Nets" in Neural Computation, showing a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time. After fine-tuning, a network with three hidden layers modeled handwritten digits well enough to classify them better than the best discriminative methods the authors compared against. The paper's author note lists Hinton as a fellow of the Canadian Institute for Advanced Research.

The recipe was to train a layer, then the next, then fine-tune. Its importance was practical: it offered a way to make depth work again, and it pointed attention back to networks with many layers.

This era's problem was not the idea but the evidence. Nothing yet showed deep networks beating the field on a benchmark everyone respected. That needed data at a scale no lab had.

For teams today: the people who keep working on an unfashionable problem are an asset. If a method matters to your business, fund a small, patient team rather than waiting for a trend.

The ImageNet Moment (2009-2016)

Data and graphics chips meet the old ideas

In 2009, a team at Princeton including Jia Deng and Li Fei-Fei presented ImageNet at CVPR. The paper introduced a large-scale ontology of images built on the WordNet structure, with 3.2 million images across 5,247 categories at that point and an aim of tens of millions. The project's own site notes the paper was later awarded a retrospective most impactful paper prize from CVPR 2009. ImageNet turned a research wish into a measurable contest. For her story, see our profile of Fei-Fei Li's leadership.

The 2012 round of the contest changed the field, a story told in full in ImageNet and AlexNet. Alex Krizhevsky, Ilya Sutskever and Hinton trained a deep convolutional network on 1.2 million labeled images and won with a top-5 error of 15.3%, against 26.2% for the second-best entry. The network had 60 million parameters and 650,000 neurons, and it took five to six days to train on two GTX 580 GPUs. The authors credit an efficient GPU implementation and non-saturating neurons for making training fast, and note a single GPU's 3GB of memory limited how large a network they could build.

Nothing in that recipe was new in kind. Convolutional layers traced to Fukushima and LeCun, backpropagation to the 1970s and 1980s, and the hardware was built for video games. What was new was the combination at scale. The 2015 review of the field by LeCun, Bengio and Hinton describes the 2012 report as a breakthrough that almost halved the error rate for object recognition and precipitated rapid adoption. ACM later pointed to deep learning's breakthroughs in computer vision, speech recognition, natural language processing and robotics.

The problem this era created was dependence. Deep learning worked, but it leaned on large labeled datasets and expensive compute.

For teams today: the winner in a technology shift is often the group with the data and the infrastructure, not the one with the cleverest idea. Audit your own data before you shop for models.

The Transformer Age (2017-present)

Attention replaces recurrence and scale takes over

In 2017, eight researchers, most at Google Brain and Google Research, published "Attention Is All You Need" at the NIPS conference. They proposed the Transformer, a network based solely on attention, dispensing with recurrence and convolutions entirely. On English-to-German translation it reached 28.4 BLEU, improving on the previous best results by over 2 BLEU, and it trained in a fraction of the time. The design dropped recurrence, which made it easier to train in parallel, and it scaled into today's large language models; see the Transformer architecture for how it works.

Recognition for the pioneers arrived as the technology spread. On March 27, 2019, ACM named Bengio, Hinton and LeCun recipients of the 2018 Turing Award, a $1 million prize, for conceptual and engineering breakthroughs that made deep neural networks a critical component of computing. On October 8, 2024, the Royal Swedish Academy of Sciences awarded the Nobel Prize in Physics to John Hopfield and Geoffrey Hinton for foundational discoveries and inventions that enable machine learning with artificial neural networks. The Academy credited Hopfield with an associative memory and Hinton with the Boltzmann machine, built on Hopfield's network, a method for finding properties in data on its own.

Today deep learning sits under most of what people call AI, from generative AI to the systems described in AI moving from research labs to work. The same questions from earlier eras return: who owns the data, who pays for the compute, and who checks the output. The neural networks at the base are the same idea McCulloch and Pitts sketched, scaled by many orders of magnitude.

For teams today: treat model choice as a replaceable layer. The architecture changed three times in this history; the data, the evaluation and the people who understand your problem lasted.

Key People

Key Facts

• McCulloch and Pitts published their neuron logic paper in December 1943, in volume 5 of the Bulletin of Mathematical Biophysics.
• Back-propagation's Nature paper was received on May 1, 1986 and issued October 9, 1986.
• The 1989 zip code network reached a 1% error rate with about a 9% reject rate.
• AlexNet's top-5 error was 15.3%, against 26.2% for the runner-up, after five to six days on two GPUs.
• The Transformer paper reported 28.4 BLEU on English-to-German translation.
• The 2024 physics Nobel went to Hopfield and Hinton, announced October 8, 2024.

The Pattern Behind the History

Across six eras, the same four moves repeat. We call the sequence the Idea-Ingredient-Believer Cycle. It's our synthesis of the history above, not a research finding.

1. The idea arrives early. Layered networks appear in 1943, 1958, the 1960s and 1980 long before they pay off.
2. A missing ingredient holds it back. Training methods, labeled data or cheap parallel compute, each absent at different times.
3. A few believers keep going. ACM notes a small group stayed committed in the early 2000s, despite skepticism.
4. A benchmark makes it undeniable. ImageNet and the 2012 contest turned a minority view into the default almost overnight.

What It Teaches Teams Today

• Separate the idea from the ingredients. Ask which of data, compute and method is the real constraint on your AI project.
• Fund patient work. The researchers behind deep learning worked through years of skepticism before the results came.
• Build a benchmark early. ImageNet shows that a shared, measurable test can move a field faster than any single model.
• Expect the architecture to change. Keep your data pipeline, evaluation set and domain knowledge portable between models.
• Read hype cycles carefully. The field's gloomiest years produced some of its most important work.

Deep learning shows that a good idea can wait seventy years for its moment and still be right. Ask of any new AI tool: which of its ingredients is actually new, and which has been waiting for the rest to catch up? For more on the origins of the field around it, see the origin of artificial intelligence and the Dartmouth conference of 1956.