The History of Neural Networks

By Victor Hoang · Updated Oct 23, 2026

In July 1958, the United States Navy's Office of Naval Research held a press event in Washington. The New York Times reported that the Navy had revealed the embryo of an electronic computer. It expected the machine would one day walk, talk and see. It would write and be conscious of its existence.

The artificial neural network began as mathematics in 1943, when Warren McCulloch and Walter Pitts described nerve cells as logic units. It got its first learning machine in 1958, when Frank Rosenblatt's perceptron made national news. It fell out of favor after 1969, returned in the mid-1980s with Hopfield networks, Boltzmann machines and backpropagation, and won wide attention only in 2012. The throughline: the idea was ready long before the evidence. Each stall lacked a way to train more than one layer, plus the data and computing power to prove it.

The machine in question could, in the demonstration, tell whether a square sat on the left or the right of a field. The distance between that promise and that demonstration is the story below. For the wider field see The History of AI; for the later deep-learning chapter see The History of Deep Learning; for the definition see what neural networks are.

Timeline at a Glance

The Wiring Diagram (1890s-1949)

Biologists draw the cell, a logician turns it into math

The first era belongs to people who never built a computer. In 1906 the Nobel Prize in Physiology or Medicine went jointly to Camillo Golgi and Santiago Ramón y Cajal in recognition of their work on the structure of the nervous system. The two men did not agree. Golgi's Nobel lecture was titled "The Neuron Doctrine, Theory and Facts", yet the Nobel Foundation's own account of the award says he held to the reticular theory, the rival view that the nervous system was one continuous net, while the neuron doctrine had gained acceptance among most neuroscientists. Cajal's lecture, a day later, was "The Structure and Connexions of Neurons". The nerve cell, as a separate unit joined to others by connections, was the picture that later machine builders borrowed.

Charles Sherrington added the circuit view. The Nobel Foundation notes that in the 1890s he showed how different reflexes are part of a complicated interplay in which the spinal cord and brain process nerve impulses and turn them into new ones. He shared the 1932 prize "for their discoveries regarding the functions of neurons".

Then came the mathematics. In December 1943, the Bulletin of Mathematical Biophysics printed a paper by Warren McCulloch, a psychiatrist at the University of Illinois, and Walter Pitts of the University of Chicago. Because of the "all-or-none" character of nervous activity, they argued, neural events and their relations can be treated by means of propositional logic. Nothing in that model learned. It described what a net of simple switches could compute.

Learning came in 1949, from psychology. Donald Hebb's book The Organization of Behavior (New York: Wiley, 1949) is cited as the source of the "cell assembly" idea in Rosenblatt's 1958 paper. The Royal Swedish Academy of Sciences summarizes Hebb's hypothesis as the idea that learning occurs because connections between neurons are reinforced when they work together, and says it is still used as a basic rule for updating artificial networks during training.

Keep precursor and concept apart. Golgi, Cajal, Sherrington, McCulloch, Pitts and Hebb did not build artificial neural networks. They supplied the picture: cells, connections and connections that change with experience. The Academy puts it plainly: the inspiration for artificial networks came from the desire to understand how the brain works.

For teams today: the first useful models of a technology often come from a neighboring field. When a vendor says "inspired by the brain," ask what was actually borrowed, and what was not.

The Perceptron Years (1957-1969)

A learning machine, a press frenzy, and a hard look

A perceptron was not a program you ran on a laptop. By the account of the Smithsonian, the Mark I Perceptron was a room-size grid of 400 light-registering sensors that could sort images, developed at Cornell by the psychologist Frank Rosenblatt, and the original machine now sits in the Smithsonian's collections. Researchers trained it with yes or no feedback that adjusted how it weighed incoming signals.

Rosenblatt's theory appeared in Psychological Review in 1958, written at the Cornell Aeronautical Laboratory under an Office of Naval Research contract. He framed it as a choice between two views of memory: stored coded images, or information held in connections that change with experience. He took the second side, citing Hebb. The sociologist Mikel Olazaran, in his history of the perceptron controversy, explains that the perceptron was a feedforward network with one layer of adjustable connections, and that Rosenblatt's learning rule guaranteed it could learn any classification its structure could express.

The press went much further than the engineering. At the July 1958 event, the demonstration was a simulation on the Weather Bureau's IBM 704, and the simulated perceptron placed 97 of 100 squares correctly on the left or right of a field. Headlines followed. And Rosenblatt was not alone. At Stanford, soon after Bernard Widrow arrived in 1959, he and his first graduate student, Marcian "Ted" Hoff, invented the Least Mean Squares algorithm. Within a year Widrow used it to train a shoebox-sized device of knobs and switches called ADALINE, an adaptive linear neuron. The two ADALINEs now held by the Smithsonian are a reminder that this was hardware you could hold. Their 1960 paper, "Adaptive switching circuits," appeared in the IRE WESCON Convention Record.

Then came the hard look. Olazaran reports that Minsky and Papert wrote that perhaps as many as a hundred groups were experimenting with the model, with generally disappointing results as tasks got harder. Marvin Minsky and Seymour Papert were also running an MIT laboratory devoted to the symbol-manipulation approach, and, as Olazaran notes while quoting Dreyfus and Dreyfus, drafts of their attack circulated from about 1965. Their book Perceptrons appeared in 1969. Olazaran's reading is careful: the book was the final push in a controversy whose main arguments were already known by the mid-1960s. The common story that the book alone ended neural network research, he argues, was the result of that closure rather than its cause.

This era solved the question of whether a machine could learn from examples. It left a new problem. A single layer of adjustable connections was too weak, and nobody had a reliable way to adjust more than one. The wider consequences for funding are covered in the first AI winter, and the rival approach that was winning attention is in the history of symbolic AI.

For teams today: a spectacular demo and a reliable product are different things. Ask what the system does on the second, harder task, not the first easy one.

The Lean Years (1969-1981)

Funding drifts away, and the missing pieces turn up in other fields

Reading the sources, the 1970s look less like a death than a drought. The Academy's summary says that at the end of the 1960s discouraging theoretical results caused many researchers to suspect these networks would never be of real use. Olazaran adds a puzzle for anyone who wants a villain. When he asked Minsky about the claim that neural network researchers could not get money, Minsky replied, "The story of DARPA and all that is just a myth." His explanation, per the same history, was that there were no good ideas, not that funders refused them.

The pieces that would end the drought were being made elsewhere, mostly without a connection to neural networks. Jürgen Schmidhuber's survey, written by a participant in this history, says efficient error backpropagation in arbitrary networks was apparently first described in a 1970 master's thesis by Seppo Linnainmaa, without reference to neural networks. It adds that Paul Werbos discussed a neural-network version in 1974 and, to Schmidhuber's knowledge, gave the first application specific to neural networks in 1981. Treat the "first" claims as Schmidhuber's judgment. David Rumelhart told Olazaran that his group learned of Werbos's mid-1970s idea only years after publishing, and that it had been hidden as far as he knew.

In Tokyo, Kunihiko Fukushima of the NHK Broadcasting Science Research Laboratories published the neocognitron in Biological Cybernetics in 1980. Its abstract describes a network self-organized by "learning without a teacher" that recognizes shapes regardless of position, with a structure similar to the visual hierarchy proposed by Hubel and Wiesel. It was a layered vision network that learned without supervision.

The same year, John Hopfield moved from Princeton to Caltech, a professorship in chemistry and biology, where he had computer resources to experiment freely. The result of that move comes next.

The era's lesson is awkward. Little was lost to enemies. The field lacked a method for training hidden layers, the computers were slow and the symbolic approach was producing results it could show off, as in the history of expert systems. For the broader pattern, see what the AI winter was.

For teams today: keep the shelved idea on a watch list with a stated condition for revisiting it. Pieces that fail together are often fixed separately, and years apart.

The Connectionist Comeback (1982-1996)

Physicists, psychologists and one reusable algorithm bring the field back

In 1982 Hopfield published "Neural networks and physical systems with emergent collective computational abilities" in the Proceedings of the National Academy of Sciences. His abstract describes a model of simple neuron-like units that produces a content-addressable memory able to return an entire memory from a sufficiently large part of it. The Royal Swedish Academy compares it to searching your mind for a half-remembered word. Its scale is the surprise. Hopfield's network had 30 nodes, 435 connections and fewer than 500 parameters, and a 100-node version was too much for the computer he had. By comparison, the Academy notes, today's large language models can contain more than one trillion parameters.

Hinton, then at Carnegie Mellon, started from Hopfield's network with Terrence Sejnowski. Using Ludwig Boltzmann's nineteenth-century statistical physics, they published the Boltzmann machine in 1985. It had a hidden layer, it learned from examples, and it was fairly inefficient. Olazaran counts it the first solution to learning in multilayer systems, rather slow but encouraging others to try. He also credits a new generation of researchers, many from physics, mathematics and cognitive science, in the University of California, San Diego "Parallel Distributed Processing" group around Rumelhart and James McClelland, with bringing neural networks back into the AI arena.

Then back-propagation. On October 9, 1986, Nature published a four-page paper (received May 1, accepted July 31) by Rumelhart, Hinton and Ronald Williams, and the same year the group's chapter appeared in the PDP volumes from MIT Press. Two details explain why it worked where earlier attempts stalled. Per Olazaran, the group derived it as a generalization of Widrow and Hoff's 1960 rule using the chain rule of calculus, and they replaced the on-or-off step function used by Rosenblatt and Widrow with a continuous, differentiable curve, which turned training multilayer networks into a problem that could be solved. Olazaran also records that David Parker in the United States and Yann LeCun in France had developed similar techniques around the same time, without open priority disputes.

A product followed within three years. At AT&T Bell Laboratories in Holmdel, New Jersey, LeCun and six colleagues trained a back-propagation network to read US Postal Service zip code digits. Their 1989 paper reported a 1% error rate and about a 9% reject rate, and made its point directly: back-propagation networks can handle real image recognition without a large, complex preprocessing stage. They also noted the architecture was highly constrained and designed for the task. Our profile of Yann LeCun's leadership follows that thread.

This era solved training for shallow networks and proved commercial value on narrow jobs. The new problem was depth. Networks with many layers were still hard to train, data was scarce, and competing statistical methods were easier to use, as the history of machine learning shows.

For teams today: a breakthrough is often a small change that unlocks an old idea. Ask what exactly changed, here a smoother activation function and a reusable training rule, before crediting a method as brand new.

The Quiet Years (1990s-2011)

Neural networks work in banks while fashion looks elsewhere

The 1990s are the part of the story that rarely gets told, because it did not look like a victory. The Academy says that during the 1990s many researchers lost interest in artificial neural networks, though Hinton was one of those who continued. A second AI downturn, with expert systems losing ground, is covered in the second AI winter.

Yet the networks were working. The 1998 paper "Gradient-Based Learning Applied to Document Recognition" by LeCun, Leon Bottou, Yoshua Bengio and Patrick Haffner describes a bank check reading system that reads several million checks per day. The paper says an independent test by systems integrators in 1995 showed it beat other commercial amount-reading systems, and that it was integrated into NCR's check readers and fielded in several US banks from June 1996. Convolutional networks, in other words, were quietly sorting money.

Sequences got a fix in 1997. Sepp Hochreiter and Jürgen Schmidhuber introduced Long Short-Term Memory in Neural Computation, a design that can bridge time lags in excess of 1,000 discrete time steps by enforcing constant error flow through special units, with gates that open and close access to it. Plain recurrent networks had failed because the learning signal decayed as it traveled back through time.

The turn came in 2006. Hinton and colleagues, per the Academy, developed a method for pretraining a network with a series of Boltzmann machines in layers, giving the connections a better starting point. The era's remaining problem was evidence. No one had yet shown a deep network beating everything else on a benchmark the whole field respected.

For teams today: the people who stay with an unfashionable method are an asset. Fund a small, patient group on any problem your business will depend on, before the market notices.

The Deep Return (2012-present)

Data and graphics chips prove the old idea

Everything the earlier eras lacked arrived together. In 2012 Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered a deep convolutional network in the ImageNet contest. It won with a top-5 error of 15.3% against 26.2% for the second-best entry, with 60 million parameters, trained for five to six days on two GPUs. The ingredients were old: layered networks from Fukushima, training from the 1980s, and hardware built for video games. What changed was scale. The Academy says the machine learning revolution began around 2010, made possible by access to vast amounts of data and enormous increases in computing power.

The detailed story of ImageNet, the Transformer and large language models belongs to the history of deep learning; the architecture underneath chat tools is explained in the Transformer architecture, and the vocabulary in the neural networks explainer. Recognition arrived late. On October 8, 2024, the Royal Swedish Academy of Sciences awarded the Nobel Prize in Physics to Hopfield and Hinton for foundational discoveries and inventions that enable machine learning with artificial neural networks.

For teams today: treat today's models as the latest layer on a long-running idea. Models change, but a clean dataset, a fair test set and people who know your problem tend to carry over.

Key People

Key Facts

• McCulloch and Pitts published their neuron logic paper in December 1943, in volume 5 of the Bulletin of Mathematical Biophysics.
• The Mark I Perceptron was a room-size grid of 400 light-registering sensors.
• Hopfield's 1982 network had 30 nodes, 435 connections and fewer than 500 parameters.
• Back-propagation's Nature paper was received on May 1, 1986 and issued October 9, 1986.
• By 1996, a convolutional check reader was fielded in US banks and reading several million checks per day.
• AlexNet's top-5 error was 15.3%, against 26.2% for the runner-up.

The Pattern Behind the History

Across six eras, the same four moves repeat. We call the sequence the Picture-Promise-Proof Cycle. It is our synthesis of the history above, not a research finding.

1. Biology lends the picture. Cells and changing connections supplied the blueprint long before any machine could use it.
2. A demo outruns the theory. The 1958 press event promised a machine that would be conscious of its existence while the hardware sorted left from right.
3. A narrow proof gets read as a wide verdict. Single-layer limits became "neural networks don't work," in a field already leaning toward symbols.
4. A training method plus new ingredients brings it back. Backpropagation in the 1980s, then data and graphics chips in the 2010s.

What It Teaches Teams Today

• Judge the second task, not the first. The perceptron aced left versus right, and the harder cases exposed the ceiling.
• Read limits narrowly. A proof about one-layer systems did not settle multilayer ones.
• Stay patient with quiet technologies. Networks read millions of checks a day in the 1990s while headlines ignored them.
• Ask which ingredient is missing. Method, data or computing power: the answer tells you whether to wait or build.
• Expect rediscovery. The same training idea surfaced in several places before it was noticed. Share and document your own work.

Neural networks show that a good idea can wait decades for its evidence and still be right. Ask of any new AI tool which of its ingredients is actually new and which has been waiting for the rest to catch up. For the field around it, see the origin of artificial intelligence.