The History of Machine Learning

By Victor Hoang · Updated Oct 23, 2026

Picture an International Business Machines (IBM) 704 in the late 1950s, playing checkers against itself hour after hour. It adjusts the weights it uses to judge a board. Arthur Samuel reported the result. It had learned to play a better game than the person who wrote the program.

Machine learning, the practice of building systems that improve from examples instead of hand-written rules, took its modern shape in the 1950s. Alan Turing sketched "learning machines" in 1950. Frank Rosenblatt described the perceptron in 1958. Samuel published his checkers-learning results in the IBM Journal in 1959. The throughline: the field kept changing who writes the knowledge, first statisticians, then engineers, then the data itself. Each change moved the hard work somewhere new.

That is the scene most histories start from, but the mathematics under it is older, and the story of what came after is the one that matters for a team deciding how to use learning systems today. For the wider field, see The History of AI and the AI timeline.

Timeline at a Glance

Statistics Before Machines (1763-1949)

Probability and curve-fitting supply the math, not the name

In 1763 the Royal Society's Philosophical Transactions printed an essay found among the papers of the late Rev. Mr. Bayes and communicated by Richard Price in a letter to John Canton. It was read to the Society on December 23, 1763. Its question was how to judge the probability of an event when all you know is how often it has and hasn't happened in the past. That is the core move of learning from examples: update a belief when evidence arrives.

Fitting a curve to messy measurements came next. According to the statistician Stephen Stigler, Adrien-Marie Legendre published the method of least squares in 1805 with a clear explanation and a worked example. Carl Friedrich Gauss published it in 1809 and claimed he had used it since 1795, which set off one of the most famous priority disputes in the history of science. Stigler lays out the evidence on both sides. For our purposes the date to remember is 1805, the first print appearance, and the point is that astronomers needed a rule for choosing the best line through imperfect observations.

Then came classification. Cortes and Vapnik, in their 1995 paper, write that R.A. Fisher suggested the first algorithm for pattern recognition in 1936. He worked with two populations of measurements and derived a decision function that separates them, and recommended a linear version when data was scarce.

Be clear about what these are. Bayes, Legendre and Fisher are precursors. Nobody called their work machine learning, and no machine ran it. What they gave the field was the vocabulary of estimation, error and generalization that later learning programs would borrow.

For teams today: much of what gets sold as AI is still regression and classification with a modern wrapper. Ask what the underlying model is before you pay for the label.

The Learning Machine (1950-1966)

Turing imagines it, Rosenblatt and Samuel build it

In 1950 Turing wrote a section of "Computing Machinery and Intelligence" titled Learning Machines. Rather than program an adult mind, he asked why not try to produce a program that simulates the child's, then educate it. He pictured the child brain as "Rather little mechanism, and lots of blank sheets." He also described teaching with rewards and punishments, noted that he had experimented with one such child machine, and observed that a teacher would often be largely ignorant of what was going on inside. Both ideas, reward-driven training and opaque internals, are still central. For the paper's wider story, see the history of the Turing Test.

The field then got its name and a meeting place at Dartmouth in 1956. Two years later, Frank Rosenblatt of the Cornell Aeronautical Laboratory published "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain" in Psychological Review. The work was sponsored by the Office of Naval Research. Rosenblatt reported that simulations of bivalent perceptron systems on the Laboratory's IBM 704 had borne out the theory's main predictions. It was a network whose connections changed with experience, the ancestor of the neural networks in use today.

The most careful early demonstration of learning came from a game. Arthur Samuel's paper, published in the IBM Journal in July 1959, studied two machine-learning procedures using checkers. His abstract says enough work had been done to verify that a computer could learn to play a better game than the person who wrote the program, and that it could do it in 8 or 10 hours of machine-playing time, given the rules, a sense of direction, and a redundant list of parameters whose correct weights were unknown.

Samuel made a design choice that reads like advice. He separated a "Neural-Net Approach" that learns through reward and punishment, which he thought should eventually give general-purpose learning machines but needed far larger networks than anyone could build, from a more efficient, specialized approach that required reprogramming for each application. He used the second. He then compared rote learning, which saved board positions with their scores, against learning by generalization, which adjusted an evaluation polynomial. Rote learning helped most when results of an action were long delayed or highly specialized techniques were needed. Generalization helped most when the possible permutations were large and consequences came quickly.

The era solved a conceptual problem: machines could improve from experience. It left a practical one, which is that every success was narrow and hand-tuned to one task.

For teams today: Samuel's checkers worked because the rules were fixed, the goal was clear, and results were easy to score. Pick learning projects that share those three traits before you tackle open-ended ones.

The Quiet Craft (1967-1989)

Nearest neighbors, decision trees and a theory, while expert systems took the spotlight

This era spans the first AI winter and overlaps the expert systems boom, when most AI money chased hand-written rules, and the symbolic AI tradition set the agenda. Learning work continued, with methods that were simple to describe and sturdy in practice.

In January 1967, Thomas Cover and Peter Hart published "Nearest neighbor pattern classification" in IEEE Transactions on Information Theory. The rule is plain: classify a new point as whatever its nearest previously classified neighbor was. Their result is that, with enough data, its error rate is bounded above by twice the Bayes probability of error, the minimum any rule could achieve. In their words, half the classification information in an infinite sample is contained in the nearest neighbor.

Decision trees gave learning a form people could read. J. Ross Quinlan's 1986 paper "Induction of Decision Trees" in the journal Machine Learning traces ID3 to a line of programs descended from Hunt's Concept Learning System. ID3 (Quinlan, 1979, per the paper) was developed in response to a challenge posed by Donald Michie: decide from pattern-based features alone whether a chess endgame position was lost within a fixed number of moves. ID3 swapped CLS's cost-driven lookahead for an information-driven choice of what to test next. Quinlan's paper opens by noting that machine learning has been a central research area since AI became a discipline in the mid-1950s, and that building knowledge-based systems by inductive inference from examples had already worked in practical applications.

In November 1984 Leslie Valiant published "A Theory of the Learnable" in Communications of the ACM. He treated learning as knowledge acquisition without explicit programming and set out a precise methodology for asking what can be learned in a reasonable (polynomial) number of steps. It gave the field a way to ask what a learner could and couldn't do. In 1986, the back-propagation paper in Nature (covered in the history of AI) showed how multilayer networks could learn their internal features.

What did this era solve? It showed learning could work on real data with understandable methods, and it began a theory. What did it leave? A bottleneck that expert-system builders knew well: the examples, not the algorithm, were now the scarce input.

For teams today: start with the simple baseline. Nearest neighbors and decision trees are easy to explain, and a complex model has to beat them to earn its cost.

The Statistical Turn (1990-2011)

Margins, ensembles and big databases make learning a business tool

By the 1990s, learning from data had changed from a side branch into the way many AI systems were built. The Stanford authors of the 2021 foundation-models report put it plainly: the rise of machine learning within AI started in the 1990s, marking a shift from specifying how to solve a task to letting a learning algorithm induce it from data.

Vladimir Vapnik led the theoretical side. Cortes and Vapnik's 1995 paper Support-Vector Networks, in Machine Learning, says the conceptual part of the optimal-hyperplane problem was solved in 1965 for separable classes, and that a 1992 paper by Boser, Guyon and Vapnik showed the order of operations could be swapped, which made rich decision surfaces affordable to compute. Their 1995 extension handled data that can't be separated without errors, and they tested it against other methods on handwritten digit sets, including a US Postal Service database of 7,300 training and 2,000 test patterns.

Ensembles were the other thread. Leo Breiman's 2001 paper "Random Forests" describes a forest as many tree predictors, each grown from a random vector, that vote for the most popular class. He reported error rates that compare favorably to AdaBoost, the boosting method of Freund and Schapire, and handle noise better. Forests and boosted trees are now widely used on tables of business data.

Learning also went to work on databases. The phrase "knowledge discovery in databases" was coined at the first KDD workshop in 1989, according to Fayyad, Piatetsky-Shapiro and Smyth's 1996 overview in AI Magazine. They describe data growing faster than people could analyze it, and cite examples such as a system called SKICAT classifying sky objects from 3 terabytes of survey images. This is where machine learning met the corporate data warehouse.

The era's public showpiece was a contest. On October 2, 2006, Netflix released about 100 million movie ratings and offered a $1 million Grand Prize. On July 26, 2009, the team BellKor's Pragmatic Chaos submitted the winning entry, a 10.06% improvement over the contest baseline, and a competing team's near-identical entry arrived twenty minutes later. Teams had merged and blended many models to get there. That showed ensembles could win, and that the last few percent cost a great deal of engineering.

This era solved the question of how to get reliable accuracy from labeled data, which is what supervised learning means in practice. It created a new dependency: hand-built "features." Someone still had to decide which measurements to feed the model.

For teams today: gradient-boosted and forest models on your own tables are still the strongest first try for scoring, churn and forecasting. Put the effort into clean features and honest testing.

The Deep Learning Breakthrough (2012-2016)

Bigger data and graphics chips let networks learn their own features

The Stanford foundation-models report dates a revival of deep neural networks to around 2010, fueled by larger datasets, more computation (notably GPUs) and greater audacity. Networks trained on raw inputs such as pixels let higher-level features emerge during training, a process called representation learning. The report points to AlexNet on the ImageNet dataset as the seminal result, and notes that the same network architecture could now serve many applications instead of a custom feature pipeline for each.

This article treats deep learning as one era among several, because it changed how features are made rather than the learning idea itself. For the dated detail, see the history of deep learning, the deep learning guide and the history of AI pillar.

What did it solve? Perception: images, speech and translation. What did it leave? A hunger for labeled data and compute, plus models that were hard to explain.

For teams today: if your input is an image, audio file or scanned document, a pretrained network will usually beat hand-built rules. Check whether you have labeled examples before you check vendors.

The Foundation Model Age (2017-present)

One pretrained model gets adapted to many tasks

In 2021, a Stanford group led by Percy Liang described models trained on broad data, generally using self-supervision at scale, that can be adapted to a wide range of downstream tasks, and gave them the name foundation models. They summarized the significance in two words: emergence and homogenization. Emergence means behavior is induced rather than explicitly constructed. Homogenization means the consolidation of methods: machine learning homogenized learning algorithms, deep learning homogenized model architectures, and foundation models homogenize the model itself. They warn this creates single points of failure.

The AI timeline covers the Transformer, GPT-3 and ChatGPT in detail. Notice what the long view adds. In our reading, Samuel's two approaches have rejoined: huge networks that learn general behavior, then specialized adaptation for each use. Reinforcement learning, the reward-and-punishment idea Turing and Samuel described, was part of how ChatGPT was trained, as the history of AI notes. And unsupervised learning, finding structure without labels, is the cousin of the self-supervision the 2021 report describes.

The new problem is not accuracy on a benchmark. It's governing a general model once everyone can build on it: who owns errors, what it may touch, and how you test something that can do many things.

For teams today: treat a foundation model like a hire with no track record at your company. Define the job, test it on your own cases, and keep a person accountable for what leaves the building.

Key People

Key Facts

• Samuel's 1959 abstract said a program learned to play better checkers than its author in 8 or 10 hours of machine-playing time.
• Cover and Hart showed the nearest neighbor rule's error is bounded above by twice the Bayes error in large samples.
• The phrase knowledge discovery in databases was coined at the first KDD workshop in 1989.
• The Netflix Prize released about 100 million ratings on October 2, 2006, and the winning entry on July 26, 2009 reached a 10.06% improvement.
• Cortes and Vapnik tested support-vector networks on a US Postal Service database of 7,300 training patterns.
• The 2021 foundation-models report names emergence and homogenization as the two defining traits.

The Pattern Behind the History

Across six eras, four moves repeat. We call the sequence the Estimate-Learn-Scale-Absorb Loop. It's our synthesis of the history above, not a research finding.

1. Estimate on small, clean problems. Bayes, Legendre and Fisher worked on bounded questions where the data was limited and the error could be written down.
2. Learn when the rules can't be written. Samuel's evaluation weights and Quinlan's trees replaced hand-built logic wherever experts couldn't spell out the steps.
3. Scale with data and compute. Support-vector methods, forests, the Netflix blend and deep networks each worked because more examples and more computing power were available.
4. Absorb the method into the platform. Each technique stopped being a project and became a library, a service or a default, and the scarce skill moved to framing the problem and judging the output.

What It Teaches Teams Today

• Start with a simple baseline. Nearest neighbors, trees and linear models are cheap to explain, and anything fancier has to beat them on your data.
• Choose tasks with clear scoring. Samuel's checkers had fixed rules and an obvious goal. Define yours before you train.
• Budget for data, not just models. From ID3 to ImageNet, the examples were the scarce input.
• Expect the last gains to be expensive. The Netflix Prize needed blended teams for the last few percent.
• Name an owner for each model. The more general the model, the more it needs someone accountable for its mistakes.

Machine learning has spent two centuries moving the work of knowing from people to data. The useful habit is to ask, for any new tool, what it learned from, how that was tested, and who answers when it's wrong.