The History of Natural Language Processing

By Victor Hoang · Updated Oct 24, 2026

Picture New York on January 7, 1954. At International Business Machines (IBM) headquarters, a computer turns Russian sentences into English for reporters. The next morning the story is on the front pages of the major American newspapers.

Natural language processing is the effort to make computers read, translate and produce human language. It started as a practical project in 1947, when Warren Weaver of the Rockefeller Foundation asked whether translation could be treated as a code to break. Its ideas are far older: grammarians were writing formal rules for a language in India more than two thousand years ago. The throughline is a swing between two bets. Write the rules of language by hand, or let the machine learn the patterns from text. Each side won for a while. Each time the deciding factor was how much text and computing power people could get their hands on.

The definition lives in what natural language processing is. This article tells how the field got there, and sits inside The History of AI. The people in the room say the problem will be solved in a few years. It took far longer, and the road there ran through a government report that nearly ended the field, a revolution in how linguists worked, and a decision to count words instead of describing them.

Timeline at a Glance

The Grammarians and Codebreakers (6th century BCE-1940s)

Rules, letter counts and a hand-counted poem

The oldest piece of this story is a grammar. The Indian grammarian Panini, placed by Britannica in the 6th to 5th century BCE (scholars give different dates), condensed the phonetics and grammar of Sanskrit into about 4,000 sutras, short rules in his Ashtadhyayi. Britannica calls the system both descriptive and generative, and notes it has been compared to a Turing machine. Keep precursor and concept apart: Panini described a language for scholars and had no thought of computers. But the form of his work, a compact set of rules that generates correct forms, is the form that rule-based NLP would return to in the twentieth century.

A second precursor came from secret writing. In Baghdad around 850, the polymath al-Kindi wrote A Manuscript on Deciphering Cryptographic Messages. Simon Singh, in an account of the Arab codebreakers, explains that al-Kindi noticed letters appear with varying frequencies in written text, so a codebreaker can count the symbols in a ciphertext and match the most common ones to the most common letters. It's the earliest case we know of in which a language is treated as data to be measured, not only as grammar to be described.

The third precursor is a hand count. On January 23, 1913, Andrey Markov told the Imperial Academy of Sciences in St. Petersburg about his study of Pushkin's Eugene Onegin. Brian Hayes's account in American Scientist describes the method: Markov took the first 20,000 letters, stripped punctuation and spaces, and counted 8,638 vowels and 11,362 consonants. If letters were independent, he would expect 3,731 double vowels. He counted 1,104. The next letter, in other words, depends on the one before it. His interest was probability theory, not language, so this is a precursor to language modeling and not language modeling itself. The long line from this count to chatbots is told in the history of large language models.

What this era left behind was two traditions that would compete for the next century: language as rules, and language as counts. Neither could be tried at scale until electronic computers existed.

For teams today: the two oldest methods in the field are still the two choices on your desk, a written rule or a measured pattern. Know which one a tool is using before you trust it.

After Weaver's Memo (1947-1966)

Translation becomes the first big job for computers

After the Second World War, Weaver, who had worked on computer design problems during the war, wrote to the MIT mathematician Norbert Wiener on March 4, 1947. His letter, quoted in his later memorandum, called translation a serious problem for UNESCO and for peace and wondered whether it could be treated as cryptography. When he looked at an article in Russian, he wrote, he could say it was really English in strange symbols and "I will now proceed to decode". Wiener's reply on April 30 was cool: he feared the boundaries of words in different languages were too vague and the connotations too wide for a mechanical scheme.

Weaver kept going. His July 15, 1949 memorandum, "Translation", reproduced from the Rockefeller Foundation archives, lays out a way to think about ambiguity that reads strangely modern. Look at one word through a mask with a hole one word wide and you can't tell whether "fast" means rapid or motionless. Widen the mask to "N words on either side" and, if N is large enough, the meaning becomes clear. That is a window of context, described in 1949. He also pointed to Claude Shannon's work on information and suggested that languages might share a common base beneath their surface, like towers resting on one basement.

Then came the demonstration. On January 7, 1954, Leon Dostert of Georgetown University and Cuthbert Hurd of IBM showed an IBM 701 translating Russian into English. John Hutchins, the historian of machine translation, reconstructs the event: Paul Garvin supplied the linguistics, IBM's Peter Sheridan did the programming, and the system had just 250 lexical items and six rules, mostly for sentences about organic chemistry. Over sixty sentences were shown. Reports ran on the front pages of the New York Times and other papers on January 8, and Hurd was quoted saying a machine built for translation could be ready within three to five years.

At the same time, a different revolution was happening in linguistics. In 1956 Noam Chomsky of MIT published "Three Models for the Description of Language". Its abstract states that no finite-state Markov process can serve as an English grammar, and that statistical approximations of higher order do not get closer to English as the order rises. His 1957 book Syntactic Structures carried the argument to a wider audience. For NLP the effect was to make formal grammar the respectable center of the field and to cast counting methods as a dead end, a judgment that shaped the field for a long time.

This era set the ambition and the problem at once. Everyone could see why translation mattered. Nobody yet knew how much knowledge a machine would need.

For teams today: a polished demo on a small, chosen sample says little about production. Ask what the demo left out, in the way the 1954 demo left out nearly all of Russian.

The ALPAC Winter (1966-1980s)

A government report ends the funding, and rules take over the labs

By the mid-1960s the sponsors wanted results. The U.S. Department of Defense, the National Science Foundation and the Central Intelligence Agency had funded machine translation projects, and they asked the National Academy of Sciences to take stock. The Automatic Language Processing Advisory Committee, chaired by John R. Pierce of Bell Telephone Laboratories, delivered Language and Machines in 1966. The report says that, while machine-aided translation existed, there was "no immediate or predictable prospect of useful machine translation". It found that translation needs were being met by a small, capable human activity.

Yet the committee did not write off the science. It recommended continued support for computational linguistics, to be treated as a science and not judged by its contribution to practical translation. It also credited machine translation work with real effects on linguistics, noting that computer parsing of ordinary sentences had shown linguists that their assumptions about syntax were mutually inconsistent. The report is remembered as the end of the first machine translation boom, and it is the downturn of the field. It overlaps with the first AI winter and shares its cause: promises made in 1954 were measured against results in 1966.

Work did not stop. It moved indoors, to artificial intelligence labs, and it moved toward meaning and small worlds. In January 1966, Joseph Weizenbaum of MIT published ELIZA, a program in which decomposition rules triggered by keywords in the input shaped its replies, an early program that conversed in natural language (see the first AI chatbots). In 1971, Terry Winograd's MIT thesis, Procedures as a Representation for Data in a Computer Program for Understanding Natural Language, described a system, later known as SHRDLU, that answered questions and carried out commands about a robot with a hand and an eye. Winograd's premise was that a computer can't deal sensibly with language unless it understands what the conversation is about.

That premise was the strength and the limit. The program worked because its world was tiny and fully described. Hand-built grammars and knowledge bases, the approach of symbolic AI and expert systems, grew expensive and brittle as soon as they met ordinary text.

For teams today: a rule-based system that works beautifully on a narrow task can be the right tool and a trap at once. Budget for the cost of maintaining the rules, not just writing them.

The Statistical Turn (1980s-1990s)

Counting wins, and the data arrives

The comeback began with speech, where statistical methods had already proven their value. In 1990 a group of eight researchers at IBM's Thomas J. Watson Research Center, Fredrick Jelinek among them, turned the same idea on translation. Their paper, "A Statistical Approach to Machine Translation", points back to Weaver: in 1949 he had suggested using statistical methods and ideas from information theory, and the authors argue that researchers abandoned the idea because computers were too weak and machine-readable text too scarce. By 1990, they write, computers were five orders of magnitude faster than in 1950.

The data came from the Canadian parliament. Its proceedings are kept in English and French by law, and the IBM team obtained about 100 million words of English with the French equivalents, from which a statistical algorithm extracted about three million sentence pairs, around 99% of them true translations of one another. No grammar of French was written. The model learned from the pairs.

Statistical methods also needed labeled text for parsing and tagging. At the University of Pennsylvania, Mitchell Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz built the Penn Treebank. Their 1993 paper reports a corpus of over 4.5 million words of American English; during the first three-year phase (1989-1992) it was tagged for part of speech, and more than half was annotated for skeletal syntactic structure. A shared, annotated corpus let research groups compare methods on the same text, a function ImageNet would later serve for vision.

Chomsky's 1956 argument had not been refuted so much as outflanked. Engineers did not need a perfect grammar of English to translate a sentence or tag a word, and they could measure their progress. The new problem was dependence on data: a method that learns from text is only as good as the text it gets, and Hansard-style parallel corpora existed for few language pairs.

For teams today: this is the lesson of the data advantage. When a method that was dismissed becomes feasible, the deciding factor is often access to data, not cleverness.

The Vector Years (2000s-2016)

Words become numbers that carry meaning

Counting methods had a weakness. A model built on counts of short word sequences cannot say anything about a sequence it never saw. In 2003, Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin of the Université de Montréal proposed to fight that curse of dimensionality by learning a distributed representation for each word, so that a sentence never seen before still scores well if it is made of words similar to ones the model has seen. They reported that this improved significantly on state-of-the-art n-gram models.

In 2011, Ronan Collobert, Jason Weston and colleagues at NEC Laboratories proposed a unified neural network for tasks including part-of-speech tagging, chunking, named entity recognition and semantic role labeling. The pitch was to avoid task-specific engineering and learn internal representations from vast amounts of mostly unlabeled text, in place of hand-designed features for each job.

In January 2013, Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean at Google posted word2vec, reporting large accuracy gains at much lower computational cost, with high-quality word vectors learned from a 1.6 billion word dataset in less than a day. Cheap, fast training made embeddings a standard part of every NLP toolkit, and the older question of how to split text into pieces, covered in tokenization, became a design choice rather than an afterthought. This sits inside the broader shift described in the history of neural networks and the history of deep learning.

The era solved the sparse-data problem and ended the need for hand-crafted features for many tasks. It left a new one: a vector for "bank" that blurs the river and the lender.

For teams today: a general-purpose representation can replace a pile of task-specific engineering. Before building custom features, ask whether a shared embedding already captures them.

The Pretraining Age (2017-present)

Attention, scale and a field that merges into language models

In June 2017, eight researchers posted "Attention Is All You Need", proposing the Transformer, a network based solely on attention mechanisms, with no recurrence or convolutions. It reported 28.4 BLEU on English-to-German translation, more than 2 BLEU above earlier best results. Fittingly, the benchmark was translation, the task Weaver had named seventy years before.

From there the story belongs to other articles. The Transformer's own lineage is told in the history of transformer models. The path from the Transformer through pretraining, GPT-3 and ChatGPT is told in the history of large language models, and the wider wave in the history of generative AI. What matters for the NLP story is what happened to the field's old separate jobs. Tagging, parsing, translation, summarizing and question answering, each once its own research area with its own methods, are now often handled by one large language model given a plain-language instruction. Interfaces that talk, covered under conversational AI, are the visible result.

Old questions return in new form. Weaver worried about meaning and context; today's models still make fluent errors. Chomsky asked what knowing a language means; the debate over whether prediction equals understanding is open.

For teams today: the tools changed and the hard parts did not. Define what a correct output is, test on your own text, and keep a human check where mistakes are costly.

Key People

Key Facts

• Markov's 1913 study counted 8,638 vowels and 11,362 consonants in 20,000 letters of Eugene Onegin.
• The 1954 Georgetown-IBM demonstration used 250 lexical items and six rules.
• The ALPAC report of 1966 was chaired by John R. Pierce of Bell Telephone Laboratories and recommended supporting computational linguistics as a science.
• IBM's 1990 translation work used about 100 million words of English and about three million sentence pairs from Canadian parliamentary proceedings.
• The Penn Treebank held over 4.5 million words of American English.
• word2vec learned vectors from a 1.6 billion word dataset in less than a day.

The Pattern Behind the History

Across these eras, the same four moves repeat. We call the sequence the Ambition-Evidence Swing. It's our synthesis of the history above, not a research finding.

1. A bold promise sets the ambition. In 1954 a handful of rules produced predictions of a solved problem within years.
2. A reckoning measures it. The 1966 ALPAC report put costs and results on paper and cut the funding.
3. The method swings with the available evidence. Hand-written rules led while data was scarce; counts took over when millions of sentence pairs became machine-readable; learned vectors followed when compute allowed.
4. A shared dataset or document resets direction. The Penn Treebank and the ALPAC report each changed what researchers worked on.

What It Teaches Teams Today

• Ask which side of the swing a tool is on. Rules, counts or learned representations fail in different ways, and your testing should match.
• Distrust the small demo. Six rules and 250 words made headlines; the real language was thousands of times bigger.
• Treat data as the asset. The Canadian Hansards and the Penn Treebank mattered more to their eras than any single clever idea.
• Fund the unglamorous science. ALPAC cut translation funding and still recommended supporting computational linguistics; the field's later gains came from that groundwork.
• Keep your evaluation set portable. Methods changed more than once in this history; the test of whether the output is right stayed with the work.

Natural language processing shows a field that kept the same goal for seventy-plus years and changed its method several times. Ask of any language tool you buy: does it work from rules, from counts or from learned patterns, and what does it do on your own text? For the surrounding story, see the AI timeline.