The History of Large Language Models

By Victor Hoang · Updated Oct 24, 2026

On January 23, 1913, Andrey Markov addressed the Imperial Academy of Sciences in St. Petersburg. He had spent hours sorting the vowels and consonants of Pushkin's Eugene Onegin.

A large language model is a neural network trained on huge amounts of text to predict what comes next. Its story starts earlier than most people expect: in 1913, when the Russian mathematician Markov counted letters in a poem by hand. The modern form arrived in 2017 with the transformer. It grew through GPT-3 in 2020 and reached the public in November 2022 with ChatGPT. The throughline: every era made the same bet, that predicting the next piece of text well enough forces a model to learn something useful. Each era's limit was set by what could be counted, stored and computed at the time. For the wider field, see The History of AI.

Over a century later, systems built on the same basic act, predicting the next symbol, are drafting emails, writing code and answering questions for hundreds of millions of people. This article follows the line between those two scenes. For the definition, see what large language models are.

Timeline at a Glance

The Counting Years (1913-2002)

Statisticians count letters long before anything learns

The earliest ancestors of the language model were not neural and not even computers. In his 1913 study, Markov took the first 20,000 letters of Eugene Onegin, stripped the spaces and punctuation, and tallied vowels and consonants. He counted 8,638 vowels and 11,362 consonants, and 1,104 vowel-vowel pairs, far fewer than independence would predict. The point was mathematical: letters in a text depend on their neighbors. Markov was not building a language model, and nobody called it one. But the idea that the next symbol can be predicted from the ones before it is the seed of everything below.

Claude Shannon turned that seed into a measurement. His 1951 paper describes a method for estimating the entropy and redundancy of a language by testing how well the next letter can be predicted when the preceding text is known. He notes that his earlier 1948 work put the entropy of English at roughly 2.3 bits per letter when effects up to eight letters are counted, and that longer-range effects, up to 100 letters, reduce it to something of the order of one bit per letter. In plain terms, English is very predictable, and a model that exploits that predictability can compress or generate it. These were measurements of text, not systems that wrote it, so treat them as precursors.

The idea reached practical machines through speech and translation. In 1990, a team at IBM's Thomas J. Watson Research Center that included Frederick Jelinek published a statistical approach to translating French into English. Their paper recalls that Warren Weaver suggested statistical methods and ideas from information theory in 1949, that researchers abandoned the approach, and that the obstacles were weak computers and a shortage of machine-readable text. They concluded that it was time to give statistical methods a chance in machine translation, since corpora were now plentiful. This is the same shift described in the history of machine learning, where data began to beat hand-written rules after the winters covered in the second AI winter.

This era solved the question of whether language has statistical structure worth modeling. It created a new problem: counting word sequences breaks down as the sequences get longer, because most long sequences never appear in the training text. Fixing that took neural networks.

For teams today: the first useful version of a technology is often a measurement, not a product. Before you build a model of your own data, count it. Simple frequency tables of your support tickets or sales emails can show whether there's a pattern worth learning.

The Vector Years (2003-2016)

Words turn into numbers, and the numbers start to mean something

In February 2003, the Journal of Machine Learning Research published a paper from the University of Montreal by Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin. It named the obstacle plainly: the curse of dimensionality, since a word sequence used for testing is likely to differ from every sequence seen in training. Their answer was to learn a distributed representation for each word, so that a sentence never seen before still gets high probability if it is made of words similar to ones the model knows. The authors reported that it improved on the best n-gram models on two text corpora and could use longer contexts. Here the modern language model is born in outline, years before the hardware could do it justice. It belongs to the theory-first stretch told in the history of neural networks.

The practical breakthrough came in January 2013, when Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean posted word2vec. They proposed two architectures for computing word vectors from very large datasets and reported that it takes less than a day to learn high quality word vectors from a 1.6-billion-word dataset. Embeddings stopped being an expensive research project and became a default building block.

Sequences were the next hurdle. In September 2014, two papers changed how machines handled long inputs. Ilya Sutskever, Oriol Vinyals and Quoc Le showed that one LSTM could read a sentence into a fixed-size vector and a second could write the translation, reaching a BLEU score of 34.8 on English-to-French against 33.3 for a phrase-based system. Days earlier, Dzmitry Bahdanau, Kyunghyun Cho and Bengio had argued that a fixed-length vector is a bottleneck and let the model softly search the source sentence for the parts relevant to each target word. That "attention" step is the hinge of the whole story. Both papers come from the deep learning wave described in the history of deep learning.

The era solved representation and gave translation its first neural wins. It left two problems: recurrent networks read text one word at a time, which made them slow to train, and they still struggled with very long passages.

For teams today: a bottleneck in one design is a research prompt for the next. When your pipeline forces everything through one narrow step, whether a single summary field or one reviewer, ask what it would look like to let downstream steps look back at the raw material.

After "Attention Is All You Need" (2017-2019)

The transformer arrives, and pre-training becomes the method

On June 12, 2017, Ashish Vaswani and seven co-authors, most at Google Brain and Google Research, posted "Attention Is All You Need." It proposed the transformer, which dispensed with recurrence entirely and relied on attention alone. The abstract reports 28.4 BLEU on English-to-German translation, over 2 BLEU above the previous best, and 41.8 BLEU on English-to-French after training for 3.5 days on eight GPUs. Because every word could be processed in parallel, the design suited the graphics hardware that deep learning already relied on. For how it works, see the transformer architecture.

Within eighteen months the field found the recipe that followed. In February 2018, Matthew Peters and colleagues posted ELMo, showing that vectors drawn from a pre-trained bidirectional language model could significantly improve the state of the art across six challenging language problems. OpenAI described a two-stage approach in 2018: first train a transformer on a very large amount of text with language modeling as the signal, then fine-tune on small supervised datasets. This model is now called GPT-1. In October 2018, Jacob Devlin and colleagues at Google posted BERT, which reported a GLUE score of 80.5%, a 7.7-point absolute improvement, with new records on eleven tasks. Two families emerged: BERT-style models that read text in both directions, and GPT-style models that predict the next word.

The next public step was a dispute, not a benchmark. OpenAI announced GPT-2 in February 2019, a transformer with 1.5 billion parameters trained to predict the next word in 40GB of Internet text, and said that, citing concerns about malicious uses, it was not releasing the trained model. A later report from OpenAI authors says the company released the 124 million parameter model in February, the 355 million model in May, the 774 million model in August, and the 1.5 billion model in November, a staged release meant to give time to study risks. Whether that caution was right is still argued, but the debate over who should get a powerful text generator started here.

This era solved training speed and made one pre-trained model reusable for many jobs. It created a new dependency: quality now rose with the size of the model and the amount of text, so the next question was how far that went.

For teams today: a general model plus small amounts of your own data is now the normal pattern. Spend your effort on the data you add, not on training from scratch.

The Scaling Years (2020-2022)

Bigger models and more data turn out to follow a curve

By early 2020, researchers had stopped guessing about size and started measuring it. On January 23, 2020, Jared Kaplan, Sam McCandlish and eight co-authors posted a study of scaling. They found that cross-entropy loss follows a power law in model size, dataset size and training compute across more than seven orders of magnitude, while details like network width or depth mattered little within a wide range. That gave labs a planning tool: more compute reliably bought a better model.

Four months later came the test. On May 28, 2020, Tom Brown and 30 co-authors posted the GPT-3 paper, describing an autoregressive language model with 175 billion parameters, ten times more than any previous non-sparse language model. Applied with no fine-tuning, only instructions and examples typed into the prompt, it was sometimes competitive with fine-tuned systems. The authors also noted that few-shot learning still struggled on some datasets and that training on large web corpora raised methodological issues. Prompting, not retraining, became the way people used a model.

The rest of the era corrected two things. In March 2022, DeepMind's Jordan Hoffmann and 21 co-authors argued that large models were undertrained. Their 70-billion-parameter Chinchilla used roughly the same compute as the 280-billion-parameter Gopher but about four times more data, and for every doubling of model size the number of training tokens should also be doubled. Size alone was not the lever; the data-to-size ratio mattered too. And in January 2022, Jason Wei and colleagues showed that generating intermediate reasoning steps, a chain of thought, substantially improves complex reasoning in large models, with a 540-billion-parameter model reaching state-of-the-art accuracy on a grade-school math benchmark using eight examples.

The biggest correction was about usefulness. A raw model continues text; it does not follow orders. On March 4, 2022, Long Ouyang and 19 co-authors at OpenAI posted InstructGPT. The recipe collected human demonstrations to fine-tune GPT-3, then human rankings to tune it further with reinforcement learning from human feedback. In their evaluations, labelers preferred the 1.3 billion parameter InstructGPT over the 175 billion parameter GPT-3, despite it having 100 times fewer parameters. Alignment, in other words, could be worth more than another jump in size.

The era solved capability and, with human feedback, usability. It created expense and risk: training now meant very large compute budgets, and a model that sounded helpful could still be wrong.

For teams today: when a vendor leads with parameter counts, ask about training data and evaluation instead. The 2022 results show that how a model was trained and tuned can matter more than its size.

The ChatGPT Moment (2022-2023)

A chat box turns a research system into a public habit

OpenAI launched ChatGPT in November 2022 as a free research preview, a model it described as one that interacts in a conversational way, able to answer follow-up questions, admit mistakes and reject inappropriate requests. It was trained with the same reinforcement learning from human feedback as InstructGPT, applied to dialogue and fine-tuned from a model in the GPT-3.5 series. OpenAI listed its own limits up front: it sometimes writes plausible-sounding but incorrect or nonsensical answers. Nothing in the underlying method was new in kind. The interface was. This era overlaps with the Scaling Years, since the research and the product ran side by side; earlier chat programs, from ELIZA onward, had never worked on open-ended questions at this level.

The adoption numbers are now documented by OpenAI itself. A paper by OpenAI economists reports that by July 2025, 18 billion messages were being sent each week by 700 million users, around 10% of the global adult population. That is the vendor's own figure for its own product, and it says nothing about the other products in the field.

Capability and openness moved fast in 2023. On March 15, 2023, OpenAI posted the GPT-4 technical report, describing a large-scale multimodal model that accepts image and text inputs and produces text outputs, including a simulated bar exam score around the top 10% of test takers. The same report says it is less capable than humans in many real-world scenarios. Images, audio and video took a parallel path, covered in the history of generative AI.

Meanwhile, the weights began to travel. On February 27, 2023, Hugo Touvron and 13 co-authors at Meta posted LLaMA, models from 7 to 65 billion parameters trained on publicly available datasets only. LLaMA-13B outperformed the 175 billion parameter GPT-3 on most benchmarks. On July 18, 2023, Meta and Microsoft announced Llama 2, available free of charge for research and commercial use, with model weights and starting code, and an acceptable use policy that restricts some uses. The accompanying paper describes models from 7 billion to 70 billion parameters, including a dialogue-tuned Llama 2-Chat. Companies could now run capable models on their own servers, a change that created the operational discipline covered in what LLMOps is.

This era solved distribution: anyone with a browser, and soon anyone with a server, could use a language model. It created questions about accuracy, data privacy and who is accountable for a model's mistakes, the same questions that AI moving from research labs to work keeps returning to.

For teams today: the interface did more for adoption than any benchmark. If you want people to use a tool, put it where they already work, and write down what it's not allowed to do.

The Age of Reasoning and Agents (2024-present)

Models spend compute on thinking, and start acting on their own

The next turn was about when the compute is spent. Until 2024, most of the cost fell on training; the answer itself arrived in a moment. On September 12, 2024, OpenAI released an early version of o1, saying that its performance consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute). On the 2024 AIME math exams, OpenAI reported that GPT-4o solved an average of 12% of problems while o1 averaged 74% with a single sample per problem. These are the vendor's own evaluations. The direction is the point: the model writes a long chain of thought before it answers, building on the 2022 chain-of-thought work above.

In January 2025, DeepSeek-AI posted a paper reporting that reasoning ability can be incentivized through pure reinforcement learning without human-labeled reasoning examples, and that behaviors such as self-reflection and verification emerged. Reasoning was no longer a feature of one lab's closed product.

Length of memory grew too. In March 2024, Google's Gemini 1.5 report described a model that recalls and reasons over millions of tokens of context, with near-perfect retrieval (above 99%) up to at least 10 million tokens, compared in the same paper with 200,000 tokens for Claude 3.0 and 128,000 for GPT-4 Turbo. Retrieval on a test is not the same as reliable reasoning over a long document, and the paper does not claim it is.

The third strand is action. In October 2022, Shunyu Yao and colleagues proposed ReAct, in which a model interleaves reasoning traces with task-specific actions, such as querying Wikipedia or an interactive environment. That pattern underlies today's AI agents, systems that call tools and carry out multi-step work. Where earlier eras ended with a model that answered, this one ends with a model that does things inside business systems.

The history is still open. The cost, reliability and oversight of these systems are the live questions, and they are the operational ones in what LLMOps is.

For teams today: treat each new capability as something to test on your own tasks. Keep a small set of real examples, rerun them whenever a model changes, and decide in advance what a human must still approve.

Key People

Key Facts

• Markov's 1913 count of 20,000 letters found 1,104 vowel-vowel pairs, far fewer than independence predicts.
• The transformer paper reported 28.4 BLEU on English-to-German translation, a gain of over 2 BLEU.
• GPT-3 had 175 billion parameters, ten times more than any previous non-sparse language model.
• Chinchilla, at 70 billion parameters trained on about four times more data than Gopher, beat the 280 billion parameter model.
• Labelers preferred a 1.3 billion parameter InstructGPT to the 175 billion parameter GPT-3.
• OpenAI's economists report 700 million weekly ChatGPT users by July 2025.

The Pattern Behind the History

Across five eras, the same loop turns. We call it the Count-Learn-Scale-Align Cycle. It's our synthesis of the history above, not a research finding.

1. Count. Someone measures how predictable text is, from Markov's letters to Shannon's entropy to IBM's translation tables.
2. Learn. A method lets the model generalize beyond what it has seen, from word vectors in 2003 to attention in 2014.
3. Scale. A cheaper architecture, the transformer, lets more data and compute flow through, and scaling studies show what that buys.
4. Align. Human feedback and reasoning training make the larger model useful, safe enough to ship, and able to act.

What It Teaches Teams Today

• Separate the ingredients. Data, compute, method and tuning each moved the field at different times. Ask which one limits your project.
• Measure before you model. Every era began with someone counting something. Know your baseline before you buy a model.
• Judge on your own tasks. Benchmarks from vendors measure their chosen tasks. Keep your own test set and rerun it when models change.
• Plan for fast churn. The leading architecture changed three times in this history. Keep your data, prompts and evaluations portable.
• Decide where a human stays in the loop. The models that act, not just answer, raise the cost of mistakes.

Large language models show that a simple goal, predict what comes next, can carry a long way when the counting, the data and the hardware catch up with it. Ask of any new model: which of those ingredients is actually new here? For the field around it, see the history of the Turing test and the AI timeline.