The History of Transformer Models

By Victor Hoang · Updated Oct 24, 2026

Consider how translation worked in 2016. The best systems read a sentence word by word. They passed a running memory forward at each step.

The transformer is a neural network design that lets every part of an input look directly at every other part. It began on June 12, 2017, when eight researchers, most of them at Google, posted "Attention Is All You Need" on arXiv. The idea of attention was three years older, and the networks it replaced were older still. The throughline: the transformer didn't win because attention was new. It won because it threw away the one-step-at-a-time structure that had held language models back, which let graphics processors (GPUs) and data do the work.

For the wider field, see The History of AI; for how the design works, see the transformer architecture. The 2017 paper's own introduction names the price: this inherently sequential nature precludes parallelization within training examples, which gets worse as sentences get longer. Everything below follows from that one complaint. The product story that came next, GPT-3 and ChatGPT, is told in the history of large language models. This article stays with the architecture.

Timeline at a Glance

The Recurrent Years (1990-2013)

Networks learn language one step at a time

The transformer's nearest real ancestors were networks that read in order. In 1990, Jeffrey Elman published "Finding Structure in Time" in Cognitive Science. His proposal was to feed hidden unit patterns back to themselves, so that a network carried a dynamic memory of what it had just seen. His simulations ran from a temporal version of XOR up to discovering syntactic and semantic features for words. That is a recurrent network learning from the order of its input.

A second thread is easy to miss. In 1992, Jürgen Schmidhuber described two feedforward nets, where the first learns to produce context-dependent weight changes for the second, whose weights vary very quickly. Nearly three decades later, Imanol Schlag, Kazuki Irie and Schmidhuber showed a formal equivalence between linearised self-attention and these early-1990s fast weight controllers, with the programming steps being outer products of what are "today called keys and values." Treat this as a precursor, not a prototype. The equivalence is for linearised variants, not for the softmax attention the 2017 paper used, and the authors are from the same lab that wrote the original. It still shows that the key-value idea had earlier homes.

The workhorse, though, was the gated recurrent net. In 1997 Sepp Hochreiter and Schmidhuber introduced Long Short-Term Memory, which could learn to bridge minimal time lags in excess of 1,000 discrete-time steps. By 2017, the Transformer paper could write that recurrent networks, LSTM and gated recurrent networks in particular, had been firmly established as state of the art in language modeling and machine translation. Our histories of neural networks and deep learning cover this stretch in full.

This era solved memory across time. It left a structural limit: a recurrent model computes step t from step t-1, so you can't process the words of one sentence at once, and a long sentence means a long chain.

For teams today: a design that works can quietly set your ceiling. Look for the step in your own pipeline that must happen one at a time, and ask whether it has to.

The Attention Years (2014-2016)

Recurrent models get a window onto the source

Two papers in September 2014 reshaped machine translation, a problem with its own long story in the history of natural language processing. On September 10, Ilya Sutskever, Oriol Vinyals and Quoc Le showed that one multilayer LSTM could read a sentence into a fixed-size vector and a second could write the translation. The result was 34.8 BLEU on English-to-French, against 33.3 for a phrase-based system. A small trick helped: reversing the order of the source words, which created many short dependencies between source and target and made optimization easier.

Nine days earlier, on September 1, Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio had named the weakness in that design. They conjectured that a fixed-length vector is a bottleneck and let the model automatically soft-search for the parts of the source sentence relevant to each target word. This is attention in an early, widely used neural form. Sequence-to-sequence set the encoder-decoder frame, and Bahdanau's paper gave that frame a way to look back.

Attention then got practical. In August 2015, Minh-Thang Luong, Hieu Pham and Christopher Manning compared a global approach that attends to all source words with a local one that looks at a subset, and reported a 5.0 BLEU gain from local attention over non-attentional systems. In June 2016, Ankur Parikh, Oscar Täckström, Dipanjan Das and Jakob Uszkoreit posted a model that used attention to decompose a natural language inference problem into subproblems solved separately, making it trivially parallelizable. The Transformer paper cites that work among its examples of self-attention in use, and it credits Uszkoreit, a co-author of both, with the proposal to replace RNNs with self-attention.

The problem this era left was subtle. In nearly all these systems, the Transformer authors observed, attention was used together with a recurrent network. The window had been added to a building that still had the old floor plan.

For teams today: patches are valuable and they're also a signal. When the fix you keep adding becomes the most useful part of the system, consider redesigning around it.

After "Attention Is All You Need" (2017-2019)

Eight authors drop the recurrence, then the field splits

The paper was posted to arXiv on June 12, 2017, and published in the proceedings of the 31st Conference on Neural Information Processing Systems (NIPS, now NeurIPS) in Long Beach, California. Its eight authors were Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser and Illia Polosukhin. The paper lists Vaswani, Shazeer and Kaiser at Google Brain, Parmar, Uszkoreit and Jones at Google Research, Gomez at the University of Toronto, and notes that Gomez's and Polosukhin's work was done while at Google Brain and Google Research respectively.

Unusually, the paper says who did what. A footnote states that all eight contributed equally, with a random listing order. It says Uszkoreit proposed replacing RNNs with self-attention and started the effort to evaluate the idea; Vaswani and Polosukhin designed and implemented the first Transformer models; and Shazeer proposed scaled dot-product attention, multi-head attention and the parameter-free position representation. Parmar tuned and evaluated countless variants, Jones built the initial codebase, and Kaiser and Gomez built tensor2tensor, which replaced it.

What the design did was concrete. A recurrent layer needs a number of sequential operations that grows with the sequence, while a self-attention layer connects all positions with a constant number of sequentially executed operations. The authors report 28.4 BLEU on English-to-German, over 2 BLEU above the previous best, and a base model trained in 12 hours on eight P100 GPUs. They say that, to their knowledge, it was the first transduction model relying entirely on self-attention, without sequence-aligned RNNs or convolution. See self-attention for the mechanism.

One caution on dates and numbers. The conference version and the arXiv version differ. The NeurIPS proceedings abstract reports 27.5 BLEU for English-to-German and 41.1 for English-to-French with a 165 million parameter model, while the current arXiv abstract, revised through August 2023, reports 28.4 and 41.8. Cite the version you mean.

The Split: encoder, decoder, both

The original design had an encoder to read and a decoder to write. Within eighteen months, labs began keeping only half of it.

OpenAI's 2018 paper by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever pre-trained a 12-layer decoder-only transformer with masked self-attention heads on unlabeled text and then fine-tuned it for each task. In October 2018 Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova posted BERT, short for Bidirectional Encoder Representations from Transformers, built to pre-train deep bidirectional representations by conditioning on both left and right context. It pushed the GLUE score to 80.5%, a 7.7-point absolute gain, and set new state-of-the-art results on eleven tasks. A year later Colin Raffel and colleagues posted T5, converting all text-based language problems into a text-to-text format.

The decoder-only line became the base of today's large language models, and the story of what it grew into is in the history of large language models. The era solved training speed and made one pre-trained model reusable for many tasks. It created a new dependency: results now tracked the size of the model and the amount of text, so the field needed to know how far that went.

For teams today: a general-purpose architecture means your advantage moves to the data and the evaluation, not the model. Decide which half of your problem is reading and which is writing, and test for each.

The Scaling Years (2020-2022)

Same block, bigger models, new kinds of data

In January 2020, Jared Kaplan and nine co-authors measured what size buys. They found that language model loss follows a power law in model size, dataset size and compute, with trends spanning more than seven orders of magnitude, while architectural details like width or depth had minimal effects within a wide range. For a team choosing where to invest, that was a planning tool. For the transformer, it was a vote of confidence: the block was stable enough that scale mattered more than redesign.

The test came four months later. GPT-3 was an autoregressive model with 175 billion parameters, and its architecture section is a reminder that the transformer kept absorbing efficiency ideas: the model used alternating dense and locally banded sparse attention patterns, similar to the Sparse Transformer. In March 2022, Jordan Hoffmann and 21 co-authors trained over 400 models and concluded that for every doubling of model size, the number of training tokens should also double. Their 70-billion-parameter Chinchilla beat the 280-billion-parameter Gopher on the same compute budget. The architecture hadn't changed; the recipe had.

The transformer also left text. On October 22, 2020, Alexey Dosovitskiy and eleven colleagues posted the Vision Transformer, showing that a pure transformer applied directly to sequences of image patches can perform very well on image classification. Pre-trained on large data, it matched state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. The title, "An Image is Worth 16x16 Words," gives the trick: treat patches as words. Uszkoreit, the Transformer paper's idea-starter, is among the authors, a link between the two papers. ImageNet, the benchmark it was measured on, has its own story in ImageNet and AlexNet.

Biology came next, with a caveat. John Jumper and his co-authors entered AlphaFold in the CASP14 assessment from May to July 2020 and published the method in Nature on July 15, 2021. Its trunk is built from a block they call the Evoformer, which contains a number of attention-based and non-attention-based components, and the paper describes a new equivariant attention architecture. AlphaFold is therefore an attention-based system, not a plain transformer. In CASP14 its structures had a median backbone accuracy of 0.96 Å r.m.s.d.. On October 9, 2024, the Royal Swedish Academy awarded half of the Nobel Prize in Chemistry jointly to Demis Hassabis and Jumper for protein structure prediction, and notes that AlphaFold2 has been used by more than two million people from 190 countries. Broader context lives in the history of generative AI and the concept of foundation models.

This era solved the question of whether one architecture could stretch across language, images and proteins. It created a bill: attention compares every position with every other, so the cost grows quickly with length.

For teams today: when a vendor says a model is better, ask whether it's bigger, trained on more tokens, or genuinely different. Those are three different claims with three different costs.

The Efficiency Race (2019-present)

The cost of looking at everything gets attacked from every side

This era overlaps with the last, because the cost problem appeared as soon as people tried longer inputs. The FlashAttention authors put it plainly: the time and memory complexity of self-attention are quadratic in sequence length. Attacks came from three directions.

Cheaper attention. In April 2019, Rewon Child, Scott Gray, Alec Radford and Ilya Sutskever posted Sparse Transformers, whose sparse factorizations of the attention matrix reduce the cost to O(n√n). Longformer followed in April 2020 with an attention mechanism that scales linearly with sequence length. In November 2019, Shazeer proposed multi-query attention to shrink the memory traffic that makes incremental decoding slow. Then in May 2022 Tri Dao and colleagues took a different route with FlashAttention, an exact method that uses tiling to reduce reads and writes between GPU memory levels and trained GPT-2 about 3 times faster in their tests. FlashAttention-2 in July 2023 reported around 2 times the speed of the original. These advances are part of why a longer context window became affordable.

Sparse experts. Shazeer and six co-authors, including Geoffrey Hinton and Jeff Dean, posted a sparsely gated mixture-of-experts layer in January 2017, months before the transformer, claiming more than 1000x improvements in model capacity with only minor losses in computational efficiency. The two lines merged in the Switch Transformer of January 2021, which simplified the routing, reported up to 7x faster pre-training, and reached trillion-parameter scale. In January 2024, Mixtral 8x7B put the idea in an openly licensed model: each layer has 8 feedforward blocks and a router picks two per token, so each token sees 47B parameters but uses 13B. See mixture of experts for the concept.

Different machinery. In December 2023, Albert Gu and Tri Dao posted Mamba. Their abstract opens by noting that foundation models are almost universally based on the Transformer architecture, then argues that state-space models have lacked content-based reasoning and fixes it by letting the model's parameters depend on the input. They report 5 times higher throughput than Transformers, linear scaling in sequence length, and a 3-billion-parameter Mamba model that matches Transformers twice its size. Six months later, the same authors showed the two families are closely related through a framework they call state space duality, noting that state-space models such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. The pragmatic response was a hybrid: Jamba, posted in March 2024, interleaves blocks of Transformer and Mamba layers and adds mixture-of-experts layers, with strong results up to a 256K context.

The papers do not show attention being retired. They show it being rebuilt, trimmed and blended. The open question is whether any rival matches the transformer at the largest scales.

For teams today: the cost to run a model often matters more than the cost to train it. Ask for inference cost at your real input lengths, and keep your stack portable enough to swap the engine.

Key People

Key Facts

• The Transformer paper was posted on arXiv on June 12, 2017 and has eight authors, listed in random order with equal contribution.
• The paper's footnote credits Jakob Uszkoreit with proposing the replacement of RNNs with self-attention.
• The arXiv abstract reports 28.4 BLEU on English-to-German and 41.8 on English-to-French after 3.5 days on eight GPUs; the NeurIPS proceedings abstract gives 27.5 and 41.1.
• BERT set new state-of-the-art results on eleven NLP tasks, including a GLUE score of 80.5%.
• Chinchilla showed that model size and training tokens should scale equally.
• Mixtral gives each token 47B parameters of access but uses 13B active.

The Pattern Behind the History

Across five eras, one loop repeats. We call it the Patch, Remove, Repeat Cycle. It's our synthesis of the history above, not a research finding.

1. A bottleneck gets named. A fixed-length vector in 2014, sequential computation in 2017, quadratic attention cost in 2022.
2. A patch is bolted on. Attention was added to recurrent networks, and it was usually used together with them.
3. Someone removes what the patch worked around. The Transformer deleted recurrence and convolutions entirely.
4. The new design exposes the next bottleneck. Parallel training made scale possible, and scale made memory and sequence length the new limits.

What It Teaches Teams Today

• Ask what the patch is hiding. If your most important feature is a workaround for the core design, the core design may be the problem.
• Treat the architecture as replaceable. Recurrent nets, transformers and now state-space models have each been the reference design in turn. Keep your data and evaluation portable.
• Credit the whole chain. The 2017 paper stands on 1990s recurrence, 2014 attention and 2016 decomposable attention. Don't fund a breakthrough as if it appeared from nowhere.
• Measure the cost you will actually pay. Training cost, inference cost and input length each have different winners.
• Read the version you cite. Even the Transformer paper's headline numbers differ between the proceedings and arXiv.

The transformer shows that a breakthrough is often a subtraction. The 2017 team didn't add a new part; they removed the one everyone assumed was necessary. Ask of any new AI system which assumption it deleted, and what it quietly made more expensive. For the surrounding story, see the AI timeline.