Attention, transformers, and the language models

Google's 'Attention Is All You Need' (2017) defined the transformer; GPT-3 (2020) showed scale worked; ChatGPT (2022) put a conversational model in front of a hundred million people in two months.

Why it matters

For the first time, machines handle language — the medium of thought — fluently enough to be useful. The questions this raises about knowledge, work and intelligence are exactly the ones the Ideas section has been asking for 2,500 years.

Threads

This event sits on Computation and Knowledge.

Related events

• 2012

  AlexNet wakes up deep learning

  A neural network trained on two gaming GPUs halved the error rate of the best computer-vision systems at ImageNet, stunning the field.

  Computing · americas

• c. 820 CE

  Al-Khwarizmi writes al-jabr

  Al-Khwarizmi systematized solving equations in The Compendious Book on Calculation by Completion and Balancing, and popularized Hindu-Arabic numerals.

  Science · middle east

• 1945

  ENIAC switches on

  The first general-purpose electronic computer filled a room, weighed 27 tons, and recalculated artillery tables in seconds instead of days.

  Computing · americas

Sources

• arXiv — Attention Is All You Need (1706.03762)