Author here. Most explanations of language models either start at backpropagation or stay at the level of transformers and how to build them. I wanted the step in between: build the intuition up from a single neuron and why this machinery works so well.
The four parts:
1. A neuron can act like an if-statement, you can build a tiny programming language into a model.
2. A bag-of-words network can codify almost any arbitrary rule; but handling sentences it has never seen requires something smarter: rules instead of look-up-tables.
3. Embeddings: how meaning gets codified at the token/word level and why that’s useful.
4. Emergent concepts: OpenAI's 2017 sentiment neuron, which grew in a model trained only to predict the next character of Amazon reviews, and how that scales up.
Apart from the hand-set weights in part 1, every number on screen comes from a real run: a training run on 60 labeled sentences, a fill-in-the-blank run on a toy corpus, real GloVe vectors, and OpenAI's released model.
Each part is a ~4-minute video, an article with interactive figures (in case you prefer reading instead of watching) and an exercise if you want to test your understanding. The videos are generated from code: the narration's timing drives the animation, the sound and the captions.
Note by "understand" (in the title) I mean in the chinese room argument style: the network builds internal features that track meaning, like that neuron. Disclosure: I'm building zairai, an AI coach, and this course is free with no sign-up. I'd love to hear where it loses you, or where an expert would refine/make any corrections.
Author here. Most explanations of language models either start at backpropagation or stay at the level of transformers and how to build them. I wanted the step in between: build the intuition up from a single neuron and why this machinery works so well.
The four parts:
1. A neuron can act like an if-statement, you can build a tiny programming language into a model.
2. A bag-of-words network can codify almost any arbitrary rule; but handling sentences it has never seen requires something smarter: rules instead of look-up-tables.
3. Embeddings: how meaning gets codified at the token/word level and why that’s useful.
4. Emergent concepts: OpenAI's 2017 sentiment neuron, which grew in a model trained only to predict the next character of Amazon reviews, and how that scales up.
Apart from the hand-set weights in part 1, every number on screen comes from a real run: a training run on 60 labeled sentences, a fill-in-the-blank run on a toy corpus, real GloVe vectors, and OpenAI's released model.
Each part is a ~4-minute video, an article with interactive figures (in case you prefer reading instead of watching) and an exercise if you want to test your understanding. The videos are generated from code: the narration's timing drives the animation, the sound and the captions.
Note by "understand" (in the title) I mean in the chinese room argument style: the network builds internal features that track meaning, like that neuron. Disclosure: I'm building zairai, an AI coach, and this course is free with no sign-up. I'd love to hear where it loses you, or where an expert would refine/make any corrections.