Connorjdavis

In a previous post on language modeling, I implemented a GPT-style transformer. Lately I've been learning mechanistic interpretability to go deeper and understand why the transformer works on a mathematical level.This post is a brain dump of what I've lea

Welcome to Part 7 of this series on language modeling. In this post, we will implement tokenization. In the previous post, we trained a one-layer transformer for maximizing the likelihood of the next token, conditioned on the previous tokens over the Tiny

In Part 6 of this series on language modeling, we upgrade our model architecture from the LSTM to the Transformer. You can see Part 5 of this series here. Our latest LSTM gave us a sample story like this:Once upon a time there were two fearful of many toy

In Part 5 of this series on language modeling, we linger a bit on the LSTM to peek under the hood in order to better understand the network's internals. If you're just joining, you can read Part 4 here. Our latest 1-layer LSTM we trained from Part 4 on th

Welcome to Part 4 of a series on language modeling. In Part 3, we got familiar using vanilla RNNs for the next-character prediction task. We saw the main advantage of adding recurrence to the network is to keep track of long-term dependencies in the input

This is Part 3 of a series on language modeling. You can read Part 2 here. In Part 2 we finished up with a couple of techniques for stabilizing training - Xavier initialization and LayerNorm. These improvements brought our perplexity down from 5.73 to 4.5

Part 2 of a series on language modeling with neural networks.

To inspire others as others have inspired me.

This is Part 2 of a series on language modeling with neural networks. You can find Part 1 here. If you recall from Part 1, our initial model had a perplexity of 434 on the hold-out set from TinyStories. This was a story sampled from the model:Story time:

This is my first post in a while.Lately I've been developing a mushroom growing app. This is an open-source project that enables citizen science for mushroom growing. For a while now, I have wanted a tool that allows me to keep track of my grows and to ru