Overview
News
Technologies
Salaries
Products
People
Growth
Financials
Overview
Use the built-in GELU, don't roll your own!. Unsurprisingly, PyTorch's own built-in GELU functionis faster than the hand-rolled one I've been using to date. But I was surprised athow much faster using it made things when training my models.
News
Use the built-in GELU, don't roll your own!
Unsurprisingly, PyTorch's own built-in GELU function is faster than the hand-rolled one I've been using to date. But I was surprised at how much faster using it made things when training my models. I discovered this accidentally just now while working on
Read more
Report
A quick(ish) Chinchilla check
I recently overtrained a couple of GPT-2 style models , training them both on 40 tokens per parameter rather than the 20 per parameter that is generally regarded as "Chinchilla-optimal". The normal heuristic is that instead of doing that, you should scale
Read more
Report
How I use AI on this blog
Inspired by this LessWrong post , I thought I'd write about how I use AI here. This is less in the interest of disclosure, more to provide a snapshot of what I'm doing right now so that I can revisit it in the future and see how it changes. And hey, maybe
Read more
Report
Why do OpenAI's GPT-2 weights beat mine? Part three: testing overtraining
The GPT-2-style models that I've been training work really well, and I've even managed to train some that perform better than the original OpenAI small model in terms of cross entropy loss on a test set. But as I wrote previously , there's a mystery: why
Read more
Report
Why do OpenAI's GPT-2 weights beat mine? Part two: the bugfix
I'm digging into why my GPT-2 style models score worse on an instruction-following eval than OpenAI's original weights; I gave the details in this post . While I was writing up the results of my first experiment into possible causes, I ran the post past C
Read more
Report
Pro access
Upgrade to see all 29 mentions
Upgrade to a paid plan to read every media mention of this company - funding news, awards, product launches and press releases from all the outlets writing about it.
Every media mention and press release
Funding news, awards and product launches
Fresh coverage from every outlet writing about the company
Upgrade now
Cancel anytime. Secure checkout. Instant activation.