Reflections on OpenAI

I have been reading so much news these days about Meta taking AI talent from OpenAI and other companies, it was fun to read this little tid-bit here:

When it comes to personnel (at least in eng), there’s a very significant Meta → OpenAI pipeline. In many ways, OpenAI resembles early Meta: a blockbuster consumer app, nascent infra, and a desire to move really quickly. Most of the infra talent I’ve seen brought over from Meta + Instagram has been quite strong.

There are other fun insights here as well. It’s seldom we get these types of look inside these companies.

How large models are trained (at a high-level). There’s a spectrum from “experimentation” to “engineering”. Most ideas start out as small-scale experiments. If the results look promising, they then get incorporated into a bigger run. Experimentation is as much about tweaking the core algorithms as it is tweaking the data mix and carefully studying the results. On the large end, doing a big run almost looks like giant distributed systems engineering. There will be weird edge cases and things you didn’t expect. It’s up to you to debug them.

Read this in full.