Ethan Caballero–Broken Neural Scaling Laws

The Inside View

Ethan Caballero is a PhD student at Mila interested in how to best scale Deep Learning models according to all downstream evaluations that matter. He is known as the fearless leader of the "Scale Is All You Need" movement and the edgiest person at MILA. His first interview is the second most popular interview on the channel and today he's back to talk about Broken Neural Scaling Laws and how to use them to superforecast AGI.

Youtube: https://youtu.be/SV87S38M1J4

Transcript: https://theinsideview.ai/ethan2

OUTLINE

(00:00) The Albert Einstein Of Scaling

(00:50) The Fearless Leader Of The Scale Is All You Need Movement

(01:07) A Functional Form Predicting Every Scaling Behavior

(01:40) A Break Between Two Straight Lines On A Log Log Plot

(02:32) The Broken Neural Scaling Laws Equation

(04:04) Extrapolating A Ton Of Large Scale Vision And Language Tasks

(04:49) Upstream And Downstream Have Different Breaks

(05:22) Extrapolating Four Digit Addition Performance

(06:11) On The Feasability Of Running Enough Training Runs

(06:31) Predicting Sharp Left Turns

(07:51) Modeling Double Descent

(08:41) Forecasting Interpretability And Controllability

(09:33) How Deception Might Happen In Practice

(10:24) Sinister Stumbles And Treacherous Turns

(11:18) Recursive Self Improvement Precedes Sinister Stumbles

(11:51) Humans In The Loop For The Very First Deception

(12:32) The Hardware Stuff Is Going To Come After The Software Stuff

(12:57) Distributing Your Training By Copy-Pasting Yourself Into Different Servers

(13:42) Automating The Entire Hardware Pipeline

(14:47) Having Text AGI Spit Out New Robotics Design

(16:33) The Case For Existential Risk From AI

(18:32) Git Re-basin

(18:54) Is Chain-Of-Thoughts Enough For Complex Reasoning In LMs?

(19:52) Why Diffusion Models Outperform Other Generative Models

(21:13) Using Whisper To Train GPT4

(22:33) Text To Video Was Only Slightly Impressive

(23:29) Last Message