Betting on methods that scale

After reading The Bitter Lesson, how do I actually put weight on scalable learning, search, and compute?

confused

I read The Bitter Lesson and I buy it: in the long run, general methods that use more compute beat clever hand-built ones. But in my own work, I can’t say what it actually means to bet on learning, search, and compute that scale.

rough view

Something like: write fewer hard rules, throw more data and compute at it. But that’s too vague to help me pick anything.

asked AI

What does the bitter lesson mean in practice, and how do I tell if a method follows it?

› show full response

In one line: hardcode less of your own judgment, and build something that gets better on its own when you give it more data, more compute, longer training, and a bigger search budget.

Three layers. Learning: let the model work out its own features and strategies from data, instead of you hand-coding them. Search: spit out options, score them, keep the good ones, repeat. You set the goal and how to grade it, not every step. Compute: the method has to batch, run across many chips, stay cheap to talk between them, and read memory in order, so it gets faster for free as the hardware grows.

Put what you know into the goal, the data, the environment, and the metrics, not into a pile of fragile if-else. Tell the model what a good answer looks like and give it lots of data and a verifier, instead of writing rules to fake being smart.

The test is simple. 10x the data, does it get better? 10x the model? 10x the training time? If the answer is mostly “no, not unless I keep tuning by hand,” it doesn’t scale well enough.

→ three layers + one test

what stuck

I didn’t push back. I read it, it made sense, I took it. Two things stuck. One, the test: does it get better on its own with 10x the data, 10x the model, 10x the training time? Two, where my own knowledge goes: into the goal, the data, the environment, and the metrics, not into a pile of fragile if-else. Tell the model what a good answer is, give it data and a verifier, don’t hand-write rules to fake being smart.

the unease

Once again I asked AI before I had a view of my own. Same problem as my earlier card on outsourcing thinking to AI. I believe the idea, but right now it’s AI’s call, not mine. To make it mine I have to take a few real ideas I’m sitting on, run them through the test, and see if it kills something I was about to build.

key takeaway

Two things stuck: judge a method by whether 10x the data, model, or training time makes it better on its own; put what you know into the goal, the data, the environment, and the metrics, not into a pile of fragile rules.