From the 2019 essay “The Bitter Lesson” by Rich Sutton:

Summary: AI research shows that leveraging computation through general methods like search and learning is far more effective than incorporating human knowledge. As computational power grows, these methods outperform human-centric approaches in fields like chess, Go, speech recognition, and vision. The key takeaway: focus on scalable computational methods, not mimicking human thought.

I wish every graduate student in AI would read “The Bitter Lesson”.1

Arguments#

I think people have generally taken this essay far too seriously.2

The point of the Bitter Lesson is that research and clever ideas are important, but people should think about how their ideas scale with data and compute rather than just relying on One Weird Trick to get them a little farther than SOTA.3

I think the objective of Sutton’s essay was to advance the science of AI, not so much career trajectories of students.

The real bitter lesson: you need money to train a model, and to make money you need to build good products, and to make good products you need to pick the loss that matters to customers.4

FYI, Rich Sutton is a well-known Reinforcement Learning for AI research scientist.