🐿️ Riding the Current - The Insanely Profitable Tech Newsletter
Originally published 27th may 2024
The phone in your pocket is far more powerful than the fastest computer of just 25 years ago. That’s just one effect of a staggering rate of technology improvement that humans aren’t built to comprehend: a doubling, every two years, of the number of electronic components we can fit on a single microchip. As we keep packing more computing power into ever-smaller packages, our software runs faster and can do much more, in accordance with an observation first made way back in 1965 by Gordon Moore. Yes, Gmail or Word may still be frustratingly slow, but for that we have to blame sloppy programmers and increasing demands, not hardware engineers.
The surprising result of all this is that for the past half-century or so, rather than sweating to optimise their code for throughput on current computers, many developers would have been better off kicking back and playing Doom on a microwave for six months–then taking delivery of a new generation of faster machines and running the same programs, unchanged, at a vastly improved speed. Clever engineers keep finding tricky ways to keep the furious shrinking on track; for instance, modern chips are actually little cooperatives, with multiple collaborating “cores” (tiny independent computers) each of which can better exploit the ever-smaller component sizes. But alas, we’re approaching a ceiling imposed by the size of the atom, so Moore’s Law will have to give out sometime soon, well before we have proton-sized computers.
I point out all this because right now, we have front-row seats to watch a similar exponential-growth process as it smashes into physical limits. We’ve had neural networks since the 1950s, and since that time their size–the number of parameters they can adjust to “learn” new behaviour–has grown geometrically, just as Moore observed that chip density did. And on top of this, there’s been a sudden further acceleration in the last four or five years, as it became clear that the complexity of very large models like ChatGPT and Gemini, with billions or trillions of parameters, made it possible for them to draw and write and speak. As a result, we have announcements every week about chatbots that can respond to facial expressions, machines that can age movie stars, and the resurrection of a dead rap singer–so you’d be forgiven for thinking that once again, despite the occasional hilarious hiccough, we just need to put our feet up and wait for tomorrow’s ever-larger models to solve our problems. But wait, don’t fire your data-science team yet! Machine learning algorithms, after all, have to run somewhere, and the LLM behemoths require proportionally gargantuan farms of ordinary computers to perform their magic. Unsurprisingly, there are already shortages of the specialised GPU chips needed to run machine-learning jobs quickly, and it’s not at all clear where we’ll find space, power, and room in our carbon-emissions budget to accommodate the $1 trillion of new data centres it seems we’ll need to meet demand. Expect a slowdown in today’s frenetic pace of AI advancement as the focus shifts to tuning up the software engine instead of just adding more and more “compute” horsepower.
This first appeared in my weekly Insanely Profitable Tech Newsletter which is received as part of the Squirrel Squadron every Monday, and was originally posted on 27th May 2024. To get my provocative thoughts and tips direct to your inbox first, sign up here: https://squirrelsquadron.com/
