This article delves into the rapid evolution of computing capabilities, measuring computational power in 'people of compute', and examining GPU advancements from the perspective of PFLOPS and energy consumption. This is an exploration of technology's potential and its implications for the future, rather than a fixed prediction of what will come.
In our journey through the age of information, we have witnessed the exponential growth of computational power. This article introduces a novel way to measure this growth: the concept of "people of compute."
The Era of the Rack Person
We define one person of compute as equivalent to 20 PFLOPS, achievable with 64 A100s or a densely packed 42U A100 rack. Today, we stand at the dawn of the rack person era, where a single rack consumes approximately 30kW to deliver those 20 PFLOPS.
Training Giants: LLaMA and GPT-4
LLaMA, trained on a cluster of 2048 A100 GPUs each boasting ~312 TFLOPS, represents a total of 639 PFLOPS or 32 people of compute. It required about 1 million GPU hours for its training, translating to 500 hours or roughly 3 weeks of continuous computation. This effort equates to approximately 2 person-years of compute work, while GPT-4 demanded an astonishing 100 person-years.
This scale of computation, while massive, remains within the realm of human comprehension. As long as this scale persists, especially over the next decade, the unit of a "person of compute" remains a valuable metric.
The Pace of Progress
Reflecting on Moore's Law, we measure the advancement of computing not just in speed but in terms of how far we leap beyond the capabilities of a modern desktop, pegged at ~50 TFLOPS. This comparison helps us appreciate the strides we're making, with NVIDIA's GPUs showing significant progress:
- 1080 (2016) = 11.3 TFLOPS
- 2080 (2018) = 14.2 TFLOPS
- 3090 (2020) = 35.6 TFLOPS
- 4090 (2022) = 82.6 TFLOPS
Despite the 4090's high power consumption, we're maintaining a pace of doubling performance roughly every two years.
The Cost of Compute
The financial aspect of compute power has also seen dramatic shifts. Today, a person of compute costs about $250k. By this metric, Google's 9 exaflop (450 person) computer is a leap into the future, embodying a computational capacity 24 years ahead of its time.
Towards a Tampa of Compute
Extrapolating these trends, we can imagine building a computer with the power of 400,000 people - equivalent to a "Tampa" of compute - at the cost of the International Space Station (ISS), roughly $100B. This leads to the thought-provoking projection that, within a few decades, the computing power equating to "One Humanity" - or 20,000 Tampas - could be achievable for the price of the ISS.
The Path Forward
As we project these trends into the future, the potential for computational growth seems boundless. Within a few decades, the power of a Humanity could sit on our desktops, revolutionizing every aspect of our lives from science and medicine to AI and beyond.
This contemplation of computing's future is not just an exercise in numbers but a window into the potential transformations in our society. It underscores the importance of technological advancement, ethical considerations, and the profound impact on humanity's trajectory.