Introduction: The Concept of Compute as a Measure
In this digital era, where computational power is the cornerstone of technological advancement, it's intriguing to measure computing capabilities in human-relative terms. We introduce the notion of a "person of compute" as a standard measure, equating to 20 PFLOPS. This is achievable through the use of 64 A100 GPUs or a single, densely-packed 42U rack of A100s. Currently, we stand at the threshold of the 1 rack person era, with a power consumption of approximately 30kW to deliver these 20 PFLOPS.
The Scale of Computational Efforts
Training Large Models: A Comparative Look
The training of LLaMA, on a cluster comprising 2048 A100 GPUs—each offering around 312 TFLOPS—presents a fascinating case study. With a total computational capability of 639 PFLOPS, this equates to 32 "people of compute." The extensive training, consuming around 1 million GPU hours, translates into a concerted effort of approximately 2 person-years of work. In contrast, the development of GPT-4 required about 100 person-years, showcasing the immense scale of computational resources dedicated to these groundbreaking AI models.
Projecting the Future of Computing
As we project into the future, the comparison of computing advancements with human-scale efforts offers a unique perspective. The narrative of doubling computational power, in line with Moore's Law, against the backdrop of budget increments provides a tangible measure of progress. The trajectory from the NVIDIA 1080 to the 4090 GPUs illustrates a consistent trend of doubling computational capability roughly every two years, albeit with increasing energy demands.
The Economic Perspective: Cost of Compute
An exploration into the cost dynamics reveals that a "person of compute" can be established with a budget of around $250k. Extrapolating this to a larger scale, Google's claim of a 9 exaflop (450 person) computer signifies an investment into a future 24 years ahead of its time. This comparison not only highlights the financial aspects but also the ambitious scale of computational projects in relation to human capabilities.
The Grand Vision: A Humanity of Compute
The ultimate aspiration, to build a computational equivalent of a "Humanity," involving 20,000 Tampas or cities of compute, unfolds a visionary future. Within the next few decades, the possibility of achieving such a monumental scale of computing power for the cost equivalent to the International Space Station paints a futuristic scenario. This progression towards a "Humanity under your desk" within 66 years underlines the exponential growth potential of computational power and its implications for technological advancement.
Conclusion: Towards a New Era of Computing
The conceptual framework of "people of compute" not only provides a tangible measure of computational power but also bridges the gap between technological progress and human understanding. As we venture further into this digital age, the continuous evolution of GPUs and the increasing efficiency of computational resources herald a new era of innovation and possibilities. The journey from the present to a "Humanity of compute" underscores the transformative impact of technology on society and the inherent potential for future breakthroughs.
Reflecting on the journey of computational advancement, we stand on the precipice of a future redefined by the limitless possibilities of technology. The notion of a "person of compute" not only encapsulates the essence of this progress but also serves as a beacon, guiding us towards a future where computational power amplifies human potential and creativity.