Why Faster Hardware Isn’t Solving AI’s Compute Problem

Wirth’s Law and the Problem With Faster Hardware

One of the central ideas in the conversation is Wirth’s law, an observation commonly summarized as: software becomes slower more quickly than hardware becomes faster.

The technology industry has benefited from decades of improvements in processors, memory, storage, and networking. But those gains do not automatically translate into proportional improvements for users. As hardware grows more powerful, software systems often become larger and more complex, absorbing much of the additional capacity.

TAHO looks at this problem through the layers of inefficiency between software workloads and the silicon executing them. Justin argues that significant computing capacity already exists but remains unavailable in practice because outdated execution methods prevent companies from using it effectively.

Damien and Justin connect Wirth’s law to Jevons paradox: the idea that improving the efficiency of a resource can increase its total consumption.

The classic example involves more efficient steam engines. Using less coal per engine did not reduce society’s overall coal consumption. It made engines more economical, encouraged more people to use them, and increased total demand.

AI may follow the same pattern.

If companies can run an AI workload for half the cost, they will not necessarily continue doing the same amount of work and simply keep the savings. Lower costs may make entirely new products, experiments, and business models economically possible.

Could More Efficient AI Create More Engineering Work?

The same principle may apply to software engineering.

A common prediction is that AI coding tools will allow companies to produce the same amount of software with fewer developers. Damien offers a more optimistic interpretation.

If AI tools allow an engineering team to build a product for $50,000 that previously would have cost $100,000, the result may not be fewer software projects. Instead, organizations that rejected the project at the higher price may decide it is now worth pursuing.

The number of viable software ideas can expand as development becomes more affordable.

Justin agrees that there will continue to be strong opportunities for engineers. He also questions whether recent technology layoffs can be attributed primarily to AI replacing technical workers. In his view, some of the pressure may instead come from rapidly increasing infrastructure expenses and companies attempting to protect margins.

AI may improve productivity, but companies still need experienced people to define problems, evaluate results, and work in areas where established solutions do not exist.

Where AI Coding Tools Reach Their Limits

TAHO uses AI tools regularly for documentation, operational systems, and content creation. However, the company relies on them much less heavily for its core engineering work.

The reason is not opposition to AI. It is the nature of the problem.

Coding assistants work particularly well when they can draw from a large body of existing examples. They can reproduce established patterns, explain familiar frameworks, and accelerate common development tasks.

TAHO, by contrast, is attempting to create new low-level execution techniques. There may be no reliable reference implementation, standard architecture, or collection of examples from which a model can infer the correct answer.

In those situations, Justin says AI tools can quickly reach a wall. They may also introduce poor-quality code or technical debt when their output is not reviewed carefully by experienced engineers.

This is an important distinction for teams adopting AI development tools. Their usefulness depends partly on how familiar the problem is. Generating another implementation of a common pattern is very different from inventing a new computing primitive.

Where TAHO Fits in the AI Stack

Justin uses Jensen Huang’s “five-layer cake” analogy to explain TAHO’s position in the AI ecosystem.

At the bottom is energy, because no digital system can operate without power. Above energy are the chips produced by companies such as NVIDIA, AMD, and Intel. The next layer is infrastructure, followed by AI models and, finally, the applications people use.

TAHO sits near the bottom of the infrastructure layer, directly above the silicon.

That position matters because many of the techniques used to organize and execute workloads were developed long before the current wave of dense AI computing. The underlying systems were not necessarily designed for the volume and complexity of modern training, inference, scientific computing, and data-processing tasks.

Traditionally, a large workload may be assigned to a processor and remain there while the processor loads and executes it. During that process, parts of the processor—or other hardware elsewhere in the environment—may remain idle.

TAHO aims to change the unit around which the work is organized.

Breaking Workloads Into Thousands of Smaller Jobs

TAHO’s software takes a workload and divides it into thousands of much smaller jobs. Those jobs can then be distributed across the customer’s computing environment.

The objective is to keep GPUs and other processors more consistently occupied, reducing the time that available resources spend waiting or sitting unused.

The type of workload is less important than its cost and execution time. Justin discusses potential uses across AI training, inference, scientific research, batch processing, and edge computing.

Another major part of TAHO’s approach is heterogeneous hardware support. Computing environments often contain resources from multiple manufacturers, but those resources can function as separate islands.

TAHO wants workloads to operate across NVIDIA, AMD, Intel, Qualcomm, and other hardware as part of a shared execution fabric. Instead of forcing a company to organize everything around a single hardware ecosystem, the software would make it easier to use the resources already available across the organization.

One example discussed in the episode involves a biotechnology company running protein-folding and other research workloads. Another involves running relatively large models across smaller edge devices by treating connected resources as a broader computing pool.

The GPU Utilization Gap

The potential value becomes clearer when Damien asks about hardware-utilization rates.

Justin points to a highly optimized published run associated with Meta and Llama that reportedly achieved approximately 43% to 47% effective utilization. That would be an exceptional outcome from an organization with substantial technical resources.

Across many other environments, Justin says utilization may be closer to 5% to 15%.

That means an organization could be spending heavily on GPUs while receiving useful work from only a fraction of its theoretical capacity.

Some waste comes from processors waiting between stages of a workload. In other cases, one section of a processor may be active while other sections remain unused. These gaps accumulate across large environments and expensive workloads.

For a company spending $100 million annually on compute, even a relatively small improvement could produce significant savings. Improving utilization may therefore be more practical than continuously purchasing additional hardware to compensate for inefficient execution.

Building Close to the Hardware

TAHO’s core technology is being written in Rust and runs on Linux.

The team is also developing adapters that allow customers to continue using their existing languages and environments while the work is ultimately translated into lower-level execution units.

Building this close to the hardware introduces a different set of challenges from conventional application development. Justin describes repeatedly reaching what appeared to be the top of a mountain, only to discover another mountain behind it.

Components that initially seemed well supported did not always behave as expected at such a low level. Assumptions about runtime and compiler layers had to be reconsidered. The team has completed several major refactors as new constraints emerged.

That process reflects the difference between implementing a known product category and inventing something for which no established blueprint exists. The difficulty can be exhausting, but it is also part of what makes the work compelling.

Chips Are Not the Only AI Bottleneck

For much of the recent AI boom, the primary infrastructure question was whether companies could obtain enough GPUs.

That is changing.

Justin and Damien discuss how power generation, grid connections, permitting, construction, and data-center capacity are becoming equally important constraints. A company may be able to purchase chips but still lack the electricity or physical infrastructure needed to operate them.

Some data-center projects planned for the next several years are already facing delays because the required power and grid infrastructure cannot be added quickly enough.

This changes the economics of efficiency.

When companies can easily add new capacity, they may tolerate poor utilization. When new power connections can take months or years, extracting more useful work from existing infrastructure becomes unavoidable.

Efficiency is therefore not only a cost-saving measure. It can determine whether an organization has enough practical capacity to launch a product, complete research, or expand an AI service.

More Intelligence per Watt

The conversation closes by connecting technical efficiency with a broader question: how much useful intelligence can the industry produce from each unit of energy?

Innovation is naturally inefficient. Research teams explore ideas that fail, build prototypes that never reach production, and follow technical paths that eventually prove unworkable.

Justin does not argue that this exploration should disappear. Instead, he sees infrastructure efficiency as a way to create more room for it.

Some savings can improve margins. Other savings can be reinvested in research, experimentation, and new products. A company that spends less to run its existing workloads may be able to attempt projects that were previously too expensive.

There is also an environmental dimension. Producing the same amount of useful computation with less electricity reduces waste and pressure on energy systems.

In that sense, efficiency and innovation are not competing priorities. Greater efficiency can expand the resources available for innovation.

The Biggest Takeaway

The AI infrastructure conversation often focuses on building more: more chips, larger data centers, additional power generation, and bigger models.

TAHO is approaching the problem from another direction.

Before adding more capacity, how much more useful work could companies extract from what they already have?

The answer could affect far more than infrastructure budgets. Cheaper and more accessible compute may allow smaller companies to compete, make new software projects viable, expand scientific research, and reduce the energy required to run increasingly capable AI systems.

The central idea of the episode is simple: the next major increase in AI capacity may not come only from manufacturing more powerful hardware. It may come from finally learning how to use existing hardware effectively.