Power as the Key Metric in AI Compute
In the early days of the AI race, raw compute and the ability level of the model were just about all that mattered. As the technology becomes more mature, and AI companies need to actually profit from their work, power use becomes a key element.
The need for energy efficiency is obvious: After depreciation, energy is by far the largest ongoing cost of an AI data centre. A more efficient AI system will be more competitive in the rapidly developing market.
Higher energy use also requires higher cooling capacity, which needs yet more energy and also increases the capital costs of the data centre As AI models scale exponentially, energy consumption, thermal limits, and power delivery have become major constraints.
Tenstorrent is at the forefront of this challenge, designing high-performance AI systems that are significantly more efficient in their use of energy.
The Solution: Automated Testing via Quarch Power Analysis
To gain the data needed, Tenstorrent began with a simple evolution: Using a GPU power analyser and a smaller single-phase AC power analyser from Quarch to test the concept.
The GPU analyser sits between an AI accelerator and the host system (capturing 3v3, 3v3_aux and 12v rails). The secondary tap fits a PCIe of 12VHPWR connector for the external 12V supply.
The combination of device-level analysis at the GPU and system-level power at the AC input gives access to a lot of data. The GPU analyser samples every 4uS (250,000 samples per second), and the AC analyser at 125uS (8,000 samples per second).
Long workloads can be recorded at high resolution. The entire trace is captured on the lower timeline, while the main trace window can be used to quickly zoom in to the more interesting details.
Data is captured in real time to a control PC over USB or LAN, so the available disc space is the main limiting factor.
Additional Requirements
Tenstorrent quickly saw the benefit in the ease of use and the additional data they could capture and moved to implement multiple GPU and AC analysers across their teams.
There were a couple of missing requirements, though:
- Multi-unit analysis Tenstorrent is required to capture data from multiple Quarch devices at the same time and correlate them accurately. To do this, Quarch provided a Python script to kick off recording on multiple analysers at the same time. Once the data was captured, it was merged and re-uploaded to Quarch Power Studio (QPS). This provided a single visual trace from any number of instruments.
As a result of the request, QPS can now upload single or even multiple files in CSV form and combine them into a visual analysis. Customers can download the Application Note AN034 to do the same. - North American AC connectors Tenstorrent racks use L15 and L21 3-phase sockets. The L15 socket is also a Delta phase configuration, while previous Quarch products were intended for EU-style Star connectors. The Quarch team worked quickly to design a new version of the product with the required inlets.
- Averaging channels Capturing at high resolution is useful for the smallest details, but at the same time, a windowed average channel was required. A channel with averaging (100mS for example) would give engineers a better idea of the general trend, especially for loads that are changing very rapidly.
This feature existed internally but was exposed to the user via ‘Synthetic channels’. Allowing any number of calculated channels from simple averages to complex multi-channel maths. See QPS 1.52
The new variants are available here
Customer Benefits
A key benefit for Tenstorrent is to characterise the input transient, average, maximum and minimum power while running various workloads. The Quarch Python automation library makes it simple to run repeat test cases with no user intervention.
“Consistent, input power measurement is essential for us to understand how much power various workloads draw.”
– Alan Wu, Principal Power Engineer @ Tenstorrent
Andy Norrie
