TensorPackAI Technical Documentation
TensorPackAI is a silicon and runtime co-design engine that cuts datacenter LLM serving power by up to 58.1% with zero latency SLA violations. By exploiting the hardware-software asymmetry between the compute-bound prefill phase and the memory-bound decode phase, it dynamically paces accelerator clock frequencies to match arithmetic intensity.
Installation
TensorPackAI Community Edition requires Linux (x86_64 or aarch64), Python 3.8+, and an NVIDIA driver (525+) with access to libnvidia-ml.so.
pip install tensorpackai
# Clone official community repository
git clone https://github.com/tensorpackai/tensorpackai-community.git
cd tensorpackai-community
# Compile C++17 Fast NVML bindings and install
pip install .
CLI Reference (`tensorpack-tune`)
The tensorpack-tune utility provides hardware inspection, telemetry verification, and frequency profile locking.
tensorpack-tune status
tensorpack-tune profile --gpu 0
# Lock GPU 0 SM clock to memory-paced decode frequency (e.g. 1095 MHz on H100)
sudo tensorpack-tune pin --gpu 0 --clock 1095
# Reset all GPUs back to manufacturer dynamic curves
sudo tensorpack-tune reset --gpu all
Prometheus Metrics Exporter
The standalone Prometheus exporter captures hardware energy dissipation and correlates it with active vLLM generation metrics to calculate real-time Joules per Token and Tokens per Watt.
# Start exporter service on port 9100 scraping vLLM metrics
tensorpack-exporter --port 9100 --interval 0.5 --vllm-url http://localhost:8000/metrics
| Metric Name | Type | Description |
|---|---|---|
| tensorpack_joules_per_token | Gauge | Windowed electrical energy consumed per generated token (ΔE / ΔN_tok). |
| tensorpack_tokens_per_watt | Gauge | Throughput energy efficiency (Tokens/s per Watt, identical to Tokens/Joule). |
| tensorpack_gpu_power_watts | Gauge | Instantaneous physical GPU power consumption in Watts. |
| tensorpack_gpu_sm_clock_mhz | Gauge | Current SM core frequency in MHz. |
Autonomous Phase-Aware Governors & SLAs
While the Community Edition provides manual clock pinning and Prometheus telemetry, TensorPackAI Enterprise executes autonomous micro-batch phase detection directly at the vLLM and Triton runtime boundaries with zero model modifications and P90 ITL < 20ms guaranteed SLAs.