TENSORPACKAI TECHNICAL DOCUMENTATION // COMMUNITY & ENTERPRISE
← Return to Home
TensorPackAI DOCS
[01] ARCHITECTURE OVERVIEW

TensorPackAI Technical Documentation

TensorPackAI is a silicon and runtime co-design engine that cuts datacenter LLM serving power by up to 58.1% with zero latency SLA violations. By exploiting the hardware-software asymmetry between the compute-bound prefill phase and the memory-bound decode phase, it dynamically paces accelerator clock frequencies to match arithmetic intensity.

Supported Silicon B200 / H100 / A100
Serving Frameworks vLLM / Triton / TGI
Telemetry Driver C++17 Direct NVML
Community License Apache 2.0
[02] GETTING STARTED

Installation

TensorPackAI Community Edition requires Linux (x86_64 or aarch64), Python 3.8+, and an NVIDIA driver (525+) with access to libnvidia-ml.so.

Method A: Install via PyPI (Recommended)
pip install tensorpackai
Method B: Direct Git Clone & Build
# Clone official community repository
git clone https://github.com/tensorpackai/tensorpackai-community.git
cd tensorpackai-community

# Compile C++17 Fast NVML bindings and install
pip install .
[03] OPERATOR TOOLING

CLI Reference (`tensorpack-tune`)

The tensorpack-tune utility provides hardware inspection, telemetry verification, and frequency profile locking.

1. Live Hardware Telemetry Inspection
tensorpack-tune status
2. Architecture Phase-Aware Recommendations
tensorpack-tune profile --gpu 0
3. Manual Clock Pinning (Requires CAP_SYS_ADMIN or sudo)
# Lock GPU 0 SM clock to memory-paced decode frequency (e.g. 1095 MHz on H100)
sudo tensorpack-tune pin --gpu 0 --clock 1095

# Reset all GPUs back to manufacturer dynamic curves
sudo tensorpack-tune reset --gpu all
[04] CLUSTER TELEMETRY

Prometheus Metrics Exporter

The standalone Prometheus exporter captures hardware energy dissipation and correlates it with active vLLM generation metrics to calculate real-time Joules per Token and Tokens per Watt.

# Start exporter service on port 9100 scraping vLLM metrics
tensorpack-exporter --port 9100 --interval 0.5 --vllm-url http://localhost:8000/metrics
Metric Name Type Description
tensorpack_joules_per_token Gauge Windowed electrical energy consumed per generated token (ΔE / ΔN_tok).
tensorpack_tokens_per_watt Gauge Throughput energy efficiency (Tokens/s per Watt, identical to Tokens/Joule).
tensorpack_gpu_power_watts Gauge Instantaneous physical GPU power consumption in Watts.
tensorpack_gpu_sm_clock_mhz Gauge Current SM core frequency in MHz.
[05] ENTERPRISE RUNTIME CO-DESIGN

Autonomous Phase-Aware Governors & SLAs

While the Community Edition provides manual clock pinning and Prometheus telemetry, TensorPackAI Enterprise executes autonomous micro-batch phase detection directly at the vLLM and Triton runtime boundaries with zero model modifications and P90 ITL < 20ms guaranteed SLAs.