# Real NVIDIA GPU telemetry (requires an NVIDIA driver on the host)
pip install "workloadtruth-cli[nvml]"

# Try it without a GPU, using the synthetic backend
pip install workloadtruth-cli

# npm launcher (thin wrapper around the PyPI package, see "Why two registries")
npx workloadtruth-cli --help

# No GPU required. Classify a synthetic "training" telemetry trace.
$ workloadtruth classify --backend synthetic --profile training --samples 10 --interval 0
workload_type : TRAINING
confidence    : 1.00
gpu_index     : 0
samples       : 10 over 9.0s
reasons:
  - avg GPU utilization 87.8% >= training threshold 65.0%
  - low GPU utilization variance (std=3.4) = training threshold 5.0
  - low power-draw variance (std=9.4W)  --help`.

## MCP Server

WorkloadTruth ships a [Model Context Protocol](https://modelcontextprotocol.io) server so an AI agent (Claude, Cursor, or any MCP-compatible client) can classify GPU workloads, run the evasion-robustness benchmark, and verify the audit log directly, without a human invoking the CLI by hand.

Install the extra:

> [!NOTE]
> The `mcp` extra requires Python 3.10+, stricter than WorkloadTruth's own 3.9 floor. `pip install "workloadtruth-cli[mcp]"` will fail to resolve on Python 3.9. Every other feature (`classify`, `watch`, `benchmark`, `verify-log`) works on Python 3.9.

Add it to your MCP client's config (for Claude Desktop, `claude_desktop_config.json`). The server is started via the `workloadtruth mcp` subcommand, not a separate console script:

The server exposes three tools over stdio:

- **`classify_workload(backend="nvml", profile="training", gpu_index=0, samples=10, interval_seconds=1.0, write_to_audit_log=False)`**: samples GPU telemetry and classifies it as `TRAINING`, `INFERENCE`, or `IDLE`. `backend` is `"nvml"` (real hardware) or `"synthetic"` (documented synthetic traces, no GPU required). Optionally appends the result to the hash-chained audit log.
- **`run_benchmark(trials=50, window=30)`**: runs the evasion-robustness benchmark against synthetic telemetry and returns per-profile accuracy under clean and evasion-obfuscated conditions.
- **`verify_audit_log(log_file="workloadtruth.log.jsonl")`**: re-derives the hash chain of a local audit log and reports whether it has been tampered with.

Example call, classifying a synthetic training trace with no GPU required:

Transport is stdio, so there is nothing to host: the MCP client spawns `workloadtruth mcp` as a local subprocess. A `.well-known/agent.json` manifest is also shipped at the repo root for A2A-style discovery, listing both the CLI and MCP interfaces and the packages that provide them. Source: [`src/workloadtruth/mcp_server.py`](src/workloadtruth/mcp_server.py).

## Audit log

`workloadtruth watch` appends a hash-chained entry to `workloadtruth.log.jsonl` on every classification window. Each entry's hash covers its own content plus the previous entry's hash, so any edit, reorder, or deletion after the fact breaks the chain from that point forward. `workloadtruth verify-log` re-derives every hash and reports the first broken link, if any.

![WorkloadTruth watch appending hash-chained entries to a local audit log, then verify-log confirming the chain hasn't been tampered with](https://raw.githubusercontent.com/RudrenduPaul/WorkloadTruth/main/docs/demo-verifylog.gif)

This proves what was classified, when, and that the local record hasn't been silently altered afterward. **It does not prove the classification itself was correct**, and it is not evidence of regulatory compliance. See below.

## Why two registries

WorkloadTruth's implementation is Python. NVML access (`pynvml`/`nvidia-ml-py`) is the mature, official way to read NVIDIA GPU telemetry, and it's also what the closest prior art ([arXiv:2606.19262](https://arxiv.org/abs/2606.19262)) uses. The npm package (`workloadtruth-cli`) is a thin launcher, not a reimplementation. It locates and execs the real `workloadtruth` binary installed from PyPI, so `npx workloadtruth-cli` works for npm-first agent tooling without duplicating the classifier in two languages. The npm package versions independently of the PyPI package since it only ships a launcher script, not the classifier itself.

## Comparison

| | WorkloadTruth | NVIDIA DCGM / `dcgm-exporter` | run:ai | Weights & Biases |
|---|---|---|---|---|
| Reads GPU telemetry | Yes (via NVML) | Yes (source) | Yes | Yes |
| Classifies workload type automatically | **Yes** | No, exposes raw metrics only | No, workload type is user-declared at job submission | No, training-run-scoped by design, no classification |
| Local, hash-chained audit trail | Yes | No | No | No |
| Evasion-robustness benchmark | Yes (documented, reproducible) | N/A | N/A | N/A |
| Requires an NVIDIA GPU | Only for the `nvml` backend; the `synthetic` backend works without one | Yes | Yes | No (general system metrics) |

Checked directly against each project's own documentation: [DCGM exporter docs](https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetry/latest/dcgm-exporter.html), [run:ai inference overview](https://run-ai-docs.nvidia.com/saas/workloads-in-nvidia-run-ai/using-inference/nvidia-run-ai-inference-overview), [W&B system metrics docs](https://docs.wandb.ai/models/ref/python/experiments/system-metrics). None of these classify workload type from telemetry alone. That gap is what WorkloadTruth fills.

## What is WorkloadTruth, and why does it exist

WorkloadTruth is an open-source command-line tool and MCP server that classifies a running GPU workload as `TRAINING`, `INFERENCE`, or `IDLE` using only GPU-level telemetry (utilization, memory pattern, power draw), with no changes to the workload's own code and no reliance on a self-reported job label.

It exists because every mainstream GPU scheduler asks the job's owner to declare its type at submission time and never checks that declaration against what the hardware is actually doing. That gap has two real consequences: cost misallocation (a job scheduled at low-priority "inference" pricing that is actually running full training) and unauthorized workload changes (an inference endpoint that quietly starts training on live traffic). WorkloadTruth closes that verification gap today, and doubles as the first open, installable implementation of a real academic research thread on verifying AI training runs from hardware telemetry (see below).

## Relationship to prior research

WorkloadTruth's core technique, classifying training vs. non-training GPU activity from telemetry, is not novel. It's the direct application of a real, active research thread:

1. Yonadav Shavit (Harvard), [*"What does it take to catch a Chinchilla?"*](https://arxiv.org/abs/2303.11341) (2023): proposed hardware-level "training transcripts" for verifying large training runs.
2. GovAI, [*"Computing Power and the Governance of AI"*](https://www.governance.ai/analysis/computing-power-and-the-governance-of-ai) (2024): surveyed compute-governance mechanisms, explicitly framed as exploratory, not endorsed policy.
3. [*"Hardware-Enabled Mechanisms for Verifying Responsible AI Development"*](https://arxiv.org/abs/2505.03742) (2025): hardware-security researchers proposing on-chip attestation.
4. Rahman & Tajdari, [*"Detecting Hidden ML Training With Zero-Overhead Telemetry"*](https://arxiv.org/abs/2606.19262) (ICML 2026 Technical AI Governance workshop): a working NVML-telemetry classifier, 98.2% accurate on unobfuscated workloads, the direct prior art for this project's core classification technique.

**What WorkloadTruth adds:** as of this project's own research (2026-07-19), no open-source, installable implementation of this research thread existed, only academic prototypes. WorkloadTruth is that packaging: a real CLI, an MCP server, a hash-chained audit log, and a reproducible evasion-robustness benchmark, built in the open. It does not claim to improve on the paper's classification technique. See the benchmark section above: the current rule-based classifier is considerably more evadable than the paper's ML approach on the one axis it measures.

## What WorkloadTruth is not

- **Not a compliance or regulatory-audit tool.** No law currently requires inference/training classification or reporting. Any future claim otherwise will name the specific enacted regulation; none exists as of this writing.
- **Not a content inspector.** WorkloadTruth reads GPU-level signals only (utilization, memory, power). It never inspects model weights, training data, prompts, or completions.
- **Not proof of "compliant" or "safe" operation.** The audit log proves what was classified and when, and that the record wasn't altered afterward, not that the classification was correct or that any policy was followed.

## FAQ

**Does this need an NVIDIA GPU?**
Only for the `nvml` backend. `--backend synthetic` runs the full classifier and CLI against documented synthetic traces, no GPU required. Useful for trying the tool or for CI.

**Can it classify AMD or Intel GPU workloads?**
Not yet. The telemetry layer is a pluggable interface (`TelemetryBackend`) specifically so a new vendor backend (AMD ROCm, Intel Level Zero) can be added without touching the classifier. See [CONTRIBUTING.md](CONTRIBUTING.md).

**Is the classifier accurate enough to bill or penalize someone based on its output?**
Not yet, and the benchmark section above is the honest reason why: 0% accuracy on evasive training workloads today. Treat `workload_type` as a signal to investigate, not a verdict.

**Why not just use the ML classifier from the paper?**
Its trained weights and dataset were never published. Reimplementing an ML classifier without real training data would produce an unvalidated accuracy claim, not a measured one. See [How classification works](#how-classification-works).

**How is this different from run:ai or NVIDIA DCGM?**
run:ai and DCGM both expose or use GPU telemetry, but neither classifies workload type from that telemetry. run:ai relies entirely on the label the job's owner declares at submission; DCGM just exposes raw utilization and memory metrics for something else to interpret. WorkloadTruth is the layer that actually looks at the telemetry and answers the question. See the [comparison table](#comparison).

**Does this work on Windows, macOS, and Linux?**
The `synthetic` backend runs anywhere Python 3.9+ runs, including this project's own macOS build environment (which has no NVIDIA GPU). The `nvml` backend requires an NVIDIA GPU and driver, which in practice means Linux or Windows with NVIDIA hardware; NVML itself is not available on macOS.

**What license is this under, and can I use it commercially?**
Apache 2.0. Commercial use, modification, and redistribution are all permitted under its terms; see [LICENSE](LICENSE).

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). Security issues: see [SECURITY.md](SECURITY.md).

## License

[Apache 2.0](LICENSE)