Tenant guidance

What was measured

2 nodes, 16 × NVIDIA H100 80GB HBM3 · driver 570.211.01 · InfiniBand 400 Gb/s, 8 ports per node

3 passive sweeps between 2026-09-08 01:44 UTC and 2026-09-08 01:45 UTC. Active probes run: RDMA bandwidth between nodes, all-reduce inside each node, all-reduce across nodes, reference training step.

Judged against reference targets (illustrative; your SLA replaces them). Other fleets used: 0 — every comparison is inside this fleet or against the stated basis, never against anyone else's hardware.

Run 64a6d6ef710a · generated 2026-09-10 21:30 UTC · 3 sweeps in store

info1

Measured against reference targets (illustrative; your SLA replaces them), 2 nodes and 16 GPUs meet 12 of 12 checks.

2/2nodes with no failed check

16/16GPUs with no failed check

12/12checks met
collection gaps on 2 node(s): nic_eth: missing ethtool

0actions

$96what you paid for 16 GPUs over 1.5 h (your rental_hours) at $3.99 per GPU-hour

$0on GPUs that failed a check — 0 GPUs

Working / idle split not shown. 3 sweeps covering 1 minutes of a 1.5-hour period is a diagnostic snapshot, not a record of how the capacity was used; utilization seen in that window is not evidence of your usage or of loss. A day of `11mac watch` alongside your jobs is what prices out. Diagnostic overhead in this window: 903 GPU-seconds of probes, not counted as work or idle.

$3.99/GPU-hour and $0.10/kWh are reference prices, not yours; set them in the expectations file.

Sections that need a decision start open; the rest are folded — click a heading to open it. Printing or saving as PDF includes every section.

Verdict

Measured against reference targets (illustrative; your SLA replaces them) — no SLA was supplied, so each device is held to what it says it can do. Pass --expectations sla.json to judge against your contract.

meets 12not met 0could not judge 0

Also worth knowing

GPU 7 on slurm-worker-slinky-0 has corrected a small number of memory errors over its life

All checks

every check, its verdict and what it measures

Targets are reference targets, not a contract. No SLA file was supplied, so each check is judged against a shipped floor with its source shown beside it (a datasheet, the card's own limit, a published measurement, or our own capture). Pass --expectations your-sla.json and every target becomes your contract's number.

Check	What this checks	Target	Where	Verdict	Measured	Impact and what to do	$/day at risk
Scale-up bandwidth: all-reduce inside one server	Scale-up: how fast the GPUs inside one server combine their results over their own links (NVLink or xGMI). Every multi-GPU step on a server waits on this.	≥ 350 GB/s reference: 350 GB/s intra-node all-reduce bus bandwidth (nccl-tests, 8 GPUs, 512 MB)	2 nodes	meets	436-437 GB/s over 8 GPUs against a floor of 350	Met. Nothing to do.	none at risk
All-reduce returns correct values	Whether the GPUs combined those results correctly. A wrong value here means training would silently go wrong.	0 incorrect reduction results nccl-tests checks every result	2 nodes	meets	nccl-tests verified every reduction result	Met. Nothing to do.	none at risk
Scale-out bandwidth: all-reduce across servers	How fast GPUs on different servers can combine their results over the network between servers. This is what a multi-node training job waits on.	≥ 100 GB/s reference: 100 GB/s across nodes (nccl-tests bus bandwidth, 16 GPUs on 2 nodes)	2 nodes	meets	130 GB/s over 16 GPUs against a floor of 100	Met. Nothing to do.	none at risk
Memory integrity	Whether any GPU's memory has thrown an error it could not correct since boot. The driver contains such errors and terminates the affected job; the count is the history of that happening.	0 uncorrectable errors since boot reference: 0 since boot	16 GPUs	meets	no uncorrectable errors, since boot or on record	Met. Nothing to do.	none at risk
GPUs fit for work	The share of GPUs that passed every check, against the share the contract promises.	≥ 97% of GPUs with no failed check reference: 0.97 of GPUs with no failed check	fleet	meets	16 of 16 (100.0%) with no failed check; target 97%	Met. Nothing to do.	none at risk
No hardware-fault messages in the accessible kernel log	Whether the readable part of the kernel log holds any message that public vendor or kernel documentation classes as a hardware fault: uncorrectable PCIe or memory errors, a device dropping off the bus, a GPU reset. Benign and application-level messages are counted but never fail this. The log covers whatever the ring buffer still held at read time.	0 fault-class messages public kernel and vendor documentation, per signature	2 nodes	meets	no fault-class signature present	Met. Nothing to do.	none at risk
PCIe links at full width	How many PCIe lanes connect each GPU to its host. Fewer than the card supports means data moves between CPU and GPU more slowly than it should.	the card's own maximum link width the card's own reported limit	16 GPUs	meets	all at rated width	Met. Nothing to do.	none at risk
Scale-out bandwidth: RDMA write between servers (host memory)	Scale-out: how fast one server's network card can write straight into another server's memory over RDMA (host memory in this test; GPUDirect is not exercised), the east-west path every multi-server job depends on, against the port's speed.	≥ 80% of the port rate reference: 0.8 of port rate: ib_write_bw over native InfiniBand reached 96% (385.8 of 400 Gb/s) on the sample cluster; 80% leaves room for a busy switch.	slurm-worker-slinky-0/mlx5_0->slurm-worker-slinky-1	meets	384.34 of 400 Gb/s (96%; floor 80%)	Met. Nothing to do.	none at risk
Communication exposed on the reference step	What share of that reference step was spent waiting for GPUs to exchange gradients rather than computing, with no overlap: a worst case.	≤ 25% of the step reference: 0.25 of the un-overlapped reference step	2 nodes	meets	14.9%-15.3% against a ceiling of 25%	Met. Nothing to do.	none at risk
Reference-step utilisation (MFU)	How much of each GPU's rated compute a fixed reference training step actually achieved (model FLOP utilisation). A GPU that is throttled or misconfigured scores low. This is our step, not your workload.	≥ 60% of the datasheet peak reference: 0.60 on the REFERENCE STEP (dense bf16 MLP, 1.07 B params, 8192 tokens per GPU)	16 GPUs	meets	68.6%-71.2% against a floor of 60%	Met. Nothing to do.	none at risk
GPU temperatures within limit	GPU temperature against the limit at which the card starts holding its clocks back.	≤ 85 °C reference: 85 C, CARRIED OVER from the A100's 'GPU Max Operating Temp'	16 GPUs	meets	all at or below 85 C	Met. Nothing to do.	none at risk
No hardware-fault Xid codes in the accessible kernel log	Whether the GPU driver has logged a code NVIDIA classes as a hardware fault (an Xid) in the part of the kernel log the tool could read. Most Xids are the application's, and are not counted here.	0 hardware fault codes NVIDIA Xid documentation	2 nodes	meets	no hardware-class Xid in the readable log	Met. Nothing to do.	none at risk

Findings

each diagnostic finding with its evidence

info

Corrected memory errors within expected rate

slurm-worker-slinky-0/gpu7

GPU 7 on slurm-worker-slinky-0 has corrected a small number of memory errors over its life. That is normal; nothing to do unless the count starts climbing.

slurm-worker-slinky-0/gpu7 has 1 lifetime corrected memory error and is not accumulating more

Action: No action; record the count so a future rise can be compared to it · Owner: vendor · Confidence: high

What this is not: Bit flips are absorbed by ECC as designed; only a trend distinguishes wear from background physics

evidence: slurm-worker-slinky-0/8d918c5f885cc361/91ea3e4352826ff1

Fleet

one row per node

Node	Status	SKU	Driver	GPUs	Measurements	Not read
slurm-worker-slinky-0	ok	NVIDIA H100 80GB HBM3	570.211.01	8	322	nic_eth: missing ethtool
slurm-worker-slinky-1	ok	NVIDIA H100 80GB HBM3	570.211.01	8	320	nic_eth: missing ethtool

"Not read" lists tools that were absent or restricted on that node. A gap is recorded as a gap, never filled with a guess.

Bottom line

the money, as a waterfall that reconciles

Every figure covers 1.5 hours (your rental_hours). GPU-hours are priced at $3.99 — what you pay for one — and electricity at $0.10 per kWh — both are placeholder prices; set gpu_hour_price_usd and power_price_usd_per_kwh in your expectations file for your real numbers.

What	For the period	How it was worked out
What you paid for these GPUs	$96	16 GPUs × $3.99 per GPU-hour × 1.5 hours. The price is a placeholder — set gpu_hour_price_usd to yours
Provider shortfall — GPUs you could not trust	$0	no GPU failed a check
Diagnostic overhead — the measurement's own probes	—	903 GPU-seconds of active probes; not counted as work or as idle
Electricity the GPUs drew (the provider's bill), measured	$0	1.15 kW across 16 GPUs × 1.5 hours × $0.10 per kWh. The electricity price is a placeholder — set power_price_usd_per_kwh. GPUs only: servers, cooling and network are not included

Measured quantities × the prices supplied. An illustration for planning, not an invoice deduction or an observed loss; it does not assign responsibility on its own.

Collective sweeps

bus bandwidth by message size, per collective

Bus bandwidth by message size, nccl-tests convention. The top number is what gets quoted; the shape is what matters to a job whose messages are small. Scale-up is inside one server (NVLink / xGMI); scale-out crosses servers.

Collective	Scope	Node	8 KB	1 MB	64 MB	Peak	Sizes swept
all-reduce	scale-up	slurm-worker-slinky-0	0.3	35.0	359.2	436.5 at 512 MB	27
all-reduce	scale-up	slurm-worker-slinky-1	0.3	35.0	359.2	436.5 at 512 MB	27
all-reduce	scale-out	slurm-worker-slinky-0	0.2	3.4	85.6	141.3 at 512 MB	27
all-reduce	scale-out	slurm-worker-slinky-1	0.2	3.4	85.6	141.3 at 512 MB	27

GB/s throughout.

Measured values

the numbers themselves

What the tools reported, per device, in this sweep. The JSON carries every measurement; this is the subset a reader checks first.

gpu

Node	Device	clocks_current_sm	clocks_max_sm	ecc.aggregate_dram_correctable	ecc.aggregate_sram_correctable	ecc.volatile_dram_uncorrectable	enforced_power_limit	pcie_link_width_current	pcie_link_width_max	power_draw	temperature_gpu	throttled	refstep.exposed_comm_fraction	refstep.mfu	refstep.tflops
slurm-worker-slinky-0	gpu0	345	1980	0	0	0	700	16	16	70.26	27	no	—	0.70	696.24
slurm-worker-slinky-0	gpu1	345	1980	0	0	0	700	16	16	68.23	28	no	—	0.70	693.77
slurm-worker-slinky-0	gpu2	345	1980	0	0	0	700	16	16	71.96	29	no	—	0.71	702.06
slurm-worker-slinky-0	gpu3	345	1980	0	0	0	700	16	16	69.69	27	no	—	0.71	704.31
slurm-worker-slinky-0	gpu4	345	1980	0	0	0	700	16	16	70.27	27	no	—	0.70	690.33
slurm-worker-slinky-0	gpu5	345	1980	0	0	0	700	16	16	68.75	28	no	—	0.71	698.22
slurm-worker-slinky-0	gpu6	345	1980	0	0	0	700	16	16	68	28	no	—	0.70	694.21
slurm-worker-slinky-0	gpu7	345	1980	0	1	0	700	16	16	71.71	27	no	—	0.71	703.78
slurm-worker-slinky-0	gpus	—	—	—	—	—	—	—	—	—	—	—	0.15	—	—
slurm-worker-slinky-1	gpu0	345	1980	0	0	0	700	16	16	77.14	36	no	—	0.70	693.56
slurm-worker-slinky-1	gpu1	345	1980	0	0	0	700	16	16	72.65	38	no	—	0.69	678.92
slurm-worker-slinky-1	gpu2	345	1980	0	0	0	700	16	16	69.52	37	no	—	0.70	692.21
slurm-worker-slinky-1	gpu3	345	1980	0	0	0	700	16	16	76.14	37	no	—	0.71	700.68
slurm-worker-slinky-1	gpu4	345	1980	0	0	0	700	16	16	75.61	36	no	—	0.70	690.54
slurm-worker-slinky-1	gpu5	345	1980	0	0	0	700	16	16	70.94	36	no	—	0.70	688.48
slurm-worker-slinky-1	gpu6	345	1980	0	0	0	700	16	16	77.83	37	no	—	0.70	691.19
slurm-worker-slinky-1	gpu7	345	1980	0	0	0	700	16	16	73.15	37	no	—	0.71	697.96
slurm-worker-slinky-1	gpus	—	—	—	—	—	—	—	—	—	—	—	0.15	—	—

nic

Node	Device	collective.allreduce.busbw.intra-node	collective.allreduce.latency_us.intra-node
slurm-worker-slinky-0	nccl	437.05	34.23
slurm-worker-slinky-1	nccl	435.98	42.90

ib

Node	Device	active_mtu	max_mtu	rate	state	collective.allreduce.busbw.cross-node	collective.allreduce.latency_us.cross-node
slurm-worker-slinky-0	mlx5_0:1	4096	4096	400	Active	—	—
slurm-worker-slinky-0	mlx5_1:1	4096	4096	400	Active	—	—
slurm-worker-slinky-0	mlx5_2:1	4096	4096	400	Active	—	—
slurm-worker-slinky-0	mlx5_3:1	4096	4096	400	Active	—	—
slurm-worker-slinky-0	mlx5_4:1	4096	4096	400	Active	—	—
slurm-worker-slinky-0	mlx5_5:1	4096	4096	400	Active	—	—
slurm-worker-slinky-0	mlx5_6:1	4096	4096	400	Active	—	—
slurm-worker-slinky-0	mlx5_7:1	4096	4096	400	Active	—	—
slurm-worker-slinky-0	nccl:2nodes	—	—	—	—	129.60	56.29
slurm-worker-slinky-1	mlx5_0:1	4096	4096	400	Active	—	—
slurm-worker-slinky-1	mlx5_1:1	4096	4096	400	Active	—	—
slurm-worker-slinky-1	mlx5_2:1	4096	4096	400	Active	—	—
slurm-worker-slinky-1	mlx5_3:1	4096	4096	400	Active	—	—
slurm-worker-slinky-1	mlx5_4:1	4096	4096	400	Active	—	—
slurm-worker-slinky-1	mlx5_5:1	4096	4096	400	Active	—	—
slurm-worker-slinky-1	mlx5_6:1	4096	4096	400	Active	—	—
slurm-worker-slinky-1	mlx5_7:1	4096	4096	400	Active	—	—
slurm-worker-slinky-1	nccl:2nodes	—	—	—	—	129.60	56.29

rdma

Node	Device	rdma.bw_gbps.unknown
slurm-worker-slinky-0	mlx5_0->slurm-worker-slinky-1	384.34

system

Node	Device	kernel_taint_flags	mce_count	xid_count
slurm-worker-slinky-0	system	out_of_tree_module	0	0
slurm-worker-slinky-1	system	out_of_tree_module	0	0

Behind the findings

the raw outliers and changes

2 intra-fleet outliers and 0 changes over time were evaluated by the rules. 3 runs in the store; change detection needs at least 6.

Topology discovery found no switch attachments (no LLDP visible), so findings cannot be attributed to a leaf or uplink.

How to read the numbers

what each SKU is compared against

GPU model	Nodes	GPUs	Compared against	Other fleets used
NVIDIA H100 80GB HBM3	2	16	this fleet only	0

Every finding compares your nodes and devices with each other. "Other fleets used: 0" means nothing here is judged against anyone else's hardware — a node is flagged only when it disagrees with its own peers.

What this run cost you

the overhead 11mac imposed

Passive sweep: 161.9 node-seconds across 240 read-only commands; no GPU time and no fabric traffic by construction.

Active probes: 903.3 GPU-seconds, summed from the probes' own records. Fabric bytes for the RDMA probe are recorded in that probe's own run report, not in this store.

Checking our work

how to verify this report

642 measurements, hash-chained. Chain root: 248f25a7b155a04a168f2f98b47a9667a50588584250b268df08fc66b90dffdd

30 of 36 raw outputs the measurements cite are present in the archive; 6 are not. Measurements without their source output are listed in the manifest as unverifiable and should be read as such.

The chain and the hashes show the numbers were not altered after they were written; they do not make the method right. Anyone can re-run the same commands and check. Verify without trusting this file: 11mac verify report.json --db fleet.db --archive ./raw recomputes every hash from the data.