| Check | What this checks | Target | Where | Verdict | Measured | Impact and what to do | $/day at risk |
|---|
| No hardware-fault messages in the accessible kernel log | Whether the readable part of the kernel log holds any message that public vendor or kernel documentation classes as a hardware fault: uncorrectable PCIe or memory errors, a device dropping off the bus, a GPU reset. Benign and application-level messages are counted but never fail this. The log covers whatever the ring buffer still held at read time. | 0 fault-class messages
public kernel and vendor documentation, per signature | sample-129-146-226-5 | does not meet | 1 × nvswitch.sxid.fatal: An NVSwitch fault NVIDIA classes as fatal. Traffic across that switch or link is affected. | A device on this node has faulted per its own driver → Confirm with: nvidia-smi nvlink -e for error counters on the GPUs behind that switch; Fabric Manager logs. (vendor) | — |
| Scale-out bandwidth: RDMA write between servers (host memory) | Scale-out: how fast one server's network card can write straight into another server's memory over RDMA (host memory in this test; GPUDirect is not exercised), the east-west path every multi-server job depends on, against the port's speed. | ≥ 80% of the port rate
reference: 0.8 of port rate: perftest ib_write_bw with 64 KB messages reaches 90%+ of line rate on a correctly configured RoCE path (active MTU 4096) | sample-129-146-226-5/mlx5_0->sample-161-153-79-0 | does not meet | 0.39 of 50 Gb/s (1%; floor 80%) | Multi-node jobs across this path get about 1% of the bandwidth expected → Raise the netdev MTU on mlx5_0's interface so the RDMA path can use its 4096-byte maximum; re-probe to confirm (operator) | — |
| Uncorrectable memory errors on lifetime record | Uncorrectable memory errors from before this boot, and whether the GPU has repaired the damaged rows since. | row-remapper: remapped and clean is at spec; pending or failed is not
NVIDIA row-remapping documentation | sample-129-146-226-5/gpu2 | meets | 4 lifetime, 1 row remapped, none since boot, no repair pending — normal operation per the driver | A history worth watching, not a fault today | none at risk |
| GPUs fit for work | The share of GPUs that passed every check, against the share the contract promises. | ≥ 97% of GPUs with no failed check
reference: 0.97 of GPUs with no failed check | fleet | meets | 32 of 32 (100.0%) with no failed check; target 97% | Met. Nothing to do. | none at risk |
| Network links at contracted rate | The speed each network port negotiated with the switch, against the speed the card can run at or the contract promises. | ≥ 50,000 Mb/s
reference: 50 Gb/s: what the sample fleet's ConnectX ports were provisioned at (a 2-lane breakout of a 100G card) | 4 ports | meets | all at or above 50,000 Mb/s | Met. Nothing to do. | none at risk |
| PCIe links at full width | How many PCIe lanes connect each GPU to its host. Fewer than the card supports means data moves between CPU and GPU more slowly than it should. | the card's own maximum link width
the card's own reported limit | 32 GPUs | meets | all at rated width | Met. Nothing to do. | none at risk |
| GPU temperatures within limit | GPU temperature against the limit at which the card starts holding its clocks back. | ≤ 85 °C
reference: 85 C is the A100-SXM4's own 'GPU Max Operating Temp' as reported by nvidia-smi -q (slowdown 89 C, shutdown 92 C) | 32 GPUs | meets | all at or below 85 C | Met. Nothing to do. | none at risk |
| No hardware-fault Xid codes in the accessible kernel log | Whether the GPU driver has logged a code NVIDIA classes as a hardware fault (an Xid) in the part of the kernel log the tool could read. Most Xids are the application's, and are not counted here. | 0 hardware fault codes
NVIDIA Xid documentation | 4 nodes | meets | no hardware-class Xid in the readable log | Met. Nothing to do. | none at risk |
| Scale-up bandwidth: all-reduce inside one server | Scale-up: how fast the GPUs inside one server combine their results over their own links (NVLink or xGMI). Every multi-GPU step on a server waits on this. | ≥ 170 GB/s
reference: 170 GB/s intra-node all-reduce bus bandwidth (nccl-tests convention, 8 GPUs, largest message) | fleet | could not judge | no collective probe was run (needs nccl-tests / rccl-tests on the node and --probe) | → Run `11mac probe --kind collective --probe` in an agreed window | — |
| Reference-step utilisation (MFU) | How much of each GPU's rated compute a fixed reference training step actually achieved (model FLOP utilisation). A GPU that is throttled or misconfigured scores low. This is our step, not your workload. | ≥ 70% of the datasheet peak
reference: 0.70 on the REFERENCE STEP (dense bf16 MLP, 1.07 B params, 8192 tokens per GPU), not on a customer workload | fleet | could not judge | no reference step was run (needs PyTorch on the node and --probe) | → Run `11mac probe --kind refstep --probe` in an agreed window | — |