all_reduce_perf probe intra-node all-reduce sweep (nccl-tests / rccl-tests), one process, N GPUs ((/[A-Za-z0-9_.-]+)+/)?all_reduce_perf -b \d+ -e \d+ -f 2 -g \d+ -n \d+ -w \d+ -c [01] amd_smi_bad_pages free retired page list amd-smi bad-pages amd_smi_metric free live metrics amd-smi metric( --throttle)? amd_smi_process free processes resident on AMD GPUs amd-smi process amd_smi_ras free RAS error counters amd-smi ras amd_smi_static free static GPU inventory amd-smi static amd_smi_version free amd-smi tool version amd-smi version cuda_version_file free CUDA runtime version where nvcc is absent cat /usr/local/cuda/version\.json dmesg_ring free kernel ring for Xid/MCE extraction dmesg -T --level=(err|warn|err,warn|crit,alert,emerg) ethtool free link speed and duplex ethtool [A-Za-z0-9][A-Za-z0-9._:@/-]* ethtool_dom free optics DOM readings ethtool -m [A-Za-z0-9][A-Za-z0-9._:@/-]* ethtool_drv free driver and firmware version ethtool -i [A-Za-z0-9][A-Za-z0-9._:@/-]* ethtool_stats free NIC counters ethtool -S [A-Za-z0-9][A-Za-z0-9._:@/-]* hostname free canonical hostname hostname( -f)? ib_read_bw probe pairwise RDMA read bandwidth (perftest) ib_read_bw -d [A-Za-z0-9_]+ -i \d+ -s \d+ -n \d+ -p \d+ -F --report_gbits( -R)?( (?!-)[A-Za-z0-9_.:-]+)? ib_write_bw probe pairwise RDMA write bandwidth (perftest); server form has no address ib_write_bw -d [A-Za-z0-9_]+ -i \d+ -s \d+ -n \d+ -p \d+ -F --report_gbits( -R)?( (?!-)[A-Za-z0-9_.:-]+)? ibnetdiscover free IB fabric topology ibnetdiscover ibstat free IB port state ibstat( [A-Za-z0-9_]+)? ibv_devinfo free RDMA device attributes: MTU, speed ibv_devinfo -d [A-Za-z0-9_]+ kernel_taint free kernel taint mask cat /proc/sys/kernel/tainted lldpctl free switch name and port per NIC lldpctl -f json ls_ib free IB device enumeration ls /sys/class/infiniband ls_net free ethernet interface enumeration ls /sys/class/net ls_sysfs free enumerate a sysfs subtree (ports, counters) ls /sys/class/(net|infiniband)(/(?!\.\.?/)(?!\.\.?$)[A-Za-z0-9_.-]+)+ lscpu free cpu topology and NUMA lscpu lspci_gpu free pci enumeration lspci -nn -D mpirun_all_reduce_perf probe cross-node all-reduce sweep via mpirun, one rank per host mpirun -np \d+ -H (?!-)[A-Za-z0-9_.-]+:1(,(?!-)[A-Za-z0-9_.-]+:1)* --bind-to none( --allow-run-as-root)?( -x NCCL_[A-Z0-9_]+=[A-Za-z0-9_,.-]+)* ((/[A-Za-z0-9_.-]+)+/)?all_reduce_perf -b \d+ -e \d+ -f 2 -g \d+ -n \d+ -w \d+ -c [01] numactl free NUMA layout numactl --hardware nvcc_version free CUDA toolkit version nvcc --version nvidia_smi_compute_apps free processes currently resident on each GPU nvidia-smi --query-compute-apps=[a-z0-9_.,]+ --format=csv,noheader,nounits nvidia_smi_nvlink free NVLink link state nvidia-smi nvlink -s nvidia_smi_q free full query dump nvidia-smi -q nvidia_smi_q_domain free domain-scoped query; PERFORMANCE carries the throttle bitmask nvidia-smi -q -d (ECC|PERFORMANCE|CLOCK|POWER|TEMPERATURE|MEMORY|ROW_REMAPPER) nvidia_smi_query free structured per-GPU fields nvidia-smi --query-gpu=[a-z0-9_.,]+ --format=csv,noheader,nounits nvidia_smi_topo free intra-node GPU topology nvidia-smi topo -m nvidia_smi_version free driver/tool version nvidia-smi --version ofed_version free Mellanox OFED version ofed_info -s os_release free distro identification cat /etc/os-release python_stdin probe reference training step: our script on stdin, args = gpus, tokens python3 - \d+ \d+ python_torch_check free is PyTorch present and how many GPUs does it see python3 -c 'import torch;print\(torch\.__version__,torch\.cuda\.device_count\(\)\)' rocm_smi free rocm-smi fallback surface rocm-smi( --show(productname|topo|nodesbw|id|meminfo))? rocm_smi_version free rocm-smi tool version rocm-smi --version rocm_version free ROCm version cat /opt/rocm/\.info/version sinfo free slurm partition state sinfo -h -o %[A-Za-z] squeue free slurm queue depth squeue -h -o %[A-Za-z]+ squeue_node free jobs allocated to one node; the strongest 'do not probe' signal squeue -h -w [A-Za-z0-9][A-Za-z0-9._:@/-]* -o %[A-Za-z]+ srun_all_reduce_perf probe cross-node all-reduce sweep via Slurm on an allocation the operator grants srun( --mpi=(pmix|pmi2))? -N \d+ --ntasks-per-node=1 ((/[A-Za-z0-9_.-]+)+/)?all_reduce_perf -b \d+ -e \d+ -f 2 -g \d+ -n \d+ -w \d+ -c [01] sysfs_counters free bulk-read IB/RoCE port counters with their names grep -rH \. /sys/class/infiniband(/(?!\.\.?/)(?!\.\.?$)[A-Za-z0-9_.-]+)+/(counters|hw_counters)/? sysfs_read free sysfs counter read cat /sys/class/(net|infiniband)(/(?!\.\.?/)(?!\.\.?$)[A-Za-z0-9_.-]+)+ uname free kernel and arch uname -[amrsv]+ uptime free boot age uptime which free tool presence probe which [A-Za-z0-9][A-Za-z0-9._:@/-]*