A PCIe error in a GPU server does not automatically mean the accelerator is defective. Some link errors are corrected by hardware without losing data; others can make a device unreachable. The important distinction is between a recovered event, a recurring pattern that needs investigation, and a failure that has already disrupted work.
PCI Express Advanced Error Reporting (AER) gives administrators evidence about the connection between a processor’s PCIe root complex and devices such as GPUs, network adapters and NVMe drives. The Linux kernel’s AER documentation distinguishes correctable errors from uncorrectable non-fatal and fatal errors. AER reports conditions in a PCIe link or transaction; it does not, by itself, diagnose a failed GPU chip.
Understand the Three PCIe AER Severities
| Reported condition | What it means | Initial operational response |
|---|---|---|
| Correctable | PCIe hardware recovered the error without losing data at the interface | Record and trend it; examine recurring or worsening patterns |
| Uncorrectable, non-fatal | A transaction may be unreliable while the link remains usable | Investigate affected work and driver recovery; assess the node before returning it to normal service |
| Uncorrectable, fatal | The link is unreliable and recovery generally requires a reset | Treat as an incident, protect workloads and follow the platform’s recovery procedure |
“Correctable” describes what happened to a particular PCIe event, not whether the surrounding hardware is in perfect condition. A single corrected event can be benign. Repeated warnings after a riser change, a new accelerator installation or an unexpected drop in link speed deserve closer attention, without an invented universal replacement threshold.
This resembles, but is not the same as, the decision described in HW Server’s guide to correctable ECC memory errors. Both require looking at patterns rather than replacing hardware after one warning. ECC protects memory data; PCIe AER describes errors along an I/O path. Their counters should not be combined or interpreted as the same failure type.
Locate the Failing Path Before Blaming a GPU
A PCIe topology may contain a root port, one or more switches, risers, retimers and an endpoint. The device named in a kernel message may be the port that reported an error rather than the board that caused it. Administrators need the bus-device-function address, the upstream/downstream relationship and the physical slot map to understand where to investigate.
For example, Red Hat’s published PCIe AER log examples show a correctable physical-layer RxErr report as well as a fatal event followed by unsuccessful device recovery. The published examples establish the meaning of the log format, not that every RxErr points to a defective cable or riser. A full diagnosis needs the surrounding records and the machine’s actual topology.
On a supported Linux host, collect the relevant kernel journal with journalctl -k, map PCIe devices with lspci, and compare negotiated link width and speed with the hardware’s documented configuration. Preserve the records before rebooting or clearing counters. A link that negotiates at less than the design expects is useful context, but it is not conclusive proof of a particular failed component.
The topology becomes even more important in dense hardware. HW Server’s analysis of a 160-bay NVMe server illustrates why many endpoints can sit behind PCIe switching rather than connect directly to a single host port. The same general mapping discipline applies to GPU risers, switched backplanes and multi-device expansion systems.
Treat NVIDIA Xid Messages as a Second Layer of Evidence
AER is a PCIe reporting mechanism. NVIDIA Xid messages come from the GPU driver and describe GPU-related events; the two can appear near each other in an incident without being interchangeable.
In NVIDIA’s Xid catalog, Xid 79 means the driver tried to reach a GPU over PCIe and found it inaccessible. The documented causes can involve the PCIe path, GPU hardware or driver problems. Xid 74, by contrast, concerns an NVLink connection, not an ordinary PCIe AER classification. Neither code alone identifies a replaceable part.
For a missing GPU, correlate the Xid timestamp with PCIe AER messages, system event logs, recent changes, GPU enumeration and the status of related devices. If the machine has lost an accelerator used by production work, isolate or drain the affected node under the site’s incident procedure rather than treating the condition as a harmless corrected warning.
Use an Evidence-Based Repair Decision
A good troubleshooting sequence minimizes unnecessary hardware swaps and preserves the information a vendor needs:
- Save the incident record. Capture kernel and GPU-driver logs, time of failure, affected PCI addresses, workload symptoms, and any out-of-band hardware events before a reboot if safe and possible.
- Check severity and recurrence. Separate isolated corrected events from sustained bursts, uncorrectable events, link resets and devices disappearing from the operating system. Kernel message rate limiting means console line counts alone are not a complete event tally.
- Map the whole connection. Compare the documented PCIe topology, risers, switches, retimers, slot assignments, link width, link speed and recent firmware or hardware changes.
- Use approved maintenance procedures. Only qualified personnel should inspect seating, connectors and supported firmware changes during the appropriate service window. Never reseat internal PCIe cards in a running server unless its manufacturer explicitly supports the operation.
- Escalate with a reproducible evidence package. NVIDIA’s GPU node-triage guidance calls for driver and OS details, full logs, debug steps and
nvidia-bug-report.shoutput when applicable. The vendor can then advise on diagnostics, resets and component replacement.
Avoid simply suppressing AER messages to make dashboards look healthy. Also avoid assigning a generic error-count threshold to every server platform; policies should reflect the vendor’s guidance, the workload and evidence of actual instability.
The useful rule is to diagnose the path before replacing the endpoint. A corrected PCIe error is evidence of successful recovery at that moment. A repeated pattern calls for investigation; a fatal event or vanished GPU calls for controlled incident response. The error label is the beginning of the diagnosis, not the result.
