Replace a failed disk on Dell PERC and monitor the rebuild

When a drive in a redundant PERC virtual disk (RAID 1, 5, 6, 10, 50 or 60) fails, the virtual disk goes to Degraded. Dell's guidance is simple: replace the failed drive and let it rebuild; when the rebuild completes the virtual disk returns to Optimal. This guide covers how to find the right drive, what replacement is acceptable, what the controller does automatically, and how to follow progress. It is based on the Dell PERC 11 User's Guide, the Dell PERC and HBA Controllers Unified User's Guide (PERC 12) and the iDRAC9 7.xx User's Guide.

Step 1 — Identify the failed drive

1. In iDRAC9, open Storage > Overview > Virtual Disks and Storage > Overview > Physical Disks. Note the degraded VD and the failed drive's slot. Storage > Overview > Summary also shows recent storage events.
2. Make the drive easy to find: on the Physical Disks page select it and click Blink (Dell requires a real-time capable controller and a backplane for this). From RACADM: racadm storage blink:<PD FQDD>.
3. At the server, compare the drive carrier LEDs with the table below. Click Unblink when done.

Each carrier also has a green activity LED that blinks when the drive is accessed. Dell notes that in AHCI mode the status LED does not light.

Step 2 — Choose a compatible replacement

The PERC 11 guide requires the same protocol and technology (a SAS hard drive replaces a SAS hard drive; NVMe replaces NVMe) and, for a rebuild, equal or greater capacity. The PERC 12 guide spells out the rules in more detail:

• Same protocol (SAS for SAS, SATA for SATA). Cross-protocol replacement is not supported.
• Same media type (HDD for HDD, SSD for SSD).
• Capacity equal to or larger than the failed drive. Extra capacity may not be used.
• Same encryption type (SED/FIPS for SED/FIPS, non-SED for non-SED).
• Equal or higher speed; for SSDs, the same or higher endurance class.

Dell's troubleshooting section names a replacement that is too small or incompatible as the usual reason a fault-tolerant VD will not rebuild. If unsure, see dealing with predictive-failure drives for when to replace before a drive actually fails.

Step 3 — Hot-swap the drive

1. Pull the failed drive (status LED blinking amber) from its slot, following the server's service manual and anti-static precautions.
2. Insert the replacement drive in the same slot.
3. Watch the status LED: it should begin blinking green slowly (rebuilding).

A drive that was in a VD and is reinserted into the same slot can appear foreign for a short time before the rebuild starts; Dell describes this as a transient state. If a drive from another server shows as foreign, see PERC foreign configuration: import or clear.

What the controller does automatically

• No hot spare: the rebuild starts on the new drive in the failed slot.
• Hot spare present: the controller uses the dedicated or global hot spare to rebuild as soon as the drive fails. When you later insert a new drive in the failed slot and the rebuild is complete, PERC copies the data back to the new drive (replace member / revertible hot spare) and the spare returns to hot spare state. The VD stays Optimal during the copyback. More in global vs dedicated hot spares.
• Different slot: automatic revert happens only when the new drive goes in the same slot; otherwise a manual replace member is needed.

Dell notes that automatic rebuild or copyback does not start while a RAID level migration or capacity expansion is running, and is held off during a full initialization until it finishes.

Step 4 — Monitor the rebuild

• In iDRAC9, check Storage > Overview > Physical Disks and Virtual Disks until the drive is Online and the VD is Optimal.
• In System Setup, Main Menu > Virtual Disk Management shows the VD status, and the HII Dashboard lists background operations in progress.
• The Lifecycle Log records storage configuration changes.

Rebuilds take longer under heavy I/O. The PERC 11 guide states there is only one rebuild I/O for every five host I/Os, and suggests reducing load or raising the controller's Rebuild Rate. iDRAC9 also lists Rebuild physical disk and Cancel rebuild as operations, but a manual rebuild applies only when automatic rebuild is disabled. Dell warns that a cancelled rebuild leaves the VD degraded, so rebuild as soon as possible. For timing and risk, read RAID rebuild time and risks.

Frequently asked questions

What if two drives fail at the same time?

Dell says several drives becoming inaccessible at once usually points to cabling or connections, not drives. Its documented recovery is to power off, check cables, reseat drives, then import the foreign configuration. See degraded or offline virtual disk.

Should I update firmware first?

For a drive in Failed state, Dell's corrective action is to update the PERC firmware to the latest version on the support site and replace the drive.

Checklist

• Backup confirmed; degraded VD and failed slot recorded.
• Failed drive identified by Blink and amber LED, not by guesswork.
• Replacement matches protocol, media, encryption, speed and is at least as large.
• Replacement drive holds no data you need.
• New drive inserted in the same slot; LED blinking green slowly.
• VD back to Optimal; hot spare (if used) back to hot spare state.

Skyline can supply, configure and maintain Dell, HPE and Huawei servers under an annual maintenance contract (AMC). Contact us.