When a drive in a RAID 1, 5, 6, 10, 50 or 60 volume fails, the controller rebuilds the missing data onto a spare or replacement drive using the mirror copy or the parity on the remaining drives. Until the rebuild ends the volume is degraded, and with RAID 1, 5 or 10 it has no redundancy left where the failure happened. This guide explains what drives the rebuild time on Dell PERC and HPE Smart Array / MR controllers, which settings you can change, and how to reduce the chance that a rebuild turns into data loss.
Before you start
- Confirm you have a current, restorable backup of the volume before you touch any drive.
- Identify the failed drive in iDRAC, iLO or the controller utility, and check the other drives in the same array for errors.
- Have a replacement of the same type (SAS or SATA, HDD or SSD) and at least the same capacity ready.
Why the degraded window matters
HPE's controller guide spells out what a second failure means during a rebuild:
- RAID 1 and 10 lose no data as long as no failed drive is mirrored to another failed drive.
- RAID 50: all data is lost if a second drive fails in the same parity group before the first rebuild finishes.
- RAID 60: all data is lost if a third drive fails in a parity group before one of the other failed drives has finished rebuilding.
- RAID 6 survives any two failures. Dell's PERC guide notes this includes a failure while a single disk is rebuilding.
A complete drive failure is not the only risk. A rebuild reads every block on the surviving drives, so a bad sector found then can stop the data from being recreated. HPE controllers handle this with a puncture: the rebuild completes and the volume stays available, but the data in the affected stripe is lost and must be restored from backup.
What sets the rebuild time
- Amount of data per drive and drive speed. HPE states that a RAID 5 or RAID 6 rebuild generally needs about 15 to 30 seconds per gigabyte. That works out to roughly 4 to 8 hours per 1,000 GB, but actual time depends on I/O activity, the number of drives in the logical drive, the rebuild priority setting and drive performance.
- Host load. Dell's PERC 11 guide says a disk under high I/O stress rebuilds more slowly, because there is only one rebuild I/O for every five host I/Os. Dell's fix is to reduce I/O stress or raise the rebuild rate.
- Strip size. HPE notes that the smaller the strip size (64 KiB or less), the longer background parity scans and rebuilds take, and the more they affect host I/O.
- Other background work. On PERC 11, automatic rebuild is held during a full initialization and starts afterwards, and a RAID level migration or capacity expansion in progress delays any automatic rebuild or copyback until it completes.
- Controller features. HPE Smart Array SR controllers include rapid rebuild technology for all RAID levels except RAID 0.
Rebuild rate and priority settings
| Controller | Setting | Where |
|---|---|---|
| Dell PERC 11 | Rebuild Rate | HII: Main Menu > Controller Management > Advanced Controller Properties > Task Rates |
| HPE Smart Array SR | Rebuild Priority: Low, Medium, Medium High, High | SSA, or SSA CLI modify rp= |
| HPE MR Gen11 | Rebuild rate, 1 to 100 | HPE MR Storage Administrator: More Actions > Set Adjustable Task Rate, Priority Percentage column |
HPE defines its priorities as follows. Low: normal system operations take priority over the rebuild. Medium: rebuilding for half of the time. Medium high: the rebuild gets higher priority than normal operations. High: the rebuild takes precedence over all other operations. On MR controllers a higher number means a faster rebuild and possibly slower system I/O.
ssacli ctrl slot=3 modify rp=highA higher setting shortens the degraded window but slows applications. Choose based on which costs more for that server: a slow application for a few hours, or a longer period without redundancy.
Step 1 — Prepare before replacing the drive
HPE's "before replacing drives" checklist applies to any controller:
- Make sure the array has a current, valid backup.
- Check that no other drive in the same array is reporting errors.
- Confirm the replacement is the same type as the degraded drive (SAS or SATA, hard drive or SSD).
- Use a replacement at least as large as the smallest drive in the array. HPE controllers immediately fail drives that are too small.
- With external storage, power the server down first and up last, so drives are not wrongly marked as failed.
Warning: Never remove more drives than the RAID level can tolerate. If fault tolerance is already lost, HPE advises delaying drive replacement until you have tried to recover the data. Back up first whenever the volume is still readable.
Step 2 — Replace the drive and monitor the rebuild
- Locate the correct bay with the locate LED, then hot-swap the failed drive.
- On Dell PERC, a replacement inserted into a slot that belongs to a redundant virtual disk starts a rebuild or copyback automatically, regardless of any data on the inserted drive. On HPE, a rebuild starts when the drive is replaced if no online spare was available.
- Watch progress in iDRAC, iLO, the controller utility or, on HPE MR, the Background Processes in Progress window.
- If you must replace more drives than the RAID level tolerates, follow HPE's rule: replace within the tolerance, wait for the rebuild to complete (shown by the drive LEDs), then continue.
Vendor-specific procedures: Dell PERC failed disk replacement and HPE Smart Array failed drive replacement.
Step 3 — After the rebuild
- Review the controller and system logs for media errors. HPE recommends reviewing the IML and OS event logs for evidence of data loss or puncture.
- Run a consistency check on Dell PERC, which verifies and corrects mirror or parity data on fault-tolerant virtual disks (see patrol read and consistency check).
- Put a new hot spare back in place if one was used.
Warning: HPE states that a punctured stripe is only removed by deleting the volume and re-creating it with Rapid Parity Initialization or after erasing the drives. That destroys the volume's data, so back up and plan a restore before doing it.
Reduce the risk before the next failure
- Replace on predictive failure. Dell's Auto Replace Member on Predictive Failure and HPE's predictive spare activation copy data off a failing drive while the array is still healthy. HPE says this is up to four times faster than a typical rebuild. See hot spares explained.
- Keep media scans on. Dell patrol read runs every seven days by default on configured SAS and SATA hard drives. HPE recommends a high surface scan priority and current drivers and firmware to minimise punctures.
- Use RAID 6 or 60 for large arrays so a second problem during a long rebuild is survivable.
Checklist
- Backup verified before any drive work.
- Other drives in the array checked for errors.
- Replacement drive matches type and is at least as large as the smallest member.
- Rebuild rate or priority set deliberately for the time of day.
- Rebuild monitored to completion; no further drives removed meanwhile.
- Logs reviewed for media errors or punctures; consistency check run.
- Hot spare replaced and predictive replacement enabled.
Skyline can supply, configure and maintain Dell, HPE and Huawei servers under an annual maintenance contract (AMC). Contact us.
Comments
0 total · 0 threads