Virtual disk degraded or offline: what to do on Dell PERC and HPE Smart Array

A degraded virtual disk (HPE calls it a logical drive or volume) is still serving data but has lost some or all of its redundancy. An offline or failed one is no longer usable. The safe response is different for each, and the most common way to turn a recoverable situation into data loss is to replace or reinitialise the wrong thing in a hurry. This guide covers what Dell and HPE documentation says each state means and what to do, in order.

What the states mean

HPE's drive procedures list how many failures each level tolerates: RAID 0 none; RAID 5 one; RAID 6 two; RAID 50 one per parity group; RAID 60 two per parity group; RAID 1 and RAID 10 several, as long as no two failed drives are mirrored to each other.

Step 1 — Identify the affected volume and drives

1. Dell: in iDRAC9 open Storage > Overview > Virtual Disks and Storage > Overview > Physical Disks, and review Maintenance > Lifecycle Log for the order in which drives failed. Command-line options are in PERCCLI status commands.
2. HPE: in iLO open System Information > Storage and Information > Integrated Management Log, or run SSA. HPE's documented signs of a failed drive include an amber drive status LED, a POST message listing failed drives, a distinctive icon in SSA and an IML entry.
3. At the server, light the drive first with Blink in iDRAC or Identify Device in SSA before you pull anything.

Step 2 — Degraded: replace the failed drive and let it rebuild

• Dell: the PERC 11 corrective action for a failed physical disk is to update the PERC firmware to the latest version and replace the drive. If a hot spare is assigned, the controller starts the rebuild on it automatically. A rebuild fails to start if the replacement is too small or incompatible. Full procedure: replace a failed disk on Dell PERC.
• HPE: with hot-plug drives in a fault-tolerant array, drive activity pauses for 1 or 2 seconds when the new drive is inserted, then data recovery starts automatically. If you replace the drive with the power off, a POST message asks you to press F1 to start automatic data recovery; until you do, the volume stays in a ready-to-recover condition. HPE also says the controller immediately fails replacement drives that are too small. Full procedure: replace a failed drive on Smart Array.

Expect the rebuild to slow down under load: Dell notes one rebuild I/O for every five host I/Os. See RAID rebuild time and risks. HPE warns that if another drive fails while fault tolerance is unavailable during a rebuild, all data on the array can be lost, which is why the backup comes first.

Step 3 — Offline or failed: check connections before replacing drives

When several drives drop out at once, the cause is often cabling or a backplane, not several dead drives.

• Dell: PERC 11 says multiple simultaneous disk errors in one array typically point to a cabling or connection failure. Turn off the system, check cables, reseat the disks, make sure all are present, then import the foreign configuration (see PERC foreign configuration). A virtual disk that went straight to Offline after a cable pull or power loss imports as Optimal without a rebuild. The controller keeps the offline disk's unwritten cache as preserved (pinned) cache until you import the disk or discard the cache.
• HPE: HPE notes that compromised fault tolerance can be caused by problems unrelated to drives, and that inserting replacement drives does not improve the condition of the logical volume. Its documented recovery is to power the whole system down and up; a marginal drive may work long enough to copy important files. If POST shows a 1779 message, HPE describes pressing F2, selecting Device Health Status, selecting 1779 and using the submenu to re-enable the logical volumes, and warns that data loss has probably occurred and the data is suspect. Copy what you can, replace failed drives, then restore from backup.

HPE also cautions that a drive the controller failed may seem to work again after a power cycle or reseat, and that continuing to use such a marginal drive can lead to data loss.

Step 4 — When to stop and call the vendor

• Fatal errors or data corruption are reported when accessing virtual disks (Dell's action: contact Global Technical Support).
• The iDRAC log shows PDR47 "A block on ... was punctured by the controller". Dell's recommended action is to back up, initialise the disk and restore from backup, so plan it with support and a verified backup.
• PERC 12 reports it is unable to rebuild because of an error on the source drive; Dell's action is to replace bad drives and recreate the virtual disk from backup.
• More drives are failed or missing than the RAID level tolerates, or the same drive keeps failing after replacement.

Collect a SupportAssist collection or an Active Health System log before opening the case.

Frequently asked questions

Can I keep running on a degraded virtual disk?

It keeps serving data, but it has lost redundancy. Dell's guidance is to replace the failed disk and rebuild; HPE warns that another failure while fault tolerance is unavailable can lose all data on the array.

Will replacing drives fix a failed HPE logical drive?

No. HPE states that when fault tolerance is compromised, inserting replacement drives does not improve the condition of the logical volume.

Checklist

• Backup confirmed before any drive is touched.
• Affected volume, failed drives and failure order identified from iDRAC/iLO logs.
• Degraded: compatible drive of equal or larger capacity installed; rebuild monitored to Optimal/OK.
• Offline or failed: cables and seating checked with power off before replacing drives.
• No initialise, delete, clear or re-enable action taken while data may be recoverable.
• Logs collected and vendor case opened for punctures, corruption or multiple failures.

Skyline can supply, configure and maintain Dell, HPE and Huawei servers under an annual maintenance contract (AMC). Contact us.