How to recover data from Punctured RAID array?

[Evans]

I am using a Dell Server with LSI controller. Unfortunately, I had a drive which is giving “Failure Predicated Warning”. I have also contacted Dell support and as per their suggestion replaced the drive and array rebuilt itself.

Today, another drive, giving the similar “Failure Predicated Warning” so I think maybe it is a bad batch of drivers. So I called Dell support again to check the problem in depth and after a brief conversation I realized that there were bad blocks on one of the drives that did not fail, but those bad blocks were copied over during rebuild. Now, the bad blocks are all over the place and they are gently destroying my array . Overall, I figured out that this is normally called as a Punctured Array.

So I think to replace all the drives, rebuild the array and restore from backup. But as I am facing this issue since last couple of weeks that means my backups may be in the worst situation and if I restore from backup, then I will miss almost month’s important data from my database which I cannot afford at the moment.

So I would like to know that, have you recovered anything from like this situation before without loss of any data?

[paullopez]

It seems like your system is still running so you can take direct backup, you can dump the disk/array, rebuild and restore from the backup.

You look too much worried about bad blocks, but bad blocks do not mean that your backups are also in the worst situation. If there are no performance issues or damaged files, then I think backups should complete enough to finish a restore.

However, it would be a lot better if you take the most recent backup and analyze your important data. If it is intact, you likely have a good backup.

Although, in this case a risk is also involved as you cannot be totally sure that your backups are in a good situation or taking backups now will not cause any file loss. However, your array will ultimately fail and force a restore anyway, so this is your only realistic option.

[dWight]

IMO, you should halt spinning backups or deleting old one from your system. You need to keep all the backups you currently have. So try to take a full server backup. If the disks are in good condition then your data may probably intact and you will not come across with any problems while running the new full backups.

Once the backup process done, check whether the data is intact or not. If it’s intact, then scrap those disks and build a new RAID array. Once that is complete, try to restore from the backup you have taken.

If that fails, then try the old backup, and the next oldest, etc.… Also make sure to check the functionality of the system, just because it boots, does not mean it’s fully operational. Specifically, check the databases for corruption.

It is OK if you need to restore the whole system from an older backup. Make sure you take the latest backups and just restore the database files and other important files. Don’t forget to test them to ensure that they work appropriately. Following this procedure will reduce the risk of entire data loss.

[Chris Jenkins]

Your system I assume is still up, so the best thing to do is make an immediate backup, dump the disks/array, rebuild, and restore from the backup.

Bad blocks don't always mean your backups are also bad. If you haven't experienced any performance problems or damaged files, then your backups should still be complete enough to finish a restore.

To test, take your most recent backup and examine your most important data. If it's still intact, you likely have a good backup.

At this point, there is a risk involved as you cannot be 100% certain that your backups are good or that backing up now won't cause file loss. However, your array will eventually fail and force a restore anyway, so this is your only real option.