Hard Disk Failing
Jul 25, 2021, 8:58 am
#1
FlyerTalk Evangelist
Original Poster
Join Date: Nov 2002
Location: ORD
Posts: 14,231
Hard Disk Failing
I wasn't really sure whom to ask but thought there are some IT pros here I could lean on. I have run an Ubuntu server in my basement for a long time now. Yesterday I discovered it was unreachable. After hooking up a monitor I saw I was getting a ton of I/O errors when it tried to boot. I thought the hard drive was failing. I ran fsck and it found and fixed a lot of errors. I ran it again, and it didn't find any more errors. Now I can boot the machine but there are odd issues, like I get a weird I/O error when I just try to run apt-get update.
The Disks utility says the disk status is OK but there are 6500 bad sectors. This seems pretty clear that it's a hard drive in the process of failing. I'm not worried about my data, which is on a separate RAID 1 array in the same machine and backed up to CrashPlan.
My questions are:
1. Am I correct in thinking this is a failing hard drive? Is there anything else I should check?
2. At this point the computer and disks are pretty old. I think I bought the computer (an HP desktop) for cheap in 2012 and set everything up then. Since we are in the process of moving I think I might just pull the RAID disks, wipe the bad one, and recycle the computer. Agree?
2a. Is there anything I should do to the RAID disks before pulling them? I'm planning to keep them so I can use them to set up a replacement computer later.
3. When we are settled and I find a deal on a new computer, i'll probably buy new larger disks for the RAID array. Is it pretty straightforward to migrate to new disks? I have the mdadm file backed up.
The Disks utility says the disk status is OK but there are 6500 bad sectors. This seems pretty clear that it's a hard drive in the process of failing. I'm not worried about my data, which is on a separate RAID 1 array in the same machine and backed up to CrashPlan.
My questions are:
1. Am I correct in thinking this is a failing hard drive? Is there anything else I should check?
2. At this point the computer and disks are pretty old. I think I bought the computer (an HP desktop) for cheap in 2012 and set everything up then. Since we are in the process of moving I think I might just pull the RAID disks, wipe the bad one, and recycle the computer. Agree?
2a. Is there anything I should do to the RAID disks before pulling them? I'm planning to keep them so I can use them to set up a replacement computer later.
3. When we are settled and I find a deal on a new computer, i'll probably buy new larger disks for the RAID array. Is it pretty straightforward to migrate to new disks? I have the mdadm file backed up.
Last edited by gfunkdave; Jul 26, 2021 at 7:32 am Reason: corrected RAID version
Jul 25, 2021, 8:48 pm
#2
Join Date: Nov 2000
Location: Upcountry Maui, HI
Posts: 13,308
I wonder if raid (which type of raid?) ever really helped you or not? (complexity vs ???) Did it?
1. Yes.
2. Agree.
2a. No idea (don't use raid, backup your critical data multiple ways)
3. No idea. (don't use raid.)
What type of RAID are you using?
-David
1. Yes.
2. Agree.
2a. No idea (don't use raid, backup your critical data multiple ways)
3. No idea. (don't use raid.)
What type of RAID are you using?
-David
Last edited by LIH Prem; Jul 25, 2021 at 9:31 pm
Jul 26, 2021, 2:52 am
#3
Join Date: Oct 2009
Programs: Don't you know who I am????
Posts: 297
A couple of thoughts:
1) As LIHPrem wrote, are those drives mirrored or striped? I would guess mirrored, but if not, if you lose or pull one drive, you lose all the data. It usually can be recovered but that can be expensive and time consuming.
2) Based on the age of that machine, if your data volume is striped, you should assume you WILL lose access to it if your boot RAID fails due to driver/hardware/firmware and chipset dependencies. You may be able to get a current copy of a driver, but does that work with the firmware on your machine? If not you might find yourself with a striped RAID that is logically and physically fine, but is inaccessible
.3) Have you tested your CP backup? Can you actually retrieve the files you need in a timely fashion? Not an indictment of CP, but no backup should be counted upon unless it has been proven to be fully restorable. That restore exercise should be repeated within the time window of the loss of data you can tolerate.
4) Have you considered making a data backup to physical media like an external USB drive, or if so equipped from this era, eSATA? Or even install a HDD internally and backup the data to that local drive, verify the backup can be restored, them pull that drive. Again, nothing wrong with CP, but the value of a local copy on an external drive cannot be over estimated.
5) Have you considered using an imaging program to create a bootable image of your excising BOOT volume on new hardware (SSD or HDD)? It sounds like you are going to transition to a new machine so this drive could be used as a boot device until you transition, then it could be used as a backup drive. Getting your machine to boot off a NEW modern HDD is essential unless you are immediately transitioning to new hardware.
6) It sounds like those are spinning disks that are over 10 years old. I would retire them as quickly as possible, or at best relegate them to a local backup role keeping your off-site backup. SSDS for boot and transactional data are the way to go. HDDs can be fine for DATA, but be sure to use a RAID rated driver if you intend to run RAID.
Best of luck, but I would act quickly.
1) As LIHPrem wrote, are those drives mirrored or striped? I would guess mirrored, but if not, if you lose or pull one drive, you lose all the data. It usually can be recovered but that can be expensive and time consuming.
2) Based on the age of that machine, if your data volume is striped, you should assume you WILL lose access to it if your boot RAID fails due to driver/hardware/firmware and chipset dependencies. You may be able to get a current copy of a driver, but does that work with the firmware on your machine? If not you might find yourself with a striped RAID that is logically and physically fine, but is inaccessible
.3) Have you tested your CP backup? Can you actually retrieve the files you need in a timely fashion? Not an indictment of CP, but no backup should be counted upon unless it has been proven to be fully restorable. That restore exercise should be repeated within the time window of the loss of data you can tolerate.
4) Have you considered making a data backup to physical media like an external USB drive, or if so equipped from this era, eSATA? Or even install a HDD internally and backup the data to that local drive, verify the backup can be restored, them pull that drive. Again, nothing wrong with CP, but the value of a local copy on an external drive cannot be over estimated.
5) Have you considered using an imaging program to create a bootable image of your excising BOOT volume on new hardware (SSD or HDD)? It sounds like you are going to transition to a new machine so this drive could be used as a boot device until you transition, then it could be used as a backup drive. Getting your machine to boot off a NEW modern HDD is essential unless you are immediately transitioning to new hardware.
6) It sounds like those are spinning disks that are over 10 years old. I would retire them as quickly as possible, or at best relegate them to a local backup role keeping your off-site backup. SSDS for boot and transactional data are the way to go. HDDs can be fine for DATA, but be sure to use a RAID rated driver if you intend to run RAID.
Best of luck, but I would act quickly.
Jul 26, 2021, 7:31 am
#4
FlyerTalk Evangelist
Original Poster
Join Date: Nov 2002
Location: ORD
Posts: 14,231
Thanks both. It's a standard Linux software RAID 1 using mdadm, mirrored across two drives. I have the mdadm.conf file and there is a RAID bitmap, which I gather is needed to recreate the array.
The drive that's failing is the boot drive, not part of the RAID. The RAID is just for data.
I have restored files from Crashplan several times over the last several years and just downloaded a folder the other day.
In response to the question about whether RAID has ever helped me, I guess the answer is no. I had set it up so that if one of the data drives failed I would still have a local copy and could pop in another drive. But it hasn't come up, since the drives haven't failed. And that's a good point about a separate boot device. I created a Ubuntu stick to get in to the computer so maybe I can just use that for a week or so.
The drive that's failing is the boot drive, not part of the RAID. The RAID is just for data.
I have restored files from Crashplan several times over the last several years and just downloaded a folder the other day.
In response to the question about whether RAID has ever helped me, I guess the answer is no. I had set it up so that if one of the data drives failed I would still have a local copy and could pop in another drive. But it hasn't come up, since the drives haven't failed. And that's a good point about a separate boot device. I created a Ubuntu stick to get in to the computer so maybe I can just use that for a week or so.
Aug 24, 2021, 12:11 pm
#5
Join Date: May 2006
Location: IAD
Programs: UA 1MM *G (recovered GS), SPG Nothing, Hilton Nothing, AA Nothing
Posts: 895
Many times that I've faced a failing drive, I ran Spinrite on it and it was fixed (either totally fixed or fixed 'enough' that I could recover data).
https://www.grc.com/sr/spinrite.htm
https://www.grc.com/sr/spinrite.htm
Sep 1, 2021, 10:13 am
#6
Join Date: Dec 2009
Location: RDU
Programs: DL DM+(segs)/MM, UA Ag, Hilton DM, Marriott Ti (life Pt), TSA Opt-out Platinum
Posts: 3,226
Many times that I've faced a failing drive, I ran Spinrite on it and it was fixed (either totally fixed or fixed 'enough' that I could recover data).
https://www.grc.com/sr/spinrite.htm
https://www.grc.com/sr/spinrite.htm
Edit: I don't know if it would work with a RAID disk though, never tried that.
Last edited by HDQDD; Sep 1, 2021 at 2:03 pm
Sep 3, 2021, 11:23 am
#7
FlyerTalk Evangelist
Join Date: Nov 2002
Location: Freeload Univ. Where are you sitting?
Posts: 14,818
).I used a separate RAID controller (Promise FastTrak) - they claimed that you could treat the RAID as an ordinary hard drive, and for some of the stuff I tied it worked. Not sure what will happen if you have to dig down into the nuts and bytes territory, but it's probably worth a try.
Sep 7, 2021, 7:19 pm
#8
FlyerTalk Evangelist
Join Date: Jun 2005
Posts: 38,410
Spinrite? Wow! I haven't heard of that since I used it on MFM drives. (You could look up MFM ).
I used a separate RAID controller (Promise FastTrak) - they claimed that you could treat the RAID as an ordinary hard drive, and for some of the stuff I tied it worked. Not sure what will happen if you have to dig down into the nuts and bytes territory, but it's probably worth a try.
).I used a separate RAID controller (Promise FastTrak) - they claimed that you could treat the RAID as an ordinary hard drive, and for some of the stuff I tied it worked. Not sure what will happen if you have to dig down into the nuts and bytes territory, but it's probably worth a try.
Sep 13, 2021, 8:14 pm
#9
Join Date: May 2006
Location: IAD
Programs: UA 1MM *G (recovered GS), SPG Nothing, Hilton Nothing, AA Nothing
Posts: 895
https://www.grc.com/sr/testimonials.htm