• Welcome to Overclockers Forums! Join us to reply in threads, receive reduced ads, and to customize your site experience!

Raid 5 (6 physical drives) recovery help..

Overclockers is supported by our readers. When you click a link to make a purchase, we may earn a commission. Learn More.

Methal

Member
Joined
Aug 5, 2008
Location
DC
I work for a pacs company, for those of you who dont know, pacs is picture archiving for medical devices (modalities) Interesting mix between medical and technology.

Last month middle of the night the primary server at a large hospital crashed hard. Near as we could tell AT LEAST the raid card, idrac, and possibly mother board and a hard drive or some combo of them bit the dust. This server is 3 months out of dell warranty. So replacement/on site service is out of the question. The site already has a new replacement server, and is running on their back up for now.

When the server went down the site lost hundreds of studies (sets of images created by CTs, MRIs, Xrays etc.) because some dip**** stopped the archiving service. (this is a windows server....ugh I know!! who runs a hospital server on windows!?!)

Long story short, they are sending me the 6 hard drives from the dead server, my company wants me to get the data off of them. :shock: I've done this before with raid 0 and failed hardware before. I know its possible with gparted and DD. But this was 6+ years ago....

Any idea where I can start?
The first problem is that the emergency IT guy pulled ALL OF THE DRIVES OUT and got them all mixed up. So I do not know which is 0, 1, 2 etc.

on the plus side I have an exact replica of the server that went bad though.

This is how fun this is going to be, if the data is lost. The hospital gets sued by patients and in turn my company gets sued by the hospital because we are contracted to store this stuff for between 5 years and 20 years after the death of the patients. :chair:

to make it more fun, surgeries are digital now. if you have a brain bleed the surgeon watches a live feed from a mobile MRI/ECHO/CT/Whatever and compares it to past images stored in Pacs. That surgery (there are dozens) will be cancelled if those images are gone.

and you all thought hospital trips were perfect =) Let me let you in on a little secret. When you are on the operating table, sometimes a doctor stick his finger in you body to hold a bleed back while his assistant calls the pacs support to find out how to look something up in pacs, or change a view. You may be sleeping soundly, but half a dozen other people are freaking the F out =) especially if there is a bug in the program that prevents them from seeing what they need to see.

Anyway thoughts? tips? (prayers...? =) )
 
This is a super ugly situation and I would honestly tell them to get professional assistance so someone can take the blame (other than you) if it goes wrong.

You'd need to start by knowing what RAID controller they were using, which may make this a lot easier; most LSI cards will pick up on configurations. If the RAID controller can pick it up and repair, you are down to filesystem restoration. If the controller can't pick up, you are in bad territory. There might be programs to help you repair that.

If this is mdadm, it should be easy to put them in a READ ONLY (critical to do this) configuration and test the different orders, or let it try out what it thinks is right.

Seriously though, I wouldn't touch this with a 10 ft pole. If I was absolutely forced to work on it, I'd take images of the drives in case something goes wrong.
 
The raid card was either a perc i6 or i5. (I can look it up) both of which I have access to. not sure if that helps, but once I boot the machine into linux, if its anything like the ones i've done before on raid 0 I'll be able to DD the entire raid onto another device. Probably will need to be a 2tb external.

DD wont change anything on the drives, but will give me something to go from. It wont be bootable obviously, that i expected.

I plan to use Linux Mint, Gparted to view the devices, and raid, and DD to copy.

If I get very lucky I can use Gparted to mount the file sytem and then simply copy the Archive folder which is all I need.
 
well had this place been smart they would have also had all data off site somewhere to back up from... shame on them for not setting up a contingency plan..

but yeah that sucks.. if you can get the correct raid card you might be okay, but at the same time if you cant find the data they may have to send off for a prof service which we all know aint cheap lol..

good luck to you. let us know how it goes.

I primarily fix dell servers and 9 times out of 10 its not the servers fault lol.. theres usually someone on the back end pushing firmware updates and screws crap up.
 
A LSI (perc) card should pick up the configuration automatically. If you put the drives on the RAID card, it will not important a foreign configuration unless you tell it to. This will tell you if the recovery is going to be remotely easy.

Make sure when you boot a Linux disk that mdadm/dmraid is disabled so it doesn't mess anything up. I'd fire up the live disk, make sure everything is off, then insert the disks.

Professional recovery services absolutely have ways to reconstruct an array, then do data recovery. I would really push for letting another company handle it.
 
What a mess!

It has taken several days of work but finally I was able to get everything off of the devices.

There ended up being a couple issues, 1 the IT on the site is not fit to be flipping hamburgers. He pulled all the drives out and played musical chairs with them and plugged them back in. leaving the server unbootable, and completely useless.

C:/ was 2x 15k drives in raid1, D:./ was 4x10k drives in Raid 5. 1 of which was bad, 1 in failure predicted.

The 4 raid 5 drives had a partition of exactly 2.0 tb. Leaving 745 gigs of allocated space. No idea why they did this. =\

Luckily we had the exact duplicate machine here at work I could just plug the drives into, import foreign config, and let the raid rebuild. all I had to do was find the dead drive and replace it.

1 of the 15k drives was also dead, and the other failure predicted. I ignored this. I didn't need anything from the C drive so I just tossed em.
booted the machine from a USB running Mint, and used DD to clone the 2tb partition to a 3TB hd. This took 48 hours + I suspect because 1 of the drives was going south.

I was able to get 100% of the crap off those drives and put on a single drive. Sent this back out to the site and used robo copy to move the data back into the archive folders on their new server.

To think though, all of this could have been avoided if the site had a competent IT guy.

To further add to his spectrum of genus, once the 3tb external got on site, he plugged it in, and hit format when windows didn't detect the encrypted volume. I seriously lost my mind at this point. His excuse: "in 15 years of IT i've never seen a drive F before!"

Sad part of all this, he makes more than double what I make =/
 
There isn't a large enough facepalm possible for that situation.

............................................________........................
....................................,.-‘”...................``~.,..................
.............................,.-”...................................“-.,............
.........................,/...............................................”:,........
.....................,?......................................................\,.....
.................../...........................................................,}....
................./......................................................,:`^`..}....
.............../...................................................,:”........./.....
..............?.....__.........................................:`.........../.....
............./__.(.....“~-,_..............................,:`........../........
.........../(_....”~,_........“~,_....................,:`........_/...........
..........{.._$;_......”=,_.......“-,_.......,.-~-,},.~”;/....}...........
...........((.....*~_.......”=-._......“;,,./`..../”............../............
...,,,___.\`~,......“~.,....................`.....}............../.............
............(....`=-,,.......`........................(......;_,,-”...............
............/.`~,......`-...............................\....../\...................
.............\`~.*-,.....................................|,./.....\,__...........
,,_..........}.>-._\...................................|..............`=~-,....
.....`=~-,_\_......`\,.................................\........................
...................`=~-,,.\,...............................\.......................
................................`:,,...........................`\..............__..
.....................................`=-,...................,%`>--==``.......
........................................_\..........._,-%.......`\...............
...................................,<`.._|_,-&``................`\..............
 
You should just send your resume to the hospital its pretty clear they need help and you could make 2x what you make now.
 
I subbed to the thread from day one.


I'm pleased to hear it worked out in the end, not an easy task well done! :thup:
As for the IT dude........


I have nothing useful to say about him. :facepalm:
 
Back