9690SA 3Ware RAID Setup Help

[mokrunka]

Hello All

I'm helping my brother setup a server he has. The server has a 3Ware 9690SA Hardware RAID card, and it has 4 x 2Tb 5400 RPM Samsung Harddrives.

When I boot into the 3Ware BIOS, it recognizes all 4 drives, no problem. What he wants to do is install them in RAID 10 for a 4 Tb mirrored volume for performance and redundancy. When I try to create a RAID array with the 4 drives, and set it to use RAID 10, it only allows me to utilize 2079 Gb (1/4 x 8 Tb), which is half of what I would expect for this RAID config.

Assuming I go ahead and allow it to build the array with this volume, we can go on to install Fedora 14 with no issue, but still only 2 Tb are available.

This is our first time setting up a RAID array like this, so I'd appreciate any advice anyone can offer.

I've also tried enabling Auto-carving to insure that I have support for physical volumes larger than 2 Tb, but to no avail. With the version of Linux we're running, and the RAID card he's got, it should support greater than 2 Tb PVs either way though.

Thanks!

 

[Automata]

I don't have experience with their cards, but I do have an idea.

I see this in the features:

Auto carving allows LUNs > 2TB to be divided into multiple units for legacy operating systems limited to 2TB max. per LUN

When you create the array, it should ask how large you want it to be. It may default to 2 TB for compatibility reasons, but you should be able to go higher.

 

[mokrunka]

I don't have experience with their cards, but I do have an idea.

I see this in the features:

When you create the array, it should ask how large you want it to be. It may default to 2 TB for compatibility reasons, but you should be able to go higher.

Yea, I was reading the guide for this card online. Unfortunately, the auto-carving feature is enabled, but it is really only pertinent if you are using an older kernel version on Linux (and older Windows systems). I did get it to boot into Fedora with a 4 Tb RAID 10 array, but for some reason, 2 Tb of the volume can't be formatted... On to more research!

I know this is a pretty specific issue, so I appreciate you responding!

 

[Automata]

Did you create it as a GPT drive in the setup? I have not done an OS install on an array larger than 2 TiB before, even Linux. I'm not sure if it will let you.

To make it easier, you could create different partitions (as you should anyway...). Create the normal /, swap and /boot partitions and keep them decently sized. I fit my CentOS 5.5 install on around 3 GiB of space, minus swap. You could then leave the rest unformatted until you get the OS installed and then format the rest and mount it.

I do use a GPT partition as a storage drive because I use a 7 drive RAID 5 array for my data.

 

[mokrunka]

Did you create it as a GPT drive in the setup? I have not done an OS install on an array larger than 2 TiB before, even Linux. I'm not sure if it will let you.

To make it easier, you could create different partitions (as you should anyway...). Create the normal /, swap and /boot partitions and keep them decently sized. I fit my CentOS 5.5 install on around 3 GiB of space, minus swap. You could then leave the rest unformatted until you get the OS installed and then format the rest and mount it.

I do use a GPT partition as a storage drive because I use a 7 drive RAID 5 array for my data.

I did not create a GPT drive. What exactly is this? I'm guessing I mis-configured something in the Fedora installer... Time to do some research I suppose. I just did the basic install, just to see how much space it would come up with in Fedora.

I noticed that I have a 500 Mb section, a 1.8 Tb section, and then a 2.2 Tb section that says "free space". So, it seems to at least recognize that there is more than 2Tb there. When I try to partition said free space, it gives me an error, of course I don't remember what it is at the moment though... Any ideas?

 

[Automata]

A MBR can only go out to 2 TiB, hence why the controller has a safety feature for backwards compatibility. More information here:

http://en.wikipedia.org/wiki/GUID_Partition_Table

I'm firing up a VM now with CentOS and Fedora 14 to see if it lets me do it as GPT.

EDIT: Ugg, VMWare hates me tonight, might be a bit.

 

[visbits]

I don't think GPT can be booted if he has another drive for the OS mount this as /home or something.

You can change it to GPT by using parted.

parted /dev/sdb
mklabel gpt
print
mkpart primary ext3 0 4000GB
quit
mkfs.ext3 /dev/sdb1

Easy as pie.

 

[Automata]

But the problem is that is the drive they are booting to. If you remove the MBR, it won't boot...

 

[Automata]

Did some looking around and it looks like you can boot to GPT partitions, but it is difficult.

http://www.rodsbooks.com/gdisk/booting.html

Honestly, I'd suggest getting another drive just for the OS. This allows you to format the RAID array as GPT and there are no problems. I'm using three 30 GB SAS Seagate Cheetas in RAID 1 and it helps keep the load off the data disk and eliminates this silly GPT/MBR issue.

 

[mokrunka]

Ok, questions.

First, I have four 2 Tb drives, so if I dedicate one as an OS, do I still need to format the other 6 Tb worth of drives as GPT?

And, if I do that, when do I format them as GPT? Is that done before or after I setup the RAID array in the 3ware BIOS?

Finally, if I'm able to setup the GPT partition table for 3 drives @ 6 Tb, why wouldn't I be able to do the same for 4 drives @ 8Tb?

Thanks guys

 

[Automata]

If you have 3 drives, you can't run the RAID 10 setup you want and that drive is far too large for the operating system. As I said, my CentOS 5.5 install is under 5 GB. I'd really suggest getting a new/different drive so you can keep your RAID 10.

Beyond that, yeah, just change it to GPT and format.

The issue isn't formatting the drives to use GPT. That is the easy part. The problem is that the operating system can't be on a drive/array that has GPT, it just doesn't work without a huge amount of work. If you install the OS to a drive that does not have GPT, then you completely bypass the issue.

 

[mokrunka]

If you have 3 drives, you can't run the RAID 10 setup you want and that drive is far too large for the operating system. As I said, my CentOS 5.5 install is under 5 GB. I'd really suggest getting a new/different drive so you can keep your RAID 10.

Beyond that, yeah, just change it to GPT and format.

The issue isn't formatting the drives to use GPT. That is the easy part. The problem is that the operating system can't be on a drive/array that has GPT, it just doesn't work without a huge amount of work. If you install the OS to a drive that does not have GPT, then you completely bypass the issue.

Ok thanks. The only issue is that the computer is a 1U server, so I only have 4 hard drive bays--if I were to use another HDD, we'd have to basically leave it sitting on the top of the box. I need to do some thinking about how I want to handle this.

How is this done in industry? Surely they have much more hard drive capacity, do they just have one box with an O/S, and somehow are able to access the additional storage on the other boxes? He's got another machine running Fedora 14, would there be a way to utilize all 4 hard drives on the server, and use the other machine for the storage space?

 

[Automata]

Is there a way to put a drive internal to the server? You could use a 2.5" laptop drive or even a small SAS drive, like what I've done. It won't be pretty, but I'd rather do that than lose a bunch of space.

Accessing it from another box would require either a working operating system (problem) or low level hardware (aka: no computer in the box). Both of these defeat the purpose of that case.

 

[mokrunka]

Thanks Thideras--sorry it took me so long to reply.

Using your well-placed advice, I've developed a solution.

I installed a 2.5" laptop HDD, the least expensive one that I could find @ $40 from Fry's and used this for the OS drive. I then used gparted to format each file system to GPT, which allows for filesystems larger than 2 Tb.

I did some more research (prompted from the earlier post about GPT) and discovered that Fedora does not allow a bootloader to reside on a RAID array.

Before I broke down and picked up the laptop drive, I tried installing Fedora on the RAID array. The install went off without a hitch, but unfortunately it wouldn't boot for the reason above. All is working now, and I have a 4 Tb volume GPT partition and a 120 Gb OS drive.

My next question is how to use the space I've just created. I went ahead and made one large partition for each drive--do I now just need to mount it to make it work so that I can store files on it?

Really appreciate your help

 

[Automata]

If you have one volume, simply create the location you want it to reside in and add it to the /etc/fstab file. From there, setup services that you want and use that directory. You do want to keep the naming simple. For example, my server's RAID array is at /mnt/hitachi, which is the make of the drives. You can pretty much put it anywhere.

I'm not sure what you mean by "one large partition for each drive". I thought the four 2 TB drives were in a RAID 10 array.

 

[mokrunka]

I'm not sure what you mean by "one large partition for each drive". I thought the four 2 TB drives were in a RAID 10 array.

Hmmm. Interesting point. When I use the disk utility, it lists out the HDDs in the left pane, and it says that the 1.8 Tb and the 2.2 Tb drives are listed as "Peripheral Devices", but they both have GPT filesystems, and the both drives are free.

I'm assuming that they (the 4 x 2 Tb drives) are still in RAID 10, since that is what the 3ware BIOS indicates when I boot; is there a way to check this to confirm that they are in a RAID array in the OS?

Also, if I understand correctly, GPT is a partition table, so now that the OS recognizes all of the storage, I could format them for ext3 or ext4 with no issue?

 

[Automata]

If they are in RAID, the operating system should see them as one hard drive. When I do "fstab -l", this is what I see:

[root@thideras ~]# fdisk -l

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.


WARNING: The size of this disk is 6.0 TB (5997921828864 bytes).
DOS partition table format can not be used on drives for volumes
larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID
partition table format (GPT).


[COLOR=Red]Disk /dev/sda: 5997.9 GB, 5997921828864 bytes
255 heads, 63 sectors/track, 729205 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1      267350  2147483647+  ee  EFI GPT[/COLOR]

Notice that it does not see each drive, just the final array. From my knowledge, the only type of RAID that does not follow this is software RAID (Intel Matrix RAID or pure software).

 

[mokrunka]

[root@physics engineer]# fdisk -l /dev/sdb

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 4000.0 GB, 3999977701376 bytes
255 heads, 63 sectors/track, 486302 cylinders, total 7812456448 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdb1 1 4294967295 2147483647+ ee GPT

Success! Thank you for your help! I had to rebuild the RAID one last time and partition the drive to ext4, it seems to be working! whew, what a day...