File Server - Onboard/Hardware RAID Questions

[JigPu]

I'm several years in the past when it comes to technology I've used (my desktop hasn't changed much at all since I built it 5 years ago :D), so even though I've done quite a bit of reading I'd like some input.

~~~~~~~~~~

As photos begin to fill my disk, I'm looking more and more intently at building a file server. I'm thinking about using RAID-5, but am not sure I want to spend the extra money for a hardware RAID card...

From what I've seen online, RAID-5 tends to perform poorly when using an onboard controller. However, would it perfom so poorly that it would bottleneck gigabit ethernet? Are there any faster networking options available (since gigabit ethernet doesn't seem all that fast compared to a RAID array)?

Also, what kind of support should I expect for Linux? The hardware cards I've seen online indicate that they have Linux support, but what does that mean? Just drivers? Array monitoring applications? What about onboard RAID?

JigPu

[thor17usa]

In terms of RAID, onboard solutions have definitely gotten better in recent years. I have been using RAID solutions for about 10 years now, starting out with SCSI RAID solutions that were passed down to me. Maintaining a SCSI solution was too costly then and still is now, so I abandoned it for SATA drives about 5 years ago.

I was using a Promise 4 drives hardware RAID controller for a few years because it was a great solution for the time, providing a good mix of software/hardware integration. The Promise S150 card was not a dedicated RAID 5 solution, but provide great performance. Really you need to think about how much you want to spend on the total solution. There are some enclosures that allow for easy RAID 5 storage, but are limited in what they can do and are sometimes slow on performance.

Another solution would be to build a storage server. I went with an Intel ICH9R based mobo with 8 SATA ports. 6 SATA ports are serving 6 750GB Seagate drives while the other ports service OS and extra storage drives. This is software/hardware solution similar to my Promise card, but the tech is much newer and performs a lot better. I have the 750GB drives in RAID 5 for 3.75GB of storage and have not had any problems yet (crossing fingers).

Whatever you decide on, just make sure you do your homework since any solution with RAID 5 will cost some cash.

[Xenocide]

Another solution would be to build a storage server. I went with an Intel ICH9R based mobo with 8 SATA ports. 6 SATA ports are serving 6 750GB Seagate drives while the other ports service OS and extra storage drives. This is software/hardware solution similar to my Promise card, but the tech is much newer and performs a lot better. I have the 750GB drives in RAID 5 for 3.75GB of storage and have not had any problems yet (crossing fingers).

Whatever you decide on, just make sure you do your homework since any solution with RAID 5 will cost some cash.

Too bad thats a software raid solution :(

You will be spending some nice cash getting a hardware based setup.

[Madwand]

Linux RAID should be better than on-board RAID, and should be sufficient for your purposes, at least when write-back caching is enabled. The bottlenecks will tend to be in the OS and file transfer protocol (esp. SMB). Getting a NIC with good drivers could help -- e.g. Intel.

[Malpine Walis]

Well JigPu, I am not sure quite what you are asking for. Your post reads to me as if you have run a few questions together and it might help if you could clarify the matter.

The first thing to look at is what comparisons you are making. Sure level 5 should be as fast as you can get, if it is implemented on a dedicated RAID card. And sure, if you implement that array on your BIOS, there will be a hit on your processor. That much should be fairly obvious.

However, not knowing what web sites you have been reading, I don't really know if that is the comparison that they are making. They could be making a comparison that is actually biased away from the one type of array that they don't care much for. If they choose their words carefully, it is fairly easy to hide that bias so that it is not really obvious. In fact, they could even be hiding the bias from themselves.

The second thing to look at is how you think that a poorly setup array is going to saturate gigbit ethernet. Realistically, I would expect bigger problems from a well setup card based level 5 array than from a poorly performing mobo based one, all else being equal.

In any case, you indicate that you are looking for a system to store and retrieve photographs. But what type of photos and how are you pulling them off the array? If you are storing sub ten megapixel photos from a consumer camera and pulling them one at a time, I really don't see a problem. If you are pulling a batch of a hundred personal pics at a time (or one of the huge mosaics from the Hubble telescope), then you might manage to fill up the FIFO buffer on the transmitting (server) end. Even so, it will be gigabit ethernet and the buffers will clear quickly enough.

Third, what else will be going on with this server? If it is just for storage and possibly some other light uses such as a voice chat for games (you get to use the soundcard in your main rig for the game sounds and the voice chat is only in your headphones, which is worth a few wootage points), then the extra load on the processor from implementing a level 5 array on the mobo should not be a huge issue.

Now, if you were trying to play Crysis on your server and had a mobo based level 5 array, that would be asking for some trouble. Every asset in the game would basically have to be processed when it goes through the controller and processed again in game. Such would not be a huge deal if you had a dedicated card controller but then you would have to build out a file server to also be a gaming rig. I am going to guess that that is not what you had in mind.

As far as Linux support, I really don't know. I can nose around in the Ubuntu menus and find what I need but it often takes me a bit of time and a couple of false starts. But if you find a card that you are interested in, would you not then check the HCL for the distro that you are using and the manufacturer's data sheet to at least get an idea of what potential problems you will run into?

Also, ebay seems to be a decent place for RAID cards. I did a search for “4 sata ii” and came up with several sellers who have cards with 4 sata 3.0 ports internally running between $25 and $50. You would, of course, have to research a specific card and you may have to wait for the right deal to come your way. I routinely wait for low priced ending today auctions on fairly expensive stuff and I find that once I have made the decision to buy something, the right auction usually comes my way in under two weeks.

[noegruts]

As photos begin to fill my disk, I'm looking more and more intently at building a file server. I'm thinking about using RAID-5, but am not sure I want to spend the extra money for a hardware RAID card...

From what I've seen online, RAID-5 tends to perform poorly when using an onboard controller. However, would it perfom so poorly that it would bottleneck gigabit ethernet?

I built a Windows Home Server machine a while ago based on an ASUS P5K-E motherboard. The ICH9R southbridge on this board gives me more than 100MB/sec *write* speed on my five-disk RAID 5 array in the server. The read speed is (of course) even higher, but the important point is that the controller is good enough to saturate the gigabit ethernet connection.

[noegruts]

Malpine, I am sure you mean well, but as tusken pointed out, you seem to state a lot of things as fact when you are just plain wrong. Some examples:

Sure level 5 should be as fast as you can get

Utter nonsense. The speed of RAID 0 will outstrip RAID 5 every time.

Realistically, I would expect bigger problems from a well setup card based level 5 array than from a poorly performing mobo based one, all else being equal.

That makes absolutely no sense.

...you might manage to fill up the FIFO buffer on the transmitting (server) end. Even so, it will be gigabit ethernet and the buffers will clear quickly enough.

Where do you get this stuff from?

[noegruts]

The bottlenecks will tend to be in the OS and file transfer protocol (esp. SMB).

Agreed 100%. SMBv1 (in pre-Vista & Windows Server 2008 machines) can only deliver about 30-40% of the throughput that gigabit ethernet offers. If you are communicating between two machines that both support SMBv2 (ie Vista or WS2k8), and you have carefully tweaked various gigabit ethernet settings (eg jumbo frames, TCP parameters etc) then you can approach the throughput potential of gigabit ethernet with SMB.

FTP has no problem saturating by gigabit ethernet connection and can deliver over 100MB/sec.

Note that SMB is the protocol used by Windows file sharing, so even if you've never heard of it, you're probably using it.

[Jon]

I'm going to give my usual plug for unRAID (http://lime-technology.com) in this situation. It's cheap, it grows with you, is simple to use, does not require new or expensive hardware and is fast enough (I use it as a media server, so I can't imagine photo storage would require the speed of striping - not to say this it isn't nice).

[Malpine Walis]

Where do you get this stuff from?

Well, you see, there is this thread at the top of the forum that is titled as the official RAID sticky. Now if I am wrong, then so is the information in the sticky. And I do not mean the posts by certain members but in the links that explain matters.

So are you saying that the sticky is full of wrong information?

That makes absolutely no sense.

OK, let me make a suggestion. Instead of posting in a combative tone, the next time you are confused, try asking me for some clarification on what it is that you do not understand. In this case, I will offer that you took that one line completely out of the context in which it was embedded. Now, in truth, it is not in my power to not have that happen from time to time but if asked, I am fully willing to clarify matters.

Ask me nicely and I may just help you to understand what you have clearly missed.

M'kay?

[Cheator]

I'm going to give my usual plug for unRAID (http://lime-technology.com) in this situation. It's cheap, it grows with you, is simple to use, does not require new or expensive hardware and is fast enough (I use it as a media server, so I can't imagine photo storage would require the speed of striping - not to say this it isn't nice).

Very cool Jon.

[noegruts]

Ask me nicely and I may just help you to understand what you have clearly missed.

Thanks for your kind offer but I think I'll pass on getting any more misinformation from somebody that has stated on several occasions that RAID 5 is faster than RAID 0.

[JigPu]

Dang Malpine! That's what I call a reply! :)

However, not knowing what web sites you have been reading, I don't really know if that is the comparison that they are making. They could be making a comparison that is actually biased away from the one type of array that they don't care much for. If they choose their words carefully, it is fairly easy to hide that bias so that it is not really obvious. In fact, they could even be hiding the bias from themselves.
Onboard being generally slower than a card (due to all XOR operations being done on the CPU) is what I was referring to. I haven't even gotten into this-card-vs-that-card yet. I'm still debating if I want to spend $300 for a RAID card with hardware XOR, or to let the CPU do all the work.

The second thing to look at is how you think that a poorly setup array is going to saturate gigbit ethernet. Realistically, I would expect bigger problems from a well setup card based level 5 array than from a poorly performing mobo based one, all else being equal.
I have no personal experience with gigabit ethernet, and no experience with RAID-5. Because of this, I can only base my thoughts on what I've seen online.

Generally, I've seen that SMB on gigabit ethernet (reviews I've seen are all for onboard NICs) is less-than-stellar. 40MB/sec would be average. A RAID-5 array I assume would be able to read at least this fast, since single hard drives can achieve that througput.

The part I'm not sure about though is if onboard RAID-5 would be able to write 40MB/sec. Writes on a RAID-5 are supposed to be slow with onboard RAID, since the CPU does all the XOR calculations. However, what is "slow"? I'd be happy with 40MB/sec, since SMB-over-gigabit isn't going to do much better anyway :)

In any case, you indicate that you are looking for a system to store and retrieve photographs. But what type of photos and how are you pulling them off the array? If you are storing sub ten megapixel photos from a consumer camera and pulling them one at a time, I really don't see a problem. If you are pulling a batch of a hundred personal pics at a time (or one of the huge mosaics from the Hubble telescope), then you might manage to fill up the FIFO buffer on the transmitting (server) end. Even so, it will be gigabit ethernet and the buffers will clear quickly enough.
The photos are RAW photos from an 8MP Rebel XT. Average photo size is 8MB. Quite obviously, transferring a single photo would pose no problem. Even 100Mbit ethernet would do quite nicely.

What I'm considering (and perhaps it would just be better to reconsider this) is essentially backing up the photos to this server. Right now, 30GB of photos takes ages to backup, so I'm feeling the need for speed :D. At 40MB/sec a backup would take only 12 minutes to complete (which is quite reasonable). If onboard RAID-5 can write that fast, awesome. If it can (for example) only do 20MB/sec, less awesome.

Third, what else will be going on with this server? If it is just for storage and possibly some other light uses such as a voice chat for games (you get to use the soundcard in your main rig for the game sounds and the voice chat is only in your headphones, which is worth a few wootage points), then the extra load on the processor from implementing a level 5 array on the mobo should not be a huge issue.

Now, if you were trying to play Crysis on your server and had a mobo based level 5 array, that would be asking for some trouble. Every asset in the game would basically have to be processed when it goes through the controller and processed again in game. Such would not be a huge deal if you had a dedicated card controller but then you would have to build out a file server to also be a gaming rig. I am going to guess that that is not what you had in mind.
Mostly its just going to be a backup & file server. Lots of photos can change between daily backups (because of editing in Lightroom), which is why I'm concerned about write performance of the RAID-5. Other files (like videos) won't change at all, so I'm not concerned about them.

It may also see use as a http/ssh/bittorrent server (which shouldn't load the disk or network too much) and as a DVR (which would hit the CPU hard).

As far as Linux support, I really don't know. I can nose around in the Ubuntu menus and find what I need but it often takes me a bit of time and a couple of false starts. But if you find a card that you are interested in, would you not then check the HCL for the distro that you are using and the manufacturer's data sheet to at least get an idea of what potential problems you will run into?
Most products list Linux support on their box, so I'd imagine they'd be in the HCL. However, better safe than sorry, so thanks for the idea :) What I'm more interested in though is the included software -- I don't want my RAID dying without any software to notify me and let me rebuild the array ;) Hardware notification (buzzers, lights, etc) may be OK, but if I'm away it'd be useless.

JigPu

[noegruts]

Generally, I've seen that SMB on gigabit ethernet (reviews I've seen are all for onboard NICs) is less-than-stellar. 40MB/sec would be average. A RAID-5 array I assume would be able to read at least this fast, since single hard drives can achieve that througput.

The part I'm not sure about though is if onboard RAID-5 would be able to write 40MB/sec. Writes on a RAID-5 are supposed to be slow with onboard RAID, since the CPU does all the XOR calculations. However, what is "slow"? I'd be happy with 40MB/sec, since SMB-over-gigabit isn't going to do much better anyway :)

As I said above, SMBv1 is limited to about 40MB/sec over gigabit ethernet. SMBv2 (in Vista & Windows Server 2008) performs much better.

My Windows Home Server machine - which has a five-drive RAID 5 array - can write at more than 100MB/sec and read even faster (I don't remember off the top of my head), so the gigE network is the bottleneck. And this is with the ICH9R southbridge on my P5K-E motherboard.

Thus, since you can saturate the network with "just" a decent onboard RAID 5 controller, the only reason IMHO to go with an add-in RAID controller is if it adds a feature that you need, such as OLCE or email notification of a drive failure (which you mentioned might be of use to you). You can get all the performance you can use over gigE with Intel's ICH9R.

By the way, I'd consider WHS as a solution to your backup requirements. It makes daily backups a total no-brainer and because it is based on Windows 2003 Server you can run whatever else you like on it too. I run my domain's web and mail server on it with no problems.

[Madwand]

The part I'm not sure about though is if onboard RAID-5 would be able to write 40MB/sec. Writes on a RAID-5 are supposed to be slow with onboard RAID, since the CPU does all the XOR calculations. However, what is "slow"? I'd be happy with 40MB/sec, since SMB-over-gigabit isn't going to do much better anyway

Here's a graph with some local RAID 5 performance measurements I've made. A key to sequential RAID 5 write performance is write caching, not CPU performance, as shown by the chart.

Without caching, RAID 5 write performance averages around 1/2 the sequential transfer rate of a single drive in theory, but performance theory always needs to be checked with measurements in context.

http://i89.photobucket.com/albums/k203/Madwand0/ir5-3vs4-wb-on-vs-off.png

SMB v1 over gigabit can do much better than 40 MB/s, but that's not commonly seen in practice, which makes around 30 MB/s a more realistic goal for a random system.

[Malpine Walis]

Dang Malpine! That's what I call a reply! :)

Well, you should know me by now.

Onboard being generally slower than a card (due to all XOR operations being done on the CPU) is what I was referring to. I haven't even gotten into this-card-vs-that-card yet. I'm still debating if I want to spend $300 for a RAID card with hardware XOR, or to let the CPU do all the work.

Well, that was actually why I was asking how you would be using the box. Honestly, if it is going to be primarily a file server with very little load on the CPU, then one consideration should probably be that you have plenty of CPU for the job. As I said before, if you were also going to be doing high end gaming on the box, that would be asking for trouble of a different sort but since you won't be doing that, that is one point for not buying an expensive card.

I have no personal experience with gigabit ethernet, and no experience with RAID-5. Because of this, I can only base my thoughts on what I've seen online.

Nor do I have experience with Gb ethernet. However, you seem to be concerned with getting raw speed. But just how important will that be?

What I'm considering (and perhaps it would just be better to reconsider this) is essentially backing up the photos to this server. Right now, 30GB of photos takes ages to backup, so I'm feeling the need for speed :D. At 40MB/sec a backup would take only 12 minutes to complete (which is quite reasonable). If onboard RAID-5 can write that fast, awesome. If it can (for example) only do 20MB/sec, less awesome.

One point to consider would be if you would really be running a 30 Gb backup every day? Or would you do that on day 1 and then just back up the changes after that?

Honestly, wouldn't it be viable to mount the array as a drive on your main box and just use it that way? You have the availability and redundancy in place and after the one time huge backup, all you are doing is saving your work across the network each time.

It may also see use as a http/ssh/bittorrent server (which shouldn't load the disk or network too much) and as a DVR (which would hit the CPU hard).

Here, you might also want to consider a tuner card with a processor. You can get those for around a hundred bucks at newegg. Then the tuner card can handle your encoding goodness and you don't have to worry about trying to record your favorite shows during a scheduled backup.

[JigPu]

Oh, I do know you Malpine :D I'm just always pleased to see detailed replies :)

I'm glad to see that onboard RAID should be capable of doing the XOR more than fast enough. And as you pointed out, since the CPU should be free, I don't imagine there will be a bandwidth problem on the RAID end.

Its true that I won't be doing full a 30GB backup daily, so I guess any bottleneck that gigabit incurs won't be that bad. Daily backups will "usually" pretty small, though it'll still be annoying on the few occasions I modify a lot of files (e.g., I add a new keyword tag to most of my photos) or modify a few huge files.

The only problem with mounting the drive locally is that Lightroom doesn't support loading catalogs from network drives. Its a really annoying limitation, and one that a lot of people have complained to Adobe about. Hence using the server for backups. For other files, I probably will have the drive mounted locally since Media Player doesn't really care where the files are coming from.

JigPu