Questions about a new 20x1.5tb array (and old data)

[elbweb]

Currently I have 9x1.5TB drives, and I am moving it to a new system, new array, new controller, everything.

The current drives do not use raid, it is essentially just a bunch of data (WHS machine).

I have a few questions about this:

First, should I consider a 2 disk raid 1 for my main os drive? or just stick it in the main array?

Second, should I use 2 10x1.5tb raid 5 arrays over 1 20x1.5tb raid 6?

Since I current have all my data on the existing drives, one possibility is to create a 10x1.5tb array, moving my data there, and then create another 10x1.5tb array with the empty drives.

I am leaning towards creating a 13x1.5tb raid 6 array, copying all my data to it, then expand to the 20x1.5tb array. This would make it easier in the future that I could always add to the single array. I am aware that this will probably take at least a week to do (but I'm still hoping it won't)

Any thoughts, questions, comments?

The new machine will be as follows (main guts salvaged from an old gaming machine):

Norco 4220
Rocketraid 4320
Chenbro CK12803 Expander Board
Gigabyte GA-X48-DQ6
Q9450
4x2gb 1066
20xSeagate 1.5tb drive
And Possibly (still debating):2x320gb 2.5"

 

[Automata]

First, should I consider a 2 disk raid 1 for my main os drive? or just stick it in the main array?

If the system has to be up all the time, I would. If you make full backups of the main drive and can image it back, then it entirely comes down to: "Do I care if it is down for 20 minutes while I re-image the drive?" If not, then don't worry about RAID.

Second, should I use 2 10x1.5tb raid 5 arrays over 1 20x1.5tb raid 6?

Again, it comes down to a few factors. If you have spare disks ready to replace failed ones, RAID 5 should work fine. But if another drive fails while repairing or in the degraded state, then you lose all your data. With that many drives, I would really suggest RAID 6 at minimum.

Any thoughts, questions, comments?

The new machine will be as follows (main guts salvaged from an old gaming machine):

Norco 4220
Rocketraid 4320
Chenbro CK12803 Expander Board
Gigabyte GA-X48-DQ6
Q9450
4x2gb 1066
20xSeagate 1.5tb drive
And Possibly (still debating):2x320gb 2.5"

Finally, someone else chose the same case that I did. If you want to see how I put mine in, check the link in my sig labeled "Rackmount Overkill".

to the forums!

 

[neonblingbling]

I would be careful having a RAID 5 array that large...

Check out this article.

Basically, it's saying most SATA drives have an unrecoverable read error rate of 10^14, or 12TB. So with a RAID 5 array that is 13TB (9 disks with data, 1 parity), your entire array might go *poof* quite easily.

RAID 6? I couldn't tell you if it's easy to fail with a large array. Two reasons: 1) it's not in the article. 2) I'm on summer break.

 

[Automata]

I would be careful having a RAID 5 array that large...

Check out this article.

Basically, it's saying most SATA drives have an unrecoverable read error rate of 10^14, or 12TB. So with a RAID 5 array that is 13TB (9 disks with data, 1 parity), your entire array might go *poof* quite easily.

RAID 6? I couldn't tell you if it's easy to fail with a large array. Two reasons: 1) it's not in the article. 2) I'm on summer break.

Please read it again. It says when it is reading 12tb of data that it might hit an error when rebuilding the array.

The key point that seems to be missed in many of the comments is that when a disk fails in a RAID 5 array and it has to rebuild there is a significant chance of a non-recoverable read error during the rebuild (BER / UER). As there is no longer any redundancy the RAID array cannot rebuild, this is not dependent on whether you are running Windows or Linux, hardware or software RAID 5, it is simple mathematics. An honest RAID controller will log this and generally abort, allowing you to restore undamaged data from backup onto a fresh array.

This doesn't mean you can't have an array over 12tb and not have issues, it just means that you have an increased chance of failure; which is common sense...

 

[elbweb]

If the system has to be up all the time, I would. If you make full backups of the main drive and can image it back, then it entirely comes down to: "Do I care if it is down for 20 minutes while I re-image the drive?" If not, then don't worry about RAID.

Again, it comes down to a few factors. If you have spare disks ready to replace failed ones, RAID 5 should work fine. But if another drive fails while repairing or in the degraded state, then you lose all your data. With that many drives, I would really suggest RAID 6 at minimum.

Finally, someone else chose the same case that I did. If you want to see how I put mine in, check the link in my sig labeled "Rackmount Overkill".

to the forums!

I think that I will just go with the large raid 6 array. This is going to be used by a 32bit OS so only 2TB volumes can be used. I think I am going to create 13 2tb volumes, and then another 1tb one that will be the os drive.. then in the future add the drives in groups of 4 (meaning 3 2tb drives).

This will be used with windows home server...

I would be careful having a RAID 5 array that large...

Check out this article.

Basically, it's saying most SATA drives have an unrecoverable read error rate of 10^14, or 12TB. So with a RAID 5 array that is 13TB (9 disks with data, 1 parity), your entire array might go *poof* quite easily.

RAID 6? I couldn't tell you if it's easy to fail with a large array. Two reasons: 1) it's not in the article. 2) I'm on summer break.

So then raid 6, I should be fine. I would have enough time to recover the missing disk if something goes horribly wrong.

 

[neonblingbling]

Please read it again. It says when it is reading 12tb of data that it might hit an error when rebuilding the array.

This doesn't mean you can't have an array over 12tb and not have issues, it just means that you have an increased chance of failure; which is common sense...

Sorry, it does sound like I meant 'data might go missing at random for no reason'. I meant to say the *poof* might happen during the rebuild.

Since I have minimal to zero experience with RAID, I was always amazed at the redundancy of RAID 5 (since it was 'cheaper' than RAID 1). But after reading the article, I was a bit, I dunno, frightened.

But even today a 7 drive RAID 5 with 1 TB disks has a 50% chance of a rebuild failure.

50% chance of rebuild failure seems like a big chance to take. Although I'm not quite sure how the author got this number... I just wanted to point out how there could be a big problem here.

So then raid 6, I should be fine. I would have enough time to recover the missing disk if something goes horribly wrong.

From what I understand, I cannot simply agree with this. If you get a read error every 12TB (on average), then a 20x1.5TB RAID 6 array would have trouble rebuilding...

Example (please feel free to point out misunderstandings): Say you have a 20x1.5TB RAID 6 (27TB of data) array and one drive fails. That's ok, because RAID 6 can handle 2 drives failing. Assume you do get a read error once every 12TB. If so, then while you are rebuilding, you'd get a read error 12TB in. That's would count as 'drive 2' in failed total. After that, at 24TB (12TB x2), you get another read error. That means at this point, you'd loose out on the last 3TB if your RAID adapter allows you to keep the first 24TB it recovered, else you loose all data.

It just doesn't sound like a good idea to me.

 

[elbweb]

Sorry, it does sound like I meant 'data might go missing at random for no reason'. I meant to say the *poof* might happen during the rebuild.

Since I have minimal to zero experience with RAID, I was always amazed at the redundancy of RAID 5 (since it was 'cheaper' than RAID 1). But after reading the article, I was a bit, I dunno, frightened.



50% chance of rebuild failure seems like a big chance to take. Although I'm not quite sure how the author got this number... I just wanted to point out how there could be a big problem here.


From what I understand, I cannot simply agree with this. If you get a read error every 12TB (on average), then a 20x1.5TB RAID 6 array would have trouble rebuilding...

Example (please feel free to point out misunderstandings): Say you have a 20x1.5TB RAID 6 (27TB of data) array and one drive fails. That's ok, because RAID 6 can handle 2 drives failing. Assume you do get a read error once every 12TB. If so, then while you are rebuilding, you'd get a read error 12TB in. That's would count as 'drive 2' in failed total. After that, at 24TB (12TB x2), you get another read error. That means at this point, you'd loose out on the last 3TB if your RAID adapter allows you to keep the first 24TB it recovered, else you loose all data.

It just doesn't sound like a good idea to me.

So basically from this, given current technology, there is no way to create a redundant data system beyond a raid 0 application (when dealing with such large amount of data).

This doesn't make any sense to me.

From your article:

I’ve clearly tapped into a rich vein of RAID folklore. Just to be clear I’m talking about a failed drive (i.e. all sectors are gone) plus an URE on another sector during a rebuild. With 12 TB of capacity in the remaining RAID 5 stripe and an URE rate of 10^14, you are highly likely to encounter a URE. Almost certain, if the drive vendors are right.

This seems like why it is OK to me. In a raid 6 scenario, I have 2 parity drives, so I can recover from two errors.

Another from the article:

RAID 6 will protect you against this quite nicely, just as RAID 5 protects against a single disk failure today. In the future, though, you will require RAID 6 to protect against single disk failures + the inevitable URE and so, effectively, RAID 6 in a few years will give you no more protection than RAID 5 does today. This isn’t RAID 6’s fault. Instead it is due to the increasing capacity of disks and their steady URE rate. RAID 5 won’t work at all, and, instead, RAID 6 will replace RAID 5.

Anyway, someone will more insight who would help me out here would be greatly appreciated.

 

[neonblingbling]

This seems like why it is OK to me. In a raid 6 scenario, I have 2 parity drives, so I can recover from two errors.

• 20 x 1.5TB RAID 6 (27TB of data)
• Assume URE rate of 10^14, which equates to 12TB
• Assume that in worse-case-world, it will happen every 12TB

Drive #1 dies. Array starts to rebuild.
12TB later, you get a read error. This counts as drive #2 failing.
12TB later, you get ANOTHER read error. This counts as drive #3 failing.

RAID 6 can only handle 2 drives failing.

 

[elbweb]

• 20 x 1.5TB RAID 6 (27TB of data)
• Assume URE rate of 10^14, which equates to 12TB
• Assume that in worse-case-world, it will happen every 12TB

Drive #1 dies. Array starts to rebuild.
12TB later, you get a read error. This counts as drive #2 failing.
12TB later, you get ANOTHER read error. This counts as drive #3 failing.

RAID 6 can only handle 2 drives failing.

So what I previously said is true? There is no way to realistically have this data redundant? unless I split it into two raid 6 arrays...? meaning 16 usable drives for 20 drives?

 

[AceNZ]

• 20 x 1.5TB RAID 6 (27TB of data)
• Assume URE rate of 10^14, which equates to 12TB
• Assume that in worse-case-world, it will happen every 12TB

Drive #1 dies. Array starts to rebuild.
12TB later, you get a read error. This counts as drive #2 failing.
12TB later, you get ANOTHER read error. This counts as drive #3 failing.

RAID 6 can only handle 2 drives failing.

A read error is not the same thing as a drive failure. RAID 5 or 6 can support multiple read errors.

 

[neonblingbling]

A read error is not the same thing as a drive failure. RAID 5 or 6 can support multiple read errors.

Even while rebuilding?

 

[MongGrel]

I'd go with RAID 6 myself on something that large from just what I'd read over time researching into RAID, seems a lot of people managing servers do not really care for 5.

I'm not in IT so I do not know, but did a lot of research setting up a couple arrays and a lot of people seem to prefer 6 for large ones due to possible faults on rebuilds.

Thideras it seems to begin with, I'm just piling on I guess.


​

 

[visbits]

Run 2 hot spares in the parity. Lookup IBM raid 5EE, I'm not sure if any sas controllers support it.


http://publib.boulder.ibm.com/infocenter/eserver/v1r2/index.jsp?topic=/diricinfo/fqy0_craid5ee.html

 

[elbweb]

Run 2 hot spares in the parity. Lookup IBM raid 5EE, I'm not sure if any sas controllers support it.


http://publib.boulder.ibm.com/infocenter/eserver/v1r2/index.jsp?topic=/diricinfo/fqy0_craid5ee.html

I think I am going to have at least 1 hot spare, but the problem I have now is that I do not have the space for it (phsyically).

Possibly create the array as a 19 drive + 1... I dont think I would need that yet though.

 

[AceNZ]

Even while rebuilding?

RAID 5 will continue working with either 1 full drive failure or arbitrary read failures. If 1 drive fails, then any additional read errors (such as during a rebuild) are not recoverable.

RAID 6 will continue working with up to 2 full drive failures, or 1 drive failure plus arbitrary read failures. So you can have 1 failed drive and any read errors will still be corrected, including during a rebuild. If 2 drives fail, then any subsequent read errors are not recoverable.

 

[Automata]

I messaged this thread to my father, and here is what he had to say. Please remember that he does work in a facility that has tons and tons of storage (Exabytes). He doesn't work directly with the systems, but he does know a lot about them. I wanted him to post, but he didn't want to just "jump" in a thread like that

I agree with the guy that said a read error is not the same as a total drive failure. The folks saying that RAID 5 or RAID 6 won't be reliable over a certain size are assuming an entire drive fails. This does not happen much. Also, a well constructed RAID has at least one hot spare which is brought in as soon as the first drive fails, starting a parity reconstruct probably before a human notices.

We've got somewhere between 1 and 2 *Petabytes* of spinning RAID, almost all RAID 5 but some RAID 1. We've not had any cases of lost data that I can recall except where a controller fails and corrupts the disk or where there was not a hot spare, a drive failed, and nobody noticed before a second drive failure.

Bottom line though is that if the data is critical you should have two or three copies. Never trust just RAID.

 

[elbweb]

I messaged this thread to my father, and here is what he had to say. Please remember that he does work in a facility that has tons and tons of storage (Exabytes). He doesn't work directly with the systems, but he does know a lot about them. I wanted him to post, but he didn't want to just "jump" in a thread like that

I agree with the guy that said a read error is not the same as a total drive failure. The folks saying that RAID 5 or RAID 6 won't be reliable over a certain size are assuming an entire drive fails. This does not happen much. Also, a well constructed RAID has at least one hot spare which is brought in as soon as the first drive fails, starting a parity reconstruct probably before a human notices.

We've got somewhere between 1 and 2 *Petabytes* of spinning RAID, almost all RAID 5 but some RAID 1. We've not had any cases of lost data that I can recall except where a controller fails and corrupts the disk or where there was not a hot spare, a drive failed, and nobody noticed before a second drive failure.

Bottom line though is that if the data is critical you should have two or three copies. Never trust just RAID.

So from my understanding, I should only run into problems if the same part of three drives fail (i.e. if its a matrix type, simple, layout: that would mean when one drive failed, and another had a read error at the first bit, another disk could have a read error anywhere beyond the first bit, and it would be fine).

So my question now is, would it be recommended to use raid 6? or a raid 5 with a hot spare?

 

[Automata]

So from my understanding, I should only run into problems if the same part of three drives fail (i.e. if its a matrix type, simple, layout: that would mean when one drive failed, and another had a read error at the first bit, another disk could have a read error anywhere beyond the first bit, and it would be fine).

So my question now is, would it be recommended to use raid 6? or a raid 5 with a hot spare?

Basing off what he said, RAID 5 should work fine for you. Remember that it is an "average" of 1 bit out of every 12TB. The whole drive isn't failing.

But, and this is me speaking, I would run RAID 6 because I'm paranoid about losing two drives before noticing the RAID was degraded.

 

[elbweb]

Basing off what he said, RAID 5 should work fine for you. Remember that it is an "average" of 1 bit out of every 12TB. The whole drive isn't failing.

But, and this is me speaking, I would run RAID 6 because I'm paranoid about losing two drives before noticing the RAID was degraded.

I agree with this, but one of the features of the raid card that I bought is that is has a built in NIC which includes a web interface and also the ability to email you if something happens with the raid array (regardless of if the OS is operating properly).

I still think I am just going to take my chances with the raid 6 array. Right now I have 9 of these drives with no redundancy at all, so this is a much better direction for me.

 

[Automata]

I agree with this, but one of the features of the raid card that I bought is that is has a built in NIC which includes a web interface and also the ability to email you if something happens with the raid array (regardless of if the OS is operating properly).

Very nice, wish my RAID card had that feature...