• Welcome to Overclockers Forums! Join us to reply in threads, receive reduced ads, and to customize your site experience!

How to plan long term active data storage on a budget?

Overclockers is supported by our readers. When you click a link to make a purchase, we may earn a commission. Learn More.

smoth

Member
Joined
Jul 26, 2009
I am currently a college student on a tight budget, but I am concerned that my current data storage solution may be setting myself up for a nightmare in years to come. I currently have a copy of all critical data (documents, pictures) at my parents house, on my school's network (documents only), and on 3 hard drives (laptop, internal, external), and on dvd, so I am not concerned about loosing critical files. My main concern is with my media collection (movies, music). Over the years it has grown to approximately 1.8TB, currently on 2 1TB hdds each with approximately 900GB and I still have stacks of movies and CDs lying around waiting to make it into my library. Since I often come up with reasons to move large numbers of files around, that is as full as I would like to go. Obviously I need more space, but I am starting to question whether this is a good approach. The easiest solution would be to go grab another drive and throw it in and start filling, but I am starting to realize the amount of time I have spent creating my digital libraries. Between ripping, encoding, deleting unwanted parts, tagging, organizing, etc I have probably invested well over 1,000 hours. Now, during high school, this was no big deal, as I did not have a job and had virtually no school work, now I am working 2 jobs, and taking 19 credits per semester, and competing on my schools ski team so if something were to happen, I would be devastated to be without my source of relaxation and would have no way of replacing the files (I keep my discs at home as I have nowhere to keep them in my dorm).

Eventually I would like one of those 16-24 hotswap bay cases with a large raid 6 array, but the money is currently nowhere to be found. Even a 4port controller starts at $350 and that give me no room to expand a raid 5 array past 3TB without replacing all of my drives (total cost right now of $550). I know I could do onboard, but I have heard horror stories about trying to recover data if the board dies, especially years down the road.

Does anyone have a suggestion on how to proceed in this situation?

The price of the controller is certainly cheaper than my time if something would go wrong, but there is no way that I can come up with that kind of money anytime in the foreseeable future.
 

visbits

Member
Joined
Jan 20, 2009
A dell perc5 could do it, you could use SAS extenders to get past the 8 drive limit of traditional fanout cables. :salute:
 
OP
smoth

smoth

Member
Joined
Jul 26, 2009
If I just get another drive for now and pray that nothing goes awry, can I purchase another drive and a controller card down the road and hook up my 3 drives and the new empty drive to the controller and build a raid 5 array without loosing my data or moving it to temporary storage? Same question but moving from raid 5-6?
 

jason4207

Senior Member
Joined
Nov 26, 2005
Location
Concord, NC
Look into unRAID. It's what I decided on, and it's working well for me so far.

http://www.lime-technology.com/joomla/

You can build your own server from cheap PC parts, and then add HDDs of any size as you need them. You do have to buy the unRAID license ($120) if you want more than 3 drives, though, but you can try it out for free until then to see if you like it. It's more fault tolerant than RAID5, but you get the same space advantage. It's also a lot less quirky IMO, and you can take 1 drive out and read all it's contents in another PC if necessary.

The LimeTech community support is also very good IMO.
 
OP
smoth

smoth

Member
Joined
Jul 26, 2009
Seems like an interesting option.

Can I create an array (with unraid, or controller cards) from hdds that already have data on them without moving all of that data to temporary storage?
 

deathman20

High Speed Premium Senior
Joined
Aug 5, 2002
The drives will need to be clean to create a Raid Array. Though as people mentioned this when I was creating my server.... Raid really isn't a viable backup solution. It will save your data and such if a drive dies (Raid1, Raid5, Raid6, Raid10) but if someone deletes something on the drive or a virus gets in there, the data is hosed.

So you might consider as well in the future to make a backup solution for longer integraty of the data. Course that will cost some money, can do cheaply just getting drives and software, more expensive if you drives, software and some sort of raid solution.
 

jason4207

Senior Member
Joined
Nov 26, 2005
Location
Concord, NC
Seems like an interesting option.

Can I create an array (with unraid, or controller cards) from hdds that already have data on them without moving all of that data to temporary storage?

Afaik you can't do that w/ unRAID (or any other RAID except maybe RAID1). In your situation you'd probably have to buy 2 more 1TB drives. Build the unRAID server using the 2 new drives w/o parity, and then transfer your data across. Then add 1 of the old 1TB drives for parity. Then add the last drive and have over 1TB of free space.
 
OP
smoth

smoth

Member
Joined
Jul 26, 2009
I realize that raid is not a backup solution, (see what I did with my critical documents), but I cannot think of any affordable way to create any robust backup of that quantity of data. Any suggestions? I figure that my main potential cause of data loss will be a dead drive and since that seems to be the cheapest "backup," I will start there. Its not like I would actually loose any data if the whole thing died as I have access to the original discs. I would be very sad, but I would just loose the convenience of having my entire library within a few clicks.

Can you move from raid5 to raid6 by just adding another drive or would that also require having sufficient temporary storage?
 

neonblingbling

Member
Joined
Aug 8, 2004
Like visbits said, I think you should look into a Dell PERC card. A common one is PERC 5/i (supports RAID 5, internal. Regular cables support 8 drives). Another option is the PERC 6/i (I believe it's the 5/i's replacement, it supports RAID 5 and RAID 6). I got a used PERC 5/i with everything I needed (minus drives) for $100 shipped, so it has a very nice bang for your buck.

What you could do, is grab a PERC 5/i ($100), then three 1TB drives (~$85 each). Set up the three drives in a RAID 5 (2TB), copy over your data, then expand your array to be 5 drives (4TB), while your data is still on the drives. You could also consider making one of the drives a hot-spare, theoretically giving you better odds of not having data loss. I'd also consider upgrading to 1.5TB drives ($110 each) and selling your current 1TB drives.
 
OP
smoth

smoth

Member
Joined
Jul 26, 2009
If I go with a separate system for data storage exclusively, what type of hardware do I need. I would like to keep power consumption and noise as low as possible.

Case: Norco RPC-450B
Raid controller: dell PERC 5 (I am thinking that for a max of 8 drives purchased over a few years, the odds of multiple simultaneous drive failures is pretty low. any thoughts on this?)
HDD: WD 1.5TB ecogreen x3 to start (raid5)
HDD OS: WD 80gb 7200rpm
Mobo: ECS BLACK SERIES A780GM-A
CPU:AMD Athlon 64 LE-1640 Lima 2.7GHz
RAM: GSkill 2x1gb ddr2800 cas5 (wow, memory got expensive, got my last set of 2x1gb ddr2800 4-4-4-12 for $5 after rebates)
PSU: Corsair 450watt is this powerful enough?

I have also been looking at old Dell (example) servers on ebay that seem to be going for about the same price. Dont know if it would be beneficial to go with server hardware or not.

Where did you find your PERC5 for $100?
 

neonblingbling

Member
Joined
Aug 8, 2004
I just had a thought: I thought the WD green drives have a feature that makes them spin down after a certain amount of time. I think that would not be good in a RAID 5 because a the RAID controller would think it dropped offline. Might be best to see if someone can verify this.

Another good member to try contact is thideras. He has a server just like what you want to build.

I am not really up to date on CPU/mobo/ram combos right now, but I do know DDR2 seems to be headed the way of DDR and everything before it (supply goes down, demand and price up). Might be worth looking at DDR3? Also, I have a socket 939 x2 setup with 5 drives (1 OS, 4 RAID) running off a Corsair 400W supply. Is it enough? Seems like it. :thup:

A problem with the old Severs on ebay is that they are probably loud/hot/power hungry. I'd personally not go with them (but hey, maybe they are better).

I got lucky and found my PERC5 in the classifieds on these forums. The guidelines to getting into the classifieds are at the bottom of this post. I feel very secure in these classifieds, but PERC 5/i's are not listed very often. Having said that, there happens to be one for sale right now. The other place to find them would probably be ebay. Maybe consider getting a PERC 6/i if you want to have a RAID 6 setup in the future. Here is a lonnnnnng thread packed with info. I only say try a PERC 6/i is because many of those members have and seem to be happy with it. :p

Also, in another thread:
It's certainly not the best RAID controller out there, but it's the best you're gonna get for under $300 or so, and it's more than good enough for the average SATA RAID setup.

People aren't stressing the point enough, I don't think, so I will say: make sure to get one with the battery backup unit. The cache won't work without it, and the BBUs aren't very easy to find by themselves.

I don't know of any particular revisions of the 5/i to look out for, but do be careful... there are a couple similarly named cards that aren't as good. I believe they're called 5/e and 5/iR.
 

jason4207

Senior Member
Joined
Nov 26, 2005
Location
Concord, NC
If I go with a separate system for data storage exclusively, what type of hardware do I need. I would like to keep power consumption and noise as low as possible.

Case: Norco RPC-450B
Raid controller: dell PERC 5 (I am thinking that for a max of 8 drives purchased over a few years, the odds of multiple simultaneous drive failures is pretty low. any thoughts on this?)
HDD: WD 1.5TB ecogreen x3 to start (raid5)
HDD OS: WD 80gb 7200rpm
Mobo: ECS BLACK SERIES A780GM-A
CPU:AMD Athlon 64 LE-1640 Lima 2.7GHz
RAM: GSkill 2x1gb ddr2800 cas5 (wow, memory got expensive, got my last set of 2x1gb ddr2800 4-4-4-12 for $5 after rebates)
PSU: Corsair 450watt is this powerful enough?

I have also been looking at old Dell (example) servers on ebay that seem to be going for about the same price. Dont know if it would be beneficial to go with server hardware or not.

Where did you find your PERC5 for $100?

For low-power I would direct you back to unRAID. It allows the disks to spin-down when not in use. RAID5 or RAID6 will require all drives to be spun-up all the time (I think).

Here's a thread I did on underclocking my E5200 for server duties. In the end the whole system consumes 52W when idle w/ the disks spun down.

http://www.ocforums.com/showthread.php?t=579229

I also just use the 8 onboard sata ports on the mobo. No need for 3rd party RAID controllers unless I want more than 8 drives, and then I can use inexpensive cards since I don't need any heavy-duty RAID5/6 support.

Do you plan on rack mounting this?

Another issue I forsee is when you run out of space and want to upgrade drives to a larger size w/o just continuing to add more drives. I'm not sure how difficult that is w/ RAID5/6, but it's easy w/ unRAID.

You don't need an OS HDD w/ unRAID. It all fits on a USB stick.

The 450W PSU should work fine.

I would suggest using Intel hardware w/ unRAID, though, and if you're a diehard AMD guy that might be a issue for you.
 

jediman

Member
Joined
Mar 11, 2003
Exactly the issue I was concerned with. Google FlexRaid. Its free software parity without striping (raid 5 without your data being spread accross multiple drives). This makes it so if one drive dies you can recover it with parity like raid 5. If 2 drives die before you replace and rebuild you only lose one drives worth of data. Also, because you aren't striping over multiple drives, you can use different size drives. Say you have a 500, 2x1 tb and 2x1.5 tb. One 1.5 tb will be a parity drive and hold parity information there will be something that behaves like a raid 5 that stretched over different areas of the drives. The 500 gig drive would generate parity along with 500 gigs from each of the other drives onto 500 gigs of the parity drive, the next 500 gigs of the 1tb drives and 1.5tb drive would generate parity on the next 500g of the parity drive, and the last 500gigs of the non parity 1.5tb would be mirrored on the last 500g of the parity drive. Its expandable, you can throw in any number of drives that are smaller or equal to the parity drive with no build time. You can replace the parity drive by removing it from the system, adding a new larger drive, rebuilding parity and place the old parity drive back in the array as free storage. Its software so you don't have to worry about incompatible controllers. Eventually he says he's adding support for multiple parity drives (raid 6 like).
 

jediman

Member
Joined
Mar 11, 2003
And you wouldn't need an expensive raid card, just a controller card.

Flexraid also shouldn't have any problem with drives spinning down for power savings (because it only needs information from all drives when it is updating parity. Ussually you are just reading info from a single drive like you normally do when using it)

oh ya, the downsides are the parity protection isn't continuous, so if you change a section of a drive you lose parity protection on that section of the array until parity is updated (on a schedule of your choosing), but this doesn't matter so much with your video storage. Also you don't get any performance increases like you would with raid.
 

neonblingbling

Member
Joined
Aug 8, 2004
Exactly the issue I was concerned with. Google FlexRaid. Its free software parity without striping ...

Sounds interesting! unRAID also sounds interesting, especially since it allows drives to spin down. Makes a hardware solution sound less and less appealing...

Also, are these software 'RAID' systems able to saturate a gigabit connection?
 

jediman

Member
Joined
Mar 11, 2003
like i said somewhere in there. Flexraid should also allow drives to spin down. As to saturating gigabit. Flexraid is the the speed of whatever single hard drive you are reading off of. So assuming you have a decent gigabit network (say 80 megs a second?) and a decenter hard drive you are reading from (bout 80 as well?) the gigaethernet should be just about saturated.
 
OP
smoth

smoth

Member
Joined
Jul 26, 2009
I am looking into these software solutions, but I have few concerns.

1) Lets say the drive with the OS installed on fails? Can I just reinstall the OS on another drive and regain access to my data?
2)What happens if my motherboard fails? Will a different sata controler cause problems? Will I need to reinstall as with Windows (back to question 1)?
3)I am having trouble finding suggested system requirments for either unraid or flexraid. What type of hardware woud I be looking at?
 

jediman

Member
Joined
Mar 11, 2003
1) unraid is the os so if it fails i would guess your in trouble but im not sure. flex raid is just a program that reads data and generates parity data. you can even flexraid some burned dvd's together and restore on any other computer if you wanted. I'm not sure how flex raid works if you are trying to protect your os with it.

2)unraid will have to support the controller as it is the os. flexraid runs in windows so windows will have to support the controller.

3)unraid you jsut need compatible hardware. browse their forums. flexraid you need a computer that will run a version of windows that is supported by flexraid. go to their website and find out. neither actually require much work be done. flex just reads off a hard drive just like you normally do (and you run some xor calculations once a day when you are away). unraid i think is updated on the fly and would likely require a bit more proccessing power if you are constantly changing things (though i don't think you are)
 

jason4207

Senior Member
Joined
Nov 26, 2005
Location
Concord, NC
I am looking into these software solutions, but I have few concerns.

1) Lets say the drive with the OS installed on fails? Can I just reinstall the OS on another drive and regain access to my data?
2)What happens if my motherboard fails? Will a different sata controler cause problems? Will I need to reinstall as with Windows (back to question 1)?
3)I am having trouble finding suggested system requirments for either unraid or flexraid. What type of hardware woud I be looking at?

1) With unRAID the OS is on a flash drive. If it does happen to fail, you can re-write 2 files to the flash drive and be back in action.

2) unRAID handles different controllers well, and makes them transparent. You won't have to re-install anything.

3) Look into a P35 or P45 mobo and you should be good. The boards w/ more sata ports are preferred for obvious reasons.
 

I.M.O.G.

Glorious Leader
Joined
Nov 12, 2002
Location
Rootstown, OH
rsync is probably the best answer for your goals. Raid isn't a very good answer for that amount of data - the pain of losing it would be too great. rsync will keep another machine in sync, but this isn't going to meet your budget goals either. It'd also require twice as much storage space as you need.