- Joined
- May 15, 2006
WARNING: This post will contain some THEORY (where noted) and may not describe features that are available in anything discussed (software or hardware). This post is made for information gathering only. If there is any information that is confirmed working, I will update the thread.
-----------------------------
I've been looking into software RAID and has been my "project" for the night. It is a very interesting idea and I have a few things that I would really like to test without risking data loss. I may pick up some drives so that I can test this out in a "live environment".
So, let's start with the basics. Mdadm us a utility available on Linux operating systems that allows you to run multiple hard drives in a software RAID array. Mdadm will not drop a drive that is unresponsive. This means that you can use drives that have large TLER values (ie: Green drives). The array will simply be unresponsive (or slow, if it reads from parity) until the drive comes back online. This means a substantially lower hardware cost as a RAID card is not needed. Where did I find this out? On Backblaze's website. They use HUGE servers (45 1.5tb driver per server) with mdadm as their RAID. They use "home drives", which is something that I found extremely interesting. In combination with their article (which I had originally read when it was released) and RJARRRPCGP's post, I've been thinking very hard on how this will function.
For example. Mdadm allows the expansion of RAID arrays through a simple command. This would allow a user to start off with a small array and grow it as their server grows. Granted, the rebuild time is something that would need to be tested and I wouldn't expect anything less than a few days once you starting getting in the 5+ TB range, but it is still better than backing up to a device that can store all the data, rebuilding the array with the new drives and moving the data back.
So, what can you do with this? What would be reasons to use mdadm over hardware RAID?
THEORY:
My question is this, though. Can you shrink an array through mdadm? If so, this would allow you to completely move data between two different mdadm arrays. For example, let's use my server and say I had a ton of money laying around (I wish!). My server has the capability of running 20 hard drives at a time. Say that I filled this up with 1tb drives for cheap and have a running capacity of 18tb (RAID 6). Say that I'm nearing the capacity of this array and I want to upgrade to larger drives. I don't want to purchase another server with 2tb drives to copy the data over. If you could shrink the mdadm RAID (by cleaning up space on the array and resizing the filesystem, then removing a drive from the array) by 4 drives (you'd need 4tb of space free, which could be moved to external drives, other internal drives or other computers), you would have the space for a RAID 6 array. So, you resize this partition and drop four of the 1tb drives out of the array. This leaves you with 14tb of data and little to no remaining space. You could shut the server down, remove the unused 1tb drives and plug in your shiny new 2tb drives. Fire up the server, build the RAID 6 array with the new four 2tb drives. This gives you 4tb of space to work with. Format the array and move over 4tb of data to it. Continue shrinking/growing the two arrays until there is no space data left on the 1tb array. Remove all the 1tb drives, plug in the 2tb drives and resize. You have successfully gone from a 18tb RAID 6 array to a 36tb RAID 6 array without using two computers or two servers. Bottom line, I know you can't do this with hardware RAID; can it be done with software?
Even without being able to remove drives from an array, software RAID would be extremely useful if it proves to be stable and fast enough for multiple data streams.
-----------------------------
I do have a few ideas for testing. Without any hardware cost, I can see how this works. "How would you do that, Thideras?", you may ask. Simple, virtual machines. I can create multiple small hard drives that would allow me to create an array. The only downside is these are "fake" hard drives and will not reflect how an actual drive responds to being in an array, but will instead show how the mdadm utility works with drives falling out of the array, adding and (possibly) removing. It should give a fair indication on how fast it can re-build/re-size and how much processor it would take up or identify bottlenecks.
To-do list:
If everything works, we need to identify a few things:
Now, I'm sorry if this doesn't make any sense, but I wrote this late on a Saturday night of hard thinking. Please let me know if there are any questions on anything I discussed. I would also like to note that this "test server" (or any server with this purpose, really) would not have "enterprise quality" drives and equipment; because that raises costs higher than most home users can afford. Myself included. The purpose of this would have massive amounts of storage, with parity, for cheap. Quality can be worked around through different methods (RAID, backups, etc).
-----------------------------
I've been looking into software RAID and has been my "project" for the night. It is a very interesting idea and I have a few things that I would really like to test without risking data loss. I may pick up some drives so that I can test this out in a "live environment".
So, let's start with the basics. Mdadm us a utility available on Linux operating systems that allows you to run multiple hard drives in a software RAID array. Mdadm will not drop a drive that is unresponsive. This means that you can use drives that have large TLER values (ie: Green drives). The array will simply be unresponsive (or slow, if it reads from parity) until the drive comes back online. This means a substantially lower hardware cost as a RAID card is not needed. Where did I find this out? On Backblaze's website. They use HUGE servers (45 1.5tb driver per server) with mdadm as their RAID. They use "home drives", which is something that I found extremely interesting. In combination with their article (which I had originally read when it was released) and RJARRRPCGP's post, I've been thinking very hard on how this will function.
For example. Mdadm allows the expansion of RAID arrays through a simple command. This would allow a user to start off with a small array and grow it as their server grows. Granted, the rebuild time is something that would need to be tested and I wouldn't expect anything less than a few days once you starting getting in the 5+ TB range, but it is still better than backing up to a device that can store all the data, rebuilding the array with the new drives and moving the data back.
So, what can you do with this? What would be reasons to use mdadm over hardware RAID?
- Cost is substantially lower
- Drives are not dropped out of the array when they go unresponsive
- Allows you to easily expand an array (some RAID cards do this too)
- Allows [virtually] an unlimited number of hard drives to be added to an array
- Easily monitor the hard drives SMART data, including temperature (can script in warnings/emails) without using proprietary software
- Array performance will be [probably] be substantially lower
- Rebuilds will take longer since you don't have an optimized dedicate processor to calculate parity (the RAID card itself)
- How does TLER respond to an array?
- If a drive is unresponsive, so is the array? What about with parity?
- Recovering an array that is good, but the server has failed (bad motherboard, operating system drive, etc)
- Recovering an array that is failed
THEORY:
My question is this, though. Can you shrink an array through mdadm? If so, this would allow you to completely move data between two different mdadm arrays. For example, let's use my server and say I had a ton of money laying around (I wish!). My server has the capability of running 20 hard drives at a time. Say that I filled this up with 1tb drives for cheap and have a running capacity of 18tb (RAID 6). Say that I'm nearing the capacity of this array and I want to upgrade to larger drives. I don't want to purchase another server with 2tb drives to copy the data over. If you could shrink the mdadm RAID (by cleaning up space on the array and resizing the filesystem, then removing a drive from the array) by 4 drives (you'd need 4tb of space free, which could be moved to external drives, other internal drives or other computers), you would have the space for a RAID 6 array. So, you resize this partition and drop four of the 1tb drives out of the array. This leaves you with 14tb of data and little to no remaining space. You could shut the server down, remove the unused 1tb drives and plug in your shiny new 2tb drives. Fire up the server, build the RAID 6 array with the new four 2tb drives. This gives you 4tb of space to work with. Format the array and move over 4tb of data to it. Continue shrinking/growing the two arrays until there is no space data left on the 1tb array. Remove all the 1tb drives, plug in the 2tb drives and resize. You have successfully gone from a 18tb RAID 6 array to a 36tb RAID 6 array without using two computers or two servers. Bottom line, I know you can't do this with hardware RAID; can it be done with software?
Even without being able to remove drives from an array, software RAID would be extremely useful if it proves to be stable and fast enough for multiple data streams.
-----------------------------
I do have a few ideas for testing. Without any hardware cost, I can see how this works. "How would you do that, Thideras?", you may ask. Simple, virtual machines. I can create multiple small hard drives that would allow me to create an array. The only downside is these are "fake" hard drives and will not reflect how an actual drive responds to being in an array, but will instead show how the mdadm utility works with drives falling out of the array, adding and (possibly) removing. It should give a fair indication on how fast it can re-build/re-size and how much processor it would take up or identify bottlenecks.
To-do list:
Install host operating system (CentOS)Update the host operating system and install utilities neededCreate multiple small hard drives (unknown size as of now, maybe 5gb in size?)Use these drives in a RAID using the mdadm utility- Test what happens when a drives suddenly stops responding (delete the drive or find another method to make it broken)
- See how quickly it can rebuild an array when forced (easy to test)
- See how quickly it adds drives to an array
- See how quickly it adds multiple drives to an array
- See if you can change the RAID level without shutting down the array
- See if you can remove drives from the array
If everything works, we need to identify a few things:
- What port multipliers can be used to interface the drives?
- What are the best drives for the array?
Now, I'm sorry if this doesn't make any sense, but I wrote this late on a Saturday night of hard thinking. Please let me know if there are any questions on anything I discussed. I would also like to note that this "test server" (or any server with this purpose, really) would not have "enterprise quality" drives and equipment; because that raises costs higher than most home users can afford. Myself included. The purpose of this would have massive amounts of storage, with parity, for cheap. Quality can be worked around through different methods (RAID, backups, etc).
Last edited: