Storage@Home

David · May 16, 2008

I saw this over at [H].

Basically Stanford are thinking of starting a distributed storage project in order to find 1PB (thats a petabyte) of cheap, redundant, scalable and fast storage for Folding@Home data (of which they generate 2TB/month).

The basics seem to be that you say what sort of line you're on, where you are, and how much space they can use. They do some sort of striping thing to ensure redundancy and have the data replicated on ten hosts. They reckon 4 of the 10 hosts could fail and data would still not be lost.

Points would be assigned for participants, at the moment this is a minor issue but the basic idea is points per unit storage plus penalties for downtime/failures etc.

Posted at [H] here: http://www.hardforum.com/showthread.php?t=1306850
Paper on the subject: http://www.stanford.edu/~beberg/[email protected]

Kevin007 · May 16, 2008

David said:
I saw this over at [H].

Basically Stanford are thinking of starting a distributed storage project in order to find 1PB (thats a petabyte) of cheap, redundant, scalable and fast storage for Folding@Home data (of which they generate 2TB/month).

The basics seem to be that you say what sort of line you're on, where you are, and how much space they can use. They do some sort of striping thing to ensure redundancy and have the data replicated on ten hosts. They reckon 4 of the 10 hosts could fail and data would still not be lost.

Points would be assigned for participants, at the moment this is a minor issue but the basic idea is points per unit storage plus penalties for downtime/failures etc.

Posted at [H] here: http://www.hardforum.com/showthread.php?t=1306850
Paper on the subject: http://www.stanford.edu/~beberg/[email protected]

That sounds like a cool idea.

Ben333 · May 16, 2008

Sounds like a cool idea but I know I won't be doing it - wouldn't I use a lot of bandwidth constantly? If so, won't it be impossible to have large farms because of the bandwidth demand?

VinnyTAMU · May 16, 2008

Have they said how often they would need to access the files stored on your computer? I have plenty of spare hard drive space and cpu cycles to spare, but I have very little upload bandwidth to spare. Have they taken this into consideration?

David · May 16, 2008

You'll want to check the paper I linked for more information.

They acknowledge the issues with limited bandwidth (especially in the US where most of their user base is). I don't think they've tackled the issue of farms but they are aiming to make it so FaH donors can also run SaH without affecting folding.

deadlysyn · May 16, 2008

I haven't taken the time to read the paper yet, but is this something meant to try to help with the server issues they seemed to have been having recently? If I had the bandwidth and the space available, I would participate, but I do have a 300GB drive and 250 of it is taken up. I do have an old P3 board I was thinking about building into a file server, but then I would be limited by my bandwidth.

WarriorII · May 16, 2008

Hhahahahahahahahaha

Yeah right, you want my storage now????

What the 24/7 computer use isn't enough?

Talking about pushing it......

Standford / Folding can apply For Grants and other Donations.
They can ADD this to their list, not mine.

Want in one hand and Wish in the other.
Guess what you got more of?

Check my box in the "Opposed" section.

I can always stop if I am doing too much for them now.

benbaked · May 16, 2008

WarriorII said:
Hhahahahahahahahaha

Yeah right, you want my storage now????

What the 24/7 computer use isn't enough?

Talking about pushing it......

Standford / Folding can apply For Grants and other Donations.
They can ADD this to their list, not mine.

Want in one hand and Wish in the other.
Guess what you got more of?

Check my box in the "Opposed" section.

I can always stop if I am doing too much for them now.

Ditto, this is one project that I won't be volunteering for. My HD space is at a premium, if they want it then they can rent it from me - paid in cash, not points.

seadave77 · May 16, 2008

I'm with WarriorII. My HD is also at a premium. It's not that much to get more space. A million isn't that much and they can probably get a grant to pay for it. I'd say the project has grown to have it's hands in to many pots, if you know what I mean. They need to work on their F@H clients and get them sorted out before they work on another one.

Rpkole · May 16, 2008

David said:
(of which they generate 2TB/month)

sweet i can hold 3 months worth of data so i wonder what the points would be like but it will be heck on them getting to it cause i have 10Mb down and 1Mb up so it would take a while for them to get the data back

Shiggity · May 16, 2008

I bet they put a giant PPD on that.

I'm still in agreement with Warrior though, I'm running out of HDD space as it is.

Godfather1138 · May 17, 2008

I also agree that they have the means to apply for funding to gain the needed storage. I think our power bills speak for themselves.

silent bob · May 17, 2008

benbaked said:
Ditto, this is one project that I won't be volunteering for. My HD space is at a premium, if they want it then they can rent it from me - paid in cash, not points.

cash in hand , and I mean Dead Presidents cash not check cash.

I could prolly free up a TB or so IF I make a server instead of multiple backups on multiple computers

Shelnutt2 · May 17, 2008

CPU cycles are one thing. Its another to have files sitting on my hdd.

Since I often bounce between OS's, they'd have a difficult time using me.

Ben333 · May 17, 2008

WarriorII said:
Yeah right, you want my storage now????

What the 24/7 computer use isn't enough?

Talking about pushing it......

Standford / Folding can apply For Grants and other Donations.
They can ADD this to their list, not mine.

Want in one hand and Wish in the other.
Guess what you got more of?

Check my box in the "Opposed" section.

I can always stop if I am doing too much for them now.

+1 - For me its not so much not having enough HD space but the bandwidth. I run a few websites, forums and game servers off my connection at home and WON'T be giving up ANY of that bandwidth.

muddocktor · May 17, 2008

I can also see this harebrained scheme giving all DC projects in general a bad name with ISP's because of the extra traffic. You already have Cox and Comcast? filtering torrents and now Stanford want's us to let them use terrabytes more bandwidth for their purpose? I'm sure the ISP people aren't total dummies and will notice the sudden upsurge in bandwidth usage to and from Stanford and if it gets widespread, they might start filtering anything coming or going from their IP addresses. And worse, they might do the same thing to any university that has a DC project running too. And for you people that presently are in college; Stanford will effectively be stealing bandwidth from the university you are going to also with this goofy idea too. That will equate to more bandwidth charges for the school you go to and they will pass the added cost along to everyone attending your school in higher tuition.

This is pretty backhanded and low, IMO.

harlam357 · May 17, 2008

Bad idea, bad scheme... buy some storage servers. However, I just got curious to see how much HD space I actually have in the house and I added it up... 3025 Gigabytes.

...2030 of that reserved for media (mpeg2 TV, DVD, avi, mkv, etc). I'm just a junior HTPC nut.

And to mud's point... glad I don't have comcast or cox cable. I'm on Road Runner, which is generally a Time Warner thing, but our local company is branded Bright House. I just got upgraded to the 10Mbps/1Mbps line and it's nice for F@H uploads.

...and so far I haven't seen them throttle any of my activity, including torrents. :cross fingers: they don't start the filter/throttle game like some other ISPs.

David · May 17, 2008

muddocktor said:
I can also see this harebrained scheme giving all DC projects in general a bad name with ISP's because of the extra traffic. You already have Cox and Comcast? filtering torrents and now Stanford want's us to let them use terrabytes more bandwidth for their purpose? I'm sure the ISP people aren't total dummies and will notice the sudden upsurge in bandwidth usage to and from Stanford and if it gets widespread, they might start filtering anything coming or going from their IP addresses. And worse, they might do the same thing to any university that has a DC project running too. And for you people that presently are in college; Stanford will effectively be stealing bandwidth from the university you are going to also with this goofy idea too. That will equate to more bandwidth charges for the school you go to and they will pass the added cost along to everyone attending your school in higher tuition. This is pretty backhanded and low, IMO.

They have already thought about the dangers of the ports they use being labelled as P2P and hence blocked by ISPs.

At the end of the day it's donor based so you have to volunteer. I'm sure college IT departments will start blocking it if it drives up bandwidth costs. However I'm not sure if there is any sort of academic WAN in the US (I think the UK has one - JANET?) which might save having the transmissions go over the internet?

It's definitely a trickier issue than processing power for FaH.

leelegend · May 17, 2008

i dont mind.....if the machines are on and folding 24/7, whats the harm in a bit of storage?

lee

RipD · May 20, 2008

Keep in mind that Stanford is also doing research. Part of this project could be to prove (or disprove) the viability of this concept, and for them to find where the issues are. I've got a couple of drives sitting around. Easy enough for me to plug them in a give Stanford to space and see what happens. For those who already bandwidth constrained, I can see why you would not want to do this.

Storage@Home

Forums Super Moderator

Member

Folding for Team 32!

Member

Forums Super Moderator

Folding Team Content Editor, Who Dolk'd my stars S

Moderator #666

Folding/SETI/Rosetta Team Member

Member

Member

Member

Member

Team 32 Folding Ghetto Hanging PC Senior

Overclockers Team Content Editor

Folding for Team 32!

Retired

Senior Fold-a-holic

Forums Super Moderator

Member

Member

Similar threads