How can you organize files on a VPN?

[oqnx]

I am considering setting up a vpn, and just wanted to know a few things before I put any time or money into a vpn. My biggest question is whether there is an effective way to organize files on a vpn. Ideally, this VPN is going to be a multimedia outlet of sorts, so I was hoping that if I connect multiple drives or computers, I could create "virtual folders" which would organize the files spaced out across all of the hard drives. This way, when you connect to the vpn, rather than seeing 20 different network locations, its just one big public "folder" which you can browse through. Also, I'm hoping that this vpn can be used by a few friends of friends in china to get through the filtering system, so is there any way to do this and spread out the load from people using the vpn for that purpose from just my computer to any other computers in unblocked countries?

 

[FX4]

First question I have to ask is do you understand what a VPN does? What you are describing is a SAN (Storage Area Network), and I don't think a home budget would justify the cost. There may be a way to do this with Linux, I don't know because it's not something that has peaked my interest. The VPN (Virtual Private Network) is a separate topic that you highlight in the second part of your post. My understanding is yes people do use VPNs to get out of China.

 

[oqnx]

First question I have to ask is do you understand what a VPN does? What you are describing is a SAN (Storage Area Network), and I don't think a home budget would justify the cost. There may be a way to do this with Linux, I don't know because it's not something that has peaked my interest. The VPN (Virtual Private Network) is a separate topic that you highlight in the second part of your post. My understanding is yes people do use VPNs to get out of China.

I know that a vpn is a virtual private network. My understanding of this is that it pretty much creates a LAN connection over the internet, and should therefore allow you to "share" files on the vpn as if you were sharing them on the lan. Maybe my assumptions are wrong. Looking into SAN now, thanks!

 

[FX4]

A properly set up VPN connection will cause the end user to feel like they are on the local area network, so yes your assumptions are correct. Where I think you are muddling things is the idea of a SAN. I did some quick looking around the Linux world and it looks to me like you either spend the big bucks or know how to code to get everything working. I by no means did a comprehensive search though so there may be something out there that is a bit more user friendly than what I found.

 

[I.M.O.G.]

It sounds like what you are essentially trying to setup is a private cloud storage network. Something like dropbox, except instead of online storage, you want the storage to be made up of different hard drives on different computers. The total storage of the network would be greater than that of any single node in your network.

A VPN isn't necessary and does not make it any easier to do what you are trying to do.

Rsync would be close, but for that to work easily, you'd need one computer that can hold all the data in the network, and that would be the central repository. That doesn't sound like its the case.

If each computer has enough storage to hold all the data in the network, then that becomes much easier.

 

[oqnx]

It sounds like what you are essentially trying to setup is a private cloud storage network. Something like dropbox, except instead of online storage, you want the storage to be made up of different hard drives on different computers. The total storage of the network would be greater than that of any single node in your network.

A VPN isn't necessary and does not make it any easier to do what you are trying to do.

Rsync would be close, but for that to work easily, you'd need one computer that can hold all the data in the network, and that would be the central repository. That doesn't sound like its the case.

If each computer has enough storage to hold all the data in the network, then that becomes much easier.

Yes, pretty much. Pretty much I'm trying to create a "virtual raid" if I understand raid. The wiki article on raid is horribly confusing for me.
Probably an even better description is I'm looking for a system like windows 7's "library" system which can be applied to shared folders in the network.

 

[FX4]

I did some more research and I think this will work for you:
http://zfsonlinux.org/
http://breden.org.uk/2009/05/10/home-fileserver-zfs-file-systems/
http://hub.opensolaris.org/bin/view/Community+Group+zfs/whatis

 

[Automata]

Are these files going to be in a central location (e.g. a server)? If not, I can't think of a way you could do this. Additionally, if these is spread out among multiple computers, you aren't looking for RAID, that is a totally different concept.

Clarification on what the environment is like will make this a lot easier to find an answer.

 

[FX4]

He is looking for a distributed file system manager that can be accessed via a VPN.

 

[Automata]

Any type of RAID (or concatenation the partition/arrays) plus file sharing (Samba, Windows file sharing, etc) will do what he wants, then. Hardware RAID, mdadm, zfs, etc. Without knowing details on the computer, it will be difficult to help more.

 

[oqnx]

I'm going to try and make a diagram of some sort in a little bit. I'll post then and maybe it'll start to make sense.

 

[I.M.O.G.]

That doesn't account for the distributed nature of the storage thid.

He definitely is not talking raid. If anything its jbod, but distributed/virtualized storage.

 

[Automata]

That doesn't account for the distributed nature of the storage thid.

That is what I'm saying, we don't have enough information to give an answer. If it is multiple computers, there isn't an ideal way to handle it.

 

[FX4]

From what I have read ZFS will handle it.

 

[Automata]

From what I have read ZFS will handle it.

Across multiple computers? I'm pretty sure that is false.

 

[I.M.O.G.]

I also don't think zfs includes the bits to glue it together. You would need something to make the remote disks available to the server running the zfs volumes.

 

[FX4]

Across multiple computers? I'm pretty sure that is false.

Yes, see the blog link I posted.

 

[oqnx]

Okay, the diagram is just pointless here. Let me try and give an example.
Let's assume only 3 computers or devices for simplicity.
Computer 1 shares 3 folders, Music, Pictures, Videos
Computer 2 shares 3 folders, Music 1, Pictures 1, Videos 1
Computer 3 shares 3 folders, Music 2, Pictures 2, Videos 2
If my understanding of vpn is correct, under normal vpn, you would see a tree with computer 1, computer 2, and computer 3. You would need to remember which files are one which computers, and then browse to that computer, then find your files.
I want a system, where you connect to the network, and rather than a list of computers, you see three main folders, Music, Pictures, Videos. Those main folders would link to organized subfolders, and in those subfolders there would be "links" to folders and files on other devices. Let's go back to the example computers. There is an album on computer 2 that I want to listen to. Rather than needing to remember that I put The Wall on computer 2, I open the network, click on "music" "classic rock" "P" "Pink Floyd" "The Wall". Is this sort of starting to make sense? I'm sorry, I'm not very good at describing things.

 

[I.M.O.G.]

The blog talks about basic home networking, with home folders and public shares based on zfs. All storage is based on the server... if it talks about remote disks adding to the total storage capacity, please excerpt that part for me. It is a long article and I may have missed it, but from a quick look I don't see any way to add a logical volume over tcpip on a remote computer.

It also doesn't do libraries like he is considering.

I don't have a solution so I appreciate your suggestion, it just doesn't fit what he wants to do exactly.

 

[FX4]

See the section about creating home directory systems. He actually glues the disparate file systems together in a couple of different ways. Some by user and some by file type.