[how to] configure lancache in ubuntu 16.04 under hyper-V

Requirements:
- windows operating system that has hyper-v role enabled and running you can get the baremetal hypervisor free from microsoft, but that's alot more advanced.
- a router that is capable of forwarding dns traffic, for this writeup i will be using pfsense
- the more ram and cpu the better
- enough storage space to hold the files you will be caching im going to end up using 1tb worth of 500gb drives in raid10
- an iso of ubuntu 16.04 i used desktop which works fine
- must have a dedicated nic for the vm's dont want the host os to be sharing with the vm's it can make stuff messy.

[note: if installing on a physical machine scroll down to where the other note is under virtual machine part II]

ok first we are going to create a virtual switch to share between vm's and a virtual machine and for our caching server.
creating virtual switch

1. open hyper-v manager and click "virtual switch manager"
2. click "new virtual network switch" make sure external is selected then click "create virtual switch"
3. it should have went to settings for the new switch, give it a name, pick a network adapter that is dedicated just for the vm's.
- uncheck the box next to "allow management operating system to share this network adapter".
- make sure external network is still selected
- make sure "enable single-root i/o virtualization is not checked
4. click apply this will create the virtual switch. then click the "+" next to the new switch and click extensions, make sure no boxes are checked in there.
5. click apply if anything needed unchecked in "extensions"

now we will move on to creating the virtual machine.

1. open hyper-v manager in the right panel click new > virtual machine a dialogue box will appear, on "before you begin" read that then click next.
2. "specify name and location" any name will work i have mine named "cache" also you can use the default storage space or if you want it to use its own dedicated hard drive click next.
3. "specify generation" make sure generation 1 is selected and click next.
4. "assign memory" id prefer to have ATLEAST 4gb of memory so "4096"MB is a good starting point the more the better as the OS will use this to store the most recently shared files and will have faster access next time they are needed before they are flushed from memory. click next.
5. "configure networking" pick the virtual switch we created in the last section.
6. "connect virtual hard disk" ok this is crucial we need to have enough storage for what we need so put the virtual disk location somewhere that will have enough make the "size" eg if you want to be able to cache 500gb worth of games you need to have the virtual disk closer to 550gb.
7. "installation options" select the radio button next to "stall an operating system from a bootable cd/dvd-rom. then select "image file" then nagivate to your ubuntu iso we downloaded earlier. then click finish.
8. start the virtual machine from the hyper -v manager do the ubuntu install im not going to cover that its simple.
9. once its installed and you have rebooted, shut the machine down and right click on it in hyper- v manager we need to change some settings.

virtual machine part II.
im breaking this off as now we have the vm created and are changing settings to make this work best.

1. you should have the vm's settings open from the last instruction. select "processor" and give it as many cores as you can afford, i have an i3 and i gave it 4 cores (2c4t cpu).
2. click apply, then click ok. we are done with configuring the vm.
3. next we will start configuring the OS for the install, so put on your big girl panties its about to get fun.



OK so now we can power the vm back on and boot into Ubuntu. we want to install some hyper-v drivers, and needed software.
source: https://technet.microsoft.com/en-us/library/dn531029.aspx
1. open a terminal window (ctrl+alt+t) and run the following commands
that is the only thing i installed from the page if you want other things to work with hyper-v such as snapshots and stuff theres more info on the microsoft page i linked.

[note: anything after this point will work for setting it up on a dedicated physical machine with a fresh ubuntu 16.04 install as well]

2. next we are going to install nginx 1.10.1 https://www.nginx.com/resources/wiki/start/topics/tutorials/install/
if there is any issue refer to the page i linked and use the secondary set of commands.

2.5 in terminal run and make sure version 1.10.1 is installed (technically anything past 1.6 is fine)

3. now that nginx is installed we are going to open the file browser and go to computer>/etc/nginx/ and delete everything but "mime.types" you might have to chmod this directory to delete them. for more information on that https://help.ubuntu.com/community/FilePermissions i usually chmod 777 the folder plus /etc/nginx/* that will give permissions to all files in the folder. some people dont condone 777 being used though do some searching and see what's best for you.

4. next we will download the new nginx config there are alot of them but this one seems promising as its used for a huge lan party "q-con" and they actively update their stuff on there. https://github.com/Q-Con/ds-lancache/archive/master.zip open that zip folder then there should be a folder inside labeled "nginx" open that and copy everything from inside there into your /etc/nginx/ folder.

5. last thing in this section we need to create some folders this will be the directory where the caches are stored, open computer> /srv/ (chmod this directory first) create a folder called www, inside that folder one called cache, inside that folder we need to put 3 folders installs, other, and tmp. so the stucture of the folders should be
/srv/www/cache/installs/
/srv/www/cache/tmp/
/srv/www/cache/other/

now everything should be ready to rock and roll just a few more steps to go bear with me.

creating the virutal nic's
this is essential as this is where our dns forwarding is going to come in handy.

1. navigate in the file browser to /etc/networking/ (will likely have to chmod the directory and directory/*) and open the "interfaces" file.
2. now we need to create some virtual nic's inside this file we need to add some lines i will post my config in here you will have to change the all of the ip's to match your local network and router ect.

if you dont understand what to assign all these addresses then read here
we need all of these because that's what our dns forwards are going to point to later on.

3. reboot the machine. to ensure everything configured correctly open a terminal window and run ifconfig and make sure you have all 6 of the network adapters listed

ok now everything should be ready on the ubuntu side, now we are going to move onto the router configuration.

router dns forwarding config

ok this is the fun part every router is different im using pfsense i will give you the example of my config in dnsmasq you will see how the ip's match the above virtual nic's and that needs to happen so what everip's you assign to your virtual nics MUST MATCH all of your dns forwards.
im not going to go much more indepth on this part besides

NOTE: if when you go to run nginx with the new config and it says "host at lancache-steam cannot be found" that means you dont have the dns forwarded correctly.
if not using pfsense you might or might not need to use .localdomain after the names depending on your router.
open /etc/nginx/vhosts/lancache-single.conf and under all of the listen tags E.G. "lancache-steam" and "lancache-riot" add .yourdomain if you are using pfsense as you can see mine is "lancache-steam.localdomain"
basically all steam traffic will be directed to 192.168.1.218 where nginx will be listening and that's how it works or something.. i know it works. lol. riot traffic goes to .219 ect.



last thing there is to do is to start nginx.

1. open a terminal window and run the following"
-if this fails due to not finding a host look above and check your dns forwarding.
-if this fails because access to a folder or folder not existing then chmod or make the directory it needs go back and check what ones we made earlier.
-if this fails because of "user permissions" you didnt use sudo. lol i did this too many times to count.

2. only if the last step passes, run:
if it fails go back to step 1 as that tests and troubleshoots the nginx config.

3. if it successfully runs then we will need to test it! open steam on another computer and download a small game like counter strike. delete the game and download it again, you should see a significant speed increase, if not you might want to invest in an SSD('s) more ram or more cpu.

edit:
final note if having any issues with game downloads pausing downloads saying they are queued for 5 minutes or what ever like i was having issues with, then go into your host os's network settings and right click ethernet adapter(on vm switch)> properties> configure> advanced tab > "large send offload v2 (ipv4)" set to "disabled" this fixed my download hangs. only other option i found was to use a legacy adapter that was slow.

welp. i found the downside to what i had to do to make it work. the legacy network adapter causes high cpu usage, and is only 100Mb/s so that is no bueno. going to work on getting the normal network adapter working right i suppose. it has to be some kind of driver issue.....

edit: ok i got the network issue fixed now i will updat the post abover in due time after i finish my testing to make sure its stable.
what i changed in the network adapter in hyper-v.
"vlan id" enable virtual lan identification not checked
"hardware acceleration" disable everything
"advanced features" nothing enabled besides protected network

edit2: this seems to be working great, only problem right now is once the game finishes downloading it isnt wanting to "finish" like the cache keeps dumping data at it and the client shows its downloading stuff, but it also says its 100% downloaded. if i hit play it will give me a connection timed out error after a long wait.


edit3: just got a steam pause using the "network adapter" going to troubleshoot summore later. it was doing some weird stuff all together. also ever since ive done the network config this go around it hangs and says "waiting for network configuration" then it says "booting without full network configuration" but then it boots and all the networking stuff works fine with either network adapter.

going to revert back to legacy network adapter though it is horribly slow and terrible on the vm's resources. it makes one of the cores run at 99% for some reason. until i can troubleshoot what is going on with just the normal network adapter.
found these errors in nginx it seems to be something with the network adapter getting confused or someting, because one nic works[legacy], the other doesnt[normal].

im giving up on getting the normal network adapter to work the legacy is working fine just super slow... still faster than my internet connection... but still slow compared to gigabit.

edit4? after all the effort ive put into this im actually going to be switching my server host to ESXI this evening it seems so much better than hyper-v all around.

ok, its been a while i finally decided to mess with it again, my intel nic has an option (in the host operating system) right click ethernet adapter> properties> configure> advanced tab > "large send offload v2 (ipv4) disabled that boom problems went away. so far so good anyways downloaded two games and everything seems kosher im gonna do a little more testing but looks promising. this is letting me use the normal gigabit network adapter in ubuntu and not pausing my steam games.

wagex said:

open steam on another computer and download a small game like counter strike. delete the game and download it again, you should see a significant speed increase, if not you might want to invest in an SSD('s) more ram or more cpu.

Click to expand...

Any idea how long this is good for? Until the cache is manually flushed? Until restart? Is there a possibility of archiving the cache for reinstallation purposes? what I mean is: if you have to wipe the OS on the gaming rig and reinstall, then would an archive of the cached file be of any use? This would be most usefull to me in that way as I do not have multiple gaming rigs that I would need to have the games cached for.

I would however be interested to hear of adding something like dnsmasq to this setup would make sense, or would that just be unneccesary?

knoober said:

Any idea how long this is good for? Until the cache is manually flushed? Until restart? Is there a possibility of archiving the cache for reinstallation purposes? what I mean is: if you have to wipe the OS on the gaming rig and reinstall, then would an archive of the cached file be of any use? This would be most usefull to me in that way as I do not have multiple gaming rigs that I would need to have the games cached for.

I would however be interested to hear of adding something like dnsmasq to this setup would make sense, or would that just be unneccesary?

Click to expand...

its good until the files online are changed... im not quite sure i understand what you are asking. there is no archiving, the files are always good, unless something changes on the other end eg game update or someting, it keeps files for up to a year or however long you set it to hold them for. it runs on a seperate machine so you doing anything on your computer has nothing to do with how the server operates. you should not be running this vm on your desktop that you are installing the game from.

you could have 1000 computers downloading the same game and it would give them all the same files.

why would you want dnsmasq?

your post is super scattered and hard to understand if my reply doesnt make much sense that is why.


edit: updated network config to normal hyper-v network adapter settings. also added note to disable large send offload v2 (ipv4)

wagex said:

its good until the files online are changed...

Click to expand...

Thanks, that's what I was wondering

you could have 1000 computers downloading the same game and it would give them all the same files.

Click to expand...

Right there with you on that concept. Great for multi rig LAN. what I want to do is take the contents of the cache and run them through Bzip or gzip, so I would be able to make a compressed the cache directory: kind of a backup of the cache so to speak. The idea would be to store the install files locally in a more permanent way. Think of it as Caching Plus. My thinking is that Caching Plus might work even without a working connection because all/most necessary files would be served from the recently restored cache archive. I'll have to investigate this on my own though.

why would you want dnsmasq?

Click to expand...

For the same reason you would set up a caching server for your games but it caches DNS entries instead. Speeds up the access of previously used DNS entries, because they are hosted locally. If not found on the local machine the request will be forwarded to another DNS server down the line. I only ask because I wasn't sure if it worked the way I thought it did or was compatible with the setup you outlined

knoober said:

Thanks, that's what I was wondering

Right there with you on that concept. Great for multi rig LAN. what I want to do is take the contents of the cache and run them through Bzip or gzip, so I would be able to make a compressed the cache directory: kind of a backup of the cache so to speak. The idea would be to store the install files locally in a more permanent way. Think of it as Caching Plus. My thinking is that Caching Plus might work even without a working connection because all/most necessary files would be served from the recently restored cache archive. I'll have to investigate this on my own though.



For the same reason you would set up a caching server for your games but it caches DNS entries instead. Speeds up the access of previously used DNS entries, because they are hosted locally. If not found on the local machine the request will be forwarded to another DNS server down the line. I only ask because I wasn't sure if it worked the way I thought it did or was compatible with the setup you outlined

Click to expand...

that's pretymuch how it works my router is running dnsmasq... that's what the dns entries above go into... up where it says dnsmasq.
the way you worded it seemed like you thought it should be running on the ubuntu machine.

as for taking the contents of the cache and running them through bzip or what ever, that's probably a no go seeing as how steam handles all its downloads in chunks, so everything on there are a bunch of chunks of the games / files of the games broken all down into ~1mb files. these are re-assembled by the steam client as it downloads.

wagex said:

that's pretymuch how it works my router is running dnsmasq... that's what the dns entries above go into... up where it says dnsmasq.
the way you worded it seemed like you thought it should be running on the ubuntu machine.

Click to expand...

I did. I meant this dnsmasq not just the settings on your router. There are others, but this one works pretty much out of the box with little config.

as for taking the contents of the cache and running them through bzip or what ever, that's probably a no go seeing as how steam handles all its downloads in chunks, so everything on there are a bunch of chunks of the games / files of the games broken all down into ~1mb files. these are re-assembled by the steam client as it downloads.

Click to expand...

I can see why that might not work then. I will report back if I find it possible to make any sort of archive, but I agree that it likely will not work. I was banking on the fact that those chunks would be stored whole in the cache and would be able to be addressed by the steam client at any time.

knoober said:

I did. I meant this dnsmasq not just the settings on your router. There are others, but this one works pretty much out of the box with little config.



I can see why that might not work then. I will report back if I find it possible to make any sort of archive, but I agree that it likely will not work. I was banking on the fact that those chunks would be stored whole in the cache and would be able to be addressed by the steam client at any time.

Click to expand...

that is the dnsmasq that is running on my router as far as i know. its running on pfsense freebsd.

and yeah nginx stores the chunks in a weird way (all files actually including riot and stuff you cannot actually locate any of the physical files i can open the directory but its broke down into so many tiny folders its almost impossible to find anything esp not a particular game directory) theres no way to manually do it, all you have to do is run steam like normal and it will pull from it instead of downloading from the internet every time unless you dont have the game cached or a particular file has been updated.


the way this works is it basically mimics steams download servers when your client goes to download from the internet dns redirects push traffic to this machine on your local network, it looks and says hey i have this file! here you go! if it doesnt have the file it downloads it and says here you go! almost no traffic besides authentication traffic makes it to the actual steam servers from your pc.

Greetings, im having problems configuring the DNS Forwarded or PFsense , I'm mainly looking to cache riot and blizzard and modified the files correctly , running; "nginx -t" completes the test correctly. But when i start the league of legends patchier it wont connect. The DNS Forwarder is sending the request to the cachebox but not through. Any help will be apreciated , I've been learning so much with this.

kaytoz said:

Greetings, im having problems configuring the DNS Forwarded or PFsense , I'm mainly looking to cache riot and blizzard and modified the files correctly , running; "nginx -t" completes the test correctly. But when i start the league of legends patchier it wont connect. The DNS Forwarder is sending the request to the cachebox but not through. Any help will be apreciated , I've been learning so much with this.

Click to expand...

that's kind of weird it sounds like something isnt configured correctly. go to your cache machine and use the browser to hit league of legends.com and download the .exe installer and see if that works fine. mostly want to see if that box has internet access.

fixed ! , now its just slow

wagex said:

that's kind of weird it sounds like something isnt configured correctly. go to your cache machine and use the browser to hit league of legends.com and download the .exe installer and see if that works fine. mostly want to see if that box has internet access.

Click to expand...

Actually fixed the problem by starting over, probably was a typo somewhere. I managed to download League of Legends completely @ 400kbps. Deleted the game and re installed the launcher and the game is downloading from the cache box but only @400-450kbps. My box consists of a 240gb ssd / core 2 duo q6000 , 4gb ram , ubuntu 14.04.
thanks in advance !

kaytoz said:

Actually fixed the problem by starting over, probably was a typo somewhere. I managed to download League of Legends completely @ 400kbps. Deleted the game and re installed the launcher and the game is downloading from the cache box but only @400-450kbps. My box consists of a 240gb ssd / core 2 duo q6000 , 4gb ram , ubuntu 14.04.
thanks in advance !

Click to expand...

that's strange mine usually almost caps the gigabit connection with riot. check the network connection while downloading on the cache machine and see what's up. i think i usually disable update from peer to peer or what ever in the launcher top right corner.

figured id update, what i ended up doing to make everything work is switching to centos, everything works seamlessly and waaaaaay better than ubuntu. the above for the most part will work in centos theres a couple permissions you have to add and you have to manually add /srv/www/cache/ folders and whatnot.

here is the config im using, i switch to unbound as well for the dns. i just created a new file in /var/unbound/lancache.conf


then in the "advanced setup" of unbound i added this bit

edit: commented out origin redirects it no longer works they switched to a https cdn system
confirmed working: steam, origin, and riot

figured id post what kind of performance im getting downloading stuff as i go this is steam so far, i might try some nginx tweaks and shoot for closer to gigabit speeds but still 9.6gb download took less than 5 minutes i was happy



EDIT: origin seems to be working now, i didnt change anything besides making sure my client was up to date >.> shh.

ok so something new has popped up that im working on, once the download has been downloaded from the cache once, and the recent files are cached inthe os's memory, upon re-download the performance is pretty bad. eg first download it pulls from hard disks and averages 45-65MB/s down pulling from the cached files in memory it drops to 20-40MBs down. not sure what is going on with that, still doing some testing with different memory levels.

anyone have any idea why this could be happening? it just doesnt make any sense to me if anything the stuff cached in memory should be much faster.

for now having the vm's memory set to 2GB seems to be best because obv the cached filesi n memory are re-written before they are needed so its only really pulling from the hard disks.

If you are caching to memory wouldn't that 2gb eventually fill up and start offloading to swap? Does an increase in memory or change in swapping effect the rates?

Perhaps memory is not purging after the first pull from the hard disk and pulls another copy for each successive use (causing the memory to fill) . Even if that is not the case, it might be wise to set the memory to purge and reload from the hard disk for each use (because the first use is 2x as fast)?

im not caching to memory the os is. the os reads the file from the hdd gives it to nginx to send to the other machine on the network. when that happens the os saves it in its cache in the memory incase its needed again, its not important data so just about anything will overwrite it if need be.

the memory shouldnt purge. it should be doing what its doing its just doing it alot slower than expected. its reading the data from memory as it should, but its reading from memory slower than its reading from the hard disks.

running iotop while running downloads.
initial download ~50Mb/s reading from disks the entire download
second download 0MB/s reading from disks

so its doing as it should just slower than it should.

also it doesnt offload to swap because its not data that's being used, its data waiting to be re-written.


if i push 8GB of ram to the VM and download a 2gb game, that entire game download is sittingin the cache, i can see it in htop as orange. downloading it again its not pulling from disks at all just memory and is somehow slower.


maybe this flow chart will help explain a bit its a bit confusing seeings how were talking about a couple different caches in the thread.
initial download:
read from hdd > stored in memory by OS (as recent file cache not nginx) > sent over nic to client

second download:
read from memory > sent over nic to client