• Welcome to Overclockers Forums! Join us to reply in threads, receive reduced ads, and to customize your site experience!

ubuntu 14.04, nginx, hyper-v, network errors? corrupt data?

Overclockers is supported by our readers. When you click a link to make a purchase, we may earn a commission. Learn More.

wagex

Chapstick Eating Premium Member
Joined
Jan 14, 2011
ubuntu 16.04 nginx lancache, pfsense dns forwarding issues.

ok, i have a vm im running trying to get to work right, no idea what i can be doing wrong. here is a quick run down. ubuntu is running nginx as a reverse proxy to cache my steam / riot / microsoft downloads. i can get this configuration to run flawlessly on a physical machine but in a virtual machine something happens i cannot pinpoint. example.
initiate steam download from a windows network pc, traffic gets directed to the vm on the server fine, it works for a while and keeps caching, then steam gets an error (corrupt steam chunk or something ) it wont continue to download.

on a physical machine it runs perfectly i can cache all day and then redownload the stuff all day no errors. so this is what is driving me insane, as its literally the same exact setup on both machines.
 
Last edited:
plan for now is to run a celeron and 4gb of ram as a solo machine to run this on for now till i can figure out what is going on with the vm, id really like to move over to a new hypervisor but i dont have time to move my other vm's over to it and reconfigure them considering half of them are using the 2012r2 datacenter activation which wont work if the host is not 2012r2 datacenter.
 
I hate when problems crop up like this. My best guess is that there is something wrong on the virtualized network stack. Have you tried virtio? I am not sure if this is an option with hyper-v but on virtualbox and vmware (although vmware calls it something different I think) it basically gives fairly close to native network stack.
 
I hate when problems crop up like this. My best guess is that there is something wrong on the virtualized network stack. Have you tried virtio? I am not sure if this is an option with hyper-v but on virtualbox and vmware (although vmware calls it something different I think) it basically gives fairly close to native network stack.

hmm, thanks! il have to look into this here in a bit im a total noob when it comes to linux so il report back with my findings!

edit: found this will try it in a while! seems so simple im going to facepalm so hard if this fixes it.
723AQvQ.png
edit 2: seems like virtio is similar to the hyper-v drivers that ive already installed but i could be wrong i cant seem to find anything about using it with a linux guest under hyper-v though :\

e.g. heres the hyper-v stuff i installed manually which was fun to do
in the background you can kind of see my network interfaces config if you see anything weird about it besides the virtual nics (nginx thing) let me know.
24eb5e30446d6fdbbb0846ab5ac1d93f.png
 
Last edited:
0 luck with changing up the network stuff unless theres something else that works with hyper-v besides the ones that are special for it.
 
when you say "it keeps crashing" what do you mean?

can you reliably reproduce the problem? if so at what point is it failing? have you checked ubuntu logs? what about hyper-v logs?
 
when you say "it keeps crashing" what do you mean?

can you reliably reproduce the problem? if so at what point is it failing? have you checked ubuntu logs? what about hyper-v logs?

sorry i didnt clarify on that, by crashing i mean something gets corrupt or delayed to the point that steam pauses the download on the client end, gives some error about corrupt data if i dig into it il have to replicate the error again to get the exact error message. and yes i can replicate it time and time again.

steps to replicate it.
1. stop nginx
2. clear all cached files
3. add in my dns forwards into pfsense
4. restart nginx
5. start steam download on client pc on the network
the amout of time it has to download to cause the error in steam is pretty random it could be after 100mb or after 1.4gb but it always ends up pausing the download and wont resume till i clear the cached files in nginx

now if i do these same exact steps on a physical machine, boom it works great. i was able to cache over 30gb of data in one day never one error or issue.

as far as logs i have dug deep into both ubuntu and nginx and i cant find anything pertaining to the issues im having. no connection errors or anything of the sort nginx even logs every single chunk weather it pulls it from cache or has to cache it i havent been able to find anything that occurs near the time of failure.
 
what about hyper-v logs?

If this was happening in a production environment I would be looking to the hypervisor to help debug the issue because thats the most likely cause at this point
 
what about hyper-v logs?

If this was happening in a production environment I would be looking to the hypervisor to help debug the issue because thats the most likely cause at this point

nothing i can see on the hyper-v end of things, though i havent quite been able to get integration with my server manager to quite work right with ubuntu, perhaps in the future i will just grow a pair and move over everything to esxi just stinks my server 2012 guests wont stay activated. i guess the main one i can run in linux as most of the software running in it is designed for it....
im gonna fire up a test machine running esxi and see if i can get it running in that environment, if i can then i will migrate my other vm's over and load it up on my main server instead of 2012 hyper-v, i just love the ease of use of hyper-v lol though i have never tried another hypervisor besides virtualbox in windows. this could be a good next step for me. :)

il keep you updated, i might try and mess with it this weekend if i can find the time between hunting and what ever else my wife wants to do... we all know how that goes. lol.
 
same set up, but dont use hyper-V
try vmware.

so you are runing nginx under ubuntu just with an esxi host? il have to look into converting my server to esxi i suppose in the future.
 
Just a shot in the dark here but I think this may be related to the problem in my post about 14,04. It seems that the RTL8111 and most related NIC setups have a broken driver package, ie a known issue, for ubuntu (and most Debian and Redhad distros, if I read correctly). There is a fix, but with my experience level I cannot get it to work correctly, perhaps you could though. If you have a realtek NIC anyway. The fix is listed in my post and does work, until a reboot.

I will also say that I had a similar experience with virtual machines. THe OS would connect just fine in a vm, but when loaded by iteself was fraught with networking problems. I hope this is your issue, because all the other stuff here in this thread is over my head.

Edit: Im very sorry. I read this a few days ago and didnt bother to re-read the whole post BEFORE i replied. I did not realize that you could run in the physical and not the vm. I posted this thinking it was the other way around. Really very sorry.
 
Last edited:
now that my isp is choking me down to 600GB a month i think im going to try and get this going again because running 10gb worth of updates on one pc in one day then running the same updates on the other computer a couple days later wont be helping. i might try with the newest release of ubuntu and go from there see if things are any better.
 
now that my isp is choking me down to 600GB a month i think im going to try and get this going again because running 10gb worth of updates on one pc in one day then running the same updates on the other computer a couple days later wont be helping. i might try with the newest release of ubuntu and go from there see if things are any better.

If it's for the updates you can try what I do (granted I use Arch)... I have the /var/cache/pacman/pkg directory as an NFS mount on my server, all my computers mount this through systemd at boot. This way they pull from the same cache of installed packages and bandwidth use after the initial update is limited to software that is specific to that machine
 
ok im back sorry split off another thread to try and figureout my pfsense issues wich were fixed. now. this is the issue i get when im running this in a VM it only does it for steam riot (league of legends) works fine and no complaints.
if im running the exact same setup on a physical machine i never get this issue. but in a vm this always happens. Id prefer to figure out how to get it working in a VM because its kind of hard to justify having another machine running when this machine is doing almost 0 heavy lifting.
the host is windows server 2012 with hyper-v, the guest os running in the vm ubuntu 16.04 using nginx version 1.10.1
dbd9d502ab0b4741d856c49ea47aa107.png
 
do you have another computer you can run say virtualbox on a linux host to see if it is something caused by Hyper-V. Unfortunately I cannot reproduce what you are doing as I have no Windows/Hyper-V. I did a similar thing several years back with Squid as a transparent proxy and didn't have an issue back then (circa 2010 ish)

I had a 5M DSL connection that had a hard time supporting the 3 or 4 steam boxes we had in our house
 
do you have another computer you can run say virtualbox on a linux host to see if it is something caused by Hyper-V. Unfortunately I cannot reproduce what you are doing as I have no Windows/Hyper-V. I did a similar thing several years back with Squid as a transparent proxy and didn't have an issue back then (circa 2010 ish)

I had a 5M DSL connection that had a hard time supporting the 3 or 4 steam boxes we had in our house

il give that a try here in just a little bit...

edit: cant till i get home virtualization is not enabled on the machine i have running. sadly cannot run hyper-v and virtualbox at the same time or id have tried that already :(

edit:
i managed to get an error today, times seem to be around the time it "paused" the downloads
Code:
 2016/06/07 10:58:15 [error] 1729#1729: *30 valve703.steamcontent.com could not be resolved (110: Operation timed out), client: 192.168.1.70, server: steam, request: "GET /depot/1/chunk/611824e741420d2cac162f1112a3865e18f5bb65 HTTP/1.1", host: "valve703.steamcontent.com"
2016/06/07 11:04:51 [error] 1729#1729: *523 upstream timed out (110: Connection timed out) while connecting to upstream, client: 192.168.1.70, server: steam, request: "GET /depot/1/chunk/c160de1267d2ddd225ee88a5c837070310284b68 HTTP/1.1", upstream: "http://162.254.195.22:80/depot/1/chunk/c160de1267d2ddd225ee88a5c837070310284b68", host: "valve712.steamcontent.com"

i have found a way to replicate the errors, if i just clear the cache and restart nginx then clear data on the download on the client pc, then re-initiate a download it will still error out with no log, but if i clear cache, reboot the machine, clear the local data in steam then restart the download i get the error. so im guessing without the reboot the corrupted data is still chilling in the OS's ram cache.
Code:
 2016/06/07 11:46:42 [error] 1863#1863: *1 upstream timed out (110: Connection timed out) while connecting to upstream, client: 192.168.1.70, server: steam, request: "GET /depot/1/chunk/6e6122f800762e40876c94ffd25a6713e2364c92 HTTP/1.1", upstream: "http://162.254.195.14:80/depot/1/chunk/6e6122f800762e40876c94ffd25a6713e2364c92", host: "valve704.steamcontent.com"
best guess so far is something to do with ubuntu, hyper-v nic, and dns handling.

next thing i will try is instead of doing a bunch of virtual nic's in the interface file just do 6x network adapters for the vm under hyper-v i have some work to do for now il try it later this afternoon or evening.

also need to update on something i was suggested from the hyper-v subreddit "disabling vmq" i disabled it under the network adapter in hyper-v settings and the only effect i can see that had was that the steam-cache-error.log no longer logs the failed upstream timeout's and the downloads still fail under steam.

ok so i went back to the hypervisor and assigned it 6 nic's on that end thinking maybe ubuntu's virtual nic's were messing with it, steam download still paused. also tried to re-enable vmq on all 6 nic's on the hyper-v side. no change. i think im going to keep the 6 different nic's from the vm's side for now just for the fact it seemed to download longer before it failed though it could be happenstance.
 
Last edited:
welp put in a new nic today an intel pro/1000 pt dual nic still have the issue, i read about realtek nic's and vm network issues.
 
Have you tried doing a wireshark on both the vm and the client?

It may give you some indication of what is happening on the network level.
 
ok, i have a vm im running trying to get to work right, no idea what i can be doing wrong. here is a quick run down. ubuntu is running nginx as a reverse proxy to cache my steam / riot / microsoft downloads. i can get this configuration to run flawlessly on a physical machine but in a virtual machine something happens i cannot pinpoint. example.

Do you using a network bridge or NAT setup for your VM?

If it's NAT it can be the problem I believe.
Try configure the direct network bridge for your VM network card and check again, or think about forwarding the ports on your hypervisor instead of it.

If it's bridged already some logs may be helpful too.
 
Last edited by a moderator:
Back