Extremely long download times??

[Ramonetb]

Has anyone else been experiencing extremely long download times for their work units?

Since starting the burn-in on the new HPC here, I've seen a drop in points from about 16M ppd to 12M. I thought something might be wrong with the hardware (heat related issues), but it turns out it's very long download times!

14:11:04:WU00:FS00:Assigned to work server 171.67.108.101
14:11:04:WU00:FS00:Requesting new work unit for slot 00: READY cpu:48 from 171.67.108.101
14:11:04:WU00:FS00:Connecting to 171.67.108.101:8080
14:11:08:WU00:FS00:Downloading 2.94MiB
14:11:46:WU00:FS00:Download 2.12%
14:11:58:WU00:FS00:Download 4.25%
14:12:10:WU00:FS00:Download 6.37%
14:12:24:WU00:FS00:Download 10.62%
14:12:39:WU00:FS00:Download 12.75%
14:13:04:WU00:FS00:Download 17.00%
14:13:11:WU00:FS00:Download 19.12%
14:13:49:WU00:FS00:Download 21.25%
14:13:55:WU00:FS00:Download 23.37%
14:14:16:WU00:FS00:Download 25.50%
14:14:53:WU00:FS00:Download 27.62%
14:15:14:WU00:FS00:Download 29.75%
14:15:34:WU00:FS00:Download 31.87%
14:24:02:WU00:FS00:Download 34.00%
14:32:04:WU00:FS00:Download 36.12%
14:32:10:WU00:FS00:Download 40.37%
14:32:18:WU00:FS00:Download 44.62%
14:32:26:WU00:FS00:Download 46.75%
14:32:34:WU00:FS00:Download 51.00%
14:32:45:WU00:FS00:Download 53.12%
14:33:16:WU00:FS00:Download 55.25%
14:33:39:WU00:FS00:Download 57.37%
14:33:52:WU00:FS00:Download 61.62%
14:33:59:WU00:FS00:Download 63.75%
14:40:30:WU00:FS00:Download 68.00%
14:41:25:WU00:FS00:Download 70.12%
14:43:34:WU00:FS00:Download 72.25%
14:43:54:WU00:FS00:Download 74.37%
14:44:10:WU00:FS00:Download 76.50%
14:44:18:WU00:FS00:Download 78.62%
14:50:46:WU00:FS00:Download 80.74%
14:50:52:WU00:FS00:Download 82.87%
14:51:00:WU00:FS00:Download 84.99%
14:52:11:WU00:FS00:Download 87.12%
14:52:31:WU00:FS00:Download 89.24%
14:52:51:WU00:FS00:Download 91.37%
14:53:05:WU00:FS00:Download 93.49%
15:03:24:WU00:FS00:Download 95.62%
15:03:41:WU00:FS00:Download 99.87%
15:03:41:WU00:FS00:Download complete
15:03:41:WU00:FS00:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:13124 run:28 clone:5 gen:39 core:0xa7

This unit, which was only about 3MB, took nearly an hour to download! I've seen some as long as three hours in the logs!
I've checked the network fabric and all appears to be in order. I've tested download speeds and consistently get about 75MB/s.

I guess what I'm wondering is if others are experiencing this or if I've got a problem with the network topology on my end?

Thanks!

-Kirk

 

[machausta]

I'm on a 1mb wireless connection, just barely in range and I'm getting better download times than you are. Interesting...

 

[torin3]

I tend to have the advanced control panel up in the second monitor and most of the time I see a unit downloaded in less than 2 minutes. Usually in the last percent of the previous WU.

12:04:42:WU02:FS01:0x21:Completed 7425000 out of 7500000 steps (99%)
12:04:43:WU00:FS01:Connecting to 171.67.108.45:80
12:04:43:WU00:FS01:Assigned to work server 140.163.4.245
12:04:43:WU00:FS01:Requesting new work unit for slot 01: RUNNING gpu:3:GP104 [GeForce GTX 1070] from 140.163.4.245
12:04:43:WU00:FS01:Connecting to 140.163.4.245:8080
12:04:44:WU00:FS01:Downloading 14.50MiB
12:04:44:WU00:FS01:Download complete
12:04:45:WU00:FS01:Received Unit: id:00 state:DOWNLOAD error:NO_ERROR project:10496 run:148 clone:14 gen:40 core:0x21 unit:0x000000308ca304f556bbb09830e10a44
12:07:00:WU01:FS00:0x21:Completed 2300000 out of 5000000 steps (46%)
12:07:09:WU02:FS01:0x21:Completed 7500000 out of 7500000 steps (100%)
12:07:10:WU02:FS01:0x21:Saving result file logfile_01.txt
12:07:10:WU02:FS01:0x21:Saving result file checkpointState.xml
12:07:10:WU02:FS01:0x21:Saving result file checkpt.crc
12:07:10:WU02:FS01:0x21:Saving result file log.txt
12:07:10:WU02:FS01:0x21:Saving result file positions.xtc
12:07:11:WU02:FS01:0x21:Folding@home Core Shutdown: FINISHED_UNIT
12:07:11:WU02:FS01:FahCore returned: FINISHED_UNIT (100 = 0x64)
12:07:11:WU02:FS01:Sending unit results: id:02 state:SEND error:NO_ERROR project:11707 run:81 clone:3 gen:65 core:0x21 unit:0x0000004e8ca304f358702f6d46613660
12:07:11:WU02:FS01:Uploading 5.85MiB to 140.163.4.243
12:07:11:WU02:FS01:Connecting to 140.163.4.243:8080
12:07:11:WU00:FS01:Starting
12:07:11:WU00:FS01:Running FahCore: "C:\Program Files (x86)\FAHClient/FAHCoreWrapper.exe" C:/Users/trich/AppData/Roaming/FAHClient/cores/fahwebx.stanford.edu/cores/Win32/AMD64/NVIDIA/Fermi/Core_21.fah/FahCore_21.exe -dir 00 -suffix 01 -version 704 -lifeline 1996 -checkpoint 15 -gpu 1 -gpu-vendor nvidia
12:07:11:WU00:FS01:Started FahCore on PID 10536
12:07:11:WU00:FS01:Core PID:6032
12:07:11:WU00:FS01:FahCore 0x21 started

Edit: that looks like a couple of seconds to download.

 

[caddi daddi]

might be the server you're assigned to.
you might try the one in torins log.

 

[Ramonetb]

Interesting!

Are those GPU assignments or CPU WU's?
I've checked some of my other systems and I'm now seeing that ALL of my A7 CPU work units are taking a very long time to download.

-Kirk

 

[torin3]

Interesting!

Are those GPU assignments or CPU WU's?
I've checked some of my other systems and I'm now seeing that ALL of my A7 CPU work units are taking a very long time to download.

-Kirk

Sorry if I misunderstood, but I'm only folding GPU WUs.

 

[Ramonetb]

Hmmm. Seems to be an issue with the 171.67.108.101 server. I'll see what I can dig up on the Stanford forums.

-Kirk

 

[WhitehawkEQ]

I've tested download speeds and consistently get about 75MB/s.

Don't you mean 75mb/s? B=Bytes, b=bits

edit: I'll check my 4p and see if I'm getting same slow downloads.
2nd edit: I checked and it looks like I'm downloading 2-6MB in less than 1 sec, then again I have 200+mb download speeds

 

[Ramonetb]

Don't you mean 75mb/s? B=Bytes, b=bits

Nope. 75MB/s.
This is a University connection, so it's close to true gigabit speeds. Speedtest.net: 916.46 Mb/s

edit: I'll check my 4p and see if I'm getting same slow downloads.
2nd edit: I checked and it looks like I'm downloading 2-6MB in less than 1 sec, then again I have 200+mb download speeds

I've been talking with Bruce over on the Stanford folding forums. We're trying some quick things to track down my issue. Traceroute returns nice values (although some servers are now not responding to it, but ping still works). I'm suspecting something on my networking end at this point.
Just frustrating since this burn-in will only last a short while and it's now taking me longer to download work units than it does to complete them!

Ah well. Hopefully we'll figure it out.

-Kirk

 

[don256us]

I've noticed some time ago that half of my CPU WU's aren't downloading at all. I usually only take note of the offense but rarely take direct action to correct it. However, when I reboot to correct a GPU issue, I noticed that sometimes the CPU issue also resolves. It seems to be hit or miss. Not sure how that helps.

Edit: You'd better get it figured out soon or I'll pass you.... wait. [*mumble*] carry the 1.... divide by a thousand... [/*mumble*] Never mind. Carry on.

 

[Ramonetb]

I've noticed some time ago that half of my CPU WU's aren't downloading at all. I usually only take note of the offense but rarely take direct action to correct it. However, when I reboot to correct a GPU issue, I noticed that sometimes the CPU issue also resolves. It seems to be hit or miss. Not sure how that helps.

Edit: You'd better get it figured out soon or I'll pass you.... wait. [*mumble*] carry the 1.... divide by a thousand... [/*mumble*] Never mind. Carry on.

Hmmm...

I'll test that out. I just rebooted a few of the compute nodes. We'll see if they last longer than the others. Worth a shot!

-Kirk