GPU_2 Problems Core_14

Jolly-Swagman · Feb 22, 2009

NEW GPU_2 Problem

Some may be experiencing a Problem that has arose on the Stanford Servers where you will get an error

Code:

[08:01:15] + Processing work unit
[08:01:15] Core required: FahCore_14.exe
[08:01:15] Core not found.
[08:01:15] - Core is not present or corrupted.
[08:01:15] - Attempting to download new core...
[08:01:15] + Downloading new core: FahCore_14.exe
[08:01:16] - Error: HTTP GET returned error code 404
[08:01:16] + Error: Could not download core
[08:01:16] + Core download error (#2), waiting before retry...
[snip]
[10:15:48] + Core download error (#13), waiting before retry...
Folding@Home Client Shutdown.

Youcan read about it here
http://foldingforum.org/viewtopic.php?f=52&t=8556

They dont want you to use the Core_14 as yet as still testing

Re: Fahcore 14

by metal03326 on Sun Feb 22, 2009 9:33 am
Yes, FahCore_14 is a beta core. It's currently tested by the beta team. This core is released only to make sure the GPUs are giving us the right answers, so don't expect it to be released to public. I don't know why you got a WU for C14. Probably PG made a mistake.

Re: Fahcore 14

by ihaque on Sun Feb 22, 2009 10:44 am

Teddy wrote:I doubt anyone cares @ Stanford that this happened, it has happened before & will happen again.
hurts the science more than anything + annoys a few people into the bargain.

Hi all,

Sorry about this. These WUs are not supposed to be going out to the non-beta public yet; we're currently testing core 14 in beta (as you've all noticed). I'm not sure why you're being assigned these WUs, as the project is set to beta-only. I'm looking into this and will try to get it resolved as soon as I can.

I understand your frustration with this kind of situation, but please understand that we do care, and we appreciate the effort that all of you make by donating your computer time. We're human and mistakes happen here too .

That said, if you've been assigned a Core 14 WU and are not a beta tester, would you mind PMing me your FAHlog.txt? It may help us figure out what's going wrong.

So Best thing to do is delete queue dat file and you will get a core_11 WU and continue on

Edward2 · Feb 22, 2009

Thanks for the info. One of my GPU's has been trying to get core_14 for the past 7 hours. Now is got a core_11, and everything is happy again.

HayesK · Feb 22, 2009

Two of my GPU had issue with core_14. Both F41-GPU2 (9600GSO) and F45-GPU1 (8800GS) were hung for a while attempting to return work and download core_14. F45-GPU1 eventually returned the work and resumed folding without intervention. F41-GPU2 was stopped and restarted between returning work and getting core_14. F41-GPU2 would likely have been OK without intervention, but it was hung when I reviewed the log this AM, but apparently was able to return the work by the time I got to the rig and restart FAH. I did not have to delete anything or lose any completed work. Both the GPU are currently running 384pt project 5900 with core_14 and making ~10% less ppd than a 511pt project. It will be interesting to see what project/core these GPU get when current is completed ~2.5 hrs.

harlam357 · Feb 22, 2009

Got one here too... looks like the client was waiting on Core 14 for some time. But its got it now. However, I can't get a reading out of FahMon since the project is not on the psummary.

I'm just gonna let it run and see what happens.

HayesK · Feb 22, 2009

The project 5900 is in psummaryC. You need to change the FAHmon pref-advanced to get client info from http://fah-web.stanford.edu/psummaryC.html

Looks like the the 5900 with core_14 on F41-G2 has crashed and went back on 511pt 5749 core_11 v1.19. F45-G1 is currently 46% on a 5900 with core_14.

Code:

[17:13:45] Completed 38%
[17:15:54] Completed 39%
[17:18:03] Completed 40%
[17:20:13] SEH code: 3221225477
[17:20:13] Run: exception thrown during GuardedRun
[17:20:13] Run: exception thrown in GuardedRun -- Gromacs cannot continue further.
[17:20:13] Going to send back what have done -- stepsTotalG=2000000
[17:20:13] Work fraction=0.4089 steps=2000000.
[17:20:17] logfile size=89445 infoLength=89445 edr=0 trr=23
[17:20:17] - Writing 89981 bytes of core data to disk...
[17:20:17] Done: 89469 -> 5010 (compressed to 5.5 percent)
[17:20:17]   ... Done.
[17:20:17] 
[17:20:17] Folding@home Core Shutdown: UNSTABLE_MACHINE
[17:20:21] CoreStatus = 7A (122)
[17:20:21] Sending work to server
[17:20:21] Project: 5900 (Run 5, Clone 50, Gen 0)
[17:20:21] - Read packet limit of 540015616... Set to 524286976.


[17:20:21] + Attempting to send results [February 22 17:20:21 UTC]
[17:20:21] - Reading file work/wuresults_06.dat from core
[17:20:21]   (Read 5522 bytes from disk)
[17:20:21] Connecting to http://171.64.122.70:8080/
[17:20:23] Posted data.
[17:20:23] Initial: 0000; - Uploaded at ~3 kB/s
[17:20:23] - Averaged speed for that direction ~71 kB/s
[17:20:23] + Results successfully sent
[17:20:23] Thank you for your contribution to Folding@Home.
[17:20:27] Trying to send all finished work units
[17:20:27] + No unsent completed units remaining.
[17:20:27] - Preparing to get new work unit...
[17:20:27] + Attempting to get work packet
[17:20:27] - Will indicate memory of 3070 MB
[17:20:27] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 15, Stepping: 11
[17:20:27] - Connecting to assignment server
[17:20:27] Connecting to http://assign-GPU.stanford.edu:8080/
[17:20:28] Posted data.
[17:20:29] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[17:20:29] + News From Folding@Home: GPU folding beta
[17:20:29] Loaded queue successfully.
[17:20:29] Connecting to http://171.67.108.11:8080/
[17:20:31] Posted data.
[17:20:31] Initial: 0000; - Receiving payload (expected size: 99223)
[17:20:32] - Downloaded at ~96 kB/s
[17:20:32] - Averaged speed for that direction ~95 kB/s
[17:20:32] + Received work.
[17:20:32] Trying to send all finished work units
[17:20:32] + No unsent completed units remaining.
[17:20:32] + Closed connections
[17:20:37] 
[17:20:37] + Processing work unit
[17:20:37] Core required: FahCore_11.exe
[17:20:37] Core found.
[17:20:37] Working on queue slot 07 [February 22 17:20:37 UTC]
[17:20:37] + Working ...
[17:20:37] - Calling '.\FahCore_11.exe -dir work/ -suffix 07 -priority 96 -nocpulock -checkpoint 30 -verbose -lifeline 3176 -version 623'

[17:20:37] 
[17:20:37] *------------------------------*
[17:20:37] Folding@Home GPU Core - Beta
[17:20:37] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[17:20:37] 
[17:20:37] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[17:20:37] Build host: amoeba
[17:20:37] Board Type: Nvidia
[17:20:37] Core      : 
[17:20:37] Preparing to commence simulation
[17:20:37] - Looking at optimizations...
[17:20:37] - Created dyn
[17:20:37] - Files status OK
[17:20:37] - Expanded 98711 -> 492276 (decompressed 498.7 percent)
[17:20:37] Called DecompressByteArray: compressed_data_size=98711 data_size=492276, decompressed_data_size=492276 diff=0
[17:20:37] - Digital signature verified
[17:20:37] 
[17:20:37] Project: 5749 (Run 14, Clone 130, Gen 129)
[17:20:37] 
[17:20:37] Assembly optimizations on if available.
[17:20:37] Entering M.D.
[17:20:44] Working on Protein
[17:20:46] Client config found, loading data.
[17:20:46] Starting GUI Server
[17:23:25] Completed 1%
[17:26:05] Completed 2%
[17:28:44] Completed 3%

harlam357 · Feb 22, 2009

Ahh, what a wise man... I forgot about those other pages.

Actually, I just recently stumbled on them a couple weeks ago... never had a reason to look anywhere else but the public psummary.

Ok, here's the data... frame times look pretty erratic. :-/

That's a 256MB card btw, shaders @ 1782.

Project : 5900
Core : Unknown
Frames : 100
Credit : 384

-- nVidia GPU - XFX 8800GT #2 --

Min. Time / Frame : 1mn 32s - 3606.26 ppd
Avg. Time / Frame : 1mn 40s - 3317.76 ppd
Cur. Time / Frame : 1mn 52s - 2962.29 ppd
R3F. Time / Frame : 1mn 48s - 3072.00 ppd
Eff. Time / Frame : 9mn 36s - 576.00 ppd

Computekinc.us · Feb 22, 2009

Uno

It only hit one of my machines, and it had to be the GTX 260 / 216

torin3 · Feb 22, 2009

It hit mine, and for some reason was downloading the core named wrong. Core_14.fah was the name. When I fixed it, to FahCore_14.exe it told me it couldn't run on 64bit windows. So I deleted queue.dat and got a normal WU.

HayesK · Feb 22, 2009

HayesK said:
Looks like the the 5900 with core_14 on F41-G2 has crashed and went back on 511pt 5749 core_11 v1.19. F45-G1 is currently 46% on a 5900 with core_14.

F45-G1 successfully completed and returned the 5900 core_14, then downloaded a p5768, 353 point, core_11.

Code:

[19:25:55] Completed 95%
[19:28:02] Completed 96%
[19:30:23] Completed 97%
[19:32:31] Completed 98%
[19:35:00] Completed 99%
[19:37:20] Completed 100%
[19:37:20] Successful run
[19:37:20] DynamicWrapper: Finished Work Unit: sleep=10000
[19:37:30] Reserved 11208 bytes for xtc file; Cosm status=0
[19:37:30] Allocated 11208 bytes for xtc file
[19:37:30] - Reading up to 11208 from "work/wudata_06.xtc": Read 11208
[19:37:30] Read 11208 bytes from xtc file; available packet space=786419256
[19:37:30] xtc file hash check passed.
[19:37:30] Reserved 23472 23472 786419256 bytes for arc file=<work/wudata_06.trr> Cosm status=0
[19:37:30] Allocated 23472 bytes for arc file
[19:37:30] - Reading up to 23472 from "work/wudata_06.trr": Read 23472
[19:37:30] Read 23472 bytes from arc file; available packet space=786395784
[19:37:30] trr file hash check passed.
[19:37:30] Allocated 560 bytes for edr file
[19:37:30] Read bedfile
[19:37:30] edr file hash check passed.
[19:37:30] Allocated 164364 bytes for logfile
[19:37:30] Read logfile
[19:37:30] GuardedRun: success in DynamicWrapper
[19:37:30] GuardedRun: done
[19:37:30] Run: GuardedRun completed.
[19:37:30] - Writing 200116 bytes of core data to disk...
[19:37:30] Done: 199604 -> 43151 (compressed to 21.6 percent)
[19:37:30]   ... Done.
[19:37:31] - Shutting down core 
[19:37:31] 
[19:37:31] Folding@home Core Shutdown: FINISHED_UNIT
[19:37:35] CoreStatus = 64 (100)
[19:37:35] Unit 6 finished with 84 percent of time to deadline remaining.
[19:37:35] Updated performance fraction: 0.928374
[19:37:35] Sending work to server
[19:37:35] Project: 5900 (Run 12, Clone 25, Gen 0)
[19:37:35] - Read packet limit of 540015616... Set to 524286976.


[19:37:35] + Attempting to send results [February 22 19:37:35 UTC]
[19:37:35] - Reading file work/wuresults_06.dat from core
[19:37:35]   (Read 43663 bytes from disk)
[19:37:35] Connecting to http://171.64.122.70:8080/
[19:37:35] Posted data.
[19:37:35] Initial: 0000; - Uploaded at ~43 kB/s
[19:37:36] - Averaged speed for that direction ~84 kB/s
[19:37:36] + Results successfully sent
[19:37:36] Thank you for your contribution to Folding@Home.
[19:37:36] + Number of Units Completed: 1709

[19:37:40] Trying to send all finished work units
[19:37:40] + No unsent completed units remaining.
[19:37:40] - Preparing to get new work unit...
[19:37:40] + Attempting to get work packet
[19:37:40] - Will indicate memory of 2047 MB
[19:37:40] - Connecting to assignment server
[19:37:40] Connecting to http://assign-GPU.stanford.edu:8080/
[19:37:40] Posted data.
[19:37:40] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[19:37:40] + News From Folding@Home: GPU folding beta
[19:37:40] Loaded queue successfully.
[19:37:40] Connecting to http://171.67.108.11:8080/
[19:37:41] Posted data.
[19:37:41] Initial: 0000; - Receiving payload (expected size: 47105)
[19:37:41] Conversation time very short, giving reduced weight in bandwidth avg
[19:37:41] - Downloaded at ~92 kB/s
[19:37:41] - Averaged speed for that direction ~77 kB/s
[19:37:41] + Received work.
[19:37:41] Trying to send all finished work units
[19:37:41] + No unsent completed units remaining.
[19:37:41] + Closed connections
[19:37:41] 
[19:37:41] + Processing work unit
[19:37:41] Core required: FahCore_11.exe
[19:37:41] Core found.
[19:37:41] Working on queue slot 07 [February 22 19:37:41 UTC]
[19:37:41] + Working ...
[19:37:41] - Calling '.\FahCore_11.exe -dir work/ -suffix 07 -priority 96 -nocpulock -checkpoint 30 -verbose -lifeline 516 -version 623'

[19:37:41] 
[19:37:41] *------------------------------*
[19:37:41] Folding@Home GPU Core - Beta
[19:37:41] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[19:37:41] 
[19:37:41] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[19:37:41] Build host: amoeba
[19:37:41] Board Type: Nvidia
[19:37:41] Core      : 
[19:37:41] Preparing to commence simulation
[19:37:41] - Looking at optimizations...
[19:37:41] - Created dyn
[19:37:41] - Files status OK
[19:37:41] - Expanded 46593 -> 252912 (decompressed 542.8 percent)
[19:37:41] Called DecompressByteArray: compressed_data_size=46593 data_size=252912, decompressed_data_size=252912 diff=0
[19:37:41] - Digital signature verified
[19:37:41] 
[19:37:41] Project: 5768 (Run 5, Clone 110, Gen 169)
[19:37:41] 
[19:37:42] Assembly optimizations on if available.
[19:37:42] Entering M.D.
[19:37:48] Working on Protein
[19:37:49] Client config found, loading data.
[19:37:49] Starting GUI Server
[19:38:57] Completed 1%
[19:40:05] Completed 2%
[19:41:14] Completed 3%
[19:42:21] Completed 4%
[19:43:29] Completed 5%
[19:44:37] Completed 6%
[19:45:45] Completed 7%
[19:46:53] Completed 8%
[19:48:01] Completed 9%
[19:49:09] Completed 10%

GIXXERGUY6 · Feb 19, 2010

I got one too

I want points

ChasR · Feb 19, 2010

Are those of you with the core_14 problem seeing it only on mancines with -advmethods turned on, or on both non -adv and -adv machines? I haven't had the problem at all, that I'm aware of, and don't have any gpus configured to -advmethods.

Norcalsteve · Feb 19, 2010

dunno if this thread is really an issue anymore, since it was started in February of last year... it may have been a problem back then... Gixx just resurrected it ;-)

ChasR · Feb 19, 2010

GIXER,
How do you find these old threads? I was totally caught out. Explains why I haven't seen this problem, at least that I can remember over a year later.

Thanks Steve.

Mark620 · Feb 19, 2010

I have several Core 14 finished that will not send...these are NEW and Current
on more than one machine....

Edit : seems 8 GPUS currently have Core 14...
It will take me a while to find out howmany Core 14 finised WU's I have from the past couple of days.

Edit 2: According to the current logs I have 23 Core 14 WU's that will not upload

Edit 3:
Typical Log:

Code:

[23:13:27] + Attempting to send results [February 19 23:13:27 UTC]
[23:13:31] - Couldn't send HTTP request to server
[23:13:31] + Could not connect to Work Server (results)
[23:13:31]     (171.67.108.21:8080)
[23:13:31] + Retrying using alternative port
[23:13:52] - Couldn't send HTTP request to server
[23:13:52] + Could not connect to Work Server (results)
[23:13:52]     (171.67.108.21:80)
[23:13:52] - Error: Could not transmit unit 08 (completed February 19) to work server.
[23:13:52] - Read packet limit of 540015616... Set to 524286976.


[23:13:52] + Attempting to send results [February 19 23:13:52 UTC]
[23:13:56] - Server does not have record of this unit. Will try again later.
[23:13:56]   Could not transmit unit 08 to Collection server; keeping in queue.
[23:13:56] - Preparing to get new work unit...
[23:13:56] + Attempting to get work packet
[23:13:56] - Connecting to assignment server
[23:13:57] - Successful: assigned to (171.64.65.20).
[23:13:57] + News From Folding@Home: Welcome to Folding@Home
[23:13:57] Loaded queue successfully.
[23:13:58] Project: 5786 (Run 8, Clone 4, Gen 19)
[23:13:58] - Read packet limit of 540015616... Set to 524286976.


[23:13:58] + Attempting to send results [February 19 23:13:58 UTC]
[23:14:02] - Couldn't send HTTP request to server
[23:14:02] + Could not connect to Work Server (results)
[23:14:02]     (171.67.108.21:8080)
[23:14:02] + Retrying using alternative port
[23:14:23] - Couldn't send HTTP request to server
[23:14:23] + Could not connect to Work Server (results)
[23:14:23]     (171.67.108.21:80)
[23:14:23] - Error: Could not transmit unit 08 (completed February 19) to work server.
[23:14:23] - Read packet limit of 540015616... Set to 524286976.


[23:14:23] + Attempting to send results [February 19 23:14:23 UTC]
[23:14:27] - Server does not have record of this unit. Will try again later.
[23:14:27]   Could not transmit unit 08 to Collection server; keeping in queue.
[23:14:27] + Closed connections
[23:14:27] 
[23:14:27] + Processing work unit
[23:14:27] Core required: FahCore_14.exe
[23:14:27] Core found.
[23:14:27] Working on queue slot 05 [February 19 23:14:27 UTC]
[23:14:27] + Working ...
[23:14:28] 
[23:14:28] *------------------------------*
[23:14:28] Folding@Home GPU Core - Beta
[23:14:28] Version 1.26 (Wed Oct 14 13:09:26 PDT 2009)
[23:14:28] 
[23:14:28] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86
[23:14:28] Build host: vspm46
[23:14:28] Board Type: Nvidia
[23:14:28] Core      : 
[23:14:28] Preparing to commence simulation
[23:14:28] - Looking at optimizations...
[23:14:28] - Created dyn
[23:14:28] - Files status OK
[23:14:28] - Expanded 66529 -> 360060 (decompressed 541.2 percent)
[23:14:28] Called DecompressByteArray: compressed_data_size=66529 data_size=360060, decompressed_data_size=360060 diff=0
[23:14:28] - Digital signature verified
[23:14:28] 
[23:14:28] Project: 5910 (Run 14, Clone 218, Gen 0)
[23:14:28] 
[23:14:28] Assembly optimizations on if available.
[23:14:28] Entering M.D.
[23:14:34] Tpr hash work/wudata_05.tpr:  2648018779 2358690084 2468980589 348568324 1229894467
[23:14:34] Working on Protein
[23:14:35] Client config found, loading data.
[23:14:35] Starting GUI Server
[23:15:55] Completed 1%
[23:17:45] Completed 2%
[23:19:31] Completed 3%
[23:21:28] Completed 4%
[23:23:10] Completed 5%
[23:25:03] Completed 6%
[23:26:46] Completed 7%
[23:28:28] Completed 8%
[23:30:17] Completed 9%
[23:32:03] Completed 10%
[23:34:00] Completed 11%
[23:35:43] Completed 12%
[23:37:29] Completed 13%
[23:39:18] Completed 14%
[23:41:00] Completed 15%
[23:42:54] Completed 16%
[23:44:40] Completed 17%
[23:46:29] Completed 18%
[23:48:15] Completed 19%
[23:50:12] Completed 20%
[23:52:13] Completed 21%
[23:53:59] Completed 22%
[23:55:55] Completed 23%
[23:57:45] Completed 24%
[23:59:35] Completed 25%
[00:01:28] Completed 26%
[00:03:10] Completed 27%
[00:05:00] Completed 28%
[00:06:49] Completed 29%
[00:08:35] Completed 30%
[00:10:29] Completed 31%
[00:12:22] Completed 32%
[00:14:11] Completed 33%
[00:15:57] Completed 34%
[00:17:47] Completed 35%
[00:19:33] Completed 36%
[00:21:19] Completed 37%
[00:23:08] Completed 38%
[00:24:54] Completed 39%
[00:26:48] Completed 40%
[00:28:34] Completed 41%
[00:30:23] Completed 42%
[00:32:09] Completed 43%
[00:34:02] Completed 44%
[00:35:48] Completed 45%
[00:37:42] Completed 46%
[00:39:28] Completed 47%
[00:41:10] Completed 48%
[00:42:56] Completed 49%
[00:44:42] Completed 50%
[00:46:24] Completed 51%
[00:48:10] Completed 52%
[00:49:52] Completed 53%
[00:51:49] Completed 54%
[00:53:35] Completed 55%
[00:55:17] Completed 56%
[00:57:22] Completed 57%
[00:59:07] Completed 58%
[01:00:53] Completed 59%
[01:02:47] Completed 60%
[01:04:33] Completed 61%
[01:06:22] Completed 62%
[01:08:12] Completed 63%
[01:10:05] Completed 64%
[01:11:55] Completed 65%
[01:13:41] Completed 66%
[01:15:31] Completed 67%
[01:17:13] Completed 68%
[01:19:02] Completed 69%
[01:20:48] Completed 70%
[01:22:34] Completed 71%
[01:24:24] Completed 72%
[01:26:25] Completed 73%
[01:28:18] Completed 74%
[01:30:04] Completed 75%
[01:31:53] Completed 76%
[01:33:39] Completed 77%
[01:35:21] Completed 78%
[01:37:07] Completed 79%
[01:39:01] Completed 80%

Macaholic · Feb 19, 2010

This is related to Dr. Pande's most recent blog post and the posts found here.

GPU_2 Problems Core_14

Jolly-Swagman

Member

Edward2

Member

HayesK

Member

harlam357

Senior Fold-a-holic

HayesK

Member

harlam357

Senior Fold-a-holic

Computekinc.us

Member

torin3

Member

HayesK

Member

GIXXERGUY6

Member

ChasR

Senior Member

Norcalsteve

Member

ChasR

Senior Member

Mark620

Member

Macaholic

Member

Similar threads