GPU - WU Series 57** Nans detected

Jolly-Swagman · Apr 9, 2009

Anybody else having Problems with this series of WU 57** Nans detected

These are all on the Gigabyte GPU 8800Gt 256MB ( BAR Donated)
@ Current PDD 4608

[15:07:02] Project: 5751 (Run 6, Clone 108, Gen 293)
[15:07:02]
[15:07:03] Assembly optimizations on if available.
[15:07:03] Entering M.D.
[15:07:10] Working on Protein
[15:07:15] Client config found, loading data.
[15:07:15] Starting GUI Server
[15:09:19] Completed 1%
[15:11:23] Completed 2%
[15:13:26] Completed 3%
[15:15:30] Completed 4%
[15:17:34] Completed 5%
[15:19:38] Completed 6%
[15:21:42] Completed 7%
[15:23:46] Completed 8%
[15:25:49] Completed 9%
[15:27:45] Completed 10%
[15:27:45] mdrun_gpu returned
[15:27:45] NANs detected on GPU

[15:59:15] Project: 5780 (Run 9, Clone 271, Gen 0)
[15:59:15]
[15:59:15] Assembly optimizations on if available.
[15:59:15] Entering M.D.
[15:59:22] Working on Protein
[15:59:24] Client config found, loading data.
[15:59:25] Starting GUI Server
[16:01:48] Completed 1%
[16:04:13] Completed 2%
[16:06:36] Completed 3%
[16:06:36] mdrun_gpu returned
[16:06:36] Nonzero force sum on GPU

[20:08:12] Project: 5777 (Run 7, Clone 286, Gen 0)
[20:08:12]
[20:08:12] Assembly optimizations on if available.
[20:08:12] Entering M.D.
[20:08:19] Working on Protein
[20:08:21] Client config found, loading data.
[20:08:21] Starting GUI Server
[20:10:45] Completed 1%
[20:13:09] Completed 2%
[20:15:33] Completed 3%
[20:17:57] Completed 4%
[20:20:21] Completed 5%
[20:22:34] Completed 6%
[20:22:34] mdrun_gpu returned
[20:22:34] NANs detected on GPU

[21:03:51] Project: 5777 (Run 7, Clone 286, Gen 0)
[21:03:51]
[21:03:51] Assembly optimizations on if available.
[21:03:51] Entering M.D.
[21:03:57] Working on Protein
[21:03:59] Client config found, loading data.
[21:03:59] Starting GUI Server
[21:06:23] Completed 1%
[21:06:58] - Autosending finished units... [April 8 21:06:58 UTC]
[21:06:58] Trying to send all finished work units
[21:06:58] + No unsent completed units remaining.
[21:06:58] - Autosend completed
[21:06:58] + Working...
[21:08:47] Completed 2%
[21:11:04] Completed 3%
[21:11:04] mdrun_gpu returned
[21:11:04] NANs detected on GPU

After getiing a few of these in a row I Nuked the work folder - Queue dat and Fah_11 / Fah-14 Cores as seemed was getting the sam WU,s and resulting the same with Nans detected, so now is folding ok again with

Code:

[06:52:40] Verifying core Core_11.fah...
[06:52:40] Signature is VALID
[06:52:40] 
[06:52:40] Trying to unzip core FahCore_11.exe
[06:52:40] Decompressed FahCore_11.exe (1843200 bytes) successfully
[06:52:45] + Core successfully engaged
[06:52:52] 
[06:52:52] + Processing work unit
[06:52:52] Core required: FahCore_11.exe
[06:52:52] Core found.
[06:52:52] Working on queue slot 01 [April 9 06:52:52 UTC]
[06:52:52] + Working ...
[06:52:52] - Calling '.\FahCore_11.exe -dir work/ -suffix 01 -priority 96 -checkpoint 15 -verbose -lifeline 3456 -version 623'

[06:52:52] 
[06:52:52] *------------------------------*
[06:52:52] Folding@Home GPU Core - Beta
[06:52:52] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[06:52:52] 
[06:52:52] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[06:52:52] Build host: amoeba
[06:52:52] Board Type: Nvidia
[06:52:52] Core      : 
[06:52:52] Preparing to commence simulation
[06:52:52] - Looking at optimizations...
[06:52:52] - Created dyn
[06:52:52] - Files status OK
[06:52:52] - Expanded 67846 -> 350980 (decompressed 517.3 percent)
[06:52:52] Called DecompressByteArray: compressed_data_size=67846 data_size=350980, decompressed_data_size=350980 diff=0
[06:52:52] - Digital signature verified
[06:52:52] 
[06:52:52] Project: 5776 (Run 2, Clone 263, Gen 7)
[06:52:52] 
[06:52:53] Assembly optimizations on if available.
[06:52:53] Entering M.D.
[06:53:00] Working on Protein
[06:53:02] Client config found, loading data.
[06:53:03] Starting GUI Server
[06:55:27] Completed 1%
[06:57:52] Completed 2%
[07:00:16] Completed 3%
[07:02:41] Completed 4%
[07:05:05] Completed 5%
[07:07:29] Completed 6%
[07:09:53] Completed 7%
[07:12:18] Completed 8%
[07:14:42] Completed 9%
[07:17:06] Completed 10%
[07:19:31] Completed 11%
[07:21:55] Completed 12%
[07:24:19] Completed 13%
[07:26:43] Completed 14%
[07:29:07] Completed 15%
[07:31:32] Completed 16%
[07:33:56] Completed 17%
[07:36:20] Completed 18%
[07:38:47] Completed 19%
[07:41:12] Completed 20%
[07:43:36] Completed 21%
[07:46:02] Completed 22%
[07:48:28] Completed 23%
[07:50:53] Completed 24%
[07:53:18] Completed 25%
[07:55:43] Completed 26%
[07:58:07] Completed 27%
[08:00:32] Completed 28%
[08:02:56] Completed 29%
[08:05:21] Completed 30%
[08:07:45] Completed 31%
[08:10:09] Completed 32%
[08:12:33] Completed 33%
[08:14:58] Completed 34%
[08:17:22] Completed 35%
[08:19:46] Completed 36%
[08:22:10] Completed 37%
[08:24:35] Completed 38%
[08:26:59] Completed 39%
[08:29:24] Completed 40%
[08:31:48] Completed 41%
[08:34:12] Completed 42%
[08:36:37] Completed 43%
[08:38:05] Opening C:\Documents and Settings\Swaggy\Application Data\Folding@home-gpu\MyFolding.html...
[08:39:01] Completed 44%

Jolly-Swagman · Apr 9, 2009

Ahhh spoke too soon that last one Crapped out @ 68% too

Have Reported these to Folding@home Forums

this post here - http://foldingforum.org/viewtopic.php?f=19&t=9423

and also hrsetrdr had previously Reported same here http://foldingforum.org/viewtopic.php?f=19&t=9254&p=93520#p93520

Hope they get them fixed,,, :bang head

EDIT -- Have SHUT DOWN GPU Clients ,,,,, Another 4 Same thing, on three Rigs, So keeping them shut down till Stanford FIX these 57** series of WU,s

as seems thats all that is being dished out at the moment on my GPU,s

IAmMoen · Apr 9, 2009

Wow that sucks man. Hope they fix them asap.

ihrsetrdr · Apr 9, 2009

Jolly-Swagman said:
Ahhh spoke too soon that last one Crapped out @ 68% too

Have Reported these to Folding@home Forums

this post here - http://foldingforum.org/viewtopic.php?f=19&t=9423

and also hrsetrdr had previously Reported same here http://foldingforum.org/viewtopic.php?f=19&t=9254&p=93520#p93520

Hope they get them fixed,,,

EDIT -- Have SHUT DOWN GPU Clients ,,,,, Another 4 Same thing, on three Rigs, So keeping them shut down till Stanford FIX these 57** series of WU,s

as seems thats all that is being dished out at the moment on my GPU,s

Yes, a couple of my GPU2 clients were plagued with NANs, when processing the p5775(r4,c75,g1) & p5775(r13,c27,g50) WUs; things pretty much settled out later on in the day.

Just a comment: The silence over on foldingforum is just deafening sometimes. toTOW had the same problem, is a site moderator that is fairly active (12+ posts per day) but yet nobody made any comment on the NAN situation, until you posted, JS.

Jolly-Swagman · Apr 9, 2009

Yeah got some more errors on the Wife,s Palit 9800GT 512MB GPU too, really sucks that they make to 96% then go belly up,,

This is the only Response so far From Stanford Forums

Report this post

Reply with quote

Re: 57** series WU,s Nans Detected

by toTOW on Thu Apr 09, 2009 10:52 pm
Strange, you completed this one fine :

Hi JollySwagman (team 32),
Your WU (P5779 R2 C244 G0) was added to the stats database on 2009-04-08 02:14:31 for 768 points of credit.

Project: 5780 (Run 9, Clone 271, Gen 0) has been completed successfully by another donor.
Project: 5751 (Run 6, Clone 108, Gen 293) has been completed successfully by another donor.

There's no data for Project: 5777 (Run 7, Clone 286, Gen 0) in the DB yet ...

I did have some success on my Palit 9800Gt 1BG GPU with P5777 (Run 13, Clone 296, Gen 0) it actually finished.

So still Got Two GPU, that are still down, as too many problematic WU,s on my Inno 8800GT 512MB and the Gigabyte 8800GT 256MB,

PeddlerOfFlesh · Apr 9, 2009

Did you check the GPU forum? Not only has this happened before, it's a huge problem. I thought that was the point of the new core. I don't seem to be getting any WUs with the new core now though

http://foldingforum.org/viewtopic.php?f=52&t=7953

http://foldingforum.org/viewtopic.php?f=52&t=7965

A lot of people have ended up shutting down their GPUs over it. Personally, I keep them folding, because it seems like after I get a few EUEs I will get a big stream of working ones. It was so nice to have nothing but Core_14 for a month or so.

Jolly-Swagman · Apr 9, 2009

Yeah I was just back in the Folding Forum

Here is the last response

Report this post

Reply with quote

Re: 57** series WU,s Nans Detected

by bruce on Fri Apr 10, 2009 3:38 am

Jolly-Swagman wrote:So there is obviously a problem with some of the P57** series WU,s

That is one possibility, but it's certainly not obvious. Perhaps these WUs are just slightly more efficient that the ones you were running when you last adjusted the overclocking of your GPU. If the better utiization of the GPU has pushed you into an unstable condition, the solution is to reduce the overclock -- or provide better cooling -- or whatever it takes to make the hardware stable.

That's just as "obvious" a possibility -- and when you got an error and somebody else completed the same WU, it's somewhat more likely than the series of WUs being the problem.

and mine

Edit post

Delete post

Report this post

Reply with quote

Re: 57** series WU,s Nans Detected

by Jolly-Swagman on Fri Apr 10, 2009 5:18 am
Well the Ironic ting is the the Only GPU the is Over Clocked slightly has had NO problems the others are @ stock settings and Both Palit 9800GT 1GB GPU,s have Massive Heat Pipe Heat sinks and Ram cooling to the Rear as well
and run @ 40-46C max The gigabyte 8800GT has Copper Heat pipe Cooling too @ 44-55C depending on WU the Inno 8800GT 512MB is stock settings and cooling gets to 65C Max depending on WU and has 120mm side fan Blowing on to GPU Will Have a TR HR-03 going on as soon as get new Thermal tape, and all GPU,s are (G92)

So as said some WU,s are having Problems and I,m not the only one that is as posted here viewtopic.php?f=19&t=9254,
Also some of my Team32 members also, have had problems.
I dont really care too much that these dont produce points as I mainly do it for the Science, and if there not completed or send bad results, its you guys at Stanford that are loosing out.
I,m just the Bunny (Pensioner)that forks out the Power bill to Run these, in the name of Science, and to help the greater cause, To Help in Finding Cures to Diseases,

Ohh i forgot too,, you already know that there was a Problem with Nans in the P57** series of WU,s posted here back in January viewtopic.php?f=52&t=7965

MARCI · Apr 9, 2009

JS, I'm NTS whats going on.. on my rig, going back 72 hours through the logs I see 5774, 5775, 5777, 5778, 5779 series work units all being completed with no errors. Admittedly it's a small sample size.

PeddlerOfFlesh · Apr 9, 2009

MARCI said:
JS, I'm NTS whats going on.. on my rig, going back 72 hours through the logs I see 5774, 5775, 5777, 5778, 5779 series work units all being completed with no errors. Admittedly it's a small sample size.

That's part of the problem. Because some people have no problems with them and some people (like myself) can do them sometimes, but can't at others, it's hard to diagnose. AFAIK Stanford's solution is to overclock less and fix your cooling issues. If you're already underclocked and your GPUs are at like 40C, well then...it works for other people! Must be something with you.

On a side note, I used to get ~5k PPD with the 353 point WUs. Now that I have one running it's getting about 1500 ppd.

HayesK · Apr 9, 2009

I have one of the 256MB Gigabyte 8800GT clocked 602-1886-757 (core/shader/mem) which has only one error past 50 WU, which is as far as the available logs go back.

Have not noticed any hung GPU or excessive errors recently, but have not done a through check of the logs either. The 20 GPU instances currently have 3x511pt, 15x768pt and 2x1888pt.

Jolly-Swagman · Apr 10, 2009

Well fired the other 2 GPU's back up after clearing out Work folders and Queue.dat and seem to be Ok again have submitted a couple of WU,s each

P5776, P5774 P5779 P5773

So for now there Good,,,,fingers crossed

ChasR · Apr 10, 2009

When you start having problems with EUEs it's time to reinstall the drivers. I checked the logs on a GTX260 and 8800GT and neither have any errors with lots of completed 577Xs. I'll check more when I have time. I don't think this is the WUs, guys.

Jolly-Swagman · Apr 10, 2009

Had another P5777 ,,, so it seems any of these WU,s go belly up.

Project: 5777 (Run 10, Clone 153, Gen 24)

[16:07:30] Project: 5777 (Run 10, Clone 153, Gen 24)
[16:07:30]
[16:07:30] Assembly optimizations on if available.
[16:07:30] Entering M.D.
[16:07:36] Working on Protein
[16:07:38] Client config found, loading data.
[16:07:38] Starting GUI Server
[16:10:03] Completed 1%
[16:12:26] Completed 2%
[16:14:51] Completed 3%
[16:17:15] Completed 4%
[16:19:39] Completed 5%
[16:22:03] Completed 6%
[16:24:27] Completed 7%
[16:26:52] Completed 8%
[16:29:15] Completed 9%
[16:31:39] Completed 10%
[16:34:04] Completed 11%
[16:36:28] Completed 12%
[16:38:52] Completed 13%
[16:41:25] Run: exception thrown during GuardedRun
[16:41:25] Run: exception thrown in GuardedRun -- Gromacs cannot continue further.
[16:41:25] Going to send back what have done -- stepsTotalG=20000000
[16:41:25] Work fraction=0.1385 steps=20000000.
[16:41:25] logfile size=13169 infoLength=13169 edr=0 trr=23
[16:41:25] - Writing 13705 bytes of core data to disk...
[16:41:25] Done: 13193 -> 4286 (compressed to 32.4 percent)
[16:41:25] ... Done.

Edit - This is The Gigabyte 8800GT 256MB 700/1700/750

And have the Latest Nvidia Drivers 182.50 Official Drivers WinXP from Nvidia Site

My Other Cards have that also had some issues with same WU have 182.08 Drivers

ChasR · Apr 10, 2009

JS,
I've got 23 GPUs running. I've checked the logs on a lot of them. I'm seeing less than 2% failure rate on 9800GTXs, 8800GTS(G92), 9600GSO, and 8800GT. 0% on GTX260 and GTX295. It simply can't be that you are so unlucky that you get all the bad WUs and I get almost none of them.

However, I might come to the same conclusion if my sample were smaller and I was looking at only two of my cards, a 8800GTS (G92) and a 8800GS that are unstable. THe 8800GTS is running at a 10% failure rate, all p577x. THe 8800GS a bit higher, but not all p577x. I can't conclude that those two GPUs are just unlucky or are magnets for bad WUs. When I look at current assignments, 100% of the current WUs are p577x (on my gpus anyway), therefore 100% of the failures on any unstable machine will be p577x.

Jolly-Swagman · Apr 10, 2009

Edit - This is The Gigabyte 8800GT 256MB 700/1700/750 (BAR Donated -GPU)

And have the Latest Nvidia Drivers 182.50 Official Drivers WinXP from Nvidia Site

My Other Cards have that also had some issues with same WU have 182.08 Drivers
----------------------------------------------------------------------------------------------------------

Had More WUs, go Belly Up on the GPU (listed above) they are the default Clocks, ,, So will Try Downclocking Core from 700Mhz and see if that helps,

More Nans Detected & Guarded Run Errors, and always on this WU Project: 5777 (Run 10, Clone 153, Gen 24) & Project: 5778 (Run 7, Clone 351, Gen 0)

Code:

--- Opening Log file [April 9 17:30:06 UTC] 


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.23

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Documents and Settings\Swaggy\Application Data\Folding@home-gpu
Arguments: -verbosity 9 

[17:30:06] - Ask before connecting: No
[17:30:06] - User name: JollySwagman (Team 32)
[17:30:06] - User ID: 1D1493BB0A79C9AE
[17:30:06] - Machine ID: 2
[17:30:06] 
[17:30:06] Loaded queue successfully.
[17:30:06] Initialization complete
[17:30:06] - Preparing to get new work unit...
[17:30:06] + Attempting to get work packet
[17:30:06] - Autosending finished units... [April 9 17:30:06 UTC]
[17:30:06] Trying to send all finished work units
[17:30:06] + No unsent completed units remaining.
[17:30:06] - Autosend completed
[17:30:06] - Will indicate memory of 3327 MB
[17:30:06] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 15, Stepping: 6
[17:30:06] - Connecting to assignment server
[17:30:06] Connecting to http://assign-GPU.stanford.edu:8080/
[17:30:07] Posted data.
[17:30:07] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[17:30:07] + News From Folding@Home: GPU folding beta
[17:30:07] Loaded queue successfully.
[17:30:07] Connecting to http://171.64.65.106:8080/
[17:30:08] Posted data.
[17:30:08] Initial: 0000; - Receiving payload (expected size: 68358)
[17:30:09] - Downloaded at ~66 kB/s
[17:30:09] - Averaged speed for that direction ~81 kB/s
[17:30:09] + Received work.
[17:30:09] + Closed connections
[17:30:09] 
[17:30:09] + Processing work unit
[17:30:09] Core required: FahCore_11.exe
[17:30:09] Core found.
[17:30:09] Working on queue slot 06 [April 9 17:30:09 UTC]
[17:30:09] + Working ...
[17:30:09] - Calling '.\FahCore_11.exe -dir work/ -suffix 06 -priority 96 -checkpoint 15 -verbose -lifeline 3456 -version 623'

[17:30:10] 
[17:30:10] *------------------------------*
[17:30:10] Folding@Home GPU Core - Beta
[17:30:10] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[17:30:10] 
[17:30:10] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[17:30:10] Build host: amoeba
[17:30:10] Board Type: Nvidia
[17:30:10] Core      : 
[17:30:10] Preparing to commence simulation
[17:30:10] - Looking at optimizations...
[17:30:10] - Created dyn
[17:30:10] - Files status OK
[17:30:10] - Expanded 67846 -> 350980 (decompressed 517.3 percent)
[17:30:10] Called DecompressByteArray: compressed_data_size=67846 data_size=350980, decompressed_data_size=350980 diff=0
[17:30:10] - Digital signature verified
[17:30:10] 
[17:30:10] Project: 5776 (Run 2, Clone 263, Gen 7)
[17:30:10] 
[17:30:10] Assembly optimizations on if available.
[17:30:10] Entering M.D.
[17:30:16] Working on Protein
[17:30:18] Client config found, loading data.
[17:30:18] Starting GUI Server
[17:35:32] Completed 1%
[17:42:55] Completed 2%
[17:50:39] Completed 3%
[17:57:53] Completed 4%
[18:05:10] Completed 5%
[18:12:24] Completed 6%
[18:19:42] Completed 7%
[18:27:19] Completed 8%
[18:34:32] Completed 9%
[18:42:11] Completed 10%
[18:49:26] Completed 11%
[18:57:04] Completed 12%
[19:04:36] Completed 13%
[19:11:49] Completed 14%
[19:19:07] Completed 15%
[19:26:27] Completed 16%
[19:33:55] Completed 17%
[19:41:26] Completed 18%
[19:48:54] Completed 19%
[19:56:19] Completed 20%
[20:03:55] Completed 21%
[20:11:11] Completed 22%
[20:18:50] Completed 23%
[20:26:02] Completed 24%
[20:33:24] Completed 25%
[20:40:57] Completed 26%
[20:48:29] Completed 27%
[20:55:39] Completed 28%
[21:03:04] Completed 29%
[21:10:18] Completed 30%
[21:17:38] Completed 31%
[21:24:56] Completed 32%
[21:32:45] Completed 33%
[21:40:15] Completed 34%
[21:47:45] Completed 35%
[21:54:49] Completed 36%
[22:02:03] Completed 37%
[22:09:38] Completed 38%
[22:16:50] Completed 39%
[22:24:17] Completed 40%
[22:31:28] Completed 41%
[22:38:57] Completed 42%
[22:46:25] Completed 43%
[22:53:51] Completed 44%
[23:01:19] Completed 45%
[23:08:31] Completed 46%
[23:15:45] Completed 47%
[23:22:57] Completed 48%
[23:30:06] - Autosending finished units... [April 9 23:30:06 UTC]
[23:30:06] Trying to send all finished work units
[23:30:06] + No unsent completed units remaining.
[23:30:06] - Autosend completed
[23:30:06] + Working...
[23:30:17] Completed 49%
[23:37:39] Completed 50%
[23:44:56] Completed 51%
[23:52:26] Completed 52%
[23:59:36] Completed 53%
[00:06:53] Completed 54%
[00:14:04] Completed 55%
[00:21:15] Completed 56%
[00:28:54] Completed 57%
[00:36:20] Completed 58%
[00:43:44] Completed 59%
[00:51:13] Completed 60%
[00:58:25] Completed 61%
[01:05:54] Completed 62%
[01:13:25] Completed 63%
[01:20:49] Completed 64%
[01:28:15] Completed 65%
[01:35:43] Completed 66%
[01:43:06] Completed 67%
[01:50:36] Completed 68%
[01:57:51] Completed 69%
[02:05:09] Completed 70%
[02:12:17] Completed 71%
[02:19:23] Completed 72%
[02:26:35] Completed 73%
[02:33:54] Completed 74%
[02:41:31] Completed 75%
[02:48:49] Completed 76%
[02:56:13] Completed 77%
[03:03:34] Completed 78%
[03:10:49] Completed 79%
[03:18:02] Completed 80%
[03:25:09] Completed 81%
[03:32:25] Completed 82%
[03:39:36] Completed 83%
[03:45:45] Completed 84%
[03:48:09] Completed 85%
[03:50:34] Completed 86%
[03:52:58] Completed 87%
[03:55:22] Completed 88%
[03:57:46] Completed 89%
[04:00:11] Completed 90%
[04:05:15] Completed 91%
[04:13:43] Completed 92%
[04:21:35] Completed 93%
[04:29:56] Completed 94%
[04:38:34] Completed 95%
[04:46:25] Completed 96%
[04:54:34] Completed 97%
[05:02:29] Completed 98%
[05:10:47] Completed 99%
[05:18:58] Completed 100%
[05:19:00] Successful run
[05:19:00] DynamicWrapper: Finished Work Unit: sleep=10000
[05:19:10] Reserved 152420 bytes for xtc file; Cosm status=0
[05:19:10] Allocated 152420 bytes for xtc file
[05:19:10] - Reading up to 152420 from "work/wudata_06.xtc": Read 152420
[05:19:10] Read 152420 bytes from xtc file; available packet space=786278044
[05:19:10] xtc file hash check passed.
[05:19:10] Reserved 23232 23232 786278044 bytes for arc file=<work/wudata_06.trr> Cosm status=0
[05:19:10] Allocated 23232 bytes for arc file
[05:19:10] - Reading up to 23232 from "work/wudata_06.trr": Read 23232
[05:19:10] Read 23232 bytes from arc file; available packet space=786254812
[05:19:10] trr file hash check passed.
[05:19:10] Allocated 560 bytes for edr file
[05:19:10] Read bedfile
[05:19:10] edr file hash check passed.
[05:19:10] Allocated 35330 bytes for logfile
[05:19:10] Read logfile
[05:19:10] GuardedRun: success in DynamicWrapper
[05:19:10] GuardedRun: done
[05:19:10] Run: GuardedRun completed.
[05:19:13] - Writing 212054 bytes of core data to disk...
[05:19:14] Done: 211542 -> 184366 (compressed to 87.1 percent)
[05:19:14]   ... Done.
[05:19:14] - Shutting down core 
[05:19:14] 
[05:19:14] Folding@home Core Shutdown: FINISHED_UNIT
[05:19:18] CoreStatus = 64 (100)
[05:19:18] Unit 6 finished with 92 percent of time to deadline remaining.
[05:19:18] Updated performance fraction: 0.917922
[05:19:18] Sending work to server
[05:19:18] Project: 5776 (Run 2, Clone 263, Gen 7)
[05:19:18] - Read packet limit of 540015616... Set to 524286976.


[05:19:18] + Attempting to send results [April 10 05:19:18 UTC]
[05:19:18] - Reading file work/wuresults_06.dat from core
[05:19:18]   (Read 184878 bytes from disk)
[05:19:18] Connecting to http://171.64.65.106:8080/
[05:19:23] Posted data.
[05:19:24] Initial: 0000; - Uploaded at ~30 kB/s
[05:19:24] - Averaged speed for that direction ~30 kB/s
[05:19:24] + Results successfully sent
[05:19:24] Thank you for your contribution to Folding@Home.
[05:19:25] + Number of Units Completed: 12

[05:19:29] Trying to send all finished work units
[05:19:29] + No unsent completed units remaining.
[05:19:29] - Preparing to get new work unit...
[05:19:29] + Attempting to get work packet
[05:19:29] - Will indicate memory of 3327 MB
[05:19:29] - Connecting to assignment server
[05:19:29] Connecting to http://assign-GPU.stanford.edu:8080/
[05:19:30] Posted data.
[05:19:30] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[05:19:30] + News From Folding@Home: GPU folding beta
[05:19:30] Loaded queue successfully.
[05:19:30] Connecting to http://171.64.65.106:8080/
[05:19:31] Posted data.
[05:19:31] Initial: 0000; - Receiving payload (expected size: 68449)
[05:19:33] - Downloaded at ~33 kB/s
[05:19:33] - Averaged speed for that direction ~71 kB/s
[05:19:33] + Received work.
[05:19:33] Trying to send all finished work units
[05:19:33] + No unsent completed units remaining.
[05:19:33] + Closed connections
[05:19:33] 
[05:19:33] + Processing work unit
[05:19:33] Core required: FahCore_11.exe
[05:19:33] Core found.
[05:19:33] Working on queue slot 07 [April 10 05:19:33 UTC]
[05:19:33] + Working ...
[05:19:33] - Calling '.\FahCore_11.exe -dir work/ -suffix 07 -priority 96 -checkpoint 15 -verbose -lifeline 3456 -version 623'

[05:19:33] 
[05:19:33] *------------------------------*
[05:19:33] Folding@Home GPU Core - Beta
[05:19:33] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[05:19:33] 
[05:19:33] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[05:19:33] Build host: amoeba
[05:19:33] Board Type: Nvidia
[05:19:33] Core      : 
[05:19:33] Preparing to commence simulation
[05:19:33] - Looking at optimizations...
[05:19:33] - Created dyn
[05:19:33] - Files status OK
[05:19:34] - Expanded 67937 -> 350980 (decompressed 516.6 percent)
[05:19:34] Called DecompressByteArray: compressed_data_size=67937 data_size=350980, decompressed_data_size=350980 diff=0
[05:19:34] - Digital signature verified
[05:19:34] 
[05:19:34] Project: 5774 (Run 3, Clone 290, Gen 7)
[05:19:34] 
[05:19:34] Assembly optimizations on if available.
[05:19:34] Entering M.D.
[05:19:45] Working on Protein
[05:20:41] Client config found, loading data.
[05:20:44] Starting GUI Server
[05:28:45] Completed 1%
[05:30:06] - Autosending finished units... [April 10 05:30:06 UTC]
[05:30:06] Trying to send all finished work units
[05:30:06] + No unsent completed units remaining.
[05:30:06] - Autosend completed
[05:30:06] + Working...
[05:36:59] Completed 2%
[05:45:05] Completed 3%
[05:53:16] Completed 4%
[06:01:22] Completed 5%
[06:09:33] Completed 6%
[06:18:14] Completed 7%
[06:26:09] Completed 8%
[06:34:22] Completed 9%
[06:42:56] Completed 10%
[06:50:55] Completed 11%
[06:59:11] Completed 12%
[07:07:41] Completed 13%
[07:15:30] Completed 14%
[07:23:51] Completed 15%
[07:32:34] Completed 16%
[07:40:48] Completed 17%
[07:49:13] Completed 18%
[07:57:26] Completed 19%
[08:05:30] Completed 20%
[08:13:51] Completed 21%
[08:22:04] Completed 22%
[08:30:39] Completed 23%
[08:38:54] Completed 24%
[08:47:22] Completed 25%
[08:55:18] Completed 26%
[09:03:28] Completed 27%
[09:11:40] Completed 28%
[09:19:52] Completed 29%
[09:28:08] Completed 30%
[09:36:36] Completed 31%
[09:44:46] Completed 32%
[09:53:21] Completed 33%
[10:01:53] Completed 34%
[10:10:23] Completed 35%
[10:19:07] Completed 36%
[10:27:16] Completed 37%
[10:35:40] Completed 38%
[10:44:24] Completed 39%
[10:52:31] Completed 40%
[11:01:04] Completed 41%
[11:09:19] Completed 42%
[11:17:30] Completed 43%
[11:26:07] Completed 44%
[11:30:07] - Autosending finished units... [April 10 11:30:07 UTC]
[11:30:07] Trying to send all finished work units
[11:30:07] + No unsent completed units remaining.
[11:30:07] - Autosend completed
[11:30:07] + Working...
[11:34:26] Completed 45%
[11:42:23] Completed 46%
[11:50:29] Completed 47%
[11:58:17] Completed 48%
[12:05:46] Completed 49%
[12:13:28] Completed 50%
[12:21:19] Completed 51%
[12:29:47] Completed 52%
[12:38:16] Completed 53%
[12:46:42] Completed 54%
[12:54:47] Completed 55%
[13:03:07] Completed 56%
[13:11:18] Completed 57%
[13:18:58] Completed 58%
[13:27:11] Completed 59%
[13:30:10] Completed 60%
[13:32:34] Completed 61%
[13:34:59] Completed 62%
[13:37:23] Completed 63%
[13:39:47] Completed 64%
[13:42:11] Completed 65%
[13:44:36] Completed 66%
[13:47:00] Completed 67%
[13:49:24] Completed 68%
[13:51:49] Completed 69%
[13:54:13] Completed 70%
[13:56:38] Completed 71%
[13:59:02] Completed 72%
[14:01:26] Completed 73%
[14:03:50] Completed 74%
[14:06:15] Completed 75%
[14:08:39] Completed 76%
[14:11:03] Completed 77%
[14:13:28] Completed 78%
[14:15:52] Completed 79%
[14:18:16] Completed 80%
[14:20:40] Completed 81%
[14:23:05] Completed 82%
[14:25:29] Completed 83%
[14:27:54] Completed 84%
[14:30:18] Completed 85%
[14:32:42] Completed 86%
[14:35:06] Completed 87%
[14:37:31] Completed 88%
[14:39:55] Completed 89%
[14:42:19] Completed 90%
[14:44:44] Completed 91%
[14:47:08] Completed 92%
[14:49:32] Completed 93%
[14:51:57] Completed 94%
[14:54:20] Completed 95%
[14:56:45] Completed 96%
[14:59:09] Completed 97%
[15:01:33] Completed 98%
[15:03:58] Completed 99%
[15:06:22] Completed 100%
[15:06:22] Successful run
[15:06:22] DynamicWrapper: Finished Work Unit: sleep=10000
[15:06:32] Reserved 152400 bytes for xtc file; Cosm status=0
[15:06:32] Allocated 152400 bytes for xtc file
[15:06:32] - Reading up to 152400 from "work/wudata_07.xtc": Read 152400
[15:06:32] Read 152400 bytes from xtc file; available packet space=786278064
[15:06:32] xtc file hash check passed.
[15:06:32] Reserved 23232 23232 786278064 bytes for arc file=<work/wudata_07.trr> Cosm status=0
[15:06:32] Allocated 23232 bytes for arc file
[15:06:32] - Reading up to 23232 from "work/wudata_07.trr": Read 23232
[15:06:32] Read 23232 bytes from arc file; available packet space=786254832
[15:06:32] trr file hash check passed.
[15:06:32] Allocated 560 bytes for edr file
[15:06:32] Read bedfile
[15:06:32] edr file hash check passed.
[15:06:32] Allocated 35213 bytes for logfile
[15:06:32] Read logfile
[15:06:32] GuardedRun: success in DynamicWrapper
[15:06:32] GuardedRun: done
[15:06:32] Run: GuardedRun completed.
[15:06:36] - Writing 211917 bytes of core data to disk...
[15:06:36] Done: 211405 -> 184193 (compressed to 87.1 percent)
[15:06:36]   ... Done.
[15:06:36] - Shutting down core 
[15:06:36] 
[15:06:36] Folding@home Core Shutdown: FINISHED_UNIT
[15:06:38] CoreStatus = 64 (100)
[15:06:38] Unit 7 finished with 93 percent of time to deadline remaining.
[15:06:38] Updated performance fraction: 0.924986
[15:06:38] Sending work to server
[15:06:38] Project: 5774 (Run 3, Clone 290, Gen 7)
[15:06:38] - Read packet limit of 540015616... Set to 524286976.


[15:06:38] + Attempting to send results [April 10 15:06:38 UTC]
[15:06:38] - Reading file work/wuresults_07.dat from core
[15:06:38]   (Read 184705 bytes from disk)
[15:06:38] Connecting to http://171.64.65.106:8080/
[15:06:44] Posted data.
[15:06:44] Initial: 0000; - Uploaded at ~25 kB/s
[15:06:45] - Averaged speed for that direction ~28 kB/s
[15:06:45] + Results successfully sent
[15:06:45] Thank you for your contribution to Folding@Home.
[15:06:45] + Number of Units Completed: 13

[15:06:50] Trying to send all finished work units
[15:06:50] + No unsent completed units remaining.
[15:06:50] - Preparing to get new work unit...
[15:06:50] + Attempting to get work packet
[15:06:50] - Will indicate memory of 3327 MB
[15:06:50] - Connecting to assignment server
[15:06:50] Connecting to http://assign-GPU.stanford.edu:8080/
[15:06:50] Posted data.
[15:06:50] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[15:06:50] + News From Folding@Home: GPU folding beta
[15:06:50] Loaded queue successfully.
[15:06:50] Connecting to http://171.64.65.106:8080/
[15:06:52] Posted data.
[15:06:52] Initial: 0000; - Receiving payload (expected size: 66765)
[15:06:53] - Downloaded at ~65 kB/s
[15:06:53] - Averaged speed for that direction ~70 kB/s
[15:06:53] + Received work.
[15:06:53] Trying to send all finished work units
[15:06:53] + No unsent completed units remaining.
[15:06:53] + Closed connections
[15:06:53] 
[15:06:53] + Processing work unit
[15:06:53] Core required: FahCore_11.exe
[15:06:53] Core found.
[15:06:53] Working on queue slot 08 [April 10 15:06:53 UTC]
[15:06:53] + Working ...
[15:06:53] - Calling '.\FahCore_11.exe -dir work/ -suffix 08 -priority 96 -checkpoint 15 -verbose -lifeline 3456 -version 623'

[15:06:53] 
[15:06:53] *------------------------------*
[15:06:53] Folding@Home GPU Core - Beta
[15:06:53] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[15:06:53] 
[15:06:53] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[15:06:53] Build host: amoeba
[15:06:53] Board Type: Nvidia
[15:06:53] Core      : 
[15:06:53] Preparing to commence simulation
[15:06:53] - Looking at optimizations...
[15:06:53] - Created dyn
[15:06:53] - Files status OK
[15:06:54] - Expanded 66253 -> 348500 (decompressed 526.0 percent)
[15:06:54] Called DecompressByteArray: compressed_data_size=66253 data_size=348500, decompressed_data_size=348500 diff=0
[15:06:54] - Digital signature verified
[15:06:54] 
[15:06:54] Project: 5777 (Run 10, Clone 153, Gen 24)
[15:06:54] 
[15:06:54] Assembly optimizations on if available.
[15:06:54] Entering M.D.
[15:07:00] Working on Protein
[15:07:02] Client config found, loading data.
[15:07:02] Starting GUI Server
[15:09:26] Completed 1%
[15:11:50] Completed 2%
[15:14:14] Completed 3%
[15:16:38] Completed 4%
[15:19:03] Completed 5%
[15:21:27] Completed 6%
[15:23:51] Completed 7%
[15:26:15] Completed 8%
[15:28:39] Completed 9%
[15:31:03] Completed 10%
[15:33:27] Completed 11%
[15:35:51] Completed 12%
[15:38:15] Completed 13%
[15:40:40] Completed 14%
[15:43:04] Completed 15%
[15:45:28] Completed 16%
[15:47:52] Completed 17%
[15:50:16] Completed 18%
[15:52:40] Completed 19%
[15:55:04] Completed 20%
[15:57:29] Completed 21%
[15:59:52] Completed 22%
[16:02:17] Completed 23%
[16:04:32] Completed 24%
[16:04:32] mdrun_gpu returned 
[16:04:32] NANs detected on GPU
[16:04:32] 
[16:04:32] Folding@home Core Shutdown: UNSTABLE_MACHINE
[16:04:35] CoreStatus = 7A (122)
[16:04:35] Sending work to server
[16:04:35] Project: 5777 (Run 10, Clone 153, Gen 24)
[16:04:35] - Read packet limit of 540015616... Set to 524286976.
[16:04:35] - Error: Could not get length of results file work/wuresults_08.dat
[16:04:35] - Error: Could not read unit 08 file. Removing from queue.
[16:04:35] Trying to send all finished work units
[16:04:35] + No unsent completed units remaining.
[16:04:35] - Preparing to get new work unit...
[16:04:35] + Attempting to get work packet
[16:04:35] - Will indicate memory of 3327 MB
[16:04:35] - Connecting to assignment server
[16:04:35] Connecting to http://assign-GPU.stanford.edu:8080/
[16:04:36] Posted data.
[16:04:36] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[16:04:36] + News From Folding@Home: GPU folding beta
[16:04:36] Loaded queue successfully.
[16:04:36] Connecting to http://171.64.65.106:8080/
[16:04:38] Posted data.
[16:04:38] Initial: 0000; - Receiving payload (expected size: 66765)
[16:04:39] - Downloaded at ~65 kB/s
[16:04:39] - Averaged speed for that direction ~69 kB/s
[16:04:39] + Received work.
[16:04:39] Trying to send all finished work units
[16:04:39] + No unsent completed units remaining.
[16:04:39] + Closed connections
[16:04:44] 
[16:04:44] + Processing work unit
[16:04:44] Core required: FahCore_11.exe
[16:04:44] Core found.
[16:04:44] Working on queue slot 09 [April 10 16:04:44 UTC]
[16:04:44] + Working ...
[16:04:44] - Calling '.\FahCore_11.exe -dir work/ -suffix 09 -priority 96 -checkpoint 15 -verbose -lifeline 3456 -version 623'

[16:04:44] 
[16:04:44] *------------------------------*
[16:04:44] Folding@Home GPU Core - Beta
[16:04:44] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[16:04:44] 
[16:04:44] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[16:04:44] Build host: amoeba
[16:04:44] Board Type: Nvidia
[16:04:44] Core      : 
[16:04:44] Preparing to commence simulation
[16:04:44] - Looking at optimizations...
[16:04:44] - Created dyn
[16:04:44] - Files status OK
[16:04:44] - Expanded 66253 -> 348500 (decompressed 526.0 percent)
[16:04:44] Called DecompressByteArray: compressed_data_size=66253 data_size=348500, decompressed_data_size=348500 diff=0
[16:04:44] - Digital signature verified
[16:04:45] 
[16:04:45] Project: 5777 (Run 10, Clone 153, Gen 24)
[16:04:45] 
[16:04:45] Assembly optimizations on if available.
[16:04:45] Entering M.D.
[16:04:51] Working on Protein
[16:04:53] Client config found, loading data.
[16:04:53] Starting GUI Server
[16:07:17] Completed 1%
[16:07:17] mdrun_gpu returned 
[16:07:17] Nonzero force sum on GPU
[16:07:17] 
[16:07:17] Folding@home Core Shutdown: UNSTABLE_MACHINE
[16:07:21] CoreStatus = 7A (122)
[16:07:21] Sending work to server
[16:07:21] Project: 5777 (Run 10, Clone 153, Gen 24)
[16:07:21] - Read packet limit of 540015616... Set to 524286976.
[16:07:21] - Error: Could not get length of results file work/wuresults_09.dat
[16:07:21] - Error: Could not read unit 09 file. Removing from queue.
[16:07:21] Trying to send all finished work units
[16:07:21] + No unsent completed units remaining.
[16:07:21] - Preparing to get new work unit...
[16:07:21] + Attempting to get work packet
[16:07:21] - Will indicate memory of 3327 MB
[16:07:21] - Connecting to assignment server
[16:07:21] Connecting to http://assign-GPU.stanford.edu:8080/
[16:07:22] Posted data.
[16:07:22] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[16:07:22] + News From Folding@Home: GPU folding beta
[16:07:22] Loaded queue successfully.
[16:07:22] Connecting to http://171.64.65.106:8080/
[16:07:23] Posted data.
[16:07:23] Initial: 0000; - Receiving payload (expected size: 66765)
[16:07:24] - Downloaded at ~65 kB/s
[16:07:24] - Averaged speed for that direction ~68 kB/s
[16:07:24] + Received work.
[16:07:24] Trying to send all finished work units
[16:07:24] + No unsent completed units remaining.
[16:07:24] + Closed connections
[16:07:29] 
[16:07:29] + Processing work unit
[16:07:29] Core required: FahCore_11.exe
[16:07:29] Core found.
[16:07:29] Working on queue slot 00 [April 10 16:07:29 UTC]
[16:07:29] + Working ...
[16:07:29] - Calling '.\FahCore_11.exe -dir work/ -suffix 00 -priority 96 -checkpoint 15 -verbose -lifeline 3456 -version 623'

[16:07:29] 
[16:07:29] *------------------------------*
[16:07:29] Folding@Home GPU Core - Beta
[16:07:29] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[16:07:29] 
[16:07:29] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[16:07:29] Build host: amoeba
[16:07:29] Board Type: Nvidia
[16:07:29] Core      : 
[16:07:29] Preparing to commence simulation
[16:07:29] - Looking at optimizations...
[16:07:29] - Created dyn
[16:07:29] - Files status OK
[16:07:30] - Expanded 66253 -> 348500 (decompressed 526.0 percent)
[16:07:30] Called DecompressByteArray: compressed_data_size=66253 data_size=348500, decompressed_data_size=348500 diff=0
[16:07:30] - Digital signature verified
[16:07:30] 
[16:07:30] Project: 5777 (Run 10, Clone 153, Gen 24)
[16:07:30] 
[16:07:30] Assembly optimizations on if available.
[16:07:30] Entering M.D.
[16:07:36] Working on Protein
[16:07:38] Client config found, loading data.
[16:07:38] Starting GUI Server
[16:10:03] Completed 1%
[16:12:26] Completed 2%
[16:14:51] Completed 3%
[16:17:15] Completed 4%
[16:19:39] Completed 5%
[16:22:03] Completed 6%
[16:24:27] Completed 7%
[16:26:52] Completed 8%
[16:29:15] Completed 9%
[16:31:39] Completed 10%
[16:34:04] Completed 11%
[16:36:28] Completed 12%
[16:38:52] Completed 13%
[16:41:25] Run: exception thrown during GuardedRun
[16:41:25] Run: exception thrown in GuardedRun -- Gromacs cannot continue further.
[16:41:25] Going to send back what have done -- stepsTotalG=20000000
[16:41:25] Work fraction=0.1385 steps=20000000.
[16:41:25] logfile size=13169 infoLength=13169 edr=0 trr=23
[16:41:25] - Writing 13705 bytes of core data to disk...
[16:41:25] Done: 13193 -> 4286 (compressed to 32.4 percent)
[16:41:25]   ... Done.
[16:41:25] 
[16:41:25] Folding@home Core Shutdown: UNSTABLE_MACHINE
[16:41:29] CoreStatus = 7A (122)
[16:41:29] Sending work to server
[16:41:29] Project: 5777 (Run 10, Clone 153, Gen 24)
[16:41:29] - Read packet limit of 540015616... Set to 524286976.


[16:41:29] + Attempting to send results [April 10 16:41:29 UTC]
[16:41:29] - Reading file work/wuresults_00.dat from core
[16:41:29]   (Read 4798 bytes from disk)
[16:41:29] Connecting to http://171.64.65.106:8080/
[16:41:30] Posted data.
[16:41:31] Initial: 0000; - Uploaded at ~2 kB/s
[16:41:31] - Averaged speed for that direction ~19 kB/s
[16:41:31] + Results successfully sent
[16:41:31] Thank you for your contribution to Folding@Home.
[16:41:35] Trying to send all finished work units
[16:41:35] + No unsent completed units remaining.
[16:41:35] - Preparing to get new work unit...
[16:41:35] + Attempting to get work packet
[16:41:35] - Will indicate memory of 3327 MB
[16:41:35] - Connecting to assignment server
[16:41:35] Connecting to http://assign-GPU.stanford.edu:8080/
[16:41:36] Posted data.
[16:41:36] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[16:41:36] + News From Folding@Home: GPU folding beta
[16:41:36] Loaded queue successfully.
[16:41:36] Connecting to http://171.67.108.11:8080/
[16:41:37] Posted data.
[16:41:37] Initial: 0000; - Receiving payload (expected size: 99337)
[16:41:38] - Downloaded at ~97 kB/s
[16:41:38] - Averaged speed for that direction ~74 kB/s
[16:41:38] + Received work.
[16:41:38] Trying to send all finished work units
[16:41:38] + No unsent completed units remaining.
[16:41:38] + Closed connections
[16:41:43] 
[16:41:43] + Processing work unit
[16:41:43] Core required: FahCore_11.exe
[16:41:43] Core found.
[16:41:43] Working on queue slot 01 [April 10 16:41:43 UTC]
[16:41:43] + Working ...
[16:41:43] - Calling '.\FahCore_11.exe -dir work/ -suffix 01 -priority 96 -checkpoint 15 -verbose -lifeline 3456 -version 623'

[16:41:43] 
[16:41:43] *------------------------------*
[16:41:43] Folding@Home GPU Core - Beta
[16:41:43] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[16:41:43] 
[16:41:43] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[16:41:43] Build host: amoeba
[16:41:43] Board Type: Nvidia
[16:41:43] Core      : 
[16:41:43] Preparing to commence simulation
[16:41:43] - Looking at optimizations...
[16:41:44] - Created dyn
[16:41:44] - Files status OK
[16:41:44] - Expanded 98825 -> 492276 (decompressed 498.1 percent)
[16:41:44] Called DecompressByteArray: compressed_data_size=98825 data_size=492276, decompressed_data_size=492276 diff=0
[16:41:44] - Digital signature verified
[16:41:45] 
[16:41:45] Project: 5751 (Run 6, Clone 108, Gen 293)
[16:41:45] 
[16:41:45] Assembly optimizations on if available.
[16:41:45] Entering M.D.
[16:41:52] Working on Protein
[16:41:56] Client config found, loading data.
[16:41:56] Starting GUI Server
[16:44:56] Completed 1%
[16:47:56] Completed 2%
[16:50:56] Completed 3%
[16:53:56] Completed 4%
[16:56:56] Completed 5%
[16:59:55] Completed 6%
[17:02:56] Completed 7%
[17:05:55] Completed 8%
[17:08:55] Completed 9%
[17:11:55] Completed 10%
[17:14:55] Completed 11%
[17:17:55] Completed 12%
[17:20:55] Completed 13%
[17:23:54] Completed 14%
[17:26:54] Completed 15%
[17:29:54] Completed 16%
[17:30:07] - Autosending finished units... [April 10 17:30:07 UTC]
[17:30:07] Trying to send all finished work units
[17:30:07] + No unsent completed units remaining.
[17:30:07] - Autosend completed
[17:30:07] + Working...
[17:32:54] Completed 17%
[17:35:54] Completed 18%
[17:38:54] Completed 19%
[17:41:54] Completed 20%
[17:44:53] Completed 21%
[17:47:53] Completed 22%
[17:50:53] Completed 23%
[17:53:53] Completed 24%
[17:56:53] Completed 25%
[17:59:53] Completed 26%
[18:02:52] Completed 27%
[18:05:53] Completed 28%
[18:08:53] Completed 29%
[18:11:52] Completed 30%
[18:14:52] Completed 31%
[18:17:52] Completed 32%
[18:20:52] Completed 33%
[18:23:52] Completed 34%
[18:26:51] Completed 35%
[18:29:51] Completed 36%
[18:32:51] Completed 37%
[18:35:51] Completed 38%
[18:38:51] Completed 39%
[18:41:51] Completed 40%
[18:44:37] + Paused
[18:44:57] ***** Got a SIGTERM signal (2)
[18:44:57] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [April 10 18:56:59 UTC] 


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.23

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Documents and Settings\Swaggy\Application Data\Folding@home-gpu
Arguments: -verbosity 9 

[18:56:59] - Ask before connecting: No
[18:56:59] - User name: JollySwagman (Team 32)
[18:56:59] - User ID: 1D1493BB0A79C9AE
[18:56:59] - Machine ID: 2
[18:56:59] 
[18:56:59] Loaded queue successfully.
[18:56:59] Initialization complete
[18:56:59] 
[18:56:59] + Processing work unit
[18:56:59] Core required: FahCore_11.exe
[18:56:59] Core found.
[18:56:59] - Autosending finished units... [April 10 18:56:59 UTC]
[18:56:59] Trying to send all finished work units
[18:56:59] + No unsent completed units remaining.
[18:56:59] - Autosend completed
[18:56:59] Working on queue slot 01 [April 10 18:56:59 UTC]
[18:56:59] + Working ...
[18:56:59] - Calling '.\FahCore_11.exe -dir work/ -suffix 01 -priority 96 -checkpoint 15 -verbose -lifeline 380 -version 623'

[18:56:59] 
[18:56:59] *------------------------------*
[18:56:59] Folding@Home GPU Core - Beta
[18:56:59] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[18:56:59] 
[18:56:59] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[18:56:59] Build host: amoeba
[18:56:59] Board Type: Nvidia
[18:56:59] Core      : 
[18:56:59] Preparing to commence simulation
[18:56:59] - Looking at optimizations...
[18:57:00] - Files status OK
[18:57:00] - Expanded 98825 -> 492276 (decompressed 498.1 percent)
[18:57:00] Called DecompressByteArray: compressed_data_size=98825 data_size=492276, decompressed_data_size=492276 diff=0
[18:57:00] - Digital signature verified
[18:57:00] 
[18:57:00] Project: 5751 (Run 6, Clone 108, Gen 293)
[18:57:00] 
[18:57:00] Assembly optimizations on if available.
[18:57:00] Entering M.D.
[18:57:06] Will resume from checkpoint file
[18:57:06] Working on Protein
[18:57:11] Client config found, loading data.
[18:57:11] Starting GUI Server
[18:57:11] Resuming from checkpoint
[18:57:11] Verified work/wudata_01.log
[18:57:11] Verified work/wudata_01.edr
[18:57:11] Verified work/wudata_01.xtc
[18:57:11] Completed 40%
[18:59:15] Completed 41%
[19:01:24] Completed 42%
[19:03:37] Completed 43%
[19:05:52] Completed 44%
[19:08:05] Completed 45%
[19:10:18] Completed 46%
[19:12:31] Completed 47%
[19:14:43] Completed 48%
[19:16:57] Completed 49%
[19:19:10] Completed 50%
[19:21:23] Completed 51%
[19:23:36] Completed 52%
[19:25:48] Completed 53%
[19:28:01] Completed 54%
[19:30:14] Completed 55%
[19:32:27] Completed 56%
[19:34:40] Completed 57%
[19:36:53] Completed 58%
[19:39:06] Completed 59%
[19:41:19] Completed 60%
[19:43:32] Completed 61%
[19:45:45] Completed 62%
[19:47:58] Completed 63%
[19:50:10] Completed 64%
[19:52:23] Completed 65%
[19:54:36] Completed 66%
[19:56:49] Completed 67%
[19:59:02] Completed 68%
[20:01:15] Completed 69%
[20:03:28] Completed 70%
[20:05:41] Completed 71%
[20:07:54] Completed 72%
[20:10:07] Completed 73%
[20:12:20] Completed 74%
[20:14:33] Completed 75%
[20:16:46] Completed 76%
[20:18:58] Completed 77%
[20:21:11] Completed 78%
[20:23:24] Completed 79%
[20:25:37] Completed 80%
[20:27:50] Completed 81%
[20:27:51] mdrun_gpu returned 
[20:27:51] Nonzero force sum on GPU
[20:27:51] 
[20:27:51] Folding@home Core Shutdown: UNSTABLE_MACHINE
[20:27:53] CoreStatus = 7A (122)
[20:27:53] Sending work to server
[20:27:53] Project: 5751 (Run 6, Clone 108, Gen 293)
[20:27:53] - Read packet limit of 540015616... Set to 524286976.
[20:27:53] - Error: Could not get length of results file work/wuresults_01.dat
[20:27:53] - Error: Could not read unit 01 file. Removing from queue.
[20:27:53] Trying to send all finished work units
[20:27:53] + No unsent completed units remaining.
[20:27:53] - Preparing to get new work unit...
[20:27:53] + Attempting to get work packet
[20:27:53] - Will indicate memory of 3327 MB
[20:27:53] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 15, Stepping: 6
[20:27:53] - Connecting to assignment server
[20:27:53] Connecting to http://assign-GPU.stanford.edu:8080/
[20:27:55] Posted data.
[20:27:55] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[20:27:55] + News From Folding@Home: GPU folding beta
[20:27:55] Loaded queue successfully.
[20:27:55] Connecting to http://171.64.65.106:8080/
[20:27:56] Posted data.
[20:27:56] Initial: 0000; - Receiving payload (expected size: 63749)
[20:27:58] - Downloaded at ~31 kB/s
[20:27:58] - Averaged speed for that direction ~65 kB/s
[20:27:58] + Received work.
[20:27:58] Trying to send all finished work units
[20:27:58] + No unsent completed units remaining.
[20:27:58] + Closed connections
[20:28:03] 
[20:28:03] + Processing work unit
[20:28:03] Core required: FahCore_11.exe
[20:28:03] Core found.
[20:28:03] Working on queue slot 02 [April 10 20:28:03 UTC]
[20:28:03] + Working ...
[20:28:03] - Calling '.\FahCore_11.exe -dir work/ -suffix 02 -priority 96 -checkpoint 15 -verbose -lifeline 380 -version 623'

[20:28:03] 
[20:28:03] *------------------------------*
[20:28:03] Folding@Home GPU Core - Beta
[20:28:03] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[20:28:03] 
[20:28:03] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[20:28:03] Build host: amoeba
[20:28:03] Board Type: Nvidia
[20:28:03] Core      : 
[20:28:03] Preparing to commence simulation
[20:28:03] - Looking at optimizations...
[20:28:04] - Created dyn
[20:28:04] - Files status OK
[20:28:04] - Expanded 63237 -> 348500 (decompressed 551.1 percent)
[20:28:04] Called DecompressByteArray: compressed_data_size=63237 data_size=348500, decompressed_data_size=348500 diff=0
[20:28:04] - Digital signature verified
[20:28:04] 
[20:28:04] Project: 5778 (Run 7, Clone 351, Gen 0)
[20:28:04] 
[20:28:04] Assembly optimizations on if available.
[20:28:04] Entering M.D.
[20:28:11] Working on Protein
[20:28:13] Client config found, loading data.
[20:28:13] Starting GUI Server
[20:30:47] Completed 1%
[20:33:22] Completed 2%
[20:35:56] Completed 3%
[20:38:30] Completed 4%
[20:41:04] Completed 5%
[20:43:38] Completed 6%
[20:46:12] Completed 7%
[20:48:47] Completed 8%
[20:51:21] Completed 9%
[20:53:55] Completed 10%
[20:56:29] Completed 11%
[20:59:03] Completed 12%
[21:01:37] Completed 13%
[21:04:11] Completed 14%
[21:06:45] Completed 15%
[21:09:19] Completed 16%
[21:11:54] Completed 17%
[21:14:28] Completed 18%
[21:17:02] Completed 19%
[21:19:36] Completed 20%
[21:22:11] Completed 21%
[21:24:45] Completed 22%
[21:27:19] Completed 23%
[21:29:53] Completed 24%
[21:32:27] Completed 25%
[21:35:01] Completed 26%
[21:37:35] Completed 27%
[21:40:10] Completed 28%
[21:42:44] Completed 29%
[21:45:18] Completed 30%
[21:47:52] Completed 31%
[21:50:26] Completed 32%
[21:53:00] Completed 33%
[21:55:34] Completed 34%
[21:58:09] Completed 35%
[22:00:43] Completed 36%
[22:03:17] Completed 37%
[22:05:52] Completed 38%
[22:08:26] Completed 39%
[22:11:00] Completed 40%
[22:13:35] Completed 41%
[22:16:09] Completed 42%
[22:18:43] Completed 43%
[22:21:17] Completed 44%
[22:23:51] Completed 45%
[22:26:26] Completed 46%
[22:29:00] Completed 47%
[22:31:34] Completed 48%
[22:34:08] Completed 49%
[22:36:42] Completed 50%
[22:39:16] Completed 51%
[22:41:50] Completed 52%
[22:44:24] Completed 53%
[22:46:59] Completed 54%
[22:49:33] Completed 55%
[22:52:07] Completed 56%
[22:54:41] Completed 57%
[22:57:15] Completed 58%
[22:59:50] Completed 59%
[23:02:24] Completed 60%
[23:04:58] Completed 61%
[23:07:32] Completed 62%
[23:10:06] Completed 63%
[23:12:40] Completed 64%
[23:15:14] Completed 65%
[23:17:48] Completed 66%
[23:20:23] Completed 67%
[23:22:57] Completed 68%
[23:25:31] Completed 69%
[23:28:05] Completed 70%
[23:30:39] Completed 71%
[23:33:13] Completed 72%
[23:35:48] Completed 73%
[23:37:55] Completed 74%
[23:37:55] mdrun_gpu returned 
[23:37:55] NANs detected on GPU
[23:37:55] 
[23:37:55] Folding@home Core Shutdown: UNSTABLE_MACHINE
[23:37:59] CoreStatus = 7A (122)
[23:37:59] Sending work to server
[23:37:59] Project: 5778 (Run 7, Clone 351, Gen 0)
[23:37:59] - Read packet limit of 540015616... Set to 524286976.
[23:37:59] - Error: Could not get length of results file work/wuresults_02.dat
[23:37:59] - Error: Could not read unit 02 file. Removing from queue.
[23:37:59] Trying to send all finished work units
[23:37:59] + No unsent completed units remaining.
[23:37:59] - Preparing to get new work unit...
[23:37:59] + Attempting to get work packet
[23:37:59] - Will indicate memory of 3327 MB
[23:37:59] - Connecting to assignment server
[23:37:59] Connecting to http://assign-GPU.stanford.edu:8080/
[23:38:00] Posted data.
[23:38:00] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[23:38:00] + News From Folding@Home: GPU folding beta
[23:38:00] Loaded queue successfully.
[23:38:00] Connecting to http://171.64.65.106:8080/
[23:38:02] Posted data.
[23:38:02] Initial: 0000; - Receiving payload (expected size: 63749)
[23:38:04] - Downloaded at ~31 kB/s
[23:38:04] - Averaged speed for that direction ~58 kB/s
[23:38:04] + Received work.
[23:38:04] Trying to send all finished work units
[23:38:04] + No unsent completed units remaining.
[23:38:04] + Closed connections
[23:38:09] 
[23:38:09] + Processing work unit
[23:38:09] Core required: FahCore_11.exe
[23:38:09] Core found.
[23:38:09] Working on queue slot 03 [April 10 23:38:09 UTC]
[23:38:09] + Working ...
[23:38:09] - Calling '.\FahCore_11.exe -dir work/ -suffix 03 -priority 96 -checkpoint 15 -verbose -lifeline 380 -version 623'

[23:38:09] 
[23:38:09] *------------------------------*
[23:38:09] Folding@Home GPU Core - Beta
[23:38:09] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[23:38:09] 
[23:38:09] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[23:38:09] Build host: amoeba
[23:38:09] Board Type: Nvidia
[23:38:09] Core      : 
[23:38:09] Preparing to commence simulation
[23:38:09] - Looking at optimizations...
[23:38:10] - Created dyn
[23:38:10] - Files status OK
[23:38:10] - Expanded 63237 -> 348500 (decompressed 551.1 percent)
[23:38:10] Called DecompressByteArray: compressed_data_size=63237 data_size=348500, decompressed_data_size=348500 diff=0
[23:38:10] - Digital signature verified
[23:38:10] 
[23:38:10] Project: 5778 (Run 7, Clone 351, Gen 0)
[23:38:10] 
[23:38:10] Assembly optimizations on if available.
[23:38:10] Entering M.D.
[23:38:17] Working on Protein
[23:38:19] Client config found, loading data.
[23:38:19] Starting GUI Server
[23:40:53] Completed 1%
[23:43:31] Completed 2%
[23:46:08] Completed 3%
[23:48:44] Completed 4%
[23:51:18] Completed 5%
[23:51:18] mdrun_gpu returned 
[23:51:18] Nonzero force sum on GPU
[23:51:18] 
[23:51:18] Folding@home Core Shutdown: UNSTABLE_MACHINE
[23:51:21] CoreStatus = 7A (122)
[23:51:21] Sending work to server
[23:51:21] Project: 5778 (Run 7, Clone 351, Gen 0)
[23:51:21] - Read packet limit of 540015616... Set to 524286976.
[23:51:21] - Error: Could not get length of results file work/wuresults_03.dat
[23:51:21] - Error: Could not read unit 03 file. Removing from queue.
[23:51:21] Trying to send all finished work units
[23:51:21] + No unsent completed units remaining.
[23:51:21] - Preparing to get new work unit...
[23:51:21] + Attempting to get work packet
[23:51:21] - Will indicate memory of 3327 MB
[23:51:21] - Connecting to assignment server
[23:51:21] Connecting to http://assign-GPU.stanford.edu:8080/
[23:51:22] Posted data.
[23:51:22] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[23:51:22] + News From Folding@Home: GPU folding beta
[23:51:22] Loaded queue successfully.
[23:51:22] Connecting to http://171.64.65.106:8080/
[23:51:23] Posted data.
[23:51:23] Initial: 0000; - Receiving payload (expected size: 63749)
[23:51:25] - Downloaded at ~31 kB/s
[23:51:25] - Averaged speed for that direction ~53 kB/s
[23:51:25] + Received work.
[23:51:25] Trying to send all finished work units
[23:51:25] + No unsent completed units remaining.
[23:51:25] + Closed connections
[23:51:30] 
[23:51:30] + Processing work unit
[23:51:30] Core required: FahCore_11.exe
[23:51:30] Core found.
[23:51:30] Working on queue slot 04 [April 10 23:51:30 UTC]
[23:51:30] + Working ...
[23:51:30] - Calling '.\FahCore_11.exe -dir work/ -suffix 04 -priority 96 -checkpoint 15 -verbose -lifeline 380 -version 623'

[23:51:30] 
[23:51:30] *------------------------------*
[23:51:30] Folding@Home GPU Core - Beta
[23:51:30] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[23:51:30] 
[23:51:30] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[23:51:30] Build host: amoeba
[23:51:30] Board Type: Nvidia
[23:51:30] Core      : 
[23:51:30] Preparing to commence simulation
[23:51:30] - Looking at optimizations...
[23:51:31] - Created dyn
[23:51:31] - Files status OK
[23:51:31] - Expanded 63237 -> 348500 (decompressed 551.1 percent)
[23:51:31] Called DecompressByteArray: compressed_data_size=63237 data_size=348500, decompressed_data_size=348500 diff=0
[23:51:31] - Digital signature verified
[23:51:31] 
[23:51:31] Project: 5778 (Run 7, Clone 351, Gen 0)
[23:51:31] 
[23:51:31] Assembly optimizations on if available.
[23:51:31] Entering M.D.
[23:51:37] Working on Protein
[23:51:39] Client config found, loading data.
[23:51:40] Starting GUI Server
[23:54:14] Completed 1%
[23:56:48] Completed 2%
[23:59:22] Completed 3%
[00:01:56] Completed 4%
[00:04:30] Completed 5%
[00:07:05] Completed 6%
[00:09:39] Completed 7%
[00:12:13] Completed 8%
[00:14:47] Completed 9%
[00:17:21] Completed 10%
[00:19:55] Completed 11%
[00:22:29] Completed 12%
[00:25:03] Completed 13%
[00:27:38] Completed 14%
[00:30:12] Completed 15%
[00:32:46] Completed 16%
[00:35:20] Completed 17%
[00:37:54] Completed 18%
[00:40:28] Completed 19%
[00:43:03] Completed 20%
[00:45:37] Completed 21%
[00:48:11] Completed 22%
[00:50:45] Completed 23%
[00:53:19] Completed 24%
[00:55:54] Completed 25%
[00:56:59] - Autosending finished units... [April 11 00:56:59 UTC]
[00:56:59] Trying to send all finished work units
[00:56:59] + No unsent completed units remaining.
[00:56:59] - Autosend completed
[00:56:59] + Working...
[00:58:02] Completed 26%
[00:58:02] mdrun_gpu returned 
[00:58:02] NANs detected on GPU
[00:58:02] 
[00:58:02] Folding@home Core Shutdown: UNSTABLE_MACHINE
[00:58:04] CoreStatus = 7A (122)
[00:58:04] Sending work to server
[00:58:04] Project: 5778 (Run 7, Clone 351, Gen 0)
[00:58:04] - Read packet limit of 540015616... Set to 524286976.
[00:58:04] - Error: Could not get length of results file work/wuresults_04.dat
[00:58:04] - Error: Could not read unit 04 file. Removing from queue.
[00:58:04] Trying to send all finished work units
[00:58:04] + No unsent completed units remaining.
[00:58:04] - Preparing to get new work unit...
[00:58:04] + Attempting to get work packet
[00:58:04] - Will indicate memory of 3327 MB
[00:58:04] - Connecting to assignment server
[00:58:04] Connecting to http://assign-GPU.stanford.edu:8080/
[00:58:05] Posted data.
[00:58:05] Initial: 40AB; - Successful: assigned to (171.64.65.106).
[00:58:05] + News From Folding@Home: GPU folding beta
[00:58:05] Loaded queue successfully.
[00:58:05] Connecting to http://171.64.65.106:8080/
[00:58:06] Posted data.
[00:58:06] Initial: 0000; - Receiving payload (expected size: 63749)
[00:58:07] - Downloaded at ~62 kB/s
[00:58:07] - Averaged speed for that direction ~55 kB/s
[00:58:07] + Received work.
[00:58:07] Trying to send all finished work units
[00:58:07] + No unsent completed units remaining.
[00:58:07] + Closed connections
[00:58:12] 
[00:58:12] + Processing work unit
[00:58:12] Core required: FahCore_11.exe
[00:58:12] Core found.
[00:58:13] Working on queue slot 05 [April 11 00:58:13 UTC]
[00:58:13] + Working ...
[00:58:13] - Calling '.\FahCore_11.exe -dir work/ -suffix 05 -priority 96 -checkpoint 15 -verbose -lifeline 380 -version 623'

[00:58:13] 
[00:58:13] *------------------------------*
[00:58:13] Folding@Home GPU Core - Beta
[00:58:13] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[00:58:13] 
[00:58:13] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[00:58:13] Build host: amoeba
[00:58:13] Board Type: Nvidia
[00:58:13] Core      : 
[00:58:13] Preparing to commence simulation
[00:58:13] - Looking at optimizations...
[00:58:14] - Created dyn
[00:58:14] - Files status OK
[00:58:14] - Expanded 63237 -> 348500 (decompressed 551.1 percent)
[00:58:14] Called DecompressByteArray: compressed_data_size=63237 data_size=348500, decompressed_data_size=348500 diff=0
[00:58:14] - Digital signature verified
[00:58:14] 
[00:58:14] Project: 5778 (Run 7, Clone 351, Gen 0)
[00:58:14] 
[00:58:14] Assembly optimizations on if available.
[00:58:14] Entering M.D.
[00:58:20] Working on Protein
[00:58:22] Client config found, loading data.
[00:58:22] Starting GUI Server
[01:00:56] Completed 1%
[01:03:31] Completed 2%
[01:06:05] Completed 3%
[01:08:39] Completed 4%
[01:11:13] Completed 5%
[01:13:47] Completed 6%
[01:16:21] Completed 7%
[01:18:56] Completed 8%
[01:21:30] Completed 9%
[01:24:04] Completed 10%
[01:26:38] Completed 11%
[01:29:12] Completed 12%
[01:31:46] Completed 13%
[01:34:21] Completed 14%
[01:36:55] Completed 15%
[01:39:29] Completed 16%
[01:41:24] Completed 17%
[01:41:24] mdrun_gpu returned 
[01:41:24] NANs detected on GPU
[01:41:24] 
[01:41:24] Folding@home Core Shutdown: UNSTABLE_MACHINE
[01:41:27] CoreStatus = 7A (122)
[01:41:27] Sending work to server
[01:41:27] Project: 5778 (Run 7, Clone 351, Gen 0)
[01:41:27] - Read packet limit of 540015616... Set to 524286976.
[01:41:27] - Error: Could not get length of results file work/wuresults_05.dat
[01:41:27] - Error: Could not read unit 05 file. Removing from queue.
[01:41:27] EUE limit exceeded. Pausing 24 hours.

Outback_Jon · Apr 11, 2009

Jolly-Swagman said:
Edit - This is The Gigabyte 8800GT 256MB 700/1700/750 (BAR Donated -GPU)

Had More WUs, go Belly Up on the GPU (listed above) they are the default Clocks, ,, So will Try Downclocking Core from 700Mhz and see if that helps,

Is that a stock "superclocked" card or something similar? The standard clocks for the 8800GTs are 600/1512/900. (I think anyway. That's what mine came stock)
Mine runs quite nicely at 640/1674/975.

Jolly-Swagman · Apr 11, 2009

Outback_Jon said:
Is that a stock "superclocked" card or something similar? The standard clocks for the 8800GTs are 600/1512/900. (I think anyway. That's what mine came stock)
Mine runs quite nicely at 640/1674/975.

Yeah checked on the Gigabyte Site and this one is there "SuperClocked" series of cards,

My other Stock 8800GT 512 is 650/1625/950 and runs fine, @ 1732 shaders

Even after Downclocking to 601/1732/750 or 601/1625/750 still goes Belly Up!!
& Temps have been fluctuating between 65-74C & 90% Fan speed -- Most of the other WU,s it runs @ 55C

Code:

 --- Opening Log file [April 11 04:23:36 UTC] 


# Windows GPU Console Edition #################################################
###############################################################################

                       Folding@Home Client Version 6.23

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: C:\Documents and Settings\Swaggy\Application Data\Folding@home-gpu
Arguments: -verbosity 9 

[04:23:36] - Ask before connecting: No
[04:23:36] - User name: JollySwagman (Team 32)
[04:23:36] - User ID: 1D1493BB0A79C9AE
[04:23:36] - Machine ID: 2
[04:23:36] 
[04:23:36] Loaded queue successfully.
[04:23:36] Initialization complete
[04:23:36] - Preparing to get new work unit...
[04:23:36] + Attempting to get work packet
[04:23:36] - Will indicate memory of 3327 MB
[04:23:36] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 15, Stepping: 6
[04:23:36] - Connecting to assignment server
[04:23:36] Connecting to http://assign-GPU.stanford.edu:8080/
[04:23:36] - Autosending finished units... [April 11 04:23:36 UTC]
[04:23:36] Trying to send all finished work units
[04:23:36] + No unsent completed units remaining.
[04:23:36] - Autosend completed
[04:23:37] Posted data.
[04:23:37] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[04:23:37] + News From Folding@Home: GPU folding beta
[04:23:37] Loaded queue successfully.
[04:23:37] Connecting to http://171.67.108.11:8080/
[04:23:39] Posted data.
[04:23:39] Initial: 0000; - Error: Bad packet type from server, expected work assignment
[04:23:39] - Attempt #1  to get work failed, and no other work to do.
Waiting before retry.
[04:23:53] + Attempting to get work packet
[04:23:53] - Will indicate memory of 3327 MB
[04:23:53] - Connecting to assignment server
[04:23:53] Connecting to http://assign-GPU.stanford.edu:8080/
[04:23:54] Posted data.
[04:23:54] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[04:23:54] + News From Folding@Home: GPU folding beta
[04:23:54] Loaded queue successfully.
[04:23:54] Connecting to http://171.67.108.11:8080/
[04:23:55] Posted data.
[04:23:55] Initial: 0000; - Receiving payload (expected size: 96985)
[04:23:56] - Downloaded at ~94 kB/s
[04:23:56] - Averaged speed for that direction ~62 kB/s
[04:23:56] + Received work.
[04:23:56] + Closed connections
[04:23:56] 
[04:23:56] + Processing work unit
[04:23:56] Core required: FahCore_11.exe
[04:23:56] Core found.
[04:23:56] Working on queue slot 06 [April 11 04:23:56 UTC]
[04:23:56] + Working ...
[04:23:56] - Calling '.\FahCore_11.exe -dir work/ -suffix 06 -priority 96 -checkpoint 15 -verbose -lifeline 3424 -version 623'

[04:23:56] 
[04:23:56] *------------------------------*
[04:23:56] Folding@Home GPU Core - Beta
[04:23:56] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[04:23:56] 
[04:23:56] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[04:23:56] Build host: amoeba
[04:23:56] Board Type: Nvidia
[04:23:56] Core      : 
[04:23:56] Preparing to commence simulation
[04:23:56] - Looking at optimizations...
[04:23:56] - Created dyn
[04:23:56] - Files status OK
[04:23:56] - Expanded 96473 -> 489240 (decompressed 507.1 percent)
[04:23:56] Called DecompressByteArray: compressed_data_size=96473 data_size=489240, decompressed_data_size=489240 diff=0
[04:23:56] - Digital signature verified
[04:23:56] 
[04:23:56] Project: 5753 (Run 10, Clone 256, Gen 217)
[04:23:56] 
[04:23:56] Assembly optimizations on if available.
[04:23:56] Entering M.D.
[04:24:03] Working on Protein
[04:24:07] Client config found, loading data.
[04:24:07] Starting GUI Server
[04:26:13] Completed 1%
[04:28:19] Completed 2%
[04:30:27] Completed 3%
[04:32:35] Completed 4%
[04:34:43] Completed 5%
[04:36:51] Completed 6%
[04:36:51] mdrun_gpu returned 
[04:36:51] Nonzero force sum on GPU
[04:36:51] 
[04:36:51] Folding@home Core Shutdown: UNSTABLE_MACHINE
[04:36:53] CoreStatus = 7A (122)
[04:36:53] Sending work to server
[04:36:53] Project: 5753 (Run 10, Clone 256, Gen 217)
[04:36:53] - Read packet limit of 540015616... Set to 524286976.
[04:36:53] - Error: Could not get length of results file work/wuresults_06.dat
[04:36:53] - Error: Could not read unit 06 file. Removing from queue.
[04:36:53] Trying to send all finished work units
[04:36:53] + No unsent completed units remaining.
[04:36:53] - Preparing to get new work unit...
[04:36:53] + Attempting to get work packet
[04:36:53] - Will indicate memory of 3327 MB
[04:36:53] - Connecting to assignment server
[04:36:53] Connecting to http://assign-GPU.stanford.edu:8080/
[04:36:54] Posted data.
[04:36:54] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[04:36:54] + News From Folding@Home: GPU folding beta
[04:36:54] Loaded queue successfully.
[04:36:54] Connecting to http://171.67.108.11:8080/
[04:36:55] Posted data.
[04:36:55] Initial: 0000; - Receiving payload (expected size: 96985)
[04:36:56] - Downloaded at ~94 kB/s
[04:36:56] - Averaged speed for that direction ~69 kB/s
[04:36:56] + Received work.
[04:36:56] Trying to send all finished work units
[04:36:56] + No unsent completed units remaining.
[04:36:56] + Closed connections
[04:37:01] 
[04:37:01] + Processing work unit
[04:37:01] Core required: FahCore_11.exe
[04:37:01] Core found.
[04:37:01] Working on queue slot 07 [April 11 04:37:01 UTC]
[04:37:01] + Working ...
[04:37:01] - Calling '.\FahCore_11.exe -dir work/ -suffix 07 -priority 96 -checkpoint 15 -verbose -lifeline 3424 -version 623'

[04:37:01] 
[04:37:01] *------------------------------*
[04:37:01] Folding@Home GPU Core - Beta
[04:37:01] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[04:37:01] 
[04:37:01] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[04:37:01] Build host: amoeba
[04:37:01] Board Type: Nvidia
[04:37:01] Core      : 
[04:37:01] Preparing to commence simulation
[04:37:01] - Looking at optimizations...
[04:37:01] - Created dyn
[04:37:01] - Files status OK
[04:37:01] - Expanded 96473 -> 489240 (decompressed 507.1 percent)
[04:37:01] Called DecompressByteArray: compressed_data_size=96473 data_size=489240, decompressed_data_size=489240 diff=0
[04:37:01] - Digital signature verified
[04:37:01] 
[04:37:01] Project: 5753 (Run 10, Clone 256, Gen 217)
[04:37:01] 
[04:37:02] Assembly optimizations on if available.
[04:37:02] Entering M.D.
[04:37:09] Working on Protein
[04:37:13] Client config found, loading data.
[04:37:13] Starting GUI Server
[04:39:21] Completed 1%
[04:41:01] Completed 2%
[04:41:01] mdrun_gpu returned 
[04:41:01] NANs detected on GPU
[04:41:01] 
[04:41:01] Folding@home Core Shutdown: UNSTABLE_MACHINE
[04:41:04] CoreStatus = 7A (122)
[04:41:05] Sending work to server
[04:41:05] Project: 5753 (Run 10, Clone 256, Gen 217)
[04:41:05] - Read packet limit of 540015616... Set to 524286976.
[04:41:05] - Error: Could not get length of results file work/wuresults_07.dat
[04:41:05] - Error: Could not read unit 07 file. Removing from queue.
[04:41:05] Trying to send all finished work units
[04:41:05] + No unsent completed units remaining.
[04:41:05] - Preparing to get new work unit...
[04:41:05] + Attempting to get work packet
[04:41:05] - Will indicate memory of 3327 MB
[04:41:05] - Connecting to assignment server
[04:41:05] Connecting to http://assign-GPU.stanford.edu:8080/
[04:41:05] Posted data.
[04:41:05] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[04:41:05] + News From Folding@Home: GPU folding beta
[04:41:05] Loaded queue successfully.
[04:41:05] Connecting to http://171.67.108.11:8080/
[04:41:06] Posted data.
[04:41:06] Initial: 0000; - Receiving payload (expected size: 96985)
[04:41:08] - Downloaded at ~47 kB/s
[04:41:08] - Averaged speed for that direction ~64 kB/s
[04:41:08] + Received work.
[04:41:08] Trying to send all finished work units
[04:41:08] + No unsent completed units remaining.
[04:41:08] + Closed connections
[04:41:13] 
[04:41:13] + Processing work unit
[04:41:13] Core required: FahCore_11.exe
[04:41:13] Core found.
[04:41:13] Working on queue slot 08 [April 11 04:41:13 UTC]
[04:41:13] + Working ...
[04:41:13] - Calling '.\FahCore_11.exe -dir work/ -suffix 08 -priority 96 -checkpoint 15 -verbose -lifeline 3424 -version 623'

[04:41:13] 
[04:41:13] *------------------------------*
[04:41:13] Folding@Home GPU Core - Beta
[04:41:13] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[04:41:13] 
[04:41:13] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[04:41:13] Build host: amoeba
[04:41:13] Board Type: Nvidia
[04:41:13] Core      : 
[04:41:13] Preparing to commence simulation
[04:41:13] - Looking at optimizations...
[04:41:13] - Created dyn
[04:41:13] - Files status OK
[04:41:13] - Expanded 96473 -> 489240 (decompressed 507.1 percent)
[04:41:13] Called DecompressByteArray: compressed_data_size=96473 data_size=489240, decompressed_data_size=489240 diff=0
[04:41:13] - Digital signature verified
[04:41:13] 
[04:41:13] Project: 5753 (Run 10, Clone 256, Gen 217)
[04:41:13] 
[04:41:14] Assembly optimizations on if available.
[04:41:14] Entering M.D.
[04:41:20] Working on Protein
[04:41:24] Client config found, loading data.
[04:41:24] Starting GUI Server
[04:43:32] Completed 1%
[04:45:40] Completed 2%
[04:47:48] Completed 3%
[04:49:56] Completed 4%
[04:52:03] Completed 5%
[04:54:11] Completed 6%
[04:54:11] mdrun_gpu returned 
[04:54:11] Nonzero force sum on GPU
[04:54:11] 
[04:54:11] Folding@home Core Shutdown: UNSTABLE_MACHINE
[04:54:14] CoreStatus = 7A (122)
[04:54:14] Sending work to server
[04:54:14] Project: 5753 (Run 10, Clone 256, Gen 217)
[04:54:14] - Read packet limit of 540015616... Set to 524286976.
[04:54:14] - Error: Could not get length of results file work/wuresults_08.dat
[04:54:14] - Error: Could not read unit 08 file. Removing from queue.
[04:54:14] Trying to send all finished work units
[04:54:14] + No unsent completed units remaining.
[04:54:14] - Preparing to get new work unit...
[04:54:14] + Attempting to get work packet
[04:54:14] - Will indicate memory of 3327 MB
[04:54:14] - Connecting to assignment server
[04:54:14] Connecting to http://assign-GPU.stanford.edu:8080/
[04:54:15] Posted data.
[04:54:15] Initial: 43AB; - Successful: assigned to (171.67.108.11).
[04:54:15] + News From Folding@Home: GPU folding beta
[04:54:15] Loaded queue successfully.
[04:54:15] Connecting to http://171.67.108.11:8080/
[04:54:15] Posted data.
[04:54:15] Initial: 0000; - Receiving payload (expected size: 96985)
[04:54:17] - Downloaded at ~47 kB/s
[04:54:17] - Averaged speed for that direction ~61 kB/s
[04:54:17] + Received work.
[04:54:17] Trying to send all finished work units
[04:54:17] + No unsent completed units remaining.
[04:54:17] + Closed connections
[04:54:22] 
[04:54:22] + Processing work unit
[04:54:22] Core required: FahCore_11.exe
[04:54:22] Core found.
[04:54:22] Working on queue slot 09 [April 11 04:54:22 UTC]
[04:54:22] + Working ...
[04:54:22] - Calling '.\FahCore_11.exe -dir work/ -suffix 09 -priority 96 -checkpoint 15 -verbose -lifeline 3424 -version 623'

[04:54:22] 
[04:54:22] *------------------------------*
[04:54:22] Folding@Home GPU Core - Beta
[04:54:22] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[04:54:22] 
[04:54:22] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[04:54:22] Build host: amoeba
[04:54:22] Board Type: Nvidia
[04:54:22] Core      : 
[04:54:22] Preparing to commence simulation
[04:54:22] - Looking at optimizations...
[04:54:23] - Created dyn
[04:54:23] - Files status OK
[04:54:23] - Expanded 96473 -> 489240 (decompressed 507.1 percent)
[04:54:23] Called DecompressByteArray: compressed_data_size=96473 data_size=489240, decompressed_data_size=489240 diff=0
[04:54:23] - Digital signature verified
[04:54:23] 
[04:54:23] Project: 5753 (Run 10, Clone 256, Gen 217)
[04:54:23] 
[04:54:23] Assembly optimizations on if available.
[04:54:23] Entering M.D.
[04:54:30] Working on Protein
[04:54:34] Client config found, loading data.
[04:54:34] Starting GUI Server
[04:56:42] Completed 1%
[04:58:50] Completed 2%
[05:00:58] Completed 3%
[05:03:05] Completed 4%
[05:05:13] Completed 5%

I Have Again had to SHUT DOWN the 8800GT 256MB For now Just keeps getting the Same Dam Wu,s over and crapping out,, even with further Downclock too, CORE and Shaders,

Yet the other GPU,s seem to be doing them OK, at the moment. :screwy:

Edit -
I have a More Serious Personal Life Issue that is Depressing me, to deal with at the Moment,

HayesK · Apr 11, 2009

Jolly-Swagman said:
Yeah checked on the Gigabyte Site and this one is there "SuperClocked" series of cards,

Even after Downclocking to 601/1732/750 or 601/1625/750 still goes Belly Up!!

Sounds like we have the same 256MB 8800GT gigabyte card. Mine has been running OK at 602-1886-757 (core/shader/mem) for long time. My recollection is that the memory on that card came set at 750 and would not run stable higher than my current setting. Perhaps your GPU memory is even worse than mine and needs to be underclocked to resolve the errors. You might even be able to get the shaders up similar to mine.

Good luck with the RL issues.

HayesK · Apr 11, 2009

ChasR said:
JS,
I've got 23 GPUs running. I've checked the logs on a lot of them. I'm seeing less than 2% failure rate on 9800GTXs, 8800GTS(G92), 9600GSO, and 8800GT. 0% on GTX260 and GTX295.

Checked the current logs on my 20 GPU last night using NoteTab to count the FINISHED and UNSTABLE occurrences. 1168 FINISHED and 30 UNSTABLE, which is 2.5% error rate. There were 8 GPU with zero errors. One GPU had 9 errors, starting about 3 days ago, most likely due to temperature issues. (It is GPU3 of 4 on a P6N and was running ~80C before repositioning the 120mm fan on top of the cards last night. Hope it helps). Excluding the 9 errors on GPU3, the total error rate drops to 1.8%.

Similar to ChasR, most of the errors were on a few problem cards.

Janus67 · Apr 30, 2009

I'm having the same problem with either one of my 2 GPUs on my 9800gx2. It appears that once one starts going the other one will fail and vice versa. Totally killing my PPD, as a lot are finishing but failing with NANs and/or EUEs, screwing me over.

GPU - WU Series 57** Nans detected

Member

Member

Member

Señor Senior Member

Member

Member

Member

Member

Member

Member

Member

Senior Member

Member

Senior Member

Member

Member

Member

Member

Member

Benching Team Leader

Similar threads