GPU - WU Series 57** Nans detected

Edward2 · Apr 30, 2009

The 57xx WU's are definitely tough on your video card. My GPU temps go up 10-15C when running them. The 511 pointers seem to be the worst, and I have been getting A LOT of these recently. Currently 8 out of 12 GPU clients are folding the 511 pointers.

One thing I would suggest is checking your folding "Work" folder for any leftover files. See ChasR's thread regarding this. I have found some of these leftover files in my work folders.
http://www.ocforums.com/showthread.php?t=602884

ChasR · Apr 30, 2009

Janus, what clocks are you running?

torin3 · Apr 30, 2009

I don't seem to be able to go over 1783 on my shaders before I start having problems. I've also seen that removing the cover on the cards helps with heating issues. I've been planning on trying it soon.

Janus67 · Apr 30, 2009

ChasR said:
Janus, what clocks are you running?

All stock since I saw them start happening

Which I think are what? 600c/1500s/1000m?

ChasR · Apr 30, 2009

I assume SLI is disabled?

Certainly clean out the work folder. There should be 9 files in it 8 associated with the WU in the current queue position plus core78.sta. All other files can be deleted with the client running. Failure to delete orphaned files will cause the client to produce an instant UM when it works around to the queue slot the orphaned files are associated with.
If orphaned files aren't the problem and it's failing at stock clocks, clean drivers are in order. Delete the current driver completely, first through control panel then Driver Cleaner or Driver Sweeper in safe mode without networking. Download and install the latest WHQL driver. It doesn't matter if you have the latest driver or not. Uninstall, run driver cleaner, reinstall.
If it still doesn't work, test the card with OCCT:

http://www.ocbase.com/perestroika_en/index.php?Download

If nothing else worked and it passes OCCT, I'm out of ideas related to the GPU.

ChasR · Apr 30, 2009

Here's a better test than OCCT:

https://simtk.org/home/memtest

Janus67 · Apr 30, 2009

ChasR said:
I assume SLI is disabled?

Certainly clean out the work folder. There should be 9 files in it 8 associated with the WU in the current queue position plus core78.sta. All other files can be deleted with the client running. Failure to delete orphaned files will cause the client to produce an instant UM when it works around to the queue slot the orphaned files are associated with.
If orphaned files aren't the problem and it's failing at stock clocks, clean drivers are in order. Delete the current driver completely, first through control panel then Driver Cleaner or Driver Sweeper in safe mode without networking. Download and install the latest WHQL driver. It doesn't matter if you have the latest driver or not. Uninstall, run driver cleaner, reinstall.
If it still doesn't work, test the card with OCCT:

http://www.ocbase.com/perestroika_en/index.php?Download

If nothing else worked and it passes OCCT, I'm out of ideas related to the GPU.

Yep, SLI is disabled and I'm running dual monitors. Every time I clear out the GPU1 folder only leaving the .dll files and the config file. I will try the memory one you linked in the followed post.

edit: here's part of my log:

Code:

[22:09:14] + Attempting to send results [April 30 22:09:14 UTC]
[22:09:19] + Results successfully sent
[22:09:19] Thank you for your contribution to Folding@Home.
[22:09:19] + Number of Units Completed: 89

[22:09:23] - Preparing to get new work unit...
[22:09:23] + Attempting to get work packet
[22:09:23] - Connecting to assignment server
[22:09:25] - Successful: assigned to (171.67.108.11).
[22:09:25] + News From Folding@Home: GPU folding beta
[22:09:25] Loaded queue successfully.
[22:09:26] + Closed connections
[22:09:26] 
[22:09:26] + Processing work unit
[22:09:26] Core required: FahCore_11.exe
[22:09:26] Core found.
[22:09:26] Working on queue slot 02 [April 30 22:09:26 UTC]
[22:09:26] + Working ...
[22:09:26] 
[22:09:26] *------------------------------*
[22:09:26] Folding@Home GPU Core - Beta
[22:09:26] Version 1.19 (Mon Nov 3 09:34:13 PST 2008)
[22:09:26] 
[22:09:26] Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
[22:09:26] Build host: amoeba
[22:09:26] Board Type: Nvidia
[22:09:26] Core      : 
[22:09:26] Preparing to commence simulation
[22:09:26] - Looking at optimizations...
[22:09:26] - Created dyn
[22:09:26] - Files status OK
[22:09:26] - Expanded 46738 -> 252912 (decompressed 541.1 percent)
[22:09:26] Called DecompressByteArray: compressed_data_size=46738 data_size=252912, decompressed_data_size=252912 diff=0
[22:09:26] - Digital signature verified
[22:09:26] 
[22:09:26] Project: 5765 (Run 1, Clone 63, Gen 530)
[22:09:26] 
[22:09:26] Assembly optimizations on if available.
[22:09:26] Entering M.D.
[22:09:32] Working on Protein
[22:09:33] Client config found, loading data.
[22:09:33] mdrun_gpu returned 
[22:09:33] NANs detected on GPU
[22:09:33] 
[22:09:33] Folding@home Core Shutdown: UNSTABLE_MACHINE
[22:09:36] CoreStatus = 7A (122)
[22:09:36] Sending work to server
[22:09:36] Project: 5765 (Run 1, Clone 63, Gen 530)
[22:09:36] - Read packet limit of 540015616... Set to 524286976.
[22:09:36] - Error: Could not get length of results file work/wuresults_02.dat
[22:09:36] - Error: Could not read unit 02 file. Removing from queue.
[22:09:36] - Preparing to get new work unit...
[22:09:36] + Attempting to get work packet
[22:09:36] - Connecting to assignment server
[22:09:37] - Successful: assigned to (171.67.108.11).
[22:09:37] + News From Folding@Home: GPU folding beta
[22:09:37] Loaded queue successfully.
[22:09:39] + Closed connections
[22:09:44] 
[22:09:44] + Processing work unit
[22:09:44] Core required: FahCore_11.exe
[22:09:44] Core found.
[22:09:44] Working on queue slot 03 [April 30 22:09:44 UTC]
[22:09:44] + Working ...
[22:09:44] 
[22:09:44] *------------------------------*

let me know if that is the wrong part or if I need to add/subtract part.

Janus67 · May 2, 2009

update: still getting EUEs and NANs on almost all 57xx WUs, even running at stock speeds :-( on my 9800gx2.

I just underclocked the GPU and memory by 10% to 550/900. I don't want to continue running at these speeds as I do play games on it and won't want to constantly change it in/out of games, will see if I still get NANs.

Bon3thugz43v3r · May 12, 2009

Here are some more 57xx WU's for the mix. Off my brand new GTX275 which I fold at below stock settings on the core and mem and then up the shader to 1620. These are the only two work units that have failed in the week I have owned the card.

Code:

*------------------------------*
Folding@Home GPU Core - Beta
Version 1.19 (Mon Nov 3 09:34:13 PST 2008)

Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
Build host: amoeba
Board Type: Nvidia
Core      : 
Preparing to commence simulation
- Looking at optimizations...
- Created dyn
- Files status OK
- Expanded 46710 -> 252912 (decompressed 541.4 percent)
Called DecompressByteArray: compressed_data_size=46710 data_size=252912, decompressed_data_size=252912 diff=0
- Digital signature verified

Project: 5765 (Run 2, Clone 225, Gen 431)

Assembly optimizations on if available.
Entering M.D.
Working on Protein
Client config found, loading data.
mdrun_gpu returned 
NANs detected on GPU

Folding@home Core Shutdown: UNSTABLE_MACHINE

Code:

*------------------------------*
Folding@Home GPU Core - Beta
Version 1.19 (Mon Nov 3 09:34:13 PST 2008)

Compiler  : Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86 
Build host: amoeba
Board Type: Nvidia
Core      : 
Preparing to commence simulation
- Looking at optimizations...
- Created dyn
- Files status OK
- Expanded 46763 -> 252912 (decompressed 540.8 percent)
Called DecompressByteArray: compressed_data_size=46763 data_size=252912, decompressed_data_size=252912 diff=0
- Digital signature verified

Project: 5768 (Run 13, Clone 227, Gen 297)

Assembly optimizations on if available.
Entering M.D.
Working on Protein
Client config found, loading data.
mdrun_gpu returned 
NANs detected on GPU

Folding@home Core Shutdown: UNSTABLE_MACHINE

PeddlerOfFlesh · May 12, 2009

I seem to be getting killed by the 353 point WUs. I went back to stock, deleted work, queue, etc, uninstalled drivers, did driver sweeper, went back to 180.48 (cause I KNOW I had finished some with those), ran memtestg80 with no errors and I'm still getting instant UMs

. This was a problem for a while, but just recently it seems like that card wants to get nothing but 353 point WUs.

Jolly-Swagman · May 12, 2009

Bon3thugz43v3r,,, you can Report those WU,s on the Stanford site in the Problems with specific WU area, seen a few of those already,

Bon3thugz43v3r · May 12, 2009

Jolly-Swagman said:
Bon3thugz43v3r,,, you can Report those WU,s on the Stanford site in the Problems with specific WU area, seen a few of those already,

Done. Hope I don't get a lot of these.

ChasR · May 12, 2009

You don't need to report them if they return results. Report them only if they are deleted after failure.

GPU - WU Series 57** Nans detected

Edward2

Member

ChasR

Senior Member

torin3

Member

Janus67

Benching Team Leader

ChasR

Senior Member

ChasR

Senior Member

Janus67

Benching Team Leader

Janus67

Benching Team Leader

Bon3thugz43v3r

Member

PeddlerOfFlesh

Member

Jolly-Swagman

Member

Bon3thugz43v3r

Member

ChasR

Senior Member

Similar threads