• Welcome to Overclockers Forums! Join us to reply in threads, receive reduced ads, and to customize your site experience!

Meet Tanker!

Overclockers is supported by our readers. When you click a link to make a purchase, we may earn a commission. Learn More.

Adak

Senior Member
Joined
Jan 9, 2006
Tanker is my new 4P folder, made from:

4 x 6272 Opteron Interlagos cpu's @ 2.1GHz. Each cpu has 16 cores, for a total of 64 cores.
http://www.newegg.com/Product/Product.aspx?Item=N82E16819113036

SuperMicro H8QGI-F-O mobo. Size is SWTX (16.48 x 13.0 inches).
http://www.superbiiz.com/detail.php?name=MB-H8QGIF#

4 x Noctua NH-U12DO A3 heatsinks/fans.
http://www.newegg.com/Product/Product.aspx?Item=N82E16835608023

16 x 2 GB (total of 32 GB's) of G.Skill RAM, unregistered DDR3 1600MHz.
http://www.newegg.com/Product/Produ...m1132X509988Xd048639699437159a7db5aee090e54a5

Corsair AX 1200 psu, rated Gold.
http://www.newegg.com/Product/Product.aspx?Item=N82E16817139014&Tpk=AX 1200

CubiTek HPTX Tank case. It's made for the SR-2 sized mobo, and JUST fits the SWTX size, after removing the two roof fans from the case (they're unneeded, since the Noctua's blow upward already). This is an all aluminum case.


The hardest thing about this build was making a horizontal bar to attach near the top of the case, to support the mobo with offsets on it's top row of support holes. Both ends of the bar had to bend 90 degrees, so it could attach to the two vertical plates that make up the back of the computer case, and the drive cages at the front of the case. (It forms an elongated U:

Code:
||                                          ||
||<= Two screws to attach it at each end => ||
||_____________________ ____________________||
||             Support  bar                 ||
||<= Back of case      Drive cage support =>||
||_____________________ ____________________||

Edit: A pic will be better to show the new support bar that was added:
DSCN0014.JPG

The bar with the U shaped end on it, just above the corner of the motherboard, is one end of the new support bar. I'm marking where the standoff holes need to be drilled on the bar, in this pic.

Three holes needed to be marked, drilled, and tapped (M3 size) in the mobo tray (backplate of the case), because the SuperMicro mobo has some different locations for it's offsets. It's soft aluminum, so that was easy.

The biggest impressions I got from this:

* Take the 1/8" flat bar stock aluminum to someone with a vise or anvil. Trying to do it with your feet or pliers, is harder than it seems it would be. I ended up (really) using the tow receiver on my pick up truck, a pair of vise grips, and a hammer. :D

* Even with just one fan on the radiating fins, the Noctua's are good coolers, and VERY quiet. I resisted choosing them for this build because of their cost, but they proved their worth. Their attachment hardware also is as good as it gets, imo.

* The Gold rated PSU, working in it's best (50%) area, is amazing. While folding with 64 threads, it's using just 565 Watts! [Edit: 590-625 Watts for bigadv work units, however.] Yes, the psu is expensive ($250-$290), but it will be a big money saver because it's so efficient. It's warranted for 7 years, and it's just (very quietly) loafing while the rig folds. I'll be surprised if it doesn't last twice that long.

*It's QUITE satisfying watching the log as the steps are announced complete, in (relatively) quick succession! :clap:

Tanker will replace an obsolete dual quad Intel server, which was folding with a far less efficient psu, and folding far fewer work units, and a single cpu folder, as well. There's no doubt that this replacement will recoup it's costs in lower electric bills, and fold a lot more productively than the two rigs it replaces.

Suppliers were:
Newegg (Noctua's & RAM), Superbiiz (mobo and cpu's), Sundial Micro (case), and Fry's Electronics (psu). Home Depot (metric die and tap set, automatic center punch, aluminum stock bar,etc.)

All the above gave me great service, and I recommend them.
 
Last edited:
Tanker Pics!

Opposite side of the case from the mobo, added bar has text on it:
DSCN0001.JPG

Back of the case, - Tanker has 10 access slots!
DSCN0002.JPG


Four Noctua's on the four cpu's, using just the single fan - cool and quiet!
DSCN0003.JPG

Folding underway for several hours (Regular SMP work units) - 563 watts! :clap:
DSCN0005.JPG

Front right corner of the case, showing the big front fan washable filter. The front facing plate is off the case, still.
Big case, and a big mobo and power supply, and not much else!

DSCN0006_2.jpg
 
Last edited:
Awesome. Glad you posted the link for the board. I had given up checking them for stock and was close to resigning myself to the tyan at egg for over $800. I pulled the trigger at superbiz :clap:
 
Awesome. Glad you posted the link for the board. I had given up checking them for stock and was close to resigning myself to the tyan at egg for over $800. I pulled the trigger at superbiz :clap:

Gratz TC on being on your way to joining the "Collective" LOL.
 
* The Gold rated PSU, working in it's best (50%) area, is amazing. While folding with 64 threads, it's using just 565 Watts! Yes, the psu is expensive ($250-$290), but it will be a big money saver because it's so efficient. It's warranted for 7 years, and it's just (very quietly) loafing while the rig folds. I'll be surprised if it doesn't last twice that long.

NICE wattage. That's about the same wattage as my 6174's are pulling, with a Corsair AX750. What's your PPD on a 6903?
 
Awesome. Glad you posted the link for the board. I had given up checking them for stock and was close to resigning myself to the tyan at egg for over $800. I pulled the trigger at superbiz :clap:

Don't be afraid of wiredzone.com either. Just got my new board from them on Friday. They were quick and great with communication. More production to follow. :thup:
 
Thank you so much for your detailed parts, WITH LINKS!, list... Big help for this guy! I have my Taxes done, and intrest in my 3960x, and the 1st will be here soon for the other guys that owe me money! Hopefully a few more weeks...

I will definately look into your links for some good deals, and a special order from Superbiiz on that SuperMicro!
 
NICE wattage. That's about the same wattage as my 6174's are pulling, with a Corsair AX750. What's your PPD on a 6903?

Tanker is folding his first bigadv wu, now. Power has climbed to 605 Watts with this 6903 wu, and holding steady. Note that Tanker has NO case fans running at this time - only the 4 fans of the Noctua heatsinks (all 120mm running at ~1300 r.p.m). There will be at least one case fan running, (the large front case fan), after the wires get "tamed" a bit, and the case gets it's sides put on.

Unfortunately, I loaded Tanker up with a slower version of Ubuntu (11.04 Natty), so the times aren't great I'm sure:

15 mins 49 seconds per frame

This is unrealistic, since I've added no time for the upload (which will be no small amount) of the wu, but it gives you a general idea of Tanker's folding performance, on this wu:
TankerPerformance1.PNG

Wow! :cool:
 
Last edited:
Tested the 32 cores vs. 64 cores on Tanker. Just one wu, but it seems clear - it's slower.

This is the data from both instances FAHlog.txt, on the same 6903 wu.

Beginning 32 core test:
Code:
--- Opening Log file [January 23 06:44:39 UTC] 


# Linux SMP Console Edition ###################################################
###############################################################################

                       Folding@Home Client Version 6.34

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/adak/folding2
Executable: ./fah6
Arguments: -smp 32 -bigadv -verbosity 9 

[06:44:39] - Ask before connecting: No
[06:44:39] - User name: Adak (Team 32)
[06:44:39] - User ID: 58F09B651DDEBC12
[06:44:39] - Machine ID: 1
[06:44:39] 
[06:44:39] Loaded queue successfully.
[06:44:39] 
[06:44:39] - Autosending finished units... [January 23 06:44:39 UTC]
[06:44:39] + Processing work unit
[06:44:39] Trying to send all finished work units
[06:44:39] Core required: FahCore_a5.exe
[06:44:39] + No unsent completed units remaining.
[06:44:39] Core found.
[06:44:39] - Autosend completed
[06:44:39] Working on queue slot 01 [January 23 06:44:39 UTC]
[06:44:39] + Working ...
[06:44:39] - Calling './FahCore_a5.exe -dir work/ -nice 19 -suffix 01 -np 32 -checkpoint 30 -verbose -lifeline 4786 -version 634'

[06:44:39] 
[06:44:39] *------------------------------*
[06:44:39] Folding@Home Gromacs SMP Core
[06:44:39] Version 2.27 (Thu Feb 10 09:46:40 PST 2011)
[06:44:39] 
[06:44:39] Preparing to commence simulation
[06:44:39] - Looking at optimizations...
[06:44:39] - Files status OK
[06:44:46] - Expanded 57247623 -> 71846524 (decompressed 50.4 percent)
[06:44:46] Called DecompressByteArray: compressed_data_size=57247623 data_size=71846524, decompressed_data_size=71846524 diff=0
[06:44:47] - Digital signature verified
[06:44:47] 
[06:44:47] Project: 6903 (Run 6, Clone 17, Gen 44)
[06:44:47] 
[06:44:47] Assembly optimizations on if available.
[06:44:47] Entering M.D.
[06:44:53] Using Gromacs checkpoints
[06:45:00] Mapping NT from 32 to 32 
[06:45:36] Resuming from checkpoint
[06:45:59] Verified work/wudata_01.log
[06:46:00] Verified work/wudata_01.trr
[06:46:00] Verified work/wudata_01.xtc
[06:46:00] Verified work/wudata_01.edr
[06:46:02] Completed 94115 out of 250000 steps  (37%)
<<<< Begin Test >>>>
[06:53:44] Completed 95000 out of 250000 steps  (38%)
[07:15:02] Completed 97500 out of 250000 steps  (39%)
[07:36:43] Completed 100000 out of 250000 steps  (40%)
[07:58:18] Completed 102500 out of 250000 steps  (41%)
<<<< End Test >>>>

And now the 64 core's FAHlog.txt for the test:

Note: Please read the license agreement (fah6 -license). Further 
use of this software requires that you have read and accepted this agreement.

64 cores detected


--- Opening Log file [January 23 08:01:06 UTC] 


# Linux SMP Console Edition ###################################################
###############################################################################

                       Folding@Home Client Version 6.34

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /home/adak/folding
Executable: ./fah6
Arguments: -smp 64 -bigadv -verbosity 9 

[08:01:06] - Ask before connecting: No
[08:01:06] - User name: Adak (Team 32)
[08:01:06] - User ID: 58F09B651DDEBC12
[08:01:06] - Machine ID: 1
[08:01:06] 
[08:01:07] Loaded queue successfully.
[08:01:07] 
[08:01:07] - Autosending finished units... [January 23 08:01:07 UTC]
[08:01:07] + Processing work unit
[08:01:07] Trying to send all finished work units
[08:01:07] Core required: FahCore_a5.exe
[08:01:07] + No unsent completed units remaining.
[08:01:07] - Autosend completed
[08:01:07] Core found.
[08:01:07] Working on queue slot 01 [January 23 08:01:07 UTC]
[08:01:07] + Working ...
[08:01:07] - Calling './FahCore_a5.exe -dir work/ -nice 19 -suffix 01 -np 64 -checkpoint 30 -verbose -lifeline 4881 -version 634'

[08:01:07] 
[08:01:07] *------------------------------*
[08:01:07] Folding@Home Gromacs SMP Core
[08:01:07] Version 2.27 (Thu Feb 10 09:46:40 PST 2011)
[08:01:07] 
[08:01:07] Preparing to commence simulation
[08:01:07] - Looking at optimizations...
[08:01:07] - Files status OK
[08:01:15] - Expanded 57247623 -> 71846524 (decompressed 50.4 percent)
[08:01:15] Called DecompressByteArray: compressed_data_size=57247623 data_size=71846524, decompressed_data_size=71846524 diff=0
[08:01:15] - Digital signature verified
[08:01:15] 
[08:01:15] Project: 6903 (Run 6, Clone 17, Gen 44)
[08:01:15] 
[08:01:16] Assembly optimizations on if available.
[08:01:16] Entering M.D.
[08:01:22] Using Gromacs checkpoints
                         :-)  G  R  O  M  A  C  S  (-:

                   Groningen Machine for Chemical Simulation

                            :-)  VERSION 4.5.3  (-:

        Written by Emile Apol, Rossen Apostolov, Herman J.C. Berendsen,
      Aldert van Buuren, Pär Bjelkmar, Rudi van Drunen, Anton Feenstra, 
        Gerrit Groenhof, Peter Kasson, Per Larsson, Pieter Meulenhoff, 
           Teemu Murtola, Szilard Pall, Sander Pronk, Roland Schulz, 
                Michael Shirts, Alfons Sijbers, Peter Tieleman,

               Berk Hess, David van der Spoel, and Erik Lindahl.

       Copyright (c) 1991-2000, University of Groningen, The Netherlands.
            Copyright (c) 2001-2010, The GROMACS development team at
        Uppsala University & The Royal Institute of Technology, Sweden.
            check out http://www.gromacs.org for more information.


                               :-)  Gromacs  (-:

[08:01:28] Mapping NT from 64 to 64 
Reading file work/wudata_01.tpr, VERSION 4.5.4-dev-20110530-cc815 (single precision)
Starting 64 threads

Reading checkpoint file work/wudata_01.cpt generated: Sun Jan 22 22:19:37 2012


Making 2D domain decomposition 8 x 8 x 1
starting mdrun 'Overlay'
11250000 steps,  45000.0 ps (continuing from step 11094115,  44376.5 ps).
[08:02:05] Resuming from checkpoint
[08:03:52] Verified work/wudata_01.log
[08:03:52] Verified work/wudata_01.trr
[08:03:53] Verified work/wudata_01.xtc
[08:03:53] Verified work/wudata_01.edr
[08:03:54] Completed 94115 out of 250000 steps  (37%)

NOTE: Turning on dynamic load balancing
<<<< Begin Test >>>>
[08:09:47] Completed 95000 out of 250000 steps  (38%)
[08:26:01] Completed 97500 out of 250000 steps  (39%)
[08:42:23] Completed 100000 out of 250000 steps  (40%)
[08:59:57] Completed 102500 out of 250000 steps  (41%)
<<<< End Test >>>>

32 Cores Summary:

38-39%: 21:18
39-40%: 21:41
40-41%: 21:35
=============
Total: 64:34

64 Cores Summary:


38-39%: 16:14
39-40%: 16:22
40-41%: 17:34
=============
Total: 50:10

Using just 32 cores does decrease the power from 600 Watts to 563 Watts. That's the same power level as folding regular smp wu's - which also uses 32 cores, on Tanker.
 
Last edited:
Great build and Great PPD Adak. :) I still haven't secured a chassis for myself and I really like what you did with the Cubitek. In fact, I may go the same route... "spreader bar" and all. Let me ask you this... is there any possibility of putting the top case fans on the outside of the case in lieu of just removing them altogether?

Second, I think you should try The Kraken and see how that improves the frame times. My 4x6174 on Ubuntu 10.10 w/The Kraken runs p6903 @ just over 14 minutes, probably averaging ~14:07 for just at 500k PPD.
 
Great build and Great PPD Adak. :) I still haven't secured a chassis for myself and I really like what you did with the Cubitek. In fact, I may go the same route... "spreader bar" and all. Let me ask you this... is there any possibility of putting the top case fans on the outside of the case in lieu of just removing them altogether?

Second, I think you should try The Kraken and see how that improves the frame times. My 4x6174 on Ubuntu 10.10 w/The Kraken runs p6903 @ just over 14 minutes, probably averaging ~14:07 for just at 500k PPD.

Thanks, H, Right back at ya! I'm sure we could find a way to add the top case fans back on. It looks like the rear top fan will fit as is, inside the top of the case, if you wanted to put it there. The forward top case fan looks like it would have to go up above the top of the case, but maybe...

Let me try some idea's on that one, and get back to you.

Yes, the Kraken is on the to do list. The first work unit is being done for comparison purposes, only. Also, I installed the wrong (slow apparently), version of Ubuntu (11.04 Natty), and will be changing that to 10.04.

Your 6174 based folder may always be quicker than Tanker's 6272's though.
 
Neither top case fan will fit, as is. RAM blocks them at the back, and case header pins block them at the front, however there are two fixes:

1) By reversing the fan attachment brackets (two screws, self tapping), you can raise the height of the fan, as shown below. Originally, the fan brackets hold the fan just below the top of the case. This shows the extra height, by reversing the brackets on a fan:
DSCN0002.JPG

That leaves the fan problem solved, but now the top panel of the case, is blocked by the top of the fan. This is OK if you run the case "topless", but otherwise, not too good.

2) The top panel (aluminum) is perforated with hundreds of small holes on the entire back half of the panel. Both fans could simply by screwed through their mounting holes, (on the fan, or using the fan brackets), to the panel.

That's what Tuigi did with his 4p system in a HAF case, but he used a single larger fan, instead of two 120mm fans.
 
Back