• Welcome to Overclockers Forums! Join us to reply in threads, receive reduced ads, and to customize your site experience!

A guide to folding under Linux

Overclockers is supported by our readers. When you click a link to make a purchase, we may earn a commission. Learn More.

Sydney

Member
Joined
Sep 28, 2009
Location
Reykjavík, Iceland
Do not use 9.10, the GPU performance is awful when the CPU client is running, I am currently trying to figure it out. Use 9.04 for maximum performance
Works on Ubuntu 9.04 as of December 11th 2009
This guide will only work for Nvidia cards, 8x00 cards or newer.
I've heard about a wrapper for ATi cards, but seeing as I don't own one I can't really test it out.


This guide was done using Ubuntu 9.04 64-bit, should work without any modification on other Debian based distros. To get SMP folding on the CPU client, you must have a 64-bit system.

Start by updating your system
Code:
sudo apt-get update 
sudo apt-get dist-upgrade

Setting up the CPU Client:
Setting up the CPU client is pretty straight-forward
Start by opening a terminal (located in Applications > Accessories > Terminal)

Then type in or copy/paste this into the terminal.

Code:
sudo apt-get install ia32-libs
mkdir -p ~/folding
cd ~/folding
wget http://www.stanford.edu/group/pandegroup/folding/release/FAH6.24beta-Linux.tgz
tar xzf FAH6.24beta-Linux.tgz
Now you have created a folder in /home/yourname/ called folding, downloaded the linux [email protected] client and extracted it.

Now, to configure the client:

Code:
./fah6 -configonly
Enter your settings, and then you can run the client using the following command

Code:
./fah6 -smp -verbosity 9
If you are using an i7, use
Code:
./fah6 -smp 8 -verbosity 9
and furthermore, if you are running your i7 stable 24/7 you can use the -bigadv flag for a point bonus.

GPU client
This is where it gets tricky, because there is no Linux GPU client, only the Windows binaries. Therefore we will use Wine to run it.

But firstly, we need to install drivers and the CUDA toolkit.

Let's download the latest CUDA enabled driver.

Go to System > Administration > Hardware Drivers and select the 180 drivers, let them download and install.

Reboot

The next step is downloading the CUDA toolkit. Open a new terminal and type in:

Code:
wget http://developer.download.nvidia.com/compute/cuda/2_0/linux/toolkit/NVIDIA_CUDA_Toolkit_2.0_ubuntu7.10_x86.run
Then, within the same terminal type:
Code:
sudo sh NVIDIA_CUDA_Toolkit_2.0_ubuntu7.10_x86.run
Just hit enter to verify install path when asked. CUDA has been set up and you are ready to go.

In case you were wondering, the reason that I'm using the 7.10 toolkit is because the GPU2 wrapper was written for that version of the toolkit, newer toolkits do not work.

Then, we need to link the toolkit.
Code:
sudo sh -c "echo '/usr/local/cuda/lib' > /etc/ld.so.conf.d/cuda.conf"
sudo ldconfig
let's grab wine

The following command will download the latest stable binary release of wine

Code:
sudo apt-get install wine

If you just installed Wine, you might want to create the directory structure before starting up the GPU client, do so by simply running notepad.
Code:
wine notepad
You need to download the CUDA wrapper, so the windows binaries can communicate with the Linux driver.
Code:
wget http://www.gpu2.twomurs.com/wrapper2ndgen/2.1/cudart.dll.so -O ~/.wine/drive_c/windows/system32/cudart.dll
ln -s ~/.wine/drive_c/windows/system32/cudart.dll ~/.wine/drive_c/windows/system32/nvcuda.dll
then run

Code:
ldd ~/.wine/drive_c/windows/system32/cudart.dll
the result should be something like this

Code:
[email protected]:~$ ldd /usr/local/lib/wine/nvcuda.dll.so
	linux-gate.so.1 =>  (0xf7fae000)
	libcudart.so.2 => /usr/local/cuda/lib/libcudart.so.2 (0xf7f34000)
	libwine.so.1 => /usr/local/lib/libwine.so.1 (0xf7dfd000)
	libm.so.6 => /lib32/libm.so.6 (0xf7dd6000)
	libc.so.6 => /lib32/libc.so.6 (0xf7c78000)
	libdl.so.2 => /lib32/libdl.so.2 (0xf7c74000)
	libpthread.so.0 => /lib32/libpthread.so.0 (0xf7c5b000)
	librt.so.1 => /lib32/librt.so.1 (0xf7c52000)
	libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf7b64000)
	libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf7b54000)
	/lib/ld-linux.so.2 (0xf7faf000)
If you see any not founds, it probably has something to do with the CUDA toolkit, make sure you installed 32-bit toolkit, not the 64-bit, even though you are running a 64-bit kernel.


Hard part is over, now it's just setting up the client. Start by downloading it. To do this open a new terminal and type this:

Code:
mkdir ~/gpu0
cd ~/gpu0
wget http://www.stanford.edu/group/pandegroup/folding/release/[email protected]_XP-623.zip
unzip [email protected]_XP-623.zip
Then run the client by typing

Code:
nice -n 19 wine [email protected] -forcegpu nvidia_g80
(even if you have a newer card than a G80, this is the flag to be given)

After configuring it, your client should be folding away.


Dual GPU Folding
If you have more than one GPU, you can use them both at the same time, just as you would do in Windows.
In terminal:
Code:
mkdir ~/gpu1/
cp ~/gpu0/[email protected]_XP-623.zip ~/gpu1/
cd ~/gpu1/
unzip [email protected]_XP-623.zip
Then run this client by typing
Code:
nice -n 19 wine [email protected] -forcegpu nvidia_g80 -gpu 1
And put machineid as 3

Scripts for running the clients.

I don't like to have a lot of programs start up with my OS, so I have a simple script to start my clients.
Run these commands to create scripts for every client (Users with only one GPU need not do the last command)
Code:
echo -e "cd ~/folding/\n./fah6 -smp -verbosity 9 -advmethods" > ~/fahcpu
Code:
echo -e "cd ~/gpu0/\nnice -n 19 wine [email protected] -forcegpu nvidia_g80 -verbosity 9 -advmethods -gpu 0" > ~/fahgpu0
Code:
echo -e "cd ~/gpu1/\nnice -n 19 wine [email protected] -forcegpu nvidia_g80 -verbosity 9 -advmethods -gpu 1" > ~/fahgpu1
Code:
echo -e "gnome-terminal -x ./fahcpu\ngnome-terminal -x ./fahgpu0\ngnome-terminal -x ./fahgpu1" > ~/fah
Code:
chmod +x fahcpu fahgpu0 fahgpu1 fah
Now you can run all your clients by typing ./fah

Running the clients on startup
Go to System > Preferences > Startup Applications
Now add the clients you want to run

CPU Client:
Name: cpu
Command: ~/./fahcpu
Comment: cpu client

GPU0 Client:
Name: gpu0
Command: ~/./fahgpu0
Comment: GPU0 client

GPU1 Client:
Name: gpu1
Command: ~/./fahgpu1
Comment: GPU1 client

Next time you reboot your system, your folding clients will be running in the background.]
However, as I stated earlier, I personally prefer running the ./fah script myself on each startup so I can monitor the output of each client, instead of having them run in the background.

Overclocking your GPU in Linux
If you are doing this, I assume you have already done so in Windows and know the safe clocks for your card(s). To overclock your video card in Linux you have to enable 'coolbits' in your nvidia control panel. To do this we will have to edit the xorg configuration
Code:
sudo gedit /etc/X11/xorg.conf
And then add 'Option "CoolBits" "1"' under 'Section: "Device"' like so:
Section "Device"
Identifier "Configured Video Device"
Driver "nvidia"
Option "NoLogo" "True"
Option "Coolbits" "1"
EndSection
Save the file and quit. Now you will have access to the CoolBits menu under your nvidia control panel
Access the control panel by going to System > Administration > NVIDIA X Server Settings, or simply typing "nvidia-settings" in a terminal. You should now see a 'Clock Frequencies' section where you can set your clocks. This has to be done every time you boot up Linux though, so the most convenient way would be to flash your video card with a BIOS with increased clock settings. Doing that is risky and is NOT RECOMMENDED UNLESS YOU KNOW EXACTLY WHAT YOU ARE DOING.

Monitoring
For monitoring I will use harlam's monitoring program, HFM.net. It was written in .net so we need to use Mono to run it under Linux. In Ubuntu 9.04 the mono in the repositories is out of date and will not support HFM.net. We need to add another repository to get mono 2.4.

Edit your package sources
Code:
sudo gedit /etc/apt/sources.list
And at the bottom of the file add
Code:
deb http://ppa.launchpad.net/directhex/monoxide/ubuntu jaunty main 
deb-src http://ppa.launchpad.net/directhex/monoxide/ubuntu jaunty main

Add the GPG key
Code:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 23DC003A

Update the repositories
Code:
sudo apt-get update

Then install the winforms libraries (will also install all the other dependencies needed to run HFM.net)
Code:
sudo apt-get install libmono-winforms2.0-cil

Now download the binaries from here and extract them to a folder of your choice. I used ~/hfm/.

Then you can run it from a terminal using
Code:
mono ~/hfm/HFM.exe

If you are experiencing missing text in HFM and are using an intel i8** or i9** video card on Ubuntu 9.10, then try this fix.
Add a new repository
Code:
sudo add-apt-repository ppa:xorg-edgers/ppa
sudo apt-get update

Then install the latest xserver-xorg-video-intel package
Code:
sudo apt-get install xserver-xorg-video-intel

Then go to System > Administration > Software Sources and under 'other software' uncheck xorg-edgers to avoid your system from unnecessarily updating more of it's packages to unstable versions.

This guide was based on information from:
[email protected] Linux SMP Guide
GPU2 Linux
NVIDIA GPU2 Linux/WINE Headless Install Guide

Feel free to improve this guide and redistribute it, giving me credit would be greatly appreciated

If you have any questions, issues or anything regarding [email protected] under Linux, just ask in this thread and I will try to help.

Troubleshooting
What to do if my xserver doesn't start?
 
Last edited:

Shelnutt2

Overclockers Team Content Editor
Joined
Jun 17, 2005
Location
/home/
You need to download the CUDA wrapper, or else the GPU client is likely to use up 100% CPU and simply freeze everything you are doing.

Only cause the CUDA wrapper started out as my baby do I care, but thats not quite right. The wrapper is the essential part of allowing for the cuda.dll calls you be redirected to the libcudart.so library. Without the wrapper the GPU client would not work. It is true though that a loop timer was built into the wrapper to cut down on CPU usage. Due to the nature of the wine, the wrapper and the linux drivers, fah ended up making more calls than were needed, like if you tried to open the same webpage in 10 different tabs. It's a waste of resources. The loop timer, put a delay in between opening the new tabs.
 
OP
Sydney

Sydney

Member
Joined
Sep 28, 2009
Location
Reykjavík, Iceland
Only cause the CUDA wrapper started out as my baby do I care, but thats not quite right. The wrapper is the essential part of allowing for the cuda.dll calls you be redirected to the libcudart.so library. Without the wrapper the GPU client would not work. It is true though that a loop timer was built into the wrapper to cut down on CPU usage. Due to the nature of the wine, the wrapper and the linux drivers, fah ended up making more calls than were needed, like if you tried to open the same webpage in 10 different tabs. It's a waste of resources. The loop timer, put a delay in between opening the new tabs.
I see, I suppose I was reading the chancelog between first and second generation wrapper instead of what it essentially does.

Fixing now.
 

Adak

Senior Member
Joined
Jan 9, 2006
That's a great guide, Sydney.

I will have to change my bookmark over to this one, now.

Thanks! :cool:
 

harlam357

Senior Fold-a-holic
Joined
Sep 22, 2004
We need to get this into our Install Sticky. However, I'll link this thread from the Intro and the FAQ in the meantime. :)

Nice Guide Sydney!!! :thup: ...and :welcome: to Team 32! :beer: New guide writers stock the fridge. :beer:
 

Shelnutt2

Overclockers Team Content Editor
Joined
Jun 17, 2005
Location
/home/
Sydney would you be willing to make a plug for HFM instead of fahmon?

Also I think I came off wrong. I didn't mean for you to have to put, "most essential part". I was just trying to inform you on what the wrapper actually did. You can word it however you want, but don't feel like you have to say it's the most essential part. I mean in reality is there really one part thats more essential than the others? It takes every step and every piece of software/hardware for it to work!
 
OP
Sydney

Sydney

Member
Joined
Sep 28, 2009
Location
Reykjavík, Iceland
Sydney would you be willing to make a plug for HFM instead of fahmon?

Also I think I came off wrong. I didn't mean for you to have to put, "most essential part". I was just trying to inform you on what the wrapper actually did. You can word it however you want, but don't feel like you have to say it's the most essential part. I mean in reality is there really one part thats more essential than the others? It takes every step and every piece of software/hardware for it to work!
Sure, I'll check it out tomorrow, fahmon isn't really all that great, since it tends to get buggy when reading GPU clients in Linux.

Wording has never been my strongest side, I'm a bit tired right now, being 2 AM and all, I'll have a look at it tomorrow.
 

harlam357

Senior Fold-a-holic
Joined
Sep 22, 2004
I wasn't going to ask for an HFM plug b/c it's almost as difficult for the novice to get the right version of Mono on their box as it is to learn to compile FahMon. :D

However, if one has Mono 2.4+... it should be pretty "plug-and-chug" at this point. :)

btw- the guide is up in both the Intro and FAQ if you want to check it out.
 

augie1111

Member
Joined
Sep 15, 2008
Location
Mirabel, Quebec, Canada
I wasn't going to ask for an HFM plug b/c it's almost as difficult for the novice to get the right version of Mono on their box as it is to learn to compile FahMon. :D

However, if one has Mono 2.4+... it should be pretty "plug-and-chug" at this point. :)

btw- the guide is up in both the Intro and FAQ if you want to check it out.

Hmm, as Homer would say 'Mono". :D Hey, as a *nix N00b, I can attest to the excellence of this guide, if followed 'exactly' with the releases shown. My GTX260 is just flying along and I haven't even OC'd it yet! Besides, I now have an SMP running too as an extra bonus to the GPU. :thup:
 

augie1111

Member
Joined
Sep 15, 2008
Location
Mirabel, Quebec, Canada
OK, so the GTX 260 is getting 5.6 PPD on a steady diet of 353 pointers ATM, as far as I can tell. I have run through the OC guide but it's Martian to me in the nVidia XServer. What would be the *nix equivalent to 'shader clock' which is all that I ever really pushed up on in Windows getting the card up to 7K PPD on average? Thanks.
 
OP
Sydney

Sydney

Member
Joined
Sep 28, 2009
Location
Reykjavík, Iceland
You can't modify shader clocks in Linux, you can only edit core and memory clocks, but shader clocks are linked to core clocks, so if you increase them, you increase shaders.

I just flashed my cards with the clocks i was happy with.
 

NV

Member
Joined
Sep 18, 2009
Code:
:~$ ldd ~/.wine/drive_c/windows/system32/cudart.dllldd: warning: you do not have execution permission for `/home/adam/.wine/drive_c/windows/system32/cudart.dll'
	linux-gate.so.1 =>  (0xf7fb8000)
	libcudart.so.2 => [COLOR="Red"]not found[/COLOR]
	libwine.so.1 => /usr/lib32/libwine.so.1 (0xf7e52000)
	libm.so.6 => /lib32/libm.so.6 (0xf7e2b000)
	libc.so.6 => /lib32/libc.so.6 (0xf7cc8000)
	libdl.so.2 => /lib32/libdl.so.2 (0xf7cc4000)
	/lib/ld-linux.so.2 (0xf7fb9000)

I copied and pasted the wget for the cuda toolkit that you posted. Yes, running 64bit ubuntu, but the wget seems to link to the 32-bit version. Not sure how to check what was installed though, I'm basically blindly following your guide because this is the only straightforward and current guide I've been able to find.

On that note, thanks for the straightforward and current guide :cool:
 
OP
Sydney

Sydney

Member
Joined
Sep 28, 2009
Location
Reykjavík, Iceland
Did you get any errors when installing the CUDA toolkit? And did you install it to the default path? (/usr/local/cuda)

Are you sure you executed these two commands?
sudo sh -c "echo '/usr/local/cuda/lib' > /etc/ld.so.conf.d/cuda.conf"
sudo ldconfig

Do you get any errors when executing
sudo sh -c "echo '/usr/local/cuda/lib' > /etc/ld.so.conf.d/cuda.conf"
 

NV

Member
Joined
Sep 18, 2009
Hmm, well I typed that command into the terminal last time. I guess I could have punched it in wrong. Copied and pasted it now, and everything worked smoothly. I now have a more extensive output and nothing not found.

Should be up and running with the GPU client in a few minutes then, I guess. Thanks!

Edit:
Working now :) Thanks for the awesome guide!
 
OP
Sydney

Sydney

Member
Joined
Sep 28, 2009
Location
Reykjavík, Iceland
Hmm, well I typed that command into the terminal last time. I guess I could have punched it in wrong. Copied and pasted it now, and everything worked smoothly. I now have a more extensive output and nothing not found.

Should be up and running with the GPU client in a few minutes then, I guess. Thanks!
Perhaps you forgot ldconfig, it updates the system with newly created dynamic links.

Anyway, let me know how it goes :)