A guide to folding under Linux

[Sydney]

Do not use 9.10, the GPU performance is awful when the CPU client is running, I am currently trying to figure it out. Use 9.04 for maximum performance
Works on Ubuntu 9.04 as of December 11th 2009
This guide will only work for Nvidia cards, 8x00 cards or newer.
I've heard about a wrapper for ATi cards, but seeing as I don't own one I can't really test it out.

This guide was done using Ubuntu 9.04 64-bit, should work without any modification on other Debian based distros. To get SMP folding on the CPU client, you must have a 64-bit system.

Start by updating your system

sudo apt-get update 
sudo apt-get dist-upgrade

Setting up the CPU Client:
Setting up the CPU client is pretty straight-forward
Start by opening a terminal (located in Applications > Accessories > Terminal)

Then type in or copy/paste this into the terminal.

sudo apt-get install ia32-libs
mkdir -p ~/folding
cd ~/folding
wget http://www.stanford.edu/group/pandegroup/folding/release/FAH6.24beta-Linux.tgz
tar xzf FAH6.24beta-Linux.tgz

Now you have created a folder in /home/yourname/ called folding, downloaded the linux F@H client and extracted it.

Now, to configure the client:

./fah6 -configonly

Enter your settings, and then you can run the client using the following command

./fah6 -smp -verbosity 9

If you are using an i7, use

./fah6 -smp 8 -verbosity 9

and furthermore, if you are running your i7 stable 24/7 you can use the -bigadv flag for a point bonus.

GPU client
This is where it gets tricky, because there is no Linux GPU client, only the Windows binaries. Therefore we will use Wine to run it.

But firstly, we need to install drivers and the CUDA toolkit.

Let's download the latest CUDA enabled driver.

Go to System > Administration > Hardware Drivers and select the 180 drivers, let them download and install.

Reboot

The next step is downloading the CUDA toolkit. Open a new terminal and type in:

wget http://developer.download.nvidia.com/compute/cuda/2_0/linux/toolkit/NVIDIA_CUDA_Toolkit_2.0_ubuntu7.10_x86.run

Then, within the same terminal type:

sudo sh NVIDIA_CUDA_Toolkit_2.0_ubuntu7.10_x86.run

Just hit enter to verify install path when asked. CUDA has been set up and you are ready to go.

In case you were wondering, the reason that I'm using the 7.10 toolkit is because the GPU2 wrapper was written for that version of the toolkit, newer toolkits do not work.

Then, we need to link the toolkit.

sudo sh -c "echo '/usr/local/cuda/lib' > /etc/ld.so.conf.d/cuda.conf"
sudo ldconfig

let's grab wine

The following command will download the latest stable binary release of wine

sudo apt-get install wine

If you just installed Wine, you might want to create the directory structure before starting up the GPU client, do so by simply running notepad.

wine notepad

You need to download the CUDA wrapper, so the windows binaries can communicate with the Linux driver.

wget http://www.gpu2.twomurs.com/wrapper2ndgen/2.1/cudart.dll.so -O ~/.wine/drive_c/windows/system32/cudart.dll
ln -s ~/.wine/drive_c/windows/system32/cudart.dll ~/.wine/drive_c/windows/system32/nvcuda.dll

then run

ldd ~/.wine/drive_c/windows/system32/cudart.dll

the result should be something like this

user@computer:~$ ldd /usr/local/lib/wine/nvcuda.dll.so
	linux-gate.so.1 =>  (0xf7fae000)
	libcudart.so.2 => /usr/local/cuda/lib/libcudart.so.2 (0xf7f34000)
	libwine.so.1 => /usr/local/lib/libwine.so.1 (0xf7dfd000)
	libm.so.6 => /lib32/libm.so.6 (0xf7dd6000)
	libc.so.6 => /lib32/libc.so.6 (0xf7c78000)
	libdl.so.2 => /lib32/libdl.so.2 (0xf7c74000)
	libpthread.so.0 => /lib32/libpthread.so.0 (0xf7c5b000)
	librt.so.1 => /lib32/librt.so.1 (0xf7c52000)
	libstdc++.so.6 => /usr/lib32/libstdc++.so.6 (0xf7b64000)
	libgcc_s.so.1 => /usr/lib32/libgcc_s.so.1 (0xf7b54000)
	/lib/ld-linux.so.2 (0xf7faf000)

If you see any not founds, it probably has something to do with the CUDA toolkit, make sure you installed 32-bit toolkit, not the 64-bit, even though you are running a 64-bit kernel.


Hard part is over, now it's just setting up the client. Start by downloading it. To do this open a new terminal and type this:

mkdir ~/gpu0
cd ~/gpu0
wget http://www.stanford.edu/group/pandegroup/folding/release/Folding@home-Win32-GPU_XP-623.zip
unzip Folding@home-Win32-GPU_XP-623.zip

Then run the client by typing

nice -n 19 wine [email protected] -forcegpu nvidia_g80

(even if you have a newer card than a G80, this is the flag to be given)

After configuring it, your client should be folding away.


Dual GPU Folding
If you have more than one GPU, you can use them both at the same time, just as you would do in Windows.
In terminal:

mkdir ~/gpu1/
cp ~/gpu0/Folding@home-Win32-GPU_XP-623.zip ~/gpu1/
cd ~/gpu1/
unzip Folding@home-Win32-GPU_XP-623.zip

Then run this client by typing

nice -n 19 wine [email protected] -forcegpu nvidia_g80 -gpu 1

And put machineid as 3

Scripts for running the clients.

I don't like to have a lot of programs start up with my OS, so I have a simple script to start my clients.
Run these commands to create scripts for every client (Users with only one GPU need not do the last command)

echo -e "cd ~/folding/\n./fah6 -smp -verbosity 9 -advmethods" > ~/fahcpu

echo -e "cd ~/gpu0/\nnice -n 19 wine [email protected] -forcegpu nvidia_g80 -verbosity 9 -advmethods -gpu 0" > ~/fahgpu0

echo -e "cd ~/gpu1/\nnice -n 19 wine [email protected] -forcegpu nvidia_g80 -verbosity 9 -advmethods -gpu 1" > ~/fahgpu1

echo -e "gnome-terminal -x ./fahcpu\ngnome-terminal -x ./fahgpu0\ngnome-terminal -x ./fahgpu1" > ~/fah

chmod +x fahcpu fahgpu0 fahgpu1 fah

Now you can run all your clients by typing ./fah

Running the clients on startup
Go to System > Preferences > Startup Applications
Now add the clients you want to run

CPU Client:
Name: cpu
Command: ~/./fahcpu
Comment: cpu client

GPU0 Client:
Name: gpu0
Command: ~/./fahgpu0
Comment: GPU0 client

GPU1 Client:
Name: gpu1
Command: ~/./fahgpu1
Comment: GPU1 client

Next time you reboot your system, your folding clients will be running in the background.]
However, as I stated earlier, I personally prefer running the ./fah script myself on each startup so I can monitor the output of each client, instead of having them run in the background.

Overclocking your GPU in Linux
If you are doing this, I assume you have already done so in Windows and know the safe clocks for your card(s). To overclock your video card in Linux you have to enable 'coolbits' in your nvidia control panel. To do this we will have to edit the xorg configuration

sudo gedit /etc/X11/xorg.conf

And then add 'Option "CoolBits" "1"' under 'Section: "Device"' like so:

Section "Device"
Identifier "Configured Video Device"
Driver "nvidia"
Option "NoLogo" "True"
Option "Coolbits" "1"
EndSection

Save the file and quit. Now you will have access to the CoolBits menu under your nvidia control panel
Access the control panel by going to System > Administration > NVIDIA X Server Settings, or simply typing "nvidia-settings" in a terminal. You should now see a 'Clock Frequencies' section where you can set your clocks. This has to be done every time you boot up Linux though, so the most convenient way would be to flash your video card with a BIOS with increased clock settings. Doing that is risky and is NOT RECOMMENDED UNLESS YOU KNOW EXACTLY WHAT YOU ARE DOING.

Monitoring
For monitoring I will use harlam's monitoring program, HFM.net. It was written in .net so we need to use Mono to run it under Linux. In Ubuntu 9.04 the mono in the repositories is out of date and will not support HFM.net. We need to add another repository to get mono 2.4.

Edit your package sources

sudo gedit /etc/apt/sources.list

And at the bottom of the file add

deb http://ppa.launchpad.net/directhex/monoxide/ubuntu jaunty main 
deb-src http://ppa.launchpad.net/directhex/monoxide/ubuntu jaunty main

Add the GPG key

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 23DC003A

Update the repositories

sudo apt-get update

Then install the winforms libraries (will also install all the other dependencies needed to run HFM.net)

sudo apt-get install libmono-winforms2.0-cil

Now download the binaries from here and extract them to a folder of your choice. I used ~/hfm/.

Then you can run it from a terminal using

mono ~/hfm/HFM.exe

If you are experiencing missing text in HFM and are using an intel i8** or i9** video card on Ubuntu 9.10, then try this fix.
Add a new repository

sudo add-apt-repository ppa:xorg-edgers/ppa
sudo apt-get update

Then install the latest xserver-xorg-video-intel package

sudo apt-get install xserver-xorg-video-intel

Then go to System > Administration > Software Sources and under 'other software' uncheck xorg-edgers to avoid your system from unnecessarily updating more of it's packages to unstable versions.

This guide was based on information from:
Folding@Home Linux SMP Guide
GPU2 Linux
NVIDIA GPU2 Linux/WINE Headless Install Guide

Feel free to improve this guide and redistribute it, giving me credit would be greatly appreciated

If you have any questions, issues or anything regarding F@H under Linux, just ask in this thread and I will try to help.

Troubleshooting
What to do if my xserver doesn't start?

 

[sneakysnowman]

I love this guide, helped me set up countless PCs

 

[Shelnutt2]

You need to download the CUDA wrapper, or else the GPU client is likely to use up 100% CPU and simply freeze everything you are doing.

Only cause the CUDA wrapper started out as my baby do I care, but thats not quite right. The wrapper is the essential part of allowing for the cuda.dll calls you be redirected to the libcudart.so library. Without the wrapper the GPU client would not work. It is true though that a loop timer was built into the wrapper to cut down on CPU usage. Due to the nature of the wine, the wrapper and the linux drivers, fah ended up making more calls than were needed, like if you tried to open the same webpage in 10 different tabs. It's a waste of resources. The loop timer, put a delay in between opening the new tabs.

 

[Sydney]

Only cause the CUDA wrapper started out as my baby do I care, but thats not quite right. The wrapper is the essential part of allowing for the cuda.dll calls you be redirected to the libcudart.so library. Without the wrapper the GPU client would not work. It is true though that a loop timer was built into the wrapper to cut down on CPU usage. Due to the nature of the wine, the wrapper and the linux drivers, fah ended up making more calls than were needed, like if you tried to open the same webpage in 10 different tabs. It's a waste of resources. The loop timer, put a delay in between opening the new tabs.

I see, I suppose I was reading the chancelog between first and second generation wrapper instead of what it essentially does.

Fixing now.

 

[Adak]

That's a great guide, Sydney.

I will have to change my bookmark over to this one, now.

Thanks!

 

[harlam357]

We need to get this into our Install Sticky. However, I'll link this thread from the Intro and the FAQ in the meantime.

Nice Guide Sydney!!! ...and to Team 32! New guide writers stock the fridge.

 

[Shelnutt2]

Sydney would you be willing to make a plug for HFM instead of fahmon?

Also I think I came off wrong. I didn't mean for you to have to put, "most essential part". I was just trying to inform you on what the wrapper actually did. You can word it however you want, but don't feel like you have to say it's the most essential part. I mean in reality is there really one part thats more essential than the others? It takes every step and every piece of software/hardware for it to work!

 

[Sydney]

Sydney would you be willing to make a plug for HFM instead of fahmon?

Also I think I came off wrong. I didn't mean for you to have to put, "most essential part". I was just trying to inform you on what the wrapper actually did. You can word it however you want, but don't feel like you have to say it's the most essential part. I mean in reality is there really one part thats more essential than the others? It takes every step and every piece of software/hardware for it to work!

Sure, I'll check it out tomorrow, fahmon isn't really all that great, since it tends to get buggy when reading GPU clients in Linux.

Wording has never been my strongest side, I'm a bit tired right now, being 2 AM and all, I'll have a look at it tomorrow.

 

[harlam357]

I wasn't going to ask for an HFM plug b/c it's almost as difficult for the novice to get the right version of Mono on their box as it is to learn to compile FahMon.

However, if one has Mono 2.4+... it should be pretty "plug-and-chug" at this point.

btw- the guide is up in both the Intro and FAQ if you want to check it out.

 

[augie1111]

I wasn't going to ask for an HFM plug b/c it's almost as difficult for the novice to get the right version of Mono on their box as it is to learn to compile FahMon.

However, if one has Mono 2.4+... it should be pretty "plug-and-chug" at this point.

btw- the guide is up in both the Intro and FAQ if you want to check it out.

Hmm, as Homer would say 'Mono". Hey, as a *nix N00b, I can attest to the excellence of this guide, if followed 'exactly' with the releases shown. My GTX260 is just flying along and I haven't even OC'd it yet! Besides, I now have an SMP running too as an extra bonus to the GPU.

 

[augie1111]

OK, so the GTX 260 is getting 5.6 PPD on a steady diet of 353 pointers ATM, as far as I can tell. I have run through the OC guide but it's Martian to me in the nVidia XServer. What would be the *nix equivalent to 'shader clock' which is all that I ever really pushed up on in Windows getting the card up to 7K PPD on average? Thanks.

 

[Sydney]

You can't modify shader clocks in Linux, you can only edit core and memory clocks, but shader clocks are linked to core clocks, so if you increase them, you increase shaders.

I just flashed my cards with the clocks i was happy with.

 

[cadman420]

This guide is great.... Thanks

 

[NV]

:~$ ldd ~/.wine/drive_c/windows/system32/cudart.dllldd: warning: you do not have execution permission for `/home/adam/.wine/drive_c/windows/system32/cudart.dll'
	linux-gate.so.1 =>  (0xf7fb8000)
	libcudart.so.2 => [COLOR="Red"]not found[/COLOR]
	libwine.so.1 => /usr/lib32/libwine.so.1 (0xf7e52000)
	libm.so.6 => /lib32/libm.so.6 (0xf7e2b000)
	libc.so.6 => /lib32/libc.so.6 (0xf7cc8000)
	libdl.so.2 => /lib32/libdl.so.2 (0xf7cc4000)
	/lib/ld-linux.so.2 (0xf7fb9000)

I copied and pasted the wget for the cuda toolkit that you posted. Yes, running 64bit ubuntu, but the wget seems to link to the 32-bit version. Not sure how to check what was installed though, I'm basically blindly following your guide because this is the only straightforward and current guide I've been able to find.

On that note, thanks for the straightforward and current guide

 

[Sydney]

Did you get any errors when installing the CUDA toolkit? And did you install it to the default path? (/usr/local/cuda)

Are you sure you executed these two commands?
sudo sh -c "echo '/usr/local/cuda/lib' > /etc/ld.so.conf.d/cuda.conf"
sudo ldconfig

Do you get any errors when executing
sudo sh -c "echo '/usr/local/cuda/lib' > /etc/ld.so.conf.d/cuda.conf"

 

[NV]

Hmm, well I typed that command into the terminal last time. I guess I could have punched it in wrong. Copied and pasted it now, and everything worked smoothly. I now have a more extensive output and nothing not found.

Should be up and running with the GPU client in a few minutes then, I guess. Thanks!

Edit:
Working now Thanks for the awesome guide!

 

[Sydney]

Hmm, well I typed that command into the terminal last time. I guess I could have punched it in wrong. Copied and pasted it now, and everything worked smoothly. I now have a more extensive output and nothing not found.

Should be up and running with the GPU client in a few minutes then, I guess. Thanks!

Perhaps you forgot ldconfig, it updates the system with newly created dynamic links.

Anyway, let me know how it goes

 

[Sydney]

Updated with HFM.net instead of fahmon.

 

[sneakysnowman]

Hooray

 

[harlam357]

Much obliged Sydney.

My only suggestion is to change the download link to send folks to the front HFM.NET Google Code Page. That way when I release another version the link won't have to be updated.

http://code.google.com/p/hfm-net/