Multi VM SMP Guide

[Cluster]

This guide is for anyone running dual or quad-core setups that need windows as their main OS, but want more PPD than the windows SMP client can provide. The reason for this is that the linux client is more effecient(even through a VM) and also the linux client has access to A2 WUs which seem to provide much better throughput and thus more PPD.

Before i begin i'd like to make a couple of notes. If you can at all, it is better to run linux as your main OS as you remove the overhead of the VM, obviously getting even better output. Now results may vary with systems and time, after going through this myself, i went from 1100ppd to 3-4k ppd, not a trivial increase. One more thing, make sure you have the ram for this. Each of my VMs uses between 550-650mb of ram.

Also, while this guide uses Ubuntu 8.04, you can use any linux distro. Ubuntu is not the lightest choice, but it is the simplest to get setup on. For those that wont to setup a VM in order to clone across multiple systems, a slimmer Linux install might be better as you can get away with much less than a 5gb disk to install the OS on.

System requirements
1.5GB of Ram
15GB HDD Space (using Ubuntu 8.04)
All your cpu cycles!

The basic process goes something like this :

1. 
   [*] Download VmWare and Ubuntu
   [*][*]Install VmWare
   [*][*][*]Install Ubuntu
   [*][*][*][*]Configure Ubuntu and download F@H
   [*][*][*][*][*]Install VmWare Toolso
   [*][*][*][*][*][*]Copy the VM
   [*][*][*][*][*][*][*]Configure the client and Fold Fold Fold!
   

The Stuff You Need

Both VmWare and Ubuntu are big downloads. VmWare is a little over 100mb, Ubuntu is around 700mb. Will give you lots of time to skim through the guide once or twice.

1. Vmware Server 150MB
   
   A Note for Vista64 users. If you try to install vmware, it is going to complain about unsigned drivers. See Appendix I for details.
   
   For those who might wish to take the leap to guinea pig status, VmWare Server 2.0 is the latest offering. It uses a Web UI instead of a regular GUI. I haven't been successful at getting the web ui to load, probably because i neuter my windows installs *shrug*
   
   
2. Ubuntu Download [700MB]
   !!! Remember to choose 64-bit !!!
   
   It does not matter if you use a 64-bit Host OS or not. Its all the same if your on 32-bit windows, but the guest linux OS needs to be 64-bit. Also it was reported there was some issue with kernel version supplied in Ubuntu 8.10. Until i hear different i'll recommend using 8.04-64bit, or until a new Ubuntu release is available to test on.

What To Do With The Stuff

1. Install VmWare Server
   
   Pretty straightforward, just install with the rest of your software. Vista64 users read Appendix I before doing this or windows will crap on you.
   
   
2. Create directories for VMs
   
   You'll want to create two directories for each VM. I named mine C:\FAH\SMP1 and C:\FAH\SMP2.
   
   
3. Configure your VM
   
   Start up the VmWare Server Console, localhost should be highlighted, select connect, then create a new virtual machine.
   
   Note: If when you boot up windows, and open the console to start your VMs, the localhost option is not shown, this means that the services have not started yet. It takes them a few seconds, close the console and try again in a minute.
   
   
   • Select Custom install
     
   • The type of Guest OS is Linux -> Ubuntu-64bit
     
   • Name the VM (SMP1) and select the directory you created earlier (C:\FAH\SMP1)
     
   • Leaving access rights and startup/shutdown at default is fine.
     
   • Select 2 Processors
     
   • Set the amount of ram for the VM (Min: 640, Recommended: 768)
     
   • Select Bridged Networking.
     
   • Create a 5GB Virtual Disk - Allocate all disk space now
     
   • Name the disk whatever you like and hit finish.
     
   • Select your VM from the Tabs at the top if its not already. In the main window below Start this VM, click Edit this VMs settings. We want to configure a CD-Rom that points to our Ubuntu ISO. Click CD-Rom on the left, and on the right there will be an option to set the drive to read an ISO image. Select the ISO option and point it to your Ubuntu ISO.
     
   • Start the VM and install Ubuntu
     
     For the most part you should be able to select the default install setup. Selecting timezone/keyboard/language as needed.
   
   
4. Configure Ubuntu
   
   Once your at the Ubuntu desktop, feel free to change your desktop, some people aren't to partial of the stork
   
   
   • Install the 32-bit math libs for F@H
     
     

      > sudo apt-get install ia32-libs

   • Open FireFox (you should see it on the taskbar at the top) and make your way to the Stanford F@H Page, download the linux client. You should see a tarball (.tar.gz file) on your desktop.
     
     
   • Double click the archive to open the archive manager, click extract, and choose a location. I put mine in /home/[username]/FAH. Do not start the folding client until you have finished and have the second VM copied. They need to be configured separately.
     
     
   • I highly recommend that you install VmWare Tools at this time. Amongst other things, it adds in time synchronization between host/guest OSes. See Appendix III for the install of this.
     
     
   • If you want to setup samba, which will allow you to share your FAH directory and make it visible to FahMon, see Appendix IV for the samba setup.

Copying Your Stuff

Now that the hard part is over with, we're almost done, the last step is to make a copy of your new VM.

1. Stop the VM you have running already.
   
   
2. Go to the directory where you created the VM and copy all of the files over to the other directory you created. (Copy from C:\FAH\SMP1 to C:\FAH\SMP2). This may take a couple of minutes as it is copying the 5gb vitual hard driver you created.
   
   
3. Hop back into the VmWare Server Console, select File > Open > Browse, point it to your _second_ vm (C:\FAH\SMP2) and select the file that shows up. It will ask you if you have copied this VM and if you would like to create a new Machine ID, select create new machine ID.
4. You should now have two VMs in VmWare. Start them up, open the command line console, go to the directory where you unzipped the client (/home/[username]/FAH), configure the clients, and enjoy your awesome PPD.
   
   

   > cd ~/FAH
   > ./fah6 -smp -verbosity 9

Appendix

I - Windows Vista 64-bit notes

Windows 64-bit users are going to have the problem of windows complaining about unsigned drivers. For this you have a couple of options.

A) Use Vmware server 2.0. I did do an install of vmware 2.0 and windows did not complain. But i have not configured a vm with it yet. In the future perhaps i will do this, or if someone else cares to, i will add the install procedure for vmware server 2.0.

B) Before windows starts loading, just after your bios finish the POST routine, hit your F8 key and it will bring up an advanced boot menu. In there is an option to disable driver signature checking. The only annoying part is, you will have to do this every time you reboot. This is what i do, but i have to go through a multi-boot menu regardless.

C)For this option you will have to perform the above operation for the install of vmware server 1.0.x, reboot, F8, select disable driver signing, and install vmware. There is also an application that will enable 'Test Mode' for vista64. This is designed for developers who are creating drivers that are not yet signed. It allows you to set Test Mode on which is basically the same as doing (B). Hit up google for enabling windows vista 64 test mode for more info.

II - Setting vmware priority/affinity

Once everything is up and running. Hop into Task Manager and find the two vmware-vmx.exe processes. If you use this system, and its not a dedicated folding rig, you'll want to set their priority to Low. Otherwise you will notice some lag as the vmware priority overrides the clients. So instead of your clients picking up idle cycles, they'll be competing for cycles like every other process, and generally they win

Now for affinities, its recommended to set one processes to cores 0/1 and the other to cores 2/3. Latest word from the new nvidia drivers is their client is using very little cpu time, so this would be a good setup even for those with nvidia gpu clients.

If you have ATI clients running, atm they're still using a good part of 1 core, then set _both_ vm processes to 0/1/2 and let the gpu client have core 3. The gpu client may not pick up all of core 3, but there are some other processes aside from the vmware-vmx ones that the vm is using. This will pick up whats left on core 3. On my system, this is what i do, and core 3 is averaging 95-97%, while cores 0/1/2 are almost solid at 100%. I tried doing core 0/1 to one vm and just core 2 to the other, and it didnt seem to work all that hot, ymmv.

III - Setting up VMWare Tools by the_cultie

1. When you have your VM open go the toolbar and click VM and select Install VMware Tools. A window will appear asking you to make sure the guest operating system is running. Click install to proceed.
   
   
   
2. On the guest OS desktop there should now be a DVD icon called VMware tools. Open it and extract the contents of the file called VMwareTools-1.0.8-xxxxxx.tar.gz to your Home Folder (also called [username] in the extract manager). It’s a good idea to go into the Home Folder and re-name the vmware-tools-distrib to something short like vmt as it saves some typing later on.
   
   
   
3. Open up Terminal and enter the following:
   
   -> cd ~/vmt
   -> sudo ./vmware-install.pl
   
   It will then ask you to select where to install the Binary Files, Init Directories, Init Scripts, Daemon Files, Library Files, and the Documentation. For each one just press enter and they will be installed in default locations. You will then be asked if you want to VMware Tools; just press enter to continue.
   
   
   
4. When the configuration starts you may get the following warning depending on the version of Linux you are using:
   
   None of the pre-built vmhgfs modules for VMware Tools is suitable for your running kernel. Do you want this program to try and build the vmhgfs module for your system (you need to have a C compiler installed on your system)? [yes]
   
   Just press enter and when it asks for the location of the directory of C header files just press enter again for the default location. The module is then built.
   When it is finished press enter again to continue. You are then asked to select a resolution for you screen by selecting a number from the list 1-13 where each number is for a different resolution. Press enter after making you selection and VMware Tools is installed. You should then re-start Linux before continuing.
   
   
   
5. After restarting Linux its time to actually configure VMware Tools to keep the time synchronized with your system time.
   Open Terminal and enter the following:
   
   -> vmware-toolbox &
   
   This will launch the VMware Tools. On the first tab of the window that open you will find an option for Time Synchronization Between the Virtual Machine and the Host Operation System. Tick the box beside it and click close. Your time will now stay synchronized with your system time.

Last Update: 12/24/08

 

[Jolly-Swagman]

Thats Great Cluster, thanks for making an easy to understand Multi VM Guide


Gets my Vote to be Added to our Install F@H Sticky

 

[ChasR]

A couple of things need clarification:

1. Whichever version of Ubuntu (7.10, 8.04 or 8.10) you install, it MUST be 64 bit. One of the more common errors in Linux installs is using a 32 bit version.

2. The command to install the 32 bit math library is sudo apt-get install ia32libs.

You're spot on with the rest. Some folks may find it easier to configure samba and the host name through the menu system. For instance, navigating to the fah folder using places (nautilus) and right clicking on the fah folder, select sharing options , check share this folder, allow others to write to this folder, will cause samba to install automatically. host name can be changed going through the networking menu under administration.

Another picky little item is the correct version of VMware is 1.0.8, not 1.8.

If you've started the fah client in VM1 before copying it to VM2, you will need to erase the work dir, queue.dat and machinedependent.dat in the fah directory of VM2 prior to starting fah in VM2.

 

[harlam357]

Great work Cluster! Very easy to follow!

Don't forget about the added benefit on Core 2 Quad cpus:

- set the affinity to 0/1 for the first vmware-vmx.exe process
- set the affinity to 2/3 for the second vmware-vmx.exe process.

This does give a small, but easily obtainable, boost to performance on a Core 2 Quad rigs that runs 2xVMs exclusively.

Also, I recommend setting VM priority to Low as a default practice. Doing so will yield better usability from the rig. If the rig is a dedicated machine then it may not matter so much, but if other tasks are performed setting the priority to Low will result in a much more transparent Folding experience.

Last suggestion, throw some Color... and some Larger sized fonts in there for section headers.

 

[Cluster]

Did some touchups to highlight the need for 64-bit and on not running the fah config until both VMs are setup.

ChasR: according to ubuntu 8.04 its ia32-libs\

More touchups, added notes that Windows Vista 64-bit users need to be aware of.

Im also thinking of reviving a custom linux distro to make this so much easier. I know Arkaine had Overclockix, derived from knoppix for setting up fah clients of all sorts. But i dont know if it ever got taken to this level, we were still in the uniproc/unicore days then. A custom distro that was merely 'zipped' and packaged for vmware installs would be much easier and faster. Although it would likely be a CLI only setup, there's really not that many commands to setting it up. Biggest part is maintaining it i suppose.

 

[harlam357]

I believe that's correct as well... but the other correction is still necessary:

> sudo apt-get install ia32-libs

 

[Cluster]

I believe that's correct as well... but the other correction is still necessary:

> sudo apt-get install ia32-libs

 

[harlam357]

Im also thinking of reviving a custom linux distro to make this so much easier. I know Arkaine had Overclockix, derived from knoppix for setting up fah clients of all sorts. But i dont know if it ever got taken to this level, we were still in the uniproc/unicore days then. A custom distro that was merely 'zipped' and packaged for vmware installs would be much easier and faster. Although it would likely be a CLI only setup, there's really not that many commands to setting it up. Biggest part is maintaining it i suppose.

Such a thing already exists... but, I've had little success with getting it to work properly. Mainly because I can't monitor it correctly using FahMon. The log only updates every 1.5 hours... at least that was my experience.

http://reilly.homeip.net/folding/vm.html

Really the VM is just a repackage of notfred's Diskless CD or USB setup- both of which work very well btw.

We don't have a large contingent, but there are a couple folks on the Team who do use notfred's Diskless approach as well.

 

[jintatsu]

I installed Xubuntu 8.10 on VM yesterday and I had problems doing the sudo apt-get install ia32-libs which i had done before when i used to be folding in 2xVM. Got it to work by doing sudo apt-get update before doing sudo apt-get install ia32-libs. Might help some who might get the same problem.

 

[harlam357]

bump... this doesn't need to get lost. Needs added to our Install Sticky, replacing LeeLegend's aging Ubuntu VM install guide.

Mods

 

[Is_907]

It's good to note that you can, if you'd like, use Debian or another distro. I'm setting this up right now (finally) with Debian 4.0r5 without X to save overhead.
My standard Kubuntu installation was losing 15% of CPU resources to Xorg so I'm really wanting my VMs to have more power.

 

[Is_907]

[15:41:08] - Calling './mpiexec -np 4 -host 127.0.0.1 ./FahCore_a2.exe -dir work/ -suffix 01 -checkpoint 15 -forceasm -verbose -lifeline 2740 -version 602'

I think this is a problem... according to the manpage for mpiexec, -n is the number of cores/cpus to be used... Now, if each of my VMs is executing "-np 4" that means I'm having all four cores used in each VM even though VMWare thinks there's only 2...
Corroboration of this: top reports four instances of FahCore_a2 in each VM.

So, am I getting screwed over because mpiexec is calling all four cores in each VM? I can edit my mpi config to only allow 2 in each... just want to know if anyone else has investigated this.

 

[Adak]

As I understand it, SMP currently always uses 4 threads, regardless of whether it's on a dual-core, a quad, or my 8 core server - in Windows, or in Linux, that's what I see.

Those 4 threads are virtualized across the available physical cores. So on a C2D, it's just two, a quad has it going across all 4 cores, and on my server, it runs across all 8 cores. Always with 4 FAH threads.

That's what it does natively, w/o VM. With VM, (the free edition), all the threads will be virtualized to two physical cores, regardless. That's all the client can reach, because of the VM.

So it's working to your benefit.

 

[Is_907]

Yeah, I'm seeing that now... Windows tells me each VM is using 50% of total CPU available so that makes sense.
Cool.
Thanks for the clarification, Adak

 

[ihrsetrdr]

Such a thing already exists... but, I've had little success with getting it to work properly. Mainly because I can't monitor it correctly using FahMon. The log only updates every 1.5 hours... at least that was my experience.

http://reilly.homeip.net/folding/vm.html

Really the VM is just a repackage of notfred's Diskless CD or USB setup- both of which work very well btw.

We don't have a large contingent, but there are a couple folks on the Team who do use notfred's Diskless approach as well.

Notfred's is great, I'm not running notfred's VMs right at the moment, but very quick n' easy to throw a notfred's VM into action...keep one on a stick.

Yea, monitoring is easiest just through a web browser, bookmarking the logfile.text.

 

[Cluster]

*bump*sticky*bump*

 

[Cluster]

bumpage

 

[ChasR]

Cluster,
I noticed another edit you need to make
The guest OS should be Ubuntu 64 bit for an Ubuntu VM rather than Other 64 bit. I'm not sure it makes a difference, but if it didn't, why would there be so many selections?

 

[harlam357]

Cluster,
I noticed another edit you need to make
The guest OS should be Ubuntu 64 bit for an Ubuntu VM rather than Other 64 bit. I'm not sure it makes a difference, but if it didn't, why would there be so many selections?

Agreed... I also use the predefined Ubuntu 64-bit profile.

 

[Cluster]

Oh, cool, i didnt think there would be one in there. I suppose regardless one should be picking Linux-64bit.

I think the difference might be for the VMWare Tools package. With Other-64bit i cant install them, but they should be installable with linux. Gonna switch mine over and see if thats the case.