[ChasR]

My post on FC Forums:

I did some testing on the effects of affinity on a Q6600 running Windows XP Pro SP2 with two virtual machines running Ubuntu 7.04. The Q6600 was running at 333 x 9, 4GB ram @1:1, 4-4-4-12. Each VM is allocated 812 MB ram.

Without affinity:
VM1:
p2605
TPF: 14:09.0
PPD: 1791.1

VM2:
p2609
TPF: 14:32.3
PPD: 1508.5
Total PPD with affinity of each VM 0,1,2,3 (no affinity): 3299.5
================================================== ===============

With Affinity
VM1:
p2605
TPF: 12:29.8
PPD: 2028.1

VM2:
p2609
TPF: 13:18.6
PPD: 1647.7
Total PPD with affinity of VM1 = 0,1 and VM2 = 2,3: 3675.8
================================================== =========

With Affinity crossed (VM1 = 0,2 and VM2 = 1,3)
VM1:
p2605
TPF: 14:11.7
PPD: 1785.5

VM2:
p2609
TPF: 13:07.3
PPD: 1671.3
Total PPD with affinity of VM1 = 0,2 and VM2 = 1,3: 3456.8
================================================== =========

Restricting affinity such that one VM is run on cpu 0 and 1 and the second VM is run on cpu 2 and 3, results in a 11.4% increase in performance on this configuration as compared to no affinity restriction. YMMV.

I'll let someone else ponder on the results of crossed affinity which resulted in a 4.8% improvement compared to no affinity and saw VM2 turn in slightly better frame times than with uncrossed affinity.

When I began the test I was running with affinity set. After running with no affinity and crossed affinity, I returned to the original settings and found I was getting the same frame times on each instance as before. So setting (or restricting) affinity does offer increased performance in the case of two VMs on a quad.

Edit: The free version of VMware only allows the VM to run on two cores, so to utilize all four cores of the quad, two VMs are necessary.

[NedClocker]

That's on a Windows rig, right?

So, if I get a quad and run linux, do I still need to run VMware?

[dfonda]

Nice write up Chasr.

[NedClocker]

Originally Posted by dfonda

Nice write up Chasr.

Yeah, but what the heck is affinity, any way?

[drshivas]

Originally Posted by NedClocker

Yeah, but what the heck is affinity, any way?

Affinity is where you set a process to run on a particular CPU, and only that CPU. By default, most processes are juggled around on the CPUs by the OS, thus creating overhead.

Right-click on a process in task manager, and you'll see a Set Affinity... option. Check it out.

[Sorin]

When I ran two VMs, I always ran that affinity setup anyways. VM1 core 0+1, VM2 core 2+3. Just because I'm a neat freak and preferred the way the processor usage bars looked when one was running and one wasn't . And because it just felt cleaner that way. Interesting to see that it actually nets more PPD.

[NedClocker]

Originally Posted by drshivas

Affinity is where you set a process to run on a particular CPU, and only that CPU. By default, most processes are juggled around on the CPUs by the OS, thus creating overhead.

Right-click on a process in task manager, and you'll see a Set Affinity... option. Check it out.

Well, if I had Windows running on a C2D or C2Q rig, I would!

That's ok. My next build will be a C2Q and I do have a brand new copy of WinXP just waiting for a computer.

Ubuntu rocks!!! But I think I will make my quad a Win rig so I can do some things on it other than fold.

[pscout]

Thanks for confirming Chas ... I had forgotton all about this

I found it interesting that once affinity is set, putting it back to the original no affinity setting did not reverse the improvement, so i guess it makes the setting sticky, at least for vmware.

It would interesting to see if there is a similar improvement running 2 clients on 1 native linux but i don't know if there is a way to set affinity in ubuntu that would match the vmware setup.

[NedClocker]

Thank you, Chas.

[ChasR]

There is also some evidence of drastic improvement in frame times in Win SMP on C2Ds. If task manager reports one of the FAH Core threads at 50% and the other threads splitting up the remaining 50%, set affinity for two threads to cpu 0 and two to cpu 1. All four threads will run at about 25% and performance should improve. At least two AMD users reported knocking 10 minutes/frame off their times by setting affinity. Sadly, when the WU completes, the threads (processes) close and you lose the affinity setting. Stanford is working on adding affinity settings to the client.

[ChasR]

Originally Posted by pscout

Thanks for confirming Chas ... I had forgotton all about this

I found it interesting that once affinity is set, putting it back to the original no affinity setting did not reverse the improvement, so i guess it makes the setting sticky, at least for vmware.

It would interesting to see if there is a similar improvement running 2 clients on 1 native linux but i don't know if there is a way to set affinity in ubuntu that would match the vmware setup.

It should change immediately, rather than being sticky. When I did the testing yesterday, I had affinity set. When I changed to no affinity, frame times immediately changed, getting about 11% longer. Every change made an immediate difference. Perhaps you accidentally changed the wrong vmware process or else don't have an updated copy of XP? (I really have no idea if the workings of affinity have been changed in XP updates, but it could explain the difference in observed behavior).

[Godfather1138]

I'm trying this on my E6600 now. I was getting a pretty steady average of 18min tpf on stock clocks. I will update here later today with any changes to that.

Update: I am not seeing an improvement on frame times due to the change. I have noticed that the first threads set to a core hare holding around 30%+ and the remaining is going to the other thread. So on core 0 one is running @ around 35% and the other would be 15%, and similar on core 1. I'm not sure if this is why I'm not seeing an improvement as it did not evenly distribute the power, but I can't say for sure.

Any thoughts?

[DreadedOne509]

I was thinking that the crossed affinity could be an indirect result of having two asynched
work unit's, one requiring (at a good moment) more cache than the other on the same core.
Since they can share the 4 MB L2 cache, since one needed the extra cache more than the other it shows an increase in ppd for them both. Had the affinity been restricted to one client on one core, the sharing of the cache would be more restrictive since the 4 threads for that client would more than likely need the cache all at the same time, or within moments of each other.

Thats looks a lot more confusing written out than I had planned, hopefully
some of you will get what I was trying to convey.

On a side note for the person asking if VMWare was required, I believe he used it
to run 2 SMP clients separate from each other since if he had just installed two (or more) SMP clients on a quad-core machine they would fight each other to a stand-still or not work at all. The Virtual Machine allowed him to run them concurrently as if they were installed on two separate dual-core machines (affinity allows the fine tuning of which cores each gets).

It would be very interesting to see how doing this with an 8-core system would affect the ppd with and without cross linking the affinity assignments. 1,3,5,7 for one client, and 2,4,6,8 on the other for one run, then 1,2,3,4 and 5,6,7,8 on the next. I'd bet the results
would surprise us, at least with cache dependent work units that is.

[ChasR]

Actually, you have to run two VMs because free VMware restricts you to 2 cores. To utilize the 4 cores two vms are required. You can run two SMP instances in native Linux and, rather than fight each other to a standstill, they fully utilize the cpu and provide up to a 1000 ppd boost over one instance on some WU combinations. However, most quads get assigned to the "quad only" server and the p305x WUs provide no boost in ppd when running two instances. Two VMs (2 cores each, thus not recognized by the AS as a quad) have the advantage of never getting a p305x. So with a Q6600 @ 3.3 GHz: in Win SMP, you're looking at 3000 ppd; in Native Linux SMP on p305x, 3300 ppd; in VM/Linux x2 SMP, 4400 ppd; and in Native Linux on two p2653s, 4650 ppd. Be advised that if you run two instances in native Linux, there are WU combos that will net you less ppd than a single SMP instance, so you will have to monitor the assignments closely and turn off one instance to allow the other to complete on a regular basis.

Dreaded may be right about the crossed affinity but it also could be that p2609 was able to hog the cache with affinity set and increase its speed at the expense of the less cache dependent p2653.

[pscout]

Yeah, just try running 2 2608's at the same time ... whether native or under vms. Looks like we could use those 12 MB cache yorktowns now.

[ChasR]

@ Pete,
Affinity appears to be sticky on XP x64 and hammers performance if set on a P4 640.

[Captain Newbie]

Originally Posted by NedClocker

That's on a Windows rig, right?

So, if I get a quad and run linux, do I still need to run VMware?

Uh, no. If you can run native, please do so.

[ChasR]

Ahh, but you may well make more ppd running Win XP/VM/Linux x2 SMP on a quad.

[Captain Newbie]

Originally Posted by ChasR

Ahh, but you may well make more ppd running Win XP/VM/Linux x2 SMP on a quad.

That's nice, but you *still* should run native if you can to get the most *real work* done.

[pscout]

Without an alternative measure of 'real work' , ppd is the only one that is useful to me and I expect most others. Stanford needs to be setting the ppd to encourage the most useful/needed wu's to be crunched. I am sure it is not easy for them to do since they only benchmark on a very limited range of configurations.

[DreadedOne509]

Originally Posted by pscout

Without an alternative measure of 'real work' , ppd is the only one that is useful to me and I expect most others. Stanford needs to be setting the ppd to encourage the most useful/needed wu's to be crunched. I am sure it is not easy for them to do since they only benchmark on a very limited range of configurations.

I think that is one of the biggest obstacles they've run into when determining
the point system. In a perfect world, awarding points based on the scientific work being done would, I think, be in direct conflict with keeping folders interested in the project; since some of the work units being released into the wild may be super easy on base systems, but score really high point-wise and vice versa. It's late, and not really sure that all makes any sense. I'll edit tomorrow when I re-read it for clarity

[NedClocker]

Originally Posted by pscout

Without an alternative measure of 'real work' , ppd is the only one that is useful to me and I expect most others. Stanford needs to be setting the ppd to encourage the most useful/needed wu's to be crunched.

I hope that's what they done with the SMP units. They must want those processed and back fast for some reason.

[Adak]

That's exactly what they have done, NedClocker - according to Vijay, the SMP WU's are indeed more valuable to their research right now, then the uniprocessor client, and that's why (amongst other reasons), they have such a high point total.

IIRC, that info was from Vijay's blog.

[ChasR]

Originally Posted by Captain Newbie

That's nice, but you *still* should run native if you can to get the most *real work* done.

I somewhat unsure what you mean by get more *real work* done. On a Q6600 at 3.3 GHz, with two VMs, you get 4400 ppd (on 2 x p2653). Thats 33% more *real work* than you get done in native Linux on a quad at 3.3 GHz on p305x.

[harlam357]

Humm, must try this... I would surmise that the increase seen on an Intel Quad with core 0, 1 set and core 2, 3 set vs cross-core (0, 2) & (1, 3) is strictly physical implementation of the dual dice.

If you can keep one set of processing inside one dice, then the data stays inside the L2 on that dice... in lieu of swapping between the two dice.

Of course setting no affinity allows the processes to swap around between all four cores and effectively slows things down even more. So I can see why cross-core affinity would yield a better result than none and why (0, 1) & (2, 3) affinity will result in the biggest increase overall.

Makes perfect sense to me... now I have to see how this will effect my AMD duals... will also be most interested to see how a Native AMD Quad will react. My guess would be that as long as affinity is set, then it won't care which core that process is assigned. No cross-core behavior due to physical implementation.