[O/C]Windows Showdown: 8 Operating Systems in 6 Benchmarks

[hokiealumnus]

Again, a well-done work comes out stronger after tackling constructive criticism. It's part of how we learn and evolve.

I believe that Guatam's work is in this category: well-done work that will emerge all the stronger.

Firewalling ourselves from differing points of view isn't healthy or conducive to understanding. If our analyses can only convince people who agree with us, then they probably aren't very good analyses. Fortunately, that's not the case here.

I think there's a good opportunity here to intermingle and strengthen the bonds within our diverse community. Again, I'd like to extend my offer to G to do something together as a follow-up. I'm learning a lot as I read through here.

Spoken like a true PhD. Thanks for your input. Even those of us that didn't write the article are getting some good advice for the future.

 

[macklin01]

I hope so, hokie. I hope I'm not just stirring up trouble.

I hope that Guatam realizes that I wouldn't even be commenting if I didn't think it was a great article worth discussing.

 

[Gautam]

Yeah, I also think that the concerns are valid.

But perhaps I should do one benchmark with something like 20-50 trials which will also exhibit that the error between results is very small, and when you have even a couple of percent worth of difference, it is significant.

 

[Omsion]

It's not "random noise" and it is consistent. Even from a statistical viewpoint, if a data point is 5 deviations from the mean then the error is certainly statistically significant. In fact I'm not clear what reasoning you guys are using to dismiss a certain percentage as being "insignificant."

Yes, that's definitely appears true for some of the comparisons (generally speaking, XP vs the rest). But for quite a few, I don't think we could conclude a statistically significant difference of mean values with the current sample size (just by looking at the graphs, no actual hypothesis testing).
For example, Vista vs Server 08 vs Win7 in 3DMark03, 3DMark05, minus Vista64 in 3DMark06, etc. Thus my original conclusion(s). I definitely agree with you that there are non-random differences in the mix.

Ok let me put it in a different perspective, if you want a new car and want advice on what is more cost/performance effective you will probably look in Car and Drivers, but if you already have the car and want to get the most out of it you will probably look in Muscle Car. You see my point, this comparo was done for a Muscle car audience not for the Car and Drivers reader. You seem to don't understand how much work it involve to get let's say 1 seconds less in wprime, and G reference guide help us accomplish that. I can assure you that switching from Vista 32 to Win 7 won't let you get your email faster

macklin01 got to this first, so I'll let his word stand.

But for myself, I learned alot here from this back and forth, and especially about what benchmarkers look for. I understand now that this is an especially great guide for choosing which OS to run when targeting different benchmakrs. This is something I wouldn't have gotten out of this without this discussion.

 

[macklin01]

Beautiful. Thanks, Omision.

 

[Gautam]

For example, Vista vs Server 08 vs Win7 in 3DMark03, 3DMark05, minus Vista64 in 3DMark06, minus Servers and Vista64 in Aquamark, etc. Thus my original conclusion in my first post, Vista and Win7 are generally statistically indistinguishable. I definitely agree with you that there are non-random differences in the mix, though.

In 03, 06 and Aquamark, yes Vista and 7 are basically equal. In 05 they are certainly not. (And in 06 the difference between 32-bit and 64-bit for Vista is significant)

How about I focus on just 05, just Vista 32 and 7 32 for example, and give them each a much larger amount of trials?

 

[Omsion]

In 03, 06 and Aquamark, yes Vista and 7 are basically equal. In 05 they are certainly not. (And in 06 the difference between 32-bit and 64-bit for Vista is significant)

How about I focus on just 05, just Vista 32 and 7 32 for example, and give them each a much larger amount of trials?

Yeah, I probably mis-wrote some of those. I plead too many graphs and quick glances

Anyway, do you think a fair conclusion given these numbers, for an average user interested in upgrading to Win7 from Vista only for performance reasons is "Don't bother - many insignificant results, couple significant ones but only resulting in small differences in both directions depending on benchmark"?

 

[I.M.O.G.]

You see G that thing should have stayed in the lounge.....

Thanks for the hard work Gautam.... I know it must have taken hours and hours to accomplish and is very much appreciated! Not many people would have bothered with such an exhaustive effort, kudos.

....sorry to see some people giving you headaches.

..... lol Bob, you might catch grief for saying that, but +1 brother I'm with you.

The response has been overwhelmingly positive, and even gautam would agree it was time to release his work. 6 major community outlets picked up his article, as well as many other smaller ones.

The negativity in response to open discussion is the only thing out of place here.

 

[macklin01]

Yeah, I probably mis-wrote some of those. I plead too many graphs and quick glances

Anyway, do you think a fair conclusion given these numbers, for an average user interested in upgrading to Win7 from Vista only for performance reasons is "Don't bother - many insignificant results, couple significant ones but only resulting in small differences in both directions depending on benchmark"?

I think that's fair; see above. I think that's what makes this so interesting--depending upon the purpose, there are two radically different sets of conclusions to draw. Also, the fact that the highest scorer varied among all the tests seems to show that overall it's a draw, when not considering specific benchmarks as the goal. I also think that another interesting result shown here is that Vista can be trimmed to perform essentially as well as XP and Win-7 (at least in benchmarks; task switching, etc. is another matter). These are interesting results outside the benching community.

It might be good to use the term "small" rather than "insignificant." The differences may well turn out to be statistically significant but not large enough to justify the time spent in an OS reinstallation. Again, depending upon the purpose of the system.

Another funny thought: for some of the benches, there may not be a statistically significant "winner." In those cases, a bencher would be better served by running the benchmark multiple times and waiting for a random event to push them higher than reinstalling their OS. That's actually kind of cool.

 

[xtreeme]

Another funny thought: for some of the benches, there may not be a statistically significant "winner." In those cases, a bencher would be better served by running the benchmark multiple times and waiting for a random event to push them higher than reinstalling their OS. That's actually kind of cool.

You are absolutely right there, I actually did hold the worldrecord in 3dMark 2001 - the result came after hours of benching. I never expected the result

Futuremark (former Mad Onion) have created a hype - I did early understand their goal ; earn money on others work.... so I just jumped off

 

[g0dM@n]

I'm sure glad I'm on Windows 7 and ditched XP.
Hopefully I can afford to implement Win 7 on all of my PCs and laptops.

GREAT ARTICLE!!

 

[xhrist]

That's a lot of good info and a lotta hard work. Thumbs up!

 

[Neuromancer]

Excellent article bump

Was just reading some of the comments and I can not believe some of the statements made "XP is just fine by the data" and it is "tangibly faster than Vista" etc.

First of all "Vista is tangibly slower" is based on a subjective conclusion, and is contrary to your claims about the article not being "scientific" and has been known since the OS was in BETA that it was a UI effect to make the OS seem more appealing.

To deal with the Vista comment. It is faster than XP, the difference is in the UI. The "aero theme" has a 1000ms delay that you can adjust. This will make Vista "tangibly appear faster" than XP. But it gets rid of the nice effects. Way back in the day XP had the same issue. They added a delay to the start menu and tweakers hacked the hell out of that OS to make it a benchable system over 2000.

(subjective)For me Vista boots faster, loads programs faster and runs a lot more solidly than XP does. I DREAD having to work on peoples PCs that still use XP. Sad but true. I am even starting to appreciate 7 a bit now that I have forced myself to use it for more than a month. Its still no Vista64 but, it might be. (7-64 would not let me run a ton of software I like so that choice was not an option )


As for the basis of the article. It is quite clear and would be too hard to read if it started at 0%. I like seeing them start at 0, and oft times when I see a review that does not start there.. I anticipate a biased report. I can see why they chose to work it how they did. Yes 5% is small in terms of "desktop readiness" it is HUGE when talking about benchmarking though. 5% boost in performance could lead to 50-2000% improvement in boints. (not a typo).

Just saying the article is great. Thanks guatum for your diligence. I find myself linking to or referring to this article quite a bit