Nvidia and Futuremark kiss and make up

[Cowboy X]

Read it here : http://hardocp.com/article.html?art=NDgx

" Futuremark Statement
For the first time in 6 months, as a result of Futuremark's White Paper on May 23rd, 2003, Futuremark and NVIDIA have had detailed discussions regarding NVIDIA GPUs and Futuremark's 3DMark03 benchmark.

Futuremark now has a deeper understanding of the situation and NVIDIA's optimization strategy. In the light of this, Futuremark now states that NVIDIA's driver design is an application specific optimization and not a cheat.

The world of 3D Graphics has changed dramatically with the latest generation of highly programmable GPUs. Much like the world of CPUs, each GPU has a different architecture and a unique optimal code path. For example, Futuremark's PCMark2002 has different CPU test compilations for AMD's AthlonXP and Intel's Pentium4 CPUs.

3DMark03 is designed as an un-optimized DirectX test and it provides performance comparisons accordingly. It does not contain manufacturer specific optimized code paths. Because all modifications that change the workload in 3DMark03 are forbidden, we were obliged to update the product to eliminate the effect of optimizations identified in different drivers so that 3DMark03 continued to produce comparable results.

However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks.

NVIDIA Statement
NVIDIA works closely with developers to optimize games for GeForceFX. These optimizations (including shader optimizations) are the result of the co-development process. This is the approach NVIDIA would have preferred also for 3DMark03.

Joint NVIDIA-Futuremark Statement
Both NVIDIA and Futuremark want to define clear rules with the industry about how benchmarks should be developed and how they should be used. We believe that common rules will prevent these types of unfortunate situations moving forward. "


Both sides have backed down , with Nvidia opening the door to participation in 3dmark again ( possible ) , Nvidia has backed down from the silly " it's a driver bug " argument and at the same time has stopped blaming Futuremark for purposely harming their performance . Future mark has done their part of the deal by conspicously removing the word cheat from their statement and calling Nvidia's cheats optimisations and code paths they will be looking into .

This avoids the lawsuits everyone expected were going to start . in my opinion Futuremark would have won but it would be a war that both sides would lose . Nvidia's reputation would be further tarnished ansd maybe permanently so in the mainstream , which would be a worse case scenario for them . While Futuremark would win the battle but probably go under financially before seeing victory .

Where does that leave us gamers .................. well we are somewhat better off since the stories of 3dmark's imminent marginalisation and death have been put to rest for now , leaving us with a good benching tool . But at the same time , if they support all these extra 'paths' and allow the sort of stuff Nvidia did , then the benchmark will be useless .

Thus I sincerely hope that they have told Nvidia behind closed doors not to do it again and made this statement to avoid lawsuits and to help Nvidia save face .

[Cowboy X]

I just saw it being actively discussed here :

http://www.beyond3d.com/forum/viewtopic.php?t=6222&start=0

Many of the people at that site are also questioning the ability to trust Futuremark's benches and reputation . What good is a benchmark if no one can enforce any sort of rules or standards .

[RedSkull]

"Thus I sincerely hope that they have told Nvidia behind closed doors not to do it again and made this statement to avoid lawsuits and to help Nvidia save face ."


thats just it, they bullied Futuremark with a team of angry lawyers into submitting this statement, Futuremark is some small company compared to Nvidia

Looks like Nvidia is following the MS business model

[PreservedSwine]

Umm, yeah, inserting clip-planes and creating buffer over-runs isn't cheating....

I'm quite glad sites like B3D exist to keep the big guys honest......

[ninthebin]

application specific optimization and not a cheat

sounds like two different ways of saying the same thing if you ask me

[Molybdym]

Originally posted by PreservedSwine
. . .inserting clip-planes and creating buffer over-runs isn't cheating....

I'm quite glad sites like B3D exist to keep the big guys honest......

I'm not huge into the video sector, but what is a clip-plane?
Anywhoo, this whole fiascio has tarnished both of these companies, but I don't think the absent-minded consumer will hear of this. I jope this does't spill over into fanboy debates, with this as ammo.

[Cowboy X]

It's like knowing the answers in an mcq and submitting them rather than actually calculting them . In this case instead of answers we are talking 3d scenes . the only thing worse than this would be to automatically start playing an avi of 'how it was meant to be rendered ' when the benchmark/demo begins instead of actually running the demo ................. lol .

[HotKoala]

As if 3DMarks was *ever* credible in the first place.

[Stumpjumper5200]

Application specific optimizations = cheating. When said application is a benchmark..........

[Molybdym]

Originally posted by Stumpjumper5200
Application specific optimizations = cheating. When said application is a benchmark..........

Here here!

[BaD CrC]

Quite frankly, I find this change from Futuremark highly suspect. First they got this detailed report with support and correlation from several other hardware sites showing (with screen captures) those clip planes and the 25% drop of performances, then, the Nvidia guys came to talk to them and the day after, Futuremark completely changed their accusations without further explanations????
To me, it looks like either they got scared, or they got loads of money from Nvidia. But they certainly did not suddenly realised that those clip planes where optimisations! I don't think I can trust Futuremark anymore.

[Rav]

Originally posted by Stumpjumper5200
Application specific optimizations = cheating. When said application is a benchmark..........

Well "optimizing" something by reducing the quality or changing the output is cheating, because no one would ever play a game or run an application like that. But, "optimizing" something by tweaking the code for that particular application without sacrificing quality or the output is just good coding practice.

NVIDIA obviously messed up the output with their optimizations. ATI optimized but did not compromise the output. I think its fair to do that because they could (and do) optimize a game that way to improve performance.

But also, I find that my results in 3dmark03 really have no relevance to the actual performance I see in my games, so none of this really matters. Or does it?

-Rav

PS could someone post a link to the screenshots of NVIDIA's optimizations?

[IR1]

Well I guess 3Dmark is only good for one thing. Comparing ATI scores... to ATI scores. Even before this... comparing ATi to Nvidia you would have to lower ATI to its worst quality and then raise Nvidias quality settings to the highest... then you would almost be apples to apples. But now... its not cheating to use clip planes and buffer overrun, ok yeah. Futuremark credibility *poof*.
\

[rosieox]

Futuremark has themselfs said that with the changing environment of the gaming industry there is no way to accuratly develope an unbiased benchmark.

I believe in the future, vid card reviews will have to run an actual game, no test runs, the actual game it's self and give picture quality and frame rate. This will be the only true way to decern between cards. It will be much harder for the reviewers, but I will believe those results more than any benchmark test.

Edit: go Giants!

[Avatar28]

Rav is right. As long as visual quality isn't sacrificed, then optimizing for the benchmark is no worse than optimizing for a game. Also, you'll note that what they're talking about is doing like the game companies do and build the optimization INTO THE BENCHMARK.

In other words, you buy Doom III it will contain optimizations for both ATI and NVidia cards (and probably anyone else who is popular at that point) over the standard DirectX. 3D Mark will do the same thing.

[IR1]

Edit: go Giants!

amen.

[DaddyB]

Originally posted by Molybdym


I'm not huge into the video sector, but what is a clip-plane?
Anywhoo, this whole fiascio has tarnished both of these companies, but I don't think the absent-minded consumer will hear of this. I jope this does't spill over into fanboy debates, with this as ammo.

It's difficult to explain what the clip plane does but I'll give it a shot. In 3dmark (and all 3d games) there is a whole 3d "world", that is you can see what's in front of you but if you turn or look up you will see more of the "world". 3dmark does the exact same thing every time so that it's results will be comparable on any system, it's kind of like the camera is on a train and it follows the same path every time BUT there is still more of the "world" that you don't see because the camera follows the same path every time. What Nvidia did is tell there drivers not to render anything that the camera doesn't see, they followed the path that the camera takes and cut out everything that you don't normally see so that there card doesn't have to waste resources rendering that stuff or trying to figure out if it should cull it or not.

In the professional version of 3dmark you can pause the benchmark and turn the camera or move it around, as if you were flying through this "world"... When you do that with an nvidia card you see that there is no "world", it just ends.

I hope I did an OK job of explaining it... in case I didn't you can read what the guys who discovered this flaw had to say, or at least look at the pictures showing how the "world" ends when you move the camera (the link is to the part about the clip plane, if you are interested you should start at page one and read the whole article and all of the cheats that were used). Link (http://www.extremetech.com/article2/0,3973,1086853,00.asp)

Back to the topic at hand

What can be said about this whole ordeal... To the objective observer it is obvious Nvidia cheated, got caught and bribed or threatend their way out of it to save some face. Im sure there will be an onslaught of uninformed fanboys saying "I told you that nvidia didn't cheat" and pointing to futuremark's statement.

[TeLoS]

Where does this leave futuremark though?... you think websites and overclockers alike will rely on 3dmark for their benchmarks now?....

Futuremark are gonna make alot of people very unhappy now, and alot of people will be finding other benchmarks i'd assume....

Curious to see what happens to futuremakrs popularity! :D

[Cowboy X]

The Inquirer has its say : http://www.theinquirer.net/?article=9810

[Oni]

Originally posted by HotKoala
As if 3DMarks was *ever* credible in the first place.

Best post in the whole thread.

[Goodie]

Bleh , see all you guys thought Nvidia was cheating and I had made a post before ssaying that it was not cheating and now the truth comes out. Nvidia didnt cheat and as a die hard Nvidia fan I'm glad I stuck by them when everyone else accused them of cheating. All they were doing was taking a different optimization strategy, bleh @ who ever thought it was cheating.....

[q149]

It was and still is cheating. I'd say Nvidia threatened Futuremark to make them change their stance...
anyway..
Let the cheating begin!
May the best cheater win!

[skid]

Shame on nVidia for cheating! Shame on Futremark for backing down!

Looks like ATI is the only one who can stand against the Nvidia Corporate Monster...

[DaddyB]

Originally posted by Goodie
Bleh , see all you guys thought Nvidia was cheating and I had made a post before ssaying that it was not cheating and now the truth comes out. Nvidia didnt cheat and as a die hard Nvidia fan I'm glad I stuck by them when everyone else accused them of cheating. All they were doing was taking a different optimization strategy, bleh @ who ever thought it was cheating.....

Originally posted by DaddyB
Im sure there will be an onslaught of uninformed fanboys saying "I told you that nvidia didn't cheat" and pointing to futuremark's statement.

And people still doubt my ability to predict the future... sigh.

[Cowboy X]

Here I was, thinking that Goodie was just being sarcastic ............. at least I hope he was .

[Cowboy X]

Extremetech has and expanded version of Futuremark's views :

http://www.extremetech.com/article2/0,3973,1116490,00.asp

Fair to say that this was seen by me here : http://forum.oc-forums.com/vb/showthread.php?s=&threadid=201686

[(HR)Titan]

what futuremark get for exuse...
why he do that mass shame on his respect
now everyone says that 3d mark sux , all the time.
so why u belived when he said that nV cheeting

imo - nV was and will have better drivers for grafic cards
and thats the true

[Cowboy X]

I wonder what Futuremark has to say about this : http://www.tech-report.com/etc/2003q2/3dmurk03/index.x?pg=1

[RedSkull]

Originally posted by HotKoala
As if 3DMarks was *ever* credible in the first place.

yeah so I guess we should just use antiquated games to benchmarks future technology hardware!!!!!!!

ooooh QUAKE 3, Srious SpAM

oooh i get 500000FPS in DOOM 3

who cares

so i guess the only current game to use is UT2K3, so they will just cheaton that becnhmark