Cheating in Shadermark and ?Q3?

[Cowboy X]

Just when you thought the cheating had ended a certain graphics company comes to prove you wrong . Tommiti Systems created a comprehensive DirectX9 shader benchmark called Shadermark . The NV30 had very poor performance compared with the R300 based products . A reviewer / member at Beyond3d ( I can't remember which ) asked Nvidia why their performance in a benchmark based on STANDARD shader code was so poor . They replied that they never heard of the benchmark and would look into it . Then comes NV35 and the cheatFX drivers which cause the great 3dMark scandal . In the background the NV35 and NV30's shader performance in Shadermark increased significantly causing some to say that Nvidia had fixed their problems and that the poor 3dmark 2003 scores showed that 3dMark was all alone in making Nvidia products look bad .

Well, the maker of the benchmark has shuffled his code to prevent and illegal optimisations , and guess what ....................... Nvidia's scores drop like a rock :

http://www.3dvelocity.com/reviews/5900u/5900u_16.htm

http://www.beyond3d.com/forum/viewtopic.php?t=6134&postdays=0&postorder=asc&start=60

http://rage3d.com/board/showthread.php?threadid=33693303&perpage=20&pagenumber=1

The really frightenning thing is how Nvidia basically disses everyone who dares to find them cheating . Their comments on the shadermark thing are definitely worth a frontpage article by Ed .


But it doesn't stop there . With all of this cheating going on many people began to wonder if there was cheating going on in game time demos as well . Nvidia is being schizophrenic here , one minute it is for 3dmark , then against , then they say no more synthetic benches , but at the same time promote synthetic benches like chameleonmark and extensively cheat in the others . ( Enough of that rant. )

Firingsquad decide to do a review of an overclocked Evga 5900 Ultra using custom demos instead of the same old timedemos . The 5900 does very well , BUT , when custom demos for Quake 3 were used I saw Nvidia lose Q3 demos even with an overclocked 5900 Ultra. Is there cheating here too ?
http://firingsquad.gamers.com/hardware/evga_e-geforce_fx_5900_ultra_review/page9.asp

http://firingsquad.gamers.com/hardware/msi_geforce_fx5900-td128_review/page9.asp

Soon all reviews will have to use custom demos and make sure that the hardware makers can't get them . And also use fraps while playing the game ( mind you Nvidia currently frowns on Fraps , strange , I thought they liked benching with real games )

So the water has been further muddied . It is now even hardeer to read a review on a vcard and think you have seen the real picture . Makes me feel like the guy in the Matrix who just wanted to go back to the way things used to be . Shoulda taken the blue pill , too much time and energy spent chasing behind cheating hardware companies .

 

[OrionRU]

:conspiracy theory:


Get a life. If the card is optimized for Q3 its cheating?

 

[james.miller]

wow. great post. excellent postartificially improving scores, and optimising are two very different things. You should read up on them.

nvidia are fast going down the drian for me. Unfortunatly though, a company that big will survive that, no matter what happens. It will take a lot more than a revolt from everybody "in the know" to dent nvidia.

 

[Zaku]

Hmmm

IMO i would call that cheating or atleast misleading.
optimizing for a demo but not for the real game.

 

[UnseenMenace]

It is now even hardeer to read a review on a vcard and think you have seen the real picture . Makes me feel like the guy in the Matrix who just wanted to go back to the way things used to be . Shoulda taken the blue pill , too much time and energy spent chasing behind cheating hardware companies .

I do not personally believe that things were ever normal, benchmarks have been Faked/Optimised/Changed for years.. I can remember when the old speed hack which was used to cheat in certain games would effect the clock of these benchmark applications and produce both nearly believeable and laughable results.. This went on for a while with many fake benchmarks posted webwide untill it became a problem.
What have benchmarks ever proved that has actually been of worth to the user.. A benchmark from one site is created in different ambient tempratures, enviroment, situation to a benchmark from another site...
Basically speaking your benchmark scores imho mean very little to me or my system, as such benchmarks are only of use as a investigative tool to measure improvements on a sigular system and not to compare one with another.
It seems strange to me that people have at various times ranted at various companys producing products and accusing them of cheating but never seem to rant towards the people producing easily flawed benchmarking software..

Perhaps we should be looking for more secure benchmarking systems, which can not be tampered with, perhaps ones done over a secure server thus also stopping sites which sell their reputations of edit results to obtain what they desire..

The problem as far as I see it, is not the companys that need addressing but the way in which we benchmark !!

Article on benchmark faking

 

[schismspeak]

:conspiracy theory:


Get a life. If the card is optimized for Q3 its cheating?

Man did you read everything? Nvidia is full of BS and no it isnt cheating as long as it doesnt reduce quality.

Oh yeah, make sure you get an Nvidia card, ATI only runs 1 out of 10 games properly.

 

[InThrees]

:conspiracy theory:


Get a life. If the card is optimized for Q3 its cheating?

In a thread I posted earlier, I intimated that it's not cheating, per se, to optimize for games.

However, in this case nVidia evidently optimized for a static demo of a game, and those optimization fail on other demos, and presumably the game itself. This means the benchmarks for the optimized demo give misleading results that can't be reproduced via random gameplay.

Now does it sound like 'cheating'?

 

[OrionRU]

I just find all these threads funny. Everyone and their dog know about this by now. So why beat the dead horse?

What are they gonna do next? AMD optimizes their procesors for benchmarks

 

[Cowboy X]

UnseenMenace ............... Note , I never really said that there was never any cheating before . What I meant is that the widespread and flagrant lying and cheating of late makes me almost wish to go back to not knowing what was going on . Hence my reference to the red and blue pills .

OrionRU ................. No need for conspiracy theories on the Shadermark issue , it is obvious that there is something strange going on there . The Quake3 thing is however a bit more iffy that is why I posed a question regarding that demo issue .

Maybe some people are tired of the almost daily cheats and illegal optimisations , but it is still important that people know . Whether we admit it or not only the rich can afford to buy a high end new Vcard without consulting benchmarks and reviews . It is the only substitute to having the card yourself . And as most people don't own computer stores or get free review samples , honest reliable reviews and benchmarks are very important . No matter what is said at [H] or in other some quarters there is a place for both synthetic and game benchmarks . Optimising for a benchmark ( synthetic or timedemo ) in a way that doesn't change the image quality or the total workload on the card to my mind is ok . But most developers have since come to the consensus that it shouldn't be done at all . Optimising for a specific game is no problem once the image output is identical or better . But optimising for a known timedemo in such a way that will not benefit the gamer on any other map or in gameplay anywherelse in the game is the worst form of deceit and is simply cheating .

UnseenMenace .............. What the community needs to do is become more transparent , equip each bench if possible with tools like 3dmark 2003 which can severly hinder cheating or at least catch it and do more actual in game fraps tests . But the main thing we need to do is make sure that companies feel it in their ears and bottom line when they cheat . If we do like [H] who first denied that Nvidia was doing anything wrong , then blamed the benchmark , and now is blaming all synthetic benches , things will never improve , since such people consistently remove the blame from the vcard maker of their choice . When more game demo cheats and low quality hacks are discoverd will [H] invalidate games as benchmarks too ???? Maybe they should invalidate themselves .

 

[Cowboy X]

Further Discussion :

http://www.beyond3d.com/forum/viewtopic.php?t=6510&start=0

I'm not the only one who disagrees with [H] :

http://www.beyond3d.com/forum/viewtopic.php?t=6578


Edit : One link was incorrect .

 

[Rudski]

:conspiracy theory:


Get a life. If the card is optimized for Q3 its cheating?

Orion you are a Tool !!!!!
Can you not see past the marketing hype?
Another no brainer with the inteligence of a Nvidia cockroach. Do you work for Nvidia or are you a long supporting customer that cannot bring himself to admit you have a ****e product. Why would you need to " optimise "? Cannot you get your hardware to perform without any smoke or mirrors. I hate it when people blow smoke up my *** and Nvidia is doing exactly that.

Get a life and start seeing things the way they are.

 

[UnseenMenace]

UnseenMenace .............. What the community needs to do is become more transparent , equip each bench if possible with tools like 3dmark 2003 which can severly hinder cheating or at least catch it and do more actual in game fraps tests . But the main thing we need to do is make sure that companies feel it in their ears and bottom line when they cheat . If we do like [H] who first denied that Nvidia was doing anything wrong , then blamed the benchmark , and now is blaming all synthetic benches , things will never improve , since such people consistently remove the blame from the vcard maker of their choice . When more game demo cheats and low quality hacks are discoverd will [H] invalidate games as benchmarks too ???? Maybe they should invalidate themselves .

I agree with your statements however it appears that the companys which produce the benchmark software have very little to say on the matter other than pointing the fingers of blame... I am certain that the people producing this software could do more to reduce the frequency of such cheating but choose not to. In these debates the software is not often dicussed or thought of as being flawed and this is perhaps something that needs reviewing.

 

[Cowboy X]

Too much $$$ flowing . And in FutureMark's case their actual agreements didn't properly define cheat ( according to THG ) leaving a loophole for Nvidia . I still think they would have won the case , but instead of being a slamdunk it could have been drawn out by Nvidia . Such a costly case would be too much for FutureMark to handle financially . They could easily go bankrupt before the case and appeals are through .


The bench makers need to set themselves up in better legal positions to call a spade a spade in public . And if needs be, sue without fear of losing , rather than their curent impotent state .

I agree that quality control software such as the developer version of 3dmark 2003 are very important and should be implemented elsewhere . I'd really love to see what would happen if an 'off the rail' viewer was used for 3dmark 2001SE . I agree tht flaws can also exist in the software , but in the case of 3dmark 2003 , none of the 'flaws ' have been verified by any independent sources ( [H] doesn't count ) . But i personally don't think that it is as big an issue , since as soon as a manufacturer is accused of cheating they will be sure to use their significant engineering knowhow to find any mud to throw back . In fact things that aren't flaws will likely be called flaws . It is for us then to evaluate the evidence , do some of our own testing and come up with a answer for ourselves .

 

[Grande Juan]

I have to agree with UnseenMenace on this one. If I owned one of these "cheating" companys I would do the same, optimize for the benchmarks to sell more cards. Perhaps if people quit putting such "blind faith" in these benchmarks as a sole reason for buying their next card, these cheating companys would stop their shananigins. After all aren't you cheating when you turn off vsync, AA and AF, to create an artifically high 3dmark score, then turn it back on for apps. They're just giving everyone what they want, a higher score than the other guy. I've always used 3dmark to optimize my own system, not compare it to others. I believe I read that somewhere on futuremarks site. How many times have we seen forum members post "I just installed xx.xx drivers and my 3dmark scored jumped xxxx points"? Did games look or play noticeably better? Likely not! Anyway I like the idea of a more secure benchmarking utility.

 

[Cowboy X]

Here is another example of what happens to the FX line when you use benchmarks other than the standard 3 or 4 , suddenly the 5900 U doesn't look as good anymore .:

http://www.gamepc.com/labs/print_content.asp?id=fx5900u

Instead of a massive performance lead it only brings Nvidia to performance parity with the R300 line . But not any true overrwhelming lead which Nvidia would like .This is what makes the need to 'optimise ' .

 

[Dragonprince]

:conspiracy theory:


Get a life. If the card is optimized for Q3 its cheating?

Well yeah...think about it. You buy a card to play alot of games right ? But review sites are limited in time/resources to do testing with a bunch of different games so they use "canned" benches that nVidia knows will be used. They optimise for those but not everything else so your left with lousy performance on any other game not covered by reviewers....thats not right, period.

You'd be ****ed if you bought a car that only cornered good in one direction because all the car mag's test new models on the same oval in the same direction right ?

Think about it

 

[Dragonprince]

I just find all these threads funny. Everyone and their dog know about this by now. So why beat the dead horse?

What are they gonna do next? AMD optimizes their procesors for benchmarks

No AMD and Intel offer the developer kits to software makers world wide to allow them to optimize for a processor. nVidia is optimizing specific benchmarks only without offering any developer kits because they are reducing image quality to get results and dont want to have to admit it. Huge difference.