How does hwbot catch fake or harvested results?

[Wastrel]

I know theres a "report fake result" link under each score but how hwbot is going to judge if the so reported fake score is actually fake or not? Its easy to say when someone with a score categorized in single 7800GTX clearly states that its running stock clocks of 430/1200 and yet has the first place in single card 7800GTX 3DMark06 score. Filtering theses kind of may I call them "untrue results" to cover both fake and mistakenly represented results solely depends on user or hwbot moderator (if there's any) reports, I think theres no automatic filtering checking results' credibility. (correct me if I'm wrong) wich may very well lead to many "untrue results" slip unnoticed.

But the problem gets worse when you have cards like X1900 that still dont get a clear clock registration in ORB or freshly released cards that dont get proper clock detection for a while (I remember one of the nV7series cards behaved that way). Dual card result can be submitted as single card and theres no way you can tell if its in fact single or dual if the GPU/Mem clocks are not available at the ORB page (single X1900 Crossfire Edition card is not banned form competing in single card category is it.) and I'm not talking about results with a simple screenshot as validation proof. Bench in multi GPU mode switch to single adjust clocks and click take a picture. Based on what mbot post says you either have to provide a verification link OR a simple screenshot.

Also harvesting results off ORB. I was searching through ORB database and realized how simple someone can pick compare URLs and present them as their own score with specs taken from ORB page and placed between [hwbot][/hwbot] tags. Howmany sores submitted to ORB? likely hundred of thousands of them and I dont think all of them know about hwbot. I am not saying this is getting actively exploited but its a potential danger. It can be easily exploited to ramp up a particular teams overall rank as even average scores add up to overall ranking.

I myself hate party poopers and I am not trying to be one, hwbot is a very innovative and interesting project I figured I share my thoughts and find a way to further filter out fake or just wrongly represented results before we reach the point that teams battling for higher ranks start (if ever) playing unfair and thats when it gets out of hand. Random individuals wont have major effect on reliability of hwbot rankings I suppose.

Oh by the way no one has reported my result as fake this whole thing came to my mind when I was browsing through hwbot results and saw some interesting scores--> reported.

EDIT:Now with Cat 6.11 its even easier to fake dual card result as single for ATi cards. The new X1900 soft Crossfire can run two X1900/950 XT(X)s wich reflect as "X1900/950 XT(X)" in compare URL's system specs. with N/A core and mem clock!

 

[El<(')>Maxi]

hwbot uses the honor system because at this point there really is no way to automatically verify results. The good news is the community is very good at catching anything that falls outside of the norm, so the bad ones don't stay around too long. As always, if you see something odd report it so a hwbot mod may review it. When submitting a score always try to provide as much of the verification info as possible.

 

[d]g[ts]

Yes the bot is moderated by humans if scores are reported.
Regaurding futuremark scores If you tag (the name section in each scores details)anything in your project manager that you publish with your forum name, and forum your benching for knowbody can claim it as there own. Now when it comes to screenshots of PI and Pifast where its just a screenshot of the bench. The top benchers or people most likely to get a screenie stolen. will use MSpaint to place the forum name in the screenie itself to prevent this.

All in all most benchers HATE to see some no talent jackass post fake crap in an attempt to look good at what they do, and will report it.

 

[Wastrel]

The problem is you just cant tell if some of those scores are fake or real. They can be fake and look quite normal without any possible tracks for anyone to follow and report.

 

[El<(')>Maxi]

Yes there are some scores that are questionable, thats why I say whenever posting a score provide all or as much validation as you can. Screen caps, ORB links, everything. For 3D I'd really prefer it if the poster had to provide the ORB link otherwise it's no good, same with 2D tests, no screen cap, no hwbot result. Right now it is easy for someone to fake on hwbot but to be honest with you I don't think they last long, especially the scores at the top. The community is constantly looking over hwbot results and most of them know what is good and what isn't for the most part. If your not sure I'd report it anyway so someone can review it. Personally I'm not gonna task hwbot too much about it when the fault lies in the people who abuse it.

All we can do is try to make it work better than it already does. Frederick is very active, and I think if you give him some good feedback he will not dissapoint you. You can visit hwbot forums anytime to discuss it. hwbot on the whole is an amazing achievment, think about what it does and its hard to say anything against it in my opinion. The site is very well done and the hwbot service provides a great resource for the OC comunity. Nowhere else can you compare the skill of overclockers worldwide in all the popular bench tests like you can on hwbot and if I had to predict, hwbot will only get better from here on out.

 

[Wastrel]

Ok thanks for the replies. As I said in the first post I agree with you hwbot is a great effort and I am not trying to pick on them at all. In fact most of the (potential) problems that I pointed out are the results of the flaws present in ORB reporting/comparing system.

I think I failed to clearly describe some of the the situations I mentioned above, long posts distarct people.

About ORB compare links Its not someone claiming a hwbot submitted result as theirs. That has been taken care of with account creation and forum checks that hwbot does.

What I was refering to was someone simply browsing through results at ORB and picking up compare links belonging to unsuspecting victims and releasing them to hwbot under their names (as independent individual or related to a crowded 3DTeam forum) where in fact they had never benched that system. The unsuspecting victim would never know that their score has been submitted if they never have used hwbot service for that score. Screenshot helps here but its not mandatory to post one afaik.

I think if hwbot requires the ORB verification links to have "for hwbot" or something like that in their description plus making it mandatory to have the exact forum username printed in the descriptions box along with "for hwbot"
most of the stolen comapre link problems would be taken care of.This way you cant just copy/paste compare links and claim that they are yours, and if the real owner didnt bother with hwbot you would be in no danger.

Yet there are big flaws that seem like ORB just dont care about them. After a rather long period of time that multi-gpu solutions are being widely used it still cant detect SLI/Crossfire and some graphics cards dont get proper clock detection. X1900 for instance, I think there are a few more cards with the same problem. This opens up another possibility to post fake results.

For example I bench with 2x X1900XTXs in crossfire mode(Cat6.11) with proper clocks that match the power of a super clocked single card. Benching finished score is up I switch to ATT and raise the clocks, even can close ATT for a sec open CCC and untick XFire reopen ATT, take a screenshot submit result on line. The system spec section in compare link shows my card as X1900XTX, the core and mem clocks N/A. How can you tell if it was in fact single or dual X1900? Even the screenshot shows single X1900 at high clocks. Same can be done with nVIDIA. ORB's flaw in action.

Again hwbot cant really do anything about it not their fault.

You're right Maxi this topic should be discussed where it belongs to, hwbot forums. I think I wont bother with that though and let it end here.

I'm not questioning any of OC Forums scores all fine and dandy nor any particular score ATM.

Smiley love to ease up this post a little.

I dont want to look like newly joined forum memebrs who get sensational over an issue and discuss it to death to paint themselves as smart a**es.

Wait a minute! that is just what I did! Exept for I had no interest in the painting section.

 

[Joe Camel]

sadly, there is little to no way to have a 100% fake proof system (that I know of)
as Maxi said, the top 10's get scrutinized 100's of times a day im sure so id like to think there would be little chance of a fake getting to stay there very long.

the only other thought i have is:
this isnt for money this is for fun. sure there is a pride factor but if you have to cheat and fake to make yourself look better, i doubt much pride comes of that.

 

[El<(')>Maxi]

NP I don't mind talking about it here Wastrel but I also think it's important for the hwbot guys to hear feedback. No doubt you have some valid questions, ones I've had myself. Maybe I'll ask on hwbot and see what ideas they have. Really they may have very good reasons for not changing things I don't know.

As far as single/dual card results on the ORB go FM has pretty much solved that with '06. I think ATI has been far better on the ORB then Nvidia with dual card setups because the Master installs the driver under a different name than slave cards. SLI however is a total ***** to get a grip on in anything other than '06. And I totally agree, not detecting card clocks is a problem they are apparently ignoring. FM gets revenue from Pro sales and now advertising, they should have this working correctly and have new release drivers approved alot sooner than they do now.