GTX 680 Single GPU - Courtesy Tech Report

Our Take On NVIDIA's FCAT GPU Testing

Add Your Comments

If you follow the GPU reviewing world, you have probably seen several articles pop up regarding a new GPU testing method using a tool called FCAT. Our editing team has discussed this extensively and we believe we’ve come to the right decision, but are of course willing to listen to our readers. Please have a look at this article, weigh the options and chime in with your two cents.

FCAT. Frame Capture Analysis Tool. In itself, it is a moderately open-source piece of software released by NVIDIA. A few select sites received copies of the software (though I’m sure they would make it available to us if we asked) and tools to use in measuring frame times.

Before we get into that tool and how it works, first we need to delve into what frame time is.

Frame Times – A Short Explanation

Before getting started, to save some space, I’m going to be making Statements with a capital S that are based on reading I’ve done. Where applicable, there will be links to the source where I read that. Rather than completely regurgitate what was said at those sources, I’ll just link them to you to verify what we’re saying.

Backing up to a year or so ago, Tech Report pioneered measuring frame time via FRAPS. Their original article, “Inside the second: A new look at game benchmarking“, went over why they felt it was a good thing to include. Their tool of choice was FRAPS, the popular FPS (and frame time) measuring software. Tech Report was the pioneer in the field, but not many review sites jumped on the bandwagon. What’s even more interesting is that Overclockers writer Kyle Lunau discovered this with FRAPS years ago, in an article published in May of 2009 – Micro Stutter: The dark secret of SLI and Crossfire.

Frame time is the time it takes your system to render one frame. You are used to FPS, frames per second. Frame time is how long within that second it took for the system to produce that frame. In our review of the GTX 650 Ti BOOST we outlaid a quick and easy reference as to what those milliseconds mean:

If you’re unfamiliar with frame times, it is how long the GPU takes to render a frame. FPS is a good measurement to tell how many frames are being produced per second on average, but as we all know, FPS can vary as you’re gaming. That’s because your GPU often takes longer than that average to produce the frame. The simplest explanation I’ve seen came from SKYMTL, who writes for Hardware Canucks from his research:

  • Below 20ms: unnoticeable
  • Rapid, cyclical fluctuations above 25ms: you’ll notice it
  • Sudden “spikes” (non-cyclical) above 40ms: you’ll notice it
  • Grey area between 21ms and 25ms: debatable. Some will notice it while others won’t.

So now you know what frame time is. Let’s move on and discuss what that has meant over the past year and a half.

Tech Report and FRAPS

Tech Report’s work, in a way, led to FCAT as it is today. You see, when they were measuring frame times, they noticed a massive problem with multiple GPU setups. While they were producing plenty of frames per second, the time it took to render frames in between other frames was huge, leading to stuttering in game that was obvious and noticeable. That’s when eyebrows started to be raised regarding this kind of testing.

FRAPS, while it is a great tool, especially when it’s free and available to the masses, is not a perfect tool. It measures frames and frame times in between the game and the GPU, the only place where it can. Here’s the handy-dandy graphic everyone’s using:

Courtesy NVIDIA

Courtesy NVIDIA

As you can see, there is a lot that happens after FRAPS that affects the final outcome. Translation: FRAPS can tell you a generally accurate average FPS number and its frame time measurements can tell you when there is a Big Problem. It can also give misleading information with regard to frame times.

AMD dislikes FRAPS as a tool. They gave Anandtech’s Ryan Smith an exclusive interview and outlook on their opinion with regard to FRAPS measurements and their driver roadmap. In it, basically, they admit to a problem with CrossfireX, say they’re working on drivers to address the problem and hope to come out with them sometime this summer. Their takeaway with FRAPS is that it shows problems where none exist, and amplifies them where they do.

Interestingly, NVIDIA agrees with AMD about FRAPS. They dislike its use as a frame time measuring tool for the very same reasons.

Enter FCAT

FCAT is software that analyzes frames captured while a game is running. It also has a component on the gaming machine that inserts colored bars on the side of the game’s frames as it is being rendered. When the overlay is running (similar to FRAPS’ FPS overlay), it looks something like this.

Image Courtesy Tech Report

Image Courtesy Tech Report

What you then need to do is run your DVI output off the GPU to a splitter. That splitter will then send out two signals – one to a monitor you’re using to run the game on your test PC, and the other to another PC set up to capture all of the frames coming from the gaming PC.

Once those frames are captured on the capture PC, you’re left with a massive file (we’re talking in the 20 GB range for 15 seconds of capturing). FCAT then analyzes the recorded frames and gives you a lot of data to parse via script. The parsing script gives you graphs that produce a frame time chart very similar to that you see produced by FRAPS, except rather than having been monitored from the game output, it has been recorded directly from the GPU itself on the business-end.

Herein lies the three practicality problems from a reviewer’s perspective and that of our site specifically:

1. The Capture PC

Number one is hardware. The capture PC is not cheap. Not in the slightest. First you have to have your standard CPU, motherboard, RAM, PSU, etc. That’s not too big of a deal. Call it $1,000 for a solid offering. Then you get to the expensive part. We’ve priced some capture cards and the one being used by most of the sites costs $1,800. By itself.

Then, as mentioned, 15 seconds from that capture card gets you over a gigabyte per second. So in order to capture the video, you’ll need a storage system capable of significantly faster rates than the fastest individual SSD on the market, and it can’t be small either. You’re talking $1,500-$2,000 on storage alone.

All told, using rough guesstimates, you’re looking at up to $5,000 on the capture PC. Now, we’re a review site. We are friendly with enough manufacturers that we could probably procure everything but the capture card, so our actual outlay would probably be $1,800. We just wanted you to see what sort of costs a layperson would need to lay out for a capture PC such that these other sites are using.

2. Time

As most of you know, we are a site run by volunteers. We have zero full-time employees writing for Overclockers.com. Our team is very dedicated and the work they do has astounded me over the past four years I’ve written for the site. That they are able to do what full-time, paid writers at other sites do on a volunteer basis is a feat in and of itself.

Analyzing FCAT data takes a very significant amount of time. Look at PC Perspective’s graphs below. Their excel work would take many hours all on its own and they’d do it for every game in every GPU review. Frankly, on a volunteer basis, that amount of extra input is a daunting prospect.

3. Multiple Reviewers

We have more than five people that write reviews regularly for this site and all of us are equipped to and can write reviews on any piece of hardware because we have standardized our test systems such that they are independent and one GPU reviewed by Lvcoyote or Bobnova on the west coast can be compared to one EarthDog or I review on the east coast. We do that because we’re community driven, always have been and always plan to be. No one person reviews all of one thing. Of course, all told, five capture systems would retail around $25,000, so that’s not an option for anybody.

The Real Reason Why FCAT Isn’t An Option

But let’s put all of those practicalities aside. Completely ignore the money, the time and the multiple reviewers. What if we could do all of that? Would we? To be blunt, the answer is no. After extensive discussion among our editing team, what we’re here to tell you today is that this is much ado about very little. Not nothing, but not much either.

Let’s start with a bunch of links to articles and then we’ll give you some quotes. There have been several Big Sites (even though we’re a large site with a large readership, these sites are Even Bigger) that have explored FCAT testing and I’ll just link you to them here.

Most definitely watch the PC Perspective video. You’ll see exactly what we saw – that FCAT benchmarking really only does one Important Thing: it tells us that CrossfireX is broken in its current form. It leads to highly increased FPS numbers due to “runt” frames. Indeed, Anandtech scored an exclusive with regard to AMD’s position on frame times and why there is a problem with CrossfireX. They are aware of and are actively pursuing a solution to their problem.

So if that’s the only problem, why spend thousands of dollars and countless hours of testing and graphing on a problem that only exists in CrossfireX. SLI is fine (which is likely why NVIDIA is pushing FCAT out right now) and single GPUs are fine. We’ve done, what, two CrossfireX reviews this year? We’ll probably get another dual GPU AMD card at some point too, I suppose. However, our main bread and butter is single card reviews up and down both AMD and NVIDIA’s product lines. Single card results have no problem. There is nothing to see here folks (with regard to single card and even SLI).

One of my favorite takes from people with FCAT capability was from Guru3D.

The reality is that this one translates in a small visible glitch for a fraction of a second on screen that you’ll hardly even notice. In fact only with the trained eye you can notice these things.

This is my message to you guys, frametime measurements have been taken out of proportion. What you see in these charts sometimes looks alarming with massive spike, whilst your average end-user will never ever even see it on screen.

Their article specific to FCAT benchmarking also had this to say:

Again I like to make the following statement strong, stutters always have been part of graphics cards. So I do plea… how important is it really? How much are you bothered by something you probably hardly even notice? I mean if you play a game and every now and then you see a small glitch, a stutter for a fraction of a second… does that ruin your game experience? We think not, and as such the stutter/glitches issues are a bit blown out of proportion with everybody jumping onto it. For multi-GPU gaming it is different though, if you pay attention to micro-stuttering, you can see it. But the question again remains the same, how many of you are actually bothered by it. Sure, if you raise enough awareness then people will revolt and address the issue continuously. But we do like to ask you to look at things in perspective.

After reading the entire Guru3D conclusion, my takeaway (paraphrased to be as short as possible) was basically, ‘Yep, we’re going to run FCAT testing because we have the tools, but it’s kind of pointless anyway unless there are Big Problems that you can see on the monitor. Most of this stuff you can’t even see.’

Tech Report did some great work and ran comparisons of FRAPS vs. FCAT and the results are telling. These were from benchmarking Skyrim in various configurations. All four graphs are courtesy Tech Report from this page.

GTX 680 Single GPU - Courtesy Tech Report

GTX 680 Single GPU – Courtesy Tech Report

Skyrim GTX 680 SLI - Courtesy Tech Report

GTX 680 SLI – Courtesy Tech Report

HD 7970 Single GPU - Courtesy Tech Report

HD 7970 Single GPU – Courtesy Tech Report

HD 7970 CrossfireX - Courtesy Tech Report

HD 7970 CrossfireX – Courtesy Tech Report

As you can see, while the results are different, they also follow each other very closely, especially in single GPU and SLI. The only problem is CrossfireX, and even as measured with FRAPS vs. FCAT, the graphs follow the same trend. While the two tools are measuring at two different places in the pipeline, they both look to be equally good at telling when you’re going to get Bad Frame Times, even with CrossfireX. The data is tighter with FCAT, but the trends are still there and still evident, with large problems shown via FRAPS.

TechPowerUP! is also a Big Site and they aren’t running FCAT testing either. Their reviewer cadaveca started a forum thread on the subject and was very blunt and straightforward. While he was obviously expressing strong opinions about CrossfireX, he has some good insights in the thread, such as:

The real unfortunate part is the cost in testing such things is still prohibitive for nearly everyone.

and, when another person mentioned reviewers would have “massive” work to do, he said

Nope. Just Ryan @ PCPER does. His new job is verifying that drivers work properly (whatever “properly” means). And as long as they are, every other reviewer can just keep on doing what they do.

…which is precisely what we intend to do.

So that is why we feel FCAT is not worth the investment and time for our site at this time. In no particular order, our main problems in a nutshell are:

  • It measures the very nearly imperceptible.
  • The only actual, real, tangible problem with frame times vs. our tried and true FPS measurement right now are AMD GPUs in CrossfireX.
  • With CFX being the sole exception, FPS measurements are accurate and still tell the whole story for single GPUs from both AMD and NVIDIA and multiple cards in SLI from NVIDIA.
  • AMD knows about this problem and is actively working to fix it. Once they do that, the whole reason for FCAT testing goes out the window.

From our point of view, pointing out a known problem that FRAPS can already show us when it’s a Big, Noticeable Problem using expensive hardware and lots of time is not a smart use of our resources. We would rather be churning out more reviews with our limited time and covering more hardware that you want to see. In every single case outside CFX (which isn’t a big portion of our reviews at all), we tend to agree with Guru3D that this entire frame time brouhaha is way overblown. Why measure what is mostly imperceptible and invest the time and money in it when you can’t even see it?

What we WILL do is continue to bring you the solid, thorough reviews we always have. We will continue to test and play games with the cards we receive extensively and watch the games as they come across our monitors. If there is a perceptible problem, we will let you know about it. Let me say that again, if it can be perceived by our human eyes, we will absolutely let you know whether there is a stuttering problem in our testing.

If, after all the GPUs we’ve reviewed over several years, we can’t notice something going on, we doubt you will either. Therein lies the crux of our problem with the entire FCAT testing kerfuffle. If we don’t notice it and you don’t notice it and assuming AMD fixes their CrossfireX runt frames problem like they should, isn’t it all just much ado about nothing?

– Jeremy Vaughan (hokiealumnus)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Discussion
  1. pwnmachine
    I was watching an interview with an Nvidia analyst on PcPer(a site which really backs fcat) and it is just dripping with bias. I understand journalists can have preffered hardware but PcPer is so blatant with its love for Nvidia they pretty much lose all credibility imo.

    As I said before this is definitely an issue that needs to be adressed but there are two sides to the story...


    yes, where people go crazy with preferred treatment, I think whatever they say turn into bs as well... I think for this issue, it is best to give it 2 months, then we come back and see if AMD fixed anything.

    I previously had a poor experience with how long it took for them to bring out Catalyst 12.0, so I feel they will need at least 2 months to fix.
    bluezero5
    okay, in all fairness, that is true. AMD admit the problem with crossfire, but at the same time promotes 7990 as legit better than GTX690 on FPS... when you say it like that, it does make AMD look bad. :)

    but they will fix it. I believe the matter is large enough for them to NOT ignore it... give them 2 months. :) Hopefully I won't have to eat my own words. :chair:


    I also have faith into AMD, they will fix this as soon as they can or they will simply loose alot of costumer. Me being the first first ... i will not buy an AMD card until this is fixed.
    trueblack
    ideally, you will expect them to have fixed them before promoting 7990.. but they choose the otherwise..


    okay, in all fairness, that is true. AMD admit the problem with crossfire, but at the same time promotes 7990 as legit better than GTX690 on FPS... when you say it like that, it does make AMD look bad. :)

    but they will fix it. I believe the matter is large enough for them to NOT ignore it... give them 2 months. :) Hopefully I won't have to eat my own words. :chair:
    I was watching an interview with an Nvidia analyst on PcPer(a site which really backs fcat) and it is just dripping with bias. I understand journalists can have preffered hardware but PcPer is so blatant with its love for Nvidia they pretty much lose all credibility imo.

    As I said before this is definitely an issue that needs to be adressed but there are two sides to the story...
    manu2b
    @TrueBlack.

    Don't mind my sig, I am no AMD fanboy (nVidia/Intel either, lol!).

    I've experienced xFire once, with a couple of 5830's, one year ago, before getting a GTX 580.

    These were no high end cards, but I can say that performance increase was obvious: you can't say it's a scam.

    I also hardly noticed any micro stuttering (most was on BF3).

    Sure, I am not a big gamer, casual at most, but when I moved from the dual 5830's to the GTX580, I roughly noticed any difference in games (FPS were roughly the same between the 2 setups, let's say 10% more in favour of the nVidia card).


    do pardon my harsh words previously. was just trying to get a point across.

    I will confess I favor nvidia over amd. so maybe that explains a little too.

    I have watched some youtube comparison video of microstuttering, and to me, I think the effect.. is small.. not unnoticeable when pointed out, but probably won't be a huge effect on gaming experience.

    that being said.

    the current performance that 7990 'use to promote sales on', is all about FPS, and since 7990 is a crossfire, and it is not fixed, in a way, that "-fps-" is a fake FPS, as the increased "fps" doesn't even help the animation, incase, harmes it slightly. and same with the benching results. the 'improved' FPS from crossfire, is in fact from poor artifacts, and the above posts before us just showed that the actual gaming observed FPS is actually.. same as non-crossfire.. I just felt people should know. and I would like to see AMD fix this as soon as possible. (ideally, you will expect them to have fixed them before promoting 7990.. but they choose the otherwise.. which is probably one of the reasons why I choose to use stronger wordings.)

    hope you didn't mind.
    @TrueBlack.

    Don't mind my sig, I am no AMD fanboy (nVidia/Intel either, lol!).

    I've experienced xFire once, with a couple of 5830's, one year ago, before getting a GTX 580.

    These were no high end cards, but I can say that performance increase was obvious: you can't say it's a scam.

    I also hardly noticed any micro stuttering (most was on BF3).

    Sure, I am not a big gamer, casual at most, but when I moved from the dual 5830's to the GTX580, I roughly noticed any difference in games (FPS were roughly the same between the 2 setups, let's say 10% more in favour of the nVidia card).
    hokiealumnus
    Did you click the link just two posts ago showing FCAT performance on the new drivers?


    yes indeed I have. thanks for showing the link.

    I am eager to see them post it soon, and not let the matter drop... (they SAY they fixed it, but it is not 'real' on the consumer level yet.)

    what I meant previously was.. if this didn't hit the surface, AMD would have never gotten the intention to fix it...

    maybe that's why nvidia brought this up.
    wait.

    How is this not a big deal?

    the above shows that AMD CrossFire is a 'scam' with increased FrameRates being no more than an artifact.

    For all we know, until CrossFire is 'proven fixed' (which we have no first hand evidence) AMD crossfire technology actually 'doesn't exist'. While AMD say they have Alpha drivers to fix it, it is all 'words' this moment. Anyone that invested in Crossfire technology is actually getting 'no better' frame sequencing than a single card, and ALL benching results on Crossfire is also 'a scam', for all we know, AMD could be inserting runt frames everywhere to increase "FPS" on their products. (meaning, all "benching results" from AMD cards can have a GOOD portion of their results being artifacts")... I call that a "major scam".

    I mean, a decent driver programmer can 'insert runt frames' at near no loss of GPU performance, and the so called 120 FPS is actually a 100FPS with 20% runts.

    I am glad this is brought into the light.

    The above claim is LEGIT, and should be known to all. While the test is too expensive, and might not be worth investing into, but until AMD -fixes- this, someone should monitor this, and warns all consumers from investing into AMD crossfire technology.

    doing this will make this website more important to people who cares what they money goes into.

    for example, the 7990 as of right now, is a outright highway robbery.
    hokiealumnus
    Aye, that video was linked from our article.

    I agree it's a very good thing that this came out. It discovered an issue and AMD is already on an Alpha driver that fixes it (with good results).

    If this piece came across as indicating any of us thought it was a bad thing that people did this, then that was not the intended impression. We're basically saying that AMD is fixing it (almost done fixing it actually) and once that's done, tried and true testing methods are back to being 100% reliable. We're not trying to minimize the importance of the discovery, just basically saying why we feel it's not worth investing in this type of system for our site as we go down the road.


    :thup:

    yup, as soon as AMD fixed CFX then this problem is outta the window!

    lol, didn't know the video is linked in the article already. thanks for pointing out!
    Aye, that video was linked from our article.

    I agree it's a very good thing that this came out. It discovered an issue and AMD is already on an Alpha driver that fixes it (with good results).

    If this piece came across as indicating any of us thought it was a bad thing that people did this, then that was not the intended impression. We're basically saying that AMD is fixing it (almost done fixing it actually) and once that's done, tried and true testing methods are back to being 100% reliable. We're not trying to minimize the importance of the discovery, just basically saying why we feel it's not worth investing in this type of system for our site as we go down the road.
    wingman99
    Looks llike a good way to test.:cool::popcorn:


    well, the result tells one thing. Crossfire is broken til they fix it.

    it is a fact for now, that crossfire 'doesn't help' animation, in fact, makes it a little worse by inserting grunts in the animation sequence. My lab mates have talked about this since a long time ago, and it is one of the reason why my machine is a GTX680 3xSLI + 7970 and not the other way around.

    I was able to observe the stuttering myself, and was able to see a clear difference in graphics properties. the stutter cannot be 'observed' so to say, but it results in the overall animation being a lot more 'fake'. Not sure how to say it, but when you have 2 machines side by side, one is crossfire, one is in SLI, and I was asked 'which feels better?', I was able to call it out 100% of the time.

    I have 6 associates in the lab, and of the 7 of us, 2 see no difference, so I must say this is a very personal 'perception' thing. :)

    for AMD cards, for now, 1 GPU is about all you need, whatever FPS you get extra with it with crossfire, actually harms animation. (great for benching though! just not for the eye)

    Here's a video explaining.. I found this recently. :)

    http://www.youtube.com/watch?feature=player_embedded&v=CsHuPxX8ZzQ
    this is new and interesting to me.

    reading all those references next. looks like AMD doesn't like this cause it clearly puts nvidia in front only. But I suppose I need to see for myself..

    are there any videos comparing this effect mentioned?
    Another thing that many of the "reviews" fail to mention that almost all issues with microstutter can be remedied via software implementation such as RadeonPro.

    Although it is still an issue that needs to be addressed on a driver level there are still plenty of ways to get around this whole misconception that micro-stuttering is a problem that can't be alleviated.

    Great article :thup: