Our Take On NVIDIA’s FCAT GPU Testing

If you follow the GPU reviewing world, you have probably seen several articles pop up regarding a new GPU testing method using a tool called FCAT. Our editing team has discussed this extensively and we believe we’ve come to the right decision, but are of course willing to listen to our readers. Please have a look at this article, weigh the options and chime in with your two cents.

FCAT. Frame Capture Analysis Tool. In itself, it is a moderately open-source piece of software released by NVIDIA. A few select sites received copies of the software (though I’m sure they would make it available to us if we asked) and tools to use in measuring frame times.

Before we get into that tool and how it works, first we need to delve into what frame time is.

Frame Times – A Short Explanation

Before getting started, to save some space, I’m going to be making Statements with a capital S that are based on reading I’ve done. Where applicable, there will be links to the source where I read that. Rather than completely regurgitate what was said at those sources, I’ll just link them to you to verify what we’re saying.

Backing up to a year or so ago, Tech Report pioneered measuring frame time via FRAPS. Their original article, “Inside the second: A new look at game benchmarking“, went over why they felt it was a good thing to include. Their tool of choice was FRAPS, the popular FPS (and frame time) measuring software. Tech Report was the pioneer in the field, but not many review sites jumped on the bandwagon. What’s even more interesting is that Overclockers writer Kyle Lunau discovered this with FRAPS years ago, in an article published in May of 2009 – Micro Stutter: The dark secret of SLI and Crossfire.

Frame time is the time it takes your system to render one frame. You are used to FPS, frames per second. Frame time is how long within that second it took for the system to produce that frame. In our review of the GTX 650 Ti BOOST we outlaid a quick and easy reference as to what those milliseconds mean:

If you’re unfamiliar with frame times, it is how long the GPU takes to render a frame. FPS is a good measurement to tell how many frames are being produced per second on average, but as we all know, FPS can vary as you’re gaming. That’s because your GPU often takes longer than that average to produce the frame. The simplest explanation I’ve seen came from SKYMTL, who writes for Hardware Canucks from his research:

• Below 20ms: unnoticeable
• Rapid, cyclical fluctuations above 25ms: you’ll notice it
• Sudden “spikes” (non-cyclical) above 40ms: you’ll notice it
• Grey area between 21ms and 25ms: debatable. Some will notice it while others won’t.

So now you know what frame time is. Let’s move on and discuss what that has meant over the past year and a half.

Tech Report and FRAPS

Tech Report’s work, in a way, led to FCAT as it is today. You see, when they were measuring frame times, they noticed a massive problem with multiple GPU setups. While they were producing plenty of frames per second, the time it took to render frames in between other frames was huge, leading to stuttering in game that was obvious and noticeable. That’s when eyebrows started to be raised regarding this kind of testing.

FRAPS, while it is a great tool, especially when it’s free and available to the masses, is not a perfect tool. It measures frames and frame times in between the game and the GPU, the only place where it can. Here’s the handy-dandy graphic everyone’s using:

As you can see, there is a lot that happens after FRAPS that affects the final outcome. Translation: FRAPS can tell you a generally accurate average FPS number and its frame time measurements can tell you when there is a Big Problem. It can also give misleading information with regard to frame times.

AMD dislikes FRAPS as a tool. They gave Anandtech’s Ryan Smith an exclusive interview and outlook on their opinion with regard to FRAPS measurements and their driver roadmap. In it, basically, they admit to a problem with CrossfireX, say they’re working on drivers to address the problem and hope to come out with them sometime this summer. Their takeaway with FRAPS is that it shows problems where none exist, and amplifies them where they do.

Interestingly, NVIDIA agrees with AMD about FRAPS. They dislike its use as a frame time measuring tool for the very same reasons.

Enter FCAT

FCAT is software that analyzes frames captured while a game is running. It also has a component on the gaming machine that inserts colored bars on the side of the game’s frames as it is being rendered. When the overlay is running (similar to FRAPS’ FPS overlay), it looks something like this.

What you then need to do is run your DVI output off the GPU to a splitter. That splitter will then send out two signals – one to a monitor you’re using to run the game on your test PC, and the other to another PC set up to capture all of the frames coming from the gaming PC.

Once those frames are captured on the capture PC, you’re left with a massive file (we’re talking in the 20 GB range for 15 seconds of capturing). FCAT then analyzes the recorded frames and gives you a lot of data to parse via script. The parsing script gives you graphs that produce a frame time chart very similar to that you see produced by FRAPS, except rather than having been monitored from the game output, it has been recorded directly from the GPU itself on the business-end.

Herein lies the three practicality problems from a reviewer’s perspective and that of our site specifically:

1. The Capture PC

Number one is hardware. The capture PC is not cheap. Not in the slightest. First you have to have your standard CPU, motherboard, RAM, PSU, etc. That’s not too big of a deal. Call it $1,000 for a solid offering. Then you get to the expensive part. We’ve priced some capture cards and the one being used by most of the sites costs $1,800. By itself.

Then, as mentioned, 15 seconds from that capture card gets you over a gigabyte per second. So in order to capture the video, you’ll need a storage system capable of significantly faster rates than the fastest individual SSD on the market, and it can’t be small either. You’re talking $1,500-$2,000 on storage alone.

All told, using rough guesstimates, you’re looking at up to $5,000 on the capture PC. Now, we’re a review site. We are friendly with enough manufacturers that we could probably procure everything but the capture card, so our actual outlay would probably be $1,800. We just wanted you to see what sort of costs a layperson would need to lay out for a capture PC such that these other sites are using.

2. Time

As most of you know, we are a site run by volunteers. We have zero full-time employees writing for Overclockers.com. Our team is very dedicated and the work they do has astounded me over the past four years I’ve written for the site. That they are able to do what full-time, paid writers at other sites do on a volunteer basis is a feat in and of itself.

Analyzing FCAT data takes a very significant amount of time. Look at PC Perspective’s graphs below. Their excel work would take many hours all on its own and they’d do it for every game in every GPU review. Frankly, on a volunteer basis, that amount of extra input is a daunting prospect.

3. Multiple Reviewers

We have more than five people that write reviews regularly for this site and all of us are equipped to and can write reviews on any piece of hardware because we have standardized our test systems such that they are independent and one GPU reviewed by Lvcoyote or Bobnova on the west coast can be compared to one EarthDog or I review on the east coast. We do that because we’re community driven, always have been and always plan to be. No one person reviews all of one thing. Of course, all told, five capture systems would retail around $25,000, so that’s not an option for anybody.

The Real Reason Why FCAT Isn’t An Option

But let’s put all of those practicalities aside. Completely ignore the money, the time and the multiple reviewers. What if we could do all of that? Would we? To be blunt, the answer is no. After extensive discussion among our editing team, what we’re here to tell you today is that this is much ado about very little. Not nothing, but not much either.

Let’s start with a bunch of links to articles and then we’ll give you some quotes. There have been several Big Sites (even though we’re a large site with a large readership, these sites are Even Bigger) that have explored FCAT testing and I’ll just link you to them here.

• Anandtech: FCAT: The Evolution of Frame Interval Benchmarking, Part 1
• Tech Report: Inside the second with Nvidia’s frame capture tools
• Guru3D: An introduction to FCAT benchmarking
• Tom’s Hardware: Challenging FPS: Testing SLI And CrossFire Using Video Capture
• PC Perspective: Frame Rating Dissected: Full Details on Capture-based Graphics Performance Testing. PCPer also did a great video going over everything that is worth a watch.

Most definitely watch the PC Perspective video. You’ll see exactly what we saw – that FCAT benchmarking really only does one Important Thing: it tells us that CrossfireX is broken in its current form. It leads to highly increased FPS numbers due to “runt” frames. Indeed, Anandtech scored an exclusive with regard to AMD’s position on frame times and why there is a problem with CrossfireX. They are aware of and are actively pursuing a solution to their problem.

So if that’s the only problem, why spend thousands of dollars and countless hours of testing and graphing on a problem that only exists in CrossfireX. SLI is fine (which is likely why NVIDIA is pushing FCAT out right now) and single GPUs are fine. We’ve done, what, two CrossfireX reviews this year? We’ll probably get another dual GPU AMD card at some point too, I suppose. However, our main bread and butter is single card reviews up and down both AMD and NVIDIA’s product lines. Single card results have no problem. There is nothing to see here folks (with regard to single card and even SLI).

One of my favorite takes from people with FCAT capability was from Guru3D.

The reality is that this one translates in a small visible glitch for a fraction of a second on screen that you’ll hardly even notice. In fact only with the trained eye you can notice these things.

This is my message to you guys, frametime measurements have been taken out of proportion. What you see in these charts sometimes looks alarming with massive spike, whilst your average end-user will never ever even see it on screen.

Their article specific to FCAT benchmarking also had this to say:

Again I like to make the following statement strong, stutters always have been part of graphics cards. So I do plea… how important is it really? How much are you bothered by something you probably hardly even notice? I mean if you play a game and every now and then you see a small glitch, a stutter for a fraction of a second… does that ruin your game experience? We think not, and as such the stutter/glitches issues are a bit blown out of proportion with everybody jumping onto it. For multi-GPU gaming it is different though, if you pay attention to micro-stuttering, you can see it. But the question again remains the same, how many of you are actually bothered by it. Sure, if you raise enough awareness then people will revolt and address the issue continuously. But we do like to ask you to look at things in perspective.

After reading the entire Guru3D conclusion, my takeaway (paraphrased to be as short as possible) was basically, ‘Yep, we’re going to run FCAT testing because we have the tools, but it’s kind of pointless anyway unless there are Big Problems that you can see on the monitor. Most of this stuff you can’t even see.’

Tech Report did some great work and ran comparisons of FRAPS vs. FCAT and the results are telling. These were from benchmarking Skyrim in various configurations. All four graphs are courtesy Tech Report from this page.

As you can see, while the results are different, they also follow each other very closely, especially in single GPU and SLI. The only problem is CrossfireX, and even as measured with FRAPS vs. FCAT, the graphs follow the same trend. While the two tools are measuring at two different places in the pipeline, they both look to be equally good at telling when you’re going to get Bad Frame Times, even with CrossfireX. The data is tighter with FCAT, but the trends are still there and still evident, with large problems shown via FRAPS.

TechPowerUP! is also a Big Site and they aren’t running FCAT testing either. Their reviewer cadaveca started a forum thread on the subject and was very blunt and straightforward. While he was obviously expressing strong opinions about CrossfireX, he has some good insights in the thread, such as:

The real unfortunate part is the cost in testing such things is still prohibitive for nearly everyone.

and, when another person mentioned reviewers would have “massive” work to do, he said…

Nope. Just Ryan @ PCPER does. His new job is verifying that drivers work properly (whatever “properly” means). And as long as they are, every other reviewer can just keep on doing what they do.

…which is precisely what we intend to do.

So that is why we feel FCAT is not worth the investment and time for our site at this time. In no particular order, our main problems in a nutshell are:

• It measures the very nearly imperceptible.
• The only actual, real, tangible problem with frame times vs. our tried and true FPS measurement right now are AMD GPUs in CrossfireX.
• With CFX being the sole exception, FPS measurements are accurate and still tell the whole story for single GPUs from both AMD and NVIDIA and multiple cards in SLI from NVIDIA.
• AMD knows about this problem and is actively working to fix it. Once they do that, the whole reason for FCAT testing goes out the window.

From our point of view, pointing out a known problem that FRAPS can already show us when it’s a Big, Noticeable Problem using expensive hardware and lots of time is not a smart use of our resources. We would rather be churning out more reviews with our limited time and covering more hardware that you want to see. In every single case outside CFX (which isn’t a big portion of our reviews at all), we tend to agree with Guru3D that this entire frame time brouhaha is way overblown. Why measure what is mostly imperceptible and invest the time and money in it when you can’t even see it?

What we WILL do is continue to bring you the solid, thorough reviews we always have. We will continue to test and play games with the cards we receive extensively and watch the games as they come across our monitors. If there is a perceptible problem, we will let you know about it. Let me say that again, if it can be perceived by our human eyes, we will absolutely let you know whether there is a stuttering problem in our testing.

If, after all the GPUs we’ve reviewed over several years, we can’t notice something going on, we doubt you will either. Therein lies the crux of our problem with the entire FCAT testing kerfuffle. If we don’t notice it and you don’t notice it and assuming AMD fixes their CrossfireX runt frames problem like they should, isn’t it all just much ado about nothing?

– Jeremy Vaughan (hokiealumnus)