Overclockers’ Updated Video Card Testing Procedure

(UPDATED ARTICLE:  http://www.overclockers.com/overclockers-gpu-testing-prodcedures) – MattNo5ss (6/27/2013)

New Graphcs Card Testbed

New Graphcs Card Testbed

With the launch of Ivy Bridge, we’ve decided to migrate our video card testbed to the new platform, primarily because of its PCIe 3.0 specification. Along with the move, we’ve updated our test suite as well to make it more relevant to our readers that always like the latest and greatest titles. Older games like HAWX 2 and STALKER just won’t cut it any longer.

We had to make some compromises for cost-effectiveness so they aren’t ALL the latest $60 titles. However, they are all still modern games that require a solid graphics setup to look good. We’ve also expanded a little bit to include not only several graphics-intensive first person shooters, but a driving game and (completely new to us) a strategy game.

As you probably know, we have reviewers across the country, literally from the east coast (NY, MD, NC) to the west (CA, WA), so this procedure is designed to maximize cross-reviewer capability. Whoever writes a review, if they use these settings, their results can be appropriately compared with all other reviewers’ results.

Testing Platform and Method

  • PCIe 3.0 motherboard (Z68 or Z77)
  • Intel i7 3770K
  • Tests are run with the CPU at 4.0GHz:
    • This is mostly due to ASUS’ complete inability to run a CPU completely at “stock”.
    • 4.0 GHz is also easily obtained by simply raising the multiplier with no other adjustments. Except for the very rare (so rare I’ve yet to see one) example, every 3770K should be able to run that at the stock Vcore, even if just using the stock cooler.
  • RAM should be set at DDR3-1866 / 9-9-9-24.
    • Here we’re aiming for a middle ground between overclockers with fast memory (DDR3-2133+) and your average gamer (who likely ends up running around DDR3-1600/1866).
  • The GPU should be run through the battery of tests at stock and overclocked to a 24/7 stable level, able to complete all game tests & benchmarks.
  • Benchmarks (where scores are in the multiple thousands or tens of thousands) should be run three times at stock and the result averaged.
  • Game tests at stock and overclocked can be run once, benchmarks when overclocked can be run once as well.
  • Lastly, crank down on both the CPU and GPU to show what sort of benchmark scores the combo can achieve. Take your pick of benches to run, preferably no less than two.

Game Tests and Settings

Game tests will all be run at 1080P resolution (1920×1080). For ease of cross-reviewer benchmarking, all games are to be run at their highest quality settings.

Aliens vs. Predator DX11 Benchmark

  • This can be run with AvP Tool or manually with the config file set properly.
    • Resolution: 1920×1080
    • Texture Quality: Very High
    • Shadow Quality: High
    • DX11 Features – Hardware Tessellation & Advanced Shadow Sampling both on.
    • Full Screen AA Samples: 4x
    • Antistropic Filtering: 16x
    • SSAO: On
AvP Result Log

AvP Result Log

Batman: Arkham City

  • Run the in-game benchmark using the following settings:
    • Resolution: 1920×1080
    • VSync: Off
    • AA: 8xMSAA
    • MVSS and HBAO
    • Tessellation: High
    • Detail Level: Extreme
    • Other Settings: Yes
    • PhysX: Off
Batman: Arkham City Result Screenshot

Batman: Arkham City Result Screenshot

Battlefield 3

  • This must be a manual run, using the following settings:
    • Resolution: 1920×1080
    • Graphics preset: Ultra (using all defaults included in that preset)
  • The method of testing is manual, but is remarkably consistent (we’re talking less than 1 FPS difference between runs in our tests).
  • Instructions:
    1. Turn on Fraps
    2. Start the first mission: Operation Semper Fidelis
    3. Begin recording FPS (F11 by default) after you pick up the pistol and are looking forward.
    4. Work through the level as normal.
    5. Stop recording FPS when you’re at the end of the level and the game takes over motion, hit F11 again when the guy puts his hands up.

Civilization V

  • This one is a bit more complicated, but here is the procedure nonetheless.
  • Go in-game (DX11) to set graphics settings as follows:
    • 1920×1080
    • 8x MSAA
    • VSync Off
    • High Detail Strategic View: Enabled
    • Other Settings: High
  • Find the config file in the “Sid Meier’s Civilization 5″ directory in your Documents folder and set “LoggingEnabled = 1″.
  • Create a shortcut on the desktop pointing to “CivilizationV_DX11.exe”.
  • Right click on the shortcut and go to Properties.
  • On the Shortcut tab, add “-benchmark LateGameView” to the end of the Target field.
  • Run Bench. It takes 3 minutes and 50 seconds to run. There’s no need to set the duration; I tested none set, 60s, and 180s. The time was 3:50 either way.
  • Get the LateGameViewBench.txt log file from the Logs directory within the ”Sid Meier’s Civilization 5″ directory.
  • There are three different scores in the log file: full render, no shadow render, and no render. The scores are the number of frames rendered in 60 seconds. The full render score is what we want to use to get FPS, and to get FPS divide the full render score by 60.
  • Record the result.
Civilization V Result Log

Civilization V Result Log

Dirt 3

  • Run the in-game benchmark using the following settings:
    • Resolution: 1920×1080
    • Anti-Aliasing: 8x MSAA
    • Settings: Everything Enabled and Maxed (High or Ultra depending on setting)
Dirt 3 Result Screenshot

Dirt 3 Result Screenshot

Metro 2033

  • Run with the included benchmark tool using the following settings:
    • Resolution: 1920×1080
    • DirectX Version: DirectX 11
    • Quality: Very High
    • Anti-Aliasing: MSAA x4
    • Texture Filtering: AF x16
    • PhysX Off
    • DOF (Depth of Field) On
    • Scene: Frontline
Metro 2033 Result Screenshot

Metro 2033 Result Screenshot

Synthetic Benchmarks

As mentioned, at stock these are run three times, then run once overclocked. Typically, these are the benches (plus Heaven DX11) that you’d run maxed out on the CPU and GPU.

3DMark03

  • Older bench, but still scales really well both with GPU power and the addition of more GPUs.
3DMark03 Screenshot

3DMark03 Screenshot

3DMark Vantage

  • A DX10 benchmark, this still scales decently with GPU power.
3DMark Vantage Screenshot

3DMark Vantage Screenshot

3DMark 11

  • Brand spanking new and DX11, this one is mostly GPU dependent still.
3DMark 11 Screenshot

3DMark 11 Screenshot

HWBot Heaven Benchmark Xtreme (DirectX 11)

  • Solid DX11 bench, this is more GPU dependent than pretty much all other benchmarks.
HWBot Heaven DX11 Screenshot

HWBot Heaven DX11 Screenshot

Added Bonuses

Both of these are performed with the GPU at stock. You can add overclocked numbers as well if you want; I don’t usually.

Temperature testing

  • Test the card both at idle and loaded.
  • To make the data worthwhile, we will need to monitor ambient temperature to normalize results.
  • Results will be normalized to 25 °C. If your ambient is 22 °C, add 3 °C to the recorded temperature of the GPU.
  • We will use HWBot Heaven DirectX 11 test as it doesn’t give the GPU a break. Use Afterburner or any constant monitoring program and record the peak temperature with the default fan profile.
  • As you can see, we’re getting away from FurMark. It’s just too unrealistic for us to use – either the temps will be too high relative to a “normal” load, or the card will throttle and throw readings off.

Power Consumption

  • Measure the system completely at idle.
  • We will record peak consumption in two situations.
  • The first is while running HWBot Heaven DX11.
  • The second is while running the 3DMark 11 Combined Physics test.
  • Again, we’re moving away from FurMark because of the potential for throttling, which will throw everything out of whack.

Multi-Monitor Setups

  • Most of our reviewers won’t be able to do this, but if you can run three monitors, run all of the game tests above preferably at 3 x 1080p (5760 x 1080) using the same quality settings as above. Hokie is the only reviewer set up to do triple-monitor setups currently, but if we get others with the capability, even better

    AMD Eyefinity / NVIDIA Surround Setup

    AMD Eyefinity / NVIDIA Surround Setup

The first review to use this method (I think) will be the ASUS GTX670 DirectCU II TOP that I’m working on now. MattNo5ss has kindly agreed to benchmark the EVGA GTX670 Superclocked using the method so we have a comparison card.

We will be re-benchmarking all of the cards we have, but it may take some time. You should see the comparison chart grow with each review until it is as built up as the one we had previously. Priority will be given to the closest competition to the card being reviewed. Thank you for your patience as we increase our comparison chart.

We’ll be using the same methodology we always have with our Approved / meh / FAIL award system. We only have three levels so they can be clearly defined, don’t rely on arbitrary numbering and -most importantly- to encourage you to read the reviews that go along with them.

While it’s pretty well set in stone at this point, with the hardware and games already purchased, feel free to chime in with questions or comments. We’re here to review this hardware for you, so let us know what you think!

- Jeremy Vaughan (hokiealumnus)

Tags: , , , , ,

17 Comments:

Lvcoyote's Avatar
Great write up Jeremy!

I can only think of one question..... the benchmarks that give FPS as the test results, some give max, minimum, and average. Are we always using average FPS for our chart data?

Thanks!
hokiealumnus's Avatar
Thanks! Average is what we'll use in recording the data for charting purposes. Maximums are pretty pointless, but minimums can bear mentioning (and/or graphing) if you're feeling ambitious or if they're really bad.
Lvcoyote's Avatar
LOL, gotcha! Thanks again for the great write-up. it will be very useful as we move forward!!
EarthDog's Avatar
Excellent write up!

I believe adding these games and removing others help keep OCF reviews relevant in today's gaming landscape. We have a little bit of everything going on here.
azuza001's Avatar
Defiantly a good uptick. Look forward to the reviews, I'm now torn between the 7850 or 670 if I save another 150$ and these reviews could help, a lot of the games I play are now in your test bed.

And minimum frames is defiantly worth mentioning over maximums, who cares if your max is 160 fps if whenever there is action you dip to the 20s? (not likely but still lol)
trekky's Avatar
sounds cool
just 1 thing why not add Bitcoin mining to the "Game Tests and Settings"
to see what the cards get when mining Bitcoins
hokiealumnus's Avatar
We're going for broad appeal and I'm not sure Bitcoin mining is used widely enough to really make it useful to a very large portion of our audience. Not a bad suggestion, I just don't think it would really appeal to enough people.
EarthDog's Avatar
Its honestly never been brought up...

We know AMD cards do better (at least that is what I recall) and part of the reason 5870's jumped in price. But the bitcoin thing, IMNSHO, has come and gone.
trekky's Avatar
well it was just a thought
bmwbaxter's Avatar
Nice write up, I think this is a very good spread of benchmarks/games

Just one question, some of the new boards will automatically enable lucid virtu, so is it to be left on or off? It can have a significant affect on some scores.
MattNo5ss's Avatar
Lucid MVP should be off. Not all boards will have it, and turning it off will give us a more direct comparison between cards. Then, for those who have Lucid MVP, they can expect performance at least as good as our results, with the possibility of better performance if they enable MVP.
bmwbaxter's Avatar
Thanks, I just wanted to make sure.
Creep-DK's Avatar
Must say i like the selection, and even more i like the 1920 x 1080 resolution, im guessing quite a few of us run that, most other sites do 1920 x 1200 .
Looking forward to reading future reviews
djscrew's Avatar
nice refresh, seems like a well thought out, balanced and up to date way to review games. I especially like the triple monitor addition for the e-peen dreams in all of us. I assume mostly it will serve as a way to educate on how much ram video cards really need, and when they need it.
PurePhire's Avatar
Nice Review. I have the Galaxy GTX670 I don't think i've even pushed it yet or even have it setup right. I see this is based on the Ivy Bridge Intel Mobo's.. Could anyone tell me if there is any AMD boards based on PCI-E 3.0?
Any info to get me started would be appreciated. Thanks in Advance.
I currently Have an Asus M5A97 mobo. DDR3 1600 Mem. AMD Phenom 2 960? X4
bmwbaxter's Avatar
no pci-e 3.0 on any AMD boards ATM. have to wait for a future CPU from them that supports it.
hokiealumnus's Avatar
Unfortunately not; the most advanced chipset to date for AMD uses the 990FX northbridge and it is still PCIe 2.0. That said, you won't be hurt by that very much, if at all.

Also, just to be clear; this was an overview of our new testing procedure, not a review of a GTX670...though we do have two of those: EVGA GTX670 SuperClocked & ASUS GTX670 DirectCU II TOP
Leave a Comment