Bench vs. System Testing: The Pros and Cons

Comments relative to watercooling component testing – Bill Adams

It is perhaps timely and appropriate to consider the respective merits, and deficiencies, of the principal methods of testing CPU watercooling components; System Testing and Bench Testing. And the word component defines the crux of the issue.

Obviously (I hope) no one would think of testing a watercooling system kit piece by piece: the manufacturer/reseller assembled it presumably to ‘balance’ the components’ performance capabilities, and that is how it should be tested to evaluate it – as it was intended to be used. But where an individual watercooling component is sold separately, it is somewhat more difficult to establish it’s performance; given the TOTAL absence of such data from the manufacturers themselves.

So the question can be re-stated: Is it possible to characterize the performance capability of a watercooling component utilizing a ‘system test’ setup?
No, or poorly at best; and the following is an explanation of the relative capabilities of system vs. bench testing for watercooling components.

Some definitions are in order to understand how certain terms are being used:

• ‘System Testing’ is here defined as ‘measuring’ a component’s performance while installed in a (real or reasonable approximation of a) functional computer system.
• ‘Bench Testing’ is defined as the insertion of a component into a test loop where all of the associated variables can be independently controlled, measured, and recorded.

{mospagebreak}

Bill Adams

With ‘System Testing’ the CPU is used, with one of a variety of software programs intended to maximally load it, as the heat source; while the other parts of the watercooling system are selected by the tester and remain unchanged throughout the test series. Presumably only one component of the system is changed, being substituted sequentially for a different model.

The effect of such substitutions are ‘measured’ by comparing the change to the net effect of the total system (with the changed component) on the CPU’s temperature as compared with the original component. Such is the intent.

How well does ‘System Testing’ work?

More to the point, how well can ‘System Testing’ work?

An evaluation of the expected accuracy is not encouraging, and even the best possible accuracy is little better. There are well known ‘problems’ with the CPU software’s loading characteristics, in addition to the generated heat (power) characterization.

The temperature measurement capability of mobos is notoriously inaccurate, and the advent of on-board diodes has simply changed one set of ‘problems’ for another of (potentially) smaller magnitude. Some testers have added thermocouples or thermistors to the coolant circuit and adjacent to the CPU.

And the preceding is what can be ‘measured’; what is not (normally) measured is the coolant flow rate which is the crucial parameter for waterblocks and pumps, and to a lesser extent radiators in conjunction with their airflow. Variations in the waterblock’s mounting and TIM application can have huge effects; but seldom do ‘System Testers’ quantify this variability, despite such being a very simple statistical analysis.

The quality of any measurement is nowadays quantified with an associated ‘measurement uncertainty value’, which is derived from the measurement system’s components’ characteristics; and strongly influenced by the calibration status of the individual pieces.

Any ‘good’ (as in valid) measurement will be made with a system having some form of (even DIY) calibration AND a definition of the associated uncertainty value. ‘System Testing’ setups cannot meet this requirement. This does not mean that there is not a test result value, only that it is not known how ‘good’ (if at all) that value might be.

But the insurmountable limitation with the ‘System Testing’ of individual watercooling components is the interactive effect due to the other parts of the cooling circuit. In the instance of waterblocks, their differing flow resistances will cause each, when installed in the cooling circuit, to cause the pump to operate at a different point on its P-Q curve; and the resulting flow rate difference will in turn impact (slightly) the cooling done by the radiator.

The actual flow through the waterblock will be unknown, and filling a bucket will not accurately replicate the flow conditions in a closed system.

Ignoring (for discussion) all of the foregoing problems, assuming that the test ‘results’ are accurate even; what then is known about the component’s capability? Performance is NOT described by a single value, given that a waterblock, for example, can be used with different pumps, radiators, hose sizes and fittings.

The performance is described by a curve, not a single datum; and the worst part is that one has not a clue as to where on the curve that datum is located when obtained by ‘System Testing’.

So why is “System Testing’ utilized for watercooling components?

• Because all the needed equipment is at hand (it’s cheap)
• Because such testing is easy for readers to relate to (Hey, I’ve got a Rio pump too)
• Because it can be done quickly (it’s cheap)
• Because it does not intellectually tax the reviewer, or the reader
• Did I mention it was cheaper?
• But if by chance one has exactly the same setup, the results are . . . well, . . . there’s still that calibration problem

In summation, ‘System Testing’ can be said to, if carefully controlled as with any type of testing, yield comparative results applicable to the system configuration utilized. It will not yield a component characterization applicable to other systems unless some quantification can be made of the flow rate. As such systems are rarely – or poorly – calibrated, the accuracy of actual values is suspect.
{mospagebreak}

Bill Adams

‘Bench Testing’ necessitates the assembly of a flow (typically) circuit with the instrumentation needed to accurately measure those parameters of interest. Such is a testing activity as contrasted with a computer activity; the requisite knowledge base is different as are the activities themselves.

Continuing with the example of the waterblock, the variables of interest are the heat load, temperatures, flow rate, and pressure drop. Related variables that must be quantified and/or controlled are the clamping force, secondary heat path losses, and the testing environment itself.

The key aspect of the test bench design is the independent control of the different variables. It is the measurement of the effect of changing one variable while holding others constant that enables the definition of a performance curve for that parameter. Why is such necessary or useful?

With access to the relevant performance curves for the waterblock, radiator, and pump it is possible to design a watercooling system, as contrasted with the buy-try-replace method of system development. In industry all systems are designed in this manner. Too complex for some? OK, there are plenty of show-and-tell reviews on the ‘net for them too.

A coolant flow bench might look like this – mine did several weeks ago:

And the following is a listing of that equipment (only) used for waterblock testing:

• Cooling / temperature control
  • Haake A82 Recirculating Chiller (to -40, +130°C with suitable liquid)
  • EDC MV106 Precision DC Millivolt Source (for remote control of chiller)
• Waterblock coolant circuit
  • Little Giant 3E-12NYS pump (to 6gpm, 9psi; larger pumps available)
  • Alfa Laval CB12-36 brazed plate heat exchanger (also submerged in the chiller bath)
  • 3 liter reservoir with low coolant level and no-flow relay for pump and die heater cutoff
  • ¾ and ½in. needle, and ½in. ball valves for flow control
  • 50µ ss filter
• Waterblock coolant instrumentation
  • Foxboro ICT pressure transducer and Newport digital gauge (for line pressure)
  • Foxboro 823DP differential pressure and Digitek digital gauge (for pressure drop)
  • Fluke 2180A Digital (RTD) Thermometer with multipoint RTD selector
  • 2 class’ A’ 4-wire RTDs to measure coolant inlet and outlet temperatures
  • Danfoss MAG 5000 flowmeter with MAG 1100 ¼in. magnetic flowtube (larger available)
• Waterblock heating circuit
  • Custom fabricated die heat simulator with 120V 400W nichrome wire cartridge heater (see here, now somewhat modified from that shown)
  • 2 class ‘A’ 0.065in. dia 4-wire RTDs, in the die simulator and waterblock baseplate
  • Midwestern HWD 60V-3A DC regulated power supply
  • Fincor RA 125V-3.2A DC regulated power supply
  • HP 3490A dmm to measure heater voltage (in mV at the heater element)
  • Fluke 8010A dmm to measure heater current (in mA, corrected for meter ‘burden’)
• Additional instrumentation
  • Fluke 2190A Digital (TC) Thermometer with multiple multipoint TC selectors (to measure heat dispersion across baseplate with arrays of contact mounted TCs)
  • Chatillon CFG10 Digital Force Gauge to measure spring load/deflection rates (larger available for use with Hammer mounting systems)

For the “Bench Testing’ of radiators, airflow and pressure instruments are substituted for the die heat simulator and its related pieces.

Not listed are the distressingly numerous pieces of calibration related equipment, instrument power supplies, or the still incomplete GPIB/data acquisition/software analysis system ‘in the works’. This list is the reason there is a virtual absence of watercooling component bench test data; the setup just costs far more than can be justified (by any rational person).

So what can ‘Bench Testing’ reveal about a waterblocks performance capability?
{mospagebreak}

Bill Adams

Some examples from a pending article (here soon!) on “A Waterblock Testing Methodology”:

The following examples of “Bench Testing” results for watercooling radiators are excerpted from a previous article on “Radiator Heat Dissipation Testing” (the revised graphs of which are here):

It is not the intent of this article to describe the many types of data that can be obtained or the information that may be gleaned through its presentation; but the above examples clearly show the differences from the bar charts of a ‘best list’ derived from ‘System Testing’.

And it must be observed that both pump and fan manufacturers provide ‘Bench Testing’ results as a matter of course. Some examples of P-Q curves:

For Iwaki pumps:

For a Nidec TA450 Model A31256 (12VDC) fan:

All this is wonderful perhaps, but then –

Why is ‘Bench Testing’ not in widespread use to characterize watercooling components?

• Because it requires extensive equipment (it’s expensive);
• Because the results can sometimes be complex, requiring a technical grasp of the parameter’s influence to understand;
• Because it requires a qualified technician or engineer to conduct (which is expensive);
• Because there is generally no ‘best’ WINNER;
• Because its very time consuming (yea, more moola);
• Because the manufacturers see no incentive (higher sales) to justify spending the money.

In summation, ‘Bench Testing’ can develop accurate performance characterizations for watercooling components; depending of course on the resolution and calibration of the instruments and adherence to effective testing procedures.

These results can be of use to individuals actually designing a system, or seeking to understand why their system does not perform as expected. Comparative ‘Bench Testing’ results can be directly utilized to identify a candidate component to improve a particular system characteristic (aka remove a ‘bottleneck’); and are really the only type of data useful to a waterblock designer.

System Testing = Show-and-Tell, perhaps a bit more with extra effort
Bench Testing = “Just the facts Ma’am, just the facts”, with a LOT of effort
Zeros are the most expensive numbers.

Be Cool!

Bill Adams