PSA: All RX5700/XT owners, remove your GPUs from SETI@home now

[gsrcrxsi]

This issue has been ongoing since the RX5700 and RX5700XT was released. OpenCL compute is broken on these GPUs (unclear at this time if it’s hardware or drivers related) and they produce invalid results.

The problem is that these RX5700s are cross validating their incorrect results with each other on occasion. If left unchecked this has serious implications for the integrity of the science database.

More information can be found here: https://setiathome.berkeley.edu/forum_thread.php?id=84508&sort_style=6&start=0

There are reports that these GPUs are causing issues with other projects as well. If you are running any of these GPUs at SETI@home, please remove it from the project immediately until a working driver update is confirmed.

 

[mackerel]

While I don't have one, I have seen similar reported for folding also. At least I believe there it fails early enough it doesn't return bad work.

I've asked similar at PrimeGrid if this is also a problem there or not, no response. I was debating getting a 5500 to play with.

 

[gsrcrxsi]

Yeah on SETI, the WUs complete processing, but with incorrect results. The vast majority of their results get invalidated, but in some cases the results from one RX5700 matches up with another RX5700 and that result gets validated.

On SETI the way the validation process works is that the same WU gets sent to 2 systems at random. Either assigned to GPU or CPU. When the results are returned from those 2 systems they get compared. If the results are different, the WU gets sent out to a third system. Whichever two systems match get the reward. So in the few cases where 2 RX5700s match up with the same wrong results, that gets validated.

It’s not a huge problem right now. But the more people that get these cards and add them to the project, the higher the probability that they will match up with each other. We’ve also been trying to get the project scientists to blanket ban these devices. Historically the project scientists aren’t very forthcoming with things going on in the background like this, but we have to assume that they aren’t banned yet due to the complexity or lack of resources to set it up.

There’s some effort being made by a few users to message each individual person with a 5700, but not everyone checks the SETI forums

 

[mackerel]

Same or very similar validation strategy at PrimeGrid. When there was a problem with nvidia drivers producing bad work a while back, they added a rule to say the suspected versions had to validate against a version that wasn't suspect. They couldn't false-verify in that scenario.

 

[gsrcrxsi]

We’ve proposed that kind of solution also. Make sure one app does not verify with the same app. But I really don’t know how complex that is to implement. Or the consequences that would have to validation time.

But the only way the project servers can easily figure out what kind of card was used is based on the app that was used. Information about exactly what card was used is harder to pinpoint on Multi-GPU systems. So say you have an RX5700 and an RX580. They both use the same app. The only way you’d know which card it was run on would be based on the stderr output, which I don’t think the system is parsing.

 

[gsrcrxsi]

The project has made some slight changes to the Windows application for AMD cards. AMD has also release new drivers that fix this issue. if you would like to run your Navi-based AMD card, please update to the newest drivers and you can process SETI@home again.

no update on Linux however. no new drivers, no updated applications. but I only ever saw 1 or 2 people with a Navi card running SETI under linux.