mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Hardware > GPU Computing

Reply
 
Thread Tools
Old 2013-06-03, 16:11   #1
Graff
 
Graff's Avatar
 
Jul 2006
USA (UT-5) via UK (UT)

22×59 Posts
Default final cudaThreadSynchronize failed

One of my GPU boxes has started spitting out errors when running
the self-test upon startup of mfaktc. Hanging of the system then
follows a few seconds later.

The system is running Ubuntu 12.04-2 LTS, the GPU is an EVGA
GTX 570 and the error is "final cudaThreadSynchronize failed".

Has anyone else had this error? Is it a sign of a failing/failed card?

Gareth
Graff is offline   Reply With Quote
Old 2013-06-03, 16:25   #2
firejuggler
 
firejuggler's Avatar
 
Apr 2010
Over the rainbow

22×3×13×17 Posts
Default

Are you Oc'ing your 570?
Your PSU is big enough?
If the air in your case is too ht, that might affect it, too. ( I know, this might not help much, but mechanical problem first)
After that, reinstall drivers, Cuda 4.2....
and after that... i'll let the expert do their deed.
firejuggler is offline   Reply With Quote
Old 2013-06-03, 21:22   #3
Graff
 
Graff's Avatar
 
Jul 2006
USA (UT-5) via UK (UT)

22·59 Posts
Default

Quote:
Originally Posted by firejuggler View Post
Are you Oc'ing your 570?
Your PSU is big enough?
If the air in your case is too ht, that might affect it, too. ( I know, this might not help much, but mechanical problem first)
After that, reinstall drivers, Cuda 4.2....
and after that... i'll let the expert do their deed.
No OC'ing. 750 W PSU. A small GPU in the same box works fine,
so the problem is unlikely to be with the drivers or CUDA.

Gareth
Graff is offline   Reply With Quote
Old 2013-06-03, 21:35   #4
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

100110111001112 Posts
Default

Quote:
Originally Posted by Graff View Post
No OC'ing. 750 W PSU. A small GPU in the same box works fine,
so the problem is unlikely to be with the drivers or CUDA.
Have you tried running the CUDALucas self test (several times if it initially passes), and/or Carl's CUDAmemtest?

I've recently found (being relatively new to actually running higher-end GPUs) that using a wide variety of tests can be very helpful.

Lastly, if the above two report errors, you might want to consider trying down-clocking. Many manufacturers seem to supply "kit" intended for "gamers" -- we who "compute" have much stricter requirements and expectations.

Last fiddled with by chalsall on 2013-06-03 at 21:37 Reason: s#and or#and/or#
chalsall is online now   Reply With Quote
Old 2013-06-04, 02:00   #5
kladner
 
kladner's Avatar
 
"Kieren"
Jul 2011
In My Own Galaxy!

2·3·1,693 Posts
Default

My experience with a 570 and a 460 running on a 750 W Bronze PSU was that it was drawing in the 690 W range (Kill-a-Watt measured). This was also with a Phenom II 1090T doing 6x P-1. When I switched to a Gold 1000 W supply the line draw dropped to about 660 W.

My point is that the combination was really loading a 750 W supply beyond the usual recommendations. I don't know if this is playing a part in your problems. I never had mfaktc errors, but did have the occasional BSOD. Haven't seen one of those in quite a while.
kladner is offline   Reply With Quote
Old 2013-06-14, 02:23   #6
Graff
 
Graff's Avatar
 
Jul 2006
USA (UT-5) via UK (UT)

3548 Posts
Default

Quote:
Originally Posted by kladner View Post
My experience with a 570 and a 460 running on a 750 W Bronze PSU was that it was drawing in the 690 W range (Kill-a-Watt measured). This was also with a Phenom II 1090T doing 6x P-1. When I switched to a Gold 1000 W supply the line draw dropped to about 660 W.

My point is that the combination was really loading a 750 W supply beyond the usual recommendations. I don't know if this is playing a part in your problems. I never had mfaktc errors, but did have the occasional BSOD. Haven't seen one of those in quite a while.
My other GPU is Quadro 600, which draws ~ 40 W. With the GT570
drawing 220 W and my CPU eating another ~ 95 W, 750 W looks to be
OK.

I ran cudamemtest. System crashed somewhere around test 6 or 7.
I caught the system crashing out of the corner of my eye while working
on another machine and didn't see exactly which test it was on (and which
GPU it was testing). Anyway, system refused to start up after this.
Removed the GT570, system came back up. Guess the card was
failing...

Guess it'll have to be another return.

Gareth
Graff is offline   Reply With Quote
Old 2013-06-14, 04:03   #7
kladner
 
kladner's Avatar
 
"Kieren"
Jul 2011
In My Own Galaxy!

2×3×1,693 Posts
Default

I agree that given those loads 750 W is plenty. Good luck on the RMA.
kladner is offline   Reply With Quote
Old 2013-08-16, 12:36   #8
Manpowre
 
"Svein Johansen"
May 2013
Norway

3×67 Posts
Default

Quote:
Originally Posted by Graff View Post
One of my GPU boxes has started spitting out errors when running
the self-test upon startup of mfaktc. Hanging of the system then
follows a few seconds later.

The system is running Ubuntu 12.04-2 LTS, the GPU is an EVGA
GTX 570 and the error is "final cudaThreadSynchronize failed".

Has anyone else had this error? Is it a sign of a failing/failed card?

Gareth
I installed driver 306.29 for the 590 boards, as the boards were crashing cuda every hour. Well, the driver did, as when I installed 306.29 the boards were rock stable.. now been running without crash for 2 days.
Manpowre is offline   Reply With Quote
Old 2013-08-16, 15:20   #9
kladner
 
kladner's Avatar
 
"Kieren"
Jul 2011
In My Own Galaxy!

2×3×1,693 Posts
Default

Quote:
Originally Posted by Manpowre View Post
I installed driver 306.29 for the 590 boards, as the boards were crashing cuda every hour. Well, the driver did, as when I installed 306.29 the boards were rock stable.. now been running without crash for 2 days.
Interesting- which were you running before? I currently have 314.22. Besides stability, which comes first, of course, did you see any performance difference?
kladner is offline   Reply With Quote
Old 2013-08-16, 18:53   #10
Manpowre
 
"Svein Johansen"
May 2013
Norway

3×67 Posts
Default

Quote:
Originally Posted by kladner View Post
Interesting- which were you running before? I currently have 314.22. Besides stability, which comes first, of course, did you see any performance difference?
I ran the latest beta driver, and latest WHQL driver that I could download a few days ago 326.41 and 320.49. both which crashed after an hour or a few hours. I tried to set power to "do not turn off monitor" but it didnt help. I also tried the grid parameter to 1.. didnt help.. only the 306.23 driver helped.

I got exactly the same performance with all drivers.

The correct driver I run on GTX 590 is 306.23 (not 29).. Im home now, so I could double check exactly the driver version. I googled this issue alot, and I saw people using cuda with 590 is using this specific driver.
Manpowre is offline   Reply With Quote
Old 2013-08-16, 21:40   #11
kladner
 
kladner's Avatar
 
"Kieren"
Jul 2011
In My Own Galaxy!

27AE16 Posts
Default

I'm glad you did not have more serious problems with the 32x.xx drivers. There are very many people swearing that those are card killers.
kladner is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Rain delays tennis final at Queen's Club davieddy Hobbies 0 2011-06-13 08:33
LLR final 3.8.4 Version is available! Jean Penné Software 5 2011-02-10 06:35
Mally -Final tribute devarajkandadai Math 0 2007-10-12 08:23
Carries too high final pass spamguy Software 0 2004-02-17 17:01
Glucas - too high final pass ??? bayanne Software 0 2003-10-22 07:12

All times are UTC. The time now is 23:04.


Sat Oct 23 23:04:16 UTC 2021 up 92 days, 17:33, 0 users, load averages: 1.26, 1.22, 1.13

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.