CUDALucas (a.k.a. MaclucasFFTW/CUDA 2.3/CUFFTW)

[kriesel]

Some CUDA library versions in combination with some GPU models produce anomalously low timings at fftlength=65536k in -fftbench in CUDALucas including the latest available, v2.06beta May 5 2017 build. Then other run lengths not a power of two, get compared to that anomalously low value, and if their accurate iteration time exceeds the anomalous time, they are absent from the resulting fft file. See the example attached. Anomalous values may be 2 or 3 times too fast, or orders of magnitude too fast. By removing usefully fast non-power-of-two fft lengths from the fft file, the anomalous timing will increase run times unnecessarily for multiple ranges of exponents.

 Device              GeForce GTX 1070
Compatibility       6.1
clockRate (MHz)     1708
memClockRate (MHz)  4004

  fft    max exp  ms/iter
    1      22133   0.0668
    2      43633   0.0667
    4      85933   0.0686
    8     169409   0.0969
   16     333803   0.0974
   32     657719   0.0969
   50    1017889   0.1241
   64    1296011   0.1257
   72    1454273   0.1281
   80    1612249   0.1293
   96    1927129   0.1321
  112    2240863   0.1388
  120    2397383   0.1667
  128    2553659   0.1689
  144    2865601   0.1914
  256    5031737   0.3427
  512    9914521   0.6101
 1024   19535569   1.1422
 2048   38492887   2.2206
 4096   75846319   4.4940
 8192  149447533   9.5765
16384  294471259  19.5743
32768  580225813  40.1492
65536 1143276383   0.1921

Since other codes were derived from CUDALucas and use the same CUDA libraries, they may also be affected. Note also that there is a somewhat analogous behavior with CUDALucas threadbench at the high end of fftlength, and another with squaring threads=1024 in CUDALucas threadbench on some GPU models.

Multiple workarounds are available:

1. Don't benchmark that high. (Easiest and quickest.)
2. Use an unaffected CUDA library version / card combination.
or
3. Hand edit the fft file after logging the stdout output via redirection to a file during the benchmark run.

[storm5510]

I found the message below when I got up this morning. It has been running around five hours, more or less.

CUDALucas.cu(1989) : cudaSafeCall() Runtime API error 30: unknown error.
Resetting device and restarting from last checkpoint.

Using threads: square 32, splice 32.
CUDALucas.cu(1115) : cudaSafeCall() Runtime API error 46: all CUDA-capable devices are busy or unavailable.

This is from version 2.06 Beta. The GPU is a GTX-480 running driver set 384.76 under Windows 10 x64 v1703.

Note the 'square' and 'splice' values. If I used the default setttings, This event would occur in just a few minutes. The lower I made them, the longer it would take for this error to happen.

[kladner]

Originally Posted by storm5510

I found the message below when I got up this morning. It has been running around five hours, more or less.

CUDALucas.cu(1989) : cudaSafeCall() Runtime API error 30: unknown error.
Resetting device and restarting from last checkpoint.

Using threads: square 32, splice 32.
CUDALucas.cu(1115) : cudaSafeCall() Runtime API error 46: all CUDA-capable devices are busy or unavailable.

This is from version 2.06 Beta. The GPU is a GTX-480 running driver set 384.76 under Windows 10 x64 v1703.

Note the 'square' and 'splice' values. If I used the default setttings, This event would occur in just a few minutes. The lower I made them, the longer it would take for this error to happen.

This is a known problem with Compute Capability 2.0 cards. I think that these include the 470, 480, 570, 580 series. My GTX 460 is unaffected. It is a CC 2.1.
When running CuLu on these cards, folks often used batch files to automatically restart the program.

@echo off
Set count=0
Set program=CUDALucas
:loop
TITLE %program% Current Reset Count = %count%
Set /A count+=1
echo %count% >> log.txt
echo %count%
%program%.exe
GOTO loop

I also attached this file, but renamed it as .txt
It was generally thought that these timeout errors and restarts did not negatively affect the results. I was always nervous about this, and stopped running CuLu on my 5xx cards.

[storm5510]

@echo off
Set count=0
Set program=CUDALucas
:loop
TITLE %program% Current Reset Count = %count%
Set /A count+=1
echo %count% >> log.txt
echo %count%
%program%.exe
GOTO loop

I"m not a stranger to batch files. I've probably written hundreds since the late 1980's. This one is a bit different. I'm not really sure how to 'break' out of this one. If I ^C, it will drop to the last line and return the the label in line 4. I must be missing something!

[kladner]

Originally Posted by storm5510

@echo off
Set count=0
Set program=CUDALucas
:loop
TITLE %program% Current Reset Count = %count%
Set /A count+=1
echo %count% >> log.txt
echo %count%
%program%.exe
GOTO loop

I"m not a stranger to batch files. I've probably written hundreds since the late 1980's. This one is a bit different. I'm not really sure how to 'break' out of this one. If I ^C, it will drop to the last line and return the the label in line 4. I must be missing something!

It has been a long time since I used it. It is certainly a more sophisticated batch file than those I have written. It may have come from FlashJH (Jerry).
Do a Google on "site: mersenneforum.org ['cudalucas timeout' or 'cudalucas restart' or related terms]"

[storm5510]

Originally Posted by kladner

It has been a long time since I used it. It is certainly a more sophisticated batch file than those I have written. It may have come from FlashJH (Jerry).
Do a Google on "site: mersenneforum.org ['cudalucas timeout' or 'cudalucas restart' or related terms]"

I get it now, This batch will restart the program after the error I posted about earlier occurs, if no one is watching. Ctrl-Break exits the batch properly. Sorry!

[kladner]

Sorry

Don't be. Happy to provide some history. I spent a good bit of time with CuLu, but that was a few years back. Now that I have an i7-6700k, that delivers 2.2ms/it in the 40-42M range, the GPUs do LLTF.

[kriesel]

Originally Posted by storm5510

@echo off
Set count=0
Set program=CUDALucas
:loop
TITLE %program% Current Reset Count = %count%
Set /A count+=1
echo %count% >> log.txt
echo %count%
%program%.exe
GOTO loop

I"m not a stranger to batch files. I've probably written hundreds since the late 1980's. This one is a bit different. I'm not really sure how to 'break' out of this one. If I ^C, it will drop to the last line and return the the label in line 4. I must be missing something!

If I recall correctly, this is the NVIDIA driver timeout for older compute level 2 cards. There are two other possible approaches.
1. Downgrade your NVIDIA driver to about 306. and convince Windows not to upgrade it automatically.
2. Increase the driver timeout threshold. That can reduce but does not totally eliminate the timeouts. see post 2246: page 205 by wombatman; post 2247 by cudalucas author owftheevil; post 2257 gives the registry key info. post 2130 by flashjh also relates.

[storm5510]

Originally Posted by kriesel

...and convince Windows not to upgrade it automatically.
...

There is a option in Windows 10 called, "Device Installation Settings." You can select the "No" option button, which is supposed to stop hardware drivers to be updated. I have it set to "No" and it installs them anyway. Really useful!

[kriesel]

Originally Posted by storm5510

There is a option in Windows 10 called, "Device Installation Settings." You can select the "No" option button, which is supposed to stop hardware drivers to be updated. I have it set to "No" and it installs them anyway. Really useful!

Yes, I remember doing battle with automatic updating too, although not on Win10. http://winsupersite.com/windows-10/s...tes-windows-10 says "obligatory" for home version. NVIDIA driver packages are big and SLOW on my low speed link.

I have a vague recollection of needing to also disable driver updating in some NVIDIA downloaded stuff too, during benchmarking versus driver version. Complete removal of higher versions from the system was required. And to make sure version stayed put, I disconnected the network cable before launching a benchmark. Not so handy for a general use machine, of course.

[kriesel]

This compilation is based on mostly my own running and testing since February on Windows, with some info from the forums mixed in. Please chime in with linux experience or in general. The absence of fft lengths greater than 8192k in the -r self test option seems like a priority item. Perhaps a separate -rbig or -r 2 option, with 1000 iterations for the big fft lengths >8192k?