Google Diet Colab Notebook

[Corbeau]

Hello all,

I have been aware of GIMPS for a decent while, but only very recently started contributing. My only computer is a Sandy Bridge quadcore laptop. Not terribly quick, and I don't want to spend money on a cloud system. However, I don't think it's necessary to do so.

Google Colaboratory is free for anyone with a Google account, and seems perfect for any work that involves GPUs. Users get free access to a Tesla K80, which is much, MUCH quicker than my poor CPU. All that is necessary is a python notebook to automate the downloading of the necessary files. Here is an example of Google Colaboratory being used in another distributed computing project, the LCZero chess engine. Would a similar notebook be of any use to the GIMPS project? I searched the entire forum, and there was no mention of Colaboratory.

(Google Colaboratory is in the cloud, so I figured that this was the right subforum. If it isn't please move it.)

[SimonVall00]

I have succesfully run mfaktc in Google Colaboratory. I have created a Github repository with the Notebook i used.



Here is a link to the repository.
https://github.com/SimonVall00/mfaktc-google-colab

[Corbeau]

Thank you for doing this! I'm having trouble with it, though; it seems to repeat the same task over and over (as in factoring the same number rather than moving on to the next one.) Also, I am getting "ERROR: cudaGetLastError() returned 8: invalid device function" errors. Do you want me to post screencaps here or on the Github?

[hansl]

I just tried this and I see the same "invalid device function"

It looks like I got assigned T4 rather than a K80
Does it require mfaktc built for 7.5 or higher in this case?

Or maybe they have disabled the ability somehow (there was some warning about mining, so maybe its too similar activity to that?).

Compiletime options
  THREADS_PER_BLOCK         256
  SIEVE_SIZE_LIMIT          32kiB
  SIEVE_SIZE                193154bits
  SIEVE_SPLIT               250
  MORE_CLASSES              enabled

Runtime options
  SievePrimes               25000
  SievePrimesAdjust         1
  SievePrimesMin            5000
  SievePrimesMax            100000
  NumStreams                3
  CPUStreams                3
  GridSize                  3
  GPU Sieving               enabled
  GPUSievePrimes            82486
  GPUSieveSize              64Mi bits
  GPUSieveProcessSize       16Ki bits
  Checkpoints               enabled
  CheckpointDelay           30s
  WorkFileAddDelay          600s
  Stages                    enabled
  StopAfterFactor           bitlevel
  PrintMode                 full
  V5UserID                  (none)
  ComputerID                (none)
  AllowSleep                no
  TimeStampInResults        no

CUDA version info
  binary compiled for CUDA  6.50
  CUDA runtime version      6.50
  CUDA driver version       10.10

CUDA device info
  name                      Tesla T4
  compute capability        7.5
  max threads per block     1024
  max shared memory per MP  65536 byte
  number of multiprocessors 40
  clock rate (CUDA cores)   1590MHz
  memory clock rate:        5001MHz
  memory bus width:         256 bit

Automatic parameters
  threads per grid          655360
  GPUSievePrimes (adjusted) 82486
  GPUsieve minimum exponent 1055144

running a simple selftest...
ERROR: cudaGetLastError() returned 8: invalid device function

[SimonVall00]

I sometimes get a K80 and sometimes a T4. It works on the K80 but not on the T4. K80 has compute capability 3.7 and T4 7.5 so I think we need mfaktc built for 7.5 to run it on the T4.

[chalsall]

Originally Posted by SimonVall00

I have succesfully run mfaktc in Google Colaboratory. I have created a Github repository with the Notebook i used.

Amazingly cool!!! Thanks for bringing this to our attention!

I wasted a good part of yesterday making friends with Colab. Although I hate Python, it was pretty interesting figuring out what you could do with this system. Even gcc and nvidia-smi are available!

I've set up a Notebook to use my Google Drive for persistent storage for mfaktc, and so I can copy over the executable and the library rather than downloading and unpacking it each time. Even checkpoint files are retained. See below.

Although it takes some extra hand-holding, you (and Google) have just given me a free 430 GHzD/D GPU to TF with!

import os.path
from google.colab import drive

if not os.path.exists('/content/drive/My Drive'):
  drive.mount('/content/drive')

%cd '/content/drive/My Drive/colab/'

!cp 'mfaktc.exe' /usr/local/bin/
!chmod 755 '/usr/local/bin/mfaktc.exe'

!mkdir /usr/local/bin/lib/
!cp 'lib/libcudart.so.6.5' /usr/local/bin/lib/

!cd '.' && LD_LIBRARY_PATH="lib:${LD_LIBRARY_PATH}" /usr/local/bin/mfaktc.exe

!cat 'results.txt'

A quick-and-dirty script to add work and/or get work out of the system, rather than uploading / downloading from Drive (run in a seperate Colab Notebook; uses the same VM and filesystem):

Insert_Work = "" #@param {type:"string"}
Delete_Results = True #@param {type:"boolean"}

import os.path
from google.colab import drive

if not os.path.exists('/content/drive/My Drive'):
  drive.mount('/content/drive')

%cd '/content/drive/My Drive/colab/'
!ls -lha
!echo

if Insert_Work:
  print("Add string is:" + Insert_Work + "\n")
  f=open("worktodo.add","w+")
  f.write(Insert_Work + "\n")
  f.close()
  !cat worktodo.add
  !echo

!cat "results.txt"
!echo

if Delete_Results:
  print("Deleting results!\n")
  !rm results.txt

!cat worktodo.txt
!wc -l worktodo.txt

[SimonVall00]

This works perfectly. Now we need mfaktc compiled for cuda 7.5 so we can run mfaktc when we are assigned the T4 GPU.

[chalsall]

Originally Posted by SimonVall00

Now we need mfaktc compiled for cuda 7.5 so we can run mfaktc when we are assigned the T4 GPU.

Indeed. It would be interesting to see what the GHzD/D is for a T4. I've only been playing with this for about a day now, but I've yet to get a T4 (unless explicitly asked for).

For anyone else trying this, be sure to set the hardware accelerator to GPU ("Runtime" menu -> "Change runtime type"). I understand that sometimes they won't be available, but I've yet to encounter this.

[chalsall]

Originally Posted by chalsall

For anyone else trying this...

I've got some "real work" to do, but just to quickly add some more information for anyone else trying this...

This is the directory contents of the "colab" folder on my Google Drive, which is mounted in the CoLab Virtual Machine:

!ls -lhaR
/content/drive/My Drive/colab
.:
total 1.4M
drwx------ 2 root root 4.0K Sep  3 19:09 lib
-rw------- 1 root root   42 Sep  4 17:24 M34196263.ckp
-rw------- 1 root root 1.1M Sep  3 18:27 mfaktc.exe
-rw------- 1 root root 8.7K Sep  3 22:49 mfaktc.ini
-rw------- 1 root root 299K Sep  4 17:24 test_1.ipynb
-rw------- 1 root root  338 Sep  4 17:01 worktodo.txt

./lib:
total 319K
-rw------- 1 root root 319K Sep  3 19:02 libcudart.so.6.5

It's basically just your standard mfaktc directory, including the x86 executable and library. These need to be moved over into the virtual machine's file system in order to be set executable.

mfaktc.ini and worktodo.txt should be edited and uploaded to your Google Drive before running the Notebook section. I found that the standard INI file is optimal.

Note that I've found that only one section of a Notebook can be run at a time, but you can launch other Notebooks (different name; different browser window/tab) to run other (shorter running, presumably) sections. All Notebooks being viewed, edited, etc by the user run in the same virtual machine, and share the file system, CPU and GPU.

And the VM appears to be a full-blown 2 core Xeon, 12 GB RAM, 360 GB HD machine with a K80 attached!

When I have more time to waste, I'm going to look at how to get mprime to run. A quick try resulted in the usual mprime first-time-run menu, which can't be interacted with directly through the python shell.

Too cool!!!

[hansl]

Originally Posted by SimonVall00

This works perfectly. Now we need mfaktc compiled for cuda 7.5 so we can run mfaktc when we are assigned the T4 GPU.

I just posted a build in the main mfaktc thread, but haven't had a chance to try it on Colab. Let me know if this helps:
https://mersenneforum.org/showpost.p...postcount=3189

[chalsall]

Originally Posted by hansl

Let me know if this helps...

Super coolness! Thanks.

I don't have time to test this at the moment, but an idea popped into my head which I just had to try... I copied the following Perl up to my Drive's colab directory:

#!/usr/bin/perl -w

@DT = gmtime(time);
$Now = sprintf("%04d.%02d.%02d %02d:%02d", 1900+$DT[5],$DT[4],$DT[3],$DT[2],$DT[1]);

print "The time is: ${Now}\n\n";

...and then in a Section ran:

!pwd
!chmod +x perl/time.pl
!ls -la perl
!perl/time.pl

...and got this:

/content/drive/My Drive/colab
total 1
-rwx------ 1 root root 160 Sep  4 19:05 time.pl
The time is: 2019.08.04 19:06

My brain exploded! (tinkle, tinkle, tingle... )