mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Cloud Computing (https://www.mersenneforum.org/forumdisplay.php?f=134)
-   -   Google Diet Colab Notebook (https://www.mersenneforum.org/showthread.php?t=24646)

Corbeau 2019-07-31 17:57

Google Diet Colab Notebook
 
Hello all,

I have been aware of GIMPS for a decent while, but only very recently started contributing. My only computer is a Sandy Bridge quadcore laptop. Not terribly quick, and I don't want to spend money on a cloud system. However, I don't think it's necessary to do so.

Google Colaboratory is free for anyone with a Google account, and seems perfect for any work that involves GPUs. Users get free access to a Tesla K80, which is much, MUCH quicker than my poor CPU. All that is necessary is a python notebook to automate the downloading of the necessary files. [URL="https://github.com/LeelaChessZero/lc0/wiki/Run-Leela-Chess-Zero-client-on-a-Tesla-K80-GPU-for-free-%28Google-Colaboratory%29"]Here[/URL] is an example of Google Colaboratory being used in another distributed computing project, the LCZero chess engine. Would a similar notebook be of any use to the GIMPS project? I searched the entire forum, and there was no mention of Colaboratory.

(Google Colaboratory is in the cloud, so I figured that this was the right subforum. If it isn't please move it.)

SimonVall00 2019-09-02 15:18

I have succesfully run mfaktc in Google Colaboratory. I have created a Github repository with the Notebook i used.



Here is a link to the repository.
[url]https://github.com/SimonVall00/mfaktc-google-colab[/url]

Corbeau 2019-09-02 23:38

Thank you for doing this! I'm having trouble with it, though; it seems to repeat the same task over and over (as in factoring the same number rather than moving on to the next one.) Also, I am getting "ERROR: cudaGetLastError() returned 8: invalid device function" errors. Do you want me to post screencaps here or on the Github?

hansl 2019-09-03 00:54

I just tried this and I see the same "invalid device function"

It looks like I got assigned T4 rather than a K80
Does it require mfaktc built for 7.5 or higher in this case?

Or maybe they have disabled the ability somehow (there was some warning about mining, so maybe its too similar activity to that?).
[code]
Compiletime options
THREADS_PER_BLOCK 256
SIEVE_SIZE_LIMIT 32kiB
SIEVE_SIZE 193154bits
SIEVE_SPLIT 250
MORE_CLASSES enabled

Runtime options
SievePrimes 25000
SievePrimesAdjust 1
SievePrimesMin 5000
SievePrimesMax 100000
NumStreams 3
CPUStreams 3
GridSize 3
GPU Sieving enabled
GPUSievePrimes 82486
GPUSieveSize 64Mi bits
GPUSieveProcessSize 16Ki bits
Checkpoints enabled
CheckpointDelay 30s
WorkFileAddDelay 600s
Stages enabled
StopAfterFactor bitlevel
PrintMode full
V5UserID (none)
ComputerID (none)
AllowSleep no
TimeStampInResults no

CUDA version info
binary compiled for CUDA 6.50
CUDA runtime version 6.50
CUDA driver version 10.10

CUDA device info
name Tesla T4
compute capability 7.5
max threads per block 1024
max shared memory per MP 65536 byte
number of multiprocessors 40
clock rate (CUDA cores) 1590MHz
memory clock rate: 5001MHz
memory bus width: 256 bit

Automatic parameters
threads per grid 655360
GPUSievePrimes (adjusted) 82486
GPUsieve minimum exponent 1055144

running a simple selftest...
ERROR: cudaGetLastError() returned 8: invalid device function
[/code]

SimonVall00 2019-09-03 05:09

I sometimes get a K80 and sometimes a T4. It works on the K80 but not on the T4. K80 has compute capability 3.7 and T4 7.5 so I think we need mfaktc built for 7.5 to run it on the T4.

chalsall 2019-09-04 14:27

[QUOTE=SimonVall00;525036]I have succesfully run mfaktc in Google Colaboratory. I have created a Github repository with the Notebook i used.[/QUOTE]

Amazingly cool!!! Thanks for bringing this to our attention!

I wasted a good part of yesterday making friends with Colab. Although I hate Python, it was pretty interesting figuring out what you could do with this system. Even gcc and nvidia-smi are available!

I've set up a Notebook to use my Google Drive for persistent storage for mfaktc, and so I can copy over the executable and the library rather than downloading and unpacking it each time. Even checkpoint files are retained. See below.

Although it takes some extra hand-holding, you (and Google) have just given me a free 430 GHzD/D GPU to TF with!

[CODE]import os.path
from google.colab import drive

if not os.path.exists('/content/drive/My Drive'):
drive.mount('/content/drive')

%cd '/content/drive/My Drive/colab/'

!cp 'mfaktc.exe' /usr/local/bin/
!chmod 755 '/usr/local/bin/mfaktc.exe'

!mkdir /usr/local/bin/lib/
!cp 'lib/libcudart.so.6.5' /usr/local/bin/lib/

!cd '.' && LD_LIBRARY_PATH="lib:${LD_LIBRARY_PATH}" /usr/local/bin/mfaktc.exe

!cat 'results.txt'[/CODE]

A quick-and-dirty script to add work and/or get work out of the system, rather than uploading / downloading from Drive (run in a seperate Colab Notebook; uses the same VM and filesystem):

[CODE]Insert_Work = "" #@param {type:"string"}
Delete_Results = True #@param {type:"boolean"}

import os.path
from google.colab import drive

if not os.path.exists('/content/drive/My Drive'):
drive.mount('/content/drive')

%cd '/content/drive/My Drive/colab/'
!ls -lha
!echo

if Insert_Work:
print("Add string is:" + Insert_Work + "\n")
f=open("worktodo.add","w+")
f.write(Insert_Work + "\n")
f.close()
!cat worktodo.add
!echo

!cat "results.txt"
!echo

if Delete_Results:
print("Deleting results!\n")
!rm results.txt

!cat worktodo.txt
!wc -l worktodo.txt[/CODE]

SimonVall00 2019-09-04 15:59

This works perfectly. Now we need mfaktc compiled for cuda 7.5 so we can run mfaktc when we are assigned the T4 GPU.

chalsall 2019-09-04 16:46

[QUOTE=SimonVall00;525162]Now we need mfaktc compiled for cuda 7.5 so we can run mfaktc when we are assigned the T4 GPU.[/QUOTE]

Indeed. It would be interesting to see what the GHzD/D is for a T4. I've only been playing with this for about a day now, but I've yet to get a T4 (unless explicitly asked for).

For anyone else trying this, be sure to set the hardware accelerator to GPU ("Runtime" menu -> "Change runtime type"). I understand that sometimes they won't be available, but I've yet to encounter this.

chalsall 2019-09-04 17:42

[QUOTE=chalsall;525169]For anyone else trying this...[/QUOTE]

I've got some "real work" to do, but just to quickly add some more information for anyone else trying this...

This is the directory contents of the "colab" folder on my Google Drive, which is mounted in the CoLab Virtual Machine:

[CODE]
!ls -lhaR
/content/drive/My Drive/colab
.:
total 1.4M
drwx------ 2 root root 4.0K Sep 3 19:09 lib
-rw------- 1 root root 42 Sep 4 17:24 M34196263.ckp
-rw------- 1 root root 1.1M Sep 3 18:27 mfaktc.exe
-rw------- 1 root root 8.7K Sep 3 22:49 mfaktc.ini
-rw------- 1 root root 299K Sep 4 17:24 test_1.ipynb
-rw------- 1 root root 338 Sep 4 17:01 worktodo.txt

./lib:
total 319K
-rw------- 1 root root 319K Sep 3 19:02 libcudart.so.6.5[/CODE]

It's basically just your standard mfaktc directory, including the x86 executable and library. These need to be moved over into the virtual machine's file system in order to be set executable.

mfaktc.ini and worktodo.txt should be edited and uploaded to your Google Drive before running the Notebook section. I found that the standard INI file is optimal.

Note that I've found that only one section of a Notebook can be run at a time, but you can launch other Notebooks (different name; different browser window/tab) to run other (shorter running, presumably) sections. All Notebooks being viewed, edited, etc by the user run in the same virtual machine, and share the file system, CPU and GPU.

And the VM appears to be a full-blown 2 core Xeon, 12 GB RAM, 360 GB HD machine with a K80 attached!

When I have more time to waste, I'm going to look at how to get mprime to run. A quick try resulted in the usual mprime first-time-run menu, which can't be interacted with directly through the python shell.

Too cool!!! :smile:

hansl 2019-09-04 18:41

[QUOTE=SimonVall00;525162]This works perfectly. Now we need mfaktc compiled for cuda 7.5 so we can run mfaktc when we are assigned the T4 GPU.[/QUOTE]

I just posted a build in the main mfaktc thread, but haven't had a chance to try it on Colab. Let me know if this helps:
[url]https://mersenneforum.org/showpost.php?p=525178&postcount=3189[/url]

chalsall 2019-09-04 19:13

[QUOTE=hansl;525179]Let me know if this helps...[/QUOTE]

Super coolness! Thanks.

I don't have time to test this at the moment, but an idea popped into my head which I just had to try... I copied the following Perl up to my Drive's colab directory:

[CODE]#!/usr/bin/perl -w

@DT = gmtime(time);
$Now = sprintf("%04d.%02d.%02d %02d:%02d", 1900+$DT[5],$DT[4],$DT[3],$DT[2],$DT[1]);

print "The time is: ${Now}\n\n";[/CODE]

...and then in a Section ran:[CODE]
!pwd
!chmod +x perl/time.pl
!ls -la perl
!perl/time.pl[/CODE]

...and got this:[CODE]
/content/drive/My Drive/colab
total 1
-rwx------ 1 root root 160 Sep 4 19:05 time.pl
The time is: 2019.08.04 19:06[/CODE]

My brain exploded! (tinkle, tinkle, tingle... :wink:)


All times are UTC. The time now is 07:11.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.