How I Compile the GPU branch of GMP-ECM in a Colaboratory Session

[EdH]

(Note: I expect to keep the first post of each of these "How I..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.)

I will take the liberty of expecting readers to already be somewhat familiar with Google's Colaboratory sessions. There are several threads already on Colab and these should be reviewed by interested readers:

Google Colaboratory Notebook?
GPU72 Notebook Integration...
Notebook Instance Reverse SSH and HTTP Tunnels.
Colab question

I do not, as of yet, have a github account, so I have not created an upload of this to github. Others may feel free to do so, if desired. The following is a manner to compile and install GMP and the GPU branch of GMP-ECM into a Colab session, so that ECM can be run making use of a GPU.

For those unfamiliar with the GPU branch of GMP-ECM, only stage 1 is handled by the GPU. All of stage 2 is still only processed by the CPU. Additionally, the stages are not run in parallel. Stage 1 runs on the GPU and then stage 2 runs on the CPU.

More information on the GPU branch of GMP-ECM can be found in the README.gpu document within the ecm main folder.

To use Colab, you need a Gmail account and will be required to log into that account to run a session.

On to the specifics:

Open a Google Colaboratory session.
Sign in with your Google/Gmail account info.
Choose New Python3 notebook:

Menu->File->New Python3 notebook (or within popup)

Change Runtime to add GPU:

Menu->Runtime->Change runtime type->Hardware accelerator->GPU->SAVE

Click Connect to start a session.
Edit title from Untitled... to whatever you like.
Paste the following into the Codeblock:

!chmod 777 /tmp
!apt update
!apt install g++ m4 make git libtool autoconf cuda-10-0
!mkdir Math
%cd Math
!wget "https://ftp.gnu.org/gnu/gmp/gmp-6.2.0.tar.bz2"
!tar -xf gmp-6.2.0.tar.bz2
!mv gmp-6.2.0 gmp
%cd gmp
!./configure
!make
############################################
# These 4 lines are recommended, but can be commented out or removed! #
############################################
!make check
!echo "Check results, then click in the"
!echo "TextBox below and press Enter to continue. . ."
!read continue in
############################################
!make install
%cd ..
!git clone https://gitlab.inria.fr/zimmerma/ecm ecm
%cd ecm
!libtoolize
!autoreconf -i
!./configure --enable-gpu --with-cuda-bin=/usr/local/cuda-10.0/bin --with-cuda=/usr/local/cuda-10.0/targets/x86_64-linux --with-cuda-lib=/usr/local/cuda-10.0/targets/x86_64-linux/lib --with-gmp=/usr/local/
!make
############################################
# These 4 lines are recommended, but can be commented out or removed! #
############################################
!make check
!echo "Check results, then click in the"
!echo "TextBox below and press Enter to continue. . ."
!read continue in
############################################
!make install
!echo "############################################"
!echo "############################################"
!echo "(2^835+1)/33" | ecm -gpu 1e4

Note: There are some checks and pauses written into the code that are recommended, but not strictly necessary. Feel free to comment out or remove those lines in the above code.

Click on the Run cell icon or use CTRL-Enter.
Either watch all the scrolling text or take a break. This will run for a few minutes. If you chose to leave the checks and pauses you can review the results of the GMP and ECM check runs before continuing. To continue at these breaks, you have to click in the text entry box below the message and hit Enter as instructed.

When everything is finished, if all went well, you should have a message similar to the following:

. . .
GMP-ECM 7.0.5-dev [configured with GMP 6.1.2, --enable-asm-redc, --enable-gpu, --enable-assert] [ECM]
 Input number is (2^835+1)/33 (250 digits)
Using B1=10000, B2=1873422, sigma=3:1745495596-3:1745496427 (832 curves)
GPU: Block: 32x32x1 Grid: 26x1x1 (832 parallel curves)
Computing 832 Step 1 took 90ms of CPU time / 4044ms of GPU time
Computing 832 Step 2 on CPU took 28406ms

At this point you can remove the code from the original code block or open a new code block to continue.

Next, in an empty code block, try the following:

!echo 115367564564210182766242534110944507919869313713243756429 | ecm -gpu 1e4

You should see a long series of successes:

GMP-ECM 7.0.5-dev [configured with GMP 6.1.2, --enable-asm-redc, --enable-gpu, --enable-assert] [ECM]
Input number is 115367564564210182766242534110944507919869313713243756429 (57 digits)
Using B1=10000, B2=1873422, sigma=3:2239793235-3:2239794066 (832 curves)
GPU: Block: 32x32x1 Grid: 26x1x1 (832 parallel curves)
GPU: factor 3387679 found in Step 1 with curve 0 (-sigma 3:2239793235)
. . .
GPU: factor 3387679 found in Step 1 with curve 830 (-sigma 3:2239794065)
Computing 832 Step 1 took 62ms of CPU time / 4012ms of GPU time
********** Factor found in step 1: 3387679
Found prime factor of 7 digits: 3387679
Prime cofactor 34055046113935288073705488067477617542827792631251 has 50 digits

If all ran as expected, you can proceed with more ECM runs using the same format. Again, for more info on running GMP-ECM with GPU support, see the README.gpu document in the main ecm folder.

[chalsall]

Originally Posted by EdH

!apt update
...

Very nice work!

Dylan figured out that it is advisable to "chmod 777 /tmp" before the "apt update".

This makes your code compatible with Kaggle.

[EdH]

Originally Posted by chalsall

Very nice work!

Dylan figured out that it is advisable to "chmod 777 /tmp" before the "apt update".

This makes your code compatible with Kaggle.

Thanks! The step is added. I haven't ventured into Kaggle yet.

[chalsall]

Originally Posted by EdH

Thanks! The step is added. I haven't ventured into Kaggle yet.

I hope you give it a go. I don't have the time at the moment to try your code there, but my experience has been that it is relatively trivial to get the Python to run in both environments. Possibly involves some additional "apt install" parms.

And... I just noticed another slight delta to your above code... You're already running as root -- there's no need to "sudo" the "apt-get" command.

[EdH]

Originally Posted by chalsall

I hope you give it a go. I don't have the time at the moment to try your code there, but my experience has been that it is relatively trivial to get the Python to run in both environments. Possibly involves some additional "apt install" parms.

And... I just noticed another slight delta to your above code... You're already running as root -- there's no need to "sudo" the "apt-get" command.

Ahh, good catch! Thanks! I probably don't need to use apt-get right after apt either. I must admit to Copy/Paste without a close review. I don't think I need p7zip or zlib1g-dev, either. But, I will have to check now that they are removed.

[Dylan14]

Nice script EdH. But there is one typo: where you have the !make install line for ecm you have an indent, which causes the script to fail. It needs to be inline with the other lines.

[EdH]

Originally Posted by Dylan14

Nice script EdH. But there is one typo: where you have the !make install line for ecm you have an indent, which causes the script to fail. It needs to be inline with the other lines.

Thanks! I don't know why those keep showing up. I've already gotten rid of half a dozen.

[PhilF]

This is great! However, I discovered that the number I wanted to work on (M1277) is too large for GPU work.

That sent me on a quest to find suitable numbers that need some ECM thrown at them that were within reach of GPU work, and I found that search difficult.

So where to find some suitable candidates for GPU based ECM curves?

I would like to compare the relative speed of a GPU-enabled GMP-ECM stage 1 with a CPU-based Prime95 stage 1.

[VBCurtis]

Originally Posted by PhilF

So where to find some suitable candidates for GPU based ECM curves?

I would like to compare the relative speed of a GPU-enabled GMP-ECM stage 1 with a CPU-based Prime95 stage 1.

GPU-ECM is limited to 1019 bits. Prime95 is limited to numbers of a certain form. There's not a ton of overlap between them, but I happen to be working on factorizations of k*2^n-1 (Riesel form).
My next two SNFS jobs are:

#expr(13*2^881-1)
466877517878596827431051486024691686855414677675172874855133940880576871475985818271788497478759019442908250077983766404430012687428346626398685854186379459430598338289874292283833496457445239
#expr(13*2^892-1)
18506805396968135611155184891404406020831889021055758328923431924399638720432241766053234118351869077918037177534464815730861966812520562472496745230825092848732924799837750268894252467544668175171

I believe these can be run by both programs. They've each had about 2t50 done already, and need something close to t55; so curves at 11e7 or bigger are helpful to my efforts.
If you're curious, you can find project status at mklasson.com/factors

[PhilF]

Perfect! That is exactly what I was looking for. I'll do some performance tests and get back with you...

[LaurV]

Originally Posted by EdH

Thanks! I don't know why those keep showing up. I've already gotten rid of half a dozen.

That's the forum, gov'nor!

Since last year when Mike changed I don't know what, I have a continuous battle with random spaces appearing in front of code lines in the code tag (therefore mis-aligning tables or space-sensitive code) or empty lines appearing in between paragraphs. Last time I edited somebody's post to align a table header I had to do it 5 or 6 times, because every time some spaces used to appear in front of a different line after I saved the (correct spaced) post. At the end I was convinced I fixed it, but revisiting the post after few days, some lines inside of the code tag are still mis-aligned. Futile to say, the post still shows I was the last guy to edit it.