mersenneforum.org  

Go Back   mersenneforum.org > Extra Stuff > Blogorrhea > EdH

Reply
 
Thread Tools
Old 2019-12-07, 19:51   #1
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

24·11·19 Posts
Default How I Create a Colab Session That Factors factordb Comps with the GPU branch of GMP-ECM and msieve

(Note: I expect to keep the first post of each of these "How I..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.)

I will take the liberty of expecting readers to already be somewhat familiar with Google's Colaboratory sessions. There are several threads already on Colab and these should be reviewed by interested readers:

Google Colaboratory Notebook?
GPU72 Notebook Integration...
Notebook Instance Reverse SSH and HTTP Tunnels.
Colab question

I do not, as of yet, have a github account, so I have not created an upload of this to github. Others may feel free to do so, if desired. The following is a manner to compile and install the GPU branch of GMP-ECM and the latest revision of msieve into a Colab session, and then use them to factor composites retrieved from factordb.com. This implementation of GMP-ECM uses the number of available cores as its curve count. Within the below code, the level setting can be used to limit how many steps of the B1 value to try. Additionally, if larger B1 values are of interest, the appropriate values can easily be changed.

For those unfamiliar with the GPU branch of GMP-ECM, only stage 1 is handled by the GPU. All of stage 2 is still only processed by the CPU. Additionally, the stages are not run in parallel. Stage 1 runs on the GPU and then stage 2 runs on the CPU.

More information on the GPU branch of GMP-ECM can be found in the README.gpu document within the ecm main folder.

This instance runs the ECM step(s) and if no factors are found by ECM, factors the composite via msieve.

To use Colab, you need a Gmail account and will be required to log into that account to run a session.

On to the specifics:

Open a Google Colaboratory session.
Sign in with your Google/Gmail account info.
Choose New Python3 notebook:
Code:
Menu->File->New Python3 notebook (or within popup)
Change Runtime to add GPU:
Code:
Menu->Runtime->Change runtime type->Hardware accelerator->GPU->SAVE
Click Connect to start a session.
Edit title from Untitled... to whatever you like.
Paste the following into the Codeblock:
Code:
#########################################################
### This Colaboratory session is designed to retrieve ###
### composites from factordb, try to factor them with ###
### GMP-ECM using the GPU branch and, if not success- ###
### ful, complete the factorization using msieve.     ###
###                                                   ###
### To adjust the number of composites to retrieve as ###
### well as the size to retrive, change the variables ###
### below this comment block.  The size of the random ###
### number to be used to help avoid collisions (1000) ###
### can also be changed, if desired.                  ###
#########################################################

compNum = 3 # Number of composites to run
compSize = 84 # Size of composites to run
ranNum = 1000 # Number for random count
level = 3 # Level to run ( 1 = 11e2, 2 = 11e3, 3 = 11e4, 4 = 43e4, 5+ = 11e5 )

import os
import random
import subprocess
import time
import urllib.request

#reports factors to factordb
def send2db(composite, factors):
  factorline = str(factors)
  sendline = 'report=' + str(composite) + '%3D' + factorline
  dbcall = sendline.encode('utf-8')
  temp2 = urllib.request.urlopen('http://factordb.com/report.php', dbcall)

#process factorT to see if factors were returned from GMP-ECM
def factorStr(factorT):
  factors = str(factorT)
  ind = factors.rfind("stdout")
  ind += 9
  factors = factors[ind:]
  factors = factors[:-4]
  return factors

#checks to see if GMP-ECM already exists
#if it does, this portion is skipped
exists = os.path.isfile('/usr/local/bin/ecm')
if exists < 1:
  print("Installing system packages. . .")
  subprocess.call(["chmod", "777", "/tmp"])
  subprocess.call(["apt", "update"])
  subprocess.call(["apt", "install", "g++", "m4", "make", "subversion", "libgmp-dev", "libtool", "p7zip", "autoconf"])
#retrieves ecm
  print("Retrieving GMP-ECM. . .")
  subprocess.call(["svn", "co", "svn://scm.gforge.inria.fr/svn/ecm/trunk", "ecm"])
  os.chdir("/content/ecm")
  subprocess.call(["libtoolize"])
  subprocess.call(["autoreconf", "-i"])
  subprocess.call(["./configure", "--enable-gpu", "--with-cuda-bin=/usr/local/cuda-10.0/bin", "--with-cuda=/usr/local/cuda-10.0/targets/x86_64-linux", "--with-cuda-lib=/usr/local/cuda-10.0/targets/x86_64-linux/lib", "", "", "", "", "--with-gmp=/usr/local/"])
  print("Compiling GMP-ECM. . .")
  subprocess.call(["make"])
  subprocess.call(["make", "install"])
  print("Finished installing GMP-ECM. . .")
  os.chdir("/content")
#retrieves msieve
  print("Retrieving msieve. . .")
  subprocess.call(["svn", "co", "https://svn.code.sf.net/p/msieve/code/trunk", "msieve"])
  os.chdir("/content/msieve")
  print("Compiling msieve. . .")
  subprocess.call(["make", "all"])
  print("Finished compiling msieve. . .")
print("Starting the factoring of", compNum, "composites. . .\n")

#main loop
for x in range(compNum):
  randnum = random.randrange(ranNum)
#fetch a number from factordb
  dbcall = 'http://factordb.com/listtype.php?t=3&mindig=' + str(compSize) + '&perpage=1&start=' + str(randnum) + '&download=1'
#some file processing to get the number into a format usable by GMP-ECM
  temp0 = urllib.request.urlopen(dbcall)
  temp1 = temp0.read()
  composite = temp1.decode(encoding='UTF-8')
  composite = composite.strip("\n")
  fstart = time.time()
#print number being worked on
  print("Composite", x + 1, "is:", composite)
  factorT = subprocess.run(['ecm', '-gpu', '-q', '11e2'], stdout=subprocess.PIPE, input=temp1)
  factors = factorStr(factorT)
  if (len(factors) == len(composite)) and level > 1:
    print("No factors found at 11e2! Trying 11e3. . .")
    factorT = subprocess.run(['ecm', '-gpu', '-q', '11e3'], stdout=subprocess.PIPE, input=temp1)
    factors = factorStr(factorT)
  if (len(factors) == len(composite)) and level > 2:
    print("No factors found at 11e3! Trying 11e4. . .")
    factorT = subprocess.run(['ecm', '-gpu', '-q', '11e4'], stdout=subprocess.PIPE, input=temp1)
  if (len(factors) == len(composite)) and level > 3:
    print("No factors found at 11e4! Trying 43e4. . .")
    factorT = subprocess.run(['ecm', '-gpu', '-q', '43e4'], stdout=subprocess.PIPE, input=temp1)
    factors = factorStr(factorT)
  if (len(factors) == len(composite)) and level > 4:
    print("No factors found at 43e4! Trying 11e5. . .")
    factorT = subprocess.run(['ecm', '-gpu', '-q', '11e5'], stdout=subprocess.PIPE, input=temp1)
    factors = factorStr(factorT)
  if (len(factors) > len(composite)):
    factors = factors.replace(' ','*')
  else:
    print("No factors found by ECM!  Running msieve. . .")
    factorTm = subprocess.run(['./msieve', '-t', '2', composite])
#find factors from the msieve run in msieve.log
    file = open('msieve.log', 'r')
    string = (" factor: ")
    fcheck = 0
    factors = ""
    for line in file:
      found = line.rfind(string)
      if found > 0:
        line = line.rstrip("\n")
        ind = found + 9
        line = line[ind:]
        if fcheck > 0:
          factors = factors + "*"
        factors = factors + line
        fcheck += 1
    os.remove("msieve.log")
  runtime = time.time() - fstart
#print factors found
  print("Factors for", x + 1, ":", factors)
  print("Elapsed time was", int(runtime / 60), "minutes and", int(runtime % 60), "seconds.\n")
#send number and factors to factordb
  send2db(composite, factors)
#all numbers are completed
print("Completed", x + 1, "composites!\n")
Click on the Run cell icon or use CTRL-Enter. This will run for a couple minutes and should proceed right into factoring three composites when msieve has completed compilation.

Subsequent runs should proceed without the compilation and installation steps.

Last fiddled with by EdH on 2020-04-19 at 14:03
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
How I Compile the GPU branch of GMP-ECM in a Colaboratory Session EdH EdH 13 2020-06-02 11:05
How I Create a Colab Session That Factors factordb Composites with the GPU branch of GMP-ECM EdH EdH 0 2019-12-04 01:49
How I Create a Colab Session That Factors factordb Composites with YAFU EdH EdH 12 2019-11-11 17:44
Someone is reporting orphan factors to FactorDB. GP2 FactorDB 6 2018-07-24 19:45
msieve No Factors in log EdH Msieve 2 2016-11-25 04:20

All times are UTC. The time now is 21:17.

Fri Sep 18 21:17:50 UTC 2020 up 8 days, 18:28, 1 user, load averages: 0.95, 1.44, 1.63

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.