mersenneforum.org  

Go Back   mersenneforum.org > Extra Stuff > Blogorrhea > EdH

Reply
 
Thread Tools
Old 2021-02-03, 19:04   #1
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3×372 Posts
Default How I Create a Colab Session That Extends Aliquot Sequences Working Directly with factordb

(Note: I expect to keep the first post of each of these "How I..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.)

I will take the liberty of expecting readers to already be somewhat familiar with Google's Colaboratory sessions. There are several threads already on Colab and these should be reviewed by interested readers:

Google Colaboratory Notebook?
GPU72 Notebook Integration...
Notebook Instance Reverse SSH and HTTP Tunnels.
Colab question

I do not, as of yet, have a github account, so I have not created an upload of this to github. Others may feel free to do so, if desired.

The following is a manner to work with factordb.com to extend an Aliquot sequence. The session will retrieve the last composite in a sequence and factor it, returning the factors to factordb.com, which in turn extends the sequence. The session will ask for the initial sequence term to be entered. The session uses YAFU to perform the factoring, which in turn uses GMP-ECM for some of its process. The initial run installs all the necessary packages.

Note: Currently, this session is only using SIQS for factoring that GMP-ECM doesn't solve. The NFS switch is not used during compilation of YAFU and neither Msieve nor the GNFS sievers are installed.

To use Colab, you need a Gmail account and will be required to log into that account to run a session.

On to the specifics:

Open a Google Colaboratory session.
Sign in with your Google/Gmail account info.
Choose New notebook:
Code:
Menu->File->New notebook (or within popup)
Click Connect to start a session.
Edit title from Untitled... to whatever you like.
Paste the following into the Codeblock:
Code:
########################################################## 
### This Colaboratory session is designed to extend an ### 
### Aliquot sequence.  It works directly with factordb ### 
### and uses YAFU to find factors, which are then sent ### 
### back to factordb.                                  ### 
###                                                    ### 
### The only interaction with this session is to enter ### 
### the initial Aliquot term when prompted.            ### 
########################################################## 
  
import os 
import subprocess 
import urllib.request 
  
#reports factors to factordb 
def send2db(composite, factors): 
  factorline = str(factors) 
  sendline = 'report=' + str(composite) + '%3D' + factorline 
  dbcall = sendline.encode('utf-8') 
  temp2 = urllib.request.urlopen('http://factordb.com/report.php', dbcall) 
  
#checks to see if yafu already exists 
#if it does, this portion is skipped 
exists = os.path.isfile('yafu') 
if exists < 1: 
  print("Installing system packages. . .") 
#  subprocess.call(["chmod", "777", "/tmp"]) 
  subprocess.call(["apt", "update"]) 
  subprocess.call(["apt", "install", "g++", "m4", "make", "subversion", "libgmp-dev", "libtool", "p7zip", "autoconf"]) 
#retrieves ecm 
  print("Retrieving GMP-ECM. . .") 
  subprocess.call(["svn", "co", "svn://scm.gforge.inria.fr/svn/ecm/trunk", "ecm"]) 
  os.chdir("/content/ecm") 
  subprocess.call(["libtoolize"]) 
  subprocess.call(["autoreconf", "-i"]) 
  subprocess.call(["./configure", "--with-gmp=/usr/local/"]) 
  print("Compiling GMP-ECM. . .") 
  subprocess.call(["make"]) 
  subprocess.call(["make", "install"]) 
  print("Finished installing GMP-ECM. . .") 
  os.chdir("/content") 
#retrieves YAFU 
  print("Retrieving YAFU. . .") 
  subprocess.call(["svn", "co", "https://svn.code.sf.net/p/yafu/code/trunk", "/content/yafu"]) 
  os.chdir("/content/yafu") 
  subprocess.call(["mv", "yafu.ini", "yafu.ini.orig"]) 
  with open("yafu.ini", "a+") as yafuini: 
    yafuini.write("B1pm1=100000\n") 
    yafuini.write("B1pp1=20000\n") 
    yafuini.write("B1ecm=11000\n") 
    yafuini.write("rhomax=200\n") 
    yafuini.write("threads=2\n") 
    yafuini.write("pretest_ratio=0.25\n") 
    yafuini.write("ecm_path=/usr/local/bin/ecm\n") 
    yafuini.write("xover=93\n") 
    yafuini.write("no_clk_test=1") 
  print("Compiling YAFU. . .") 
  subprocess.call(["make", "x86_64", "USE_SSE41=1", "USE_AVX512=1"]) 
  print("Finished compiling YAFU. . .") 
  with open("/content/yafu/AliWork.sh", "a+") as AliWork: 
    AliWork.write("#!/bin/bash/\n") 
    AliWork.write("\n") 
    AliWork.write("function getComp(){\n") 
    AliWork.write("    wget -q -U Mozilla/5.0 \"http://factordb.com/sequences.php?se=1&aq=${seq}&action=last\" -O dbTemp\n") 
    AliWork.write("    exec <\"dbTemp\"\n") 
    AliWork.write("    while read line2\n") 
    AliWork.write("      do\n") 
    AliWork.write("        case $line2 in\n") 
    AliWork.write("          *\"index.php?id=\"*) ind=${line2##*002099}\n") 
    AliWork.write("          ind1=${#line2}\n") 
    AliWork.write("          let ind1=${ind1}-${#ind}\n") 
    AliWork.write("          let ind1=${ind1}-41\n") 
    AliWork.write("          ind2=http://factordb.com/index.php?id=${line2:${ind1}:19}\n") 
    AliWork.write("          ;;\n") 
    AliWork.write("        esac\n") 
    AliWork.write("      done\n") 
    AliWork.write("\n") 
    AliWork.write("    wget -q -U Mozilla/5.0 $ind2 -O dbTemp\n") 
    AliWork.write("\n") 
    AliWork.write("    exec <\"dbTemp\"\n") 
    AliWork.write("    while read line3\n") 
    AliWork.write("      do\n") 
    AliWork.write("        case $line3 in\n") 
    AliWork.write("          *\"query\"*) comp=${line3##*\'value=\"\'}\n") 
    AliWork.write("          comp2=${comp:0:${#comp}-2}\n") 
    AliWork.write("          complen=${#comp2}\n") 
    AliWork.write("          ;;\n") 
    AliWork.write("        esac\n") 
    AliWork.write("      done\n") 
    AliWork.write("}\n") 
    AliWork.write("\n") 
    AliWork.write("rm /content/yafu/stopAliWork 2>/dev/null\n") 
    AliWork.write("cd /content/yafu\n") 
    AliWork.write("test=\"\"\n") 
    AliWork.write("\n") 
    AliWork.write("if [ ${#1} -lt 3 ]\n") 
    AliWork.write("  then\n") 
    AliWork.write("    printf \"Enter seq#: \"\n") 
    AliWork.write("    read line in\n") 
    AliWork.write("    seq=$line\n") 
    AliWork.write("  else\n") 
    AliWork.write("    seq=$1\n") 
    AliWork.write("fi\n") 
    AliWork.write("\n") 
    AliWork.write("while [ ! -e ~/stopAliWork ]\n") 
    AliWork.write("  do\n") 
    AliWork.write("    rm *.log 2>/dev/null\n") 
    AliWork.write("    rm *.dat 2>/dev/null\n") 
    AliWork.write("    rm factors 2>/dev/null\n") 
    AliWork.write("    rm db* 2>/dev/null\n") 
    AliWork.write("    rm nfs.* 2>/dev/null\n") 
    AliWork.write("    getComp\n") 
    AliWork.write("\n")     
    AliWork.write("    echo sequence is $seq and composite is ${comp2:0:10}...${comp2:${complen}-2}\'<    \'${complen}\'>\'\n") 
    AliWork.write("    echo \"$comp2\" | ./yafu -of factors\n") 
    AliWork.write("    exec <\"factors\"\n") 
    AliWork.write("      read line in\n") 
    AliWork.write("    report=http://factordb.com/report.php?report=$(echo ${line:1:$complen}%3D${line:$complen+3} | tr -s \'/\' \'*\')\n") 
    AliWork.write("    wget -q -U Mozilla/5.0 \"$report\" -O dbRsave\n") 
    AliWork.write("\n") 
    AliWork.write("  done\n") 
print("All compilations completed!\n") 
%cd /content/yafu/ 
print("Enter sequence into box and press Enter:\n") 
!bash AliWork.sh
!rm ~/stopAliWork
Click on the Run cell icon or use CTRL-Enter. The first run will install packages for a few minutes and then prompt for the seq# (initial term). It will then work toward extending the sequence.

Subsequent runs should proceed without the compilation and installation steps.

Last fiddled with by EdH on 2021-02-12 at 19:27
EdH is offline   Reply With Quote
Old 2021-02-04, 03:34   #2
LaurV
Romulan Interpreter
 
LaurV's Avatar
 
"name field"
Jun 2011
Thailand

980810 Posts
Default



I am trying to run this, it works fine for small composites, as long as the internal ECM is used, but it seems to crash when it comes to external ECM and GGNFS steps (after running the third internal "pm1" step). Are you sure the path to the ecm is in usr/bin and not in the home/content/ecm/blahblah? Ad do you need the ggnfs files in Linux? (in Windows you do, you can't run yafu without them, for composites larger than ~~110 digits, but maybe they are installed by default in Linux? I have no idea).

Anyhow, a nice beginning.

Last fiddled with by LaurV on 2021-02-04 at 03:37
LaurV is offline   Reply With Quote
Old 2021-02-04, 14:24   #3
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3·372 Posts
Default

Quote:
Originally Posted by LaurV View Post


I am trying to run this, it works fine for small composites, as long as the internal ECM is used, but it seems to crash when it comes to external ECM and GGNFS steps (after running the third internal "pm1" step). Are you sure the path to the ecm is in usr/bin and not in the home/content/ecm/blahblah? Ad do you need the ggnfs files in Linux? (in Windows you do, you can't run yafu without them, for composites larger than ~~110 digits, but maybe they are installed by default in Linux? I have no idea).

Anyhow, a nice beginning.
Ah, sorry, but this does not run NFS, only SIQS ATM. I forgot to mention that it is compiled without the NFS switch (done now). I need to see if B2 has reinstated the link to his sievers and possibly add NFS. (I would also have to add Msieve.) But, in the overall scheme, will a Colab session survive the time required for two threads to perform an NFS run?

As to the GMP-ECM, it should be in the normal linux place. From a Colab instance:
Code:
!whereis ecm
ecm: /usr/local/bin/ecm
i haven't seen any complaint about not finding GMP-ECM during any of the runs.


For fun, I started a Colab instance against 4788 to see what it would do (or not do):
Code:
fac: factoring 521522190388785541331160787195976801952432005240028067707250869703871311778809493591678394522430885815662329064194127806602359010987662524451917763292547675317210495917002745173292704484659 
fac: using pretesting plan: normal 
fac: no tune info: using qs/gnfs crossover of 93 digits 
div: primes less than 10000 
rho: x^2 + 3, starting 200 iterations on C189 
rho: x^2 + 2, starting 200 iterations on C189 
rho: x^2 + 1, starting 200 iterations on C189 
pm1: starting B1 = 150K, B2 = gmp-ecm default on C189 
ecm: 30/30 curves on C189, B1=2K, B2=gmp-ecm default ecm: 
74/74 curves on C189, B1=11K, B2=gmp-ecm default ecm: 
214/214 curves on C189, B1=50K, B2=gmp-ecm default, ETA: 1 sec 
pm1: starting B1 = 3750K, B2 = gmp-ecm default on C189 
ecm: 430/430 curves on C189, B1=250K, B2=gmp-ecm default, ETA: 2 sec 
 pm1: starting B1 = 15M, B2 = gmp-ecm default on C189
Then I tried something more reasonable:
Code:
fac: factoring 141210648600325541705449110385440866025525895296676434777837870153850236136035700644737172874485493349 
fac: using pretesting plan: normal 
fac: no tune info: using qs/gnfs crossover of 93 digits 
div: primes less than 10000 
rho: x^2 + 3, starting 200 iterations on C102  
rho: x^2 + 2, starting 200 iterations on C102  
rho: x^2 + 1, starting 200 iterations on C102  
pm1: starting B1 = 150K, B2 = gmp-ecm default on C102 
ecm: 30/30 curves on C102, B1=2K, B2=gmp-ecm default 
ecm: 74/74 curves on C102, B1=11K, B2=gmp-ecm default 
ecm: 214/214 curves on C102, B1=50K, B2=gmp-ecm default, ETA: 0 sec  
pm1: starting B1 = 3750K, B2 = gmp-ecm default on C102 
ecm: 430/430 curves on C102, B1=250K, B2=gmp-ecm default, ETA: 1 sec  
pm1: starting B1 = 15M, B2 = gmp-ecm default on C102 
ecm: 170/170 curves on C102, B1=1M, B2=gmp-ecm default, ETA: 5 sec  
nfs has not been enabled 
 
starting SIQS on c102: 141210648600325541705449110385440866025525895296676434777837870153850236136035700644737172874485493349 
 
==== sieving in progress ( 2 threads):  131760 relations needed ==== 
====            Press ctrl-c to abort and save state            ====
Thanks for the reply.
EdH is offline   Reply With Quote
Old 2021-02-04, 18:00   #4
firejuggler
 
firejuggler's Avatar
 
"Vincent"
Apr 2010
Over the rainbow

2·32·149 Posts
Default

And a reallly small trouble : if the sequence finish or is prime, it doesn't work.
Attached Thumbnails
Click image for larger version

Name:	Screenshot_9.jpg
Views:	111
Size:	77.5 KB
ID:	24267  

Last fiddled with by firejuggler on 2021-02-04 at 18:38
firejuggler is offline   Reply With Quote
Old 2021-02-04, 18:54   #5
firejuggler
 
firejuggler's Avatar
 
"Vincent"
Apr 2010
Over the rainbow

2·32·149 Posts
Default

Ah! it get stuck if the largest factor is prime and one of the other is compositre. aka sequence

20210202

Last fiddled with by firejuggler on 2021-02-04 at 18:55
firejuggler is offline   Reply With Quote
Old 2021-02-04, 19:02   #6
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3×372 Posts
Default

Quote:
Originally Posted by firejuggler View Post
Ah! it get stuck if the largest factor is prime and one of the other is compositre. aka sequence

20210202
Ah, yes. I forgot about that trouble. I will have to address that. Thanks for reminding me.
EdH is offline   Reply With Quote
Old 2021-02-05, 10:49   #7
LaurV
Romulan Interpreter
 
LaurV's Avatar
 
"name field"
Jun 2011
Thailand

24·613 Posts
Default

I already put some terms to sequence 2880388 using colab and your script (yes, I properly reserved it, the sequences I was working were all at NFS level, but this is still SIQS-ing OK).

Got the mentioned loop (at index 756, the 41 digit factor is prime, the cofactor is 42 digits composite) but the DB elves got me out of it without my intervention. If you report the factors, the elves will factor the composite after a while, and the colab script will continue normally. Of course, this is not the desired solution, as the time in between is lost with looping.

Please fix it and post it back at your convenience.

Last fiddled with by LaurV on 2021-02-05 at 10:54
LaurV is offline   Reply With Quote
Old 2021-02-05, 15:23   #8
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

100B16 Posts
Default

Quote:
Originally Posted by LaurV View Post
I already put some terms to sequence 2880388 using colab and your script (yes, I properly reserved it, the sequences I was working were all at NFS level, but this is still SIQS-ing OK).

Got the mentioned loop (at index 756, the 41 digit factor is prime, the cofactor is 42 digits composite) but the DB elves got me out of it without my intervention. If you report the factors, the elves will factor the composite after a while, and the colab script will continue normally. Of course, this is not the desired solution, as the time in between is lost with looping.

Please fix it and post it back at your convenience.
Thanks for the reply. I have a fix that I did somewhere else, but I haven't had a chance to dig it out and modify it for Colab. I think I had to do something odd like look for a color (representing composite) instead of the last term.

I hope to do this soon, but other things are distracting me.
EdH is offline   Reply With Quote
Old 2021-02-05, 15:30   #9
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

100111001110002 Posts
Default

Quote:
Originally Posted by EdH View Post
I hope to do this soon, but other things are distracting me.
"Life? Don't talk to me about life."
chalsall is online now   Reply With Quote
Old 2021-02-06, 21:23   #10
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3·372 Posts
Default

I believe I have fixed the composite retrieve issue and edited the first post to reflect the changes. It should now retrieve the composite whether it is the last entry or not.

Please let me know if this is run into again.

Thanks for all the feedback!
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
How I Create a Colab Session That Factors factordb Composites with YAFU EdH EdH 19 2021-07-29 22:56
How I Create a Colab Session That Improves Some Polynomials EdH EdH 0 2020-12-11 20:33
How I Create a Colab Session That Factors factordb Small Composites with PARI/GP EdH EdH 0 2020-11-21 17:12
How I Create a Colab Session That Factors factordb Comps with the GPU branch of GMP-ECM and msieve EdH EdH 0 2019-12-07 19:51
How I Create a Colab Session That Factors factordb Composites with the GPU branch of GMP-ECM EdH EdH 0 2019-12-04 01:49

All times are UTC. The time now is 18:24.


Mon Nov 29 18:24:15 UTC 2021 up 129 days, 12:53, 0 users, load averages: 1.49, 1.34, 1.30

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.