mersenneforum.org > EdH How I install Aliqueit onto my Ubuntu Machines
 Register FAQ Search Today's Posts Mark Forums Read

 2020-06-15, 13:02 #1 EdH     "Ed Hall" Dec 2009 Adirondack Mtns 3,541 Posts How I install Aliqueit onto my Ubuntu Machines (Note: I expect to keep the first post of each of these "How I Install..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.) This thread will explain the steps I use to install* Aliqueit onto a computer which is already running Ubuntu. This procedure should work for other linux distributions as well but the only other one I've currently tested is Debain. *In this instance "install" is referring to the acquiring and compilation of the Aliqueit package only. The binary will have to be called using its respective path. I will expect the user of these steps to be able to use the sudo command. I will use a directory called Math for these installations. You may elect to replace "Math" with your own choice. These instructions can be used as a reference only, if you want, but I will provide specifics that work for me. If you follow these steps as I provide them, you should end up with a working installation. Special Note: The Aliqueit package currently has an obsolete item permanently embedded in the compiled linux version. I don't know about the Windows version. This item can be worked around and only effects certain things, but I will address this item in the editing and recompilation of the source code. B2scale, embedded within the code, is no longer supported by GMP-ECM. Using the -y option to let YAFU perform all factoring will bypass this. First, follow the procedures in: How I Install GMP onto my Ubuntu Machines , How I Install GMP-ECM onto my Ubuntu Machines , How I Install Msieve onto my Ubuntu Machines , How I Install YAFU onto my Ubuntu Machines and How I Install ggnfs onto my Ubuntu Machines (I will actually cover several options, to include using CADO-NFS. If the -y option for YAFU will be used, ggnfs sievers are needed.) Finally, obtain Aliqueit (named aliqueit-master.zip) from the following link: https://github.com/ChristianBeer/aliqueit/archive/master.zip In keeping with the theme of my other "How I. . ." pages, I will suggest creating a folder within the Math folder for all your Aliqueit work. You may, of course, choose a completely different approach. Extract aliqueit-master folder into the Math folder. Rename the aliqueit-master directory to Aliqueit. In a terminal move into the Math/Aliqueit/src folder, copy gmpxx.h to the include directory, compile aliqueit, move the binary into the Aliqueit folder and test it: Code: cd ~/Math/Aliqueit/src cp ~/Math/gmp/gmpxx.h ~/Math/Aliqueit/3rdParty/include/. make mv aliqueit ~/Math/Aliqueit/aliqueit cd .. ./aliqueit Before you can do much else, aliqueit.ini will need to be edited. This will be addressed in the next section. Basic installation (run aliqueit using YAFU[-y]): In a terminal type the following to find your user path. These examples are based on my Math/ directory structure: Code: echo $HOME You should see something like the following: Code: /home/ You will use this information to set up the aliqueit.ini file. Open the aliqueit.ini file with your preferred editor and change the following items: Code: //Name (and path, if needed) of your yafu executable. yafu_cmd = /Math/yafu/yafu //Name (and path, if needed) of your msieve executable. msieve_cmd = /Math/msieve/msieve Note, that although you have given the path for the YAFU binary, in order for YAFU to find the yafu.ini file, it must exist in the calling directory. Place a copy of yafu.ini in the Aliqueit folder. Let's look at the basic invocation of Aliqueit again: Code: ./aliqueit This shows a page about Aliqueit with a list of options that can be supplied on the command line. Note the option -y which tells Aliqueit to perform all external factoring routines via YAFU. Also note -t which we will use briefly for testing. The basic use for earlier versions was to give Aliqueit a starting number and turn it loose. If it found a corresponding .elf file, it would check its integrity and continue from there. Otherwise, it would start fresh generating a new .elf file. This has changed somewhat in that if an .elf file is not found, it stops. You must now explicitly tell Aliqueit to start from scratch. This will be shown using the -i option below. The default name for the .elf file is alq_.elf. This is alterable in the .ini file, but I will not cover this here and simply work with the default. If you download an existing .elf file from factordb, remember to add the "alq_" prefix before you start Aliqueit. Now, let's check out the workings. Try the following from within the Aliqueit folder: Code: ./aliqueit -i 0 123456789 123456789 You should see the following: Code: Reading config file... Precalcing primes for trial factoring... Preloading mersenne primes for trial factoring... seq = 123456789 index = 0 value = 123456789 (9 digits) 0 . c9 = 123456789 = 3^2 * 3607 * 3803 : No driver 1 . c8 = 54966027 = 3 * 2459 * 7451 : Decreased! Sequence merges with earlier sequence 54966027 at index 1. Note that everything stopped as the value for the next line fell below the value of the base number, signalling a merge. This behavior can be changed within the .ini file, but we'll leave the default for now. Now that we have an .elf file, let's try: Code: ./aliqueit -t 123456789 This should simply verify the .elf file and the merge. At this point, everything should be ready to simply use the following to run Aliqueit using YAFU for all external factoring processes: Code: ./aliqueit -y  If you wish Aliqueit to stop at a particular size or after the first new line, you can review the list of options and use -d, -c, etc. Source code editing to bring Aliqueit more up-to-date: Editing and recompiling should only be done if you need to for the particular function described below. All editing will be done on the aliqueit.cc file within the "src" directory. The reason and its description follow: If Aliqueit will be used to call for ECM, either directly or via ecm.py, all references to B2scale must be removed from aliqueit.cc. The references to B2scale within the aliqueit.ini file do not need to be edited. Edits to make: Line 253 in the aliqueit.cc file - delete the highlighted portion: Code: system( ( "echo " + input_number + " | " + cfg.ecm_cmd + " -pm1 -B2scale " Line 254 in the aliqueit.cc file - delete the highlighted portion: Code: + tostring( cfg.b2scale_pm1 ) + " " + tostring( pb1 ) + " > " + cfg.ecm_tempfile ).c_str() ); Line 267 in the aliqueit.cc file - delete the highlighted portion: Code: system( ( "echo " + input_number + " | " + cfg.ecm_cmd + " -one -pp1 -c 3 -B2scale " Line 268 in the aliqueit.cc file - delete the highlighted portion: Code: + tostring( cfg.b2scale_pp1 ) + " " + tostring( pb1 ) + " > " + cfg.ecm_tempfile ).c_str() ); Line 283 in the aliqueit.cc file - delete the highlighted portion: Code: + " -B2scale " + tostring( cfg.b2scale_ecm )+ " -out " + cfg.ecm_tempfile + " " + tostring( b1 ) ).c_str() ); Line 286 in the aliqueit.cc file - delete the highlighted portion: Code: + " -B2scale " + tostring( cfg.b2scale_ecm ) + " " + tostring( b1 ) + " > " + cfg.ecm_tempfile ).c_str() ); Once the above edits are complete Aliqueit should be compiled by simply issuing the make command via a terminal within the "src" folder. Adding ecm.py and more via the aliqueit.ini Once Aliqueit is edited to fix ECM calls, it can be used with other packages than YAFU. By calling YAFU via the "-y" option, YAFU handles all the factoring by itself and simply reports the factors back to Aliqueit. This works well and is how many users have it set up. However, Aliqueit was created to also use other packages. By itself Aliqueit can invoke ECM directly, but it cannot include multi-threading of ECM. It can, however, use ecm.py to add multi-threading to the ECM call by calling ecm.py. To make this work, you need to have a copy of ecm.py which can be retrieved from this thread: A Python Diver for GMP-ECM... To get the latest version, look at or near the last post in the thread. You will also have to tell Aliqueit to use ecm.py via its aliqueit.ini file settings. Let's start there: Open the aliqueit.ini file and edit these entries: Code: //Name (and path, if needed) of your ecm.py executable. //ecmpy_cmd = python ecm.py ecmpy_cmd = python /ecm.py //Use ecm.py (for easy ecm multithreading) instead of regular ecm? use_ecmpy = true Now, let's edit the ecm.py script to set it up for your system. Open the ecm.py script and edit these entries: Code: ECM_PATH = '/usr/local/bin/' ECM_THREADS = <# of threads to use> In order to actually use ecm.py, Aliqueit will need to be run without the -y option, so YAFU isn't called for everything. However, without the -y option, you will need to specify a package to use for any gnfs that Aliqueit will need to perform. There are two basic packages that can be used. The first is a Perl package and the second is a Python version: factMsieve.pl: This can be found in the Math/ggnfs/tests folder. Place a copy in the Aliqueit folder and edit the following entry: Code: $GGNFS_BIN_PATH="/Math/ggnfs/bin"; Also, place a copy of msieve from your Math/msieve folder in the Math/ggnfs/bin folder. factmsieve.py: This can be gotten from the following link: http://brg.a2hosted.com/oldsite/computing/factmsieve.py Place it in the Aliqueit folder and edit the following lines: Code: # Set binary directory paths GGNFS_PATH = '/Math/ggnfs/bin/' MSIEVE_PATH = '/Math/msieve/msieve' # Set the number of CPU cores and threads NUM_CORES = 4 THREADS_PER_CORE = 1 USE_CUDA = False GPU_NUM = 0 MSIEVE_POLY_TIME_LIMIT = 0 Now, all that's left is to set up the following entries in aliqueit.ini: Code: //Name (and path, if needed) of your ggnfs executable. ggnfs_cmd = perl /Math/Aliqueit/factMsieve.pl or ggnfs_cmd = pyhton /Math/Aliqueit/factmsieve.py //Use msieve to find a gnfs poly? If false, ggnfs will have to find its own. use_msieve_polyfind = true //Use GNFS for composites >= this many digits. gnfs_cutoff = 95 At this point, Aliqueit should be at least running on all the threads. Of course, the -y option to use YAFU for everything was able to do that! But, YAFU isn't currently able to distribute to more than one machine. With a little modification, the previous use of factmsieve/factMsieve can be distributed to multiple machines. For more information on a manner to do this, see the thread titled: How I Run a Larger Factorization Using Msieve, gnfs and factmsieve.py on Several Ubuntu Machines I'll leave such implementation as an exercise for the reader. Although Aliqueit labels its ecm and ggnfs commands as ecm.py and factmsieve.py, in actuality they are just scripts that accept an input from Aliqueit, call appropriate programs and place factors how and where Aliqueit can find them when control is returned. By calling your own scripts, this can be expanded to run whatever you like. You don't really even need to run ECM when Aliqueit calls for ECM, but in this case, an ECM program is better, since Aliqueit steps through a variety of curves and B1 values driven by the size of the composite. In the following section, I will be explaining how to use ecmpi across multiple machines in place of ecm.py, and how to choose CADO-NFS as the ggnfs package of choice. As already mentioned, factmsieve/factMsieve can be invoked with multiple machines, but CADO-NFS is even easier to use in that capacity. Since this particular post has gotten extremely long, the next (and, final) section will be continued in the next post. . . Last fiddled with by EdH on 2020-08-09 at 15:28
 2020-06-19, 18:47 #2 EdH     "Ed Hall" Dec 2009 Adirondack Mtns DD516 Posts How I install Aliqueit onto my Ubuntu Machines (continued) Setting up Aliqueit to distribute both ECM and GNFS across several machines For this section, if you don't already have a working ecmpi cluster, you should follow: How I Install and Run ecmpi Across Several Ubuntu Machines Also make sure your Aliqueit (server) main machine can freely comunicate with the main ecmpi node. You will need to send (scp) files back and forth between the two. You will also need a working set of CADO-NFS machines to distribute outside the local machine. In this case, if you set up the CADO-NFS main machine, it can factor by itself, but you should also, easlily be able to add clients. Again, if you need to set up a CADO-NFS system, you can review the information here: How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines I'll cover ecmpi first. The suggested use for the ECM programs is to pipe the composite into the ECM program with various switches to provide additional control. All that is needed to redirect this call to an ecmpi cluster is a script to capture the call, transform it to what ecmpi needs and translate any factors found into somewhere that Aliqueit can recognize it when the script returns control. In practice, I have two scripts and they might not be well formed. But, they work for me as is and I do adjust them, if needed. I would consider them a starting point that you can build from for your own desires. The first script (aliECM.sh) works directly with Aliqueit. It captures the call for ECM and translates the call into composite, curves and B1 value, which are then written to an intermediate file (aliqueitECM). The intermediate file is sent (via scp) to the ecmpi controlling node. Then, aliECM.sh waits for the results (alqueitECMresults) from the ecmpi run. When the results are returned, the script looks in the last line for an asterisk. If one is found, this signifies a factor was returned. In actuality, at least two are returned if any are found, but I just work with the last one. This factor is reformatted to lines that Aliqueit can recognize and placed in a temp file (the name of which was also provided in the ECM call) for Aliqueit to find. The waiting for both scripts uses inotifywait. You will need to install inotify-tools to use it. The script to call the ecmpi process (aliqueitECM.sh) sits in a loop waiting for the aliqueitECM file to appear. When it does, the values from the intermediate file are used to call the ecmpi cluster to action and the output is piped into aliqueitECMresults. When the full ecmpi run is complete, the result file is sent to the Aliqueit machine where it is processed as described in the previous paragraph. The first thing you'll want to do is to change the following in aliqueit.ini: Code: ecmpy_cmd = bash /Math/Aliqueit/aliECM.sh aliECM.sh: Code: #/bin/bash/ # delete results file so inotifywait will wait rm aliECMresults # capture composite from pipe read lastpipe in # send composite to aliqueitECM file echo "N: $lastpipe" >aliqueitECM # send # of curves to run to aliqueitECM file # The actual number run will be the next multiple # of cores above the # of curves requested echo "curves:$3" >>aliqueitECM # send B1 value to aliqueitECM echo "B1: $6" >>aliqueitECM # send aliqueitECM to the master node of the # ecmpi cluster scp aliqueitECM mpi@:ecmpi/work/. # wait for aliECMresults echo "Waiting for results. . ." while [ ! -e aliECMresults ] do inotifywait -qq ./ done # check aliECMresults to see if factors were returned check=$(cat aliECMresults | grep "\*") # if yes, process the last line into Aliqueit familiar # lines and tack them to the end of the temp filename # provided by Aliqueit with the call if [ ${#check} -gt 1 ] then ind=$(echo expr index "$check" \*) factor=${check:$ind+1} echo "********** Factor found in step 2:$factor" >>$5 echo "Found prime factor of${#factor} digitis: $factor" >>$5 fi # give everything a chance to settle before returning control # to Aliqueit. (not really needed) sleep 1 Here's aliqueitECM.sh, the script that handles the ecmpi process: Code: #!/bin/bash/ # this finds out how many cores (threads, actually) are available function countcores { ccount=0 exec <"../hostfile" while read line do let ccount=${ccount}+${line:${#line}-1} done # echo "$ccount cores available" } # This provides an "endless" loop while [ ! -e aliqueitECMstop ] do # This waits for aliqueitECM to show up with the values while [ ! -e aliqueitECM ] do inotifywait -qqt 600 ./ done # clear the previous results rm aliECMresults # another check that aliqueitECM has appeared if [ -e aliqueitECM ] then # This extracts the composite, curves and B1 to use exec <"aliqueitECM" while read line do case $line in "curves:"*) curves=${line:8};; esac case $line in "N: "*) comp=${line:3};; esac case $line in "B1: "*) b1=${line:4};; esac done # echo "comp is ${comp:0:7}...${comp:-2}<${#comp}>" countcores echo "Running$curves curves with $ccount cores @$b1 on ${comp:0:7}...${comp:${#comp}-2}<${#comp}>" # This invokes ecmpi and sends the output to the result file mpirun -np $ccount --hostfile ../hostfile ../ecmpi -N$comp -nb $curves -B1$b1 >>aliECMresults # This sends the result file back to the Aliqueit machine scp aliECMresults :Math/GamAli/. # Clear current aliqueitECM to wait for the next one rm aliqueitECM fi done Let's move to the GNFS process now. Aliqueit uses a call to either factmsieve.py or factMsieve.pl to perform GNFS, We're going to use that call to invoke CADO-NFS instead. In this case, Aliqueit creates a directory of the form "ggnfs_" and places a file (test.n) in the directory. The file test.n has only "n: " in it. Then it changes to the ggnfs... directory and calls the chosen program with the name "test" as part of the call. Then Aliqueit waits for factors to show up in the ggnfs... directory in a file called "ggnfs.log" upon returned control. The script alitest.sh will be used to call CADO-NFS with the composite and then retrieve the factors from the log file and place them in ggnfs.log. First, you need a working setup of CADO-NFS. If you need to review anything to get your setup running, you can check: How I Install CADO-NFS onto my Ubuntu Machines and How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines Next, edit the aliqueit.ini file: Code: ggnfs_cmd = bash /Math/Aliqueit/alitest.sh CADO-NFS, by default, uses a temporary directory, which it creates with a random name. It also chooses the base name for all the files, from the name given in the appropriate parameters file. This is very useful in keeping jobs separated, but it makes it challenging for an external program to harvest data. On the positive side, keeping jobs separated helps CADO-NFS to perform smoothly and keep from crashing. In order to make retrieval of the factors easier, we will provide the working directory name and base name for our runs, but we also need to ensure we delete whatever may be left to corrupt a new run. The alitest.sh script will take care of this: alitest.sh: Code: #!/bin/bash/ #retrieve the composite from the call comp=$(cat$1.n) comp=${comp:3} #remove work from prior number so we can use the same names rm -R /tmp/alicado #move into CADO-NFS directory cd$HOME/Math/cado-nfs #run CADO-NFS (change options as desired) #CADO-NFS will provide a line to use to attach clients ./cado-nfs.py $comp server.whitelist= server.port= tasks.workdir=/tmp/alicado name=alicado #harvest factors for CADO-NFS's log file factors=$(cat /tmp/alicado/alicado.log | grep "Factors: ") factors=${factors:54} IFS=' ' read -r -a farray <<< "$factors" numfacts=${#farray[@]} cd$HOME/Math/GamAli/ggnfs_${comp} #create ggnfs.log file for Aliqueit to find the factors echo "Number: test" >ggnfs.log echo "N =$comp" >>ggnfs.log fcount=1 while [ $fcount -lt$numfacts ] do echo "factor: ${farray[$fcount]}" >>ggnfs.log let fcount=${fcount}+1 done At this point a fully working setup should have been achieved and you can set Aliqueit to work. Note that using several machines in the above fashion means that a failure of any of the machines during ecmpi or the polyselect portion of CADO-NFS can interrupt the factoring process. Last fiddled with by EdH on 2020-08-09 at 15:54  2020-06-20, 02:20 #3 LaurV Romulan Interpreter Jun 2011 Thailand 22·2,287 Posts Very nice tutorial Ed. You almost made me to want to switch to Linux I kinda made endless roundabouts in the past to convince Aliqueit to report to the right server (using wget and a batch file, I think I posted about it). 2020-06-20, 02:54 #4 EdH "Ed Hall" Dec 2009 Adirondack Mtns 354110 Posts Quote:  Originally Posted by LaurV Very nice tutorial Ed. You almost made me to want to switch to Linux I kinda made endless roundabouts in the past to convince Aliqueit to report to the right server (using wget and a batch file, I think I posted about it). Thanks! You did and I considered linking to it. I may still later on. Wait until I add some more sections, up to my current setup with Aliqueit running ecmpi across 20 machines and CADO-NFS in place of factmsieve/ggnfs running even more than twenty for all but LA.  2020-07-01, 21:01 #5 Mr. Odd Mar 2010 5·11 Posts Following your instructions, I get this error, even after I copied gmpxx.h from the gmp folder to the src/aliqueit folder: g++ -O3 -I../gmp/include -c -o aliqueit.o aliqueit.cc In file included from aliqueit.cc:21: aliqueit.h:11:10: fatal error: gmpxx.h: No such file or directory 11 | #include | ^~~~~~~~~ compilation terminated. make: *** [Makefile:21: aliqueit.o] Error 1 2020-07-01, 21:14 #6 EdH "Ed Hall" Dec 2009 Adirondack Mtns 3,541 Posts Quote:  Originally Posted by Mr. Odd Following your instructions, I get this error, even after I copied gmpxx.h from the gmp folder to the src/aliqueit folder: g++ -O3 -I../gmp/include -c -o aliqueit.o aliqueit.cc In file included from aliqueit.cc:21: aliqueit.h:11:10: fatal error: gmpxx.h: No such file or directory 11 | #include | ^~~~~~~~~ compilation terminated. make: *** [Makefile:21: aliqueit.o] Error 1 Aliqueit will not compile with GMP-6.2.0. Are you using the latest GMP? It will run with GMP-6.2.0 if it was compiled with GMP-6.1.2. That would take reinstalling GMP-6.1.2, compiling Aliqueit and then upgrading GMP. Edit: I do reference the GMP trouble, but if this isn't the issue, let me know. 2020-07-01, 21:17 #7 Mr. Odd Mar 2010 5·11 Posts Quote:  Originally Posted by EdH Aliqueit will not compile with GMP-6.2.0. Are you using the latest GMP? It will run with GMP-6.2.0 if it was compiled with GMP-6.1.2. That would take reinstalling GMP-6.1.2, compiling Aliqueit and then upgrading GMP. Edit: I do reference the GMP trouble, but if this isn't the issue, let me know. I stuck with 6.1.2 when I did my compile in prepping for Yafu. 2020-07-01, 21:28 #8 EdH "Ed Hall" Dec 2009 Adirondack Mtns 3,541 Posts Quote:  Originally Posted by Mr. Odd I stuck with 6.1.2 when I did my compile in prepping for Yafu. I'm going to have to think about this one. I have not had this trouble with any of my machines that I recently worked with - over 20. You must have done "sudo make install" for gmp, if you compiled ECM. It almost seems like I vaguely remember this issue from long ago, but not recently. Let me do some digging. . .  2020-07-01, 21:43 #9 EdH "Ed Hall" Dec 2009 Adirondack Mtns 3,541 Posts Just out of curiosity, does your Makefile have the following lines with the same order:? Code: aliqueit:$(OBJ) $(CC)$(CFLAGS) $(OBJ)$(LIBS) -o aliqueit A long time ago, two of these were swapped and it caused compilation troubles, but I don't recall what those troubles were.
 2020-07-01, 21:58 #10 Mr. Odd   Mar 2010 5×11 Posts My Makefile has the lines as quoted. Hmm, looking closer at the Makefile, I don't have a 'gmp/include' or 'gmp/lib' folder, is that the issue? Should it point to /usr/local/ or something?
2020-07-01, 22:59   #11
EdH

"Ed Hall"
Dec 2009

1101110101012 Posts

Quote:
 Originally Posted by Mr. Odd My Makefile has the lines as quoted. Hmm, looking closer at the Makefile, I don't have a 'gmp/include' or 'gmp/lib' folder, is that the issue? Should it point to /usr/local/ or something?
As long as you used "sudo make install," you shouldn't need to change any of the gmp references in the Makefile. Having said that, though, I notice I do have paths to the directory where I compiled GMP. Maybe you should try that:
Code:
GMP_DIR = /usr/local/lib

LIBS = -L/usr/local/lib -lgmp
or, the folder where you compiled GMP. If either of those work, I'll have to add that to my first post.

 Similar Threads Thread Thread Starter Forum Replies Last Post EdH EdH 3 2019-06-24 03:42 EdH EdH 12 2019-04-16 09:28 EdH EdH 0 2019-04-04 22:33 EdH EdH 0 2018-02-22 03:31 EdH EdH 0 2018-02-21 23:48

All times are UTC. The time now is 23:30.

Wed Jan 20 23:30:48 UTC 2021 up 48 days, 19:42, 0 users, load averages: 1.10, 1.50, 1.93