How I install Aliqueit onto my Ubuntu Machines

EdH · 2020-06-15, 13:02

(Note: I expect to keep the first post of each of these "How I Install..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.)

This thread will explain the steps I use to install* Aliqueit onto a computer which is already running Ubuntu. This procedure should work for other linux distributions as well but the only other one I've currently tested is Debain.

*In this instance "install" is referring to the acquiring and compilation of the Aliqueit package only. The binary will have to be called using its respective path.

I will expect the user of these steps to be able to use the sudo command.

I will use a directory called Math for these installations. You may elect to replace "Math" with your own choice.

These instructions can be used as a reference only, if you want, but I will provide specifics that work for me. If you follow these steps as I provide them, you should end up with a working installation.

Special Note: The Aliqueit package currently has an obsolete item permanently embedded in the compiled linux version. I don't know about the Windows version. This item can be worked around and only effects certain things, but I will address this item in the editing and recompilation of the source code.

B2scale, embedded within the code, is no longer supported by GMP-ECM. Using the -y option to let YAFU perform all factoring will bypass this.

First, follow the procedures in:

How I Install GMP onto my Ubuntu Machines
,
How I Install GMP-ECM onto my Ubuntu Machines
,
How I Install Msieve onto my Ubuntu Machines
,
How I Install YAFU onto my Ubuntu Machines
and
How I Install ggnfs onto my Ubuntu Machines (I will actually cover several options, to include using CADO-NFS. If the -y option for YAFU will be used, ggnfs sievers are needed.)

Finally, obtain Aliqueit (named aliqueit-master.zip) from the following link:

https://github.com/ChristianBeer/aliqueit/archive/master.zip

In keeping with the theme of my other "How I. . ." pages, I will suggest creating a folder within the Math folder for all your Aliqueit work. You may, of course, choose a completely different approach.

Extract aliqueit-master folder into the Math folder. Rename the aliqueit-master directory to Aliqueit.

In a terminal move into the Math/Aliqueit/src folder, copy gmpxx.h to the include directory, compile aliqueit, move the binary into the Aliqueit folder and test it:

Code:

cd ~/Math/Aliqueit/src
cp ~/Math/gmp/gmpxx.h ~/Math/Aliqueit/3rdParty/include/.
make
mv aliqueit ~/Math/Aliqueit/aliqueit
cd ..
./aliqueit

Before you can do much else, aliqueit.ini will need to be edited. This will be addressed in the next section.

Basic installation (run aliqueit using YAFU[-y]):

In a terminal type the following to find your user path. These examples are based on my Math/<package> directory structure:

Code:

echo $HOME

You should see something like the following:

Code:

/home/<username>

You will use this information to set up the aliqueit.ini file. Open the aliqueit.ini file with your preferred editor and change the following items:

Code:

//Name (and path, if needed) of your yafu executable.
yafu_cmd = <use the $HOME info here>/Math/yafu/yafu

//Name (and path, if needed) of your msieve executable.
msieve_cmd = <use the $HOME info here>/Math/msieve/msieve

Note, that although you have given the path for the YAFU binary, in order for YAFU to find the yafu.ini file, it must exist in the calling directory. Place a copy of yafu.ini in the Aliqueit folder.

Let's look at the basic invocation of Aliqueit again:

Code:

./aliqueit

This shows a page about Aliqueit with a list of options that can be supplied on the command line. Note the option -y which tells Aliqueit to perform all external factoring routines via YAFU. Also note -t which we will use briefly for testing.

The basic use for earlier versions was to give Aliqueit a starting number and turn it loose. If it found a corresponding .elf file, it would check its integrity and continue from there. Otherwise, it would start fresh generating a new .elf file. This has changed somewhat in that if an .elf file is not found, it stops. You must now explicitly tell Aliqueit to start from scratch. This will be shown using the -i option below.

The default name for the .elf file is alq_<base number>.elf. This is alterable in the .ini file, but I will not cover this here and simply work with the default. If you download an existing .elf file from factordb, remember to add the "alq_" prefix before you start Aliqueit.

Now, let's check out the workings. Try the following from within the Aliqueit folder:

Code:

./aliqueit -i 0 123456789 123456789

You should see the following:

Code:

Reading config file...
Precalcing primes for trial factoring...
Preloading mersenne primes for trial factoring...
seq = 123456789
index = 0
value = 123456789 (9 digits)
0 .      c9 = 123456789 = 3^2 * 3607 * 3803 : No driver
1 .      c8 = 54966027 = 3 * 2459 * 7451 : Decreased!
Sequence merges with earlier sequence 54966027 at index 1.

Note that everything stopped as the value for the next line fell below the value of the base number, signalling a merge. This behavior can be changed within the .ini file, but we'll leave the default for now.

Now that we have an .elf file, let's try:

Code:

./aliqueit -t 123456789

This should simply verify the .elf file and the merge.

At this point, everything should be ready to simply use the following to run Aliqueit using YAFU for all external factoring processes:

Code:

./aliqueit -y <base sequence number>

If you wish Aliqueit to stop at a particular size or after the first new line, you can review the list of options and use -d, -c, etc.

Source code editing to bring Aliqueit more up-to-date:

Editing and recompiling should only be done if you need to for the particular function described below. All editing will be done on the aliqueit.cc file within the "src" directory. The reason and its description follow:

If Aliqueit will be used to call for ECM, either directly or via ecm.py, all references to B2scale must be removed from aliqueit.cc. The references to B2scale within the aliqueit.ini file do not need to be edited.

Edits to make:
Line 253 in the aliqueit.cc file - delete the highlighted portion:

Code:

system( ( "echo " + input_number + " | " + cfg.ecm_cmd + " -pm1 -B2scale "

Line 254 in the aliqueit.cc file - delete the highlighted portion:

Code:

+ tostring( cfg.b2scale_pm1 ) + " " + tostring( pb1 ) + " > " + cfg.ecm_tempfile ).c_str() );

Line 267 in the aliqueit.cc file - delete the highlighted portion:

Code:

system( ( "echo " + input_number + " | " + cfg.ecm_cmd + " -one -pp1 -c 3 -B2scale "

Line 268 in the aliqueit.cc file - delete the highlighted portion:

Code:

+ tostring( cfg.b2scale_pp1 ) + " " + tostring( pb1 ) + " > " + cfg.ecm_tempfile ).c_str() );

Line 283 in the aliqueit.cc file - delete the highlighted portion:

Code:

+ " -B2scale " + tostring( cfg.b2scale_ecm )+ " -out " + cfg.ecm_tempfile + " " + tostring( b1 ) ).c_str() );

Line 286 in the aliqueit.cc file - delete the highlighted portion:

Code:

+ " -B2scale " + tostring( cfg.b2scale_ecm ) + " " + tostring( b1 ) + " > " + cfg.ecm_tempfile ).c_str() );

Once the above edits are complete Aliqueit should be compiled by simply issuing the make command via a terminal within the "src" folder.

Adding ecm.py and more via the aliqueit.ini

Once Aliqueit is edited to fix ECM calls, it can be used with other packages than YAFU. By calling YAFU via the "-y" option, YAFU handles all the factoring by itself and simply reports the factors back to Aliqueit. This works well and is how many users have it set up.

However, Aliqueit was created to also use other packages. By itself Aliqueit can invoke ECM directly, but it cannot include multi-threading of ECM. It can, however, use ecm.py to add multi-threading to the ECM call by calling ecm.py.

To make this work, you need to have a copy of ecm.py which can be retrieved from this thread:

A Python Diver for GMP-ECM...

To get the latest version, look at or near the last post in the thread. You will also have to tell Aliqueit to use ecm.py via its aliqueit.ini file settings. Let's start there:

Open the aliqueit.ini file and edit these entries:

Code:

//Name (and path, if needed) of your ecm.py executable.
//ecmpy_cmd = python ecm.py
ecmpy_cmd = python <directory path>/ecm.py

//Use ecm.py (for easy ecm multithreading) instead of regular ecm?
use_ecmpy = true

Now, let's edit the ecm.py script to set it up for your system. Open the ecm.py script and edit these entries:

Code:

ECM_PATH = '/usr/local/bin/'

ECM_THREADS = <# of threads to use>

In order to actually use ecm.py, Aliqueit will need to be run without the -y option, so YAFU isn't called for everything. However, without the -y option, you will need to specify a package to use for any gnfs that Aliqueit will need to perform. There are two basic packages that can be used. The first is a Perl package and the second is a Python version:

factMsieve.pl: This can be found in the Math/ggnfs/tests folder. Place a copy in the Aliqueit folder and edit the following entry:

Code:

$GGNFS_BIN_PATH="<whatever $HOME shows>/Math/ggnfs/bin";

Also, place a copy of msieve from your Math/msieve folder in the Math/ggnfs/bin folder.

factmsieve.py: This can be gotten from the following link:

http://brg.a2hosted.com/oldsite/computing/factmsieve.py

Place it in the Aliqueit folder and edit the following lines:

Code:

# Set binary directory paths
GGNFS_PATH = '<whatever $HOME shows>/Math/ggnfs/bin/'
MSIEVE_PATH = '<whatever $HOME shows>/Math/msieve/msieve'

# Set the number of CPU cores and threads
NUM_CORES = 4
THREADS_PER_CORE = 1

USE_CUDA = False
GPU_NUM = 0
MSIEVE_POLY_TIME_LIMIT = 0

Now, all that's left is to set up the following entries in aliqueit.ini:

Code:

//Name (and path, if needed) of your ggnfs executable.

ggnfs_cmd = perl <whatever $HOME shows>/Math/Aliqueit/factMsieve.pl

or

ggnfs_cmd = pyhton <whatever $HOME shows>/Math/Aliqueit/factmsieve.py

//Use msieve to find a gnfs poly? If false, ggnfs will have to find its own.
use_msieve_polyfind = true

//Use GNFS for composites >= this many digits.
gnfs_cutoff = 95

At this point, Aliqueit should be at least running on all the threads. Of course, the -y option to use YAFU for everything was able to do that! But, YAFU isn't currently able to distribute to more than one machine. With a little modification, the previous use of factmsieve/factMsieve can be distributed to multiple machines. For more information on a manner to do this, see the thread titled:

How I Run a Larger Factorization Using Msieve, gnfs and factmsieve.py on Several Ubuntu Machines

I'll leave such implementation as an exercise for the reader.

Although Aliqueit labels its ecm and ggnfs commands as ecm.py and factmsieve.py, in actuality they are just scripts that accept an input from Aliqueit, call appropriate programs and place factors how and where Aliqueit can find them when control is returned. By calling your own scripts, this can be expanded to run whatever you like. You don't really even need to run ECM when Aliqueit calls for ECM, but in this case, an ECM program is better, since Aliqueit steps through a variety of curves and B1 values driven by the size of the composite.

In the following section, I will be explaining how to use ecmpi across multiple machines in place of ecm.py, and how to choose CADO-NFS as the ggnfs package of choice. As already mentioned, factmsieve/factMsieve can be invoked with multiple machines, but CADO-NFS is even easier to use in that capacity.

Since this particular post has gotten extremely long, the next (and, final) section will be continued in the next post. . .

EdH · 2020-06-19, 18:47

Setting up Aliqueit to distribute both ECM and GNFS across several machines

For this section, if you don't already have a working ecmpi cluster, you should follow:

How I Install and Run ecmpi Across Several Ubuntu Machines

Also make sure your Aliqueit (server) main machine can freely comunicate with the main ecmpi node. You will need to send (scp) files back and forth between the two.

You will also need a working set of CADO-NFS machines to distribute outside the local machine. In this case, if you set up the CADO-NFS main machine, it can factor by itself, but you should also, easlily be able to add clients. Again, if you need to set up a CADO-NFS system, you can review the information here:

How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines

I'll cover ecmpi first. The suggested use for the ECM programs is to pipe the composite into the ECM program with various switches to provide additional control. All that is needed to redirect this call to an ecmpi cluster is a script to capture the call, transform it to what ecmpi needs and translate any factors found into somewhere that Aliqueit can recognize it when the script returns control. In practice, I have two scripts and they might not be well formed. But, they work for me as is and I do adjust them, if needed. I would consider them a starting point that you can build from for your own desires.

The first script (aliECM.sh) works directly with Aliqueit. It captures the call for ECM and translates the call into composite, curves and B1 value, which are then written to an intermediate file (aliqueitECM). The intermediate file is sent (via scp) to the ecmpi controlling node. Then, aliECM.sh waits for the results (alqueitECMresults) from the ecmpi run. When the results are returned, the script looks in the last line for an asterisk. If one is found, this signifies a factor was returned. In actuality, at least two are returned if any are found, but I just work with the last one. This factor is reformatted to lines that Aliqueit can recognize and placed in a temp file (the name of which was also provided in the ECM call) for Aliqueit to find.

The waiting for both scripts uses inotifywait. You will need to install inotify-tools to use it. The script to call the ecmpi process (aliqueitECM.sh) sits in a loop waiting for the aliqueitECM file to appear. When it does, the values from the intermediate file are used to call the ecmpi cluster to action and the output is piped into aliqueitECMresults. When the full ecmpi run is complete, the result file is sent to the Aliqueit machine where it is processed as described in the previous paragraph.

The first thing you'll want to do is to change the following in aliqueit.ini:

Code:

ecmpy_cmd = bash <whatever $HOME shows>/Math/Aliqueit/aliECM.sh

aliECM.sh:

Code:

#/bin/bash/

# delete results file so inotifywait will wait
rm aliECMresults
# capture composite from pipe
read lastpipe in
# send composite to aliqueitECM file
echo "N: $lastpipe" >aliqueitECM
# send # of curves to run to aliqueitECM file
# The actual number run will be the next multiple
# of cores above the # of curves requested
echo "curves: $3" >>aliqueitECM
# send B1 value to aliqueitECM
echo "B1: $6" >>aliqueitECM
# send aliqueitECM to the master node of the
# ecmpi cluster
scp aliqueitECM mpi@<IP address>:ecmpi/work/.
# wait for aliECMresults
echo "Waiting for results. . ."
while [ ! -e aliECMresults ]
  do
    inotifywait -qq ./
  done
# check aliECMresults to see if factors were returned
check=$(cat aliECMresults | grep "\*")
# if yes, process the last line into Aliqueit familiar
# lines and tack them to the end of the temp filename
# provided by Aliqueit with the call
if [ ${#check} -gt 1 ]
  then
    ind=$(echo `expr index "$check" \*`)
    factor=${check:$ind+1}
    echo "********** Factor found in step 2: $factor" >>$5
    echo "Found prime factor of ${#factor} digitis: $factor" >>$5
fi
# give everything a chance to settle before returning control
# to Aliqueit. (not really needed)
sleep 1

Here's aliqueitECM.sh, the script that handles the ecmpi process:

Code:

#!/bin/bash/

# this finds out how many cores (threads, actually) are available
function countcores {
  ccount=0
  exec <"../hostfile"
    while read line
      do
        let ccount=${ccount}+${line:${#line}-1}
      done
#  echo "$ccount cores available"
}

# This provides an "endless" loop
while [ ! -e aliqueitECMstop ]
  do
# This waits for aliqueitECM to show up with the values
    while [ ! -e aliqueitECM ]
      do
        inotifywait -qqt 600 ./
      done
# clear the previous results
    rm aliECMresults
# another check that aliqueitECM has appeared
    if [ -e aliqueitECM ]
      then
# This extracts the composite, curves and B1 to use
        exec <"aliqueitECM"
          while read line
            do
              case $line in
                "curves:"*) curves=${line:8};;
              esac
              case $line in
                "N: "*) comp=${line:3};;
              esac
              case $line in
                "B1: "*) b1=${line:4};;
              esac
            done
#        echo "comp is ${comp:0:7}...${comp:-2}<${#comp}>"
        countcores
        echo "Running $curves curves with $ccount cores @ $b1 on ${comp:0:7}...${comp:${#comp}-2}<${#comp}>"
# This invokes ecmpi and sends the output to the result file
        mpirun -np $ccount --hostfile ../hostfile ../ecmpi -N $comp -nb $curves -B1 $b1 >>aliECMresults
# This sends the result file back to the Aliqueit machine
        scp aliECMresults <Aliqueit machine>:Math/GamAli/.
# Clear current aliqueitECM to wait for the next one
        rm aliqueitECM
    fi
  done

Let's move to the GNFS process now. Aliqueit uses a call to either factmsieve.py or factMsieve.pl to perform GNFS, We're going to use that call to invoke CADO-NFS instead. In this case, Aliqueit creates a directory of the form "ggnfs_<composite>" and places a file (test.n) in the directory. The file test.n has only "n: <composite>" in it. Then it changes to the ggnfs... directory and calls the chosen program with the name "test" as part of the call. Then Aliqueit waits for factors to show up in the ggnfs... directory in a file called "ggnfs.log" upon returned control. The script alitest.sh will be used to call CADO-NFS with the composite and then retrieve the factors from the log file and place them in ggnfs.log.

First, you need a working setup of CADO-NFS. If you need to review anything to get your setup running, you can check:

How I Install CADO-NFS onto my Ubuntu Machines
and
How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines

Next, edit the aliqueit.ini file:

Code:

ggnfs_cmd = bash <whatever $HOME shows>/Math/Aliqueit/alitest.sh

CADO-NFS, by default, uses a temporary directory, which it creates with a random name. It also chooses the base name for all the files, from the name given in the appropriate parameters file. This is very useful in keeping jobs separated, but it makes it challenging for an external program to harvest data. On the positive side, keeping jobs separated helps CADO-NFS to perform smoothly and keep from crashing. In order to make retrieval of the factors easier, we will provide the working directory name and base name for our runs, but we also need to ensure we delete whatever may be left to corrupt a new run. The alitest.sh script will take care of this:

alitest.sh:

Code:

#!/bin/bash/

#retrieve the composite from the call
comp=$(cat $1.n)
comp=${comp:3}
#remove work from prior number so we can use the same names
rm -R /tmp/alicado
#move into CADO-NFS directory
cd $HOME/Math/cado-nfs
#run CADO-NFS (change options as desired)
#CADO-NFS will provide a line to use to attach clients
./cado-nfs.py $comp server.whitelist=<IP addresses of potential clients> server.port=<port#> tasks.workdir=/tmp/alicado name=alicado
#harvest factors for CADO-NFS's log file
factors=$(cat /tmp/alicado/alicado.log | grep "Factors: ")
factors=${factors:54}
IFS=' ' read -r -a farray <<< "$factors"
numfacts=${#farray[@]}
cd $HOME/Math/GamAli/ggnfs_${comp}
#create ggnfs.log file for Aliqueit to find the factors
echo "Number: test" >ggnfs.log
echo "N = $comp" >>ggnfs.log
fcount=1
while [ $fcount -lt $numfacts ]
  do
    echo "factor: ${farray[$fcount]}" >>ggnfs.log
    let fcount=${fcount}+1
  done

At this point a fully working setup should have been achieved and you can set Aliqueit to work.

Note that using several machines in the above fashion means that a failure of any of the machines during ecmpi or the polyselect portion of CADO-NFS can interrupt the factoring process.

LaurV · 2020-06-20, 02:20

Very nice tutorial Ed. You almost made me to want to switch to Linux

I kinda made endless roundabouts in the past to convince Aliqueit to report to the right server (using wget and a batch file, I think I posted about it).

EdH · 2020-06-20, 02:54

Quote:

Originally Posted by LaurV

Very nice tutorial Ed. You almost made me to want to switch to Linux

I kinda made endless roundabouts in the past to convince Aliqueit to report to the right server (using wget and a batch file, I think I posted about it).

Thanks! You did and I considered linking to it. I may still later on.

Wait until I add some more sections, up to my current setup with Aliqueit running ecmpi across 20 machines and CADO-NFS in place of factmsieve/ggnfs running even more than twenty for all but LA.

Mr. Odd · 2020-07-01, 21:01

Following your instructions, I get this error, even after I copied gmpxx.h from the gmp folder to the src/aliqueit folder:

g++ -O3 -I../gmp/include -c -o aliqueit.o aliqueit.cc
In file included from aliqueit.cc:21:
aliqueit.h:11:10: fatal error: gmpxx.h: No such file or directory
11 | #include <gmpxx.h>
| ^~~~~~~~~
compilation terminated.
make: *** [Makefile:21: aliqueit.o] Error 1

EdH · 2020-07-01, 21:14

Quote:

Originally Posted by Mr. Odd

Following your instructions, I get this error, even after I copied gmpxx.h from the gmp folder to the src/aliqueit folder:

g++ -O3 -I../gmp/include -c -o aliqueit.o aliqueit.cc
In file included from aliqueit.cc:21:
aliqueit.h:11:10: fatal error: gmpxx.h: No such file or directory
11 | #include <gmpxx.h>
| ^~~~~~~~~
compilation terminated.
make: *** [Makefile:21: aliqueit.o] Error 1

Aliqueit will not compile with GMP-6.2.0. Are you using the latest GMP? It will run with GMP-6.2.0 if it was compiled with GMP-6.1.2. That would take reinstalling GMP-6.1.2, compiling Aliqueit and then upgrading GMP.

Edit: I do reference the GMP trouble, but if this isn't the issue, let me know.

Mr. Odd · 2020-07-01, 21:17

Quote:

Originally Posted by EdH

Aliqueit will not compile with GMP-6.2.0. Are you using the latest GMP? It will run with GMP-6.2.0 if it was compiled with GMP-6.1.2. That would take reinstalling GMP-6.1.2, compiling Aliqueit and then upgrading GMP.

Edit: I do reference the GMP trouble, but if this isn't the issue, let me know.

I stuck with 6.1.2 when I did my compile in prepping for Yafu.

EdH · 2020-07-01, 21:28

Quote:

Originally Posted by Mr. Odd

I stuck with 6.1.2 when I did my compile in prepping for Yafu.

I'm going to have to think about this one. I have not had this trouble with any of my machines that I recently worked with - over 20. You must have done "sudo make install" for gmp, if you compiled ECM. It almost seems like I vaguely remember this issue from long ago, but not recently. Let me do some digging. . .

EdH · 2020-07-01, 21:43

Just out of curiosity, does your Makefile have the following lines with the same order:?

Code:

aliqueit: $(OBJ)
    $(CC) $(CFLAGS) $(OBJ) $(LIBS) -o aliqueit

A long time ago, two of these were swapped and it caused compilation troubles, but I don't recall what those troubles were.

Mr. Odd · 2020-07-01, 21:58

My Makefile has the lines as quoted.

Hmm, looking closer at the Makefile, I don't have a 'gmp/include' or 'gmp/lib' folder, is that the issue? Should it point to /usr/local/ or something?

EdH · 2020-07-01, 22:59

Quote:

Originally Posted by Mr. Odd

My Makefile has the lines as quoted.

Hmm, looking closer at the Makefile, I don't have a 'gmp/include' or 'gmp/lib' folder, is that the issue? Should it point to /usr/local/ or something?

As long as you used "sudo make install," you shouldn't need to change any of the gmp references in the Makefile. Having said that, though, I notice I do have paths to the directory where I compiled GMP. Maybe you should try that:

Code:

GMP_DIR = /usr/local/lib

LIBS = -L/usr/local/lib -lgmp

or, the folder where you compiled GMP. If either of those work, I'll have to add that to my first post.

2020-06-15, 13:02	#1
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2²·5·181 Posts	How I install Aliqueit onto my Ubuntu Machines (Note: I expect to keep the first post of each of these "How I Install..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.) This thread will explain the steps I use to install* Aliqueit onto a computer which is already running Ubuntu. This procedure should work for other linux distributions as well but the only other one I've currently tested is Debain. In this instance "install" is referring to the acquiring and compilation of the Aliqueit package only. The binary will have to be called using its respective path. I will expect the user of these steps to be able to use the sudo command. I will use a directory called Math for these installations. You may elect to replace "Math" with your own choice. These instructions can be used as a reference only, if you want, but I will provide specifics that work for me. If you follow these steps as I provide them, you should end up with a working installation. Special Note:* The Aliqueit package currently has an obsolete item permanently embedded in the compiled linux version. I don't know about the Windows version. This item can be worked around and only effects certain things, but I will address this item in the editing and recompilation of the source code. B2scale, embedded within the code, is no longer supported by GMP-ECM. Using the -y option to let YAFU perform all factoring will bypass this. First, follow the procedures in: How I Install GMP onto my Ubuntu Machines , How I Install GMP-ECM onto my Ubuntu Machines , How I Install Msieve onto my Ubuntu Machines , How I Install YAFU onto my Ubuntu Machines and How I Install ggnfs onto my Ubuntu Machines (I will actually cover several options, to include using CADO-NFS. If the -y option for YAFU will be used, ggnfs sievers are needed.) Finally, obtain Aliqueit (named aliqueit-master.zip) from the following link: https://github.com/ChristianBeer/aliqueit/archive/master.zip In keeping with the theme of my other "How I. . ." pages, I will suggest creating a folder within the Math folder for all your Aliqueit work. You may, of course, choose a completely different approach. Extract aliqueit-master folder into the Math folder. Rename the aliqueit-master directory to Aliqueit. In a terminal move into the Math/Aliqueit/src folder, copy gmpxx.h to the include directory, compile aliqueit, move the binary into the Aliqueit folder and test it: Code: cd ~/Math/Aliqueit/src cp ~/Math/gmp/gmpxx.h ~/Math/Aliqueit/3rdParty/include/. make mv aliqueit ~/Math/Aliqueit/aliqueit cd .. ./aliqueit Before you can do much else, aliqueit.ini will need to be edited. This will be addressed in the next section. Basic installation (run aliqueit using YAFU[-y]): In a terminal type the following to find your user path. These examples are based on my Math/<package> directory structure: Code: echo $HOME You should see something like the following: Code: /home/<username> You will use this information to set up the aliqueit.ini file. Open the aliqueit.ini file with your preferred editor and change the following items: Code: //Name (and path, if needed) of your yafu executable. yafu_cmd = <use the $HOME info here>/Math/yafu/yafu //Name (and path, if needed) of your msieve executable. msieve_cmd = <use the $HOME info here>/Math/msieve/msieve Note, that although you have given the path for the YAFU binary, in order for YAFU to find the yafu.ini file, it must exist in the calling directory. Place a copy of yafu.ini in the Aliqueit folder. Let's look at the basic invocation of Aliqueit again: Code: ./aliqueit This shows a page about Aliqueit with a list of options that can be supplied on the command line. Note the option -y which tells Aliqueit to perform all external factoring routines via YAFU. Also note -t which we will use briefly for testing. The basic use for earlier versions was to give Aliqueit a starting number and turn it loose. If it found a corresponding .elf file, it would check its integrity and continue from there. Otherwise, it would start fresh generating a new .elf file. This has changed somewhat in that if an .elf file is not found, it stops. You must now explicitly tell Aliqueit to start from scratch. This will be shown using the -i option below. The default name for the .elf file is alq_<base number>.elf. This is alterable in the .ini file, but I will not cover this here and simply work with the default. If you download an existing .elf file from factordb, remember to add the "alq_" prefix before you start Aliqueit. Now, let's check out the workings. Try the following from within the Aliqueit folder: Code: ./aliqueit -i 0 123456789 123456789 You should see the following: Code: Reading config file... Precalcing primes for trial factoring... Preloading mersenne primes for trial factoring... seq = 123456789 index = 0 value = 123456789 (9 digits) 0 . c9 = 123456789 = 3^2 * 3607 * 3803 : No driver 1 . c8 = 54966027 = 3 * 2459 * 7451 : Decreased! Sequence merges with earlier sequence 54966027 at index 1. Note that everything stopped as the value for the next line fell below the value of the base number, signalling a merge. This behavior can be changed within the .ini file, but we'll leave the default for now. Now that we have an .elf file, let's try: Code: ./aliqueit -t 123456789 This should simply verify the .elf file and the merge. At this point, everything should be ready to simply use the following to run Aliqueit using YAFU for all external factoring processes: Code: ./aliqueit -y <base sequence number> If you wish Aliqueit to stop at a particular size or after the first new line, you can review the list of options and use -d, -c, etc. Source code editing to bring Aliqueit more up-to-date: Editing and recompiling should only be done if you need to for the particular function described below. All editing will be done on the aliqueit.cc file within the "src" directory. The reason and its description follow: If Aliqueit will be used to call for ECM, either directly or via ecm.py, all references to B2scale must be removed from aliqueit.cc. The references to B2scale within the aliqueit.ini file do not need to be edited. Edits to make: Line 253 in the aliqueit.cc file - delete the highlighted portion: Code: system( ( "echo " + input_number + " \| " + cfg.ecm_cmd + " -pm1 -B2scale " Line 254 in the aliqueit.cc file - delete the highlighted portion: Code: + tostring( cfg.b2scale_pm1 ) + " " + tostring( pb1 ) + " > " + cfg.ecm_tempfile ).c_str() ); Line 267 in the aliqueit.cc file - delete the highlighted portion: Code: system( ( "echo " + input_number + " \| " + cfg.ecm_cmd + " -one -pp1 -c 3 -B2scale " Line 268 in the aliqueit.cc file - delete the highlighted portion: Code: + tostring( cfg.b2scale_pp1 ) + " " + tostring( pb1 ) + " > " + cfg.ecm_tempfile ).c_str() ); Line 283 in the aliqueit.cc file - delete the highlighted portion: Code: + " -B2scale " + tostring( cfg.b2scale_ecm )+ " -out " + cfg.ecm_tempfile + " " + tostring( b1 ) ).c_str() ); Line 286 in the aliqueit.cc file - delete the highlighted portion: Code: + " -B2scale " + tostring( cfg.b2scale_ecm ) + " " + tostring( b1 ) + " > " + cfg.ecm_tempfile ).c_str() ); Once the above edits are complete Aliqueit should be compiled by simply issuing the make command via a terminal within the "src" folder. Adding ecm.py and more via the aliqueit.ini Once Aliqueit is edited to fix ECM calls, it can be used with other packages than YAFU. By calling YAFU via the "-y" option, YAFU handles all the factoring by itself and simply reports the factors back to Aliqueit. This works well and is how many users have it set up. However, Aliqueit was created to also use other packages. By itself Aliqueit can invoke ECM directly, but it cannot include multi-threading of ECM. It can, however, use ecm.py to add multi-threading to the ECM call by calling ecm.py. To make this work, you need to have a copy of ecm.py which can be retrieved from this thread: A Python Diver for GMP-ECM... To get the latest version, look at or near the last post in the thread. You will also have to tell Aliqueit to use ecm.py via its aliqueit.ini file settings. Let's start there: Open the aliqueit.ini file and edit these entries: Code: //Name (and path, if needed) of your ecm.py executable. //ecmpy_cmd = python ecm.py ecmpy_cmd = python <directory path>/ecm.py //Use ecm.py (for easy ecm multithreading) instead of regular ecm? use_ecmpy = true Now, let's edit the ecm.py script to set it up for your system. Open the ecm.py script and edit these entries: Code: ECM_PATH = '/usr/local/bin/' ECM_THREADS = <# of threads to use> In order to actually use ecm.py, Aliqueit will need to be run without the -y option, so YAFU isn't called for everything. However, without the -y option, you will need to specify a package to use for any gnfs that Aliqueit will need to perform. There are two basic packages that can be used. The first is a Perl package and the second is a Python version: factMsieve.pl: This can be found in the Math/ggnfs/tests folder. Place a copy in the Aliqueit folder and edit the following entry: Code: $GGNFS_BIN_PATH="<whatever $HOME shows>/Math/ggnfs/bin"; Also, place a copy of msieve from your Math/msieve folder in the Math/ggnfs/bin folder. factmsieve.py: This can be gotten from the following link: http://brg.a2hosted.com/oldsite/computing/factmsieve.py Place it in the Aliqueit folder and edit the following lines: Code: # Set binary directory paths GGNFS_PATH = '<whatever $HOME shows>/Math/ggnfs/bin/' MSIEVE_PATH = '<whatever $HOME shows>/Math/msieve/msieve' # Set the number of CPU cores and threads NUM_CORES = 4 THREADS_PER_CORE = 1 USE_CUDA = False GPU_NUM = 0 MSIEVE_POLY_TIME_LIMIT = 0 Now, all that's left is to set up the following entries in aliqueit.ini: Code: //Name (and path, if needed) of your ggnfs executable. ggnfs_cmd = perl <whatever $HOME shows>/Math/Aliqueit/factMsieve.pl or ggnfs_cmd = pyhton <whatever $HOME shows>/Math/Aliqueit/factmsieve.py //Use msieve to find a gnfs poly? If false, ggnfs will have to find its own. use_msieve_polyfind = true //Use GNFS for composites >= this many digits. gnfs_cutoff = 95 At this point, Aliqueit should be at least running on all the threads. Of course, the -y option to use YAFU for everything was able to do that! But, YAFU isn't currently able to distribute to more than one machine. With a little modification, the previous use of factmsieve/factMsieve can be distributed to multiple machines. For more information on a manner to do this, see the thread titled: How I Run a Larger Factorization Using Msieve, gnfs and factmsieve.py on Several Ubuntu Machines I'll leave such implementation as an exercise for the reader. Although Aliqueit labels its ecm and ggnfs commands as ecm.py and factmsieve.py, in actuality they are just scripts that accept an input from Aliqueit, call appropriate programs and place factors how and where Aliqueit can find them when control is returned. By calling your own scripts, this can be expanded to run whatever you like. You don't really even need to run ECM when Aliqueit calls for ECM, but in this case, an ECM program is better, since Aliqueit steps through a variety of curves and B1 values driven by the size of the composite. In the following section, I will be explaining how to use ecmpi across multiple machines in place of ecm.py, and how to choose CADO-NFS as the ggnfs package of choice. As already mentioned, factmsieve/factMsieve can be invoked with multiple machines, but CADO-NFS is even easier to use in that capacity. Since this particular post has gotten extremely long, the next (and, final) section will be continued in the next post. . . Last fiddled with by EdH on 2020-08-09 at 15:28

2020-06-19, 18:47	#2
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2²×5×181 Posts	How I install Aliqueit onto my Ubuntu Machines (continued) Setting up Aliqueit to distribute both ECM and GNFS across several machines For this section, if you don't already have a working ecmpi cluster, you should follow: How I Install and Run ecmpi Across Several Ubuntu Machines Also make sure your Aliqueit (server) main machine can freely comunicate with the main ecmpi node. You will need to send (scp) files back and forth between the two. You will also need a working set of CADO-NFS machines to distribute outside the local machine. In this case, if you set up the CADO-NFS main machine, it can factor by itself, but you should also, easlily be able to add clients. Again, if you need to set up a CADO-NFS system, you can review the information here: How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines I'll cover ecmpi first. The suggested use for the ECM programs is to pipe the composite into the ECM program with various switches to provide additional control. All that is needed to redirect this call to an ecmpi cluster is a script to capture the call, transform it to what ecmpi needs and translate any factors found into somewhere that Aliqueit can recognize it when the script returns control. In practice, I have two scripts and they might not be well formed. But, they work for me as is and I do adjust them, if needed. I would consider them a starting point that you can build from for your own desires. The first script (aliECM.sh) works directly with Aliqueit. It captures the call for ECM and translates the call into composite, curves and B1 value, which are then written to an intermediate file (aliqueitECM). The intermediate file is sent (via scp) to the ecmpi controlling node. Then, aliECM.sh waits for the results (alqueitECMresults) from the ecmpi run. When the results are returned, the script looks in the last line for an asterisk. If one is found, this signifies a factor was returned. In actuality, at least two are returned if any are found, but I just work with the last one. This factor is reformatted to lines that Aliqueit can recognize and placed in a temp file (the name of which was also provided in the ECM call) for Aliqueit to find. The waiting for both scripts uses inotifywait. You will need to install inotify-tools to use it. The script to call the ecmpi process (aliqueitECM.sh) sits in a loop waiting for the aliqueitECM file to appear. When it does, the values from the intermediate file are used to call the ecmpi cluster to action and the output is piped into aliqueitECMresults. When the full ecmpi run is complete, the result file is sent to the Aliqueit machine where it is processed as described in the previous paragraph. The first thing you'll want to do is to change the following in aliqueit.ini: Code: ecmpy_cmd = bash <whatever $HOME shows>/Math/Aliqueit/aliECM.sh aliECM.sh: Code: #/bin/bash/ # delete results file so inotifywait will wait rm aliECMresults # capture composite from pipe read lastpipe in # send composite to aliqueitECM file echo "N: $lastpipe" >aliqueitECM # send # of curves to run to aliqueitECM file # The actual number run will be the next multiple # of cores above the # of curves requested echo "curves: $3" >>aliqueitECM # send B1 value to aliqueitECM echo "B1: $6" >>aliqueitECM # send aliqueitECM to the master node of the # ecmpi cluster scp aliqueitECM mpi@<IP address>:ecmpi/work/. # wait for aliECMresults echo "Waiting for results. . ." while [ ! -e aliECMresults ] do inotifywait -qq ./ done # check aliECMresults to see if factors were returned check=$(cat aliECMresults \| grep "\") # if yes, process the last line into Aliqueit familiar # lines and tack them to the end of the temp filename # provided by Aliqueit with the call if [ ${#check} -gt 1 ] then ind=$(echo `expr index "$check" \`) factor=${check:$ind+1} echo "********** Factor found in step 2: $factor" >>$5 echo "Found prime factor of ${#factor} digitis: $factor" >>$5 fi # give everything a chance to settle before returning control # to Aliqueit. (not really needed) sleep 1 Here's aliqueitECM.sh, the script that handles the ecmpi process: Code: #!/bin/bash/ # this finds out how many cores (threads, actually) are available function countcores { ccount=0 exec <"../hostfile" while read line do let ccount=${ccount}+${line:${#line}-1} done # echo "$ccount cores available" } # This provides an "endless" loop while [ ! -e aliqueitECMstop ] do # This waits for aliqueitECM to show up with the values while [ ! -e aliqueitECM ] do inotifywait -qqt 600 ./ done # clear the previous results rm aliECMresults # another check that aliqueitECM has appeared if [ -e aliqueitECM ] then # This extracts the composite, curves and B1 to use exec <"aliqueitECM" while read line do case $line in "curves:") curves=${line:8};; esac case $line in "N: ") comp=${line:3};; esac case $line in "B1: ") b1=${line:4};; esac done # echo "comp is ${comp:0:7}...${comp:-2}<${#comp}>" countcores echo "Running $curves curves with $ccount cores @ $b1 on ${comp:0:7}...${comp:${#comp}-2}<${#comp}>" # This invokes ecmpi and sends the output to the result file mpirun -np $ccount --hostfile ../hostfile ../ecmpi -N $comp -nb $curves -B1 $b1 >>aliECMresults # This sends the result file back to the Aliqueit machine scp aliECMresults <Aliqueit machine>:Math/GamAli/. # Clear current aliqueitECM to wait for the next one rm aliqueitECM fi done Let's move to the GNFS process now. Aliqueit uses a call to either factmsieve.py or factMsieve.pl to perform GNFS, We're going to use that call to invoke CADO-NFS instead. In this case, Aliqueit creates a directory of the form "ggnfs_<composite>" and places a file (test.n) in the directory. The file test.n has only "n: <composite>" in it. Then it changes to the ggnfs... directory and calls the chosen program with the name "test" as part of the call. Then Aliqueit waits for factors to show up in the ggnfs... directory in a file called "ggnfs.log" upon returned control. The script alitest.sh will be used to call CADO-NFS with the composite and then retrieve the factors from the log file and place them in ggnfs.log. First, you need a working setup of CADO-NFS. If you need to review anything to get your setup running, you can check: How I Install CADO-NFS onto my Ubuntu Machines and How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines Next, edit the aliqueit.ini file: Code: ggnfs_cmd = bash <whatever $HOME shows>/Math/Aliqueit/alitest.sh CADO-NFS, by default, uses a temporary directory, which it creates with a random name. It also chooses the base name for all the files, from the name given in the appropriate parameters file. This is very useful in keeping jobs separated, but it makes it challenging for an external program to harvest data. On the positive side, keeping jobs separated helps CADO-NFS to perform smoothly and keep from crashing. In order to make retrieval of the factors easier, we will provide the working directory name and base name for our runs, but we also need to ensure we delete whatever may be left to corrupt a new run. The alitest.sh script will take care of this: alitest.sh: Code: #!/bin/bash/ #retrieve the composite from the call comp=$(cat $1.n) comp=${comp:3} #remove work from prior number so we can use the same names rm -R /tmp/alicado #move into CADO-NFS directory cd $HOME/Math/cado-nfs #run CADO-NFS (change options as desired) #CADO-NFS will provide a line to use to attach clients ./cado-nfs.py $comp server.whitelist=<IP addresses of potential clients> server.port=<port#> tasks.workdir=/tmp/alicado name=alicado #harvest factors for CADO-NFS's log file factors=$(cat /tmp/alicado/alicado.log \| grep "Factors: ") factors=${factors:54} IFS=' ' read -r -a farray <<< "$factors" numfacts=${#farray[@]} cd $HOME/Math/GamAli/ggnfs_${comp} #create ggnfs.log file for Aliqueit to find the factors echo "Number: test" >ggnfs.log echo "N = $comp" >>ggnfs.log fcount=1 while [ $fcount -lt $numfacts ] do echo "factor: ${farray[$fcount]}" >>ggnfs.log let fcount=${fcount}+1 done At this point a fully working setup should have been achieved and you can set Aliqueit to work. Note that using several machines in the above fashion means that a failure of any of the machines during ecmpi or the polyselect portion of CADO-NFS can interrupt the factoring process. Last fiddled with by EdH on 2020-08-09 at 15:54*

2020-07-01, 21:43	#9
EdH "Ed Hall" Dec 2009 Adirondack Mtns 7044₈ Posts	Just out of curiosity, does your Makefile have the following lines with the same order:? Code: aliqueit: $(OBJ) $(CC) $(CFLAGS) $(OBJ) $(LIBS) -o aliqueit A long time ago, two of these were swapped and it caused compilation troubles, but I don't recall what those troubles were.

2020-06-20, 02:20	#3
LaurV Romulan Interpreter Jun 2011 Thailand 2³×19×61 Posts	Very nice tutorial Ed. You almost made me to want to switch to Linux I kinda made endless roundabouts in the past to convince Aliqueit to report to the right server (using wget and a batch file, I think I posted about it).

2020-07-01, 21:01	#5
Mr. Odd Mar 2010 5·11 Posts	Following your instructions, I get this error, even after I copied gmpxx.h from the gmp folder to the src/aliqueit folder: g++ -O3 -I../gmp/include -c -o aliqueit.o aliqueit.cc In file included from aliqueit.cc:21: aliqueit.h:11:10: fatal error: gmpxx.h: No such file or directory 11 \| #include <gmpxx.h> \| ^~~~~~~~~ compilation terminated. make: *** [Makefile:21: aliqueit.o] Error 1

2020-07-01, 21:58	#10
Mr. Odd Mar 2010 5·11 Posts	My Makefile has the lines as quoted. Hmm, looking closer at the Makefile, I don't have a 'gmp/include' or 'gmp/lib' folder, is that the issue? Should it point to /usr/local/ or something?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How I Install CADO-NFS onto my Ubuntu Machines	EdH	EdH	3	2019-06-24 03:42
How I Install GMP-ECM onto my Ubuntu Machines	EdH	EdH	12	2019-04-16 09:28
How I Install and Run ecmpi Across Several Ubuntu Machines	EdH	EdH	0	2019-04-04 22:33
How I Install ggnfs onto my Ubuntu Machines	EdH	EdH	0	2018-02-22 03:31
How I Install GMP onto my Ubuntu Machines	EdH	EdH	0	2018-02-21 23:48