How I install fastECPP onto my Ubuntu Machines

EdH · 2022-12-02, 16:06

(Note: I expect to keep the first post of each of these "How I Install..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.)

This thread will explain the steps I use to install fastECPP onto a computer which is already running Ubuntu. This procedure should work for other linux distributions as well but the only other one I've currently tested is Debain.

Note: Although fastECPP can run single threaded, installing it to use MPI greatly enhances its efficiency by multi-threading and I will describe it with MPI included.

I will expect the user of these steps to be able to use the sudo command.

I will use a directory called Math for these installations. You may elect to replace "Math" with your own choice.

These instructions can be used as a reference only, if you want, but I will provide specifics that work for me. If you follow these steps as I provide them, you should end up with a working installation.

First, follow the procedure in:

How I Install GMP onto my Ubuntu Machines

or, install GMP from the repository.

The packages CM and MPFRCX from multiprecision.org will need to be acquired and installed. This can be accomplished on their site, but if you want to use fastECPP to produce Primo certificates for use with factordb.com, you will need the "dev" version, which is only listed as "tarball" in the News section (second paragraph) of the Introduction page for CM. The mpfrcx package can be acquired from the Download section for its page. You should dowload the two files (currently cm-0.4.1dev-41c4bce.tar.gz and mpfrcx-0.6.3.tar.gz) and place them in the Math folder.

Extract both cm and mpfrcx into their respective folders within the Math folder.

Open a terminal and Enter:

Code:

sudo apt update

You should be prompted for your password. This prompt should only appear once for your terminal session, unless you leave it idle for a long time.

Note: Versions of Ubuntu earlier than 20.04 do NOT have a recent enough Pari/GP in their repository. To compile on Ubuntu 18.04 or earlier, manually install Pari/GP from https://pari.math.u-bordeaux.fr/download.html and leave out the two packages pari-gp and libpari-dev in the following apt install.

After the update completes and the user prompt reappears, type the following. (If you have already installed any of these, you may omit them here, but if you don't omit them, all will still be fine):

Code:

sudo apt install mpc libmpc-dev libmpfr-dev pari-gp libpari-dev openmpi-bin openmpi-common libopenmpi-dev

Accept the prompt.

Move into the Math/mpfrcx folder and use the following to install it:

Code:

./configure
make
make check

If all tests pass, perform:

Code:

sudo make install

Now, check that libmpfrcx.so.1 is available and add its location to the LD_LIBRARY_PATH environment variable:

Code:

whereis libmpfrcx.so.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/

Move into the Math/cm folder and use the following to install it:

Code:

./configure --enable-mpi
make
make check

If all tests pass, perform:

Code:

sudo make install

You should now be able to run ecpp. The method I use is probably not the best but it works nicely and for PRP candidates above a couple thousand digits, it greatly outperforms Primo (if using the MPI version). In the src folder within the cm folder, there is a script called ecpp and one called ecpp-mpi. I run these with the "bash" command. They need to be run from their build directory.

ecpp-mpi can be run across a cluster as well as just across available CPU cores/threads. For info on adding external machines, check my thread on ecmpi.

ecpp[-mpi] can accept a short version for candidates and I'll use one such candidate for some tests. In a terminal within the src folder, try:

Code:

bash ecpp -n '(288879084+253!-2^253)/4' -v -f test

After lots of screen output, there should be a couple lines that read:

Code:

Writing to 'test'.
Writing to 'test.primo'.

These mean success. I have found that factordb accepts the ".primo" certificate without modification, using its upload web page. However, I have only ever tested this with candidates that are already in the db's list of PRPs. Others appear to have needed to modify their certificates by name change and N to N$ within the certificate.

To test using all the CPU cores:

Code:

mpirun bash ecpp-mpi -n '(288879084+253!-2^253)/4' -v -f testmpi

After lots of screen output, there should be a couple lines that read:

Code:

Writing to 'testmpi'.
Writing to 'testmpi.primo'.

Again, these mean success.

Aside from the increase in speed over Primo above about 1300 decimal digits, fastECPP is easier to use with scripts. Below I'm providing such a script for use in automatically processing factordb PRPs.

Notes in no particular order:
- Certificates will be processed as Anonymous unless modifications are made.
- The size to start with and the number to process can be given on the command line during the call.
- - Otherwise defaults are used.
- The script must be used in the cm/src directory or a copy of that directory, including the two hidden folders.
- - If a copy of the src folder is used, it only needs the two hidden folders and the two scripts: ecpp and ecpp-mpi, and the following script, which, for my use, I call ecppfdb.sh.

ecppfdb.sh script:

Code:

#!/bin/bash
#############################################################
# This script is designed to use fastECPP to automatically
# process PRPs from factordb.  The starting size and quantity
# can be provided on the command line when calling.  More
# information can be found at:
#    https://www.mersenneforum.org/showthread.php?t=28270
#############################################################

# default if no value found on command line
if [ ${#1} -lt 1 ]
  then
    start=300
  else
    start=$1
fi
# default if no value found on command line
if [ ${#2} -lt 1 ]
  then
    todo=100
  else
    todo=$2
fi
# cutoff to change over to mpi
if [ $start -gt 1200 ]
  then
    mpi=1
  else
    mpi=0
fi

# maximum size of random value to add
srand=250

# removes temporary files
function cleanup {
  rm primo-* 2>/dev/null
  rm primo_* 2>/dev/null
  rm *.zip 2>/dev/null
  rm temp* 2>/dev/null
  rm result 2>/dev/null
}

ccount=0
printf "Working on first candidate.\r"
while [ $todo -gt 0 ]
  do
    elapsed=$SECONDS
    ocount=0
    cleanup
    csize=$(echo $[ $RANDOM % $srand + $start ])
    wget "http://factordb.com/primobatch.php?digits=${csize}&files=1&start=Generate%20Zip" -q -O ecppIN.zip
    unzip -q ecppIN.zip
    pfile=$(ls *.in)
    size=$(grep "digits" $pfile)
    size=${size:63:4}
    can=$(tail -n 1 *.in)
    can="0x${can:3}"
    if [ $mpi -eq 0 ]
      then
        bash ecpp -n $can -f temp >/dev/null 2>/dev/null
      else
        mpirun bash ecpp-mpi -n $can -f temp >/dev/null 2>/dev/null
    fi
    sleep 1
    curl -s -F cert=@"temp.primo" http://factordb.com/uploadcert.php >>result 2>>result
    saved=$(cat result | grep "Saved" | wc -l)
    if [ $saved -lt 1 ]
      then
        echo "Certificate not accepted by factordb!     "
    fi
    let ccount=${ccount}+1
    let elapsed=${SECONDS}-${elapsed}
    printf "Processed: %d - Last Size: %d (%d:%02d/%d:%0.2d)        \r" $ccount $size $((elapsed/60)) $((elapsed%60)) $((SECONDS/60)) $((SECONDS%60))
    let todo=${todo}-1
  done
echo ""
cleanup

Here are some examples (with default start 300):

Code:

$ bash ecppfdb.sh
Processed: 100 - Last Size: 353 (0:04/6:47)        
$ bash ecppfdb.sh 375 7
Processed: 7 - Last Size: 380 (0:11/0:42)

Extra notes:
- The values in parentheses are time for last PRP and total elapsed time.
- If a certificate is not accepted for any reason, the message, "Certificate not accepted by factordb!" is displayed.
- - Unfortunately, this script will keep retrieving the same candidate if factordb has not removed it.
- You can troubleshoot by commenting the final cleanup call and reviewing the temporary files.
- - The file called "result" is the factordb .html file returned from the curl upload.
- A random value was aded to help minimize collisions of script users.
- - The "srand" value is the maximum allowed to be added to the starting size.

At some point, I intend to add a Colab session using this script. The Colab sessions will only use one core, but should be capable of certifying lots of the smaller PRPs during times when the db has been flooded.

In my tests, obviously, the mpi version runs slower than single-threaded, but for larger candidates mpi can be far superior. If you include a hostfile, you can invoke the use of all the threads, but my tests have shown only a small improvement with all threads over one per core. Then again, when you're looking at an elapsed time of months, small improvement may mean weeks.

Running across several machines requires a little extra work (building an mpi cluster with all the same userids, etc.). For now this extra work can be found in the ecmpi thread.

paulunderwood · 2022-12-02, 16:41

Great, EdH. Now can we have a post on how to run ecpp-mpi on more than one host?

EdH · 2022-12-02, 18:06

Quote:

Originally Posted by paulunderwood

Great, EdH. Now can we have a post on how to run ecpp-mpi on more than one host?

That is actually a bit complicated from scratch, but I hope to add it in eventually. What I need to learn is how to use fastECPP outside its "ecpp" shell script. As it stands, using the ecpp script as I do would require duplicate accounts on all the machines with identical directories and all using ssh interconnection. I will try to work on some steps, but this may take a bit of time. Sorry for the delay.

Thanks for the feedback (and all the help).

silvio · 2022-12-03, 20:38

I cannot make cm:
make[2]: *** [Makefile:456: libcm.la] Error 1
make[2]: Leaving directory '/home/silvio/Math/cm/lib'
make[1]: *** [Makefile:406: all-recursive] Error 1
make[1]: Leaving directory '/home/silvio/Math/cm'
make: *** [Makefile:338: all] Error 2
Any idea?
TIA

EdH · 2022-12-03, 21:45

Have you tried a reboot? That helped one of my machines, but I haven't seen that particular error. Did everything else run smoothly to that point?

silvio · 2022-12-03, 22:56

Quote:

Originally Posted by EdH

Have you tried a reboot? That helped one of my machines, but I haven't seen that particular error. Did everything else run smoothly to that point?

Yep. Thank You

EdH · 2022-12-03, 23:29

Sorry I'm not of more help. Please post any updates, progress, etc. It's possible I've already got something installed for one of the other packages that you may not. GMP comes to mind. Do you have GMP-6.2.0 manually installed or are you using libgmp-dev? All of my machines have the manual installation, but I have seen something installing the dev library to satisfy one of the other packages.

gorignak · 2022-12-04, 04:09

Quote:

Originally Posted by silvio

I cannot make cm:
make[2]: *** [Makefile:456: libcm.la] Error 1
make[2]: Leaving directory '/home/silvio/Math/cm/lib'
make[1]: *** [Makefile:406: all-recursive] Error 1
make[1]: Leaving directory '/home/silvio/Math/cm'
make: *** [Makefile:338: all] Error 2
Any idea?
TIA

The actual error will be farther up. Can you post more of the build log?

silvio · 2022-12-04, 11:44

attached is the output of make. I'm on wsl2 in windows11
TIA

gorignak · 2022-12-05, 04:24

That shows no errors, so if that's all the output it looks like your build was successful.

EdH · 2022-12-05, 14:24

I'm afraid I have no familiarity with WSL2, so I don't think I can offer much, but I will, of course, watch these posts and see.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How I install YAFU onto my Ubuntu Machines	EdH	EdH	118	2021-09-29 12:42
How I Install GMP-ECM onto my Ubuntu Machines	EdH	EdH	12	2019-04-16 09:28
How I Install and Run ecmpi Across Several Ubuntu Machines	EdH	EdH	0	2019-04-04 22:33
How I Install msieve onto my Ubuntu Machines	EdH	EdH	0	2018-02-23 14:43
How I Install GMP onto my Ubuntu Machines	EdH	EdH	0	2018-02-21 23:48

2022-12-02, 16:06	#1
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2²·5·263 Posts	How I install fastECPP onto my Ubuntu Machines (Note: I expect to keep the first post of each of these "How I Install..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.) This thread will explain the steps I use to install fastECPP onto a computer which is already running Ubuntu. This procedure should work for other linux distributions as well but the only other one I've currently tested is Debain. Note: Although fastECPP can run single threaded, installing it to use MPI greatly enhances its efficiency by multi-threading and I will describe it with MPI included. I will expect the user of these steps to be able to use the sudo command. I will use a directory called Math for these installations. You may elect to replace "Math" with your own choice. These instructions can be used as a reference only, if you want, but I will provide specifics that work for me. If you follow these steps as I provide them, you should end up with a working installation. First, follow the procedure in: How I Install GMP onto my Ubuntu Machines or, install GMP from the repository. The packages CM and MPFRCX from multiprecision.org will need to be acquired and installed. This can be accomplished on their site, but if you want to use fastECPP to produce Primo certificates for use with factordb.com, you will need the "dev" version, which is only listed as "tarball" in the News section (second paragraph) of the Introduction page for CM. The mpfrcx package can be acquired from the Download section for its page. You should dowload the two files (currently cm-0.4.1dev-41c4bce.tar.gz and mpfrcx-0.6.3.tar.gz) and place them in the Math folder. Extract both cm and mpfrcx into their respective folders within the Math folder. Open a terminal and Enter: Code: sudo apt update You should be prompted for your password. This prompt should only appear once for your terminal session, unless you leave it idle for a long time. Note: Versions of Ubuntu earlier than 20.04 do NOT have a recent enough Pari/GP in their repository. To compile on Ubuntu 18.04 or earlier, manually install Pari/GP from https://pari.math.u-bordeaux.fr/download.html and leave out the two packages pari-gp and libpari-dev in the following apt install. After the update completes and the user prompt reappears, type the following. (If you have already installed any of these, you may omit them here, but if you don't omit them, all will still be fine): Code: sudo apt install mpc libmpc-dev libmpfr-dev pari-gp libpari-dev openmpi-bin openmpi-common libopenmpi-dev Accept the prompt. Move into the Math/mpfrcx folder and use the following to install it: Code: ./configure make make check If all tests pass, perform: Code: sudo make install Now, check that libmpfrcx.so.1 is available and add its location to the LD_LIBRARY_PATH environment variable: Code: whereis libmpfrcx.so.1 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/ Move into the Math/cm folder and use the following to install it: Code: ./configure --enable-mpi make make check If all tests pass, perform: Code: sudo make install You should now be able to run ecpp. The method I use is probably not the best but it works nicely and for PRP candidates above a couple thousand digits, it greatly outperforms Primo (if using the MPI version). In the src folder within the cm folder, there is a script called ecpp and one called ecpp-mpi. I run these with the "bash" command. They need to be run from their build directory. ecpp-mpi can be run across a cluster as well as just across available CPU cores/threads. For info on adding external machines, check my thread on ecmpi. ecpp[-mpi] can accept a short version for candidates and I'll use one such candidate for some tests. In a terminal within the src folder, try: Code: bash ecpp -n '(288879084+253!-2^253)/4' -v -f test After lots of screen output, there should be a couple lines that read: Code: Writing to 'test'. Writing to 'test.primo'. These mean success. I have found that factordb accepts the ".primo" certificate without modification, using its upload web page. However, I have only ever tested this with candidates that are already in the db's list of PRPs. Others appear to have needed to modify their certificates by name change and N to N$ within the certificate. To test using all the CPU cores: Code: mpirun bash ecpp-mpi -n '(288879084+253!-2^253)/4' -v -f testmpi After lots of screen output, there should be a couple lines that read: Code: Writing to 'testmpi'. Writing to 'testmpi.primo'. Again, these mean success. Aside from the increase in speed over Primo above about 1300 decimal digits, fastECPP is easier to use with scripts. Below I'm providing such a script for use in automatically processing factordb PRPs. Notes in no particular order: - Certificates will be processed as Anonymous unless modifications are made. - The size to start with and the number to process can be given on the command line during the call. - - Otherwise defaults are used. - The script must be used in the cm/src directory or a copy of that directory, including the two hidden folders. - - If a copy of the src folder is used, it only needs the two hidden folders and the two scripts: ecpp and ecpp-mpi, and the following script, which, for my use, I call ecppfdb.sh. ecppfdb.sh script: Code: #!/bin/bash ############################################################# # This script is designed to use fastECPP to automatically # process PRPs from factordb. The starting size and quantity # can be provided on the command line when calling. More # information can be found at: # https://www.mersenneforum.org/showthread.php?t=28270 ############################################################# # default if no value found on command line if [ ${#1} -lt 1 ] then start=300 else start=$1 fi # default if no value found on command line if [ ${#2} -lt 1 ] then todo=100 else todo=$2 fi # cutoff to change over to mpi if [ $start -gt 1200 ] then mpi=1 else mpi=0 fi # maximum size of random value to add srand=250 # removes temporary files function cleanup { rm primo-* 2>/dev/null rm primo_* 2>/dev/null rm .zip 2>/dev/null rm temp 2>/dev/null rm result 2>/dev/null } ccount=0 printf "Working on first candidate.\r" while [ $todo -gt 0 ] do elapsed=$SECONDS ocount=0 cleanup csize=$(echo $[ $RANDOM % $srand + $start ]) wget "http://factordb.com/primobatch.php?digits=${csize}&files=1&start=Generate%20Zip" -q -O ecppIN.zip unzip -q ecppIN.zip pfile=$(ls .in) size=$(grep "digits" $pfile) size=${size:63:4} can=$(tail -n 1 .in) can="0x${can:3}" if [ $mpi -eq 0 ] then bash ecpp -n $can -f temp >/dev/null 2>/dev/null else mpirun bash ecpp-mpi -n $can -f temp >/dev/null 2>/dev/null fi sleep 1 curl -s -F cert=@"temp.primo" http://factordb.com/uploadcert.php >>result 2>>result saved=$(cat result \| grep "Saved" \| wc -l) if [ $saved -lt 1 ] then echo "Certificate not accepted by factordb! " fi let ccount=${ccount}+1 let elapsed=${SECONDS}-${elapsed} printf "Processed: %d - Last Size: %d (%d:%02d/%d:%0.2d) \r" $ccount $size $((elapsed/60)) $((elapsed%60)) $((SECONDS/60)) $((SECONDS%60)) let todo=${todo}-1 done echo "" cleanup Here are some examples (with default start 300): Code: $ bash ecppfdb.sh Processed: 100 - Last Size: 353 (0:04/6:47) $ bash ecppfdb.sh 375 7 Processed: 7 - Last Size: 380 (0:11/0:42) Extra notes: - The values in parentheses are time for last PRP and total elapsed time. - If a certificate is not accepted for any reason, the message, "Certificate not accepted by factordb!" is displayed. - - Unfortunately, this script will keep retrieving the same candidate if factordb has not removed it. - You can troubleshoot by commenting the final cleanup call and reviewing the temporary files. - - The file called "result" is the factordb .html file returned from the curl upload. - A random value was aded to help minimize collisions of script users. - - The "srand" value is the maximum allowed to be added to the starting size. At some point, I intend to add a Colab session using this script. The Colab sessions will only use one core, but should be capable of certifying lots of the smaller PRPs during times when the db has been flooded. In my tests, obviously, the mpi version runs slower than single-threaded, but for larger candidates mpi can be far superior. If you include a hostfile, you can invoke the use of all the threads, but my tests have shown only a small improvement with all threads over one per core. Then again, when you're looking at an elapsed time of months, small improvement may mean weeks. Running across several machines requires a little extra work (building an mpi cluster with all the same userids, etc.). For now this extra work can be found in the ecmpi thread. Last fiddled with by EdH on 2022-12-14 at 15:00

2022-12-02, 16:41	#2
paulunderwood Sep 2002 Database er0rr 5·29·31 Posts	Great, EdH. Now can we have a post on how to run ecpp-mpi on more than one host?

2022-12-03, 20:38	#4
silvio Nov 2022 17 Posts	I cannot make cm: make[2]: * [Makefile:456: libcm.la] Error 1 make[2]: Leaving directory '/home/silvio/Math/cm/lib' make[1]: * [Makefile:406: all-recursive] Error 1 make[1]: Leaving directory '/home/silvio/Math/cm' make: *** [Makefile:338: all] Error 2 Any idea? TIA

2022-12-03, 21:45	#5
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2²×5×263 Posts	Have you tried a reboot? That helped one of my machines, but I haven't seen that particular error. Did everything else run smoothly to that point?

2022-12-03, 23:29	#7
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2²×5×263 Posts	Sorry I'm not of more help. Please post any updates, progress, etc. It's possible I've already got something installed for one of the other packages that you may not. GMP comes to mind. Do you have GMP-6.2.0 manually installed or are you using libgmp-dev? All of my machines have the manual installation, but I have seen something installing the dev library to satisfy one of the other packages.

2022-12-05, 04:24	#10
gorignak Oct 2022 5 Posts	That shows no errors, so if that's all the output it looks like your build was successful.

2022-12-05, 14:24	#11
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2²·5·263 Posts	I'm afraid I have no familiarity with WSL2, so I don't think I can offer much, but I will, of course, watch these posts and see.