mersenneforum.org  

Go Back   mersenneforum.org > Extra Stuff > Blogorrhea > EdH

Reply
 
Thread Tools
Old 2022-12-02, 16:06   #1
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

526110 Posts
Default How I install fastECPP onto my Ubuntu Machines

(Note: I expect to keep the first post of each of these "How I Install..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.)

This thread will explain the steps I use to install fastECPP onto a computer which is already running Ubuntu. This procedure should work for other linux distributions as well but the only other one I've currently tested is Debain.

Note: Although fastECPP can run single threaded, installing it to use MPI greatly enhances its efficiency by multi-threading and I will describe it with MPI included.

I will expect the user of these steps to be able to use the sudo command.

I will use a directory called Math for these installations. You may elect to replace "Math" with your own choice.

These instructions can be used as a reference only, if you want, but I will provide specifics that work for me. If you follow these steps as I provide them, you should end up with a working installation.

First, follow the procedure in:

How I Install GMP onto my Ubuntu Machines

or, install GMP from the repository.

The packages CM and MPFRCX from multiprecision.org will need to be acquired and installed. This can be accomplished on their site, but if you want to use fastECPP to produce Primo certificates for use with factordb.com, you will need the "dev" version, which is only listed as "tarball" in the News section (second paragraph) of the Introduction page for CM. The mpfrcx package can be acquired from the Download section for its page. You should dowload the two files (currently cm-0.4.1dev-41c4bce.tar.gz and mpfrcx-0.6.3.tar.gz) and place them in the Math folder.

Extract both cm and mpfrcx into their respective folders within the Math folder.

Open a terminal and Enter:
Code:
sudo apt update
You should be prompted for your password. This prompt should only appear once for your terminal session, unless you leave it idle for a long time.

Note: Versions of Ubuntu earlier than 20.04 do NOT have a recent enough Pari/GP in their repository. To compile on Ubuntu 18.04 or earlier, manually install Pari/GP from https://pari.math.u-bordeaux.fr/download.html and leave out the two packages pari-gp and libpari-dev in the following apt install.

After the update completes and the user prompt reappears, type the following. (If you have already installed any of these, you may omit them here, but if you don't omit them, all will still be fine):
Code:
sudo apt install mpc libmpc-dev libmpfr-dev pari-gp libpari-dev openmpi-bin openmpi-common libopenmpi-dev
Accept the prompt.

Move into the Math/mpfrcx folder and use the following to install it:
Code:
./configure
make
make check
If all tests pass, perform:
Code:
sudo make install
Now, check that libmpfrcx.so.1 is available and add its location to the LD_LIBRARY_PATH environment variable:
Code:
whereis libmpfrcx.so.1
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/
Move into the Math/cm folder and use the following to install it:
Code:
./configure --enable-mpi
make
make check
If all tests pass, perform:
Code:
sudo make install
You should now be able to run ecpp. The method I use is probably not the best but it works nicely and for PRP candidates above a couple thousand digits, it greatly outperforms Primo (if using the MPI version). In the src folder within the cm folder, there is a script called ecpp and one called ecpp-mpi. I run these with the "bash" command. They need to be run from their build directory.

ecpp-mpi can be run across a cluster as well as just across available CPU cores/threads. For info on adding external machines, check my thread on ecmpi.

ecpp[-mpi] can accept a short version for candidates and I'll use one such candidate for some tests. In a terminal within the src folder, try:
Code:
bash ecpp -n '(288879084+253!-2^253)/4' -v -f test
After lots of screen output, there should be a couple lines that read:
Code:
Writing to 'test'.
Writing to 'test.primo'.
These mean success. I have found that factordb accepts the ".primo" certificate without modification, using its upload web page. However, I have only ever tested this with candidates that are already in the db's list of PRPs. Others appear to have needed to modify their certificates by name change and N to N$ within the certificate.

To test using all the CPU cores:
Code:
mpirun bash ecpp-mpi -n '(288879084+253!-2^253)/4' -v -f testmpi
After lots of screen output, there should be a couple lines that read:
Code:
Writing to 'testmpi'.
Writing to 'testmpi.primo'.
Again, these mean success.

Aside from the increase in speed over Primo above about 1300 decimal digits, fastECPP is easier to use with scripts. Below I'm providing such a script for use in automatically processing factordb PRPs.

Notes in no particular order:
- Certificates will be processed as Anonymous unless modifications are made.
- The size to start with and the number to process can be given on the command line during the call.
- - Otherwise defaults are used.
- The script must be used in the cm/src directory or a copy of that directory, including the two hidden folders.
- - If a copy of the src folder is used, it only needs the two hidden folders and the two scripts: ecpp and ecpp-mpi, and the following script, which, for my use, I call ecppfdb.sh.

ecppfdb.sh script:
Code:
#!/bin/bash
#############################################################
# This script is designed to use fastECPP to automatically
# process PRPs from factordb.  The starting size and quantity
# can be provided on the command line when calling.  More
# information can be found at:
#    https://www.mersenneforum.org/showthread.php?t=28270
#############################################################

# default if no value found on command line
if [ ${#1} -lt 1 ]
  then
    start=300
  else
    start=$1
fi
# default if no value found on command line
if [ ${#2} -lt 1 ]
  then
    todo=100
  else
    todo=$2
fi
# cutoff to change over to mpi
if [ $start -gt 1200 ]
  then
    mpi=1
  else
    mpi=0
fi

# maximum size of random value to add
srand=250

# removes temporary files
function cleanup {
  rm primo-* 2>/dev/null
  rm primo_* 2>/dev/null
  rm *.zip 2>/dev/null
  rm temp* 2>/dev/null
  rm result 2>/dev/null
}

ccount=0
printf "Working on first candidate.\r"
while [ $todo -gt 0 ]
  do
    elapsed=$SECONDS
    ocount=0
    cleanup
    csize=$(echo $[ $RANDOM % $srand + $start ])
    wget "http://factordb.com/primobatch.php?digits=${csize}&files=1&start=Generate%20Zip" -q -O ecppIN.zip
    unzip -q ecppIN.zip
    pfile=$(ls *.in)
    size=$(grep "digits" $pfile)
    size=${size:63:4}
    can=$(tail -n 1 *.in)
    can="0x${can:3}"
    if [ $mpi -eq 0 ]
      then
        bash ecpp -n $can -f temp >/dev/null 2>/dev/null
      else
        mpirun bash ecpp-mpi -n $can -f temp >/dev/null 2>/dev/null
    fi
    sleep 1
    curl -s -F cert=@"temp.primo" http://factordb.com/uploadcert.php >>result 2>>result
    saved=$(cat result | grep "Saved" | wc -l)
    if [ $saved -lt 1 ]
      then
        echo "Certificate not accepted by factordb!     "
    fi
    let ccount=${ccount}+1
    let elapsed=${SECONDS}-${elapsed}
    printf "Processed: %d - Last Size: %d (%d:%02d/%d:%0.2d)        \r" $ccount $size $((elapsed/60)) $((elapsed%60)) $((SECONDS/60)) $((SECONDS%60))
    let todo=${todo}-1
  done
echo ""
cleanup
Here are some examples (with default start 300):
Code:
$ bash ecppfdb.sh
Processed: 100 - Last Size: 353 (0:04/6:47)        
$ bash ecppfdb.sh 375 7
Processed: 7 - Last Size: 380 (0:11/0:42)
Extra notes:
- The values in parentheses are time for last PRP and total elapsed time.
- If a certificate is not accepted for any reason, the message, "Certificate not accepted by factordb!" is displayed.
- - Unfortunately, this script will keep retrieving the same candidate if factordb has not removed it.
- You can troubleshoot by commenting the final cleanup call and reviewing the temporary files.
- - The file called "result" is the factordb .html file returned from the curl upload.
- A random value was aded to help minimize collisions of script users.
- - The "srand" value is the maximum allowed to be added to the starting size.

At some point, I intend to add a Colab session using this script. The Colab sessions will only use one core, but should be capable of certifying lots of the smaller PRPs during times when the db has been flooded.


In my tests, obviously, the mpi version runs slower than single-threaded, but for larger candidates mpi can be far superior. If you include a hostfile, you can invoke the use of all the threads, but my tests have shown only a small improvement with all threads over one per core. Then again, when you're looking at an elapsed time of months, small improvement may mean weeks.

Running across several machines requires a little extra work (building an mpi cluster with all the same userids, etc.). For now this extra work can be found in the ecmpi thread.

Last fiddled with by EdH on 2022-12-14 at 15:00
EdH is offline   Reply With Quote
Old 2022-12-02, 16:41   #2
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

24·281 Posts
Default

Great, EdH. Now can we have a post on how to run ecpp-mpi on more than one host?
paulunderwood is offline   Reply With Quote
Old 2022-12-02, 18:06   #3
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

526110 Posts
Default

Quote:
Originally Posted by paulunderwood View Post
Great, EdH. Now can we have a post on how to run ecpp-mpi on more than one host?
That is actually a bit complicated from scratch, but I hope to add it in eventually. What I need to learn is how to use fastECPP outside its "ecpp" shell script. As it stands, using the ecpp script as I do would require duplicate accounts on all the machines with identical directories and all using ssh interconnection. I will try to work on some steps, but this may take a bit of time. Sorry for the delay.

Thanks for the feedback (and all the help).
EdH is offline   Reply With Quote
Old 2022-12-03, 20:38   #4
silvio
 
Nov 2022

17 Posts
Default

I cannot make cm:
make[2]: *** [Makefile:456: libcm.la] Error 1
make[2]: Leaving directory '/home/silvio/Math/cm/lib'
make[1]: *** [Makefile:406: all-recursive] Error 1
make[1]: Leaving directory '/home/silvio/Math/cm'
make: *** [Makefile:338: all] Error 2
Any idea?
TIA
silvio is offline   Reply With Quote
Old 2022-12-03, 21:45   #5
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

5,261 Posts
Default

Have you tried a reboot? That helped one of my machines, but I haven't seen that particular error. Did everything else run smoothly to that point?
EdH is offline   Reply With Quote
Old 2022-12-03, 22:56   #6
silvio
 
Nov 2022

1116 Posts
Default

Quote:
Originally Posted by EdH View Post
Have you tried a reboot? That helped one of my machines, but I haven't seen that particular error. Did everything else run smoothly to that point?
Yep. Thank You
silvio is offline   Reply With Quote
Old 2022-12-03, 23:29   #7
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

5,261 Posts
Default

Sorry I'm not of more help. Please post any updates, progress, etc. It's possible I've already got something installed for one of the other packages that you may not. GMP comes to mind. Do you have GMP-6.2.0 manually installed or are you using libgmp-dev? All of my machines have the manual installation, but I have seen something installing the dev library to satisfy one of the other packages.
EdH is offline   Reply With Quote
Old 2022-12-04, 04:09   #8
gorignak
 
Oct 2022

5 Posts
Default

Quote:
Originally Posted by silvio View Post
I cannot make cm:
make[2]: *** [Makefile:456: libcm.la] Error 1
make[2]: Leaving directory '/home/silvio/Math/cm/lib'
make[1]: *** [Makefile:406: all-recursive] Error 1
make[1]: Leaving directory '/home/silvio/Math/cm'
make: *** [Makefile:338: all] Error 2
Any idea?
TIA
The actual error will be farther up. Can you post more of the build log?
gorignak is offline   Reply With Quote
Old 2022-12-04, 11:44   #9
silvio
 
Nov 2022

17 Posts
Default

attached is the output of make. I'm on wsl2 in windows11
TIA
Attached Files
File Type: txt error.txt (1.9 KB, 20 views)
silvio is offline   Reply With Quote
Old 2022-12-05, 04:24   #10
gorignak
 
Oct 2022

5 Posts
Default

That shows no errors, so if that's all the output it looks like your build was successful.
gorignak is offline   Reply With Quote
Old 2022-12-05, 14:24   #11
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

526110 Posts
Default

I'm afraid I have no familiarity with WSL2, so I don't think I can offer much, but I will, of course, watch these posts and see.
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
How I install YAFU onto my Ubuntu Machines EdH EdH 118 2021-09-29 12:42
How I Install GMP-ECM onto my Ubuntu Machines EdH EdH 12 2019-04-16 09:28
How I Install and Run ecmpi Across Several Ubuntu Machines EdH EdH 0 2019-04-04 22:33
How I Install msieve onto my Ubuntu Machines EdH EdH 0 2018-02-23 14:43
How I Install GMP onto my Ubuntu Machines EdH EdH 0 2018-02-21 23:48

All times are UTC. The time now is 17:38.


Tue Feb 7 17:38:34 UTC 2023 up 173 days, 15:07, 1 user, load averages: 0.95, 1.07, 1.05

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

≠ ± ∓ ÷ × · − √ ‰ ⊗ ⊕ ⊖ ⊘ ⊙ ≤ ≥ ≦ ≧ ≨ ≩ ≺ ≻ ≼ ≽ ⊏ ⊐ ⊑ ⊒ ² ³ °
∠ ∟ ° ≅ ~ ‖ ⟂ ⫛
≡ ≜ ≈ ∝ ∞ ≪ ≫ ⌊⌋ ⌈⌉ ∘ ∏ ∐ ∑ ∧ ∨ ∩ ∪ ⨀ ⊕ ⊗ 𝖕 𝖖 𝖗 ⊲ ⊳
∅ ∖ ∁ ↦ ↣ ∩ ∪ ⊆ ⊂ ⊄ ⊊ ⊇ ⊃ ⊅ ⊋ ⊖ ∈ ∉ ∋ ∌ ℕ ℤ ℚ ℝ ℂ ℵ ℶ ℷ ℸ 𝓟
¬ ∨ ∧ ⊕ → ← ⇒ ⇐ ⇔ ∀ ∃ ∄ ∴ ∵ ⊤ ⊥ ⊢ ⊨ ⫤ ⊣ … ⋯ ⋮ ⋰ ⋱
∫ ∬ ∭ ∮ ∯ ∰ ∇ ∆ δ ∂ ℱ ℒ ℓ
𝛢𝛼 𝛣𝛽 𝛤𝛾 𝛥𝛿 𝛦𝜀𝜖 𝛧𝜁 𝛨𝜂 𝛩𝜃𝜗 𝛪𝜄 𝛫𝜅 𝛬𝜆 𝛭𝜇 𝛮𝜈 𝛯𝜉 𝛰𝜊 𝛱𝜋 𝛲𝜌 𝛴𝜎𝜍 𝛵𝜏 𝛶𝜐 𝛷𝜙𝜑 𝛸𝜒 𝛹𝜓 𝛺𝜔