FastECPP software and >50000 digit primality proof (reposted from NMBRTHRY)

[paulunderwood]

Originally Posted by xilman

The step after that is to learn how to drive mpi properly.

I am running mpi-ecpp on a 6-core machine. Without additional information it fired up one master process and 5 workers. Because of hyperthreading two processes could be interleaved at relatively little cost on each physical processor.

That said, each process will grab its own share of memory physical and virtual and you may not be able to run as many processes as processors, let alone twice that.

I a'm not going to fiddle with a(n apparently) working invocation of mpi-ecpp but now wonder whether "mpi -np 7" might be a suitable incantation for a similar job, on the grounds that there may (just) be enough RAM available.

The above assumes that the remaining machines on the local net are not set up to run mpi jobs. This is a valid assumption right now. Whether it is worth the effort of setting them up remains to be seen.

Perhaps EdH can give us a synopsis on how to run client boxes.

[EdH]

Originally Posted by paulunderwood

Perhaps EdH can give us a synopsis on how to run client boxes.

I need to read over the rest of the thread, but all my details for ecmpi are in my, "How I Install and Run ecmpi Across Several Ubuntu Machines" thread. I use openmpi, in case you're using something different.

Basically, I have an exact same user account on all the machines, with an exact same directory tree, and all machines are key linked via ssh with every other machine. This linking of all machines to all others is important. ecmpi is compiled on each machine, and the hostfile lists the localhost slots= and the user@IP slots= for each client.

I can get more detailed later. Let me look over the rest of the thread. Or, I'll try to answer more questions.

[EdH]

I'm not familiar enough with all the required libraries to implement this, yet. But, it appears that others are running openmpi and xilman knows how to incorporate LAN connected clients (which I already suspected). I'll address further questions, if any arise.

[Batalov]

I found an interesting feature in this implementation - it doesn't back-track. Could be something that is left for later work. This has a downside of sometimes taking more time in one single step than all the others combined.

In proving a 12k digit number, one step took 10 hrs (already near 10k digit size, i.e. similar to Greg's number above which took 20hrs in all, and on similar node) and then proceeded fast. This is deterministic, too - when repeated again from that step, it took ~10 hours again.

I sent the debug case input to Andreas.

[frmky]

Certification of a 15k digit prime, 2^49831-3637, took 29.7 hours using 12x Xeon E5-2670 v2 cpus with a total of 120 cores and 240 hyperthreads. The longest step of the second phase took 6.3 hours to complete.

It appears that idle processes near the end of the second phase continue to use processor time. I will next try a run with one process per core rather than two.

[Cybertronic]

Wow , 15k in 30h ..that is fast !


So I have 3 milestone 20k numbers for testing !


Smallest 20000 digit prime and prime twins.


10^19999+110949
10^19999+1514722609+d, d=0,2


Should be done, the prover extended this page...
https://pzktupel.de/smx000.html


regards

[paulunderwood]

Originally Posted by EdH

I'm not familiar enough with all the required libraries to implement this, yet. But, it appears that others are running openmpi and xilman knows how to incorporate LAN connected clients (which I already suspected). I'll address further questions, if any arise.

Regarding mpi, can ecpp-mpi be run on non-uniform processors on a network. Can clients drop out without affecting the server? Likewise, can clients be added at runtime?

As far as I can tell, the clients require passwordless ssh keys along with identical project directories and an installation of mpi. Do the clients require ecpp-mpi too?

[frmky]

Originally Posted by paulunderwood

Regarding mpi, can ecpp-mpi be run on non-uniform processors on a network.

Yes

Originally Posted by paulunderwood

Can clients drop out without affecting the server?

No

Originally Posted by paulunderwood

Likewise, can clients be added at runtime?

No

Originally Posted by paulunderwood

As far as I can tell, the clients require passwordless ssh keys along with identical project directories and an installation of mpi.

That's right. In a homogenous cluster one typically uses a shared file system to host the binaries and input files, but identical directories on separate file systems will work.

Originally Posted by paulunderwood

Do the clients require ecpp-mpi too?

Yes, each compiled for the processor on which it will be run.

[frmky]

Originally Posted by frmky

Certification of a 15k digit prime, 2^49831-3637, took 29.7 hours using 12x Xeon E5-2670 v2 cpus with a total of 120 cores and 240 hyperthreads. The longest step of the second phase took 6.3 hours to complete.

It appears that idle processes near the end of the second phase continue to use processor time. I will next try a run with one process per core rather than two.

Running one process per core reduced the time for the second phase to 3.7 hours (the time of the longest step), and the total time to 27 hours.

[xilman]

Originally Posted by frmky

Running one process per core reduced the time for the second phase to 3.7 hours (the time of the longest step), and the total time to 27 hours.

Interesting.

Looks like my choice of 1 process per core was (accidentally) correct. Still wondering whether n+1 on a n-core machine might be a better choice to soak up some of the hyperthreading and have n workers alongside the single master. Next time, perhaps.

[kruoli]

On a fixed system, fastECPP was around double the speed of Primo for a ~10k digit prime.