A distributed ECM project for Fermat numbers

ET_ · 2017-02-04, 15:55

Code:

--- Message crossposted on Factoring and FermatSearch threads ---
---         Please follow up on FermatSearch only             ---

Hi folks. I have been asked by many users to push and enhance the usability of the ECM method for the Fermat factors search.
Many asked to "boinc-ify" the ecm stage, or set up an ECM server and distribute the work.
Many (Mark, Carlos and Yoyo among them) offered time and resourcees to come up with a new solution.
Unfortunately, Fermat numbers are quite BIG, and require a lot of time and free memory to be correctly "ecm-ed".
They also lay (far) above the limit of 1018 digits imposed by GPU-ECM.

But I think a good way to come up with a solution exists.
Prime95 writes a savefile whenever an ecm run in interrupted. The size of the savefile is related to the FFT used.
I prepared a short table with the relevant data:

Code:

Number   Size of savefile
-------------------------
F12    |     1.1 kB
F13    |     2.1 kB
F14    |     4.2 kB
F15    |     8.3 kB
F16    |    16.5 kB
F17    |    32.9 kB
F18    |    65.6 kB
F19    |   131,1 kB 
F20    |   262.2 kB
F21    |   524.4 kB
F22    |     1.0 MB
F23    |     2.1 MB
F24    |     4.2 MB
F25    |     8.4 MB
F26    |    16.8 MB
----------------------

The Idea
Instead of distributing workunits of one or more curves, often requiring days or even weeks to complete, let's redefine the meaning of a workunit like this: "How long should a PC sit on a single WU?". If we agree for (say) 3-4 hours, we might distribute a triple composed by the exponent, the correlated sigma and the actual savefile on each WU - where no savefile means a new curve.

More handling required, but...
We might also create 2 different subprojects: one for smaller PCs doing stage1, and one for stronger workstations doing high-memory stage2, possibly extending a single stage1 into more than 1 "telescopic" stage2s, using Prime95 for stage1 and gmp-ecm for stage2...

Another good reason
I noticed that more and more people is setting up microfarms using the new Single Board Computer platforms (Arduinos, Pines, Corteces, Oranges): Such SBCs would be perfect for a distributed work like this.

The requirements
Once such ambitious project is decided, we shall need the following sections to be managed:

1 - Hardware. An online server to share the WU and maintain the project.
2 - Application software. Something like a boinc wrapper, or an ecmnet server whatever the easier.
3 - Web management software. The management of the online archives and the statistics.
4 - A couple of administrators. They will monitor the traffic, get in charge of queue management and take care of the system.
5 - A lot of volunteers. They will participate and eventually spread the word on the net.

What I can do
I can handle the points 1, 3, half of 4 and be a part of 5.
My time and actual skills would prevent me from delivering a working software in less than 3 years, and I could have more RealLife(TM) issues on the way.
Moreover, my knowledge of the subject is more historical than technical (I never deep-delved into ECM): I could start now, but it would be easier with a good company on my side.
Finally, I would like to discuss pros and cons with you before applying any change to the actual (working) state of things: there are geniuses of programming, web developing, IT security, distributed computing among you, and I am asking your volunteering time and dedication for free (or, better, for a higher scope: Fermat factors discovery!).

Anybody interested in (at least) a theoretical discussion about feasibility?

Luigi

yoyo · 2017-02-04, 20:03

How long does stage 1 of a single curve runs for the required curves and composites?
How long does stage 2 runs if we run it with -maxmem 10G?

yoyo

xilman · 2017-02-04, 21:25

I could probably run a server.

I'm commissioning new hardware right now which will take over from the current Homogeneous Cunningham server.

I also know at least a little bit about security.

ET_ · 2017-02-08, 15:05

Quote:

Originally Posted by yoyo

How long does stage 1 of a single curve runs for the required curves and composites?
How long does stage 2 runs if we run it with -maxmem 10G?

yoyo

I will give you the timings as I get home for the weekend.

Luigi

ET_ · 2017-02-20, 13:22

Quote:

Originally Posted by yoyo

How long does stage 1 of a single curve runs for the required curves and composites?
How long does stage 2 runs if we run it with -maxmem 10G?

The timings for mprime are the following:

F12 : 6.00 hours
F13 : 6.14 hours
F14 : 10.35 hours
F15 : 13.09 hours
F16 : 33.25 hours

I'll post the results for F17 - F23 shortly.

Consider that a full curve of F12 or F13 would require just 3 WU of 2 hours each.

I am gathering more timings related to gmp-ecm as well, using the -chkpnt and -maxmem switches.

I will keep updating data here...

Luigi

2017-02-04, 15:55	#1
ET_ Banned "Luigi" Aug 2002 Team Italia 2·7⁴ Posts	A distributed ECM project for Fermat numbers Code: --- Message crossposted on Factoring and FermatSearch threads --- --- Please follow up on FermatSearch only --- Hi folks. I have been asked by many users to push and enhance the usability of the ECM method for the Fermat factors search. Many asked to "boinc-ify" the ecm stage, or set up an ECM server and distribute the work. Many (Mark, Carlos and Yoyo among them) offered time and resourcees to come up with a new solution. Unfortunately, Fermat numbers are quite BIG, and require a lot of time and free memory to be correctly "ecm-ed". They also lay (far) above the limit of 1018 digits imposed by GPU-ECM. But I think a good way to come up with a solution exists. Prime95 writes a savefile whenever an ecm run in interrupted. The size of the savefile is related to the FFT used. I prepared a short table with the relevant data: Code: Number Size of savefile ------------------------- F12 \| 1.1 kB F13 \| 2.1 kB F14 \| 4.2 kB F15 \| 8.3 kB F16 \| 16.5 kB F17 \| 32.9 kB F18 \| 65.6 kB F19 \| 131,1 kB F20 \| 262.2 kB F21 \| 524.4 kB F22 \| 1.0 MB F23 \| 2.1 MB F24 \| 4.2 MB F25 \| 8.4 MB F26 \| 16.8 MB ---------------------- The Idea Instead of distributing workunits of one or more curves, often requiring days or even weeks to complete, let's redefine the meaning of a workunit like this: "How long should a PC sit on a single WU?". If we agree for (say) 3-4 hours, we might distribute a triple composed by the exponent, the correlated sigma and the actual savefile on each WU - where no savefile means a new curve. More handling required, but... We might also create 2 different subprojects: one for smaller PCs doing stage1, and one for stronger workstations doing high-memory stage2, possibly extending a single stage1 into more than 1 "telescopic" stage2s, using Prime95 for stage1 and gmp-ecm for stage2... Another good reason I noticed that more and more people is setting up microfarms using the new Single Board Computer platforms (Arduinos, Pines, Corteces, Oranges): Such SBCs would be perfect for a distributed work like this. The requirements Once such ambitious project is decided, we shall need the following sections to be managed: 1 - Hardware. An online server to share the WU and maintain the project. 2 - Application software. Something like a boinc wrapper, or an ecmnet server whatever the easier. 3 - Web management software. The management of the online archives and the statistics. 4 - A couple of administrators. They will monitor the traffic, get in charge of queue management and take care of the system. 5 - A lot of volunteers. They will participate and eventually spread the word on the net. What I can do I can handle the points 1, 3, half of 4 and be a part of 5. My time and actual skills would prevent me from delivering a working software in less than 3 years, and I could have more RealLife(TM) issues on the way. Moreover, my knowledge of the subject is more historical than technical (I never deep-delved into ECM): I could start now, but it would be easier with a good company on my side. Finally, I would like to discuss pros and cons with you before applying any change to the actual (working) state of things: there are geniuses of programming, web developing, IT security, distributed computing among you, and I am asking your volunteering time and dedication for free (or, better, for a higher scope: Fermat factors discovery!). Anybody interested in (at least) a theoretical discussion about feasibility? Luigi

2017-02-04, 21:25	#3
xilman Bamboozled! "𒉺𒌌𒇷𒆷𒀭" May 2003 Down not across 3×3,529 Posts	I could probably run a server. I'm commissioning new hardware right now which will take over from the current Homogeneous Cunningham server. I also know at least a little bit about security. Last fiddled with by xilman on 2017-02-04 at 21:26

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
AlphaZero style distributed computing project for Go and Chess	MooMoo2	Lounge	5	2019-05-28 13:22
A distributed ECM project for Fermat numbers	ET_	FermatSearch	1	2017-02-22 10:42
distributed project search	drakkar67	Prime Sierpinski Project	17	2005-11-19 01:29
A distributed-computing project to optimize GIMPS FFT? Genetic algorithms	GP2	Software	10	2003-12-09 20:41
POLL - How many other distributed project are you currently	delta_t	Lounge	9	2002-12-10 09:48

2017-02-04, 20:03	#2
yoyo Oct 2006 Berlin, Germany 607 Posts	How long does stage 1 of a single curve runs for the required curves and composites? How long does stage 2 runs if we run it with -maxmem 10G? yoyo