A Python Diver for GMP-ECM...

[EdH]

Sorry if this is a same query I might have asked before.

Has anyone expanded ecm.py to work across more than one machine. I am trying to do so, but if someone else has already, I'd be interested in their work, since I'm sure it would be better than anything I cobble together.

I'd prefer to stay away from an mpi implementation, since I'd be using machines with different capabilities and openmpi was still broken in Ubuntu 18.04, last I checked. I have run ecmpi and may even use it again on my 16.04 machines, but would prefer another route.

The direction I'm currently leaning toward is a controlling script (probably bash) that works as a server holding the basic info and keeping track of returned curve count. Then, clients grab the basic info, perform the curves and report back to the server. When the count is complete the server reports success (factors) or failure.

[RichD]

This sort of sounds like ECMnet...

[EdH]

Originally Posted by RichD

This sort of sounds like ECMnet...

I'm going to have to look into that again. I looked at it a few years ago, but don't remember why I didn't continue with it. Thanks. For now, I actually did go with ecmpi, but only with some of the similar machines. There are several more I'd like to be able to add, but they would slow down all the others. ECMnet might just allow adding them at their own pace.

[xilman]

Originally Posted by EdH

I'm going to have to look into that again. I looked at it a few years ago, but don't remember why I didn't continue with it. Thanks. For now, I actually did go with ecmpi, but only with some of the similar machines. There are several more I'd like to be able to add, but they would slow down all the others. ECMnet might just allow adding them at their own pace.

I've been running and ECMNET server for years. Email me if you think I may be able to help.

[EdH]

Originally Posted by xilman

I've been running and ECMNET server for years. Email me if you think I may be able to help.

Thanks! I'm looking it all over, but in the meantime, I've reconfigured ecmpi with a select set of better machines. I'll see where I get and contact you with any questions.

[SethTro]

@WraithX

I added ecm.py to my misc-scripts github repo (let me know if you're not happy with this).

I also made a number of changes: cleaning up unused functions, adding some tests, simplifying functions in places

Years ago I planned to make a local db with all existing B1:param:sigma to keep track of what had finished and what was queued.

[SuikaPredator]

Bug: when ecm.py receives -c XXX -threads YYY as params and 0<XXX<YYY, it will run curves indefinitely.

Potential fix: change line 2094 from

ecm_args1 += ' -c {0:d}'.format(ecm_c//intNumThreads)

into

ecm_args1 += ' -c {0:d}'.format(max(ecm_c//intNumThreads,1))

I am not sure about whether to modify variable actual_num_threads or not.

[swellman]

I must ask why would one request more threads than curves to be run? User error I suppose but never by intention, surely?

[SuikaPredator]

Originally Posted by swellman

I must ask why would one request more threads than curves to be run? User error I suppose but never by intention, surely?

When the -one option is not set and the number of remaining curves is less than the number of threads, ecm.py will enqueue that many curves to the remaining composite and cause that bug. Actually I just met that bug in this case and found out that it had ran thousands of unneeded curves.

[WraithX]

That line 2094 is from Seth's version of ecm.py. I just tried to download and run his version, but it gave me errors with both python 2 and 3, so I couldn't confirm the error.

When I tested my version of ecm.py (v0.44), with -c 3 -threads 4, it only ran 3 curves and finished as expected. However, just to be sure another code path doesn't trigger this issue I updated the referenced line to:
ecm_args1 += ' -c {0:d}'.format(max(ecm_c//intNumThreads,1))

This is version 0.46.

[SuikaPredator]

Found more bugs on max memory setting... (Seth's version on github, with possible modifications)

When the program is doing extra curves after finding the first factor, it will decrease maxmem on each number by a factor of #threads. I haven't figured out any possible fix yet.

Logs:

->=============================================================================
-> Working on number: 994165605052757388...557428370002287411 (798 digits)
-> Currently working on: job0360.txt
-> Starting 12 instances of GMP-ECM...
-> ecm -maxmem 341 -c 10 25e4 < job0360.txt > job0360_t00.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t01.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t02.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t03.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t04.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t05.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t06.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t07.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t08.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t09.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t10.txt
-> ecm -maxmem 341 -c 9 25e4 < job0360.txt > job0360_t11.txt

Using B1=250000, B2=183032866, polynomial Dickson(3), 12 threads
GMP-ECM 7.0.4 [configured with GMP 6.1.2, --enable-asm-redc] [ECM]
____________________________________________________________________________
 Curves Complete |   Average seconds/curve   |    Runtime    |      ETA
-----------------|---------------------------|---------------|--------------
   109 of    109 | Stg1 23.16s | Stg2 7.248s |   0d 00:04:54 |   0d 00:00:10

factor_data: Using B1=250000, B2=183032866, polynomial Dickson(3), sigma=1:3888514963
-> *** No factor found.


->=============================================================================
-> Working on number: 448142163283738247...836780873159362451 (803 digits)
-> Currently working on: job1983.txt
-> Starting 12 instances of GMP-ECM...
-> ecm -maxmem 28 -c 16 25e4 < job1983.txt > job1983_t00.txt
-> ecm -maxmem 28 -c 16 25e4 < job1983.txt > job1983_t01.txt
-> ecm -maxmem 28 -c 16 25e4 < job1983.txt > job1983_t02.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t03.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t04.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t05.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t06.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t07.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t08.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t09.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t10.txt
-> ecm -maxmem 28 -c 15 25e4 < job1983.txt > job1983_t11.txt

Using B1=250000, B2=146985312, polynomial Dickson(3), 12 threads
GMP-ECM 7.0.4 [configured with GMP 6.1.2, --enable-asm-redc] [ECM]
____________________________________________________________________________
 Curves Complete |   Average seconds/curve   |    Runtime    |      ETA
-----------------|---------------------------|---------------|--------------
   183 of    183 | Stg1 22.36s | Stg2 7.534s |   0d 00:08:01 |   0d 00:00:03

factor_data: Using B1=250000, B2=146985312, polynomial Dickson(3), sigma=1:1882742718
-> *** No factor found.


->=============================================================================
-> Working on number: 114127662372551733...201108166477746793 (811 digits)
-> Currently working on: job7908.txt
-> Starting 12 instances of GMP-ECM...
-> ecm -maxmem 2 -c 36 25e4 < job7908.txt > job7908_t00.txt
-> ecm -maxmem 2 -c 36 25e4 < job7908.txt > job7908_t01.txt
-> ecm -maxmem 2 -c 36 25e4 < job7908.txt > job7908_t02.txt
-> ecm -maxmem 2 -c 36 25e4 < job7908.txt > job7908_t03.txt
-> ecm -maxmem 2 -c 35 25e4 < job7908.txt > job7908_t04.txt
-> ecm -maxmem 2 -c 35 25e4 < job7908.txt > job7908_t05.txt
-> ecm -maxmem 2 -c 35 25e4 < job7908.txt > job7908_t06.txt
-> ecm -maxmem 2 -c 35 25e4 < job7908.txt > job7908_t07.txt
-> ecm -maxmem 2 -c 35 25e4 < job7908.txt > job7908_t08.txt
-> ecm -maxmem 2 -c 35 25e4 < job7908.txt > job7908_t09.txt
-> ecm -maxmem 2 -c 35 25e4 < job7908.txt > job7908_t10.txt
-> ecm -maxmem 2 -c 35 25e4 < job7908.txt > job7908_t11.txt

Using B1=250000, B2=124690660, polynomial Dickson(3), 12 threads
GMP-ECM 7.0.4 [configured with GMP 6.1.2, --enable-asm-redc] [ECM]
____________________________________________________________________________
 Curves Complete |   Average seconds/curve   |    Runtime    |      ETA
-----------------|---------------------------|---------------|--------------
   424 of    424 | Stg1 23.70s | Stg2 54.71s |   0d 00:47:22 |   0d 00:00:04

factor_data: Using B1=250000, B2=124690660, polynomial Dickson(3), sigma=1:2818859778
-> *** No factor found.


->=============================================================================
-> Working on number: 401539325668293637...329189871599519489 (807 digits)
-> Currently working on: job9196.txt
-> Starting 12 instances of GMP-ECM...
-> ecm -maxmem 0 -c 34 25e4 < job9196.txt > job9196_t00.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t01.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t02.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t03.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t04.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t05.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t06.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t07.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t08.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t09.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t10.txt
-> ecm -maxmem 0 -c 33 25e4 < job9196.txt > job9196_t11.txt

Using B1=250000, B2=183032866, polynomial Dickson(3), 12 threads
GMP-ECM 7.0.4 [configured with GMP 6.1.2, --enable-asm-redc] [ECM]
____________________________________________________________________________
 Curves Complete |   Average seconds/curve   |    Runtime    |      ETA
-----------------|---------------------------|---------------|--------------
   397 of    397 | Stg1 22.55s | Stg2 7.073s |   0d 00:16:59 |   0d 00:00:03

factor_data: Using B1=250000, B2=183032866, polynomial Dickson(3), sigma=1:1550168464
-> *** No factor found.

Also, it is not confirmed as a bug of v0.46 because I am not running that version. Maybe I will try to use v0.46 from today.