mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   GMP-ECM (https://www.mersenneforum.org/forumdisplay.php?f=55)
-   -   A Python Diver for GMP-ECM... (https://www.mersenneforum.org/showthread.php?t=15508)

WraithX 2011-04-09 21:41

A Python Diver for GMP-ECM...
 
I saw mention over in the "A Python Driver for GGNFS and MSIEVE" thread that Mini-Geek was wanting a python driver that would run multiple copies of gmp-ecm on a single number. I too have been wanting this and want to try and create that python script based off of Brian Gladman's factmsieve.py script. I don't need this for aliquiet, like Mini-Geek, I would just like a way to run many many curves simultaneously without having a bunch of command prompts open. [especially when I get a computer with dual 16-core processors ;) ]

For basic logic, I was thinking of the following:
[CODE]
on startup, check for resume and/or output files

run num_threads version of gmp-ecm
split -c over threads
split -maxmem over threads
give -save, -savea, -resume, -chkpnt, and -treefile unique filenames
[/CODE]
And while it was running I wanted it to give a little bit of output, maybe something like:
[CODE]
Using B1=3000000000, B2=85024954914916, polynomial Dickson(30), 4 threads
finished 150/1000; avg s/curve: stg1 29188s, stg2 14827s; runtime: 6602250s
[/CODE]
Where the first line would be static, and that second line would update "however often".

Is this something that others would be interested in? If so, what do you think of the basic description up above? What sort of features would you like to see in something like this?

If you'd like to see some of the previous talk about this, you can see the above mentioned thread here:
[url]http://www.mersenneforum.org/showthread.php?t=12981&page=8[/url] (starting at post #181)

I have a question for anyone knowledgeable in python, what would be the best way to monitor the output of each instance of gmp-ecm without blocking the main thread from being able to update its own output? Should I just redirect the output of each gmp-ecm to a file, or can I attach to their stdout and monitor it that way (without blocking)?

WraithX 2011-04-19 02:58

Ok, I have what appears to be a working version of this driver ready. This release will be version 0.01, but hopefully it will function well for everyone. Like factMsieve.py, this requires Python 2.6 or higher to run. Here are some notes about its use. A typical command line will look like one of the following:

D:\Programming\python\ecm_py>c:\python26\python.exe ecm.py -c 100 -one -maxmem 1500 -threads 2 -out all_out.txt 1000000 < test2.txt
or
D:\Programming\python\ecm_py>echo 189137687020123159261852192605885322671854225839439913905089 | c:\python26\python.exe ecm.py -c 100 -one -maxmem 1500 -threads 2 -out all_out.txt 1000000

The command line has to have /path/to/python.exe before ecm.py. This is needed because of a limitation in Windows (possibly Linux also) where just writing ecm.py will not allow redirection to work properly. This seems to be based on a mishandling of starting python scripts through file associations. However, if the program is called as python.exe ecm.py (if python is in your path), or /path/to/python.exe ecm.py, redirection will work correctly. This includes both piping-in (echo ... | python ecm.py) and file redirection (python ecm.py ... < num.txt).

You can use all the standard gmp-ecm command line options.
If you use -c N (run N curves), then each gmp-ecm instance will get -c (N/num_threads) curves to run, with the first few instances running more if that division is not exact.
If you use -maxmem N (limit ecm.exe memory usage in Stage 2 to < N MB of memory), then each instance of gmp-ecm will use only -maxmem (N/num_threads) to make sure that the total memory used by all instances of gmp-ecm doesn't go over your limit.

When this script is factoring a number, it creates a job file representing how much work you've asked to complete on that number. It also keeps track of how many curves have been run on this number, and timings to help you know how long each stage is taking on average. Each instance of gmp-ecm will write to its own file. The script monitors these files to see how much work has been done, and updates a counter on the screen to let you know how far along the job is.

The script has an auto-resume feature. If the job is interrupted by a power outage or with Ctrl-C, you don't have to worry about losing work. The next time you start up the job with the same command line on the same number, it will read in the old job files to see how much work had been done, and then pick up where it left off. If you turn off auto-resume, the script won't check for previous runs, it will just start a new job.

Here are the options available that are specific to the script:
-threads N This will run N copies of gmp-ecm on the input number(s) you specify.
-r <file> Resume the specified job file. After finishing this job, the script will stop.
-out <file> Save work done (on all input numbers) to the specified file.

These command line options can be intermixed with regular gmp-ecm command line options. Also, please make sure that B1 is the last option in your ecm command line. If not, the program will not behave as expected. In future releases I think I can post a proper error if B1 isn't the last option, but for now, a correct command line should work as expected.

Before running, please set the path to your gmp-ecm executable. You can also set a default number of threads to use if you dont' want to specify that on the command line. Also, you can run multiple copies of this script in the same directory. There shouldn't be any collisions between separate jobs being run.

If anyone is interested in running this, I'd like to hear how well it works for you. If you encounter a problem, I can try to find a solution and post updated versions as I have time. Now, let the fun begin!

WraithX 2011-04-20 12:46

Okay, I have made a couple of changes to ecm.py, which are:

New feature:
- Print more info if a factor is found.
Originally I only printed the 'Factor found' line. Now I include the two lines after that, which includes info like primality/compositeness/length of the factor and cofactor.

Bug fix:
- I have just found that once in a while, my script might print the wrong sigma for a factor found. Fixed that.

Here is the updated version 0.02.

(P.S. could a mod please remove the zip file for 0.01 in post #2? Thank you.)

WraithX 2011-04-21 12:24

Announcing ecm.py version 0.03, the changes are:

[B]New Features[/B]:
- I've added in a -pollfiles N option which will only read data from the job files once every N seconds. This will help the user control the amount of disk I/O based on how long this job will probably run. I currently have the following recommended settings for this option:
# For quick jobs (less than a couple of hours): between 3 and 15 seconds
# For small jobs (less than a day): between 15 and 45 seconds
# For medium jobs (less than a week): between 45 and 120 seconds
# For large jobs (less than a month): between 120 and 360 seconds
(the default value is 15 seconds)

- The program now updates the info line every second, to more easily tell that it is running.

[B]Bug fix[/B]:
- I had made a mistake in [U]displaying[/U] how many curves had been completed. This didn't affect how many curves were actually completed or how long the program would run. Displayed curve counts and times are accurate now.

Mini-Geek 2011-04-30 12:56

I'm getting this error with version 0.03 of your script running on Python 3.1.3:
[CODE]C:\Files\Prime\aliquot>echo 82330147092934135248787768444414746486667369907934935367302358
441167097150481130751779642415163252839011 | python \files\prime\bin\ecmpy.py -c 10 -maxme
m 500 -out out.txt 1000000
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.03 (Python 2.6 or later) 21st Apr 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 823301470929341352487877684444147464866673699079349353673023584411670971504811307517796
42415163252839011
Traceback (most recent call last):
File "\files\prime\bin\ecmpy.py", line 1077, in <module>
parse_ecm_options(sys.argv, set_args = True)
File "\files\prime\bin\ecmpy.py", line 1015, in parse_ecm_options
ecm_args1 += ' -c {0:d}'.format(ecm_c/intNumThreads)
ValueError: Unknown format code 'd' for object of type 'float'[/CODE]
I surrounded the numbers in that line and the three similar lines after it in int() and it made it work.
This is my Python version:
[CODE]Python 3.1.3 (r313:86834, Nov 27 2010, 18:30:53) [MSC v.1500 32 bit (Intel)] on win32[/CODE]
Does anyone know how much work it would be to make this aliqueit-compatible, or make aliqueit ecm-py-compatible? I think that could greatly improve its use. :smile:

WraithX 2011-04-30 14:37

[CODE]
File "\files\prime\bin\ecmpy.py", line 1015, in parse_ecm_options
ecm_args1 += ' -c {0:d}'.format(ecm_c/intNumThreads)
ValueError: Unknown format code 'd' for object of type 'float'
[/CODE]
Interesting, it looks like Python 3.x is returning a float from that integer division, instead of returning an int like in Python 2.6. Thank you for the report. This has been fixed in ecm.py v0.04.

As for making it work with aliqueit, I saw in a different thread where you mention that the configuration looks like the following:

--------------------------
"echo " + input_number + " | " + cfg.ecm_cmd + " -one -c " + tostring( curves )
+ " -B2scale " + tostring( cfg.b2scale_ecm ) + " " + tostring( b1 ) + " > " + cfg.ecm_tempfile ).c_str()
e.g.: "echo 123456789000123333 | ecm -one -c 214 -B2scale 1.0 50000 > aliqueit_ecm_temp.log"
--------------------------

If you can set cfg.ecm_cmd to "python.exe ecm.py" or "/path/to/python ecm.py",
and then change the command line to look like the following:

--------------------------
"echo " + input_number + " | " + cfg.ecm_cmd + " -one -c " + tostring( curves )
+ " -B2scale " + tostring( cfg.b2scale_ecm ) + " -out " + cfg.ecm_tempfile ).c_str() + " " + tostring( b1 )
e.g.: "echo 123456789000123333 | python.exe ecm.py -one -c 214 -B2scale 1.0 -out aliqueit_ecm_temp.log 50000"
--------------------------

I think it might work. Well, it's worth a try, at least. ;)

WraithX 2011-04-30 14:47

1 Attachment(s)
Announcing ecm.py version 0.04, the changes are:

[B]Bug fix[/B]:
- Updated a print statement to be compatible with Python 3.x.

Walter Nissen 2011-05-01 18:56

[QUOTE=WraithX;258976]The command line has to have /path/to/python.exe before ecm.py. This is needed because of a limitation in Windows (possibly Linux also) where just writing ecm.py will not allow redirection to work properly. This seems to be based on a mishandling of starting python scripts through file associations. However, if the program is called as python.exe ecm.py (if python is in your path), or /path/to/python.exe ecm.py, redirection will work correctly.[/QUOTE]

I haven't run ecm.py , but I simply followed the instructions in
[url]http://gilchrist.ca/jeff/factoring/nfs_beginners_guide.html[/url]
for factmsieve.py where I saw nothing about a path to
python.exe , and simply proceeded with :
..\factMsieve.py example

Those instructions lump ggnfs , python , msieve and factmsieve
all together in a single directory which also contains a
working directory .
But when I saw :
..\factMsieve.py example
it didn't seem plausible DOS would find python , but it did .
There's a python27.dll in my path , but no python.exe .

Could the difference between command.com and cmd.exe
account for your experience , and mine ?
I have only the vaguest idea how they differ .
Very early in my Windows XP experience , cmd.exe seemed
better , so I've been using it since , although in a few ways
it's not as good as the command.com in Windows '98 .

mklasson 2011-05-01 20:30

Hi,

Mini-Geek pointed me to this thread and asked for a small fix to aliqueit to make it work with ecm.py. Could someone try it out and see if it seems ok?

Change use_ecmpy to true in aliqueit.ini and make sure ecmpy_cmd is set correctly.

[url]http://mklasson.com/aliqueit110a.zip[/url]

WraithX 2011-05-02 01:31

[QUOTE=Walter Nissen;260143]Those instructions lump ggnfs , python , msieve and factmsieve
all together in a single directory which also contains a
working directory .
But when I saw :
..\factMsieve.py example
it didn't seem plausible DOS would find python , but it did .
There's a python27.dll in my path , but no python.exe .[/QUOTE]

factmsieve.py does not use file redirection or command line piping to get any input. Like you have seen, it is run like 'factmsieve.py example', it then looks for different files that start with the name example (like, example.n, or example.poly, or example.fb, etc) So, you don't have to put python.exe in front of this because 1) there is a file association built into windows (when you install python) (so you can just type factmsieve.py and it will run) and 2) it does not use redirection.

Now, you can just type ecm.py at the command line, but this starts ecm.py through the file association mechanism, and in that case neither file redirection nor command line piping work correctly. This is why you need to have python.exe in front of ecm.py. This problem may exist in a Linux environment also, but I have no way to test that so I'm not sure.

Walter Nissen 2011-05-05 18:22

Thank you for this excellent explanation .

Very curious .
Then again , a lot of things are curious in DOS boxes .
Maybe something about Windows features being cobbled onto DOS .

Cheers ,

Walter

WraithX 2011-05-08 18:21

1 Attachment(s)
Announcing ecm.py version 0.05, the changes are:

[B]New Feature[/B]:
- Now prints the GMP-ECM configuration line in the console window and in the output file. (ie, GMP-ECM 6.3 [configured with GMP 5.0.1] [ECM])

[B]Bug Fix[/B]:
- The -pollfiles command line option wasn't actually being applied, meaning only the default value was used during runtime. This has been fixed.

P.S. Can a Mod remove the zip file (v0.04) from post number 7? Thank you.

WraithX 2011-06-10 01:27

1 Attachment(s)
Announcing ecm.py version 0.06, the changes are:

[B]New Features[/B]:
- Updated output to console and files to be more like GMP-ECM
- Print number of digits for each input number
- Print curve number that found the factor
- Print stage 1 and stage 2 timing for the curve that found a factor

P.S. Can a Mod please remove the zip file v0.04 from post #7 and v0.05 from post #12? Thank you.

WraithX 2011-09-08 01:51

1 Attachment(s)
Announcing ecm.py version 0.07, the changes are:

[B]Minor Bug Fixes[/B]:
- When the input to GMP-ECM was an equation, the number of digits reported by the script was wrong. This has been corrected.
- If saving of output is requested, save the output from previous work that had been completed on this job. It was previously counted and then discarded.
- The previous command line parsing didn't pass the '-one' flag to GMP-ECM. This has been corrected.

[B]New Feature[/B]:
- Added some basic input checking before passing an input line to GMP-ECM. This allows us to skip comment lines in a file (lines starting with #), to skip lines with 'invalid characters', and it helps us to count the number of digits for an input that is an equation.

Mini-Geek 2011-09-22 23:39

[QUOTE=mklasson;260148]Hi,

Mini-Geek pointed me to this thread and asked for a small fix to aliqueit to make it work with ecm.py. Could someone try it out and see if it seems ok?

Change use_ecmpy to true in aliqueit.ini and make sure ecmpy_cmd is set correctly.

[url]http://mklasson.com/aliqueit110a.zip[/url][/QUOTE]

I finally found some time to try this and ecm-py out, and they seem to work great together! They ECMd a c109 on my quad in no time, and I ran a smaller (~70 digits) sequence a little to make sure the factors were detected correctly.

Mini-Geek 2011-09-25 01:47

[QUOTE=WraithX;258976]-r <file> Resume the specified job file. After finishing this job, the script will stop.[/QUOTE]

If you use -r without resending the number to ECM, the script exits with no messages in non-verbose mode, or with the following in verbose mode: (which says it will send "job9096.txt" to GMP-ECM)
[code]C:\Files\Prime\aliquot>ecm.py -r job9096.txt
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.07 (Python 2.6 or later) 7th Sept 2011. |
-> |_________________________________________________________________|

-> Original command line was:
-> job9096.txt
-> New command line(s) will be either:
-> job9096.txt
-> job9096.txt[/code]The "for ecm_n in number_list:" loop is never entered because number_list is empty. If I echo in the number, I get one of the following errors, depending on if I call ecm.py directly, or call python and give it ecm.py (.py files are associated with python.exe, but some weird things with inputs mean it can be a little different):
[code]C:\Files\Prime\aliquot\ecm>echo 3327046125926007069399831196172582927914127361035798047985
95753750302739930552267248499144293948801482976343539281 | ecm.py -r job9096.txt
Traceback (most recent call last):
File "C:\Files\Prime\bin\ecm.py", line 1124, in <module>
parse_ecm_options(sys.argv, set_args = True)
File "C:\Files\Prime\bin\ecm.py", line 1041, in parse_ecm_options
if not sys.stdin.isatty() and set_args and intResume != 1:
AttributeError: 'NoneType' object has no attribute 'isatty'
The process tried to write to a nonexistent pipe.

C:\Files\Prime\aliquot\ecm>echo 3327046125926007069399831196172582927914127361035798047985
95753750302739930552267248499144293948801482976343539281 | python /files/prime/bin/ecm.py
-r job9096.txt
The process tried to write to a nonexistent pipe.[/code]Auto-resuming works fine. Unfortunately this isn't too much help when running through Aliqueit, since it starts from the lowest level of ECM, so all that's saved is the portion of the current level that you've done.

Mini-Geek 2011-09-25 19:40

1 Attachment(s)
I found the non-verbose level of output to be too high for my tastes while running Aliqueit, so I added a quiet level to the VERBOSE user flag. The modified script is attached. The output looks like this: Edit: adjusted the number abbreviation (which is quiet-mode-only) to take up as much space as possible without wrapping to two lines on a standard 80 character line, e.g. "-> Working on number: 267894106254354723342199974696983903161332... (68 digits)" (a character under in this case; assumed 3 digit length instead of bothering to figure it exactly).
[code]c102: running 74 ecm curves at B1=11e3...
-> Working on number: 111...471 (102 digits)

GMP-ECM 6.3 [configured with GMP 5.0.1 and --enable-asm-redc] [ECM]
Using B1=11000, B2=1873422, polynomial x^1, 3 threads
Finished 74/74; avg s/curve: stg1 0.056s, stg2 0.049s; runtime: 4s[/code]
Or when resuming:
[code]c102: running 214 ecm curves at B1=5e4...
-> Working on number: 111...471 (102 digits)
-> Found previous job file job1153.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 214 more curves.

GMP-ECM 6.3 [configured with GMP 5.0.1 and --enable-asm-redc] [ECM]
Using B1=50000, B2=12746592, polynomial x^2, 3 threads
Finished 6/214; avg s/curve: stg1 0.246s, stg2 0.174s; runtime: 1s[/code]
Or when finding a factor:
[code]c71: running 84 ecm curves at B1=5e4...
-> Working on number: 205...329 (71 digits)

GMP-ECM 6.3 [configured with GMP 5.0.1 and --enable-asm-redc] [ECM]
Using B1=50000, B2=12746592, polynomial x^2, 3 threads
Finished 16/84; avg s/curve: stg1 0.173s, stg2 0.124s; runtime: 4s

********** Factor found in step 2: 11273477395685202763693
Found probable prime factor of 23 digits: 11273477395685202763693
Composite cofactor 1821672576763466122668846803768085979236448803253 has 49 digits
[/code]
Instead of this for no factor:
[code]-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.07 (Python 2.6 or later) 7th Sept 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 1110042291838354308811195059079495973007266473263641255266850395399993578389209519659321426138548
51471 (102 digits)
->=============================================================================
-> Working on number: 111004229183835430881119505907949597300726647326364125526685039539999357838920
951965932142613854851471 (102 digits)
-> Currently working on: job7476.txt
-> Starting 3 instances of GMP-ECM...

GMP-ECM 6.3 [configured with GMP 5.0.1 and --enable-asm-redc] [ECM]
Using B1=11000, B2=1873422, polynomial x^1, 3 threads
Finished 74/74; avg s/curve: stg1 0.054s, stg2 0.049s; runtime: 4s

-> *** No factor found.[/code]

WraithX 2011-09-26 02:35

[QUOTE=Mini-Geek;272664]If you use -r without resending the number to ECM, the script exits with no messages in non-verbose mode, or with the following in verbose mode: (which says it will send "job9096.txt" to GMP-ECM)

Auto-resuming works fine. Unfortunately this isn't too much help when running through Aliqueit, since it starts from the lowest level of ECM, so all that's saved is the portion of the current level that you've done.[/QUOTE]

Thank you for letting me know about this. I've never used the -r to resume and so never caught that it was broken. I've been trying to fix it, but I think it may require a big rewrite to get it correct.

I like what you have done with the changes to verbose mode. I will include your changes in my future releases.

I have a question. Did you try "ecm.py -r jobNNNN.txt" manually, or does aliquiet try that? I guess in either case, since it is broken it won't work, I was just curious.

And speaking of aliquiet resuming, is there a way for aliquiet to check for a job file for a given number, and then re-issue the curves for that number at the ecm level specified in the job file? Basically, instead of aliquiet starting ecm from scratch, it could trigger the auto-resume feature of ecm.py (which we know works). However, this will probably require a change to aliquiet.

I will keep working on fixing the -r option of ecm.py and will hopefully have a fix sometime this week.

Mini-Geek 2011-09-26 11:56

[QUOTE=WraithX;272736]Thank you for letting me know about this. I've never used the -r to resume and so never caught that it was broken. I've been trying to fix it, but I think it may require a big rewrite to get it correct.
...
I have a question. Did you try "ecm.py -r jobNNNN.txt" manually, or does aliquiet try that? I guess in either case, since it is broken it won't work, I was just curious.
...
I will keep working on fixing the -r option of ecm.py and will hopefully have a fix sometime this week.[/QUOTE]
I did it manually. Aliqueit doesn't use the -r command, since it doesn't know in advance that there's a job file to resume for the number; it just calls ecm.py with the same parameters as last time, and so ecm.py finds the file and autoresumes it.
Considering I don't do a whole lot of resuming, it's not too hard to copy/paste the old parameters saved in the job file to the command line; don't bother to fix it if you feel it's for only my sake.
[QUOTE=WraithX;272736]And speaking of aliquiet resuming, is there a way for aliquiet to check for a job file for a given number, and then re-issue the curves for that number at the ecm level specified in the job file? Basically, instead of aliquiet starting ecm from scratch, it could trigger the auto-resume feature of ecm.py (which we know works). However, this will probably require a change to aliquiet.[/QUOTE]
I'm sure it would require a change to aliqueit, but it strikes me as being entirely possible (how much work? I don't know). Its impact to performance would be tiny if it was only checked for on the first iteration if small factors were already known, instead of checking every number every time. (probably a trivial performance hit anyway, especially if it looked for them once and cached the results)
[QUOTE=WraithX;272736]I like what you have done with the changes to verbose mode. I will include your changes in my future releases.[/QUOTE]
Thank you, glad to hear that. :smile:

WraithX 2011-10-20 02:16

1 Attachment(s)
Announcing ecm.py version 0.08, the changes are:

[B]Bug Fix[/B]:
Fixed resuming a specified file when using the -r option. Thank you to Mini-Geek for finding and reporting this.

[B]New Feature[/B]:
New VERBOSE levels to give better control over how much information the script will output while running. You can set a default level in the script at the following line:
[CODE]VERBOSE = 1 #set to 0, 1, or 2 for quiet, normal, or verbose[/CODE]
Thank you to Mini-Geek for writing and sharing this.

Dubslow 2012-09-16 19:08

:huh:
[code]∰∂ taskset f nice -n 19 ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1224, in <module>
start_ecm_threads()
File "./ecm.py", line 521, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 325, in run_exe
p = subprocess.Popen(args.split(' '), **al)
File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied[/code]
The binary is executable, i.e. `./ecm` works fine, and it's the same binary YAFU uses. This is Ubuntu 11.04, with "#! /usr/bin/python" which is version 2.7.1. (To be sure, I did chmod +x ./ecm, but that didn't do anything.)

Dubslow 2012-09-16 20:52

[QUOTE=Dubslow;311872]:huh:

The binary is executable, i.e. `./ecm` works fine, and it's the same binary YAFU uses. This is Ubuntu 11.04, with "#! /usr/bin/python" which is version 2.7.1. (To be sure, I did chmod +x ./ecm, but that didn't do anything.)[/QUOTE]

For kicks I tried "#! /usr/bin/python3" (v 3.2).
[code]-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1224, in <module>
start_ecm_threads()
File "./ecm.py", line 521, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 325, in run_exe
p = subprocess.Popen(args.split(' '), **al)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1331, in _execute_child
raise child_exception_type(err_msg)
RuntimeError: Exception occurred in preexec_fn.[/code]

WraithX 2012-09-17 04:00

[QUOTE=Dubslow;311872]
[code]
∰∂ taskset f nice -n 19 ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
...
OSError: [Errno 13] Permission denied
[/code]
[/QUOTE]

Hmmm, I haven't seen this before, but looking at the error from your python 3.2 run:
[code]RuntimeError: Exception occurred in preexec_fn.[/code]
This might be related to your use of taskset and nice. Can you find nice on your system and set the variable:
[code] NICE_PATH = ''[/code]
in your ecm.py script? (should be around line 95 in ecm.py)

If that doesn't work I have some more things to try, but let's see where this gets us first.

Dubslow 2012-09-17 04:07

[QUOTE=WraithX;311925]Hmmm, I haven't seen this before, but looking at the error from your python 3.2 run:
[code]RuntimeError: Exception occurred in preexec_fn.[/code]
This might be related to your use of taskset and nice. Can you find nice on your system and set the variable:
[code] NICE_PATH = ''[/code]
in your ecm.py script? (should be around line 95 in ecm.py)

If that doesn't work I have some more things to try, but let's see where this gets us first.[/QUOTE]

It's already like that (as is EXE_SUFFIX or some such variable).
[code]ECM = 'ecm'

if sys.platform.startswith('win'):
EXE_SUFFIX = '.exe'
else:
EXE_SUFFIX = ''
NICE_PATH = ''
ECM = './' + ECM[/code]

WraithX 2012-09-17 12:15

[QUOTE=Dubslow;311926]It's already like that (as is EXE_SUFFIX or some such variable).[/QUOTE]

[QUOTE=WraithX]Can you find nice on your system and set the variable:[/QUOTE]

Once again, with feeling. Can you find nice (the executable) on your system (maybe in /usr/bin/nice, etc) and then set the value of NICE_PATH accordingly? For the above example (/usr/bin/nice), try both /usr/bin/nice and just /usr/bin/ to see if either works.

Come to think of it...
Since this code mostly comes from factmsieve.py, I had never used or thought much about the NICE_PATH option. But thinking about it, I believe that you would set the NICE_PATH option to '/usr/bin/nice -19', instead of running with taskset/nice etc. Can you also give this a try?

Dubslow 2012-09-17 18:40

[QUOTE=WraithX;311960]Once again, with feeling. Can you find nice (the executable) on your system (maybe in /usr/bin/nice, etc) and then set the value of NICE_PATH accordingly? For the above example (/usr/bin/nice), try both /usr/bin/nice and just /usr/bin/ to see if either works.

Come to think of it...
Since this code mostly comes from factmsieve.py, I had never used or thought much about the NICE_PATH option. But thinking about it, I believe that you would set the NICE_PATH option to '/usr/bin/nice -19', instead of running with taskset/nice etc. Can you also give this a try?[/QUOTE]

Whoops :blush:

No dice though.
[code]ECM = 'ecm'

if sys.platform.startswith('win'):
EXE_SUFFIX = '.exe'
else:
EXE_SUFFIX = ''
NICE_PATH = '/usr/bin/nice -n 19' # No difference run this way or with '/usr/bin/nice 19' or 'nice -19'
ECM = './' + ECM[/code]
[code]bill@Gravemind:~/yafu/ecm∰∂ which nice
/usr/bin/nice
[1]+ Done gedit ecm.py
bill@Gravemind:~/yafu/ecm∰∂ less ~/.bash_history
bill@Gravemind:~/yafu/ecm∰∂ taskset f ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1224, in <module>
start_ecm_threads()
File "./ecm.py", line 521, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 325, in run_exe
p = subprocess.Popen(args.split(' '), **al)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1331, in _execute_child
raise child_exception_type(err_msg)
RuntimeError: Exception occurred in preexec_fn.[/code]

chris2be8 2012-09-18 16:20

What happens if you run it without the "taskset f" at the start of the command line?

Chris

Dubslow 2012-09-20 05:24

[QUOTE=chris2be8;312090]What happens if you run it without the "taskset f" at the start of the command line?

Chris[/QUOTE]

[code]bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1224, in <module>
start_ecm_threads()
File "./ecm.py", line 521, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 325, in run_exe
p = subprocess.Popen(args.split(' '), **al)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1331, in _execute_child
raise child_exception_type(err_msg)
RuntimeError: Exception occurred in preexec_fn.[/code]
Doesn't matter what the value of NICE_PATH is.

WraithX 2012-09-21 02:37

[QUOTE=Dubslow;312213][code]bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num[/code]
Doesn't matter what the value of NICE_PATH is.[/QUOTE]

I wonder if the following will make a difference. Try to put the python executable in front of the ecm.py script, like so:
[CODE]/path/to/python ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num[/CODE]
And, let's also try that with different values in NICE_PATH.

Dubslow 2012-09-21 03:11

[QUOTE=WraithX;312267]I wonder if the following will make a difference. Try to put the python executable in front of the ecm.py script, like so:
[CODE]/path/to/python ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num[/CODE]
And, let's also try that with different values in NICE_PATH.[/QUOTE]

[code]bill@Gravemind:~/yafu/ecm∰∂ /usr/bin/python3 ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1224, in <module>
start_ecm_threads()
File "./ecm.py", line 521, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 325, in run_exe
p = subprocess.Popen(args.split(' '), **al)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1331, in _execute_child
raise child_exception_type(err_msg)
RuntimeError: Exception occurred in preexec_fn.[/code]
I tried it with NICE_PATH = '', '/usr/bin/nice', and '/usr/bin/nice -n 19'.

WraithX 2012-09-22 20:37

[QUOTE=Dubslow;312269]I tried it with NICE_PATH = '', '/usr/bin/nice', and '/usr/bin/nice -n 19'.[/QUOTE]

Man, I'm sorry this is taking a while to figure out. Ok, lets try the following:
With the following command line:
ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
Set NICE_PATH = 'lambda:os.nice(20)'

If that one does not work, then:
With the following command line:
ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
[CODE]
Set NICE_PATH = ''
Change the following line in ecm.py from this:
p = subprocess.Popen(args.split(' '), **al)
To this:
p = subprocess.Popen(args.split(' '), preexec_fn=lambda:os.nice(20), **al)
[/CODE]
And if neither of those work, can you also try with /usr/bin/python3 in front of each command line?

Dubslow 2012-09-22 20:54

[QUOTE=WraithX;312459]Man, I'm sorry this is taking a while to figure out. Ok, lets try the following:
With the following command line:
ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
Set NICE_PATH = 'lambda:os.nice(20)'

If that one does not work, then:
With the following command line:
ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
[CODE]
Set NICE_PATH = ''
Change the following line in ecm.py from this:
p = subprocess.Popen(args.split(' '), **al)
To this:
p = subprocess.Popen(args.split(' '), preexec_fn=lambda:os.nice(20), **al)
[/CODE]
And if neither of those work, can you also try with /usr/bin/python3 in front of each command line?[/QUOTE]
I have a few comments:

First, I'm pretty sure /usr/bin/python3 won't make a difference. In Linux, the file type is NOT associated with the python executable; rather the shell (bash, in my and most cases) looks at the first line for the so-called "shebang" (I prefer "hashbang") which looks like `#! path-to-exe`; the shell then calls `path-to-exe $0 $@`, which is exactly the same as if I did it myself. In my case:
[code]bill@Gravemind:~/yafu/ecm∰∂ head -n 1 ecm.py
#! /usr/bin/python3[/code]
So I won't try it with the /usr/bin/python3 in front.

As for the nice, when I tried it with NICE_PATH='' and without a nice command of my own, it still didn't work, even though it should have just started with normal priority; doesn't that mean that the problem isn't with nice-ing it? I'm less sure about this, so I'll try it with your suggestions. (I had no idea os has a nice(), though of course it makes sense :razz:) (PS Wouldn't the **al overwrite the keyword arg? PPS I was sort of right, see second attempt error)

[code]~/yafu/ecm∰∂ cat ecm.py | grep NICE_PATH
NICE_PATH = 'lambda: os.nice(20)' #'/usr/bin/nice -19'
al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1224, in <module>
start_ecm_threads()
File "./ecm.py", line 521, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 325, in run_exe
p = subprocess.Popen(args.split(' '), **al)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1331, in _execute_child
raise child_exception_type(err_msg)
RuntimeError: Exception occurred in preexec_fn.[/code]
[code]bill@Gravemind:~/yafu/ecm∰∂ cat ecm.py | grep "NICE_PATH"
NICE_PATH = '' #'/usr/bin/nice -19'
al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ man grep
bill@Gravemind:~/yafu/ecm∰∂ cat ecm.py | grep "NICE_PATH"
NICE_PATH = '' #'/usr/bin/nice -19'
al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ cat ecm.py | grep "Popen"
p = subprocess.Popen(args.split(' '), preexec_fn=lambda: os.nice(20), **al)
#p = subprocess.Popen(args.split(' '), **al)
#p = subprocess.Popen([ex] + args.split(' '), **al)
#p = subprocess.Popen(cs.split(' '), **al)
bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1225, in <module>
start_ecm_threads()
File "./ecm.py", line 522, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 325, in run_exe
p = subprocess.Popen(args.split(' '), preexec_fn=lambda: os.nice(20), **al)
TypeError: type object got multiple values for keyword argument 'preexec_fn'[/code]
[code]bill@Gravemind:~/yafu/ecm∰∂ cat ecm.py | grep "NICE_PATH"
NICE_PATH = '' #'/usr/bin/nice -19'
pass #al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ cat ecm.py | grep "Popen"
p = subprocess.Popen(args.split(' '), preexec_fn=lambda: os.nice(20), **al)
#p = subprocess.Popen(args.split(' '), **al)
#p = subprocess.Popen([ex] + args.split(' '), **al)
#p = subprocess.Popen(cs.split(' '), **al)
bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1225, in <module>
start_ecm_threads()
File "./ecm.py", line 522, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 325, in run_exe
p = subprocess.Popen(args.split(' '), preexec_fn=lambda: os.nice(20), **al)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1330, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 13] Permission denied[/code]
The last error is the same as with python2.7 .

Dubslow 2012-09-22 21:10

Char limit reached.
[code]bill@Gravemind:~/yafu/ecm∰∂ cat ecm.py | grep "NICE_PATH"
NICE_PATH = '' #'/usr/bin/nice -19'
pass #al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ cat ecm.py | grep "Popen"
#p = subprocess.Popen(args.split(' '), preexec_fn=lambda: os.nice(20), **al)
p = subprocess.Popen(args.split(' '), **al) # Don't bother with preexec
#p = subprocess.Popen([ex] + args.split(' '), **al)
#p = subprocess.Popen(cs.split(' '), **al)
bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
Traceback (most recent call last):
File "./ecm.py", line 1225, in <module>
start_ecm_threads()
File "./ecm.py", line 522, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 326, in run_exe
p = subprocess.Popen(args.split(' '), **al) # Don't bother with preexec
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1330, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 13] Permission denied[/code]

WraithX 2012-09-22 21:36

[QUOTE=Dubslow;312461]In Linux, the file type is NOT associated with the python executable; rather the shell (bash, in my and most cases) looks at the first line for the so-called "shebang" (I prefer "hashbang") which looks like `#! path-to-exe`; the shell then calls `path-to-exe $0 $@`, which is exactly the same as if I did it myself[/QUOTE]

Ah, this is good to know. Thanks for that info.

Well, the next thing to try is to:
Set NICE_PATH to different values, I'm not sure which one to put here
and change the Popen line to:
p = subprocess.Popen(args.split(' '), shell=True, **al)

And also, can you add the following two lines right above the Popen line:
[CODE]
print('-> args: {0:s}', str(args))
print('-> al: {0:s}', str(al))
[/CODE]
I want to make sure nothing fishy is going on there.

Dubslow 2012-09-22 21:52

[QUOTE=WraithX;312463]Ah, this is good to know. Thanks for that info.[/quote]
[url]http://en.wikipedia.org/wiki/Shebang_(Unix[/url])
[url]http://docs.python.org/tutorial/interpreter.html#executable-python-scripts[/url]
[QUOTE=WraithX;312463]
Well, the next thing to try is to:
Set NICE_PATH to different values, I'm not sure which one to put here
and change the Popen line to:
p = subprocess.Popen(args.split(' '), shell=True, **al)

And also, can you add the following two lines right above the Popen line:
[CODE]
print('-> args: {0:s}', str(args))
print('-> al: {0:s}', str(al))
[/CODE]
I want to make sure nothing fishy is going on there.[/QUOTE]
There's the problem. Should be ./ecm, not ../ecm.
[code]bill@Gravemind:~/yafu/ecm∰∂ grep "NICE_PATH" ecm.py
NICE_PATH = '' #'/usr/bin/nice -19'
pass #al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ grep -B 2 "Popen" ecm.py
print('-> args: {0:s}', str(args))
print('-> al: {0:s}', str(al))
p = subprocess.Popen(args.split(' '), shell=True, **al)
#p = subprocess.Popen(args.split(' '), **al)
#p = subprocess.Popen([ex] + args.split(' '), **al)
#p = subprocess.Popen(cs.split(' '), **al)
bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2011, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.08 (Python 2.6 or later) 19th Oct 2011. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
-> args: {0:s} -one -c 1889 43000000
-> al: {0:s} {'executable': '../ecm', 'stdin': <_io.TextIOWrapper name='job3362.txt' mode='r' encoding='UTF-8'>, 'stdout': <_io.TextIOWrapper name='job3362_t00.txt' mode='w' encoding='UTF-8'>}
Traceback (most recent call last):
File "./ecm.py", line 1227, in <module>
start_ecm_threads()
File "./ecm.py", line 524, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 327, in run_exe
p = subprocess.Popen(args.split(' '), shell=True, **al)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1330, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 13] Permission denied[/code]
[code]ECM_PATH = '.'
#...
ECM = 'ecm'

if sys.platform.startswith('win'):
EXE_SUFFIX = '.exe'
else:
EXE_SUFFIX = ''
NICE_PATH = ''
ECM = './' + ECM
#...
ex = (ECM_PATH + exe + EXE_SUFFIX)
al['executable'] = ex[/code]
I don't have a clean copy of the script, so I don't know if this is my fault or not (though it probably is).

WraithX 2012-09-22 22:17

[QUOTE=Dubslow;312467]There's the problem. Should be ./ecm, not ../ecm.

I don't have a clean copy of the script, so I don't know if this is my fault or not (though it probably is).[/QUOTE]

Well, you can always download a clean copy from earlier in the thread. :wink:

For the path argument, if ecm is in the same folder, I have always used "./" and not ".". This should be all you need to change to get the ecm executable working.

As for nice, I would suggest the following:
NICE_PATH = '/usr/bin/nice -n 19'

WraithX 2012-09-30 17:41

1 Attachment(s)
Announcing ecm.py version 0.10, the changes are:
[QUOTE]
- added an extra check to help make sure ecm_path is set correctly
- added an extra verbose level (3) that will print some debug info to try to help diagnose any problems
- changed how the number of digits in an input number is calculated
-- * it is now based on math.log10() instead of counting characters in a string
-- * this is a big speed improvement for numbers with {1's,10's,100's,...} of thousands of digits
- changed time averaging so it no longer averages in the very rare negative time reported by gmp-ecm
[/QUOTE]

jwaltos 2012-10-04 03:38

ecm.py attachment
 
Thanks WraithX.

EdH 2013-05-05 03:17

Is it possible to convert ecm.py such that it will operate across other machines?

I'm toying with several ideas along this line, but unsure whether to look into the workings of the script myself or not. I'm not well versed in Python and could possibly use this as a way to learn some.

Anyway, as it runs, everything is done within the original terminal. Could the separate threads be run in separate terminals, such as called by gnome-terminal? If so, could these threads be run as ssh connections on remote machines? Or, if run as totally independent processes on the alternate machines, would there be a way of catching a factor, if found on one of the other machines?

Would the data transfer overhead or the variance in thread processing negate any possible gains of creating several threads across differing equipment types?

Additionally, if the above is at least a possibility, would there be a way to scale the work allocated such that faster machines get a bigger allocation?

Thanks for any ideas (and especially for a great script)...

schickel 2013-05-05 05:23

[QUOTE=EdH;339294]Is it possible to convert ecm.py such that it will operate across other machines?

I'm toying with several ideas along this line, but unsure whether to look into the workings of the script myself or not. I'm not well versed in Python and could possibly use this as a way to learn some.

Anyway, as it runs, everything is done within the original terminal. Could the separate threads be run in separate terminals, such as called by gnome-terminal? If so, could these threads be run as ssh connections on remote machines? Or, if run as totally independent processes on the alternate machines, would there be a way of catching a factor, if found on one of the other machines?

Would the data transfer overhead or the variance in thread processing negate any possible gains of creating several threads across differing equipment types?

Additionally, if the above is at least a possibility, would there be a way to scale the work allocated such that faster machines get a bigger allocation?

Thanks for any ideas (and especially for a great script)...[/QUOTE]I would say that if you're going to look into running more than one machine, ECMNET would be the better way to go.

I have my network setup with a master server here on my machine, with two dedicated machines that boot up with a slave server/client setup. That way, even if I have trouble with the network or this machine, the ECM boxes keep on running. The network overhead is miniscule, and you can just load up a bunch of low priority work along with your higher priority work, that way if something factors, there's some fall back work.

If you go with v2.0, the backend for the server is a simple text file, no DB engine required. And the clients keep the workload balanced, since each can run independently, no manual intervention required....

debrouxl 2013-05-05 09:37

Agreed, ECMNet is pretty good, even on a single host :smile:
I'm sticking to 2.x due to the lack of dependency on a SQL engine.

xilman 2013-05-05 12:38

[QUOTE=debrouxl;339312]Agreed, ECMNet is pretty good, even on a single host :smile:
I'm sticking to 2.x due to the lack of dependency on a SQL engine.[/QUOTE]Seconded. It just works --- the V2.x anyway. We never managed to get the DB interface working on my Linux box. One day, perhaps, Mark and I will return to the issue but we're each busy with other things ATM.

If you would like assistance setting up ECMNET clients and/or master server and/or slave server just ask. My fee is very reasonable: a few curves contributed to 83.217.167.177:8194 :wink:


Paul

PhilF 2013-05-05 13:02

[QUOTE=xilman;339316] My fee is very reasonable: a few curves contributed to 83.217.167.177:8194 :wink:[/QUOTE]
Now *that* is a reasonable fee! :coffee:

EdH 2013-05-05 13:20

Thanks!

I will definitely look into this. Currently, I have scripts across several machines that "wait" for sieving assignments from one of them. The assignments are initiated by a script running Aliqueit on a separate machine. However, there is just too much wasted time in between sieving assignments. I had considered adjusting the waiting machines to do some ECM work during the off times. And, then I thought of expanding the ecm.py script that the Aliqueit machine runs. Perhaps ECMNET is what I'm looking for, since you mention being able to add in other jobs between assignments, as well.

I was running curves on the c173 from 4788 continuously on all the machines until my attention span waned after a couple months. (The paint just wouldn't dry...) Then, I wrote all the sieving scripts and am running those, but looking for something else...

Unfortunately, I have a lot of non-math/computer stuff building up to keep me busy, so the more automated, the better.

schickel 2013-05-05 14:29

[QUOTE=EdH;339319]Thanks!

I will definitely look into this. Currently, I have scripts across several machines that "wait" for sieving assignments from one of them. The assignments are initiated by a script running Aliqueit on a separate machine. However, there is just too much wasted time in between sieving assignments. I had considered adjusting the waiting machines to do some ECM work during the off times. And, then I thought of expanding the ecm.py script that the Aliqueit machine runs. Perhaps ECMNET is what I'm looking for, since you mention being able to add in other jobs between assignments, as well.

I was running curves on the c173 from 4788 continuously on all the machines until my attention span waned after a couple months. (The paint just wouldn't dry...) Then, I wrote all the sieving scripts and am running those, but looking for something else...

Unfortunately, I have a lot of non-math/computer stuff building up to keep me busy, so the more automated, the better.[/QUOTE]Hmm; you'd have to test the speed loss, but ECMNET is configured to run the clients at idle priority, so if your sievers run at normal, they would take precedence over the ECM thingies. They still might not run quite at top speed, though.

EdH 2013-05-05 19:36

[QUOTE=schickel;339329]Hmm; you'd have to test the speed loss, but ECMNET is configured to run the clients at idle priority, so if your sievers run at normal, they would take precedence over the ECM thingies. They still might not run quite at top speed, though.[/QUOTE]
These machines have no other work, so maybe I can set them up with ECMNET running 24/7, with assignments coming from the same machine that handles the sieving assignments. The assignments will come based on a separate machine that is running Aliqueit or similar. While Aliqueit is running ecm.py (or something to "see" ECMNET results), the machines will get ECM assignments, when it switches to gnfs, sieving assignments will take over the machines which should put the ECMNET clients to sleep.

I'll have to work out the details of signalling what to run, communication with the Aliqueit machine and how to cancel ECM when switching, etc., but it should all be doable.

Now if I can just find some time and keep my attention directed...

debrouxl 2013-05-06 04:42

The ECMNet clients can run threads at any priority, at least under Linux. I often use nice 19, but I've occasionally used nice 0. If ECMNet ran as root, it might even run processes at negative nice values, but then it would hardly make a difference wrt. nice 0 anyway.

WraithX 2014-12-12 03:47

1 Attachment(s)
Announcing ecm.py version 0.30, the changes are:
[CODE]
-New feature that shows ETA (estimated time to job completion)
* Since it is based on the average runtime of the job so far, the estimate can be a little early or late. But, in general gives a good idea of how long the current job will take.

-New format for the running output. And you can select between different displays of Runtime and ETA like so:
# Examples of how you can mix and match Runtime and ETA outputs:
#____________________________________________________________________________
# Curves Complete | Average seconds/curve | Runtime | ETA
#-----------------|---------------------------|---------------|--------------
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 1945628s | 41d 08:37:51
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 22.518d | 41d 08:37:51
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 22d 12:27:08 | 3573471s
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 22d 12:27:08 | 41.359d
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 22d 12:27:08 | 41d 08:37:51

-New email feature.
* This option will send out an email when the job completes, either when it finds a factor or when all curves have finished with no factor found. The email will let you know which event happened.
* This option can also email out progress reports at regular intervals, so you can have a record of how far along remote machines are in their job.

-New log file feature.
* This option keeps a record of the work that ecm.py has done. Each line in the log file gets a time stamp so you can know when jobs were started and when they finished.
* This option can also record the above "runtime information" at regular intervals.
[/CODE]

wombatman 2014-12-12 04:20

Awesome stuff!

wombatman 2014-12-30 22:18

I'm running into an error using the newest ecm.py. I get an index out of range error for this line:

[CODE]using_line = '{0:s},{1:s},{2:s}, {3:d} thread{4:s}'.format(ud[0],ud[1],ud[2],actual_num_threads, '' if (actual_num_threads == 1) else 's')[/CODE]

I made sure there were no job files, and it always fails at this line, regardless of the number of threads. Interestingly, if I start ecm.py with no "number of curves" to run (that is, no -c parameter), it starts up fine. If I run with "-c 1", though, it crashes at the line above. Not sure if that helps, but I figured I would mention it.

WraithX 2014-12-31 03:53

[QUOTE=wombatman;391276]I'm running into an error using the newest ecm.py. I get an index out of range error for this line:

[CODE]using_line = '{0:s},{1:s},{2:s}, {3:d} thread{4:s}'.format(ud[0],ud[1],ud[2],actual_num_threads, '' if (actual_num_threads == 1) else 's')[/CODE][/QUOTE]

Hmmm, can you post what a full run looks like for your gmp-ecm binary? Try it with a simple command line like:
ecm.exe -c 2 50e3 < num.txt

For me, that looks like:
[CODE]
GMP-ECM 7.0-dev [configured with GMP 6.0.0, --enable-asm-redc] [ECM]
Input number is 6962815472410550045247870144481111677555048961510968048443680963
65759227064152650920222099941253589429607814503738760057809654611250765236890819
4481257411644162747345175089477132247 (181 digits)
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:3093725542
Step 1 took 343ms
Step 2 took 235ms
Run 2 out of 2:
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:201126966
Step 1 took 312ms
Step 2 took 250ms
[/CODE]

wombatman 2014-12-31 06:01

Running with the same number you used and the following command: [CODE] echo "number" | ecm.exe -c 2 50e3[/CODE]

I get: [CODE]GMP-ECM 7.0-dev [configured with GMP 6.0.0] [ECM]
Input number is 69628154724105500452478701444811116775550489615109680484436809636575922
706415265092022209994125358942960781450373876005780965461125076523689081944812574116441
62747345175089477132247 (181 digits)
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:2321917342
Step 1 took 234ms
Step 2 took 187ms
Run 2 out of 2:
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:1182338425
Step 1 took 234ms
Step 2 took 172ms[/CODE]

WraithX 2014-12-31 12:45

Ok, that output looks like what is normally expected. Can you send me some extra information about your environment?
What is your operating system?
What version of python are you using?
What is the exact command line you are using?
Can you edit ecm.py and change VERBOSE to 3 at line 136 (in the User Defined Variables), run the program again, let it crash, and then post the full output (from both ecm.py and the python error afterwards)?

That should help me start diagnosing this problem. Thanks!

wombatman 2014-12-31 16:53

You bet!

OS is Windows 7 Pro 64-bit, but I'm running the script inside of MinGW-64. Previous version worked perfectly like this.

Python version is 3.3.3.

Exact command line is: [CODE]py ecm.py -v -c 2 110e6 < input.txt[/CODE]

Result, with the verbose change you requested, is: [CODE]$ py ecm.py -v -c 2 110e6 < c178_hp2.ini 48111167755504896
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is Copyright, 2012, David Cleaver and |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman. Version 0.30 (Python 2.6 or later) 30th Nov 2014. |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 267584724287272660632588026709919587698209050387915658332777631397485002205943824599
399098348970270487342146242128808502451298903034661323335707683105729279570983958104088
3292553 (178 digits)
-> Original command line was:
-> -v -c 2 110e6
-> New command line(s) will be either:
-> -v 110e6
-> -v 110e6

->=============================================================================
-> Working on number: 26758472428727266063258802670991958769820905038791565833277763139
748500220594382459939909834897027048734214624212880850245129890303466132333570768310572
92795709839581040883292553 (178 digits)
-> Currently working on: job4858.txt
-> Original command line was:
-> -v -c 2 110e6
-> New command line(s) will be either:
-> -c 1 110e6
-> -c 2 110e6

-> Starting 2 instances of GMP-ECM...
-> ecm -c 1 110e6 < job4858.txt > job4858_t00.txt
-> ecm args: -c 1 110e6
-> ecm al: {'stdin': <_io.TextIOWrapper name='job4858.txt' mode='r' encoding='cp1252'
>, 'executable': './ecm.exe', 'creationflags': 64, 'stdout': <_io.TextIOWrapper name='j
ob4858_t00.txt' mode='w' encoding='cp1252'>}
-> ecm -c 1 110e6 < job4858.txt > job4858_t01.txt
-> ecm args: -c 1 110e6
-> ecm al: {'stdin': <_io.TextIOWrapper name='job4858.txt' mode='r' encoding='cp1252'
>, 'executable': './ecm.exe', 'creationflags': 64, 'stdout': <_io.TextIOWrapper name='j
ob4858_t01.txt' mode='w' encoding='cp1252'>}

GMP-ECM 7.0-dev [configured with GMP 6.0.0] [ECM]
Using B1=110000000, B2=776278396540, polynomial Dickson(30), 2 threads[/CODE]



....and it appears to work fine like that. And even with "-c 100" it works fine. And passing in "-threads 4" works too.

So basically, with VERBOSE=3, it works fine.

Edit: And changing VERBOSE to 1, it still appears to work fine. I'm so confused now.

WraithX 2015-01-01 15:32

ecm.py v0.31
 
1 Attachment(s)
[QUOTE=wombatman;391329]....and it appears to work fine like that. And even with "-c 100" it works fine. And passing in "-threads 4" works too.

So basically, with VERBOSE=3, it works fine.

Edit: And changing VERBOSE to 1, it still appears to work fine. I'm so confused now.[/QUOTE]

My best guess is that there was a race condition where ecm.exe was writing to an output file and ecm.py was reading in that same file. In this case ecm.py happened to read the file where the line:
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:1182338425
Was incomplete. Perhaps only
Using B1=50000, B2=
had been written to the file at the time of reading. Since my code was expecting a whole line there (with two or more commas) it crashed on you. And, if you tried to resume from that, it probably would have crashed in the same place.

I've updated the code to try to avoid this happening in the future. It's hard to test since the problem should only happen very rarely. But, if successful, you won't see this problem ever again! :smile:

Thanks for finding this, and let me know how the new version works for you!

wombatman 2015-01-01 16:31

Thanks! I'll try it out and let you know!

Antonio 2015-02-01 17:11

[QUOTE=WraithX;389816]Announcing ecm.py version 0.30, the changes are:
[CODE]
-New feature that shows ETA (estimated time to job completion)
* Since it is based on the average runtime of the job so far, the estimate can be a little early or late. But, in general gives a good idea of how long the current job will take.

-New format for the running output. And you can select between different displays of Runtime and ETA like so:
# Examples of how you can mix and match Runtime and ETA outputs:
#____________________________________________________________________________
# Curves Complete | Average seconds/curve | Runtime | ETA
#-----------------|---------------------------|---------------|--------------
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 1945628s | 41d 08:37:51
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 22.518d | 41d 08:37:51
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 22d 12:27:08 | 3573471s
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 22d 12:27:08 | 41.359d
# 2114 of 6000 | Stg1 2983s | Stg2 693.5s | 22d 12:27:08 | 41d 08:37:51

-New email feature.
* This option will send out an email when the job completes, either when it finds a factor or when all curves have finished with no factor found. The email will let you know which event happened.
* This option can also email out progress reports at regular intervals, so you can have a record of how far along remote machines are in their job.

-New log file feature.
* This option keeps a record of the work that ecm.py has done. Each line in the log file gets a time stamp so you can know when jobs were started and when they finished.
* This option can also record the above "runtime information" at regular intervals.
[/CODE][/QUOTE]

Great work thanks, works like a charm. Couple of suggestions: -

1. Can it be made to accept the ecm -inp command line option rather than redirected stdin?

2. Could it automatically rebalance the thread load if {(curves completed by fastest thread) - (curves completed by slowest thread)} > (number of threads),
currently I stop and restart to achieve this but it would be nice if I could leave it completely unattended.

xilman 2015-02-02 14:23

[QUOTE=Antonio;394179]
1. Can it be made to accept the ecm -inp command line option rather than redirected stdin?
[/QUOTE]Just curious, but why?

Either "ecm < foo" or "cat foo | ecm" do exactly the same thing (though 'cat' is spelled 'type' on Windoze systems).

I ask because Paul Z. (i.e. the other Paul) has recently suggested doing away with -inp and several other options in order to simplify the code base and, as the examples show above, it is trivially implemented anyway.

Antonio 2015-02-02 19:03

[QUOTE=xilman;394232]Just curious, but why?

Either "ecm < foo" or "cat foo | ecm" do exactly the same thing (though 'cat' is spelled 'type' on Windoze systems).

I ask because Paul Z. (i.e. the other Paul) has recently suggested doing away with -inp and several other options in order to simplify the code base and, as the examples show above, it is trivially implemented anyway.[/QUOTE]

No real reason, other than personal preference I suppose.

WraithX 2015-02-14 15:22

ecm.py v0.32
 
1 Attachment(s)
[QUOTE=Antonio;394179]Great work thanks, works like a charm. Couple of suggestions: -

1. Can it be made to accept the ecm -inp command line option rather than redirected stdin?

2. Could it automatically rebalance the thread load if {(curves completed by fastest thread) - (curves completed by slowest thread)} > (number of threads),
currently I stop and restart to achieve this but it would be nice if I could leave it completely unattended.[/QUOTE]

Announcing ecm.py version 0.32. The only change was to add the -inp feature request from above. Starting the program can look like either of the following now:
[CODE]
python.exe ecm.py [gmp-ecm options] [ecm.py options] B1 < <in_file>
ecm.py -inp <in_file> [gmp-ecm options] [ecm.py options] B1
[/CODE]

Implementing number 1 was simple enough.

Implementing number 2 would be really difficult, and potentially not beneficial. Currently I start different ecm.exe processes each with num_curves/num_threads curves of work to do. To "rebalance the load", I'd have to kill all running instances of ecm.exe and then start them up again. Each time you rebalance, you will probably lose between (num_threads/2) to (num_threads) number of curves of work (due to killing ecm.exe when it is partially through its current curve). And, starting ecm.exe incurs some initialization costs (extra time), which will build up with each rebalance. It was really interesting to hear this suggestion and think about the pros/cons, but I've decided not to implement it.

Antonio 2015-03-05 10:41

[QUOTE=WraithX;395503]Announcing ecm.py version 0.32. The only change was to add the -inp feature request from above. Starting the program can look like either of the following now:
[CODE]
python.exe ecm.py [gmp-ecm options] [ecm.py options] B1 < <in_file>
ecm.py -inp <in_file> [gmp-ecm options] [ecm.py options] B1
[/CODE]

Implementing number 1 was simple enough.

Implementing number 2 would be really difficult, and potentially not beneficial. Currently I start different ecm.exe processes each with num_curves/num_threads curves of work to do. To "rebalance the load", I'd have to kill all running instances of ecm.exe and then start them up again. Each time you rebalance, you will probably lose between (num_threads/2) to (num_threads) number of curves of work (due to killing ecm.exe when it is partially through its current curve). And, starting ecm.exe incurs some initialization costs (extra time), which will build up with each rebalance. It was really interesting to hear this suggestion and think about the pros/cons, but I've decided not to implement it.[/QUOTE]

Thanks for implementing (1), I can understand the reason for rejecting (2), however, have you considered the impact of only rebalancing the load if a thread runs out of work (and there are more than, say, 2*threads number of curves remaining to be done?

Now for something more serious:-

As an exercise I was trying out ecm.py on 23*10^20371+7 and found a composite factor, 570237 (3*67*2837) so I tried
(23*10^20371+7)/(3*67*2837) and this produced the following:

[CODE]F:\ECM>rem t20
F:\ECM>ecm.py -inp num2ecm.txt -c 80 11000
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is copyright, 2011-2015, David Cleaver |
-> | and is a conversion of factmsieve.py that is Copyright, 2010, |
-> | Brian Gladman. Version 0.32 (Python 2.6 or later) 14th Feb 2015 |
-> |_________________________________________________________________|
inp_file = num2ecm.txt
-> Number(s) to factor:
Traceback (most recent call last):
File "F:\ECM\ecm.py", line 1493, in <module>
parse_ecm_options(sys.argv, set_args = True, first = True)
File "F:\ECM\ecm.py", line 1397, in parse_ecm_options
print('-> {0:s} ({1:d} digits)'.format(line, num_digits(eval(line.replace('^', '**')))))
File "<string>", line 1, in <module>
OverflowError: integer division result too large for a float[/CODE]

This is on Windows 7 sp1 64bit with Python version: -
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40)

Any ideas?

casevh 2015-03-05 13:58

In Python 2, the division operator performs integer division, but in Python 3 was changed to always return a float. Both versions of Python support "//" to perform integer division. Can you try changing your input to use "//"?

Antonio 2015-03-05 19:26

Changing from / to // allows the python script to work, but when the number is passed to ecm.exe it doesn't parse the // correctly and ignores it and all that follows. So it just finds the 570237 composite factor again.

casevh 2015-03-05 20:46

Untested, but try changing the offending line (1397 ??) from

[CODE]print('-> {0:s} ({1:d} digits)'.format(line, num_digits(eval(line.replace('^', '**')))))[/CODE]

to

[CODE]print('-> {0:s} ({1:d} digits)'.format(line, num_digits(eval(line.replace('^', '**').replace('/', '//')))))[/CODE]

Antonio 2015-03-05 22:58

[QUOTE=casevh;397108]Untested, but try changing the offending line (1397 ??) from

[CODE]print('-> {0:s} ({1:d} digits)'.format(line, num_digits(eval(line.replace('^', '**')))))[/CODE]

to

[CODE]print('-> {0:s} ({1:d} digits)'.format(line, num_digits(eval(line.replace('^', '**').replace('/', '//')))))[/CODE][/QUOTE]


I applied the suggested change to lines
987
1384
1397
1523
1573
and
1634
all now seems to be working ok
many thanks.

WraithX 2015-03-06 03:35

Announcing ecm.py v0.33...
 
1 Attachment(s)
[QUOTE=casevh;397108][CODE]print('-> {0:s} ({1:d} digits)'.format(line, num_digits(eval(line.replace('^', '**').replace('/', '//')))))[/CODE][/QUOTE]

[QUOTE=Antonio;397124]I applied the suggested change to lines
987, 1384, 1397, 1523, 1573, and 1634
all now seems to be working ok
many thanks.[/QUOTE]

A big thanks to casevh for suggesting and Antonio for verifying this solution. I've decided to simplify the code a bit and have put the eval(n.replace('^', '**').replace('/', '//')) in the num_digits function, so now the use of num_digits looks like: format(ecm_n, num_digits(ecm_n)). This looks a lot cleaner, and any future changes can be made in one place.

Here is the updated version, ecm.py v0.33.

Antonio 2015-03-06 21:19

[QUOTE=WraithX;397137]A big thanks to casevh for suggesting and Antonio for verifying this solution. I've decided to simplify the code a bit and have put the eval(n.replace('^', '**').replace('/', '//')) in the num_digits function, so now the use of num_digits looks like: format(ecm_n, num_digits(ecm_n)). This looks a lot cleaner, and any future changes can be made in one place.

Here is the updated version, ecm.py v0.33.[/QUOTE]

Glad to be of service, thanks for the update.

Antonio 2015-03-09 19:30

Is there a way to specify B1 AND B2 when using the python script?
The standard ECM command line doesn't work.

[CODE]
F:\ECM>ecm.py -maxmem 12288 -inp num2ecm.txt -c 2100 250000 425000000
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is copyright, 2011-2015, David Cleaver |
-> | and is a conversion of factmsieve.py that is Copyright, 2010, |
-> | Brian Gladman. Version 0.33 (Python 2.6 or later) 5th Mar 2015 |
-> |_________________________________________________________________|
inp_file = num2ecm.txt
-> Number(s) to factor:
-> (85*2^67652-1)/(3*924972809) (20358 digits)
->=============================================================================
-> Working on number: (85*2^67652-1)/(3*924972809) (20358 digits)
-> Currently working on: job4826.txt
-> Starting 3 instances of GMP-ECM...
-> ecm 250000 -c 700 -maxmem 4096 425000000 < job4826.txt > job4826_t00.txt
-> ecm 250000 -c 700 -maxmem 4096 425000000 < job4826.txt > job4826_t01.txt
-> ecm 250000 -c 700 -maxmem 4096 425000000 < job4826.txt > job4826_t02.txt
Invalid B2 value: -c
Invalid B2 value: -c
Invalid B2 value: -c
GMP-ECM 7.0-dev [configured with GMP 6.0.0, --enable-asm-redc] [ECM]
-> *** Error: unexpected return value: 1[/CODE]

WraithX 2015-03-15 19:16

Announcing ecm.py v0.34...
 
1 Attachment(s)
[QUOTE=Antonio;397337]Is there a way to specify B1 AND B2 when using the python script?
The standard ECM command line doesn't work.
[CODE]F:\ECM>ecm.py -maxmem 12288 -inp num2ecm.txt -c 2100 250000 425000000[/CODE][/QUOTE]

Thank you for putting ecm.py through its paces! It should be possible to specify both B1 and B2 now.

Announcing ecm.py v0.34:
[CODE]
New features:
- Can now specify both B1 and B2 (B2 is optional, if not specified gmp-ecm will choose a value)
- Can now echo numbers from the command line into ecm.py, like so:
echo <num> | python.exe ecm.py [gmp-ecm options] [ecm.py options] B1 [B2]

And as a reminder, ecm.py can be used in any of the following three ways:
USAGE: python.exe ecm.py [gmp-ecm options] [ecm.py options] B1 [B2] < <in_file>
or: echo <num> | python.exe ecm.py [gmp-ecm options] [ecm.py options] B1 [B2]
or: ecm.py -inp <in_file> [gmp-ecm options] [ecm.py options] B1 [B2]
where <in_file> is a file with the number(s) to factor (one per line)
[/CODE]

cgy606 2016-06-08 04:32

[QUOTE=WraithX;258976]Ok, I have what appears to be a working version of this driver ready. This release will be version 0.01, but hopefully it will function well for everyone. Like factMsieve.py, this requires Python 2.6 or higher to run. Here are some notes about its use. A typical command line will look like one of the following:

D:\Programming\python\ecm_py>c:\python26\python.exe ecm.py -c 100 -one -maxmem 1500 -threads 2 -out all_out.txt 1000000 < test2.txt
or
D:\Programming\python\ecm_py>echo 189137687020123159261852192605885322671854225839439913905089 | c:\python26\python.exe ecm.py -c 100 -one -maxmem 1500 -threads 2 -out all_out.txt 1000000

The command line has to have /path/to/python.exe before ecm.py. This is needed because of a limitation in Windows (possibly Linux also) where just writing ecm.py will not allow redirection to work properly. This seems to be based on a mishandling of starting python scripts through file associations. However, if the program is called as python.exe ecm.py (if python is in your path), or /path/to/python.exe ecm.py, redirection will work correctly. This includes both piping-in (echo ... | python ecm.py) and file redirection (python ecm.py ... < num.txt).

You can use all the standard gmp-ecm command line options.
If you use -c N (run N curves), then each gmp-ecm instance will get -c (N/num_threads) curves to run, with the first few instances running more if that division is not exact.
If you use -maxmem N (limit ecm.exe memory usage in Stage 2 to < N MB of memory), then each instance of gmp-ecm will use only -maxmem (N/num_threads) to make sure that the total memory used by all instances of gmp-ecm doesn't go over your limit.

When this script is factoring a number, it creates a job file representing how much work you've asked to complete on that number. It also keeps track of how many curves have been run on this number, and timings to help you know how long each stage is taking on average. Each instance of gmp-ecm will write to its own file. The script monitors these files to see how much work has been done, and updates a counter on the screen to let you know how far along the job is.

The script has an auto-resume feature. If the job is interrupted by a power outage or with Ctrl-C, you don't have to worry about losing work. The next time you start up the job with the same command line on the same number, it will read in the old job files to see how much work had been done, and then pick up where it left off. If you turn off auto-resume, the script won't check for previous runs, it will just start a new job.

Here are the options available that are specific to the script:
-threads N This will run N copies of gmp-ecm on the input number(s) you specify.
-r <file> Resume the specified job file. After finishing this job, the script will stop.
-out <file> Save work done (on all input numbers) to the specified file.

These command line options can be intermixed with regular gmp-ecm command line options. Also, please make sure that B1 is the last option in your ecm command line. If not, the program will not behave as expected. In future releases I think I can post a proper error if B1 isn't the last option, but for now, a correct command line should work as expected.

Before running, please set the path to your gmp-ecm executable. You can also set a default number of threads to use if you dont' want to specify that on the command line. Also, you can run multiple copies of this script in the same directory. There shouldn't be any collisions between separate jobs being run.

If anyone is interested in running this, I'd like to hear how well it works for you. If you encounter a problem, I can try to find a solution and post updated versions as I have time. Now, let the fun begin![/QUOTE]

Hi WraithX,

I was wondering if you (or somebody else) could help me running the python script. So I am using python 2.6 for Mac OSX Yosemite 10.10.5. I have gmp-ecm 7.0 installed and the python script ecm.py in the same folder. I am running the python script using the command:

bash-3.2$ echo "2^521-1" | python ecm.py -c 100 -threads 2 1000000
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is copyright, 2011-2015, David Cleaver |
-> | and is a conversion of factmsieve.py that is Copyright, 2010, |
-> | Brian Gladman. Version 0.34 (Python 2.6 or later) 14th Mar 2015 |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 2^521-1 (157 digits)
-> Could not find the program: ./Users/cgy606/Documents/yafu/ecm-7.0./ecm
-> Did you set the path properly in this script?
-> It is currently set to:
-> ECM_BIN_PATH = ./Users/cgy606/Documents/yafu/ecm-7.0

Apparently their is an issue with the ecm binary path. However, ecm 7.0 does work:

bash-3.2$ echo "2^521-1" | ecm -c 2 1000000
GMP-ECM 6.4 [configured with GMP 6.0.0, --enable-asm-redc] [ECM]
Input number is 2^521-1 (157 digits)
Using B1=1000000, B2=1045563762, polynomial Dickson(6), sigma=3061204579
Step 1 took 6590ms
Step 2 took 3210ms
Run 2 out of 2:
Using B1=1000000, B2=1045563762, polynomial Dickson(6), sigma=717933293
Step 1 took 6671ms
Step 2 took 3162ms

Although it says gmp-ecm 6.4, I downloaded the latest version. Any thoughts on how to get this to work?

VBCurtis 2016-06-08 04:58

Did you compile the new ECM, or just download the files? Your second invocation does not indicate what folder you are in when invoking ecm, so I can't tell if that old ECM binary is located in a different folder. Seems likely, since it's 6.4.

cgy606 2016-06-08 05:16

[QUOTE=VBCurtis;435772]Did you compile the new ECM, or just download the files? Your second invocation does not indicate what folder you are in when invoking ecm, so I can't tell if that old ECM binary is located in a different folder. Seems likely, since it's 6.4.[/QUOTE]

Okay, so I was able to configure by running that script, now I tried to run install-sh:

bash-3.2$ ls
AUTHORS addlaws.h config.log ecm.c libtool pentium4 stamp-h1
COPYING aprtcle config.m4 ecm.h listz.c pm1.c stamp-h2
COPYING.LIB arm config.status ecm.h.in longlong.h pm1fs2.c test.ecm
ChangeLog athlon config.sub ecm.py ltmain.sh polyeval.c test.pm1
Fgw.c auxarith.c configure ecm.xml lucas.c powerpc32 test.pp1
INSTALL auxi.c configure.ac ecm2.c m4 powerpc64 test_mulredc.c
INSTALL-ecm auxlib.c cudakernel.cu ecm_int.h main.c pp1.c testlong.ecm
M877.save b1_ainc.c cudakernel.h ecm_ntt.c median.c prime95.save testlong.pm1
M997.save basicdefs.h cudawrapper.c ecm_py.log memusage.c random.c testlong.pp1
Makefile batch.c depcomp ecmfactor.c mips resume.c torsions.h
Makefile.am bench_mulredc.c dummy.save eval.c missing rho.c tune.c
Makefile.in bestd.c dummy2.save factor.c mpmod.c schoen_strass.c x86
NEWS c155 ecm-ecm.h generic mpmod.h sets_long.c x86_64
README candi.c ecm-gmp.h getprime_r.c mpzspm.c sp.c
README.lib champions.h ecm-gpu.h getprime_r.h mpzspv.c sp.h
TODO compile ecm-impl.h hppa mul_lo.c sparc64
Z2102.n config.guess ecm-params.h ia64 ntt_gfp.c spm.c
acinclude.m4 config.h ecm-py_v0.34.zip install-sh numbers.txt spv.c
aclocal.m4 config.h.in ecm.1 ks-multiply.c parametrizations.c stage2.c
bash-3.2$ ./install-sh
./install-sh: no input file specified.
bash-3.2$ sh install-sh
install-sh: no input file specified.
bash-3.2$ sh INSTALL
INSTALL: line 1: Installation: command not found
INSTALL: line 2: AUTHORS: command not found
INSTALL: line 4: syntax error near unexpected token `C'
INSTALL: line 4: `Copyright (C) 1994-1996, 1999-2002, 2004-2013 Free Software Foundation,'
bash-3.2$ sh INSTALL-ecm
INSTALL-ecm: line 1: Instructions: command not found
INSTALL-ecm: line 3: syntax error near unexpected token `)'
INSTALL-ecm: line 3: `0) you first need to install the GNU MP (GMP) library.'
bash-3.2$

I pretty sure I am doing something boneheaded...

Dubslow 2016-06-08 07:42

INSTALL is a text file. Open it and read its contents to tell you how to install GMP-ECM. It should be something like this:

[code]./configure
make
make test
make install[/code]

R. Gerbicz 2016-06-08 08:03

[QUOTE=cgy606;435769]However, ecm 7.0 does work:

bash-3.2$ echo "2^521-1" | ecm -c 2 1000000
GMP-ECM 6.4 [configured with GMP 6.0.0, --enable-asm-redc] [ECM]
Input number is 2^521-1 (157 digits)
[/QUOTE]

I would make no (ecm) factorization work on a known Mersenne prime.

cgy606 2016-06-08 15:25

[QUOTE=Dubslow;435787]INSTALL is a text file. Open it and read its contents to tell you how to install GMP-ECM. It should be something like this:

[code]./configure
make
make test
make install[/code][/QUOTE]

Thanks for the help. I built gmp-ecm 7.0 (and I believe its a proper built, see the terminal output below). However, I still think there is something wrong with the path set to the ecm binaries. Has anybody actually ran this python script on a mac (or unix)?

bash-3.2$ echo "116! + 5" | ecm -c 1 1000000
GMP-ECM 7.0-rc2 [configured with GMP 6.0.0, --enable-asm-redc] [ECM]
Input number is 116!+5 (191 digits)
Using B1=1000000, B2=1045563762, polynomial Dickson(6), sigma=1:2621072992
Step 1 took 9334ms
********** Factor found in step 1: 2045
Found composite factor of 4 digits: 2045
Composite cofactor (116!+5)/2045 has 188 digits
bash-3.2$ pwd
/Users/cgy606/Documents/yafu/ecm-7.0
bash-3.2$ echo "16592218505877252817595384320726519331245298560877363179444333539424579615734146863694149619172215730904092217599971446347242578938012841662398382297100491858685120782396088019559902200489" | python ecm.py -c 16 -threads 4 1000000
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is copyright, 2011-2015, David Cleaver |
-> | and is a conversion of factmsieve.py that is Copyright, 2010, |
-> | Brian Gladman. Version 0.34 (Python 2.6 or later) 14th Mar 2015 |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 16592218505877252817595384320726519331245298560877363179444333539424579615734146863694149619172215730904092217599971446347242578938012841662398382297100491858685120782396088019559902200489 (188 digits)
-> Could not find the program: ./Users/cgy606/Documents/yafu/ecm-7.0./ecm
-> Did you set the path properly in this script?
-> It is currently set to:
-> ECM_BIN_PATH = ./Users/cgy606/Documents/yafu/ecm-7.0

Dubslow 2016-06-08 21:10

Edit the script and change this line:

[code]ECM_BIN_PATH = ./Users/cgy606/Documents/yafu/ecm-7.0[/code]

to this line:

[code]ECM_BIN_PATH = ecm[/code]

If that doesn't work, then get the shell output of [c]which ecm[/c] and set [c]ECM_BIN_PATH[/c] to that.

cgy606 2016-06-09 05:39

[QUOTE=Dubslow;435850]Edit the script and change this line:

[code]ECM_BIN_PATH = ./Users/cgy606/Documents/yafu/ecm-7.0[/code]

to this line:

[code]ECM_BIN_PATH = ecm[/code]

If that doesn't work, then get the shell output of [c]which ecm[/c] and set [c]ECM_BIN_PATH[/c] to that.[/QUOTE]

That didn't work:

[code]bash-3.2$ which ecm
/usr/local/bin/ecm
bash-3.2$ echo "16592218505877252817595384320726519331245298560877363179444333539424579615734146863694149619172215730904092217599971446347242578938012841662398382297100491858685120782396088019559902200489" | python ecm.py -c 16 -threads 4 1000000
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is copyright, 2011-2015, David Cleaver |
-> | and is a conversion of factmsieve.py that is Copyright, 2010, |
-> | Brian Gladman. Version 0.34 (Python 2.6 or later) 14th Mar 2015 |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 16592218505877252817595384320726519331245298560877363179444333539424579615734146863694149619172215730904092217599971446347242578938012841662398382297100491858685120782396088019559902200489 (188 digits)
-> Could not find the program: ./usr/local/bin./ecm
-> Did you set the path properly in this script?
-> It is currently set to:
-> ECM_BIN_PATH = ./usr/local/bin
bash-3.2$

wombatman 2016-06-09 07:22

This line:

[CODE] ./usr/local/bin[B][U].[/U][/B]/ecm[/CODE]

would seem to be the issue, no? Can you run "./ecm" from the command line? If so, set the BIN_PATH to "" and it should work.

Dubslow 2016-06-09 08:39

[c]ECM_BIN_PATH = /usr/local/bin/ecm[/c]

cgy606 2016-06-09 16:43

[QUOTE=wombatman;435880]This line:

[CODE] ./usr/local/bin[B][U].[/U][/B]/ecm[/CODE]

would seem to be the issue, no? Can you run "./ecm" from the command line? If so, set the BIN_PATH to "" and it should work.[/QUOTE]

So I tried that and I get an error:

[CODE] bash-3.2$ ./ecm
Invalid arguments. See ./ecm --help.
[/CODE]

I am concerned that maybe my ecm installation might be wrong...

cgy606 2016-06-09 16:47

[QUOTE=Dubslow;435885][c]ECM_BIN_PATH = /usr/local/bin/ecm[/c][/QUOTE]

So I looked in the python script and you do not set "ECM_BIN_PATH". Rather one sets "ECM_PATH"

[CODE] # ###############################################################
# User Defined Variables, set before using...
# ###############################################################
# Set binary directory paths
ECM_PATH = './usr/local/bin'
[/CODE]

And then the script prints out "ECM_BIN_PATH" (if their is an error in locating the binaries):

[CODE]
# check that an executable file exists

def check_binary(exe):
if CHECK_BINARIES:
if not os.path.exists(ECM_PATH + exe + EXE_SUFFIX):
print('-> Could not find the program: {0:s}'.format(ECM_PATH + exe + EXE_SUFFIX))
print('-> Did you set the path properly in this script?')
print('-> It is currently set to:')
print('-> ECM_BIN_PATH = {0:s}'.format(ECM_PATH))
sys.exit(-1)
if not os.path.isfile(ECM_PATH + exe + EXE_SUFFIX):
print('-> The following is not a file: {0:s}'.format(ECM_PATH + exe + EXE_SUFFIX))
print('-> Did you set the path properly in this script?')
print('-> It is currently set to:')
print('-> ECM_BIN_PATH = {0:s}'.format(ECM_PATH))
sys.exit(-1)
[/CODE]

I searched the entire script to make sure their are no other instances of "ECM_BIN_PATH", I couldn't find any (BTW I am using v 034 of the python script). Perhaps I am missing something?

wombatman 2016-06-09 16:58

[QUOTE=cgy606;435904]So I tried that and I get an error:

[CODE] bash-3.2$ ./ecm
Invalid arguments. See ./ecm --help.
[/CODE]

I am concerned that maybe my ecm installation might be wrong...[/QUOTE]

No, that's actually good. Try changing the ECM_PATH to "" and run the script.

cgy606 2016-06-09 17:02

[QUOTE=wombatman;435907]No, that's actually good. Try changing the ECM_PATH to "" and run the script.[/QUOTE]

It works!!! I knew I was doing something boneheaded :bounce:

[CODE]

bash-3.2$ echo "16592218505877252817595384320726519331245298560877363179444333539424579615734146863694149619172215730904092217599971446347242578938012841662398382297100491858685120782396088019559902200489" | python ecm.py -c 16 -threads 4 1000000
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is copyright, 2011-2015, David Cleaver |
-> | and is a conversion of factmsieve.py that is Copyright, 2010, |
-> | Brian Gladman. Version 0.34 (Python 2.6 or later) 14th Mar 2015 |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 16592218505877252817595384320726519331245298560877363179444333539424579615734146863694149619172215730904092217599971446347242578938012841662398382297100491858685120782396088019559902200489 (188 digits)
->=============================================================================
-> Working on number: 165922185058772528...088019559902200489 (188 digits)
-> Currently working on: job9870.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -c 4 1000000 < job9870.txt > job9870_t00.txt
-> ./ecm -c 4 1000000 < job9870.txt > job9870_t01.txt
-> ./ecm -c 4 1000000 < job9870.txt > job9870_t02.txt
-> ./ecm -c 4 1000000 < job9870.txt > job9870_t03.txt

GMP-ECM 7.0-rc2 [configured with GMP 6.0.0, --enable-asm-redc] [ECM]
Using B1=1000000, B2=1045563762, polynomial Dickson(6), 4 threads
____________________________________________________________________________
Curves Complete | Average seconds/curve | Runtime | ETA
-----------------|---------------------------|---------------|--------------
8 of 16 | Stg1 16.90s | Stg2 8.497s | 0d 00:01:16 | 0d 00:02:15

Run 8 out of 16:
Using B1=1000000, B2=1045563762, polynomial Dickson(6), sigma=1:1972960141
Step 1 took 16815ms
********** Factor found in step 1: 99966551448308840149
Found prime factor of 20 digits: 99966551448308840149
Composite cofactor 165977702196287452743884566421762233309052750989118251380454425629289120250968365614593763832099318096182205447167223938283861744093049940708376659370742242048186838661 has 168 digits

[/CODE]

It even found a factor. Now quick question, it finds the factor after 8 curves, and then quits. Is their a way to force the script to keep running until all the curves are done or all factors are prime (i.e. the number is fully factored)?

VBCurtis 2016-06-09 18:31

The problem with having it keep running is that only the thread that found the factor would "know" about it; the other threads would continue to try to factor the original number. This is slower, often by a fair bit, than running curves on the new cofactor.

I usually run up to t40 with regular single-threaded ECM (which *does* continue after finding a factor), then use the python script for higher bounds; this reduces the incidence of ecm.py stopping its work due to a found factor (both because there are lower chances per day to find a factor once t40 has been run, and higher chance the cofactor is composite when I do find a 43+ digit factor).

cgy606 2016-06-09 19:53

Well, I guess I could modify the script so that it does this. What I had in mind is that if one of the threads finds a factor, it first tests the primality of the large cofactor (ecm already does this), and if it is still composite, the script could kill all the other process, recalculate the input number (ecm should spit this out), and then reschedule the remains curves. I know yafu does this (or a process similar to this), but I was never able to get yafu to work on my macbook pro. Anyways, this should be a good exercise for me to do since I should learn python scripting anyways, for future jobs in the financial sector...

VBCurtis 2016-06-09 22:06

If you do produce this mod, please post your work! I'd be happy to use that.

WraithX 2016-06-10 03:37

[QUOTE=cgy606;435908]It works!!! I knew I was doing something boneheaded :bounce:

It even found a factor. Now quick question, it finds the factor after 8 curves, and then quits. Is their a way to force the script to keep running until all the curves are done or all factors are prime (i.e. the number is fully factored)?[/QUOTE]

I believe I can add this functionality. However, there are some questions about how to proceed. There are several cases that need to be considered:
1) If the factor is prime and the cofactor is prime, I think it's obvious we can stop.
2) If the factor is prime and the cofactor is composite
3) If the factor is composite and the cofactor is prime
4) If the factor is composite and the cofactor is composite

Should we start running the original number of curves on the new composite? Or only run the remaining number of curves on the composite? My guess from what you are asking is to "run the remaining number of curves" on the new composite. And then also, if two (or more) composites are found, should we run the remaining number of curves on each new composite, or run the original number of curves on each composite?

The second part of your question (keep running until the number is completely factored) I think is outside the scope of ecm.py. Much like it is outside the scope of ecm.exe. There is no logic (in ecm.py or ecm.exe) that can keep running curves, and keep choosing bounds to find potentially larger and larger factors that can adequately take into account: the amount of ram a system has available (or will have available in the future), and when the best time to switch over to different factoring methods, like gnfs/snfs, would be. That is more in the realm of yafu.

I can implement the first part of your question, but won't be implementing the second part. Also, since I'm now using the gpu capabilities of ecm, I need a way to start multiple ecm's to resume those saved stage1 residues. So, I'll be adding in the ability for ecm.py to recognize and work with the "-resume" option.

Also, one thing you should know about how paths work. In a previous post you wrote that you set:
[C]ECM_PATH = './usr/local/bin'[/C]
That would not point to the [C]/usr/local/bin[/C] directory, because you have a "." in front of it. The "." means start from the current working directory and look into these sub folders for what you want. So, if you ran the ecm.py script from the (made up) directory
[C]/Users/cgy606/Documents/ecmpy/[/C] and had set [C]ECM_PATH = './usr/local/bin'[/C], it would look for the ecm executable in the directory:
[C]/Users/cgy606/Documents/ecmpy/usr/local/bin/[/C]
Which proabably doesn't exist, and so will definitely fail. You could have set:
[C]ECM_PATH = '/usr/local/bin'[/C]
Without the leading "." and that would have worked because it is an absolute path, and not a relative path. Since you've got it working now, you don't need to change it, but I wanted to let you know about the difference between absolute paths and relative paths.

WraithX 2016-06-10 03:52

Announcing ecm.py v0.35...
 
1 Attachment(s)
Recently I noticed that the ecm.py script would crash if it was given factorial or primorial input strings. This is because the python "eval" function can't handle these characters. So, instead of writing my own equation parser to figure out how many digits are in these input strings, I've just grabbed the output from ecm.exe to see how many digits it reports are in the input number. So,

Announcing ecm.py v0.35:
[CODE]
Fixed:
- ecm.py no longer calculates number of digits on its own, it reads this information from the ecm executable. This fixed a problem where the python "eval" function would crash when it encountered factorial or primorial characters.
ie, you can now do: [C]echo "140!+1" | python ecm.py 1e6[/C] and it will work correctly, without crashing.
- also fixed the output when the ecm binary is not found. It will no longer print out the misleading "ECM_BIN_PATH", it will print out "ECM_PATH" to match the variable name in the python code.
[/CODE]

cgy606 2016-06-10 05:38

[QUOTE=WraithX;435938]Recently I noticed that the ecm.py script would crash if it was given factorial or primorial input strings. This is because the python "eval" function can't handle these characters. So, instead of writing my own equation parser to figure out how many digits are in these input strings, I've just grabbed the output from ecm.exe to see how many digits it reports are in the input number. So,

Announcing ecm.py v0.35:
[CODE]
Fixed:
- ecm.py no longer calculates number of digits on its own, it reads this information from the ecm executable. This fixed a problem where the python "eval" function would crash when it encountered factorial or primorial characters.
ie, you can now do: [C]echo "140!+1" | python ecm.py 1e6[/C] and it will work correctly, without crashing.
- also fixed the output when the ecm binary is not found. It will no longer print out the misleading "ECM_BIN_PATH", it will print out "ECM_PATH" to match the variable name in the python code.
[/CODE][/QUOTE]

Great, I have been factoring factorial +/- numbers.

cgy606 2016-06-10 06:06

[QUOTE=WraithX;435937]I believe I can add this functionality. However, there are some questions about how to proceed. There are several cases that need to be considered:
1) If the factor is prime and the cofactor is prime, I think it's obvious we can stop.
2) If the factor is prime and the cofactor is composite
3) If the factor is composite and the cofactor is prime
4) If the factor is composite and the cofactor is composite

Should we start running the original number of curves on the new composite? Or only run the remaining number of curves on the composite? My guess from what you are asking is to "run the remaining number of curves" on the new composite. And then also, if two (or more) composites are found, should we run the remaining number of curves on each new composite, or run the original number of curves on each composite?

The second part of your question (keep running until the number is completely factored) I think is outside the scope of ecm.py. Much like it is outside the scope of ecm.exe. There is no logic (in ecm.py or ecm.exe) that can keep running curves, and keep choosing bounds to find potentially larger and larger factors that can adequately take into account: the amount of ram a system has available (or will have available in the future), and when the best time to switch over to different factoring methods, like gnfs/snfs, would be. That is more in the realm of yafu.

I can implement the first part of your question, but won't be implementing the second part. Also, since I'm now using the gpu capabilities of ecm, I need a way to start multiple ecm's to resume those saved stage1 residues. So, I'll be adding in the ability for ecm.py to recognize and work with the "-resume" option.

Also, one thing you should know about how paths work. In a previous post you wrote that you set:
[C]ECM_PATH = './usr/local/bin'[/C]
That would not point to the [C]/usr/local/bin[/C] directory, because you have a "." in front of it. The "." means start from the current working directory and look into these sub folders for what you want. So, if you ran the ecm.py script from the (made up) directory
[C]/Users/cgy606/Documents/ecmpy/[/C] and had set [C]ECM_PATH = './usr/local/bin'[/C], it would look for the ecm executable in the directory:
[C]/Users/cgy606/Documents/ecmpy/usr/local/bin/[/C]
Which proabably doesn't exist, and so will definitely fail. You could have set:
[C]ECM_PATH = '/usr/local/bin'[/C]
Without the leading "." and that would have worked because it is an absolute path, and not a relative path. Since you've got it working now, you don't need to change it, but I wanted to let you know about the difference between absolute paths and relative paths.[/QUOTE]

So some food for thought on how to proceed. Clearly we need not be concerned about case '1' as in principal that is already implemented... we check the factor found and the cofactor for primality, if they both pass, kill the script and drink a beer.

Case '2' is the most fruitful of our efforts. The 'best' way to proceed IMHO is to kill all the threads running (assuming that the script starts and stops at roughly the same time at the start of each curve, the way that yafu works), test the primality of each factor. If the larger one is composite (WLOG let us assume that the smaller factor is the one found), then the script determines how many curves have been completed (at the current B1/B2 values), calculates how many curves remain in order to complete the original input, and then reschedules the remain number of curves given the number of threads being used. To illustrate:

Factoring C170 B1 = 3e6 B2 =### threads = 4 total curves remaining = 2352
factor found prp37 (curve 221 thread = 1 sigma = ###)
composite cofactor C134

Factoring C134 B1 = 3e6 B2 =### threads = 4 total curves = 1468

I think you get the idea...

Case '3' is nothing more than a transpose of case '2'. We found a "small" composite factor and a large probable prime factor. In principal we could reschedule the curves like we did in case '2' but their is probably a better way to factor this number, which I will explain in case '4'

For case '4' we find 2 composite cofactors. Let's assume for the sake of argument that one is larger (i.e. more decimal digits than the other). We could continue factoring that one in the same fashion as we did in case '2', but let's turn our attention to the smaller one. What does it mean when ecm finds two composite cofactors? Usually it means that smaller factors were not eliminated from the beginning (i.e. with some other factoring method like trial division or rho P+1/P-1) and thus B1 was selected so high that in a single curve, it effectively found 2 factors and not one (we would like to claim this was intentional but no one would believe this statement)! Anyways, if it finds a factor of N digits, then the smallest cofactor of this composite number can have at most ~N/2 digits. But how large is this composite cofactor of the original number we are factoring expected to be? Well, the current ecm record is 82 digits (I think). For the sake of argument, let's be a little conservative and assume that somebody out their runs a curve at B1 = 25e9 (or something crazy like that) on a C300 number and finds a C90 and a C210 (lucky!!!). Clearly, one of the C90 factors should have been found be about 5k curves at B1 = 11e6 (on average of course). In principal we could run ecm on the C90 until B1 = 11e6 bounds or we could let SIQS or some other factoring algorithm hack at it (perhaps even trial division given that maybe even smaller factors were not eliminated from the C300 to begin with). Anyways, the story I am trying to paint here is that if two composites are found, it basically means that a very large B1 bound was selected while at the same time small factors were not eliminated. Given that this cofactor is not large (less than 90 digits or so), we should focus on the larger (and more important cofactor to continuing factoring on) and reschedule the remaining curves for that guy analogous to case '2'.

I hope this makes sense...

WraithX 2016-06-18 20:25

Announcing ecm.py v0.36...
 
1 Attachment(s)
Announcing ecm.py v0.36:
[CODE]
New Feature:
- Added the ability for ecm.py to perform the remaining number of requested curves on any composite factors found.
(You can activate this by setting "find_one_factor_and_stop = 0", it is 1 by default)
[/CODE]

I've added the ability for ecm.py to continue working on any composite factors found, it will perform the remaining number of requested curves. I've run quite a few tests locally and it seems to work well. However, if you do run into any problems, please let me know.

WraithX 2016-07-10 04:24

Announcing ecm.py v0.38...
 
1 Attachment(s)
Announcing ecm.py v0.38:
[CODE]
New feature:
- Added the ability for ecm.py to resume a GMP-ECM (compatible) save file, and
it will evenly distribute the resume lines across several instances of GMP-ECM
[/CODE]
Calling this can be as simple as:
[C]ecm.py -resume resume.txt[/C]

Or you can use additional options, like:
[CODE]
ecm.py -threads 3 -out output.txt -maxmem 300 -pollfiles 60 -resume resume.txt
------------------------------------------------------------------------------
Which would spread the resume lines from resume.txt across 3 instances of gmp-ecm,
and give each one the command line option "-maxmem 100" ( = 300/3)
and poll the output files every 60 seconds to look for factors, or see if a gmp-ecm instance has finished
and save all gmp-ecm output to the file output.txt
* Like always, you can specify the "threads" and "pollfiles" options inside the script
[/CODE]

Here is a description of this new feature, which can also be found in the script:
[CODE]
# If we are using the "-resume" feature of gmp-ecm, we will make some assumptions about the job...
# 1) This is designed to be a _simple_ way to speed up resuming ecm by running several resume jobs in parallel.
# ie, we will not try to replicate all resume capabilities of gmp-ecm
# 2) If we find identical lines in our resume file, we will only resume one of them and skip the others
# - If this happens, we will print out a notice to the user (if VERBOSE >= v_normal) so they know what is going on
# 3) We will use the B1 value in the resume file, and not resume with higher values of B1
# 4) We will let gmp-ecm determine which B2 value to use, which can be affected by "-maxmem" and "-k"
# 5) We will try to split up the resume work evenly between the threads.
# - We will put total/num_threads resume lines into each file, and total%num_threads files will each get one extra line.
# At the end of a job or when restarting a job, we will write any completed resume lines out to a "finished file"
# This "finished file" will be used to help us keep track of work done, in case we are interrupted and need to (re)resume later
# We will query the output files once every poll_file_delay seconds.
# resume_job_<filename>_inp_t00.txt # input resume file for use by gmp-ecm in thread 0
# resume_job_<filename>_inp_t01.txt # input resume file for use by gmp-ecm in thread 1
# ...etc...
# resume_job_<filename>_out_t00.txt # output file for resume job of gmp-ecm in thread 0
# resume_job_<filename>_out_t01.txt # output file for resume job of gmp-ecm in thread 1
# ...etc...
# resume_job_<filename>_finished.txt # file where we write out each resume line that we have finished with gmp-ecm
# where <filename> is based on the resume file name, but with any "." characters replaced by a dash.
[/CODE]

I know this skips over v0.37. I have created a version 0.37 with similar functionality, but it put each resume line into its own file (one at a time, not all at once) and would give that input file to gmp-ecm to resume, and save the output to another file. Once that resume line was finished processing, it would delete both the input and output files, and then move on to the next resume line. So, if a resume file had 1000 lines to resume, then the script would created/delete 1000 input files and 1000 output files. I didn't want to tax any filesystems by creating/deleting so many files, so I rewrote it as detailed above.

wombatman 2016-07-10 05:29

This is AWESOME. Thanks for making this update. :smile:

swellman 2016-07-14 12:45

+1

Fantastic functionality. Love it!

UBR47K 2016-08-03 06:14

Is there anyway to specify B1 value when using the "-resume" switch?
I'd like to use GMP-ECM for stage 2 with Prime95 stage 1 results.txt

cgy606 2016-08-05 23:46

I tried running the script on a resume file produced from a gpu stage 1 run. I am getting the following error:


python ecm.py -threads 8 -resume gpu.save
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work |
-> | on a single machine. It is copyright, 2011-2016, David Cleaver |
-> | and is a conversion of factmsieve.py that is Copyright, 2010, |
-> | Brian Gladman. Version 0.38 (Python 2.6 or later) 7th Jul 2016 |
-> |_________________________________________________________________|

-> Resuming work from resume file: gpu.save
-> Spreading the work across 8 thread(s)
->=============================================================================
-> Working on the number(s) in the resume file: gpu.save
-> Using up to 8 instances of GMP-ECM...
-> Found 1024 unique resume lines to work on.
-> Will start working on the 1024 resume lines.
Traceback (most recent call last):
File "ecm.py", line 2393, in <module>
parse_ecm_options(sys.argv, set_args = True, first = True)
File "ecm.py", line 2235, in parse_ecm_options
run_ecm_resume_job()
File "ecm.py", line 1850, in run_ecm_resume_job
threadList = [[i, '', 0, '', '', [], False] for i in xrange(intNumThreads)]
NameError: name 'xrange' is not defined

Any ideas about what is going wrong?

VBCurtis 2016-08-06 00:19

Looks like you didn't give B1 or B2 parameters to ecm.py.

When I do stage 2 from a GPU'ed stage 1, I put on the command line the same B1 value I ran Stage 1 on (note you can put a higher one here, and it'll use the CPU to extend B1 before starting stage 2).


All times are UTC. The time now is 14:20.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.