mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > GMP-ECM

Reply
 
Thread Tools
Old 2013-05-05, 14:29   #45
schickel
 
schickel's Avatar
 
"Frank <^>"
Dec 2004
CDP Janesville

2×1,061 Posts
Default

Quote:
Originally Posted by EdH View Post
Thanks!

I will definitely look into this. Currently, I have scripts across several machines that "wait" for sieving assignments from one of them. The assignments are initiated by a script running Aliqueit on a separate machine. However, there is just too much wasted time in between sieving assignments. I had considered adjusting the waiting machines to do some ECM work during the off times. And, then I thought of expanding the ecm.py script that the Aliqueit machine runs. Perhaps ECMNET is what I'm looking for, since you mention being able to add in other jobs between assignments, as well.

I was running curves on the c173 from 4788 continuously on all the machines until my attention span waned after a couple months. (The paint just wouldn't dry...) Then, I wrote all the sieving scripts and am running those, but looking for something else...

Unfortunately, I have a lot of non-math/computer stuff building up to keep me busy, so the more automated, the better.
Hmm; you'd have to test the speed loss, but ECMNET is configured to run the clients at idle priority, so if your sievers run at normal, they would take precedence over the ECM thingies. They still might not run quite at top speed, though.
schickel is offline   Reply With Quote
Old 2013-05-05, 19:36   #46
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

2×29×71 Posts
Default

Quote:
Originally Posted by schickel View Post
Hmm; you'd have to test the speed loss, but ECMNET is configured to run the clients at idle priority, so if your sievers run at normal, they would take precedence over the ECM thingies. They still might not run quite at top speed, though.
These machines have no other work, so maybe I can set them up with ECMNET running 24/7, with assignments coming from the same machine that handles the sieving assignments. The assignments will come based on a separate machine that is running Aliqueit or similar. While Aliqueit is running ecm.py (or something to "see" ECMNET results), the machines will get ECM assignments, when it switches to gnfs, sieving assignments will take over the machines which should put the ECMNET clients to sleep.

I'll have to work out the details of signalling what to run, communication with the Aliqueit machine and how to cancel ECM when switching, etc., but it should all be doable.

Now if I can just find some time and keep my attention directed...
EdH is offline   Reply With Quote
Old 2013-05-06, 04:42   #47
debrouxl
 
debrouxl's Avatar
 
Sep 2009

97810 Posts
Default

The ECMNet clients can run threads at any priority, at least under Linux. I often use nice 19, but I've occasionally used nice 0. If ECMNet ran as root, it might even run processes at negative nice values, but then it would hardly make a difference wrt. nice 0 anyway.
debrouxl is offline   Reply With Quote
Old 2014-12-12, 03:47   #48
WraithX
 
WraithX's Avatar
 
Mar 2006

2·35 Posts
Default

Announcing ecm.py version 0.30, the changes are:
Code:
-New feature that shows ETA (estimated time to job completion)
 * Since it is based on the average runtime of the job so far, the estimate can be a little early or late.  But, in general gives a good idea of how long the current job will take.  

-New format for the running output.  And you can select between different displays of Runtime and ETA like so:
# Examples of how you can mix and match Runtime and ETA outputs:
#____________________________________________________________________________
# Curves Complete |   Average seconds/curve   |    Runtime    |      ETA
#-----------------|---------------------------|---------------|--------------
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |      1945628s |  41d 08:37:51
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |       22.518d |  41d 08:37:51
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |  22d 12:27:08 |      3573471s
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |  22d 12:27:08 |       41.359d
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |  22d 12:27:08 |  41d 08:37:51

-New email feature.
 * This option will send out an email when the job completes, either when it finds a factor or when all curves have finished with no factor found.  The email will let you know which event happened.
 * This option can also email out progress reports at regular intervals, so you can have a record of how far along remote machines are in their job.

-New log file feature.
 * This option keeps a record of the work that ecm.py has done.  Each line in the log file gets a time stamp so you can know when jobs were started and when they finished.
 * This option can also record the above "runtime information" at regular intervals.
Attached Files
File Type: zip ecm-py_v0.30.zip (14.2 KB, 256 views)
WraithX is offline   Reply With Quote
Old 2014-12-12, 04:20   #49
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

3×5×7×17 Posts
Default

Awesome stuff!
wombatman is offline   Reply With Quote
Old 2014-12-30, 22:18   #50
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

33718 Posts
Default

I'm running into an error using the newest ecm.py. I get an index out of range error for this line:

Code:
using_line = '{0:s},{1:s},{2:s}, {3:d} thread{4:s}'.format(ud[0],ud[1],ud[2],actual_num_threads, '' if (actual_num_threads == 1) else 's')
I made sure there were no job files, and it always fails at this line, regardless of the number of threads. Interestingly, if I start ecm.py with no "number of curves" to run (that is, no -c parameter), it starts up fine. If I run with "-c 1", though, it crashes at the line above. Not sure if that helps, but I figured I would mention it.
wombatman is offline   Reply With Quote
Old 2014-12-31, 03:53   #51
WraithX
 
WraithX's Avatar
 
Mar 2006

2·35 Posts
Default

Quote:
Originally Posted by wombatman View Post
I'm running into an error using the newest ecm.py. I get an index out of range error for this line:

Code:
using_line = '{0:s},{1:s},{2:s}, {3:d} thread{4:s}'.format(ud[0],ud[1],ud[2],actual_num_threads, '' if (actual_num_threads == 1) else 's')
Hmmm, can you post what a full run looks like for your gmp-ecm binary? Try it with a simple command line like:
ecm.exe -c 2 50e3 < num.txt

For me, that looks like:
Code:
GMP-ECM 7.0-dev [configured with GMP 6.0.0, --enable-asm-redc] [ECM]
Input number is 6962815472410550045247870144481111677555048961510968048443680963
65759227064152650920222099941253589429607814503738760057809654611250765236890819
4481257411644162747345175089477132247 (181 digits)
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:3093725542
Step 1 took 343ms
Step 2 took 235ms
Run 2 out of 2:
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:201126966
Step 1 took 312ms
Step 2 took 250ms
WraithX is offline   Reply With Quote
Old 2014-12-31, 06:01   #52
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

33718 Posts
Default

Running with the same number you used and the following command:
Code:
 echo "number" | ecm.exe -c 2 50e3
I get:
Code:
GMP-ECM 7.0-dev [configured with GMP 6.0.0] [ECM]
Input number is 69628154724105500452478701444811116775550489615109680484436809636575922
706415265092022209994125358942960781450373876005780965461125076523689081944812574116441
62747345175089477132247 (181 digits)
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:2321917342
Step 1 took 234ms
Step 2 took 187ms
Run 2 out of 2:
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:1182338425
Step 1 took 234ms
Step 2 took 172ms
wombatman is offline   Reply With Quote
Old 2014-12-31, 12:45   #53
WraithX
 
WraithX's Avatar
 
Mar 2006

2×35 Posts
Default

Ok, that output looks like what is normally expected. Can you send me some extra information about your environment?
What is your operating system?
What version of python are you using?
What is the exact command line you are using?
Can you edit ecm.py and change VERBOSE to 3 at line 136 (in the User Defined Variables), run the program again, let it crash, and then post the full output (from both ecm.py and the python error afterwards)?

That should help me start diagnosing this problem. Thanks!
WraithX is offline   Reply With Quote
Old 2014-12-31, 16:53   #54
wombatman
I moo ablest echo power!
 
wombatman's Avatar
 
May 2013

110111110012 Posts
Default

You bet!

OS is Windows 7 Pro 64-bit, but I'm running the script inside of MinGW-64. Previous version worked perfectly like this.

Python version is 3.3.3.

Exact command line is:
Code:
py ecm.py -v -c 2 110e6 < input.txt
Result, with the verbose change you requested, is:
Code:
$ py ecm.py -v -c 2 110e6 < c178_hp2.ini 48111167755504896
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work   |
-> | on a single machine.  It is Copyright, 2012, David Cleaver and  |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman.   Version 0.30 (Python 2.6 or later) 30th Nov 2014.    |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 267584724287272660632588026709919587698209050387915658332777631397485002205943824599
399098348970270487342146242128808502451298903034661323335707683105729279570983958104088
3292553 (178 digits)
-> Original command line was:
->  -v -c 2 110e6
-> New command line(s) will be either:
->  -v 110e6
->  -v 110e6

->=============================================================================
-> Working on number: 26758472428727266063258802670991958769820905038791565833277763139
748500220594382459939909834897027048734214624212880850245129890303466132333570768310572
92795709839581040883292553 (178 digits)
-> Currently working on: job4858.txt
-> Original command line was:
->  -v -c 2 110e6
-> New command line(s) will be either:
->  -c 1 110e6
->  -c 2 110e6

-> Starting 2 instances of GMP-ECM...
-> ecm -c 1 110e6 < job4858.txt > job4858_t00.txt
-> ecm args:  -c 1 110e6
-> ecm   al: {'stdin': <_io.TextIOWrapper name='job4858.txt' mode='r' encoding='cp1252'
>, 'executable': './ecm.exe', 'creationflags': 64, 'stdout': <_io.TextIOWrapper name='j
ob4858_t00.txt' mode='w' encoding='cp1252'>}
-> ecm -c 1 110e6 < job4858.txt > job4858_t01.txt
-> ecm args:  -c 1 110e6
-> ecm   al: {'stdin': <_io.TextIOWrapper name='job4858.txt' mode='r' encoding='cp1252'
>, 'executable': './ecm.exe', 'creationflags': 64, 'stdout': <_io.TextIOWrapper name='j
ob4858_t01.txt' mode='w' encoding='cp1252'>}

GMP-ECM 7.0-dev [configured with GMP 6.0.0] [ECM]
Using B1=110000000, B2=776278396540, polynomial Dickson(30), 2 threads


....and it appears to work fine like that. And even with "-c 100" it works fine. And passing in "-threads 4" works too.

So basically, with VERBOSE=3, it works fine.

Edit: And changing VERBOSE to 1, it still appears to work fine. I'm so confused now.

Last fiddled with by wombatman on 2014-12-31 at 16:55
wombatman is offline   Reply With Quote
Old 2015-01-01, 15:32   #55
WraithX
 
WraithX's Avatar
 
Mar 2006

2·35 Posts
Default ecm.py v0.31

Quote:
Originally Posted by wombatman View Post
....and it appears to work fine like that. And even with "-c 100" it works fine. And passing in "-threads 4" works too.

So basically, with VERBOSE=3, it works fine.

Edit: And changing VERBOSE to 1, it still appears to work fine. I'm so confused now.
My best guess is that there was a race condition where ecm.exe was writing to an output file and ecm.py was reading in that same file. In this case ecm.py happened to read the file where the line:
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:1182338425
Was incomplete. Perhaps only
Using B1=50000, B2=
had been written to the file at the time of reading. Since my code was expecting a whole line there (with two or more commas) it crashed on you. And, if you tried to resume from that, it probably would have crashed in the same place.

I've updated the code to try to avoid this happening in the future. It's hard to test since the problem should only happen very rarely. But, if successful, you won't see this problem ever again!

Thanks for finding this, and let me know how the new version works for you!
Attached Files
File Type: zip ecm-py_v0.31.zip (14.3 KB, 301 views)
WraithX is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Python Coding Help? kelzo Programming 3 2016-11-27 05:16
PHP vs. Python vs. C (all with GMP) daxmick Programming 2 2014-02-10 01:45
Python... Xyzzy Programming 20 2009-09-08 15:51
using libecm from python yqiang GMP-ECM 2 2007-04-22 00:14
Help w/ python. a216vcti Programming 7 2005-10-30 00:37

All times are UTC. The time now is 06:35.


Wed Dec 8 06:35:03 UTC 2021 up 138 days, 1:04, 1 user, load averages: 0.74, 1.09, 1.12

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.