mersenneforum.org A Python Diver for GMP-ECM...
 Register FAQ Search Today's Posts Mark Forums Read

2013-05-05, 14:29   #45
schickel

"Frank <^>"
Dec 2004
CDP Janesville

84A16 Posts

Quote:
 Originally Posted by EdH Thanks! I will definitely look into this. Currently, I have scripts across several machines that "wait" for sieving assignments from one of them. The assignments are initiated by a script running Aliqueit on a separate machine. However, there is just too much wasted time in between sieving assignments. I had considered adjusting the waiting machines to do some ECM work during the off times. And, then I thought of expanding the ecm.py script that the Aliqueit machine runs. Perhaps ECMNET is what I'm looking for, since you mention being able to add in other jobs between assignments, as well. I was running curves on the c173 from 4788 continuously on all the machines until my attention span waned after a couple months. (The paint just wouldn't dry...) Then, I wrote all the sieving scripts and am running those, but looking for something else... Unfortunately, I have a lot of non-math/computer stuff building up to keep me busy, so the more automated, the better.
Hmm; you'd have to test the speed loss, but ECMNET is configured to run the clients at idle priority, so if your sievers run at normal, they would take precedence over the ECM thingies. They still might not run quite at top speed, though.

2013-05-05, 19:36   #46
EdH

"Ed Hall"
Dec 2009

32×13×31 Posts

Quote:
 Originally Posted by schickel Hmm; you'd have to test the speed loss, but ECMNET is configured to run the clients at idle priority, so if your sievers run at normal, they would take precedence over the ECM thingies. They still might not run quite at top speed, though.
These machines have no other work, so maybe I can set them up with ECMNET running 24/7, with assignments coming from the same machine that handles the sieving assignments. The assignments will come based on a separate machine that is running Aliqueit or similar. While Aliqueit is running ecm.py (or something to "see" ECMNET results), the machines will get ECM assignments, when it switches to gnfs, sieving assignments will take over the machines which should put the ECMNET clients to sleep.

I'll have to work out the details of signalling what to run, communication with the Aliqueit machine and how to cancel ECM when switching, etc., but it should all be doable.

Now if I can just find some time and keep my attention directed...

 2013-05-06, 04:42 #47 debrouxl     Sep 2009 977 Posts The ECMNet clients can run threads at any priority, at least under Linux. I often use nice 19, but I've occasionally used nice 0. If ECMNet ran as root, it might even run processes at negative nice values, but then it would hardly make a difference wrt. nice 0 anyway.
2014-12-12, 03:47   #48
WraithX

Mar 2006

1D816 Posts

Announcing ecm.py version 0.30, the changes are:
Code:
-New feature that shows ETA (estimated time to job completion)
* Since it is based on the average runtime of the job so far, the estimate can be a little early or late.  But, in general gives a good idea of how long the current job will take.

-New format for the running output.  And you can select between different displays of Runtime and ETA like so:
# Examples of how you can mix and match Runtime and ETA outputs:
#____________________________________________________________________________
# Curves Complete |   Average seconds/curve   |    Runtime    |      ETA
#-----------------|---------------------------|---------------|--------------
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |      1945628s |  41d 08:37:51
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |       22.518d |  41d 08:37:51
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |  22d 12:27:08 |      3573471s
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |  22d 12:27:08 |       41.359d
#  2114 of   6000 | Stg1  2983s | Stg2 693.5s |  22d 12:27:08 |  41d 08:37:51

-New email feature.
* This option will send out an email when the job completes, either when it finds a factor or when all curves have finished with no factor found.  The email will let you know which event happened.
* This option can also email out progress reports at regular intervals, so you can have a record of how far along remote machines are in their job.

-New log file feature.
* This option keeps a record of the work that ecm.py has done.  Each line in the log file gets a time stamp so you can know when jobs were started and when they finished.
* This option can also record the above "runtime information" at regular intervals.
Attached Files
 ecm-py_v0.30.zip (14.2 KB, 159 views)

 2014-12-12, 04:20 #49 wombatman I moo ablest echo power!     May 2013 1,741 Posts Awesome stuff!
 2014-12-30, 22:18 #50 wombatman I moo ablest echo power!     May 2013 1,741 Posts I'm running into an error using the newest ecm.py. I get an index out of range error for this line: Code: using_line = '{0:s},{1:s},{2:s}, {3:d} thread{4:s}'.format(ud[0],ud[1],ud[2],actual_num_threads, '' if (actual_num_threads == 1) else 's') I made sure there were no job files, and it always fails at this line, regardless of the number of threads. Interestingly, if I start ecm.py with no "number of curves" to run (that is, no -c parameter), it starts up fine. If I run with "-c 1", though, it crashes at the line above. Not sure if that helps, but I figured I would mention it.
2014-12-31, 03:53   #51
WraithX

Mar 2006

23·59 Posts

Quote:
 Originally Posted by wombatman I'm running into an error using the newest ecm.py. I get an index out of range error for this line: Code: using_line = '{0:s},{1:s},{2:s}, {3:d} thread{4:s}'.format(ud[0],ud[1],ud[2],actual_num_threads, '' if (actual_num_threads == 1) else 's')
Hmmm, can you post what a full run looks like for your gmp-ecm binary? Try it with a simple command line like:
ecm.exe -c 2 50e3 < num.txt

For me, that looks like:
Code:
GMP-ECM 7.0-dev [configured with GMP 6.0.0, --enable-asm-redc] [ECM]
Input number is 6962815472410550045247870144481111677555048961510968048443680963
65759227064152650920222099941253589429607814503738760057809654611250765236890819
4481257411644162747345175089477132247 (181 digits)
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:3093725542
Step 1 took 343ms
Step 2 took 235ms
Run 2 out of 2:
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:201126966
Step 1 took 312ms
Step 2 took 250ms

 2014-12-31, 06:01 #52 wombatman I moo ablest echo power!     May 2013 1,741 Posts Running with the same number you used and the following command: Code:  echo "number" | ecm.exe -c 2 50e3 I get: Code: GMP-ECM 7.0-dev [configured with GMP 6.0.0] [ECM] Input number is 69628154724105500452478701444811116775550489615109680484436809636575922 706415265092022209994125358942960781450373876005780965461125076523689081944812574116441 62747345175089477132247 (181 digits) Using B1=50000, B2=12746592, polynomial x^2, sigma=1:2321917342 Step 1 took 234ms Step 2 took 187ms Run 2 out of 2: Using B1=50000, B2=12746592, polynomial x^2, sigma=1:1182338425 Step 1 took 234ms Step 2 took 172ms
 2014-12-31, 12:45 #53 WraithX     Mar 2006 23·59 Posts Ok, that output looks like what is normally expected. Can you send me some extra information about your environment? What is your operating system? What version of python are you using? What is the exact command line you are using? Can you edit ecm.py and change VERBOSE to 3 at line 136 (in the User Defined Variables), run the program again, let it crash, and then post the full output (from both ecm.py and the python error afterwards)? That should help me start diagnosing this problem. Thanks!
 2014-12-31, 16:53 #54 wombatman I moo ablest echo power!     May 2013 6CD16 Posts You bet! OS is Windows 7 Pro 64-bit, but I'm running the script inside of MinGW-64. Previous version worked perfectly like this. Python version is 3.3.3. Exact command line is: Code: py ecm.py -v -c 2 110e6 < input.txt Result, with the verbose change you requested, is: Code: \$ py ecm.py -v -c 2 110e6 < c178_hp2.ini 48111167755504896 -> ___________________________________________________________________ -> | Running ecm.py, a Python driver for distributing GMP-ECM work | -> | on a single machine. It is Copyright, 2012, David Cleaver and | -> | is a conversion of factmsieve.py that is Copyright, 2010, Brian | -> | Gladman. Version 0.30 (Python 2.6 or later) 30th Nov 2014. | -> |_________________________________________________________________| -> Number(s) to factor: -> 267584724287272660632588026709919587698209050387915658332777631397485002205943824599 399098348970270487342146242128808502451298903034661323335707683105729279570983958104088 3292553 (178 digits) -> Original command line was: -> -v -c 2 110e6 -> New command line(s) will be either: -> -v 110e6 -> -v 110e6 ->============================================================================= -> Working on number: 26758472428727266063258802670991958769820905038791565833277763139 748500220594382459939909834897027048734214624212880850245129890303466132333570768310572 92795709839581040883292553 (178 digits) -> Currently working on: job4858.txt -> Original command line was: -> -v -c 2 110e6 -> New command line(s) will be either: -> -c 1 110e6 -> -c 2 110e6 -> Starting 2 instances of GMP-ECM... -> ecm -c 1 110e6 < job4858.txt > job4858_t00.txt -> ecm args: -c 1 110e6 -> ecm al: {'stdin': <_io.TextIOWrapper name='job4858.txt' mode='r' encoding='cp1252' >, 'executable': './ecm.exe', 'creationflags': 64, 'stdout': <_io.TextIOWrapper name='j ob4858_t00.txt' mode='w' encoding='cp1252'>} -> ecm -c 1 110e6 < job4858.txt > job4858_t01.txt -> ecm args: -c 1 110e6 -> ecm al: {'stdin': <_io.TextIOWrapper name='job4858.txt' mode='r' encoding='cp1252' >, 'executable': './ecm.exe', 'creationflags': 64, 'stdout': <_io.TextIOWrapper name='j ob4858_t01.txt' mode='w' encoding='cp1252'>} GMP-ECM 7.0-dev [configured with GMP 6.0.0] [ECM] Using B1=110000000, B2=776278396540, polynomial Dickson(30), 2 threads ....and it appears to work fine like that. And even with "-c 100" it works fine. And passing in "-threads 4" works too. So basically, with VERBOSE=3, it works fine. Edit: And changing VERBOSE to 1, it still appears to work fine. I'm so confused now. Last fiddled with by wombatman on 2014-12-31 at 16:55
2015-01-01, 15:32   #55
WraithX

Mar 2006

23·59 Posts
ecm.py v0.31

Quote:
 Originally Posted by wombatman ....and it appears to work fine like that. And even with "-c 100" it works fine. And passing in "-threads 4" works too. So basically, with VERBOSE=3, it works fine. Edit: And changing VERBOSE to 1, it still appears to work fine. I'm so confused now.
My best guess is that there was a race condition where ecm.exe was writing to an output file and ecm.py was reading in that same file. In this case ecm.py happened to read the file where the line:
Using B1=50000, B2=12746592, polynomial x^2, sigma=1:1182338425
Was incomplete. Perhaps only
Using B1=50000, B2=
had been written to the file at the time of reading. Since my code was expecting a whole line there (with two or more commas) it crashed on you. And, if you tried to resume from that, it probably would have crashed in the same place.

I've updated the code to try to avoid this happening in the future. It's hard to test since the problem should only happen very rarely. But, if successful, you won't see this problem ever again!

Thanks for finding this, and let me know how the new version works for you!
Attached Files
 ecm-py_v0.31.zip (14.3 KB, 197 views)

 Similar Threads Thread Thread Starter Forum Replies Last Post kelzo Programming 3 2016-11-27 05:16 daxmick Programming 2 2014-02-10 01:45 Xyzzy Programming 20 2009-09-08 15:51 yqiang GMP-ECM 2 2007-04-22 00:14 a216vcti Programming 7 2005-10-30 00:37

All times are UTC. The time now is 22:33.

Mon Mar 1 22:33:54 UTC 2021 up 88 days, 18:45, 0 users, load averages: 2.01, 2.49, 2.52