mersenneforum.org A Python Diver for GMP-ECM...
 Register FAQ Search Today's Posts Mark Forums Read

2012-09-22, 21:36   #34
WraithX

Mar 2006

23×59 Posts

Quote:
 Originally Posted by Dubslow In Linux, the file type is NOT associated with the python executable; rather the shell (bash, in my and most cases) looks at the first line for the so-called "shebang" (I prefer "hashbang") which looks like #! path-to-exe; the shell then calls path-to-exe $0$@, which is exactly the same as if I did it myself
Ah, this is good to know. Thanks for that info.

Well, the next thing to try is to:
Set NICE_PATH to different values, I'm not sure which one to put here
and change the Popen line to:
p = subprocess.Popen(args.split(' '), shell=True, **al)

And also, can you add the following two lines right above the Popen line:
Code:
  print('-> args: {0:s}', str(args))
print('->   al: {0:s}', str(al))
I want to make sure nothing fishy is going on there.

2012-09-22, 21:52   #35
Dubslow

"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88

3×29×83 Posts

Quote:
 Originally Posted by WraithX Ah, this is good to know. Thanks for that info.
http://en.wikipedia.org/wiki/Shebang_(Unix)
http://docs.python.org/tutorial/inte...python-scripts
Quote:
 Originally Posted by WraithX Well, the next thing to try is to: Set NICE_PATH to different values, I'm not sure which one to put here and change the Popen line to: p = subprocess.Popen(args.split(' '), shell=True, **al) And also, can you add the following two lines right above the Popen line: Code:  print('-> args: {0:s}', str(args)) print('-> al: {0:s}', str(al)) I want to make sure nothing fishy is going on there.
There's the problem. Should be ./ecm, not ../ecm.
Code:
bill@Gravemind:~/yafu/ecm∰∂ grep "NICE_PATH" ecm.py
NICE_PATH = '' #'/usr/bin/nice -19'
pass #al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ grep -B 2 "Popen" ecm.py
print('-> args: {0:s}', str(args))
print('->   al: {0:s}', str(al))
p = subprocess.Popen(args.split(' '), shell=True, **al)
#p = subprocess.Popen(args.split(' '), **al)
#p = subprocess.Popen([ex] + args.split(' '), **al)
#p = subprocess.Popen(cs.split(' '), **al)
bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work   |
-> | on a single machine.  It is Copyright, 2011, David Cleaver and  |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman.   Version 0.08 (Python 2.6 or later) 19th Oct 2011.    |
-> |_________________________________________________________________|

-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
-> args: {0:s}  -one -c 1889 43000000
->   al: {0:s} {'executable': '../ecm', 'stdin': <_io.TextIOWrapper name='job3362.txt' mode='r' encoding='UTF-8'>, 'stdout': <_io.TextIOWrapper name='job3362_t00.txt' mode='w' encoding='UTF-8'>}
Traceback (most recent call last):
File "./ecm.py", line 1227, in <module>
File "./ecm.py", line 524, in start_ecm_threads
procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
File "./ecm.py", line 327, in run_exe
p = subprocess.Popen(args.split(' '), shell=True, **al)
File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.2/subprocess.py", line 1330, in _execute_child
raise child_exception_type(errno_num, err_msg)
OSError: [Errno 13] Permission denied
Code:
ECM_PATH = '.'
#...
ECM = 'ecm'

if sys.platform.startswith('win'):
EXE_SUFFIX = '.exe'
else:
EXE_SUFFIX = ''
NICE_PATH = ''
ECM = './' + ECM
#...
ex = (ECM_PATH + exe + EXE_SUFFIX)
al['executable'] = ex
I don't have a clean copy of the script, so I don't know if this is my fault or not (though it probably is).

Last fiddled with by Dubslow on 2012-09-22 at 21:57

2012-09-22, 22:17   #36
WraithX

Mar 2006

23·59 Posts

Quote:
 Originally Posted by Dubslow There's the problem. Should be ./ecm, not ../ecm. I don't have a clean copy of the script, so I don't know if this is my fault or not (though it probably is).

For the path argument, if ecm is in the same folder, I have always used "./" and not ".". This should be all you need to change to get the ecm executable working.

As for nice, I would suggest the following:
NICE_PATH = '/usr/bin/nice -n 19'

2012-09-30, 17:41   #37
WraithX

Mar 2006

23×59 Posts

Announcing ecm.py version 0.10, the changes are:
Quote:
 - added an extra check to help make sure ecm_path is set correctly - added an extra verbose level (3) that will print some debug info to try to help diagnose any problems - changed how the number of digits in an input number is calculated -- * it is now based on math.log10() instead of counting characters in a string -- * this is a big speed improvement for numbers with {1's,10's,100's,...} of thousands of digits - changed time averaging so it no longer averages in the very rare negative time reported by gmp-ecm
Attached Files
 ecm-py_v0.10.zip (11.0 KB, 269 views)

 2012-10-04, 03:38 #38 jwaltos     Apr 2012 367 Posts ecm.py attachment Thanks WraithX.
 2013-05-05, 03:17 #39 EdH     "Ed Hall" Dec 2009 Adirondack Mtns E2B16 Posts Is it possible to convert ecm.py such that it will operate across other machines? I'm toying with several ideas along this line, but unsure whether to look into the workings of the script myself or not. I'm not well versed in Python and could possibly use this as a way to learn some. Anyway, as it runs, everything is done within the original terminal. Could the separate threads be run in separate terminals, such as called by gnome-terminal? If so, could these threads be run as ssh connections on remote machines? Or, if run as totally independent processes on the alternate machines, would there be a way of catching a factor, if found on one of the other machines? Would the data transfer overhead or the variance in thread processing negate any possible gains of creating several threads across differing equipment types? Additionally, if the above is at least a possibility, would there be a way to scale the work allocated such that faster machines get a bigger allocation? Thanks for any ideas (and especially for a great script)... Last fiddled with by EdH on 2013-05-05 at 03:17
2013-05-05, 05:23   #40
schickel

"Frank <^>"
Dec 2004
CDP Janesville

84A16 Posts

Quote:
 Originally Posted by EdH Is it possible to convert ecm.py such that it will operate across other machines? I'm toying with several ideas along this line, but unsure whether to look into the workings of the script myself or not. I'm not well versed in Python and could possibly use this as a way to learn some. Anyway, as it runs, everything is done within the original terminal. Could the separate threads be run in separate terminals, such as called by gnome-terminal? If so, could these threads be run as ssh connections on remote machines? Or, if run as totally independent processes on the alternate machines, would there be a way of catching a factor, if found on one of the other machines? Would the data transfer overhead or the variance in thread processing negate any possible gains of creating several threads across differing equipment types? Additionally, if the above is at least a possibility, would there be a way to scale the work allocated such that faster machines get a bigger allocation? Thanks for any ideas (and especially for a great script)...
I would say that if you're going to look into running more than one machine, ECMNET would be the better way to go.

I have my network setup with a master server here on my machine, with two dedicated machines that boot up with a slave server/client setup. That way, even if I have trouble with the network or this machine, the ECM boxes keep on running. The network overhead is miniscule, and you can just load up a bunch of low priority work along with your higher priority work, that way if something factors, there's some fall back work.

If you go with v2.0, the backend for the server is a simple text file, no DB engine required. And the clients keep the workload balanced, since each can run independently, no manual intervention required....

 2013-05-05, 09:37 #41 debrouxl     Sep 2009 977 Posts Agreed, ECMNet is pretty good, even on a single host I'm sticking to 2.x due to the lack of dependency on a SQL engine.
2013-05-05, 12:38   #42
xilman
Bamboozled!

"𒉺𒌌𒇷𒆷𒀭"
May 2003
Down not across

295B16 Posts

Quote:
 Originally Posted by debrouxl Agreed, ECMNet is pretty good, even on a single host I'm sticking to 2.x due to the lack of dependency on a SQL engine.
Seconded. It just works --- the V2.x anyway. We never managed to get the DB interface working on my Linux box. One day, perhaps, Mark and I will return to the issue but we're each busy with other things ATM.

If you would like assistance setting up ECMNET clients and/or master server and/or slave server just ask. My fee is very reasonable: a few curves contributed to 83.217.167.177:8194

Paul

2013-05-05, 13:02   #43
PhilF

Feb 2005

2·5·59 Posts

Quote:
 Originally Posted by xilman My fee is very reasonable: a few curves contributed to 83.217.167.177:8194
Now *that* is a reasonable fee!

 2013-05-05, 13:20 #44 EdH     "Ed Hall" Dec 2009 Adirondack Mtns 70538 Posts Thanks! I will definitely look into this. Currently, I have scripts across several machines that "wait" for sieving assignments from one of them. The assignments are initiated by a script running Aliqueit on a separate machine. However, there is just too much wasted time in between sieving assignments. I had considered adjusting the waiting machines to do some ECM work during the off times. And, then I thought of expanding the ecm.py script that the Aliqueit machine runs. Perhaps ECMNET is what I'm looking for, since you mention being able to add in other jobs between assignments, as well. I was running curves on the c173 from 4788 continuously on all the machines until my attention span waned after a couple months. (The paint just wouldn't dry...) Then, I wrote all the sieving scripts and am running those, but looking for something else... Unfortunately, I have a lot of non-math/computer stuff building up to keep me busy, so the more automated, the better.

 Similar Threads Thread Thread Starter Forum Replies Last Post kelzo Programming 3 2016-11-27 05:16 daxmick Programming 2 2014-02-10 01:45 Xyzzy Programming 20 2009-09-08 15:51 yqiang GMP-ECM 2 2007-04-22 00:14 a216vcti Programming 7 2005-10-30 00:37

All times are UTC. The time now is 21:34.

Mon Mar 1 21:34:44 UTC 2021 up 88 days, 17:46, 0 users, load averages: 2.59, 2.68, 2.76