A Python Diver for GMP-ECM...

[WraithX]

Originally Posted by Dubslow

In Linux, the file type is NOT associated with the python executable; rather the shell (bash, in my and most cases) looks at the first line for the so-called "shebang" (I prefer "hashbang") which looks like `#! path-to-exe`; the shell then calls `path-to-exe $0 $@`, which is exactly the same as if I did it myself

Ah, this is good to know. Thanks for that info.

Well, the next thing to try is to:
Set NICE_PATH to different values, I'm not sure which one to put here
and change the Popen line to:
p = subprocess.Popen(args.split(' '), shell=True, **al)

And also, can you add the following two lines right above the Popen line:

  print('-> args: {0:s}', str(args))
  print('->   al: {0:s}', str(al))

I want to make sure nothing fishy is going on there.

[Dubslow]

Originally Posted by WraithX

Ah, this is good to know. Thanks for that info.

http://en.wikipedia.org/wiki/Shebang_(Unix)
http://docs.python.org/tutorial/inte...python-scripts

Originally Posted by WraithX

Well, the next thing to try is to:
Set NICE_PATH to different values, I'm not sure which one to put here
and change the Popen line to:
p = subprocess.Popen(args.split(' '), shell=True, **al)

And also, can you add the following two lines right above the Popen line:

  print('-> args: {0:s}', str(args))
  print('->   al: {0:s}', str(al))

I want to make sure nothing fishy is going on there.

There's the problem. Should be ./ecm, not ../ecm.

bill@Gravemind:~/yafu/ecm∰∂ grep "NICE_PATH" ecm.py
  NICE_PATH = '' #'/usr/bin/nice -19'
    pass #al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ grep -B 2 "Popen" ecm.py
  print('-> args: {0:s}', str(args))
  print('->   al: {0:s}', str(al))
  p = subprocess.Popen(args.split(' '), shell=True, **al)
  #p = subprocess.Popen(args.split(' '), **al)
  #p = subprocess.Popen([ex] + args.split(' '), **al)
  #p = subprocess.Popen(cs.split(' '), **al)
bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work   |
-> | on a single machine.  It is Copyright, 2011, David Cleaver and  |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman.   Version 0.08 (Python 2.6 or later) 19th Oct 2011.    |
-> |_________________________________________________________________|
 
-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
-> args: {0:s}  -one -c 1889 43000000
->   al: {0:s} {'executable': '../ecm', 'stdin': <_io.TextIOWrapper name='job3362.txt' mode='r' encoding='UTF-8'>, 'stdout': <_io.TextIOWrapper name='job3362_t00.txt' mode='w' encoding='UTF-8'>}
Traceback (most recent call last):
  File "./ecm.py", line 1227, in <module>
    start_ecm_threads()
  File "./ecm.py", line 524, in start_ecm_threads
    procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
  File "./ecm.py", line 327, in run_exe
    p = subprocess.Popen(args.split(' '), shell=True, **al)
  File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.2/subprocess.py", line 1330, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 13] Permission denied

ECM_PATH = '.'
#...
ECM = 'ecm'

if sys.platform.startswith('win'):
  EXE_SUFFIX = '.exe'
else:
  EXE_SUFFIX = ''
  NICE_PATH = ''
  ECM = './' + ECM
#...
  ex = (ECM_PATH + exe + EXE_SUFFIX)
  al['executable'] = ex

I don't have a clean copy of the script, so I don't know if this is my fault or not (though it probably is).

[WraithX]

Originally Posted by Dubslow

There's the problem. Should be ./ecm, not ../ecm.

I don't have a clean copy of the script, so I don't know if this is my fault or not (though it probably is).

Well, you can always download a clean copy from earlier in the thread.

For the path argument, if ecm is in the same folder, I have always used "./" and not ".". This should be all you need to change to get the ecm executable working.

As for nice, I would suggest the following:
NICE_PATH = '/usr/bin/nice -n 19'

[WraithX]

Announcing ecm.py version 0.10, the changes are:

- added an extra check to help make sure ecm_path is set correctly
- added an extra verbose level (3) that will print some debug info to try to help diagnose any problems
- changed how the number of digits in an input number is calculated
-- * it is now based on math.log10() instead of counting characters in a string
-- * this is a big speed improvement for numbers with {1's,10's,100's,...} of thousands of digits
- changed time averaging so it no longer averages in the very rare negative time reported by gmp-ecm

[jwaltos]

Thanks WraithX.

[EdH]

Is it possible to convert ecm.py such that it will operate across other machines?

I'm toying with several ideas along this line, but unsure whether to look into the workings of the script myself or not. I'm not well versed in Python and could possibly use this as a way to learn some.

Anyway, as it runs, everything is done within the original terminal. Could the separate threads be run in separate terminals, such as called by gnome-terminal? If so, could these threads be run as ssh connections on remote machines? Or, if run as totally independent processes on the alternate machines, would there be a way of catching a factor, if found on one of the other machines?

Would the data transfer overhead or the variance in thread processing negate any possible gains of creating several threads across differing equipment types?

Additionally, if the above is at least a possibility, would there be a way to scale the work allocated such that faster machines get a bigger allocation?

Thanks for any ideas (and especially for a great script)...

[schickel]

Originally Posted by EdH

Is it possible to convert ecm.py such that it will operate across other machines?

I'm toying with several ideas along this line, but unsure whether to look into the workings of the script myself or not. I'm not well versed in Python and could possibly use this as a way to learn some.

Anyway, as it runs, everything is done within the original terminal. Could the separate threads be run in separate terminals, such as called by gnome-terminal? If so, could these threads be run as ssh connections on remote machines? Or, if run as totally independent processes on the alternate machines, would there be a way of catching a factor, if found on one of the other machines?

Would the data transfer overhead or the variance in thread processing negate any possible gains of creating several threads across differing equipment types?

Additionally, if the above is at least a possibility, would there be a way to scale the work allocated such that faster machines get a bigger allocation?

Thanks for any ideas (and especially for a great script)...

I would say that if you're going to look into running more than one machine, ECMNET would be the better way to go.

I have my network setup with a master server here on my machine, with two dedicated machines that boot up with a slave server/client setup. That way, even if I have trouble with the network or this machine, the ECM boxes keep on running. The network overhead is miniscule, and you can just load up a bunch of low priority work along with your higher priority work, that way if something factors, there's some fall back work.

If you go with v2.0, the backend for the server is a simple text file, no DB engine required. And the clients keep the workload balanced, since each can run independently, no manual intervention required....

[debrouxl]

Agreed, ECMNet is pretty good, even on a single host
I'm sticking to 2.x due to the lack of dependency on a SQL engine.

[xilman]

Originally Posted by debrouxl

Agreed, ECMNet is pretty good, even on a single host
I'm sticking to 2.x due to the lack of dependency on a SQL engine.

Seconded. It just works --- the V2.x anyway. We never managed to get the DB interface working on my Linux box. One day, perhaps, Mark and I will return to the issue but we're each busy with other things ATM.

If you would like assistance setting up ECMNET clients and/or master server and/or slave server just ask. My fee is very reasonable: a few curves contributed to 83.217.167.177:8194


Paul

[PhilF]

Originally Posted by xilman

My fee is very reasonable: a few curves contributed to 83.217.167.177:8194

Now *that* is a reasonable fee!

[EdH]

Thanks!

I will definitely look into this. Currently, I have scripts across several machines that "wait" for sieving assignments from one of them. The assignments are initiated by a script running Aliqueit on a separate machine. However, there is just too much wasted time in between sieving assignments. I had considered adjusting the waiting machines to do some ECM work during the off times. And, then I thought of expanding the ecm.py script that the Aliqueit machine runs. Perhaps ECMNET is what I'm looking for, since you mention being able to add in other jobs between assignments, as well.

I was running curves on the c173 from 4788 continuously on all the machines until my attention span waned after a couple months. (The paint just wouldn't dry...) Then, I wrote all the sieving scripts and am running those, but looking for something else...

Unfortunately, I have a lot of non-math/computer stuff building up to keep me busy, so the more automated, the better.