mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > GMP-ECM

Reply
 
Thread Tools
Old 2012-09-22, 21:36   #34
WraithX
 
WraithX's Avatar
 
Mar 2006

2×35 Posts
Default

Quote:
Originally Posted by Dubslow View Post
In Linux, the file type is NOT associated with the python executable; rather the shell (bash, in my and most cases) looks at the first line for the so-called "shebang" (I prefer "hashbang") which looks like `#! path-to-exe`; the shell then calls `path-to-exe $0 $@`, which is exactly the same as if I did it myself
Ah, this is good to know. Thanks for that info.

Well, the next thing to try is to:
Set NICE_PATH to different values, I'm not sure which one to put here
and change the Popen line to:
p = subprocess.Popen(args.split(' '), shell=True, **al)

And also, can you add the following two lines right above the Popen line:
Code:
  print('-> args: {0:s}', str(args))
  print('->   al: {0:s}', str(al))
I want to make sure nothing fishy is going on there.
WraithX is offline   Reply With Quote
Old 2012-09-22, 21:52   #35
Dubslow
Basketry That Evening!
 
Dubslow's Avatar
 
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88

3×29×83 Posts
Default

Quote:
Originally Posted by WraithX View Post
Ah, this is good to know. Thanks for that info.
http://en.wikipedia.org/wiki/Shebang_(Unix)
http://docs.python.org/tutorial/inte...python-scripts
Quote:
Originally Posted by WraithX View Post
Well, the next thing to try is to:
Set NICE_PATH to different values, I'm not sure which one to put here
and change the Popen line to:
p = subprocess.Popen(args.split(' '), shell=True, **al)

And also, can you add the following two lines right above the Popen line:
Code:
  print('-> args: {0:s}', str(args))
  print('->   al: {0:s}', str(al))
I want to make sure nothing fishy is going on there.
There's the problem. Should be ./ecm, not ../ecm.
Code:
bill@Gravemind:~/yafu/ecm∰∂ grep "NICE_PATH" ecm.py
  NICE_PATH = '' #'/usr/bin/nice -19'
    pass #al['preexec_fn'] = NICE_PATH
bill@Gravemind:~/yafu/ecm∰∂ grep -B 2 "Popen" ecm.py
  print('-> args: {0:s}', str(args))
  print('->   al: {0:s}', str(al))
  p = subprocess.Popen(args.split(' '), shell=True, **al)
  #p = subprocess.Popen(args.split(' '), **al)
  #p = subprocess.Popen([ex] + args.split(' '), **al)
  #p = subprocess.Popen(cs.split(' '), **al)
bill@Gravemind:~/yafu/ecm∰∂ ./ecm.py -c 7553 -one -threads 4 -out EMS.job 43000000 < num
-> ___________________________________________________________________
-> | Running ecm.py, a Python driver for distributing GMP-ECM work   |
-> | on a single machine.  It is Copyright, 2011, David Cleaver and  |
-> | is a conversion of factmsieve.py that is Copyright, 2010, Brian |
-> | Gladman.   Version 0.08 (Python 2.6 or later) 19th Oct 2011.    |
-> |_________________________________________________________________|
 
-> Number(s) to factor:
-> 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
->=============================================================================
-> Working on number: 96829488818499592481168771836336683023181156945795350980834458372199490598743221067775290195641203125439681639536219726888871822435629511515837059837171813128663335953886175536897367740550240372528813404899458874513057418332695709006061299277468749241875966062032012477732299909160292749026996368849279816035027111164073836173908645011 (335 digits)
-> Found previous job file job3362.txt, will resume work...
-> *** Already completed 0 curves on this number...
-> *** Will run 7553 more curves.
-> Currently working on: job3362.txt
-> Starting 4 instances of GMP-ECM...
-> ./ecm -one -c 1889 43000000 < job3362.txt > job3362_t00.txt
-> args: {0:s}  -one -c 1889 43000000
->   al: {0:s} {'executable': '../ecm', 'stdin': <_io.TextIOWrapper name='job3362.txt' mode='r' encoding='UTF-8'>, 'stdout': <_io.TextIOWrapper name='job3362_t00.txt' mode='w' encoding='UTF-8'>}
Traceback (most recent call last):
  File "./ecm.py", line 1227, in <module>
    start_ecm_threads()
  File "./ecm.py", line 524, in start_ecm_threads
    procs.append(run_exe(ECM, ecm_args2, in_file = ecm_job, out_file = file_name, wait = False))
  File "./ecm.py", line 327, in run_exe
    p = subprocess.Popen(args.split(' '), shell=True, **al)
  File "/usr/lib/python3.2/subprocess.py", line 736, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.2/subprocess.py", line 1330, in _execute_child
    raise child_exception_type(errno_num, err_msg)
OSError: [Errno 13] Permission denied
Code:
ECM_PATH = '.'
#...
ECM = 'ecm'

if sys.platform.startswith('win'):
  EXE_SUFFIX = '.exe'
else:
  EXE_SUFFIX = ''
  NICE_PATH = ''
  ECM = './' + ECM
#...
  ex = (ECM_PATH + exe + EXE_SUFFIX)
  al['executable'] = ex
I don't have a clean copy of the script, so I don't know if this is my fault or not (though it probably is).

Last fiddled with by Dubslow on 2012-09-22 at 21:57
Dubslow is offline   Reply With Quote
Old 2012-09-22, 22:17   #36
WraithX
 
WraithX's Avatar
 
Mar 2006

7468 Posts
Default

Quote:
Originally Posted by Dubslow View Post
There's the problem. Should be ./ecm, not ../ecm.

I don't have a clean copy of the script, so I don't know if this is my fault or not (though it probably is).
Well, you can always download a clean copy from earlier in the thread.

For the path argument, if ecm is in the same folder, I have always used "./" and not ".". This should be all you need to change to get the ecm executable working.

As for nice, I would suggest the following:
NICE_PATH = '/usr/bin/nice -n 19'
WraithX is offline   Reply With Quote
Old 2012-09-30, 17:41   #37
WraithX
 
WraithX's Avatar
 
Mar 2006

48610 Posts
Default

Announcing ecm.py version 0.10, the changes are:
Quote:
- added an extra check to help make sure ecm_path is set correctly
- added an extra verbose level (3) that will print some debug info to try to help diagnose any problems
- changed how the number of digits in an input number is calculated
-- * it is now based on math.log10() instead of counting characters in a string
-- * this is a big speed improvement for numbers with {1's,10's,100's,...} of thousands of digits
- changed time averaging so it no longer averages in the very rare negative time reported by gmp-ecm
Attached Files
File Type: zip ecm-py_v0.10.zip (11.0 KB, 358 views)
WraithX is offline   Reply With Quote
Old 2012-10-04, 03:38   #38
jwaltos
 
jwaltos's Avatar
 
Apr 2012
Gracie on alert.

34·5 Posts
Default ecm.py attachment

Thanks WraithX.
jwaltos is offline   Reply With Quote
Old 2013-05-05, 03:17   #39
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3·372 Posts
Default

Is it possible to convert ecm.py such that it will operate across other machines?

I'm toying with several ideas along this line, but unsure whether to look into the workings of the script myself or not. I'm not well versed in Python and could possibly use this as a way to learn some.

Anyway, as it runs, everything is done within the original terminal. Could the separate threads be run in separate terminals, such as called by gnome-terminal? If so, could these threads be run as ssh connections on remote machines? Or, if run as totally independent processes on the alternate machines, would there be a way of catching a factor, if found on one of the other machines?

Would the data transfer overhead or the variance in thread processing negate any possible gains of creating several threads across differing equipment types?

Additionally, if the above is at least a possibility, would there be a way to scale the work allocated such that faster machines get a bigger allocation?

Thanks for any ideas (and especially for a great script)...

Last fiddled with by EdH on 2013-05-05 at 03:17
EdH is offline   Reply With Quote
Old 2013-05-05, 05:23   #40
schickel
 
schickel's Avatar
 
"Frank <^>"
Dec 2004
CDP Janesville

41128 Posts
Default

Quote:
Originally Posted by EdH View Post
Is it possible to convert ecm.py such that it will operate across other machines?

I'm toying with several ideas along this line, but unsure whether to look into the workings of the script myself or not. I'm not well versed in Python and could possibly use this as a way to learn some.

Anyway, as it runs, everything is done within the original terminal. Could the separate threads be run in separate terminals, such as called by gnome-terminal? If so, could these threads be run as ssh connections on remote machines? Or, if run as totally independent processes on the alternate machines, would there be a way of catching a factor, if found on one of the other machines?

Would the data transfer overhead or the variance in thread processing negate any possible gains of creating several threads across differing equipment types?

Additionally, if the above is at least a possibility, would there be a way to scale the work allocated such that faster machines get a bigger allocation?

Thanks for any ideas (and especially for a great script)...
I would say that if you're going to look into running more than one machine, ECMNET would be the better way to go.

I have my network setup with a master server here on my machine, with two dedicated machines that boot up with a slave server/client setup. That way, even if I have trouble with the network or this machine, the ECM boxes keep on running. The network overhead is miniscule, and you can just load up a bunch of low priority work along with your higher priority work, that way if something factors, there's some fall back work.

If you go with v2.0, the backend for the server is a simple text file, no DB engine required. And the clients keep the workload balanced, since each can run independently, no manual intervention required....
schickel is offline   Reply With Quote
Old 2013-05-05, 09:37   #41
debrouxl
 
debrouxl's Avatar
 
Sep 2009

2·3·163 Posts
Default

Agreed, ECMNet is pretty good, even on a single host
I'm sticking to 2.x due to the lack of dependency on a SQL engine.
debrouxl is offline   Reply With Quote
Old 2013-05-05, 12:38   #42
xilman
Bamboozled!
 
xilman's Avatar
 
"𒉺𒌌𒇷𒆷𒀭"
May 2003
Down not across

24·13·53 Posts
Default

Quote:
Originally Posted by debrouxl View Post
Agreed, ECMNet is pretty good, even on a single host
I'm sticking to 2.x due to the lack of dependency on a SQL engine.
Seconded. It just works --- the V2.x anyway. We never managed to get the DB interface working on my Linux box. One day, perhaps, Mark and I will return to the issue but we're each busy with other things ATM.

If you would like assistance setting up ECMNET clients and/or master server and/or slave server just ask. My fee is very reasonable: a few curves contributed to 83.217.167.177:8194


Paul
xilman is offline   Reply With Quote
Old 2013-05-05, 13:02   #43
PhilF
 
PhilF's Avatar
 
"6800 descendent"
Feb 2005
Colorado

2·11·31 Posts
Default

Quote:
Originally Posted by xilman View Post
My fee is very reasonable: a few curves contributed to 83.217.167.177:8194
Now *that* is a reasonable fee!
PhilF is online now   Reply With Quote
Old 2013-05-05, 13:20   #44
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3×372 Posts
Default

Thanks!

I will definitely look into this. Currently, I have scripts across several machines that "wait" for sieving assignments from one of them. The assignments are initiated by a script running Aliqueit on a separate machine. However, there is just too much wasted time in between sieving assignments. I had considered adjusting the waiting machines to do some ECM work during the off times. And, then I thought of expanding the ecm.py script that the Aliqueit machine runs. Perhaps ECMNET is what I'm looking for, since you mention being able to add in other jobs between assignments, as well.

I was running curves on the c173 from 4788 continuously on all the machines until my attention span waned after a couple months. (The paint just wouldn't dry...) Then, I wrote all the sieving scripts and am running those, but looking for something else...

Unfortunately, I have a lot of non-math/computer stuff building up to keep me busy, so the more automated, the better.
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Python Coding Help? kelzo Programming 3 2016-11-27 05:16
PHP vs. Python vs. C (all with GMP) daxmick Programming 2 2014-02-10 01:45
Python... Xyzzy Programming 20 2009-09-08 15:51
using libecm from python yqiang GMP-ECM 2 2007-04-22 00:14
Help w/ python. a216vcti Programming 7 2005-10-30 00:37

All times are UTC. The time now is 21:10.


Tue Nov 30 21:10:40 UTC 2021 up 130 days, 15:39, 0 users, load averages: 2.04, 1.90, 1.66

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.