Distributed polynomial selection.

[chris2be8]

Hello,

I've updated factMsieve.pl to support distributed polynomial selection. Use the attached file renamed to remove .txt from the end. Customise paths, number of CPUs, etc. If you are set up to use more than 1 system I recommend setting USE_LOCKDIR to 1, then you can always add helpers if a job is taking too long.

Set up a shared directory that all systems can update, put the NAME.n file in it and run factMsieve.pl in it on each system.

On the master run as follows:
factMsieve.pl NAME.n 1 4

On the helper systems:
factMsieve.pl NAME.n 2 4

The "2" should be 3 for the third helper, etc.

The last number is the total number of systems. It's actually cosmetic, you don't have to make it the same on all systems.

The only vital point is that each system gets a different host number. If you set USE_LOCKDIR to 1 the master doesn't have to know there are any other systems helping it.

It is safe to start the helpers before the master. Eg if the master is doing the linear algebra on the previous number.

It will work fine if the systems have different speeds. Eg 6 core master and 2 core helper.

It also copes with "matrix needs more columns than rows" and sieves a few more records. This isn't tested very well, I can't make it happen when I want it to.

Chris K

PS. The code in it to automatically start helper systems isn't tested. It should work but I can't guarantee it.

[chris2be8]

Hello,

Here's my latest version. Enhancements are:

Allow users to specify the range of leading coefficients to search during polynomial selection. You can either put it into factMsieve.pl or into NAME.n (lcmin: lcstep: or lcmax:).

You can put logs etc into another dir from the data files. Eg a local dir on helpers to reduce network load.

You can do line sieving on one of the helper systems by setting DOCLASSICAL=2 (only on 1 helper though!) You get best overall throughput if you do it on a single core system that can't run the 64 bit lattice sieve (and that's probably the only case where it's a clear benefit).

I have several items on my to-do list. But I'm short of time to work on them.

Chris K

[debrouxl]

Great

[chris2be8]

A small fix, check that the paths to binaries etc are correct before polynomial selection, so you don't come back to a job and find it halted after polynomial selection ended because the path to the lattice siever is wrong. This version checks the paths before checking how many parms it's passed, so running it without parms will tell you if it can find its programs etc.

Another minor enhancement, it no longer uses cat (really just tidying up).

Chris K

[chris2be8]

A few more fixes:
It always checks it can find def-par.txt, even if $CHECK_BINARIES=0.
It does not save the factor base when sieving below alim/rlim.
It will save the factor base (name.afb.[01]) into logdir if you specify logdir. Which is more network friendly.
Fix a few bugs running with only 1 CPU.
Optionally check the factors are prime with ECPP. Linux only though.

While writing that I found that the factor base is always called the name of the job file with .afb.[01] on the end. The -b option is just another way to say what the job file is called. So putting the factor base into logdir requires the job file to be generated there too.

Chris K

PS. Does anyone know what the -M, -P, -i and -t flags tell gnfs-lasieve4e.c to do? I don't know C well enough to work it out from the code.

[chris2be8]

Hello,

The latest version which should be 2-3% faster on the helper systems because:
Helpers do special-Q ranges 1/10 as large as the master. So they don't do as much wasted sieving after the master has built the matrix.

Helper threads start a new range as soon as they finish a range, instead of waiting for all threads to end their range before starting a new set of ranges. So all CPUs are busy all the time.

Classical sieving will be done on 1 CPU wile the others do lattice sieving. It's still not much use but at least won't slow sieving down (the previous version was slower on a multi-CPU system).

I'm looking for a way to multi-thread trial post processing and sieving. But it's not easy to ensure it'll be an improvement.

Happy factoring and Happy Christmas!

Chris K

[pinhodecarlos]

Using your script under windows gnfs-lasieve siever runs at normal priority. How do I set it to idle priority?

# Run the binaries at low priority?
my $NICE="what to add here to run binaries at low priority";
#my $NICE="nice -n 19 ";

[schickel]

Originally Posted by pinhodecarlos

Using your script under windows gnfs-lasieve siever runs at normal priority. How do I set it to idle priority?

# Run the binaries at low priority?
my $NICE="what to add here to run binaries at low priority";
#my $NICE="nice -n 19 ";

What version? It'll be START with some added options. On XP (or 7, just checked), it would be:

start /b /low /wait <command here> <params here>"

with the LOW indicating priority, WAIT meaning wait for the command to finish, and B meaning run in the same window.

[pinhodecarlos]

I already tried that under Vista without success.

[Andi47]

Originally Posted by pinhodecarlos

I already tried that under Vista without success.

After opening the command window, go to the task manager and set the cmd window to lowest priority. When you start ggnfs or whatever from this cmd window afterwards, this process (and each of its child processes) will inherit the idle priority from the cmd window.

[schickel]

Originally Posted by Andi47

After opening the command window, go to the task manager and set the cmd window to lowest priority. When you start ggnfs or whatever from this cmd window afterwards, this process (and each of its child processes) will inherit the idle priority from the cmd window.

That's one way. The other way is to turn the priority down if you're using the PC (that's what I do).

Originally Posted by pinhodecarlos

I already tried that under Vista without success.

Yeah, I just did a little experimenting and I think the problem is all the parameters that get fed to the siever.

I found one way to do it.....but the cmd window hangs around. Put the siever command line in a .bat file and do it this way: