mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Aliquot Sequences (https://www.mersenneforum.org/forumdisplay.php?f=90)
-   -   Using Several Instances of Aliqueit for a large gnfs job (https://www.mersenneforum.org/showthread.php?t=16332)

EdH 2011-12-12 22:34

Using Several Instances of Aliqueit for a large gnfs job
 
For those who like the automation of Aliqueit and would like to work on larger numbers, but not wait quite so long for completion, I have been playing with a way to use several machines on the same gnfs job via Aliqueit. I'm not sure of the results, though and am asking for some comments on the following:

I recently ran a c135 on a 64-bit linux machine, with help from several 32-bit linux machines. What I did was to start the 64-bit machine and once the poly selection was done and relations were being added, I copied the ggnfs_###... directory and the ###.elf file to the other machines. I then got rid of the spairs files on those other machines. (I did some other things due to the dual thread work by the 64-bit machine, but those items will be addressed if and when I create a "how-to" about this effort.)

Next, I waited for the first completion of q range on the 64-bit machine to see what percentage was completed. I used this info to calculate how high the q should get if left alone to 100%. I then used this value to adjust each of the other machines to work above this value and separate from each other, by editing the test.job.T0 and test.job.resume files. Then I started Aliqueit with the -e switch on each 32-bit machine.

As each machine completed a cycle and created test.dat, I removed it to a holding area where I concatenated all of them into spairs.add. I then placed spairs.add into the 64-bit ggnfs_##... directory.

All seemed to work well, but the 64-bit machine needed to go back out several times for 1000000 more relations. It ended up needing to get to 113.9% of the original estimate. Is there a way I can tell if this is due to the added relations being no good, or if this is just due to an underestimate in the 64-bit machine's factmsieve.py script?

I realize that I can just run gnfs-lasieve with the same results, but I'm exploring whether this might be an easier way for those that may not want to go the gnfs-lasieve route...

Thanks for any comments...

bsquared 2011-12-12 22:52

If the relations were "no good" then you would have either had skads of error message during filtering or perhaps much higher than usual duplicate relations (30% or more, say). If you saw neither of these situations, then the relations were probably fine and the 113% is just due to do a low initial guess or something.

As long as you are doing things manually on linux anyway, that doesn't sound any easier than just working directly with gnfs-lasieve*. Although I suppose it avoids figuring out starting Q and min rels figures.

Batalov 2011-12-12 23:36

[QUOTE=EdH;281994]... or if this is just due to an underestimate in the 64-bit machine's factmsieve.py script?
[/QUOTE]
"That's a bingo. Is that the way you say it?"

EdH 2011-12-13 03:12

Thanks Guys,

I saw no error messages and these are the duplicate removals:
[code]
Mon Dec 12 06:30:50 2011 found 3884965 hash collisions in 23001181 relations
Mon Dec 12 06:31:29 2011 added 36 free relations
Mon Dec 12 06:31:29 2011 commencing duplicate removal, pass 2
Mon Dec 12 06:33:09 2011 found 3653999 duplicates and 19347218 unique
[/code]I guess that's about 16% duplication?

I'm going to evaluate the ease of both methods and "maybe" write some steps to take to add machines. That way I'll know where to remind myself how and maybe someone else can find it useful.

Batalov 2011-12-13 03:33

Seriously speaking, there's also a possibility that you evaluated the necessary Q-range on the admission that the relation yield is a constant. But it[I] isn't[/I], and it is not easy to guesstimate it. Generally, it goes down as Q goes up, but the question is - how fast.

One way (frequently used before launching large projects) is a dense set of spot checking runs (with many starting Qs and a span of 2000 or a 1000), followed by a spline (or better yet with normalization the by number of reported special_q's), and a guesstimate from experience with similar runs of what redundancy is going to be.

EdH 2011-12-13 04:58

[QUOTE=Batalov;282013]Seriously speaking, there's also a possibility that you evaluated the necessary Q-range on the admission that the relation yield is a constant. But it[I] isn't[/I], and it is not easy to guesstimate it. Generally, it goes down as Q goes up, but the question is - how fast.

One way (frequently used before launching large projects) is a dense set of spot checking runs (with many starting Qs and a span of 2000 or a 1000), followed by a spline (or better yet with normalization the by number of reported special_q's), and a guesstimate from experience with similar runs of what redundancy is going to be.[/QUOTE]
I actually follow all this, but haven't the experience to make use of it. I therefore made use of the following logic:

For example, let's say q is going up by 1M each time and relations are growing at a rate of 5% for each 1M. And, it started at 20M. 100% (in a perfect world) would place the top at 40M. So, let's start machine 2 at 40M. I'm hoping that the relations turned up by machine 2 will offset the 40M top of machine 1 downward more so than the diminishing relations will affect the overall count. The trickier part is figuring out the starting points for machines 3, 4, 5, etc. I don't want any overlap there either, but the further away from the machine 1 range, the less return.

bchaffin 2011-12-13 18:58

Note that factmseive.py already has some support for running multiple threads on each of multiple machines. It doesn't hook directly into aliqueit, but as bsquared said if you're doing a bunch of manual work anyway it may be simpler just to throw the number to factmsieve and then pass the answer back to aliqueit.


All times are UTC. The time now is 03:50.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.