Using Several Instances of Aliqueit for a large gnfs job

EdH · 2011-12-12, 22:34

For those who like the automation of Aliqueit and would like to work on larger numbers, but not wait quite so long for completion, I have been playing with a way to use several machines on the same gnfs job via Aliqueit. I'm not sure of the results, though and am asking for some comments on the following:

I recently ran a c135 on a 64-bit linux machine, with help from several 32-bit linux machines. What I did was to start the 64-bit machine and once the poly selection was done and relations were being added, I copied the ggnfs_###... directory and the ###.elf file to the other machines. I then got rid of the spairs files on those other machines. (I did some other things due to the dual thread work by the 64-bit machine, but those items will be addressed if and when I create a "how-to" about this effort.)

Next, I waited for the first completion of q range on the 64-bit machine to see what percentage was completed. I used this info to calculate how high the q should get if left alone to 100%. I then used this value to adjust each of the other machines to work above this value and separate from each other, by editing the test.job.T0 and test.job.resume files. Then I started Aliqueit with the -e switch on each 32-bit machine.

As each machine completed a cycle and created test.dat, I removed it to a holding area where I concatenated all of them into spairs.add. I then placed spairs.add into the 64-bit ggnfs_##... directory.

All seemed to work well, but the 64-bit machine needed to go back out several times for 1000000 more relations. It ended up needing to get to 113.9% of the original estimate. Is there a way I can tell if this is due to the added relations being no good, or if this is just due to an underestimate in the 64-bit machine's factmsieve.py script?

I realize that I can just run gnfs-lasieve with the same results, but I'm exploring whether this might be an easier way for those that may not want to go the gnfs-lasieve route...

Thanks for any comments...

bsquared · 2011-12-12, 22:52

If the relations were "no good" then you would have either had skads of error message during filtering or perhaps much higher than usual duplicate relations (30% or more, say). If you saw neither of these situations, then the relations were probably fine and the 113% is just due to do a low initial guess or something.

As long as you are doing things manually on linux anyway, that doesn't sound any easier than just working directly with gnfs-lasieve*. Although I suppose it avoids figuring out starting Q and min rels figures.

Batalov · 2011-12-12, 23:36

Quote:

Originally Posted by EdH

... or if this is just due to an underestimate in the 64-bit machine's factmsieve.py script?

"That's a bingo. Is that the way you say it?"

EdH · 2011-12-13, 03:12

Thanks Guys,

I saw no error messages and these are the duplicate removals:

Code:

Mon Dec 12 06:30:50 2011  found 3884965 hash collisions in 23001181 relations
Mon Dec 12 06:31:29 2011  added 36 free relations
Mon Dec 12 06:31:29 2011  commencing duplicate removal, pass 2
Mon Dec 12 06:33:09 2011  found 3653999 duplicates and 19347218 unique

I guess that's about 16% duplication?

I'm going to evaluate the ease of both methods and "maybe" write some steps to take to add machines. That way I'll know where to remind myself how and maybe someone else can find it useful.

Batalov · 2011-12-13, 03:33

Seriously speaking, there's also a possibility that you evaluated the necessary Q-range on the admission that the relation yield is a constant. But it isn't, and it is not easy to guesstimate it. Generally, it goes down as Q goes up, but the question is - how fast.

One way (frequently used before launching large projects) is a dense set of spot checking runs (with many starting Qs and a span of 2000 or a 1000), followed by a spline (or better yet with normalization the by number of reported special_q's), and a guesstimate from experience with similar runs of what redundancy is going to be.

EdH · 2011-12-13, 04:58

Quote:

Originally Posted by Batalov

Seriously speaking, there's also a possibility that you evaluated the necessary Q-range on the admission that the relation yield is a constant. But it isn't, and it is not easy to guesstimate it. Generally, it goes down as Q goes up, but the question is - how fast.

One way (frequently used before launching large projects) is a dense set of spot checking runs (with many starting Qs and a span of 2000 or a 1000), followed by a spline (or better yet with normalization the by number of reported special_q's), and a guesstimate from experience with similar runs of what redundancy is going to be.

I actually follow all this, but haven't the experience to make use of it. I therefore made use of the following logic:

For example, let's say q is going up by 1M each time and relations are growing at a rate of 5% for each 1M. And, it started at 20M. 100% (in a perfect world) would place the top at 40M. So, let's start machine 2 at 40M. I'm hoping that the relations turned up by machine 2 will offset the 40M top of machine 1 downward more so than the diminishing relations will affect the overall count. The trickier part is figuring out the starting points for machines 3, 4, 5, etc. I don't want any overlap there either, but the further away from the machine 1 range, the less return.

bchaffin · 2011-12-13, 18:58

Note that factmseive.py already has some support for running multiple threads on each of multiple machines. It doesn't hook directly into aliqueit, but as bsquared said if you're doing a bunch of manual work anyway it may be simpler just to throw the number to factmsieve and then pass the answer back to aliqueit.

2011-12-12, 22:34	#1
EdH "Ed Hall" Dec 2009 Adirondack Mtns 5×727 Posts	Using Several Instances of Aliqueit for a large gnfs job For those who like the automation of Aliqueit and would like to work on larger numbers, but not wait quite so long for completion, I have been playing with a way to use several machines on the same gnfs job via Aliqueit. I'm not sure of the results, though and am asking for some comments on the following: I recently ran a c135 on a 64-bit linux machine, with help from several 32-bit linux machines. What I did was to start the 64-bit machine and once the poly selection was done and relations were being added, I copied the ggnfs_###... directory and the ###.elf file to the other machines. I then got rid of the spairs files on those other machines. (I did some other things due to the dual thread work by the 64-bit machine, but those items will be addressed if and when I create a "how-to" about this effort.) Next, I waited for the first completion of q range on the 64-bit machine to see what percentage was completed. I used this info to calculate how high the q should get if left alone to 100%. I then used this value to adjust each of the other machines to work above this value and separate from each other, by editing the test.job.T0 and test.job.resume files. Then I started Aliqueit with the -e switch on each 32-bit machine. As each machine completed a cycle and created test.dat, I removed it to a holding area where I concatenated all of them into spairs.add. I then placed spairs.add into the 64-bit ggnfs_##... directory. All seemed to work well, but the 64-bit machine needed to go back out several times for 1000000 more relations. It ended up needing to get to 113.9% of the original estimate. Is there a way I can tell if this is due to the added relations being no good, or if this is just due to an underestimate in the 64-bit machine's factmsieve.py script? I realize that I can just run gnfs-lasieve with the same results, but I'm exploring whether this might be an easier way for those that may not want to go the gnfs-lasieve route... Thanks for any comments...

2011-12-13, 03:12	#4
EdH "Ed Hall" Dec 2009 Adirondack Mtns 7063₈ Posts	Thanks Guys, I saw no error messages and these are the duplicate removals: Code: Mon Dec 12 06:30:50 2011 found 3884965 hash collisions in 23001181 relations Mon Dec 12 06:31:29 2011 added 36 free relations Mon Dec 12 06:31:29 2011 commencing duplicate removal, pass 2 Mon Dec 12 06:33:09 2011 found 3653999 duplicates and 19347218 unique I guess that's about 16% duplication? I'm going to evaluate the ease of both methods and "maybe" write some steps to take to add machines. That way I'll know where to remind myself how and maybe someone else can find it useful.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Aliqueit.exe discussion	bsquared	Aliquot Sequences	591	2020-09-09 16:51
c5 instances are now available	GP2	Cloud Computing	19	2018-02-12 16:23
Resuming aliqueit	johnadam74	Aliquot Sequences	4	2016-03-28 12:32
Advice for large GNFS jobs?	WraithX	Factoring	59	2013-07-30 01:13
2 instances	brandonriffel	Software	3	2007-02-15 16:15

2011-12-12, 22:52	#2
bsquared "Ben" Feb 2007 6455₈ Posts	If the relations were "no good" then you would have either had skads of error message during filtering or perhaps much higher than usual duplicate relations (30% or more, say). If you saw neither of these situations, then the relations were probably fine and the 113% is just due to do a low initial guess or something. As long as you are doing things manually on linux anyway, that doesn't sound any easier than just working directly with gnfs-lasieve*. Although I suppose it avoids figuring out starting Q and min rels figures.

2011-12-13, 03:33	#5
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 10010001110110₂ Posts	Seriously speaking, there's also a possibility that you evaluated the necessary Q-range on the admission that the relation yield is a constant. But it isn't, and it is not easy to guesstimate it. Generally, it goes down as Q goes up, but the question is - how fast. One way (frequently used before launching large projects) is a dense set of spot checking runs (with many starting Qs and a span of 2000 or a 1000), followed by a spline (or better yet with normalization the by number of reported special_q's), and a guesstimate from experience with similar runs of what redundancy is going to be.

2011-12-13, 18:58	#7
bchaffin Sep 2010 Portland, OR 2²×3×31 Posts	Note that factmseive.py already has some support for running multiple threads on each of multiple machines. It doesn't hook directly into aliqueit, but as bsquared said if you're doing a bunch of manual work anyway it may be simpler just to throw the number to factmsieve and then pass the answer back to aliqueit.