mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Msieve (https://www.mersenneforum.org/forumdisplay.php?f=83)
-   -   Postprocessing error: "too many merge attempts" (https://www.mersenneforum.org/showthread.php?t=11555)

mdettweiler 2009-03-03 06:35

Postprocessing error: "too many merge attempts"
 
1 Attachment(s)
Just today I was using msieve to postprocess a GNFS job on a C101 which, by accident, had been very, very oversieved. During the filtering stage it kept giving me a "too many merge attempts" error; here's the sequence of events of how I encountered this error message and attempted to fix it:

-Sieving had collected almost 7 million relations, compared to about 5.5 million which was required for this job. I started msieve v1.34 (why I was using such an antiquated version is a somewhat long story) and it ran until it gave the "too many merge attempts" error after attempting to improve the matrix a number of times. The log file from this attempt is included in the attached archive as "msieve.log.134".

-I then tried re-running the job with msieve v1.40 beta. This also failed; by my admittedly quite limited understanding of how all this works, it appeared to do a little better in its attempts to improve the matrix, but still ultimately failed. The log file from this attempt is included as "msieve.log.140".

-Remembering a thread a while back that discussed problems that cropped up with humongously oversieved jobs in postprocessing, I tried whacking the # of relations down to about 6.5 million and ran it with v1.40. I couldn't tell whether this improved things much (though my later results indicated that this probably did help at least to some extent), and didn't save a log file from this attempt.

-I cut the # of relations down even more, to 5.75 million, and ran postprocessing with v1.40. This time it worked (somewhat fitfully, it appeared, though again my knowledge of how this stuff works is still somewhat limited and thus I could be dead wrong on this :smile:) and produced factors. The log from this attempt is included as "msieve.log.140.final".

It appears that the problem lies in the sheer amount of oversieving that was done on this number. I'm pretty sure that msieve's behavior in circumstances like this has been reported at least once before, but I figured I'd report this occurrence in case it ends up being helpful to improve the program. :smile:

One other, somewhat unrelated bug(?) that I noticed during the final and successful attempt to run the postprocessing was that despite the fact that I ran msieve with the "-t 2" command line flag to run on both cores of my computer, it only ran single-threaded during the linear algebra stage. I'm not sure if this is a bug, or if this is somehow this was due to the earlier wonkiness due to all the extra relations.

Anyway, just figured I'd post the logs from this somewhat difficult job in case it proves enlightening to anyone. :smile:

Max :smile:

10metreh 2009-03-03 07:22

[quote=mdettweiler;164472]Just today I was using msieve to postprocess a GNFS job on a C101 which, by accident, had been very, very oversieved. During the filtering stage it kept giving me a "too many merge attempts" error; here's the sequence of events of how I encountered this error message and attempted to fix it:

-Sieving had collected almost 7 million relations, compared to about 5.5 million which was required for this job. I started msieve v1.34 (why I was using such an antiquated version is a somewhat long story) and it ran until it gave the "too many merge attempts" error after attempting to improve the matrix a number of times. The log file from this attempt is included in the attached archive as "msieve.log.134".

-I then tried re-running the job with msieve v1.40 beta. This also failed; by my admittedly quite limited understanding of how all this works, it appeared to do a little better in its attempts to improve the matrix, but still ultimately failed. The log file from this attempt is included as "msieve.log.140".

-Remembering a thread a while back that discussed problems that cropped up with humongously oversieved jobs in postprocessing, I tried whacking the # of relations down to about 6.5 million and ran it with v1.40. I couldn't tell whether this improved things much (though my later results indicated that this probably did help at least to some extent), and didn't save a log file from this attempt.

-I cut the # of relations down even more, to 5.75 million, and ran postprocessing with v1.40. This time it worked (somewhat fitfully, it appeared, though again my knowledge of how this stuff works is still somewhat limited and thus I could be dead wrong on this :smile:) and produced factors. The log from this attempt is included as "msieve.log.140.final".

It appears that the problem lies in the sheer amount of oversieving that was done on this number. I'm pretty sure that msieve's behavior in circumstances like this has been reported at least once before, but I figured I'd report this occurrence in case it ends up being helpful to improve the program. :smile:

One other, somewhat unrelated bug(?) that I noticed during the final and successful attempt to run the postprocessing was that despite the fact that I ran msieve with the "-t 2" command line flag to run on both cores of my computer, it only ran single-threaded during the linear algebra stage. I'm not sure if this is a bug, or if this is somehow this was due to the earlier wonkiness due to all the extra relations.

Anyway, just figured I'd post the logs from this somewhat difficult job in case it proves enlightening to anyone. :smile:

Max :smile:[/quote]

[URL="http://www.mersenneforum.org/showthread.php?t=11271"]www.mersenneforum.org/showthread.php?t=11271[/URL]
[URL="http://www.mersenneforum.org/showthread.php?p=152180"]www.mersenneforum.org/showthread.php?p=152180[/URL]

Batalov 2009-03-03 07:35

[quote=mdettweiler;164472]One other, somewhat unrelated bug(?) that I noticed during the final and successful attempt to run the postprocessing was that despite the fact that I ran msieve with the "-t 2" command line flag to run on both cores of my computer, it only ran single-threaded during the linear algebra stage. I'm not sure if this is a bug, or if this is somehow this was due to the earlier wonkiness due to all the extra relations.[/quote]
Well, you know, the help of more than 1 thread can turn into a disservice, that's why it is not a bug, but a feature. It is built-in to do so (under a certain size). It is not related.

You know, in short, BL scaling up is [I]very[/I] sublinear, and the less the job, the less sense -t 2 makes. So it is forced to be -t 1 in the code under a certain size. Even for large (but not very large) jobs, on a more-than-4-cpu cluster, I've learned to live and let live and not use more than -t 4. [SIZE=1](live and let live? Just watched Bond-21... Rubbish. :smile:[/SIZE][SIZE=1])[/SIZE]

jasonp 2009-03-03 13:12

The linear algebra uses one thread for any matrix whose size is smaller than 250k; multithreaded support adds a lot of overhead that would make small jobs take longer.

mdettweiler 2009-03-03 16:53

[quote=10metreh;164476][URL="http://www.mersenneforum.org/showthread.php?t=11271"]www.mersenneforum.org/showthread.php?t=11271[/URL]
[URL="http://www.mersenneforum.org/showthread.php?p=152180"]www.mersenneforum.org/showthread.php?p=152180[/URL][/quote]
Thanks for the links--those do indeed seem to be the same problem I had and they proved to be quite enlightening. :smile:
[quote=Batalov;164477]Well, you know, the help of more than 1 thread can turn into a disservice, that's why it is not a bug, but a feature. It is built-in to do so (under a certain size). It is not related.

You know, in short, BL scaling up is [I]very[/I] sublinear, and the less the job, the less sense -t 2 makes. So it is forced to be -t 1 in the code under a certain size. Even for large (but not very large) jobs, on a more-than-4-cpu cluster, I've learned to live and let live and not use more than -t 4. [SIZE=1](live and let live? Just watched Bond-21... Rubbish. :smile:[/SIZE][SIZE=1])[/SIZE][/quote]
[quote=jasonp;164490]The linear algebra uses one thread for any matrix whose size is smaller than 250k; multithreaded support adds a lot of overhead that would make small jobs take longer.[/quote]
Ah, I get it now. Thanks guys for explaining. :smile:


All times are UTC. The time now is 06:26.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.