mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > YAFU

Reply
 
Thread Tools
Old 2010-08-16, 22:13   #397
jasonp
Tribal Bullet
 
jasonp's Avatar
 
Oct 2004

2·3·19·31 Posts
Default

QS postprocessing has not changed in literally years; the LA is much more efficient now but only for matrices much larger than QS would generate.
jasonp is offline   Reply With Quote
Old 2010-08-17, 16:39   #398
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

337110 Posts
Default

Quote:
Originally Posted by bsquared View Post
It took a while to find one, but I now have a repeatable test case which locks up. I'll start looking into what's causing that.
The bug which was causing large SIQS jobs to hang occasionally should now be fixed in version 1.19.2.

The other issues mentioned were
1.) multi-threaded loses cores
2.) batch files don't resume as expected.

I'm not sure what to do about 1.), but I'll keep looking into it. Maybe the thread pool architecture needs to be re-thought...

As for 2.), I guess I need to be educated as to how people expect resuming work in a batchfile to behave. Do you want/expect it to pick up where it left off? Would you be ok with the program modifying the batchfile (deleting rows as they are completed, for example)? Right now, manual modification of the batchfile to remove lines which have already been completed is the correct thing to do.
bsquared is online now   Reply With Quote
Old 2010-08-17, 19:55   #399
em99010pepe
 
em99010pepe's Avatar
 
Sep 2004

2×5×283 Posts
Default

Quote:
Originally Posted by bsquared View Post

As for 2.), I guess I need to be educated as to how people expect resuming work in a batchfile to behave. Do you want/expect it to pick up where it left off? Would you be ok with the program modifying the batchfile (deleting rows as they are completed, for example)?
I expect the client to pick up the work where it left off just by deleting the numbers (row) completed so far.
em99010pepe is offline   Reply With Quote
Old 2010-08-17, 20:03   #400
xilman
Bamboozled!
 
xilman's Avatar
 
"π’‰Ίπ’ŒŒπ’‡·π’†·π’€­"
May 2003
Down not across

3·3,529 Posts
Default

Quote:
Originally Posted by bsquared View Post
As for 2.), I guess I need to be educated as to how people expect resuming work in a batchfile to behave. Do you want/expect it to pick up where it left off? Would you be ok with the program modifying the batchfile (deleting rows as they are completed, for example)? Right now, manual modification of the batchfile to remove lines which have already been completed is the correct thing to do.
I don't know how applicable the analogy may be, but the CWI LA "just works". That is, no-one needs to edit anything and the computation continues from the last checkpoint with no human intervention other than re-starting with the relevant checkpoint.


Paul
xilman is online now   Reply With Quote
Old 2010-08-18, 20:51   #401
em99010pepe
 
em99010pepe's Avatar
 
Sep 2004

2×5×283 Posts
Default

Quote:
Originally Posted by bsquared View Post
The bug which was causing large SIQS jobs to hang occasionally should now be fixed in version 1.19.2.
I'm sorry but could you upload yafu-1.19.2.zip with the 64-bit versions?
em99010pepe is offline   Reply With Quote
Old 2010-08-18, 21:00   #402
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

3,371 Posts
Default

Quote:
Originally Posted by em99010pepe View Post
I'm sorry but could you upload yafu-1.19.2.zip with the 64-bit versions?
Sorry, I just got them from Brian Gladman today, in fact. I'm still having trouble getting my express edition 2010 on win7 to compile 64 bit code.

Anyway, they should now be in the 1.19.2 zip file for download.
bsquared is online now   Reply With Quote
Old 2010-08-19, 21:14   #403
em99010pepe
 
em99010pepe's Avatar
 
Sep 2004

54168 Posts
Default

Thank you. I'm running some Yafu tasks, I'll let you know more about the losing cores issue. We need to understand why it happens and fix it.

Last fiddled with by em99010pepe on 2010-08-19 at 21:15
em99010pepe is offline   Reply With Quote
Old 2010-08-19, 21:29   #404
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

3,371 Posts
Default

Quote:
Originally Posted by em99010pepe View Post
Thank you. I'm running some Yafu tasks, I'll let you know more about the losing cores issue. We need to understand why it happens and fix it.
Agreed. Thanks, that will help.

Here is some interesting data that is maybe related.

On a machine running windows server 2008 and a nehalem based CPU (x5570), the scheduler seems to do a horrible job with yafu and performance really suffers. Looking at the task manager I see that every core is partially utilized, no matter how many threads I specify.

On a machine running windows server 2008 and a core2 based CPU (xeon 5160), the scheduler seems to do a decent job with yafu and performance is fine. Looking at the task manager I see that every core is partially utilized, same as in the nehalem case.

Is the problem that windows doesn't know the difference between a hyperthread and a physical core? Is linux smarter than this, or just lucky in the way it enumerates cores?
bsquared is online now   Reply With Quote
Old 2010-08-19, 21:36   #405
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

3,371 Posts
Default

As Brian stated earlier, we can fix the scheduling issue for nehalems by programmatically assigning an affinity mask for the thread. But nothing informs this decision, so what if we assign the thread to an already loaded core? It doesn't seem like a very graceful fix, but the only other fix seems to be inside the windows scheduler. I'm open to ideas here... can we detect the utlization of a core during runtime in order to inform the affinity mask (i.e. try to hack in a scheduler within yafu)? is there a way to detect in windows whether a core is a hyperthread or not?

em99010pepe: do you see "core loss" behavior if you disable hyperthreading in the BIOS?
bsquared is online now   Reply With Quote
Old 2010-08-19, 21:40   #406
em99010pepe
 
em99010pepe's Avatar
 
Sep 2004

2·5·283 Posts
Default

Quote:
Originally Posted by bsquared View Post

em99010pepe: do you see "core loss" behavior if you disable hyperthreading in the BIOS?
My machines don't have hyperthreading.
em99010pepe is offline   Reply With Quote
Old 2010-08-19, 21:42   #407
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

337110 Posts
Default

Quote:
Originally Posted by em99010pepe View Post
My machines don't have hyperthreading.
oh, sorry. nevermind.
bsquared is online now   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Running YAFU via Aliqueit doesn't find yafu.ini EdH YAFU 8 2018-03-14 17:22
YAFU-1.34 bsquared YAFU 119 2015-11-05 16:24
Yafu bug. storflyt32 YAFU 2 2015-06-29 05:19
yafu-1.33 bsquared YAFU 12 2012-11-08 04:12
yafu-1.32.1 bsquared YAFU 21 2012-09-04 19:44

All times are UTC. The time now is 20:32.

Thu Feb 25 20:32:51 UTC 2021 up 84 days, 16:44, 1 user, load averages: 1.68, 1.60, 1.68

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.