Yafu

[bsquared]

Originally Posted by bsquared

Here is some interesting data that is maybe related.
...

msieve has similar problems.

nehalem based cpu:
222 seconds to factor a c76

core2 based cpu:
134 seconds to factor the same number.

The nehalem *should* be faster!

[em99010pepe]

Here on a core i5 and on a Q6600 CPU gets to 99% when sieving on all cores (4) and when doing square root only uses one core (25%). Normal behave.
One question:

1) Will Yafu use all cores on Lanczos iteration?

[em99010pepe]

Originally Posted by bsquared

msieve has similar problems.

nehalem based cpu:
222 seconds to factor a c76

core2 based cpu:
134 seconds to factor the same number.

The nehalem *should* be faster!

For what version of Yafu did you do that bench?

[bsquared]

Originally Posted by em99010pepe

For what version of Yafu did you do that bench?

That was actually msieve version 1.46. I was just trying to show that the slowdown-due-to-inefficient-scheduling-issue isn't a YAFU specific issue.

[jasonp]

Originally Posted by bsquared

is there a way to detect in windows whether a core is a hyperthread or not?

My understanding is that there is no official way to determine whether a core is 'virtual' or 'physical', i.e. is hyperthreaded or not. There are MSDN articles where people claim otherwise, but all you have to work with is a bitfield that maps to the total number of CPUs (virtual+physical) available.

[em99010pepe]

Originally Posted by em99010pepe

Thank you. I'm running some Yafu tasks, I'll let you know more about the losing cores issue. We need to understand why it happens and fix it.

I'm testing Yafu on a Win XP 64 bit version and this time 2 out or 4 cores were lost during a Yafu task when this is a dedicated machine for crunching and the client was running at normal priority. I had this "core loss" behave issue on a Win 7 64-but machine, I really don't know what's going on!

[rekcahx]

Hi. I sometimes (frequency about 1/100) yafu 1.19-2 crashes.

Here is backtrace:

*** glibc detected *** ./y: double free or corruption (!prev): 0x00000000016c6090 ***
 ======= Backtrace: =========
/lib/libc.so.6(+0x71b16)[0x7f2c16b9bb16]
/lib/libc.so.6(cfree+0x6c)[0x7f2c16ba088c]
./y[0x44cff3]
./y[0x44fc3f]
./y[0x466244]
./y[0x4679de]
./y[0x461873]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f2c16b48c4d]
./y[0x401c69]
======= Memory map: ========
00400000-00474000 r-xp 00000000 fe:01 87892192                           /home/rekcahx/yafu119/auto/y
00674000-00675000 rw-p 00074000 fe:01 87892192                           /home/rekcahx/yafu119/auto/y
00675000-00676000 rw-p 00000000 00:00 0 
0147a000-01cb0000 rw-p 00000000 00:00 0                                  [heap]
7f2c08000000-7f2c08021000 rw-p 00000000 00:00 0 
7f2c08021000-7f2c0c000000 ---p 00000000 00:00 0 
7f2c0eff1000-7f2c0f007000 r-xp 00000000 fe:01 32776200                   /lib/libgcc_s.so.1
7f2c0f007000-7f2c0f206000 ---p 00016000 fe:01 32776200                   /lib/libgcc_s.so.1
7f2c0f206000-7f2c0f207000 rw-p 00015000 fe:01 32776200                   /lib/libgcc_s.so.1
7f2c0f231000-7f2c1527b000 rw-p 00000000 00:00 0 
7f2c1527d000-7f2c1527e000 ---p 00000000 00:00 0 
7f2c1527e000-7f2c15a7e000 rw-p 00000000 00:00 0 
7f2c15a7e000-7f2c15a7f000 ---p 00000000 00:00 0 
7f2c15a7f000-7f2c1627f000 rw-p 00000000 00:00 0 
7f2c1627f000-7f2c16280000 ---p 00000000 00:00 0 
7f2c16280000-7f2c16a80000 rw-p 00000000 00:00 0 
7f2c16b2a000-7f2c16c82000 r-xp 00000000 fe:01 32778260                   /lib/libc-2.11.1.so
7f2c16c82000-7f2c16e81000 ---p 00158000 fe:01 32778260                   /lib/libc-2.11.1.so
7f2c16e81000-7f2c16e85000 r--p 00157000 fe:01 32778260                   /lib/libc-2.11.1.so
7f2c16e85000-7f2c16e86000 rw-p 0015b000 fe:01 32778260                   /lib/libc-2.11.1.so
7f2c16e86000-7f2c16e8b000 rw-p 00000000 00:00 0 
7f2c16e8b000-7f2c16ea2000 r-xp 00000000 fe:01 32778259                   /lib/libpthread-2.11.1.so
7f2c16ea2000-7f2c170a1000 ---p 00017000 fe:01 32778259                   /lib/libpthread-2.11.1.so
7f2c170a1000-7f2c170a2000 r--p 00016000 fe:01 32778259                   /lib/libpthread-2.11.1.so
7f2c170a2000-7f2c170a3000 rw-p 00017000 fe:01 32778259                   /lib/libpthread-2.11.1.so
7f2c170a3000-7f2c170a7000 rw-p 00000000 00:00 0 
7f2c170a7000-7f2c17127000 r-xp 00000000 fe:01 32776401                   /lib/libm-2.11.1.so
7f2c17127000-7f2c17327000 ---p 00080000 fe:01 32776401                   /lib/libm-2.11.1.so
7f2c17327000-7f2c17328000 r--p 00080000 fe:01 32776401                   /lib/libm-2.11.1.so
7f2c17328000-7f2c17329000 rw-p 00081000 fe:01 32776401                   /lib/libm-2.11.1.so
7f2c17329000-7f2c17347000 r-xp 00000000 fe:01 32776398                   /lib/ld-2.11.1.so
7f2c1746d000-7f2c17507000 rw-p 00000000 00:00 0 
7f2c17516000-7f2c1751a000 rw-p 00000000 00:00 0 
7f2c17542000-7f2c17546000 rw-p 00000000 00:00 0 
7f2c17546000-7f2c17547000 r--p 0001d000 fe:01 32776398                   /lib/ld-2.11.1.so
7f2c17547000-7f2c17548000 rw-p 0001e000 fe:01 32776398                   /lib/ld-2.11.1.so
7f2c17548000-7f2c17549000 rw-p 00000000 00:00 0 
7fff94880000-7fff94895000 rw-p 00000000 00:00 0                          [stack]
7fff94983000-7fff94984000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

64 bit linux, self compiled binary without any modifications to source. All observed crashes are from ./y "siqs(some-number)".

In this case, factored number was
140738203611255550057741915720905198497801794190307899963

[ldesnogu]

I built with both gcc 4.1.2 and gcc 4.4.0 under CentOS 5.5 64-bit. valgrind didn't find anything with 4.1.2 but this with 4.4:

==23657== Conditional jump or move depends on uninitialised value(s)
==23657==    at 0x4A082E0: strcpy (mc_replace_strmem.c:303)
==23657==    by 0x464331: tokenize (calc.c:373)
==23657==    by 0x464880: calc (calc.c:457)
==23657==    by 0x45E7E2: main (driver.c:655)

The offending line 373 is:

            strcpy(tmp,&ch);

ch being declared as a char that's indeed broken
I guess something like this would work:

tmp[0] = ch; tmp[1] = '\0';

OTOH valgrind didn't find anything that could explain rekcahx's issue.

It also looks like fp_montgomery_setup could be broken:

tfm/fp_montgomery_setup.c:46: warning: left shift count >= width of type

Hope this helps.

[bsquared]

Originally Posted by ldesnogu

Hope this helps.

Thanks!

The left shift count warning has always been there since I first imported TFM. While obviously concerning, it doesn't appear to ever happen in practice (1 is left shifted by a variable quantity which *could* be equal to 64, but in practice never is).

I'll fix the issue with the strcpy, but agree that it is unlikely to be the cause of the problem.

@ rekcahx
I was unable to reproduce the problem with yafu built with either gcc-4.1.2 or gcc-3.2.3, but this is not terribly surprising since bugs lately appear to be edge conditions that only arise in certain states of the RNG. Do you still have the log file for that crash? If it printed any random seed info, post it here. In the meantime I'll start factoring random numbers of similar size and see if I get a lucky test case.

[ldesnogu]

Originally Posted by bsquared

The left shift count warning has always been there since I first imported TFM. While obviously concerning, it doesn't appear to ever happen in practice (1 is left shifted by a variable quantity which *could* be equal to 64, but in practice never is).

Unless I missed something the shift isn't variable:

*rho = (((fp_word) 1 << ((fp_word) DIGIT_BIT)) - ((fp_word)x));

[bsquared]

Originally Posted by ldesnogu

Unless I missed something the shift isn't variable:

*rho = (((fp_word) 1 << ((fp_word) DIGIT_BIT)) - ((fp_word)x));

No, you didn't miss anything. Hmm. Well, my explanation was what I thought was happening up until now - I didn't look at the parens close enough. I'll look at this more, thanks!