YAFU 2.0

[kruoli]

Since I am on Zen 3, I only built with AVX2 and BMI2. It also stated "using AVX2 enabled 32k sieve core" etc. I double checked and rebuilt with the parameters you suggested (ommiting the last one), and got a crossover of 93.0 bits this time.

When editing some files in vim, it occured to me that a lot of lines (e.g. the Makefile) end with ^M, but it seems to be handled alright.

The next error is in ecm.c, where it states:

factor/gmp-ecm/ecm.c:825:17: error: non-void function 'ecm_do_one_curve' should return a value [-Wreturn-type]
                return;

In my local copy, I assumed return NULL; here. Now it built! Interestingly, it gets detected as Built with GCC 4.
Unfortunately, I get a segmentation fault when I am in the first SIQS of tuning. I rebuilt with -g and got (gdb and bt):

Program received signal SIGSEGV, Segmentation fault.
0x0000000000357844 in __gmpn_divrem_1 ()
(gdb) bt
#0  0x0000000000357844 in __gmpn_divrem_1 ()
#1  0x0000000000001832 in ?? ()
#2  0x0000000000001832 in ?? ()
#3  0x0000000002f207d0 in ?? ()
#4  0x0000000000001831 in ?? ()
#5  0x00000000003555bf in __gmpz_tdiv_q_ui ()
#6  0x000000000029ab1e in tdiv_LP_avx2 (report_num=<optimized out>, parity=<optimized out>, bnum=<optimized out>, sconf=<optimized out>, dconf=<optimized out>) at factor/qs/tdiv_large.c:604
#7  0x01cf01ce01cd01cc in ?? ()
#8  0x01d301d201d101d0 in ?? ()
#9  0x01d701d601d501d4 in ?? ()
#10 0x01db01da01d901d8 in ?? ()
#11 0x01df01de01dd01dc in ?? ()
#12 0x01e301e201e101e0 in ?? ()
#13 0x01e701e601e501e4 in ?? ()
#14 0x01eb01ea01e901e8 in ?? ()
#15 0x01ef01ee01ed01ec in ?? ()
#16 0x0000000200000002 in ?? ()
#17 0x0000000000355506 in __gmpz_tdiv_q_2exp ()
#18 0x000000000028f7db in process_poly (vptr=<optimized out>, vptr@entry=0x2c17c90) at factor/qs/SIQS.c:1108
#19 0x000000000028f00e in SIQS (fobj=fobj@entry=0x25a2ef0) at factor/qs/SIQS.c:828
#20 0x0000000000276d21 in factor_tune (inobj=0x25a2ef0) at factor/tune.c:140
#21 0x00000000002839b4 in feval (funcnum=funcnum@entry=72, nargs=<optimized out>, nargs@entry=0, metadata=<optimized out>, metadata@entry=0x7fffffffce60) at top/cmdParser/calc.c:2869
#22 0x000000000028194a in calc (in=in@entry=0x7fffffffccd0, metadata=<optimized out>, metadata@entry=0x7fffffffce60) at top/cmdParser/calc.c:1946
#23 0x0000000000280822 in calc_with_assignment (in=in@entry=0x25aa5e0, metadata=<optimized out>, metadata@entry=0x7fffffffce60, force_quiet=force_quiet@entry=0) at top/cmdParser/calc.c:1526
#24 0x000000000028069e in process_expression (input_exp=<optimized out>, metadata=<optimized out>, metadata@entry=0x7fffffffce60, force_quiet=<optimized out>, no_convert_result=<optimized out>, no_convert_result@entry=0)
    at top/cmdParser/calc.c:1472
#25 0x000000000027404d in main (argc=<optimized out>, argv=<optimized out>) at top/driver.c:390

Weird, it comes from GMP? I ran make check there and all went fine.

[kruoli]

The increase in SIQS time is giving me a headache:
I found old logs and picked 188859359342907532180448190281668092516979292823033816845808258496473426383608039973872193 (c90) as my example. The log stated 27.8 s in SIQS phase, I retried with the options as explained and the current GIT version and got 60.8 s. I retried with all GIT versions that identified as v2.05 and all segmentation faulted (see below why it might).

Conflicting processes can be ruled out by restarts and htop run as root.

I cannot say for sure that I had BMI2 and also AVX2 enabled in my original build. But it would seem strange that this increased my performance. There is one single thing that changed, my Debian version, with that the GCC version. The log file that I have found was executed on Debian 11, so this alone is not the problem. But GCC 10 might be (as hinted to before in this thread). For example, my v2.05 versions built with GCC 10 were crashing in SIQS, my old GCC 8 version never crashed (in SIQS).

[bsquared]

Originally Posted by kruoli

The increase in SIQS time is giving me a headache:
I found old logs and picked 188859359342907532180448190281668092516979292823033816845808258496473426383608039973872193 (c90) as my example. The log stated 27.8 s in SIQS phase, I retried with the options as explained and the current GIT version and got 60.8 s. I retried with all GIT versions that identified as v2.05 and all segmentation faulted (see below why it might).

Conflicting processes can be ruled out by restarts and htop run as root.

I cannot say for sure that I had BMI2 and also AVX2 enabled in my original build. But it would seem strange that this increased my performance. There is one single thing that changed, my Debian version, with that the GCC version. The log file that I have found was executed on Debian 11, so this alone is not the problem. But GCC 10 might be (as hinted to before in this thread). For example, my v2.05 versions built with GCC 10 were crashing in SIQS, my old GCC 8 version never crashed (in SIQS).

With gcc-11.1.0 on a Xeon 6254 for the sieving portion of siqs with 16 threads I get:
54 seconds with SSE2 (this is default)
49 seconds with SSE4.1
40 seconds with AVX/BMI2
29 seconds with AVX-512

linear algebra then adds about 8 seconds to all of these. The various instruction set flags do make a difference. With your timing you must be using a lot of threads, so LA might take an appreciable amount of time compared to sieving. It's a stretch, but are you comparing sieve time to total time between these measurements?

This is basically the same as with version 2.05, as far as I can see. I can't see how it would be different on a Zen 3, but crazier things have happened in yafu before. I don't have an AMD to test on, so thanks for bearing with me.

This weekend I'll install clang and work on the build.

[kruoli]

Thank you!

There was another difference I oversaw: I had -O3 in my recent builds. I reverted this to -O2 and now the SIQS times are fine again!

Elapsed time: 23.8909 sec
QS elapsed time = 23.8910 seconds.
[...]
SIQS elapsed time = 28.9506 seconds.

That's better than before!

Regarding ^M, this seems to be \r in some files. So there are files that contain mixed line ends. It should be quick to convert all files to \n only.

[bsquared]

Originally Posted by kruoli

Thank you!

There was another difference I oversaw: I had -O3 in my recent builds. I reverted this to -O2 and now the SIQS times are fine again!

Elapsed time: 23.8909 sec
QS elapsed time = 23.8910 seconds.
[...]
SIQS elapsed time = 28.9506 seconds.

That's better than before!

Regarding ^M, this seems to be \r in some files. So there are files that contain mixed line ends. It should be quick to convert all files to \n only.

Great news, thanks!

I vaguely recall having issues like this with -O3 in the past, which is why it defaults to -O2, but I would not have thought of that as a culprit. Glad you found it.

[kruoli]

When running YAFU2 (recent GIT version) with aliqueit, I need to run a script that checks for running 0 auto-increasing ecm curves... in the output. Are you interested in finding a workaround for these? When I was "present" when something like this occured, it seemed always like a msieve problem (SIQS postprocessing was not able to finish or some other problem).

[EdH]

I have had similar occurrences on occasion and traced them to rare SIQS crashes. I've only briefly seen the actual messages from YAFU due to the way my scripts run, but the messages seemed to be a row vs. column matrix issue. I haven't pursued it much, because I set Msieve to run if YAFU fails (in aliqueit.ini). Now, if Aliqueit doesn't find any returned factors from YAFU (via factor.log), it runs Msieve. That has seemed to take care of it for me. But, I'm not using the -y YAFU switch, which might change things.

[BudgieJane]

Originally Posted by BudgieJane

I downloaded 2.08 and I've got the same problem I had with 2.07 when that was first available, in that it would appear that the Windows .exe file is missing a few bytes ...

I have bought myself a new laptop, and this one runs Windows 10 rather than Windows 7 that my old laptop was running. The good news is that Yafu 2.08 no longer crashes. However, there is some bad news:

JANELT5 C:\Users\Jane\Documents\APLapps\Numbers > ..\..\Maths\yafu\Versions\yafu-2.08\yafu-x64


YAFU Version 2.08
Built with Microsoft Visual Studio 1922
Using GMP-ECM 7.0.4, Powered by MPIR 3.0.0
Detected 11th Gen Intel(R) Core(TM) i5-11300H @ 3.10GHz
Detected L1 = 49152 bytes, L2 = 8388608 bytes, CL = 64 bytes
Using 1 random witness for Rabin-Miller PRP checks
Cached 664579 primes; max prime is 9999991
Could not parse yafu.ini from C:\Users\Jane\Documents\APLapps\Numbers

Which is all very well, but Yafu.ini is definitely there, and I can read it, as this shows:

JANELT5 C:\Users\Jane\Documents\APLapps\Numbers > type yafu.ini
% This is yafu.ini from JaneLT5 as amended by tune. It is in C:\Users\Jane\Documents\APLapps\Numbers\
B1pm1=100000
B1pp1=20000
B1ecm=11000
rhomax=1000
threads=4
pretest_ratio=0.20
ggnfs_dir=..\..\Maths\ggnfs-x64\
ecm_path=..\..\Maths\ecm\ecm.exe
ext_ecm=10000
%tune_info=Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz,WIN64,2.61946e-005,0.196468,0.240753,0.106352,95.27,2450.77
%tune_info=Intel(R) Core(TM) i7-4710MQ CPU @ 2.50GHz,WIN64,2.52059e-06,0.2279,6.39174,0.080593,100.104,42
tune_info=Intel(R) Core(TM) i5-11300H @ 3.10GHz,WIN64,2.52059e-06,0.2279,6.39174,0.080593,100.104,42

JANELT5 C:\Users\Jane\Documents\APLapps\Numbers >

Can anyone see where I might have made an error in the .ini file? I don't think I have, because it works OK with Yafu v.2.07.

Anyway, I ran tune, and it's obvious that I got the cpu-string wrong, because tune has ignored my tune_info (which was a copy of the previous version with what I thought was the proper cpu_string) and it has added another line to the file (which is the proof that it can read and write the file, even if it can't parse it, as shown above).

tune_info=Intel(R) Core(TM) i5-11300H @ 3.10GHz,WIN64,2.52059e-06,0.2279,6.39174,0.080593,100.104,42
tune_info=11th Gen Intel(R) Core(TM) i5-11300H @ 3.10GHz,WIN64,7.11117e-06,0.200817,6.67832,0.0763865,110.525,42

So, what's going on here? I would like to run the latest version. As usual, if you want any more information, just ask.

[James Heinrich]

Originally Posted by BudgieJane

Can anyone see where I might have made an error in the .ini file?

No. I tried it myself, and replicated what you got with Could not parse yafu.ini using my 2.07 yafu.ini, but worked with the shipping version. So I went through it line by line, checking each change that might make it not parse. And none of them broke it.

Edit: Aha! I found the problem line. But I don't understand it.
If you comment out the v on the verbosity line then it can't parse. Leave the single uncommented v there and it parses.

[bsquared]

Originally Posted by James Heinrich

No. I tried it myself, and replicated what you got with Could not parse yafu.ini using my 2.07 yafu.ini, but worked with the shipping version. So I went through it line by line, checking each change that might make it not parse. And none of them broke it.

Edit: Aha! I found the problem line. But I don't understand it.
If you comment out the v on the verbosity line then it can't parse. Leave the single uncommented v there and it parses.

I found the problem. For some dumb reason I surrounded the getcwd() command with a check for verbosity>0. So you have to have at least one -v for it to parse the .ini.

Hopefully that workaround is good enough for now; I can get the exe updated soon, hopefully.

[bsquared]

Originally Posted by bsquared

I found the problem. For some dumb reason I surrounded the getcwd() command with a check for verbosity>0. So you have to have at least one -v for it to parse the .ini.

Hopefully that workaround is good enough for now; I can get the exe updated soon, hopefully.

Actually, I'm wrong, you have to have one -v for it to *say* that it parsed the .ini file. It will parse it correctly regardless of what it says, it's just a reporting problem.