YAFU 2.0

[kruoli]

I tried running it in Visual Studio and got an exception at 0x00007FF6B3ECD044 in yafu-mingw-avx2.exe: 0xC000001D: Illegal Instruction.

It highlights line 50 of tiny.c, maybe the sqrt(...) is the problem here? It will take some time until I have it running under WSL, but maybe this helps?

The line number might differ if there were some changes after the .exe was built, since I used the latest git code.

[kruoli]

Just got it!
It is 00007FF6B3ECD044 vcvtusi2sd xmm6,xmm6,ebx.

[ldesnogu]

Originally Posted by kruoli

Just got it!
It is 00007FF6B3ECD044 vcvtusi2sd xmm6,xmm6,ebx.

That's an AVX512F instruction: https://www.felixcloutier.com/x86/vcvtusi2sd

[bsquared]

Originally Posted by kruoli

Just got it!
It is 00007FF6B3ECD044 vcvtusi2sd xmm6,xmm6,ebx.

That is a AVX512F instruction, weird. Thank you for tracking that down.

I didn't realize you were using the mingw executable. Do you have the same problem if you use the yafu-x64 version? That was built by visual studio and I would expect it to be more portable.

[kruoli]

Originally Posted by kruoli

The non-AVX version works fine.

The mingw version was the only AVX2 version I saw. The version without mingw is running fine on all machines so far. It also is remarkably smaller.

[bsquared]

Originally Posted by kruoli

The mingw version was the only AVX2 version I saw. The version without mingw is running fine on all machines so far. It also is remarkably smaller.

Ok, sounds good. Just use the -x64 version. I might just remove the mingw versions from git.

[extra info]
A long-running background project for yafu is to get all assembly optimizations working in visual studio, so I can forget about the mingw path. mingw was nice when that was the fastest option for windows, but now with WSL2 and better assembly support (via intrinsics) for visual studio, I'd like to move away from mingw.

[bsquared]

A couple updates, one related to a msieve change that I've just noticed.

1)
As of msieve 1023, gnfs cpu (and maybe gpu?) poly selection now defaults to a massive 8640000 seconds deadline per coefficient searched. That can be changed with a "poly_deadline" parameter, which I didn't know about before. Noticed it recently when trying to gnfs a c120 or so, and having it spend hours in multi-threaded poly select for a 30 minute job. The update now uses the poly_deadline parameter and it seems to work better.

Although, it finds way fewer polys than before. The parameter tables changed in msieve 1023 as well, and some seem to have pretty strict minimum E-score requirements.

2)
When running snfs with an input poly file, yafu will do a quick search for a better skew value. So far it doesn't do anything with this info other than print it to the screen, but after some more testing I might automatically update the input poly with the improved skew, if one is found.

3)
I'm no longer supporting -psearch deep or -psearch wide for multi-threaded gnfs poly selection. Not sure if these were even used by anyone... I know I didn't use them. Now all searches are "fast", which was the default, and which divides the search time by the number of threads. -psearch {min | avg | good} can still be used to modify acceptable E-scores.* For small jobs I've found -psearch avg or even -psearch min to be faster, as any old poly will sieve almost as fast as another and you would like to get on with the sieving asap.

[edit]
* the -psearch min | avg capability is where conflicts might occur with the new msieve poly parameter tables. I need to do more testing to know for sure. But for that c120 job I did recently, it found exactly one poly after 10 minutes of searching on 40 threads (over 6 thread-hours of searching). Specifying -psearch min won't do any good if msieve won't even return polys found near that score.

[charybdis]

Latest yafu segfaults when resuming NFS. Here's the gdb output:

>> nfs: checking for job file - job file found, testing for matching input
nfs: number in job file matches input
nfs: checking for data file
nfs: previous data file found
nfs: commencing search for last special-q
line 0 = 1631079863,1741:fb1cb3,26a433,4903:156d15,c5f,ec3,1387,a1d71,ce251,e01fb

line 1 = 1638952149,2035:19f8559,be9,11ab,3121,c775:38e243,13af,2e2d,3161,3491,27e51,e01fb

line 2 = 1736339753,1015:7f3ae3,2cde80d,116f,274e5:100019,2971ac5,4255,a3ff,af7f,e01fb

line 3 = 1733728719,1543:26a084d,18b5637,1999,48f7:305e33f,1d41f,61d11,6d3,e01fb

nfs: parsing special-q from .dat file
nfs: commencing nfs on c100: 1522605027922533360535618378132637429718068114961380688657908494580122963258952897654000350692006139
nfs: resumesieve; last_spq = 918011, nfs_phases = 0
nfs: found skew: 1108606.390000
nfs: found c[0]: -537120876901310626660353125
nfs: found c[1]: 20224876429498235072250
nfs: found c[2]: 13491534070563215
nfs: found c[3]: -19666557336
nfs: found c[4]: 8820
nfs: found Y0: -644590365547202992686312
nfs: found Y1: 11026585360387
nfs: parsed lpbr = 26, lpba = 26

Program received signal SIGSEGV, Segmentation fault.
parse_job_file (fobj=fobj@entry=0x5555578f9000, job=job@entry=0x7fffffffa1d0)
    at factor/nfs/nfs_filemanip.c:1372
1372            job->snfs->sdifficulty = job->snfs->difficulty = 0;
(gdb) backtrace
#0  parse_job_file (fobj=fobj@entry=0x5555578f9000, job=job@entry=0x7fffffffa1d0) at factor/nfs/nfs_filemanip.c:1372
#1  0x00005555555da4cb in nfs (fobj=fobj@entry=0x5555578f9000) at factor/nfs/nfs.c:874
#2  0x000055555556a0b2 in feval (funcnum=funcnum@entry=71, nargs=1, metadata=metadata@entry=0x7fffffffb6c0) at top/cmdParser/calc.c:2857
#3  0x00005555555696f9 in calc (in=in@entry=0x7fffffffb500, metadata=metadata@entry=0x7fffffffb6c0) at top/cmdParser/calc.c:1943
#4  0x00005555555699b0 in calc_with_assignment (in=in@entry=0x5555578f7850, metadata=metadata@entry=0x7fffffffb6c0, force_quiet=force_quiet@entry=0)
    at top/cmdParser/calc.c:1531
#5  0x0000555555569b8a in process_expression (input_exp=<optimized out>, metadata=0x7fffffffb6c0, force_quiet=0, no_convert_result=0) at top/cmdParser/calc.c:1477
#6  0x000055555555a878 in main (argc=<optimized out>, argv=<optimized out>) at top/driver.c:366

[charybdis]

Despite the recent changes to make poly selection faster, it still continues up until the deadline for most 109-digit numbers:

07/05/21 07:12:47, nfs: commencing nfs on c109: 1090856851194159102847462612480990401998890638794326582448210291834957096186057950134759387351816485448746449
07/05/21 07:12:47, nfs: commencing poly selection with 6 threads
07/05/21 07:12:47, nfs: setting deadline of 2964 seconds
07/05/21 07:12:47, nfs: expecting degree 4 poly E from 3.14e-09 to > 3.62e-09
07/05/21 07:12:47, nfs: searching for avg quality poly E > 3.26e-09
07/05/21 08:04:29, nfs: completed 146 ranges of size 60 in 3102.7585 seconds
07/05/21 08:04:29, nfs: best poly = # norm 3.444816e-10 alpha -7.184646 e 1.363e-09 rroots 3

Notice the huge difference between the expected E values and what we actually get. For numbers between 10^108 and 2^362 (≈9.39*10^108), yafu thinks msieve will use degree 4 but it actually uses degree 5. So an "average" quality polynomial never turns up and the search continues until the deadline.

[bsquared]

Originally Posted by charybdis

Despite the recent changes to make poly selection faster, it still continues up until the deadline for most 109-digit numbers:

07/05/21 07:12:47, nfs: commencing nfs on c109: 1090856851194159102847462612480990401998890638794326582448210291834957096186057950134759387351816485448746449
07/05/21 07:12:47, nfs: commencing poly selection with 6 threads
07/05/21 07:12:47, nfs: setting deadline of 2964 seconds
07/05/21 07:12:47, nfs: expecting degree 4 poly E from 3.14e-09 to > 3.62e-09
07/05/21 07:12:47, nfs: searching for avg quality poly E > 3.26e-09
07/05/21 08:04:29, nfs: completed 146 ranges of size 60 in 3102.7585 seconds
07/05/21 08:04:29, nfs: best poly = # norm 3.444816e-10 alpha -7.184646 e 1.363e-09 rroots 3

Notice the huge difference between the expected E values and what we actually get. For numbers between 10^108 and 2^362 (≈9.39*10^108), yafu thinks msieve will use degree 4 but it actually uses degree 5. So an "average" quality polynomial never turns up and the search continues until the deadline.

This is good to know, I will have to add a check to see what degree is actually being used.

Also, the bug you previously reported has been found and fixed (thanks for the detailed report!). I still need to commit the changes... hopefully will get a chance tonight.

[jasonp]

If Yafu has specific needs about configuring NFS polynomial, maybe it should add a configuration string of its own when running Msieve. That would let it control the polynomial degree and all the bounds used. The bounds and deadlines changed quite a while ago and reflected a lot of best practice here, but they can't cover every use case.