YAFU 2.0

[bsquared]

Originally Posted by James Heinrich

I'm back home now and able to test v2.06 on my Win10/i8-8100 machine.
It works, but I've noticed a couple things (which entirely could be me doing something wrong).

1) ECM runs as a single thread of yafu-x64.exe
I don't care if ECM runs within yafu or spawns external ecm.exe as long as it uses the appropriate number of threads.
Relevant yafu.ini entries:
threads=4
ecm_path=c:\users\user\desktop\yafu2\ecm\ecm.exe
If I specify an invalid path for ecm.exe then it appropriately tells me "ecm: ECM executable does not exist at c:\invalid\path\ecm.exe" and "using internal single threaded ECM...", but even when the path to ecm.exe is valid it appears to be using the internal, single-threaded ECM (with no warning).

This is something I can fix, but in the meantime use -ext_ecm X on the command line or .ini file, where X is the B1 crossover point to start using external ecm (something like 50000 works well).

The problem is that this value is set really big when avx-ecm is available during compilation. I need to add a check that it is also available at runtime...

Originally Posted by James Heinrich

2) Number of witnesses for PRP is always "1" no even if I set something like nprp=20 in yafu.ini

I will fix this. However for most factoring jobs aprcl is used instead of gmp's prp function and aprcl doesn't use or need the witnesses option.

Originally Posted by James Heinrich

3) Increased verbosity doesn't seem to work (at least in yafu.ini). The description says "Note that more v's increase the verbosity" but if I specify v v it tells me "invalid option v v" and spews the list of valid options and exits

Put the v's on separate lines in the .ini file. Each line is interpreted as one option.

[James Heinrich]

Originally Posted by bsquared

The problem is that this value is set really big when avx-ecm is available during compilation. I need to add a check that it is also available at runtime...

I was able to use your suggested config setting to spawn the external ecm.exe, but I'm not certain how it's supposed to work with the internal ECM -- is that not multithreaded? Is there something I'm missing to be able to use the built-in ECM but make use of available threads?

Originally Posted by bsquared

Put the v's on separate lines in the .ini file. Each line is interpreted as one option.

This wasn't obvious to me, perhaps an amendment to the .ini file to make that clearer would help.

[kruoli]

On my installations, I always use prefer_gmpecm and this works as intended, also with 2.06.

[bsquared]

Originally Posted by James Heinrich

I was able to use your suggested config setting to spawn the external ecm.exe, but I'm not certain how it's supposed to work with the internal ECM -- is that not multithreaded? Is there something I'm missing to be able to use the built-in ECM but make use of available threads?

This wasn't obvious to me, perhaps an amendment to the .ini file to make that clearer would help.

Before ecm version 7 only single threads were allowed, so yafu does a check on the ecm version linked in during compile. I'm pretty sure I'm linking with version 7.0.4 in the executable I provide but it's possible there is a bug in detecting that version. I'll look it over.

Yes I'll add some text to the .ini file description.

[bsquared]

Originally Posted by kruoli

On my installations, I always use prefer_gmpecm and this works as intended, also with 2.06.

That should only matter if you have AVX-512, in which case yafu will default to using AVX-ECM.

[bur]

edit: I saw this was reported earlier and should be fixed?

I pulled the newest version and did a make clean and make NFS=1 USE_SSE41=1 USE_AVX2=1 and building quits with the following error:

/usr/bin/ld: ./libyecm.a(avx_ecm_main.o): in function `vec_ecm_main':
/home/florian/Math/yafu/factor/avx-ecm/avx_ecm_main.c:592: undefined reference to `vecmulmod52_fixed1040_bfips'
/usr/bin/ld: /home/florian/Math/yafu/factor/avx-ecm/avx_ecm_main.c:593: undefined reference to `vecsqrmod52_fixed1040_bfips'
/usr/bin/ld: /home/florian/Math/yafu/factor/avx-ecm/avx_ecm_main.c:594: undefined reference to `vec_simul_addsub52_fixed1040'
[... lots of similar lines]

I also tried your previous suggestion to use the USE_BMI2=1 flag. And I used the flags during make clean.

Do you know why I still run into the problem?

[bsquared]

Originally Posted by bur

Do you know why I still run into the problem?

Because for some reason my push/commit didn't take... I'm still learning this whole git thing... I'll try again tonight.

[James Heinrich]

I wouldn't have noticed this except for some error checking in my automated parsing of factor.log

09/13/21 08:12:06, ****************************
09/13/21 08:12:06, Starting factorization of 11745986472276496655662247734956521108703066873589956247227553585499089637967910706104797828678082669016193648910144450016469882935490998359642649527365479
09/13/21 08:12:06, using pretesting plan: deep
09/13/21 08:12:06, using specified qs/gnfs crossover of 100 digits
09/13/21 08:12:06, using specified qs/snfs crossover of 75 digits
09/13/21 08:12:06, ****************************
09/13/21 08:12:06, rho: x^2 + 3, starting 1000 iterations on C155
09/13/21 08:12:06, prp5 = 91291
09/13/21 08:12:06, rho: x^2 + 3, starting 1000 iterations on C150
09/13/21 08:12:06, prp7 = 7805879
09/13/21 08:12:06, rho: x^2 + 3, starting 1000 iterations on C143
09/13/21 08:12:06, rho: x^2 + 2, starting 1000 iterations on C143
09/13/21 08:12:07, rho: x^2 + 1, starting 1000 iterations on C143
09/13/21 08:12:07, pm1: starting B1 = 150K, B2 = gmp-ecm default on C143
09/13/21 08:12:07, current ECM pretesting depth: 0.00
09/13/21 08:12:07, scheduled 30 curves at B1=2000 toward target pretesting depth of 47.67
09/13/21 08:12:07, Finished 30 curves using GMP-ECM method on C143 input, B1=2k, B2=gmp-ecm default
09/13/21 08:12:07, current ECM pretesting depth: 15.18
09/13/21 08:12:07, scheduled 74 curves at B1=11000 toward target pretesting depth of 47.67
09/13/21 08:12:10, Finished 74 curves using GMP-ECM method on C143 input, B1=11k, B2=gmp-ecm default
09/13/21 08:12:10, current ECM pretesting depth: 20.24
09/13/21 08:12:10, scheduled 214 curves at B1=50000 toward target pretesting depth of 47.67
09/13/21 08:12:11, prp26 = 23304102950704118347000297 (curve 2 stg2 B1=50000 sigma=3874399757 thread=1)
09/13/21 08:12:11, Finished 4 curves using GMP-ECM method on C143 input, B1=50k, B2=gmp-ecm default
09/13/21 08:12:11, current ECM pretesting depth: 20.34
09/13/21 08:12:11, scheduled 210 curves at B1=50000 toward target pretesting depth of 39.00
09/13/21 08:12:22, Finished 210 curves using GMP-ECM method on C117 input, B1=50k, B2=gmp-ecm default
09/13/21 08:12:22, pm1: starting B1 = 3750K, B2 = gmp-ecm default on C117
09/13/21 08:12:23, current ECM pretesting depth: 25.33
09/13/21 08:12:23, scheduled 430 curves at B1=250000 toward target pretesting depth of 39.00
09/13/21 08:13:47, Finished 430 curves using GMP-ECM method on C117 input, B1=250k, B2=gmp-ecm default
09/13/21 08:13:47, pm1: starting B1 = 15M, B2 = gmp-ecm default on C117
09/13/21 08:13:50, current ECM pretesting depth: 30.45
09/13/21 08:13:50, scheduled 904 curves at B1=1000000 toward target pretesting depth of 39.00
09/13/21 08:25:10, Finished 904 curves using GMP-ECM method on C117 input, B1=1M, B2=gmp-ecm default
09/13/21 08:25:10, current ECM pretesting depth: 35.56
09/13/21 08:25:10, scheduled 1619 curves at B1=3000000 toward target pretesting depth of 39.00
09/13/21 08:31:42, prp35 = 10458847902056956740364765053965417 (curve 195 stg2 B1=3000000 sigma=2620919999 thread=0)
09/13/21 08:31:49, Finished 197 curves using GMP-ECM method on C117 input, B1=3M, B2=gmp-ecm default
09/13/21 08:31:49, final ECM pretested depth: 35.98
09/13/21 08:31:49, prp84 cofactor = 67627522805179106773859180025103681923504306209525858835440580638970766947677849939
09/13/21 08:31:49, Total factoring time = 1183.1567 seconds

Note the second-last line:
prp84 cofactor = 67627522805179106773859180025103681923504306209525858835440580638970766947677849939
That number is only 83 characters long, but claims to be a prp84.

For comparison, this is what it shows on v1.34.5:

09/13/21 20:25:23 v1.34.5 @ 3930K, ****************************
09/13/21 20:25:23 v1.34.5 @ 3930K, Starting factorization of 11745986472276496655662247734956521108703066873589956247227553585499089637967910706104797828678082669016193648910144450016469882935490998359642649527365479
09/13/21 20:25:23 v1.34.5 @ 3930K, using pretesting plan: deep
09/13/21 20:25:23 v1.34.5 @ 3930K, using tune info for qs/gnfs crossover
09/13/21 20:25:23 v1.34.5 @ 3930K, ****************************
09/13/21 20:25:23 v1.34.5 @ 3930K, rho: x^2 + 3, starting 1000 iterations on C155
09/13/21 20:25:23 v1.34.5 @ 3930K, rho: x^2 + 2, starting 1000 iterations on C155
09/13/21 20:25:23 v1.34.5 @ 3930K, prp5 = 91291
09/13/21 20:25:23 v1.34.5 @ 3930K, rho: x^2 + 2, starting 1000 iterations on C150
09/13/21 20:25:23 v1.34.5 @ 3930K, prp7 = 7805879
09/13/21 20:25:23 v1.34.5 @ 3930K, rho: x^2 + 2, starting 1000 iterations on C143
09/13/21 20:25:23 v1.34.5 @ 3930K, rho: x^2 + 1, starting 1000 iterations on C143
09/13/21 20:25:23 v1.34.5 @ 3930K, pm1: starting B1 = 150K, B2 = gmp-ecm default on C143
09/13/21 20:25:23 v1.34.5 @ 3930K, current ECM pretesting depth: 0.00
09/13/21 20:25:23 v1.34.5 @ 3930K, scheduled 30 curves at B1=2000 toward target pretesting depth of 47.67
09/13/21 20:25:23 v1.34.5 @ 3930K, Finished 30 curves using Lenstra ECM method on C143 input, B1=2K, B2=gmp-ecm default
09/13/21 20:25:23 v1.34.5 @ 3930K, current ECM pretesting depth: 15.18
09/13/21 20:25:23 v1.34.5 @ 3930K, scheduled 74 curves at B1=11000 toward target pretesting depth of 47.67
09/13/21 20:25:27 v1.34.5 @ 3930K, Finished 74 curves using Lenstra ECM method on C143 input, B1=11K, B2=gmp-ecm default
09/13/21 20:25:27 v1.34.5 @ 3930K, current ECM pretesting depth: 20.24
09/13/21 20:25:27 v1.34.5 @ 3930K, scheduled 214 curves at B1=50000 toward target pretesting depth of 47.67
09/13/21 20:25:35 v1.34.5 @ 3930K, prp26 = 23304102950704118347000297 (curve 22 stg2 B1=50000 sigma=3377483298 thread=2)
09/13/21 20:25:35 v1.34.5 @ 3930K, Finished 132 curves using Lenstra ECM method on C143 input, B1=50K, B2=gmp-ecm default
09/13/21 20:25:35 v1.34.5 @ 3930K, current ECM pretesting depth: 23.33
09/13/21 20:25:35 v1.34.5 @ 3930K, scheduled 82 curves at B1=50000 toward target pretesting depth of 39.00
09/13/21 20:25:39 v1.34.5 @ 3930K, Finished 84 curves using Lenstra ECM method on C117 input, B1=50K, B2=gmp-ecm default
09/13/21 20:25:39 v1.34.5 @ 3930K, pm1: starting B1 = 3750K, B2 = gmp-ecm default on C117
09/13/21 20:25:41 v1.34.5 @ 3930K, current ECM pretesting depth: 25.34
09/13/21 20:25:41 v1.34.5 @ 3930K, scheduled 430 curves at B1=250000 toward target pretesting depth of 39.00
09/13/21 20:27:04 v1.34.5 @ 3930K, Finished 432 curves using Lenstra ECM method on C117 input, B1=250K, B2=gmp-ecm default
09/13/21 20:27:04 v1.34.5 @ 3930K, pm1: starting B1 = 15M, B2 = gmp-ecm default on C117
09/13/21 20:27:10 v1.34.5 @ 3930K, current ECM pretesting depth: 30.46
09/13/21 20:27:10 v1.34.5 @ 3930K, scheduled 904 curves at B1=1000000 toward target pretesting depth of 39.00
09/13/21 20:30:19 v1.34.5 @ 3930K, prp35 = 10458847902056956740364765053965417 (curve 40 stg2 B1=1000000 sigma=886406094 thread=1)
09/13/21 20:30:19 v1.34.5 @ 3930K, Finished 240 curves using Lenstra ECM method on C117 input, B1=1M, B2=gmp-ecm default
09/13/21 20:30:19 v1.34.5 @ 3930K, final ECM pretested depth: 31.78
09/13/21 20:30:19 v1.34.5 @ 3930K, scheduler: switching to sieve method
09/13/21 20:30:19 v1.34.5 @ 3930K, prp83 = 67627522805179106773859180025103681923504306209525858835440580638970766947677849939
09/13/21 20:30:19 v1.34.5 @ 3930K, Total factoring time = 296.4042 seconds

[bur]

Because for some reason my push/commit didn't take...

Thanks!

[bsquared]

Originally Posted by James Heinrich

I wouldn't have noticed this except for some error checking in my automated parsing of factor.log

Note the second-last line:
prp84 cofactor = 67627522805179106773859180025103681923504306209525858835440580638970766947677849939
That number is only 83 characters long, but claims to be a prp84.

This should be fixed now, as well as the previous problem building on avx2.

I verified the latest commits made it to github, we are now at 2.07.

In this version I've committed changes to the MSVC project files and linux makefile that revert back to the "normal" directory structure of yafu and its dependencies (no more ".git" or "trunk" in the relative paths of libraries). And removed some confusing stuff in the makefile. Hopefully working toward something that is easier to build.

Two other things of note:
1) I was made aware that aprcl wasn't running on factors found, this is now fixed and factors displayed to screen should now be "P" instead of "PRP"
2) AVX-ECM below 1040 bits is about 10% faster.

Regarding the "P" vs. "PRP" labeling... the .log file still generally gets "prp" printed. Which could be fixed, but before I do that I wanted to see if that would break anyone's scripting/parsing?

[James Heinrich]

Originally Posted by bsquared

Regarding the "P" vs. "PRP" labeling... the .log file still generally gets "prp" printed. Which could be fixed, but before I do that I wanted to see if that would break anyone's scripting/parsing?

It wouldn't break anything I do, my regex already looks like ^(prp|p)([0-9]+)( cofactor)? = ([0-9]+)

We already have to handle the different format for small factors: div: found prime factor = 3
Possibly this could be changed to standardize it such that all factors follow the format (p|prp|c)<len> = <factor>