AVX-ECM

[mathwiz]

Originally Posted by bsquared

I've checked the latest code into r393 if anyone wants to do some testing. You will need a avx512 capable cpu.

Is https://github.com/bbuhrow/avx-ecm/ the latest repo? Unless I'm reading wrong, the only commit recently (3 days ago) was just to README.md?

[bsquared]

Originally Posted by mathwiz

Is https://github.com/bbuhrow/avx-ecm/ the latest repo? Unless I'm reading wrong, the only commit recently (3 days ago) was just to README.md?

Ah, good question. Sorry, I've folded it all into yafu, so it is yafu r393 that you'll need. It is probably a good idea to update the avx-ecm repo as well since that is significantly easier to build. I can do that tonight, hopefully.

[VBCurtis]

One more reason to buy one of those Phi barebones systems!

Nice work.

[mathwiz]

Originally Posted by bsquared

Ah, good question. Sorry, I've folded it all into yafu, so it is yafu r393 that you'll need. It is probably a good idea to update the avx-ecm repo as well since that is significantly easier to build. I can do that tonight, hopefully.

Gotcha. Is there an easy way to build just AVX-ECM from the latest yafu sources? Or, if you could update the avx-ecm git repo, that'd be great too!

I tried:

$ svn checkout https://svn.code.sf.net/p/yafu/code/branches/wip yafu-code
$ make -j 8 SKYLAKEX=1

But I get some linker errors like:

gcc-9: error: /sppdg/scratch/buhrow/projects/gmp_install/gmp-6.2.0/lib/libgmp.a: No such file or directory

[bsquared]

Originally Posted by VBCurtis

One more reason to buy one of those Phi barebones systems!

Nice work.

Thanks!

I actually have some data for the KNL, linearly scaled from a run with B1=7e6.

7210 KNL:
2048 simultaneous curves with 256 threads
354 hours/run
10.4 minutes/curve

7250 KNL
2176 simultaneous curves with 272 threads
353 hours/run
9.7 minutes/curve

B2 then takes about 3/4 of that to go up to 100*B1, another 264 hours, so that's a lengthy ~25 day runtime per batch. But you get 2000+ curves at the end of it (B1=7G).
[edit]
Running with fewer threads and curves trades off throughput for lower wall-clock time, if that is a concern. I don't have data for the exact trades there, yet. Also, it will save intermediate checkpoint files every 100M primes for each curve. The savefiles can be fed to GMP-ECM for stage 2, if you'd rather run stage 2 that way. Run with -saveB1 on the command line to enable saving.

Memory use is pretty minimal for both stages; won't be a problem to stay in MCDRAM.

[bsquared]

Originally Posted by mathwiz

Gotcha. Is there an easy way to build just AVX-ECM from the latest yafu sources? Or, if you could update the avx-ecm git repo, that'd be great too!

I tried:

$ svn checkout https://svn.code.sf.net/p/yafu/code/branches/wip yafu-code
$ make -j 8 SKYLAKEX=1

But I get some linker errors like:

No easy way to just build part of yafu. Makefiles are not my strong suit. I very much recommend EdH's guides on building yafu. It requires a small amount of makefile editing to point to your msieve, ecm, and gmp locations.

IMO there are reasons to prefer the yafu version of avx-ecm. It integrates into factor(), so it's fire and forget on any input. It will auto-recognize Mersennes and determine the fastest approach (sometimes it's better to just use Montgomery arithmetic). It uses the NFS algebraic factor extraction framework that's already there, so you could feed it, say, (2^1265-1) / 121441 / 533831 / 136149590957161 / 556590676780973231 and it would pull out all the annoying algebraic factors and go to work on the remaining C223. You can tell it to pretest up to a specific t-level, etc.

avx-ecm has none of that, but I will update it with the new vector math at least, with maybe a command line switch to turn it on.

[wombatman]

Posting here only because the issue seems to be related to avx-ecm. I'm trying to build the latest revision of YAFU, and no matter what I pass during make, it's failing with errors like:

/usr/bin/ld: factor/avx-ecm/avx_ecm_main.o: in function `vec_ecm_main':
/home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:266: undefined reference to `snfs_init'
/usr/bin/ld: /home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:272: undefined reference to `find_primitive_factor'
/usr/bin/ld: /home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:279: undefined reference to `snfs_clear'
/usr/bin/ld: /home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:464: undefined reference to `vecmulmod52_mersenne'
/usr/bin/ld: /home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:465: undefined reference to `vecsqrmod52_mersenne'
collect2: error: ld returned 1 exit status
make: *** [Makefile:397: all] Error 1

I'm passing in this:

 make clean && make ECM=1 USE_SSE41=1

This error is recent, though I can't say exactly which revision I was able to successfully build. Any suggestions?

[bsquared]

Originally Posted by wombatman

Posting here only because the issue seems to be related to avx-ecm. I'm trying to build the latest revision of YAFU, and no matter what I pass during make, it's failing with errors like:

/usr/bin/ld: factor/avx-ecm/avx_ecm_main.o: in function `vec_ecm_main':
/home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:266: undefined reference to `snfs_init'
/usr/bin/ld: /home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:272: undefined reference to `find_primitive_factor'
/usr/bin/ld: /home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:279: undefined reference to `snfs_clear'
/usr/bin/ld: /home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:464: undefined reference to `vecmulmod52_mersenne'
/usr/bin/ld: /home/wombat/yafu-code/branches/wip/factor/avx-ecm/avx_ecm_main.c:465: undefined reference to `vecsqrmod52_mersenne'
collect2: error: ld returned 1 exit status
make: *** [Makefile:397: all] Error 1

I'm passing in this:

 make clean && make ECM=1 USE_SSE41=1

This error is recent, though I can't say exactly which revision I was able to successfully build. Any suggestions?

Looks like you just need to use more makefile options. You need at least NFS=1 to bring in the nfs stuff and SKYLAKEX=1 for the avx512 vector arithmetic stuff. And if you use SKYLAKEX=1 then you also need USE_AVX2=1. My typical build line looks like this:

make NFS=1 USE_AVX2=1 USE_BMI2=1 SKYLAKEX=1

also, ECM=1 is a msieve thing, not a yafu thing, so you can leave that off.

[bsquared]

Another quick usage note...
Normally, yafu will switch over from using an internal GMP-ECM to an external GMP-ECM (pointed to by the ecm_path variable in yafu.ini) at B1=50k. With AVX-ECM enabled, by default that switchover happens at B1=40M instead. If you want to use AVX-ECM at higher B1, put that crossover value into the variable "ext_ecm" in yafu.ini.

For example if I want to use B1=110M with AVX-ECM and not with the external GMP-ECM, set ext_ecm=120000000, or anything larger than 110M.

[mathwiz]

(waits patiently for the old avx-ecm git repo to be updated with the latest code... it's so much easier to build :)

[bsquared]

Originally Posted by mathwiz

(waits patiently for the old avx-ecm git repo to be updated with the latest code... it's so much easier to build :)

Thanks for your patience

Now updated, tested, seems to work. Also checked in a pre-built version for windows. The mingw compiler doesn't do as good of a job, but something for windows is better than nothing.