ggnfs improvements

[henryzz]

Originally Posted by bsquared

More frustrations. Trying to make the f-versions of the sievers fails because input-poly-orig doesn't exist (none of .w .c or .h).

But there is an input-poly.orig in the lasieve4 source! (note the .orig instead of -orig in the name). Ok, copy over input-poly.orig.c, rename it to input-poly-orig.c, create an input-poly-orig.h which has a declaration for the input_poly_orig function and attempt to make allf. And it works! But, the sievers won't run with the error:

Cannot read number which is to be factored: Success

Attempt 2:
Rename input-poly.c and input-poly.h, which do exist in the repo, to input-poly-orig.c and input-poly-orig.h. Build works again!

But, the sievers won't run with the error:

Input line =
Rational sieve parameters like
sievemin FBbound sieverest_multiplier max_prime_bits max_factorbits
eg. 20     1e6    3.2                  30             50
lacking from input file nfs300_650k_I11.job

Anyone know how to obtain a correct copy of input-poly-orig.c and input-poly-orig.h so that the f- and g-versions of the siever can be built?

The f and g versions probably don't have many of the ggnfs modifications although there have been working versions of the f siever around in the past. The f version allows composite special q and sieving special q below the fb limits. Sieving only prime special q it was a slightly slower. Composite special q tend to increase the duplication rate. There may be situations where this is useful to improve overall yield on very large jobs.
The g version looks like it adds batch trial division. I don't know whether this is an improvement.

I think I did get both f and g to compile at one point(maybe 10 years ago). I am not sure that I got g working. Will have a look at what I have from the distant past tomorrow.
I remember f was a bit of a fiddle to get working.

[wreck]

I have done the sieve from 316M to 317M of R1340L, 959668 relations collected,
while 315M to 316M collect 1291314 relations, about 331646 less relations.
316000000 to 316492661 sieved using original gnfs binary under windows 10 (628194 relations, 1.27 rel/q),
while 316492661 to 317000000 sieved using avx512-gnfs under Ubuntu 20.04
(331474 relations, 0.653 rel/q; if remove the q from 316724521 to 316963841,
then 268019 special-Q, 1.23 rel/q, about the same as original ggnfs).


I dig to found that no q between 316724521 to 316963841, there exist five relations
in the relation file.

-279653911,270203771:2C47EC05,5455,5117,D955FD7,792367,6CA4A7:26D57BED,8D97BD69,E1EA112D,655,FBBBA0D,19C8371,13541B1,C33305,12E0D529
355978387,122689099:821,570B,1EC7,882D177,10E0F7,4467B,8C5D:1DFEA7DD,96D50931,8AD,B5D2F55,594CBB9,FF9539,8E8D85,528D49,20E55,12E0D529
-128782835,236497531:1048C1371,11395B29,4DC2601,415FAB,1C0C9:-1440211,431542:10AA4D87,66A862F,388838F,348CCD,11919D:F69C5DDD,1C5CBE439,2FF9,655,3E470D,11DFDD,12E47C01
12213311,17980164:DF3,3F1,1D71,1CFC1539,17BA9CD1,30383F:13BBD45D,63D9,26D5,565,267AF3D,13D092D,8CF335,5904D,12E47C01
3312109,19371270:99E70D45,1996DCB,5EBC01,31D3B3,F8CB:22FE5F21,DBC93D,2268AD,1CC9ED,1584FD,71035,99AD,12E47C01

I re-sieve q from 316724521 to 316724621, the relations are correct, the right five relations are

-279653911,270203771:2C47EC05,5455,5117,D955FD7,792367,6CA4A7:26D57BED,8D97BD69,E1EA112D,655,FBBBA0D,19C8371,13541B1,C33305,12E0D529
355978387,122689099:821,570B,1EC7,882D177,10E0F7,4467B,8C5D:1DFEA7DD,96D50931,8AD,B5D2F55,594CBB9,FF9539,8E8D85,528D49,20E55,12E0D529
-128782835,236497531:1048C1371,11395B29,4DC2601,415FAB,1C0C9:884E1065,46C9,BB9,D2B0A41,C501139,21E218D,4460B1,20D781,35071,12E0D529
-81467213,471237169:577,7E24763,2028C6B,ABCFFD,56F7B,195EF:16F2EA5D5,88B50671,8E9,5281,DE6A729,A34815,5FAD49,1717A5,C289,12E0D529
78951107,56007340:40D52CC5,73CA173,287E079,601BBF,9BEC7:16A53A29,CED58865,7BFD,2A1D,2635,5CA4235,26D80B1,14062D,12E0D529

I try to reproduce the problem with file R1340L_16e_a_316500000_317000000.out's content is

-279653911,270203771:2C47EC05,5455,5117,D955FD7,792367,6CA4A7:26D57BED,8D97BD69,E1EA112D,655,FBBBA0D,19C8371,13541B1,C33305,12E0D529
355978387,122689099:821,570B,1EC7,882D177,10E0F7,4467B,8C5D:1DFEA7DD,96D50931,8AD,B5D2F55,594CBB9,FF9539,8E8D85,528D49,20E55,12E0D529
-128782835,236497531:1048C1371,11395B29,4DC2601,415FAB,1C0C9:

And with the command
./gnfs-lasieve4I16e -v -f 316500000 -c 500000 -o R1340L_16e_a_316500000_317000000.out -a R1340L_poly.txt -R
But the relations seems correct this time,

-279653911,270203771:2C47EC05,5455,5117,D955FD7,792367,6CA4A7:26D57BED,8D97BD69,E1EA112D,655,FBBBA0D,19C8371,13541B1,C33305,12E0D529
355978387,122689099:821,570B,1EC7,882D177,10E0F7,4467B,8C5D:1DFEA7DD,96D50931,8AD,B5D2F55,594CBB9,FF9539,8E8D85,528D49,20E55,12E0D529
-128782835,236497531:1048C1371,11395B29,4DC2601,415FAB,1C0C9:
2772363,13511359:287F6A29,E3B69429,4EB67B,56DC7,F9FD:BA04BBF9,4DB5FFE9,EB1,216B0C9,48A2D9,FFF29,12E0D529
15943849,3724237:689CBD57,39E5,2F27,6739B8F,2C9EF9,80BD:78892B29,C8B44AB1,85D,1069,47DE495,1A010A9,1C14D,9B8D,12E0D529
-971678175,1191188861:8B5A9FFF,18600551F,D401D,6C4D3,F265:12E965A1D,7A504FF5,E76C0A75,C030961,77A02A5,42681B9,185999,1413C5,12E0D529

The original command I use is
./gnfs-lasieve4I16e -v -f 316000000 -c 1000000 -o R1340L_16e_a_316000000_317000000.out -a R1340L_poly.txt -R
I write this command as a bash file
#!/bin/bash
let StartQ=316000000
let SieveQ=1000000
let EndQ=StartQ+SieveQ
echo "./gnfs-lasieve4I16e -v -f $StartQ -c $SieveQ -o R1340L_16e_a_${StartQ}_${EndQ}.out -a R1340L_poly.txt -R"
./gnfs-lasieve4I16e -v -f $StartQ -c $SieveQ -o R1340L_16e_a_${StartQ}_${EndQ}.out -a R1340L_poly.txt -R

I open and close the dos's bat or Linux's bash file about 50 times (there is a time that I open the bash file twice).
Under Windows I use Control + C, it will ask me whether I want to close the window, I type y,
And the next time I open it, there is no complain.
Under Linux I use Control + C or Control + Z, the programe close immediately, and when next time I
open it, it complains another new line
' Warning: an incomplete line in the original file; if just a few, it's ok, they will be skipped'.

Not sure what happened here, perhaps a parse error.
I'm ready to sieve q from 316724521 to 316963841 again.

[henryzz]

I am struggling to get the code I have to compile. I am getting many multiple definition errors for functions like asm_zero, asm_copy etc.

The strange thing is that I am getting these errors on code that I compiled in 2014.

I have found instructions for compiling on https://www.mersenneforum.org/showth...t=19711&page=4 post 64
The attachment link seems to be broken, so I have attached what I believe to be the referenced file. Can anyone else compile following these instructions?

Once I get it to compile, I believe I have an edited file for the 5f siever.

[frmky]

Originally Posted by henryzz

I am getting many multiple definition errors for functions like asm_zero, asm_copy etc.

That's new in gcc 10 and higher. Use gcc 9 or lower. I haven't taken the time to track that down.

[henryzz]

Originally Posted by frmky

That's new in gcc 10 and higher. Use gcc 9 or lower. I haven't taken the time to track that down.

Thanks for the hint.
I think it is something to do with montgomery_mul.h being referenced from multiple places. It seems to be in both strategy.o and liblasieve.a.

[bsquared]

The same thing happened in yafu when I switched to gcc 11 from 9. It has to do with the function pointers "declared" in the header of montgomery_mul.h. I put that in quotes because I guess newer gcc's don't consider those declarations anymore. ICX also throws errors. ICC is fine with it. I'll have to look up how I fixed that in yafu; I hope it would work here too.

[SethTro]

Originally Posted by bsquared

For GGNFS that time is labeled MPQS and is reported when running with -v. Posts 34 and 48 show examples where this time is a noticeable fraction of the overall time. In those examples it was 8% and 3%, respectively, when using my new AVX512 vectorized ECM code. Without that it would be a little higher. When the large prime bounds are large and three large primes are allowed on one side or the other is when you'll run into this case. For smaller jobs that don't use 3LP, the MPQS (cofactorization) time is usually negligible (e.g, post 1 @ 420 bits where it is 0.2%).

So it may indeed help on big jobs. The key will be finding out if device transfer overhead ruins the benefits or not. And also working the gpu into the overall 3LP factorization strategy, which has a number of steps. A stage 2 algorithm in the gpu is also probably a must.

For the biggest of jobs, I've been thinking about ways to get Jasonp's Batch GCD method involved. I use it in yafu's 3LP variation for a really nice speedup.

Thank you for the detailed explanations in your responses.

It doesn't sound like it would be a big improvement at the current time. If at some future point this number is larger (15%? 25%?) It might be worth the work to extract the cuda code from gmp-ecm, make it support multiple numbers at the same time, and tune it to minimize setup overhead.

[Gimarel]

Originally Posted by frmky

That's new in gcc 10 and higher. Use gcc 9 or lower. I haven't taken the time to track that down.

Originally Posted by gcc-10 release notes

GCC now defaults to -fno-common. As a result, global variable accesses are more efficient on various targets. In C, global variables with multiple tentative definitions now result in linker errors. With -fcommon such definitions are silently merged during linking

It seems that the option -fcommon should solve the problem.

[bsquared]

Originally Posted by Gimarel

It seems that the option -fcommon should solve the problem.

This worked for me to build the e-versions using ICX, thank you!

lasieve5_64 is now checked in and has all of the same optimizations as lasieve4_64. Plus a couple more that provide maybe another few percent speedup.

What I should have done is make the changes to the cweb files in lasieve 5 instead of working with the ctangled code. That is a chore. I was porting from ctangled code in lasieve4, so that's what made sense at the time. I might circle back and see if I can do that.

Probably next on the agenda is seeing about a windows build, and doing some more testing.

[bsquared]

Fixed a silly bug that was preventing a small number of relations from being discovered.

As a test I ran all special Q from 316000000 to 316100000 using wreck's poly for R1340L with the lasieve5_64 16e siever with the new code and the original code.

Original:
123939 relations found in 174079 core seconds.

New:
123689 relations found in 139200 core seconds.

A 20% speed improvement at a cost of 250 (0.2%) missing relations.

I verified that all relations from the new version are the same as from the original. All but 3 of the missing relations were 3LP's. The other 3 were 62+ bit 2LP's. These latter 3 could be picked at very low cost up by adding a backup method to the 2LP factorization strategy if uecm fails.

[unconnected]

Hi, I've tried new binaries and got an error after several successful factorizations.
For the reference - I use old good factMsieve.pl script which getting .poly-file or even number and do the rest
Poly for c103 number:

$ cat c103.poly
n: 1525306632396272654256201398577639453116991431104195043037446924921873712961837838587765194333349697233
skew: 2272170.41
c0: 44470936714402386191207041425
c1: 64665054240226483277526
c2: 39035884984131219
c3: -20580193202
c4: 9240
Y0: -3584449133208765940001434
Y1: 16673014653679
rlim: 2330000
alim: 2330000
lpbr: 26
lpba: 26
mfbr: 52
mfba: 52
rlambda: 2.5
alambda: 2.5
type: gnfs
qintsize: 50000

One of the job-files (these are intermediate files which used by the main script to sieve):

$ cat c103-2.job.T1
n: 1525306632396272654256201398577639453116991431104195043037446924921873712961837838587765194333349697233
m: 
c0: 44470936714402386191207041425
c1: 64665054240226483277526
c2: 39035884984131219
c3: -20580193202
c4: 9240
Y0: -3584449133208765940001434
Y1: 16673014653679
skew: 2272170.41
rlim: 2330000
alim: 1315000
lpbr: 26
lpba: 26
mfbr: 52
mfba: 52
rlambda: 2.5
alambda: 2.5
q0: 1315001
qintsize: 12499
#q1:1327500

Error:

$ ./gnfs-lasieve4I12e -k -o test.out -v -n0 -a c103-2.job.T1
gnfs-lasieve4I12e (with asm64,avx-512 mmx-td,lasetup,lasched,sieve1,ecm,tds0,search0,tdsched): L1_BITS=15
FBsize 100777+0 (deg 4), 171534+0 (deg 1)
Sorted factor base on side 0: 1: 32495 2: 32397
Sorted factor base on side 1: 1: 168023
total yield: 56627, q=1319407 (0.00110 sec/rel; ETA 0h01m)  td error : 5081 does not divide
Aborted (core dumped)

Error is also reproducible with lasieve4 binary (but on another job-file):

$ ./gnfs-lasieve4I12e -k -o test2.out -v -n0 -a c103-2.job.T2
gnfs-lasieve4I12e (with asm64,avx-512 mmx-td,lasetup,lasched,sieve1,ecm,tds0,search0,tdsched): L1_BITS=15
FBsize 93653+0 (deg 4), 171534+0 (deg 1)
I,J: 11,10, n_i,n_j = 2048,1024, j_per_strip,n_strips=16,64
total yield: 67817, q=1233781 (0.00114 sec/rel; ETA 0h01m)  td error : 5081 does not divide
Aborted (core dumped)