Just a Curiosity

[EdH]

I recently rebuilt msieve and YAFU to fix a crash and noticed this:

05/31/20 15:36:25 v1.35-beta @ math27, System/Build Info: 
Using GMP-ECM 7.0.5-dev, Powered by GMP 6.1.2
detected        Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz
detected L1 = 32768 bytes, L2 = 3145728 bytes, CL = 64 bytes
measured cpu frequency ~= 2494.385440

When I ran "tune:"

best exponential fit is y = 0.492662 * exp(0.0994914 * x)
QS/NFS crossover occurs at 100.3 digits
Adding tune_info entry for LINUX64 -        Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz

Tune info in yafu.ini:

tune_info=       Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz,LINUX64,2.54497e-05,0.197911,0.492662,0.0994914,100.294,2494.37

When I ran ./yafu via a Perl script:

fac: using specified qs/gnfs crossover of 93 digits
fac: using specified qs/snfs crossover of 75 digits

Edit: I also just noticed this:

. . .
nfs: commencing polynomial search over range: 17692 - 17942
 deadline: 8640000 CPU-seconds per coefficient
error generating or reading NFS polynomials
nfs: commencing polynomial search over range: 17942 - 18192
deadline: 8640000 CPU-seconds per coefficient
coeff 18060 specialq 1 - 946643 other 6855 - 16452
aprogs: 427 entries, 1206 roots
hashtable: 1024 entries,  0.02 MB
nfs: commencing polynomial search over range: 18192 - 18442
deadline: 8640000 CPU-seconds per coefficient
error generating or reading NFS polynomials
. . .

[bsquared]

I'm seeing normal usage of tune info after testing just now:

fac: using pretesting plan: normal
fac: using tune info for qs/gnfs crossover

Make sure that yafu.ini doesn't have lines containing "xover=" or "snfs_xover=" in it. Yafu will also try to verify that the computer/OS info matches between the current CPU and the tune info text. Based on what you posted they should match, so if it doesn't turn out to be a "xover=" line then I'm not sure what's going on.

I see the same "error generating or reading NFS polynomials" messages during poly search, but it appears to generate polynomials and complete NFS factorizations just fine. I'll file that as low priority for now.

In related news, after the new SIQS updates tune() is now showing a crossover of 106.6 digits for LINUX64! (using a skylake-x.) One should take that number with a large grain of salt given all the assumptions tune() makes, but still, that's higher than I've ever seen it for 64-bit linux ggnfs.

I ran the same C100 through both routines and got 641 seconds for SIQS and 681 for NFS, so 106.6 digits is probably a little optimistic toward SIQS. I'll have to re-visit some of the assumptions tune() makes. (I may have shortened poly select time but not reflected that in tune()).

[EdH]

Originally Posted by bsquared

. . .
Make sure that yafu.ini doesn't have lines containing "xover=" or "snfs_xover=" in it. Yafu will also try to verify that the computer/OS info matches between the current CPU and the tune info text. Based on what you posted they should match, so if it doesn't turn out to be a "xover=" line then I'm not sure what's going on.

I remember leaving those lines in several of my .ini files. These are probably those machines.

I have noticed the latest .ini version you've created. I haven't delved into the inner portions, but is there a reason that "threads" is commented (%) out, as well as being set to 1?

I've been using your msieve modification to cure segfaults on my GMP 6.2.0 machines. Thanks!

And, thanks for all else, as well.

[bsquared]

Originally Posted by EdH

I have noticed the latest .ini version you've created. I haven't delved into the inner portions, but is there a reason that "threads" is commented (%) out, as well as being set to 1?

In my mind, the .ini file should be used to set *permanent* parameters. Things that you always need (for example the ecm path, ggnfs binary folder, and ecm pretesting preferences). Anything in the .ini file can be overridden on the command line. For me, threads is something I frequently change, depending on what I'm trying to do or what networked system I'm currently running on, so I always set that from the command line. Just my preference: customize the .ini as you see fit. The new .ini is simply meant to document the possible options.

[EdH]

Originally Posted by bsquared

In my mind, the .ini file should be used to set *permanent* parameters. Things that you always need (for example the ecm path, ggnfs binary folder, and ecm pretesting preferences). Anything in the .ini file can be overridden on the command line. For me, threads is something I frequently change, depending on what I'm trying to do or what networked system I'm currently running on, so I always set that from the command line. Just my preference: customize the .ini as you see fit. The new .ini is simply meant to document the possible options.

I just remembered something else to bring up, but will have to wait for another tune session to show an example. I get a repetitive skewed error message with the new .ini file. All seems to run fine afterward, though. This was with and without removing the existing tune data. I suppose it might have to do with the threads edit. I'll post an example later, unless it never shows again.

[bsquared]

Originally Posted by EdH

I just remembered something else to bring up, but will have to wait for another tune session to show an example. I get a repetitive skewed error message with the new .ini file. All seems to run fine afterward, though. This was with and without removing the existing tune data. I suppose it might have to do with the threads edit. I'll post an example later, unless it never shows again.

I saw this too. I think it has to do with blank lines in the .ini. I changed the ReadINI function to ignore these but the writeINI function that is called as part of tune did not get changed and doesn't like them. In mine the blank line were removed, in addition to adding the tune_info line, as an after effect of running tune(). It seemed Mostly Harmless ^(TM), but I will try to fix at some point anyway.

If you are referring to something else let me know.

[EdH]

Originally Posted by bsquared

I saw this too. I think it has to do with blank lines in the .ini. I changed the ReadINI function to ignore these but the writeINI function that is called as part of tune did not get changed and doesn't like them. In mine the blank line were removed, in addition to adding the tune_info line, as an after effect of running tune(). It seemed Mostly Harmless ^(TM), but I will try to fix at some point anyway.

If you are referring to something else let me know.

I'm pretty sure that's what it was. The blank line was removed in the one I just ran and there was no error - all looked correct.

[EdH]

Yep, an extra blank line (or two) prior to the previous tune info (and 100.1 digit crossover:)

QS/NFS crossover occurs at 100.1 digits
Invalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywords
. . .<38 copies of these lines>. . .
Invalid line in yafu.ini, use Keyword=Value pairsSee docfile.txt for valid keywords
found OS = LINUX64 and CPU = Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz in tune_info field
Adding tune_info entry for LINUX64 -        Intel(R) Core(TM) i5-2520M CPU @ 2.50GHz

[EdH]

I think your larger crossover might explain the results of an experiment I've been conducting:

Two machines "virtually" identical.

Machine 1: Aliqueit running YAFU for qs, ecm.py for ECM and CADO-NFS for GNFS

Machine 2: Aliqueit running option -y (YAFU for all)

Both were started on an existing sequence that was around 105 digits.

Although Machine 1 seemed to be doing a lot more ECM than Machine 2, it ran away, leaving machine 2 many lines behind. At this time, the sequence had taken a downturn and was shedding digits.

I swapped the setups and copied the longer sequence to Machine 2, so they would start even again. This time Machine 1, running YAFU (-y) ran away by over 30 indices, at first! As the digits rose again, Machine 2 caught up with and passed Machine 1, even though it had been 30 indices behind.

I will have to do some more comparisons to really conclude anything, but the preliminary shows CADO-NFS to be promising. As to my using CADO-NFS, for my Aliqueit runs, I'm implementing distributed polyselect and sieving across >20 machines of various architecture. I was able to do this with factmsieve.py, too, but CADO-NFS is simpler and now it seems faster, also.

I noticed a query as to the possibility of YAFU using CADO-NFS and know this would be quite a bit of work, but from my playing around and other tests, etc., if you ever do consider CADO-NFS, I would suggest still letting Msieve do the LA.

[VBCurtis]

Originally Posted by EdH

..... if you ever do consider CADO-NFS, I would suggest still letting Msieve do the LA.

I think on small inputs the difference in LA time isn't enough to be worth much programming effort, since it's more complicated to use a hybrid versus cado-as-black-box-factorer. I don't bother to break CADO to invoke msieve for LA below 150 digits, and by that size lots of folks aren't using YAFU anyway. msieve may be 2/3rds the time, but the matrix time for a 130 digit number just isn't that long.

[EdH]

Originally Posted by VBCurtis

I think on small inputs the difference in LA time isn't enough to be worth much programming effort, since it's more complicated to use a hybrid versus cado-as-black-box-factorer. I don't bother to break CADO to invoke msieve for LA below 150 digits, and by that size lots of folks aren't using YAFU anyway. msieve may be 2/3rds the time, but the matrix time for a 130 digit number just isn't that long.

But, if the programming takes care of the hybridization, why not try for the fastest factoring possible? I would think if you already have msieve doing LA and you want to inject CADO into a package for polyselect and sieving, it would be best to just inject that faster part. Why cripple LA, even a little?