Improved params files for CADO

[bur]

Yes, I'll go with msieve for filtering/matrix building since as mentioned in the other thread I ran into a memory problem with CADO in that stage before. Even though unfortunately it seems msieve is less efficient in filtering, I always found it to require more rels than CADO to build a matrix. Which is why you suggested 40M additional rels?

That means in total it will likely take a lot longer (25% more sieving time maybe?) than using CADO exclusively, but I think I have no choice due to the limited RAM. If RAM wouldn't be of concern, would strategy2 / 240M rels / msieve still be faster than strategy1 / 210M rels / CADO?

[charybdis]

No, I'm not suggesting that msieve will require 40M more relations than CADO to build a matrix, nor am I suggesting that you actually keep sieving until reaching 250M relations.

The issue is that if CADO starts filtering when you reach 210M, the server may crash due to not having enough memory, and it won't resume sieving until you restart the server yourself. So it's better to set rels_wanted artificially high so that CADO doesn't start filtering itself, and to run msieve filtering manually when you get to ~210M relations. For example, you reach 210M relations overnight, and in this case you'd rather start msieve filtering with 220M relations in the morning than have CADO crash overnight and you're still stuck on 210M. If you find that you need to sieve a bit more, you can restart the CADO server (making sure to set tasks.wutimeout high enough that the tasks you interrupted don't expire).

Msieve linear algebra is much faster than CADO, so even if memory is not an issue it is still better to wait until msieve can build a matrix. In addition, you may find that when you only just have enough relations to build a matrix, you will save more time in linear algebra by collecting 5-10M more relations and getting a smaller matrix than you spend sieving to collect those relations. This is the logic behind VBCurtis's tasks.filter.required_excess = 0.05, which tells CADO to continue sieving if it can only just build a matrix.

[EdH]

IIRC, when I had trouble with the square root memory crash some time ago, I was able to recover by adding a large swap file. Would such a swap file allow CADO-NFS to perform the earlier portions of filtering/LA?

[VBCurtis]

Filtering: yes. I had an nvme-SSD swap file when I did the C207 job, and CADO filtering ran about 50x slower than all-RAM. I think CADO was trying to use ~90-100GB on a 64GB machine.

Matrix: You're better off massively oversieving to create a matrix that fits in RAM than trying to make LA run using swap. This might happen to you if you tried your farm on e.g. a C200+.

[bur]

I see, thanks. I know about over-sieving and matrix time, though it gets less valuable the fewer cores you have for sieving. And for some jobs on a 10-core it felt to me as if just going with CADO and less rels was faster than using msieve despite the slower LA, but I can't quantify it.

So using these parameters is considered the best choice?

tasks.A = 28
tasks.qmin = 15000000
tasks.lim0 = 50000000
tasks.lim1 = 70000000
tasks.lpb0 = 31
tasks.lpb1 = 32
tasks.sieve.mfb0 = 58
tasks.sieve.mfb1 = 63
tasks.sieve.ncurves0 = 20
tasks.sieve.ncurves1 = 25
tasks.sieve.adjust_strategy = 2

Would it be efficient to do test-sieving for strategy 1 vs 2?


I have a poly with cownoise score of 3.7e-13. The record is 3.9e-13, is that close enough or should I search some more?

[VBCurtis]

Those look good to me! I don't think test-sieving the strategies is worthwhile; expect strategy 2 to be 3-5% faster over the entire job, though it may not be faster at the lowest Q's.
I'd start trying to build an msieve matrix around 205M relations, and I'd want a matrix 11M or smaller to stop sieving.
Good luck!

Edit: a poly score within 5% of the record is easily good enough to sieve; we can't break records on every job, especially if the number does not begin with a 1.

[bur]

Ok, I'll do that. I did some more poly searching today and up to 3e6 nothing else came up, so I guess that's good enough.


You didn't specify lambda, should I leave it to CADO?

[VBCurtis]

Originally Posted by bur

You didn't specify lambda, should I leave it to CADO?

Yes. Using tight lambda settings seems to boost speed on small jobs with loose large-prime bounds- my files use LP bounds often 2 bits larger than tradition / GGNFS-factmsieve specify, but tight-lambda helps less when LP choice is relatively smaller compared to lim size.

As our job sizes grow, our LP choice gets ever closer to "standard" choices because the extra matrix complexity from larger large-prime relations costs more than the boost to sieve performance. Instead of using lambda to control which cofactors to split, I use a small mfb choice- note that 2 * 31 is 62, but mfb0 is 58. Setting mfb0 to 59 might be faster, but that's not easily testable because one expects to need more relations when one's relations contain (on average) larger large-prime primes.

[bur]

Thanks for the info, I still don't know enough about GNFS to understand why the settings are what they are. I get the general concept, but things like the factorbase I'll have to do some more reading about.


Sieving is going fine so far, 9.4M relations from the first 2M q. Current ETA (for 240M rels) is Jan, 10th, so all in all I expect Jan, 19th or so for the first attempt at a matrix.

[VBCurtis]

I finally have the time, human and machine, to finish up some comparisons and get files published for c145 to c170. While I haven't done enough tests at these sizes to be confident there aren't speed gains to be found (for one, I haven't determined the right size to start using 3LP, because I haven't tested yet), it benefits the community to get "pretty good" params out there while we search for greatness.

Over the next week or so, I'll get drafts published for review by the regulars who may have tried other settings, or just want to suggest alternatives to test. First up is c145, attached here.

[EdH]

I'm definitely interested in c170. I recently ran a c172 with the default params and it took about twice what I expected. I modified the default for next time, trying to set values between what I have for the c165 and c175. I was stuck with whether to choose siever 14 or 15, though.