CADONFS Data Harvesting
This is mostly a question for VBCurtis:
If I'm only running CADONFS through sieving, are there any bits of timing data vs. composite sizes you may be interested in? I won't be able to flag whether I'm using a params file modified by you or me, or an original, but I can probably harvest certain details contained in the log file, or even from the snapshot. This is the CADONFS portion of a normal run for my scripts:  CADONFS is called by a script and given a few standard items. The rest are supplied by the params files.  CADONFS performs the Polyselect and Lattice Sieving via server/clients.  A side script is started (depending on the composite being >125 digits) that examines the relations and performs Msieve filtering until a matrix can be built.   Once successful, CADONFS is told to shut down.    The shutdown may occur anywhere from Lattice Sieving to LA.  If the composite is <125 digits, CADONFS completes the factorization. In light of the fact the process may or may not get to/through filtering, is there info that would be of value to gather? 
Yes, if you can connect the timing data to the params used for the job and to the composite size (including first digit).
Also, small jobs are easy to collect data, and I think I'm done with params below 135140 digits. So, maybe only for 150+ digit jobs? That data can be used to build a bit of a size  vs  sievetime curve, and any outliers mean "find better params for that size, please." If you'd like to do that, I'll be happy to review the data to see where I likely have suboptimal params. 
It's only for a c140, but how would this look for data:[code]N = 336... <140 digits>
tasks.I = 14 tasks.lim0 = 8800000 tasks.lim1 = 14400000 tasks.lpb0 = 30 tasks.lpb1 = 31 tasks.qmin = 900000 tasks.filter.target_density = 130.0 tasks.filter.purge.keep = 180 tasks.polyselect.P = 182500 tasks.polyselect.admax = 146e3 tasks.polyselect.admin = 1800 tasks.polyselect.degree = 5 tasks.polyselect.incr = 120 tasks.polyselect.nq = 15625 tasks.polyselect.nrkeep = 66 tasks.polyselect.ropteffort = 16 tasks.sieve.lambda0 = 1.82 tasks.sieve.lambda1 = 1.81 tasks.sieve.mfb0 = 56 tasks.sieve.mfb1 = 58 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 24 tasks.sieve.qrange = 10000 Polynomial Selection (root optimized): Total time: 6113.74 Polynomial Selection (root optimized): Rootsieve time: 6112.81 Generate Factor Base: Total cpu/real time for makefb: 22.71/1.43919 Lattice Sieving: Total number of relations: 100246494 Lattice Sieving: Total time: 322777s Filtering  Duplicate Removal, splitting pass: CPU time for dup1: 245.5s[/code]Anything missing or not really of interest? 
[QUOTE=EdH;603076][code]Lattice Sieving: Total time: 322777s[/code][/QUOTE]
Unhelpfully, while the total time for a completed job is given in wallclock time and in threadtime, this value for the sieving step is neither of these: it's clienttime, so should be about threadtime/2 unless you've changed threadsperclient from the default. Just something that needs to be kept in mind when making comparisons. 
[QUOTE=charybdis;603079]Unhelpfully, while the total time for a completed job is given in wallclock time and in threadtime, this value for the sieving step is neither of these: it's clienttime, so should be about threadtime/2 unless you've changed threadsperclient from the default. Just something that needs to be kept in mind when making comparisons.[/QUOTE]I'm a tiny bit confused, but all my clients are based on four threads, except an occasional two thread machine that is very rarely engaged. I could make that rarely into never without issue, since it is also my GPU machine and I'm normally doing other stuff with it. Are you saying I should divide this value by 4? Or, should I just note on that line, 4 threads per client?
Also, would it be helpful if I translate that value into 89:39:37? 
A note of 4 threads per client is enough I can double the time if I compare to my own machines, or just leave it asis when comparing to other runs of yours.
I do think you should have everything run 4threaded so that the mix of 2threaded clients and 4threaded clients doesn't dirty the data. Polyselect params aren't really of interest, but poly score is of interest. Alas, Cado's score report is only comparable to other polys that use the exactsame siever & lim's, which is annoying. Poly select time is useful, as I am still not convinced I'm doing the right amount of poly select relative to sieve time! You may be using some older params files before I learned that the ratio of last Q sieved to startingQ should be no more than 8. If you notice any finalQ that's much higher than 8x the startingQ for that params file, boost startingQ accordingly. I'm seeing best performance with this ratio around 6 for C140+ jobs, a bit higher ratio works fine for smaller jobs. 
I was also thinking a note about clients would be better, since then you know. If I simply adjusted the value, we'd never be sure. And, I can leave out the 2thread easy enough, although I wonder the actual effect of one, 2thread client alongside 5779, fourthread clients. Then, again, what's its contribution among the set? I doubt it would be missed.
Should I add in the full polynomial? There are at least two different polynomials (of three) in my current sample. I would expect the final one to be the one used. I should be able to harvest that. (I'm pretty sure) I could add in a cownoise score. I'm currently running a c160, that is supposed to finish tonight. I can start using it as my sample, instead of the current c140. If I understand, I can drop all tasks.polyselect values. Is there interest in the separate Rootsieve time? What about the Factor Base value? ETA: If I achieve my goal of having CADONFS continue sieving until I tell it to stop, there will be no filtering time. I will probably remove that item from my data list. 
If there's a cownoise score, the actual poly has no use to the data summary.
I agree that 12 twothreaded instances won't color the data from a 50+ client farm! 
Here's another run against the c140:[code]N = 336... <140 digits>
tasks.I = 14 tasks.lim0 = 8800000 tasks.lim1 = 14400000 tasks.lpb0 = 30 tasks.lpb1 = 31 tasks.qmin = 900000 tasks.filter.target_density = 130.0 tasks.filter.purge.keep = 180 tasks.sieve.lambda0 = 1.82 tasks.sieve.lambda1 = 1.81 tasks.sieve.mfb0 = 56 tasks.sieve.mfb1 = 58 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 24 tasks.sieve.qrange = 10000 Polynomial Selection (root optimized): Total time: 6113.74 Lattice Sieving: Total number of relations: 100246494 Lattice Sieving: Total time: 322777s (all clients used 4 threads) Found 55764577 unique, 23084551 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 2.51691527e11[/code]The discrepancy in relations counts is due to the filtering tests for Msieve running while CADONFS is still sieving. Would a ratio of duplication be a less confusing value? Or, would you rather be able to look at the actual numbers as shown? 
[QUOTE=EdH;603094]Is there interest in the separate Rootsieve time?[/QUOTE]
Surely there's not much point in reporting it unless you report the time for the rest of polyselect too? 
[QUOTE=charybdis;603115]Surely there's not much point in reporting it unless you report the time for the rest of polyselect too?[/QUOTE]Found it! Is this better?:[code]N = 336... <140 digits>
tasks.I = 14 tasks.lim0 = 8800000 tasks.lim1 = 14400000 tasks.lpb0 = 30 tasks.lpb1 = 31 tasks.qmin = 900000 tasks.filter.target_density = 130.0 tasks.filter.purge.keep = 180 tasks.sieve.lambda0 = 1.82 tasks.sieve.lambda1 = 1.81 tasks.sieve.mfb0 = 56 tasks.sieve.mfb1 = 58 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 24 tasks.sieve.qrange = 10000 Polynomial Selection (size optimized): Total time: 29411.9 Polynomial Selection (root optimized): Total time: 6113.74 Lattice Sieving: Total number of relations: 100246494 Lattice Sieving: Total time: 322777s (all clients used 4 threads) Found 55764577 unique, 23084551 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 2.51691527e11[/code] 
Today is not turning ot to be a "better day!" I'm causing duplication of wok in another thread, and now I'm finding out that if CADONFS is told to stop prior to its filtering, it doesn't give a tme for las. I will have to sort out something else. For now, the c160 will not be useful for much. I might just take a break. . .

Maybe I've found a solution. Here's the data for the c160:[code]N = 516... <160 digits>
tasks.I = 14 tasks.lim0 = 45000000 tasks.lim1 = 70000000 tasks.lpb0 = 31 tasks.lpb1 = 32 tasks.qmin = 17000000 tasks.filter.target_density = 150.0 tasks.filter.purge.keep = 190 tasks.sieve.lambda0 = 1.84 tasks.sieve.mfb0 = 59 tasks.sieve.mfb1 = 62 tasks.sieve.ncurves0 = 20 tasks.sieve.ncurves1 = 25 tasks.sieve.qrange = 10000 Polynomial Selection (size optimized): Total time: 505021 Polynomial Selection (root optimized): Total time: 26267.6 Lattice Sieving: Total number of relations: 212441669 Lattice Sieving: Total time: 3.16477e+06s (all clients used 4 threads) Found 149733097 unique, 40170110 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 1.43789954e12[/code] 
Here's a c162:[code]N = 235... <162 digits>
tasks.I = 14 tasks.lim0 = 45000000 tasks.lim1 = 70000000 tasks.lpb0 = 31 tasks.lpb1 = 32 tasks.qmin = 17000000 tasks.filter.target_density = 150.0 tasks.filter.purge.keep = 190 tasks.sieve.lambda0 = 1.84 tasks.sieve.mfb0 = 59 tasks.sieve.mfb1 = 62 tasks.sieve.ncurves0 = 20 tasks.sieve.ncurves1 = 25 tasks.sieve.qrange = 10000 Polynomial Selection (size optimized): Total time: 508246 Polynomial Selection (root optimized): Total time: 25518.1 Lattice Sieving: Total time: 3.77171e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 218448391 Found 149733097 unique, 40170110 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 1.16869325e12[/code] 
Here's a c168:[code]N = 385... <168 digits>
tasks.I = 14 tasks.lim0 = 65000000 tasks.lim1 = 100000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 10000000 tasks.filter.target_density = 170.0 tasks.filter.purge.keep = 160 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 60 tasks.sieve.ncurves0 = 19 tasks.sieve.ncurves1 = 25 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 999726 Polynomial Selection (root optimized): Total time: 6873.68 Lattice Sieving: Total time: 6.3694e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 179907757 Found 149733097 unique, 40170110 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 5.83275752e13[/code] 
Here's a c161:[code]N = 235... <161 digits>
tasks.I = 14 tasks.lim0 = 45000000 tasks.lim1 = 70000000 tasks.lpb0 = 31 tasks.lpb1 = 32 tasks.qmin = 17000000 tasks.filter.target_density = 150.0 tasks.filter.purge.keep = 190 tasks.sieve.lambda0 = 1.84 tasks.sieve.mfb0 = 59 tasks.sieve.mfb1 = 62 tasks.sieve.ncurves0 = 20 tasks.sieve.ncurves1 = 25 tasks.sieve.qrange = 10000 Polynomial Selection (size optimized): Total time: 493855 Polynomial Selection (root optimized): Total time: 27925.7 Lattice Sieving: Total time: 2.84944e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 202173233 Found 149733097 unique, 40170110 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 1.47600121e12[/code] 
I'm currently running a c164 with [C]A = 28[/C] and [C]adjust_strategy = 2[/C]. Will the data from this one compare to the data from others. What additional things might I need to mention, if any?

I don't think any. We're looking for deviances from the trendline of "twice as hard every 5.5 digits". When a job is above that trend, it's a sign the params for that job size might benefit from some more attention.
At least, that's what I've found working my way up from 95 to 150 digits. Charybdis occasionally runs two jobs very similar in length with one setting changed between them, as an A/B comparison to determine which setting is better. This is timeconsuming at 160+, but it's the sort of work that lets us refine the params set. For instance, we *still* don't have a clear idea of when 3LP pulls ahead of 2LP for CADO. In principle, there should be a single cutoff above which we always use 3LP. If you find yourself running a second job in the 160s the same size as one you've already documented, give 3LP a shot (I can be more specific on settings if you like). 
I've got a pretty large pool of numbers I'm playing with. If you can get me specific params you'd like me to use, I'll try them on another 164 digit or close. The current one is 345. . . I've got about a dozen to check for something close, that I'll hope (ironically) doesn't ECM. I have no idea how to use 3LP, so please be quite specific in what I should do.

[QUOTE=EdH;604411]I've got a pretty large pool of numbers I'm playing with. . .[/QUOTE]Of course that meant that all the c164s are falling to ECM, now.* If I don't find a suitable c164, would you prefer I move up or down a digit? The leading digits are 345... on the current one, which should be finished tomorrow.
[SIZE=2]* The best way to get them to succeed at ECM is to look for GNFS candidates, unless you actually try that. . .[/SIZE] [SIZE=2] [/SIZE][SIZE=2]Edit: By posting the above I hope that the final c164 will fail ECM.[SIZE=1] But then because I posted such, it will succeed. But. . .:smile:[/SIZE] [/SIZE] 
Here's my 3LP settings for params.c165, tested exactly once:
[code]tasks.I = 14 tasks.qmin = 10000000 tasks.lim0 = 40000000 tasks.lim1 = 60000000 tasks.sieve.lambda0 = 1.83 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 88 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 tasks.sieve.rels_wanted = 175000000[/code] If you can compare sieve time to your current settings, that would help us decide if C165 is big enough to run 3LP. If you have multiple jobs, please also try mfb1=89, as 88 might be too small. 3LP makes the sieve faster, at the expense of a jump in matrix size. It's not to our benefit to log a 10% improvement in sieve time if we lose 50% to matrix time! Hopefully that's an exaggeration, but that's why we take data. 
Are you sure that swapping the lims won't improve yield? I thought larger lim on the 2LP side was pretty well established by now. Too lazy to dig up an old polynomial and testsieve it myself.

[QUOTE=VBCurtis;604513]Here's my 3LP settings for params.c165, tested exactly once:
. . ..[/QUOTE]Sorry for my 3LP ignorance, but will these params tell it to use 3LP or do I need to add something else? Also, do you need the full 175M relations? If Msieve successfully filters earlier than 175M, do you still want the rest? BTW, I found a c164 candidate. I think it has a 6 leading digit. The current c164 should be done tomorrow, so I can start the 3LP job after that. 
[QUOTE=charybdis;604514]Are you sure that swapping the lims won't improve yield? I thought larger lim on the 2LP side was pretty well established by now. Too lazy to dig up an old polynomial and testsieve it myself.[/QUOTE]
Definitely not sure. With GGNFS that's clear, but CADO sieves below the factor base so I didn't make any assumptions. Ed Nothing else needs to be changed. mfb at 88 is the key setting that causes 3LP (any setting larger than 3 * log_2(lim) will do it). I don't mind if you don't get to 175M; whatever your scripts do is just fine with me all the better to compare to a previous run with your script. May wish to swap lim's per Charybdis' suggestion, though. 
Haven't done any big GNFS jobs myself for a while, but larger lims on the 2LP side definitely sieve better for the big SNFS jobs I've been doing instead. I think we used larger lims on the 2LP side for 3,748+ too.

Here's the first c164 (note: A=28 and adjust_strategy=2):[code]N = 345... <164 digits>
tasks.lim0 = 50000000 tasks.lim1 = 70000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 10000000 tasks.filter.target_density = 170.0 tasks.filter.purge.keep = 160 tasks.sieve.lambda0 = 2.07 tasks.sieve.lambda1 = 2.17 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 61 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 25 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 529277 Polynomial Selection (root optimized): Total time: 31468 Lattice Sieving: Total time: 4.6221e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 171561952 Found 149733097 unique, 40170110 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 8.37946014e13[/code]Anything else I should grab before I do the next one? 
One last question:
Should I have a [C]tasks.sieve.lambda1[/C] value? I currently have 2.17 (as can be seen above). Should I just keep that? 
No lambda if you did use one, it would have to be close to 3 for 3LP. Best leave it default.

[QUOTE=VBCurtis;604538]No lambda if you did use one, it would have to be close to 3 for 3LP. Best leave it default.[/QUOTE]
Thanks! It is in work. Here are the significant parts of the snapshot:[code]N = 685. . .<164 digits> tasks.I = 14 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 10000000 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 88 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 tasks.sieve.rels_wanted = 175000000[/code] 
Here is the latest c164:[code]N = 685... <164 digits>
tasks.I = 14 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 10000000 tasks.filter.target_density = 170.0 tasks.filter.purge.keep = 160 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 88 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 526394 Polynomial Selection (root optimized): Total time: 31614.9 Lattice Sieving: Total time: 4.67967e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 175012772 Found 149733097 unique, 40170110 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 8.31589954e13[/code]I don't see too much difference. Murphy_E is lower and sieving took a little bit longer. However, the previous c164 used strategy 2, while this one did not. What effect would that have had at this size? If you like, provide some changes and I'll put them in the params file for the next ~c165 composite. It may not be real soon, but maybe this upcoming week. 
Try with strategy2, please? I don't use that setting because it seems to trigger errors with CADO postprocessing, so I forgot to include it for you.
My guess is 4% faster from strat2? Your next data point will tell us. :) Was the resulting matrix notably bigger than your previous C164? 
[QUOTE=VBCurtis;604613]Try with strategy2, please? I don't use that setting because it seems to trigger errors with CADO postprocessing, so I forgot to include it for you.
My guess is 4% faster from strat2? Your next data point will tell us. :) Was the resulting matrix notably bigger than your previous C164?[/QUOTE]I'll run the next one with A=28 and strategy 2, then.* I didn't bring up strategy 2 because you used I=14. Here are the matrix sections from the two logs  first c164:[code]Thu Apr 21 08:30:37 2022 matrix is 9822977 x 9823172 (3015.8 MB) with weight 932199937 (94.90/col) Thu Apr 21 08:30:37 2022 sparse part has weight 672685141 (68.48/col) Thu Apr 21 08:32:25 2022 filtering completed in 2 passes Thu Apr 21 08:32:27 2022 matrix is 9792967 x 9793156 (3013.3 MB) with weight 931103496 (95.08/col) Thu Apr 21 08:32:27 2022 sparse part has weight 672400393 (68.66/col) Thu Apr 21 08:33:10 2022 matrix starts at (0, 0) Thu Apr 21 08:33:11 2022 matrix is 9792967 x 9793156 (3013.3 MB) with weight 931103496 (95.08/col) Thu Apr 21 08:33:11 2022 sparse part has weight 672400393 (68.66/col) Thu Apr 21 08:33:11 2022 saving the first 48 matrix rows for later Thu Apr 21 08:33:12 2022 matrix includes 64 packed rows Thu Apr 21 08:33:13 2022 matrix is 9792919 x 9793156 (2895.1 MB) with weight 745879127 (76.16/col)[/code]and, the second c164:[code]Sat Apr 23 07:29:12 2022 matrix is 10949079 x 10949259 (3349.2 MB) with weight 1042919866 (95.25/col) Sat Apr 23 07:29:12 2022 sparse part has weight 746571916 (68.18/col) Sat Apr 23 07:32:13 2022 filtering completed in 2 passes Sat Apr 23 07:32:17 2022 matrix is 10934410 x 10934588 (3348.1 MB) with weight 1042422445 (95.33/col) Sat Apr 23 07:32:17 2022 sparse part has weight 746467122 (68.27/col) Sat Apr 23 07:33:16 2022 matrix starts at (0, 0) Sat Apr 23 07:33:19 2022 matrix is 10934410 x 10934588 (3348.1 MB) with weight 1042422445 (95.33/col) Sat Apr 23 07:33:19 2022 sparse part has weight 746467122 (68.27/col) Sat Apr 23 07:33:19 2022 saving the first 48 matrix rows for later Sat Apr 23 07:33:21 2022 matrix includes 64 packed rows Sat Apr 23 07:33:23 2022 matrix is 10934362 x 10934588 (3228.6 MB) with weight 832280012 (76.11/col)[/code]Yes, the matrix is a little bit larger, but is that just due to when Msieve happened to succeed in its filtering tests? * I'm guessing that's the only change you would like for the next ~c164 run (I don't have another c164 handy just yet), or do you want something else modified, too? 
Please try A=28 separately from strat 2. I'd like to know the speed gained from start 2 on I=14.
I expect A=28 would be slower than I=14 here, anyway; perhaps we can testsieve that rather than run a full job. 
[QUOTE=VBCurtis;604640]Please try A=28 separately from strat 2. I'd like to know the speed gained from start 2 on I=14.
I expect A=28 would be slower than I=14 here, anyway; perhaps we can testsieve that rather than run a full job.[/QUOTE]OK. I have a c164 (685...) candidate and I should be able to start it running tomorrow. I'll run I=14 [B]with[/B] adjust_strategy=2. (For some reason, I had it in the back of my head that strategy=2 wouldn't work with I=14.) 
[QUOTE=VBCurtis;604640]I expect A=28 would be slower than I=14 here, anyway; perhaps we can testsieve that rather than run a full job.[/QUOTE]
Unfortunately I don't think that's something you can figure out by testsieving, because A=28 should have a lower duplication rate. The crossover point for "time to find rels_wanted raw relations" is probably a few digits higher than the true crossover. @EdH: I think you might need to take a look at your script, as all your summaries seem to include [code]Found 149733097 unique, 40170110 duplicate, and 0 bad relations.[/code] 
[QUOTE=charybdis;604646]. . .
@EdH: I think you might need to take a look at your script, as all your summaries seem to include [code]Found 149733097 unique, 40170110 duplicate, and 0 bad relations.[/code][/QUOTE]Indeed! I can't find where the report gets written in any of my scripts, but I do have a file with those values from sometime, that I harvest for each run. Thanks for catching that. I will definitely have to work on it. I might have to skip remdups4 and let Msieve report duplication and harvest the values from there. 
OK, I'm losing it! The new candidate is the one I just factored. It got mixed into the list because it wasn't finished yet. I need to do some more work before I get to the next candidate.

I have a c164 underway:[code]N = 712...<164 digits>
tasks.I = 14 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 10000000 tasks.sieve.adjust_strategy = 2 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 88 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 tasks.sieve.rels_wanted = 175000000[/code] 
Here's the next c164 (I=14 and adjust_strategy=2):[code]N = 712... <164 digits>
tasks.I = 14 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 10000000 tasks.filter.target_density = 170.0 tasks.filter.purge.keep = 160 tasks.sieve.adjust_strategy = 2 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 88 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 524425 Polynomial Selection (root optimized): Total time: 30333.8 Lattice Sieving: Total time: 4.46548e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 175001545 Found 122488916 unique, 45564734 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 8.11818879e13[/code] 
Poly score 2.5% worse, but sieve time roughly 5% better. Nice!
The next settings to test are A=28 and mfb1 = 89. A=28 is more important a test (mfb should not change sieve time very much). 
[QUOTE=VBCurtis;604744]. . .
The next settings to test are A=28 and mfb1 = 89. A=28 is more important a test (mfb should not change sieve time very much).[/QUOTE]Are you saying I should only change to A=28 first, or go ahead and change both, and with or without strategy=2? 
I think / hope each change should be independent that is, you have determined strat 2 is faster (really, Charybdis determined this over a year ago), now it's the default. Next, try A = 28; once we know the best setting there, try mfb's.
One change at a time with A/B comparisons give us "clear" evidence for what to use; once the big settings like A and lp are set, the little settings (mfb, starting Q, lambda, target rels) can be dialed in hopes of finding a few more % of speed. 
[QUOTE=VBCurtis;604756]I think / hope each change should be independent that is, you have determined strat 2 is faster (really, Charybdis determined this over a year ago), now it's the default. Next, try A = 28; once we know the best setting there, try mfb's.
One change at a time with A/B comparisons give us "clear" evidence for what to use; once the big settings like A and lp are set, the little settings (mfb, starting Q, lambda, target rels) can be dialed in hopes of finding a few more % of speed.[/QUOTE]I just wanted to make sure I'm on the same page. The only thing I'll change for next time is A=28. Again, I'm out of c164 candidates, I may have some lower c165s. 
[QUOTE=EdH;604764]I just wanted to make sure I'm on the same page. The only thing I'll change for next time is A=28.[/QUOTE]
A=28 should probably have a slightly lower qmin  maybe 7M?  as you'll be sieving a smaller range of Q. We'll see what Curtis says. 
Excellent point we want the ratio of qmin to qmax to be something between 6 and 8. I suspect your C164 with I=14 ran Q roughly 1060M; since A=28 should yield 40% better, a Qrange of 743M might be expected.
So, I agree with Charybdis' suggestion to change qmin to 7M. Good idea! 
[QUOTE=VBCurtis;604771]Excellent point we want the ratio of qmin to qmax to be something between 6 and 8. I suspect your C164 with I=14 ran Q roughly 1060M; since A=28 should yield 40% better, a Qrange of 743M might be expected.
So, I agree with Charybdis' suggestion to change qmin to 7M. Good idea![/QUOTE]It has been written  it will be done.:smile: One more c164 (231...) has shown up in my work listings and I should be able to start it tomorrow afternoon. 
The latest is underway:[code]N = 231...<164 digits>
tasks.A = 28 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.adjust_strategy = 2 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 88 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 tasks.sieve.rels_wanted = 175000000[/code]In my reports, is there any interest in the two "filter" lines? I'm thinking of removing them. 
The filter lines have no effect, since you're using msieve for postprocessing.

[QUOTE=VBCurtis;604815]The filter lines have no effect, since you're using msieve for postprocessing.[/QUOTE]Those were my thoughts, but wondered if they may have any use, anyway. Thanks!

The latest c164:[code]N = 231... <164 digits>
tasks.A = 28 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.adjust_strategy = 2 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 88 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 496571 Polynomial Selection (root optimized): Total time: 31172.8 Lattice Sieving: Total time: 4.28663e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 169728713 Found 119819021 unique, 42731179 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 8.60775398e13[/code] 
Looks like a 6% better scoring poly than the last test, but only 4% lower sieve time.
Nearly a wash, but I=14 is also lower memory while being notslower. 
[QUOTE=VBCurtis;604891]Looks like a 6% better scoring poly than the last test, but only 4% lower sieve time.
Nearly a wash, but I=14 is also lower memory while being notslower.[/QUOTE]Sounds like a suggestion to move back to I=14. I'm thinking I'm going to have to move up or down a digit for my next few tests. Is there a preference if I move to mfb1=89? 
As long as the input number is close, comparing Escore is pretty accurate (as I did in my previous post). It shouldn't matter whether you go with 163 or 165 or 166 next time for the mfb 89 trial.
Note that a larger mfb "should" improve yield, but more raw relations are likely necessary. The tradeoff is murky if sec/rel is not better when testsieving, I go with the smaller mfb. When sec/rel is better, a full factorization is likely to educate us. So, please try a full factorization as you planned. 
All my runs are full runs, unless they break. At these sizes, full runs are less than 48 hours, so my patience is still holding.

I've started a c165. I went back to I=14, but forgot to take qmin back up to 10M. Will that throw things off?:[code]N = 309...<165>
tasks.I = 14 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.adjust_strategy = 2 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 tasks.sieve.rels_wanted = 175000000[/code] 
Not really; you'll have a slightly higher duplicate rate, meaning you may need an extra 13M raw relations to get the matrix size you expect. Sieve time will barely change.

Here's the c165:[code]N = 309... <165 digits>
tasks.I = 14 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.adjust_strategy = 2 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 523377 Polynomial Selection (root optimized): Total time: 35527 Lattice Sieving: Total time: 4.96142e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 175009615 Found 122644730 unique, 53361072 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 7.58493957e13[/code]Although I have another c165 available, I'm going to move back to working on the easier Aliquot sequences list for a little bit, since the number of sequences is diminishing and I want to try to make sure it doesn't get too low. 
If I've got the scaling of poly score vs sieve time, looks like 88 is quicker.

[QUOTE=VBCurtis;604981]If I've got the scaling of poly score vs sieve time, looks like 88 is quicker.[/QUOTE]Not at all empirical, but I had the feeling this was more difficult. I encountered a CADONFS filtering crash (possibly the strategy 2 trouble you mentioned?) and Msieve LA came back with over 32.5 hours ETA with 40 threads. I should probably save a copy of the Msieve log just for later perusal. (I wonder if I should see if a Colab instance would give me a good GPU and what it might show for ETA. . .)
ETA: I'm planning to try the Colab session, but the compression and upload times, as well as Colab inialization, are going to take quite a while, so I'm not sure there's an advantage. ETA2: After nearly an hour and nowhere near completion, I gave up on my msieve.dat.mat.tbz upload to Google Drive for a Colab test. It had already taken over a halfhour to copy and compress it. 
Here's a c162 I just finished:[code]N = 314... <162 digits>
tasks.I = 14 tasks.lim0 = 45000000 tasks.lim1 = 70000000 tasks.lpb0 = 31 tasks.lpb1 = 32 tasks.qmin = 17000000 tasks.sieve.lambda0 = 1.84 tasks.sieve.mfb0 = 59 tasks.sieve.mfb1 = 62 tasks.sieve.ncurves0 = 20 tasks.sieve.ncurves1 = 25 tasks.sieve.qrange = 10000 Polynomial Selection (size optimized): Total time: 494305 Polynomial Selection (root optimized): Total time: 28582 Lattice Sieving: Total time: 3.96443e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 209104683 Found 146771345 unique, 45989579 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 1.11487621e12 [/code]I'm guessing I should adjust some of these values to be closer to the params for the previous c165? 
Looks like 78% slower than the scaling one would expect for a C162 given the timing data from the C165 (that is, using the "double effort every 5.5 digits" heuristic).
I'd expect 3.72 megaseconds based on the 4.96 megasecond time for C165, calculated by inverting the ratio of the poly scores; the run was 3.96 megaseconds, 8% slower than target. This may indicate 3 large primes is valuable on the C160 file, or perhaps some other parameter is not ideal. I'd try the lims and mfbs from the C165 run, with 31/31LP and 5% lower target relations (because the job is smaller/easier). I'd also use the qmin from the C165. Ignoring the changes made for 3LP, I've been scaling the lim's to double every 10 digits. I'm not sure what is best for 3LP cases, but C165 used 60M/40M; perhaps 50M for lim0 makes sense on C160 file. 
Thanks! I've possibly got a c160 (615...) coming up. I think I've got the changes made to the params.* I'll post what turns up.
* I've got a high rels_wanted, but Msieve is going to break in when a matrix is possible. 
Here's the latest:[code]N = 615... <160 digits>
tasks.I = 14 tasks.lim0 = 50000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 20 tasks.sieve.ncurves1 = 25 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 493434 Polynomial Selection (root optimized): Total time: 27103.8 Lattice Sieving: Total time: 3.40375e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 169962886 Found 119992365 unique, 35401629 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 1.64176010e12[/code] 
Hrmmm.... poly score nearly 50% better than the last one, but only 15% less sieve time. That's bad especially when raw relations needed went from 209M to 169M.
Aha! I forgot: Breaking 3LP cofactors uses a small ncurves value. Leaving ncurves1 at 25 may have caused this slow job. I think the CADO default is to use ncurves of 9 or 10 on the 3LP side (2025 on the 2LP side is still good). So, when using mfb1 > 2 * lpb1 which is how we do 3LP on that side, drop ncurves1 from 25 to 10ish. 
I'm running a sequence that is currently at 169 digits on its way down. It has a good chance of providing some more 16x composites. I'll drop ncurves1 to 10 for the next one, which could actually be any of the three param files (160, 165, 170). I'm somewhat set for 160 and 165, but what would you suggest for 170 if I start with the 165 params?

Another c163
[code]N = 344... <163 digits>
tasks.I = 14 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 529204 Polynomial Selection (root optimized): Total time: 30833.4 Lattice Sieving: Total time: 4.95571e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 185882690 Found 125042962 unique, 43355067 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 9.45211518e13[/code] 
Weird; two digits easier than the C165 from a few posts ago, with the same params, yet needed 10M more raw relations, more unique relations, and took the same amount of time as a poly that scored 20%+ worse.

[QUOTE=VBCurtis;607821]Weird; two digits easier than the C165 from a few posts ago, with the same params, yet needed 10M more raw relations, more unique relations, and took the same amount of time as a poly that scored 20%+ worse.[/QUOTE]
Is all of this skewed because of my use of Msieve telling CADONFS when to stop sieving? Edit: Msieve runs filtering to test for a matrix success and then waits a short time and tests again. 
No, I think your way is actually more accurate, since it measures when a matrix can be built of a certain density rather than using userset guesses for number of relations to find.

One more:[code]N = 164... <160 digits>
tasks.I = 14 tasks.lim0 = 50000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 20 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 480487 Polynomial Selection (root optimized): Total time: 27393 Lattice Sieving: Total time: 3.16854e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 166474932 Found 113244194 unique, 35828316 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 1.62682689e12[/code] 
And, a c172:[code]N = 237... <172 digits>
tasks.I = 14 tasks.lim0 = 65000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 19 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 985587 Polynomial Selection (root optimized): Total time: 10003 Lattice Sieving: Total time: 1.28816e+07s (all clients used 4 threads) Lattice Sieving: Total number of relations: 240648323 Found 151223351 unique, 81169538 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 3.16608561e13[/code] 
Here's a c173:[code]N = 107... <173 digits>
tasks.A = 28 tasks.lim0 = 65000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.adjust_strategy = 2 tasks.sieve.mfb0 = 59 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 19 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 994417 Polynomial Selection (root optimized): Total time: 9945.86 Lattice Sieving: Total time: 1.28671e+07s (all clients used 4 threads) Lattice Sieving: Total number of relations: 233696828 Found 145036125 unique, 73955642 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 2.69267673e13[/code]What changes, other than increasing qmin would be helpful? A suggestion for qmin=15M has been made. What about strategy = 2? Is that good here? It seems we removed that in the past. 
[QUOTE=EdH;611693]What changes, other than increasing qmin would be helpful? A suggestion for qmin=15M has been made. What about strategy = 2? Is that good here? It seems we removed that in the past.[/QUOTE]
Strategy 2 is pretty much essential when using even values of A. The performance benefit is over 10% if I remember correctly. As already discussed in this thread, it also tends to be a few percent faster at this size even for odd values of A (i.e. when I is used instead of A). Presumably strategy 2 was removed because of the occasional CADO filtering crashes. But with your script setup that shouldn't be an issue, as long as CADO filtering doesn't run until msieve can build a matrix. Apart from increasing qmin, the other thing to try for small c17x would be raising lpb1 to 32, and correspondingly mfb1 to 92. 
Thanks! I've been trying to adjust rels_wanted to make sure it exceeds Msieve matrix build relations needed, so I'll stick with strategy 2. Part of this is so the CADO filtering isn't fighting for resources while Msieve filtering is testing for a matrix. I'll adjust qmin to 15M, and raise lpb1 and mfb1 for my next c17x run, whenever I do another one.
I do run CADO filtering briefly at the end to gather the data for this thread. At that point, if it crashes, it doesn't bother my scripts, but I might miss some of the values in these posts. I can look at that later if it does occur. 
Here's another c160:[code]N = 126... <160 digits>
tasks.I = 14 tasks.lim0 = 50000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 7000000 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 20 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 488894 Polynomial Selection (root optimized): Total time: 25992.8 Lattice Sieving: Total time: 3.49294e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 172663455 Found 118538364 unique, 41243958 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 1.50177701e12[/code]I have since, increased qmin for both c160 and c165 params. I should have another c165 tomorrow. BTW, I have been harvesting all my runs for a while now. If you have some specific sizes you'd be interested in, let me know and I'll post them. 
Comparing with earlier in the thread, it looks to me like 3LP is slower at c160.

Here's a c165:[code]N = 322... <165 digits>
tasks.I = 14 tasks.lim0 = 60000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 12000000 tasks.sieve.lambda0 = 1.83 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 18 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 513630 Polynomial Selection (root optimized): Total time: 34447.3 Lattice Sieving: Total time: 5.58991e+06s (all clients used 4 threads) Lattice Sieving: Total number of relations: 174610477 Found 128257371 unique, 38152963 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 7.81486672e13[/code]How does this compare? I had less duplication with 12M, but I can't say how time turned out. It seems to be a bit longer than a c163, as expected. You think I should increase qmin some more and/or take mfb1 down a bit? 
[QUOTE=EdH;611938]How does this compare?
I had less duplication with 12M, but I can't say how time turned out. It seems to be a bit longer than a c163, as expected. You think I should increase qmin some more and/or take mfb1 down a bit?[/QUOTE] This is actually a bit slower than your previous c165 from post 57, which had the same params except qmin=7M and strategy 2. My comment (in a PM, I think) about raising qmin was intended for the small c17x numbers, not c160165. Granted, strategy 2 is probably responsible for some of the difference. For 2LP at c165, I'd drop mfb1 to 61 and bump lim1 up to something like 80M to compensate. Also ncurves1 back up to 25. Let's wait to see what VBCurtis thinks. 
[QUOTE=charybdis;611943]This is actually a bit slower than your previous c165 from post 57, which had the same params except qmin=7M and strategy 2. My comment (in a PM, I think) about raising qmin was intended for the small c17x numbers, not c160165. Granted, strategy 2 is probably responsible for some of the difference.
For 2LP at c165, I'd drop mfb1 to 61 and bump lim1 up to something like 80M to compensate. Also ncurves1 back up to 25. Let's wait to see what VBCurtis thinks.[/QUOTE] I think all of this is accurate: 58/61 for mfb's sounds right, with 59/61 also worth a test. Ditto for doubling lim1 for 2LP compared to 3LP. I'm finding best speed/duplicate compromise when Qmax is 6 to 7 times Qmin. If Qmax exceeds 8 * Qmin, I raise Qmin accordingly on the params file for that size. For 31bit LP and no 3LP, I'd use ncurves of 21 and 25, but up or down a couple seems to not matter. Edit: Try lambda0 = 1.86 for mfb0 = 58, and 1.89 for mfb0 = 59. That should improve yield. 
Here's the latest, a c171:[code]N = 902... <171 digits>
tasks.I = 14 tasks.lim0 = 65000000 tasks.lim1 = 40000000 tasks.lpb0 = 31 tasks.lpb1 = 31 tasks.qmin = 15000000 tasks.sieve.mfb0 = 58 tasks.sieve.mfb1 = 89 tasks.sieve.ncurves0 = 19 tasks.sieve.ncurves1 = 10 tasks.sieve.qrange = 5000 Polynomial Selection (size optimized): Total time: 979823 Polynomial Selection (root optimized): Total time: 9797.37 Lattice Sieving: Total time: 1.25403e+07s (all clients used 4 threads) Lattice Sieving: Total number of relations: 225750239 Found 136126593 unique, 61871366 duplicate, and 0 bad relations. cownoise Best MurphyE for polynomial is 2.98902086e13[/code] 
Let me know what size you plan to run next; I'll do my first batch of las testsieving on that size so we can make progress on these settings. I plan to test a bunch of settings, and then have you run a full job on the ones I think are fastest.
There remain good reasons to test full jobs for instance, testsieving doesn't indicate just how many relations will be needed when I change LP bounds. Our experience provides reasonable estimates, but when I'm trying to eke out 5% more speed an estimation error of 5% can swamp whatever speed I think I'm finding. 
All times are UTC. The time now is 02:58. 
Powered by vBulletin® Version 3.8.11
Copyright ©2000  2022, Jelsoft Enterprises Ltd.