20191107, 06:30  #1 
Feb 2011
2^{5} Posts 
trivial snfs bug; advice ~9 days into job
As I mentioned in the other subforum, I decided to try snfs for the first time, on the 203 digit cofactor of the 235digit Pell(613). At first I didn't put the difficulty in my job file, and so I found that while factor/nfs/nfs.c tells you
Code:
nfs: detected snfs job but no snfs difficulty; assuming size of number is the snfs difficulty (BTW why not make some attempt to calculate snfs difficulty using the polynomials (as factmsieve does) rather than make such an assumption?) After I put size:235 in and ran a short test, I wrote that post asking for advice re params, but I didn't get much. I tried perturbing a couple parameterss away from yafu's defaults and doing nt test sieving, but the defaults appeared best so I stuck with them. Maybe this isn't optimal, e.g. I found this thread suggesting lp31. A bit late to change now. I now have half the required relations. The sieving slowed down quite a bit, with the reported ETA staying almost constant (~330h) for around 7 days and only then beginning to drop, and I thought perhaps I'd try speeding things up a bit by doing some work on another computer and copying that to rels.add. But it looks like the only mechanism for telling one instance of yafu to do a different range is ns start,end, which ends up with each siever sieving the entire (endstart)/#threads at once rather than biting off manageable chunks and checkpointing progress as nfs() normally does. Any recommendations on doing a rudimentary job of distributing some work? 
20191107, 13:45  #2 
Jun 2012
2×5×17^{2} Posts 
Can you post your nfs.job and Yafu.ini files?
What command did you use to invoke Yafu? Yafu is a great tool, but it’s parameters often need tweaking at higher difficulties. The above files should help us understand what is happening with your factorization. 
20191107, 16:58  #3 
Feb 2011
2^{5} Posts 
The poly and skew are the same as in the other thread, so the job file just adds size:235 and the parameters yafu filled in by default:
Code:
n: 41992954986653668251498160346091865848259963960853060020750860632150880435788410917184988498688690850646615751221247309950434587076089709352088108916792779996577130820982289749315443146420724344354422429 Y1: 390458526450928779826062879981346977190 Y0: 942650270102046130733437746596776286089 skew: 1.75 type: snfs size: 235 c6: 1 c4: 15 c3: 40 c2: 75 c1: 72 c0: 29 rlim: 41800000 alim: 41800000 lpbr: 30 lpba: 30 mfbr: 60 mfba: 60 rlambda: 2.6 alambda: 2.6 Code:
B1pm1=100000 B1pp1=20000 B1ecm=11000 rhomax=1000 threads=8 pretest_ratio=0.25 v=1 ggnfs_dir=C:\Users\Dan\Downloads\factor\GGNFS\ ecm_path=C:\Users\Dan\Downloads\factor\gmpecmsvn3052znver1\ecm.exe tune_info=AMD Ryzen 5 2400G with Radeon Vega Graphics ,WIN64,9.0141e006,0.205255,0.280054,0.10029,98.5466,3594.43 Code:
.\yafu "nfs(41992954986653668251498160346091865848259963960853060020750860632150880435788410917184988498688690850646615751221247309950434587076089709352088108916792779996577130820982289749315443146420724344354422429)" job snfs.job R Code:
nfs: found 49858664 relations, need at least 91912198 (filtering ETA: 242h 44m), continuing with sieving ... 
20191107, 17:14  #4 
"Curtis"
Feb 2005
Riverside, CA
2×7×11×29 Posts 
What q did yafu start at? You might profit by using another machine to sieve low q values, while leaving this one to proceed normally.
You could sieve q as low as 5M, though the duplicate rate may jump (which means you would need more than the forecast number of relations to build a matrix by how much, we don't know). 
20191107, 17:45  #5 
Feb 2011
2^{5} Posts 
I don't have any record of that, but I can simply see what it does on another machine without any savefile:
Code:
nfs: continuing with sieving  could not determine last special q; using default startq nfs: commencing rational side lattice sieving over range: 21020000  21060000 nfs: commencing rational side lattice sieving over range: 20900000  20940000 Code:
Total yield: 17840 0/0 mpqs failures, 14796/11935 vain mpqs milliseconds total: Sieve 1038043 Sched 0 medsched 403247 TD 172674 (Init 1976, MPQS 11586) SieveChange 1242786 TD side 0: init/small/medium/large/search: 22399 9895 10894 32381 8687 sieve: init/small/medium/large/search: 9896 144202 13329 320269 20051 TD side 1: init/small/medium/large/search: 24662 4282 9705 31614 4451 sieve: init/small/medium/large/search: 13712 176266 13310 324282 2726 total yield: 17977, q=67452023 (0.15930 sec/rel) ETA 0h00m) 1122 Special q, 7839 reduction iterations reports: 322740438>12683945>11614796>7695020>4282943>2623772 Number of relations with k rational and l algebraic primes for (k,l)=: 
20191107, 18:12  #6 
"Curtis"
Feb 2005
Riverside, CA
2·7·11·29 Posts 
That leaves you Q from 10M20.9M (or lower, if you wish) that you can run manually on another machine, without worrying about trying to tell the main yafu process to skip any q values.
So, you can use other machines like you wanted, while also reducing the number of q you'll have to run above 2*lim (that's the rough bound where sieving starts to get quite inefficient, and your job will slow even more than it has already). I'd strongly prefer to run Q from 880M than 20110M, for example. Q from 10M20.9M should be quite productive compared to your current progress; Q's under 10M may have the extra duplicates I mentioned earlier. That is, the progress will look fast, but you may need more relations (maybe 5% more than the expected total number, it's hard to predict). The lower you go, the more duplicates you should expect. 810M will be almost normal, but 12M may not help the overall job much. Edit: if you really get desperate, you could manually run some of the low Q's with 15e instead of 14e. It'll run slower, but (1) you'll get about twice as many relations per range, and (2) 15e on small Q is likely still faster than 14e up at Q=90M. Don't run the same range with 14e and 15e; 15e will repeat all the 14e relations. So, maybe run 1520.9M on one machine with 14e, and 1315M on another with 15e, and then see how close the job is to completing? Last fiddled with by VBCurtis on 20191107 at 18:17 
20191107, 18:44  #7 
Jun 2012
B4A_{16} Posts 
For a 30 bit job, use higher values for rlim and alim. Many use the rule of thumb rlim=2^(lpbr4), same with alim. Yafu usually underestimates rlim and alim.
The key metric is yield, i.e. relations per Q. Should be 1.5 to < 3. I suspect 30bit is a tad low for a job of this difficulty, but like you said you’re committed now. Grunt it out is your best bet if you want to guarantee eventual success. In the past, I’ve stopped Yafu then tried to resume it only to have yafu fail to find the last stopping point and subsequently overwrite my data file with a new sieving effort. Good times... You can break up a job on different machines with Yafu using the ns flag. Test sieve a bit to estimate the total yields (yield tends to drop as Q increases). Then specify a Qrange on each machine. After sieving, combine all the data files into one data file and start filtering. CADO is much better at managing a job over many machines. Yafu was never intended to run jobs of such size. Personally I love Yafu. Just takes some manual intervention. Last fiddled with by swellman on 20191107 at 18:46 
20191107, 19:44  #8 
Feb 2011
20_{16} Posts 
Thanks to both of you for your advice.
For a second machine, as far as siever  or perhaps a J flag  does the difference between the rational and algebraic norms affect what I should choose there? 
20191107, 20:09  #9 
"Curtis"
Feb 2005
Riverside, CA
2×7×11×29 Posts 
I think you'll be okay with the 14e siever just like the regular job is using, but you can use J 14 with 14e to gain a little extra yield. That's, in effect, the 14.5 siever. A square sieve region (2^14 by 2^14 using J 14 rather than default 2^14 by 2^13) isn't optimal, but neither are your parameters so we have to make do. I don't consider the norms when making such a choice; just the yield and the estimate of the number of relations I'll need.
14e will be fastest. 14e J 14 (or 15e J 13, which is 2^15 by 2^13 sieve region, again in effect a 14.5 siever) will be next fastest, and will find lots of relations on small Q. 15e will be slower still, but will find tons of relations. A little experimenting can show you the best path; I'd use the smallest siever I could such that Q from 520.9M produced enough relations to save me from Q>80M. I expect that's something like 1020.9M on 14e, and 510M (or maybe 610M) on 15e. I changed my GGNFS script to use starting q of lim/5 for SNFS jobs like this; that is, I would have started my job at 8M. You have the chance to make that change now, but you can't profitably change alim, rlim, lpbr or lpba without starting over (or making a mess). Knowing what you know now about the size of the job, choosing alim = rlim = 67M, with Q from 12M till it finished and lpb of 30 or 31 would be about right for this job. I always choose a bigger lpb when it's close, but that's not really the mainstream opinion, so lpb=30 is likely standard. lim's just under powers of 2 are faster than just above; 67M is a bit below 2^26. That effect doesn't matter on small jobs, but becomes substantial at 134M or 268M (to the point that I don't ever select a lim between 134M and 200M it either fits in the faster 134M selection, or I skip up 50%). Last fiddled with by VBCurtis on 20191107 at 20:10 
20191107, 22:55  #10  
Jun 2012
2·5·17^{2} Posts 
Quote:
But usually it’s best to pick the r or a side and sieve on all machines on that side. And remember VBCurtis is the master siever. He’s forgotten more than I know! 

20191108, 00:13  #11 
"Ben"
Feb 2007
13·257 Posts 
Thanks swellman and VBCurtis for the help. Someday YAFU's NFS will get an overhaul to address some of these usability issues. It does fairly well for fire&forget type jobs (up to 130/140 digits, say) and SNFS poly generation is pretty capable. But for large job there are no doubt better tools. Backup data files whenever restarting if possible!
Thanks for all of the feedback too, I can't fix things I don't know about. (I don't do NFS work much anymore; focusing on QS and ECM when I get time.) 
Thread Tools  
Similar Threads  
Thread  Thread Starter  Forum  Replies  Last Post 
two (trivial) bugs for yafu 1.34  tmp  YAFU  11  20180530 22:12 
Advice for large SNFS jobs?  ryanp  Factoring  69  20130430 00:28 
Trivial question  davieddy  Information & Answers  6  20111208 08:07 
Trivial bug: repeated PM notifications  Christenson  Forum Feedback  0  20110321 03:49 
SNFS trivial factorization  fetofs  Factoring  39  20060726 12:32 