mersenneforum.org > YAFU trivial snfs bug; advice ~9 days into job
 User Name Remember Me? Password
 Register FAQ Search Today's Posts Mark Forums Read

 2019-11-07, 06:30 #1 Belteshazzar   Feb 2011 25 Posts trivial snfs bug; advice ~9 days into job As I mentioned in the other subforum, I decided to try snfs for the first time, on the 203 digit cofactor of the 235-digit Pell(613). At first I didn't put the difficulty in my job file, and so I found that while factor/nfs/nfs.c tells you Code: nfs: detected snfs job but no snfs difficulty; assuming size of number is the snfs difficulty it actually does no such thing, instead assuming the snfs difficulty is 0, which gets translated to a gnfs difficulty of 30 (the .56*snfs_d +30 formula is nonsense for snfs_d< 68) and accordingly bizarre parameter choices. (BTW why not make some attempt to calculate snfs difficulty using the polynomials (as factmsieve does) rather than make such an assumption?) After I put size:235 in and ran a short test, I wrote that post asking for advice re params, but I didn't get much. I tried perturbing a couple parameterss away from yafu's defaults and doing -nt test sieving, but the defaults appeared best so I stuck with them. Maybe this isn't optimal, e.g. I found this thread suggesting lp31. A bit late to change now. I now have half the required relations. The sieving slowed down quite a bit, with the reported ETA staying almost constant (~330h) for around 7 days and only then beginning to drop, and I thought perhaps I'd try speeding things up a bit by doing some work on another computer and copying that to rels.add. But it looks like the only mechanism for telling one instance of yafu to do a different range is -ns start,end, which ends up with each siever sieving the entire (end-start)/#threads at once rather than biting off manageable chunks and checkpointing progress as nfs() normally does. Any recommendations on doing a rudimentary job of distributing some work?
 2019-11-07, 13:45 #2 swellman     Jun 2012 52·7·17 Posts Can you post your nfs.job and Yafu.ini files? What command did you use to invoke Yafu? Yafu is a great tool, but it’s parameters often need tweaking at higher difficulties. The above files should help us understand what is happening with your factorization.
 2019-11-07, 16:58 #3 Belteshazzar   Feb 2011 25 Posts The poly and skew are the same as in the other thread, so the job file just adds size:235 and the parameters yafu filled in by default: Code: n: 41992954986653668251498160346091865848259963960853060020750860632150880435788410917184988498688690850646615751221247309950434587076089709352088108916792779996577130820982289749315443146420724344354422429 Y1: 390458526450928779826062879981346977190 Y0: -942650270102046130733437746596776286089 skew: 1.75 type: snfs size: 235 c6: 1 c4: 15 c3: -40 c2: 75 c1: -72 c0: 29 rlim: 41800000 alim: 41800000 lpbr: 30 lpba: 30 mfbr: 60 mfba: 60 rlambda: 2.6 alambda: 2.6 I've done nothing fancy with yafu.ini or invoking yafu: Code: B1pm1=100000 B1pp1=20000 B1ecm=11000 rhomax=1000 threads=8 pretest_ratio=0.25 v=1 ggnfs_dir=C:\Users\Dan\Downloads\factor\GGNFS\ ecm_path=C:\Users\Dan\Downloads\factor\gmpecm-svn3052-znver1\ecm.exe tune_info=AMD Ryzen 5 2400G with Radeon Vega Graphics ,WIN64,9.0141e-006,0.205255,0.280054,0.10029,98.5466,3594.43 Code: .\yafu "nfs(41992954986653668251498160346091865848259963960853060020750860632150880435788410917184988498688690850646615751221247309950434587076089709352088108916792779996577130820982289749315443146420724344354422429)" -job snfs.job -R The most recent progress indicator is Code: nfs: found 49858664 relations, need at least 91912198 (filtering ETA: 242h 44m), continuing with sieving ... nfs.dat is 5.1GB and it's currently working on q around 67,500,000.
 2019-11-07, 17:14 #4 VBCurtis     "Curtis" Feb 2005 Riverside, CA 22·7·132 Posts What q did yafu start at? You might profit by using another machine to sieve low q values, while leaving this one to proceed normally. You could sieve q as low as 5M, though the duplicate rate may jump (which means you would need more than the forecast number of relations to build a matrix- by how much, we don't know).
 2019-11-07, 17:45 #5 Belteshazzar   Feb 2011 2016 Posts I don't have any record of that, but I can simply see what it does on another machine without any savefile: Code: nfs: continuing with sieving - could not determine last special q; using default startq nfs: commencing rational side lattice sieving over range: 21020000 - 21060000 nfs: commencing rational side lattice sieving over range: 20900000 - 20940000 etc, with the lowest q in the second one. so 20,900,000 seems to be the default start. also, in case it's informative, here's some typical output from its current range of q: Code: Total yield: 17840 0/0 mpqs failures, 14796/11935 vain mpqs milliseconds total: Sieve 1038043 Sched 0 medsched 403247 TD 172674 (Init 1976, MPQS 11586) Sieve-Change 1242786 TD side 0: init/small/medium/large/search: 22399 9895 10894 32381 8687 sieve: init/small/medium/large/search: 9896 144202 13329 320269 20051 TD side 1: init/small/medium/large/search: 24662 4282 9705 31614 4451 sieve: init/small/medium/large/search: 13712 176266 13310 324282 2726 total yield: 17977, q=67452023 (0.15930 sec/rel) ETA 0h00m) 1122 Special q, 7839 reduction iterations reports: 322740438->12683945->11614796->7695020->4282943->2623772 Number of relations with k rational and l algebraic primes for (k,l)=:
 2019-11-07, 18:12 #6 VBCurtis     "Curtis" Feb 2005 Riverside, CA 22·7·132 Posts That leaves you Q from 10M-20.9M (or lower, if you wish) that you can run manually on another machine, without worrying about trying to tell the main yafu process to skip any q values. So, you can use other machines like you wanted, while also reducing the number of q you'll have to run above 2*lim (that's the rough bound where sieving starts to get quite inefficient, and your job will slow even more than it has already). I'd strongly prefer to run Q from 8-80M than 20-110M, for example. Q from 10M-20.9M should be quite productive compared to your current progress; Q's under 10M may have the extra duplicates I mentioned earlier. That is, the progress will look fast, but you may need more relations (maybe 5% more than the expected total number, it's hard to predict). The lower you go, the more duplicates you should expect. 8-10M will be almost normal, but 1-2M may not help the overall job much. Edit: if you really get desperate, you could manually run some of the low Q's with 15e instead of 14e. It'll run slower, but (1) you'll get about twice as many relations per range, and (2) 15e on small Q is likely still faster than 14e up at Q=90M. Don't run the same range with 14e and 15e; 15e will repeat all the 14e relations. So, maybe run 15-20.9M on one machine with 14e, and 13-15M on another with 15e, and then see how close the job is to completing? Last fiddled with by VBCurtis on 2019-11-07 at 18:17
 2019-11-07, 18:44 #7 swellman     Jun 2012 52×7×17 Posts For a 30 bit job, use higher values for rlim and alim. Many use the rule of thumb rlim=2^(lpbr-4), same with alim. Yafu usually underestimates rlim and alim. The key metric is yield, i.e. relations per Q. Should be 1.5 to < 3. I suspect 30-bit is a tad low for a job of this difficulty, but like you said you’re committed now. Grunt it out is your best bet if you want to guarantee eventual success. In the past, I’ve stopped Yafu then tried to resume it only to have yafu fail to find the last stopping point and subsequently overwrite my data file with a new sieving effort. Good times... You can break up a job on different machines with Yafu using the -ns flag. Test sieve a bit to estimate the total yields (yield tends to drop as Q increases). Then specify a Q-range on each machine. After sieving, combine all the data files into one data file and start filtering. CADO is much better at managing a job over many machines. Yafu was never intended to run jobs of such size. Personally I love Yafu. Just takes some manual intervention. Last fiddled with by swellman on 2019-11-07 at 18:46
 2019-11-07, 19:44 #8 Belteshazzar   Feb 2011 25 Posts Thanks to both of you for your advice. For a second machine, as far as siever - or perhaps a -J flag - does the difference between the rational and algebraic norms affect what I should choose there?
 2019-11-07, 20:09 #9 VBCurtis     "Curtis" Feb 2005 Riverside, CA 22·7·132 Posts I think you'll be okay with the 14e siever just like the regular job is using, but you can use -J 14 with 14e to gain a little extra yield. That's, in effect, the 14.5 siever. A square sieve region (2^14 by 2^14 using -J 14 rather than default 2^14 by 2^13) isn't optimal, but neither are your parameters so we have to make do. I don't consider the norms when making such a choice; just the yield and the estimate of the number of relations I'll need. 14e will be fastest. 14e -J 14 (or 15e -J 13, which is 2^15 by 2^13 sieve region, again in effect a 14.5 siever) will be next fastest, and will find lots of relations on small Q. 15e will be slower still, but will find tons of relations. A little experimenting can show you the best path; I'd use the smallest siever I could such that Q from 5-20.9M produced enough relations to save me from Q>80M. I expect that's something like 10-20.9M on 14e, and 5-10M (or maybe 6-10M) on 15e. I changed my GGNFS script to use starting q of lim/5 for SNFS jobs like this; that is, I would have started my job at 8M. You have the chance to make that change now, but you can't profitably change alim, rlim, lpbr or lpba without starting over (or making a mess). Knowing what you know now about the size of the job, choosing alim = rlim = 67M, with Q from 12M till it finished and lpb of 30 or 31 would be about right for this job. I always choose a bigger lpb when it's close, but that's not really the mainstream opinion, so lpb=30 is likely standard. lim's just under powers of 2 are faster than just above; 67M is a bit below 2^26. That effect doesn't matter on small jobs, but becomes substantial at 134M or 268M (to the point that I don't ever select a lim between 134M and 200M- it either fits in the faster 134M selection, or I skip up 50%). Last fiddled with by VBCurtis on 2019-11-07 at 20:10
2019-11-07, 22:55   #10
swellman

Jun 2012

52×7×17 Posts

Quote:
 Originally Posted by Belteshazzar Thanks to both of you for your advice. For a second machine, as far as siever - or perhaps a -J flag - does the difference between the rational and algebraic norms affect what I should choose there?
FWIW, one trick is to sieve the same region on both the -r and -a sides simultaneously (one on each machine). This usually isn’t worth doing, as the “B-side” often has terrible yield. But it can save a job if yield fades out at high Q on the primary side.

But usually it’s best to pick the -r or -a side and sieve on all machines on that side.

And remember VBCurtis is the master siever. He’s forgotten more than I know!

 2019-11-08, 00:13 #11 bsquared     "Ben" Feb 2007 41×83 Posts Thanks swellman and VBCurtis for the help. Someday YAFU's NFS will get an overhaul to address some of these usability issues. It does fairly well for fire&forget type jobs (up to 130/140 digits, say) and SNFS poly generation is pretty capable. But for large job there are no doubt better tools. Backup data files whenever restarting if possible! Thanks for all of the feedback too, I can't fix things I don't know about. (I don't do NFS work much anymore; focusing on QS and ECM when I get time.)

 Similar Threads Thread Thread Starter Forum Replies Last Post tmp YAFU 11 2018-05-30 22:12 ryanp Factoring 69 2013-04-30 00:28 davieddy Information & Answers 6 2011-12-08 08:07 Christenson Forum Feedback 0 2011-03-21 03:49 fetofs Factoring 39 2006-07-26 12:32

All times are UTC. The time now is 08:40.

Tue Apr 13 08:40:52 UTC 2021 up 5 days, 3:21, 1 user, load averages: 1.42, 1.42, 1.45