mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > YAFU

Reply
 
Thread Tools
Old 2019-11-07, 06:30   #1
Belteshazzar
 
Feb 2011

25 Posts
Default trivial snfs bug; advice ~9 days into job

As I mentioned in the other subforum, I decided to try snfs for the first time, on the 203 digit cofactor of the 235-digit Pell(613). At first I didn't put the difficulty in my job file, and so I found that while factor/nfs/nfs.c tells you
Code:
nfs: detected snfs job but no snfs difficulty; assuming size of number is the snfs difficulty
it actually does no such thing, instead assuming the snfs difficulty is 0, which gets translated to a gnfs difficulty of 30 (the .56*snfs_d +30 formula is nonsense for snfs_d< 68) and accordingly bizarre parameter choices.

(BTW why not make some attempt to calculate snfs difficulty using the polynomials (as factmsieve does) rather than make such an assumption?)

After I put size:235 in and ran a short test, I wrote that post asking for advice re params, but I didn't get much. I tried perturbing a couple parameterss away from yafu's defaults and doing -nt test sieving, but the defaults appeared best so I stuck with them. Maybe this isn't optimal, e.g. I found this thread suggesting lp31. A bit late to change now.

I now have half the required relations. The sieving slowed down quite a bit, with the reported ETA staying almost constant (~330h) for around 7 days and only then beginning to drop, and I thought perhaps I'd try speeding things up a bit by doing some work on another computer and copying that to rels.add. But it looks like the only mechanism for telling one instance of yafu to do a different range is -ns start,end, which ends up with each siever sieving the entire (end-start)/#threads at once rather than biting off manageable chunks and checkpointing progress as nfs() normally does.

Any recommendations on doing a rudimentary job of distributing some work?
Belteshazzar is offline   Reply With Quote
Old 2019-11-07, 13:45   #2
swellman
 
swellman's Avatar
 
Jun 2012

2×5×172 Posts
Default

Can you post your nfs.job and Yafu.ini files?

What command did you use to invoke Yafu?

Yafu is a great tool, but it’s parameters often need tweaking at higher difficulties. The above files should help us understand what is happening with your factorization.
swellman is online now   Reply With Quote
Old 2019-11-07, 16:58   #3
Belteshazzar
 
Feb 2011

25 Posts
Default

The poly and skew are the same as in the other thread, so the job file just adds size:235 and the parameters yafu filled in by default:
Code:
n: 41992954986653668251498160346091865848259963960853060020750860632150880435788410917184988498688690850646615751221247309950434587076089709352088108916792779996577130820982289749315443146420724344354422429
Y1: 390458526450928779826062879981346977190
Y0: -942650270102046130733437746596776286089
skew: 1.75
type: snfs
size: 235
c6: 1
c4: 15
c3: -40
c2: 75
c1: -72
c0: 29

rlim: 41800000
alim: 41800000
lpbr: 30
lpba: 30
mfbr: 60
mfba: 60
rlambda: 2.6
alambda: 2.6
I've done nothing fancy with yafu.ini or invoking yafu:
Code:
B1pm1=100000
B1pp1=20000
B1ecm=11000
rhomax=1000
threads=8
pretest_ratio=0.25
v=1
ggnfs_dir=C:\Users\Dan\Downloads\factor\GGNFS\
ecm_path=C:\Users\Dan\Downloads\factor\gmpecm-svn3052-znver1\ecm.exe
tune_info=AMD Ryzen 5 2400G with Radeon Vega Graphics    ,WIN64,9.0141e-006,0.205255,0.280054,0.10029,98.5466,3594.43
Code:
.\yafu "nfs(41992954986653668251498160346091865848259963960853060020750860632150880435788410917184988498688690850646615751221247309950434587076089709352088108916792779996577130820982289749315443146420724344354422429)" -job snfs.job -R
The most recent progress indicator is
Code:
nfs: found 49858664 relations, need at least 91912198 (filtering ETA: 242h 44m), continuing with sieving ...
nfs.dat is 5.1GB and it's currently working on q around 67,500,000.
Belteshazzar is offline   Reply With Quote
Old 2019-11-07, 17:14   #4
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

2×7×11×29 Posts
Default

What q did yafu start at? You might profit by using another machine to sieve low q values, while leaving this one to proceed normally.
You could sieve q as low as 5M, though the duplicate rate may jump (which means you would need more than the forecast number of relations to build a matrix- by how much, we don't know).
VBCurtis is offline   Reply With Quote
Old 2019-11-07, 17:45   #5
Belteshazzar
 
Feb 2011

25 Posts
Default

I don't have any record of that, but I can simply see what it does on another machine without any savefile:
Code:
nfs: continuing with sieving - could not determine last special q; using default startq
nfs: commencing rational side lattice sieving over range: 21020000 - 21060000
nfs: commencing rational side lattice sieving over range: 20900000 - 20940000
etc, with the lowest q in the second one. so 20,900,000 seems to be the default start. also, in case it's informative, here's some typical output from its current range of q:
Code:
Total yield: 17840
0/0 mpqs failures, 14796/11935 vain mpqs
milliseconds total: Sieve 1038043 Sched 0 medsched 403247
TD 172674 (Init 1976, MPQS 11586) Sieve-Change 1242786
TD side 0: init/small/medium/large/search: 22399 9895 10894 32381 8687
sieve: init/small/medium/large/search: 9896 144202 13329 320269 20051
TD side 1: init/small/medium/large/search: 24662 4282 9705 31614 4451
sieve: init/small/medium/large/search: 13712 176266 13310 324282 2726
total yield: 17977, q=67452023 (0.15930 sec/rel) ETA 0h00m)
1122 Special q, 7839 reduction iterations
reports: 322740438->12683945->11614796->7695020->4282943->2623772
Number of relations with k rational and l algebraic primes for (k,l)=:
Belteshazzar is offline   Reply With Quote
Old 2019-11-07, 18:12   #6
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

2·7·11·29 Posts
Default

That leaves you Q from 10M-20.9M (or lower, if you wish) that you can run manually on another machine, without worrying about trying to tell the main yafu process to skip any q values.

So, you can use other machines like you wanted, while also reducing the number of q you'll have to run above 2*lim (that's the rough bound where sieving starts to get quite inefficient, and your job will slow even more than it has already). I'd strongly prefer to run Q from 8-80M than 20-110M, for example.

Q from 10M-20.9M should be quite productive compared to your current progress; Q's under 10M may have the extra duplicates I mentioned earlier. That is, the progress will look fast, but you may need more relations (maybe 5% more than the expected total number, it's hard to predict). The lower you go, the more duplicates you should expect. 8-10M will be almost normal, but 1-2M may not help the overall job much.

Edit: if you really get desperate, you could manually run some of the low Q's with 15e instead of 14e. It'll run slower, but (1) you'll get about twice as many relations per range, and (2) 15e on small Q is likely still faster than 14e up at Q=90M. Don't run the same range with 14e and 15e; 15e will repeat all the 14e relations. So, maybe run 15-20.9M on one machine with 14e, and 13-15M on another with 15e, and then see how close the job is to completing?

Last fiddled with by VBCurtis on 2019-11-07 at 18:17
VBCurtis is offline   Reply With Quote
Old 2019-11-07, 18:44   #7
swellman
 
swellman's Avatar
 
Jun 2012

B4A16 Posts
Default

For a 30 bit job, use higher values for rlim and alim. Many use the rule of thumb rlim=2^(lpbr-4), same with alim. Yafu usually underestimates rlim and alim.

The key metric is yield, i.e. relations per Q. Should be 1.5 to < 3. I suspect 30-bit is a tad low for a job of this difficulty, but like you said you’re committed now. Grunt it out is your best bet if you want to guarantee eventual success. In the past, I’ve stopped Yafu then tried to resume it only to have yafu fail to find the last stopping point and subsequently overwrite my data file with a new sieving effort. Good times...

You can break up a job on different machines with Yafu using the -ns flag. Test sieve a bit to estimate the total yields (yield tends to drop as Q increases). Then specify a Q-range on each machine. After sieving, combine all the data files into one data file and start filtering.

CADO is much better at managing a job over many machines. Yafu was never intended to run jobs of such size.

Personally I love Yafu. Just takes some manual intervention.

Last fiddled with by swellman on 2019-11-07 at 18:46
swellman is online now   Reply With Quote
Old 2019-11-07, 19:44   #8
Belteshazzar
 
Feb 2011

2016 Posts
Default

Thanks to both of you for your advice.


For a second machine, as far as siever - or perhaps a -J flag - does the difference between the rational and algebraic norms affect what I should choose there?
Belteshazzar is offline   Reply With Quote
Old 2019-11-07, 20:09   #9
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

2×7×11×29 Posts
Default

I think you'll be okay with the 14e siever just like the regular job is using, but you can use -J 14 with 14e to gain a little extra yield. That's, in effect, the 14.5 siever. A square sieve region (2^14 by 2^14 using -J 14 rather than default 2^14 by 2^13) isn't optimal, but neither are your parameters so we have to make do. I don't consider the norms when making such a choice; just the yield and the estimate of the number of relations I'll need.

14e will be fastest. 14e -J 14 (or 15e -J 13, which is 2^15 by 2^13 sieve region, again in effect a 14.5 siever) will be next fastest, and will find lots of relations on small Q.
15e will be slower still, but will find tons of relations.

A little experimenting can show you the best path; I'd use the smallest siever I could such that Q from 5-20.9M produced enough relations to save me from Q>80M. I expect that's something like 10-20.9M on 14e, and 5-10M (or maybe 6-10M) on 15e.

I changed my GGNFS script to use starting q of lim/5 for SNFS jobs like this; that is, I would have started my job at 8M. You have the chance to make that change now, but you can't profitably change alim, rlim, lpbr or lpba without starting over (or making a mess).

Knowing what you know now about the size of the job, choosing alim = rlim = 67M, with Q from 12M till it finished and lpb of 30 or 31 would be about right for this job. I always choose a bigger lpb when it's close, but that's not really the mainstream opinion, so lpb=30 is likely standard.
lim's just under powers of 2 are faster than just above; 67M is a bit below 2^26. That effect doesn't matter on small jobs, but becomes substantial at 134M or 268M (to the point that I don't ever select a lim between 134M and 200M- it either fits in the faster 134M selection, or I skip up 50%).

Last fiddled with by VBCurtis on 2019-11-07 at 20:10
VBCurtis is offline   Reply With Quote
Old 2019-11-07, 22:55   #10
swellman
 
swellman's Avatar
 
Jun 2012

2·5·172 Posts
Default

Quote:
Originally Posted by Belteshazzar View Post
Thanks to both of you for your advice.


For a second machine, as far as siever - or perhaps a -J flag - does the difference between the rational and algebraic norms affect what I should choose there?
FWIW, one trick is to sieve the same region on both the -r and -a sides simultaneously (one on each machine). This usually isn’t worth doing, as the “B-side” often has terrible yield. But it can save a job if yield fades out at high Q on the primary side.

But usually it’s best to pick the -r or -a side and sieve on all machines on that side.

And remember VBCurtis is the master siever. He’s forgotten more than I know!
swellman is online now   Reply With Quote
Old 2019-11-08, 00:13   #11
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

13·257 Posts
Default

Thanks swellman and VBCurtis for the help. Someday YAFU's NFS will get an overhaul to address some of these usability issues. It does fairly well for fire&forget type jobs (up to 130/140 digits, say) and SNFS poly generation is pretty capable. But for large job there are no doubt better tools. Backup data files whenever restarting if possible!

Thanks for all of the feedback too, I can't fix things I don't know about. (I don't do NFS work much anymore; focusing on QS and ECM when I get time.)
bsquared is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
two (trivial) bugs for yafu 1.34 tmp YAFU 11 2018-05-30 22:12
Advice for large SNFS jobs? ryanp Factoring 69 2013-04-30 00:28
Trivial question davieddy Information & Answers 6 2011-12-08 08:07
Trivial bug: repeated PM notifications Christenson Forum Feedback 0 2011-03-21 03:49
SNFS trivial factorization fetofs Factoring 39 2006-07-26 12:32

All times are UTC. The time now is 16:09.

Tue Nov 24 16:09:11 UTC 2020 up 75 days, 13:20, 4 users, load averages: 1.58, 1.72, 1.71

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.