mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > CADO-NFS

Reply
 
Thread Tools
Old 2021-12-22, 08:04   #56
bur
 
bur's Avatar
 
Aug 2020
79*6581e-4;3*2539e-3

659 Posts
Default

Yes, I'll go with msieve for filtering/matrix building since as mentioned in the other thread I ran into a memory problem with CADO in that stage before. Even though unfortunately it seems msieve is less efficient in filtering, I always found it to require more rels than CADO to build a matrix. Which is why you suggested 40M additional rels?

That means in total it will likely take a lot longer (25% more sieving time maybe?) than using CADO exclusively, but I think I have no choice due to the limited RAM. If RAM wouldn't be of concern, would strategy2 / 240M rels / msieve still be faster than strategy1 / 210M rels / CADO?
bur is offline   Reply With Quote
Old 2021-12-22, 14:11   #57
charybdis
 
charybdis's Avatar
 
Apr 2020

3A116 Posts
Default

No, I'm not suggesting that msieve will require 40M more relations than CADO to build a matrix, nor am I suggesting that you actually keep sieving until reaching 250M relations.

The issue is that if CADO starts filtering when you reach 210M, the server may crash due to not having enough memory, and it won't resume sieving until you restart the server yourself. So it's better to set rels_wanted artificially high so that CADO doesn't start filtering itself, and to run msieve filtering manually when you get to ~210M relations. For example, you reach 210M relations overnight, and in this case you'd rather start msieve filtering with 220M relations in the morning than have CADO crash overnight and you're still stuck on 210M. If you find that you need to sieve a bit more, you can restart the CADO server (making sure to set tasks.wutimeout high enough that the tasks you interrupted don't expire).

Msieve linear algebra is much faster than CADO, so even if memory is not an issue it is still better to wait until msieve can build a matrix. In addition, you may find that when you only just have enough relations to build a matrix, you will save more time in linear algebra by collecting 5-10M more relations and getting a smaller matrix than you spend sieving to collect those relations. This is the logic behind VBCurtis's tasks.filter.required_excess = 0.05, which tells CADO to continue sieving if it can only just build a matrix.
charybdis is offline   Reply With Quote
Old 2021-12-22, 14:33   #58
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

22·5·263 Posts
Default

IIRC, when I had trouble with the square root memory crash some time ago, I was able to recover by adding a large swap file. Would such a swap file allow CADO-NFS to perform the earlier portions of filtering/LA?
EdH is offline   Reply With Quote
Old 2021-12-22, 17:41   #59
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

564310 Posts
Default

Filtering: yes. I had an nvme-SSD swap file when I did the C207 job, and CADO filtering ran about 50x slower than all-RAM. I think CADO was trying to use ~90-100GB on a 64GB machine.

Matrix: You're better off massively oversieving to create a matrix that fits in RAM than trying to make LA run using swap. This might happen to you if you tried your farm on e.g. a C200+.

Last fiddled with by VBCurtis on 2021-12-22 at 17:42
VBCurtis is offline   Reply With Quote
Old 2021-12-29, 08:17   #60
bur
 
bur's Avatar
 
Aug 2020
79*6581e-4;3*2539e-3

659 Posts
Default

I see, thanks. I know about over-sieving and matrix time, though it gets less valuable the fewer cores you have for sieving. And for some jobs on a 10-core it felt to me as if just going with CADO and less rels was faster than using msieve despite the slower LA, but I can't quantify it.

So using these parameters is considered the best choice?

Code:
tasks.A = 28
tasks.qmin = 15000000
tasks.lim0 = 50000000
tasks.lim1 = 70000000
tasks.lpb0 = 31
tasks.lpb1 = 32
tasks.sieve.mfb0 = 58
tasks.sieve.mfb1 = 63
tasks.sieve.ncurves0 = 20
tasks.sieve.ncurves1 = 25
tasks.sieve.adjust_strategy = 2
Would it be efficient to do test-sieving for strategy 1 vs 2?


I have a poly with cownoise score of 3.7e-13. The record is 3.9e-13, is that close enough or should I search some more?
bur is offline   Reply With Quote
Old 2021-12-29, 17:46   #61
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

130138 Posts
Default

Those look good to me! I don't think test-sieving the strategies is worthwhile; expect strategy 2 to be 3-5% faster over the entire job, though it may not be faster at the lowest Q's.
I'd start trying to build an msieve matrix around 205M relations, and I'd want a matrix 11M or smaller to stop sieving.
Good luck!

Edit: a poly score within 5% of the record is easily good enough to sieve; we can't break records on every job, especially if the number does not begin with a 1.

Last fiddled with by VBCurtis on 2021-12-29 at 17:47
VBCurtis is offline   Reply With Quote
Old 2021-12-29, 18:53   #62
bur
 
bur's Avatar
 
Aug 2020
79*6581e-4;3*2539e-3

65910 Posts
Default

Ok, I'll do that. I did some more poly searching today and up to 3e6 nothing else came up, so I guess that's good enough.


You didn't specify lambda, should I leave it to CADO?
bur is offline   Reply With Quote
Old 2021-12-29, 21:10   #63
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

33×11×19 Posts
Default

Quote:
Originally Posted by bur View Post
You didn't specify lambda, should I leave it to CADO?
Yes. Using tight lambda settings seems to boost speed on small jobs with loose large-prime bounds- my files use LP bounds often 2 bits larger than tradition / GGNFS-factmsieve specify, but tight-lambda helps less when LP choice is relatively smaller compared to lim size.

As our job sizes grow, our LP choice gets ever closer to "standard" choices because the extra matrix complexity from larger large-prime relations costs more than the boost to sieve performance. Instead of using lambda to control which cofactors to split, I use a small mfb choice- note that 2 * 31 is 62, but mfb0 is 58. Setting mfb0 to 59 might be faster, but that's not easily testable because one expects to need more relations when one's relations contain (on average) larger large-prime primes.
VBCurtis is offline   Reply With Quote
Old 2021-12-30, 06:48   #64
bur
 
bur's Avatar
 
Aug 2020
79*6581e-4;3*2539e-3

659 Posts
Default

Thanks for the info, I still don't know enough about GNFS to understand why the settings are what they are. I get the general concept, but things like the factorbase I'll have to do some more reading about.


Sieving is going fine so far, 9.4M relations from the first 2M q. Current ETA (for 240M rels) is Jan, 10th, so all in all I expect Jan, 19th or so for the first attempt at a matrix.
bur is offline   Reply With Quote
Old 2022-01-27, 03:03   #65
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

33·11·19 Posts
Default

I finally have the time, human and machine, to finish up some comparisons and get files published for c145 to c170. While I haven't done enough tests at these sizes to be confident there aren't speed gains to be found (for one, I haven't determined the right size to start using 3LP, because I haven't tested yet), it benefits the community to get "pretty good" params out there while we search for greatness.

Over the next week or so, I'll get drafts published for review by the regulars who may have tried other settings, or just want to suggest alternatives to test. First up is c145, attached here.
Attached Files
File Type: txt params.c145.txt (2.1 KB, 37 views)
VBCurtis is offline   Reply With Quote
Old 2022-01-27, 03:23   #66
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

22×5×263 Posts
Default

I'm definitely interested in c170. I recently ran a c172 with the default params and it took about twice what I expected. I modified the default for next time, trying to set values between what I have for the c165 and c175. I was stuck with whether to choose siever 14 or 15, though.
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Posting log files or other text files Xyzzy Forum Feedback 3 2018-12-30 19:37
Improved NFS polynomial selection jasonp Operation Kibibit 5 2014-09-07 11:02
CADO-NFS skan Information & Answers 1 2013-10-22 07:00
could oddperfect's ecm progress page be improved? jasong GMP-ECM 11 2007-05-30 03:08
Factoring progress has really improved! eepiccolo Lone Mersenne Hunters 3 2003-04-12 02:04

All times are UTC. The time now is 23:08.


Mon Feb 6 23:08:01 UTC 2023 up 172 days, 20:36, 1 user, load averages: 1.42, 1.28, 1.13

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

≠ ± ∓ ÷ × · − √ ‰ ⊗ ⊕ ⊖ ⊘ ⊙ ≤ ≥ ≦ ≧ ≨ ≩ ≺ ≻ ≼ ≽ ⊏ ⊐ ⊑ ⊒ ² ³ °
∠ ∟ ° ≅ ~ ‖ ⟂ ⫛
≡ ≜ ≈ ∝ ∞ ≪ ≫ ⌊⌋ ⌈⌉ ∘ ∏ ∐ ∑ ∧ ∨ ∩ ∪ ⨀ ⊕ ⊗ 𝖕 𝖖 𝖗 ⊲ ⊳
∅ ∖ ∁ ↦ ↣ ∩ ∪ ⊆ ⊂ ⊄ ⊊ ⊇ ⊃ ⊅ ⊋ ⊖ ∈ ∉ ∋ ∌ ℕ ℤ ℚ ℝ ℂ ℵ ℶ ℷ ℸ 𝓟
¬ ∨ ∧ ⊕ → ← ⇒ ⇐ ⇔ ∀ ∃ ∄ ∴ ∵ ⊤ ⊥ ⊢ ⊨ ⫤ ⊣ … ⋯ ⋮ ⋰ ⋱
∫ ∬ ∭ ∮ ∯ ∰ ∇ ∆ δ ∂ ℱ ℒ ℓ
𝛢𝛼 𝛣𝛽 𝛤𝛾 𝛥𝛿 𝛦𝜀𝜖 𝛧𝜁 𝛨𝜂 𝛩𝜃𝜗 𝛪𝜄 𝛫𝜅 𝛬𝜆 𝛭𝜇 𝛮𝜈 𝛯𝜉 𝛰𝜊 𝛱𝜋 𝛲𝜌 𝛴𝜎𝜍 𝛵𝜏 𝛶𝜐 𝛷𝜙𝜑 𝛸𝜒 𝛹𝜓 𝛺𝜔