mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Cunningham Tables

Reply
 
Thread Tools
Old 2021-11-16, 16:37   #254
RichD
 
RichD's Avatar
 
Sep 2008
Kansas

52·139 Posts
Default

Nice work on the stats. How can I tell how much I contributed? (I'm sure it is less than 1%.)
RichD is offline   Reply With Quote
Old 2021-11-16, 18:59   #255
charybdis
 
charybdis's Avatar
 
Apr 2020

11·53 Posts
Default

Quote:
Originally Posted by kruoli View Post
Also, Seth added a graph which shows the WU yield against Q. You can quickly spot where we changed the sieving strategy (at least I thought the first big jump is that, but IIRC we switched the strategy later, didn't we?), where I experimented with A=31 and maybe even the few outliers because of A=32.
Nice work! The jump is from the initial change from strategy 2 to strategy 0. The later change back to strategy 2 had a much smaller effect and is basically impossible to spot.
charybdis is offline   Reply With Quote
Old 2021-11-16, 20:15   #256
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

10100001000012 Posts
Default

Yay for Seth-stats! Also, nice chart for yield per WU.

Boo for learning how little I contributed to the job, but seeing how slow (in rel/sec) my machine sieves is a nice reminder to maybe go Ryzen-shopping in 2022.

What should our target be for matrix size to say "good enough, let's stop sieving and run the matrix"? 55M? 50?

The C207 team sieve was 72M matrix using msieve. Our uniques ratio had gone to hell because we shifted from I=16 to I=15 too early, so sieving more for a smaller matrix wasn't productive.
VBCurtis is offline   Reply With Quote
Old 2021-11-17, 08:20   #257
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

2×5×83 Posts
Default

Quote:
Originally Posted by RichD View Post
How can I tell how much I contributed?
PM'ed you. For the final stats, I will add names of those who were okay with this manually and strip out the clients manually or something like this.

Quote:
Originally Posted by charybdis View Post
Nice work!
Thanks.
Quote:
Originally Posted by charybdis View Post
The jump is from the initial change from strategy 2 to strategy 0. The later change back to strategy 2 had a much smaller effect and is basically impossible to spot.
Ah, this is interesting. I wonder how the graph would have turned out if we never changed to strategy 0.

Quote:
Originally Posted by VBCurtis View Post
What should our target be for matrix size to say "good enough, let's stop sieving and run the matrix"? 55M? 50?
Regarding test-(filtering and building a matrix), I tried to learn how to do this yesterday. Please comment on my approach. I used the README.msieve file, but at least some things there changed.
  1. I copy all *.gz files in the upload directory of the work directory to a safe temporary work space (second drive in my case).
  2. Create a file list of these (one file per line).
  3. Compile convert_poly (this should help me avoiding manual work error) and use it to change the poly into msieve format.
  4. Call ./check_rels -poly {CADO poly} -out fixed.gz -lpb0 33 -lpb1 34 -filelist {file list file} and let it run. It is quite slow at around quarter a million relations per second and only uses one core. Is this optimisable? Then I have msieve-able relations in fixed.gz, gunzip fixed.gz; mv fixed msieve.dat.
  5. Remove free relations (How? Does the grep -aEv '^[0-9]+,0$' relations > relations_fixed approach work here, too?) and precede the relations file with N {number to factor}.
  6. Call msieve -nc -v {number to be factored} and look out for the matrix size. If too large, Ctrl+C (I just now realised I wrote Strg+C in this thread multiple times…).
kruoli is online now   Reply With Quote
Old 2021-11-17, 11:25   #258
charybdis
 
charybdis's Avatar
 
Apr 2020

11·53 Posts
Default

Quote:
Originally Posted by kruoli View Post
Regarding test-(filtering and building a matrix), I tried to learn how to do this yesterday. Please comment on my approach. I used the README.msieve file, but at least some things there changed.
  1. I copy all *.gz files in the upload directory of the work directory to a safe temporary work space (second drive in my case).
  2. Create a file list of these (one file per line).
  3. Compile convert_poly (this should help me avoiding manual work error) and use it to change the poly into msieve format.
  4. Call ./check_rels -poly {CADO poly} -out fixed.gz -lpb0 33 -lpb1 34 -filelist {file list file} and let it run. It is quite slow at around quarter a million relations per second and only uses one core. Is this optimisable? Then I have msieve-able relations in fixed.gz, gunzip fixed.gz; mv fixed msieve.dat.
  5. Remove free relations (How? Does the grep -aEv '^[0-9]+,0$' relations > relations_fixed approach work here, too?) and precede the relations file with N {number to factor}.
  6. Call msieve -nc -v {number to be factored} and look out for the matrix size. If too large, Ctrl+C (I just now realised I wrote Strg+C in this thread multiple times…).
I just run cat *.gz >> [msieve directory path]/msieve.dat.gz in the upload directory and then msieve -nc -v -t [threads] in the msieve directory. If you want to avoid creating a completely new msieve.dat.gz each time you run filtering, you can control the Q ranges of the files you're cat-ing, e.g. cat 3_748plus.4????????-?????????.????????.gz >> ... for the 400M-500M range (or in this case even cat 3_748plus.4*.gz, as there aren't any other ranges starting with a 4). Msieve can read gzipped relations as long as it was compiled without NO_ZLIB=1.

There is no need for check_rels. Msieve will check the relations itself. You may end up with some bogus bad relations where the joins between the original *.gz files were; don't worry, you aren't losing any real relations.

CADO does not put free relations in the upload directory, so there won't be any for you to remove. Msieve will add them the first time you run filtering.

Finally, make sure to reverse the signs of all the algebraic coefficients (A6 to A0) in the msieve.fb file, because otherwise msieve will complain about the negative leading coefficient when it finally gets to sqrt.

Last fiddled with by charybdis on 2021-11-17 at 11:28
charybdis is offline   Reply With Quote
Old 2021-11-17, 12:16   #259
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

2×5×83 Posts
Default

Quote:
Originally Posted by charybdis View Post
I just run cat *.gz >> [msieve directory path]/msieve.dat.gz […]
Unfortunately, the command line flows over in this case. I tried find . -name '*.gz' | while read line; do cat $line >> {target path}/msieve.dat.gz; done, but this was really slow (60 MB/s), then I used find . -name '*.gz' | xargs -d'\n' -L 128 cat >> {target path}/msieve.dat.gz and this worked great (> 1 GB/s)!

Edit: Thanks for the suggestions!

Last fiddled with by kruoli on 2021-11-17 at 12:21 Reason: Additions.
kruoli is online now   Reply With Quote
Old 2021-11-17, 12:35   #260
charybdis
 
charybdis's Avatar
 
Apr 2020

11×53 Posts
Default

Quote:
Originally Posted by kruoli View Post
Unfortunately, the command line flows over in this case. I tried find . -name '*.gz' | while read line; do cat $line >> {target path}/msieve.dat.gz; done, but this was really slow (60 MB/s), then I used find . -name '*.gz' | xargs -d'\n' -L 128 cat >> {target path}/msieve.dat.gz and this worked great (> 1 GB/s)!

Edit: Thanks for the suggestions!
Well I can't say I've ever tried to cat >200k files at once before Glad you found a good solution.

Of course I should have said "A5 to A0" rather than "A6 to A0" in my previous post. Not that it matters much.
charybdis is offline   Reply With Quote
Old 2021-11-17, 13:13   #261
swellman
 
swellman's Avatar
 
Jun 2012

2×3×13×43 Posts
Default

Quote:
Originally Posted by charybdis View Post
Finally, make sure to reverse the signs of all the algebraic coefficients (A6 to A0) in the msieve.fb file, because otherwise msieve will complain about the negative leading coefficient when it finally gets to sqrt.
Quote:
Of course I should have said "A5 to A0" rather than "A6 to A0" in my previous post. Not that it matters much.
I believe this issue has been fixed.
swellman is offline   Reply With Quote
Old 2021-11-17, 13:29   #262
charybdis
 
charybdis's Avatar
 
Apr 2020

58310 Posts
Default

Quote:
Originally Posted by swellman View Post
I believe this issue has been fixed.
Only in Greg's version of msieve, not the "official" one.
charybdis is offline   Reply With Quote
Old 2021-11-17, 15:16   #263
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

47·89 Posts
Default

An alternate I use for the relations:
Code:
for i in <directory/relsprefix>*.gz
  do
    cat "$i" >> total.gz
  done
zcat total.gz | remdups4 1000 >msieve.dat
The remdups4 can be left out. I use it to save Msieve some work and to get an early idea about the uniques.

I also wrote a routine to swap the poly to Msieve format, but I believe that CADO-NFS provides one as well. From the CADO-NFS README.msieve:
Code:
1) Create a file msieve.fb, which contains:

      N <number to be factored>
      R0  <coeff of x^0, rational side>
      R1  <coeff of x^1, rational side>
      A0  <coeff of x^0, algebraic side>
      A1  <etc>
      A2  <etc>
      A3  <etc>
      A4  <etc>
      A5  <etc>

   This can be done with:

      $ ./convert_poly -of msieve < cxxx.poly > msieve.fb
Seth's page looks great! I was working on a way to add some of the data to the original page, but, of course this is much better.

Thanks, Seth, for a great resource and helping it work for this project!
EdH is offline   Reply With Quote
Old 2021-11-17, 15:33   #264
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

47×89 Posts
Default

Quote:
Originally Posted by RichD View Post
Nice work on the stats. How can I tell how much I contributed? (I'm sure it is less than 1%.)
This may not be entirely correct, but you should be able to figure out your specific client(s) with a little bit of observation, by checking the last workunit submitted date/time against the date/time shown on you client's las command line. You still need to account for the delay in refresh of the page, and they may be off a little, but I "seem" to be able to identify my clients in that manner.

Edit: Actually the date/time is in the "Subprocess has PID..." line.

Last fiddled with by EdH on 2021-11-17 at 15:38
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Using 16e on smaller numbers fivemack Factoring 3 2017-09-19 08:52
NFS on smaller numbers? skan YAFU 6 2013-02-26 13:57
Bernoulli(200) c204 akruppa Factoring 114 2012-08-20 14:01
checking smaller number fortega Data 2 2005-06-16 22:48
Factoring Smaller Numbers marc Factoring 6 2004-10-09 14:17

All times are UTC. The time now is 19:57.


Tue Jan 18 19:57:53 UTC 2022 up 179 days, 14:26, 0 users, load averages: 1.69, 1.84, 1.76

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

≠ ± ∓ ÷ × · − √ ‰ ⊗ ⊕ ⊖ ⊘ ⊙ ≤ ≥ ≦ ≧ ≨ ≩ ≺ ≻ ≼ ≽ ⊏ ⊐ ⊑ ⊒ ² ³ °
∠ ∟ ° ≅ ~ ‖ ⟂ ⫛
≡ ≜ ≈ ∝ ∞ ≪ ≫ ⌊⌋ ⌈⌉ ∘ ∏ ∐ ∑ ∧ ∨ ∩ ∪ ⨀ ⊕ ⊗ 𝖕 𝖖 𝖗 ⊲ ⊳
∅ ∖ ∁ ↦ ↣ ∩ ∪ ⊆ ⊂ ⊄ ⊊ ⊇ ⊃ ⊅ ⊋ ⊖ ∈ ∉ ∋ ∌ ℕ ℤ ℚ ℝ ℂ ℵ ℶ ℷ ℸ 𝓟
¬ ∨ ∧ ⊕ → ← ⇒ ⇐ ⇔ ∀ ∃ ∄ ∴ ∵ ⊤ ⊥ ⊢ ⊨ ⫤ ⊣ … ⋯ ⋮ ⋰ ⋱
∫ ∬ ∭ ∮ ∯ ∰ ∇ ∆ δ ∂ ℱ ℒ ℓ
𝛢𝛼 𝛣𝛽 𝛤𝛾 𝛥𝛿 𝛦𝜀𝜖 𝛧𝜁 𝛨𝜂 𝛩𝜃𝜗 𝛪𝜄 𝛫𝜅 𝛬𝜆 𝛭𝜇 𝛮𝜈 𝛯𝜉 𝛰𝜊 𝛱𝜋 𝛲𝜌 𝛴𝜎 𝛵𝜏 𝛶𝜐 𝛷𝜙𝜑 𝛸𝜒 𝛹𝜓 𝛺𝜔