mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Factoring

Reply
 
Thread Tools
Old 2021-01-11, 17:58   #870
charybdis
 
Apr 2020

11×17 Posts
Default

Quote:
Originally Posted by ryanp View Post
Wedged again for about 12 hours so far:

Code:
Sun Jan 10 19:56:49 2021  commencing in-memory singleton removal
Sun Jan 10 19:58:41 2021  begin with 621912609 relations and 624555581 unique ideals
Sun Jan 10 20:33:55 2021  reduce to 616867386 relations and 611731201 ideals in 17 passes
Sun Jan 10 20:33:55 2021  max relations containing the same ideal: 167
Sun Jan 10 20:37:11 2021  relations with 0 large ideals: 289519
Sun Jan 10 20:37:11 2021  relations with 1 large ideals: 190512
Sun Jan 10 20:37:11 2021  relations with 2 large ideals: 1663725
Sun Jan 10 20:37:11 2021  relations with 3 large ideals: 9813804
Sun Jan 10 20:37:11 2021  relations with 4 large ideals: 36668291
Sun Jan 10 20:37:11 2021  relations with 5 large ideals: 89881466
Sun Jan 10 20:37:11 2021  relations with 6 large ideals: 145991448
Sun Jan 10 20:37:11 2021  relations with 7+ large ideals: 332368621
Sun Jan 10 20:37:11 2021  commencing 2-way merge
I'm going to give it a bit more time, then trying adding more relations and repeating.
Seems worrying that it's stuck at 2-way merge rather than full merge. A bug in msieve, perhaps?

Also I'm thinking "well over 100M" may be an understatement...
charybdis is offline   Reply With Quote
Old 2021-01-11, 22:13   #871
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

1000001112 Posts
Default

Quote:
Originally Posted by charybdis View Post
Seems worrying that it's stuck at 2-way merge rather than full merge. A bug in msieve, perhaps?

Also I'm thinking "well over 100M" may be an understatement...
I'm letting it stay wedged until I manage to gather new relations + merge + remdups4 and gzip, which could be a few more hours at least. At that point, I'll again kill the msieve run and try again.. need to better automate all this!
ryanp is online now   Reply With Quote
Old 2021-01-11, 22:58   #872
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

2×5×467 Posts
Default

Has anyone had success letting CADO filter, and then running msieve on the resulting matrix files? I think CADO produces smaller matrices from a given dataset, and more of the filtering is multi-threaded.
VBCurtis is online now   Reply With Quote
Old 2021-01-11, 23:02   #873
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

26310 Posts
Default

Quote:
Originally Posted by VBCurtis View Post
Has anyone had success letting CADO filter, and then running msieve on the resulting matrix files? I think CADO produces smaller matrices from a given dataset, and more of the filtering is multi-threaded.
I haven't tried it before, but would be curious to give it a shot. At the bottom of README.msieve there's this blurb:

Code:
up from msieve (svn) revision 891, msieve can read a cycle file produced by
CADO-NFS. To use it, you will have to:

- use CADO-NFS for the filtering. In what follows, let 'prefix' be
  the prefix used for all the CADO filenames
- use the CADO 'replay' binary with --for_msieve to produce
  a file <prefix>.cyc
- concatenate all the relation files specified by purge.log in
  the order specified, and name the file <prefix> in the same
  directory as all the other CADO intermediate files. If Msieve was
  compiled with zlib support, the files do not have to be uncompressed
- create a <prefix>.fb file with the polynomials in Msieve format
- create worktodo.ini with a single line containing N
- run Msieve LA with

  -v -nf <prefix>.fb -s <prefix> -nc2 "cado_filter=1"
ryanp is online now   Reply With Quote
Old 2021-01-11, 23:07   #874
charybdis
 
Apr 2020

11×17 Posts
Default

According to jasonp the CADO filtering format has changed since that code was written. An old version of CADO could work but I don't know whether it had been sufficiently tested on jobs this big back then, and this would certainly have been before the multithreaded merge code was written.

Last fiddled with by charybdis on 2021-01-11 at 23:07
charybdis is offline   Reply With Quote
Old 2021-01-11, 23:08   #875
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

263 Posts
Default

So I successfully cloned the latest CADO repo and built the replay binary, but not quite sure how to invoke it.

Code:
The available parameters are the following:
    -purged    input purged file
    -his       input history file
    -out       basename for output matrices
    -skip      number of heaviest columns that go to the dense matrix (default 32)
    -index     file containing description of rows (relations-sets) of the matrix
    -ideals    file containing correspondence between ideals and matrix columns
    -force-posix-threads (switch) force the use of posix threads, do not rely on platform memory semantics
    -path_antebuffer path to antebuffer program
    -for_msieve (switch) output matrix in msieve format
    -Nmax      stop at Nmax number of rows (default 0)
    -col0      print only columns with index >= col0
    -colmax    print only columns with index < colmax
    -verbose_flags fine grained control on which messages get printed
I've got an input.dat.gz containing all my relations (gzipped) and already have msieve's input.fb and input.ini. Any advice?
ryanp is online now   Reply With Quote
Old 2021-01-12, 16:36   #876
chris2be8
 
chris2be8's Avatar
 
Sep 2009

7D116 Posts
Default

fivemack wrote a program called remsing to remove singleton relations. It might help.

See https://mersenneforum.org/showthread...ing#post525512 and related posts.

Chris
chris2be8 is offline   Reply With Quote
Old 2021-01-12, 18:33   #877
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

10716 Posts
Default

Quote:
Originally Posted by chris2be8 View Post
fivemack wrote a program called remsing to remove singleton relations. It might help.

See https://mersenneforum.org/showthread...ing#post525512 and related posts.

Chris
Thanks - to be clear, is the suggestion that after "remdups4" (I now have ~2.1B uniques) to then run that through remsing before handing the relations to msieve?
ryanp is online now   Reply With Quote
Old 2021-01-12, 18:55   #878
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

467010 Posts
Default

Yes. The idea, I think (?) is to run the first pass of singleton removal externally, so msieve handles a smaller dataset. This speeds things up when e.g. trying a handful of target densities, and in the case where relations counts are really big like this job it may help msieve handle the dataset.

You can run it multiple times, just like msieve runs multiple singleton-removal passes, but I found a single run helpful for the GNFS-207 team-sieve job.
VBCurtis is online now   Reply With Quote
Old 2021-01-12, 19:01   #879
ryanp
 
ryanp's Avatar
 
Jun 2012
Boulder, CO

10716 Posts
Default

I've got it running now. The output is currently at:

Code:
$ ~/remsing/remsing input.dat input.rem
1449910000 read  1102385169 bad
Does that ration of "bad" to "read" seem reasonable?
ryanp is online now   Reply With Quote
Old 2021-01-12, 19:09   #880
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

2×5×467 Posts
Default

No. The numbers should look quite similar to the first pass of msieve, such as these from your log on 9th Jan:
Code:
commencing in-memory singleton removal
begin with 2069585205 relations and 2015728165 unique ideals
reduce to 940662702 relations and 730460437 ideals in 20 passes
Using this log as a very rough guide, you should get output of about half your input for the first pass through remsing.

Edit: I think that happened to me the first time I tried it; an updated remsing was later posted that worked fine for me. The link posted here was to the post #608 that had the updated remsing, though. Hrmmm.....

Last fiddled with by VBCurtis on 2021-01-12 at 19:10
VBCurtis is online now   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Passive Pascal Xyzzy GPU Computing 1 2017-05-17 20:22
Tesla P100 — 5.4 DP TeraFLOPS — Pascal Mark Rose GPU Computing 52 2016-07-02 12:11
Nvidia Pascal, a third of DP firejuggler GPU Computing 12 2016-02-23 06:55
Calculating perfect numbers in Pascal Elhueno Homework Help 5 2008-06-12 16:37
Factorization attempt to a c163 - a new Odd Perfect Number roadblock jchein1 Factoring 30 2005-05-30 14:43

All times are UTC. The time now is 22:14.

Thu Feb 25 22:14:55 UTC 2021 up 84 days, 18:26, 1 user, load averages: 2.20, 1.94, 1.72

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.