Pascal's OPN roadblock files

[ryanp]

Originally Posted by VBCurtis

No. The numbers should look quite similar to the first pass of msieve, such as these from your log on 9th Jan:

commencing in-memory singleton removal
begin with 2069585205 relations and 2015728165 unique ideals
reduce to 940662702 relations and 730460437 ideals in 20 passes

Using this log as a very rough guide, you should get output of about half your input for the first pass through remsing.

Edit: I think that happened to me the first time I tried it; an updated remsing was later posted that worked fine for me. The link posted here was to the post #608 that had the updated remsing, though. Hrmmm.....

I just grabbed the latest code from https://github.com/fivemack/factorisation-tools. Maybe fivemack (if he's reading this) can chime in? The final output was about 1/4 the original rels, which definitely seems wrong.

[charybdis]

Another suggestion: if the huge volume of relations is posing problems for msieve, then you could try turning this into a 33/35 job by removing all the relations with 36-bit algebraic factors using grep. Granted, you might need to sieve a bit more, but it ought to be simpler than finding and fixing a bug.

[VBCurtis]

Or, you could just let CADO run the filtering and see how big that matrix is. I think msieve generally creates a matrix slightly bigger than cado; so if CADO's matrix comes out a tolerable size, you can stop sieving and focus on getting the dataset through msieve.

Though, Charybdis' last suggestion might mean sieving a little more anyway if msieve proves to be the problem.

[ryanp]

Originally Posted by VBCurtis

Or, you could just let CADO run the filtering and see how big that matrix is.

I would, but I'm not quite sure how to run the replay binary? (also I thought there was some question about current. compatibility between CADO's output and msieve's LA?)

In any event, here's where things currently stand with the latest msieve run, 2.1B unique's and target_density=110:

Tue Jan 12 12:21:48 2021  found 11 duplicates and 2110664553 unique relations
Tue Jan 12 12:21:48 2021  memory use: 31280.0 MB
Tue Jan 12 12:21:49 2021  reading ideals above 1591017472
Tue Jan 12 12:21:49 2021  commencing singleton removal, initial pass
Tue Jan 12 16:06:06 2021  memory use: 41024.0 MB
Tue Jan 12 16:06:07 2021  reading all ideals from disk
Tue Jan 12 16:07:00 2021  memory use: 39808.5 MB
Tue Jan 12 16:08:59 2021  commencing in-memory singleton removal
Tue Jan 12 16:10:33 2021  begin with 2110664552 relations and 2029607823 unique ideals
Tue Jan 12 16:27:46 2021  reduce to 992995519 relations and 761340257 ideals in 20 passes
Tue Jan 12 16:27:46 2021  max relations containing the same ideal: 35
Tue Jan 12 16:28:38 2021  reading ideals above 720000
Tue Jan 12 16:28:38 2021  commencing singleton removal, initial pass

[ryanp]

msieve keeps getting wedged... Trying again with 2.1B uniques and target_density=90.

[henryzz]

Would jasonp be able to give any advice on how to get past this blockage in the merge stage? I don't know whether he is reading this thread.

[wreck]

I'd suggest to do the 'NFS Filtering' step 50 hours first , to look what will happen then. Since the relations count is 2000 million, it will meed more patience than 1000 million.

[wreck]

Repeate messages.
Some background date:
1. filter step for 2,1076+ snfs 324 (688M relations) : 15 hours.
2. filter step for 10,459- snfs 306 (737M relations) : 22 hours.

[ryanp]

Originally Posted by wreck

I'd suggest to do the 'NFS Filtering' step 50 hours first , to look what will happen then. Since the relations count is 2000 million, it will meed more patience than 1000 million.

Thanks for the tip! I'll try to let this run (with 2.1B uniques and TD=90) sit for a while.

[ryanp]

Fri Jan 15 12:21:24 2021  commencing in-memory singleton removal
Fri Jan 15 12:22:01 2021  begin with 590365520 relations and 594141896 unique ideals
Fri Jan 15 12:35:29 2021  reduce to 585813821 relations and 580608996 ideals in 21 passes
Fri Jan 15 12:35:29 2021  max relations containing the same ideal: 155
Fri Jan 15 12:37:14 2021  relations with 0 large ideals: 307095
Fri Jan 15 12:37:14 2021  relations with 1 large ideals: 215009
Fri Jan 15 12:37:14 2021  relations with 2 large ideals: 1852650
Fri Jan 15 12:37:14 2021  relations with 3 large ideals: 10607404
Fri Jan 15 12:37:14 2021  relations with 4 large ideals: 38306704
Fri Jan 15 12:37:14 2021  relations with 5 large ideals: 90474854
Fri Jan 15 12:37:14 2021  relations with 6 large ideals: 141486075
Fri Jan 15 12:37:14 2021  relations with 7+ large ideals: 302564030
Fri Jan 15 12:37:14 2021  commencing 2-way merge

So we're now at 72+ hours and it's still running... I'll give it a few more days, but starting to lose hope.

[pinhodecarlos]

Maybe get in touch with Greg from NFS@Home to see if he can give any support or advise?!