Team sieve #36: c162 from 4788:i5141 - Page 5

EdH · 2013-02-01, 17:04

Bummer! The bad news - the machine that assigns and compiles relations from all the others crashed.

The good news - all the other machines completed their parts.

The missing range is now being sieved by the entire set, but not sure how soon I can get it uploaded - probably tonight...

jrk · 2013-02-01, 21:15

Quote:

Originally Posted by jrk

FYI the relations from my ranges will be posted on Friday afternoon (CST).

As promised:

1) 25M-29M
2) 36M-38M

RichD · 2013-02-02, 03:08

48-50M is posted.

From remdups4:

Code:

Found 3552598 unique, 21482 duplicate, and 0 bad relations.

EdH · 2013-02-02, 05:24

46-48M uploaded...

Dubslow · 2013-02-02, 09:38

I'll take 44-46M.

Matrix ETA when that 38-44 block finishes.

Dubslow · 2013-02-03, 23:15

Well, the filtering went weird... there are 106,330,986 raw rels, which I remdups'd to 90,469,046 unique (with ~5 bad rels), but when I ran filtering I got millions of "error -11" in at least two separate chunks of rels (one clump in the 11M region, another in the 82M region, and possibly more whose output I missed). All in all it only read ~78M (unique) relations to work with.

Code:

read 90M relations
found 2742930 hash collisions in 78584372 relations
added 122080 free relations
commencing duplicate removal, pass 2
found 0 duplicates and 78706452 unique relations
memory use: 253.2 MB
reading ideals above 68222976
commencing singleton removal, initial pass
memory use: 1506.0 MB
reading all ideals from disk
memory use: 1385.9 MB
commencing in-memory singleton removal
begin with 78706452 relations and 75158728 unique ideals
reduce to 36237253 relations and 26360491 ideals in 24 passes
max relations containing the same ideal: 35
reading ideals above 720000
commencing singleton removal, initial pass
memory use: 753.0 MB
reading all ideals from disk
memory use: 1342.4 MB
keeping 34193525 ideals with weight <= 200, target excess is 192519
commencing in-memory singleton removal
begin with 36237256 relations and 34193525 unique ideals
reduce to 36085410 relations and 34041566 ideals in 15 passes
max relations containing the same ideal: 200
removing 3629799 relations and 3229800 ideals in 400000 cliques
commencing in-memory singleton removal
begin with 32455611 relations and 34041566 unique ideals
reduce to 32178761 relations and 30530696 ideals in 13 passes
max relations containing the same ideal: 193
removing 2671620 relations and 2271620 ideals in 400000 cliques
commencing in-memory singleton removal
begin with 29507141 relations and 30530696 unique ideals
reduce to 29336594 relations and 28086355 ideals in 9 passes
max relations containing the same ideal: 186
removing 2363397 relations and 1963397 ideals in 400000 cliques
commencing in-memory singleton removal
begin with 26973197 relations and 28086355 unique ideals
reduce to 26828040 relations and 25975971 ideals in 9 passes
max relations containing the same ideal: 174
removing 2197351 relations and 1797351 ideals in 400000 cliques
commencing in-memory singleton removal
begin with 24630689 relations and 25975971 unique ideals
reduce to 24494234 relations and 24040345 ideals in 8 passes
max relations containing the same ideal: 163
removing 1335768 relations and 1105202 ideals in 230566 cliques
commencing in-memory singleton removal
begin with 23158466 relations and 24040345 unique ideals
reduce to 23107254 relations and 22883497 ideals in 7 passes
max relations containing the same ideal: 155
relations with 0 large ideals: 673
relations with 1 large ideals: 710
relations with 2 large ideals: 10935
relations with 3 large ideals: 115280
relations with 4 large ideals: 662403
relations with 5 large ideals: 2273171
relations with 6 large ideals: 4858904
relations with 7+ large ideals: 15185178
commencing 2-way merge
reduce to 14147427 relation sets and 13923670 unique ideals
commencing full merge
memory use: 1677.3 MB
found 7702363 cycles, need 7671870
weight of 7671870 cycles is about 537342638 (70.04/cycle)
distribution of cycle lengths:
1 relations: 1100989
2 relations: 1016855
3 relations: 976231
4 relations: 860264
5 relations: 740361
6 relations: 635374
7 relations: 538777
8 relations: 445021
9 relations: 360699
10+ relations: 997299
heaviest cycle: 20 relations
commencing cycle optimization
start with 39547684 relations
pruned 848951 relations
memory use: 1352.8 MB
distribution of cycle lengths:
1 relations: 1100989
2 relations: 1039607
3 relations: 1008635
4 relations: 876531
5 relations: 753570
6 relations: 638685
7 relations: 539933
8 relations: 440813
9 relations: 354547
10+ relations: 918560
heaviest cycle: 20 relations
RelProcTime: 2737
nfs: commencing msieve linear algebra

commencing linear algebra
read 7671870 cycles
cycles contain 22919148 unique relations
read 22919148 relations
using 20 quadratic characters above 1073741372
building initial matrix
memory use: 3171.3 MB
read 7671870 cycles
matrix is 7671693 x 7671870 (2337.2 MB) with weight 734199137 (95.70/col)
sparse part has weight 520621508 (67.86/col)
filtering completed in 2 passes
matrix is 7669404 x 7669581 (2337.0 MB) with weight 734119508 (95.72/col)
sparse part has weight 520606495 (67.88/col)
matrix starts at (0, 0)
matrix is 7669404 x 7669581 (2337.0 MB) with weight 734119508 (95.72/col)
sparse part has weight 520606495 (67.88/col)
saving the first 48 matrix rows for later
matrix includes 64 packed rows
matrix is 7669356 x 7669581 (2249.2 MB) with weight 584705896 (76.24/col)
sparse part has weight 512909036 (66.88/col)
using block size 262144 for processor cache size 8192 kB
commencing Lanczos iteration (8 threads)
memory use: 2923.8 MB
linear algebra at 0.0%, ETA 107h37m69581 dimensions (0.0%, ETA 107h37m)    
checkpointing every 80000 dimensions9581 dimensions (0.0%, ETA 108h 7m)    
^Cnear algebra completed 18982 of 7669581 dimensions (0.2%, ETA 106h41m)

That was 8 threaded LA, and I tested it down to 3 threads being slight faster than 2 threads, but I'm using two anyways.

Code:

commencing linear algebra
matrix starts at (0, 0)
matrix is 7669404 x 7669581 (2337.0 MB) with weight 734119508 (95.72/col)
sparse part has weight 520606495 (67.88/col)
saving the first 48 matrix rows for later
matrix includes 64 packed rows
matrix is 7669356 x 7669581 (2249.2 MB) with weight 584705896 (76.24/col)
sparse part has weight 512909036 (66.88/col)
using block size 262144 for processor cache size 8192 kB
commencing Lanczos iteration (2 threads)
memory use: 2572.7 MB
restarting at iteration 621 (dim = 39285)
checkpointing every 120000 dimensions9581 dimensions (0.5%, ETA 63h43m)    
linear algebra at 0.5%, ETA 65h16m7669581 dimensions (0.5%, ETA 65h16m)    
linear algebra completed 57246 of 7669581 dimensions (0.7%, ETA 65h34m)

So factors will be around 2.5-3 days.

RichD · 2013-02-04, 05:22

Quote:

Originally Posted by Dubslow

Code:

using block size 262144 for processor cache size 8192 kB

How did you get 262K block size? I'm pretty sure I used LARGEBLOCKS=1 for msieve and I only get 65K.

Dubslow · 2013-02-04, 06:09

Quote:

Originally Posted by RichD

How did you get 262K block size? I'm pretty sure I used LARGEBLOCKS=1 for msieve and I only get 65K.

*shrug* I can send you my Msieve compile if you want. This processor does have 8 MB L3 cache, as reported in that line.

Batalov · 2013-02-04, 06:20

Quote:

Originally Posted by Dubslow

... but when I ran filtering I got millions of "error -11" in at least two separate chunks of rels (one clump in the 11M region, another in the 82M region, and possibly more whose output I missed)...

Someone sieved with the wrong poly. Parse the output, for exact start and finish lines of blocks of irrelevant relations, cut them out of the *.dat file (with head+tail or with a script), then grep from whose input file they came.

Remdups doesn't ask for the .poly or .fb file - so you could deduce that it has no way of figuring which of the relations (though formally fine in their formatting) are irrelevant to the project. msieve, of course, can do that and surely does.

Dubslow · 2013-02-04, 07:35

It seems it's split into many more smaller chunks than I had imagined.

Code:

11604711 - 13030164
14803967 - 16175268
17944493 - 19274840
21029847 - 22200053
24071610 - 25127113
25781307 - 25892310
43131497 - 46224830
76554918 - 77215789 
81046237 - 82590802

I'll finish investigating which and whose rels these are tomorrow.

EdH · 2013-02-04, 16:07

It looks like one of my machines wasn't getting the correct poly.

(The nine chunks kind of of gave it away - I did nine runs.)

Unfortunately, it had to be the one working the largest sections, too!

Sorry! I'll try to slink back into the shadows for now...

Off to see why and correct it!!!

Later: I believe it to be fixed.

2013-02-02, 09:38	#49
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 16065₈ Posts	I'll take 44-46M. Matrix ETA when that 38-44 block finishes. Last fiddled with by Dubslow on 2013-02-02 at 09:44

2013-02-04, 16:07	#55
EdH "Ed Hall" Dec 2009 Adirondack Mtns 3634₁₀ Posts	It looks like one of my machines wasn't getting the correct poly. (The nine chunks kind of of gave it away - I did nine runs.) Unfortunately, it had to be the one working the largest sections, too! Sorry! I'll try to slink back into the shadows for now... Off to see why and correct it!!! Later: I believe it to be fixed. Last fiddled with by EdH on 2013-02-04 at 16:20 Reason: Accuracy and Followup...

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Team Sieve #28: c162 from 4788:2714	jrk	Aliquot Sequences	27	2011-10-25 08:51
Team sieve #23: c172 from 4788:i2617	schickel	Aliquot Sequences	64	2011-02-19 02:28
Team sieve #21: c162 from 4788:2602	jrk	Aliquot Sequences	31	2010-12-30 21:33
Team sieve #20: c170 from 4788:2549	schickel	Aliquot Sequences	153	2010-11-09 07:39
Team sieve #5: c140 from 4788:2407	10metreh	Aliquot Sequences	77	2009-05-27 20:39

2013-02-01, 17:04	#45
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2·23·79 Posts	Bummer! The bad news - the machine that assigns and compiles relations from all the others crashed. The good news - all the other machines completed their parts. The missing range is now being sieved by the entire set, but not sure how soon I can get it uploaded - probably tonight...

2013-02-02, 03:08	#47
RichD Sep 2008 Kansas 2²·821 Posts	48-50M is posted. From remdups4: Code: Found 3552598 unique, 21482 duplicate, and 0 bad relations.

2013-02-02, 05:24	#48
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2·23·79 Posts	46-48M uploaded...

2013-02-04, 07:35	#54
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 7221₁₀ Posts	It seems it's split into many more smaller chunks than I had imagined. Code: 11604711 - 13030164 14803967 - 16175268 17944493 - 19274840 21029847 - 22200053 24071610 - 25127113 25781307 - 25892310 43131497 - 46224830 76554918 - 77215789 81046237 - 82590802 I'll finish investigating which and whose rels these are tomorrow.