Pascal's OPN roadblock files - Page 84

Happy5214 · 2021-02-06, 12:36

That seems promising. Assuming you're working with Ryan's data, if you can successfully filter it, is it possible to send him the msieve output files so he doesn't have to redo the work?

ryanp · 2021-02-06, 14:23

If frmky is able to form a matrix, either he or I can run the LA. I'd also be interested to get the msieve patch so I can try the filtering step myself.

frmky · 2021-02-07, 17:39

Code:

Sat Feb  6 23:19:11 2021  commencing 2-way merge
Sat Feb  6 23:28:47 2021  reduce to 334640603 relation sets and 328431256 unique ideals
Sat Feb  6 23:28:47 2021  commencing full merge
Sun Feb  7 04:51:15 2021  memory use: 38947.4 MB
Sun Feb  7 04:52:10 2021  found 147393213 cycles, need 147295456
Sun Feb  7 04:53:37 2021  weight of 147295456 cycles is about 19148588285 (130.00/cycle)
Sun Feb  7 04:53:37 2021  distribution of cycle lengths:
Sun Feb  7 04:53:37 2021  1 relations: 11327174
Sun Feb  7 04:53:37 2021  2 relations: 10934036
Sun Feb  7 04:53:37 2021  3 relations: 11762737
Sun Feb  7 04:53:37 2021  4 relations: 11269419
Sun Feb  7 04:53:37 2021  5 relations: 10840433
Sun Feb  7 04:53:37 2021  6 relations: 10270300
Sun Feb  7 04:53:37 2021  7 relations: 9536741
Sun Feb  7 04:53:37 2021  8 relations: 8923986
Sun Feb  7 04:53:37 2021  9 relations: 8177647
Sun Feb  7 04:53:37 2021  10+ relations: 54252983
Sun Feb  7 04:53:37 2021  heaviest cycle: 28 relations
Sun Feb  7 04:54:54 2021  commencing cycle optimization
Sun Feb  7 05:08:34 2021  start with 1239155413 relations
Sun Feb  7 07:05:08 2021  pruned 42215931 relations
Sun Feb  7 07:05:10 2021  memory use: 34774.5 MB
Sun Feb  7 07:05:11 2021  distribution of cycle lengths:
Sun Feb  7 07:05:11 2021  1 relations: 11327174
Sun Feb  7 07:05:11 2021  2 relations: 11209087
Sun Feb  7 07:05:11 2021  3 relations: 12224988
Sun Feb  7 07:05:11 2021  4 relations: 11643676
Sun Feb  7 07:05:11 2021  5 relations: 11239934
Sun Feb  7 07:05:11 2021  6 relations: 10573210
Sun Feb  7 07:05:11 2021  7 relations: 9830741
Sun Feb  7 07:05:11 2021  8 relations: 9126063
Sun Feb  7 07:05:11 2021  9 relations: 8350544
Sun Feb  7 07:05:11 2021  10+ relations: 51770039
Sun Feb  7 07:05:11 2021  heaviest cycle: 28 relations
Sun Feb  7 07:13:57 2021  RelProcTime: 199990
Sun Feb  7 07:13:57 2021  elapsed time 55:33:18

147.3M matrix! That would take my little 8 nodes about 2.5 months to solve!
Patch is attached. Edit: Had to add .txt to the end to get the forum to accept it.

ryanp · 2021-02-07, 18:01

Quote:

Originally Posted by frmky

147.3M matrix! That would take my little 8 nodes about 2.5 months to solve!

Any chance you'd be able to run it?

Otherwise, I'll test out the patch and can try sieving some more.

frmky · 2021-02-07, 21:52

It's now in the official msieve SVN, so no need for the patch file. Just grab the latest code.

frmky · 2021-02-08, 03:15

Code:

commencing linear algebra
read 147295456 cycles
cycles contain 531217644 unique relations
read 531217644 relations
using 8 quadratic characters above 4294917295
building initial matrix
memory use: 68478.7 MB
read 147295456 cycles
matrix is 147295267 x 147295456 (74519.1 MB) with weight 19922810683 (135.26/col)
sparse part has weight 18061791563 (122.62/col)
filtering completed in 2 passes
matrix is 147292440 x 147292629 (74518.9 MB) with weight 19922748759 (135.26/col)
sparse part has weight 18061766673 (122.63/col)
matrix starts at (0, 0)
matrix is 147292440 x 147292629 (74518.9 MB) with weight 19922748759 (135.26/col)
sparse part has weight 18061766673 (122.63/col)
saving the first 48 matrix rows for later
matrix includes 64 packed rows
matrix is 147292392 x 147292629 (71275.3 MB) with weight 17993852562 (122.16/col)
sparse part has weight 17211463600 (116.85/col)
using block size 8192 and superblock size 1966080 for processor cache size 20480 kB
commencing Lanczos iteration (16 threads)
memory use: 73902.4 MB
linear algebra at 0.0%, ETA 35242h 7m92629 dimensions (0.0%, ETA 35242h 7m)

I'll see you in 4 years...

VBCurtis · 2021-02-08, 05:57

Quote:

Originally Posted by frmky

Code:

linear algebra at 0.0%, ETA 35242h 7m92629 dimensions (0.0%, ETA 35242h 7m)

I'll see you in 4 years...

Holy {censored}!!!!

I mean, wow. 147M dimensions.... But the ETA seems off- shouldn't this take around 4x as long as 73M dimensions? I solved one that size for 2330L in about 3100hr on a single 10-core Ivy Bridge socket, and around 2000hr when run naively (without MPI) on both 10-core sockets.

I'm not volunteering for ~8000 machine-hours, but it seems an 8-socket mini-cluster should be around 10x faster than this!

Thanks for finding and fixing this bug- I'm happy that we're likely to have msieve at our disposal for jobs as big as we can solve matrices for!

frmky · 2021-02-08, 06:49

I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers.

I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE.

ryanp · 2021-02-08, 23:13

Quote:

Originally Posted by frmky

I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers.

I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE.

If you're able to run this on your cluster, that would be greatly appreciated; if not, I also understand. Meanwhile I will continue sieving to see if I can form something resembling a reasonable matrix.

ryanp · 2021-02-10, 05:36

Verified the fix in the latest SVN update, and I was able to produce a matrix myself. For comparison, here's an ETA on a 64-core workstation:

Code:

commencing linear algebra
read 147295123 cycles
cycles contain 531218272 unique relations
read 531218272 relations
using 8 quadratic characters above 4294917295
building initial matrix
memory use: 68478.7 MB
read 147295123 cycles
matrix is 147294934 x 147295123 (74519.4 MB) with weight 19922890703 (135.26/col)
sparse part has weight 18061865192 (122.62/col)
filtering completed in 2 passes
matrix is 147292104 x 147292281 (74519.2 MB) with weight 19922828489 (135.26/col)
sparse part has weight 18061839994 (122.63/col)
matrix starts at (0, 0)
matrix is 147292104 x 147292281 (74519.2 MB) with weight 19922828489 (135.26/col)
sparse part has weight 18061839994 (122.63/col)
saving the first 240 matrix rows for later
matrix includes 256 packed rows
matrix is 147291864 x 147292281 (69444.9 MB) with weight 16505103766 (112.06/col)
sparse part has weight 15847879420 (107.59/col)
using block size 8192 and superblock size 608256 for processor cache size 25344 kB
commencing Lanczos iteration (64 threads)
memory use: 98165.2 MB
linear algebra at 0.0%, ETA 4273h 0m292281 dimensions (0.0%, ETA 4273h 0m)    
checkpointing every 40000 dimensions292281 dimensions (0.0%, ETA 4287h45m)    
linear algebra completed 81656 of 147292281 dimensions (0.1%, ETA 4336h46m)

Unfortunately I don't think I can tie up this machine for ~half year, but hopefully some more sieving can bring down the matrix size a bit.

henryzz · 2021-02-10, 11:00

Quote:

Originally Posted by ryanp

Verified the fix in the latest SVN update, and I was able to produce a matrix myself. For comparison, here's an ETA on a 64-core workstation:

Is this using MPI? On a multi-cpu system that should help.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Passive Pascal	Xyzzy	GPU Computing	1	2017-05-17 20:22
Tesla P100 — 5.4 DP TeraFLOPS — Pascal	Mark Rose	GPU Computing	52	2016-07-02 12:11
Nvidia Pascal, a third of DP	firejuggler	GPU Computing	12	2016-02-23 06:55
Calculating perfect numbers in Pascal	Elhueno	Homework Help	5	2008-06-12 16:37
Factorization attempt to a c163 - a new Odd Perfect Number roadblock	jchein1	Factoring	30	2005-05-30 14:43

2021-02-06, 12:36	#914
Happy5214 "Alexander" Nov 2008 The Alamo City 2³×71 Posts	That seems promising. Assuming you're working with Ryan's data, if you can successfully filter it, is it possible to send him the msieve output files so he doesn't have to redo the work?

2021-02-06, 14:23	#915
ryanp Jun 2012 Boulder, CO 2²×3×23 Posts	If frmky is able to form a matrix, either he or I can run the LA. I'd also be interested to get the msieve patch so I can try the filtering step myself.

2021-02-07, 21:52	#918
frmky Jul 2003 So Cal 2×3×347 Posts	It's now in the official msieve SVN, so no need for the patch file. Just grab the latest code.

2021-02-08, 06:49	#921
frmky Jul 2003 So Cal 2·3·347 Posts	I was being a bit facetious. I've found it scales closer to d^2.5, so say a little under 6 times as long. And that computer isn't particularly fast. A rough estimate is about 2.5 months on the cluster I most use for NFS@Home numbers. I'll ask around for cluster time to not set NFS@Home back too far. I doubt any of the currently sieved numbers can be done without at least a small cluster with IB or at least 10GbE.