3,748+ c204 Smaller-but-Needed

[EdH]

We're closing in on our primary goal!

Now it's time to show a small portion of my ignorance:

- If A=32 performed so much better returning relations, why was A=30 a better choice? Did the duplicates outweigh the overall return?

- I thought that CPU time was set against an old standard CPU benchmark. If newer CPUs are multiples of that benchmark, why is host #2 (on Seth's page) showing 50% less CPU time over host #3, while showing 50% more productivity? Was that much efficiency really a difference in CPU capabilities?

[kruoli]

A=32 took much longer to run a single WU (less relations per time) and used much more memory.

Filtering and LA with 1.255B relations gave me (TD 120):

commencing linear algebra
read 73594760 cycles
cycles contain 241224346 unique relations
read 241224346 relations
using 20 quadratic characters above 4294917295
building initial matrix
memory use: 33714.0 MB
read 73594760 cycles
matrix is 73594580 x 73594760 (27697.0 MB) with weight 8500291408 (115.50/col)
sparse part has weight 6377472282 (86.66/col)
filtering completed in 2 passes
matrix is 73565842 x 73566019 (27695.0 MB) with weight 8499247339 (115.53/col)
sparse part has weight 6377274484 (86.69/col)
matrix starts at (0, 0)
matrix is 73565842 x 73566019 (27695.0 MB) with weight 8499247339 (115.53/col)
sparse part has weight 6377274484 (86.69/col)
saving the first 48 matrix rows for later
matrix includes 64 packed rows
matrix is 73565794 x 73566019 (26838.6 MB) with weight 7031611414 (95.58/col)
sparse part has weight 6299920567 (85.64/col)
using block size 8192 and superblock size 6291456 for processor cache size 65536 kB
commencing Lanczos iteration (16 threads)
memory use: 27168.9 MB
Linear algebra completed 1007 of 73566019 dimensions (0.0%, ETA 3612h 5m)

[charybdis]

Originally Posted by EdH

- If A=32 performed so much better returning relations, why was A=30 a better choice? Did the duplicates outweigh the overall return?

The A/I value determines the size of the sieve region, i.e. how hard we search for relations at each special-Q. Higher A values will always have higher yield but worse rels/sec, but if you use too small a sieve region then the duplication rate will get too high because you'll need to sieve over a very large range of Q.

Originally Posted by EdH

- I thought that CPU time was set against an old standard CPU benchmark. If newer CPUs are multiples of that benchmark, why is host #2 (on Seth's page) showing 50% less CPU time over host #3, while showing 50% more productivity? Was that much efficiency really a difference in CPU capabilities?

There's no adjustment for relative CPU speeds, it's literally just plain old CPU-time. This means, for example, that you'll see lower rels/sec if you use hyperthreading. My machines (host #2) do not have HT so this may account for part of the efficiency difference. The rest of the difference is probably a combination of clock speed differences and newer CPUs being more efficient. (If you look at my individual clients you'll see a few of them are substantially faster than the rest. That's not because they've got a higher clock speed - they haven't - it's because they're the newest.)

Originally Posted by kruoli

Linear algebra completed 1007 of 73566019 dimensions (0.0%, ETA 3612h 5m)

Ouch - looks like we might need to go up to 1.4-1.5G relations. Even given the huge matrix, that ETA feels worse than I'd have expected. Were you still sieving on the other 16 threads?

[kruoli]

Originally Posted by charybdis

Ouch - looks like we might need to go up to 1.4-1.5G relations. Even given the huge matrix, that ETA feels worse than I'd have expected. Were you still sieving on the other 16 threads?

Yes, this was not designed to be a benchmark. I could have used one thread as well, but I was not sure if building the matrix is also parallelised. It looked like it is not.

Our sieving speed (available cores) is still good. We should be able to tackle this quickly.

[EdH]

Thanks for the answers. That does help educate me a bit.

Oliver, are you planning to take the server down when you reach the initial goal? I'm not sure how my scripts will react. If you plan to drop the server during my overnight (which will include your early morning), I will tell all my clients to stop before I head to bed.

[kruoli]

Originally Posted by EdH

Oliver, are you planning to take the server down when you reach the initial goal?

No, I am going to wait at least until our experts here agree that we can stop, in that case I would announce my intention and will shut down my server after one hour without no new work sent out or 24 h ignoring existing connections, whichever comes first.

[VBCurtis]

CADO should keep running until we get a manageable matrix. 73M at 1.25G relations suggests we have another day or three to go; hopefully filtering can be run once a day until the matrix doesn't shrink much from the day's extra relations.

1.16G -> 87M matrix (default TD)
1.255G -> 73M matrix (TD 120)
So ~100M extra relations dropped 16% from matrix size. Another 50M relations from a day's sieving should reduce another 6% or so; that is, we might lose ~4M dimensions from one more day's sieving.

Projecting from these two data points, 1.4G might yield a matrix around 61-64M. Let's use TD 124 next time, to make filtering work a little harder.

[kruoli]

This can be done. I will start the next run at around 8 PM UTC. The filtering ran 15 h last time since the machine is busy, so results should be there around 2 PM UTC.

[VBCurtis]

Also, you are correct that filtering and matrix-building is single-threaded.
CADO's is multi-threaded, but that doesn't help us here since CADO matrix-solving is so much slower.

If only we knew how to use the results from CADO filtering to run with msieve matrix-solving; on the C207 team sieve I got a 60M matrix from CADO but 72M from msieve.

[EdH]

Sounds good. Will you have the server send out a 410 message, or just stop its process? I might pull all my clients on Monday night, if we're still sieving then, due to expecting to be tied up Tuesday.

[EdH]

Originally Posted by VBCurtis

. . .
If only we knew how to use the results from CADO filtering to run with msieve matrix-solving; on the C207 team sieve I got a 60M matrix from CADO but 72M from msieve.

The CADO-NFS docs explain how, if we'd like to try it:

Originally Posted by README.msieve

III) Using msieve linear algebra after CADO-NFS filtering

[should work with CADO-NFS revision aca5658]

up from msieve (svn) revision 891, msieve can read a cycle file produced by
CADO-NFS. To use it, you will have to:

- use CADO-NFS for the filtering. In what follows, let 'prefix' be
the prefix used for all the CADO filenames
- use the CADO 'replay' binary with --for_msieve to produce
a file <prefix>.cyc
- concatenate all the relation files specified by purge.log in
the order specified, and name the file <prefix> in the same
directory as all the other CADO intermediate files. If Msieve was
compiled with zlib support, the files do not have to be uncompressed
- create a <prefix>.fb file with the polynomials in Msieve format
- create worktodo.ini with a single line containing N
- run Msieve LA with

-v -nf <prefix>.fb -s <prefix> -nc2 "cado_filter=1"

The string at the end may get extra options depending on whether
the LA has more tweaking, like using MPI. The .cyc file gets
overwritten during the LA, so re-running the LA does not require
cado_filter=1.