Reserved for MF - Sequence 4788 - Page 283

bur · 2021-09-16, 14:42

Thanks, to find the number of uniques I just did a wc -l on the lines not starting with #, does that yield the correct result? Is there a faster way, does remdup output the number somewhere?

EdH · 2021-09-16, 15:00

Quote:

Originally Posted by bur

Thanks, to find the number of uniques I just did a wc -l on the lines not starting with #, does that yield the correct result? Is there a faster way, does remdup output the number somewhere?

I can't answer the first part, but remdups4 outputs counts to the terminal as it runs.

bur · 2021-09-16, 15:56

You're right, even with a summary. I must have been blind. Thanks again and sorry for the off-topic.

ryanp · 2021-09-16, 17:09

I typically run remdups4 after gathering new relations files, though I haven't been keeping track of the % dups.

Currently at 1B uniques as of this morning.

ryanp · 2021-09-20, 01:41

Latest run is going:

Code:

Sun Sep 19 16:01:22 2021  commencing relation filtering
Sun Sep 19 16:01:22 2021  setting target matrix density to 120.0
Sun Sep 19 16:01:22 2021  estimated available RAM is 192016.8 MB
Sun Sep 19 16:01:22 2021  commencing duplicate removal, pass 1
Sun Sep 19 19:12:14 2021  found 498163917 hash collisions in 1834320482 relations
Sun Sep 19 19:12:45 2021  commencing duplicate removal, pass 2
Sun Sep 19 19:36:34 2021  found 107391008 duplicates and 1726929474 unique relations
Sun Sep 19 19:36:34 2021  memory use: 16280.0 MB
Sun Sep 19 19:36:35 2021  reading ideals above 1259929600
Sun Sep 19 19:36:35 2021  commencing singleton removal, initial pass

We'll see what happens by morning. Not sure if I quite have enough unique's yet, but it's getting there!

bur · 2021-09-20, 09:43

Good to know.

Quote:

found 107391008 duplicates and 1726929474 unique relations

Why is the duplicate percentage so low, is that after an inital removal already?

ryanp · 2021-09-20, 14:16

Quote:

Originally Posted by bur

Good to know.

Why is the duplicate percentage so low, is that after an inital removal already?

I had been running remdups up until now, but the relations file is taking up too much disk space at this point, so now I'm just appending new rel's and handing the file right to msieve.

ryanp · 2021-09-21, 00:06

The msieve run with 1.72B uniques succeeded, but generated a 169M x 169M matrix. Trying again now, with 1.9B uniques.

charybdis · 2021-09-21, 00:19

What Q value have you sieved up to, and what is the overall duplication rate like? I'd be interested to know how well my estimates from test-sieving have held up.

Batalov · 2021-09-21, 01:08

Quote:

Originally Posted by ryanp

The msieve run with 1.72B uniques succeeded, but generated a 169M x 169M matrix. Trying again now, with 1.9B uniques.

That's the famous cusp (of convergence)

bur · 2021-09-21, 17:27

Quote:

That's the famous cusp (of convergence)

Interesting, is it valid for most sizes of numbers? That'd be quite helpful.

2021-09-16, 14:42	#3103
bur Aug 2020 796581e-4;32539e-3 2²×7×19 Posts	Thanks, to find the number of uniques I just did a wc -l on the lines not starting with #, does that yield the correct result? Is there a faster way, does remdup output the number somewhere? Last fiddled with by bur on 2021-09-16 at 14:42

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Reserved for MF - Sequence 3408	RichD	Aliquot Sequences	488	2022-06-10 01:32
Reserved for MF - Sequence 3366	RichD	Aliquot Sequences	535	2021-12-04 01:13
Reserved for MF - Sequence 276	kar_bon	Aliquot Sequences	136	2021-10-21 16:17
Assignments are reserved but not showing up	prism019	GPU to 72	6	2020-09-21 22:11
80M to 64 bits ... but not really reserved	petrw1	Lone Mersenne Hunters	82	2010-01-11 01:57

2021-09-16, 15:56	#3105
bur Aug 2020 796581e-4;32539e-3 1024₈ Posts	You're right, even with a summary. I must have been blind. Thanks again and sorry for the off-topic.

2021-09-16, 17:09	#3106
ryanp Jun 2012 Boulder, CO 2×199 Posts	I typically run remdups4 after gathering new relations files, though I haven't been keeping track of the % dups. Currently at 1B uniques as of this morning.

2021-09-21, 00:06	#3110
ryanp Jun 2012 Boulder, CO 110001110₂ Posts	The msieve run with 1.72B uniques succeeded, but generated a 169M x 169M matrix. Trying again now, with 1.9B uniques.

2021-09-21, 00:19	#3111
charybdis Apr 2020 2²×11×17 Posts	What Q value have you sieved up to, and what is the overall duplication rate like? I'd be interested to know how well my estimates from test-sieving have held up.