Contributing to Cunningham Project - Page 21

charybdis · 2021-01-18, 21:23

Just thrown in 45 more cores to help speed things along to the finish line.

VBCurtis · 2021-01-20, 00:44

We reached 540M relations, and CADO crashed on the first filtering step with this error:

Code:

Error:Filtering - Duplicate Removal, removal pass: 	terminate called after throwing an instance of 'renumber_t::corrupted_table'
Error:Filtering - Duplicate Removal, removal pass: 	  what():  Renumber table is corrupt: cannot find data for prime 0x28e17329 ; note: isprime(p)==0
Error:Filtering - Duplicate Removal, removal pass: 	Note: above 2^32 ideals or relations, add FLAGS_SIZE="-DSIZEOF_P_R_VALUES=8 -DSIZEOF_INDEX=8" to local.sh

I've seen this error on small (~C110) jobs with recent checkouts on another machine, which caused me to stick with the May-2020 version that works on that other machine. I used a fresh git-checkout for this job, and am disappointed to see that it's a bigger bug than I guessed.

In our case, it shouldn't much matter- I'm going to use msieve for postprocessing anyway. I'll let y'all know what I've got after filtering.

Edit: remdups leaves me 368.0M uniques, a pleasant 68%. Q-range was 30M to 264.6M; yield fell off quite a bit at the end. If I did this size of number again, I think I'd use A=30 for Q=30-50M to boost yield at the bottom, and I'd increase lim's by 25% or so (I used 130/180M on this job).

VBCurtis · 2021-01-20, 04:19

First filtering pass:

Code:

Tue Jan 19 20:02:16 2021  begin with 368134973 relations and 408812974 unique ideals
Tue Jan 19 20:08:56 2021  reduce to 136356813 relations and 139341382 ideals in 25 passes

That leaves us a bit short of the necessary relations. I've restarted the CADO server, asking for 570M total relations (30M more).

Looks like Charybdis' machines clients were still waiting, as they got work immediately. Mine are running too, so if you've moved on to other tasks don't worry about your clients. 30M relations won't take long.

charybdis · 2021-01-20, 13:12

Quote:

Originally Posted by VBCurtis

First filtering pass:

Code:

Tue Jan 19 20:02:16 2021  begin with 368134973 relations and 408812974 unique ideals
Tue Jan 19 20:08:56 2021  reduce to 136356813 relations and 139341382 ideals in 25 passes

That leaves us a bit short of the necessary relations. I've restarted the CADO server, asking for 570M total relations (30M more).

Looks like Charybdis' machines clients were still waiting, as they got work immediately. Mine are running too, so if you've moved on to other tasks don't worry about your clients. 30M relations won't take long.

That looks a lot more than 30M relations short to me. To get a good matrix I'd want the initial number of relations to be a bit bigger than the number of ideals, so you probably want 100M more raw relations to get another ~70M unique.

Would be a good idea to set rels_wanted to something like 800M so the clients keep going, and keep running msieve filtering until you get a decent matrix.

PS. most of my machines rebooted overnight to install updates, apologies

EdH · 2021-01-20, 14:30

I'll be tasking my farm back to you shortly. The "night owl" machines were running a night project.

VBCurtis · 2021-01-20, 17:24

OK, I changed target rels to 670M. I'll try msieve-filtering this afternoon at ~570M raw rels and report back.

VBCurtis · 2021-01-21, 04:10

578M raw relations got me 391M uniques, and a 39.98M matrix at TD=84.

There was a deficit of 27M in relations vs ideals in the first step of filtering, so I'm continuing to sieve. I think 610M or so should be enough, and we're at 600M now, so feel free to repurpose your machines any time.

I'll leave the server to run till morning (USA-pacific time), and will run a final filtering job with however many relations I have when I wake up.

VBCurtis · 2021-01-21, 15:37

597M relations built a 34.4M matrix at TD 100 overnight.

I lost internet connection to the CADO server overnight, so we quit serving workunits at 615M relations. That'll be enough, so we're "officially" done sieving. I'll report on the actual matrix size and ETA later today.

Edit: 615M raw relations yielded 414M uniques, enough for a 31M matrix at TD 116. No ETA yet, I'll edit again in a few hours once the matrix is running.

EDIT2: ETA is just under 10 days, running on all 20 cores without MPI. I recently updated that machine to Ubuntu 20.04, so I may try to get the MPI fix applied and see how msieve-MPI performs on it.

EdH · 2021-01-22, 18:31

Quote:

Originally Posted by VBCurtis

. . . I recently updated that machine to Ubuntu 20.04, so I may try to get the MPI fix applied and see how msieve-MPI performs on it.

You shouldn't need to fix anything (if you're referring to openmpi), but you will probably need to have the same Ubuntu version on any other machines you plan to include in your cluster. The following should be the only thing necessary and it should really already be accomplished if you already had openmpi installed in the earlier version:

Code:

apt update
apt install openmpi-bin openmpi-common libopenmpi-dev

VBCurtis · 2021-01-22, 19:57

No plans to use another machine- simply to MPI across the two sockets, as the reduction in cross-messaging is reported to give 15-20% speedup compared to the current "msieve -t 20" for the two ten-core sockets.

I'll play with it tonight- gotta go pick up a Ferrari this afternoon! The very best kind of errand, especially when it's not my car. :)

swellman · 2021-02-06, 21:36

Yoyo@Home is finishing the batch of base-2 Cunningham numbers (1987 edition). Should be completely finished in another week or two. Each composite in this group has survived 12000 curves @B1 = 850e6. One iteration down, five more to go. Of course that’s not counting any earlier ECM work.

Next step? Following a strategy of finish-what-you-started, I can just queue all the surviving composites back into Yoyo for another round of 0.16t65 (ETA ~13 months). Or we could select a small subset of “interesting” cases for focused ECM and eventual NFS by Greg on NFS@Home. There are two quartics - start with them? Or others?

With the recent extension of the base-2 tables, do we leave some of Yoyo’s resources available to work that effort? Or is ECM on Yoyo even needed with this latest batch of Cunninghams?

My opinion is to split the baby - focus on a few of the 1987 base-2 Cunninghams and leave room for other work. Feeds Greg a few more candidates by this fall and avoids flooding Yoyo’s queue. But I’m certainly no Solomon...

2021-01-20, 00:44	#222
VBCurtis "Curtis" Feb 2005 Riverside, CA 2·2,339 Posts	We reached 540M relations, and CADO crashed on the first filtering step with this error: Code: Error:Filtering - Duplicate Removal, removal pass: terminate called after throwing an instance of 'renumber_t::corrupted_table' Error:Filtering - Duplicate Removal, removal pass: what(): Renumber table is corrupt: cannot find data for prime 0x28e17329 ; note: isprime(p)==0 Error:Filtering - Duplicate Removal, removal pass: Note: above 2^32 ideals or relations, add FLAGS_SIZE="-DSIZEOF_P_R_VALUES=8 -DSIZEOF_INDEX=8" to local.sh I've seen this error on small (~C110) jobs with recent checkouts on another machine, which caused me to stick with the May-2020 version that works on that other machine. I used a fresh git-checkout for this job, and am disappointed to see that it's a bigger bug than I guessed. In our case, it shouldn't much matter- I'm going to use msieve for postprocessing anyway. I'll let y'all know what I've got after filtering. Edit: remdups leaves me 368.0M uniques, a pleasant 68%. Q-range was 30M to 264.6M; yield fell off quite a bit at the end. If I did this size of number again, I think I'd use A=30 for Q=30-50M to boost yield at the bottom, and I'd increase lim's by 25% or so (I used 130/180M on this job). Last fiddled with by VBCurtis on 2021-01-20 at 02:44

2021-01-20, 04:19	#223
VBCurtis "Curtis" Feb 2005 Riverside, CA 2×2,339 Posts	First filtering pass: Code: Tue Jan 19 20:02:16 2021 begin with 368134973 relations and 408812974 unique ideals Tue Jan 19 20:08:56 2021 reduce to 136356813 relations and 139341382 ideals in 25 passes That leaves us a bit short of the necessary relations. I've restarted the CADO server, asking for 570M total relations (30M more). Looks like Charybdis' machines clients were still waiting, as they got work immediately. Mine are running too, so if you've moved on to other tasks don't worry about your clients. 30M relations won't take long.

2021-01-21, 15:37	#228
VBCurtis "Curtis" Feb 2005 Riverside, CA 2×2,339 Posts	597M relations built a 34.4M matrix at TD 100 overnight. I lost internet connection to the CADO server overnight, so we quit serving workunits at 615M relations. That'll be enough, so we're "officially" done sieving. I'll report on the actual matrix size and ETA later today. Edit: 615M raw relations yielded 414M uniques, enough for a 31M matrix at TD 116. No ETA yet, I'll edit again in a few hours once the matrix is running. EDIT2: ETA is just under 10 days, running on all 20 cores without MPI. I recently updated that machine to Ubuntu 20.04, so I may try to get the MPI fix applied and see how msieve-MPI performs on it. Last fiddled with by VBCurtis on 2021-01-21 at 20:11

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Cunningham ECM efforts	pinhodecarlos	Cunningham Tables	7	2017-12-21 13:29
Cunningham ECM Now Futile?	R.D. Silverman	GMP-ECM	4	2012-04-25 02:45
Cunningham Project on YouTube	Batalov	Cunningham Tables	0	2012-02-26 02:58
Extended Cunningham or so	rekcahx	Factoring	6	2011-08-19 12:45
Introduction: ECM work done on Cunningham Project composites	garo	Cunningham Tables	2	2005-01-20 10:06

2021-01-18, 21:23	#221
charybdis Apr 2020 193 Posts	Just thrown in 45 more cores to help speed things along to the finish line.

2021-01-20, 14:30	#225
EdH "Ed Hall" Dec 2009 Adirondack Mtns 5×727 Posts	I'll be tasking my farm back to you shortly. The "night owl" machines were running a night project.

2021-01-20, 17:24	#226
VBCurtis "Curtis" Feb 2005 Riverside, CA 1001001000110₂ Posts	OK, I changed target rels to 670M. I'll try msieve-filtering this afternoon at ~570M raw rels and report back.

2021-01-21, 04:10	#227
VBCurtis "Curtis" Feb 2005 Riverside, CA 2·2,339 Posts	578M raw relations got me 391M uniques, and a 39.98M matrix at TD=84. There was a deficit of 27M in relations vs ideals in the first step of filtering, so I'm continuing to sieve. I think 610M or so should be enough, and we're at 600M now, so feel free to repurpose your machines any time. I'll leave the server to run till morning (USA-pacific time), and will run a final filtering job with however many relations I have when I wake up.

2021-01-22, 19:57	#230
VBCurtis "Curtis" Feb 2005 Riverside, CA 2·2,339 Posts	No plans to use another machine- simply to MPI across the two sockets, as the reduction in cross-messaging is reported to give 15-20% speedup compared to the current "msieve -t 20" for the two ten-core sockets. I'll play with it tonight- gotta go pick up a Ferrari this afternoon! The very best kind of errand, especially when it's not my car. :)

2021-02-06, 21:36	#231
swellman Jun 2012 B94₁₆ Posts	Yoyo@Home is finishing the batch of base-2 Cunningham numbers (1987 edition). Should be completely finished in another week or two. Each composite in this group has survived 12000 curves @B1 = 850e6. One iteration down, five more to go. Of course that’s not counting any earlier ECM work. Next step? Following a strategy of finish-what-you-started, I can just queue all the surviving composites back into Yoyo for another round of 0.16t65 (ETA ~13 months). Or we could select a small subset of “interesting” cases for focused ECM and eventual NFS by Greg on NFS@Home. There are two quartics - start with them? Or others? With the recent extension of the base-2 tables, do we leave some of Yoyo’s resources available to work that effort? Or is ECM on Yoyo even needed with this latest batch of Cunninghams? My opinion is to split the baby - focus on a few of the 1987 base-2 Cunninghams and leave room for other work. Feeds Greg a few more candidates by this fall and avoids flooding Yoyo’s queue. But I’m certainly no Solomon...