Improved params files for CADO

[charybdis]

I'm fairly confident that A=28 with adjust_strategy=2 beats both I=14 and I=15 at c172. The I=14/A=28 crossover is probably between c165 and c170 IMO. There's been a bit of discussion in the c165-c170 params thread.

[bur]

Removed, better suited for the c165-c170 discussion thread.

[EdH]

Originally Posted by charybdis

I'm fairly confident that A=28 with adjust_strategy=2 beats both I=14 and I=15 at c172. The I=14/A=28 crossover is probably between c165 and c170 IMO. There's been a bit of discussion in the c165-c170 params thread.

I thought about that a little bit, but I'm too ignorant of some of the associated parameters and I'm trying for a blanket params file that will work in all my varied machines without too many individualizations. Perhaps it's time to rework the scripts to allow for all of that. I am already having the client scripts adjusting thread/process counts based on memory. I can probably do the same for strategy. . .

(. . .wandering off trying to remember why "A=" didn't work the last time I tried it. . .)

[charybdis]

Originally Posted by EdH

(. . .wandering off trying to remember why "A=" didn't work the last time I tried it. . .)

Possibly the strategy-2 composite factors bug. Once in every few hundred million relations, a relation will have a composite factor, and this makes CADO filtering quit, taking down the server with it. The solution is to use msieve instead, but that makes automation more difficult.

The other solution would be for CADO to throw out bad relations without quitting. Maybe I should suggest this to them someday...

[VBCurtis]

I think in Ed's case it's that at some point CADO stopped compiling for core-2 era chips, so some of his farm uses a CADO version from before A= sievers were implemented.
I ran into this myself on my ancient dell C6100 32-core core2 server. In my case, I chose to run ECM on the core2 chips during the 4-6 weeks a year I fire that beast up as an electric heater, giving up on CADO on that old old hardware.

[EdH]

Originally Posted by charybdis

Possibly the strategy-2 composite factors bug. Once in every few hundred million relations, a relation will have a composite factor, and this makes CADO filtering quit, taking down the server with it. The solution is to use msieve instead, but that makes automation more difficult.

The other solution would be for CADO to throw out bad relations without quitting. Maybe I should suggest this to them someday...

Originally Posted by VBCurtis

I think in Ed's case it's that at some point CADO stopped compiling for core-2 era chips, so some of his farm uses a CADO version from before A= sievers were implemented.
I ran into this myself on my ancient dell C6100 32-core core2 server. In my case, I chose to run ECM on the core2 chips during the 4-6 weeks a year I fire that beast up as an electric heater, giving up on CADO on that old old hardware.

Curtis' memory is obviously better than mine. That is correct, however the Core 2 issue was fixed in later versions and all my machines are now running whatever I had to upgrade to for the project Oliver ran.

I do have automation running Msieve for >c125. Msieve is only run if a matrix can be built. If successful, it kills CADO and proceeds. If unsuccessful it lets CADO finish. The Msieve script is triggered by a local client receiving the 410 signal. I even have a second (and third) machine setup to automatically start other processes when the main (or second) machine goes into LA.

I will take a look at "A=" soon, but I think my trials will be delayed past a pending c168 that looks like it will run in a few minutes. Do I simply swap "tasks.I = 14" to "tasks.A = 28" and what differences in other params?

Thanks.

[VBCurtis]

Originally Posted by EdH

I will take a look at "A=" soon, but I think my trials will be delayed past a pending c168 that looks like it will run in a few minutes. Do I simply swap "tasks.I = 14" to "tasks.A = 28" and what differences in other params?

Thanks.

Yep- you're going up "half" a siever, so yield will go up by about 40%, and you'll need about 70% of the Q you would have needed for I=14. However, since bur's tests are showing that Q-max to Q-min ratio should be smaller than previously designed, I'd leave Q-min alone.

[charybdis]

Originally Posted by EdH

Do I simply swap "tasks.I = 14" to "tasks.A = 28" and what differences in other params?

Apart from what Curtis said, make sure to add tasks.sieve.adjust_strategy=2 if you don't have it already.

[EdH]

Thanks Guys! I will work on this to try and implement it for the next run. I missed this one and it's already sieving. I still need to do some Msieve tuning again, also. I stepped from 24 to 40 threads and want to test the MPI workings again. For some reason, I haven't been using MPI. I think it was due to factor harvesting from the log file, but I have so many things to rework, I'm almost lost with where to start.

[EdH]

I'm looking at the implementation of strategy 2 for my machines and have a couple questions. I'm hoping to add the fewest scripts possible to my mix.

- In the case of adding strategy 2, if I start by using the default, can I add strategy 2 via my clients at first?

- Is strategy 2 dependent on "A=" or will it be of use for other "I=" runs?

- I think there was discussion with HenryZ over a memory limit switch. Is that correct, or is my memory failing me yet again? If available, should I try using it with my "lesser" machines, or just go without strategy 2?

- Now the complicated one to describe: I'm running all my clients as 4-threaded right now and then filling the available machines with clients, e.g. 40-thread machine has 10 clients. If I go with strategy 2, is this still a good setup, for those machines that can handle the memory?

[charybdis]

Originally Posted by EdH

I'm looking at the implementation of strategy 2 for my machines and have a couple questions. I'm hoping to add the fewest scripts possible to my mix.

- In the case of adding strategy 2, if I start by using the default, can I add strategy 2 via my clients at first?

Yes, "--override adjust-strategy 2" should work fine. Note the hyphen rather than underscore here.

- Is strategy 2 dependent on "A=" or will it be of use for other "I=" runs?

It's useful for "I=" (i.e. odd A value) on large enough numbers. For c150+ it usually increases yield by a few percent without sacrificing much speed. But the advantage is bigger for even values of A.

- I think there was discussion with HenryZ over a memory limit switch. Is that correct, or is my memory failing me yet again? If available, should I try using it with my "lesser" machines, or just go without strategy 2?

This is -bkthresh1. Lower values use less RAM but are slower. Above max(lim0, lim1) it has no effect. You'll need to test different values out to see if it's at all useful for your setup. For ~c170 you might want to try 5M, 10M, 20M. But I suspect if you have enough memory for strategy 0 then -bkthresh1 will sieve too slowly to be helpful.

- Now the complicated one to describe: I'm running all my clients as 4-threaded right now and then filling the available machines with clients, e.g. 40-thread machine has 10 clients. If I go with strategy 2, is this still a good setup, for those machines that can handle the memory?

Nothing wrong with this as long as there's enough memory. And indeed above 4 threads/client you probably won't see full CPU usage.