Let's change the NFS@home sievers!

VBCurtis · 2020-08-31, 02:39

After nearly 10 years in service, the 14e queue has outlived its usefulness. Most of the workunits (not jobs) in 14e are for numbers that would be faster on 15e, but the 15e queue is lengthy and the 14e queue often runs dry, so they're sent to 14e somewhat wastefully.

So, why don't we retire 14e, and change lasieved to the 15e siever? In order to maintain three BOINC projects within NFS@home, I have a couple of ideas:

1. lasieved w/15e jobs should have lim's restricted to maintain a low-memory option for users on older machines. Limits of 134/200M or 134/134M would keep memory use about where it is now (note some 14e jobs use lim's of 268M!) Note this limit would not be hard-coded, so we could let it rise over the years as jobs and memory capacities get bigger.

2.If we change d to 15e, the lasieve-e BOINC project should change to 16e (or, failing that, 15e -J 15). Many of the current "e" jobs would be more efficient on 16e, and the queue is long in part because of these huge jobs. Further, there are quite a few GNFS-190+ or SNFS-275+ jobs of interest among the projects we collectively care about, so this queue would stay well-fed and yet faster than the current arrangement (since each job of that size is faster on 16e than 15e).

For me, the second change is not required by the first- we could have 15e-small and 15e-big for the two queues and still come out more efficient overall.

Note that going to larger sievers also tends to make smaller matrices for each job, or equivalently require somewhat fewer relations to yield a matrix.

Upsides:
1. Lots of small-15e jobs are currently homeless, either waiting months in the 15e queue or wasting resources on 14e. These jobs are very common in our work now, and the jobs continue to grow over time. It's worth it to change now, and will become more valuable in the future.

2. We have the hardware to handle matrices of 50M size, and using 16e would allow us to handle up to GNFS-205 or SNFS-300 (? Not sure, I don't do much SNFS) without troubling Greg on the big queue.

Downsides:
1. Some small jobs, e.g. 14e with 30LP or smaller, will be less efficient on 15e than 14e. This computational waste is much less than the savings available from jobs currently run as 14e/32, e.g. GNFS-175ish. The really small ones (e.g.. GNFS 155 or less) can be done on a home desktop anyway.

2. Memory use might rise on the "e" queue. We should be careful to keep 16e lim's small at the outset to make sure memory use doesn't drastically change if we do this.

3. If "d" queue has a strict lim cap, there will still be a size of job that's homeless, with 16e slow but 15e/134M too restrictive. We can handle this as it happens, likely by sending to whichever queue is shorter.

4. Greg might not have time nor interest in the effort required to make this change happen.

Thoughts?

xilman · 2020-08-31, 07:46

Quote:

Originally Posted by VBCurtis

...

Thoughts?

I would support this.

Some time ago NFS@home ran a number of 14e SNFS for me when there will still suitable candidates. After they dried up I stopped submitting because the 15e queue was even then over-subscribed.

pinhodecarlos · 2020-08-31, 08:14

Seconded.
The only recommendation is to also update the memory requirements on the project, since beginning people complain about this, just take a look at the forum. 14e using more memory than 15e and 16e, or 15e more than 16e.

swellman · 2020-08-31, 12:53

I’m all for such changes.

VBCurtis · 2020-08-31, 15:55

Quote:

Originally Posted by pinhodecarlos

Seconded.
The only recommendation is to also update the memory requirements on the project, since beginning people complain about this, just take a look at the forum. 14e using more memory than 15e and 16e, or 15e more than 16e.

Carlos-
As our resident BOINC-people expert, what should be the max memory use for the "d" subproject, and for "e"? I can work out maximum lim's on 15e for "d" from your suggestion.

Or, should we go about it the other way, measuring RAM use for a typical new "d" job and then claim memory needs are a little higher than that?

What do the rest of you think of a cap at GNFS-185 for the new "d"? That can be run with lim's of 134M. What would a corresponding SNFS difficulty be? 275 (assuming small sextic-poly coefficients)?

If frmky doesn't reply here within a week or so, I'll PM him about this idea before I get too carried away with memory measurements/etc.

jyb · 2020-08-31, 16:24

Yes, I believe the time has come for this change. I support it. Thank you for getting the ball rolling, Mike.

pinhodecarlos · 2020-08-31, 16:26

Curtis,

Beyond memory requirements there’s also the credits requirements and badge requirements. Everything is related, in the beginning Greg wanted to credit more for the app using more memory (and higher difficulty jobs) the 16e, but when you guys started to run 32-bit jobs with higher lims on the 14e all started to fall apart, that’s when the complains started.

Badge lower granularity is also a problem, it is impossible for comum mortals to achieve them, only with computer farms.

Regarding memory requirements what I’m seeing is that the number of cores increase is not
followed by more RAM per core. It’s stalled to 2GB/core, so I would say you could at the limit double the memory requirements as they are now stated on the projects details. I would suggest the following but it is better to request Greg to query the database to see what’s the average memory available per core.

-d version to 0.75 GB
-e version to 1.25 GB
-f version to 1.75 GB

pinhodecarlos · 2020-08-31, 16:34

Everything needs to be balanced, it is not only about the memory. It’s a trade off between memory usage, wu size, wu processing timings, level of granularity of the available badges.

frmky · 2020-09-01, 07:52

Would all of the needs be met by the creation of lasievee_small (15e smaller numbers) and lasievef_small (16e smaller numbers) queues? On the back end these would be the same as the existing lasievee and lasievef queues, but guidelines for size and memory requirements for submitted jobs could be determined by the community and evolve over time as needed.

These new queues could have their own separate stats and badges, or could be combined with the existing queues' stats (d+e_small and e+f_small for example). Carlos?

pinhodecarlos · 2020-09-01, 09:03

Quote:

Originally Posted by frmky

Would all of the needs be met by the creation of lasievee_small (15e smaller numbers) and lasievef_small (16e smaller numbers) queues? On the back end these would be the same as the existing lasievee and lasievef queues, but guidelines for size and memory requirements for submitted jobs could be determined by the community and evolve over time as needed.

These new queues could have their own separate stats and badges, or could be combined with the existing queues' stats (d+e_small and e+f_small for example). Carlos?

Separate queues, separate badges otherwise clients will not run it for own milestones.

VBCurtis · 2020-09-01, 15:00

Quote:

Originally Posted by pinhodecarlos

Separate queues, separate badges otherwise clients will not run it for own milestones.

Is this true even if the old d and e queues cease to exist?

That is, if the 15e-small replaces d and counts for d badges, they would still stop running it?

Doesn't matter to us queue-feeders what badges are given, I merely wish to make sure we're talking about the same thing.

2020-08-31, 02:39	#1
VBCurtis "Curtis" Feb 2005 Riverside, CA 13555₈ Posts	Let's change the NFS@home sievers! After nearly 10 years in service, the 14e queue has outlived its usefulness. Most of the workunits (not jobs) in 14e are for numbers that would be faster on 15e, but the 15e queue is lengthy and the 14e queue often runs dry, so they're sent to 14e somewhat wastefully. So, why don't we retire 14e, and change lasieved to the 15e siever? In order to maintain three BOINC projects within NFS@home, I have a couple of ideas: 1. lasieved w/15e jobs should have lim's restricted to maintain a low-memory option for users on older machines. Limits of 134/200M or 134/134M would keep memory use about where it is now (note some 14e jobs use lim's of 268M!) Note this limit would not be hard-coded, so we could let it rise over the years as jobs and memory capacities get bigger. 2.If we change d to 15e, the lasieve-e BOINC project should change to 16e (or, failing that, 15e -J 15). Many of the current "e" jobs would be more efficient on 16e, and the queue is long in part because of these huge jobs. Further, there are quite a few GNFS-190+ or SNFS-275+ jobs of interest among the projects we collectively care about, so this queue would stay well-fed and yet faster than the current arrangement (since each job of that size is faster on 16e than 15e). For me, the second change is not required by the first- we could have 15e-small and 15e-big for the two queues and still come out more efficient overall. Note that going to larger sievers also tends to make smaller matrices for each job, or equivalently require somewhat fewer relations to yield a matrix. Upsides: 1. Lots of small-15e jobs are currently homeless, either waiting months in the 15e queue or wasting resources on 14e. These jobs are very common in our work now, and the jobs continue to grow over time. It's worth it to change now, and will become more valuable in the future. 2. We have the hardware to handle matrices of 50M size, and using 16e would allow us to handle up to GNFS-205 or SNFS-300 (? Not sure, I don't do much SNFS) without troubling Greg on the big queue. Downsides: 1. Some small jobs, e.g. 14e with 30LP or smaller, will be less efficient on 15e than 14e. This computational waste is much less than the savings available from jobs currently run as 14e/32, e.g. GNFS-175ish. The really small ones (e.g.. GNFS 155 or less) can be done on a home desktop anyway. 2. Memory use might rise on the "e" queue. We should be careful to keep 16e lim's small at the outset to make sure memory use doesn't drastically change if we do this. 3. If "d" queue has a strict lim cap, there will still be a size of job that's homeless, with 16e slow but 15e/134M too restrictive. We can handle this as it happens, likely by sending to whichever queue is shorter. 4. Greg might not have time nor interest in the effort required to make this change happen. Thoughts?

2020-08-31, 16:26	#7
pinhodecarlos "Carlos Pinho" Oct 2011 Milton Keynes, UK 3×7×13×19 Posts	Curtis, Beyond memory requirements there’s also the credits requirements and badge requirements. Everything is related, in the beginning Greg wanted to credit more for the app using more memory (and higher difficulty jobs) the 16e, but when you guys started to run 32-bit jobs with higher lims on the 14e all started to fall apart, that’s when the complains started. Badge lower granularity is also a problem, it is impossible for comum mortals to achieve them, only with computer farms. Regarding memory requirements what I’m seeing is that the number of cores increase is not followed by more RAM per core. It’s stalled to 2GB/core, so I would say you could at the limit double the memory requirements as they are now stated on the projects details. I would suggest the following but it is better to request Greg to query the database to see what’s the average memory available per core. -d version to 0.75 GB -e version to 1.25 GB -f version to 1.75 GB Last fiddled with by pinhodecarlos on 2020-08-31 at 16:28

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
ECM change	Prime95	PrimeNet	28	2020-09-02 08:16
Compiling GNFS sievers on AArch64 platform	wombatman	Programming	11	2017-03-11 03:12
gnfs asm version sievers illegal instruction	EdH	Factoring	32	2016-10-12 20:49
Name Change?	Fred	Lounge	8	2016-01-31 17:42
Calling all 64-bit Linux sievers!	frmky	NFS@Home	25	2013-10-16 15:58

2020-08-31, 08:14	#3
pinhodecarlos "Carlos Pinho" Oct 2011 Milton Keynes, UK 5187₁₀ Posts	Seconded. The only recommendation is to also update the memory requirements on the project, since beginning people complain about this, just take a look at the forum. 14e using more memory than 15e and 16e, or 15e more than 16e.

2020-08-31, 12:53	#4
swellman Jun 2012 7×11×53 Posts	I’m all for such changes.

2020-08-31, 16:24	#6
jyb Aug 2005 Seattle, WA 2×929 Posts	Yes, I believe the time has come for this change. I support it. Thank you for getting the ball rolling, Mike.

2020-08-31, 16:34	#8
pinhodecarlos "Carlos Pinho" Oct 2011 Milton Keynes, UK 3·7·13·19 Posts	Everything needs to be balanced, it is not only about the memory. It’s a trade off between memory usage, wu size, wu processing timings, level of granularity of the available badges.

2020-09-01, 07:52	#9
frmky Jul 2003 So Cal 2682₁₀ Posts	Would all of the needs be met by the creation of lasievee_small (15e smaller numbers) and lasievef_small (16e smaller numbers) queues? On the back end these would be the same as the existing lasievee and lasievef queues, but guidelines for size and memory requirements for submitted jobs could be determined by the community and evolve over time as needed. These new queues could have their own separate stats and badges, or could be combined with the existing queues' stats (d+e_small and e+f_small for example). Carlos?