Modifications to LL assignment rules!!!

[Prime95]

This thread is for discussion of modifications to the LL assignment rules. After a plan of action is decided upon we will re-visit DC assignment rules. Refer to milestones thread for some of the recent discussions on this issue. As a result of those discussions, I've proposed in this post a new SQL query to run nightly that will give us more metrics for more intelligent decision making.


Background:

1) Cat 1 & 2 assignments were reserved for those users that promised to complete assignments quickly.
2) We assumed GIMPS completed 100,000 LL tests a years. Thus, by setting the cat 3 cutoff at the first 100,000 exponents, a user getting a cat 4 assignment would have at least a year to complete before getting in the way of a milestone.
3) We selected 90/150/270/360 days to complete an assignment. Exponents not recycled until they reach category 1.

Current problems:

1) The number of computers signed up for cat 1 & 2 assignments is too low.
2) Some cat 1 assignments are going to computers that are too slow, suspect, or have a history of abandoning assignments.
3) The very smallest exponents (a.k.a. cat 0) might need even stricter limits to advance milestones faster are discourage poaching.

Old metrics:

1) CPU speed (adjusted by rolling average, hours-per-day, and number of workers)
2) Number of recent LL completions
3) Computer's days-of-work-get value. NOTE, we do not have access to the days between-checkins value.
4) Computer's reliability value -- a rather gross estimate of how error-prone the computer is.

New metrics (from a new SQL query run once a day):

1) Computer's rank ordered by LL/DC GHZ-days completed over the last 90 days.
2) Computer's number completed LL/DC attempts for last 90 days.
3) Computer's number of suspect LL/DC results for the last 90 days.
4) Computer's number of expired assignments for the last 90 days.

Data:

Today's cat cutoffs (and available assignments, not including those that GPU-to-72 has):
cat 3: 76.15M 15,866
cat 2: 68.83M 2,962
cat 1: 67.76M 3,137

Same info one year ago:
cat 3: 71.46M 62,752
cat 2: 64.45M 5,833
cat 1: 58.46M 2,141

Assignments in the last 30 days:
cat 1: 837 (639 active, 198 completed)
cat 2: 16 (all still active)
cat 3: 6244 (5924 active, 320 completed)
cat 4: 5128 (5020 active, 108 completed)

Total completed LL tests in last 30 days grouped by exponent's category the day it was assigned:
cat 1: 799
cat 2: 22
cat 3: 4655
cat 4: 3036


Goals:

1) Regular progress on milestones
2) Reduce the vast number of available cat 1/2/3 assignments (i.e. compress the ranges if possible)
3) Allow users with slower computers to get LL tests without getting in the way of milestones.
4) If necessary, adjust manual testing ranges
5) Don't piss off users by recycling assignments that are making decent enough progress.
6) Where possible, rules should be self-adjusting. Examples: rather than "computer should have a CPU speed of 3GHz" we have a rule of "computer is ranked in the top 1000 producers over the last 90 days". Another example: instead of "cat 2 is the first 10000 exponents", we could try "cat 2 is the number of exponents we expect the top 500 computers could complete in the next 120 days".

[Prime95]

I will update the first post as the discussion develops. Thus, you can refer back to see the current state of affairs without re-reading the entire thread.

First up, do we need more data to help in our thinking?

[chalsall]

Originally Posted by Prime95

First up, do we need more data to help in our thinking?

Thanks for re-opening this discussion George. Much needed.

Just to put on the table, GPU72 has recently been focusing on building up a buffer of candidates TF'ed to 75 bits to give to P-1'ers. Some of these are in the high Cat 3 range (75M), with the rest in the low Cat 4 range (76M & 77M) to feed to the "churners" (which have now largely stabilized after the latest MP announcement).

The goal has been to ensure that no candidate is assigned for P-1'ing without first being appropriately TF'ed, and that no candidate is assigned for LL'ing without first being appropriately P-1'ed.

We have also been working on TF'ing the low Cat 3 range (which already has P-1 done). I've redirected some additional amount of our firepower down there as our aforementioned "to be P-1'ed" buffer is looking pretty comfortable at the moment.

[Mark Rose]

I would restrict access to Cat 1/2 DC to computers who have produced verified results in two of the last three results, and no suspect or bad results in the last three results.

[Uncwilly]

Can we also open discussion on what folks call Cat 0?

When an exponent in cat 1 expires (or a DC has a mismatch), it gets thrown back in the pool of cat 1's for another 90 days. This has happened twice to the same exponent. That stretches the time out to nearly 270 days.

Also, exponents that are picked up in Cat 2 and expire after showing progress then stalling become 'late' cat 1 entries. Then if they expire, they are hold outs.

A cat 1* or cat 1' or cat 0 would be for such stragglers (Cat 1 expirees and Cat 2's that expire into the lower part of Cat 1). They would get handed out to top 25 machines or to forumites that have pledged 25 day max turn around (on some sort of fair basis).

[Prime95]

Originally Posted by Uncwilly

Can we also open discussion on what folks call Cat 0?

Yes, kinda. I'd like the discussion somewhat limited to LL rules. So question #1 is do we feel there is a need for a cat 0 in first-time LL testing?? If we agree there is a need, then let's come up with proposed rules keeping in mind that it would be great if such a rule would also make sense for DC.

That said, I like your solution. Something like a top 100 machine in addition to a reduced time limit. Possibly in addition to limiting it to users that have checked the smallest exponents box.

[retina]

Originally Posted by Prime95

5) Don't piss off users ...

I think this is the most important rule. The rule to rule them all. Without the users you got nothing.

[cuBerBruce]

Originally Posted by Prime95

1) Computer's rank ordered by LL/DC GHZ-days completed over the last 90 days.

I have no idea where my computers currently rank among all the GIMPS computers. Would the CPUs page be modified to show us the ranking of each computer in the last 90 days?

With the upward surge in the cat 1 exponent wavefront in the last few months, some older computers of mine are nearing the point where they might not finish them in the required time limit, using 1 core per worker. I've been thinking of switching them to two cores per worker, but under the proposed new rules, I don't know if that will allow them to qualify for cat 1 LLs. I may be forced to switch them to cat 2 work.

Originally Posted by Prime95

Yes, kinda. I'd like the discussion somewhat limited to LL rules. So question #1 is do we feel there is a need for a cat 0 in first-time LL testing?? If we agree there is a need, then let's come up with proposed rules keeping in mind that it would be great if such a rule would also make sense for DC.

With at least four users poaching very low exponents in recent weeks to (apparently) prevent them from the possibility of another "lengthy" assignment, and at least two exponents being double-poached, I kind of think this suggests something is needed.

[axn]

Originally Posted by Prime95

1) The number of computers signed up for cat 1 & 2 assignments is too low.

I don't think this assertion is correct. From the other thread, there was a datapoint that suggested appr 25 completed cat1 LLs/day, so < 6 months to advance thru the cat 1 range. That seems entirely acceptable.
I don't know about cat 2 completion rate, but I have yet to hear an argument about what _should_ be the progress of cat 1/2 waves (and justifications for it). Perhaps we should start there.

Originally Posted by Prime95

2) Some cat 1 assignments are going to computers that are too slow, suspect, or have a history of abandoning assignments.

This would definitely be a problem. IMO, the whole idea of category system was to avoid "sticking nails" exponents blocking milestones for a long time. Anything that can be done to reduce this problem would be good. But even here, things have improved drastically. Right now, we're looking at a few months to achieve a milestone instead of years.

Originally Posted by Prime95

3) The very smallest exponents (a.k.a. cat 0) might need even stricter limits to advance milestones faster are discourage poaching.

Since the number of exponents that needs babysitting are small, manual intervention is perfectly acceptable. We don't need to automate every little thing.

Originally Posted by Prime95

1) Progress on milestones faster

No. Keeping _regular_ progress of milestones is the goal (i.e. preventing extreme hold ups). Actual milestone progress is a function of the combined LL completion from all the categories. It might appear that cat 1 & 2 are the ones achieving milestones, but that is short-sighted view. If the cat 3 & 4 have done a good job, then cat 1 & 2 will go thru that range faster when they eventually get to it. Moving machines from cat3/4 to cat1/2 only helps with milestones in the short run. To reiterate "Keeping _regular_ progress of milestones is the goal"

Prime95: Noted and fixed in original post

Originally Posted by Prime95

2) Reduce the vast number of available cat 1/2/3 assignments (i.e. compress the ranges if possible)

This is something that can be easily done. There is nothing sacrosanct about 4k/10k cutoffs. We can adjust it up/down based on the (perceived) progress of the milestones.

Originally Posted by Prime95

3) Allow users with slower computers to get LL tests without getting in the way of milestones.

The correct focus!

Originally Posted by Prime95

First up, do we need more data to help in our thinking?

Absolutely. I am yet to see an actual well-reasoned argument that there is a problem with the current system. It seems to be a case of "perfect is the enemy of good enough". More data can surely help.

Originally Posted by cuBerBruce

With at least four users poaching very low exponents in recent weeks to (apparently) prevent them from the possibility of another "lengthy" assignment, and at least two exponents being double-poached, I kind of think this suggests something is needed.

Four is a very small number. We have to ask ourselves if it is really worth it to address it.

[chalsall]

Originally Posted by axn

I don't think this assertion is correct. From the other thread, there was a datapoint that suggested appr 25 completed cat1 LLs/day, so < 6 months to advance thru the cat 1 range. That seems entirely acceptable I don't know about cat 2 completion rate, but I have yet to hear an argument about what _should_ be the progress of cat 1/2 waves (and justifications for it). Perhaps we should start there.

When you look at the current situation and see that there are 6,163 candidates in the Cat 1 and Cat 2 ranges awaiting to be assigned (plus another 1,013 Cat 2s about to be processed and released by GPU72 for assignment) out of the 10,000 candidates in the two ranges, it seems clear (at least to me) that something isn't optimal.

Please note (as also mentioned previously) that /very/ few Cat 2s are currently being assigned; almost all current Cat 2s were originally Cat 3s when assigned. Further, currently any Cat 2s which are assigned almost immediately become Cat 1s.

What this means is that at the current rate of ~27 completions of Cat 1s a day it's actually going to take ~370 days to finish up to where the Cat 3 range starts currently. Do we really want to still be working below 69M in a years time?

[Madpoo]

Originally Posted by axn

...
Since the number of exponents that needs babysitting are small, manual intervention is perfectly acceptable. We don't need to automate every little thing.
...

Yeah, people may have noticed I have no problem with doing manual interventions when I see a user is working on an assignment and will miss their expiration by just a few days. In cases like that I would personally like to see manual interventions in the way of "stretching" the expiration by a few days to allow them to complete the work.

The alternative in those situations is to let it expire, go to someone else, but then the original assignee finishes anyway in a few days and the new person gets "poached".

Some of these things could be fixed by merely making sure only the "quality" machines are getting those things in the first place, but even then there will be the odd instance of a category 4 assignment taking really long and making its way into a category zero situation, where it's the last one left blocking a milestone (or several). It's happened.

I guess that would mean we were completing cat 1/2/3 work too fast, if a cat 4 assignment caught up with them before 1 year had lapsed. Nice problem to have?