Factor found that should have been found by P-1

[manfred4]

Originally Posted by James Heinrich

The list is updated once per day, roughly 11pm Eastern time.

If you notice a particular exponent that should be on the list but isn't, and the factor you found was reported to mersenne.org more than 24h ago, please let me know.

This one: http://www.mersenne.ca/exponent/20943323
It hasn't found its way into your database of missed factors

[James Heinrich]

Originally Posted by manfred4

This one: http://www.mersenne.ca/exponent/20943323
It hasn't found its way into your database of missed factors

I'm not sure, perhaps the regenerator was feeling grumpy last night, I ran it manually and it's showing there now.

edit: confession: I forgot to check the page to confirm that it wasn't there before I regenerated the data

[tha]

Originally Posted by James Heinrich

I'm not sure, perhaps the regenerator was feeling grumpy last night, I ran it manually and it's showing there now.

OK, how about this one? (I still find a substantial amount of them every week and many don't pop up.)


P-1 found a factor in stage #1, B1=6501.
.......M12815641 has a factor: 80707884945969040271

[James Heinrich]

Originally Posted by tha

OK, how about this one?

Hmm, what's going on. There's no reason that shouldn't be included, and indeed if I run it manually it shows up fine. I've turned on some additional logging for the automated run tonight to see if I can figure out what's happening. I'll see what I find out tomorrow morning.

Two related notes:
a) Exponent detail pages for affected exponents now include a note next to the P-1 record if a factor is known to have been missed (e.g.)

b) I've recently sent my spiders hunting for the details for 37000 exponents which had incomplete data on P-1 runs. Notably when a P-1 effort is reported on recent results or recent cleared reports it doesn't include information on bounds used. My spider has already gone through all the affected exponents from 2012-2015 and is working its way back in time. As new P-1 results are found they will be reasonably quickly (within an hour or so) updated with the correct bounds.

[tha]

Originally Posted by James Heinrich

Hmm, what's going on. There's no reason that shouldn't be included, and indeed if I run it manually it shows up fine. I've turned on some additional logging for the automated run tonight to see if I can figure out what's happening. I'll see what I find out tomorrow morning.

If tracking is still turned on see how it handles exponent 12911999

[James Heinrich]

Remind me tomorrow

[tha]

Originally Posted by tha

If tracking is still turned on see how it handles exponent 12911999

New day, new exponent (12930067) ready to be analysed. Looks like some unreliable (overclocked?) machine did a lot of P-1 at the time.

[James Heinrich]

Originally Posted by tha

If tracking is still turned on see how it handles exponent 12911999

It was corrected added to the list of missed factors in the automated overnight run.

[tha]

Originally Posted by James Heinrich

I just discovered that my list of missing exponents... was missing some exponents

So last night's 1160 exponents is now down to 295.

The list has shrunk further to 82. Only the last two exponents in the 12M range are still in there. All the others are gone. Could it be that exponents make it into the list and then disappear immediately or later?

I noticed the B1 and B2 values have been integrated differently recently as well. Does the program take into account the day of the finding of the exponent and the last P-1 run before that date?

[James Heinrich]

Originally Posted by tha

The list has shrunk further to 82.

I had to re-spider all the exponents in this (and another) list to sync with PrimeNet data for another reason, hence the recent fluctuations.

There are two huge problems with making this list sane, both due to lacking data from PrimeNet:

• old (pre-v5) P-1 runs were not really recorded, just the max-bounds-used was updated, but the data of who ran with what bounds and when was just tossed
• When a factor is found (any method) the previously used P-1 bounds (and TF levels) are purged

PrimeNet considers an exponent with a factor no longer interesting and tosses out "unnecessary" historical data that it really shouldn't. But I don't have control over that so I just have to muddle through with what I have.

The "missed P-1" list now has ~20k entries, including a whole bunch of small (under 1M) exponents where one or more small factors was already known and then later additional P-1 was done to find more factors (while ignoring already-known factors). These shouldn't be in the "missed P-1" list, but since PrimeNet refuses to record dates for when TF and P-1 was done on exponents with known factors, I can't differentiate between a factor found by TF and missed by P-1, vs a factor found by TF and then explicitly ignored on a later P-1 run looking for additional factors. Random example (of thousands) M367541.

The only real solution is to
a) stop PrimeNet from throwing out data
b) try and recover previously tossed data by re-parsing all the submitted data from logs dating back as far as they go.

I doubt this will happen anytime soon, if ever.

[Madpoo]

Originally Posted by James Heinrich

I had to re-spider all the exponents in this (and another) list to sync with PrimeNet data for another reason, hence the recent fluctuations.

There are two huge problems with making this list sane, both due to lacking data from PrimeNet:

• old (pre-v5) P-1 runs were not really recorded, just the max-bounds-used was updated, but the data of who ran with what bounds and when was just tossed
• When a factor is found (any method) the previously used P-1 bounds (and TF levels) are purged

PrimeNet considers an exponent with a factor no longer interesting and tosses out "unnecessary" historical data that it really shouldn't. But I don't have control over that so I just have to muddle through with what I have.

The "missed P-1" list now has ~20k entries, including a whole bunch of small (under 1M) exponents where one or more small factors was already known and then later additional P-1 was done to find more factors (while ignoring already-known factors). These shouldn't be in the "missed P-1" list, but since PrimeNet refuses to record dates for when TF and P-1 was done on exponents with known factors, I can't differentiate between a factor found by TF and missed by P-1, vs a factor found by TF and then explicitly ignored on a later P-1 run looking for additional factors. Random example (of thousands) M367541.

The only real solution is to
a) stop PrimeNet from throwing out data
b) try and recover previously tossed data by re-parsing all the submitted data from logs dating back as far as they go.

I doubt this will happen anytime soon, if ever.

I keep meaning to set something up to get XMLs put together of all old info. I thought about doing it on a per-day basis... would that be a hassle to parse, having to go through 365 of them for a 1-year period? Would it make it better to do it on a monthly basis instead, with the tradeoff being larger XML files?

Apart from that, is there another data dump I can send your way to help you backfill anything so you can avoid having to crawl it from the website?

And then, last question, what data is missing when a factor is found that you would like to see retained? There's a good chance maybe it's still there, but the website just isn't showing it any longer.

I have a feeling that with the server constraints in the past, there was an actual need to keep the DB size from growing too much. As it is right now, there's more space. I don't know if it's good to have the existing DB get so large that doing backups on it or other maintenance becomes a chore ,but perhaps instead of just deleting the "irrelevant" details it could be archived to a separate DB on it's own backup schedule or whatever.

Specifically I'd imagine things like assignment histories. Expired assignments are kept but assignments that finish up regularly are removed, so there's a loss of info like when it was assigned in the first place.