Register FAQ Search Today's Posts Mark Forums Read

2015-04-26, 23:40   #958
lycorn

Sep 2002
Oeiras, Portugal

23×52×7 Posts

Quote:
 Originally Posted by Mark Rose I was also trying to look at the full details of M1277 when the error started.
I´ve tried several times to have a look at those, but never managed. It appears to freeze while waiting for the results, until I give up and close the page. Prior to releasing the page, every request (in different tabs) I do just hangs, and is not serviced.

2015-04-27, 04:11   #959
Serpentine Vermin Jar

Jul 2014

63128 Posts

Quote:
 Originally Posted by Mark Rose I have spiders that submit results every minute, if there are any. I was also trying to look at the full details of M1277 when the error started. All my logged in requests would hang, until I stopped the spiders for a while, and whatever was locking things got released.
Ugh... yeah, M1277.

The problem with that one (and it pre-dated any changes I made) is that the full history has SOOOO many ECM curve results that loading it takes a very long time.

Even omitting the full history will still take a while.

I would recommend *avoiding* looking at M1277 until me or George can look at the queries involved and see if it can be sped up.

2015-04-27, 04:13   #960
Serpentine Vermin Jar

Jul 2014

2·1,637 Posts

Quote:
 Originally Posted by Mark Rose Bleh. Looks like I lost 825 GHz-days of work or so. Ah well :/ I keep about 300 GHz-d/d in reserve (where I have to pay for power)... so I'll press that into service.
Oh, and if you let George know about that, he can hopefully correct that situation for ya.

2015-04-27, 04:36   #961
Prime95
P90 years forever!

Aug 2002
Yeehaw, FL

5×1,433 Posts

Quote:
 Originally Posted by Madpoo Oh, and if you let George know about that, he can hopefully correct that situation for ya.
There is a secret web page I can use to transfer TF work from anonymous back to you. Send me the results.txt for the lost work and your user id.

2015-04-27, 14:01   #962
James Heinrich

"James Heinrich"
May 2004
ex-Northern Ontario

2·72·31 Posts

Quote:
 Originally Posted by Madpoo The problem with that one (and it pre-dated any changes I made) is that the full history has SOOOO many ECM curve results that loading it takes a very long time.
My approach in a case like that, where generating a page with a lot of slow queries but the contents are mostly static, is to cache the generated HTML to a database table. The generate will be slow the first time, but once the first person has looked at it it's lightning-fast for everyone else. Then whenever new data is submitted that would affect the displayed page (new ECM run, factor found, whatever) that row is easily cleared from the database. In a case like this you could even hardcode an exception for known-problem exponents like M1277 (or any of the small exponents with much ECM) to not clear the cache on a trivial update (yet another ECM curve run) but only if a factor is found or similar major change.

2015-04-27, 14:30   #963
Mark Rose

"/X\(‘-‘)/X\"
Jan 2013

2×1,433 Posts

Quote:
 Originally Posted by Madpoo Oh, and if you let George know about that, he can hopefully correct that situation for ya.
No, the results didn't get submitted as anonymous. They were simply lost. The script I was using lost the results when I killed all the hanging copies. It had removed it from the results.txt before having successfully submitted the results. I consider it a bug that I'll look at fixing.

I've already reprocessed a third of the lost assignments using my "reserve" hardware. I really just want to keep consistent throughput. :)

2015-04-27, 23:03   #964
Serpentine Vermin Jar

Jul 2014

2×1,637 Posts

Quote:
 Originally Posted by James Heinrich My approach in a case like that, where generating a page with a lot of slow queries but the contents are mostly static, is to cache the generated HTML to a database table. The generate will be slow the first time, but once the first person has looked at it it's lightning-fast for everyone else. Then whenever new data is submitted that would affect the displayed page (new ECM run, factor found, whatever) that row is easily cleared from the database. In a case like this you could even hardcode an exception for known-problem exponents like M1277 (or any of the small exponents with much ECM) to not clear the cache on a trivial update (yet another ECM curve run) but only if a factor is found or similar major change.
Well, I take it back. It actually is something I added that made something like M1277 go so slow.

There's a query that pulls all the data and it uses some loose JOINS that involves duplicates being generated for certain sets of data. In the case of M1277 although there are *only* 1023 log entries for ECM results, by the time the joins to certain other non-distinct tables are involved, the report for M1277 involves no less than 91,520 rows.

Once PHP gets a hold of it all, it removes dupe entries based on the type of data being displayed... still not really optimal but it's also worth noting that this works equally well for single or multiple exponents... all things said it's not too bad.

Where I messed up was when I tried to "fix" the slow display where it would show *all* of the ECM info by letting you toggle the display of "no factors found by ECM" on or off. But I thought I would helpfully calculate how many curves in total had been run as some sort of probably un-useful tip.

Trouble is, I did that in a subselect of the main query and I think it was doing a big SUM calculation on each row... 91,520 times.

So... yeah, it's no wonder that if you tried to include all of the ECM data (which would still sum up the # of curves) it would basically time out.

If you left the "full ECM history" box unchecked then there's only 1496 rows, and while it still does my stupid sum in each row it will finish in 30-60 seconds if you're patient enough.

Solution: I'm removing the mostly useless "recent curves run" from being displayed. Be aware that if you still feel like looking at all the times someone did ECM on a number and found nothing, in the most extreme case like M1277 it will still take a while to show everything (maybe 30-40 seconds).

My advice? If you're crawling for importing data into your own setup, please do yourself a favor and use the XML version of the exponent report. e.g.:
http://www.mersenne.org/report_expon...hp?exp_lo=1277

I wrote that from scratch and it's just using raw SQL "for xml" and it's VERY fast. That reminds me that I meant to put just a little link on each report page so you could click and view the XML data for that exponent. Right now it's kind of a hidden feature.

 2015-04-27, 23:59 #965 Prime95 P90 years forever!     Aug 2002 Yeehaw, FL 716510 Posts I just added tons of v4 log entries to the results history. This may slow down the full history when using report exponent. Also, some of the old log entries might be misformatted or incorrect. Please let us know of any problems!
2015-04-28, 04:46   #966
Mark Rose

"/X\(‘-‘)/X\"
Jan 2013

54628 Posts

Quote:
 Originally Posted by Prime95 I just added tons of v4 log entries to the results history. This may slow down the full history when using report exponent. Also, some of the old log entries might be misformatted or incorrect. Please let us know of any problems!
Are any from sannerud's laptop? If so, did the v4 entries make it into this list?

2015-04-28, 05:15   #967
Serpentine Vermin Jar

Jul 2014

2×1,637 Posts

Quote:
 Originally Posted by Mark Rose Are any from sannerud's laptop? If so, did the v4 entries make it into this list?
Oh, good question.

Unfortunately it's a little difficult to map the old v4 entries to their v5 user id, if they mapped their v4/v5 usernames. I looked at doing that but there's a few tricky bits.

But I can say that by looking at the old logs, I see 4664 distinct exponents where sannerud did some TF and didn't find anything.

4352 of those still have zero factors found. I don't know what parameters George used previously to identify the work to redo? Was it just the exponent, or was it narrowed down to the bit ranges in particular? It'd also be good to check these and see if any had already been re-checked by someone. A very crude look shows "most" of those have been checked by someone else, but I don't know if it was the same bit ranges or not.

584 exponents, on the other hand, have *only* been checked by sannerud, in any range. Bear in mind, I'm tired and I should know better than to do SQL queries with any authority when I'm tired.

For George's benefit, the query looks something like (somewhat obfuscated just for whatever reason):
Code:
select distinct exponent from <v4 log> where <user text> like 'sanner%' and <result>=<no factor by TF>
and exponent not in (select distinct exponent from <factors>)
and exponent not in (select distinct exponent from <v5 log> where <result>=<no factor by TF> and <id>  != <sannerud's user id )
order by exponent
He (she? I have no idea) TF'd some exponents multiple times for whatever reason.

Here's one for someone to run with and see what happens:
M23562031

It was tested 6 times by sannerud between 2003-12-02 and 2004-08-04 but in each case the result logged is like "...M23562031 no factor to 2^67..."

2015-04-28, 05:58   #968
Mark Rose

"/X\(‘-‘)/X\"
Jan 2013

2·1,433 Posts

Quote:
 Originally Posted by Madpoo I see 4664 distinct exponents where sannerud did some TF and didn't find anything. 4352 of those still have zero factors found.
That's a factor found rate of 6.7%, which is low. It's worth investigating further, I think.

 Similar Threads Thread Thread Starter Forum Replies Last Post ewmayer Lounge 39 2015-05-19 01:08 ewmayer Science & Technology 41 2014-04-16 11:54 cheesehead Soap Box 56 2013-06-29 01:42 cheesehead Soap Box 61 2013-06-11 04:30 Dubslow Programming 19 2012-05-31 17:49

All times are UTC. The time now is 10:49.

Tue Sep 29 10:49:43 UTC 2020 up 19 days, 8 hrs, 0 users, load averages: 1.39, 1.34, 1.44