mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > PrimeNet

Reply
 
Thread Tools
Old 2015-04-26, 23:40   #958
lycorn
 
lycorn's Avatar
 
Sep 2002
Oeiras, Portugal

101011111102 Posts
Default

Quote:
Originally Posted by Mark Rose View Post
I was also trying to look at the full details of M1277 when the error started.
I´ve tried several times to have a look at those, but never managed. It appears to freeze while waiting for the results, until I give up and close the page. Prior to releasing the page, every request (in different tabs) I do just hangs, and is not serviced.
lycorn is offline   Reply With Quote
Old 2015-04-27, 04:11   #959
Madpoo
Serpentine Vermin Jar
 
Madpoo's Avatar
 
Jul 2014

63158 Posts
Default

Quote:
Originally Posted by Mark Rose View Post
I have spiders that submit results every minute, if there are any. I was also trying to look at the full details of M1277 when the error started. All my logged in requests would hang, until I stopped the spiders for a while, and whatever was locking things got released.
Ugh... yeah, M1277.

The problem with that one (and it pre-dated any changes I made) is that the full history has SOOOO many ECM curve results that loading it takes a very long time.

Even omitting the full history will still take a while.

I would recommend *avoiding* looking at M1277 until me or George can look at the queries involved and see if it can be sped up.
Madpoo is offline   Reply With Quote
Old 2015-04-27, 04:13   #960
Madpoo
Serpentine Vermin Jar
 
Madpoo's Avatar
 
Jul 2014

29×113 Posts
Default

Quote:
Originally Posted by Mark Rose View Post
Bleh. Looks like I lost 825 GHz-days of work or so. Ah well :/

I keep about 300 GHz-d/d in reserve (where I have to pay for power)... so I'll press that into service.
Oh, and if you let George know about that, he can hopefully correct that situation for ya.
Madpoo is offline   Reply With Quote
Old 2015-04-27, 04:36   #961
Prime95
P90 years forever!
 
Prime95's Avatar
 
Aug 2002
Yeehaw, FL

2·43·83 Posts
Default

Quote:
Originally Posted by Madpoo View Post
Oh, and if you let George know about that, he can hopefully correct that situation for ya.
There is a secret web page I can use to transfer TF work from anonymous back to you. Send me the results.txt for the lost work and your user id.
Prime95 is offline   Reply With Quote
Old 2015-04-27, 14:01   #962
James Heinrich
 
James Heinrich's Avatar
 
"James Heinrich"
May 2004
ex-Northern Ontario

60658 Posts
Default

Quote:
Originally Posted by Madpoo View Post
The problem with that one (and it pre-dated any changes I made) is that the full history has SOOOO many ECM curve results that loading it takes a very long time.
My approach in a case like that, where generating a page with a lot of slow queries but the contents are mostly static, is to cache the generated HTML to a database table. The generate will be slow the first time, but once the first person has looked at it it's lightning-fast for everyone else. Then whenever new data is submitted that would affect the displayed page (new ECM run, factor found, whatever) that row is easily cleared from the database. In a case like this you could even hardcode an exception for known-problem exponents like M1277 (or any of the small exponents with much ECM) to not clear the cache on a trivial update (yet another ECM curve run) but only if a factor is found or similar major change.
James Heinrich is online now   Reply With Quote
Old 2015-04-27, 14:30   #963
Mark Rose
 
Mark Rose's Avatar
 
"/X\(‘-‘)/X\"
Jan 2013

23·359 Posts
Default

Quote:
Originally Posted by Madpoo View Post
Oh, and if you let George know about that, he can hopefully correct that situation for ya.
No, the results didn't get submitted as anonymous. They were simply lost. The script I was using lost the results when I killed all the hanging copies. It had removed it from the results.txt before having successfully submitted the results. I consider it a bug that I'll look at fixing.

I've already reprocessed a third of the lost assignments using my "reserve" hardware. I really just want to keep consistent throughput. :)
Mark Rose is offline   Reply With Quote
Old 2015-04-27, 23:03   #964
Madpoo
Serpentine Vermin Jar
 
Madpoo's Avatar
 
Jul 2014

CCD16 Posts
Default

Quote:
Originally Posted by James Heinrich View Post
My approach in a case like that, where generating a page with a lot of slow queries but the contents are mostly static, is to cache the generated HTML to a database table. The generate will be slow the first time, but once the first person has looked at it it's lightning-fast for everyone else. Then whenever new data is submitted that would affect the displayed page (new ECM run, factor found, whatever) that row is easily cleared from the database. In a case like this you could even hardcode an exception for known-problem exponents like M1277 (or any of the small exponents with much ECM) to not clear the cache on a trivial update (yet another ECM curve run) but only if a factor is found or similar major change.
Well, I take it back. It actually is something I added that made something like M1277 go so slow.

There's a query that pulls all the data and it uses some loose JOINS that involves duplicates being generated for certain sets of data. In the case of M1277 although there are *only* 1023 log entries for ECM results, by the time the joins to certain other non-distinct tables are involved, the report for M1277 involves no less than 91,520 rows.

Once PHP gets a hold of it all, it removes dupe entries based on the type of data being displayed... still not really optimal but it's also worth noting that this works equally well for single or multiple exponents... all things said it's not too bad.

Where I messed up was when I tried to "fix" the slow display where it would show *all* of the ECM info by letting you toggle the display of "no factors found by ECM" on or off. But I thought I would helpfully calculate how many curves in total had been run as some sort of probably un-useful tip.

Trouble is, I did that in a subselect of the main query and I think it was doing a big SUM calculation on each row... 91,520 times.

So... yeah, it's no wonder that if you tried to include all of the ECM data (which would still sum up the # of curves) it would basically time out.

If you left the "full ECM history" box unchecked then there's only 1496 rows, and while it still does my stupid sum in each row it will finish in 30-60 seconds if you're patient enough.

Solution: I'm removing the mostly useless "recent curves run" from being displayed. Be aware that if you still feel like looking at all the times someone did ECM on a number and found nothing, in the most extreme case like M1277 it will still take a while to show everything (maybe 30-40 seconds).

My advice? If you're crawling for importing data into your own setup, please do yourself a favor and use the XML version of the exponent report. e.g.:
http://www.mersenne.org/report_expon...hp?exp_lo=1277

I wrote that from scratch and it's just using raw SQL "for xml" and it's VERY fast. That reminds me that I meant to put just a little link on each report page so you could click and view the XML data for that exponent. Right now it's kind of a hidden feature.
Madpoo is offline   Reply With Quote
Old 2015-04-27, 23:59   #965
Prime95
P90 years forever!
 
Prime95's Avatar
 
Aug 2002
Yeehaw, FL

2×43×83 Posts
Default

I just added tons of v4 log entries to the results history. This may slow down the full history when using report exponent. Also, some of the old log entries might be misformatted or incorrect. Please let us know of any problems!
Prime95 is offline   Reply With Quote
Old 2015-04-28, 04:46   #966
Mark Rose
 
Mark Rose's Avatar
 
"/X\(‘-‘)/X\"
Jan 2013

287210 Posts
Default

Quote:
Originally Posted by Prime95 View Post
I just added tons of v4 log entries to the results history. This may slow down the full history when using report exponent. Also, some of the old log entries might be misformatted or incorrect. Please let us know of any problems!
Are any from sannerud's laptop? If so, did the v4 entries make it into this list?
Mark Rose is offline   Reply With Quote
Old 2015-04-28, 05:15   #967
Madpoo
Serpentine Vermin Jar
 
Madpoo's Avatar
 
Jul 2014

63158 Posts
Default

Quote:
Originally Posted by Mark Rose View Post
Are any from sannerud's laptop? If so, did the v4 entries make it into this list?
Oh, good question.

Unfortunately it's a little difficult to map the old v4 entries to their v5 user id, if they mapped their v4/v5 usernames. I looked at doing that but there's a few tricky bits.

But I can say that by looking at the old logs, I see 4664 distinct exponents where sannerud did some TF and didn't find anything.

4352 of those still have zero factors found. I don't know what parameters George used previously to identify the work to redo? Was it just the exponent, or was it narrowed down to the bit ranges in particular? It'd also be good to check these and see if any had already been re-checked by someone. A very crude look shows "most" of those have been checked by someone else, but I don't know if it was the same bit ranges or not.

584 exponents, on the other hand, have *only* been checked by sannerud, in any range. Bear in mind, I'm tired and I should know better than to do SQL queries with any authority when I'm tired.

For George's benefit, the query looks something like (somewhat obfuscated just for whatever reason):
Code:
select distinct exponent from <v4 log> where <user text> like 'sanner%' and <result>=<no factor by TF>
   and exponent not in (select distinct exponent from <factors>)
   and exponent not in (select distinct exponent from <v5 log> where <result>=<no factor by TF> and <id>  != <sannerud's user id )
 order by exponent
He (she? I have no idea) TF'd some exponents multiple times for whatever reason.

Here's one for someone to run with and see what happens:
M23562031

It was tested 6 times by sannerud between 2003-12-02 and 2004-08-04 but in each case the result logged is like "...M23562031 no factor to 2^67..."
Madpoo is offline   Reply With Quote
Old 2015-04-28, 05:58   #968
Mark Rose
 
Mark Rose's Avatar
 
"/X\(‘-‘)/X\"
Jan 2013

23×359 Posts
Default

Quote:
Originally Posted by Madpoo View Post
I see 4664 distinct exponents where sannerud did some TF and didn't find anything.

4352 of those still have zero factors found.
That's a factor found rate of 6.7%, which is low. It's worth investigating further, I think.
Mark Rose is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Official "Faits erronés dans de belles-lettres" thread ewmayer Lounge 39 2015-05-19 01:08
Official "all-Greek-to-me Fiction Literature and Cinema" Thread ewmayer Science & Technology 41 2014-04-16 11:54
Official "Lasciate ogne speranza" whinge-thread cheesehead Soap Box 56 2013-06-29 01:42
Official "Ernst is a deceiving bully and George is a meanie" thread cheesehead Soap Box 61 2013-06-11 04:30
Official "String copy Statement Considered Harmful" thread Dubslow Programming 19 2012-05-31 17:49

All times are UTC. The time now is 18:57.

Mon Oct 26 18:57:36 UTC 2020 up 46 days, 16:08, 0 users, load averages: 1.67, 1.88, 2.00

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.