Severe outrages - Page 55

gd_barnes · 2021-08-20, 04:44

Stats are back in maintenance mode.

AMDave · 2021-08-20, 14:10

At some point there was an event that caused corruption the MySQL and that corruption was retained in the backups so restoring from back up could not excise the problem.
The old prpnet1470 database was already unrecoverable, but I had to forcibly remove it to restore integrity.

The processing logs were really quite perplexing until the timestamps revealed the picture.
Somewhere in MySQL was enough of a problem to cause the sql steps to take longer and longer to complete, until they were too long.
Eventually the daily and hourly processes overlapped.
As one process created the update tables another one removed them.
That is how the stats summaries disappeared.

I re-optimised the configuration of the MySQL server itself,
I have re-optimised all of the databases: both ports and stats.
It was close to optimal but I have refined it further.
Bounced it and bench marked it several times.
Its looking better, for an old dog. (It is a great grandfather now!)

I have merged the daily and hourly stats processes so there is now only one process.
That comes at a small cost in processing time.
The original hourly process target was 3 minutes and the daily process time was 10 minutes.
The single process will now run every hour including midnight and take up to 20 minutes depending on how many results you can pump into it :)
I have removed the weakest point in the process (the 'blind-update-and-switch' summary step) which is where the web stats were disappearing from.

And finally I have added a 'hard' semaphore that prevents the process from running over itself.
So the process will attempt to run each hour, as before, but if a massive slowdown occurs again it will run to completion without interruption and will not start again until the start of the hour after it has completed.

I will have to come back and change that hard semaphore to a soft semaphore at a more convenient time.

Back out of maintenance mode.
Monitoring the log files closely.

gd_barnes · 2021-08-20, 20:09

Thank you very much, Dave, for that tremendous effort.

I had seen that prpnet1470 caused a big issue over a year ago. Hence I never attempted to run it again. It's too bad that its corruption caused the backups to become corrupted.

To everyone: The NPLB stats pages look good now!

odicin · 2021-08-27, 05:19

Very nice...

I noticed that there are still some incorrect entries on the free-dc stats page (Top 10 Users, Show All entries). Possibly, the export for this also has to be touched again.

Regards Odi

AMDave · 2021-09-08, 12:44

I have posted on FDC asking Bok to refresh the project stats from the latest NPLB extracts

AMDave · 2021-09-09, 11:47

Bok advised that his new stats server no longer refreshes the stats of non-BOINC projects.
However he is looking into making NPLB an exception.
There is a bit of work and goodwill involved so if it happens don't forget to tell him thank you :)

2021-08-20, 14:10	#596
AMDave Jan 2006 deep in a while-loop 666₁₀ Posts	At some point there was an event that caused corruption the MySQL and that corruption was retained in the backups so restoring from back up could not excise the problem. The old prpnet1470 database was already unrecoverable, but I had to forcibly remove it to restore integrity. The processing logs were really quite perplexing until the timestamps revealed the picture. Somewhere in MySQL was enough of a problem to cause the sql steps to take longer and longer to complete, until they were too long. Eventually the daily and hourly processes overlapped. As one process created the update tables another one removed them. That is how the stats summaries disappeared. I re-optimised the configuration of the MySQL server itself, I have re-optimised all of the databases: both ports and stats. It was close to optimal but I have refined it further. Bounced it and bench marked it several times. Its looking better, for an old dog. (It is a great grandfather now!) I have merged the daily and hourly stats processes so there is now only one process. That comes at a small cost in processing time. The original hourly process target was 3 minutes and the daily process time was 10 minutes. The single process will now run every hour including midnight and take up to 20 minutes depending on how many results you can pump into it :) I have removed the weakest point in the process (the 'blind-update-and-switch' summary step) which is where the web stats were disappearing from. And finally I have added a 'hard' semaphore that prevents the process from running over itself. So the process will attempt to run each hour, as before, but if a massive slowdown occurs again it will run to completion without interruption and will not start again until the start of the hour after it has completed. I will have to come back and change that hard semaphore to a soft semaphore at a more convenient time. Back out of maintenance mode. Monitoring the log files closely. Last fiddled with by AMDave on 2021-08-20 at 14:11

2021-08-20, 20:09	#597
gd_barnes May 2007 Kansas; USA 10517₁₀ Posts	Thank you very much, Dave, for that tremendous effort. I had seen that prpnet1470 caused a big issue over a year ago. Hence I never attempted to run it again. It's too bad that its corruption caused the backups to become corrupted. To everyone: The NPLB stats pages look good now! Last fiddled with by gd_barnes on 2021-08-20 at 20:09

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
prime64 causing severe unresponsiveness	Freightyard	Software	14	2011-11-11 00:22

2021-08-20, 04:44	#595
gd_barnes May 2007 Kansas; USA 13×809 Posts	Stats are back in maintenance mode.

2021-08-27, 05:19	#598
odicin Sep 2011 Potsdam, Germany 2×61 Posts	Very nice... I noticed that there are still some incorrect entries on the free-dc stats page (Top 10 Users, Show All entries). Possibly, the export for this also has to be touched again. Regards Odi

2021-09-08, 12:44	#599
AMDave Jan 2006 deep in a while-loop 29A₁₆ Posts	I have posted on FDC asking Bok to refresh the project stats from the latest NPLB extracts

2021-09-09, 11:47	#600
AMDave Jan 2006 deep in a while-loop 2×3²×37 Posts	Bok advised that his new stats server no longer refreshes the stats of non-BOINC projects. However he is looking into making NPLB an exception. There is a bit of work and goodwill involved so if it happens don't forget to tell him thank you :)