Server outrages - Page 55

gd_barnes · 2021-08-20, 04:44

Stats are back in maintenance mode.

AMDave · 2021-08-20, 14:10

At some point there was an event that caused corruption the MySQL and that corruption was retained in the backups so restoring from back up could not excise the problem.
The old prpnet1470 database was already unrecoverable, but I had to forcibly remove it to restore integrity.

The processing logs were really quite perplexing until the timestamps revealed the picture.
Somewhere in MySQL was enough of a problem to cause the sql steps to take longer and longer to complete, until they were too long.
Eventually the daily and hourly processes overlapped.
As one process created the update tables another one removed them.
That is how the stats summaries disappeared.

I re-optimised the configuration of the MySQL server itself,
I have re-optimised all of the databases: both ports and stats.
It was close to optimal but I have refined it further.
Bounced it and bench marked it several times.
Its looking better, for an old dog. (It is a great grandfather now!)

I have merged the daily and hourly stats processes so there is now only one process.
That comes at a small cost in processing time.
The original hourly process target was 3 minutes and the daily process time was 10 minutes.
The single process will now run every hour including midnight and take up to 20 minutes depending on how many results you can pump into it :)
I have removed the weakest point in the process (the 'blind-update-and-switch' summary step) which is where the web stats were disappearing from.

And finally I have added a 'hard' semaphore that prevents the process from running over itself.
So the process will attempt to run each hour, as before, but if a massive slowdown occurs again it will run to completion without interruption and will not start again until the start of the hour after it has completed.

I will have to come back and change that hard semaphore to a soft semaphore at a more convenient time.

Back out of maintenance mode.
Monitoring the log files closely.

gd_barnes · 2021-08-20, 20:09

Thank you very much, Dave, for that tremendous effort.

I had seen that prpnet1470 caused a big issue over a year ago. Hence I never attempted to run it again. It's too bad that its corruption caused the backups to become corrupted.

To everyone: The NPLB stats pages look good now!

odicin · 2021-08-27, 05:19

Very nice...

I noticed that there are still some incorrect entries on the free-dc stats page (Top 10 Users, Show All entries). Possibly, the export for this also has to be touched again.

Regards Odi

AMDave · 2021-09-08, 12:44

I have posted on FDC asking Bok to refresh the project stats from the latest NPLB extracts

AMDave · 2021-09-09, 11:47

Bok advised that his new stats server no longer refreshes the stats of non-BOINC projects.
However he is looking into making NPLB an exception.
There is a bit of work and goodwill involved so if it happens don't forget to tell him thank you :)

Mini-Geek · 2022-04-17, 17:39

Looks like NPLB servers are down since 11am.

gd_barnes · 2022-04-17, 19:56

My internet service is down. I am putting in a call to Spectrum now.

Sorry about the outage.

gd_barnes · 2022-04-17, 21:54

I definitely am having a problem with my router. I’ve done all the usual reboots and restarts. Unplugging and plugging back in. Etc.

Spectrum confirmed the issue. Likely the earliest that they can be here is around 11am CDT US on Monday. (4pm GMT). There is a small chance that they may be able to come out this evening.

I will continue to keep everyone updated.

gd_barnes · 2022-04-18, 00:45

Spectrum will not be here until Monday around 11am local time.

In the meantime if you are running an NPLB PRPnet server you can try to connect with the following:
server=G9000:100:1:192.168.0.110:9000

This would be entered in your prpclient.ini file if you are running port 9000.

You can also try connecting to the main NPLB page by entering 192.168.0.110 for the URL.

I have extended the expiration on all the work units to 48 hours.

WraithX · 2022-04-18, 01:55

Quote:

Originally Posted by gd_barnes

You can also try connecting to the main NPLB page by entering 192.168.0.110 for the URL.

Just to let you know, 192.168.x.x is not routable on the internet, it is a local-only (private) subnet. No one but you will be able to reach that server via that ip address.

You can read more about private networks on wikipedia: https://en.wikipedia.org/wiki/Private_network

2021-08-20, 14:10	#596
AMDave Jan 2006 deep in a while-loop 29A₁₆ Posts	At some point there was an event that caused corruption the MySQL and that corruption was retained in the backups so restoring from back up could not excise the problem. The old prpnet1470 database was already unrecoverable, but I had to forcibly remove it to restore integrity. The processing logs were really quite perplexing until the timestamps revealed the picture. Somewhere in MySQL was enough of a problem to cause the sql steps to take longer and longer to complete, until they were too long. Eventually the daily and hourly processes overlapped. As one process created the update tables another one removed them. That is how the stats summaries disappeared. I re-optimised the configuration of the MySQL server itself, I have re-optimised all of the databases: both ports and stats. It was close to optimal but I have refined it further. Bounced it and bench marked it several times. Its looking better, for an old dog. (It is a great grandfather now!) I have merged the daily and hourly stats processes so there is now only one process. That comes at a small cost in processing time. The original hourly process target was 3 minutes and the daily process time was 10 minutes. The single process will now run every hour including midnight and take up to 20 minutes depending on how many results you can pump into it :) I have removed the weakest point in the process (the 'blind-update-and-switch' summary step) which is where the web stats were disappearing from. And finally I have added a 'hard' semaphore that prevents the process from running over itself. So the process will attempt to run each hour, as before, but if a massive slowdown occurs again it will run to completion without interruption and will not start again until the start of the hour after it has completed. I will have to come back and change that hard semaphore to a soft semaphore at a more convenient time. Back out of maintenance mode. Monitoring the log files closely. Last fiddled with by AMDave on 2021-08-20 at 14:11

2021-08-20, 20:09	#597
gd_barnes May 2007 Kansas; USA 10,889 Posts	Thank you very much, Dave, for that tremendous effort. I had seen that prpnet1470 caused a big issue over a year ago. Hence I never attempted to run it again. It's too bad that its corruption caused the backups to become corrupted. To everyone: The NPLB stats pages look good now! Last fiddled with by gd_barnes on 2021-08-20 at 20:09

2022-04-18, 00:45	#604
gd_barnes May 2007 Kansas; USA 10,889 Posts	Spectrum will not be here until Monday around 11am local time. In the meantime if you are running an NPLB PRPnet server you can try to connect with the following: server=G9000:100:1:192.168.0.110:9000 This would be entered in your prpclient.ini file if you are running port 9000. You can also try connecting to the main NPLB page by entering 192.168.0.110 for the URL. I have extended the expiration on all the work units to 48 hours. Last fiddled with by gd_barnes on 2022-04-18 at 00:51

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Gerbicz Error Checking - Originally: Would a "rack server" run the same as a desktop server	EdH	Hardware	16	2021-03-29 16:43
Use of a virtual private server (VPS) - normally a web server	drkirkby	Hardware	11	2021-03-15 09:11
is the server down	Unregistered	Information & Answers	1	2008-11-23 14:05
server down?	masser	Sierpinski/Riesel Base 5	2	2008-05-24 05:55
New Server Hardware and price quotes, Funding the server	Angular	PrimeNet	32	2002-12-09 01:12

2021-08-20, 04:44	#595
gd_barnes May 2007 Kansas; USA 10,889 Posts	Stats are back in maintenance mode.

2021-08-27, 05:19	#598
odicin Sep 2011 Potsdam, Germany 211₈ Posts	Very nice... I noticed that there are still some incorrect entries on the free-dc stats page (Top 10 Users, Show All entries). Possibly, the export for this also has to be touched again. Regards Odi

2021-09-08, 12:44	#599
AMDave Jan 2006 deep in a while-loop 2×3²×37 Posts	I have posted on FDC asking Bok to refresh the project stats from the latest NPLB extracts

2021-09-09, 11:47	#600
AMDave Jan 2006 deep in a while-loop 666₁₀ Posts	Bok advised that his new stats server no longer refreshes the stats of non-BOINC projects. However he is looking into making NPLB an exception. There is a bit of work and goodwill involved so if it happens don't forget to tell him thank you :)

2022-04-17, 17:39	#601
Mini-Geek Account Deleted "Tim Sorbera" Aug 2006 San Antonio, TX USA 2×3×23×31 Posts	Looks like NPLB servers are down since 11am.

2022-04-17, 19:56	#602
gd_barnes May 2007 Kansas; USA 10,889 Posts	My internet service is down. I am putting in a call to Spectrum now. Sorry about the outage.

2022-04-17, 21:54	#603
gd_barnes May 2007 Kansas; USA 10,889 Posts	I definitely am having a problem with my router. I’ve done all the usual reboots and restarts. Unplugging and plugging back in. Etc. Spectrum confirmed the issue. Likely the earliest that they can be here is around 11am CDT US on Monday. (4pm GMT). There is a small chance that they may be able to come out this evening. I will continue to keep everyone updated.