OFFICIAL "SERVER PROBLEMS" THREAD

[tha]

The recent cleared list shows only C for composite and no F for factor for a couple of hours, which is very counterintuitive, or a sign of something being wrong.

[kladner]

Originally Posted by tha

The recent cleared list shows only C for composite and no F for factor for a couple of hours, which is very counterintuitive, or a sign of something being wrong.

The factors were the hangup. This morning (my time) the entire set of results went through. All the NF results came up with Error 40, not needed. The two factors cleared and displayed credit.

[Madpoo]

Originally Posted by kladner

The factors were the hangup. This morning (my time) the entire set of results went through. All the NF results came up with Error 40, not needed. The two factors cleared and displayed credit.

Having looked at the current server now, I can see that there are times when the disk subsystem really starts to get overwhelmed... it will start to page excessively and that results in SQL queries taking longer... if there are any writes involved, then the transaction log starts growing since it can't keep up, and everything else on the system starts to crawl.

I don't know exactly what kinds of things can trigger those episodes, but it is interesting to see the cascade effect in action.

The replacement system has much faster and more disks. More memory and 64-bit helps too because I think the real problem right now is the memory crunch from the large SQL dataset. It could possibly be helped by limiting SQL to 1 GB of the 2 GB total... the overall system could be more responsive in general and not page so much, but SQL does work better the more memory it has. I just wonder how much it really helps to say SQL can use 1GB or nearly 2GB when the database is in the dozens of GB in size.

I'd look at that angle more, reducing the SQL memory allocations, but hopefully that old server's days are numbered.

I'm not as available this coming week for testing but it's set up enough that George is doing more functionality tests... so far I've had v4 and v5 clients connect and get/check-in results and I think all of the website stuff is working. There were a few (minor, I think) PHP incompatibilities with the upgrade from 5.2 to 5.5 along the way but those would have needed fixing at some point anyway, so that's good.

In summary, better things are coming.

[LaurV]

Good work! Eagerly waiting...

[Mark Rose]

Originally Posted by Madpoo

The replacement system has much faster and more disks. More memory and 64-bit helps too because I think the real problem right now is the memory crunch from the large SQL dataset. It could possibly be helped by limiting SQL to 1 GB of the 2 GB total... the overall system could be more responsive in general and not page so much, but SQL does work better the more memory it has. I just wonder how much it really helps to say SQL can use 1GB or nearly 2GB when the database is in the dozens of GB in size.

I have no experience with MSSQL, but in general, databases can more efficiently manage memory with larger internal caches than relying on an OS file system cache because they have additional insight into the data the OS lacks. If you reduce the memory allocated to MSSQL, the OS will use more memory to cache files recently written to, which is pointless if those files are also cached in MSSQL, and MSSQL will end up doing more disk reads as its caches are smaller.

Running a separate virtual machine for the database server is a good way to avoid having other processes page the database memory out. I would certainly do it.

[M29]

Originally Posted by Madpoo

I can see that there are times when the disk subsystem really starts to get overwhelmed...

The system seems to be out picking daisies when returning the Top Producers >> Totals Overall.

I think it is supposed to return the Top 500 ("/report_top_500/"). Instead it returns all 4500+.

[kladner]

In summary, better things are coming.

That's very good to hear. Thanks for all your efforts and contributions.

[Madpoo]

Originally Posted by M29

The system seems to be out picking daisies when returning the Top Producers >> Totals Overall.

I think it is supposed to return the Top 500 ("/report_top_500/"). Instead it returns all 4500+.

Oh, weird. You're right. That's more than 500.

That's one of the reports that gets generated hourly, but the server right now does not do any gzip compression, so that large > 1MB of data gets sent to the client via the scenic route.

I did test out switching the current server to use the built in PHP compression and that works, but I wasn't entirely sure if the v4/v5 API communications will work okay if the client gets gzipped data back from it's requests.

On the replacement server I avoided that by just using built in IIS compression for the main site and have it disabled for all of the API related calls. It might work anyway... the clients are probably using whatever built-in HTTP for it's OS which *should* be able to decompress any gzipped responses but I'm not going to test all of them just for that, and those calls don't normally return much data.

Suffice to say, when doing a webpagetest.org check of that page on the current server it can take 10-20 seconds. On the test server it was taking 1-2 seconds tops. 1+ MB of data compressed down to 150KB or so.

I guess if that report was fixed to just generate the top 500 and not the larger set, it wouldn't take as long anyway.

Another fun stat is how long it takes the server to run the SQL tasks that generate those hourly stats. On the current server it could take anywhere from 3 minutes to as much as 20 minutes, just depending on how much disk thrashing was going on, but on average I think I saw it taking 4-5 minutes. The test server is consistently doing it in 35-40 *seconds*.

That's not to say the final server replacement will do exactly the same, since we're testing this all on a virtual machine right now since that was easier for me to get going, but it is a close match to the physical box itself.

[Madpoo]

Originally Posted by Madpoo

...when doing a webpagetest.org check of that page on the current server it can take 10-20 seconds.

If you're curious, here's the webpagetest I did on the current server for that page:
http://www.webpagetest.org/result/140805_E3_3QR/

[VictordeHolland]

The site seems to be more responsive and uploading manual results was faster than the past two weeks, did you guys do anything?

[Prime95]

Originally Posted by VictordeHolland

The site seems to be more responsive and uploading manual results was faster than the past two weeks, did you guys do anything?

Nope, you were just lucky.

The site does seem happier with 5GB of free disk space. Madpoo is out of town this week so there won't be any work on the new server til he gets back.