OFFICIAL "SERVER PROBLEMS" THREAD

[Mark Rose]

Originally Posted by Madpoo

Weird... 502 error is a gateway related error, like a proxy server in the path had an issue. There's no proxy directly in front of the current website... traffic goes directly to IIS from the outside world, not through a load balancer/caching proxy/whatever.

Any proxy seems like it'd be on the client's side, but the IIS/5.0 is kind of a giveaway that it did manage to hit the server in some way... might have been a request going through a proxy that timed out or something...that's my guess.

While not IIS, I've seen 502's happen with an Nginx+PHP sitation, where PHP fails. If PHP is not in-process, like it often is in Apache, in IIS, a PHP issue may have been what caused that.

[Mark Rose]

Originally Posted by James Heinrich

How was the result submitted? Manually? with misfit?

By calling this file: https://github.com/MarkRose/primetoo...ster/mfloop.py

It should send every line from mfakt?'s results.txt with a mersenne number string (M123456...) in it.

[chalsall]

Originally Posted by Madpoo

Weird... 502 error is a gateway related error, like a proxy server in the path had an issue. There's no proxy directly in front of the current website... traffic goes directly to IIS from the outside world, not through a load balancer/caching proxy/whatever.

I can support the observation that "502s" are quite common. Is it possible that Primenet is behind a proxy you don't know about?

I can further say but (quickly checking some of my spider's logs) can't document that such errors are highly correlative with other errors, such as "CGI Timeouts", "Database unavailable", etc.

If it would help I could instrument my spiders to collect deeper error statistics.

[James Heinrich]

Originally Posted by sdbardwick

It looks like a TF that was interpreted by PrimeNet as P-1

I would tend to agree. I would poke a little closer at the data... if mersenne.org was working right now.

Once we get the server migrated and more responsive I'll re-implement my results parsing code and hopefully put the whole misinterpreted results issue behind us.

[Madpoo]

Originally Posted by James Heinrich

I would tend to agree. I would poke a little closer at the data... if mersenne.org was working right now.

Once we get the server migrated and more responsive I'll re-implement my results parsing code and hopefully put the whole misinterpreted results issue behind us.

I'm trying to login to the current server right now and it's being exceptionally slow, so I'm not sure what's up. I'll probably just look and not touch in case George is already on there as well checking on it. Unless I see something obviously broken, there's the "too many cooks" thing and George would know from past experience what the problem probably is.

[chalsall]

Originally Posted by Madpoo

Unless I see something obviously broken, there's the "too many cooks" thing and George would know from past experience what the problem probably is.

Or not...

All of my spiders, both GPU72's and worker client's, have been experiencing many errors today. I suspect many others have been so observing as well.

Perhaps you could keep trying to log into Primenet (and iff finally succesful) to simple observe what's going on without changing anything?

BTW, I would be very happy to underwrite the transport of your new donated server to its new home via UPS or FedEx.

[Madpoo]

Originally Posted by chalsall

All of my spiders, both GPU72's and worker client's, have been experiencing many errors today. I suspect many others have been so observing as well.

Perhaps you could keep trying to log into Primenet (and iff finally succesful) to simple observe what's going on without changing anything?

I got in and there's still 3.5 GB of free space, so it's not crunched for that. It's not a lot, but enough. CPU use is low, but the system is still super laggy and I'm assuming it's doing a lot of paging again due to the memory pressure. I'd start up performance monitor and check but it's so slow it might take a while and only add to the overburdened system.

Originally Posted by chalsall

BTW, I would be very happy to underwrite the transport of your new donated server to its new home via UPS or FedEx.

Once I get back from my super fun work trip near Newark (</sarcasm>) I'll check back in with George and James to see how things are looking. James supplied some fun new PHP to eliminate the old methods that used a SQL extended proc to verify found factors and run msieve, which I think was the last bit of thing we were dealing with.

I was talking to George about the migration plans to the new system and one approach we were tossing back and forth was to make this test system "live" for long enough to send the new box to Scott and get it swapped out. If we go that route then we could see things get better even sooner.

But as I'm sure we'd all agree, we really just want to make sure that all of the functionality is still good... v4 and v5 clients are able to do everything they need to do and the web site works as expected, just faster, faster and faster.

[Prime95]

Originally Posted by Madpoo

Unless I see something obviously broken, there's the "too many cooks" thing and George would know from past experience what the problem probably is.

Don't worry about getting in this cooks way. I have no idea what is wrong. I follow the only recipe I know: kill IIS, restart MSSQL, restart IIS.

Reboot complete - sorry it took so long, I was out having fun this morning.

[Madpoo]

Originally Posted by Prime95

Don't worry about getting in this cooks way. I have no idea what is wrong. I follow the only recipe I know: kill IIS, restart MSSQL, restart IIS.

Reboot complete - sorry it took so long, I was out having fun this morning.

I'm doing some additional perfmon analysis... I've confirmed that really, truly, it's just memory pressure and excessive swapping. It's nothing that can really be solved given the 32-bit nature of the system.

One thing I think might help is to limit how much memory SQL is able to use so that, for instance, the website running on there as well won't be constantly fighting with SQL over how much physical RAM they can get.

I can tell by looking that SQL is already hitting memory limits with just 2 GB so limiting it to 1.5 GB and leaving that other 512MB for IIS and the OS isn't going to matter much to SQL but could matter a whole lot to the other components. In other words, it won't make SQL faster, but won't really make it much slower either so it's a benefit to everything else.

Page faults/sec is averaging 1600 or so... if you've ever had to diagnose memory pressure in Windows you'll know that's pretty high. :) That's with a current SQL tps of just 28-29 on average at the moment.

Anyway, to that end I set SQL to use just 1.5 GB (1536 MB) and restarted it, so we should see soon if it's having any impact on the responsiveness of everything else.

I probably should have set SQL's minimum memory to the same thing, but anyway, right now SQL is still filling it's memory buffer based on the current transactions it's dealing with. Things will reach a steady state, maybe 30-60 minutes, I'm guessing. Then we'll know if this is any better.

[M29]

Originally Posted by Prime95

sorry it took so long, I was out having fun this morning.

Did you play 9 or 18?

[Prime95]

Originally Posted by M29

Did you play 9 or 18?

Guilty as charged -- 18.