feature request

tha · 2003-12-11, 08:32

L.S.,

Nice almost empty forum space here. Time to request a server feature.

I would like to see the status page on the Mersenne website, the table and the associated files listed below the table to be server generated, say once a day. It will save some valueble time for George and make the availability more consistent.

YotN,

Henk.

wblipp · 2003-12-11, 14:38

Is there any thought about making it possible for projects to run their own "slave server" that can be configured to work on different tasks but consolidate results back to a common master server? Tim Charron's ECM Server/Client works this way.

There are projects that use the Prime95 program in a distributed but manually coordinated way. Some, like LMH, are directly part of the search for Mersenne Primes. Others, like GIMPS ECM and P-1 and ElevenSmooth are factoring Mersenne numbers known to be composite. Slave servers might also solve the bottleneck problems from large manual submissions - these could set up a slave server that accepts the results and then reports to the master using a protocol that the master can throttle.

PrimeCruncher · 2003-12-11, 22:30

^^ I'd like a feature like that for PrimeNet. That is, a "mini-server" that can be set up inside a network that connects to the PrimeNet server and gets/updates/returns exponents. Then PCs on the inside of said network could connect to the mini-server to get and return their exponents. This would be very useful, especially for large farms.

Complex33 · 2003-12-11, 22:44

PrimeCruncher, maybe this would be of some use?

http://www.mersenne.org/ips/proxy.html

PrimeCruncher · 2003-12-11, 22:48

Quote:

Originally posted by Complex33
PrimeCruncher, maybe this would be of some use?

http://www.mersenne.org/ips/proxy.html

Complex33, I've looked at that before. But it doesn't buffer the packet being sent back or get new packets to distribute. I was just thinking something like that would be good if there was, for some reason, a prolonged downtime of the server. It would be good for small farms on dial-up connections and large farms for coordination. It essentially would be a proxy server but it would buffer work and messages.

Old man PrimeNet · 2003-12-11, 23:42

For clarity let's assume we are discussing GIMPS clients that have PrimeNet-integrated networking support.

The best strategy is in fact the one already in use, specifically:

(a) each program batch-queues/buffers enough of its own work to keep busy for at least as long as it maximally expects it could be out of contact with the server under routine operations, and

(b) each program manages its own state synchronization.

For grid designs where large volumes of data - those where data movement cost in $ or time is significant (on the order of 100MB or more) - can benefit from distributed caching and/or peer-to-peer, logically-local (intra-LAN, etc.) data exchanges.

For GIMPS, adding an intermediate 'mini-server' having stateful data adds nothing useful. The data exchanges are too small and zippy and the work units too long (hours or more) to benefit from a more complicated design.

A non-caching proxy, however, performs a useful function by concentrating traffic from identically-configured clients through a single point that can be monitored and logged as an authorized portal. That the proxy does not change the client batch-queuing/buffering & state synchronization model is an excellent property of the overall design.

For what real reason would a small farm or cluster need to use anything else? A proxy needs a single, many-to-one network connection to PrimeNet, the same as a 'mini-server' would. Conversely, an episode of relatively long-term server unavailability would affect similarly configured work-queued clients and a work-caching mini-server identically - sooner or later they both run out of work.

As such, the only possible answer driving a need for a stateful 'mini-server' is if there is no network connection on the farm or cluster. Unless the person running it felt compelled to stay 'in the loop', refusing to trust PrimeNet to manage those resources.

PrimeCruncher · 2003-12-11, 23:56

Quote:

Originally posted by Old man PrimeNet
For GIMPS, adding an intermediate 'mini-server' having stateful data adds nothing useful.

I would disagree.

Quote:

Conversely, an episode of relatively long-term server unavailability would affect similarly configured work-queued clients and a work-caching mini-server identically - sooner or later they both run out of work.

As such, the only possible answer driving a need for a stateful 'mini-server' is if there is no network connection on the farm or cluster. Unless the person running it felt compelled to stay 'in the loop', refusing to trust PrimeNet to manage those resources.

Yes, but with a mini-server the work list can be lengthened. I'm not saying replace the Prime95 method of buffering, but add to it. Thus worktodo.ini + mini-server = PrimeNet server can potentially be down for months with no problem. And some farms may not have network connections. The Prime95 extremists out there (okay, I guess I may be one of them

) would disable auto communication to save cycles on nodes where there is only a dial-up connection that is on say, once a week, in the traditional model being used now. Whereas with the mini-server, the auto communication would not need to be disabled since it would contact the mini-server once to submit the work. Then at the specified connection time the mini-server would sync with the PrimeNet server, thus making the node to PrimeNet communication more efficient. I realize this would be a lot of work for something few people would use but if the v5 server is going to have redundancy/fail-over capacity/etc aren't we basically already doing this on a much larger scale?

I hope you're getting what I'm trying to say here...

Old man PrimeNet · 2003-12-12, 00:17

How would increasing the client work queue fail to achieve the identical result? You already agree the result is invariant by falling back on an appeal to limited server availability. Moreover, a failover design supports disaster risk management, not routine operations.

I assert that the correct course of action is the one with the greatest leverage, and that in this case high server availability - something we need to provide anyway - makes any other requirement vanish.

PrimeCruncher · 2003-12-12, 00:45

Quote:

Originally posted by Old man PrimeNet
How would increasing the client work queue fail to achieve the identical result? You already agree the result is invariant by falling back on an appeal to limited server availability. Moreover, a failover design supports disaster risk management, not routine operations.

I assert that the correct course of action is the one with the greatest leverage, and that in this case high server availability - something we need to provide anyway - makes any other requirement vanish.

Increasing the client work queue would prevent it from running out of work as quick. I agree that it (obviously) is an invariable end result but the point is that it lengthens the queue time.

I thought the idea of failover is that if the primary server fails, the secondary server takes over. While v5 will undoubtably be more stable than v4 I see that as disaster risk management AND routine operations.

However, since you are in charge of developing v5 while I will probably have no impact on it whatsoever, the decision is ultimately yours. Since I can see that you are set against this, I will end this discussion now and we can just agree to disagree on this issue.

wblipp · 2003-12-12, 01:54

Quote:

Originally posted by Old man PrimeNet
For GIMPS, adding an intermediate 'mini-server' having stateful data adds nothing useful.

This is probably right for the core functionality of assigning exponents for LL tests and directly related factoring on that particular exponent. There are other uses of the Prime95 program that could benefit from slave servers. Providing support for these other uses is a dangerous quagmire that could imperil the timely delivery of core functionality, so I understand completely if the decision is to require these users to continue with whatever methods they presently use. My experience running a Prime95 ECM factoring project is that it's difficult to get timely reports about completed curves that didn't find a factor. I imagine other uses of Prime95, including LMH, would find life simpler with a slave server possibility, too.

But making Prime95 a powerful simple tool for "whatever people want to work on" isn't really the GIMPS charter, so I understand completely if the decision is to ignore these ancilliary applications of Prime95 - but I wanted it to be a conscious design decision, not an accidently overlooked opportunity.

Paulie · 2003-12-12, 15:37

There is also the trust level of the proxy server for work returned. Having control of the box and the code, is safer for the project. The server trusts the client communicating with it, why should it trust some queue proxy?

Remember, the majority of the work is still done by people who "set it and forget it", and the core is designed with that in mind. The flexability in the queing for the client now is great, lets teams and single crunchers do what they will with their own work queues. I'd rather spend some time on anti-poaching features. :)

One plus for a queing server, atleast for TPR, would be automating the work type specifically for the team, to optimize production for the team. The ability to check in/out, for example, P-1 ready exponents, or exponents at certain FFT ranges to specific processor types, would be very useful. But is this for the core server, or could someone make a seperate queue and client, to handle editing the local worktodo.ini files? If you're at the level of manually editing your worktodo, having another program that handles adding work for you is no hardship. But let a trusted client handle request/result transactions with the core server.

Maybe once the core is done, there could be a secondary project server(s), on a different server or IP port, settable via prime.ini, for things like ECM. So the logs can be consolidated with a single stats server. But the primary purpose isn't supporting every math project available (Prime95 is very nice in allowing for work beyond GIMPS), this is about GIMPS. The donated time/money/space/electricity/backups are for GIMPS.

Failover at the core could be handled via clustering and a web accelerator FEP.

2003-12-11, 08:32	#1
tha Dec 2002 2×5²×17 Posts	feature request L.S., Nice almost empty forum space here. Time to request a server feature. I would like to see the status page on the Mersenne website, the table and the associated files listed below the table to be server generated, say once a day. It will save some valueble time for George and make the availability more consistent. YotN, Henk.

2003-12-11, 14:38	#2
wblipp "William" May 2003 New Haven 2,371 Posts	Slave Server Feature Is there any thought about making it possible for projects to run their own "slave server" that can be configured to work on different tasks but consolidate results back to a common master server? Tim Charron's ECM Server/Client works this way. There are projects that use the Prime95 program in a distributed but manually coordinated way. Some, like LMH, are directly part of the search for Mersenne Primes. Others, like GIMPS ECM and P-1 and ElevenSmooth are factoring Mersenne numbers known to be composite. Slave servers might also solve the bottleneck problems from large manual submissions - these could set up a slave server that accepts the results and then reports to the master using a protocol that the master can throttle.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Feature request	tcharron	Software	3	2018-10-03 20:08
Feature request	TheMawn	PrimeNet	3	2013-06-17 02:32
Feature request	JuanTutors	Software	22	2013-03-11 19:23
Feature Request	moo	Software	24	2005-11-26 22:08
Feature request	JuanTutors	Software	2	2005-07-04 22:02

2003-12-11, 22:30	#3
PrimeCruncher Sep 2003 Borg HQ, Delta Quadrant 2·3³·13 Posts	^^ I'd like a feature like that for PrimeNet. That is, a "mini-server" that can be set up inside a network that connects to the PrimeNet server and gets/updates/returns exponents. Then PCs on the inside of said network could connect to the mini-server to get and return their exponents. This would be very useful, especially for large farms.

2003-12-11, 22:44	#4
Complex33 Aug 2002 Texas 5·31 Posts	PrimeCruncher, maybe this would be of some use? http://www.mersenne.org/ips/proxy.html

2003-12-11, 23:42	#6
Old man PrimeNet Jan 2003 Altitude>12,500 MSL 1100101₂ Posts	For clarity let's assume we are discussing GIMPS clients that have PrimeNet-integrated networking support. The best strategy is in fact the one already in use, specifically: (a) each program batch-queues/buffers enough of its own work to keep busy for at least as long as it maximally expects it could be out of contact with the server under routine operations, and (b) each program manages its own state synchronization. For grid designs where large volumes of data - those where data movement cost in $ or time is significant (on the order of 100MB or more) - can benefit from distributed caching and/or peer-to-peer, logically-local (intra-LAN, etc.) data exchanges. For GIMPS, adding an intermediate 'mini-server' having stateful data adds nothing useful. The data exchanges are too small and zippy and the work units too long (hours or more) to benefit from a more complicated design. A non-caching proxy, however, performs a useful function by concentrating traffic from identically-configured clients through a single point that can be monitored and logged as an authorized portal. That the proxy does not change the client batch-queuing/buffering & state synchronization model is an excellent property of the overall design. For what real reason would a small farm or cluster need to use anything else? A proxy needs a single, many-to-one network connection to PrimeNet, the same as a 'mini-server' would. Conversely, an episode of relatively long-term server unavailability would affect similarly configured work-queued clients and a work-caching mini-server identically - sooner or later they both run out of work. As such, the only possible answer driving a need for a stateful 'mini-server' is if there is no network connection on the farm or cluster. Unless the person running it felt compelled to stay 'in the loop', refusing to trust PrimeNet to manage those resources.

2003-12-12, 00:17	#8
Old man PrimeNet Jan 2003 Altitude>12,500 MSL 1100101₂ Posts	How would increasing the client work queue fail to achieve the identical result? You already agree the result is invariant by falling back on an appeal to limited server availability. Moreover, a failover design supports disaster risk management, not routine operations. I assert that the correct course of action is the one with the greatest leverage, and that in this case high server availability - something we need to provide anyway - makes any other requirement vanish.

2003-12-12, 15:37	#11
Paulie Aug 2002 337₈ Posts	There is also the trust level of the proxy server for work returned. Having control of the box and the code, is safer for the project. The server trusts the client communicating with it, why should it trust some queue proxy? Remember, the majority of the work is still done by people who "set it and forget it", and the core is designed with that in mind. The flexability in the queing for the client now is great, lets teams and single crunchers do what they will with their own work queues. I'd rather spend some time on anti-poaching features. :) One plus for a queing server, atleast for TPR, would be automating the work type specifically for the team, to optimize production for the team. The ability to check in/out, for example, P-1 ready exponents, or exponents at certain FFT ranges to specific processor types, would be very useful. But is this for the core server, or could someone make a seperate queue and client, to handle editing the local worktodo.ini files? If you're at the level of manually editing your worktodo, having another program that handles adding work for you is no hardship. But let a trusted client handle request/result transactions with the core server. Maybe once the core is done, there could be a secondary project server(s), on a different server or IP port, settable via prime.ini, for things like ECM. So the logs can be consolidated with a single stats server. But the primary purpose isn't supporting every math project available (Prime95 is very nice in allowing for work beyond GIMPS), this is about GIMPS. The donated time/money/space/electricity/backups are for GIMPS. Failover at the core could be handled via clustering and a web accelerator FEP.