LLRnet and PRPnet servers for automated LLR

[mdettweiler]

Hi all,

A few weeks ago, Oddball sent me a PM asking about how to set up an LLRnet server. To sum up a somewhat lengthy discussion, I offered to host a server on the existing NPLB/CRUS server equipment, and Oddball decided to go with side-by-side LLRnet and PRPnet servers to give users the option of using either. The sieve depth on the variable-n effort has advanced somewhat since then, and thus we have now set up the servers.

LLRnet and PRPnet are quite similar in concept. They have both a server and client part: the server keeps a queue of k/n pairs to be tested for potential primes, and clients connect to it over the internet to request pairs to work on. The client then feeds the number to LLR, which tests the number; then the client sends the result back to the server where it is stored.

LLRnet was the "original" automated LLR testing software. Years ago it was developed by the Riesel Sieve project and has gone through various changes over the years, during which it's been made use of by many projects in the prime search world. However, about 5 years ago (IIRC), Riesel Sieve ceased actively developing LLRnet due to their shifting more to a BOINC setup. At the time the LLRnet client had its LLR testing code built in, so it was stuck on LLR version 3.5.0 even as newer, faster, LLR versions came out.

In December 2008, Mark Rodenkirch (a.k.a. rogue on this forum) developed PRPnet, based on his existing ECMnet factoring software, as a replacement for LLRnet that would be modular, allowing for new LLR versions to be easily switched in as they came out. It also supported the use of other applications such as Phrot and PFGW, which at the time were faster for some types of numbers, and in the case of Phrot, supported non-x86 architectures. Since its initial release, PRPnet has developed into a solid, robust platform for automated primality testing and has addressed many of the shortcomings of LLRnet.

In March 2010, in response to the recent release of LLR v3.8, which improved speed an additional 4% or so over 3.7.1c (which was in turn 6% faster than the 3.5.0 used in LLRnet), Karsten Bonath (kar_bon on the forum), with the assistance of myself and Gary Barnes (gd_barnes), developed a new version of the LLRnet client which, like PRPnet, is modular and not tied to a particular LLR version. The server side of LLRnet was unmodified, but the new client remains backwards-compatible with the old one so that it can work on any LLRnet server. While this still does not address some of the other shortcomings of LLRnet (no native support for twin prime searches, no built-in stats, no email notifications of prime found, among others), it did provide another option at a time when PRPnet was not yet capable of supporting as high loads as LLRnet.

Since then, development of a PRPnet version capable of the same load capacity of LLRnet has completed, so they are now both solid options for high- and low-volume projects alike. While the servers are rather different in design, the PRPnet client and the "new" LLRnet client both work on a similar paradigm. Nonetheless, some users have a preference of one over the other, so we're therefore offering both at TPS.

Both servers are loaded with work from the n=480K-500K variable-n subproject. They are located at:

noprimeleftbehind.net port 12000 for PRPnet (currently working on n=480K-485K, k=100K-200K)
noprimeleftbehind.net port 12050 for LLRnet (currently working on n=485K-490K, k<100K)

A web page summarizing the current status of both servers can be found at http://www.noprimeleftbehind.net/tps/. It contains links to the PRPnet server's built-in web pages for detailed information on its status, and to repositories where the results files from both servers will be copied off daily (and in the case of port 12000, converted to LLRnet format for consistency). The deadline for returning results on both servers is 3 days.

For instructions on how to set up and run each client, see:
PRPnet instructions
LLRnet instructions

If you have any questions about the server or need to report a problem with them, post in this thread--I'll be monitoring it. If you need to reach me directly, email me at max@noprimeleftbehind.net or send me a PM here at the forum. (Email is preferred so that I don't have to type my reply in a itty bitty text box, but PMs are OK too. )

Max

[mdettweiler]

The PRPnet client is not difficult to set up and use. Those of you familiar with BOINC should get the hang of it quite easily; similar to how BOINC can juggle multiple projects and divvy up CPU time between them, PRPnet does the same with individual servers.



To set up the client:

• Download the latest client for your operating system from http://pgllr.mine.nu/PRPNet/. (Binaries provided courtesy of PrimeGrid.) Windows clients are 32-bit (since LLR's calculations don't benefit from 64-bit, there's no point in making a 64-bit binary), and Linux is 64-bit (probably because the PrimeGrid people don't have a 32-bit Linux system on hand to make the binaries--let me know if you need one). Binaries are also available for Mac, both Intel and PPC, and even for Playstation 3's if you've got one of those.
• Extract the zip file to a new folder.
• Open the file prpclient.ini.
• Plug in your email address and username near the top of the file. It is recommended that you replace "clientID" with a unique identifier for this particular computer; that way you can trace a particular result back to the computer it came from in the event that you submit a bad result.
• If you wish to have your participation affiliated with a team (this does not affect individual stats, only the separate team stats), enter the team's name in the appropriate field, right under the client ID. Otherwise, leave it blank and your stats will be counted under team "None" in the team stats. Note: if your team name contains spaces, leave them out when entering it here. For instance, my team "Raiders of the Lost Primes" would be entered as "RaidersOfTheLostPrimes".
• Below the area where you just put your user information, you'll see a section where you can configure what servers the client gets work from. Here, you can configure the proportions and queue sizes for each individual server. The instructions provided in the prpclient.ini file are rather self-explanatory, so I won't go into detail about that here. Those familiar with BOINC will feel right at home here. Add the following entry to the list of servers:
  TPS12000:100:1:noprimeleftbehind.net:12000
  You may want to keep one or more of the PrimeGrid servers around as backup in case the TPS server goes offline or runs out of work. They should be already configured with "0" in the second field (indicating that they should only be used as backup), but you'll want to check to make sure this is the case; unlike BOINC, PRPnet's resource share numbers must total 100 for all servers. For a list of what kind of work is in each of the PrimeGrid servers, see here. Also, you may want to consider adding servers from the NPLB and CRUS projects as backup too (yeah, I know, shameless plug): see here and here respectively for their currently active servers.
• Farther down the file, you'll see two options, "startoption" and "stopoption". By default, each time the client is stopped or started, it asks you what to do with any work left in its queue. This can become rather redundant during normal operation, and can be especially annoying on Windows systems where PRPnet will hold up a system shutdown until you answer its prompt. I recommend configuring the client to startoption=2 (complete assigned work units), and either stopoption=2 (return completed workunits, keep incomplete workunits in queue, and shut down) or stopoption=3 (shut down and hold everything in queue, to be picked up where it left off next time you run the program), depending on your preference. Note that stopoption=2 can delay a shutdown a bit if there are a lot of workunits to return, or if you're on dial-up; if this is important in your scenario, I'd recommend using stopoption=3.
• Once you're satisfied with the configuration, save the file.
• Now, run prpclient.exe (or ./prpclient from a terminal on Linux). The client will run, fetch its first batch of work from whatever server has the highest work proportion, and start working. When that batch is done, it will return the results and grab a new batch to work on. Et cetera, et cetera, ad infinitum.
• To stop the client, press Ctrl-C. The client will stop according to your specified stopoption.

That's all there is to it! If some of the prpclient.ini options look confusing, ignore them--they're quite safely set at their defaults. If you've got any questions, feel free to post them in this thread and either I or someone else knowledgeable of PRPnet will answer your question promptly.

Max

[mdettweiler]

The main difference to note between PRPnet and the "new" LLRnet client is that the LLRnet client only can be configured with one server, whereas PRPnet can split its time between any number of servers and have backup servers in case its primary servers go down. Thus LLRnet can be slightly simpler to configure, at the expense of some flexibility. (IMHO, they're both just about as easy to configure, but opinions on this vary both ways.)


To set up the client:

• Download the latest client for your operating system here. There are a few different versions available there, but for Windows you want v0.73 and for Linux you want v0.71. (Ignore the disparity in version numbers, they're really completely separate programs.)

At this point the instructions differ depending on your operating system. For Windows:

• First of all, note that the Windows client package includes a preconfigured LLRnet server setup alongside the client. For most users this is not needed and you should ignore it entirely to avoid confusion. Extract just the contents of the "LLRnet_client" subdirectory to a new folder on your computer. You can also extract LICENSE.txt and README.txt from the root directory to the same place if you'd like.
• Right-click the file do.bat and click Edit. Do NOT double-click on it; this will run the batch file and start the program (which won't work yet since you haven't configured it).
• At the top of the file you'll see the following lines:
  set op_Iterations=10000
  set op_Timeout=60
  set op_Beep=TRUE
  set op_indPrimeLog=FALSE
  set op_connect=FALSE
  These are your configuration options. They're (sort of) self-explanatory: op_Iterations controls LLR's OutputIterations option (how often it should update the % complete display on the screen); op_Timeout adjusts how long the client should wait before retrying if it can't connect to the server; op_Beep sets whether the computer will make a "BEEP!" sound out of the system speaker when a prime is found (FYI, this often happens even if your volume is muted); op_indPrimeLog sets whether the program should log primes to a file in the client's own directory (TRUE) or in the directory above it (FALSE--useful for multicore systems); op_connect sets whether the client should keep waiting for an unresponsive server to come back online (TRUE), checking every op_Timeout seconds, or shut down after the first failure (FALSE). I recommend leaving op_Iterations and op_Timeout be, setting op_indPrimeLog to true if your system has only one core (but leave it at false if otherwise), and setting op_connect to true. As for op_Beep, that's truly a matter of personal preference.
• Save and close the file. Now open llr-clientconfig.txt in a text editor (you can double-click this one).
• Change the server and port lines as follows:
  server = "noprimeleftbehind.net"
  port = 12050
  Set username to your name here on the forum and WUCacheSize to the amount of pairs you'd like the client to do in each batch. If in doubt here, 1 is a safe option.
• Save and close the file.
• Double-click on do.bat to run it. It will fetch the number of pairs specified in WUCacheSize, process them, return them, grab another batch and do it all over again until you hit Ctrl-C to stop the client.
• If you shut down the client and don't plan to restart it for more than 3 days, run _cancel.bat to tell the server that you won't be processing them (otherwise, it will assume that after 3 days has elapsed). Then run _new.bat to clean out any junk left in the folder. The client will now be emptied out and ready to start afresh on new work when you run do.bat in the future.

For Linux:

• Note: you must have Perl installed on your system to run this. (Specifically, Perl 5.8 or greater, though it may work with earlier versions and I just don't know it.) Most Linux systems have Perl pre-installed.
• Extract the zip file to a new directory on your computer.
• Open do.pl in a text editor. Starting at line 15 you'll see the following:
  # Configuration variables, set as desired
  $llrPath = './llr';
  $llrnetPath = './llrnet';
  $beepOnPrime = "false";
  $individualPrimeLog = "false";
  $iniOptions = "OutputIterations=10000\n";
  $errorTimeout = 60;
  llrPath and llrnetPath can be safely left alone. beepOnPrime specifies whether the client should beep the PC speaker when it finds a prime. (Note that this often will occur even if your speakers are muted.) individualPrimeLog tells it whether to log primes to a file in the client's own directory, or in the next directory up (useful for multicore systems). iniOptions lets you specify custom options for LLR, such as OutputIterations (how often LLR should update the % complete figure on the screen); if you want to add more, be sure to affix each with "\n". errorTimeout is how long the client should wait before attempting to reconnect to a downed server.
• After editing the configuration variables however you'd like, save and close the file. Open llr-clientconfig.txt in a text editor.
• Set server and port as follows:
  server = "noprimeleftbehind.net"
  port = 12050
  Set username to your name here on the forum and WUCacheSize to the amount of pairs you'd like the client to do in each batch. If in doubt here, 1 is a safe option. The other options can be safely ignored.
• Save and close the file.
• Open a terminal and run the command "perl do.pl". Alternately, you can run "chmod +x do.pl" and thereafter "./do.pl" will work the same. The client will fetch WUCacheSize k/n pairs, process them, return them, and do another batch until you press Ctrl-C to stop the client.
• To cancel any unfinished pairs in queue, run "perl do.pl -c" (or, "./do.pl -c" if you've done the chmod thing).

Note that the Linux client package currently contains LLR v3.8.0. The latest version is 3.8.1, which has a few minor bugfixes. The more major ones don't really affect the kind of numbers TPS does, so you should be OK with 3.8.0, but if you want to upgrade to the latest version you can grab the 3.8.1 for Linux binary at http://jpenne.free.fr/index2.html and replace the one in your LLRnet directory with it.

If you have any questions about the LLRnet client, post them here and I or someone else knowledgeable of this stuff will answer them.

Max

[Oddball]

The PRPnet server seems to be down, and I can't get LLRnet to work either

I'm going to do some sieving for the time being.

[mdettweiler]

Originally Posted by Oddball

The PRPnet server seems to be down, and I can't get LLRnet to work either

I'm going to do some sieving for the time being.

Yes, as I just posted in the NPLB forum a moment ago, Gary's having a power and internet outage over on his end. (The server and his router equipment are on UPS backup, so since the internet doesn't work the cable line must be out too.)

Gary first notified me of the outage an hour ago via text message (though I didn't receive it until just now), and (obviously) it hasn't been fixed yet. Based on how long it usually takes my power company to repair a downed line, I'm optimistic that it will be fixed by morning CDT but that's not necessarily guaranteed. And of course there's the cable company who's got to fix their end as well...so it might be mid-morning or noon CDT by the time it's back up.

[Oddball]

Originally Posted by mdettweiler

Yes, as I just posted in the NPLB forum a moment ago, Gary's having a power and internet outage over on his end.

It seems that you posted that message in the NPLB forum a minute or two after I posted my message, so I didn't get to see your post.

it might be mid-morning or noon CDT by the time it's back up.

The server is back up and we're running low on work. Could you load n=480301-480500 into the PRPnet server?

[mdettweiler]

Originally Posted by Oddball

It seems that you posted that message in the NPLB forum a minute or two after I posted my message, so I didn't get to see your post.

Yeah, I was going to post here next but you beat me to it.

The server is back up and we're running low on work. Could you load n=480301-480500 into the PRPnet server?

Sure thing--it's now loaded and ready to go again.

BTW @Lennart: if you're planning to do another big dump on the TPS PRPnet server, be sure to post about it here so we know to load some more work in advance!

[Lennart]

Originally Posted by mdettweiler

Yeah, I was going to post here next but you beat me to it. Sure thing--it's now loaded and ready to go again.

BTW @Lennart: if you're planning to do another big dump on the TPS PRPnet server, be sure to post about it here so we know to load some more work in advance!

I am sorry ! I did not check how much was left

Lennart

[mdettweiler]

Originally Posted by Lennart

I am sorry ! I did not check how much was left

Lennart

No problem--that's the nice thing about PRPnet, you can dry out one server and nobody goes idle.

Do you by chance know how long you'll be staying on the TPS server with this number of cores? I can then calculate an approximate figure for how much work will need to be loaded in to cover that time frame.

[Lennart]

Originally Posted by mdettweiler

No problem--that's the nice thing about PRPnet, you can dry out one server and nobody goes idle.

Do you by chance know how long you'll be staying on the TPS server with this number of cores? I can then calculate an approximate figure for how much work will need to be loaded in to cover that time frame.

I am not sure but I have not that many on now. I have some work to do on SR5 again I will not be on with that much core after today. I shall post here if I put many on the server.

Lennart

[Lennart]

I have emailed Max that I will put some more on the server. I see now that it will soon dry out so I may have to move them for a while.

Lennart