mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > PrimeNet

Reply
 
Thread Tools
Old 2019-01-13, 14:22   #1
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

24·239 Posts
Default Primenet API

I looked again at the API document recently, particularly Primenet API Assignment Progress, at http://v5.mersenne.org/v5design/v5webAPI_0.97.html#ap
and noticed several things. I think they mean the document could use an update to incorporate some extensions already implemented. It's indicated as Kurowski/Woltman 11/14/2007 RELEASE CANDIDATE 0.97(c) That's from before the first gpu-based GIMPS software, originated in 2009, and long before the addition of the Jacobi check, or the discovery of the Gerbicz check and rollout of PRP3 code to capitalize on it.

5.6.5.1.1 GIMPS Request Parameters
stage permitted work_unit_values does not include 'PRP'

5.7.5.1.1 how are the error counts encoded? what error types are counted?

7.2 does not include PRP related work types PRP, PRP-DC, PRP-CF, PRP-CF-D

7.3 does not include PRP related work types

8.2 does not include PRP related work types
8.2 has a defined result_type 0, so
"GIMPS result_type values include the range 1-256:" probably should say 0 not 1.

Assignment result_type values 257-1023 appear to be undefined; neither reserved for future purposes, nor excluded, nor part of the assignable result type range.

There's nothing there about the new longer 2048-bit residues either.

Is there some other document that contains errata/addenda?

https://www.mersenne.org/worktypes/ has a list of work types but not the corresponding numerical codes.

Last fiddled with by kriesel on 2019-01-13 at 14:27
kriesel is offline   Reply With Quote
Old 2019-01-13, 15:13   #2
GP2
 
GP2's Avatar
 
Sep 2003

2·1,289 Posts
Default

Quote:
Originally Posted by kriesel View Post
There's nothing there about the new longer 2048-bit residues either.

Is there some other document that contains errata/addenda?
The definitive answers for any questions about the Primenet protocol are to be found in the mprime/Prime95 source code. Any would-be re-implementer had best study that very closely.

In case of any contradictions between an outdated protocol definition document and the source code of its sole implementation, it is the source code that will prevail.
GP2 is offline   Reply With Quote
Old 2019-01-13, 15:34   #3
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

24×239 Posts
Default

Quote:
Originally Posted by GP2 View Post
The definitive answers for any questions about the Primenet protocol are to be found in the mprime/Prime95 source code. Any would-be re-implementer had best study that very closely.

In case of any contradictions between an outdated protocol definition document and the source code of its sole implementation, it is the source code that will prevail.
Thanks. I imagine in the case of multiple implementations, that the dominant one, created by one of the Primenet API creators would prevail. (Woltman, prime95 source code). Reference implementations are good to have and use.

It's unfortunate that the API documentation is 11 years out of date. I'm repeatedly referred to it, during the occasional discussions of interfacing gpu applications to primenet, and it seems singularly well suited to CPU applications, and rather unsuited to GPU applications, at least without some gpu-oriented extensions.
kriesel is offline   Reply With Quote
Old 2019-01-13, 19:49   #4
Madpoo
Serpentine Vermin Jar
 
Madpoo's Avatar
 
Jul 2014

CBD16 Posts
Default

Quote:
Originally Posted by kriesel View Post
5.7.5.1.1 how are the error counts encoded? what error types are counted?
I skimmed your post but this part caught my eye because I'd recently asked George for some clarification on the error counts.

The source code has a section in there that details how the bitmapped 32-bit error code is split up (8 bits here, 6 bits there, single-bit things for a few items, etc).

FYI, the high-bit (32) is actually reserved to indicate whether the work was started on an old client without error checking. The next highest bit is currently unused.

Also FYI, the encoding of errors changed in version 29.3 with the introduction of Jacobi/Gerbicz stuff, just to make things even more interesting.

My own interest in it came from trying to analyze whether certain errors are more likely to result in a bad residue, and whether we can use that data to do a better job of marking tests suspect. I'm still analyzing things, but I realize now we need to change that "unverified or suspect" check to take into account the version of software.

Anyway, search the source code for the "inc_error_count" function (I don't remember which .c file it shows up in).
Madpoo is offline   Reply With Quote
Old 2019-01-13, 20:05   #5
Madpoo
Serpentine Vermin Jar
 
Madpoo's Avatar
 
Jul 2014

3·1,087 Posts
Default

Quote:
Originally Posted by kriesel View Post
Thanks. I imagine in the case of multiple implementations, that the dominant one, created by one of the Primenet API creators would prevail. (Woltman, prime95 source code). Reference implementations are good to have and use.

It's unfortunate that the API documentation is 11 years out of date. I'm repeatedly referred to it, during the occasional discussions of interfacing gpu applications to primenet, and it seems singularly well suited to CPU applications, and rather unsuited to GPU applications, at least without some gpu-oriented extensions.
While I'm at it... I don't remember the details since my last deep-dive on the API was a while back, but I'm pretty sure the main functions are all unchanged (things like "uc = update computer" or "ap = assignment progress" --- introduction of new features or assignment types may alter some of the parameters here and there, but otherwise it's a good place to start. As noted, the source code is a good place to look for clarification, but the API itself is a great place to start.

And when in doubt, use Fiddler and make sure the client is setup to use your system proxy ( or point it specifically to the proxy that Fiddler sets up) and you can see first-hand how the client will talk to the primenet service.

One problem I do remember coming up was that some things require you to supply the cpu's GUID when updating the progress of an assignment, which is all well and good for an actual Prime95 client. However, manual assignments do in fact have a GUID assigned internally, but there's no way for an end user to see what their manual machine's GUID is. The solution to that, odd as it is, is to use the API itself to create a new machine using that "uc" function and keep track of the GUID that way, rather then doing it as a manual assignment like GPU software would currently do. In other words, the GPU clients can become fully functional clients just like Prime95 if they do all the same API stuff, but it becomes harder if we wanted to just implement a halfway solution.

Some stuff in the API isn't really fleshed out that well in the description. Like, what is the hardware GUID, or what makes up the Windows hardware hash? That's the thing that lets the client know if the setup was moved to a new machine so it can generate a new unique ID, etc. That's where you'd have to look at the source code to actually see what goes into it.

Some of the other bits there about CPU type, cache sizes, etc should follow some accepted guidelines when it comes to GPUs but that would be a better discussion for the devs of the GPU software and the community to decide what details to include. Internally, the CPU details and cache sizes are used to get an idea of how quickly an assignment could be expected to complete - I don't remember if that goes into category assignments or if we're only looking at past throughput for a machine. In other words, it may not be a big deal but standardizing the format would be much preferred.

The source code is out there and may actually be able to be integrated into other programs without entirely reinventing the wheel. Food for thought there. I'm just assuming it's in the current source as a library of functions that could be called with whatever parameters, with some (hopefully) minor tweaking for specific solutions.
Madpoo is offline   Reply With Quote
Old 2019-01-13, 22:24   #6
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

24·239 Posts
Default

An updated and extended version of my first post in this thread is at https://www.mersenneforum.org/showthread.php?t=23992

If I decipher any of those questions, my discoveries (including possible misconceptions) may appear here but is more likely it will appear there.

Last fiddled with by kriesel on 2019-01-13 at 22:24
kriesel is offline   Reply With Quote
Old 2019-02-23, 20:44   #7
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

382410 Posts
Default Primenet API and documentation

Please review https://www.mersenneforum.org/showth...845#post505845
Post 1: primenet practice in prime95 has outpaced the documentation
2: the existing API is cpu and prime95 oriented, which does not fit gpu applications well
3: early incomplete draft regarding extending the primenet API to support gpu applications, including multi-gpu per system configurations.

The Primenet API documentation as it is, seems daunting.

Last fiddled with by kriesel on 2019-02-23 at 20:44
kriesel is offline   Reply With Quote
Old 2019-02-26, 06:00   #8
Madpoo
Serpentine Vermin Jar
 
Madpoo's Avatar
 
Jul 2014

CBD16 Posts
Default

Quote:
Originally Posted by kriesel View Post
Please review https://www.mersenneforum.org/showth...845#post505845
Post 1: primenet practice in prime95 has outpaced the documentation
2: the existing API is cpu and prime95 oriented, which does not fit gpu applications well
3: early incomplete draft regarding extending the primenet API to support gpu applications, including multi-gpu per system configurations.

The Primenet API documentation as it is, seems daunting.
I've been working with Ernst on exploring some of the finer points of using the API.

One sticking point is the requirement for using security hashes for some features. A hash that requires trusted client code, which means it couldn't/shouldn't be included in open source.

If we lived in a world where everyone was chill and didn't try to game the system or be jerks, we wouldn't need that, but... here we are.

That said, there are some things I'm learning along the way about what's possible.

One idea I've tinkered with is to add the ability for a user to logon to the website and create a CPU account in there rather than doing that solely with the Prime95 client.

What that would do is be able to create the computer in a way that we can do as much as possible without needing those security hashes.

Right now, you can't create a computer using the API without that hash, but we have been able to do things like update the progress on an assignment using the "Manual Testing" computer that each user gets when they check in/out results manually.

The tricky bits are that the "manual testing" has a guid attached to it that isn't visible to end users, and there's also a "feature" in the API that requires a computer to have had it's status updated within the past XX days, otherwise it won't accept any data for that CPU.

Prime95 will send a "uc" (update computer) as the first command to the API whenever it does anything, to make sure that data is fresh. It could just be sending the same thing as before about the processor, memory, etc.

So for Ernst I sent him the GUID for that, manually updated the timestamp on when that "cpu" was last updated, and it'll be good for now (until it's once again 'stale' based on that timestamp). Since the "Manual Testing" cpu doesn't have a secret key associated with it, it's basically treated as an untrusted client so it doesn't need the hashes. I haven't actually tried sending a result that way or getting assignments... I'm just assuming those would work, but baby steps for now.

So in my grand scheme, I thought of some kind of thing to create a "Third Party Client" cpu through the website which would show you the GUID, and you could cut/paste that GUID into whatever config file the client uses to talk to the API. That special type wouldn't check the freshness of the last time it was updated because it doesn't make sense.

Essentially this is no different, security-wise, than accepting 3rd party client results through the manual result page... those don't even have the checksum on LL/PRP that can be used to weed out bogus submissions.

Now, if a 3rd party dev wanted to be able to use the API *and* benefit from a more secure checking, both to auth to the server that it's a bona fide client request, and maybe also include their own checksum process of the result, we could probably get something going as long as those bits don't make it into the published source.

Prime95 is like that... the rest is available but the security bits aren't. If you want to compile your own, you can, but you lose the built in client communication I suppose, and your results come in without the checksum so they're treated as essentially untrusted like other clients.

Thoughts or ideas?

My huge idea was (as an aside) to just "whip up" Primenet API version 6, which would use modern API endpoint processes, doing POSTs and passing data back and forth with something structured like JSON. It's not a huge change since it's not altering the way it works really beyond *how* the client/server talk, and not so much what they're saying to each other. Far less than the v4 -> v5 change was.

But that's pie-in-the-sky and would still take a good deal of effort and changes to the client as well. So... tucking that idea away for a series of rainy days.
Madpoo is offline   Reply With Quote
Old 2019-02-26, 16:09   #9
Mark Rose
 
Mark Rose's Avatar
 
"/X\(‘-‘)/X\"
Jan 2013
Ͳօɾօղէօ

1010111000102 Posts
Default

What does a trusted client actually mean for us?

I gather what we're doing is validating that the work is actually done.

Couldn't we just submit the penultimate state for LL and PRP and rerun the final iteration on the server instead?
Mark Rose is offline   Reply With Quote
Old 2019-02-26, 17:38   #10
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

24·239 Posts
Default

Quote:
Originally Posted by Mark Rose View Post
What does a trusted client actually mean for us?

I gather what we're doing is validating that the work is actually done.

Couldn't we just submit the penultimate state for LL and PRP and rerun the final iteration on the server instead?
Just for mlucas, or for all client applications? On my slow internet, that's 10 minutes or more of uplink time per 10M save file. When it's working well and there's no other traffic. Last week I had to drive to town to download some files because even 75MB files were failing at the 1 to 2MB point. It's asymmetric DSL 768kbps down, 128kbps up, so is likely to fail before 1MB in uploads, while interim residue save files are of order 10MB or 30M for prime95 8xM exponents.

Last fiddled with by kriesel on 2019-02-26 at 17:40
kriesel is offline   Reply With Quote
Old 2019-02-26, 17:46   #11
Mark Rose
 
Mark Rose's Avatar
 
"/X\(‘-‘)/X\"
Jan 2013
Ͳօɾօղէօ

2·7·199 Posts
Default

Quote:
Originally Posted by kriesel View Post
Just for mlucas, or for all client applications? On my slow internet, that's 10 minutes or more of uplink time per 10M save file. When it's working well and there's no other traffic. Last week I had to drive to town to download some files because even 75MB files were failing at the 1 to 2MB point. It's asymmetric DSL 768kbps down, 128kbps up, so is likely to fail before 1MB in uploads, while interim residue save files are of order 10MB or 30M for prime95 8xM exponents.
16 KB/s up... that's painful! Reminds me of the 90's.

I guess Prime95/mprime could continue to do as it does, but this mechanism could be used as an alternative to make a result as "trusted".
Mark Rose is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Primenet and GMP-ECM ET_ PrimeNet 9 2018-07-04 20:28
Hello PrimeNet!! SeeD419 PrimeNet 7 2011-07-11 18:09
56.0-57.x on PrimeNet v5 ckdo Lone Mersenne Hunters 0 2008-09-04 05:54
47.0-48.0 on PrimeNet ckdo Lone Mersenne Hunters 0 2008-02-14 20:05
New stats on PrimeNet! ET_ PrimeNet 18 2008-01-24 01:21

All times are UTC. The time now is 06:17.

Mon Jun 1 06:17:54 UTC 2020 up 68 days, 3:50, 1 user, load averages: 1.08, 1.24, 1.37

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.