GpuOwl PRP-Proof changes

[preda]

A mention of recent changes related to PRP-Proof in GpuOwl as of 2020-07-19:

Proof generation is now enabled by default, no need to do anything for that.

1. Power
The proof power can be controlled with -proof <power> argument, in the range 6-9. By default (i.e. when not using -proof) the power is 8. If low on disk space use 7 or 6, otherwise the default 8 should be good. The minimum disk use (at power 6) is about 0.8 GB for a 100M exponent.

On startup the program validates the list of temporary proof residues that should be there already, and tries to find a proof power that matches the existing residues. If the needed residues aren't there, proof generation is disabled for the current exponent (but will probably work correctly for the next exponent).

It's not possible to change the proof power on an ongoing exponent -- but the program should detect and ignore such change attempts (but they will be applied on the next exponent started from the start).

2. Storage
By default the temporary residues (many files) are stored in:
working-directory-of-gpuowl/<exponent>/proof/

Because these temporary residues may take up a lot of space, there were requests to allow storing them to a different location (such as a NFS mount) that may have more space. This is possible with the new -tmpDir argument, in which case the tmp residues are store into:
<tmpDir>/<exponent>/proof/

The temporary storage is not cleaned yet -- but this should be done soon.

3. Results
When a PRP test that has proof enabled is finished, the proof will be generated automatically. The proof generation is fast, takes on the order of minutes. The proof file location should be displayed on logs, but it goes to:
gpuowl-working-dir/proof/ if not using -pool <dir>, or to:
pool-dir/proof/ if using -pool <dir>

The proof file name is formed from exponent and power like this: 97309379-8.proof

If for some reason the proof for the same exponent&power is re-generated, the existing proof is overwritten.

The results JSON for a PRP test that generated a proof will have a new section "proof" that looks like this:
"proof":{"version":"1", "power":"8", "hashsize":"64", "MD5":"5818c5af84ab468717dae240ffffaaaa"}
which includes the MD5 of the proof file.

There will be a mechanism for uploading the proof files themselves soon, but for now they should just be stored. (i.e. don't delete the .proof files yet)

[preda]

There may be bugs. It's still in flux, there are still changes coming in (such as the cleaning of the temporary residues), so a general-availability build is not needed yet -- the intention was to let everybody know of the upcoming changes early.

[preda]

Now the proof temporary storage should be cleaned up properly after proof generation.

This means:
- the directory <base>/<exponent>/proof/ will be emptied and removed
- the directory <base>/<exponent> will be removed only if empty

where <base> is the temporary-storage directory if specified (-tmpDir), otherwise the GpuOwl's working directory (-dir).

An undocumented option exists, "-keep proof", that disables the above cleanup. This option is not considered generally useful (thus not documented), it's intended for debugging proof generation, and will probably be dropped in the future.

[kriesel]

https://www.mersenne.org/report_expo...7000009&full=1
PRP -proof 8. Proof file 150.5 MB cached. Temporary files total 4.08 GB.

[preda]

It is now possible to upload proofs, here's how:

1. Manual upload of a single proof
Use the new script tools/upload.py , invoke it like this:
upload.py <user-ID> <proof-file>
where <user-ID> is your mersenne.org ID.

2. Integrated into tools/primenet.py
The old primenet.py script will now watch the "proof/" subfolder in the watched dirs. If .proof files are found there, the script attempts to upload them. If uploaded successfully, the proofs are moved to a subfolder "uploaded/". (this "uploaded" subfolder is created if not existing)

The primenet.py script works best in conjunction with the -pool <dir> option to gpuowl, which has the effect that the script can be run with a single folder to watch (the pool directory) where all the results and proofs from all the gpuowl instances accumulate.

Note: the PRP JSON textual result now contains a "proof" section mentioning the generated proof. The old JSONs didn't have this part. It is possible that proof files exist for JSONs without a "proof" section -- they should still be accepted by primenet, but it is possible that the upload fails for this case ATM (this situation is investigated). It may be useful to report upload failures in any case, just to double check the "why" this being rather new.

Kudos to George, Aaron & James who did the heavy lifting server-side with such speed.

[paulunderwood]

Two problems:

1. I can't compile the latest gpuOwl:

 make gpuowl
echo \"`git describe --long --dirty --always`\" > version.new
diff -q -N version.new version.inc >/dev/null || mv version.new version.inc
echo Version: `cat version.inc`
Version: "v6.11-361-g2274207-dirty"
g++-8 -MT Pm1Plan.o -MMD -MP -MF .d/Pm1Plan.Td -Wall -O2 -std=c++17   -c -o Pm1Plan.o Pm1Plan.cpp
g++-8 -MT GmpUtil.o -MMD -MP -MF .d/GmpUtil.Td -Wall -O2 -std=c++17   -c -o GmpUtil.o GmpUtil.cpp
g++-8 -MT Worktodo.o -MMD -MP -MF .d/Worktodo.Td -Wall -O2 -std=c++17   -c -o Worktodo.o Worktodo.cpp
g++-8 -MT common.o -MMD -MP -MF .d/common.Td -Wall -O2 -std=c++17   -c -o common.o common.cpp
g++-8 -MT main.o -MMD -MP -MF .d/main.Td -Wall -O2 -std=c++17   -c -o main.o main.cpp
g++-8 -MT Gpu.o -MMD -MP -MF .d/Gpu.Td -Wall -O2 -std=c++17   -c -o Gpu.o Gpu.cpp
In file included from Gpu.cpp:4:
ProofSet.h:161:8: error: ‘unordered_map’ in namespace ‘std’ does not name a template type
   std::unordered_map<u32, Words> pending;
        ^~~~~~~~~~~~~
ProofSet.h:161:3: note: ‘std::unordered_map’ is defined in header ‘<unordered_map>’; did you forget to ‘#include <unordered_map>’?
ProofSet.h:16:1:
+#include <unordered_map>
 
ProofSet.h:161:3:
   std::unordered_map<u32, Words> pending;
   ^~~
ProofSet.h: In member function ‘void ProofCache::flush()’:
ProofSet.h:206:20: error: ‘pending’ was not declared in this scope
     for (auto it = pending.cbegin(), end = pending.cend(); it != end && write(it->first, it->second); it = pending.erase(it));
                    ^~~~~~~
ProofSet.h:207:10: error: ‘pending’ was not declared in this scope
     if (!pending.empty()) {
          ^~~~~~~
ProofSet.h: In member function ‘void ProofCache::save(u32, const Words&)’:
ProofSet.h:218:9: error: ‘pending’ was not declared in this scope
     if (pending.empty() && write(k, words)) { return; }
         ^~~~~~~
ProofSet.h:219:5: error: ‘pending’ was not declared in this scope
     pending[k] = words;
     ^~~~~~~
ProofSet.h: In member function ‘Words ProofCache::load(u32, u32) const’:
ProofSet.h:224:15: error: ‘pending’ was not declared in this scope
     auto it = pending.find(k);
               ^~~~~~~
ProofSet.h: In member function ‘void ProofCache::clear()’:
ProofSet.h:228:18: error: ‘pending’ was not declared in this scope
   void clear() { pending.clear(); }
                  ^~~~~~~
make: *** [Makefile:30: Gpu.o] Error 1

2. I Can't upload:

 ./upload.py "PaulUnderwood" ../../../../gpuowl/104982617/104982617-9.proof 
Uploading M104982617 from "../../../../gpuowl/104982617/104982617-9.proof"
http://mersenne.org/proof_upload/?UserID=PaulUnderwood&Exponent=104982617&FileSize=131228338&FileMD5=98b1ab8540e226083ada38ef1abb7ad5
{'error_status': 401, 'error_message': 'Unauthorized', 'error_description': ''}

[kruoli]

On my end, the uploading works fine.

[preda]

Originally Posted by paulunderwood

Two problems:

1. I can't compile the latest gpuOwl:
2. I Can't upload:

Please re-try the compilation, I attempted a fix.

The upload problem you see may be related to what I mentioned, the JSON result without "proof" -- under investigation.

[paulunderwood]

Originally Posted by preda

Please re-try the compilation, I attepted a fix.

The upload problem you see may be related to what I mentioned, the JSON result without "proof" -- under investigation.

Compiled well.

Edit: Uploading requires the new format

[paulunderwood]

Originally Posted by paulunderwood

Edit: Uploading requires the new format

Uploading seems to work now for the old proof result format.

[paulunderwood]

Uploading ability seems to be intermittent. 401 error. I will try later.