mersenneforum.org  

Go Back   mersenneforum.org > Extra Stuff > Blogorrhea > kriesel

Closed Thread
 
Thread Tools
Old 2020-06-01, 16:45   #23
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

2×3×11×67 Posts
Default Setting up in Linux

There's a separate post in the Google Colab thread for setting that up.

Ewmayer has recently created a comprehensive guide, at https://mersenneforum.org/showthread.php?t=25601

Earlier:
Prime95 suggested for setting up on a local Linux system, https://mersenneforum.org/showpost.p...5&postcount=76
and "also do a sudo apt install libncurses5" https://www.mersenneforum.org/showpo...postcount=2242
Ernst's needed packages list https://mersenneforum.org/showpost.p...postcount=2243


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=23391
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2020-07-16 at 19:21
kriesel is offline  
Old 2020-07-30, 15:17   #24
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

2×3×11×67 Posts
Default Gpuowl gpu ram use scaling with exponent

GPU ram occupancy by Gpuowl v6.11-340 was observed using GPU-Z v2.33 on Windows 10 during performance of PRP iterations, for several scattered exponents. Idle gpu ram occupancy was entered for exponent 0. V6.11-364 was spot tested and observed essentially identical for the same exponent. At 100Mdigit, LL gpu ram usage was observed to be about 80% that of PRP. Note, no PRP proof computations have yet been observed for gpu ram usage. System ram usage for the process was observed to be less than gpu ram usage.
The regression fit was somewhat better plotted against FFT length than against exponent, not surprisingly.


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=23391
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Files
File Type: pdf gpuowl gpu ram use.pdf (11.1 KB, 11 views)

Last fiddled with by kriesel on 2020-08-10 at 02:08
kriesel is offline  
Old 2020-08-10, 00:30   #25
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

2×3×11×67 Posts
Default Using gpuowl's primenet.py on Windows

The script requires Python 3. Programming environments are not standard parts of a default Windows install, so Python, perl, C etc may be absent. Python may have been already installed as part of a MS Visual Studio install, or separately.

Check whether your target system already has Python, and which version.
On Windows 7:
Start, Control Panel, Programs, Uninstall a Program, scroll down the list of installed programs to see if a version of Python is installed.
Or on Windows 10:
Settings, Apps, scroll down the list of installed programs to see if a version of Python is installed.
If no Python 3.x is installed, follow the installation directions at https://docs.python.org/3/using/windows.html
Test that it can be easily invoked from a command prompt.
At a command prompt: python
If python is included in the path environment variable, it will respond with something like
Code:
Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
 >>>
(Ctl-Z Enter to exit interactive python.)
If it does not recognize python, it might just be missing from the path environment string.
On Windows 7:
Fix the path in a permanent way so that Python can be run in any future command prompt:
Start, Control Panel, System, Advanced System Settings, Environment Variables...
In the System variables portion, if it's not present, append something like the following to add the location of the Python folder on your system to the path
;C:\Program Files\Python36
(Scroll down to Variable "Path", click the line, click "Edit...", scroll to and click on the right end of the existing path, paste in the necessary additional text, click OK.
(This was necessary in my case because Python was not present in the path, and the cmd line setx /M did not work because the path was already beyond its 255-char limit, thanks to a lot of NVIDIA related content, and ordinary set can handle the long path length but is temporary, as is the command prompt box in which it is set.)

Make a hi.py file to test the Python install, containing
Quote:
print ('Hello!')
and try it:
Code:
>python hi.py
Hello!

>
If it worked, continue. If not, check that path modification.

If Gpuowl's primenet.py and upload.py are not present, copy them in. For test purposes I put mine in a gpuowl working directory.
If you have previously reported results in the results.txt file, now is a good time to move or remove them. (Otherwise they will generate "already reported" errors.)

Try running primenet.py with no parameters:
Quote:
>python primenet.py
If the PYTHONPATH environment variable is not set properly or at all, it will produce something like
Code:
Traceback (most recent call last):
  File "primenet.py", line 9, in <module>
    import requests
ModuleNotFoundError: No module named 'requests'
Create new system environment variable PYTHONPATH, per a hint from Mihai, since I found it did not exist.
On Windows 7,
Start, Control Panel, System and Security, System, Advanced system settings, Environment Variables..., scroll the system variables list. If there's no PYTHONPATH there, click New..., enter the Variable name PYTHONPATH, and enter a string corresponding to the location of requests.py as the Variable value.

By experimentation, the value needed seems to be in my case,
Quote:
C:\Program Files\Python36\Lib\site-packages\pip\_vendor
There should already be a proof subfolder in the working directory. Otherwise it will generate an error from the script.
When the script runs it will create subfolders __pycache__ and uploaded.
It requires at least a username. Following is the beginning of a successful run.
Code:
>python primenet.py -u primenet-uid

Work type: 150
Will fetch ahead 2 tasks. Check every 3600 sec.
User: primenet-uid
Watched dirs:  ./
Primenet password: primenet-pwd
It will wait and loop until terminated.

For reference, from gpuowl's readme.md, primenet.py's case-sensitive options are
Code:
## Primenet.py Arguments
-h, --help            show this help message and exit\
-u USERNAME           Primenet user name\
-p PASSWORD           Primenet password\
-t TIMEOUT            Seconds to sleep between updates\
--dirs DIR \[DIR ...\]  GpuOwl directories to scan\
--tasks NTASKS        Number of tasks to fetch ahead\
-w \{PRP,PM1,LL_DC,PRP_DC,PRP_WORLD_RECORD,PRP_100M\}   GIMPS work type
Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=23391
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1

Last fiddled with by kriesel on 2020-08-10 at 02:30
kriesel is offline  
Old 2020-08-10, 13:28   #26
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

2·3·11·67 Posts
Default Methods of uploading proof files

Gpuowl now supports generation of PRP proof files. These require uploading to the server for verification. Preda has provided primenet.py and upload.py for performing the upload. Installing Python on every system to use them may be unappealing or impractical for some. Here are some alternatives. Note, upload the JSON results records first, then the corresponding proof files later.
  1. Python interpreter, primenet.py, upload.py on each system. (Might not be allowed by corporate policy, disk space, available user time, etc.) This is I think the preferred method supported and created by Preda, author of Gpuowl.
  2. Python interpreter, primenet.py, upload.py on one system. Manually or by small script copy proof files over from other systems, to a file share on the Python-equipped system in which primenet.py operates.
  3. Upload via a prime95 or mprime install or multiple installs. As George described it (mostly) at https://www.mersenneforum.org/showpo...&postcount=220
    Code:
    1) Manually submit your results.json.txt first, before giving prime95 or mprime a chance at the proof file.
    2) move the proofs to a folder where prime95 v30.2 (or newer, or mprime) is running.
    3) wait for the proofs to get uploaded (I archive my proofs using advanced resource limits dialog box).
    Note that if there's an MD5 mismatch, it may just loop on retries ~65 minutes apart until you manually intervene, logged in results.txt, in prime95 V30.3b3, resembling the following.
    Code:
    [Fri Sep  4 07:51:19 2020]
    Unexpected error during 58000013-8.proof upload: {"error_status":401,"error_message":"Unauthorized","error_description":""}
    A nonnull error description would be helpful there; something like "MD5 mismatch detected", or whatever the case may be. I think this is a server-side consideration.
  4. Use the standalone uploader.exe program Prime95 previously provided for Windows at https://www.mersenneforum.org/showpo...&postcount=154. (No longer available on dropbox from that link. See attachment on this post below. Uploader reposted with George's permission.) Usage is
    Code:
    uploader user_id proof_filename[ chunk_size[  upload_rate_limit]]
    with chunk_size expressed in MB and upload_rate_limit expressed in Mbps apparently.
    A file share and tiny batch script can expedite this.
    Code:
    for %%a in ( *.proof ) do uploader your-primenet-uid %%a
    Or for a bit of logging of the uploads,
    Code:
    for %%a in (*.proof) do powershell ".\uploader.exe your-primenet-uid %%a | tee uplog.txt -Append"
    To include time stamps in the logging, and console output:
    Code:
    prompt $d $t $p$g 
    for %%a in (*.proof) do powershell "echo start upload %%a %date% %time% >> uplog.txt;.\uploader.exe kriesel %%a | tee uplog.txt -Append; echo end upload %%a %date% %time% >>uplog.txt"
    Retry in case of upload failure is manual; remove or rename successfully uploaded proof files or exclude them with the filespec. Delete the local proof copies after successful upload, if at all. (I'm currently using uploader.exe on a file share for occasional manual upload of gpuowl-produced proof files, and deleting local proof files after successful upload. Earlier I was waiting to delete until after a successful verification was manually confirmed.) Note, while prime95 and mprime's upload capability is being maintained, the standalone uploader may not be, and error handling may be somewhat less effective. If I recall correctly, this was created as a stopgap, before prime95 included proof file upload capability. Mprime/prime95 upload is preferred over the standalone uploader. The standalone uploader may require more than one user initiated try, as in the following.
    Code:
    Thu 08/27/2020 10:59:53.11 E:\PRP-PROOF-CACHE>for %a in (*.proof) do uploader kriesel %a
    
    Thu 08/27/2020 10:59:53.12 E:\PRP-PROOF-CACHE>uploader kriesel 160456789-8.proof
    MD5 of 160456789-8.proof is 5f8f1c693b35765a465072735a042649
    Proof file exponent is 160456789
    Filesize of 160456789-8.proof is 180513949
    No entries in need list: {"error_status":401,"error_message":"Unauthorized","error_description":""}
    
    Thu 08/27/2020 10:59:55.76 E:\PRP-PROOF-CACHE>up
    
    Thu 08/27/2020 11:01:23.41 E:\PRP-PROOF-CACHE>for %a in (*.proof) do uploader kriesel %a
    
    Thu 08/27/2020 11:01:23.42 E:\PRP-PROOF-CACHE>uploader kriesel 160456789-8.proof
    MD5 of 160456789-8.proof is 5f8f1c693b35765a465072735a042649
    Proof file exponent is 160456789
    Filesize of 160456789-8.proof is 180513949
    Success!
    
    Thu 08/27/2020 11:01:46.65 E:\PRP-PROOF-CACHE>
    Occasionally an upload will report an error, but a retry will report it has already been uploaded.
    Code:
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 39000037-8.proof | tee uplog.txt -Append"
    MD5 of 39000037-8.proof is ed9f4f30202027b27a6bbed5caa84691
    Proof file exponent is 39000037
    Filesize of 39000037-8.proof is 43875102
    CURL library error: Operation timed out after 180012 milliseconds with 0 bytes received
    
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
    MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
    Proof file exponent is 41000051
    Filesize of 41000051-8.proof is 46125120
    CURL library error: Operation timed out after 180011 milliseconds with 0 bytes received
    E:\PRP-PROOF-CACHE>up
    
    E:\PRP-PROOF-CACHE>for %a in (*.proof) do powershell ".\uploader.exe kriesel %a | tee uplog.txt -Append"
    
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 39000037-8.proof | tee uplog.txt -Append"
    MD5 of 39000037-8.proof is ed9f4f30202027b27a6bbed5caa84691
    Proof file exponent is 39000037
    Filesize of 39000037-8.proof is 43875102
    Server response missing URLToUse: {"error_status":409,"error_message":"Conflict","error_description":"Proof already uploaded"}
    
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
    MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
    Proof file exponent is 41000051
    Filesize of 41000051-8.proof is 46125120
    CURL library error: Operation timed out after 180005 milliseconds with 0 bytes received
    E:\PRP-PROOF-CACHE>up
    
    E:\PRP-PROOF-CACHE>for %a in (*.proof) do powershell ".\uploader.exe kriesel %a | tee uplog.txt -Append"
    
    E:\PRP-PROOF-CACHE>powershell ".\uploader.exe kriesel 41000051-8.proof | tee uplog.txt -Append"
    MD5 of 41000051-8.proof is 640129af8806cfb84b84084bb84bf284
    Proof file exponent is 41000051
    Filesize of 41000051-8.proof is 46125120
    Server response missing URLToUse: {"error_status":409,"error_message":"Conflict","error_description":"Proof already uploaded"}
    E:\PRP-PROOF-CACHE>
    In my experience these do go on to get certs run, successfully, so seem not to be an issue. Note that if there's an MD5 mismatch, it may look sort of like this:
    Code:
    MD5 of 58000013-8.proof is fed107d6833f4c8e1f67a450e15272bc
    Proof file exponent is 58000013
    Filesize of 58000013-8.proof is 65250075
    Server response missing URLToUse: {"error_status":401,"error_message":"Unauthorized","error_description":""}
    A nonnull error description would be helpful there; something like "MD5 mismatch detected", or whatever the case may be. I think this was a server-side security consideration. It has been changed to give a list of possible reasons, which the end user can check.
  5. Roll your own uploader script using curl or perl or whatever. For example, https://www.mersenneforum.org/showpo...&postcount=271
  6. Translation of the python scripts to a preferred language, such as perl. This is unappealing, as it would create a recurring maintenance workload whenever Preda revises his python scripts. Also knowing both the preferred language and knowing or learning python.
  7. Compilation of the gpuowl primenet.py, upload.py into a standalone executable. (I'm exploring this. If it works out, the primenet executable could be included with future Windows gpuowl build posts.) This would approximate #1 above, without the need to install a Python interpreter on every system running gpuowl.
Possibly some client management applications will add proof uploading capability at some point, also. (Mlucas primenet.py seems likely)
The total number of different uploaders ought be limited to a manageable set for the server maintainers to support on that end.


Top of this reference thread: https://www.mersenneforum.org/showthread.php?t=23391
Top of reference tree: https://www.mersenneforum.org/showpo...22&postcount=1
Attached Files
File Type: zip uploader.zip (1.93 MB, 13 views)

Last fiddled with by kriesel on 2020-09-17 at 13:42 Reason: added uploader logging via powershell tee
kriesel is offline  
Closed Thread

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Reference material discussion thread kriesel kriesel 60 2020-07-19 21:15
Mersenne Prime GPU Computing reference material kriesel kriesel 31 2020-07-09 14:04
CUDALucas-specific reference material kriesel kriesel 9 2020-05-28 23:32
Mfaktc-specific reference material kriesel kriesel 8 2020-04-17 03:50
CUDAPm1-specific reference material kriesel kriesel 12 2019-08-12 15:51

All times are UTC. The time now is 03:11.

Thu Sep 24 03:11:58 UTC 2020 up 14 days, 22 mins, 0 users, load averages: 1.70, 1.71, 1.55

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.