mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > News

Reply
 
Thread Tools
Old 2020-07-13, 16:18   #100
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

114516 Posts
Default Revisiting shift, DC, LL, discovery confirmation

Given the development of PRP-proof and verification for computing effort that's a small fraction of what a full PRP DC or LL requires:

Software enhancement priorities change
Does this significantly lower the value of nonzero shift as a possible future enhancement of gpuowl PRP?
Does it significantly lower the value of improved error detection and correction in gpuowl LL DC?
Does it significantly lower the value of nonzero shift as a possible future enhancement of gpuowl LLDC?

Does it by comparison raise the relative value of improved error detection and correction in P-1 factoring? (It lowers the cost of missing a factor due to error, by almost a factor of two after extensive PRP-proof adoption, but perhaps reduces future competition for programming time, by reducing incentives for adding PRP shift, LL shift, LL error detection and correction etc. to gpuowl.)

PrimeNet work assignment types change
PRP verifications will be introduced as a new work type eventually (~1 year)
Will we do single PRP-verifications per exponent, or DC PRP-verifications? Possibly a separate PRP-V-DC work type?
PRP first tests will continue. They may eventually (years from now) require the ability to do proof generation.
PRP DC will continue a while, but may phase out before LLDC.
LL will phase out. LLDC will phase out. These phaseouts will take years.

LL proof/Verification
Gerbicz gives a method for LL of verifying in (O)sqrt(p). https://mersenneforum.org/showpost.p...3&postcount=10 It takes more time and more disk space than the PRP-proof. So even if this were implemented, PRP would have a proof and verification speed advantage and adoption advantage over LL with its fastest verification.
For p~108, sqrt(p)/log2(p) = 104/26.58 =376:1 (and worsens as p increases)

So we would prioritize PRP&proof over LL for ordinary testing. The stronger time savings in PRP with proof and verification versus conventional DC or LL with a slower proof/verification possibility imply phaseout of LL as a first-test as quickly as practical. Also sqrt(p) residues, 120GB per exponent is onerous for the LL case in ordinary use.

But, an actual verification/certificate approach would be more persuasive proof of primality via confirming LL test, for the known Mersenne primes and new discoveries, than the usual approach of LL multiple times on different software and hardware and see if they agree.

PRP & proof generation 1.004 x an ordinary PRP (GEC present in both cases)
PRP verification 0.004 x
LL & proof ~2 x
LL verification ?
Past Mersenne prime confirmations have usually been by 3 or 4 duplicate LL runs on different hardware and software. We can now run mprime/prime95, mlucas, CUDALucas, cllucas, and gpuowl, 5 distinct software programs. And possibly glucas, which has been used in the past for confirmations.
kriesel is offline   Reply With Quote
Old 2020-07-15, 05:30   #101
Mark Rose
 
Mark Rose's Avatar
 
"/X\(‘-‘)/X\"
Jan 2013

2·1,429 Posts
Default

If possible, can the temporary files be written to in large chunks, such as 4 MB at a time? There are a few reasons for this:

1. SSDs suffer from write amplification, where writes smaller than 4 KB involve writing a whole 4 KB page. At a very minimum, the writes for these files should be 4 KB. But it's better if the writes can fully fill an erase block on the SSD, which can be up to 4 MB in size these days: when a file is later deleted, the whole erase block can be marked as unused, and this reduces the amount of garbage collection and rewriting SSDs must do after deleting the file.

2. Large IO reduces system calls, which have gotten expensive with Spectre and Meltdown mitigations that require flushing on context switches, which is what system calls are. This may also impact the continued processing in Prime95 at every system call.

3. If overwriting in place, large IO is much kinder to copy-on-write file systems, such as Btrfs and ZFS.

Normally I wouldn't be too concerned about this, but Prime95 has the potential to wear out SSDs, especially low end consumer drives with low endurance media or without a lot of free space for wear leveling.
Mark Rose is offline   Reply With Quote
Old 2020-07-15, 06:25   #102
preda
 
preda's Avatar
 
"Mihai Preda"
Apr 2015

24·3·52 Posts
Default

Quote:
Originally Posted by Mark Rose View Post
If possible, can the temporary files be written to in large chunks, such as 4 MB at a time? There are a few reasons for this:

1. SSDs suffer from write amplification, where writes smaller than 4 KB involve writing a whole 4 KB page. At a very minimum, the writes for these files should be 4 KB. But it's better if the writes can fully fill an erase block on the SSD, which can be up to 4 MB in size these days: when a file is later deleted, the whole erase block can be marked as unused, and this reduces the amount of garbage collection and rewriting SSDs must do after deleting the file.

2. Large IO reduces system calls, which have gotten expensive with Spectre and Meltdown mitigations that require flushing on context switches, which is what system calls are. This may also impact the continued processing in Prime95 at every system call.

3. If overwriting in place, large IO is much kinder to copy-on-write file systems, such as Btrfs and ZFS.

Normally I wouldn't be too concerned about this, but Prime95 has the potential to wear out SSDs, especially low end consumer drives with low endurance media or without a lot of free space for wear leveling.
GpuOwl does the bulk of the file writes through C's fwrite(), and in this situation (writing residues) a full residue is fwrite()'ed at once (that would be ~12MB for 100M exponents). BUT this fwrite() does not go to disk, it goes to the OS's cache which is in RAM. Even after the file is closed it may be a while before the data reaches the disk (because I don't fsync() it). We can assume that the OS is quite smart about when and how to flush the cached&dirty data to disk (should do it in big chunks if the hardware likes that).

It would be a problem only if the software would do small writes *and* would also force flush-to-disk with a fsync-like. (without fsync, the OS would still aggregate whatever sequential writes happen to a file, before they reach the disk)
preda is offline   Reply With Quote
Old 2020-07-20, 01:47   #103
Prime95
P90 years forever!
 
Prime95's Avatar
 
Aug 2002
Yeehaw, FL

157618 Posts
Default

Primenet's first official number verified by proof: 97009879

This went through all the steps. Prime95 PRP test with proof generation, file upload by separate program, server proof processor manually launched exponent marked awailable for certification, prime95 got a certification assignment, prime95 ran and uploaded the certification.

Not close to being ready for everyday use. Error handling is far from robust, the proof file uploader is broken again, server proof processor needs to be launched automatically, prime95 settings for resource management need inventing. And on and on.
Prime95 is online now   Reply With Quote
Old 2020-07-20, 02:57   #104
petrw1
1976 Toyota Corona years forever!
 
petrw1's Avatar
 
"Wayne"
Nov 2006
Saskatchewan, Canada

22×3×5×73 Posts
Default

Quote:
Originally Posted by Prime95 View Post
Primenet's first official number verified by proof: 97009879
Woot Woot
petrw1 is online now   Reply With Quote
Old 2020-07-20, 09:57   #105
ET_
Banned
 
ET_'s Avatar
 
"Luigi"
Aug 2002
Team Italia

2×2,383 Posts
Default

Quote:
Originally Posted by Prime95 View Post
Primenet's first official number verified by proof: 97009879

This went through all the steps. Prime95 PRP test with proof generation, file upload by separate program, server proof processor manually launched exponent marked awailable for certification, prime95 got a certification assignment, prime95 ran and uploaded the certification.

Not close to being ready for everyday use. Error handling is far from robust, the proof file uploader is broken again, server proof processor needs to be launched automatically, prime95 settings for resource management need inventing. And on and on.
"A small step for a programmer, a giant leap for the GIMPS" (C)
ET_ is offline   Reply With Quote
Old 2020-07-20, 11:58   #106
M344587487
 
M344587487's Avatar
 
"Composite as Heck"
Oct 2017

2·313 Posts
Default

Quote:
Originally Posted by Mark Rose View Post
...
If wear is a problem and you can spare the RAM (and accept the drawback of an awkward-to-interrupt/uninterruptible test), doing the test from a RAM disk avoids writing temp files to storage. You just need to be mindful to move the proof file to storage after every test, you could do this pretty easily with a script that polls the temp fs for proof files if there isn't already a copy mechanism in place.
M344587487 is offline   Reply With Quote
Old 2020-07-20, 14:07   #107
Prime95
P90 years forever!
 
Prime95's Avatar
 
Aug 2002
Yeehaw, FL

23×311 Posts
Default

Quote:
Originally Posted by M344587487 View Post
If wear is a problem and you can spare the RAM (and accept the drawback of an awkward-to-interrupt/uninterruptible test), doing the test from a RAM disk avoids writing temp files to storage. You just need to be mindful to move the proof file to storage after every test, you could do this pretty easily with a script that polls the temp fs for proof files if there isn't already a copy mechanism in place.
Both gpuowl and prime95 allow specifying a separate directory for the residues. No scripts necessary.
Prime95 is online now   Reply With Quote
Old 2020-07-20, 21:13   #108
preda
 
preda's Avatar
 
"Mihai Preda"
Apr 2015

120010 Posts
Default

Quote:
Originally Posted by Prime95 View Post
Both gpuowl and prime95 allow specifying a separate directory for the residues. No scripts necessary.
That would work like this with gpuowl:
1. specify -tmpDir to a non-existent folder. GpuOwl will check whether the folder exist, and log a warning if it doesn't exist, but accept it nevertheless. (this is the planned behavior, not implemented)

2. because the residues can't be written to the -tmpDir location, they'll be kept in RAM until the proof is generated.

The above technique is dangerous -- in the case of a power cut, or process kill/crash, the residues are gone and the proof can't be generated.
preda is offline   Reply With Quote
Old 2020-07-21, 08:24   #109
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

293 Posts
Default

That seems somewhat counterintuitive. When passing a non-existing folder, one would expect the program to create it.

Last fiddled with by kruoli on 2020-07-21 at 08:26 Reason: Grammar.
kruoli is offline   Reply With Quote
Old 2020-07-21, 08:29   #110
retina
Undefined
 
retina's Avatar
 
"The unspeakable one"
Jun 2006
My evil lair

25·179 Posts
Default

Quote:
Originally Posted by kruoli View Post
That seems somewhat counterintuitive. When passing a non-existing folder, one would expect the program to create it.
I agree.

Perhaps make it "-tmpdir -" or similar if you want to use RAM.

I also hope that tmpDir and tmpdir are considered the same.
retina is online now   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Your help wanted - Let's buy GIMPS a KNL development system! airsquirrels Hardware 313 2019-10-29 22:51
Is GMP-ECM still under active development? mathwiz GMP-ECM 0 2019-05-15 01:06
LLR 3.8.6 Development version Jean Penné Software 0 2011-06-16 20:05
LLR 3.8.5 Development version Jean Penné Software 6 2011-04-28 06:21
LLR 3.8.4 development version is available! Jean Penné Software 4 2010-11-14 17:32

All times are UTC. The time now is 05:49.

Wed Sep 23 05:49:20 UTC 2020 up 13 days, 3 hrs, 0 users, load averages: 2.07, 1.62, 1.62

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.