How does one prove that a mersenne prime found with CUDALucas is really prime?

[kriesel]

Originally Posted by preda

I was thinking the header, which could be the first line if single-line, or the first few lines otherwise, could be a textual format. So the exponent would be a string of ASCII digits, part of the header.

After the header would come the "bits" which would be binary to save space and simplify save/load.

This would allow easy human inspection of the savefile.

High human readability sounds great.

Bits: binary saves space, ASCII hexadecimal makes it readable and displayable in normal editors. (I'd rather not binhex as a density compromise; the readability is poor.) Since these would be used much less often than n-interim-savefiles-for-every-exponent, transmission size and time is less of a concern, and the entire file or small set of them can be put in a compressed archive.

Getting close. This format needs a name. "Mersenne neutral file format"?

Does something like this exist elsewhere in the number theory community? (Could we avoid some reinventing the wheel, or copy some good design features?)

Do you see software supporting both import and export?

Export would be handy for debugging on occasion, either in code development or in use.
For example, if a variant were available for P-1, in CUDAPm1, I'd be using it right now to try to understand what's going on with some bad runs on large exponents. Some problems due to limited memory are transferable from a gpu to another gpu model with much more memory, contained in the save file from the first gpu. The files are unreadable binary. Run parameters chosen on the first gpu are unnecessarily restrictive on the second. And some runs fail in the same way on the larger memory gpu as the first.

Import maybe should require some security key or authentication.

[preda]

Originally Posted by kriesel

Do you see software supporting both import and export?

The format would be worth only if there's buy-in from more than 1 developer.

If I consider it a good format, I would just use it for gpuOwl completely, as its only format. (thus, no separate import/export would be needed).

One piece of info that I put now in owl's files, that's not included in the portable format, is "error count", which was requested by George. Maybe the format would have a set of optional "comments" that are not essential, but are preserved by the software that does not understand them.

[ewmayer]

Originally Posted by kriesel

"Serge Batalov also verified it using Ernst Mayer's MLucas software on two Intel Xeon 18-core Amazon EC2 servers in 3.5 days." https://www.mersenne.org/primes/?press=M74207281
Not first and second half of iterations, but duplicate full runs?

I'd expect the n-parallel approach to work well with prime95/mprime since it is very common.

Yes, Serge's (and more recently, Anders') dual verify runs at different FFT lengths were both end-to-end.

Here is the current Mlucas file format:

o 1 byte for test type, numerically, i.e. 256 possible values, mapped to an internal table;
o 1 byte for modulus type, currently only Mersennes and Fermats supported;
o 8 bytes for iteration count of the residue stored in the file;
o ceiling(p/8) bytes for the residue R - i.e. maximally byte-compact, endian-and-FFT-length-of-run-independent;
o 8 bytes for Res64 = R (mod 2^64), which should match the leading 8 full-residue bytes in the above bytewise form);
o 5 bytes for R (mod 2^35-1);
o 5 bytes for R (mod 2^36-1) [these last 2 a.k.a. the Selfridge-Hurwitz residues, based on those guy's Fermat-number work, using a 36-bit-hardware-integer machine; SH also used R (mod 2^36), but that is just the low 36 bits of GIMPS' Res64];

After reading R, I directly compute the two SH residues and compare to the above file-stored checksums; this gives me an md5/sha1-style integrity check of the whole residue R, which the Res64 does not.

For v18, I am adding several new fields:
o 3 bytes for FFT-length-in-Kdoubles which the code was using at time of savefile write. This is so that if the code switches to a larger-than-default FFT length mid-run based on ROE behavior for the exponent in question, it will immediately resume using the larger FFT length on restart-from-interrupt, rather than resuming using the smaller default FFT length as the current release does.
o 8 bytes for circular-shift to apply to the (unshifted) residue read from the file. I include the shift-count-at-iteration-of-savefile-write because [a] the code will choose a random shift count at run-start time (i.e. since this is not specified by the Primenet server, it cannot be read from the worktodo file), and [b] it saves the need for taking an initial-shift value s from the savefile and computing s * 2^iter (mod p). I remove the shift from R prior to the savefile write, so in fact there's really no need to store s to the file (i.e. I could resume-from-interrupt using an entirely different random shift value, applied to R after reading it from the savefile), but for aesthetic reasons I like the idea of doing the whole run based on a single initial value of s, rather than as-many-values-of-s-as-there-were-run-interrupts.

[kriesel]

Originally Posted by ewmayer

Yes, Serge's (and more recently, Anders') dual verify runs at different FFT lengths were both end-to-end.

Here is the current Mlucas file format:

o 1 byte for test type, numerically, i.e. 256 possible values, mapped to an internal table;
o 1 byte for modulus type, currently only Mersennes and Fermats supported;
o 8 bytes for iteration count of the residue stored in the file;
o ceiling(p/8) bytes for the residue R - i.e. maximally byte-compact, endian-and-FFT-length-of-run-independent;
o 8 bytes for Res64 = R (mod 2^64), which should match the leading 8 full-residue bytes in the above bytewise form);
o 5 bytes for R (mod 2^35-1);
o 5 bytes for R (mod 2^36-1) [these last 2 a.k.a. the Selfridge-Hurwitz residues, based on those guy's Fermat-number work, using a 36-bit-hardware-integer machine; SH also used R (mod 2^36), but that is just the low 36 bits of GIMPS' Res64];

After reading R, I directly compute the two SH residues and compare to the above file-stored checksums; this gives me an md5/sha1-style integrity check of the whole residue R, which the Res64 does not.

For v18, I am adding several new fields:
o 4 bytes for FFT-length-in-K which the code was using at time of savefile write. This is so that if the code switches to a larger-than-default FFT length mid-run based on ROE behavior for the exponent in question, it will immediately resume using the larger FFT length on restart-from-interrupt, rather than resuming using the smaller default FFT length as the current release does.
o 8 bytes for circular-shift to apply to the (unshifted) residue read from the file. I include the shift-count-at-iteration-of-savefile-write because [a] the code will choose a random shift count at run-start time (i.e. since this is not specified by the Primenet server, it cannot be read from the worktodo file), and [b] it saves the need for taking an initial-shift value s from the savefile and computing s * 2^iter (mod p). I remove the shift from R prior to the savefile write, so in fact there's really no need to store s to the file (i.e. I could resume-from-interrupt using an entirely different random shift value, applied to R after reading it from the savefile), but for aesthetic reasons I like the idea of doing the whole run based on a single initial value of s, rather than as-many-values-of-s-as-there-were-run-interrupts.

That's such a clear and well organized format description, I think I'd like to quote you in my reference material threads.

[ewmayer]

Originally Posted by kriesel

That's such a clear and well organized format description, I think I'd like to quote you in my reference material threads.

Be my guest - note I make no claim to my format being as complete as is desirable, for example it lacks any kind of errors-during-the-run encoding - but I think most of the key features (byte-compact, endian-and-FFT-length-independent, some kind of whole-residue checksum included) should be requirements for any common savefile format which may end up getting agreed on.

[kriesel]

Originally Posted by ewmayer

o 8 bytes for circular-shift to apply to the (unshifted) residue read from the file. I include the shift-count-at-iteration-of-savefile-write because [a] the code will choose a random shift count at run-start time (i.e. since this is not specified by the Primenet server, it cannot be read from the worktodo file), and [b] it saves the need for taking an initial-shift value s from the savefile and computing s * 2^iter (mod p). I remove the shift from R prior to the savefile write, so in fact there's really no need to store s to the file (i.e. I could resume-from-interrupt using an entirely different random shift value, applied to R after reading it from the savefile), but for aesthetic reasons I like the idea of doing the whole run based on a single initial value of s, rather than as-many-values-of-s-as-there-were-run-interrupts.

It seems to me that recording the circular shift value provides much greater reproducibility from run to run. That could be very useful if debugging some shift-dependent case.