v19 pre-release discussion

ewmayer · 2019-08-25, 23:00

Update on v19 code development: At long last I have the basic Mersenne-mod PRP Gerbicz-checking working in Mlucas - here from my gdb session stepping through the latest code, this fiddled things so as to update the Gerbicz checkproduct (mod-product of PRP-test residues at fixed user-set iteration intervals) every 1000 squarings, and perform the final up-squaring step and comparison of the 2 resulting mod-products at the end of that. The two displayed Res64 values compare just the bottom limb of the 2 residue-length quantities being compared ... the mi64_cmp_eq() call compares the full vectors:

Code:

Gerbicz check: B[] Res64: 7681E95DC8B1C8A6

Breakpoint 10, ernstMain (mod_type=1, test_type=2, exponent=173431, fft_length=8, radix_set=2, maxFFT=8, iterations=10000, sh0=0x7fff5fbff910, sh1=0x7fff5fbff908, sh2=0x7fff5fbff900, scrnFlag=0, runtime=0x7fff5fbff480) at ../Mlucas.c:1871
1871				for(i = 1; i < PRP_BASE; i++) {
(gdb) n
1872					if(mi64_cmpult(c_uint64_ptr,d_uint64_ptr,j)) break;
(gdb) 
1876				ASSERT(HERE, mi64_cmpult(c_uint64_ptr,d_uint64_ptr,j), "Gerbicz checkproduct reduction (mod 2^p-1) failed!");
(gdb) 
1877	fprintf(stderr,"Gerbicz check: D[] Res64: %016llX\n",c_uint64_ptr[0]);
(gdb) 
Gerbicz check: D[] Res64: 7681E95DC8B1C8A6
1878				if(mi64_cmp_eq((uint64*)arrtmp,c_uint64_ptr,j))
(gdb) 
1879					fprintf(stderr,"Gerbicz check passed!\n");
(gdb) 
Gerbicz check passed!

The defaults I intend to implement for the code release are as above to update the Gerbicz checkproduct every 1000 squarings (and save it along with the main PRP-test residue to savefile every 10000 or 100000 squarings), and perform the final up-squaring step and comparison of the 2 resulting mod-products every 10^6 squarings. That gives a near-optimal extra-work expense equivalent to 1000 2-input modmuls (for the every-1000-iter updates) plus 1000 modsquares (for the final up-squaring step), roughly equivalent to 3000 modsquares total, every 10^6 PRP iterations, or 0.3% overhead to implement the added check. Should said check detect a residue error during the PRP test there will of course be extra work involved in the rollback to the last 1M-iter savefile. But the % of runs which suffer such an error-detected and rollback should be about the same as the current level of bad-LL-test residues, so assuming the latter suffer fewer than 1 G-check rollback every million iterations, it's a win.

Still remaining to be done prior to code release:

1. New 2-operand modmul code in the real/complex-FFT-wrapper-and-dyadic-square step occurring between the forward and inverse FFTs needs to be SIMD-ized, which means sse2, avx, avx-512 and ARM asimd assembly-code macros need to be written and debugged;

2. Gerbicz-checkproduct write/read and integrity-checking needs to be added to the savefile write/read code;

3. The rollback-on-error handling mechanism needs to be coded up;

4. Said savefile-code needs to play nice with the v18-added premature-checkpoint-on-signal-and-exit handling ... when such a signal is cuaght we need to update the PRP residue in the savefile but not the Gerbicz checkproduct;

5. The Gerbicz-check computations need to be fiddled to play nice in the cintext of circularly-shifted PRP residues - all my code-dev so far has been for the easier shift = 0 case.

kriesel · 2019-08-29, 12:38

(.... crickets, for days....)
6. Update documentation, including file format

Any guess when v19 release may occur?
With or without primenet integration?

ewmayer · 2019-08-29, 19:29

Quote:

Originally Posted by kriesel

(.... crickets, for days....)
6. Update documentation, including file format

Any guess when v19 release may occur?
With or without primenet integration?

I should've provided some kind of estimated timeline in my above - it's gonna be at least a month before a beta release, and that's only absent the kinds of time-costing hurdles that nearly always crop up along the way.

The primenet.py script that's been shipping since v17 makes tight primenet integration unnecessary, IMO. The one added feature that would be nice to have in said script would be the ability to do regular assignment progress updates ... I actually have the code needed for that in place and debugged, but the current server set-up is such that it needed Aaron (madpoo) to do some manual intervention at the server end (in effect simulating the result of a v5 API "update computer info" transaction) for me to be able to test and use that new feature.

paulunderwood · 2019-08-29, 22:27

A month should coincide with the end of my first N2 wavefront LL test. I look forward to using the PRP beta version.

ET_ · 2019-08-30, 10:48

Quote:

Originally Posted by paulunderwood

A month should coincide with the end of my first N2 wavefront LL test. I look forward to using the PRP beta version.

So do I, if it allows for PRP on Mersenne composites

ewmayer · 2019-08-30, 19:05

Quote:

Originally Posted by ET_

So do I, if it allows for PRP on Mersenne composites

PRP-C (composite cofactor PRP testing) will have to wait until v20 - based on past experience and this being a 1-man coding show, I try to limit myself to one major new feature per release. Sorry!

ET_ · 2019-08-30, 19:32

Quote:

Originally Posted by ewmayer

PRP-C (composite cofactor PRP testing) will have to wait until v20 - based on past experience and this being a 1-man coding show, I try to limit myself to one major new feature per release. Sorry!

I will wait. Though I guess Raspy users would have chosen PRP-C over plain vanillina PRP. Shorter running times, you know...

kriesel · 2019-09-30, 22:59

Quote:

Originally Posted by ewmayer

I should've provided some kind of estimated timeline in my above - it's gonna be at least a month before a beta release, and that's only absent the kinds of time-costing hurdles that nearly always crop up along the way.

How goes it? Presumably this is still taking priority over the SP LL experiment https://www.mersenneforum.org/showth...t=23926&page=4

paulunderwood · 2019-10-03, 22:26

My first LLR test on the N2 is nearly done. Is PRP-3 mlucas imminent?

ewmayer · 2019-10-04, 19:02

Quote:

Originally Posted by paulunderwood

My first LLR test on the N2 is nearly done. Is PRP-3 mlucas imminent?

Alas, no - I have all the new core-math-code infrastructure (needed to support generic 2-imput FFT-modmul ... my code was 100% geared toward 1-input FFT-autosquare up til now) in place including 6 custom versions of the key optimized code macros (scalar-double, ARMv8 SIMD, sse2,avx,avx2/fma,avx-512) but wrestling with all the control logic needed for the new execution path is taking longer than I had hoped. So please bear with me and just queue up more LL-test work as your current jobs finish.

ewmayer · 2019-11-16, 21:20

I am doing final shakedown tests of the v19 beta release on the hardware available to me. Since the fellow who physically hosted the GIMPS KNL workstation has gone AWOL, I could use remote access to a Skylake-X system running Linux in order to test the new PRP+Gerbicz code under avx-512.

2019-08-25, 23:00	#1
ewmayer ∂²ω=0 Sep 2002 República de California 2²×3×5×193 Posts	v19 pre-release discussion Update on v19 code development: At long last I have the basic Mersenne-mod PRP Gerbicz-checking working in Mlucas - here from my gdb session stepping through the latest code, this fiddled things so as to update the Gerbicz checkproduct (mod-product of PRP-test residues at fixed user-set iteration intervals) every 1000 squarings, and perform the final up-squaring step and comparison of the 2 resulting mod-products at the end of that. The two displayed Res64 values compare just the bottom limb of the 2 residue-length quantities being compared ... the mi64_cmp_eq() call compares the full vectors: Code: Gerbicz check: B[] Res64: 7681E95DC8B1C8A6 Breakpoint 10, ernstMain (mod_type=1, test_type=2, exponent=173431, fft_length=8, radix_set=2, maxFFT=8, iterations=10000, sh0=0x7fff5fbff910, sh1=0x7fff5fbff908, sh2=0x7fff5fbff900, scrnFlag=0, runtime=0x7fff5fbff480) at ../Mlucas.c:1871 1871 for(i = 1; i < PRP_BASE; i++) { (gdb) n 1872 if(mi64_cmpult(c_uint64_ptr,d_uint64_ptr,j)) break; (gdb) 1876 ASSERT(HERE, mi64_cmpult(c_uint64_ptr,d_uint64_ptr,j), "Gerbicz checkproduct reduction (mod 2^p-1) failed!"); (gdb) 1877 fprintf(stderr,"Gerbicz check: D[] Res64: %016llX\n",c_uint64_ptr[0]); (gdb) Gerbicz check: D[] Res64: 7681E95DC8B1C8A6 1878 if(mi64_cmp_eq((uint64)arrtmp,c_uint64_ptr,j)) (gdb) 1879 fprintf(stderr,"Gerbicz check passed!\n"); (gdb) Gerbicz check passed! The defaults I intend to implement for the code release are as above to update the Gerbicz checkproduct every 1000 squarings (and save it along with the main PRP-test residue to savefile every 10000 or 100000 squarings), and perform the final up-squaring step and comparison of the 2 resulting mod-products every 10^6 squarings. That gives a near-optimal extra-work expense equivalent to 1000 2-input modmuls (for the every-1000-iter updates) plus 1000 modsquares (for the final up-squaring step), roughly equivalent to 3000 modsquares total, every 10^6 PRP iterations, or 0.3% overhead to implement the added check. Should said check detect a residue error during the PRP test there will of course be extra work involved in the rollback to the last 1M-iter savefile. But the % of runs which suffer such an error-detected and rollback should be about the same as the current level of bad-LL-test residues, so assuming the latter suffer fewer than 1 G-check rollback every million iterations, it's a win. Still remaining to be done prior to code release: 1. New 2-operand modmul code in the real/complex-FFT-wrapper-and-dyadic-square step occurring between the forward and inverse FFTs needs to be SIMD-ized, which means sse2, avx, avx-512 and ARM asimd assembly-code macros need to be written and debugged; 2. Gerbicz-checkproduct write/read and integrity-checking needs to be added to the savefile write/read code; 3. The rollback-on-error handling mechanism needs to be coded up; 4. Said savefile-code needs to play nice with the v18-added premature-checkpoint-on-signal-and-exit handling ... when such a signal is cuaght we need to update the PRP residue in the savefile but not the Gerbicz checkproduct; 5. The Gerbicz-check computations need to be fiddled to play nice in the cintext of circularly-shifted PRP residues - all my code-dev so far has been for the easier shift = 0 case. Last fiddled with by ewmayer on 2019-08-25 at 23:00*

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
v18 pre-release discussion	ewmayer	Mlucas	11	2019-03-01 04:10
Next GMP release?	Andi47	GMP-ECM	6	2007-11-26 07:29
v24.13 release candidate 1	Prime95	Software	13	2005-07-14 23:29
V24.12 release candidate 3	Prime95	Software	45	2005-07-02 19:13
V24.12 Release 1 Crashing on P4 M	njcroquet1	Software	8	2005-06-24 14:40

2019-08-29, 12:38	#2
kriesel "TF79LL86GIMPS96gpu17" Mar 2017 US midwest 29×167 Posts	(.... crickets, for days....) 6. Update documentation, including file format Any guess when v19 release may occur? With or without primenet integration?

2019-08-29, 22:27	#4
paulunderwood Sep 2002 Database er0rr 2·5·353 Posts	A month should coincide with the end of my first N2 wavefront LL test. I look forward to using the PRP beta version.

2019-10-03, 22:26	#9
paulunderwood Sep 2002 Database er0rr 6712₈ Posts	My first LLR test on the N2 is nearly done. Is PRP-3 mlucas imminent?

2019-11-16, 21:20	#11
ewmayer ∂²ω=0 Sep 2002 República de California 2²·3·5·193 Posts	I am doing final shakedown tests of the v19 beta release on the hardware available to me. Since the fellow who physically hosted the GIMPS KNL workstation has gone AWOL, I could use remote access to a Skylake-X system running Linux in order to test the new PRP+Gerbicz code under avx-512.