Primo reservation thread - Page 4

paulunderwood · 2021-01-03, 07:33

It would be a nice feature if Primo showed % done and an ETA.

Have you experimented with running more than one instance of Primo, maybe at different priorities? There are a lot of free cycles during phase 1.

Since ryanp is certifying M78737 cofactor, I'll have a stab at running M82939 cofactor (prp24948) at a low priority in parrallel with R49081,

Batalov · 2021-01-03, 08:09

There are pros and cons. Yes, you can run some other computations, but then (just my guess) primo master thread doesn't serve slaves with that share memory that it holds (that and queuing the spawned kids is its only work) all too well, and primo progress slows down disproportionally.

If you observed "top" then you can see mysterious parameters passed onto "stk*1" workers. I think the second to last parameter is a reference to its private offset into shared mem. They all borrow data from master by reference (master holds a couple Gb of RAM); the spawns are born small and then suddenly show 1.5g of RAM, too.

Ok, I'll tell you more. I once tried to distribute slaves across cluster. (I wrote wrappers that passed all parameters and managed quasi-queues.) But it was all in vain because they could not access memory of the master, and all quit.
Conclusion: the 1st stage has to be done on one physical node (with shmem).

I was able to run 2nd stage faster (if you have many nodes), though it is tedious and easy to make a mistake. YOU'VE BEEN WARNED; if you don't have a firm hand - don't do it. And disclaimer: that was with old versions - pre 4.3, so all .t files were processed in order.

Here is the sketch of how:

keep in mind that what 2nd stage does is: it takes any 'uncooked' .t file, and cooks it, and writes it back. That is embarrassingly parallel and doesn't have any if/else branching.
have at least one .t file reprocessed in master node (it becomes smaller and its datestamp is newer than the .tmp file)*
clone the complete work folder to slave node
on the slave node: copy the finished .t over half of the unfinished .t files (this will make the state of the "proof object" patently violated, but you will later only use all other .t files, and send them back to master)
start another primo on the slave node.
the slave primo will skip over half of the "fake-done" .t files (they have the 'done' bit set in their header) and will do all others. (and then it will fail and will probably delete all temp files, so you have to babysit and stop it before it is fully done.)
At that time master will still be busy. Stop it and transfer only the "really done" .t files from the slave, overwrite the corresponding undone .t files
restart master primo
it will finish the half that is its own and will find all others already done and will combine all .t files into the .out file.
Caution: one wrong move and you shot yourself in the foot**. Have a legitimate backup of non-faked native primo work folder.

TL;DR version: it is not worth the saved time. Don't do it.
____
* perhaps a cooked .t file can be taken frozen for some previously done project. Never tried. Who knows - maybe primo master does light sanity check on the cooked .t file? maybe it only reads its magic header (several bytes). Maybe it checks a few more things and then it will reject a foreign .t file.

** there was a comparative list of programming languages; what happens when the programmer shoots himself in the foot. There was one language where the foot then explodes. That is similar to what will happen here.

paulunderwood · 2021-01-03, 08:23

Hmm, I have just created a second Primo folder and am running M-cofactor at niceness 19. I have 128 threads after all. But you seem to be right about top -- I'll monitor the situation for a week or two,

Batalov · 2021-01-03, 08:37

What Primo does is it takes "an onion" and peels layers, down to the inner core. (Well for the onion, the volume is R^3, but this is a nearly good analogy. Think of it as if volume formula was R^4.5.)
So, here is a neat rule of thumb: if you peel off only 1/7 of the diameter, you have done half of the job.

The reverse is true:
Q: suppose you wanted to do a series of projects each twice as hard as the last.
A: Recipe to achieve that: always add +1/6 of the size of the last number.

In "Bits", primo shows how small the current onion has become. It doesn't matter for primo if you are just beginning a 70,000-bit number or if you peeled your 88,000-bit number to a 70,000-bit intermediate number. If I started right now another 70,000-bit number on another sibling machine, it would progress nearly exactly like this number in progress.

2021-01-03, 07:33	#34
paulunderwood Sep 2002 Database er0rr 3·11·107 Posts	It would be a nice feature if Primo showed % done and an ETA. Have you experimented with running more than one instance of Primo, maybe at different priorities? There are a lot of free cycles during phase 1. Since ryanp is certifying M78737 cofactor, I'll have a stab at running M82939 cofactor (prp24948) at a low priority in parrallel with R49081, Last fiddled with by paulunderwood on 2021-01-03 at 07:49

2021-01-03, 08:09	#35
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 47×197 Posts	There are pros and cons. Yes, you can run some other computations, but then (just my guess) primo master thread doesn't serve slaves with that share memory that it holds (that and queuing the spawned kids is its only work) all too well, and primo progress slows down disproportionally. If you observed "top" then you can see mysterious parameters passed onto "stk1" workers. I think the second to last parameter is a reference to its private offset into shared mem. They all borrow data from master by reference (master holds a couple Gb of RAM); the spawns are born small and then suddenly show 1.5g of RAM, too. Ok, I'll tell you more. I once tried to distribute slaves across cluster. (I wrote wrappers that passed all parameters and managed quasi-queues.) But it was all in vain because they could not access memory of the master, and all quit. Conclusion: the 1st stage has to be done on one physical node (with shmem). I was able to run 2nd stage faster (if you have many nodes), though it is tedious and easy to make a mistake. YOU'VE BEEN WARNED; if you don't have a firm hand - don't do it. And disclaimer: that was with old versions - pre 4.3, so all .t files were processed in order. Here is the sketch of how: keep in mind that what 2nd stage does is: it takes any 'uncooked' .t file, and cooks it, and writes it back. That is embarrassingly parallel and doesn't have any if/else branching. have at least one .t file reprocessed in master node (it becomes smaller and its datestamp is newer than the .tmp file) clone the complete work folder to slave node on the slave node: copy the finished .t over half of the unfinished .t files (this will make the state of the "proof object" patently violated, but you will later only use all other .t files, and send them back to master) start another primo on the slave node. the slave primo will skip over half of the "fake-done" .t files (they have the 'done' bit set in their header) and will do all others. (and then it will fail and will probably delete all temp files, so you have to babysit and stop it before it is fully done.) At that time master will still be busy. Stop it and transfer only the "really done" .t files from the slave, overwrite the corresponding undone .t files restart master primo it will finish the half that is its own and will find all others already done and will combine all .t files into the .out file. Caution: one wrong move and you shot yourself in the foot. Have a legitimate backup of non-faked native primo work folder. TL;DR version*: it is not worth the saved time. Don't do it. ____ perhaps a cooked .t file can be taken frozen for some previously done project. Never tried. Who knows - maybe primo master does light sanity check on the cooked .t file? maybe it only reads its magic header (several bytes). Maybe it checks a few more things and then it will reject a foreign .t file. ** there was a comparative list of programming languages; what happens when the programmer shoots himself in the foot. There was one language where the foot then explodes. That is similar to what will happen here.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
15*2^n-1, n>1M Reservation Thread	Kosmaj	Riesel Prime Search	702	2018-04-11 07:36
5*2^n-1 Reservation Thread	VBCurtis	Riesel Prime Search	657	2017-09-29 15:52
Octoproth Reservation Thread	Greenbank	Octoproth Search	2	2007-12-26 09:58
Dodecaproth Reservation Thread	Greenbank	Octoproth Search	30	2006-02-09 00:33
Hexadecaproth Reservation Thread (n=76)	Greenbank	Octoproth Search	0	2006-01-25 13:41

2021-01-03, 08:23	#36
paulunderwood Sep 2002 Database er0rr 3×11×107 Posts	Hmm, I have just created a second Primo folder and am running M-cofactor at niceness 19. I have 128 threads after all. But you seem to be right about top -- I'll monitor the situation for a week or two,

2021-01-03, 08:37	#37
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 22053₈ Posts	What Primo does is it takes "an onion" and peels layers, down to the inner core. (Well for the onion, the volume is R^3, but this is a nearly good analogy. Think of it as if volume formula was R^4.5.) So, here is a neat rule of thumb: if you peel off only 1/7 of the diameter, you have done half of the job. The reverse is true: Q: suppose you wanted to do a series of projects each twice as hard as the last. A: Recipe to achieve that: always add +1/6 of the size of the last number. In "Bits", primo shows how small the current onion has become. It doesn't matter for primo if you are just beginning a 70,000-bit number or if you peeled your 88,000-bit number to a 70,000-bit intermediate number. If I started right now another 70,000-bit number on another sibling machine, it would progress nearly exactly like this number in progress.