mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > News

Reply
 
Thread Tools
Old 2021-08-21, 22:52   #1
ssybesma
 
"Steve Sybesma"
May 2012
Brighton, CO USA

89 Posts
Default 12 machines permanently sidelined: LL was better than PRP on storage space

Unless you guys can come up with a way to limit the amount of storage PRP wants to eat up, I'm done with the GIMPS project as far as most of my machines. I have 20 machines and now 12 have been forced out.

For a long time I was running Intel Compute Sticks (1GB RAM and 8GB storage) with a stripped down version of Peppermint OS and I had many completed LL tests doing it that way. It took about 4 months...and that was fine with me.

Forcing PRP is the end of first-time testing for me...12 machines have to be taken offline permanently that can't do first-time tests.

I didn't get into this project to do double-checks exclusively.

Do you have any suggestion to limit what PRP uses to no more than 512MB of space?

LL did not use that much storage space for its temp files. It was a fraction of what PRP uses.

Last fiddled with by ssybesma on 2021-08-21 at 22:53
ssybesma is offline   Reply With Quote
Old 2021-08-21, 23:32   #2
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

F5F16 Posts
Default

Have you thought about PXE booting SLAX Linux from one HDD?

Last fiddled with by paulunderwood on 2021-08-21 at 23:37
paulunderwood is offline   Reply With Quote
Old 2021-08-21, 23:48   #3
ssybesma
 
"Steve Sybesma"
May 2012
Brighton, CO USA

89 Posts
Default PXE booting

Quote:
Originally Posted by paulunderwood View Post
Have you thought about PXE booting SLAX Linux from one HDD?
Ugh...I wouldn't even know how to begin doing that. You're saying one machine acts as a server for the smaller machines to run mprime on?

All I really need is a way to put a limitation on how much disk space the PRP checks consume.

If I can do that, I can do PRP checks with all my machines but they are still a little slower, but at least they won't freeze up my machines. A few months ago I figured out out to slim down Peppermint OS even more but it's not really enough to make PRP happy. It almost eats up everything you have, even if you make more space.

If I can put a hard limit on PRP disk space consumption to say 400MB, that would be ideal...that's all the temp files combined.

I think now that amount is way over 700MB and I've already removed 9 machines from the project so all the files are gone.

I still have 11 machines finishing up but those will all be forced onto PRP after they complete.

Last fiddled with by ssybesma on 2021-08-21 at 23:48
ssybesma is offline   Reply With Quote
Old 2021-08-22, 00:11   #4
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

5·787 Posts
Default

Quote:
Originally Posted by ssybesma View Post
Ugh...I wouldn't even know how to begin doing that. You're saying one machine acts as a server for the smaller machines to run mprime on?
Yes my first suggestion was a too drastic. Since you have Peppermint OS installed, why not set up a NFS to and from an HDD'd machine?

Last fiddled with by paulunderwood on 2021-08-22 at 00:19
paulunderwood is offline   Reply With Quote
Old 2021-08-22, 00:23   #5
Prime95
P90 years forever!
 
Prime95's Avatar
 
Aug 2002
Yeehaw, FL

768510 Posts
Default

You can put "ProofPower=5" in prime.txt. This would limit disk usage to about 32*14MB = 448MB which will go up slowly as the first-test wavefront advances.
You may also need to add "ProofPowerMult=1"

Alternatively, if you have a reasonably sized network disk drive mounted, you can ask prime95 to store the temporary files on the network disk. I do this with several of my computers.

Also: Tell prime95 to create fewer save files.

Last fiddled with by Prime95 on 2021-08-22 at 00:25
Prime95 is offline   Reply With Quote
Old 2021-08-22, 00:27   #6
ssybesma
 
"Steve Sybesma"
May 2012
Brighton, CO USA

89 Posts
Default

Quote:
Originally Posted by paulunderwood View Post
Yes my first suggestion was a too drastic. Since you have Peppermint OS installed, why not set up a NFS to and from an HDD'd machine?
OK, NFS being file sharing...that is setup. I can access files on any device from any another.

So how do I point mprime on each device to look at a folder on larger (more storage space) machine for its temp files?

That would certainly solve the problem.

Steve

Last fiddled with by ssybesma on 2021-08-22 at 00:27
ssybesma is offline   Reply With Quote
Old 2021-08-22, 00:37   #7
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

5·787 Posts
Default

If the server is set up with something like /nfs_share/stick1, /nfs_share/stick2 etc then stick machine 1 accesses its share which contains all it needs for running mprime including the executable,

Last fiddled with by paulunderwood on 2021-08-22 at 00:44
paulunderwood is offline   Reply With Quote
Old 2021-08-22, 00:37   #8
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

22×1,481 Posts
Default

I think you have some choices. (Some of this is already covered somewhat by previous posts made by others while I was writing this)

Assuming 1GB ram, 8GB SD storage, Linux, mprime, & wifi equipped (802.11b/g/n):

A) mprime & PRP / GEC at low proof power is better than none. Run a low proof power to reduce storage space required for proof generation.
At power 7, that's ~3.2GB temp space, 400MB proof file, for p=100M, proportional to exponent (so ~3.5GB & 440MB at p=110M) both of which can get cleared automatically after proof is uploaded to the server. Still saves ~99.2% of DC effort.
At power 6, that's 1.6GB temp space, ~800MB proof file, and still saves ~98.5% of DC effort. (~1.76GB & 880MB =2.64GB at p=110M)
Minimum proof power in prime95/mprime is 5; 800MB temp space, & still saves ~97% of DC effort.
(George please correct any wrong numbers given)
How many GB of storage does your minimum install use up?

B) use other options to limit mprime's local footprint.
Put the executable, the temp file or proof file or all on a LAN server, and perhaps the more actively used working files stay on the stick's working directory.
Use Wifi or other connectivity options to access server storage. Use the separate settings for "optional directory to hold large temporary files" and "optional directory to hold archived proofs"
And as George said, reduce mprime number of backup files. Maybe you already have.

C) Use Windows which can support bigger SD cards (32GB) as local storage (but maybe that requires swapping out the Intel compute sticks or more cost than tolerable)

D) (Not recommended due to low efficiency, ~50%, sort of a waste of time compared to PRP/GEC/proof-gen once) Run PRP/GEC without proof, and PRP DC your own PRP first tests or partner with another contributor to DC each other's.
Recent Mlucas, or mprime V29.8 don't implement proof generation, so could be run on hardware that lacks space for proof gen and also lacks access to network shares with sufficient space. No proof generation means a full PRP DC is required.

E) Sell your idled 12 sticks, and buy newer hardware with adequate capacity for today's workloads.
I haven't used them, but some like Raspberry Pi. Recent Raspberry Pis have gigabit wired ethernet jacks.
Those could probably be network booted and use server hosted working directories, so make use of TB space available for sharing, temp space, proof file, backup files, etc. They're ARM based so would involve using Mlucas.
Mlucas does not have PRP proof file generation, yet.
Another way to go is a modest priced AMD GPU which could do the equivalent of 12-20 sticks of computing or more depending on models, in one unit to manage.
(For scale, 1 RadeonVII GPU can do about a wavefront PRP a day. An RX480 in ~5 days; RX550 ~18 days.)

F) Run some P-1 factoring. There's always a need for that at the wavefront. If I could do it with 1GB ram in a Windows laptop, it should be feasible in 1GB ram in Linux.

G) (Not recommended because it's very inefficient and less reliable)
Convert legitimate PRP first test assignments into bootleg LL first test,
settle for the 50% detection rate of errors that the Jacobi symbol check provides,
and stick the project with 102% or higher DC overhead.


Have fun!

Last fiddled with by kriesel on 2021-08-22 at 01:24
kriesel is offline   Reply With Quote
Old 2021-08-22, 00:52   #9
ssybesma
 
"Steve Sybesma"
May 2012
Brighton, CO USA

10110012 Posts
Default

Quote:
Originally Posted by paulunderwood View Post
If the server is set up with something like /nfs_share/stick1, /nfs_share/stick2 etc then stick machine 1 accesses its share which contains all it needs for running mprime including the executable,

I get that part.
I haven't seen how to setup where mprime looks.
I assume that prime.txt or local.txt would hold the path to the individual destination folder where it will keep its files.
That seems like a VERY clean option you gave me and that way the PRP temp files can be as big as they want.
I assume not all of them are being accessed constantly at once but I have excellent network speed and not worried about the host machine's resources.
ssybesma is offline   Reply With Quote
Old 2021-08-22, 00:55   #10
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

592410 Posts
Default

Quote:
Originally Posted by ssybesma View Post
I get that part.
I haven't seen how to setup where mprime looks.
./mprime -d
14
scroll through the choices, then at Advanced, say N don't skip

As a test I did this:
Code:
Your choice: 14

Consult readme.txt prior to changing any of these settings.

Temporary disk space limit in GB/worker (6.000000):
Upload bandwidth limit in Mbps (0.250000):
Upload large files time period start (00:00):
Upload large files time period end (24:00):
Download limit for certification work in MB/day (40):
Skip advanced resource settings (Y): n
Optional directory to hold large temporary files: ./largetmp
Optional directory to hold archived proofs: ./archivedproofs
Daytime P-1/ECM stage 2 memory in GB (0.300000): 1
Nighttime P-1/ECM stage 2 memory in GB (0.300000): 1
Max emergency memory in GB/worker (1.000000):
Priority -- 1 is highly recommended, see readme.txt (1):
Certification work limit in % of CPU time (10):
Use hyperthreading for trial factoring (recommended) (Y):
Use hyperthreading for PRP, LL, P-1, ECM (not recommended) (N):

Accept the answers above? (Y): y
and afterward found in local.txt:
Code:
ProofResiduesDir=./largetmp
ProofArchiveDir=./archivedproofs
I did not find those Dir entries referenced in readme.txt or undoc.txt. Maybe I just missed them.
That's for splitting up the mprime storage space.
If you want it all on a file share, which would be simpler but maybe not quite the performance a split setup may offer, set up the share, cd to the share, as in a shell script, run from there, & don't set separate archive, temporary, or working directories in the mprime local.txt or prime.txt.

Last fiddled with by kriesel on 2021-08-22 at 01:07
kriesel is offline   Reply With Quote
Old 2021-08-22, 00:59   #11
paulunderwood
 
paulunderwood's Avatar
 
Sep 2002
Database er0rr

75378 Posts
Default

Quote:
Originally Posted by ssybesma View Post
I get that part.
I haven't seen how to setup where mprime looks.
I assume that prime.txt or local.txt would hold the path to the individual destination folder where it will keep its files.
That seems like a VERY clean option you gave me and that way the PRP temp files can be as big as they want.
I assume not all of them are being accessed constantly at once but I have excellent network speed and not worried about the host machine's resources.
The share becomes part of the local file system for the stick. With this extension, everything needed to run mprime -- ini, local,, GMP, temp files, proofs mprime -- are all stored on the share. From the client stick you just see another directory. -- set up to share from the server.; It is mounted by the client -- tested with the mount command and automated by an entry in the /etc/fstab.

Last fiddled with by paulunderwood on 2021-08-22 at 01:16
paulunderwood is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
I run out of disk space - will this automatically now there's more space? drkirkby Information & Answers 5 2021-02-12 08:41
Efficient storage of all primes up to some n hansl Computer Science & Computational Number Theory 23 2019-05-12 21:11
How can I permanently deactivate/delete my mersenneforum account? Arkadiusz Forum Feedback 19 2016-03-11 22:35
Prime gaps and storage HellGauss Computer Science & Computational Number Theory 18 2015-11-16 14:21
A Lot Of Storage moo Hardware 0 2005-11-23 03:16

All times are UTC. The time now is 02:16.


Sun Dec 5 02:16:58 UTC 2021 up 134 days, 20:45, 0 users, load averages: 1.14, 1.19, 1.22

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.