Real easy cloud computing set-ups

[Uncwilly]

It would be great to have some real easy set-ups for cloud computing to help GIMPS. What I would like to see is something that hits the following. (And this thread might be the place to house these.) In a single clear post for each set-up we would have:

• Which cloud service is specifically called out.
• How to sign-up for the service, with links needed.
• Is there a Windows interface or only Linux?
• A link to an established ready repository (precompiled programs, directory structures, and scripts [and any needed bits of an OS for those not using the same OS]).
• Simple, clear, step by step instructions for newbs.
• How to link to a users PrimeNet ID (if possible) or GPU72 and how to have persistent CPU/GPU ID's (if possible) (such that work can be tracked and assignments followed up on.)
• How to save work and files either to cloud storage or to a local drive. (again scripts)
• Costs and how to get the needed hardware and the best value on the platform.
• Which different tests can I run with it? PRP, TF, P-1, LL(DC only), ECM, CF-PRP, etc.
• Other important factors.

This would help the more casual user to spin up a session or 3 to help do some targeted testing or throw a few dollars at the project when they have the funds.
[this post is subject to editing to add clarification, more bullet points, or links to such posts.]

[Chuck]

Originally Posted by Uncwilly

It would be great to have some real easy set-ups for cloud computing to help GIMPS. What I would like to see is something that hits the following. (And this thread might be the place to house these.) In a single clear post for each set-up we would have:...

Amen to that. I once started reading the how-to guide at the beginning of this topic and it is so mind-numbingly complex that I gave up.

[chalsall]

Originally Posted by Chuck

Amen to that. I once started reading the how-to guide at the beginning of this topic and it is so mind-numbingly complex...

That's what makes it fun!

[kriesel]

Agreed, good well organized and maintained documentation would be useful, especially if it's easily found.

Something that's an attempt at that (and in revision tonight) is the Google Colaboratory reference thread https://www.mersenneforum.org/showthread.php?t=24839
It uses a whole thread because there are so many GIMPS apps that might be run on it. Putting it into a single post would make it huge.

Volunteers with experience in or learning now the other cloud computing choices are needed.
Other possibilities listed in https://mersenneforum.org/showthread...t521922include

• Amazon EC2 https://www.mersenneforum.org/showthread.php?t=21444
• Amazon SageMaker (I have no idea if this is useful) https://aws.amazon.com/sagemaker/
• Google Compute Engine https://www.mersenneforum.org/showthread.php?t=21433
• IBM DataPlatform Notebooks. Watson Studio: default Python 3.6 free; 1 vcpu, 4GB ram https://dataplatform.cloud.ibm.com/d...ironments.html
• Kaggle initially ok'ed use for prime hunting, https://www.mersenneforum.org/showpo...&postcount=537 then started blocking accounts doing so, then clarified to chalsall they'd had a change of policy https://www.mersenneforum.org/showpo...&postcount=608 So I think the thing to do with Kaggle is ignore it going forward. So long, and thanks for all the cycles.
• Microsoft Azure (some free use) https://www.mersenneforum.org/showthread.php?t=21440
• Perhaps GPU Manufacturers' gpu test drive programs such as https://www.nvidia.com/en-us/data-ce...pu-test-drive/
• Paperspace (some free use) https://docs.paperspace.com/gradient...free-instances
• Lambda Labs https://mersenneforum.org/showthread.php?t=26525

Some moderators seem to forget eventually that ordinary forumites can't do some things moderators can:

• edit our own posts past one hour from initial posting (outside our own blog areas)
• send private message to more than 5 recipients in a single message
• edit polls after they're created
• (and probably more)

some of which makes maintaining useful info in an ordinary thread outside the blog area difficult to do. (I know they forget these things, because they sometimes tell me to do what is not possible; update old posts, and pm up to ten, for example.)
Moderators will sometimes perform edits ordinary forumites can't, especially the simple quick copy/paste type, past the hour deadline, if someone asks nicely, after doing the hard work of developing or refining a set of instructions or documentation.

[Uncwilly]

Originally Posted by kriesel

Agreed, good well organized and maintained documentation would be useful, especially if it's easily found.

Putting it as a sticky under Cloud Computing would make it easy to find when people look for it.

It uses a whole thread because there are so many GIMPS apps that might be run on it. Putting it into a single post would make it huge.

If there needs to be more than a single post per service, that could be handled. The first post in the thread would be an index. The mods would add the links to the particular posts. Something a kin to:

iCloud Computing:

Setting up mprime
Setting gpuowl

Some moderators seem to forget eventually that ordinary forumites can't do some things moderators can:

Quite the opposite, mods would be helpful keeping the thread up to date. Posting in other threads that accomplish the goal would be brought into the thread. Mike might even provide MersenneForurm·org urls and storage for the repositories.

• edit our own posts past one hour from initial posting (outside our own blog areas)

That would be a none issue as something like this would be happily maintained by mods.

At the end of each post would be a link(s) to any thread(s) that would be useful for further reading or where to ask questions. The posts in the thread would be enough for someone (who is not conversant in linux or a comp sci person, but can follow instructions) to get a service up and running. Then any extra things, questions, suggestions, or a little 'hand holding' (help with misunderstandings would be handled there. pdfs with screen shots (with circles and arrows) etc could be attached to the posts to make it even easier. The idea of being able to buy $25 or $50 of machine time in a month to do a special project (like getting the PRP-CF-DC caught up to the PRP-CF-DC, or dealing with exponents that need an LL-TC) would be great. That would allow someone to bring more firepower than they have on their normal machine to bear on a project, not buy another machine, or leave their normal machines doing their normal work.

[tdulcet]

Originally Posted by Uncwilly

It would be great to have some real easy set-ups for cloud computing to help GIMPS.

I wrote Linux install scripts for MPrime, Mlucas and CUDALucas, which should work on all cloud providers. They can all be run with a single command and will automatically download, build, setup and run the respective GIMPS program. After running the install script, users can monitor the program on the GIMPS website CPUs page and should never need to log into the cloud provider again.

Originally Posted by Uncwilly

• A link to an established ready repository (precompiled programs, directory structures, and scripts [and any needed bits of an OS for those not using the same OS]).

The scripts are all hosted at this GitHub repository: https://github.com/tdulcet/Distribut...puting-Scripts Pull requests are welcome!

Originally Posted by Uncwilly

• How to link to a users PrimeNet ID (if possible) or GPU72

The scripts all allow the user to specify their PrimeNet User ID. See the README for usage information (I do not want to post it here, in case it changes in the future). For the Mlucas install script, there are also detailed instructions on @ewmayer's Mlucas README.

Originally Posted by Uncwilly

• Costs and how to get the needed hardware and the best value on the platform.

The optimization/tuning is automatically done during the setup for the Mlucas and CUDALucas install scripts.

Originally Posted by Uncwilly

• Which different tests can I run with it? PRP, TF, P-1, LL(DC only), ECM, CF-PRP, etc.

The scripts support all the worktypes that are supported by the respective GIMPS program, although they are primarily for primality testing.

Here are links to my original posts about each script:

• Prime95/MPrime: https://www.mersenneforum.org/showth...112#post491112
• Mlucas: https://www.mersenneforum.org/showth...949#post545949
• CUDALucas: https://www.mersenneforum.org/showth...167#post490167

The MPrime and CUDALucas install scripts are used by @danc2 and I's Google Colab Notebooks for Primality Testing, which is what I would recommend for newbies since it only requires an internet connection and a free Google account (see step by step instructions here).

Google Cloud Platform provides one always free VM and Oracle Cloud provides two always free VMs (also see here). They are very slow and take roughly six months to do a wavefront first time primality test with MPrime, but if you sign up for both you can do about six per year for free. Many other cloud providers provide free trials or credits.