How-to guide for running LL tests on the Amazon EC2 cloud

[Cheetahgod]

Yeah I'm using marine. Maybe I didn't set it up right. Has anyone used the latest version on aws?

[Cheetahgod]

I got it working. I should do PRP-WR tests right?

[Uncwilly]

Sure, all first time checks are in the World Record range right now.

[storm5510]

I have had an AWS account for a while. I have not done anything with it. Using Ubuntu with mprime in the past few months made it a bit clearer in some areas. I have been doing P-1 work for James Heinrich. This means manual reserve and submit. I do not know if this is possible in this instance. If files can be pasted and copied then this would be different.

What I am looking at now is in the attached image. It is the EC2 dashboard. I really do not know where to go from here.

[chalsall]

Originally Posted by storm5510

What I am looking at now is in the attached image. It is the EC2 dashboard. I really do not know where to go from here.

Create an SSH key-pair. Spin up an instance. SSH into the instance. Profit!

[storm5510]

Originally Posted by chalsall

Create an SSH key-pair. Spin up an instance. SSH into the instance. Profit!

I now have one running instance, one volume, one key pair, and two security groups. "SSH into the instant." I take it this would be "Connect." I gave the instance a name. Connect presents three options. I chose the third. It refused to connect so I stopped the instance.

I think I will let this go. It all looks great, but sails over this old head.

[chalsall]

Originally Posted by storm5510

I think I will let this go. It all looks great, but sails over this old head.

Never give up! You human; it deterministic computer. You're smarter than it is; prove it!

(I said this to someone else last year while they were making friends with Colab. They replied "Yeah, but I think it's more patient..." )

[storm5510]

Originally Posted by chalsall

Never give up! You human; it deterministic computer. You're smarter than it is; prove it!

(I said this to someone else last year while they were making friends with Colab. They replied "Yeah, but I think it's more patient..." )

I've never been one go give up on something like this. I'll mess with it until I figure it out.

[keisentraut]

I think the EFS/EBS method in this guide is not optimal for Spot instance because you have fixed costs if you stop for a long time when spot prices are high.

1. I have a S3 bucket, where mprime, the configuration and its savefiles are stored. The monthy costs for this will be at most 1ct, because it's a few hundered MBs only and S3 is cheap.
2. I use the userdata script to run a bash script which copies the mprime folder from S3, then runs mprime in it and watches for the termination notice.
3. If the 2min termination notice triggers (or some maximum runtime is done), the script will immediately stop mprime, wait 10 seconds, copy back everything to the S3 bucket and shutdown the machine.
4. In the future, the userdata script will also immediately re-file a new spot instance request with a very low price, but for now it stops and will not continue.
5. Assignments are done by PrimeNet, when one is finished, it will simply get a new one.

This way, my EC2 spot instances do not need any persistent EFS/EBS storage and I can set it to the shutdown behaviour to terminate. I tested that it works and plan to write a guide after I am finished with an actual candidate.

What I'm currently need help with is

• what the settings in prime.txt and local.txt do I need for optimal performance? I don't understand which number of workers and affinity masks I will need for a c5.large instances, only found it for the older c4. Or is it still more efficient to run c4?
• Also, do I need to run the time-consuming "Jacobi-check" on every restart of mprime? Can I disable this or should I leave it as is? Overhead is probably negligible anyway as long as the average runtime of a spot instance is at least half an hour.

[drkirkby]

I thought about this sentance


"NOTE: if your IP address ever changes in the future, you will need to repeat this step. Otherwise your ssh login attempts will time out and fail."

Generally speaking, if one has a dynamic IP, it will not change a huge amount. The first couple of numbers are highly likely to remain fixed. My current IP is 213.205.240.207 I can be fairly confident, it will remain 213.205.xxx.yyy, so I can set the range of IPs allows to SSH to the Amazon instance to be 213.205.0.0/16. That allows access from 256^2 -2= 65534 IP addresses, but not the whole world. To my mind that offers a reasonable compromise between the poor security of letting anyone in the world try to hack it, and the inconvenience of having to change the IP on the Amazon website every time your service provider gives you a new address.

Certainly with EE, who I use, I've never had any problems with this, as they always seem to give me some IP address in the range 213.205.0.1 to 213.205.255.254.

Dave

[kriesel]

Originally Posted by chalsall

They replied "Yeah, but I think it's more patient..."

The silicon its can be very patient 24/7/365. It is easier for them, with no emotions, no sense of evenutal unavoidable decline and mortality. They can be very forgiving too, none of that geez, user still doesn't get me, I'm so sick of being mistreated with wrong commands, etc. That inexhaustible forgivingness is one of my favorite features of computers. Once humans add enough AI to the OS though, that may turn for the worse.
It may be almost infinitely patient. But we can be sufficiently stubborn / determined to reach our suitably chosen goals with it.