View Single Post
Old 2016-07-14, 21:21   #3
GP2
 
GP2's Avatar
 
Sep 2003

5·11·47 Posts
Default Using the Amazon EC2 cloud for Lucas-Lehmer testing

Regions and Availability zones

Amazon EC2 offers servers that are located in various different regions ("AWS regions") around the world, and within each region there are various "availability zones".

Before you can actually get started with Lucas-Lehmer testing, a number of preliminary configuration steps are necessary, and they need to be performed separately for each AWS region that you intend to use. Detailed instructions are given in the later sections of this guide.

The methods described here require a feature called Elastic File System (EFS). As of August 2017, EFS is only available in certain AWS regions so far: us-east-1 (N. Virginia), us-east-2 (Ohio), us-west-2 (Oregon), eu-central-1 (Frankfurt), eu-west-1 (Ireland), ap-southeast-2 (Sydney, Australia). Click here for an up-to-date list of AWS regions with EFS.

However, it makes no difference what part of the world you yourself live in, you can use servers in nearly all AWS regions around the world without restriction (the exceptions are regions in China, and a region reserved for US government customers).

Within each region, there are different "availability zones", which have a letter designation. For instance, the us-east-2 region has availability zones named us-east-2a, us-east-2b, us-east-2c.

Note: these letter designations are not the same for each user. What you see as "-2a" might be labeled "-2b" for another Amazon EC2 customer. It is thought that Amazon scrambles these names in order to prevent everyone from just choosing zone "a" by default, and thereby overcrowding some availability zones while others remain underutilized.

Prices

For the type of work we will be doing, we will incur hourly charges and use so-called "spot instances".

Like prices on a stock market, the "spot market" prices can fluctuate considerably, and are driven by market forces. These spot prices will vary (sometimes greatly) by region and by availability zone, and there are often very large discrepancies between different regions (or even different availability zones within the same region) that can persist for many months.

To understand this pricing discrepancy phenomenon (the lack of so-called "arbitrage"), remember that many Amazon cloud customers run websites and other online services on their servers, and need to locate them in specific geographical areas where the majority of their customers live in order to provide good response times (low latency). Also many countries have privacy regulations that require online service providers to store personal data about their citizens on servers that are geographically located within the country. Many online services also have strict uptime and availability requirements, and it is impractical for them to shut down every now and then to migrate to a cheaper server.

However, for the type of work we will be doing, we are free to select any region and availability zone, and migrate as often as we wish in search of the lowest spot prices.

Spot price information can be found on the Amazon EC2 Spot Instances Pricing page. You can use-the drop-down menu to view prices in the different regions: Ohio, N. Virginia, Oregon, Northern California, and the various regions in other countries around the world. Look at the column labeled Linux/UNIX Usage and scroll down to the section titled Compute Optimized - Current Generation, and in particular the line for "c4.large".

It is important to remember that the spot prices you see displayed on that page are valid only for the current moment, and can fluctuate at any time. If you are familiar with Google Compute Engine and its fixed prices for preemptible virtual machines, note that Amazon EC2 spot prices are not fixed but are determined by market forces.

As of June 2017, the us-east-2 (Ohio) region is consistently less expensive for our purposes than the other AWS regions, often by a wide margin, and this situation has persisted for many months, with relatively stable pricing which has remained fairly similar among the various availability zones. Therefore it is recommended that you choose this region.

However, because spot prices can, in principle, change very rapidly, and because EFS might become available in new AWS regions in the future, it is a good idea to keep an eye on new developments and price changes.

The old way

(This subsection is aimed at people who already have experience using EC2 for LL testing the "old way", before EFS was available. If you don't understand it, just skip this section)

With EFS, running mprime in Amazon EC2 is much more convenient than before. You can now pretty much "set it and forget it", just like you would with physical computers.

In the old way, or in regions that don't have EFS yet, you would launch a spot instance with two EBS volumes: one 8 GB delete-on-termination root volume with a standard operating system AMI such as Amazon Linux, and then an additional 1 GB do-not-delete-on-termination volume that contains the mprime work directory with the executable, the configuration files and worktodo and save files.

When a spot instance terminates, the 1 GB volume is "orphaned". A small amount of manual intervention is then required to recover its data: the orphaned volume has to be attached to some other instance and then mounted, and its worktodo and save files copied over. This gets annoying if it has to be done often on a regular basis (for instance, if you set an aggressively low limit price that gets hit frequently).

With EFS, there are no 1 GB volumes anymore. Rather, all the worktodo and save files for all the instances exist together on the EFS filesystem, as sibling subdirectories of one another. The EFS filesystem is permanent and is unaffected by the termination of any of the instances that use it.

When spot instances terminate and new ones launch, it is now easy for the new instances to locate and take over the work of the old terminated instances. No manual intervention or recovery is needed.

Prerequisites

Amazon EC2 offers server computers with either Windows or Linux pre-installed. However, the spot prices for Windows versions are typically four or five times more expensive. Therefore we will run Linux on our Amazon EC2 servers.

Some basic familiarity with the Linux command-line interface (the "shell" known as "bash") will be very helpful. Also, it will help if you are familiar with using ssh client programs to log into a Linux server remotely. If not, there is plenty of information about these topics on the Internet.

However, your own computer (the one that you will use to log into Amazon EC2) does not have to run Linux, it can run Windows or MacOS or anything else.

If you are familiar with Google Compute Engine, note that Amazon EC2 does not offer SSH in a browser window, so you will have to use an ssh client program. If you use Windows 10, the WSL (Windows Subsystem for Linux) has an ssh client program, or you can download free software. One popular program is PuTTY, which can be downloaded at http://www.chiark.greenend.org.uk/~sgtatham/putty/

Your AWS account

You can access your Amazon AWS account (or sign up for one) at https://aws.amazon.com/ . You can log in with the same username and password used for shopping at Amazon.com

You will need to provide a payment method (e.g. credit card) when you sign up for AWS.


Next section: Configure ssh for the default security group

Last fiddled with by GP2 on 2017-08-05 at 08:37
GP2 is offline   Reply With Quote