View Single Post
Old 2020-06-23, 19:33   #70
Mark Rose
 
Mark Rose's Avatar
 
"/X\(‘-‘)/X\"
Jan 2013
Ͳօɾօղէօ

532 Posts
Default

Regarding using AWS, I would implement something like this:

1. Have the PrimeNet server generate a signed URL for uploading to S3. Pass this to the client, and the client uploads straight to S3. Compose the URL with the exponent and some random number to allow multiple results for the same result (it will happen).
2. Have new upload events on S3 trigger a message via Simple Notification Service (SNS), which then publishes a message to a Simple Queue Service (SQS) queue.
3. Configure an EC2 Auto Scaling Group to scale based on the number of messages in the queue. Make it scale down aggressively to zero when there are no messages, but scale up slowly when there are many in the queue.
4. When the verifier running on the instance is finished, post a result to another SQS queue.
5. Use an AWS Lambda function to process events from that SQS queue and post them to PrimeNet. Decoupling this from the verifier allows PrimeNet to go down and not lose results or having expensive verifier instances kept running while waiting for work.

I expect much of the above could be implemented with simple python or JS outside of the verifier.

Uploads to S3 have free bandwidth, but cost $0.000005 per PUT request. So 10000 results per day would cost a couple dollars. Transferring to the EC2 instances is free.

Storage costs get expensive overtime. If each result is 100 MB, that's about 1 TB per day. A trivial amount of bandwidth to handle on AWS, but that will soon cost huge money: even using the Infrequent Access Tier of S3 Intelligent Tiering, each month would generate ~$400/month/month in expenses. Even cheaper alternatives like BackBlaze B2 would get expensive quickly. So holding on to the results long-term may not be feasible.

Last fiddled with by Mark Rose on 2020-06-23 at 19:44
Mark Rose is offline   Reply With Quote