How about using Amazon's hardware instead?

[GP2]

Originally Posted by Mark Rose

The processor has a design power of 135 watts. Throw in some memory and chipset watts and powersupply inefficiency and you're probably close to 175 watts per processor.

Hmmm....

I found the Xeon E5-2690 v4 info here but wasn't sure about the overhead. OK, let's go with 175 watts...

Assuming an average price of $0.125 per kW·h for electricity (see this table for individual US states), this means you'd be spending:

$0.125  / kW·h    ×   0.175 kW    =   $0.022  per hour per processor

With Amazon EC2, among the various availability zones the cheapest spot price for a c4.large is about $0.018 per hour. That is 1 core out of an 18-core Xeon E5-2666 v3 (Haswell), which seems to be a custom processor made specifically for Amazon (some specs here and here).

Maybe I'm comparing apples to oranges, or maybe the math is off, and I certainly don't want to sound like an Amazon shill, but it seems that if you're willing to put up with the no-warning interruptibility that goes with spot pricing, you can get cloud computing for barely more than the cost of electricity alone (which Amazon pays for). And no expense for extra air-conditioning, no upfront expense to actually buy any hardware, automatic "upgrade" whenever Amazon introduces a new generation of instance types, flexibility to ramp up or scale down at any time. Some say Google Cloud Platform is even cheaper, I haven't properly investigated it yet. And this article says that cloud instance prices have been dropping by an average of 13% year over year since 2007.

And electricity is even more expensive than average in some states: New England = almost 20 cents per kilowatt-hour, New York State and California about 17 cents.

At the moment I have only one computer at home running GIMPS and everything else is in the cloud on a per-hour basis. It's a bit scary to watch the daily billing add up, but at least it's transparent instead of being hidden in a higher monthly utility bill. I'm starting to buy into the idea that this is actually how computing will be done going forward. No more corporate data centers, no more home server farms. The cloud oligopoly players like Amazon, Google and Microsoft have economies of scale and probably tax breaks and other sweetheart deals that the average person or even fairly large companies just can't compete with. And you're probably pretty much guaranteed to get no bad LL results, because corporations doing their mission-critical computer modeling and simulation are starting to rely on this more and more.

Those of you running home server farms, what benefits can you highlight? Are you able to get low power consumption and high throughput with barebones no-frills setups? Maybe I'm missing some part of the picture...

[retina]

Originally Posted by GP2

Those of you running home server farms, what benefits can you highlight? Are you able to get low power consumption and high throughput with barebones no-frills setups? Maybe I'm missing some part of the picture...

I don't think it is about benefits. More likely it is about control, bait-and-switch and loss-leader things. It also relies upon the continuing existence of the company you decide to trust with your computing. What if the company is badly managed and goes bankrupt? What if they decide to start charging more after everyone has finally ditched their own computers? What if your data/code is sensitive and/or confidential? What if there are no more cheap spots available because the platform becomes saturated? What if rogue employees decide to "steal" any prime results for themselves?

[Mark Rose]

It's still cheaper in the long run for me at $0.15/kWh to buy and run computers at home versus Amazon if my time-frame is long.

Where Amazon is cheaper is if you have to pay for all the other stuff, like networking, redundancy, technicians, etc. If your requirements change, you're not stuck with ill-fitting hardware. And you can get results faster if your task is distributable. Additionally, there's no capital tied up, so if you're trying to avoid shareholder dilution when raising money, the less you spend up front, the better.

Amazon prices do indeed drop by about 13% a year. My own analysis shows it's actually not worth getting three-year reservations for EC2, because the frequent price drops and improving hardware performance means a one-year reservation saves nearly as much.

Amazon Web Services is very well managed. It does have different failure modes than a traditional data center, and it forces you to think a little more abstractly about computing, but it results in extra redundancy and uptime if you follow the paradigms. It's also helpful that the second large ecommerce site in the world uses the same tools, so they tend to work very well. There can be some vendor lock-in if you go that way, but it's easy to avoid tightly coupling to anything AWS specific if you're smart about it. The tradeoff is development time versus time-to-market.

[henryzz]

After having a quick look around the price is around $0.16/kWh via British Gas. By shopping around it may be reducable to around $0.12/kWh in the UK. I know some reduce the price as you use more. I would guess we would benefit from that.

[fivemack]

Originally Posted by GP2

With Amazon EC2, among the various availability zones the cheapest spot price for a c4.large is about $0.018 per hour. That is 1 core out of an 18-core Xeon E5-2666 v3 (Haswell), which seems to be a custom processor made specifically for Amazon. Maybe I'm comparing apples to oranges, or maybe the math is off

So, for slightly less than the price of the electricity you need to run 14 cores of your own, you can get Amazon to run one core for you. This really does not sound a stellar deal.

[GP2]

Originally Posted by fivemack

So, for slightly less than the price of the electricity you need to run 14 cores of your own, you can get Amazon to run one core for you. This really does not sound a stellar deal.

Wait, could you run all 14 cores on a total of 175 watts, with all of them doing non-stop LL testing?? When it was quoted as 175 watts per processor, I assumed that meant per core...

[axn]

Originally Posted by GP2

Wait, could you run all 14 cores on a total of 175 watts

Yes. TDP is quoted for CPU, not for individual cores.

[GP2]

Originally Posted by axn

Yes. TDP is quoted for CPU, not for individual cores.

Well, I'm pretty clueless about hardware, that's why I try not to operate my own.

Still, I think this processor must sell for a couple thousand brand new, for the CPU alone. Then you surely need pricey top-notch memory so that mprime isn't memory-bandwidth-bound, plus the kind of motherboard that supports it. Probably still cheaper in the long run, although that sustained 13% reduction year-on-year trendline for cloud instance prices does look promising.

[fivemack]

But if you're comparing the cost of Amazon equipment to the cost of operating your own, the relevant cost is of the equipment you'd actually buy if operating your own - i7/4790K boxes at $800 each and using probably 120W for about the performance of eight of Amazon's cores.

It seems a basic principle of capitalism that paying someone else to operate a machine for you will cost more than operating it yourself; you might still want to pay someone else if you can't afford the machine or if you wouldn't be using all of its capabilities 24/365 - I use a machine-shop rather than owning my own CNC lathe, I hire someone to shampoo my carpet rather than owning a carpet-shampooer - but both of those are false for running mprime on your own computer in your own house.

[GP2]

Originally Posted by fivemack

It seems a basic principle of capitalism that paying someone else to operate a machine for you will cost more than operating it yourself

Actually, it's a basic principle of capitalism that if you buy in bulk you can usually get a better price, and if you buy in truly massive quantities and operate on an immense scale you can get huge economies of scale and squeeze out the competition. I recall that owners of small corner stores would often acquire their inventory by literally shopping at their local Walmart, because Walmart would sell stuff at retail for less than the price the small store owners could obtain it from distributors.

In this case, Amazon has bought enough processors that they got Intel to make a special version of a CPU optimized according to Amazon's specifications. They probably get good bulk rates for electricity, they probably get tax breaks from states and counties anxious to create high-tech jobs, they probably have some kind of custom designed cooling system or even a custom designed building that cools entire gigantic racks for the lowest possible cost per server, and so forth.

It would not be at all paradoxical if the oligopoly cloud companies (Amazon, Microsoft and Google) eventually achieve sufficient economies of scale that they could provide computing to individuals for less than it costs you to do it yourself, and still turn a profit. Indeed, I think that's the endgame that's already coming into view, although we're not there yet; however, for corporations with the additional overhead of operating their own data center buildings and a highly-paid IT staff, the tipping point is a lot closer. In general I think mobile computing and desktop computing will take different paths; your smartphone will keep getting smarter but your desktop might migrate into the cloud.

[Madpoo]

Originally Posted by GP2

How many watts does one of these use when it's doing LL tests at full capacity? Not including the additional air-conditioning burden, which may be hard to calculate directly.

Well... that's a good question and my answer may seem glib: I don't care.

I mean, I do, but our setup involves colocating where we pay per cabinet, power, network. The cooling is facility wide and for business reasons we spec out the power by looking at the peak usage of all equipment and then making sure it can run entirely on one circuit (there are 2 power circuits though, for redundancy... all the equipment has dual power supplies going to each of the PDUs).

On a fully loaded server with a bunch of RAM and anywhere from 8-16 hard drives, typical power might be in the 300-400W range and will go to 450-500W when Prime95 is running. I think that's about right. Let me check one real quick...

Yeah, on the dual E5-2697v3 it's sucking 505W with Prime95 going. Prime95 stopped it's at 348W.

It has a pair of 800W supplies which is perfect because it needs to be able to handle a full load on just one PSU (N+1 redundancy).