George's dream build

[bgbeuning]

I had one of the ebay power connectors fail. (see 1st pic)
And redid my power (see 2nd pic)
Powering 9 boards with one VGA tap from the PSU is
pushing it a bit, and I need to reduce that.

[Mark Rose]

Originally Posted by Mark Rose

Whelp, I just ordered me a cluster: four i5-6600's with 32 GB DDR4-2133 each. I found Gigabyte GA-H100M-A motherboards for $30, so it wasn't worth getting the fancy ASRock boards at $92.50. I decided to up the CPU to make the cluster more useful for other applications and to improve resale value: nobody wants a low end chip (the i5-6600 is the same speed as a stock clock i5-6600K). Going with 32 GB was only 3 times the cost of 8 GB, plus I won't have to worry about getting rid of 4 GB sticks in the future. They're all going to be powered by a single EVGA 210-GQ-0650-V1 650W 80Plus Gold using ATX splitters. The splitters won't arrive until I'm back from holiday, so pictures will have to wait until after then. I already have all the needed networking stuff.

The total cost is about $2250 or about $1725 US. Had I got with i5-6400 and 8 GB, it would have been $1625 or $1250 US.

So the 32 GB RAM kits I bought for $120 are now retailing for $320. Yikes.

[henryzz]

Originally Posted by Mark Rose

So the 32 GB RAM kits I bought for $120 are now retailing for $320. Yikes.

The question is how much could you sell them for. If prices are going to continue rising it may be worth investing.

[Mysticial]

Originally Posted by henryzz

The question is how much could you sell them for. If prices are going to continue rising it may be worth investing.

I did something similar to that back in March when Ryzen launched. Because of the initial Ryzen memory issues, I ended up with an extra set of 8 x 16GB @ 3300 MHz G.Skill TridentZ's which I got on sale for only $800.

I decided to keep that 128GB kit in anticipation of a Skylake X build. Now that same set is listing on Newegg for $1440.

I wish I had gotten more back in March. Not to sell for a profit, but because there's a real possibility that I might be needing a second 128GB system later this year. But with ram prices through the roof, that's going to be tough.

[henryzz]

Originally Posted by Mysticial

I did something similar to that back in March when Ryzen launched. Because of the initial Ryzen memory issues, I ended up with an extra set of 8 x 16GB @ 3300 MHz G.Skill TridentZ's which I got on sale for only $800.

I decided to keep that 128GB kit in anticipation of a Skylake X build. Now that same set is listing on Newegg for $1440.

I wish I had gotten more back in March. Not to sell for a profit, but because there's a real possibility that I might be needing a second 128GB system later this year. But with ram prices through the roof, that's going to be tough.

Is this temporary due to shortages or going to continue long term?

[Mysticial]

Originally Posted by henryzz

Is this temporary due to shortages or going to continue long term?

I'd say give it a least another year before things start to look better. But don't hold your breathe though.

What we're seeing now with ram is very similar to what happened to hard drives following the 2009 Thailand floods.

It starts with a shortage:

• In 2009, the floods knocked out a large portion of the hard drive supply chain.
• In late 2016 for ram, increased demand from smart phones and a shift of manufacturing capacity to SSDs caused supply to drop.

The shortage leads to extreme price hikes:

• Hard drives went up by 2-3x.
• Right now, we're nearing that 3x point for low-end memory and 2x for high-end memory.

The suppliers realize that the demand is price-insensitive. So they make no effort to ramp up supply and let the revenue flow in. This is usually when price-fixing happens. And suppliers "make up" reasons for not being able to increase supply in order to keep the feds off of them.

After a few years, market dynamics take over again and prices drop back to normal. I don't see this happening any time soon for DRAM. So I wouldn't be surprised if prices stay high like this for another 2 - 3 years. It took hard drives a good 5 years to recover. And even then, Moore's Law for GB/$ has stopped for some 3 years now.

(Though when I worked at Google a few years back, they were "bragging" about how Google eats up the majority of the world's hard drive supply because of YouTube. That might have had something to do with it. And I sure as hell didn't find as funny as everyone else there.)


EDIT:

That hard drive shortage back in 2009 hit me particularly hard as I'm a big customer of them. I had about 30 2TB drives made before the floods which I ended up using for almost 8 years because I couldn't replace them until last year. By then, enough of them had failed and/or degraded that I couldn't wait much longer. And prices were low again.

I have a bad feeling about this ram situation since I need a lot of it. Most of my builds for the past 10 years have had maxed out ram configurations.

[Mark Rose]

For anyone building a cluster, I encourage you to look at the upcoming i3-8100 processor paired with dual channel DDR4-2400 memory on the cheapest motherboard as the sweet spot. The processor should retail for US$117.

[petrw1]

Originally Posted by Mark Rose

For anyone building a cluster, I encourage you to look at the upcoming i3-8100 processor paired with dual channel DDR4-2400 memory on the cheapest motherboard as the sweet spot. The processor should retail for US$117.

What about the i5-8400?
2 more cores and supports DDR4-2666 Memory for only $70 more.

Interesting thought that the CPU is so much slower....to the point that Ghz X Cores is not a lot more than the i3 you mentioned.

[Mark Rose]

Originally Posted by petrw1

What about the i5-8400?
2 more cores and supports DDR4-2666 Memory for only $70 more.

Interesting thought that the CPU is so much slower....to the point that Ghz X Cores is not a lot more than the i3 you mentioned.

The sweet spot seems to be roughly:

fma3/clock x GHz * cores * 333 MHz = MHz channels of DDR4 required.

So Intel with fma3/clock * 3.6 * 4 * 333 = 4800 MHz, or 2400 MHz dual channel.

But the 6 core chips would need 7200 MHz or 3600 MHz dual channel.

The 8 core chips at 3.6 GHz with quad channel 2400 MHz DDR4 are also balanced, but the price is higher than two of the proposed i3 systems.


With Ryzen, the chips have half speed FMA3, so the memory requirements are much less:

Ryzen 1600X: .5 * 4.0 * 6 * 333 = 3996 MHz, or two channel DDR4-2000/2133.

Ryzen 1700: .5 * 3.7 * 8 * 333 = 4928 MHz, or two channel DDR4-2400/2666.

Ryzen 1800X: .5 * 4.0 * 8 * 333 = 5328 MHz, or two channel DDR4-2666/2933.

But the Ryzen systems will produce much less throughput per dollar, and turn more electricity into heat per work done.

[preda]

Originally Posted by Mark Rose

The sweet spot seems to be roughly:

fma3/clock x GHz * cores * 333 MHz = MHz channels of DDR4 required.

How is the "sweet spot" affected by running a single test on all cores vs. one test per core?

[Mark Rose]

Originally Posted by preda

How is the "sweet spot" affected by running a single test on all cores vs. one test per core?

For Skylake and newer, a single worker is a few percent better at current exponents with 4 cores. I'm not sure to how many cores where a single worker stays better. For Haswell and earlier, 1 worker per core is better.

But in either case, the memory bandwidth requirements are basically the same.