New GPU Compute System

[airsquirrels]

Well I managed to snag 8 Titan Blacks and water blocks for just $2400 on eBay this morning, and I've got an empty SuperMicro 4027TR that has just been running CPUs on air. I might finally be going team green, even with much older hardware....

I really enjoyed building the AMD systems last year with Fury X cards - 8 Fury X cards put out about 8000 GhzDay/Day in TF and averages around 500 GhzDay/Day of LL. Unfortunately AMD's Polaris architecture really disappointed this year and even though NVIDIA Pascal is proving to be extremely performant it would take somewhere around $13,000 for 8x Titan X Pascals on water (650 GhzDay/Day) or $6-7000 for 8x 1080's on water (500 GhzDay/Day).

In both cases the old Titan Blacks actually perform nearly as well in LL, yielding around 700 GhzDay/Day at the 4096K FFT that I've been referencing. $2400 sounds like a great deal!

As must as I wish I could just throw down for 8x P100 PCIe whenever they come out, for > $30,000, I think I will stick to eBay.

[Mark Rose]

That is a good deal!

I'm tempted to pick up more GTX 580s for TF as they're getting really cheap now, but power consumption makes that an unfriendly proposition. I'd also need to build a new system to plug them into and find an electrical circuit with enough spare capacity.

How many GPUs do you have now?

[airsquirrels]

Originally Posted by Mark Rose

...and find an electrical circuit with enough spare capacity.?

That one always seems to be the challenge! Although we've got a larger solar array going in soon that should help with that one...

The number of cards ebbs and flows, I tend to lose about 1 card every other month or so from heat death, liquid/condensation issues, or just general failures. We have stabilized a lot of things so I'm hoping that number goes down. My graveyard currently has:
1x HD7990 - PCIe extender shorted, card smoked
4x Fury X (All lost in the same liquid leak - insurance covered)
3x Titan (infamous memory mosfet failure, technically repairable but I lack the SMD oven)
1x 295X2 - Stock cooler, raw heat death, just let out the smoke one day

We have 81 live GPUs + the 8 I just won and 1 Fury X on order, so that will make 90.

29 NVIDIA and 61 AMD.

[VictordeHolland]

You are truly the King of GPUS!

[LaurV]

Originally Posted by airsquirrels

3x Titan (infamous memory mosfet failure, technically repairable but I lack the SMD oven)

You can send them to me I have the knowledge and capacity to repair them, and can find a mobo and some intestines to put them to work...

[diep]

Originally Posted by airsquirrels

That one always seems to be the challenge! Although we've got a larger solar array going in soon that should help with that one...

The number of cards ebbs and flows, I tend to lose about 1 card every other month or so from heat death, liquid/condensation issues, or just general failures. We have stabilized a lot of things so I'm hoping that number goes down. My graveyard currently has:
1x HD7990 - PCIe extender shorted, card smoked
4x Fury X (All lost in the same liquid leak - insurance covered)
3x Titan (infamous memory mosfet failure, technically repairable but I lack the SMD oven)
1x 295X2 - Stock cooler, raw heat death, just let out the smoke one day

We have 81 live GPUs + the 8 I just won and 1 Fury X on order, so that will make 90.

29 NVIDIA and 61 AMD.

Cool man 8 cards Titan Black.

Your cards only get turned on when sun shines?

[airsquirrels]

Ohio is net-metering, so we run the cards 24/7 and get paid for excess power generation during the day, pay for power at night. It balances out to a net positive.

[chalsall]

Originally Posted by airsquirrels

Ohio is net-metering, so we run the cards 24/7 and get paid for excess power generation during the day, pay for power at night. It balances out to a net positive.

If I may please rant just a little bit about this...

The Generation, Transmission and Distribution (GTD) Industry have a bit of a legitimate complaint about this.

You generate power and feed it back into the grid when you can't use it at the same price you buy it for, but you don't pay for any of the infrastructure (cables, poles, generators, management of same). And, possibility, when you are generating power and can't use it others also can't use it.

Have you thought about storing excess energy on your own premises for your own use with a temporal delta?

The provider should not be assumed to be a capacitor to your variable production and demands.

[diep]

Originally Posted by airsquirrels

Ohio is net-metering, so we run the cards 24/7 and get paid for excess power generation during the day, pay for power at night. It balances out to a net positive.

Wow those politicians are FUBAR there as well - as buying in power from the net for those electricity companies during surges will soon be 100 dollar a megawatt hour, when the Sun isn't shining in november, whereas when there is surplus power it's worth less than 1 dollar a megawatthour, if you can sell it at all.

Over here the energy companies need to pay a fine when they generate wind or solar panel power when there is a surpluse on the market :)

In short energy companies basically build energy centrals solely for the surges there are on the net.

So this sort of subsidy can't last forever.

[diep]

Originally Posted by chalsall

If I may please rant just a little bit about this...

The Generation, Transmission and Distribution (GTD) Industry have a bit of a legitimate complaint about this.

You generate power and feed it back into the grid when you can't use it at the same price you buy it for, but you don't pay for any of the infrastructure (cables, poles, generators, management of same). And, possibility, when you are generating power and can't use it others also can't use it.

Have you thought about storing excess energy on your own premises for your own use with a temporal delta?

The provider should not be assumed to be a capacitor to your variable production and demands.

Yeah well i also have milling machines here and lathe. Those eat major power of course when used (and i don't even have a real good one that eats easily 30+ kilowatt). The entire industry has designed its machines such that they eat a constant amount of power. Only the peak power usage you have a year you pay for. Is that 7 megawatt? then you pay for 7 megawatt all year.

I'm always amazed by the high amounts of cash a megawatt hour government needs to pay for towards the energy companies - just because of so called energy savings - whereas it makes their bill 5x higher.

For companies eating lots of power, you get a constant megawatt for about 25 euro a megawatt hour now.

That's like 2.5 eurocent a kilowatthour?

On other hand for this household the first 10k kilowatthour we consume we pay a lot of taxes. Can easily go to 35 eurocent a kilowatthour because of the 25 eurocent taxes on top of it which hardly are there for industry of course.

[airsquirrels]

Originally Posted by chalsall

If I may please rant just a little bit about this

The provider should not be assumed to be a capacitor to your variable production and demands.

We actually still get billed the transmission fee. Our bill is split between metered kWh and generation and transmission.

In the short term it is cheaper to pay the transmission fee than invest in and maintain on-site batteries/inverters etc. I do actually have ambitions of setting that up and running DC systems, it just doesn't make anywhere close to economic sense right now.