Help needed - needing a Nvidia Tesla K80 in a Dell 7920 tower workstation

[paulunderwood]

Originally Posted by tServo

I believe you may have 2 problems: power and cooling. It's hard to say which is the culprit.

(1) cooling: I agree with frmky that these cards in a 7920 will need some sort of additional cooling. The 7920 box does NOT have sufficient airflow to cool these red hot beasts.
They were designed to be put in a "pizza box" server that has about 20 high power fans that scream so loud it sounds like a jet taking off. Years ago, I had a friend that put one of these in a 7910 ( I think ) and he cobbled a VERY high speed fan to do the job. The fan fit by the exhaust slots and pulled the air out. I can't remember how he attached it to the machine, he may have used duct tape ( The handyman's secret weapon ).
You can search ebay for "tesla k80 fan" for some guys who use printers to make fans that attach to the input side of the card.

(2) the cards need a very special cable that takes 2 standard pcie 8 pin power cables and output a special 8 pin cable. I would try using 2 different pcie power cables ( not just 1 cable split into 2 ). search ebay for "K80 power cable".

Good luck!

[drkirkby]

Originally Posted by Xyzzy

There is no way that 300W can be drawn from the PCIe slot. You need at least an 8 pin PCIe connector. What connectors are on the card? The PDF you linked shows a single 6 pin cord.

Note the PCIe requirement here: https://www.techpowerup.com/gpu-specs/tesla-k80.c2616

Pictures of your card and PSU would help a bunch!

Someone gave me a technical reference manual on this workstation. It's on another computer which I can't access now, but that should have the specification. But this video, which is a review of the workstation, says 7 minutes and 50 seconds in that it supports 3 x 300 W or 2 x 375 W GPUs


https://www.youtube.com/watch?v=jP65i_Iqml8


I will add some pictures later.

[tServo]

Originally Posted by drkirkby

Someone gave me a technical reference manual on this workstation. It's on another computer which I can't access now, but that should have the specification. But this video, which is a review of the workstation, says 7 minutes and 50 seconds in that it supports 3 x 300 W or 2 x 375 W GPUs


https://www.youtube.com/watch?v=jP65i_Iqml8


I will add some pictures later.

I believe Xyzzy's point was ( correct me if I'm wrong, Mike ) that the power cannot come solely via the card's pcie bus, you must also attach pcie power connectors to get that much power. I believe the pcie spec says the pcie bus can only supply 75 watts. The rest must come fron the pcie power connectors.

[drkirkby]

I did have another power cable connected - I will take a photograph of the card and connector tomorrow, but at the moment I don't want to shut the machine down, but unfortunately opening the case powers it off.

Dave

[Xyzzy]

Originally Posted by drkirkby

I did have another power cable connected - I will take a photograph of the card and connector tomorrow, but at the moment I don't want to shut the machine down, but unfortunately opening the case powers it off.

There is a switch somewhere in your computer called a "chassis intrusion" switch. You can permanently jump that switch so you can open the computer while it is running.

[paulunderwood]

Originally Posted by drkirkby

Someone gave me a technical reference manual on this workstation. It's on another computer which I can't access now, but that should have the specification. But this video, which is a review of the workstation, says 7 minutes and 50 seconds in that it supports 3 x 300 W or 2 x 375 W GPUs


https://www.youtube.com/watch?v=jP65i_Iqml8


I will add some pictures later.

The k80 has a non-standard power socket. This is why we strongly recommend a power feed from two PCI-e to K80 (special) which can get from eBay for a fiver. Search: "K80 power cable"..

[Xyzzy]

But he only has three 6-pin cables, right?



Edit: He has at least two 8-pin cables.

[kriesel]

Cooling: most Tesla cards are passively cooled, designed for modest volume higher pressure air flowing lengthwise (high impedance), as in rack mount servers that use low volume high pressure fans to force air from rack front (cold aisle) to back (hot aisle).
That's a completely different fan design than the usual consumer gpu or home or office system case low pressure high volume fan (for low impedance), more commonly called a blower, typically a squirrel cage centrifugal fan. See the discussion about Xeon Phi including coprocessor cards, which are also MOSTLY designed for passively cooling. An exception with builtin blower can be seen here. It takes a squirrel-cage blower style fan to produce the much higher air pressure to cool a GPU lengthwise than the axial-flow low pressure fans that are common in desktop or tower workstation or consumer cases or consumer grade gpus that have fans on the big face. (The difference between a car radiator fan or window fan, and a vacuum cleaner or furnace or leaf blower.)
Little of the above should be a surprise to the OP. Sent to him in a PM 2021-01-11:
"How do you plan to cool the K80? Does your workstation case have the high pressure fans required?"

All graphics cards shown in the video link given in post 13 include built-in fans, not passive cooling design.

Note, there's a cottage industry 3d printing adapters to passive cooled cards. Some appear to include a low pressure fan not suitable for the task. This one appears to use a proper blower but handicaps it with a sharp 180 degree U-turn which will create a lot of added flow resistance.

I bought a used workstation some time ago that came with a Tesla C2075 and Quadro 4000 that coexisted. That Tesla was also an actively cooled GPU with blower style fan built in. (It's gone to recycling long ago after first making the box unstable, then unbootable, and failing testing on another system also.)

Power connections: PCIe slots are specified to provide no more than 75 watts max.
A standard six pin auxiliary plug that connects separately to a PCIe module (such as a GPU or other card) has 75 watt max rating. A standard eight pin auxiliary plug that connects separately to a PCIe module has 150 watt max rating.

These are additive: PCIe 75 + 6-aux 75 +8-aux 150 =300 for example.

Some GPUs use:
PCIe power only. Low power cards such as Quadro 2000 or RX550.
PCIe + 6-aux; 150W max
PCIe + pair of 6; 225W max
PCIe + 6 and 8; 300W max
PCIe + pair of 8; 375W max. Some 300W nominal GPUs also use this approach.

RTX3xxxx introduced a higher density and power rating 12-contact aux connector.
Apparently some Teslas used nonstandard power connection.

It's quite common for the higher power GPUs (Radeon VII, GTX1080, RTX2080 etc) to have multiple aux power connectors, and refuse to operate when a single connection is made.
It's quite common for large-capacity PSU GPU Aux power cables to have multiple connectors per cable on the GPU end; (6+2)x2 is common. One cable, 4 connectors usable in various combinations; 6, 8, dual-6, 8+6, dual-8, on one GPU, or split for 2 adjacent modest-power gpus.

Power supply capacity is probably adequate, depending on other components present.
A KillaWatt line power input meter or equivalent could be useful to check the typical draw, although that would not likely identify short sub-second peaks in power demand that exceed the PSU rating. Such meters are available for free loan from some public libraries. They're also good to own and don't cost much compared to any decent CPU or GPU.

A low end GPU with display output perhaps temporarily installed could be useful to observe what is going on during the BIOS startup and any beginnings of OS load, before an SSH server is up.

Recording boot displays with a digital camera can be helpful, allowing later review of a message that flashes on just before it all goes dark.

Drivers: If I recall correctly, from reading the NVIDIA site long ago, and one previous Tesla experience, Teslas can be run either with a driver that also works with Quadros with display capability, or with a special compute-oriented driver. https://docs.nvidia.com/gameworks/co...te_cluster.htm
Note, NVIDIA allows one NVIDIA GPU driver per system. The version must be compatible with all NVIDIA GPU models you hope to use simultaneously. I've had to move GPUs to other systems to arrange the simultaneous solution to that.

[drkirkby]

Originally Posted by kriesel

Cooling: most Tesla cards are passively cooled, designed for modest volume higher pressure air flowing lengthwise (high impedance), as in rack mount servers that use low volume high pressure fans to force air from rack front (cold aisle) to back (hot aisle).
That's a completely different fan design than the usual consumer gpu or home or office system case low pressure high volume fan (for low impedance), more commonly called a blower, typically a squirrel cage centrifugal fan.

Yes, maybe cooling is the problem. The fans themselves must be able to supply enough air, I accept at a lower pressure. The design of the workstation version of the Dell 7920 tower workstation is very much like a consumer PC in appearance, although not in internal design - for example, it only takes ECC RAM. There is a 2U rackmount version of the Dell 7920, which is like a typcal server, although its called a workstation.

I'm attaching a copy of the Dell 7920 Technical reference. I'm not aware of anywhere else this document can be found. It will have information about the connectors in it. (I had to use an online compression tool, as the original size was over 4 MB, which is the limit of the forum).

I will try to upload some photographs later. I have had problems on some forums (I think including this one), where the file size is very limited. I'm not sure if this is the forum, but I tried compressing a jpeg both with my iphone and Gimp on a PC, and the forum said the file format was not valid. I maybe thinking of another forum. I will try to attach some pictures later, but for now here's the technical reference, which is better than the users guide. For example, the users guide only shows how to configure the machine with identical RAM modules, but this shows that RAM modules of different sizes can be mixed, and how to mix them.

[chalsall]

Originally Posted by drkirkby

The design of the workstation version of the Dell 7920 tower workstation is very much like a consumer PC in appearance, although not in internal design - for example, it only takes ECC RAM. There is a 2U rackmount version of the Dell 7920, which is like a typcal server, although its called a workstation.

To share... While I ***love*** Dell rack-mount kit, I hate their workstations. That also goes for HP, Lenovo, et al.

As you mentioned, nothing is readily Commerical Off-The-Shelf (COTS). Not even the PSU!

A client of mine recently had their Lenovo-based accounting workstation (read: mission-critical) fail. It was most likely the PSU, but a replacement would have had to be ordered (and, being Bimshire, imported with a temporal latency of about a month) at a cost of only slightly less than a full new build from commodity parts.

I had them back up and running by building a brand new machine (migrating the HD) in a day. The Lenovo was taken to a local recycler who does e-waste properly.

[jawesker]

Hi, did you get any walkaround for this problem? I have almost the same problem. I own a dell t7820 tower station and recently bought a tesla card k80, when I plug it in with proper ventilation and with the proper power supply the workstation starts but no video and stays like that just doing nothing. I checked and the indicator LEDs in the tesla card are on. After a couple of minues I turn the system off and I could feel that the tesla card was a bit more than warm so I think that the power supply is doing his job. Any one have some experience with this card?

Kind regards