Intel Xeon PHI? - Page 14

kriesel · 2020-12-02, 03:11

https://en.wikipedia.org/wiki/MCDRAM It's not just on the board, it's in the same package as the processor cores. Like modern caches. https://en.wikipedia.org/wiki/Xeon_P...QHL6_DSCx3.jpg

ewmayer · 2020-12-02, 03:38

Quote:

Originally Posted by kriesel

https://en.wikipedia.org/wiki/MCDRAM It's not just on the board, it's in the same package as the processor cores. Like modern caches. https://en.wikipedia.org/wiki/Xeon_P...QHL6_DSCx3.jpg

Yes, I took 'onboard' to mean in the CPU package proper, because for it to be used to imply 'on the (mother)board' would be inane - as opposed to 'onshore'? - but apparently the mobo manual's usage means just that. AFAICT the MCDRAM frequency is not fiddlable, unless it's somehow tied to the CPU frequency. Anyhow, found this doc from Colfax Research - the folks who made the GIMPS KNL workstation model - re. the 3 usage modes (I've added numbering for clarity):

Quote:

Advantages and disadvantages of the three modes are:

[1] Cache mode — No work required to use, but may have lower performance than flat mode in case of frequent misses in HBM as cache.
[2] Flat mode — May offer better performance than cache mode, but requires modifications of the code and/or execution environment.
[3] Hybrid mode — Benefit of both Flat mode and Cache mode, but smaller sizes for each.

Since I have 0 RAM aside from the MCDRAM, HBM-as-cache misses are not in play, thus choosing any usage mode other than [1] would be silly: if 0 cache misses in mode [1], then [2] offers no performance benefit and requires both the prorietary Intel toolchain and major recode, and [3] offers the worst of both worlds: smaller L3 cache than 16GB, and needing recode.

VBCurtis · 2020-12-02, 06:01

I interpreted that Colfax document as explaining that cache mode would really be a cache, and not seen by the OS as main memory; while flat would treat the on-package RAM as main memory but a separate space from the DDR4 modules- it's the "two main memories" that I thought would require special coding.
In your case, with no DDR to code for, I expected flat to be the choice.

Interesting that "cache" mode is still seen by the OS as main memory.

kriesel · 2020-12-02, 20:09

Quote:

Originally Posted by VBCurtis

I interpreted that Colfax document as explaining that cache mode would really be a cache, and not seen by the OS as main memory; while flat would treat the on-package RAM as main memory but a separate space from the DDR4 module.

I concur after a light reading of it. Somehow the BIOS, Windows 10 and prime95 knew what to do with the MCDRAM in the absence of any DIMMS in my system, and use it as addressable memory. Sounds like a 1-legged flat mode imposed by the OS to me. Windows and Linux and prime95 and presumably Mlucas have been dealing with multiple NUMA banks for a while now. Looking at both source codes for how dual processor package systems are handled might be educational.

ewmayer · 2020-12-02, 20:33

Just did a reboot and had a look at the CPU Configuration menu in the BIOs:

o CPU freq = 1.4GHz and not BIOS-adjustable (nor the min.max ratios), L1 = 4352KB (4.25MB), L2 = 34816KB (34MB),

o HT is enabled by default, since I found no gain from using it in my Mlucas timing tests, disabled it to see if that makes a difference for my current 1-thread-per-core running

Under the Advanced Power Management Configuration Power Technology menu:

o Config TDP submenus not shown

o EIST (P-states): "EIST (Enhanced Intel SpeedStep Technology) allows the system to automatically adjust processor voltage and core frequency to reduce power consumption and heat dissipation. The options are Disable and *Enable*" -- basically, a form of auto-throttling, I disabled, which should be safe to do with my case in side-panel-removed form with the resulting low CPU temps (I ordered the baking sheet as my first side-opening mesh-covering solution attempt, btw). Note the sidebar infopane adds "When disabled, CPU frequency is set at max non-turbo", which means the aforementioned 1.4GHz. But "non-Turbo" implies there might be an available "turbo" mode - that might refer to a particular version of the CPU which has the clock unlocked, though.

o CPU C State Control -- see nothing of interest there

o CPU T State Control (Available when Power Technology is set to Custom): has a single submenu, ACPI (Advanced Configuration Power Interface) T-States: "Select Enable to support CPU throttling by the operating system to reduce power consumption. The options are *Enable* and Disable" -- switched to Disable, saved changes, booted the OS, fired up big 64M-FFT F30 job on 64 cores, wait for next ~10-min checkpoint update ... and no discernible speed change. So unless someone else has any bright ideas, looks like we are more or less maxed out already at the default settings.

kriesel · 2020-12-02, 20:40

I did not mess with the BIOS in regard to cpu speed limits, that I recall. Windows 10 Task Manager has shown it at as I recall at least 1.42, 1.44, and 1.5Ghz; the first two under heavy prime95 load and side cover off. So Ernst you may want to look to CentOS or reconsider SpeedStep. Reviewing the thread, I find
1.05 https://mersenneforum.org/attachment...9&d=1605147324
1.13 https://mersenneforum.org/attachment...3&d=1605231308
1.4 https://mersenneforum.org/attachment...1&d=1605151520
1.42 and 1.44 observed https://mersenneforum.org/showpost.p...&postcount=124
1.5 https://mersenneforum.org/attachment...2&d=1606566337

Disabling may be a plus for stable clock during benchmarking. But subtracting a bit of throttle-UP at times.

Re the baking sheet, that may keep the cats out, but as silicone coated fiberglass will do very little for EMI control.

kriesel · 2020-12-05, 16:44

Highest indicated clock rate I've seen from the Xeon Phi 7250 is 1.55 Ghz (Windows Task Manager, a small fraction of the cores in use). Square waves have odd harmonics of the fundamental frequency with declining amplitude. So up to 1.55, 4.65, 7.75....Ghz at amplitude 1, 1/3, 1/5,... How high should be blocked by screening to replace the side panel?
The removable side panels are nonmagnetic, as are top, front, and bottom. Back panel and power supply case are magnetic; back panel has about 200 quantity 6mm hexagonal holes closely spaced for ventilation. Presumably that constitutes a sufficiently bad slot antenna. Assume that perforated aluminum or steel for a new side panel can have the same size hole (that electrical vs. magnetic shielding is not an issue).

Slots are of order 0.1 wavelengths wide by 0.5 wavelengths long in slot antenna use.
6mm/0.1 = 60mm corresponding to an estimated upper bound of 300e6m/sec/.06m = 5Ghz, only 3.2 times the maximum observed clock rate, which seems rather low.

Typical microwave oven frequency is 2.45GHz. There the shielding has holes ~2mm in size. 0.002m / ( 300e6m/s / 2.45e9/s ) = .002/.122 = 0.016 wavelengths.

These results suggest that for somewhat quieting the EMI from a miner frame system, with 3+Ghz CPU and up to ~1.8Ghz gpus, hardware cloth with its 1/4" wire pitch would be a much better choice than 1"x2" aperture welded wire fence roll, for constructing a 5-sided cover shield.

octonion · 2021-02-03, 07:33

I just bought one of these Xeon Phi workstations - does anyone know if the Supermicro K1SPE motherboard might be capable of supporting 128GB 8Rx4 (rank 8) LRDIMMs? It's listed as supporting 64GB LRDIMMs, for a total of 384GB of RAM, but I'm wondering if it might be capable of 768GB of RAM.

ewmayer · 2021-02-04, 23:33

Quote:

Originally Posted by octonion

I just bought one of these Xeon Phi workstations - does anyone know if the Supermicro K1SPE motherboard might be capable of supporting 128GB 8Rx4 (rank 8) LRDIMMs? It's listed as supporting 64GB LRDIMMs, for a total of 384GB of RAM, but I'm wondering if it might be capable of 768GB of RAM.

Search for "contact" in the Mobo manual PDF, that shows contact info on p17, Tech Support e-mail is support@supermicro.com .

You don't happen to have one of those monster 128GB modules lying around which you could plug in and see-if-it-boots-and-recognizes-the-RAM, do you?

Just curious, roughly what $ are we talking about for 768GB of such RAM?

octonion · 2021-02-06, 00:43

I'm trying to find someone with a 128GB DDR4 8Rx4 LRDIMM module to borrow for about a week.

octonion · 2021-02-11, 05:49

Sadly, I can't get my Xeon Phi workstation to display output. Reseated the power connections, checked the jumpers, cleared the CMOS and reset the BIOS, tried three different monitors with VGA input. It may simply be a bad VGA cable, despite it being brand new. I'd hate to have to return it.

2020-12-02, 20:40	#149
kriesel "TF79LL86GIMPS96gpu17" Mar 2017 US midwest 4902₁₀ Posts	I did not mess with the BIOS in regard to cpu speed limits, that I recall. Windows 10 Task Manager has shown it at as I recall at least 1.42, 1.44, and 1.5Ghz; the first two under heavy prime95 load and side cover off. So Ernst you may want to look to CentOS or reconsider SpeedStep. Reviewing the thread, I find 1.05 https://mersenneforum.org/attachment...9&d=1605147324 1.13 https://mersenneforum.org/attachment...3&d=1605231308 1.4 https://mersenneforum.org/attachment...1&d=1605151520 1.42 and 1.44 observed https://mersenneforum.org/showpost.p...&postcount=124 1.5 https://mersenneforum.org/attachment...2&d=1606566337 Disabling may be a plus for stable clock during benchmarking. But subtracting a bit of throttle-UP at times. Re the baking sheet, that may keep the cats out, but as silicone coated fiberglass will do very little for EMI control. Last fiddled with by kriesel on 2020-12-02 at 20:57

2020-12-05, 16:44	#150
kriesel "TF79LL86GIMPS96gpu17" Mar 2017 US midwest 2·3·19·43 Posts	Highest indicated clock rate I've seen from the Xeon Phi 7250 is 1.55 Ghz (Windows Task Manager, a small fraction of the cores in use). Square waves have odd harmonics of the fundamental frequency with declining amplitude. So up to 1.55, 4.65, 7.75....Ghz at amplitude 1, 1/3, 1/5,... How high should be blocked by screening to replace the side panel? The removable side panels are nonmagnetic, as are top, front, and bottom. Back panel and power supply case are magnetic; back panel has about 200 quantity 6mm hexagonal holes closely spaced for ventilation. Presumably that constitutes a sufficiently bad slot antenna. Assume that perforated aluminum or steel for a new side panel can have the same size hole (that electrical vs. magnetic shielding is not an issue). Slots are of order 0.1 wavelengths wide by 0.5 wavelengths long in slot antenna use. 6mm/0.1 = 60mm corresponding to an estimated upper bound of 300e6m/sec/.06m = 5Ghz, only 3.2 times the maximum observed clock rate, which seems rather low. Typical microwave oven frequency is 2.45GHz. There the shielding has holes ~2mm in size. 0.002m / ( 300e6m/s / 2.45e9/s ) = .002/.122 = 0.016 wavelengths. These results suggest that for somewhat quieting the EMI from a miner frame system, with 3+Ghz CPU and up to ~1.8Ghz gpus, hardware cloth with its 1/4" wire pitch would be a much better choice than 1"x2" aperture welded wire fence roll, for constructing a 5-sided cover shield. Last fiddled with by kriesel on 2020-12-05 at 16:58

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
AMD vs Intel	dtripp	Software	3	2013-02-19 20:20
Intel NUC	nucleon	Hardware	2	2012-05-10 23:53
Intel RNG API?	R.D. Silverman	Programming	19	2011-09-17 01:43
AMD or Intel	mack	Information & Answers	7	2009-09-13 01:48
Intel Mac?	penguain	NFSNET Discussion	0	2006-06-12 01:31

2020-12-02, 03:11	#144
kriesel "TF79LL86GIMPS96gpu17" Mar 2017 US midwest 2×3×19×43 Posts	https://en.wikipedia.org/wiki/MCDRAM It's not just on the board, it's in the same package as the processor cores. Like modern caches. https://en.wikipedia.org/wiki/Xeon_P...QHL6_DSCx3.jpg

2020-12-02, 06:01	#146
VBCurtis "Curtis" Feb 2005 Riverside, CA 2×5×467 Posts	I interpreted that Colfax document as explaining that cache mode would really be a cache, and not seen by the OS as main memory; while flat would treat the on-package RAM as main memory but a separate space from the DDR4 modules- it's the "two main memories" that I thought would require special coding. In your case, with no DDR to code for, I expected flat to be the choice. Interesting that "cache" mode is still seen by the OS as main memory.

2020-12-02, 20:33	#148
ewmayer ∂²ω=0 Sep 2002 República de California 2×7×829 Posts	Just did a reboot and had a look at the CPU Configuration menu in the BIOs: o CPU freq = 1.4GHz and not BIOS-adjustable (nor the min.max ratios), L1 = 4352KB (4.25MB), L2 = 34816KB (34MB), o HT is enabled by default, since I found no gain from using it in my Mlucas timing tests, disabled it to see if that makes a difference for my current 1-thread-per-core running Under the Advanced Power Management Configuration Power Technology menu: o Config TDP submenus not shown o EIST (P-states): "EIST (Enhanced Intel SpeedStep Technology) allows the system to automatically adjust processor voltage and core frequency to reduce power consumption and heat dissipation. The options are Disable and Enable" -- basically, a form of auto-throttling, I disabled, which should be safe to do with my case in side-panel-removed form with the resulting low CPU temps (I ordered the baking sheet as my first side-opening mesh-covering solution attempt, btw). Note the sidebar infopane adds "When disabled, CPU frequency is set at max non-turbo", which means the aforementioned 1.4GHz. But "non-Turbo" implies there might be an available "turbo" mode - that might refer to a particular version of the CPU which has the clock unlocked, though. o CPU C State Control -- see nothing of interest there o CPU T State Control (Available when Power Technology is set to Custom): has a single submenu, ACPI (Advanced Configuration Power Interface) T-States: "Select Enable to support CPU throttling by the operating system to reduce power consumption. The options are Enable and Disable" -- switched to Disable, saved changes, booted the OS, fired up big 64M-FFT F30 job on 64 cores, wait for next ~10-min checkpoint update ... and no discernible speed change. So unless someone else has any bright ideas, looks like we are more or less maxed out already at the default settings.

2021-02-03, 07:33	#151
octonion Feb 2021 4₁₀ Posts	I just bought one of these Xeon Phi workstations - does anyone know if the Supermicro K1SPE motherboard might be capable of supporting 128GB 8Rx4 (rank 8) LRDIMMs? It's listed as supporting 64GB LRDIMMs, for a total of 384GB of RAM, but I'm wondering if it might be capable of 768GB of RAM.

2021-02-06, 00:43	#153
octonion Feb 2021 4₈ Posts	I'm trying to find someone with a 128GB DDR4 8Rx4 LRDIMM module to borrow for about a week.

2021-02-11, 05:49	#154
octonion Feb 2021 2² Posts	Sadly, I can't get my Xeon Phi workstation to display output. Reseated the power connections, checked the jumpers, cleared the CMOS and reset the BIOS, tried three different monitors with VGA input. It may simply be a bad VGA cable, despite it being brand new. I'd hate to have to return it.