Memory settings questions

[Prime95]

In testing 27.3, my Sandy Bridge appears to be limited by memory bandwidth. I'm not an expert in this area. Can someone confirm that my memory settings are optimal (or at least reasonable)?

I have two sticks of OCZ OCZ3P2000LV2G. The chip is I5-2500K with a multiplier of 41 instead of the stock 33. CPU-Z says memory is running at 800 MHz, FSB:DRAM ratio is 1:6, CAS-RAS-etc is 9-9-9-20-1T.

Would adding two more sticks improve memory bandwidth?

[Prime95]

I just tried to up the memory to PC1866. It boots but prime95 gives a BSOD. Maybe if I up the memory voltages or play with the timings, I can get this to work.

[kladner]

My own observations indicate that those timings are in a normal range. My partner's i7-920 runs triple channel at 8-8-8-?, I think. (Can't check at the moment.) It is all stock settings, and the RAM is actually rated (and running) at 1333, or 667MHz. My PhenomII 1090T has Corsair (4x4GB) dual channel, which is rated at 667MHz, but runs happily at 800. The SPD timings are 9-9-9-24-41 2T, but it's running at 1T without problems.

It didn't apply to this PhenomII setup, but on an earlier 2 channel board, going from 2 sticks to 4 forced changing from 1T to2T. I just added two sticks to the current board, and was able to keep the 1T.

Unless adding 2 more sticks let you have more channels running, I don't think it would improve bandwidth.

There is probably someone around here who knows i5 setups better than I do.

EDIT: Would reducing latency at the same frequency improve your situation?

[Dubslow]

What are the symptoms that make you think it's memory bandwidth? I generally don't notice anything out of the ordinary on my 2600K (typically it's at 39 multiplier equivalent) and I'm running memory at 667Hz/DDR3-1333 settings (and I think standard 9-9-9-24 timings).

[Prime95]

Originally Posted by Dubslow

What are the symptoms that make you think it's memory bandwidth? I generally don't notice anything out of the ordinary on my 2600K (typically it's at 39 multiplier equivalent) and I'm running memory at 667Hz/DDR3-1333 settings (and I think standard 9-9-9-24 timings).

Running version 27.3, one core has per iteration time of about 15 ms. Running 3 cores, I get about 16 ms. each. Running 4 cores I get about 19 ms. each.

[Ralf Recker]

Originally Posted by Dubslow

What are the symptoms that make you think it's memory bandwidth? I generally don't notice anything out of the ordinary on my 2600K (typically it's at 39 multiplier equivalent) and I'm running memory at 667Hz/DDR3-1333 settings (and I think standard 9-9-9-24 timings).

Originally Posted by Prime95

Running version 27.3, one core has per iteration time of about 15 ms. Running 3 cores, I get about 16 ms. each. Running 4 cores I get about 19 ms. each.

I have seen this over at PrimeGrid. A non-hyperthreaded Core i7-2600K @ 4.4 GHz (8 MB L3 cache) was more than 90 minutes faster (9.75 hours instead of 11.25) when running 4 321-LLR tasks (LLR 3.8.x / gwnumlib 26.x) than a Core i5-2500K @ 4.4 GHz (6 MB L3 cache). With only 3 tasks running the Core i5-2500K was as fast as the Core i7-2600K with 4 tasks running.

[Prime95]

Originally Posted by Prime95

Running version 27.3, one core has per iteration time of about 15 ms. Running 3 cores, I get about 16 ms. each. Running 4 cores I get about 19 ms. each.

With my memory running at 800 MHz, I timed a simple memory read and memory read/write loop. Reads takes 15 clocks per 64-byte cache line. Read/writes take about 24 clocks per cache line.

2560K FFT has 20MB of data. A 2-pass FFT reads/writes this data twice. It must also read about 8MB of sin/cos data. So best case time in the 4 core case on my 4.1GHz machine is: ((20M * 2) / 64 * 24 + 8M / 64 * 15) / 4.1G * 4, or about 17 ms.

Thus, unless I can improve my memory bandwidth I won't be able to get 4 cores to run as fast as the single core case.

[Dubslow]

I've always experienced something like that, though usually 4-core-times/1-core-times < 1.1, so it's not too bad. Either way, 4*.9 > 2*.95 > 1. The nature of our computer tech means that we'll generally be memory limited, unless either CPU caches get larger or memory gets significantly faster. (I've been asking myself for the last 6 months how much the extra 2 MB L3 cache on 2600 vs. 2500 is, i.e. is it worth the extra $100 price difference?)

[Prime95]

I can live with a 4-cores/1-cores timing ratio of 1.1. A 19/15 ratio of 1.26 leaves a lot of potential performance on the table.

I need to ponder whether there are any changes I can make to the FFT that increase the number of floating-point ops but decrease memory usage. This would slow down the FFT in the 1-core case, but increase throughput in the 4-core case.

Remember I'm not optimizing for my machine, I'm trying to improve the throughput of all Sandy Bridge systems. I started this thread primarily to make sure my system is configured properly and thus represents a typical Sandy Bridge system.

P.S. I'm now running at 930MHz with timings of 9-9-9-30 with a command rate of 2T. This is a little bit faster than the previous 800 MHz setup.

[ldesnogu]

Originally Posted by Prime95

I just tried to up the memory to PC1866. It boots but prime95 gives a BSOD. Maybe if I up the memory voltages or play with the timings, I can get this to work.

There are some answers on OCZ forum.

As far as increasing bandwidth goes, IIRC i5 2500K only supports dual channel, so if you already have 2 DIMM, one for each channel, I don't think adding DIMMs will increase the bandwidth.

[TheJudger]

Originally Posted by Prime95

I have two sticks of OCZ OCZ3P2000LV2G. The chip is I5-2500K with a multiplier of 41 instead of the stock 33. CPU-Z says memory is running at 800 MHz, FSB:DRAM ratio is 1:6, CAS-RAS-etc is 9-9-9-20-1T.

Would adding two more sticks improve memory bandwidth?

Well, I guess you've populated your two modules correct (one module per memory channel). When you add another pair of memory you have more ranks per channel. For a fixed clockrate the theoretical bandwidth is the same but the practical bandwidth can be a little bit higher because read & writes to different ranks have a lower latency than a read & write to the same rank. With more ranks there is a higher chance that memory acceses are not on the same rank. On the other hand more ranks usually means lower clockrate which has a much bigger penalty. As an example here you can see the validated clockrates for unbuffered DIMMs on a Xeon 55xx series CPU. Those numbers may change for other CPUs but generally more modules(ranks) per channel yield a lower clockrate. For best performance you usually want a single dual-rank module per channel with the highest supported/possible clockrate.

Originally Posted by Prime95

I can live with a 4-cores/1-cores timing ratio of 1.1. A 19/15 ratio of 1.26 leaves a lot of potential performance on the table.

Keep in mind that you have to share L3 capacity and bandwidth aswell. Another reason for bad scaling can be the turbomode (higher clockspeed for a single core, lower clockspeed when all four cores are busy) but I guess this is not the reason in your case.

Oliver