mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Software

Reply
 
Thread Tools
Old 2010-07-08, 16:22   #1
gabrieltt
 
Jul 2010
planet earth

22 Posts
Default 32 cores limitation

hi all!

i have a problem using mprime on a system with 48 cores. it is a four cpu amd opteron 12 core system running linux. when i try to configure mprime to use all of my cores, i get strange integers for cores >32 in the configuration dialog, looks like a buffer overflow. running mprime anyway will just use 32 of my cores.

is it possible, that mprime can't handle more than 32 cores? and if so, would it be possible somehow to modify mprime to compute on all cores?

any ideas?
gabriel
gabrieltt is offline   Reply With Quote
Old 2010-07-08, 17:20   #2
firejuggler
 
firejuggler's Avatar
 
Apr 2010
Over the rainbow

26·37 Posts
Default

i can offer you an alternative solution : make another folder, copy your binary there and run another session from it on 16 cores. Or 15, if you need to use it for another purpose

Last fiddled with by firejuggler on 2010-07-08 at 17:21
firejuggler is online now   Reply With Quote
Old 2010-07-09, 02:25   #3
Prime95
P90 years forever!
 
Prime95's Avatar
 
Aug 2002
Yeehaw, FL

32·19·41 Posts
Default

Quote:
Originally Posted by gabrieltt View Post
is it possible, that mprime can't handle more than 32 cores?
Yes.

It might be possible to change mprime. I'd have to look into Linux's (and Windows') set affinity system call to see if it now handles more than 32 cores. At the very least I should make a future version that allows more than 32 worker windows / threads and just sets threads to run on any core.
Prime95 is online now   Reply With Quote
Old 2010-07-09, 11:49   #4
gabrieltt
 
Jul 2010
planet earth

410 Posts
Default

hi, first of all thanks for your fast reply!

Quote:
Originally Posted by firejuggler View Post
make another folder, copy your binary there and run another session from it on 16 cores.
i tried this. making two different folders starting the first one with 32 cores, waiting until they were addressed, then starting the second one with 16 cores, but it remains the same. there were just 32 cores in use as you can see here:
Code:
 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 7417 root      30  10  389m  14m 1268 S 2131  0.0 739:26.61  mprime                                                                  
 7480 root      30  10  246m 7996 1268 S 1075  0.0 172:46.18 mprime
where 2131 and 1075 are cpu-percentage here. in sum not exceeding 3200%. but obviously i have 16 cores left. since:
Code:
cat /proc/cpuinfo | grep -c processor
48
Quote:
Originally Posted by Prime95 View Post
It might be possible to change mprime. I'd have to look into Linux's (and Windows') set affinity system call to see if it now handles more than 32 cores. At the very least I should make a future version that allows more than 32 worker windows / threads and just sets threads to run on any core.
that would be very great if you would have the time to do that.

thanks again,
gabriel
gabrieltt is offline   Reply With Quote
Old 2010-07-09, 12:11   #5
gabrieltt
 
Jul 2010
planet earth

22 Posts
Default general core availability

i'm currently trying to eliminate other causes. so i started a prime program i've written on my own. i configured it to use all 48 cores and it does as you can see:
Code:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                
 7708 root      20   0 85.3g  84g  960 S 4790 67.7  50:37.54 prime3
gabrieltt is offline   Reply With Quote
Old 2010-07-10, 16:42   #6
chris2be8
 
chris2be8's Avatar
 
Sep 2009

22×3×5×31 Posts
Default

Quote:
Originally Posted by gabrieltt View Post
i have a problem using mprime on a system with 48 cores. it is a four cpu amd opteron 12 core system running linux. when i try to configure mprime to use all of my cores, i get strange integers for cores >32 in the configuration dialog, looks like a buffer overflow. running mprime anyway will just use 32 of my cores.
I'm thinking of getting a new system, that sounds a nice one. Where did you get it and how much did it cost?

And how much RAM has it got? Could it be running out of memory at 32 cores?

Chris K
chris2be8 is offline   Reply With Quote
Old 2010-07-11, 09:26   #7
chris2be8
 
chris2be8's Avatar
 
Sep 2009

74416 Posts
Default

To get back on topic, what Linux distribution and release are you running? Output from "uname -a" and the contents of /etc/*release would tell us.

Chris K
chris2be8 is offline   Reply With Quote
Old 2010-07-11, 10:29   #8
TheJudger
 
TheJudger's Avatar
 
"Oliver"
Mar 2005
Germany

45316 Posts
Default

Chris,

it is not related to the Linux distribution.
George uses a 32 bit integer as CPU mask (function sched_setaffinity()) which allows only 32 CPUs/cores. When mprime says "running on any CPU" its more like "running on the first 32 CPUs/cores" (if available). Seems to be a similar issue on Windows, too.
---
On Linux one could try to preload sched_setaffinity() with a dummy function.
---
Gabriel:
you could try to start mprime with 16 threads and once they are started use taskset to move the individual threads to CPU cores 32..47. You can us 'ps -L -C mprime' to receive the "PIDs" of the individual tasks.
Once you've moved those 16 threads to CPUs 32..47 start another instace of mprime (using 32 threads) running on CPU 0..31.

Of course this has (at least) two issues:
- there is a high chance that the threads on CPU 32..47 don't use "local memory" (NUMA) which results in lower performance
- it might not last long until mprime resets the affinity masks (I didn't take a deeper look into the code).
---
Another possible workaround:
Grab the code, disable the calls of sched_setaffinity() and recompile.
Run (in your case) 8 instances of mprime, each one with 6 threads and use tools like numactl or taskset to start each instance on the individual NUMA nodes. When talking about AMDs 8 and 12 core CPUs: they are actually double 4 and 6 core CPUs.
When talking about NUMA your system with 4 12-cores acts like a system with 8 6-core CPUs!

Oliver
TheJudger is offline   Reply With Quote
Old 2010-07-11, 13:07   #9
xilman
Bamboozled!
 
xilman's Avatar
 
May 2003
Down not across

27B016 Posts
Default

Quote:
Originally Posted by TheJudger View Post
Another possible workaround: ...
And another: use the remaining 16 cores to run a completely different program ;-)

If you would like to factor integers, a number of people here will welcome you with open arms and, more to the point, provide you with software and tasks.

Paul
xilman is offline   Reply With Quote
Old 2010-07-12, 02:35   #10
joblack
 
joblack's Avatar
 
Oct 2008
n00bville

52·29 Posts
Default

Quote:
Originally Posted by TheJudger View Post
Chris,
it is not related to the Linux distribution.
George uses a 32 bit integer as CPU mask (function sched_setaffinity()) which allows only 32 CPUs/cores.
As far as I know there is a different new function which supports the cpu groups. It might be not that hard to implement that feature.

Man, I would like to have a 48-core, too ;).
joblack is offline   Reply With Quote
Old 2010-07-12, 03:35   #11
mdettweiler
A Sunny Moo
 
mdettweiler's Avatar
 
Aug 2007
USA (GMT-5)

3×2,083 Posts
Default

Hmm...since the problem seems to be in the affinity code, how would it work if the OP just set Prime95 to use 48 worker windows, but without actually specifying affinities for each one? In my experience Windows and Linux usually handle the distribution of tasks to particular CPUs pretty well automatically, but then again my experience is limited to <=4 cores so it may not apply at this level. (Perhaps this isn't so much of a problem when one is simply running multiple instances of a single-threaded program, as I more often do, rather than running one multithreaded mprime/Prime95 instance to cover all cores.)
mdettweiler is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
AffinityScramble2 limitation aurashift PrimeNet 20 2016-09-25 02:21
Hardware limitation? bcp19 Hardware 1 2013-02-18 04:15
CPU cores Unregistered Information & Answers 7 2009-11-02 08:27
6 Intel Cores petrw1 Hardware 3 2008-09-16 16:33
A program that uses all the CPU-cores Primix Hardware 7 2008-09-06 21:09

All times are UTC. The time now is 19:49.

Wed Aug 5 19:49:08 UTC 2020 up 19 days, 15:35, 2 users, load averages: 1.01, 1.37, 1.52

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.