mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Hardware > Cloud Computing

Reply
 
Thread Tools
Old 2020-02-09, 20:35   #892
xx005fs
 
"Eric"
Jan 2018
USA

22×53 Posts
Default

Is there anyway to run 1 session but execute 2 programs at once, say executing primenet.py and GPUOWL to manage assignments or MISFIT and mfactx? or is there a need to open up a second session to run the assignment managers and will those affect my GPU session run time?

Last fiddled with by xx005fs on 2020-02-09 at 20:37
xx005fs is offline   Reply With Quote
Old 2020-02-09, 20:50   #893
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

947110 Posts
Default

Quote:
Originally Posted by xx005fs View Post
Is there anyway to run 1 session but execute 2 programs at once, say executing primenet.py and GPUOWL to manage assignments or MISFIT and mfactx?
Well, MISFIT is going to be a problem, since it's Winblows only...

But to answer your question, sure. Just launch it (or anything(s)) you want into the background.

But... be aware that everything launched (even when fork()'ed) is terminated when the Section finishes running, so you need to keep the launching script running (possibly just sleep(forever)).
chalsall is online now   Reply With Quote
Old 2020-02-09, 21:01   #894
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

132×29 Posts
Default

Quote:
Originally Posted by xx005fs View Post
Is there any way to run 1 session but execute 2 programs at once
Yes. Or 3 or more.

Launch task A as a background task, go on to launch task B as a background task, and task C showing periodic top updates is my normal MO in each colab session. The colab cpus are fast enough that if you're a frequent colab user, a 90M primality test will complete in time before assignment expiration.

It's linux on a VM, so multitasking is built in, no need for separate Colab sessions for each task. My task A is mprime to occupy the cpu core; B a gpu app if I can get a gpu, otherwise omitted; C is top repeating at 2-minute intervals in the foreground, that shows uptime and whether mprime and the gpu app are running and if so how much cpu time each is getting. That gives pretty good resolution of how long a session lasted.

Another background task could be wedged in after the launch of task B to run on cpu also such as a primenet.py or other helper app. Or an additional gpu task on the same allocated gpu, to get the small total throughput gains that are common even in well tuned TF applications on the faster gpus. (Commonly 1-3% boost with a second instance.)

If the gpu times out, the session times out.

If the gpu runs out of work, the gpu app and its background process terminate, while mprime keeps going until cpu use times out.

Top regularly repeating (!top -d 120 or however many seconds interval you like, as long as it's shorter than Colab's idle-session detection) keeps the session going.

By all means run mprime along with the gpu app; don't leave the cpu idle/wasted.

Do LL DC if nothing else; there's a several year backlog.
Sample code for multiple-task sections has been posted.

Follow the link in https://mersenneforum.org/showpost.p...&postcount=891 or see https://www.mersenneforum.org/showpo...73&postcount=8 etc.

Quote:
, say executing primenet.py and GPUOWL to manage assignments or MISFIT and mfactx? or is there a need to open up a second session to run the assignment managers and will those affect my GPU session run time?
I have no running experience on MISFIT. Getting it to run on colab might be a challenge; MS Forms & .net required. If I recall correctly there is also a graphical interface. I've seen claims it may be runnable on linux with mono. Never tried it myself.

There are alternatives. See https://www.mersenneforum.org/showpo...92&postcount=3
Another possibility is to add softare on your pc to make a Google drive appear local, and run MISFIT or another management app locally.
In my work mix, running TF to 76 bits usually, the work addition and results reporting only needs to be daily or less, hardly worth scripting into colab. It would be different at 73 bits or even lower.

In general, for reference info, see https://mersenneforum.org/showthread.php?t=24607

Last fiddled with by kriesel on 2020-02-09 at 21:22
kriesel is offline   Reply With Quote
Old 2020-02-10, 08:42   #895
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

132×29 Posts
Default how many cpus/cores?

I thought a Colab session only got one cpu core. But I notice today it looks like 2, with HT.
Code:
Architecture:        x86_64 CPU 
op-mode(s):      32-bit, 64-bit 
Byte Order:          Little Endian 
CPU(s):              2 
On-line CPU(s) list: 0,1 
Thread(s) per core:  2 
Core(s) per socket:  1 
Socket(s):           1 
NUMA node(s):        1 
Vendor ID:           GenuineIntel 
CPU family:          6 Model:               79 
Model name:          Intel(R) Xeon(R) CPU @ 2.20GHz

Last fiddled with by kriesel on 2020-02-10 at 08:44
kriesel is offline   Reply With Quote
Old 2020-02-10, 09:47   #896
axn
 
axn's Avatar
 
Jun 2003

485610 Posts
Default

Quote:
Originally Posted by kriesel View Post
I thought a Colab session only got one cpu core. But I notice today it looks like 2, with HT.
Not sure whether you think 2 cores/4 threads or 1 core/2 threads, but it is in fact the latter.
For P95 use, it is just one core.
axn is offline   Reply With Quote
Old 2020-02-10, 18:06   #897
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

24FF16 Posts
Default

Quote:
Originally Posted by Chuck View Post
Finally. I've signed up. Let's see how this works out.
Well, things have /definitely/ changed on Colab. For the better; at least at the moment.

A massive amount of compute is being given. Personally, after almost a month getting effectively nothing, I'm at ~3.5 THzD/D for the last five days.

Thanks, Google!!!

Chuck... How are you finding the "paid tier"? I'd try it myself, but I don't have any US-based financial routes.
chalsall is online now   Reply With Quote
Old 2020-02-10, 23:52   #898
chalsall
If I May
 
chalsall's Avatar
 
"Chris Halsall"
Sep 2002
Barbados

3×7×11×41 Posts
Default

Quote:
Originally Posted by chalsall View Post
Personally, after almost a month getting effectively nothing, I'm at ~3.5 THzD/D for the last five days.
A new personal record... Six (6#) T4s running simultaneously!!!

And over-night, one T4 lasted 11 hours, while three others lasted 10 hours each!
chalsall is online now   Reply With Quote
Old 2020-02-11, 00:09   #899
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

132·29 Posts
Default

Quote:
Originally Posted by chalsall View Post
A new personal record... Six (6#) T4s running simultaneously!!!

And over-night, one T4 lasted 11 hours, while three others lasted 10 hours each!
So that's where they've all gone.
kriesel is offline   Reply With Quote
Old 2020-02-11, 01:04   #900
Chuck
 
Chuck's Avatar
 
May 2011
Orange Park, FL

32·97 Posts
Default

Quote:
Originally Posted by chalsall View Post

Chuck... How are you finding the "paid tier"? I'd try it myself, but I don't have any US-based financial routes.
It's working well so far. It disconnects after 24 hours. I had a T4 the first session and P100 thereafter.
Chuck is offline   Reply With Quote
Old 2020-02-11, 02:22   #901
xx005fs
 
"Eric"
Jan 2018
USA

22×53 Posts
Default

After the colab premium launch, it has been a complete disaster for me since the only GPU I am receiving is the Tesla T4s, and I can't even get a half decent GPU like the K80 to run my PRP tasks. Even after 30 session resets I am still getting useless T4s, which I would much rather hand it to someone else doing TF since I am not going to run TF workloads. It would've been much better if google decided to leave the K80 GPUs for free users to tinker with.
xx005fs is offline   Reply With Quote
Old 2020-02-11, 03:03   #902
axn
 
axn's Avatar
 
Jun 2003

113708 Posts
Default

Quote:
Originally Posted by xx005fs View Post
After the colab premium launch, it has been a complete disaster for me since the only GPU I am receiving is the Tesla T4s, and I can't even get a half decent GPU like the K80 to run my PRP tasks. Even after 30 session resets I am still getting useless T4s, which I would much rather hand it to someone else doing TF since I am not going to run TF workloads. It would've been much better if google decided to leave the K80 GPUs for free users to tinker with.
In my timing runs, a T4 was only about 15% slower than K80 (since you get only half a K80 in colab).

Try this - when you get a T4, use it run your PRP. After 1-2 hours, kill the run and ask for a new GPU session. There is a high chance you will get a P100 or a K80.
axn is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Alternatives to Google Colab kriesel Cloud Computing 11 2020-01-14 18:45
Notebook enzocreti enzocreti 0 2019-02-15 08:20
Computer Diet causes Machine Check Exception -- need heuristics help Christenson Hardware 32 2011-12-25 08:17
Computer diet - Need help garo Hardware 41 2011-10-06 04:06
Workunit diet ? dsouza123 NFSNET Discussion 5 2004-02-27 00:42

All times are UTC. The time now is 16:52.

Thu Feb 25 16:52:00 UTC 2021 up 84 days, 13:03, 0 users, load averages: 1.68, 1.92, 1.92

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.