mersenneforum.org  

Go Back   mersenneforum.org > Extra Stuff > Programming

Reply
 
Thread Tools
Old 2005-02-07, 09:55   #1
leifbk
 
leifbk's Avatar
 
May 2004
Oslo, Norway

7816 Posts
Lightbulb Home-grown GIMPS database with Python/MySQL

I've recently joined the Lone Mersenne Hunters gang, doing some pre-factoring up to 62 bits. I reserved a batch of 2525 exponents. I'm running this on four different computers, all on my local 192.168.0.x network. I split the original worktodo file into 200-lines chunks, and I'm appending these to each worktodo.ini file as needed.

It's a bit tedious to keep track on all the results and assignments manually, so I decided to write a tiny database application. As this is a domestic, extremely non-critical application, I've kept error checking and user interface code at an absolute minimum.

The database engine used here is MySQL. The database is located on a fifth computer, an elderly 133 MHz Pentium with 128 MB RAM that is running Debian Linux. It's named 'samson', IP 192.168.0.10. By the way, I stumbled into a major snag as I tried to make the MySQL db accessible from the network on this box. Buried deep within the /etc/mysql/my.cnf on this distro you'll find this line:

[pre]skip-networking[/pre]

Now, put a # in front of this line, save & restart mysql. Log into mysql as root and run:

Code:
create database gimps;
grant all on gimps.* to 'gimps'@'192.168.0.%' identified by 'gimps';
flush privileges;
use gimps;
create table lmh (
  exponent bigint(20) not null default '0',
  computer varchar(10) not null default '',
  finished timestamp(14) not null,
  residual varchar(20) not null default '',
  factor bigint(20) unsigned default '0',
  primary key  (exponent)
);
You may now verify the connection by logging in from another computer with MySQL installed:

Code:
leif@balapapa lmh $ mysql -u gimps -p gimps -h 192.168.0.10
Enter password: (gimps)
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 130 to server version: 4.0.23_Debian-3-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> desc lmh;
+----------+---------------------+------+-----+---------+-------+
| Field    | Type                | Null | Key | Default | Extra |
+----------+---------------------+------+-----+---------+-------+
| exponent | bigint(20)          |      | PRI | 0       |       |
| computer | varchar(10)         |      |     |         |       |
| finished | timestamp(14)       | YES  |     | NULL    |       |
| residual | varchar(20)         |      |     |         |       |
| factor   | bigint(20) unsigned | YES  |     | 0       |       |
+----------+---------------------+------+-----+---------+-------+
5 rows in set (0.00 sec)

mysql>
Next: Feeding the table
leifbk is offline   Reply With Quote
Old 2005-02-07, 09:57   #2
leifbk
 
leifbk's Avatar
 
May 2004
Oslo, Norway

23×3×5 Posts
Default The Feeder

To stuff all the exponents already done into the lmh table, I had to write a
script that reads the results.txt from end to end:

Code:
#! /usr/bin/env python
# db-feed.py - feed results.txt to db
# leifbk 2005

import time, os, MySQLdb, fileinput

def convertDate(mystring):
    mytimeobj = time.strptime(mystring, "%b %d %H:%M:%S %Y")
    return time.strftime("%Y-%m-%d %H:%M:%S", mytimeobj)

c = MySQLdb.connect(host="192.168.0.10", port=3306, \
            user="gimps", passwd="gimps", db="gimps")
sql = c.cursor()

computer = 'buldrian'
filename = 'buldrian.txt'

for rawline in fileinput.input(filename):
    line = rawline.strip()
    if line[:1] == '[':     # line contains datetime
        timestring = line[5:25]
        finished = convertDate(timestring)
        gotexponent = 0
    index = line.find('M296')
    if index != -1: # then line doesn't contain an exponent we want
        exponent = line[index+1:index+9]
        index = line.find('has a factor')
        if index == -1:
            residual = line[line.find('WZ'):]
            factor = '0'
        else:
            residual = ''
            factor = line[line.find('factor: ')+8:]
        gotexponent = 1
    if gotexponent:
        print exponent, computer, finished, residual, factor
        sql.execute("insert into lmh \
            (exponent, computer, finished, residual, factor) \
            values (%s, %s, %s, %s, %s)", \
            (exponent, computer, finished, residual, factor))
c.close()
Next: The client
leifbk is offline   Reply With Quote
Old 2005-02-07, 09:59   #3
leifbk
 
leifbk's Avatar
 
May 2004
Oslo, Norway

12010 Posts
Default The Client

Here's the client that polls the results.txt and feeds new data into the database as they arrive. You'll need to have MySQL installed on the computers running it (as well as Python). However, it works perfectly all right if you just map up the Prime95/mprime directory over the network, and alter the 'filename' parameter accordingly.

Code:
#! /usr/bin/env python
# db-client.py - poll results.txt for updates and send result to db
# leifbk 2005

import time, os, MySQLdb

c = MySQLdb.connect(host="192.168.0.10", port=3306, \
                    user="gimps", passwd="gimps", db="gimps")
sql = c.cursor()
computer = 'balapapa'   # change this name for each client!

#Set the filename and open the file
# supposed to be run from same dir as mprime/Prime95,
# otherwise, include qualified path
filename = 'results.txt'
file = open(filename,'r')

#Find the size of the file and move to the end
st_results = os.stat(filename)
st_size = st_results[6]
file.seek(st_size)

while 1:
    where = file.tell()
    rawline = file.readline()
    if not rawline:
        time.sleep(1)
        file.seek(where)
    else:
        line = rawline.rstrip() # trim off newline
        index = line.find('M296')
        if index != -1:
            exponent = line[index+1:index+10]
            index = line.find('has a factor')
            if index == -1:
                residual = line[line.find('WZ'):]
                factor = '0'
            else:
                residual = ''
                factor = line[line.find('factor: ')+8:]
            sql.execute("insert into lmh \
                    (exponent, computer, residual, factor) \
                    values (%s, %s, %s, %s)", \
                    (exponent, computer, residual, factor))
c.close()
This script is supposed to run continuously. On Linux, you may run it as background process:

[pre]leif@balapapa lmh $ nohup ./dbc-balapapa.py &[/pre]

On a Windows box, I'll just open a cmd shell, issue the command, and keep the cmd minimized. But there are of course more elegant ways.

Next: The Monitor
leifbk is offline   Reply With Quote
Old 2005-02-07, 10:04   #4
leifbk
 
leifbk's Avatar
 
May 2004
Oslo, Norway

23·3·5 Posts
Default The Monitor

The Monitor program: LMH-stats

Finally, we need a monitor that shows the progress. This is somewhat simplistic, in a later version I plan to use the curses library to produce a non-scrolling view:

Code:
#! /usr/bin/env python
# lmh-stats.py - shows summary from lmh-db. leifbk 2005
# this monitor can be run in a tiny xterm window in a corner of the screen.
# select advanced -> keep above others, to desktop -> all desktops.
# set font size to small, remove tab bar, scroll bar and menu, resize frame
# to fit exactly around the latest screenful.
# close() & window_title() are for linux and do not work on billware.

import MySQLdb, signal
from time import sleep
from os import system
from sys import exit

def close(signal, frame):               # clean exit on Ctrl-C
    exit()

def window_title(s):
    # this works with bash in KDE Konsole, untested on other systems
    title = 'echo -ne "\033]0;' + s + '\007"'
    system(title)

def rest(s, i):
    # display what's left of current worktodo.ini
    # update whenever number of assignments changes!
    if   s == 'balapapa': a = 400
    elif s == 'buldrian': a = 400
    elif s == 'bulbasar': a = 400
    elif s == 'barbara' : a = 200
    return a - i

c = MySQLdb.connect(host="192.168.0.10", port=3306, \
                            user="gimps", passwd="gimps", db="gimps")
sql = c.cursor()
tot = 2525                              # no. of exponents in batch
oldcnt = 0
dashes = '-' * 52
window_title('LMH-status')
signal.signal(signal.SIGINT, close)
while 1:                                # infinite loop
    sql.execute('select count(*) from lmh')
    r = sql.fetchone()
    cnt = r[0]
    if cnt == oldcnt:
        sleep(2)
    else:
        sql.execute('select now()')     # get current time from db
        r = sql.fetchone()
        print '\nStatus of current LMH batch %s' % \
                (r[0].strftime("%Y-%m-%d %H:%M:%S"),)
        print dashes
        sql.execute('(select * from lmh \
                            order by finished desc limit 10) \
                            order by finished')
        r = sql.fetchall()
        for row in r:                   # print latest 10 results
            if row[4] == 0:             # no factor found,
                res = row[3]            # display residual
            else:
                res = str(row[4])       # display factor
            print '%s %-8s %d %s' % \
                            (row[2].strftime("%m-%d %H:%M:%S"), \
                                    row[1], row[0], res)
        print dashes
        print 'Processed %d exponents of total %d, %d to go.' % \
                            (cnt, tot, tot - cnt)
        sql.execute('select count(*) from lmh where factor > 0')
        r = sql.fetchone()
        print 'Found %d factors so far.\n' % (r[0],)
        sql.execute('select computer, count(computer) as done \
                            from lmh \
                            group by computer \
                            order by done desc')
        r = sql.fetchall()
        for row in r:
            print '%-10s %5d %5d' % (row[0], row[1], rest(row[0], row[1]))
        oldcnt = cnt
c.close()
I planted a view of this monitor running on my KDE desktop in the Lounge: "You just might be addicted to GIMPS if ..." thread.

I hope that someone may find some of the things explored here useful.

regards, Leif.
leifbk is offline   Reply With Quote
Old 2005-02-08, 05:46   #5
moo
 
moo's Avatar
 
Jul 2004
Nowhere

80910 Posts
Default

i have a thought this is something im randomly pceing togeather at 11:45 at night well here it goes set up a intranet server that has php then you pass a line over the url so it could enter the entire line in the php would talk to mysql just a thought i dont know how to pass the assignments out tho.
moo is offline   Reply With Quote
Old 2005-02-08, 07:37   #6
leifbk
 
leifbk's Avatar
 
May 2004
Oslo, Norway

23×3×5 Posts
Default

Quote:
Originally Posted by moo
i have a thought this is something im randomly pceing togeather at 11:45 at night well here it goes set up a intranet server that has php then you pass a line over the url so it could enter the entire line in the php would talk to mysql just a thought
Yes, that's of course a possibility if you don't want the client to access a remote MySQL server, but part of my motivation for doing this was to explore how to interact with MySQL directly across a network. It turns out that the interaction is totally transparent, and the only difference between remote and local access, in programming terms, is the hostname specification in the connection string.

I'm probably wrong in assuming that MySQL has to be installed on the client computer, though. If somebody here knows the answer, I'd be thankful for some enlightenment. You'll of course need the Python MySQLdb module, but I'm not entirely sure if this code is self-supported, or if it requires local access to some MySQL code. My Linux computers will install MySQL automatically as a dependency if you ask for the python-mysql module.

About PHP: Yes, that's definitely an alternative. (Some would even prefer Perl ) I developed my entire genealogy website with PHP/MySQL, but there's actually nothing that PHP can do that Python can't do better

Quote:
Originally Posted by moo
i dont know how to pass the assignments out tho.
I've given that some thought, too, but at present I don't care much. I've put my 200-line chunks in an ftp directory on the same computer that's running the MySQL server. I fetch them from there manually, and then delete them on the server. I've could of course have automated this as well to some extent, but it's such a minor amount of work that I don't feel like it's worth it. But of course, I'm a compulsive code hacker, so I just might ...

regards, Leif.
leifbk is offline   Reply With Quote
Old 2005-02-09, 02:43   #7
moo
 
moo's Avatar
 
Jul 2004
Nowhere

14518 Posts
Default

now to think of it that would easilly work use collection with the intranet and pass it out like seti they just tell the cleint the location then it gets the chunk that could work also. use ftp or http to do that...
moo is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Extending the database/limits of GIMPS NBtarheel_33 Data 9 2010-11-29 06:19
Python... Xyzzy Programming 20 2009-09-08 15:51
MySQL error... Xyzzy mersennewiki 2 2006-05-22 18:16
Help w/ python. a216vcti Programming 7 2005-10-30 00:37
C program to rapidly verify all factors in GIMPS database GP2 Programming 8 2005-01-03 07:49

All times are UTC. The time now is 14:56.

Thu Apr 15 14:56:15 UTC 2021 up 7 days, 9:37, 0 users, load averages: 1.87, 2.13, 2.25

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.