mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Cunningham Tables

Reply
 
Thread Tools
Old 2021-10-26, 21:08   #155
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

2×3×127 Posts
Default

Have a look, your addition is implemented. The script took nearly nine minutes (!) to generate the HTML file the last time. Before that, it was less than thirty seconds. I cannot see why your extension should change this so drastically.

The range is not extracted correctly. It states letter n, space, hyphen, and then a part of the log file name. Thus, I invalidated it.
kruoli is offline   Reply With Quote
Old 2021-10-27, 00:28   #156
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3·372 Posts
Default

Quote:
Originally Posted by kruoli View Post
Have a look, your addition is implemented. The script took nearly nine minutes (!) to generate the HTML file the last time. Before that, it was less than thirty seconds. I cannot see why your extension should change this so drastically.

The range is not extracted correctly. It states letter n, space, hyphen, and then a part of the log file name. Thus, I invalidated it.
I was working with a locally generated log. If you could, copy a section of your log that has a couple of my clients in it with a found relations line (or two) and post those in a code block in a PM. That will help me adjust the script for your log.
EdH is offline   Reply With Quote
Old 2021-10-27, 01:29   #157
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

100B16 Posts
Default

Quote:
Originally Posted by kruoli View Post
. . . The script took nearly nine minutes (!) to generate the HTML file the last time. Before that, it was less than thirty seconds. I cannot see why your extension should change this so drastically. . .
That's because I was doing too much unneeded processing within the case block. I've moved that processing to later. After I adjust for the log differences, it should run much faster.
EdH is offline   Reply With Quote
Old 2021-10-27, 12:34   #158
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3·372 Posts
Default

Good - the fix "seems" simple. I also knocked some processing time off the routine, but the changes might be confusing. I'll paste the two sections first and then a complete script that you can make your time modification to, in case that's easier.

This should take the place of the previous case block:
Code:
      *"Lattice Sieving: Found"*) total=${line}
        found=${line:50:25}
        d=$(echo `expr index "$found" d`)
        found=${found:${d}+1}
        r=$(echo `expr index "$found" r`)
        found=${found:0:${r}-2}
        if [ $found -gt $maxfound ]
          then
            maxfound=$found
            rangemax=${total:100}
        fi
        if [ $found -lt $minfound ]
          then
            minfound=$found
            rangemin=${total:100}
        fi
        ;;
And, this should be changed in the appropriate section below the case block:
Code:
rindh=$(echo `expr index "$rangemax" -`)
rindp=$(echo `expr index "${rangemax:${rindh}}" .`)
rangemax=${rangemax:${rindh}-${rindp}:${rindp}+${rindp}-1}
rindh=$(echo `expr index "$rangemin" -`)
rindp=$(echo `expr index "${rangemin:${rindh}}" .`)
rangemin=${rangemin:${rindh}-${rindp}:${rindp}+${rindp}-1}
echo "<h3>$eta</h3>" >>TeamSieve.html
echo "<p>Maximum relations returned from single WU: ${maxfound} ($rangemax)</p>" >>TeamSieve.html
echo "<p>Minimum relations returned from single WU: ${minfound} ($rangemin)</p>" >>TeamSieve.html
echo "</center>" >>TeamSieve.html
Here's my whole script ATM. There's some extra code for displaying the numbers of relations that doesn't include the echo for it. You can work on that if you wish, or I'll add that in later, in case it's of interest. I don't think it was in the original script, but I didn't check. You may have just removed that:
Code:
#!/bin/bash

log="CADOhtml.log"
maxfound=0
minfound=100000

echo "<!DOCTYPE html>" >TeamSieve.html
echo "<html>" >>TeamSieve.html
echo "<head>" >>TeamSieve.html
echo "<title>Team Sieve data for 3,748+</title>" >>TeamSieve.html
echo "</head>" >>TeamSieve.html
echo "<body>" >>TeamSieve.html
echo "<center>" >>TeamSieve.html
echo "<h1>Current percentage done and ETA</h1>" >>TeamSieve.html

exec <"$log"
while read line
  do
    case $line in
      *"Lattice Sieving: Marking workunit"*) eta=${line:80}
        ;;
      *"Lattice Sieving: Found"*) total=${line}
        found=${line:50:25}
        d=$(echo `expr index "$found" d`)
        found=${found:${d}+1}
        r=$(echo `expr index "$found" r`)
        found=${found:0:${r}-2}
        if [ $found -gt $maxfound ]
          then
            maxfound=$found
            rangemax=${total:100}
        fi
        if [ $found -lt $minfound ]
          then
            minfound=$found
            rangemin=${total:100}
        fi
        ;;
    esac
  done
k=$(echo `expr index "$eta" k`)
eta=${eta:${k}+2:-1}
total=${total##*w}
div=$(echo `expr index "$total" /`)
totalf=${total:0:${div}-1}
totaln=${total:${div}}
rindh=$(echo `expr index "$rangemax" -`)
rindp=$(echo `expr index "${rangemax:${rindh}}" .`)
rangemax=${rangemax:${rindh}-${rindp}:${rindp}+${rindp}-1}
rindh=$(echo `expr index "$rangemin" -`)
rindp=$(echo `expr index "${rangemin:${rindh}}" .`)
rangemin=${rangemin:${rindh}-${rindp}:${rindp}+${rindp}-1}
echo "<h3>$eta</h3>" >>TeamSieve.html
echo "<p>Maximum relations returned from single WU: ${maxfound} ($rangemax)</p>" >>TeamSieve.html
echo "<p>Minimum relations returned from single WU: ${minfound} ($rangemin)</p>" >>TeamSieve.html
echo "</center>" >>TeamSieve.html
echo "</body>" >>TeamSieve.html
echo "</html>" >>TeamSieve.html
I really should make this more modular if we're going to add/edit much more. . .
EdH is offline   Reply With Quote
Old 2021-10-27, 12:50   #159
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

2·3·127 Posts
Default

The script now took 5m 50s, which means I definitely try to get "my version" (not with BASH) working ASAP. If Seth is faster, we might skip this step entirely.

Quote:
Originally Posted by EdH View Post
There's some extra code for displaying the numbers of relations that doesn't include the echo for it.
In your current version neither; that was not in the original version and I have not removed it.
kruoli is offline   Reply With Quote
Old 2021-10-27, 13:02   #160
Gimarel
 
Apr 2010

101110112 Posts
Default

I think the loop
Code:
exec <"$log"
while read line
  do
...
  done
is to slow. I would use something like
Code:
line=$(grep "Lattice Sieving: Marking workunit" "$log" | tail -n 1)
eta=${line:80}
grep "Lattice Sieving: Found" "$log" |
while read line
  do
...
  done
and ommit the "case".
But this doesn't work because of the pipe.

Last fiddled with by Gimarel on 2021-10-27 at 13:04
Gimarel is offline   Reply With Quote
Old 2021-10-27, 22:11   #161
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3·372 Posts
Default

@Gimarel: I did use your grep suggestion for the ETA, but I'm trying something a little different for the Max/Min to see if it will speed things up at all.


I've made a bunch of changes in the script and added comments, so here's the whole thing now:
Code:
#!/bin/bash

log="CADOhtml.log"
maxfound=0
minfound=100000

# Initialize HTML page
echo "<!DOCTYPE html>" >TeamSieve.html
echo "<html>" >>TeamSieve.html
echo "<head>" >>TeamSieve.html
echo "<title>Team Sieve data for 3,748+</title>" >>TeamSieve.html
echo "</head>" >>TeamSieve.html
echo "<body>" >>TeamSieve.html
echo "<center>" >>TeamSieve.html
echo "<h1>Current percentage done and ETA</h1>" >>TeamSieve.html

# Harvest found relations data
exec <"$log"
while read line
  do
    case $line in
      *"Lattice Sieving: Found"*) linearr=($line)
        found="${linearr[6]}"
        if [ $found -gt $maxfound ]
          then
            maxfound=$found
            rangemax=${line:100}
        fi
        if [ $found -lt $minfound ]
          then
            minfound=$found
            rangemin=${line:100}
        fi
        ;;
    esac
  done

# Harvest ETA data
eta=$(grep "Lattice Sieving: Marking workunit" "$log" | tail -n 1)
etaarr=($eta)
echo "<h3>${etaarr[10]:1} ${etaarr[11]} ${etaarr[12]} ${etaarr[13]} ${etaarr[14]} ${etaarr[15]} ${etaarr[16]} ${etaarr[17]:0:-1}</h3>" >>TeamSieve.html

# Add local generation time
echo "<p> Local time of creating this report: " >>TeamSieve.html
date >>TeamSieve.html
echo "</p>" >>TeamSieve.html

# Harvest Maximum and Minimum
IFS="."
rmaxarr=($rangemax)
rminarr=($rangemin)
unset IFS
echo "<p>Maximum relations returned from single WU: ${maxfound} (${rmaxarr[2]})</p>" >>TeamSieve.html
echo "<p>Minimum relations returned from single WU: ${minfound} (${rminarr[2]})</p>" >>TeamSieve.html

# Finalize HTML page
echo "</center>" >>TeamSieve.html
echo "</body>" >>TeamSieve.html
echo "</html>" >>TeamSieve.html
Let me know of the timing change. If it's longer to run now, I'll look at other approaches. I'm also questioning the usefulness of the max/min ATM. I added in the local time data, too.
EdH is offline   Reply With Quote
Old 2021-10-27, 22:50   #162
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

13C916 Posts
Default

Quote:
Originally Posted by EdH View Post
I'm also questioning the usefulness of the max/min ATM. I added in the local time data, too.
I don't think the max has any value- I experimented with -override flag to see what could be overridden, and ran a client-day with A=31. Pretty sure no A=30 workunit will find 12,000 relations!

Kind of tempted to run a few with A=32 to see how high I can push that record. :)
VBCurtis is offline   Reply With Quote
Old 2021-10-27, 23:57   #163
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3·372 Posts
Default

Quote:
Originally Posted by VBCurtis View Post
I don't think the max has any value- I experimented with -override flag to see what could be overridden, and ran a client-day with A=31. Pretty sure no A=30 workunit will find 12,000 relations!

Kind of tempted to run a few with A=32 to see how high I can push that record. :)
I kind of wondered about that number shown, compared to the minimum. I have had really small returns with the first range, but that was because it was smaller than the rest for whatever CADO was doing at the time. I did wonder about this run, though.

Still wondering what data would be of use. Time taken for WUs is of interest, but I'm doing that locally for my machines and we'd be displaying too much if it was done for all clients.
EdH is offline   Reply With Quote
Old 2021-10-28, 00:11   #164
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

506510 Posts
Default

Stats I find interesting as a job proceeds:

total relations found

ETA

Q value of current workunit

average yield for job

current yield for job (say, past 100 or 400 workunits)

time spent so far on job, in units of client-days or some such.
Time spent carries little meaning in a team job like this, since various machines run at variable speeds.
VBCurtis is offline   Reply With Quote
Old 2021-10-28, 01:27   #165
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

10000000010112 Posts
Default

Quote:
Originally Posted by VBCurtis View Post
Stats I find interesting as a job proceeds:

total relations found

ETA

Q value of current workunit

average yield for job

current yield for job (say, past 100 or 400 workunits)

time spent so far on job, in units of client-days or some such.
Time spent carries little meaning in a team job like this, since various machines run at variable speeds.
I will look at this, but not sure how many I can easily implement other than the first, which I was working on, but quit. I'll revisit it. I did add something different I'll post separately about. Let's see if it has any interest.
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Using 16e on smaller numbers fivemack Factoring 3 2017-09-19 08:52
NFS on smaller numbers? skan YAFU 6 2013-02-26 13:57
Bernoulli(200) c204 akruppa Factoring 114 2012-08-20 14:01
checking smaller number fortega Data 2 2005-06-16 22:48
Factoring Smaller Numbers marc Factoring 6 2004-10-09 14:17

All times are UTC. The time now is 09:30.


Sun Nov 28 09:30:54 UTC 2021 up 128 days, 3:59, 0 users, load averages: 0.76, 0.97, 0.95

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.