3,748+ c204 Smaller-but-Needed - Page 25

kruoli · 2021-11-17, 19:47

Our "old" status page has now rankings without anonymisation and without client details, so perfect! Also, I was able to squeeze out a factor of 3-4x in speed gains again, so basically 3s.

Edit: The rankings work by some processing on data from Seth's script.

Code:

project_name="3_748plus"
log="workdir/$project_name.log"
HTML_FILE="status.html"
WUcountmax=0
WUacount=0
declare -a WUdates
declare -a WUcounts

# Initialize HTML page
echo "<!DOCTYPE html>" > $HTML_FILE
echo "<html>" >> $HTML_FILE
echo "<head>" >> $HTML_FILE
echo "<title>Team Sieve data for 3,748+</title>" >> $HTML_FILE
echo "</head>" >> $HTML_FILE
echo "<body>" >> $HTML_FILE
echo "<center>" >> $HTML_FILE
echo "<h1>Current percentage done and ETA</h1>" >> $HTML_FILE

# Harvest found relations and WU data
wuDatesFile="wuDates.log"
relationCountsLogFile="relations.log"
sortedRelationCountsLogFile="sorted.log"
grep -a "Lattice Sieving: Found" "$log" | cut -d' ' -f7 > "$relationCountsLogFile"
sort -n "$relationCountsLogFile" > "$sortedRelationCountsLogFile"
maxfound=$(tail -n1 $sortedRelationCountsLogFile)
minfound=$(head -n1 $sortedRelationCountsLogFile)
rm -f "$sortedRelationCountsLogFile"
today="$(date +%Y-%m-%d)"
grep -a " as ok (" "$log" | cut -d' ' -f2 | uniq -c > "$wuDatesFile"
while read line
  do
    linearr=($line)
    WUdates[$WUacount]="${linearr[1]}"
    WUcounts[$WUacount]="${linearr[0]}"
    if [ $WUcountmax -lt ${linearr[0]} ]
      then
        WUcountmax=${linearr[0]}
    fi
    if [ "${linearr[1]}" == "$today" ] # if the last group is today, we want to estimate total WUs for today based on the current rate
      then
        WUcounts[$WUacount]="${linearr[0]} (est. $(((60*60*24*${linearr[0]})/($(date +%s)-$(date +%s --date "${linearr[1]} 00:00:00")))))"
      else
        WUcounts[$WUacount]="${linearr[0]}"
    fi
    let WUacount=${WUacount}+1
  done < "$wuDatesFile"
rm -f "$wuDatesFile"

# Harvest ETA data
eta=$(tac "$log" | grep -am1 "Lattice Sieving: Marking workunit")
etaarr=($eta)
echo "<h3>${etaarr[10]:1} ${etaarr[11]} ${etaarr[12]} ${etaarr[13]} ${etaarr[14]} ${etaarr[15]} ${etaarr[16]} ${etaarr[17]:0:-1}</h3>" >> $HTML_FILE

# Add local generation time
echo "<p>Local time of creating this report: $(date)</p>" >> $HTML_FILE

# Harvest Maximum and Minimum
rmax=$(grep -am1 "Found $maxfound" "$log" | rev | cut -d. -f3 | rev)
rmin=$(grep -am1 "Found $minfound" "$log" | rev | cut -d. -f3 | rev)
echo "<p>Maximum relations returned from single WU: ${maxfound} ($rmax)<br/>" >> $HTML_FILE
echo "Minimum relations returned from single WU: ${minfound} ($rmin)</p>" >> $HTML_FILE

# Display WU counts for each day
echo "<table style=\"width:400pt;text-align:center\">" >> $HTML_FILE
echo -e "<tr>\n<th>Day</th>\n<th>Valid WUs</th>\n<th><pre>          </pre></th>\n<th>Day</th>\n<th>Valid WUs</th>\n</tr>" >> $HTML_FILE
for WUindex in $(seq 0 $(((1+$WUacount)/2-1)))
  do
    echo "<tr>" >> $HTML_FILE
    echo "<td>${WUdates[$WUindex]}</td>" >> $HTML_FILE
    if [ "${WUcounts[$WUindex]}" == "$WUcountmax" ] # == instead of -eq because of the special "today" format
      then
        echo "<td><b>${WUcounts[$WUindex]}</b></td>" >> $HTML_FILE
      else
        echo "<td>${WUcounts[$WUindex]}</td>" >> $HTML_FILE
    fi
    echo "<td/>" >> $HTML_FILE # spacer column
    echo "<td>${WUdates[$(((1+$WUacount)/2+$WUindex))]}</td>" >> $HTML_FILE # should be empty for the last entry if WUacount is odd
    if [ "${WUcounts[$(((1+$WUacount)/2+$WUindex))]}" == "$WUcountmax" ]
      then
        echo "<td><b>${WUcounts[$(((1+$WUacount)/2+$WUindex))]}</b></td>" >> $HTML_FILE
      else
        echo "<td>${WUcounts[$(((1+$WUacount)/2+$WUindex))]}</td>" >> $HTML_FILE
    fi
    echo "</tr>" >> $HTML_FILE
  done
echo "</table>" >> $HTML_FILE

# Harvest relations data
relsarr=($(tac "$log" | grep -am1 "Lattice Sieving: Found"))
slash=$(echo `expr index "${relsarr[13]}" /`)
relsf=${relsarr[13]:0:${slash}-1}
relsn=${relsarr[13]:${slash}}
echo "<h4>$relsf relations have been found &ndash; the goal is at least 1300000000 (the server is set to $relsn)</h4>" >> $HTML_FILE
spr=$((${EPOCHSECONDS}-$(date +%s --date "2021-10-19 12:00:00")))
spr=$((${spr}*10000))
spr=$((${spr}/${relsf}))
relsleft=$((1300000000-${relsf}))
tleft=$((${spr}*${relsleft}))
tleft=$((${tleft}/10000))
tleft=$((${EPOCHSECONDS}+${tleft}))
relspdone=$((${relsf}/1300000))
echo -n "<p>Based on 1.3B goal: ${relspdone:0:2}.${relspdone:2}% finished &ndash; ETA: " >> $HTML_FILE
printf "%(%Y-%m-%d %H:%M:%S)T</p>\n" $tleft >> $HTML_FILE

# Average relations found for last 100, 1000 and all WUs
smallCount=1000
bigCount=25000
avcount=$(wc -l < "$relationCountsLogFile")
avsum=$({ sed 's/^/x+=/' "$relationCountsLogFile"; echo x; } | bc)
avsmall=$(($({ tail -n $smallCount "$relationCountsLogFile" | sed 's/^/x+=/'; echo x; } | bc)/$smallCount))
avbig=$(($({ tail -n $bigCount "$relationCountsLogFile" | sed 's/^/x+=/'; echo x; } | bc)/$bigCount))
rm -f "$relationCountsLogFile"
let avtotal=$avsum/$avcount
echo -n "<p>" >> $HTML_FILE
if [ $avcount -gt $smallCount ]
  then
    echo "Average relations returned for last $smallCount WUs: $avsmall<br/>" >> $HTML_FILE
    if [ $avcount -gt $bigCount ]
      then
        echo "Average relations returned for last $bigCount WUs: $avbig<br/>" >> $HTML_FILE
    fi
fi
echo "Average relations returned for all WUs: $avtotal</p>" >> $HTML_FILE

# Total CPU time so far
totalCPU=$(tac "$log" | grep -am1 "Lattice Sieving: Combined stats")
tCPUarr=($totalCPU)
CPUs=${tCPUarr[13]:1:-2}
period=$(echo `expr index "$CPUs" .`)
CPUs=${CPUs:0:${period}-1}
CPUy=$((CPUs/31536000))
CPUd=$((CPUs/86400%365))
CPUh=$((CPUs/3600%24))
CPUm=$((CPUs/60%60))
CPUs=$((CPUs%60))
echo "<p>CPU time logged so far: ${CPUy}y ${CPUd}d ${CPUh}h ${CPUm}m ${CPUs}s</p>" >> $HTML_FILE

# Last WU sent out
lastWU=$(tac "$log" | grep -am1 "Sending workunit")
lWUarr=($lastWU)
lastWU=${lWUarr[8]}
lastWU=${lastWU##*_}
testdate=$(date +"%Y-%m-%d %H:%M:%S")
diffb="${lWUarr[1]} ${lWUarr[2]:0:8}"
diffs="$(($(date -d "$testdate" '+%s') - $(date -d "$diffb" '+%s')))"
if [ $diffs -gt 600 ]
  then
    echo "<h4 style=\"color: #ff0000\">Latest WU sent out at ${lWUarr[2]:0:8} local time: $lastWU</h4>" >> $HTML_FILE
  else
    echo "<p>Latest WU sent out at ${lWUarr[2]:0:8} local time: $lastWU</p>" >> $HTML_FILE
fi

# Check whether server is running
if ! pgrep -f -a nfs.py
  then
    echo "<p style=\"color: #ff0000\"><b>Attention! The server script could not be found in the process list! The server crashed most likely!</b></p>" >> $HTML_FILE
fi

# Link to Seth's version of the stats page
echo "<p><a href=\"Seth/\">Seth's status page</a></p>" >> $HTML_FILE

# Basic top tier list
echo "<h2>Rankings</h2>" >> $HTML_FILE
echo "<table style=\"width:200pt;text-align:center\">" >> $HTML_FILE
echo -e "<tr>\n<th>Place</th>\n<th>User</th>\n<th>Valid WUs</th>\n</tr>" >> $HTML_FILE
i=0;
head -n 20 refresher.log | grep 'workunits$' | while read line
  do
    linearr=($line)
    echo "</tr>" >> $HTML_FILE
    echo "<td>${i}.</td>" >> $HTML_FILE
    echo "<td>${linearr[0]}</td>" >> $HTML_FILE
    echo "<td>${linearr[1]:1}</td>" >> $HTML_FILE
    echo "<tr>" >> $HTML_FILE
    let i=i+1
  done
echo "</table>" >> $HTML_FILE

# Finalize HTML page
echo "</center>" >> $HTML_FILE
echo "</body>" >> $HTML_FILE
echo -n "</html>" >> $HTML_FILE

charybdis · 2021-11-17, 20:59

Do rankings usually start at 0th place in Germany?

(nice work again btw)

kruoli · 2021-11-17, 21:10

Quote:

Originally Posted by charybdis

Do rankings usually start at 0th place in Germany?

(nice work again btw)

xD
Au, weia! That is not my language (I love all of German language's quirks and anomalies, I'd like to post over my joy all week, but nobody likes such spam, but this is not something different in German). That is programmers language, I always think from 0 and languages where arrays start at index 1 are weird. Fixed.

(Thanks!)

EdH · 2021-11-18, 00:58

Quote:

Originally Posted by charybdis

Do rankings usually start at 0th place in Germany?

(nice work again btw)

What?!? Did I miss seeing my "eFarm" in 1st place (in name only)?

I do wonder if the next Team Project will also be able to make use of these scripts, pages, etc. Not that I'm really ready to jump into another when this one is finished. I think a break to play elsewhere will be welcome.

I do like these additions as a little bit of relief from the boredom at this point in our effort.

Great work on all fronts. Thanks for all the effort all around.

kruoli · 2021-11-18, 07:31

We have had a major breakdown of computing power. Maybe some clients got stuck again? For me personally, two were stuck. The server has not crashed for a few days.

Yes, I would be willing to share my scripts and learning process doing this project. I am also working on releasing my client script that pins processes to cores. For larger machines, I saw significant improvements with this method. Especially on multi-chip CPUs.

The log is nearing half a GB in size. It is intriguing how we can parse it in less than three seconds!

kruoli · 2021-11-18, 09:09

Quote:

Originally Posted by kruoli

For me personally, two were stuck.

This was really weird. I pipe their output to a temporary file. It was not able to write to them anymore, although I could touch, rm, cat (to and from) the files. After I changed the file name, it worked.

I had msieve running over night with 1.16B relations for getting used to it. Maybe this data point is helpful still?

Code:

Thu Nov 18 04:00:18 2021  commencing linear algebra
Thu Nov 18 04:00:27 2021  read 87934102 cycles
Thu Nov 18 04:04:54 2021  cycles contain 297844075 unique relations
Thu Nov 18 04:24:22 2021  read 297844075 relations
Thu Nov 18 04:30:31 2021  using 20 quadratic characters above 4294917295
Thu Nov 18 04:41:02 2021  building initial matrix
Thu Nov 18 05:08:44 2021  memory use: 42248.2 MB
Thu Nov 18 05:09:02 2021  read 87934102 cycles
Thu Nov 18 05:09:12 2021  matrix is 87933921 x 87934102 (33193.4 MB) with weight 10167235282 (115.62/col)
Thu Nov 18 05:09:12 2021  sparse part has weight 7646253497 (86.95/col)
Thu Nov 18 05:18:17 2021  filtering completed in 2 passes
Thu Nov 18 05:18:29 2021  matrix is 87886893 x 87887072 (33190.0 MB) with weight 10165476782 (115.67/col)
Thu Nov 18 05:18:29 2021  sparse part has weight 7645921498 (87.00/col)
Thu Nov 18 05:20:14 2021  matrix starts at (0, 0)
Thu Nov 18 05:20:23 2021  matrix is 87886893 x 87887072 (33190.0 MB) with weight 10165476782 (115.67/col)
Thu Nov 18 05:20:23 2021  sparse part has weight 7645921498 (87.00/col)
Thu Nov 18 05:20:23 2021  saving the first 48 matrix rows for later
Thu Nov 18 05:20:29 2021  matrix includes 64 packed rows
Thu Nov 18 05:20:38 2021  matrix is 87886845 x 87887072 (32168.6 MB) with weight 8420654619 (95.81/col)
Thu Nov 18 05:20:38 2021  sparse part has weight 7553924701 (85.95/col)
Thu Nov 18 05:20:38 2021  using block size 8192 and superblock size 6291456 for processor cache size 65536 kB
Thu Nov 18 05:22:58 2021  commencing Lanczos iteration
Thu Nov 18 05:22:58 2021  memory use: 32862.9 MB

Then I killed it when displaying an ETA of over 17,000 hours.

charybdis · 2021-11-18, 11:54

Quote:

Originally Posted by kruoli

We have had a major breakdown of computing power. Maybe some clients got stuck again? For me personally, two were stuck. The server has not crashed for a few days.

My clients - and seemingly everyone else's - were unable to connect to the server for 3 hours last night, until about 07:10 your time. The logs show the same "Connection timed out" error as they do when the server goes down. Whatever it was, let's hope it doesn't happen again.

Quote:

Originally Posted by kruoli

Code:

Thu Nov 18 05:20:38 2021  matrix is 87886845 x 87887072 (32168.6 MB) with weight 8420654619 (95.81/col)
Thu Nov 18 05:20:38 2021  sparse part has weight 7553924701 (85.95/col)
Thu Nov 18 05:20:38 2021  using block size 8192 and superblock size 6291456 for processor cache size 65536 kB
Thu Nov 18 05:22:58 2021  commencing Lanczos iteration
Thu Nov 18 05:22:58 2021  memory use: 32862.9 MB

Then I killed it when displaying an ETA of over 17,000 hours.

Nice to see we can at least produce a matrix. It is of course far too big, though the ETA wouldn't be quite so long if you weren't running msieve single-threaded! Was this with the default target_density?

kruoli · 2021-11-18, 12:12

Quote:

Originally Posted by charybdis

It is of course far too big, though the ETA wouldn't be quite so long if you weren't running msieve single-threaded! Was this with the default target_density?

This is true, but since I would not leave it running this way I started it single threaded. Yes, I simply let msieve decide on what to do with the data. For later, we can of course choose other parameters.

charybdis · 2021-11-18, 12:53

You'll probably want to wait at least until a matrix can be built with target_density=120; I expect VBCurtis will chime in later with his thoughts. Chances are if you tried TD=120 now you would get "too few cycles, matrix probably cannot build".

How is the duplication rate looking?

kruoli · 2021-11-18, 13:04

Wed Nov 17 19:18:16 2021 found 300859501 duplicates and 861473338 unique relations

That looks good, I would guess?

EdH · 2021-11-18, 14:38

I noticed the server had stopped serving when I checked my overnight machines at about 04:30 server time. The check message was red saying the last assignment was (I think) 04:11.

All but one of my clients was running fine this morning. Many had run the timeout reset a couple times. Some of my clients didn't have the new script and all of them but one, were running fine. The one was stuck, as before. All should be good now and I think "timeout" will be the answer to my query about gracefully quiting CADO-NFS. Thanks for that.

Are you going to employ MPI in Msieve LA? I'm not sure of your hardware setup, but I used mpi across two Xeons for a larger run a while back. It did save some time, but it was difficult to learn the nuances. charybdis was quite helpful with it. Unfortunately, I can't locate the posts about getting my setup to work best.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Using 16e on smaller numbers	fivemack	Factoring	3	2017-09-19 08:52
NFS on smaller numbers?	skan	YAFU	6	2013-02-26 13:57
Bernoulli(200) c204	akruppa	Factoring	114	2012-08-20 14:01
checking smaller number	fortega	Data	2	2005-06-16 22:48
Factoring Smaller Numbers	marc	Factoring	6	2004-10-09 14:17

2021-11-17, 20:59	#266
charybdis Apr 2020 7·107 Posts	Do rankings usually start at 0th place in Germany? (nice work again btw)

2021-11-18, 07:31	#269
kruoli "Oliver" Sep 2017 Porta Westfalica, DE 2⁴×5×13 Posts	We have had a major breakdown of computing power. Maybe some clients got stuck again? For me personally, two were stuck. The server has not crashed for a few days. Yes, I would be willing to share my scripts and learning process doing this project. I am also working on releasing my client script that pins processes to cores. For larger machines, I saw significant improvements with this method. Especially on multi-chip CPUs. The log is nearing half a GB in size. It is intriguing how we can parse it in less than three seconds!

2021-11-18, 12:53	#273
charybdis Apr 2020 7·107 Posts	You'll probably want to wait at least until a matrix can be built with target_density=120; I expect VBCurtis will chime in later with his thoughts. Chances are if you tried TD=120 now you would get "too few cycles, matrix probably cannot build". How is the duplication rate looking?

2021-11-18, 13:04	#274
kruoli "Oliver" Sep 2017 Porta Westfalica, DE 2⁴·5·13 Posts	Wed Nov 17 19:18:16 2021 found 300859501 duplicates and 861473338 unique relations That looks good, I would guess?

2021-11-18, 14:38	#275
EdH "Ed Hall" Dec 2009 Adirondack Mtns 2²×1,151 Posts	I noticed the server had stopped serving when I checked my overnight machines at about 04:30 server time. The check message was red saying the last assignment was (I think) 04:11. All but one of my clients was running fine this morning. Many had run the timeout reset a couple times. Some of my clients didn't have the new script and all of them but one, were running fine. The one was stuck, as before. All should be good now and I think "timeout" will be the answer to my query about gracefully quiting CADO-NFS. Thanks for that. Are you going to employ MPI in Msieve LA? I'm not sure of your hardware setup, but I used mpi across two Xeons for a larger run a while back. It did save some time, but it was difficult to learn the nuances. charybdis was quite helpful with it. Unfortunately, I can't locate the posts about getting my setup to work best.