mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Cunningham Tables

Reply
 
Thread Tools
Old 2021-11-17, 19:47   #265
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

24·5·13 Posts
Default

Our "old" status page has now rankings without anonymisation and without client details, so perfect! Also, I was able to squeeze out a factor of 3-4x in speed gains again, so basically 3s.

Edit: The rankings work by some processing on data from Seth's script.

Code:
project_name="3_748plus"
log="workdir/$project_name.log"
HTML_FILE="status.html"
WUcountmax=0
WUacount=0
declare -a WUdates
declare -a WUcounts

# Initialize HTML page
echo "<!DOCTYPE html>" > $HTML_FILE
echo "<html>" >> $HTML_FILE
echo "<head>" >> $HTML_FILE
echo "<title>Team Sieve data for 3,748+</title>" >> $HTML_FILE
echo "</head>" >> $HTML_FILE
echo "<body>" >> $HTML_FILE
echo "<center>" >> $HTML_FILE
echo "<h1>Current percentage done and ETA</h1>" >> $HTML_FILE

# Harvest found relations and WU data
wuDatesFile="wuDates.log"
relationCountsLogFile="relations.log"
sortedRelationCountsLogFile="sorted.log"
grep -a "Lattice Sieving: Found" "$log" | cut -d' ' -f7 > "$relationCountsLogFile"
sort -n "$relationCountsLogFile" > "$sortedRelationCountsLogFile"
maxfound=$(tail -n1 $sortedRelationCountsLogFile)
minfound=$(head -n1 $sortedRelationCountsLogFile)
rm -f "$sortedRelationCountsLogFile"
today="$(date +%Y-%m-%d)"
grep -a " as ok (" "$log" | cut -d' ' -f2 | uniq -c > "$wuDatesFile"
while read line
  do
    linearr=($line)
    WUdates[$WUacount]="${linearr[1]}"
    WUcounts[$WUacount]="${linearr[0]}"
    if [ $WUcountmax -lt ${linearr[0]} ]
      then
        WUcountmax=${linearr[0]}
    fi
    if [ "${linearr[1]}" == "$today" ] # if the last group is today, we want to estimate total WUs for today based on the current rate
      then
        WUcounts[$WUacount]="${linearr[0]} (est. $(((60*60*24*${linearr[0]})/($(date +%s)-$(date +%s --date "${linearr[1]} 00:00:00")))))"
      else
        WUcounts[$WUacount]="${linearr[0]}"
    fi
    let WUacount=${WUacount}+1
  done < "$wuDatesFile"
rm -f "$wuDatesFile"

# Harvest ETA data
eta=$(tac "$log" | grep -am1 "Lattice Sieving: Marking workunit")
etaarr=($eta)
echo "<h3>${etaarr[10]:1} ${etaarr[11]} ${etaarr[12]} ${etaarr[13]} ${etaarr[14]} ${etaarr[15]} ${etaarr[16]} ${etaarr[17]:0:-1}</h3>" >> $HTML_FILE

# Add local generation time
echo "<p>Local time of creating this report: $(date)</p>" >> $HTML_FILE

# Harvest Maximum and Minimum
rmax=$(grep -am1 "Found $maxfound" "$log" | rev | cut -d. -f3 | rev)
rmin=$(grep -am1 "Found $minfound" "$log" | rev | cut -d. -f3 | rev)
echo "<p>Maximum relations returned from single WU: ${maxfound} ($rmax)<br/>" >> $HTML_FILE
echo "Minimum relations returned from single WU: ${minfound} ($rmin)</p>" >> $HTML_FILE

# Display WU counts for each day
echo "<table style=\"width:400pt;text-align:center\">" >> $HTML_FILE
echo -e "<tr>\n<th>Day</th>\n<th>Valid WUs</th>\n<th><pre>          </pre></th>\n<th>Day</th>\n<th>Valid WUs</th>\n</tr>" >> $HTML_FILE
for WUindex in $(seq 0 $(((1+$WUacount)/2-1)))
  do
    echo "<tr>" >> $HTML_FILE
    echo "<td>${WUdates[$WUindex]}</td>" >> $HTML_FILE
    if [ "${WUcounts[$WUindex]}" == "$WUcountmax" ] # == instead of -eq because of the special "today" format
      then
        echo "<td><b>${WUcounts[$WUindex]}</b></td>" >> $HTML_FILE
      else
        echo "<td>${WUcounts[$WUindex]}</td>" >> $HTML_FILE
    fi
    echo "<td/>" >> $HTML_FILE # spacer column
    echo "<td>${WUdates[$(((1+$WUacount)/2+$WUindex))]}</td>" >> $HTML_FILE # should be empty for the last entry if WUacount is odd
    if [ "${WUcounts[$(((1+$WUacount)/2+$WUindex))]}" == "$WUcountmax" ]
      then
        echo "<td><b>${WUcounts[$(((1+$WUacount)/2+$WUindex))]}</b></td>" >> $HTML_FILE
      else
        echo "<td>${WUcounts[$(((1+$WUacount)/2+$WUindex))]}</td>" >> $HTML_FILE
    fi
    echo "</tr>" >> $HTML_FILE
  done
echo "</table>" >> $HTML_FILE

# Harvest relations data
relsarr=($(tac "$log" | grep -am1 "Lattice Sieving: Found"))
slash=$(echo `expr index "${relsarr[13]}" /`)
relsf=${relsarr[13]:0:${slash}-1}
relsn=${relsarr[13]:${slash}}
echo "<h4>$relsf relations have been found &ndash; the goal is at least 1300000000 (the server is set to $relsn)</h4>" >> $HTML_FILE
spr=$((${EPOCHSECONDS}-$(date +%s --date "2021-10-19 12:00:00")))
spr=$((${spr}*10000))
spr=$((${spr}/${relsf}))
relsleft=$((1300000000-${relsf}))
tleft=$((${spr}*${relsleft}))
tleft=$((${tleft}/10000))
tleft=$((${EPOCHSECONDS}+${tleft}))
relspdone=$((${relsf}/1300000))
echo -n "<p>Based on 1.3B goal: ${relspdone:0:2}.${relspdone:2}% finished &ndash; ETA: " >> $HTML_FILE
printf "%(%Y-%m-%d %H:%M:%S)T</p>\n" $tleft >> $HTML_FILE

# Average relations found for last 100, 1000 and all WUs
smallCount=1000
bigCount=25000
avcount=$(wc -l < "$relationCountsLogFile")
avsum=$({ sed 's/^/x+=/' "$relationCountsLogFile"; echo x; } | bc)
avsmall=$(($({ tail -n $smallCount "$relationCountsLogFile" | sed 's/^/x+=/'; echo x; } | bc)/$smallCount))
avbig=$(($({ tail -n $bigCount "$relationCountsLogFile" | sed 's/^/x+=/'; echo x; } | bc)/$bigCount))
rm -f "$relationCountsLogFile"
let avtotal=$avsum/$avcount
echo -n "<p>" >> $HTML_FILE
if [ $avcount -gt $smallCount ]
  then
    echo "Average relations returned for last $smallCount WUs: $avsmall<br/>" >> $HTML_FILE
    if [ $avcount -gt $bigCount ]
      then
        echo "Average relations returned for last $bigCount WUs: $avbig<br/>" >> $HTML_FILE
    fi
fi
echo "Average relations returned for all WUs: $avtotal</p>" >> $HTML_FILE

# Total CPU time so far
totalCPU=$(tac "$log" | grep -am1 "Lattice Sieving: Combined stats")
tCPUarr=($totalCPU)
CPUs=${tCPUarr[13]:1:-2}
period=$(echo `expr index "$CPUs" .`)
CPUs=${CPUs:0:${period}-1}
CPUy=$((CPUs/31536000))
CPUd=$((CPUs/86400%365))
CPUh=$((CPUs/3600%24))
CPUm=$((CPUs/60%60))
CPUs=$((CPUs%60))
echo "<p>CPU time logged so far: ${CPUy}y ${CPUd}d ${CPUh}h ${CPUm}m ${CPUs}s</p>" >> $HTML_FILE

# Last WU sent out
lastWU=$(tac "$log" | grep -am1 "Sending workunit")
lWUarr=($lastWU)
lastWU=${lWUarr[8]}
lastWU=${lastWU##*_}
testdate=$(date +"%Y-%m-%d %H:%M:%S")
diffb="${lWUarr[1]} ${lWUarr[2]:0:8}"
diffs="$(($(date -d "$testdate" '+%s') - $(date -d "$diffb" '+%s')))"
if [ $diffs -gt 600 ]
  then
    echo "<h4 style=\"color: #ff0000\">Latest WU sent out at ${lWUarr[2]:0:8} local time: $lastWU</h4>" >> $HTML_FILE
  else
    echo "<p>Latest WU sent out at ${lWUarr[2]:0:8} local time: $lastWU</p>" >> $HTML_FILE
fi

# Check whether server is running
if ! pgrep -f -a nfs.py
  then
    echo "<p style=\"color: #ff0000\"><b>Attention! The server script could not be found in the process list! The server crashed most likely!</b></p>" >> $HTML_FILE
fi

# Link to Seth's version of the stats page
echo "<p><a href=\"Seth/\">Seth's status page</a></p>" >> $HTML_FILE

# Basic top tier list
echo "<h2>Rankings</h2>" >> $HTML_FILE
echo "<table style=\"width:200pt;text-align:center\">" >> $HTML_FILE
echo -e "<tr>\n<th>Place</th>\n<th>User</th>\n<th>Valid WUs</th>\n</tr>" >> $HTML_FILE
i=0;
head -n 20 refresher.log | grep 'workunits$' | while read line
  do
    linearr=($line)
    echo "</tr>" >> $HTML_FILE
    echo "<td>${i}.</td>" >> $HTML_FILE
    echo "<td>${linearr[0]}</td>" >> $HTML_FILE
    echo "<td>${linearr[1]:1}</td>" >> $HTML_FILE
    echo "<tr>" >> $HTML_FILE
    let i=i+1
  done
echo "</table>" >> $HTML_FILE

# Finalize HTML page
echo "</center>" >> $HTML_FILE
echo "</body>" >> $HTML_FILE
echo -n "</html>" >> $HTML_FILE

Last fiddled with by kruoli on 2021-11-17 at 20:03 Reason: Clarifications.
kruoli is online now   Reply With Quote
Old 2021-11-17, 20:59   #266
charybdis
 
charybdis's Avatar
 
Apr 2020

7·107 Posts
Default

Do rankings usually start at 0th place in Germany?

(nice work again btw)
charybdis is offline   Reply With Quote
Old 2021-11-17, 21:10   #267
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

104010 Posts
Default

Quote:
Originally Posted by charybdis View Post
Do rankings usually start at 0th place in Germany?

(nice work again btw)
xD
Au, weia! That is not my language (I love all of German language's quirks and anomalies, I'd like to post over my joy all week, but nobody likes such spam, but this is not something different in German). That is programmers language, I always think from 0 and languages where arrays start at index 1 are weird. Fixed.

(Thanks!)
kruoli is online now   Reply With Quote
Old 2021-11-18, 00:58   #268
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

22·1,151 Posts
Default

Quote:
Originally Posted by charybdis View Post
Do rankings usually start at 0th place in Germany?

(nice work again btw)
What?!? Did I miss seeing my "eFarm" in 1st place (in name only)?

I do wonder if the next Team Project will also be able to make use of these scripts, pages, etc. Not that I'm really ready to jump into another when this one is finished. I think a break to play elsewhere will be welcome.

I do like these additions as a little bit of relief from the boredom at this point in our effort.

Great work on all fronts. Thanks for all the effort all around.
EdH is offline   Reply With Quote
Old 2021-11-18, 07:31   #269
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

24×5×13 Posts
Default

We have had a major breakdown of computing power. Maybe some clients got stuck again? For me personally, two were stuck. The server has not crashed for a few days.

Yes, I would be willing to share my scripts and learning process doing this project. I am also working on releasing my client script that pins processes to cores. For larger machines, I saw significant improvements with this method. Especially on multi-chip CPUs.

The log is nearing half a GB in size. It is intriguing how we can parse it in less than three seconds!
kruoli is online now   Reply With Quote
Old 2021-11-18, 09:09   #270
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

24×5×13 Posts
Default

Quote:
Originally Posted by kruoli View Post
For me personally, two were stuck.
This was really weird. I pipe their output to a temporary file. It was not able to write to them anymore, although I could touch, rm, cat (to and from) the files. After I changed the file name, it worked.

I had msieve running over night with 1.16B relations for getting used to it. Maybe this data point is helpful still?
Code:
Thu Nov 18 04:00:18 2021  commencing linear algebra
Thu Nov 18 04:00:27 2021  read 87934102 cycles
Thu Nov 18 04:04:54 2021  cycles contain 297844075 unique relations
Thu Nov 18 04:24:22 2021  read 297844075 relations
Thu Nov 18 04:30:31 2021  using 20 quadratic characters above 4294917295
Thu Nov 18 04:41:02 2021  building initial matrix
Thu Nov 18 05:08:44 2021  memory use: 42248.2 MB
Thu Nov 18 05:09:02 2021  read 87934102 cycles
Thu Nov 18 05:09:12 2021  matrix is 87933921 x 87934102 (33193.4 MB) with weight 10167235282 (115.62/col)
Thu Nov 18 05:09:12 2021  sparse part has weight 7646253497 (86.95/col)
Thu Nov 18 05:18:17 2021  filtering completed in 2 passes
Thu Nov 18 05:18:29 2021  matrix is 87886893 x 87887072 (33190.0 MB) with weight 10165476782 (115.67/col)
Thu Nov 18 05:18:29 2021  sparse part has weight 7645921498 (87.00/col)
Thu Nov 18 05:20:14 2021  matrix starts at (0, 0)
Thu Nov 18 05:20:23 2021  matrix is 87886893 x 87887072 (33190.0 MB) with weight 10165476782 (115.67/col)
Thu Nov 18 05:20:23 2021  sparse part has weight 7645921498 (87.00/col)
Thu Nov 18 05:20:23 2021  saving the first 48 matrix rows for later
Thu Nov 18 05:20:29 2021  matrix includes 64 packed rows
Thu Nov 18 05:20:38 2021  matrix is 87886845 x 87887072 (32168.6 MB) with weight 8420654619 (95.81/col)
Thu Nov 18 05:20:38 2021  sparse part has weight 7553924701 (85.95/col)
Thu Nov 18 05:20:38 2021  using block size 8192 and superblock size 6291456 for processor cache size 65536 kB
Thu Nov 18 05:22:58 2021  commencing Lanczos iteration
Thu Nov 18 05:22:58 2021  memory use: 32862.9 MB
Then I killed it when displaying an ETA of over 17,000 hours.
kruoli is online now   Reply With Quote
Old 2021-11-18, 11:54   #271
charybdis
 
charybdis's Avatar
 
Apr 2020

7×107 Posts
Default

Quote:
Originally Posted by kruoli View Post
We have had a major breakdown of computing power. Maybe some clients got stuck again? For me personally, two were stuck. The server has not crashed for a few days.
My clients - and seemingly everyone else's - were unable to connect to the server for 3 hours last night, until about 07:10 your time. The logs show the same "Connection timed out" error as they do when the server goes down. Whatever it was, let's hope it doesn't happen again.

Quote:
Originally Posted by kruoli View Post
Code:
Thu Nov 18 05:20:38 2021  matrix is 87886845 x 87887072 (32168.6 MB) with weight 8420654619 (95.81/col)
Thu Nov 18 05:20:38 2021  sparse part has weight 7553924701 (85.95/col)
Thu Nov 18 05:20:38 2021  using block size 8192 and superblock size 6291456 for processor cache size 65536 kB
Thu Nov 18 05:22:58 2021  commencing Lanczos iteration
Thu Nov 18 05:22:58 2021  memory use: 32862.9 MB
Then I killed it when displaying an ETA of over 17,000 hours.
Nice to see we can at least produce a matrix. It is of course far too big, though the ETA wouldn't be quite so long if you weren't running msieve single-threaded! Was this with the default target_density?
charybdis is offline   Reply With Quote
Old 2021-11-18, 12:12   #272
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

24×5×13 Posts
Default

Quote:
Originally Posted by charybdis View Post
It is of course far too big, though the ETA wouldn't be quite so long if you weren't running msieve single-threaded! Was this with the default target_density?
This is true, but since I would not leave it running this way I started it single threaded. Yes, I simply let msieve decide on what to do with the data. For later, we can of course choose other parameters.
kruoli is online now   Reply With Quote
Old 2021-11-18, 12:53   #273
charybdis
 
charybdis's Avatar
 
Apr 2020

7·107 Posts
Default

You'll probably want to wait at least until a matrix can be built with target_density=120; I expect VBCurtis will chime in later with his thoughts. Chances are if you tried TD=120 now you would get "too few cycles, matrix probably cannot build".

How is the duplication rate looking?
charybdis is offline   Reply With Quote
Old 2021-11-18, 13:04   #274
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

24·5·13 Posts
Default

Wed Nov 17 19:18:16 2021 found 300859501 duplicates and 861473338 unique relations

That looks good, I would guess?
kruoli is online now   Reply With Quote
Old 2021-11-18, 14:38   #275
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

22×1,151 Posts
Default

I noticed the server had stopped serving when I checked my overnight machines at about 04:30 server time. The check message was red saying the last assignment was (I think) 04:11.

All but one of my clients was running fine this morning. Many had run the timeout reset a couple times. Some of my clients didn't have the new script and all of them but one, were running fine. The one was stuck, as before. All should be good now and I think "timeout" will be the answer to my query about gracefully quiting CADO-NFS. Thanks for that.

Are you going to employ MPI in Msieve LA? I'm not sure of your hardware setup, but I used mpi across two Xeons for a larger run a while back. It did save some time, but it was difficult to learn the nuances. charybdis was quite helpful with it. Unfortunately, I can't locate the posts about getting my setup to work best.
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Using 16e on smaller numbers fivemack Factoring 3 2017-09-19 08:52
NFS on smaller numbers? skan YAFU 6 2013-02-26 13:57
Bernoulli(200) c204 akruppa Factoring 114 2012-08-20 14:01
checking smaller number fortega Data 2 2005-06-16 22:48
Factoring Smaller Numbers marc Factoring 6 2004-10-09 14:17

All times are UTC. The time now is 19:58.


Sat Jul 2 19:58:36 UTC 2022 up 79 days, 17:59, 1 user, load averages: 0.85, 1.19, 1.18

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

≠ ± ∓ ÷ × · − √ ‰ ⊗ ⊕ ⊖ ⊘ ⊙ ≤ ≥ ≦ ≧ ≨ ≩ ≺ ≻ ≼ ≽ ⊏ ⊐ ⊑ ⊒ ² ³ °
∠ ∟ ° ≅ ~ ‖ ⟂ ⫛
≡ ≜ ≈ ∝ ∞ ≪ ≫ ⌊⌋ ⌈⌉ ∘ ∏ ∐ ∑ ∧ ∨ ∩ ∪ ⨀ ⊕ ⊗ 𝖕 𝖖 𝖗 ⊲ ⊳
∅ ∖ ∁ ↦ ↣ ∩ ∪ ⊆ ⊂ ⊄ ⊊ ⊇ ⊃ ⊅ ⊋ ⊖ ∈ ∉ ∋ ∌ ℕ ℤ ℚ ℝ ℂ ℵ ℶ ℷ ℸ 𝓟
¬ ∨ ∧ ⊕ → ← ⇒ ⇐ ⇔ ∀ ∃ ∄ ∴ ∵ ⊤ ⊥ ⊢ ⊨ ⫤ ⊣ … ⋯ ⋮ ⋰ ⋱
∫ ∬ ∭ ∮ ∯ ∰ ∇ ∆ δ ∂ ℱ ℒ ℓ
𝛢𝛼 𝛣𝛽 𝛤𝛾 𝛥𝛿 𝛦𝜀𝜖 𝛧𝜁 𝛨𝜂 𝛩𝜃𝜗 𝛪𝜄 𝛫𝜅 𝛬𝜆 𝛭𝜇 𝛮𝜈 𝛯𝜉 𝛰𝜊 𝛱𝜋 𝛲𝜌 𝛴𝜎𝜍 𝛵𝜏 𝛶𝜐 𝛷𝜙𝜑 𝛸𝜒 𝛹𝜓 𝛺𝜔