3,748+ c204 Smaller-but-Needed

[EdH]

Excellent! I was getting ready to post a three column version today, but no extra as you've done. I like the trimming. I've also been playing around with adding a unique clients count per day, but not very aggressively. I'm not really sure it would add anything. I see you changed the 400 to 1000 for comparison. I've been wondering if 400 might be better increased.

[kruoli]

Originally Posted by EdH

I've also been playing around with adding a unique clients count per day, but not very aggressively. I'm not really sure it would add anything.

This would likely cause really weird results because I had some clients running with --single in a loop and not having --clientid set. So I have hundreds of "unique" clients.

[EdH]

Originally Posted by kruoli

This would likely cause really weird results because I had some clients running with --single in a loop and not having --clientid set. So I have hundreds of "unique" clients.

I had thought about that as well. My machines are all running --single, but I provide an ID. OTOH, all those unique IDs would show a lot of interest at that time!

[VBCurtis]

Originally Posted by EdH

Excellent! I was getting ready to post a three column version today, but no extra as you've done. I like the trimming. I've also been playing around with adding a unique clients count per day, but not very aggressively. I'm not really sure it would add anything. I see you changed the 400 to 1000 for comparison. I've been wondering if 400 might be better increased.

When I requested 100 and 400, I had dropped a zero in my mental conversion from q-range to number of workunits.
Glad you changed 400 to 1000.
Comparing current yield to overall-job-yield tells us most of the relevant info about how yield is falling off, anyway.

[EdH]

Originally Posted by VBCurtis

When I requested 100 and 400, I had dropped a zero in my mental conversion from q-range to number of workunits.
Glad you changed 400 to 1000.
Comparing current yield to overall-job-yield tells us most of the relevant info about how yield is falling off, anyway.

Credit Oliver with the change. Should it be even greater?

Curiosity: I have a Colab instance running ATM:

INFO:root:Running /content/cado-nfs/build/e202773b5d6e/sieve/las
 -poly download/3_748plus.poly -q0 356442000 -A 30 -q1 356444000 -lim0 500000000 -lim1 300000000
 -lpb0 33 -lpb1 34 -mfb0 66 -mfb1 98 -ncurves0 25 -ncurves1 20 -fb1 download/3_748plus.roots1.gz
 -out Colab.eFarm.work/3_748plus.356442000-356444000.gz -t 2 -adjust-strategy 2 -stats-stderr
INFO:root:[Mon Nov  8 16:36:57 2021] Subprocess has PID 12961

The resources show only 2.14GB of 12.69 available, in use. I didn't think RAM use was tied to threads, or at least not too closely. Something odd here?

[VBCurtis]

Originally Posted by EdH

Credit Oliver with the change. Should it be even greater?
I didn't think RAM use was tied to threads, or at least not too closely. Something odd here?

I think 1000 and 10000 would be better than 400 and 1000; but it may be that yield just doesn't change much even over 10000 workunits, and just one "most recent yield" calculation has any value.

As for RAM: Seems odd to me. My guess is that CADO doesn't "see" all the available ram due to some colab-specific setting, and is running in minimal-ram configuration. My production runs have used 4,6,8,10 threads per client, and all have similar memory use.

[kruoli]

Originally Posted by VBCurtis

I think 1000 and 10000 would be better than 400 and 1000 [...].

The values are now at 1000 and 25000. If this is too large I can change it back.

Originally Posted by VBCurtis

[...] all have similar memory use.

I saw a small increase of memory footprint when increasing the thread count. I also had a test with 16 threads that was slower than a single worker with 8 cores on a machine with 16 physical cores.

[charybdis]

Also mystified by the low RAM usage on Colab. It is possible to get CADO to use less memory using -bkthresh1, but this comes at a huge cost in speed, and I don't think it ever gets triggered automatically. Are you sure you didn't catch the RAM usage during initialization when las hadn't started sieving yet?

Also I see the average relations per WU figure has gone up from ~5000 to ~6000. Is somebody running clients with A=31?

[VBCurtis]

Originally Posted by charybdis

Also I see the average relations per WU figure has gone up from ~5000 to ~6000. Is somebody running clients with A=31?

Not guilty on A=31.

Did the yield jump right at Q=300M?

[EdH]

The Colab instances are running rather slow at around 40 minutes per WU, but they are limited to two threads:

INFO:root:Running /content/cado-nfs/build/e202773b5d6e/sieve/las -poly download/3_748plus.poly -q0 362096000 -A 30 -q1 362098000 -lim0 500000000 -lim1 300000000 -lpb0 33 -lpb1 34 -mfb0 66 -mfb1 98 -ncurves0 25 -ncurves1 20 -fb1 download/3_748plus.roots1.gz -out Colab.eFarm.work/3_748plus.362096000-362098000.gz -t 2 -adjust-strategy 2 -stats-stderr
INFO:root:[Tue Nov  9 00:01:07 2021] Subprocess has PID 18277
. . .
INFO:root:Running /content/cado-nfs/build/e202773b5d6e/sieve/las -poly download/3_748plus.poly -q0 362614000 -A 30 -q1 362616000 -lim0 500000000 -lim1 300000000 -lpb0 33 -lpb1 34 -mfb0 66 -mfb1 98 -ncurves0 25 -ncurves1 20 -fb1 download/3_748plus.roots1.gz -out Colab.eFarm.work/3_748plus.362614000-362616000.gz -t 2 -adjust-strategy 2 -stats-stderr
INFO:root:[Tue Nov  9 00:40:25 2021] Subprocess has PID 18749

As you can see, las was started at 00:40:25 and I took the following check at 00:50:40:

Waiting to finish the current execution.
Python 3 Google Compute Engine backend

RAM: 3.16 GB/12.69 GB Disk: 46.70 GB/107.72 GB

I just did another check at 00:55:42 and it was identical.

Wouldn't " -bkthresh1" be seen in the invocation for las, if it was used?

[charybdis]

Aha, I think I've got it. I noticed that the averages seemed to be stuck. And indeed:

Originally Posted by kruoli

lastarr=($(cat "$relationCountsLogFile")) <-- should be "tac"?

...

lastarrsize=${#lastarr[*]}
while [ $avcount -lt $lastarrsize ]
  do
    let avsum=${avsum}+${lastarr[$avcount]}
    let avcount=${avcount}+1
    if [ $avcount -eq $smallCount ]
      then
        let avsmall=${avsum}/$smallCount
    fi
    if [ $avcount -eq $bigCount ]
      then
        let avbig=${avsum}/$bigCount
    fi
  done
let avtotal=$avsum/$avcount

Looks an awful lot like it's averaging the first 1000 WUs rather than the last 1000...