Inefficient behaviour in yafu when doing large NFS with lots of threads

[2147483647]

I'm using yafu to do SNFS on some large numbers (about 200 digits) with 48 threads for sieving, and I've noticed that yafu starts 48 sievers but then doesn't start any more until all 48 have finished. This is a problem because some sievers finish after 5 minutes, almost all of them finish after about 7 minutes, but there are usually 1 or 2 processes that don't finish until about 10 minutes. During these 3 minutes, only these 1 or 2 processes are actually doing anything, and the other cpu cores are not being used at all, so the sieving is taking much longer than it should do.

Has anyone else dealt with this issue? Would this be simple to fix?

[xilman]

Originally Posted by 2147483647

Has anyone else dealt with this issue? Would this be simple to fix?

FactMsieve has the same problem (I don't use yafu myself) so for big factorizations I use a home-grown client-server harness called cabald/cabalc. The name is a relic of a collaboration running in the 90's when RSA-140 was the bleeding edge. It has been used elsewhere, most recently (AFAIK) at EPFL for part of the RSA-768 factorization

One machine runs a cabald daemon which hands out tasks to an arbitrarily large number of cabalc clients. An arbitrary number of cabalc clients can run on each machine, including that which runs cabald.

The harness is very simple and handles only the allocation and processing of sieving tasks. All other duties are up to you, including polynomial finding and the consolidation of relation files for subsequent linear algebra and square root extraction. I normally use factMsieve.pl for all those tasks.

Source code, basic documentation and very limited hand-holding available on request. You are assumed to know what you are doing when running a NFS factorization so assistance is limited to the cabald{c,d} software alone.

Another alternative is CADO, which is a much more professional and powerful client/server version of NFS. It is also complex to drive and the siever is markedly slower than ggnfs. However, once set up it gives a fire-and-forget interface more akin to yafu and factMsieve.pl in that very little human attention is required.

[bsquared]

Originally Posted by 2147483647

I'm using yafu to do SNFS on some large numbers (about 200 digits) with 48 threads for sieving, and I've noticed that yafu starts 48 sievers but then doesn't start any more until all 48 have finished. This is a problem because some sievers finish after 5 minutes, almost all of them finish after about 7 minutes, but there are usually 1 or 2 processes that don't finish until about 10 minutes. During these 3 minutes, only these 1 or 2 processes are actually doing anything, and the other cpu cores are not being used at all, so the sieving is taking much longer than it should do.

Has anyone else dealt with this issue? Would this be simple to fix?

It is a known issue, yes, and I have a fix planned. Not sure when I'll get around to it but hopefully soon.

[yoyo]

Some of my users blaming the same that many cores are idle.