even more parallelisation

[aqua]

Hi,

I noticed that yafu calculates stuff in threads before sieving and when prints out thousands of numbers (func print_snfs ).
Is there a way to run a program instead of a thread in those cases too like ecm and lasieve ?
The reason is that a program with parameters can be easily spawned on another computers.
When you look into the source code you might think that author should be very annoyed by writing tons of code for string and file handling instead of doing very interesting parts. Why not break it apart (if its breakable)?

[LaurV]

It is breakable as it is. The "program" you need to run instead of threads, is called "yafu". You can split your run in a billion computers, just run yafu in each, with suitable parameters. If you have the computers and don't know how to do it, either RTFM, or send the computers to us, and we'll do it for you

[aqua]

LaurV,

Thank you for your reply! Do you mean it is possible to spawn more yafus on different computers to effectively factorize same number ?
This day was spent by doing a spawning for lasieves and ecms on another computer, but ... there is a better way... oh no.. :)
I think I do read the documentation, but it requires understanding of math behind all of that. It is slower to understand.

[bsquared]

Originally Posted by aqua

LaurV,

Thank you for your reply! Do you mean it is possible to spawn more yafus on different computers to effectively factorize same number ?
This day was spent by doing a spawning for lasieves and ecms on another computer, but ... there is a better way... oh no.. :)
I think I do read the documentation, but it requires understanding of math behind all of that. It is slower to understand.

Out of curiosity, how big is your number?

More yafu's can be spawned, but toward the end of that process there will be some fiddly work that must be done by hand (recombining files, testing if you are done, rinsing and repeating). There is also non-trivial setup work to do to make sure the work is compatible (making sure one job file is used everywhere), and that each process is started with the appropriate flags/parameters so you don't duplicate work. If you are thinking of going to this level, you are probably better off using the individual tools manually.

The benefit of yafu is that you can fire and forget on one computer. The drawback is that that approach runs out of gas at a certain point, depending on your patience and hardware. If you want to go further then you either wait longer with yafu or learn how to run the 3rd party tools manually (lasieve, msieve, and/or CADO).

[LaurV]

Originally Posted by aqua

Do you mean it is possible to spawn more yafus on different computers to effectively factorize same number ?

Yes, that's usually how we collaborate to factor those large 200 digit numbers, but it is not easy. Sieving part is the most time-consuming, about 85% of the time, as Curtis replied in your other thread, and this is usually parallelized in multiple computers, possibly by just one or many forum members who contribute to factorization. Each computer or yafu instance can sieve a different Q-range using the same poly, get its own millions of relations, then you put all together and run LA in a big cluster (LA can't be easily parallelized, but that is not more than 10-15 percent of the effort).

Edit: crosspost with bb, he is the developer, so trust what he says better than what I say
@bb: he mentioned in the other thread that he tries to factor 384 and 512 bit numbers (probably, some illegal low-level RSA cracking, games or something )

[VBCurtis]

The manual effort to make sure each computer is doing separate Q-ranges, and then combining the files and running the post-processing manually is rather tedious, and usually not used for jobs that take a month or less; we don't view the manual labor as worth it compared to just being patient.

However, the CADO package is set up to handle this automatically- one machine is designated the CADO-server, and any number of other machines can be run as clients for the poly select and sieving steps. The CADO documentation is pretty good for directions, though the help files are somewhat hidden in various folders (for instance, the detailed explanation of the job parameter options is in ./parameters/factor/c90.params

The catch to CADO, which is an alternative package to YAFU, is that it's linux-only. If you run Linux, it is likely faster than YAFU, and the individual sub-programs are actively developed. The external binaries YAFU calls are no longer developed (msieve for poly select and post processing, lasieve for sieving).

[kruoli]

With Windows 10, CADO can be run in WSL or WSL2. I already used it that way. So I'd vote for at least giving it a try!

With WSL2, the server-client-feature for multiple machines is a bit more complicated to set up, but still far from imnpossible. So if not needed, you should use the "normal" WSL version 1.