mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > YAFU

Reply
 
Thread Tools
Old 2020-11-18, 14:03   #1
aqua
 
Nov 2020

3 Posts
Default even more parallelisation

Hi,

I noticed that yafu calculates stuff in threads before sieving and when prints out thousands of numbers (func print_snfs ).
Is there a way to run a program instead of a thread in those cases too like ecm and lasieve ?
The reason is that a program with parameters can be easily spawned on another computers.
When you look into the source code you might think that author should be very annoyed by writing tons of code for string and file handling instead of doing very interesting parts. Why not break it apart (if its breakable)?
aqua is offline   Reply With Quote
Old 2020-11-18, 14:11   #2
LaurV
Romulan Interpreter
 
LaurV's Avatar
 
Jun 2011
Thailand

2×5×7×131 Posts
Default

It is breakable as it is. The "program" you need to run instead of threads, is called "yafu". You can split your run in a billion computers, just run yafu in each, with suitable parameters. If you have the computers and don't know how to do it, either RTFM, or send the computers to us, and we'll do it for you
LaurV is online now   Reply With Quote
Old 2020-11-18, 14:19   #3
aqua
 
Nov 2020

310 Posts
Default

LaurV,

Thank you for your reply! Do you mean it is possible to spawn more yafus on different computers to effectively factorize same number ?
This day was spent by doing a spawning for lasieves and ecms on another computer, but ... there is a better way... oh no.. :)
I think I do read the documentation, but it requires understanding of math behind all of that. It is slower to understand.
aqua is offline   Reply With Quote
Old 2020-11-18, 14:37   #4
bsquared
 
bsquared's Avatar
 
"Ben"
Feb 2007

1101001000112 Posts
Default

Quote:
Originally Posted by aqua View Post
LaurV,

Thank you for your reply! Do you mean it is possible to spawn more yafus on different computers to effectively factorize same number ?
This day was spent by doing a spawning for lasieves and ecms on another computer, but ... there is a better way... oh no.. :)
I think I do read the documentation, but it requires understanding of math behind all of that. It is slower to understand.
Out of curiosity, how big is your number?

More yafu's can be spawned, but toward the end of that process there will be some fiddly work that must be done by hand (recombining files, testing if you are done, rinsing and repeating). There is also non-trivial setup work to do to make sure the work is compatible (making sure one job file is used everywhere), and that each process is started with the appropriate flags/parameters so you don't duplicate work. If you are thinking of going to this level, you are probably better off using the individual tools manually.

The benefit of yafu is that you can fire and forget on one computer. The drawback is that that approach runs out of gas at a certain point, depending on your patience and hardware. If you want to go further then you either wait longer with yafu or learn how to run the 3rd party tools manually (lasieve, msieve, and/or CADO).

Last fiddled with by bsquared on 2020-11-18 at 14:38
bsquared is offline   Reply With Quote
Old 2020-11-18, 14:39   #5
LaurV
Romulan Interpreter
 
LaurV's Avatar
 
Jun 2011
Thailand

2·5·7·131 Posts
Default

Quote:
Originally Posted by aqua View Post
Do you mean it is possible to spawn more yafus on different computers to effectively factorize same number ?
Yes, that's usually how we collaborate to factor those large 200 digit numbers, but it is not easy. Sieving part is the most time-consuming, about 85% of the time, as Curtis replied in your other thread, and this is usually parallelized in multiple computers, possibly by just one or many forum members who contribute to factorization. Each computer or yafu instance can sieve a different Q-range using the same poly, get its own millions of relations, then you put all together and run LA in a big cluster (LA can't be easily parallelized, but that is not more than 10-15 percent of the effort).

Edit: crosspost with bb, he is the developer, so trust what he says better than what I say
@bb: he mentioned in the other thread that he tries to factor 384 and 512 bit numbers (probably, some illegal low-level RSA cracking, games or something )

Last fiddled with by LaurV on 2020-11-18 at 14:46
LaurV is online now   Reply With Quote
Old 2020-11-18, 16:59   #6
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

11×421 Posts
Default

The manual effort to make sure each computer is doing separate Q-ranges, and then combining the files and running the post-processing manually is rather tedious, and usually not used for jobs that take a month or less; we don't view the manual labor as worth it compared to just being patient.

However, the CADO package is set up to handle this automatically- one machine is designated the CADO-server, and any number of other machines can be run as clients for the poly select and sieving steps. The CADO documentation is pretty good for directions, though the help files are somewhat hidden in various folders (for instance, the detailed explanation of the job parameter options is in ./parameters/factor/c90.params

The catch to CADO, which is an alternative package to YAFU, is that it's linux-only. If you run Linux, it is likely faster than YAFU, and the individual sub-programs are actively developed. The external binaries YAFU calls are no longer developed (msieve for poly select and post processing, lasieve for sieving).
VBCurtis is offline   Reply With Quote
Old 2020-11-18, 17:07   #7
kruoli
 
kruoli's Avatar
 
"Oliver"
Sep 2017
Porta Westfalica, DE

5×83 Posts
Default

With Windows 10, CADO can be run in WSL or WSL2. I already used it that way. So I'd vote for at least giving it a try!

With WSL2, the server-client-feature for multiple machines is a bit more complicated to set up, but still far from imnpossible. So if not needed, you should use the "normal" WSL version 1.

Last fiddled with by kruoli on 2020-11-18 at 17:07 Reason: Too much words.
kruoli is online now   Reply With Quote
Reply

Thread Tools


All times are UTC. The time now is 09:30.

Thu Jan 28 09:30:19 UTC 2021 up 56 days, 5:41, 0 users, load averages: 1.29, 1.77, 2.09

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.