mersenneforum.org > EdH How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines
 Register FAQ Search Today's Posts Mark Forums Read

 2018-02-25, 18:00 #1 EdH     "Ed Hall" Dec 2009 Adirondack Mtns 64208 Posts How I Run a Larger Factorization Via CADO-NFS on Several Ubuntu Machines (Note: I expect to keep the first post of each of these "How I..." threads up-to-date with the latest version. Please read the rest of each thread to see what may have led to the current set of instructions.) This thread will explain the steps I use to run CADO-NFS on several computers which are already running Ubuntu and have CADO-NFS installed* and tested per: How I Install CADO-NFS on my Ubuntu Machines This procedure should work for other linux distributions as well but the only other one I've currently tested is Debain. In my case I needed to install python3 from the repository on my Debian based machines . More details about CADO-NFS and my early experiments can be found in the CADO-NFS thread. *In this instance "install" is referring to the acquiring and compilation of the CADO-NFS package only. The scripts will have to be called using their respective paths. I will be referencing the folder Math/cado-nfs as described in the installation routine above. Make any local changes for your computer. Determine which machine will be the server and which will be clients. Open a terminal on each, move to the appropriate folder and test each with the suggested run from the README file: Code: cd Math/cado-nfs ./cado-nfs.py 90377629292003121684002147101760858109247336549001090677693 For me, the final results are: Code: Info:root: Using default parameter file ./parameters/factor/params.c60 Info:root: No database exists yet Info:root: Created temporary directory /tmp/cado.3nu5mplc Info:Database: Opened connection to database /tmp/cado.3nu5mplc/c60.db Info:root: Set tasks.threads=8 based on detected logical cpus Info:root: tasks.polyselect.threads = 2 Info:root: tasks.sieve.las.threads = 2 Info:root: slaves.scriptpath is /home/math79/Math/cado-nfs Info:root: Command line parameters: ./cado-nfs.py 90377629292003121684002147101760858109247336549001090677693 ... Info:HTTP server: Shutting down HTTP server Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 55.2/37.7927 Info:root: Cleaning up computation data in /tmp/cado.3nu5mplc 260938498861057 760926063870977 773951836515617 588120598053661 Now, let's choose a slightly larger composite: Code: 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 The machine we are going to use as the server could factor this number without too much trouble, but the completion can be sped up by adding clients. In order to add clients, some extra options will need to be added to the command line on the server machine. There will also need to be some extra options for our clients, too, but for now let's work with the server. For my setup, I am using ssh and all my machines are able to connect with each other already so I don't use the --server option. However, ssh isn't necessary or described here. If interested, ssh has many pages on the web. For now, we'll use the --server option for these steps. I prefer to let the default files, that are based on digit size, be called, rather than constructing my own, so the composite will be entered on the command line rather than within a parameter file. A necessary option to add clients is server.whitelist. The whitelist will need to include the IP of any possible client. This can be individual or in a masked manner. By masked manner, I am referring to 192.168.1.0/24 meaning any IP from 192.168.1.0 through 192.168.1.255. With the bare minimum from above provided, you should be able to start the server running with the command: Code: ./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 --server The following should be returned: Code: ./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 --server Info:root: Using default parameter file ./parameters/factor/params.c95 Info:root: No database exists yet Info:root: Created temporary directory /tmp/cado.4thcy1o_ Info:Database: Opened connection to database /tmp/cado.4thcy1o_/c95.db Info:root: Set tasks.threads=8 based on detected logical cpus Info:root: tasks.polyselect.threads = 2 Info:root: tasks.sieve.las.threads = 2 Info:root: Command line parameters: ./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 --server Info:root: If this computation gets interrupted, it can be resumed with ./cado-nfs.py /tmp/cado.4thcy1o_/c95.parameters_snapshot.0 Info:HTTP server: Using non-threaded HTTPS server Info:HTTP server: Using whitelist: 192.168.1.0/24 Info:Complete Factorization: Factoring 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 Info:HTTP server: serving at https://math79:42668 (0.0.0.0) Info:HTTP server: For debugging purposes, the URL above can be accessed if the server.only_registered=False parameter is added Info:HTTP server: You can start additional cado-nfs-client.py scripts with parameters: --server=https://math79:42668 --certsha1=c2ee63f9850b3e23d0707a760762f39a83f48628 Info:HTTP server: If you want to start additional clients, remember to add their hosts to server.whitelist ... If the server machine was left alone, it would fully factor the composite all by itself: Code: Info:HTTP server: Shutting down HTTP server Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 6014.18/949.3 Info:root: Cleaning up computation data in /tmp/cado.nprm037c 179231227423414197451601378315047105853969879 11022834899950977366949652871606409040980556071039 But, since this is supposed to show how I use multiple machines, I'm going to add two clients after a brief mention of two other options: Check out the two bold sections of the following portion of the preceding run: Code: Info:HTTP server: You can start additional cado-nfs-client.py scripts with parameters: --server=https://math79:42668 --certsha1=c2ee63f9850b3e23d0707a760762f39a83f48628 The first to look at is the port address (42668). In this case, the number was chosen by the server. When you run the clients, you need to know this number. This means you'll have to wait until the server is running and try to catch this information as it scrolls by. Alternatively, you can assign a number of your own choosing, which is my preferred method. To do this add server.port=. For the next example I'll use 56789. The second option to look at is the --certsha1=... This is created by the server and can be used to validate clients. However, you need to catch this as it scrolls to use it in your client commands. If only machines from your local net are in use you may wish to decline using the ssl certificate validation. This can be accomplished with the option server.ssl=no. This is what I will use in the following example. Let's start the server with all the options discussed and see what it shows: Code: ./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 server.port=56789 server.ssl=no --server Info:root: Using default parameter file ./parameters/factor/params.c95 Info:root: No database exists yet Info:root: Created temporary directory /tmp/cado.e7kdh_gs Info:Database: Opened connection to database /tmp/cado.e7kdh_gs/c95.db Info:root: Set tasks.threads=8 based on detected logical cpus Info:root: tasks.polyselect.threads = 2 Info:root: tasks.sieve.las.threads = 2 Info:root: Command line parameters: ./cado-nfs.py 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 server.whitelist=192.168.1.0/24 server.port=56789 server.ssl=no --server Info:root: If this computation gets interrupted, it can be resumed with ./cado-nfs.py /tmp/cado.e7kdh_gs/c95.parameters_snapshot.0 Info:HTTP server: Using non-threaded HTTP server Info:HTTP server: Using whitelist: 192.168.1.0/24 Info:Complete Factorization: Factoring 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 Info:HTTP server: serving at http://math79:56789 (0.0.0.0) Info:HTTP server: For debugging purposes, the URL above can be accessed if the server.only_registered=False parameter is added Info:HTTP server: You can start additional cado-nfs-client.py scripts with parameters: --server=http://math79:56789 Info:HTTP server: If you want to start additional clients, remember to add their hosts to server.whitelist ... Now that the server is running, let's move to the clients. There's a line above that reads: Code: Info:HTTP server: You can start additional cado-nfs-client.py scripts with parameters: --server=http://math79:56789 It's kind of right, but we'll actually have to add a little to the address to finish it. There is also one more option that is important if you are using clients with different hardware from your server; --bindir. This option tells your client to use its own programs rather than download them from the server. If you used the standard compilation of the programs, they will be in cado-nfs/build/. Here is the command line I will use on both of the two clients. This line can be used on all your clients by changing the bindir path appropriately: Code: ./cado-nfs-client.py --server=http://math79.local:56789 --bindir=build/math77/ I've highlighted the .local above, to explain that if the clients are on the same local network as the server, so you can use .local to connect. You can also connect via the server's IP: @192..., which is what you'll need to use if connecting from outside the local network. Note: The above line starts one client with two threads on its client machine. If you have more threads available start more clients wih the same command line. My three machines all have 8 threads, so I will be starting four clients on each client machine using the above command in four separate terminals. Each client should show something like: Code: INFO:root:Starting client math77.4b87c3d8 INFO:root:Python version is 2.7.12 INFO:root:Downloading http://math79.local:56789/cgi-bin/getwu?clientid=math77.4b87c3d8 to download/WU.math77.4b87c3d8 (cafile = None) INFO:root:Result file math77.4b87c3d8.work/c95.polyselect1.40000-50000 does not exist INFO:root:Running 'build/math77/polyselect/polyselect' -P 10000 -N 1975636228803860706131861386351317508435774072460176838764200263234956507563682801432890234281 -degree 4 -t 2 -admin 40000 -admax 50000 -incr 60 -nq 256 > 'math77.4b87c3d8.work/c95.polyselect1.40000-50000' INFO:root:Attaching file math77.4b87c3d8.work/c95.polyselect1.40000-50000 to upload INFO:root:Sending result for workunit c95_polyselect1_40000-50000 to http://math79.local:56789/cgi-bin/upload.py INFO:root:Cleaning up for workunit c95_polyselect1_40000-50000 ... Eventually, they will show something like: Code: INFO:root:Running 'build/math77/sieve/las' -I 13 -poly 'download/c95.poly' -q0 54000 -q1 56000 -lim0 3660220 -lim1 2758600 -lpb0 23 -lpb1 23 -mfb0 22 -mfb1 45 -ncurves0 2 -ncurves1 13 -fb 'download/c95.roots.gz' -out 'math77.4b87c3d8.work/c95.54000-56000.gz' -t 2 -stats-stderr INFO:root:Attaching file math77.4b87c3d8.work/c95.54000-56000.gz to upload INFO:root:Attaching stderr for command 0 to upload INFO:root:Sending result for workunit c95_sieving_54000-56000 to http://math79.local:56789/cgi-bin/upload.py INFO:root:Cleaning up for workunit c95_sieving_54000-56000 After enough sieving has been reported, the server will move to the Linear Algebra work and the clients will be told they are no longer needed: Code: Info:Lattice Sieving: Cancelling remaining workunits Info:Filtering - Singleton removal: Total cpu/real time for purge: 7.67/2.33534 Info:Filtering - Merging: Starting Info:Filtering - Merging: Merged matrix has 85097 rows and total weight 14466502 (170.0 entries per row on average) Info:Filtering - Merging: Total cpu/real time for merge: 22.66/17.3496 Info:Filtering - Merging: Total cpu/real time for replay: 2.91/2.44187 Info:Linear Algebra: Starting Code: INFO:root:Client finishing: Received 410 from server. Bye. And, finally, the server completes the rest on its own: Code: Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 5273.74/470.649 Info:root: Cleaning up computation data in /tmp/cado.o8quwcb4 179231227423414197451601378315047105853969879 11022834899950977366949652871606409040980556071039 Now, let's compare times for factoring between one machine and three of the same architecture. Single machine: Code: Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 6014.18/949.3 One server and two client machines. Each machine actually ran four clients, each client with two threads: Code: Info:Complete Factorization: Total cpu/elapsed time for entire factorization: 5273.74/470.649 With three machines, it took roughly half as long to factor the 94 digt composite. At first look, you may wonder why it didn't take one-third as long. The reason is mostly due to the fact that the Linear Algebra steps are only performed on the server. Overhead also plays a small role. For further details on some of my testing, specifically the factorization of RSA-130 see the above referenced thread on CADO-NFS, starting with post 137. Last fiddled with by EdH on 2018-03-17 at 16:08

 Similar Threads Thread Thread Starter Forum Replies Last Post EdH EdH 7 2019-08-21 02:26 EdH EdH 3 2019-06-24 03:42 EdH EdH 12 2019-04-16 09:28 EdH EdH 0 2018-02-21 23:48 akruppa Factoring 14 2008-09-18 23:52

All times are UTC. The time now is 20:41.

Fri Sep 18 20:41:28 UTC 2020 up 8 days, 17:52, 1 user, load averages: 1.58, 1.65, 1.74