Resume linear algebra

Timic · 2018-01-21, 14:35

I used Brian Gladman's python script which uses GGNFS and MSIEVE to factor 155 digit number, but it threw errors when it came to linear algebra step (I assumed it was because of RAM), so I moved it to another machine with more RAM.
When I ran the script there, it started sieving from beginning (so the .dat file was removed).

Do I have to run some command from cmd to resume linear algebra step?

By the way, I followed this guide.

chris2be8 · 2018-01-21, 17:09

If you still have the .dat file somewhere, eg on the original machine, you can probably restart linear algebra (you may have to re-build the matrix though). If you have lost the .dat file you will have to re-run the whole job. Make a backup copy of the .dat file (and everything else in the directory) before you start experimenting though.

Can you post the log from the failing run so we can see what went wrong. Also a directory listing of where you were running it.

You probably need to re-run the msieve step in the log, replacing -nc2 with -ncr, if you have all the files from the failing attempt at linear algebra. If that fails try running msieve with -nc to rebuild the matrix.

I assume you are running Windows from your mention of running things under cmd. What version is it though?

Chris

VictordeHolland · 2018-01-21, 18:32

Without the relations and the matrix you can't finish the factorization. You need the matrix to resume linear algebra (-ncr to resume or -nc2 to rebuild) and the relations in the sqr fase (-nc3).

Dubslow · 2018-01-21, 19:25

The important part is that -ncr is required to resume. Without -ncr you will overwrite the data, sadly. More than one forum user has lost an in progress matrix in this manner before

EdH · 2018-01-21, 20:04

But, if he didn't rerun on the original machine, the relations should still exist there. The OP would also need the original poly file, or fb, IIRC. On the original machine, there should be a compressed file with all the original relations, too.

Or, is my memory really bad, or the comparison between Win and linux too far removed?

VictordeHolland · 2018-01-21, 21:19

The dir should look something like this on Windows (note this is an old dir)

Code:

18-04-2016  18:47    12.761.511.205 C204_119_99.dat.gz
30-12-2015  22:47            99.223 Changes
31-12-2015  22:45           864.110 libmsieve.a
11-09-2015  01:38            56.978 libwinpthread-1.dll
31-12-2015  22:45         3.175.900 msieve-svn988-x64-sandybridge.exe
22-04-2016  01:13    23.917.535.851 msieve.dat
29-04-2016  06:01       570.056.176 msieve.dat.bak.chk
29-04-2016  07:03       570.056.176 msieve.dat.chk
23-04-2016  00:51       509.858.000 msieve.dat.cyc
29-04-2016  07:36       114.010.464 msieve.dat.dep
23-04-2016  00:51     6.132.670.652 msieve.dat.mat
18-04-2016  15:21               429 msieve.fb
18-04-2016  15:21               502 msieve.poly
11-09-2014  14:01           185.976 pthreadGC2.dll
11-09-2014  14:01            82.944 pthreadVC2.dll
30-12-2015  22:47            25.290 Readme
30-12-2015  22:47            57.946 Readme.nfs
30-12-2015  22:47             7.987 Readme.qs
18-04-2016  15:21               205 worktodo.ini
17-08-2015  17:19            90.250 zlib1.dll

Timic · 2018-01-22, 15:43

Hi, I'm sorry for the late reply.

The problem is, this script ran on a really bad virtual machine, which has 4GB RAM and 4 core CPU with 1 thread each. Windows already used 50% of RAM so I guess msieve didn't have enough memory to complete linear algebra. I have Windows 2012 R2 by the way.

So, the minute I stopped the linear algebra step, I saved everything. I attached 2 pictures that show all files (it's zipped, so I couldn't ls it through cmd).

The full log is in attachments.

wombatman · 2018-01-22, 15:54

What RSA number are you trying to break?

Timic · 2018-01-22, 17:09

This one:
1772352223689553499523382657591107586734440334822954296365208356693849378633858324876899946660923527875550194354495596668114847388151261943936272297881447 (154 digits)

chris2be8 · 2018-01-22, 17:10

I only use Linux, so I'm guessing a bit. But try the following after extracting the files onto another system:
msieve -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -ncr

If that fails try restarting -nc2 from scratch:
msieve -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -nc2

If either of those worked then run:
msieve -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -nc3

But if they both failed try rebuilding the matrix (this will try to run all msieve steps in one go):
msieve -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -nc

Obviously keep the backup zip files safe until you have factored the number.

If you run:
msieve -h
you will get a brief summary of what options msieve accepts. That should help you work out what it wants (eg after -l put the name of the log file.

Chris

Timic · 2018-01-22, 18:11

I ran (added verbose too)
msieve -v -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -ncr
and it threw an error: cannot open matrix checkpoint file, so I guess that won't work. Tried the -nc2, it built the matrix and is now performing linear algebra, BUT I get these errors:

lanczos halted after x iterations (dim = x)
linear algebra failed; retrying
memory use: 868.8 MB
lanczos error: submatrix is not invertible

Those were the same errors as on the virtual machine, I thought they were because of memory, but I have 16 GB of RAM here...

Is this normal? By the way, I'm using the same .dat and .dat.cyc.

2018-01-21, 14:35	#1
Timic Jan 2018 11 Posts	Resume linear algebra I used Brian Gladman's python script which uses GGNFS and MSIEVE to factor 155 digit number, but it threw errors when it came to linear algebra step (I assumed it was because of RAM), so I moved it to another machine with more RAM. When I ran the script there, it started sieving from beginning (so the .dat file was removed). Do I have to run some command from cmd to resume linear algebra step? By the way, I followed this guide.

2018-01-21, 17:09	#2
chris2be8 Sep 2009 2439₁₀ Posts	If you still have the .dat file somewhere, eg on the original machine, you can probably restart linear algebra (you may have to re-build the matrix though). If you have lost the .dat file you will have to re-run the whole job. Make a backup copy of the .dat file (and everything else in the directory) before you start experimenting though. Can you post the log from the failing run so we can see what went wrong. Also a directory listing of where you were running it. You probably need to re-run the msieve step in the log, replacing -nc2 with -ncr, if you have all the files from the failing attempt at linear algebra. If that fails try running msieve with -nc to rebuild the matrix. I assume you are running Windows from your mention of running things under cmd. What version is it though? Chris Last fiddled with by chris2be8 on 2018-01-21 at 17:14 Reason: Clarify to back up every related file.

2018-01-21, 18:32	#3
VictordeHolland "Victor de Hollander" Aug 2011 the Netherlands 3²×131 Posts	Without the relations and the matrix you can't finish the factorization. You need the matrix to resume linear algebra (-ncr to resume or -nc2 to rebuild) and the relations in the sqr fase (-nc3). Last fiddled with by VictordeHolland on 2018-01-21 at 18:33

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
restarting nfs linear algebra	cubaq	YAFU	2	2017-04-02 11:35
Linear algebra at 600%	CRGreathouse	Msieve	8	2009-08-05 07:25
Linear algebra crashes	10metreh	Msieve	3	2009-02-02 08:34
Linear algebra proof	Damian	Math	8	2007-02-12 22:25
Linear algebra in MPQS	R1zZ1	Factoring	2	2007-02-02 06:45

2018-01-21, 19:25	#4
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 3×29×83 Posts	The important part is that -ncr is required to resume. Without -ncr you will overwrite the data, sadly. More than one forum user has lost an in progress matrix in this manner before

2018-01-21, 20:04	#5
EdH "Ed Hall" Dec 2009 Adirondack Mtns 7×773 Posts	But, if he didn't rerun on the original machine, the relations should still exist there. The OP would also need the original poly file, or fb, IIRC. On the original machine, there should be a compressed file with all the original relations, too. Or, is my memory really bad, or the comparison between Win and linux too far removed?

2018-01-22, 15:54	#8
wombatman I moo ablest echo power! May 2013 5·7·53 Posts	What RSA number are you trying to break?

2018-01-22, 17:09	#9
Timic Jan 2018 13₈ Posts	This one: 1772352223689553499523382657591107586734440334822954296365208356693849378633858324876899946660923527875550194354495596668114847388151261943936272297881447 (154 digits)

2018-01-22, 17:10	#10
chris2be8 Sep 2009 3²·271 Posts	I only use Linux, so I'm guessing a bit. But try the following after extracting the files onto another system: msieve -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -ncr If that fails try restarting -nc2 from scratch: msieve -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -nc2 If either of those worked then run: msieve -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -nc3 But if they both failed try rebuilding the matrix (this will try to run all msieve steps in one go): msieve -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -nc Obviously keep the backup zip files safe until you have factored the number. If you run: msieve -h you will get a brief summary of what options msieve accepts. That should help you work out what it wants (eg after -l put the name of the log file. Chris

2018-01-22, 18:11	#11
Timic Jan 2018 11 Posts	I ran (added verbose too) msieve -v -s .\rsa/rsa.dat -l .\rsa/rsa.log -i .\rsa/rsa.ini -nf .\rsa/rsa.fb -t 4 -ncr and it threw an error: cannot open matrix checkpoint file, so I guess that won't work. Tried the -nc2, it built the matrix and is now performing linear algebra, BUT I get these errors: lanczos halted after x iterations (dim = x) linear algebra failed; retrying memory use: 868.8 MB lanczos error: submatrix is not invertible Those were the same errors as on the virtual machine, I thought they were because of memory, but I have 16 GB of RAM here... Is this normal? By the way, I'm using the same .dat and .dat.cyc.