mersenneforum.org Filtering
 Register FAQ Search Today's Posts Mark Forums Read

 2010-08-04, 12:11 #1 R.D. Silverman     Nov 2003 22·5·373 Posts Filtering The CWI filtering code does an excellent job of filtering data and producing small matrices. When the operating system lets it. I continue to have problems with filtering under Windoze. The filter code sometimes want to grab more memory and does a realloc(). However, this realloc() frequently fails because Windoze insists that that whatever memory is requested is contiguous. I now have a 64-bit version of XP running on my laptop. I have compiled the filter code with LARGEADDRESSAWARE turned on. My system has 4GB of real memory. And the filter code still has realloc() failures. This is very frustrating when you set up a filter pass, it runs for 7 hours and then fails. You have to reset the filter parameters and try again. It is one royal pain in the gluteus medimus.
2010-08-04, 13:13   #2
Andi47

Oct 2004
Austria

2·17·73 Posts

Quote:
 Originally Posted by R.D. Silverman It is one royal pain in the gluteus medimus.
I thought, the "ass muscle" is named "gluteus medius" (without an m between i and u).

Sorry for offtopic - and sorry, I can't help out with the CWI filtering questions, as I'm using msieve for postprocessing (which usually works, even in windoze).

2010-08-04, 14:16   #3
R.D. Silverman

Nov 2003

22·5·373 Posts

Quote:
 Originally Posted by Andi47 I thought, the "ass muscle" is named "gluteus medius" (without an m between i and u). Sorry for offtopic - and sorry, I can't help out with the CWI filtering questions, as I'm using msieve for postprocessing (which usually works, even in windoze).
I would switch to msieve. If I could find the time to do it. Real Work (TM)
keeps me busy.

2010-08-04, 15:07   #4
Andi47

Oct 2004
Austria

2·17·73 Posts

Quote:
 Originally Posted by R.D. Silverman I would switch to msieve. If I could find the time to do it. Real Work (TM) keeps me busy.
If you have your relations in GGNFS format (I think, there are programs out there to convert them from GGNFS to CWI and vice versa, unfortunately I have never dealt with them), and you are working under windows:

* copy (or move) your relations to your msieve folder: the file holding your relations should be named "msieve.dat", and its first line should read "N <your number to factor>"
* create a file named "msieve.fb" which holds your polynomial in the following format:

Code:
N 78821071490818005708357125967410599676235556965930189693232012070393059400352932620218122808136572098113999958161214144948302153279747171639
SKEW 2933331.78
R0 -1537303692875719383652715852
R1  3484803096777727
A0  68619560356658365509867543692766345
A1  1116685541029260068075862175208
A2 -80186249681430126429982
A3 -1044959199363040132
A4  12133739733
A5  9180
* create a file called "msieve.ini" which holds the number which you want to factor

in a command line window, change to your msieve directory and type:

Code:
msieve -p -v -nc -t <number of threads>
whereas the command line flags are meaning:

-p: run at lowest priority
-v: verbose mode
-nc: do filtering, linalg and sqrt (if the matrix solving step is interrupted for some reason, you can restart by using -ncr instead of -nc)
-t: number of threads (not needed if only one thread is used)

when the run is finished, you can find the factors at the bottom of the "msieve.log" file.

Last fiddled with by Andi47 on 2010-08-04 at 15:08 Reason: typo

2010-08-04, 16:12   #5
R.D. Silverman

Nov 2003

22·5·373 Posts

Quote:
 Originally Posted by Andi47 If you have your relations in GGNFS format (I think, there are programs out there to convert them from GGNFS to CWI and vice versa, unfortunately I have never dealt with them), and you are working under windows: * download msieve.exe * copy (or move) your relations to your msieve folder: the file holding your relations should be named "msieve.dat", and its first line should read "N " * create a file named "msieve.fb" which holds your polynomial in the following format: Code: N 78821071490818005708357125967410599676235556965930189693232012070393059400352932620218122808136572098113999958161214144948302153279747171639 SKEW 2933331.78 R0 -1537303692875719383652715852 R1 3484803096777727 A0 68619560356658365509867543692766345 A1 1116685541029260068075862175208 A2 -80186249681430126429982 A3 -1044959199363040132 A4 12133739733 A5 9180 * create a file called "msieve.ini" which holds the number which you want to factor in a command line window, change to your msieve directory and type: Code: msieve -p -v -nc -t  whereas the command line flags are meaning: -p: run at lowest priority -v: verbose mode -nc: do filtering, linalg and sqrt (if the matrix solving step is interrupted for some reason, you can restart by using -ncr instead of -nc) -t: number of threads (not needed if only one thread is used) when the run is finished, you can find the factors at the bottom of the "msieve.log" file.
Why is N needed in both the .dat and .fb files?
Inserting N into the head of (an existing) (say) 7 GB file seems like a nusiance.

My siever emits CWI format. Rather than use a converter, I would prefer
just to modify my code to emit the msieve format. Is it specified anywhere??

2010-08-04, 16:27   #6
axn

Jun 2003

23×233 Posts

Quote:
 Originally Posted by R.D. Silverman Why is N needed in both the .dat and .fb files? Inserting N into the head of (an existing) (say) 7 GB file seems like a nusiance.
Not needed for the dat file. msieve will work fine without it.

 2010-08-04, 16:29 #7 jasonp Tribal Bullet     Oct 2004 5×709 Posts The value of N is in both the ini file and the dat and FB files because the driver for the library allows batches of numbers to be input in the ini file. The dat file therefore needs the value of N at the top to make sure that the relations correspond to the current number from the ini file. If you ask for sieving, and the N in the dat file is not the number from the ini file, then the dat file is wiped out and sieving starts over again (i.e. you're factoring another number and the dat file is leftover from a previous run, not a restart). For NFS postprocessing the dat file is never truncated. N is also in the FB file to detect whether the FB file is from an old factorization. It's somewhat convoluted, and maybe I should change it because nobody batch factors multiple NFS jobs using just msieve. But the full library, including the QS code, does run for each number in the ini file in turn, and I've occaisionally used batch factoring that didn't involve NFS. Last fiddled with by jasonp on 2010-08-04 at 17:31
2010-08-04, 16:33   #8
axn

Jun 2003

23×233 Posts

Quote:
 Originally Posted by R.D. Silverman My siever emits CWI format. Rather than use a converter, I would prefer just to modify my code to emit the msieve format. Is it specified anywhere??
a,b:rational side factors:algebraic side factors

a,b in decimal. factors in hex (case insensitive), comma separated. you may leave out small factors < 1000.

Last fiddled with by jasonp on 2010-08-04 at 17:31 Reason: Also: factors only need to occur once in the format

2010-08-04, 17:22   #9
Andi47

Oct 2004
Austria

2×17×73 Posts

Quote:
 Originally Posted by jasonp The value of N is in both the ini file and the dat and FB files because the driver for the library allows batches of numbers to be input in the ini file. The dat file therefore needs the value of N at the top to make sure that the relations correspond to the current number from the ini file. If you ask for sieving, and the N in the dat file is not the number from the ini file, then the dat file is wiped out and sieving starts over again (i.e. you're factoring another number and the dat file is leftover from a previous run, not a restart). For NFS postprocessing the dat file is never truncated. N is also in the FB file to detect whether the FB file is from an old factorization.
When N is not specified in the .dat file, I have seen msieve (not sure which version) giving an error "error -xy in relation 0" and continuing anyway, but I wouldn't rely on it.

BTW: @JasonP: some time ago I accidently copied an old relations file into the .dat file and got gazillions of relation errors. So: What about stopping with a message like "too many relation errors - are you sure you have copied the right relations file?" if hundreds of relation errors in a row are popping out - instead (or additional to) checking if N is in the first line of the .dat file? (so there would be no need to add it to a multi-GB-file which can be pain in the gluteus maximus...)

Note: how to add N to the .dat file in windows:
1.) rename the relations file to 1.out
2.) create a file named 0.out (so that it is prior to the relns file in alphabetic sorting!!) which contains the line: "N <your number>". Note: This will also work if you want to concatenate multiple relation files, e.g. named 1.out, 2.out, xy.out, etc...
3.) make sure that 0.out and your <relations>.out files are the only files in the directory with the extension .out
4.) in the command line type "copy *.out msieve.dat"
5.) go for a coffee - depending of the size of your relation file(s), it might take some minutes, but it works even for multi-GB-relation-files.
6.) done.

Last fiddled with by Andi47 on 2010-08-04 at 17:24

 2010-08-04, 17:35 #10 jasonp Tribal Bullet     Oct 2004 5×709 Posts Not sure I would want to prematurely abort reading relations in...what if you are concatenating hundreds of files and one of them is from a previous factorization? You get a block of relations that will never work but hundreds of millions that do. Another thing on my wish list for relation reading is recursive dat files, i.e. why have to concatenate anything? Just list the files and let msieve read them one after the other. Admittedly, in an era where everyone supports huge files then being forced to build a huge file is a small inconvenience.
2010-08-04, 18:26   #11
axn

Jun 2003

23×233 Posts

Quote:
 Originally Posted by jasonp Another thing on my wish list for relation reading is recursive dat files, i.e. why have to concatenate anything? Just list the files and let msieve read them one after the other.
This feature would be much appreciated. Read the files from a file list and use the non-existent dat file (-s option) as a scratch pad (for example, free relations).

EDIT:- File copying with AV scanners running is not a pretty thing (esp. under Windoze)

Last fiddled with by axn on 2010-08-04 at 18:27

 Similar Threads Thread Thread Starter Forum Replies Last Post jasonp Msieve 36 2018-05-07 19:55 Stargate38 YAFU 4 2016-04-20 16:53 Dubslow Msieve 20 2016-02-05 14:00 richs Msieve 8 2015-01-18 17:40 Sleepy Msieve 25 2011-08-04 15:05

All times are UTC. The time now is 04:17.

Sun May 22 04:17:24 UTC 2022 up 38 days, 2:18, 0 users, load averages: 1.23, 1.56, 1.54