mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Cunningham Tables

Reply
 
Thread Tools
Old 2010-08-04, 12:11   #1
R.D. Silverman
 
R.D. Silverman's Avatar
 
Nov 2003

22·5·373 Posts
Default Filtering

The CWI filtering code does an excellent job of filtering data and producing small matrices.

When the operating system lets it.

I continue to have problems with filtering under Windoze.
The filter code sometimes want to grab more memory and
does a realloc(). However, this realloc() frequently fails because
Windoze insists that that whatever memory is requested is contiguous.


I now have a 64-bit version of XP running on my laptop. I have
compiled the filter code with LARGEADDRESSAWARE turned on. My
system has 4GB of real memory. And the filter code still has
realloc() failures.

This is very frustrating when you set up a filter pass, it runs for 7
hours and then fails. You have to reset the filter parameters and
try again. It is one royal pain in the gluteus medimus.
R.D. Silverman is offline   Reply With Quote
Old 2010-08-04, 13:13   #2
Andi47
 
Andi47's Avatar
 
Oct 2004
Austria

2·17·73 Posts
Default

Quote:
Originally Posted by R.D. Silverman View Post
It is one royal pain in the gluteus medimus.
I thought, the "ass muscle" is named "gluteus medius" (without an m between i and u).

Sorry for offtopic - and sorry, I can't help out with the CWI filtering questions, as I'm using msieve for postprocessing (which usually works, even in windoze).
Andi47 is offline   Reply With Quote
Old 2010-08-04, 14:16   #3
R.D. Silverman
 
R.D. Silverman's Avatar
 
Nov 2003

22·5·373 Posts
Default

Quote:
Originally Posted by Andi47 View Post
I thought, the "ass muscle" is named "gluteus medius" (without an m between i and u).

Sorry for offtopic - and sorry, I can't help out with the CWI filtering questions, as I'm using msieve for postprocessing (which usually works, even in windoze).
I would switch to msieve. If I could find the time to do it. Real Work (TM)
keeps me busy.
R.D. Silverman is offline   Reply With Quote
Old 2010-08-04, 15:07   #4
Andi47
 
Andi47's Avatar
 
Oct 2004
Austria

2·17·73 Posts
Default

Quote:
Originally Posted by R.D. Silverman View Post
I would switch to msieve. If I could find the time to do it. Real Work (TM)
keeps me busy.
If you have your relations in GGNFS format (I think, there are programs out there to convert them from GGNFS to CWI and vice versa, unfortunately I have never dealt with them), and you are working under windows:

* download msieve.exe
* copy (or move) your relations to your msieve folder: the file holding your relations should be named "msieve.dat", and its first line should read "N <your number to factor>"
* create a file named "msieve.fb" which holds your polynomial in the following format:

Code:
N 78821071490818005708357125967410599676235556965930189693232012070393059400352932620218122808136572098113999958161214144948302153279747171639
SKEW 2933331.78
R0 -1537303692875719383652715852
R1  3484803096777727
A0  68619560356658365509867543692766345
A1  1116685541029260068075862175208
A2 -80186249681430126429982
A3 -1044959199363040132
A4  12133739733
A5  9180
* create a file called "msieve.ini" which holds the number which you want to factor

in a command line window, change to your msieve directory and type:

Code:
msieve -p -v -nc -t <number of threads>
whereas the command line flags are meaning:

-p: run at lowest priority
-v: verbose mode
-nc: do filtering, linalg and sqrt (if the matrix solving step is interrupted for some reason, you can restart by using -ncr instead of -nc)
-t: number of threads (not needed if only one thread is used)

when the run is finished, you can find the factors at the bottom of the "msieve.log" file.

Last fiddled with by Andi47 on 2010-08-04 at 15:08 Reason: typo
Andi47 is offline   Reply With Quote
Old 2010-08-04, 16:12   #5
R.D. Silverman
 
R.D. Silverman's Avatar
 
Nov 2003

22·5·373 Posts
Default

Quote:
Originally Posted by Andi47 View Post
If you have your relations in GGNFS format (I think, there are programs out there to convert them from GGNFS to CWI and vice versa, unfortunately I have never dealt with them), and you are working under windows:

* download msieve.exe
* copy (or move) your relations to your msieve folder: the file holding your relations should be named "msieve.dat", and its first line should read "N <your number to factor>"
* create a file named "msieve.fb" which holds your polynomial in the following format:

Code:
N 78821071490818005708357125967410599676235556965930189693232012070393059400352932620218122808136572098113999958161214144948302153279747171639
SKEW 2933331.78
R0 -1537303692875719383652715852
R1  3484803096777727
A0  68619560356658365509867543692766345
A1  1116685541029260068075862175208
A2 -80186249681430126429982
A3 -1044959199363040132
A4  12133739733
A5  9180
* create a file called "msieve.ini" which holds the number which you want to factor

in a command line window, change to your msieve directory and type:

Code:
msieve -p -v -nc -t <number of threads>
whereas the command line flags are meaning:

-p: run at lowest priority
-v: verbose mode
-nc: do filtering, linalg and sqrt (if the matrix solving step is interrupted for some reason, you can restart by using -ncr instead of -nc)
-t: number of threads (not needed if only one thread is used)

when the run is finished, you can find the factors at the bottom of the "msieve.log" file.
Why is N needed in both the .dat and .fb files?
Inserting N into the head of (an existing) (say) 7 GB file seems like a nusiance.

My siever emits CWI format. Rather than use a converter, I would prefer
just to modify my code to emit the msieve format. Is it specified anywhere??
R.D. Silverman is offline   Reply With Quote
Old 2010-08-04, 16:27   #6
axn
 
axn's Avatar
 
Jun 2003

23×233 Posts
Default

Quote:
Originally Posted by R.D. Silverman View Post
Why is N needed in both the .dat and .fb files?
Inserting N into the head of (an existing) (say) 7 GB file seems like a nusiance.
Not needed for the dat file. msieve will work fine without it.
axn is online now   Reply With Quote
Old 2010-08-04, 16:29   #7
jasonp
Tribal Bullet
 
jasonp's Avatar
 
Oct 2004

5×709 Posts
Default

The value of N is in both the ini file and the dat and FB files because the driver for the library allows batches of numbers to be input in the ini file. The dat file therefore needs the value of N at the top to make sure that the relations correspond to the current number from the ini file. If you ask for sieving, and the N in the dat file is not the number from the ini file, then the dat file is wiped out and sieving starts over again (i.e. you're factoring another number and the dat file is leftover from a previous run, not a restart). For NFS postprocessing the dat file is never truncated. N is also in the FB file to detect whether the FB file is from an old factorization.

It's somewhat convoluted, and maybe I should change it because nobody batch factors multiple NFS jobs using just msieve. But the full library, including the QS code, does run for each number in the ini file in turn, and I've occaisionally used batch factoring that didn't involve NFS.

Last fiddled with by jasonp on 2010-08-04 at 17:31
jasonp is offline   Reply With Quote
Old 2010-08-04, 16:33   #8
axn
 
axn's Avatar
 
Jun 2003

23×233 Posts
Default

Quote:
Originally Posted by R.D. Silverman View Post
My siever emits CWI format. Rather than use a converter, I would prefer
just to modify my code to emit the msieve format. Is it specified anywhere??
a,b:rational side factors:algebraic side factors

a,b in decimal. factors in hex (case insensitive), comma separated. you may leave out small factors < 1000.

Last fiddled with by jasonp on 2010-08-04 at 17:31 Reason: Also: factors only need to occur once in the format
axn is online now   Reply With Quote
Old 2010-08-04, 17:22   #9
Andi47
 
Andi47's Avatar
 
Oct 2004
Austria

2×17×73 Posts
Default

Quote:
Originally Posted by jasonp View Post
The value of N is in both the ini file and the dat and FB files because the driver for the library allows batches of numbers to be input in the ini file. The dat file therefore needs the value of N at the top to make sure that the relations correspond to the current number from the ini file. If you ask for sieving, and the N in the dat file is not the number from the ini file, then the dat file is wiped out and sieving starts over again (i.e. you're factoring another number and the dat file is leftover from a previous run, not a restart). For NFS postprocessing the dat file is never truncated. N is also in the FB file to detect whether the FB file is from an old factorization.
When N is not specified in the .dat file, I have seen msieve (not sure which version) giving an error "error -xy in relation 0" and continuing anyway, but I wouldn't rely on it.

BTW: @JasonP: some time ago I accidently copied an old relations file into the .dat file and got gazillions of relation errors. So: What about stopping with a message like "too many relation errors - are you sure you have copied the right relations file?" if hundreds of relation errors in a row are popping out - instead (or additional to) checking if N is in the first line of the .dat file? (so there would be no need to add it to a multi-GB-file which can be pain in the gluteus maximus...)

Note: how to add N to the .dat file in windows:
1.) rename the relations file to 1.out
2.) create a file named 0.out (so that it is prior to the relns file in alphabetic sorting!!) which contains the line: "N <your number>". Note: This will also work if you want to concatenate multiple relation files, e.g. named 1.out, 2.out, xy.out, etc...
3.) make sure that 0.out and your <relations>.out files are the only files in the directory with the extension .out
4.) in the command line type "copy *.out msieve.dat"
5.) go for a coffee - depending of the size of your relation file(s), it might take some minutes, but it works even for multi-GB-relation-files.
6.) done.

Last fiddled with by Andi47 on 2010-08-04 at 17:24
Andi47 is offline   Reply With Quote
Old 2010-08-04, 17:35   #10
jasonp
Tribal Bullet
 
jasonp's Avatar
 
Oct 2004

5×709 Posts
Default

Not sure I would want to prematurely abort reading relations in...what if you are concatenating hundreds of files and one of them is from a previous factorization? You get a block of relations that will never work but hundreds of millions that do.

Another thing on my wish list for relation reading is recursive dat files, i.e. why have to concatenate anything? Just list the files and let msieve read them one after the other. Admittedly, in an era where everyone supports huge files then being forced to build a huge file is a small inconvenience.
jasonp is offline   Reply With Quote
Old 2010-08-04, 18:26   #11
axn
 
axn's Avatar
 
Jun 2003

23×233 Posts
Default

Quote:
Originally Posted by jasonp View Post
Another thing on my wish list for relation reading is recursive dat files, i.e. why have to concatenate anything? Just list the files and let msieve read them one after the other.
This feature would be much appreciated. Read the files from a file list and use the non-existent dat file (-s option) as a scratch pad (for example, free relations).

EDIT:- File copying with AV scanners running is not a pretty thing (esp. under Windoze)

Last fiddled with by axn on 2010-08-04 at 18:27
axn is online now   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
NFS filtering in the era of big data jasonp Msieve 36 2018-05-07 19:55
NFS filtering error... Stargate38 YAFU 4 2016-04-20 16:53
The big filtering bug strikes again (I think) Dubslow Msieve 20 2016-02-05 14:00
Filtering Error richs Msieve 8 2015-01-18 17:40
Filtering Sleepy Msieve 25 2011-08-04 15:05

All times are UTC. The time now is 04:17.


Sun May 22 04:17:24 UTC 2022 up 38 days, 2:18, 0 users, load averages: 1.23, 1.56, 1.54

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

≠ ± ∓ ÷ × · − √ ‰ ⊗ ⊕ ⊖ ⊘ ⊙ ≤ ≥ ≦ ≧ ≨ ≩ ≺ ≻ ≼ ≽ ⊏ ⊐ ⊑ ⊒ ² ³ °
∠ ∟ ° ≅ ~ ‖ ⟂ ⫛
≡ ≜ ≈ ∝ ∞ ≪ ≫ ⌊⌋ ⌈⌉ ∘ ∏ ∐ ∑ ∧ ∨ ∩ ∪ ⨀ ⊕ ⊗ 𝖕 𝖖 𝖗 ⊲ ⊳
∅ ∖ ∁ ↦ ↣ ∩ ∪ ⊆ ⊂ ⊄ ⊊ ⊇ ⊃ ⊅ ⊋ ⊖ ∈ ∉ ∋ ∌ ℕ ℤ ℚ ℝ ℂ ℵ ℶ ℷ ℸ 𝓟
¬ ∨ ∧ ⊕ → ← ⇒ ⇐ ⇔ ∀ ∃ ∄ ∴ ∵ ⊤ ⊥ ⊢ ⊨ ⫤ ⊣ … ⋯ ⋮ ⋰ ⋱
∫ ∬ ∭ ∮ ∯ ∰ ∇ ∆ δ ∂ ℱ ℒ ℓ
𝛢𝛼 𝛣𝛽 𝛤𝛾 𝛥𝛿 𝛦𝜀𝜖 𝛧𝜁 𝛨𝜂 𝛩𝜃𝜗 𝛪𝜄 𝛫𝜅 𝛬𝜆 𝛭𝜇 𝛮𝜈 𝛯𝜉 𝛰𝜊 𝛱𝜋 𝛲𝜌 𝛴𝜎𝜍 𝛵𝜏 𝛶𝜐 𝛷𝜙𝜑 𝛸𝜒 𝛹𝜓 𝛺𝜔