My msieve cuda setup as a newbie

pdazzl · 2014-04-27, 02:42

So I went thru a lot of trial, error, pain, reading and hours of lost sieving work in order to get msieve cuda and gnfs to work in a stable way on my system. Here are my cliff notes version of how I got it to work for anyone setting it up for the first time in windows if it'll help anyone wanting to use their nvidia card.

(Here's Jeff's software links - http://gilchrist.ca/jeff/factoring/index.html)

Initial msieve steps
- Download the latest cuda msieve 1.52 in it's own directory
- Save number to be factored in worktodo
- Go to that msieve directory in a dos prompt
- Make your ms (dat) file by running "msieve -np1 -nps" (-t 2,4 or 8 may speed up the graphics card threading). Run this as long as necessary or till completion.
- Make your p (dat) and consequently your fb file when the job terminates with "msieve -npr"

GNFS with baked msieve steps
- Download gnfs into it's own directory, then download msieve 1.51(1.52 isn't stable for me here) and copy/overwrite it's files in the gnfs directory. You'll also need to install a perl program (I used ActivePerl 5.16.3)
- Edit factMsieve.pl in a text editor and modify "$GGNFS_BIN_PATH" to wherever you've got this directory on your computer
- Create a file name "example.poly" in this format:

Example:
n: 5113381927664564679524313774533181907274615658199099980710867282929897381772360403788270780127706236362305468664699371
Y0: -104013864515029424875978
Y1: 176683580069
c0: 672090960188934118490208355035
c1: 11140739048969871965364872
c2: -264578001851380470603
c3: -116849997785592
c4: 1582639468
c5: 420
skew: 385223.03
type: gnfs

- Overwrite these values from the fb file you created in the cuda msieve directory earlier; the format uses variables that are different in msieve. You can check this link for reference on converting it from msieve to gnfs:
http://gilchrist.ca/jeff/factoring/n...oly_guide.html
- In a dos prompt go to the gnfs diretory and run "factMsieve.pl example.poly" and hopefully you'll eventually get your factor. I've run a couple of 118, 119, 123 digit factorization and the second phase takes a good day or two (I have an i7 950). Smaller 70-90 digit factorizations can be done rather quickly just on regular msieve

Hopefully this helps anyone trying to get this to work on their system, using cuda to prep your polynomial def seems to help speed things up.

VBCurtis · 2014-04-28, 05:37

The python script converts automagically from msieve format to GGNFS format.
I haven't used the perl script, so I can't list the other improvements made; but I've been meaning to write a post of tweaks I've found that improve the python script's efficiency. Perhaps I'll find the time this week to add that post to this thread- or, better, maybe someone else will beat me to it.

henryzz · 2014-04-28, 20:12

Is there any particular reason why msieve has never been changed to use the ggnfs format?
It doesn't really make sense to have two formats around for no reason.

Batalov · 2014-04-28, 20:34

There's also the CWI format.

prss · 2016-03-13, 15:40

Hi,

I am testing if gpu run polys selection faster than cpu. I have a cpu core i5 2.5 Ghz and gpu nvidia optimus 630m with 96 cuda cores. However, gpu run slower than cpu for same input.

Follow below results for msieve 1.52 CPU version:

Code:

> msieve152.exe -t 2 -v -np1
Sun Mar 13 10:40:20 2016  
Sun Mar 13 10:40:20 2016  
Sun Mar 13 10:40:20 2016  Msieve v. 1.52 (SVN 958)
Sun Mar 13 10:40:20 2016  random seeds: 58bb0280 671cb68d
Sun Mar 13 10:40:20 2016  factoring 1383685099763667105632400713678004435475890381619245554755184129298742220082186377 (82 digits)
Sun Mar 13 10:40:21 2016  searching for 15-digit factors
Sun Mar 13 10:40:24 2016  commencing number field sieve (82-digit input)
Sun Mar 13 10:40:24 2016  commencing number field sieve polynomial selection
Sun Mar 13 10:40:24 2016  polynomial degree: 4
Sun Mar 13 10:40:24 2016  max stage 1 norm: 5.07e+013
Sun Mar 13 10:40:24 2016  max stage 2 norm: 2.34e+013
Sun Mar 13 10:40:24 2016  min E-value: 9.06e-008
Sun Mar 13 10:40:24 2016  poly select deadline: 281
Sun Mar 13 10:40:24 2016  time limit set to 0.08 CPU-hours
Sun Mar 13 10:40:24 2016  expecting poly E from 1.73e-007 to > 1.99e-007
Sun Mar 13 10:40:24 2016  searching leading coefficients from 1 to 1925891
Sun Mar 13 10:46:28 2016  polynomial selection complete
Sun Mar 13 10:46:28 2016  elapsed time 00:06:08

Follow below results for msieve 1.52 GPU version:

Code:

> msieve152_gpu.exe -g 0 -t 2 -v -np1
Sun Mar 13 10:29:19 2016  
Sun Mar 13 10:29:19 2016  
Sun Mar 13 10:29:19 2016  Msieve v. 1.52 (SVN 958)
Sun Mar 13 10:29:19 2016  random seeds: b6c3de70 6e9dbc8d
Sun Mar 13 10:29:19 2016  factoring 1383685099763667105632400713678004435475890381619245554755184129298742220082186377 (82 digits)
Sun Mar 13 10:29:20 2016  searching for 15-digit factors
Sun Mar 13 10:29:23 2016  commencing number field sieve (82-digit input)
Sun Mar 13 10:29:23 2016  commencing number field sieve polynomial selection
Sun Mar 13 10:29:23 2016  polynomial degree: 4
Sun Mar 13 10:29:23 2016  max stage 1 norm: 5.07e+013
Sun Mar 13 10:29:23 2016  max stage 2 norm: 2.34e+013
Sun Mar 13 10:29:23 2016  min E-value: 9.06e-008
Sun Mar 13 10:29:23 2016  poly select deadline: 281
Sun Mar 13 10:29:23 2016  time limit set to 0.08 CPU-hours
Sun Mar 13 10:29:23 2016  expecting poly E from 1.99e-007 to > 2.29e-007
Sun Mar 13 10:29:23 2016  searching leading coefficients from 1 to 1925891
Sun Mar 13 10:29:23 2016  using GPU 0 (GeForce GT 630M)
Sun Mar 13 10:29:23 2016  selected card has CUDA arch 2.1
Sun Mar 13 10:38:31 2016  polynomial selection complete
Sun Mar 13 10:38:31 2016  elapsed time 00:09:12

I am executing bimay files on windows 8 64 bits.
I dont install cuda toolkit or any cuda software.
I have installed just last nvidia driver.

what I am doing wrong?

chris2be8 · 2016-03-13, 17:03

The GPU search should have produced more stage 1 hits that the CPU search (count how many records each run put into msieve.dat.m). So when you have run -nps and -npr on the output from stage 1 you should end up with a better poly from the GPU search.

You can adjust the time -np1 takes by telling msieve what range of leading coefficients to search.

It's probably best to run -nps, then sort msieve.dat.ms on the last field and run -npr on the top few hundred records (as Readme.nfs says). Otherwise -npr could take hours.

I run it on Linux so I can't help with details of how to sort records etc on Windows.

Chris

prss · 2016-03-13, 18:30

Thanks for reply.

Sorry, but i dont understand good. have i that set min_coeff and max_coeff when
run msieve gpu -np1, sort integers in .m file and after run msieve gpu -npr?

Googulator · 2016-03-14, 11:25

Be careful with the binaries on Jeff's page. The GGNFS binaries there are old, and will sieve a lot slower than the newer optimized sievers. Also, Msieve SVN 939 and 942 are buggy and can't complete the linear algebra step at the end.

jasonp · 2016-03-14, 11:39

GPU msieve only makes sense for comparatively large problems, 140 digits and up. It will not in general be faster for problems that are smaller, because the GPU is so efficient at stage 1 that you will spend huge amounts of time running stage 2. As problems get larger stage 1 finds many fewer results, so that stage 2 can keep up better.

Stage 2 needs 100x longer than stage 1 to process a single hit, so if you use the GPU for small problems the only practical way to do so is to run stage 1 and part of stage 2, the size optimization (you run msieve with '-np1 -nps'). Size optimization runs only maybe 5x slower than stage 1 and produces a msieve.dat.ms file with the results. Sort the lines in the file and run the root optimization part of stage 2 (with -npr) only on the lines of the .ms file whose last field is the smallest, maybe the smallest 100 or so entries. This is pretty much guaranteed to find the best polynomial across the whole dataset withut spending all day running stage 2.

All that manual work isn't worth it for small problems where the entire polynomial selection would take 5 minutes; it only make sense to go to the trouble for larger problems where polynomial selection might take a day and there are millions of stage 1 results.

prss · 2016-03-14, 13:08

Thank you very much jasonp by clarifications!

Googulator, i am using msieve gpu binary of jasonp sourge forge page.

I see that msieve search by default coeffs of 1 to 1925891 for my RSA-82 digits.
In polys selection of RSA-704 search by coeffs only from 20000 to 21000.

How can i choose range for coeffs search?

jasonp · 2016-03-14, 13:15

The list of NFS options is in Readme.nfs in the source.

To search between A and B you can do

-np1 -nps "min_coeff=A max_coeff=B"

(Note the quotes)

2014-04-27, 02:42	#1
pdazzl Apr 2014 7×17 Posts	My msieve cuda setup as a newbie So I went thru a lot of trial, error, pain, reading and hours of lost sieving work in order to get msieve cuda and gnfs to work in a stable way on my system. Here are my cliff notes version of how I got it to work for anyone setting it up for the first time in windows if it'll help anyone wanting to use their nvidia card. (Here's Jeff's software links - http://gilchrist.ca/jeff/factoring/index.html) Initial msieve steps - Download the latest cuda msieve 1.52 in it's own directory - Save number to be factored in worktodo - Go to that msieve directory in a dos prompt - Make your ms (dat) file by running "msieve -np1 -nps" (-t 2,4 or 8 may speed up the graphics card threading). Run this as long as necessary or till completion. - Make your p (dat) and consequently your fb file when the job terminates with "msieve -npr" GNFS with baked msieve steps - Download gnfs into it's own directory, then download msieve 1.51(1.52 isn't stable for me here) and copy/overwrite it's files in the gnfs directory. You'll also need to install a perl program (I used ActivePerl 5.16.3) - Edit factMsieve.pl in a text editor and modify "$GGNFS_BIN_PATH" to wherever you've got this directory on your computer - Create a file name "example.poly" in this format: Example: n: 5113381927664564679524313774533181907274615658199099980710867282929897381772360403788270780127706236362305468664699371 Y0: -104013864515029424875978 Y1: 176683580069 c0: 672090960188934118490208355035 c1: 11140739048969871965364872 c2: -264578001851380470603 c3: -116849997785592 c4: 1582639468 c5: 420 skew: 385223.03 type: gnfs - Overwrite these values from the fb file you created in the cuda msieve directory earlier; the format uses variables that are different in msieve. You can check this link for reference on converting it from msieve to gnfs: http://gilchrist.ca/jeff/factoring/n...oly_guide.html - In a dos prompt go to the gnfs diretory and run "factMsieve.pl example.poly" and hopefully you'll eventually get your factor. I've run a couple of 118, 119, 123 digit factorization and the second phase takes a good day or two (I have an i7 950). Smaller 70-90 digit factorizations can be done rather quickly just on regular msieve Hopefully this helps anyone trying to get this to work on their system, using cuda to prep your polynomial def seems to help speed things up. Last fiddled with by pdazzl on 2014-04-27 at 02:47

2016-03-14, 11:39	#9
jasonp Tribal Bullet Oct 2004 110111001110₂ Posts	GPU msieve only makes sense for comparatively large problems, 140 digits and up. It will not in general be faster for problems that are smaller, because the GPU is so efficient at stage 1 that you will spend huge amounts of time running stage 2. As problems get larger stage 1 finds many fewer results, so that stage 2 can keep up better. Stage 2 needs 100x longer than stage 1 to process a single hit, so if you use the GPU for small problems the only practical way to do so is to run stage 1 and part of stage 2, the size optimization (you run msieve with '-np1 -nps'). Size optimization runs only maybe 5x slower than stage 1 and produces a msieve.dat.ms file with the results. Sort the lines in the file and run the root optimization part of stage 2 (with -npr) only on the lines of the .ms file whose last field is the smallest, maybe the smallest 100 or so entries. This is pretty much guaranteed to find the best polynomial across the whole dataset withut spending all day running stage 2. All that manual work isn't worth it for small problems where the entire polynomial selection would take 5 minutes; it only make sense to go to the trouble for larger problems where polynomial selection might take a day and there are millions of stage 1 results. Last fiddled with by jasonp on 2016-03-14 at 11:41

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
error when running msieve 1.53 with cuda	aein	Msieve	9	2019-02-25 14:09
Can anyone help me about msieve 1.53 with CUDA?	Seto	Msieve	8	2016-09-24 12:54
Problem in running msieve with CUDA	mohamed	Msieve	20	2013-08-01 08:27
Help with msieve CUDA (C1060)	toorandom	Msieve	7	2012-11-28 17:37
CUDA_ERROR_LAUNCH_OUT_OF_RESOURCES when running msieve 1.5.0 with CUDA	ryanp	Msieve	3	2012-06-12 03:27

2014-04-28, 05:37	#2
VBCurtis "Curtis" Feb 2005 Riverside, CA 1001000000001₂ Posts	The python script converts automagically from msieve format to GGNFS format. I haven't used the perl script, so I can't list the other improvements made; but I've been meaning to write a post of tweaks I've found that improve the python script's efficiency. Perhaps I'll find the time this week to add that post to this thread- or, better, maybe someone else will beat me to it.

2014-04-28, 20:12	#3
henryzz Just call me Henry "David" Sep 2007 Cambridge (GMT/BST) 2×2,897 Posts	Is there any particular reason why msieve has never been changed to use the ggnfs format? It doesn't really make sense to have two formats around for no reason.

2014-04-28, 20:34	#4
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 22053₈ Posts	There's also the CWI format.

2016-03-13, 17:03	#6
chris2be8 Sep 2009 2²·3²·5·11 Posts	The GPU search should have produced more stage 1 hits that the CPU search (count how many records each run put into msieve.dat.m). So when you have run -nps and -npr on the output from stage 1 you should end up with a better poly from the GPU search. You can adjust the time -np1 takes by telling msieve what range of leading coefficients to search. It's probably best to run -nps, then sort msieve.dat.ms on the last field and run -npr on the top few hundred records (as Readme.nfs says). Otherwise -npr could take hours. I run it on Linux so I can't help with details of how to sort records etc on Windows. Chris

2016-03-13, 18:30	#7
prss Jun 2011 Brazil 2×11 Posts	Thanks for reply. Sorry, but i dont understand good. have i that set min_coeff and max_coeff when run msieve gpu -np1, sort integers in .m file and after run msieve gpu -npr?

2016-03-14, 11:25	#8
Googulator Dec 2015 10100₂ Posts	Be careful with the binaries on Jeff's page. The GGNFS binaries there are old, and will sieve a lot slower than the newer optimized sievers. Also, Msieve SVN 939 and 942 are buggy and can't complete the linear algebra step at the end.

2016-03-14, 13:08	#10
prss Jun 2011 Brazil 26₈ Posts	Thank you very much jasonp by clarifications! Googulator, i am using msieve gpu binary of jasonp sourge forge page. I see that msieve search by default coeffs of 1 to 1925891 for my RSA-82 digits. In polys selection of RSA-704 search by coeffs only from 20000 to 21000. How can i choose range for coeffs search?

2016-03-14, 13:15	#11
jasonp Tribal Bullet Oct 2004 DCE₁₆ Posts	The list of NFS options is in Readme.nfs in the source. To search between A and B you can do -np1 -nps "min_coeff=A max_coeff=B" (Note the quotes)