mersenneforum.org  

Go Back   mersenneforum.org > Factoring Projects > Factoring

Reply
 
Thread Tools
Old 2018-03-18, 22:21   #12
VBCurtis
 
VBCurtis's Avatar
 
"Curtis"
Feb 2005
Riverside, CA

22·19·59 Posts
Default

My experience with frustrating "cannot be read" errors is often linked to using the same folder as a previous factorization, and not knowing/remembering that .params is a hidden file in that directory. So, I neglect to delete the old .params, which causes the python script to get confused and not create files they way they're supposed to be created, and I get awfully frustrated.

I didn't think of this until schickel mentioned .params, which brought back many unpleasant memories.
VBCurtis is online now   Reply With Quote
Old 2018-03-19, 03:39   #13
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3,449 Posts
Default

Quote:
Originally Posted by VBCurtis View Post
My experience with frustrating "cannot be read" errors is often linked to using the same folder as a previous factorization, and not knowing/remembering that .params is a hidden file in that directory. So, I neglect to delete the old .params, which causes the python script to get confused and not create files they way they're supposed to be created, and I get awfully frustrated.

I didn't think of this until schickel mentioned .params, which brought back many unpleasant memories.
THANK YOU!!

That's why it worked once and never again. I've been writing .fb and .poly so much it never dawned on me that Frank meant a hidden file when he mentioned .params.

I removed .params and everything worked just as it was supposed to. Now I have to go reconstruct my scripts I trashed and retest a couple runs.

Thanks All!
EdH is offline   Reply With Quote
Old 2018-03-19, 04:45   #14
schickel
 
schickel's Avatar
 
"Frank <^>"
Dec 2004
CDP Janesville

2×1,049 Posts
Default

Quote:
Originally Posted by EdH View Post
THANK YOU!!

That's why it worked once and never again. I've been writing .fb and .poly so much it never dawned on me that Frank meant a hidden file when he mentioned .params.

I removed .params and everything worked just as it was supposed to. Now I have to go reconstruct my scripts I trashed and retest a couple runs.

Thanks All!
Ah, I run under Windows so I didn't realize that the .params file would be hidden under a *nix OS. That would definitely explain why you would have trouble if you were to reuse a directory.

Not sure why the file was named that way, although I would imagine it probably started out using a "working name" for the current set of parameters and in one of the ports from the original shell script someone dropped the name and just the extension was left.
schickel is offline   Reply With Quote
Old 2018-03-19, 14:52   #15
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3,449 Posts
Default

Quote:
Originally Posted by schickel View Post
Ah, I run under Windows so I didn't realize that the .params file would be hidden under a *nix OS. That would definitely explain why you would have trouble if you were to reuse a directory.

Not sure why the file was named that way, although I would imagine it probably started out using a "working name" for the current set of parameters and in one of the ports from the original shell script someone dropped the name and just the extension was left.
Not only is it hidden from view, but from all non-specific actions. And, the Ubuntu file manager defaults to hide hidden files whenever it is run, whether you have changed it to show, or not. I should have paid a little closer attention to your post.

But, it appears to be working as I had originally planned now.

Many Thanks!
EdH is offline   Reply With Quote
Old 2018-03-19, 16:46   #16
chris2be8
 
chris2be8's Avatar
 
Sep 2009

5×389 Posts
Default

If you are using msieve to do post processing then there is no need for the .params file or the associated logic to change parameters. I was able to delete it all from factMsieve.pl and msieve copes happily with everything I throw at it (my script automatically changes parameters if it's getting too low a yield). Thanks Jason.

Chris
chris2be8 is offline   Reply With Quote
Old 2018-03-19, 20:01   #17
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3,449 Posts
Default

Quote:
Originally Posted by chris2be8 View Post
If you are using msieve to do post processing then there is no need for the .params file or the associated logic to change parameters. I was able to delete it all from factMsieve.pl and msieve copes happily with everything I throw at it (my script automatically changes parameters if it's getting too low a yield). Thanks Jason.

Chris
Does your script distribute to multiple machines? I'm using factmsieve.py across 32 machines, but it only distributes the sieving. I do have an earlier version I modified to use mpi for the Linear Algebra, but after the Pentium level computers, even connecting two Core2 level machines directly via their Gigabit interfaces, I lost, rather than gained, speed.

With your poly script, I'm currently running poly selection on all 32 machines and then using the refindpoly.pl script to create the best poly, and then run factmsieve.py to completion.

So, basically, I initiate a bash script on all machines. They all search for poly pairs via msieve with a time limit. I have a minute buffer for the main machine to wait so all the others have an opportunity to finish.

The main one then runs the refindpoly.pl script to create the <name>.poly file and runs the factmsieve.py script as client 1.

All the other machines wait for the <name>.poly file to exist and then invoke factmsieve.py with their unique client ids.

The main machine gathers enough relations and goes on to the LA stages, while the clients finish their sieving runs and close down.

Only the main machine finishes the final steps.

After all of that, I use grep to pull the factors from the <name>.log file.

I hope soon to "correct" my "How I..." thread back to the above description.
EdH is offline   Reply With Quote
Old 2018-03-20, 17:30   #18
chris2be8
 
chris2be8's Avatar
 
Sep 2009

194510 Posts
Default

Yes, it distributes .poly selection and sieving over several machines. I'm set up so the fastest (master) machine exports a directory (ggnfs/trunk/tests in my case) and the other systems (the helpers) mount it by NFS. I run domaster.sh on the master in that dir and dohelper.sh on the helpers in the same dir.

I just have to put a .poly or .n file in that dir and they will all start work on it once the last number is finished. Note the helpers will do .poly selection and usually start sieving while the master does LA on the previous number.

Since I got a GPU I've used that for .poly selection, then put the generated .poly into the work dir and let the scripts do the rest. But the code should still work.

See attachment for the code. If you want to use it put a copy of factMsieve.pl on each system, updating number of CPUs etc as appropriate. Put domaster.sh on the master system. And a copy of dohelper.sh with the call of factMsieve.pl passing a different number to each helper (2, 3, 4, etc).

It can automatically post the results to factordb.com if you put a line in the .n or .poly file like this:
Code:
factordb: (88^122+1)/300304489707608253860713737369565
Ignore the code to do ECM. That's still work in progress and won't run if you have more than one system.

Chris
Attached Files
File Type: zip factMsieve.zip (28.8 KB, 70 views)
chris2be8 is offline   Reply With Quote
Old 2018-03-20, 17:47   #19
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

3,449 Posts
Default

Thanks Chris,

I will look this over and maybe I'll replace my scripts. I'll have to see....

ECM-wise, before I got heavily involved in the Primo and composite workings for the db, I had a set of scripts that would run the Bayesian script and split the suggestions up into smaller packets, which available "clients" would check out and process. When the last packet of a particular size was checked out, the main system would step to the next size. If any factor was found, it was reported back to the main system and all further work was stopped. I didn't really have any verification in place to make sure a client fully processed its assignment, but it worked pretty well for all my runs.

I would really need to study it again to see what I was doing, though. I keep changing directions and forgetting the particulars of previous work...

Ed
EdH is offline   Reply With Quote
Old 2018-03-21, 01:32   #20
Dubslow
Basketry That Evening!
 
Dubslow's Avatar
 
"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88

3·2,399 Posts
Default

http://www.catb.org/esr/writings/uni...s/prodigy.html
Dubslow is offline   Reply With Quote
Old 2018-03-21, 16:55   #21
chris2be8
 
chris2be8's Avatar
 
Sep 2009

36318 Posts
Default

Hello,

I've just remembered that I renamed the sievers from gnfs-lasieve4I1* to lasieve4I1* so the process table in the system monitor would show which siever was running (it truncated the longer name). So you either need to do that as well or update factMsieve.pl around line 190 to call the original names. Sorry for the omission.

It checks if it can find the sievers etc before checking what parms it's been passed. So just run it without parms to check if all the paths are correct.

It should only need the lattice sievers, def-par.txt and msieve to run, nothing else from ggnfs.

Chris
chris2be8 is offline   Reply With Quote
Old 2018-03-21, 18:29   #22
EdH
 
EdH's Avatar
 
"Ed Hall"
Dec 2009
Adirondack Mtns

D7916 Posts
Default

Quote:
Originally Posted by chris2be8 View Post
Hello,

I've just remembered that I renamed the sievers from gnfs-lasieve4I1* to lasieve4I1* so the process table in the system monitor would show which siever was running (it truncated the longer name). So you either need to do that as well or update factMsieve.pl around line 190 to call the original names. Sorry for the omission.

It checks if it can find the sievers etc before checking what parms it's been passed. So just run it without parms to check if all the paths are correct.

It should only need the lattice sievers, def-par.txt and msieve to run, nothing else from ggnfs.

Chris
Thanks! I'm sidetracked ATM, so I hadn't played with it yet. I've never really done anything with factMsieve.pl, other than testing that it worked for my install procedure. I came onto the scene just about the time Brian was finishing up factmsieve.py and I went in that direction. Now, as you have seen, I'm playing with other things like CADO-NFS, as well, even wondering if I can incorporate CADO-NFS distributed poly selection, factmsieve.py distributed sieving, (or, maybe factmMsieve.pl), and msieve distributed LA. From my initial studying, it looks like everthing that's in place is focused in the wrong direction. i.e. CADO-NFS can import a poly rather than export. But, I do have more study pending...
EdH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Base-6 speed for prime testing vs. base-2 jasong Conjectures 'R Us 36 2010-08-03 06:25
trial division over a factor base Peter Hackman Factoring 7 2009-10-26 18:27
Algebraic factor issues base 24 michaf Conjectures 'R Us 18 2008-05-21 10:08
GNFS - Factor Base Golem86 Factoring 11 2007-07-17 18:08
Quadratic Sieve - How large should the factor base be? hallstei Factoring 5 2005-04-19 11:58

All times are UTC. The time now is 20:52.

Fri Nov 27 20:52:41 UTC 2020 up 78 days, 18:03, 3 users, load averages: 0.58, 1.03, 1.28

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.