Why can't I start factmsieve.py with a poly file, but no factor base?

[VBCurtis]

My experience with frustrating "cannot be read" errors is often linked to using the same folder as a previous factorization, and not knowing/remembering that .params is a hidden file in that directory. So, I neglect to delete the old .params, which causes the python script to get confused and not create files they way they're supposed to be created, and I get awfully frustrated.

I didn't think of this until schickel mentioned .params, which brought back many unpleasant memories.

[EdH]

Originally Posted by VBCurtis

My experience with frustrating "cannot be read" errors is often linked to using the same folder as a previous factorization, and not knowing/remembering that .params is a hidden file in that directory. So, I neglect to delete the old .params, which causes the python script to get confused and not create files they way they're supposed to be created, and I get awfully frustrated.

I didn't think of this until schickel mentioned .params, which brought back many unpleasant memories.

THANK YOU!!

That's why it worked once and never again. I've been writing .fb and .poly so much it never dawned on me that Frank meant a hidden file when he mentioned .params.

I removed .params and everything worked just as it was supposed to. Now I have to go reconstruct my scripts I trashed and retest a couple runs.

Thanks All!

[schickel]

Originally Posted by EdH

THANK YOU!!

That's why it worked once and never again. I've been writing .fb and .poly so much it never dawned on me that Frank meant a hidden file when he mentioned .params.

I removed .params and everything worked just as it was supposed to. Now I have to go reconstruct my scripts I trashed and retest a couple runs.

Thanks All!

Ah, I run under Windows so I didn't realize that the .params file would be hidden under a *nix OS. That would definitely explain why you would have trouble if you were to reuse a directory.

Not sure why the file was named that way, although I would imagine it probably started out using a "working name" for the current set of parameters and in one of the ports from the original shell script someone dropped the name and just the extension was left.

[EdH]

Originally Posted by schickel

Ah, I run under Windows so I didn't realize that the .params file would be hidden under a *nix OS. That would definitely explain why you would have trouble if you were to reuse a directory.

Not sure why the file was named that way, although I would imagine it probably started out using a "working name" for the current set of parameters and in one of the ports from the original shell script someone dropped the name and just the extension was left.

Not only is it hidden from view, but from all non-specific actions. And, the Ubuntu file manager defaults to hide hidden files whenever it is run, whether you have changed it to show, or not. I should have paid a little closer attention to your post.

But, it appears to be working as I had originally planned now.

Many Thanks!

[chris2be8]

If you are using msieve to do post processing then there is no need for the .params file or the associated logic to change parameters. I was able to delete it all from factMsieve.pl and msieve copes happily with everything I throw at it (my script automatically changes parameters if it's getting too low a yield). Thanks Jason.

Chris

[EdH]

Originally Posted by chris2be8

If you are using msieve to do post processing then there is no need for the .params file or the associated logic to change parameters. I was able to delete it all from factMsieve.pl and msieve copes happily with everything I throw at it (my script automatically changes parameters if it's getting too low a yield). Thanks Jason.

Chris

Does your script distribute to multiple machines? I'm using factmsieve.py across 32 machines, but it only distributes the sieving. I do have an earlier version I modified to use mpi for the Linear Algebra, but after the Pentium level computers, even connecting two Core2 level machines directly via their Gigabit interfaces, I lost, rather than gained, speed.

With your poly script, I'm currently running poly selection on all 32 machines and then using the refindpoly.pl script to create the best poly, and then run factmsieve.py to completion.

So, basically, I initiate a bash script on all machines. They all search for poly pairs via msieve with a time limit. I have a minute buffer for the main machine to wait so all the others have an opportunity to finish.

The main one then runs the refindpoly.pl script to create the <name>.poly file and runs the factmsieve.py script as client 1.

All the other machines wait for the <name>.poly file to exist and then invoke factmsieve.py with their unique client ids.

The main machine gathers enough relations and goes on to the LA stages, while the clients finish their sieving runs and close down.

Only the main machine finishes the final steps.

After all of that, I use grep to pull the factors from the <name>.log file.

I hope soon to "correct" my "How I..." thread back to the above description.

[chris2be8]

Yes, it distributes .poly selection and sieving over several machines. I'm set up so the fastest (master) machine exports a directory (ggnfs/trunk/tests in my case) and the other systems (the helpers) mount it by NFS. I run domaster.sh on the master in that dir and dohelper.sh on the helpers in the same dir.

I just have to put a .poly or .n file in that dir and they will all start work on it once the last number is finished. Note the helpers will do .poly selection and usually start sieving while the master does LA on the previous number.

Since I got a GPU I've used that for .poly selection, then put the generated .poly into the work dir and let the scripts do the rest. But the code should still work.

See attachment for the code. If you want to use it put a copy of factMsieve.pl on each system, updating number of CPUs etc as appropriate. Put domaster.sh on the master system. And a copy of dohelper.sh with the call of factMsieve.pl passing a different number to each helper (2, 3, 4, etc).

It can automatically post the results to factordb.com if you put a line in the .n or .poly file like this:

factordb: (88^122+1)/300304489707608253860713737369565

Ignore the code to do ECM. That's still work in progress and won't run if you have more than one system.

Chris

[EdH]

Thanks Chris,

I will look this over and maybe I'll replace my scripts. I'll have to see....

ECM-wise, before I got heavily involved in the Primo and composite workings for the db, I had a set of scripts that would run the Bayesian script and split the suggestions up into smaller packets, which available "clients" would check out and process. When the last packet of a particular size was checked out, the main system would step to the next size. If any factor was found, it was reported back to the main system and all further work was stopped. I didn't really have any verification in place to make sure a client fully processed its assignment, but it worked pretty well for all my runs.

I would really need to study it again to see what I was doing, though. I keep changing directions and forgetting the particulars of previous work...

Ed

[Dubslow]

http://www.catb.org/esr/writings/uni...s/prodigy.html

[chris2be8]

Hello,

I've just remembered that I renamed the sievers from gnfs-lasieve4I1* to lasieve4I1* so the process table in the system monitor would show which siever was running (it truncated the longer name). So you either need to do that as well or update factMsieve.pl around line 190 to call the original names. Sorry for the omission.

It checks if it can find the sievers etc before checking what parms it's been passed. So just run it without parms to check if all the paths are correct.

It should only need the lattice sievers, def-par.txt and msieve to run, nothing else from ggnfs.

Chris

[EdH]

Originally Posted by chris2be8

Hello,

I've just remembered that I renamed the sievers from gnfs-lasieve4I1* to lasieve4I1* so the process table in the system monitor would show which siever was running (it truncated the longer name). So you either need to do that as well or update factMsieve.pl around line 190 to call the original names. Sorry for the omission.

It checks if it can find the sievers etc before checking what parms it's been passed. So just run it without parms to check if all the paths are correct.

It should only need the lattice sievers, def-par.txt and msieve to run, nothing else from ggnfs.

Chris

Thanks! I'm sidetracked ATM, so I hadn't played with it yet. I've never really done anything with factMsieve.pl, other than testing that it worked for my install procedure. I came onto the scene just about the time Brian was finishing up factmsieve.py and I went in that direction. Now, as you have seen, I'm playing with other things like CADO-NFS, as well, even wondering if I can incorporate CADO-NFS distributed poly selection, factmsieve.py distributed sieving, (or, maybe factmMsieve.pl), and msieve distributed LA. From my initial studying, it looks like everthing that's in place is focused in the wrong direction. i.e. CADO-NFS can import a poly rather than export. But, I do have more study pending...