10,375- LA discussion

Raman · 2008-11-25, 18:35

Code:

Tue Nov 25 20:50:42 2008  Msieve v. 1.38
Tue Nov 25 20:50:42 2008  random seeds: 4b6f5c70 ba6361fd
Tue Nov 25 20:50:42 2008  factoring 21877563764774520766443734689964017774797543957040097751031466722004501741746069689447656345311301031586769230692476490788893736747545384191961681472150167189343222044540938001 (176 digits)
Tue Nov 25 20:50:44 2008  searching for 15-digit factors
Tue Nov 25 20:50:48 2008  commencing number field sieve (176-digit input)
Tue Nov 25 20:50:48 2008  R0: -100000000000000000000000000000000000000000000000001
Tue Nov 25 20:50:48 2008  R1:  10000000000000000000000000
Tue Nov 25 20:50:48 2008  A0:  1
Tue Nov 25 20:50:48 2008  A1:  4
Tue Nov 25 20:50:48 2008  A2: -4
Tue Nov 25 20:50:48 2008  A3: -1
Tue Nov 25 20:50:48 2008  A4:  1
Tue Nov 25 20:50:48 2008  size score = 7.079708e-020, Murphy alpha = 2.443568, combined = 2.663938e-020
Tue Nov 25 20:50:49 2008  
Tue Nov 25 20:50:49 2008  commencing relation filtering
Tue Nov 25 20:50:49 2008  commencing duplicate removal, pass 1
<relation errors removed>
Tue Nov 25 21:12:56 2008  found 30049442 hash collisions in 102079376 relations
Tue Nov 25 21:13:50 2008  added 3656801 free relations
Tue Nov 25 21:13:50 2008  commencing duplicate removal, pass 2
Tue Nov 25 21:22:06 2008  found 33213551 duplicates and 72522626 unique relations
Tue Nov 25 21:22:06 2008  memory use: 504.8 MB
Tue Nov 25 21:22:06 2008  reading rational ideals above 40042496
Tue Nov 25 21:22:06 2008  reading algebraic ideals above 40042496
Tue Nov 25 21:22:06 2008  commencing singleton removal, pass 1
Tue Nov 25 21:40:08 2008  relations with 0 large ideals: 1985031
Tue Nov 25 21:40:08 2008  relations with 1 large ideals: 9658122
Tue Nov 25 21:40:08 2008  relations with 2 large ideals: 26296163
Tue Nov 25 21:40:08 2008  relations with 3 large ideals: 24293848
Tue Nov 25 21:40:08 2008  relations with 4 large ideals: 7241562
Tue Nov 25 21:40:08 2008  relations with 5 large ideals: 3047900
Tue Nov 25 21:40:08 2008  relations with 6 large ideals: 0
Tue Nov 25 21:40:08 2008  relations with 7+ large ideals: 0
Tue Nov 25 21:40:08 2008  72522626 relations and about 48219389 large ideals
Tue Nov 25 21:40:08 2008  commencing singleton removal, pass 2
Tue Nov 25 21:58:18 2008  found 13457956 singletons
Tue Nov 25 21:58:18 2008  current dataset: 59064670 relations and about 33726394 large ideals
Tue Nov 25 21:58:18 2008  commencing singleton removal, pass 3
Tue Nov 25 22:13:09 2008  relations with 0 large ideals: 1985031
Tue Nov 25 22:13:09 2008  relations with 1 large ideals: 8745835
Tue Nov 25 22:13:09 2008  relations with 2 large ideals: 21815752
Tue Nov 25 22:13:09 2008  relations with 3 large ideals: 18877978
Tue Nov 25 22:13:09 2008  relations with 4 large ideals: 5342457
Tue Nov 25 22:13:09 2008  relations with 5 large ideals: 2297617
Tue Nov 25 22:13:09 2008  relations with 6 large ideals: 0
Tue Nov 25 22:13:09 2008  relations with 7+ large ideals: 0
Tue Nov 25 22:13:09 2008  59064670 relations and about 40536373 large ideals
Tue Nov 25 22:13:09 2008  commencing singleton removal, pass 4
Tue Nov 25 22:28:06 2008  found 8332310 singletons
Tue Nov 25 22:28:06 2008  current dataset: 50732360 relations and about 31725608 large ideals
Tue Nov 25 22:28:06 2008  commencing singleton removal, pass 5
Tue Nov 25 22:43:01 2008  found 1749418 singletons
Tue Nov 25 22:43:01 2008  current dataset: 48982942 relations and about 29953379 large ideals
Tue Nov 25 22:43:01 2008  commencing singleton removal, pass 6
Tue Nov 25 22:56:10 2008  found 319375 singletons
Tue Nov 25 22:56:10 2008  current dataset: 48663567 relations and about 29633183 large ideals
Tue Nov 25 22:56:10 2008  commencing singleton removal, final pass
Tue Nov 25 23:12:23 2008  memory use: 803.9 MB
Tue Nov 25 23:12:24 2008  commencing in-memory singleton removal
Tue Nov 25 23:12:31 2008  begin with 48663567 relations and 32742963 unique ideals
Tue Nov 25 23:13:53 2008  reduce to 45900517 relations and 29945133 ideals in 11 passes
Tue Nov 25 23:13:53 2008  max relations containing the same ideal: 48
Tue Nov 25 23:14:07 2008  reading rational ideals above 720000
Tue Nov 25 23:14:07 2008  reading algebraic ideals above 720000
Tue Nov 25 23:14:07 2008  commencing singleton removal, final pass
Tue Nov 25 23:39:17 2008  keeping 31211450 ideals with weight <= 20, new excess is 3605389
failed to reallocate 888143872 bytes

With 30 bit lpba and 29 bit lpbr, I assume that 65 million unique relations suffice and I have 72 million,
so I think that it is certainly ready for post processing.

How much additional memory do I need so?
Or given my laptop and desktop, each having 2 GB RAM, is it possible to share the memory which are being connected so by Ethernet LAN cable?

I think that I made a mistake keeping the large prime limits too high, it is probably the cause for requiring too much memory. Probably the matrix might also be too large, because there will be too many primes in the factor base. I think that it would have been always better if I kept so the large primes to about 28.

Any method or possible software available online to use so for shared memory?
Or should I try the old way with /3GB switch and safe mode? Or try so for some 64 bit Linux, which has 3 GB virtual memory size? If I have to install Linux, I have to format Windows with all the hard disk data.

jasonp · 2008-11-25, 20:57

Quote:

Originally Posted by Raman

How much additional memory do I need so?

Probably the whole thing could fit in 2.5GB. It might even fit in 2GB if you are lucky; try booting windows into safe mode. If that doesn't work, you need a machine with more memory, and switching to a different OS will not give you more memory.

Quote:

Or given my laptop and desktop, each having 2 GB RAM, is it possible to share the memory which are being connected so by Ethernet LAN cable?

No. There are packages that can do it (PVM, MPI, etc), and they even run on windows, but the code you run must be aware of them; msieve is not.

Quote:

I think that I made a mistake keeping the large prime limits too high, it is probably the cause for requiring too much memory. Probably the matrix might also be too large, because there will be too many primes in the factor base. I think that it would have been always better if I kept so the large primes to about 28.

Yes, 29 bit large primes would have made the postprocessing a lot easier.

Batalov · 2008-11-25, 21:20

(Just to state what you most probably know: do not use the perl script, from this point; most of its version will give you problems)

You can do filtering and the start of "-nc2" on a machine with 4Gb of memory -- you will need it (and all the files) only for a few hours, so you can probably find one.
When the process is already in BL, check the memory footprint (it gets smaller at that point, and you may fit into 2Gb... very unlikely for this number).
Then if successful, stop the "-nc2", transfer _all_ files back to your own machine and proceed with "-ncr" (probably 10-15 days) and then "-nc3".

You will be better off with another 2Gb of memory, of course. That's a small story in itself, albeit rather obvious... you know, match it to the existing, or better get a new 2x2Gb. Then again, you will be able to use 3.5Gb with Windows XP... etc etc etc. YMMV.

<S>

P.S. An afterthought: This is a big job. It may happen that you could read the whole story above while doubling the numbers, ok? I.e. you may need a short-term access to a 8Gb machine, but then finish on yours with 3.5Gb memory; but you have to try to see.

henryzz · 2008-11-26, 07:30

am i right in thinking that oversieving reduces the size of the matrix but increases the memory usage for the filtering

jasonp · 2008-11-26, 12:58

Oversieving has a better chance of producing a smaller matrix because you can throw away collections of relations that add too many matrix entries, and still wind up with a matrix large enough for the math to work.

Whether the filtering has a harder job as a result depends on the quality of the filtering implementation. More oversieving gives you more flexibility that good filtering has to exploit, but less oversieving means the dataset going into the matrix construction phase is needlessly larger, so that oversieving could actually make filtering less memory-intensive.

Raman · 2008-11-26, 17:55

No thought about oversieving. I don't want to spend any more time with 10,375- and already continued sieving with 6,343- on my Desktop.

I have already over 7 million relations in excess than the required amount and I think that it is enough.

The filtering is running in safe mode and I will report about the details soon. Filtering is going on rather slow in safe mode (Like the retrograde rotation of Venus about its axis). One full day for filtering in the safe mode against the three hours time in the normal mode.

The matrix is expected to have about 3335254 rows or columns, and is expected to take about three and a half days to complete so the task.

Probably I think that Windows in safe mode permits infinite amount of virtual memory for each program. Well, the countdown begins so...

Raman · 2008-12-01, 17:14

Linear algebra for 10,375- completed about 3 hours ago.
and it recovered 43 dependencies.
When I started square root, it took a long time with

cycles contain 5222168 unique relations

I didn't have patience, so I restarted my Laptop, with the /3GB
switch in the boot.ini file. (That step took only 30 minutes with
7,295- and I had been waiting for over 45 minutes with smaller
number of relations per cycle). And the virtual memory allocated
had been very slow, hardly few hundred KB per second, compared
to 1 GB it had to allocate totally.

When I restarted my Laptop, it neither boots properly in Safe Mode
nor normal mode.

I don't know what to do now.

Quote:

A problem has been detected and Windows has been shut down to prevent damage to your computer.

If this is the first time you've seen this Stop error screen, restart your computer. If this screen appears again, follow these steps:

Check for viruses on your computer. Remove any newly installed hard drives or hard drive controllers. Check your hard drive to make sure it is properly configured and terminated. Run CHKDSK /F to check for hard drive corruption, and then restart your computer.

Technical information:

*** STOP: 0x0000007B (0xF6CB7524,0xC00000034,0x00000000,0x00000000)

The dependencies and the cycles file is struck up in my Laptop. My Desktop contains so, only the relations, and if I were to run in my Desktop, I have to run the entire filtering and Linear Algebra again. No way, at all. No chance for redoing it up again.

And my Core 2 Quad desktop has high probabilities of running into LA failures, mainly running into non-invertible blocks of the submatrices, as it happened so for even a small matrix such as 10,312+ whose dimensions were less than 2 million rows or columns.

Bad Luck! What to do so... Need some more patience

Is there any way to revert back to the original boot.ini file,
without booting or logging on into Windows at all? Command prompt option is not there with F8,

and system recovery fails so.

S485122 · 2008-12-01, 18:47

Quote:

Originally Posted by Raman

Is there any way to revert back to the original boot.ini file,without booting or logging on into Windows at all? Command prompt option is not there with F8, and system recovery fails so.

Yes : take your disk out of the portable and use an usb adapter to mount it on your desktop computer, edit the boot.ini file (on the disk of portable of course), eject the disk and put it back in your laptop. If the faulty boot.ini file was the only problem all should be OK. I think that the /3GB option is only valid if you have at least 3 GB of physical memory...

There are some Microsoft Knowledge based articles about this problem : Advanced troubleshooting for "Stop 0x0000007B" errors in Windows XP is the most relevant but not very helpful in your case.

Jacob

jasonp · 2008-12-02, 12:43

You only need the .cyc and .dep files from the laptop, and these are much smaller than the full set of files produced in earlier stages

Raman · 2008-12-02, 13:46

Quote:

Originally Posted by S485122

Yes : take your disk out of the portable and mount it on your desktop computer, edit the boot.ini file (on the disk of portable of course), eject the disk and put it back in your laptop.

No, I tried it, but laptop hard disk size is too small to fit effectively into the Desktop hard disk drive slot. I thought that you had been aware of it before itself. Perhaps that I should buy an USB adapter???? Too costly?? Or some other alternate ways available so?

Quote:

Originally Posted by jasonp

You only need the .cyc and .dep files from the laptop, and these are much smaller than the full set of files produced in earlier stages

What to do so? I wasn't clever enough in backing up those .cyc and .dep files to my Desktop as soon as the Linear Algebra ended so. Now, I am not even able to boot into Windows, even in safe mode to edit the boot.ini file, or copy those files even to a flash drive. Nor does command prompt option is being available so on pressing F8 during the booting process.

And it also requested so for the .dat file, besides the .cyc and .dep files, and it is approximately 11.2 GB (with sizes of 30 bit lpbr and 29 bit lpba). And the process of backing up wasn't needed so for the past three numbers that I had done so. Those numbers had been safely working so.

Any quick suggestions to edit the boot.ini file without logging on into Windows, or back up the files, when even the command prompt doesn't work so at all with F8, or to make the some of the system recovery process to succeed so, or how to run CHKDSK /F under this condition??

henryzz · 2008-12-02, 13:51

an dos boot disk might do it
http://www.ultimatebootcd.com/

2008-11-25, 21:20	#3
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 2²·2,549 Posts	(Just to state what you most probably know: do not use the perl script, from this point; most of its version will give you problems) You can do filtering and the start of "-nc2" on a machine with 4Gb of memory -- you will need it (and all the files) only for a few hours, so you can probably find one. When the process is already in BL, check the memory footprint (it gets smaller at that point, and you may fit into 2Gb... very unlikely for this number). Then if successful, stop the "-nc2", transfer _all_ files back to your own machine and proceed with "-ncr" (probably 10-15 days) and then "-nc3". You will be better off with another 2Gb of memory, of course. That's a small story in itself, albeit rather obvious... you know, match it to the existing, or better get a new 2x2Gb. Then again, you will be able to use 3.5Gb with Windows XP... etc etc etc. YMMV. <S> P.S. An afterthought: This is a big job. It may happen that you could read the whole story above while doubling the numbers, ok? I.e. you may need a short-term access to a 8Gb machine, but then finish on yours with 3.5Gb memory; but you have to try to see. Last fiddled with by Batalov on 2008-11-25 at 22:14 Reason: (double-up)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
P-1 discussion	kladner	GPU to 72	43	2012-01-27 20:43
Discussion about dates	Flatlander	Twin Prime Search	12	2011-11-17 09:40
P-1 discussion	AntonVrba	Prime Cullen Prime	5	2007-04-04 04:59
factexcl.txt discussion	hhh	Prime Sierpinski Project	5	2006-11-22 17:50
New .dat discussion	VJS	Prime Sierpinski Project	7	2006-07-25 14:31

2008-11-26, 07:30	#4
henryzz Just call me Henry "David" Sep 2007 Liverpool (GMT/BST) 6,131 Posts	am i right in thinking that oversieving reduces the size of the matrix but increases the memory usage for the filtering

2008-11-26, 12:58	#5
jasonp Tribal Bullet Oct 2004 DED₁₆ Posts	Oversieving has a better chance of producing a smaller matrix because you can throw away collections of relations that add too many matrix entries, and still wind up with a matrix large enough for the math to work. Whether the filtering has a harder job as a result depends on the quality of the filtering implementation. More oversieving gives you more flexibility that good filtering has to exploit, but less oversieving means the dataset going into the matrix construction phase is needlessly larger, so that oversieving could actually make filtering less memory-intensive.

2008-11-26, 17:55	#6
Raman Noodles "Mr. Tuch" Dec 2007 Chennai, India 3×419 Posts	No thought about oversieving. I don't want to spend any more time with 10,375- and already continued sieving with 6,343- on my Desktop. I have already over 7 million relations in excess than the required amount and I think that it is enough. The filtering is running in safe mode and I will report about the details soon. Filtering is going on rather slow in safe mode (Like the retrograde rotation of Venus about its axis). One full day for filtering in the safe mode against the three hours time in the normal mode. The matrix is expected to have about 3335254 rows or columns, and is expected to take about three and a half days to complete so the task. Probably I think that Windows in safe mode permits infinite amount of virtual memory for each program. Well, the countdown begins so...

2008-12-02, 12:43	#9
jasonp Tribal Bullet Oct 2004 5×23×31 Posts	You only need the .cyc and .dep files from the laptop, and these are much smaller than the full set of files produced in earlier stages

2008-12-02, 13:51	#11
henryzz Just call me Henry "David" Sep 2007 Liverpool (GMT/BST) 6,131 Posts	an dos boot disk might do it http://www.ultimatebootcd.com/