20050825, 00:11  #1 
"Bob Silverman"
Nov 2003
North of Boston
16510_{8} Posts 
Interesting Screwup
The linear algebra just finished on 2,969+ and it failed to find any true
dependencies. It did find 64 pseudo dependencies. I just found out why. The filtering first does two passes, looking at the number of excess relations after the first pass to adjust parameters for the second. The output files only contain primes above a certain threshold (typically 250k). It is therefore necessary to completely factor all relations before doing a final filter pass and then building/solving the matrix. Apparently, something caused the refactor to fail. I suspect a power failure in the middle of the computation. I failed to notice that it had failed, and started up the final filter pass. Therefore the final matrix only contained primes greater than 250K. Ooops! I am redoing the entire process now. Maybe this will teach me to be more careful. 
20050825, 01:21  #2 
Aug 2004
New Zealand
230_{10} Posts 
You have my sympathy.
Unfortunately, you can be almost certain that something similar will happen in the future. It's pretty hard to get everything right all the time. When I did a C147 last year, I similarly wasted 20+ days solving a matrix that turned out to be broken. The cause ended up being a corrupt file left over from a machine crash that the filtering stage had picked up and included. It took me a while to track the cause down. 
20050825, 08:44  #3  
Bamboozled!
"๐บ๐๐ท๐ท๐ญ"
May 2003
Down not across
2×5,791 Posts 
Quote:
When NFSNET factored 2,811 the dataset was so large I had serious problems filtering. One problem arose from a bug in Windows' IO library  after reading more than 4G of text files (WIndows distinguishes between text and data) it lost track of where the line endings (CRLF on Windows, contrast with CR on MacOS and LF on Unices). The relations files became subtly corrupted and I was unable to build a "small" matrix. In desperation, I ran an enormous but very sparse matrix on the cluster. I forget the details, but it ran for a long time on a 32cpu cluster, only to produce useless dependencies. Paul 

20050825, 15:48  #4 
Dec 2004
13·23 Posts 
Bob,
May I ask you:  how long the entire process took  Computers  HD space  memory requirements 2,969+ seems pretty big... 
Thread Tools  
Similar Threads  
Thread  Thread Starter  Forum  Replies  Last Post 
Some interesting patterns regarding mod  CuriousKit  Miscellaneous Math  24  20150406 18:40 
An interesting series  Citrix  Math  0  20051102 05:33 
A new interesting thing about 15k  robert44444uk  15k Search  0  20050406 23:00 
very interesting reading  stippix  Hardware  0  20040927 06:19 
Something Interesting  clowns789  Hardware  1  20031220 12:36 