Featured request

[Sergei Chernykh]

Originally Posted by bsquared

I've verified this works too, and removed the single-thread limitation if linked against ECM 7+.

It is unlikely there will be another official release. The best I can do anymore is fix bugs and occasionally work on something interesting that hopefully doesn't break something else. If you can't compile from SVN for your platform, chances are someone can and would be willing to help out.

I can compile it, but the problem is that the resulting binary is a bit slower than the official yafu 1.34. I can't get proper compiler options to make it at least as fast.

[bsquared]

Originally Posted by Sergei Chernykh

I can compile it, but the problem is that the resulting binary is a bit slower than the official yafu 1.34. I can't get proper compiler options to make it at least as fast.

I assume you mean SIQS is slower? The speed of ECM and NFS will be dictated by builds of gmp, gmp-ecm, msieve, and ggnfs.

There are only a couple things you can do to impact SIQS speed. The first and least important is the march flag, which you can change on line 19 of the makefile. The second and the most important are the following arguments to the make process:

To add sse41 assembly optimizations make like this

make x86_64 NFS=1 USE_SSE41=1

Or if you have a haswell system or above you can now do

make x86_64 NFS=1 USE_AVX2=1

which is 10-15% faster than sse41.

The last is the compiler. Either of the above works with gcc. If you have the Intel compiler it will make everything a little bit faster yet. (COMPILER=icc during make, if you have it)

[Mr. Odd]

As ever, if someone is willing to compile it for Ubuntu/Linux, that'd be greatly appreciated.

[Joe O]

Originally Posted by richs

Is it possible to write the status of the ECM effort to the factor.log file when one interrupts an ECM run with Control-C, for example:

t15: 78.14
t20: 36.48
t25: 5.09
t30: 0.54
t35: 0.04
sum of completed work is t27.68

Thanks!

Has this been implemented? Can it be? It would be a great help for me, and possibly others that have to share machines.

[bsquared]

Originally Posted by Joe O

Has this been implemented? Can it be? It would be a great help for me, and possibly others that have to share machines.

This has now been implemented; latest code is in \braches\wip in SVN 358.

[Joe O]

Originally Posted by bsquared

This has now been implemented; latest code is in \braches\wip in SVN 358.

Thank You!

[LaurV]

Arrr.. Huston? We have a problem... I remember long arguments about packed versus unpacked relations, but I don't remember the result. I may have been the only one arguing for unpacked relations, that's why.

Was there a command line switch implemented like "-unpack" or so? I generally avoid using the newest versions of yafu which had the "bad habit" of packing my beautiful relations, and as I argued at the time of the discussion, I see no benefit in packing them. Today's disks (ramdisk, ssd) are so fast that it is no benefit in having smaller file, and there is no speedup of one version to another, and if you have 1 gig on the disk for the packed file, then you also have 3 gigs for the unpacked one... Most factorizations we (general we) do anyhow (in the 120-130 digits), need only a tenth of that. If you are concerned about sending the relation files to third party, guess what, a "dedicated" external packer as **zip or **rar can do wonders, and compress better and faster than internal yafu's lzw, not counting the crc-ing or encrypting etc. the file for a better and safer transfer.

On the other side, unpacked relations file is "safer", if the file gets corrupted, just open it with pn2 or your favorite text editor and delete the corrupted relations, but you can save the most of the other and don't need to re-do all the days/weeks of sieving. In a packed file, corruption usually means discarding all the file, or at best, the lzw block where the byte was corrupted. And from the "luck" point of view, at a size of gigabytes and a lot of writing, and ramdisks mounting and unmounting, there is a high chance that will happen sooner or later.

Therefore I was quite happy with my "unpacking" yafu. Till today. Some time ago I installed a 32-bit yafu in a win32 computer (no, it can not be upgraded, due to other tasks running there, I already answered to this, thanks!) which was crunching some aliquots in its 4 stupid and slow cores. Yesterday I realized that there is no progress there for the last week, and moved my buttons to go there and have a look. My toy was crunching the C150 from aliquot 714192 index 2420, and it already tried ~20 times to go into LA, crashing every time at memory allocation. That computer only has 4 gigs of RAM, from which win32 only uses 3, so the crash was normal.

found 3777669 cycles, need 3759956
weight of 3759956 cycles is about 263367037 (70.05/cycle)
....
commencing cycle optimization
start with 20429912 relations
pruned 382710 relations
memory use: 563.3 MB
distribution of cycle lengths:
1 relations: 537300
2 relations: 485096
....
9 relations: 169164
10+ relations: 552040
heaviest cycle: 22 relations
RelProcTime: 1374
nfs: commencing msieve linear algebra

commencing linear algebra
read 3759956 cycles
cycles contain 12204676 unique relations
failed to allocate 292912224 bytes
WARNING: couldn't find factor in factor.log

And "da capo al fine", starts again with filtering. For ever. Grrrr.. Here is aliqueit.exe's fault, it should give a warning and stop, or switch to other sequence, not call yafu forever. But that is another story.

I decided to move the nfs.dat with all afferent job files to another computer, running yafu in 64 bits and having reasonable amount of RAM. And fucking surprize! The nfs.dat is compressed. I had no idea that I installed one "bad yafu" in that computer, all my other installations are "good yafu" which do not compress the relation file.

The ugly part is that new yafu with "uncompressing" does not realize that the .dat file is compressed, and when you resume with it, it will stay 20 minutes (the compressed file has 2.9 GB) looking for the "special q", of course it does not find any, as it doesn't understand the content of the zipped, binary-looking stuff, therefore it starts sieving again from the beginning, adding uncompressed relations at the end of the compressed file. And you only realize that after wasting 30 minutes of "special q" search.



Suggestions? Can I decompress them by hand? (standard compression algorithm? I can look into the source, but I had to ask first). Do I have to look for a 64-bit yafu which uses compressed relations? Can I use any kind of command line switch as mentioned above? Can you (bb) implement one? (in case is not already and I am stupid, but I didn't find it in the docs). (that is the reason I posted in feature requests thread).

[bsquared]

The compression is a msieve thing, yafu doesn't have anything to do about it. Other than, of course, the fact that it uses msieve routines for post processing :)

For the reasons you mention, none of the yafu builds that I've offered over the years use compression in the msieve portion, as far as I can remember. But it's been a real long time since I've built 32-bit yafu and I can't remember what I did the last time I included one in a download file. If it uses compression then it wasn't intentional...

If you build your own yafu then you'll need to remember to build msieve with NO_ZLIB=1.

As for the mess you currently have on your hands, I can't provide a recipe for unraveling it. Maybe you'll need to continue with factmsieve or something that uses a ZLIB enabled msieve.

[LaurV]

Thanks for your very fast reply. You already helped me with the msieve tip and the magic word "zlib"! I was skimming the yafu code last evening without finding the culprit and then gave up as I was tired from job. Now I know how to do.

[LaurV]

yarrr
Thanks again B².

nfs: commencing msieve linear algebra

commencing linear algebra
read 3759956 cycles
cycles contain 12204676 unique relations
read 12204676 relations
using 20 quadratic characters above 536870840
building initial matrix
memory use: 1579.0 MB
read 3759956 cycles
matrix is 3759778 x 3759956 (1133.2 MB) with weight 350932464 (93.33/col)
sparse part has weight 255697659 (68.01/col)
filtering completed in 2 passes
matrix is 3757164 x 3757342 (1133.0 MB) with weight 350847953 (93.38/col)
sparse part has weight 255679362 (68.05/col)
matrix starts at (0, 0)
matrix is 3757164 x 3757342 (1133.0 MB) with weight 350847953 (93.38/col)
sparse part has weight 255679362 (68.05/col)
saving the first 48 matrix rows for later
matrix includes 64 packed rows
matrix is 3757116 x 3757342 (1099.8 MB) with weight 281198460 (74.84/col)
sparse part has weight 250723726 (66.73/col)
using block size 65536 for processor cache size 3072 kB
commencing Lanczos iteration (2 threads)
memory use: 907.2 MB
linear algebra at 0.0%, ETA 17h50m757342 dimensions (0.0%, ETA 17h50m)
checkpointing every 210000 dimensions342 dimensions (0.1%, ETA 18h 0m)
linear algebra completed 10385 of 3757342 dimensions (0.3%, ETA 18h 2m)

(bitchy side of me: please note a printing anomaly at the one-before-last line, the overprinted line is shorted and it didn't erase to the end of line )

[chris2be8]

Originally Posted by LaurV

(bitchy side of me: please note a printing anomaly at the one-before-last line, the overprinted line is shorted and it didn't erase to the end of line )

That's msieve again. I get output like that when running msieve directly (not called by YAFU).

Chris