YAFU 2.0

[bsquared]

Originally Posted by LaurV

So, again, what was that trick to run yafu in such a way that it doesn't compress the rels.dat file?

Sorry for your troubles... I think you have to build msieve with the NO_ZLIB=1 option during make. That's how I build mine and I've never had relations compressed. I'm assuming you are building your own copy and not using my provided windows version.

That said, I'm not sure how you are getting both compressed and uncompressed relations in a single file.

[bsquared]

Checked in new update and new windows exe:

+ fixed input of batchfiles, the file name wasn't getting copied to where it needed to be
+ by default json output is now all on one line, -jsonpretty option to pretty print like before (each category on separate line)

[kotenok2000]

curl -L "https://github.com/bbuhrow/yafu/raw/master/bin/x64/Release/yafu-x64.exe" -o "yafu.exe"

[kotenok2000]

Why does yafu remove entry from batchfile when i manually interrupt yafu wuth ctrl+c?

[bsquared]

Originally Posted by kotenok2000

Why does yafu remove entry from batchfile when i manually interrupt yafu wuth ctrl+c?

It removes lines after they are completed... it doesn't check that they were completed successfully.

I've always been on the fence about whether it should remove lines at all... on the one hand it's nice to know where you are in the process if something gets interrupted (e.g., power outage, in which case the line wouldn't be removed). On the other there are situations like this that are difficult to handle.

Since I don't want to deal with difficult situations I will probably just get rid of the removing-lines-as-they-finish feature unless there are strong opinions about keeping it.

[LaurV]

Originally Posted by bsquared

That said, I'm not sure how you are getting both compressed and uncompressed relations in a single file.

I don't. They are all compressed. But the machines run other things too, and in a not very stable environment, and the relations files get HUUGE. Sometimes a chunk inside of such a file is damaged, and yafu/gzip can not go behind of it. In a not-compressed file the things would be as easy as deleting few lines of text, with a text editor that can open such large files, or sometimes, not even that, because yafu will just skip lines with the bad relations. Also, when the computer crashes and rels0, rels1, ... rels_n (which are always uncompressed) remain hanging, on a not-compressed environment I could concatenate all these files to the rels.dat with a simple DOS copy command, and salvage the relations, before resuming the sieving. That is why my pledge for a non-compressed dat file. With a compressed dat file, I can not do any of those, and editing the file when things crash is more tedious (as explained in the previous post, something the "normal" user, who doesn't eat hex and binary for breakfast as we eat, can not do). The only disadvantage of an uncompressed dat file would be the larger size, and a bit longer time to read the relations when filtering (the relations are read from disk, as opposite to the zipped file which is decompressed in ram, faster), but nowadays everybody has large disks with lots of free space, and the additional 5-10 minutes, 3 or 4 times per day, won't be an issue.

I am using executable yafu, not compiled by myself. Which compressed the relations. I remember now that you told me once about recompiling msieve, but that I never did...

[James Heinrich]

Originally Posted by bsquared

I will probably just get rid of the removing-lines-as-they-finish feature unless there are strong opinions about keeping it.

I have a strong preference to keep it for my own use. If you want to disable that by default then please leave the code in there behind a --auto-remove-batch-lines or similar switch, please.

[chris2be8]

To recover data from a damaged .gz file try gzrecover. My script to recover from rubbish in relations from NFS@Home uses:

gzrecover -p $GZ | remdups 1250 -v >$DAT

$GZ is file to process.
$DAT is where to put the recovered relations.

[James Heinrich]

Just ran into something in today's v2.09 that I haven't seen before:

=== Starting work on batchfile expression ===
factor(9893910732172879371097056787104792163954805874908431656489378038530926857)
=============================================
fac: factoring 9893910732172879371097056787104792163954805874908431656489378038530926857
fac: using pretesting plan: deep
fac: using specified qs/gnfs crossover of 100 digits
fac: using specified qs/snfs crossover of 75 digits
div: primes less than 10000
fmt: 1000000 iterations
rho: x^2 + 3, starting 1000 iterations on C73
rho: x^2 + 2, starting 1000 iterations on C73
rho: x^2 + 1, starting 1000 iterations on C73
pm1: starting B1 = 150K, B2 = gmp-ecm default on C73
ecm: 30/30 curves on C73, B1=2k, B2=gmp-ecm default
ecm: 74/74 curves on C73, B1=11k, B2=gmp-ecm default
fopen error: Too many open files
could not open factor.log for appending
ecm: ECM executable does not exist at C:\Users\User\Desktop\yafu2\ecm\ecm.exe
ecm: using internal single threaded ECM...
ecm: 74/74 curves on C73, B1=11k, B2=gmp-ecm default
fopen error: Too many open files
could not open factor.log for appending
ecm: ECM executable does not exist at C:\Users\User\Desktop\yafu2\ecm\ecm.exe
ecm: using internal single threaded ECM...
ecm: 74/74 curves on C73, B1=11k, B2=gmp-ecm default
fopen error: Too many open files
could not open factor.log for appending
ecm: ECM executable does not exist at C:\Users\User\Desktop\yafu2\ecm\ecm.exe
ecm: using internal single threaded ECM...
ecm: 74/74 curves on C73, B1=11k, B2=gmp-ecm default
fopen error: Too many open files
could not open factor.log for appending
ecm: ECM executable does not exist at C:\Users\User\Desktop\yafu2\ecm\ecm.exe
ecm: using internal single threaded ECM...
ecm: 74/74 curves on C73, B1=11k, B2=gmp-ecm default
fopen error: Too many open files
could not open factor.log for appending
ecm: ECM executable does not exist at C:\Users\User\Desktop\yafu2\ecm\ecm.exe
ecm: using internal single threaded ECM...
ecm: 74/74 curves on C73, B1=11k, B2=gmp-ecm default
fopen error: Too many open files
could not open factor.log for appending
ecm: ECM executable does not exist at C:\Users\User\Desktop\yafu2\ecm\ecm.exe
ecm: using internal single threaded ECM...
ecm: 74/74 curves on C73, B1=11k, B2=gmp-ecm default
fopen error: Too many open files
<loop ad infinitum>

and it kept looping until I cancelled it.
(The ECM executable does exist there, I think it just couldn't open it due to too many open files. And of course factor.log does exist.)
I would suggest that "too many open files" should be a fatal error and YAFU should simply exit at that point rather than looping.
The screen output would just keep overwriting the ecm: xx/74 curves on C73 line over and over, however factor.log seemed to contain a full/normal run log for repeated factoring:

08/10/22 17:07:55, ****************************
08/10/22 17:07:55, Starting factorization of 9893910732172879371097056787104792163954805874908431656489378038530926857
08/10/22 17:07:55, using pretesting plan: deep
08/10/22 17:07:55, using specified qs/gnfs crossover of 100 digits
08/10/22 17:07:55, using specified qs/snfs crossover of 75 digits
08/10/22 17:07:55, ****************************
08/10/22 17:07:55, rho: x^2 + 3, starting 1000 iterations on C73
08/10/22 17:07:55, rho: x^2 + 2, starting 1000 iterations on C73
08/10/22 17:07:55, rho: x^2 + 1, starting 1000 iterations on C73
08/10/22 17:07:55, pm1: starting B1 = 150K, B2 = gmp-ecm default on C73
08/10/22 17:07:55, current ECM pretesting depth: 0.000000
08/10/22 17:07:55, scheduled 30 curves at B1=2000 toward target pretesting depth of 24.333333
08/10/22 17:07:55, Finished 30 curves using GMP-ECM method on C73 input, B1=2k, B2=gmp-ecm default
08/10/22 17:07:55, current ECM pretesting depth: 15.177725
08/10/22 17:07:55, scheduled 74 curves at B1=11000 toward target pretesting depth of 24.333333
08/10/22 17:07:56, Finished 74 curves using GMP-ECM method on C73 input, B1=11k, B2=gmp-ecm default
08/10/22 17:07:56, current ECM pretesting depth: 20.242996
08/10/22 17:07:56, scheduled 176 curves at B1=50000 toward target pretesting depth of 24.333333
08/10/22 17:08:00, Finished 176 curves using GMP-ECM method on C73 input, B1=50k, B2=gmp-ecm default
08/10/22 17:08:00, final ECM pretested depth: 24.355146
08/10/22 17:08:00, scheduler: switching to sieve method
08/10/22 17:08:00, starting SIQS on c73: 9893910732172879371097056787104792163954805874908431656489378038530926857
08/10/22 17:08:00, random seed: 10805957936155471635
08/10/22 17:08:00, ==== sieve params ====
08/10/22 17:08:00, n = 76 digits, 251 bits
08/10/22 17:08:00, factor base: 21952 primes (max prime = 527987)
08/10/22 17:08:00, single large prime cutoff: 44878895 (85 * pmax)
08/10/22 17:08:00, double large prime range from 278770272169 to 24607855715335
08/10/22 17:08:00, DLP MFB = 1.75
08/10/22 17:08:00, allocating 6 large prime slices of factor base
08/10/22 17:08:00, buckets hold 2048 elements
08/10/22 17:08:00, large prime hashtables have 491520 bytes
08/10/22 17:08:00, using AVX2 enabled 32k sieve core
08/10/22 17:08:00, sieve interval: 5 blocks of size 32768
08/10/22 17:08:00, polynomial A has ~ 9 factors
08/10/22 17:08:00, using multiplier of 233
08/10/22 17:08:00, using multiplier of 233
08/10/22 17:08:00, using Q2(x) polynomials for kN mod 8 = 1
08/10/22 17:08:00, using SPV correction of 21 bits, starting at offset 34
08/10/22 17:08:00, trial factoring cutoff at 81 bits
08/10/22 17:08:00, ==== sieving started ( 4 threads) ====
08/10/22 17:08:08, trial division touched 2830496 sieve locations out of 15199764480
08/10/22 17:08:08, total reports = 2830496, total surviving reports = 1092798
08/10/22 17:08:08, total blocks sieved = 463860, avg surviving reports per block = 2.36
08/10/22 17:08:08, dlp-ecm: 0 failures, 54248 attempts, 572102 outside range, 359414 prp, 53453 useful
08/10/22 17:08:08, 22363 relations found: 9439 full + 12924 from 151048 partial, using 46386 polys (333 A polys)
08/10/22 17:08:08, on average, sieving found 3.46 rels/poly and 20068.37 rels/sec
08/10/22 17:08:08, trial division touched 2830496 sieve locations out of 15199764480
08/10/22 17:08:08, ==== post processing stage (msieve-1.38) ====
08/10/22 17:08:08, QS elapsed time = 7.9990 seconds.
08/10/22 17:08:08, begin singleton removal with 160487 relations
08/10/22 17:08:08, reduce to 38458 relations in 7 passes
08/10/22 17:08:08, recovered 38458 relations
08/10/22 17:08:08, recovered 26065 polynomials
08/10/22 17:08:08, attempting to build 22363 cycles
08/10/22 17:08:08, found 22363 cycles from 38458 relations in 3 passes
08/10/22 17:08:08, distribution of cycle lengths:
08/10/22 17:08:08,    length 1 : 9439
08/10/22 17:08:08,    length 2 : 7879
08/10/22 17:08:08,    length 3 : 3353
08/10/22 17:08:08,    length 4 : 1161
08/10/22 17:08:08,    length 5 : 383
08/10/22 17:08:08,    length 6 : 106
08/10/22 17:08:08,    length 7 : 28
08/10/22 17:08:08,    length 9+: 14
08/10/22 17:08:08, largest cycle: 10 relations
08/10/22 17:08:08, matrix is 21952 x 22363 (3.6 MB) with weight 758624 (33.92/col)
08/10/22 17:08:08, sparse part has weight 758624 (33.92/col)
08/10/22 17:08:08, filtering completed in 3 passes
08/10/22 17:08:08, matrix is 17396 x 17460 (2.9 MB) with weight 619158 (35.46/col)
08/10/22 17:08:08, sparse part has weight 619158 (35.46/col)
08/10/22 17:08:08, saving the first 48 matrix rows for later
08/10/22 17:08:09, matrix is 17348 x 17460 (2.4 MB) with weight 510149 (29.22/col)
08/10/22 17:08:09, sparse part has weight 461081 (26.41/col)
08/10/22 17:08:09, matrix includes 64 packed rows
08/10/22 17:08:09, commencing Lanczos iteration
08/10/22 17:08:09, memory use: 3.1 MB
08/10/22 17:08:10, lanczos halted after 276 iterations (dim = 17348)
08/10/22 17:08:10, recovered 17 nontrivial dependencies
08/10/22 17:08:10, prp40 = 1360456298247937862016941462358767177779
08/10/22 17:08:10, prp34 = 7272494342460497523675029892412883
08/10/22 17:08:10, Lanczos elapsed time = 2.0160 seconds.
08/10/22 17:08:10, Sqrt elapsed time = 0.0110 seconds.
08/10/22 17:08:10, SIQS elapsed time = 10.0265 seconds.
08/10/22 17:08:10,
08/10/22 17:08:10,
08/10/22 17:08:10, Total factoring time = 15.4097 seconds

And factor.json also contains multiple entries, however I found it unexpected that it seems to alternate the order of the found factors:

{ 	"input-expression":"factor(9893910732172879371097056787104792163954805874908431656489378038530926857)", 	"input-decimal":"9893910732172879371097056787104792163954805874908431656489378038530926857", 	"input-argument-string":"factor(@) -batchfile random_composites.txt ", 	"factors-prime":["1360456298247937862016941462358767177779","7272494342460497523675029892412883"], 	"pm1-curves" : {"150000":1}, 	"ecm-curves" : {"2000":30,"11000":74,"50000":176}, 	"ecm-levels" : {"t15":32.31,"t20":9.42,"t25":0.87,"t30":0.06}, 	"runtime" : {"total":14.9000, "ecm":3.7169, "pm1":0.0352, "pp1":0.0000, "siqs":10.3100, "nfs-total":0.0000, "nfs-poly":0.0000, "nfs-sieve":0.0000, "nfs-filter":0.0000, "nfs-la":0.0000, "nfs-sqrt":0.0000}, 	"time-start" : "2022-08-10  17:06:52", 	"time-end" : "2022-08-10  17:07:07", 	"info":{"compiler":"MSVC 1931","ECM-version":"7.0.4","MPIR-version":"3.0.0","yafu-version":"2.09"} }
{ 	"input-expression":"factor(9893910732172879371097056787104792163954805874908431656489378038530926857)", 	"input-decimal":"9893910732172879371097056787104792163954805874908431656489378038530926857", 	"input-argument-string":"factor(@) -batchfile random_composites.txt ", 	"factors-prime":["7272494342460497523675029892412883","1360456298247937862016941462358767177779"], 	"pm1-curves" : {"150000":1}, 	"ecm-curves" : {"2000":30,"11000":74,"50000":176}, 	"ecm-levels" : {"t15":32.31,"t20":9.42,"t25":0.87,"t30":0.06}, 	"runtime" : {"total":14.2596, "ecm":4.0419, "pm1":0.0347, "pp1":0.0000, "siqs":9.3449, "nfs-total":0.0000, "nfs-poly":0.0000, "nfs-sieve":0.0000, "nfs-filter":0.0000, "nfs-la":0.0000, "nfs-sqrt":0.0000}, 	"time-start" : "2022-08-10  17:07:07", 	"time-end" : "2022-08-10  17:07:21", 	"info":{"compiler":"MSVC 1931","ECM-version":"7.0.4","MPIR-version":"3.0.0","yafu-version":"2.09"} }
{ 	"input-expression":"factor(9893910732172879371097056787104792163954805874908431656489378038530926857)", 	"input-decimal":"9893910732172879371097056787104792163954805874908431656489378038530926857", 	"input-argument-string":"factor(@) -batchfile random_composites.txt ", 	"factors-prime":["1360456298247937862016941462358767177779","7272494342460497523675029892412883"], 	"pm1-curves" : {"150000":1}, 	"ecm-curves" : {"2000":30,"11000":74,"50000":176}, 	"ecm-levels" : {"t15":32.31,"t20":9.42,"t25":0.87,"t30":0.06}, 	"runtime" : {"total":16.6600, "ecm":4.5211, "pm1":0.0445, "pp1":0.0000, "siqs":10.7723, "nfs-total":0.0000, "nfs-poly":0.0000, "nfs-sieve":0.0000, "nfs-filter":0.0000, "nfs-la":0.0000, "nfs-sqrt":0.0000}, 	"time-start" : "2022-08-10  17:07:21", 	"time-end" : "2022-08-10  17:07:38", 	"info":{"compiler":"MSVC 1931","ECM-version":"7.0.4","MPIR-version":"3.0.0","yafu-version":"2.09"} }
{ 	"input-expression":"factor(9893910732172879371097056787104792163954805874908431656489378038530926857)", 	"input-decimal":"9893910732172879371097056787104792163954805874908431656489378038530926857", 	"input-argument-string":"factor(@) -batchfile random_composites.txt ", 	"factors-prime":["7272494342460497523675029892412883","1360456298247937862016941462358767177779"], 	"pm1-curves" : {"150000":1}, 	"ecm-curves" : {"2000":30,"11000":74,"50000":176}, 	"ecm-levels" : {"t15":32.31,"t20":9.42,"t25":0.87,"t30":0.06}, 	"runtime" : {"total":16.7173, "ecm":5.0581, "pm1":0.0360, "pp1":0.0000, "siqs":10.4897, "nfs-total":0.0000, "nfs-poly":0.0000, "nfs-sieve":0.0000, "nfs-filter":0.0000, "nfs-la":0.0000, "nfs-sqrt":0.0000}, 	"time-start" : "2022-08-10  17:07:38", 	"time-end" : "2022-08-10  17:07:55", 	"info":{"compiler":"MSVC 1931","ECM-version":"7.0.4","MPIR-version":"3.0.0","yafu-version":"2.09"} }
{ 	"input-expression":"factor(9893910732172879371097056787104792163954805874908431656489378038530926857)", 	"input-decimal":"9893910732172879371097056787104792163954805874908431656489378038530926857", 	"input-argument-string":"factor(@) -batchfile random_composites.txt ", 	"factors-prime":["1360456298247937862016941462358767177779","7272494342460497523675029892412883"], 	"pm1-curves" : {"150000":1}, 	"ecm-curves" : {"2000":30,"11000":74,"50000":176}, 	"ecm-levels" : {"t15":32.31,"t20":9.42,"t25":0.87,"t30":0.06}, 	"runtime" : {"total":15.4097, "ecm":4.5138, "pm1":0.0402, "pp1":0.0000, "siqs":10.0265, "nfs-total":0.0000, "nfs-poly":0.0000, "nfs-sieve":0.0000, "nfs-filter":0.0000, "nfs-la":0.0000, "nfs-sqrt":0.0000}, 	"time-start" : "2022-08-10  17:07:55", 	"time-end" : "2022-08-10  17:08:10", 	"info":{"compiler":"MSVC 1931","ECM-version":"7.0.4","MPIR-version":"3.0.0","yafu-version":"2.09"} }
{ 	"input-expression":"factor(9893910732172879371097056787104792163954805874908431656489378038530926857)", 	"input-decimal":"9893910732172879371097056787104792163954805874908431656489378038530926857", 	"input-argument-string":"factor(@) -batchfile random_composites.txt ", 	"factors-prime":["7272494342460497523675029892412883","1360456298247937862016941462358767177779"], 	"pm1-curves" : {"150000":1}, 	"ecm-curves" : {"2000":30,"11000":74,"50000":176}, 	"ecm-levels" : {"t15":32.31,"t20":9.42,"t25":0.87,"t30":0.06}, 	"runtime" : {"total":14.0025, "ecm":3.7470, "pm1":0.0336, "pp1":0.0000, "siqs":9.3725, "nfs-total":0.0000, "nfs-poly":0.0000, "nfs-sieve":0.0000, "nfs-filter":0.0000, "nfs-la":0.0000, "nfs-sqrt":0.0000}, 	"time-start" : "2022-08-10  17:08:10", 	"time-end" : "2022-08-10  17:08:24", 	"info":{"compiler":"MSVC 1931","ECM-version":"7.0.4","MPIR-version":"3.0.0","yafu-version":"2.09"} }

Of course the too many open files might be Windows misbehaving rather than YAFU (some update was trying to apply itself in the background at the time I noticed the YAFU problem by chance), but if the error is detected it should exit rather than loop.

[bsquared]

How many entries does your batchfile have? I'm wondering if something isn't getting closed properly when looping over batchfile entries.

[James Heinrich]

Originally Posted by bsquared

How many entries does your batchfile have? I'm wondering if something isn't getting closed properly when looping over batchfile entries.

I process batches of 100 at a time, submit results, fetch new, re-launch YAFU against the next batch.

I've done this successfully for long periods with v1.34.5 but only just started this on YAFU 2.09 today because the -batchfile option was broken until this morning.

I'm not fully convinced that it's a YAFU-error rather than a Windows-error, just that YAFU's handling of the error condition should be better.