mersenneforum.org

mersenneforum.org (https://www.mersenneforum.org/index.php)
-   Software (https://www.mersenneforum.org/forumdisplay.php?f=10)
-   -   mtsieve (https://www.mersenneforum.org/showthread.php?t=23042)

Happy5214 2022-12-17 16:18

[QUOTE=rogue;620022]Unlikely. That would require large changes to the framework. I do not understand why the log file is an issue. Do you have a script reading from it? You can run a copy of the programs from different directories. They will not share a log file. I do not understand why you would want to suppress it.[/QUOTE]

Actually, I do have scripts reading my logs (to automatically determine a stopping point based on factor removal rate and LLR test time). Since my data is all stored in folders (one per dataset) and symlinked into the working directory, I guess I'll just have the script change the symlinks. They'll need to read the [I]q[/I] pretest log output anyway to determine which [I]q[/I] to use.

Trying2Sieve 2022-12-19 05:03

[QUOTE=rogue;619942]None of the sieves in the framework sieve repdigits, which is what this appears to be. Which option in newpgen sieves that format?

For pfgw you would want the ABC format. The first line would be something like this:
ABC 7*10^($a+1)+1
40
45
48

Again, nothing in the framework that does that.

For distributing the work across many clients, if they are all on the same network, PRPNet is an option, but you would have to use the generic pfgw format in the server as the server doesn't support repdigit searches.[/QUOTE]

FINALLY SOLVED!

Indeed, you were correct sir, so thank you!

The ABC format does give me what I need for massive parallelization using my custom utilities along with mtsieve and pfgw.

These are the parameters that were ultimately successful (tested on my laptop)

[CODE]srsieve2.exe -W 6 -s "7*10^n+1" -n 1000000 -N 8000000 -f A -o "8_million.txt"[/CODE]

I set srsieve to use 6 cores
The equation is 7 x 10^n + 1
The range of n is one million to eight million

[B]-f A was the magic setting for ABC format.[/B]

It reached the prime [CODE]1 913 595 054 221[/CODE] after about 24 hours on the laptop.

This sieved [CODE]6 269 705[/CODE] of the [CODE]7 000 001[/CODE] factors.

If I stop there and use my 365 cores that will be 2000 candidates per core.

I'm now parallelizing the sieving and starting over.

Thanks again for your help!

rogue 2022-12-19 13:50

[QUOTE=Trying2Sieve;620281][CODE]srsieve2.exe -W 6 -s "7*10^n+1" -n 1000000 -N 8000000 -f A -o "8_million.txt"[/CODE]

I set srsieve to use 6 cores[/QUOTE]

Make sure that you sieve deeply enough otherwise you are wasting time with PRP testing. With numbers of that size you should estimate the time for a test for around n=5000000 and run srsieve2 until the removal rate is close to that.

If you have a GPU use srsieve2cl as it might be faster than the CPU, even if running on 6 cores.

Trying2Sieve 2022-12-22 10:34

[QUOTE=rogue;620302]Make sure that you sieve deeply enough otherwise you are wasting time with PRP testing. With numbers of that size you should estimate the time for a test for around n=5000000 and run srsieve2 until the removal rate is close to that.

If you have a GPU use srsieve2cl as it might be faster than the CPU, even if running on 6 cores.[/QUOTE]

Can an interrupted sieving session be restarted with like a possible -i option?

It was not clear based on the available online instructions.

rogue 2022-12-22 13:35

[QUOTE=Trying2Sieve;620556]Can an interrupted sieving session be restarted with like a possible -i option?

It was not clear based on the available online instructions.[/QUOTE]

Yes. The first line will tell you the last prime that was tested. Unless you using -p, it will start from that prime.

For all sieves in the framework -i is used to continue sieving. The input file needs to have a format that is compatible with the specific sieve you are using. Some sieves can read and write multiple formats. Some sieves have a format that is unique to that sieve.

rogue 2022-12-22 13:49

I have posted srsieve2 1.6.8 (and only srsieve2) over at sourceforge. This fixes an issue where an invalid factor is found due to an inadvertent conversion of a 32-bit value to a 16-bit value. This seems to only occur when overriding -U/-V/-X.

All invalid factors immediately stop srsieve2, so if srsieve2 doesn't stop, then you don't have to update.

rebirther 2022-12-22 16:01

Is there a possibility for srsieve2 to write the sieve output every 15min / 30min not 1h?

rogue 2022-12-22 16:34

[QUOTE=rebirther;620577]Is there a possibility for srsieve2 to write the sieve output every 15min / 30min not 1h?[/QUOTE]

Not without a code change. Please explain the need.

rebirther 2022-12-22 16:52

[QUOTE=rogue;620578]Not without a code change. Please explain the need.[/QUOTE]

The time loosing is less for different reasons like crashes. But if its too difficulty or time consuming then let it so.

Happy5214 2022-12-22 17:20

[QUOTE=rebirther;620579]The time loosing is less for different reasons like crashes. But if its too difficulty or time consuming then let it so.[/QUOTE]

If you have the ability to build the code yourself, it can be changed by modifying the constant [C]CHECKPOINT_SECONDS[/C] defined in line 17 of [C]core/FactorApp.cpp[/C].

Citrix 2022-12-22 18:48

[QUOTE=rogue;620567]I have posted srsieve2 1.6.8 (and only srsieve2) over at sourceforge. This fixes an issue where an invalid factor is found due to an inadvertent conversion of a 32-bit value to a 16-bit value. This seems to only occur when overriding -U/-V/-X.

All invalid factors immediately stop srsieve2, so if srsieve2 doesn't stop, then you don't have to update.[/QUOTE]

Thank you!
What is the difference between -U/-V/-X and -q/-Q?
I believe the code could be improved upon so it does not use so much memory for large values and the program can calculate the fastest values automatically.

rogue 2022-12-22 18:58

[QUOTE=rebirther;620579]The time loosing is less for different reasons like crashes. But if its too difficulty or time consuming then let it so.[/QUOTE]

I assume you mean a crash of the computer and not of the software.

You lose an hour, at most, if you experience a crash.

rogue 2022-12-22 19:12

[QUOTE=Citrix;620590]Thank you!
What is the difference between -U/-V/-X and -q/-Q?
I believe the code could be improved upon so it does not use so much memory for large values and the program can calculate the fastest values automatically.[/QUOTE]

Look at the details I have in CHANGES.txt for version 1.6.0. That explains, although it does not have all of the gory details.

-U/-V/-X will impact the potential q that srsieve2 can use. If you change those values and use with -Q you will see it output a different set of potential q for each.

Are you referring to CPU memory or GPU memory?

I'm not certain what you mean by "calculate the fastest values automatically". I think you are suggesting that the program run thru all combinations of -U/-V/-X to find the best Q. If you are, then I haven't thought about it. I'm not saying it is impossible. I'm just saying that I haven't thought about it. I do not think it would be a small task. I do know (based upon testing) that although some values for -U/-V/-X/q can be faster than others, there is no guarantee that you can use those values when running in the GPU. In other words, some combinations work fine and others cannot allocate enough GPU memory. One of the limitations is that I have no programmatic way of determining if the kernel is going to use too much memory until I create it. I might be able to catch the "out of resources" when it creates the kernel, but handling that and choosing different inputs is a lot of work.

Citrix 2022-12-22 19:34

[CODE]
BASE_MULTIPLE: least exponent Q for which sieving in subsequence base b^Q will be allowed.
[/CODE]

If you are defining the Q value with -U then what is the purpose of -Q?

[CODE]
It is recommend to play with these multipliers to find the optimal values as the provided
defaults are from sr1sieve and sr2sieve and some values can provide faster sieving.
The recommendation is to sieve a range of 1e9 and see how long it takes (use sr2sieve.log
to see how long it actually takes). Vary -U, -V, and-X evaluate the results. With a
script and multiple cores, this could done relatively quickly for most inputs.
[/CODE]

The program should be able to look at differences of consecutive terms to find the best Q/U/V/X value automatically. The best U/V/X value can be calculated than checking every value and then deciding which one is best.

rogue 2022-12-22 20:16

[QUOTE=Citrix;620597][CODE]
BASE_MULTIPLE: least exponent Q for which sieving in subsequence base b^Q will be allowed.
[/CODE]

If you are defining the Q value with -U then what is the purpose of -Q?

[CODE]
It is recommend to play with these multipliers to find the optimal values as the provided
defaults are from sr1sieve and sr2sieve and some values can provide faster sieving.
The recommendation is to sieve a range of 1e9 and see how long it takes (use sr2sieve.log
to see how long it actually takes). Vary -U, -V, and-X evaluate the results. With a
script and multiple cores, this could done relatively quickly for most inputs.
[/CODE]

The program should be able to look at differences of consecutive terms to find the best Q/U/V/X value automatically. The best U/V/X value can be calculated than checking every value and then deciding which one is best.[/QUOTE]

I'm sorry for the confusion between -q and -Q. I flipped their usage more than once. -Q will list the possible Q values associated with BASE_MULTIPLE. -q will let you specify which Q to use instead of the one with the lowest estimated work.

Although it would be nice to choose the best Q/U/V/X automatically, but it does not do that at this time. I do not know the effort to do that. I will look into it.

You will probably see that BASE_MULTIPLE != Q. BASE_MULTIPLE is used to determine the possible Q. I didn't define it well enough in CHANGES.txt.

Happy5214 2022-12-23 14:43

[QUOTE=rogue;620595]-U/-V/-X will impact the potential q that srsieve2 can use. If you change those values and use with -Q you will see it output a different set of potential q for each.[/QUOTE]

I tested -U/-V/-X with a bunch of different values and a dataset with 5 sequences, and it generated the exact same Q sets each time (at least when the values were valid/not out of range), up to having the same MD5 hashes.

rogue 2022-12-23 16:39

[QUOTE=Happy5214;620682]I tested -U/-V/-X with a bunch of different values and a dataset with 5 sequences, and it generated the exact same Q sets each time (at least when the values were valid/not out of range), up to having the same MD5 hashes.[/QUOTE]

That is not what I have seen:

[code]
E:\stuff\mtsieve>srsieve2 -i sr_2.abcd -p 10747286000000 -osr_2_new.abcd -U49 -Q
srsieve2 v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Sieving with single sequence c=1 logic for p >= 10747286000000
BASE_MULTIPLE = 98, POWER_RESIDUE_LCM = 2352, LIMIT_BASE = 2352
q = 98 with 2 subseq yields work = 544186
q = 196 with 2 subseq yields work = 410726
q = 294 with 2 subseq yields work = 362611
q = 392 with 4 subseq yields work = 364112
q = 588 with 2 subseq yields work = 333178
q = 784 with 8 subseq yields work = 406627
q = 1176 with 4 subseq yields work = 402298
q = 2352 with 8 subseq yields work = 618324
Split 1 base 2 sequence into 2 base 2^588 sequences.
[/code]

[code]
E:\stuff\mtsieve>srsieve2 -i sr_2.abcd -p 10747286000000 -osr_2_new.abcd -U49 -V270 -Q
srsieve2 v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Sieving with single sequence c=1 logic for p >= 10747286000000
BASE_MULTIPLE = 98, POWER_RESIDUE_LCM = 26460, LIMIT_BASE = 26460
q = 98 with 2 subseq yields work = 4897670
q = 196 with 2 subseq yields work = 3696538
q = 294 with 2 subseq yields work = 3263501
q = 490 with 9 subseq yields work = 3744367
q = 588 with 2 subseq yields work = 2998598
q = 882 with 2 subseq yields work = 3175200
q = 980 with 9 subseq yields work = 3800437
q = 1470 with 9 subseq yields work = 4320313
q = 1764 with 2 subseq yields work = 4315853
q = 2646 with 6 subseq yields work = 5879299
q = 2940 with 9 subseq yields work = 6505909
q = 4410 with 9 subseq yields work = 8959759
q = 5292 with 6 subseq yields work = 10366839
q = 8820 with 9 subseq yields work = 16683496
q = 13230 with 27 subseq yields work = 24822390
q = 26460 with 27 subseq yields work = 48591513
[/code]

[code]
E:\stuff\mtsieve>srsieve2 -i sr_2.abcd -p 10747286000000 -osr_2_new.abcd -Q
srsieve2 v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Sieving with single sequence c=1 logic for p >= 10747286000000
BASE_MULTIPLE = 30, POWER_RESIDUE_LCM = 720, LIMIT_BASE = 720
q = 30 with 9 subseq yields work = 373119
q = 60 with 9 subseq yields work = 266145
q = 90 with 9 subseq yields work = 219699
q = 120 with 18 subseq yields work = 235598
q = 180 with 9 subseq yields work = 162150
q = 240 with 36 subseq yields work = 229270
q = 360 with 18 subseq yields work = 152909
q = 720 with 36 subseq yields work = 166099
Split 1 base 2 sequence into 18 base 2^360 sequences.
[/code]

Citrix 2022-12-23 17:19

The -U by default should be set to gcd(difference of all consecutive candidates in the file). For the file I sent you it should be -U441. You can try running the code with -U441 -Q to find the best value then. There might not be a significant difference.

Once that is set
BASE_MULTIPLE = 2*U
POWER_RESIDUE_LCM = 60
LIMIT_BASE=x*BASE_MULTIPLE; where x<30

Assuming U is set correctly:-
Generally the larger the selected LIMIT_BASE the less amount of work will need to be done but the overhead goes up making the program slower. A lower value of x might be better.

For POWER_RESIDUE_LCM -the higher the value the faster it should be but the overhead goes up. Rarely is it significantly beneficial to keep it above 60.
The BASE_MULTIPLE does not need to be included in the POWER_RESIDUE_LCM
POWER_RESIDUE_LCM is also independent of LIMIT_BASE

Happy5214 2022-12-23 19:02

1 Attachment(s)
[QUOTE=rogue;620692]That is not what I have seen:

[code]
E:\stuff\mtsieve>srsieve2 -i sr_2.abcd -p 10747286000000 -osr_2_new.abcd -U49 -Q
[ciip]
[/code]

[code]
E:\stuff\mtsieve>srsieve2 -i sr_2.abcd -p 10747286000000 -osr_2_new.abcd -U49 -V270 -Q
[clip]
[/code]

[code]
E:\stuff\mtsieve>srsieve2 -i sr_2.abcd -p 10747286000000 -osr_2_new.abcd -Q
[clip]
[/code][/QUOTE]

I had the old SVN build (v1.6.6), but updating to the latest version didn't help. I adapted each of your calls to my folder and re-ran, and here's what I got:
[code]$../../../bin/srsieve2-svn-live -i b2_n.abcd -p 10747286000000 -osr_2_new.abcd -U49 -Q
srsieve2 v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Must use generic sieving logic because -l was not specified for mutiple sequences
Sieving with generic logic for p >= 10747286000000
q = 2 with 5 subseq yields bs = 1116, gs = 224, work = 2239
q = 4 with 5 subseq yields bs = 791, gs = 158, work = 1585
q = 8 with 8 subseq yields bs = 711, gs = 88, work = 1423
q = 16 with 16 subseq yields bs = 711, gs = 44, work = 1432
q = 32 with 32 subseq yields bs = 711, gs = 22, work = 1450
q = 64 with 64 subseq yields bs = 711, gs = 11, work = 1485
q = 3 with 6 subseq yields bs = 998, gs = 167, work = 2004
q = 6 with 6 subseq yields bs = 707, gs = 118, work = 1421
q = 12 with 6 subseq yields bs = 502, gs = 83, work = 1010
q = 24 with 9 subseq yields bs = 434, gs = 48, work = 886
q = 48 with 18 subseq yields bs = 434, gs = 24, work = 906
q = 96 with 36 subseq yields bs = 434, gs = 12, work = 947
q = 192 with 72 subseq yields bs = 435, gs = 6, work = 1030
q = 9 with 13 subseq yields bs = 855, gs = 65, work = 1711
q = 18 with 13 subseq yields bs = 604, gs = 46, work = 1219
q = 36 with 13 subseq yields bs = 421, gs = 33, work = 880
q = 72 with 18 subseq yields bs = 348, gs = 20, work = 765
q = 144 with 36 subseq yields bs = 348, gs = 10, work = 823
q = 288 with 72 subseq yields bs = 348, gs = 5, work = 938
q = 576 with 144 subseq yields bs = 435, gs = 2, work = 1183
q = 5 with 21 subseq yields bs = 1449, gs = 69, work = 2909
q = 10 with 21 subseq yields bs = 1021, gs = 49, work = 2065
q = 20 with 21 subseq yields bs = 736, gs = 34, work = 1472
q = 40 with 32 subseq yields bs = 625, gs = 20, work = 1305
q = 80 with 64 subseq yields bs = 625, gs = 10, work = 1346
q = 160 with 128 subseq yields bs = 625, gs = 5, work = 1428
q = 320 with 256 subseq yields bs = 782, gs = 2, work = 1620
q = 15 with 26 subseq yields bs = 926, gs = 36, work = 1882
q = 30 with 26 subseq yields bs = 667, gs = 25, work = 1348
q = 60 with 26 subseq yields bs = 463, gs = 18, work = 983
q = 120 with 37 subseq yields bs = 379, gs = 11, work = 884
q = 240 with 74 subseq yields bs = 417, gs = 5, work = 984
q = 480 with 148 subseq yields bs = 348, gs = 3, work = 1187
q = 960 with 296 subseq yields bs = 522, gs = 1, work = 1608
q = 45 with 58 subseq yields bs = 794, gs = 14, work = 1660
q = 90 with 58 subseq yields bs = 556, gs = 10, work = 1222
q = 180 with 58 subseq yields bs = 397, gs = 7, work = 952
q = 360 with 76 subseq yields bs = 348, gs = 4, work = 934
q = 720 with 152 subseq yields bs = 348, gs = 2, work = 1216
q = 1440 with 304 subseq yields bs = 348, gs = 1, work = 1781
q = 2880 with 608 subseq yields bs = 175, gs = 1, work = 3042
q = 1 with 5 subseq yields bs = 1582, gs = 316, work = 3164
Split 5 base 2 sequences into 18 base 2^72 sequences.
Sieve started: 10747286e6 < p < 2^62 with 12777 terms (500117 < n < 999973, k*2^n-1)
Increasing worksize to 256000 since each chunk is tested in less than a second
^CCTRL-C accepted. Threads will stop after sieving to 10747294163971
Sieve interrupted at p=10747294163971.
CPU time: 1.70 sec. (0.01 sieving) (1.00 cores)
12777 terms written to sr_2_new.abcd
Primes tested: 272000. Factors found: 0. Remaining terms: 12777. Time: 1.70 seconds.
[/code]

[code]$../../../bin/srsieve2-svn-live -i b2_n.abcd -p 10747286000000 -osr_2_new.abcd -U49 -V270 -Q
[i]won't bother copying here, since it's the exact same output as above apart from the timing details[/i]
[/code]

[code]$ ../../../bin/srsieve2-svn-live -i b2_n.abcd -p 10747286000000 -osr_2_new.abcd -Q
[i]Ditto as above[/i]
[/code]

I originally ran the tests with srsieve2cl 1.6.6 with similar results, but the above output was from regular CPU srsieve2 1.6.8 (from SVN) to match your output. It's on Linux, if that makes a difference. I'll attach the input file here.

PS I noticed your input was a single sequence, while mine was multiple sequences. Could that make a difference?

Citrix 2022-12-23 19:36

[QUOTE=rogue;620692]

[code]
E:\stuff\mtsieve>srsieve2 -i sr_2.abcd -p 10747286000000 -osr_2_new.abcd -U49 -Q
srsieve2 v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Sieving with single sequence c=1 logic for p >= 10747286000000
BASE_MULTIPLE = 98, POWER_RESIDUE_LCM = 2352, LIMIT_BASE = 2352
q = 98 with 2 subseq yields work = 544186
q = 196 with 2 subseq yields work = 410726
q = 294 with 2 subseq yields work = 362611
q = 392 with 4 subseq yields work = 364112
q = 588 with 2 subseq yields work = 333178
q = 784 with 8 subseq yields work = 406627
q = 1176 with 4 subseq yields work = 402298
q = 2352 with 8 subseq yields work = 618324
Split 1 base 2 sequence into 2 base 2^588 sequences.
[/code]

Assuming you are using the file I gave you...the number of subseq seem incorrect. It should be:-

[CODE]
q = 98 with 2 subseq ...
q = 196 with 3 subseq
q = 294 with 2 subseq
q = 392 with 6 subseq
q = 588 with 3 subseq
q = 784 with 12 subseq
Split 1 base 2 sequence into 3 base 2^588 sequences.
[/CODE]

rogue 2022-12-23 22:15

[QUOTE=Citrix;620696]The -U by default should be set to gcd(difference of all consecutive candidates in the file). For the file I sent you it should be -U441. You can try running the code with -U441 -Q to find the best value then. There might not be a significant difference.

Once that is set
BASE_MULTIPLE = 2*U
POWER_RESIDUE_LCM = 60
LIMIT_BASE=x*BASE_MULTIPLE; where x<30

Assuming U is set correctly:-
Generally the larger the selected LIMIT_BASE the less amount of work will need to be done but the overhead goes up making the program slower. A lower value of x might be better.

For POWER_RESIDUE_LCM -the higher the value the faster it should be but the overhead goes up. Rarely is it significantly beneficial to keep it above 60.
The BASE_MULTIPLE does not need to be included in the POWER_RESIDUE_LCM
POWER_RESIDUE_LCM is also independent of LIMIT_BASE[/QUOTE]

You are correct on LIMIT_BASE. I have to fix that.

2 subseq is correct. Both sr1sieve and srsieve2 give the same number of subsequences. I'm not certain what you are basing that on.

-U/-V/-X are only used if you have a single sequence or if you have multiple sequences and are using Legendre tables.

Happy5214 2022-12-23 23:11

[QUOTE=rogue;620724]-U/-V/-X are only used if you have a single sequence or if you have multiple sequences and are using Legendre tables.[/QUOTE]

That isn't really documented well in the CHANGES.txt (and really there needs to be a single place for up-to-date user documentation and parameter usage hints that doesn't require looking through changelogs or forum posts). I'll try it with [c]-l[/c] tomorrow. Are Citrix's suggested parameters sensible as a scripting guide (or are you actually planning to add automated parameterization to the code itself, making scripting unnecessary)?

Citrix 2022-12-23 23:12

[QUOTE=rogue;620724]

2 subseq is correct. Both sr1sieve and srsieve2 give the same number of subsequences. I'm not certain what you are basing that on.

[/QUOTE]

If you take all the n values in the file and calculate n%588-I get the following 3 residue values- 0, 294, 441
That would be 3 subseq

rogue 2022-12-24 01:07

[QUOTE=Citrix;620729]If you take all the n values in the file and calculate n%588-I get the following 3 residue values- 0, 294, 441
That would be 3 subseq[/QUOTE]

I will take a look. Since srsieve2 is based upon sr1sieve, then sr1sieve is the original of the mistake.

rogue 2022-12-24 01:08

[QUOTE=Happy5214;620728]That isn't really documented well in the CHANGES.txt (and really there needs to be a single place for up-to-date user documentation and parameter usage hints that doesn't require looking through changelogs or forum posts). I'll try it with [c]-l[/c] tomorrow. Are Citrix's suggested parameters sensible as a scripting guide (or are you actually planning to add automated parameterization to the code itself, making scripting unnecessary)?[/QUOTE]

[URL="https://www.mersenneforum.org/rogue/mtsieve.html"]This[/URL] has the best documentation, although it isn't perfect. I can certain add details for -U/-V/-X.

Citrix 2022-12-24 06:12

[QUOTE=rogue;620753]I will take a look. Since srsieve2 is based upon sr1sieve, then sr1sieve is the original of the mistake.[/QUOTE]

The sr1sieve code seems correct though the first does produce a wrong subseq and the second one does not. I ran both with -Q4 option.

[CODE]
ABCD 21*2^$a+1 [1000] // Sieved to 1000000000 with srsieve
4004
2002
5005
3003
2002
5005
3003
2002
5005
3003
2002
5005
3003
[/CODE]

[CODE]
ABCD 21*2^$a+1 [1000] // Sieved to 1000000000 with srsieve
4
2
5
3
2
5
3
2
5
3
2
5
3
2
5
3
[/CODE]

Happy5214 2022-12-24 11:33

[QUOTE=Citrix;620831]The sr1sieve code seems correct though the first does produce a wrong subseq and the second one does not. I ran both with -Q4 option.[/QUOTE]

Passing 4 to [C]-Q[/C] doesn't do anything useful. Did you mean to pass it to [C]-q[/C] (note the uppercase/lowercase distinction)?

[QUOTE=rogue;620754][URL="https://www.mersenneforum.org/rogue/mtsieve.html"]This[/URL] has the best documentation, although it isn't perfect. I can certain add details for -U/-V/-X.[/QUOTE]

In addition to a basic description of what the parameters mean, a detailed guide on how to tune them (perhaps pulling from this thread and the changelogs) would be helpful.

Citrix 2022-12-24 11:54

[QUOTE=Happy5214;620846]Passing 4 to [C]-Q[/C] doesn't do anything useful. Did you mean to pass it to [C]-q[/C] (note the uppercase/lowercase distinction)?
[/QUOTE]

This is for sr1sieve as mentioned and not srsieve2.
I would recommend you read the source code to better understand on how to fine tune the variables.

Happy5214 2022-12-24 12:46

[QUOTE=Citrix;620847]This is for sr1sieve as mentioned and not srsieve2.
I would recommend you read the source code to better understand on how to fine tune the variables.[/QUOTE]

Sorry, I blew past that note. Perhaps switching the case between sr1sieve and srsieve2 is the source of Mark's [C]-q[/C]/[C]-Q[/C] conflation in the docs (along with the obvious similarity of the letters)?

For whatever reason, I cannot really grasp srsieve's basic algorithms. The variables are in practical effect parameters to a black box to me. (So perhaps a basic high-level algorithmic explanation should go with the documentation too, if that's reasonable.)

Happy5214 2022-12-24 13:35

[QUOTE=Citrix;620729]If you take all the n values in the file and calculate n%588-I get the following 3 residue values- 0, 294, 441
That would be 3 subseq[/QUOTE]

294^[I]i[/I] mod 588 = 0 for all integers [I]i[/I] > 1. I don't know if that's in play here.

storm5510 2022-12-24 16:55

Regarding [I]srsieve2[/I]: Is there a relationship between the number of threads a CPU has and the -W switch? I have an older machine with 4 cores, or 4 threads. I set the -W switch to 6. It runs. I was not expecting it to. :huh:

Happy5214 2022-12-24 17:25

[QUOTE=storm5510;620864]Regarding [I]srsieve2[/I]: Is there a relationship between the number of threads a CPU has and the -W switch? I have an older machine with 4 cores, or 4 threads. I set the -W switch to 6. It runs. I was not expecting it to. :huh:[/QUOTE]

The number of threads on your CPU (real and virtual; sieving behaves well with hyperthreading, if your CPU has that feature) should be the ceiling for the [C]-W[/C] value. Setting it any higher will cause the worker threads to clash with each other.

storm5510 2022-12-24 17:56

[QUOTE=Happy5214;620865]The number of threads on your CPU (real and virtual; sieving behaves well with hyperthreading, if your CPU has that feature) should be the ceiling for the [C]-W[/C] value. Setting it any higher will cause the worker threads to clash with each other.[/QUOTE]

I went back and checked it. 4 threads. It is running fine at -W6 so I will leave it alone. I am running a job for CRUS that I would prefer not to stop.

MisterBitcoin 2022-12-25 11:36

[CODE]C:\Users\Sydekum\Documents\other stuff\srbsieve_full>srsieve2cla.exe -i b3_n.abcd -P 250e9 -w4000
srsieve2cl v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Must use generic sieving logic because -l was not specified for mutiple sequences
Sieving with generic logic for p >= 233657417857
Split 439 base 3 sequences into 2772 base 3^24 sequences.
1 warning generated.
GPU primes per worker is 49152
Sieve started: 233657417857 < p < 25e10 with 3780612 terms (500000 < n < 1000000, k*3^n+1) (expecting 9739 factors)

OpenCL Error: Out of resources
in call to clEnqueueReadBuffer
argument: factorCount

C:\Users\Sydekum\Documents\other stuff\srbsieve_full>[/CODE]

Looks like the input is a little bit too large :smile: I used -w to decrease the size, but that didnt help.
2k bases seem to work.

[CODE]C:\Users\Sydekum\Documents\other stuff\srbsieve_full>srsieve2cla.exe -i sr_61.abcd -P 1e16
srsieve2cl v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Must use generic sieving logic because -l was not specified for mutiple sequences
Sieving with generic logic for p >= 1000000000000000
Split 2 base 61 sequences into 14 base 61^30 sequences.
1 warning generated.
GPU primes per worker is 49152
Sieve started: 1e15 < p < 1e16 with 16620 terms (600003 < n < 999951, k*61^n-1) (expecting 1039 factors)
p=1000003259582399, 778.2K p/sec, no factors found, 0.0% done. ETC 2033-08-28 07:44[/CODE]

Also; how can I see that 1 warning?

rogue 2022-12-25 15:10

[QUOTE=MisterBitcoin;620897][CODE]C:\Users\Sydekum\Documents\other stuff\srbsieve_full>srsieve2cla.exe -i b3_n.abcd -P 250e9 -w4000
srsieve2cl v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Must use generic sieving logic because -l was not specified for mutiple sequences
Sieving with generic logic for p >= 233657417857
Split 439 base 3 sequences into 2772 base 3^24 sequences.
1 warning generated.
GPU primes per worker is 49152
Sieve started: 233657417857 < p < 25e10 with 3780612 terms (500000 < n < 1000000, k*3^n+1) (expecting 9739 factors)

OpenCL Error: Out of resources
in call to clEnqueueReadBuffer
argument: factorCount

C:\Users\Sydekum\Documents\other stuff\srbsieve_full>[/CODE]

Looks like the input is a little bit too large :smile: I used -w to decrease the size, but that didnt help.
2k bases seem to work.

[CODE]C:\Users\Sydekum\Documents\other stuff\srbsieve_full>srsieve2cla.exe -i sr_61.abcd -P 1e16
srsieve2cl v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Must use generic sieving logic because -l was not specified for mutiple sequences
Sieving with generic logic for p >= 1000000000000000
Split 2 base 61 sequences into 14 base 61^30 sequences.
1 warning generated.
GPU primes per worker is 49152
Sieve started: 1e15 < p < 1e16 with 16620 terms (600003 < n < 999951, k*61^n-1) (expecting 1039 factors)
p=1000003259582399, 778.2K p/sec, no factors found, 0.0% done. ETC 2033-08-28 07:44[/CODE]

Also; how can I see that 1 warning?[/QUOTE]

The warning is not generated by anything in the mtsieve framework.

-w affects worksize for CPU workers. -g affects worksize for GPU workers. In this case you are not using a CPU worker so -w has no affect.

What I have written below assumes you are not running multiple GPU heavy programs concurrently as they will "fight" one another for memory.

You have a few options for reducing kernel memory usage:
[LIST][*]Use -Q to see how many subsequences there are for each q then force the selection a Q with fewer subsequences by using -q.[*]Use -b to adjust the number of baby steps. The default is 1.0. Reduce to decrease the number of baby steps.[*]Use -K to split the sequences across multiple kernels. This will have the biggest impact on speed.[*]Use -M to reduce the size of the buffer for holding factors.[/LIST]
These can be used in any combination. Determining the optimal values for your system will require experimentation.

storm5510 2022-12-25 18:45

There is one thing with [I]srsieve2/srsieve2cl[/I] which aggravates the crap out of me to no end! It is the overwriting of an existing line on the screen by a new line. This prevents any comparisons with anything above. Administrator Command Prompt or Admistrator Powershell, it makes no difference! :censored:

rogue 2022-12-25 18:50

[QUOTE=storm5510;620917]There is one thing with [I]srsieve2/srsieve2cl[/I] which aggravates the crap out of me to no end! It is the overwriting of an existing line on the screen by a new line. This prevents any comparisons with anything above. Administrator Command Prompt or Admistrator Powershell, it makes no difference! :censored:[/QUOTE]

Why?

rebirther 2022-12-25 19:07

[QUOTE=storm5510;620917]There is one thing with [I]srsieve2/srsieve2cl[/I] which aggravates the crap out of me to no end! It is the overwriting of an existing line on the screen by a new line. This prevents any comparisons with anything above. Administrator Command Prompt or Admistrator Powershell, it makes no difference! :censored:[/QUOTE]

Iam using srsieve2 1.6.6 and you have line by line but you dont have the new options there. I have seen this with the same version of llr with win7 vs win10, one is overwriting the first line the other is using a new line.

kruoli 2022-12-25 19:10

What he is requesting is the possibility to use [C]\n[/C] instead of [C]\r[/C] at the end of the status update line. In Linux, one could use [C]tr[/C] to circumvent this, but this is not so easy on Windows when having no programming skills.

The problem with [C]\r[/C] is that we lose previous status lines and cannot get them back, since they are also not written to a log file or something. Of course this is not required for the program's function, but I can understand that some might be interested in having a look on how the removal rate has changed during a run. Currently, you would have to manually write intermediate values down to not lose them.

MisterBitcoin 2022-12-25 20:58

[QUOTE=rogue;620909]The warning is not generated by anything in the mtsieve framework.

-w affects worksize for CPU workers. -g affects worksize for GPU workers. In this case you are not using a CPU worker so -w has no affect.

What I have written below assumes you are not running multiple GPU heavy programs concurrently as they will "fight" one another for memory.

You have a few options for reducing kernel memory usage:
[LIST][*]Use -Q to see how many subsequences there are for each q then force the selection a Q with fewer subsequences by using -q.[*]Use -b to adjust the number of baby steps. The default is 1.0. Reduce to decrease the number of baby steps.[*]Use -K to split the sequences across multiple kernels. This will have the biggest impact on speed.[*]Use -M to reduce the size of the buffer for holding factors.[/LIST]
These can be used in any combination. Determining the optimal values for your system will require experimentation.[/QUOTE]

Thanks for the reply, fortunatly I was able to get it to work with the following line.
[CODE]srsieve2cl.exe -i b3_n.abcd -g4 -P 500e9 -M 10 -b 0.2500[/CODE]

However that reduces the speed by a lot, resulting in 8.928K p/sec only. (I have 47.28K p/sec when using 10 out of 20 threads from my CPU).
Looks like my GPU is getting old, so I might have to replace it next year. So far for gaming stuff it still works, so thats okay!

MisterBitcoin 2022-12-25 21:14

Just as question:

Why is srsieve2 1.1 sieving with >50K per sec and newest srsieve2 with "just" 17K per sec with the same command?
I have dumped my process done with 1.1 and continued the sieving where I started it; just in case.

Citrix 2022-12-25 21:38

[QUOTE=Happy5214;620848]

For whatever reason, I cannot really grasp srsieve's basic algorithms. The variables are in practical effect parameters to a black box to me. (So perhaps a basic high-level algorithmic explanation should go with the documentation too, if that's reasonable.)[/QUOTE]

[url]https://en.wikipedia.org/wiki/Discrete_logarithm[/url]

If there is a particular sequence you need help with ... you can post the input file and I can suggest the best parameters.

rogue 2022-12-26 00:10

[QUOTE=MisterBitcoin;620927]Just as question:

Why is srsieve2 1.1 sieving with >50K per sec and newest srsieve2 with "just" 17K per sec with the same command?
I have dumped my process done with 1.1 and continued the sieving where I started it; just in case.[/QUOTE]

Good question. I do not know the answer.

rogue 2022-12-26 00:21

[QUOTE=kruoli;620920]What he is requesting is the possibility to use [C]\n[/C] instead of [C]\r[/C] at the end of the status update line. In Linux, one could use [C]tr[/C] to circumvent this, but this is not so easy on Windows when having no programming skills.

The problem with [C]\r[/C] is that we lose previous status lines and cannot get them back, since they are also not written to a log file or something. Of course this is not required for the program's function, but I can understand that some might be interested in having a look on how the removal rate has changed during a run. Currently, you would have to manually write intermediate values down to not lose them.[/QUOTE]

While sieving it gives you the removal rate over a period of time. You will see the text "last xx min" which references how many minutes were used to compute the removal rate.

I honestly do not see a value in knowing how much that rate has changed over time. To me this is purely statistical for someone using the software. All I can suggest is that you install a compiler, download the source, change the code, build, and then use what you build. I think you will quickly see that using \n creates a huge amount of useless noise when the programs are running.

storm5510 2022-12-26 03:48

[QUOTE=rogue;620918]Why?[/QUOTE]

I will not bother to explain the purpose further. I am used to the output of [I]mfaktc, gpuOwl[/I], and [I]Prime95[/I] I suppose.

storm5510 2022-12-27 00:29

[I]Srsieve2cl:[/I]

[QUOTE]Must use generic sieving logic because -l was not specified for multiple sequences.[/QUOTE]

This is for Lengendre tables. How do you specify [B]-l[/B] for multiple sequences?

Trying2Sieve 2022-12-27 10:42

[QUOTE=Happy5214;620850]294^[I]i[/I] mod 588 = 0 for all integers [I]i[/I] > 1. I don't know if that's in play here.[/QUOTE]

I have 20 threads on my laptop and I can usually only get [CODE]-W 6[/CODE] to remain stable.

I'm fine with that though, it cranks pretty fast.

I'm out to p > 15 trillion over an exponent range of 1,000,000 to 8,000,000 after only about one week.

Trying2Sieve 2022-12-27 10:49

[QUOTE=MisterBitcoin;620927]Just as question:

Why is srsieve2 1.1 sieving with >50K per sec and newest srsieve2 with "just" 17K per sec with the same command?
I have dumped my process done with 1.1 and continued the sieving where I started it; just in case.[/QUOTE]

It might depend on the features of your processor.

[URL="https://gothicchess.info/images/sieve_rate.png"]https://gothicchess.info/images/sieve_rate.png[/URL]

rogue 2022-12-27 14:02

[QUOTE=storm5510;620983][I]Srsieve2cl:[/I]

This is for Lengendre tables. How do you specify [B]-l[/B] for multiple sequences?[/QUOTE]

Just add -l with some non-zero value. But srsievecl does not have a kernel for Legendre tables + multiple sequences. It can only run in the CPU. In the future I will create an OpenCL kernel to support multiple sequences with the Legendre tables. I just don't know when I'm going to do it.

Happy5214 2022-12-27 18:27

[QUOTE=Citrix;620928][url]https://en.wikipedia.org/wiki/Discrete_logarithm[/url]

If there is a particular sequence you need help with ... you can post the input file and I can suggest the best parameters.[/QUOTE]

I know a bit about the discrete log. It may be laziness on my part for not spending enough time looking at the code?

In any event, I ran it for a few hours with the default parameters on my GPU, then ran it on sr2sieve (the original program). sr2sieve was twice as fast. I simply can't get my GeForce RTX 2060 to run any faster with srsieve2 than the old programs on my 10th-gen eight-core Intel Core i7. It's not even competitive.

rogue 2022-12-27 22:29

[QUOTE=Happy5214;621090]I know a bit about the discrete log. It may be laziness on my part for not spending enough time looking at the code?

In any event, I ran it for a few hours with the default parameters on my GPU, then ran it on sr2sieve (the original program). sr2sieve was twice as fast. I simply can't get my GeForce RTX 2060 to run any faster with srsieve2 than the old programs on my 10th-gen eight-core Intel Core i7. It's not even competitive.[/QUOTE]

I assume you mean sr2sieve vs srsieve2cl.

When you say "twice as fast", what are you measuring? If you are measuring p/sec, then that is not a valid comparison because they are computed differently. The proper way to compare two two is to run a range of 1e10, e.g. -p1e10 -P2e10, and see which one completes the range first.

Note that I'm not disagreeing with your results, just that you need to ensure that you are comparing apples to apples and not apples to oranges.

What command line options are you using for srsieve2cl?

Happy5214 2023-01-01 20:29

[QUOTE=rogue;621109]I assume you mean sr2sieve vs srsieve2cl.

When you say "twice as fast", what are you measuring? If you are measuring p/sec, then that is not a valid comparison because they are computed differently. The proper way to compare two two is to run a range of 1e10, e.g. -p1e10 -P2e10, and see which one completes the range first.

Note that I'm not disagreeing with your results, just that you need to ensure that you are comparing apples to apples and not apples to oranges.

What command line options are you using for srsieve2cl?[/QUOTE]

On an srsieve2cl run with default parameters except [c]-g 100[/c] (that was a leftover in my runner script from an earlier dataset) and [c]-G 1[/c], it completed the range 1e12 to 12e11 (a range of size 2e11) in 6588.37 seconds. Earlier tests (with length of 1e9) on this dataset showed that the default [i]q[/i] of 72 was optimal/fastest, and that changing [c]-g[/c] and [c]-G[/c] had only a small impact on runtime. I did have some success during testing with the [C]-b[/C] flag, as setting that to 0.3 shaved the testing time from 35.16 seconds to 30.81 seconds (both with default [c]-g[/c] and [c]-G 1[/c]), but that wouldn't make up for the disparity seen below.

sr2sieve completed the range 14e11 to 16e11 (same size as above), with 4 threads, cached Legendre tables, and default settings otherwise, in 3201.256 seconds. As you can see, it's more than a 2:1 margin.

rogue 2023-01-01 21:38

[QUOTE=Happy5214;621409]On an srsieve2cl run with default parameters except [c]-g 100[/c] (that was a leftover in my runner script from an earlier dataset) and [c]-G 1[/c], it completed the range 1e12 to 12e11 (a range of size 2e11) in 6588.37 seconds. Earlier tests (with length of 1e9) on this dataset showed that the default [i]q[/i] of 72 was optimal/fastest, and that changing [c]-g[/c] and [c]-G[/c] had only a small impact on runtime. I did have some success during testing with the [C]-b[/C] flag, as setting that to 0.3 shaved the testing time from 35.16 seconds to 30.81 seconds (both with default [c]-g[/c] and [c]-G 1[/c]), but that wouldn't make up for the disparity seen below.

sr2sieve completed the range 14e11 to 16e11 (same size as above), with 4 threads, cached Legendre tables, and default settings otherwise, in 3201.256 seconds. As you can see, it's more than a 2:1 margin.[/QUOTE]

This isn't exactly a fair comparison since you are running multiple CPU threads and have only one GPU. Nevertheless this does surprise me a little.

Happy5214 2023-01-01 22:13

[QUOTE=rogue;621411]This isn't exactly a fair comparison since you are running multiple CPU threads and have only one GPU. Nevertheless this does surprise me a little.[/QUOTE]

The best result I could muster for srsieve2cl with that dataset over 1e9 was 29.18 seconds with the following parameters: [c]-q 144 -g 32 -G 2 -b 0.3[/c] ([C]-q 144[/C] became slightly faster than the default 72 once I retested with [C]-g[/C] and [C]-b[/C]). [c]-G 4[/c] produced the same time, while values of 1, 3, and 8 were all slower.

Do you mean one GPU chip or one GPU worker? Multiple GPU workers didn't seem to help that much beyond the second one.

rogue 2023-01-02 01:11

[QUOTE=Happy5214;621414]The best result I could muster for srsieve2cl with that dataset over 1e9 was 29.18 seconds with the following parameters: [c]-q 144 -g 32 -G 2 -b 0.3[/c] ([C]-q 144[/C] became slightly faster than the default 72 once I retested with [C]-g[/C] and [C]-b[/C]). [c]-G 4[/c] produced the same time, while values of 1, 3, and 8 were all slower.

Do you mean one GPU chip or one GPU worker? Multiple GPU workers didn't seem to help that much beyond the second one.[/QUOTE]

Based upon my testing, there doesn't appear to be much value in overriding -G.

storm5510 2023-01-02 16:56

[QUOTE=rogue;621430]Based upon my testing, there doesn't appear to be much value in overriding -G.[/QUOTE]

Testing on my hardware, RTX 2080, the higher the value of [I]-g[/I], the slower the test. [I]-g8[/I] turned out to be the best, [I]-G[/I] seemed to have no effect. I fail to understand the value of using [I]-q[/I].

rogue 2023-01-02 17:14

[QUOTE=storm5510;621466]Testing on my hardware, RTX 2080, the higher the value of [I]-g[/I], the slower the test. [I]-g8[/I] turned out to be the best, [I]-G[/I] seemed to have no effect. I fail to understand the value of using [I]-q[/I].[/QUOTE]

When it starts, it will compute the different possible values for q. Each q will yield a different number of subsequences, baby steps, and giant steps needed for the discrete log. A simple formula is used to "estimate" the amount of work for each q. srsieve2 chooses the q with the lowest estimate. Since it is just an estimate, it can be wrong. -Q will tell you the estimated work for each possible q. -Q also shows you the number of subsquences, baby steps and giant steps for each possible q. But you have to run a test range for each of those possible q values (using -q) to determine which q is going to do the work in the shortest period of time.

storm5510 2023-01-02 19:10

[QUOTE=rogue;621470]When it starts, it will compute the different possible values for q. Each q will yield a different number of subsequences, baby steps, and giant steps needed for the discrete log. A simple formula is used to "estimate" the amount of work for each q. srsieve2 chooses the q with the lowest estimate. Since it is just an estimate, it can be wrong. -Q will tell you the estimated work for each possible q. -Q also shows you the number of subsquences, baby steps and giant steps for each possible q. But you have to run a test range for each of those possible q values (using -q) to determine which q is going to do the work in the shortest period of time.[/QUOTE]

Many thanks! I need to do more tests.

[U]Edit[/U]: I tried using various values for [I]-q[/I]. Each seemed to slow my process by roughly one-third. Perhaps this is something I had better leave alone.

storm5510 2023-01-03 18:28

1 Attachment(s)
It does linefeed on occasion. What triggers it, I don't know.

rogue 2023-01-03 18:37

[QUOTE=storm5510;621623]It does linefeed on occasion. What triggers it, I don't know.[/QUOTE]

Neither do I.

rogue 2023-01-03 19:44

I have posted mtsieve 2.4.0 over at sourceforge. Here are the changes:

[code]
xyyxsieve/xyysievecl: version 1.8.1
Changed -D to -V for disabling AVX as -D is already used.

srsieve2/srsieve2cl: version 1.6.8
Fix issue where a single abs(c)=1 sequence finds an invalid factor. This
is caused by code that an unnecessary cast of a 32-bit value to a 16-bit value.

ccsieve: version 1.0
Initial release. This is sieve that can be used to assist in Cunningham Chain
searches. There are some differences from newpgen including:
ccsieve is slower then newpgen for primorials (because it validates factors).
ccsieve supports larger files.
ccsieve supports factorial type Cunningham Chains.
ccsieve can generate the CC file format. This is a new format supported
by pfgw which is used specifically for Cunningham Chains. This format
is far easier to read than the newpgen format.

These are the parameters specific to ccsieve:
-c 1 = first kind (first term end with -1), 2 = second kind (first term end with +1)
-t 1 = b^n, 2 = primorial, 3 = factorial
-k Minimum k to search
-K Maximum k to search
-l Cunningham Chain length
-b Base for b^n type searches
-n n of b^n, n# for primorial, n! for factorial
-f Format of output file (C=CC (default), N=NEWPGEN)

The maximum chain length supported by ccsieve is 30, although the longest chain
you are likely to find with help from ccsieve is 10 as the first k in the chain
is limited to 62 bits and records for longer chains have much larger k.

When using the CC format, pfgw can support longer Cunningham Chains up with up to 30
terms. pfgw will not test past the fifth term when using the newpgen format.

The newpgen output format support has not been tested.

Cunningham Chain records can be found at:
https://primes.utm.edu/top20/page.php?id=19
https://primes.utm.edu/top20/page.php?id=20
https://www.pzktupel.de/JensKruseAndersen/cc.htm
[/code]

I'm particularly excited because of ccsieve. Some of the Cunningham Chain records are old. I think there is a good chance to set some new records with help from ccsieve and the latest pfgw. There are no Cunningham Chains with factorials and as far as I know, nobody has tested them. I was able to find some of length 3 with testing, but nothing record worthy. I can imagine Serge (Batalov) wanting to add to his existing records. :smile:

NorbSchneider 2023-01-06 10:48

rogue: In the older version of xyyxsievecl were an automatic save of the remaining terms in 5 minutes. In this version
can I also set a time with automatic save? In the file xyyxsieve.log were saved the terms and their factors in the older version. Can I save in this version also the terms and their factors?

henryzz 2023-01-06 13:19

Cunningham chains look interesting. Sieving them efficiently looks like it has its challenges, though. Sieves can easily end up very sparse(0.01% or less of candidates remaining is quite realistic for chains of length 8), and candidates can overlap if the sieve range is more than a factor of 2.

Looking at [url]http://primerecords.dk/Cunningham_Chain_records.htm#largest[/url] many of the record ks for Balyakin's records have high powers of 2 as factors. I wonder if sieving only odd ks but sieving very large CCs only removing if there is a remaining subset CC of the desired length(e.g. sieving a length 20 CC(maybe max length of 64 as a bitarray?) but only removing if there is no subCC of length 5 remaining) would be more efficient. Those records suggest that something like this was done. Theoretically, this could be coded efficiently when primes are larger than the sieve range as it would just be necessary to determine which term of the CC the modulus applies to(if the relevant k would not be a fraction). It would be necessary to add multiples of p rather than just p for later terms of the CC as each one is 2x further apart.

Another possibility is similar to polysieve(this program sieved for prime tuples). See [URL="https://mersenneforum.org/showthread.php?t=16705&highlight=polysieve"]thread[/URL] and original [URL="https://sites.google.com/site/robertgerbicz/polysieve"]code[/URL].
This used a wheel sieve(for primorials sieve k*p#/q#+-1 with wheel sieve upto q) and a hash-based sieve.
Polysieve used a wheel sieve to generate candidates(potentially with multiple consecutive sieve regions per element of the wheel sieve) and saved a very sparse list of remaining candidates.
From this list, it created a hash table based on the last 30ish(configurable) bits of k("a" in polysieve) if there was a hit when sieving the large primes, then a binary search was used to check whether the hit was actually in the list. This allows efficient sieving of a very large sparse range that wouldn't otherwise fit in memory(currently, memory limits make it necessary to run ccsieve many times in order to sieve a very large range of k).
Hopefully using a wheel sieve will help generating many candidates with no small factors. Currently a lot of time is spent on removing candidates with small factors.

Apologies for the rambling post, please ask questions if I have been unclear.

rogue 2023-01-06 13:47

[QUOTE=NorbSchneider;621828]rogue: In the older version of xyyxsievecl were an automatic save of the remaining terms in 5 minutes. In this version
can I also set a time with automatic save? In the file xyyxsieve.log were saved the terms and their factors in the older version. Can I save in this version also the terms and their factors?[/QUOTE]

The remaining terms are written once per hour. This cannot be affected by any command line options. If you are using -O, factors are written when they are found although the I/O buffer is not flushed for each factor. The factor file will remain open until the process stops.

rogue 2023-01-06 14:04

[QUOTE=henryzz;621833]Cunningham chains look interesting. Sieving them efficiently looks like it has its challenges, though. Sieves can easily end up very sparse(0.01% or less of candidates remaining is quite realistic for chains of length 8), and candidates can overlap if the sieve range is more than a factor of 2.

Looking at [url]http://primerecords.dk/Cunningham_Chain_records.htm#largest[/url] many of the record ks for Balyakin's records have high powers of 2 as factors. I wonder if sieving only odd ks but sieving very large CCs only removing if there is a remaining subset CC of the desired length(e.g. sieving a length 20 CC(maybe max length of 64 as a bitarray?) but only removing if there is no subCC of length 5 remaining) would be more efficient. Those records suggest that something like this was done. Theoretically, this could be coded efficiently when primes are larger than the sieve range as it would just be necessary to determine which term of the CC the modulus applies to(if the relevant k would not be a fraction). It would be necessary to add multiples of p rather than just p for later terms of the CC as each one is 2x further apart.

Another possibility is similar to polysieve(this program sieved for prime tuples). See [URL="https://mersenneforum.org/showthread.php?t=16705&highlight=polysieve"]thread[/URL] and original [URL="https://sites.google.com/site/robertgerbicz/polysieve"]code[/URL].
This used a wheel sieve(for primorials sieve k*p#/q#+-1 with wheel sieve upto q) and a hash-based sieve.
Polysieve used a wheel sieve to generate candidates(potentially with multiple consecutive sieve regions per element of the wheel sieve) and saved a very sparse list of remaining candidates.
From this list, it created a hash table based on the last 30ish(configurable) bits of k("a" in polysieve) if there was a hit when sieving the large primes, then a binary search was used to check whether the hit was actually in the list. This allows efficient sieving of a very large sparse range that wouldn't otherwise fit in memory(currently, memory limits make it necessary to run ccsieve many times in order to sieve a very large range of k).
Hopefully using a wheel sieve will help generating many candidates with no small factors. Currently a lot of time is spent on removing candidates with small factors.

Apologies for the rambling post, please ask questions if I have been unclear.[/QUOTE]

Right now it does not exclude even k for base 2 or odd k for odd bases. I am in the process of making that change. This does not affect primorial or factorial sieving.

I agree that memory is definitely a problem especially for longer chains as so few terms will remain. I wonder if it makes sense to segment the range then switch from a bitmap to a hashmap once the number of terms falls below a certain count.

I have not looked at polysieve and wasn't aware of it. At some point I will might take a look at it and see if there is anything that ccsieve can borrow.

henryzz 2023-01-06 14:42

[QUOTE=rogue;621839]Right now it does not exclude even k for base 2 or odd k for odd bases. I am in the process of making that change. This does not affect primorial or factorial sieving.

I agree that memory is definitely a problem especially for longer chains as so few terms will remain. I wonder if it makes sense to segment the range then switch from a bitmap to a hashmap once the number of terms falls below a certain count.

I have not looked at polysieve and wasn't aware of it. At some point I will might take a look at it and see if there is anything that ccsieve can borrow.[/QUOTE]

I suspect that segmenting for fitting in cache for small primes and fitting in memory for medium primes would probably be sensible. Changing to a hashmap would probably benefit larger primes. Unfortunately, this increases complexity quite a bit, but it should speed up larger searches significantly.
Would a bucket sieve be worth considering for primes>sieve length? I am not sure how this would compare with the hashmap sieve. Maybe it would suit medium primes better(i.e. segment to fit in cache and then bucket sieve between there and the hashmap sieve). The hashmap sieve suits best when the sieve is very sparse.

Polysieve is an interesting sieve that might be nice to port to mtsieve one day. I might do this at some point when I have time.

Am I correct in thinking resuming sieves doesn't work currently for ccsieve?

kruoli 2023-01-06 15:46

[QUOTE=rogue;621625]Neither do I.[/QUOTE]

This just happened to me, too. It happened when p got a digit longer.

rogue 2023-01-06 16:13

[QUOTE=henryzz;621843]I suspect that segmenting for fitting in cache for small primes and fitting in memory for medium primes would probably be sensible. Changing to a hashmap would probably benefit larger primes. Unfortunately, this increases complexity quite a bit, but it should speed up larger searches significantly.
Would a bucket sieve be worth considering for primes>sieve length? I am not sure how this would compare with the hashmap sieve. Maybe it would suit medium primes better(i.e. segment to fit in cache and then bucket sieve between there and the hashmap sieve). The hashmap sieve suits best when the sieve is very sparse.

Polysieve is an interesting sieve that might be nice to port to mtsieve one day. I might do this at some point when I have time.

Am I correct in thinking resuming sieves doesn't work currently for ccsieve?[/QUOTE]

If you choose to implement the polysieve logic into mtsieve, that would be cool. If you are familiar with the polysieve code you could probably do it much faster than I could. That code is pretty dense.

Resuming might work with the CC format, but I haven't tested it. I haven't tested newpgen format at all. I intend to do both before the next release. I also want to add a feature that will allow one to split the remaining k across multiple files.

kruoli 2023-01-06 17:21

Trying to resume a CC file I created simply prints the program's header, then takes a while and then exits without crash without further notice. Additionally, I had the problem that [C]ccsieve[/C] stated [C]Fatal Error: Something is wrong. Counted terms (289524078) != expected terms (283967206)[/C] once when trying to write the first CC file after an hour.

Happy5214 2023-01-06 17:56

1 Attachment(s)
I've just decided to post my entire testing log, with all the parameters I tried, along with an sr2sieve run with the same data (which I posted in an earlier post). I deleted boilerplate lines for brevity.

Edit: I'm pretty sure I passed the thread argument to sr2sieve incorrectly, as it's way slower than the four-threaded run I mentioned in a prior post with similar data. It likely ran with just 1 thread.

rogue 2023-01-06 18:41

[QUOTE=Happy5214;621860]I've just decided to post my entire testing log, with all the parameters I tried, along with an sr2sieve run with the same data (which I posted in an earlier post). I deleted boilerplate lines for brevity.

Edit: I'm pretty sure I passed the thread argument to sr2sieve incorrectly, as it's way slower than the four-threaded run I mentioned in a prior post with similar data. It likely ran with just 1 thread.[/QUOTE]

What was the min p that sieving started at?

Do you have command line output for the four-threaded run you mentioned?

You can use "top" on a command line to see how many cores sr2sieve is using while it is running.

For sr2sieve, I don't recall if the last line is the wall time from start to finish or the CPU time from start to finish. Do you know? I haven't run it multi-threaded on my Mac, so I haven't looked.

rogue 2023-01-06 18:42

[QUOTE=kruoli;621855]Trying to resume a CC file I created simply prints the program's header, then takes a while and then exits without crash without further notice. Additionally, I had the problem that [C]ccsieve[/C] stated [C]Fatal Error: Something is wrong. Counted terms (289524078) != expected terms (283967206)[/C] once when trying to write the first CC file after an hour.[/QUOTE]

Hmm. I haven't run into this. What was the command line? If I can reproduce it should be easy to fix.

kruoli 2023-01-06 19:06

[QUOTE=rogue;621864]Hmm. I haven't run into this. What was the command line? If I can reproduce it should be easy to fix.[/QUOTE]

Thanks. :smile: [C]ccsieve -W 8 -o remaining8kp7.cc -O factors8kp7.log -n 1994 -c 1 -t 3 -k 1 -K 1000000000 -l 4 -P 1e9[/C] When I removed the [C]-O[/C] part, it worked fine. It is noteworthy that in this case the factor log file only had entries of this format: [C]{some number} | {some gibberish, but always the same}[/C].

Happy5214 2023-01-06 19:11

[QUOTE=rogue;621863]What was the min p that sieving started at?[/QUOTE]

They all started at the then-current progress of the [C]abcd[/C] file, which was 1e12 = 1000e9, so they all ran for 1e9.

[QUOTE=rogue;621863]Do you have command line output for the four-threaded run you mentioned?[/QUOTE]

It wouldn't be helpful, because it ran within my normal workflow (i.e. it was for 2e11 instead of 1e9, started at a higher [I]p[/I] of 14e11, and excluded candidates eliminated from the 1000e9-1400e9 range). It would have included a bunch of found factors, and in any event it's beyond my backscroll range. Here's the log output though:

[code]Sat Dec 24 14:09:21 2022 WARNING: --pmin=1400000000000 from command line overrides pmin=1271633311499 from `b2_n.abcd'
Sat Dec 24 14:09:21 2022 sr2sieve 1.9.3 started: 500189 <= n <= 999973, 1400000000000 <= p <= 1600000000000
Sat Dec 24 15:02:42 2022 sr2sieve 1.9.3 stopped: at p=1600000000000 because range is complete.
Sat Dec 24 15:02:42 2022 Found factors for 54 terms in 3201.256 sec. (expected about 59.99)
[/code]

(The [I]p_min[/I] is where srsieve2cl was stopped.)

The command is in a shell script, and it reads like:

[code]./sr2sieve -i 'b2_n.abcd' -C 'sr2cache.bin' -t "${threads}" ${p_min} -P "${pmax}e9" && echo "${pmax}" > ./current/progress && ./updatefactors && ./prp[/code]

[QUOTE=rogue;621863]You can use "top" on a command line to see how many cores sr2sieve is using while it is running.

For sr2sieve, I don't recall if the last line is the wall time from start to finish or the CPU time from start to finish. Do you know? I haven't run it multi-threaded on my Mac, so I haven't looked.[/QUOTE]

The [I]p[/I]/sec on line 167 (approx. 16.07M/sec) is around one-fourth of the [I]p[/I]/sec I computed from my logs for the four-threaded 1400e9 to 1600e9 run (approx. 62.48M/sec) with similar data, so that seems to confirm that the sr2sieve run in the posted log was single-threaded. An assumption that the time is the wall-clock time matches with the result of dividing the [I]p[/I] range by the [I]p[/I]/sec value (which is by range rather than primes checked) output by the program in intermediate reports, and it scales appropriately but non-linearly based on thread count (i.e. shorter times for higher thread counts).

rogue 2023-01-06 19:39

I have no explanation why -t=8 would run in a single thread. Maybe the cores had contention with other processes. You likely notice that changing -G has no impact on throughput. Since you have so few subsequences -b has almost no impact.

As for the miscount I'm fairly certain I know why that is happening. It it writing the remaining terms file at the same time it is removing terms. In essence a term is written to the file then removed before that file is closed, which is where the check triggering the error you saw is being made. I'll change the behavior to prevent that.

kruoli 2023-01-06 20:07

For me, multiple threads are only used when the prime to be sieved has reached a certain threshold.

There was another command line without [C]-O[/C] that had a similar problem: [C]ccsieve -W 16 -o remaining2701pow1994.cc -b 2701 -n 1994 -c 1 -t 1 -k 1 -K 1e11 -l 4[/C] lead to [C]Fatal Error: Something is wrong. Counted terms (3698237) != expected terms (3698192)[/C].

rogue 2023-01-06 21:34

[QUOTE=kruoli;621872]For me, multiple threads are only used when the prime to be sieved has reached a certain threshold.

There was another command line without [C]-O[/C] that had a similar problem: [C]ccsieve -W 16 -o remaining2701pow1994.cc -b 2701 -n 1994 -c 1 -t 1 -k 1 -K 1e11 -l 4[/C] lead to [C]Fatal Error: Something is wrong. Counted terms (3698237) != expected terms (3698192)[/C].[/QUOTE]

Right. That limit set to 1e8. In other words it sieves to 1e8 with a single thread before it can use multiple threads. I have reduced that to 1e6, but I have also changed the logic to avoid writing and removing concurrently.

storm5510 2023-01-07 00:40

Something I have been meaning to ask, if I can word it properly:

Example: You have a series or a .abcd file with 250,000 terms. As [I]srsieve2(cl)[/I] goes along, some terms are determined to be factors. Does the program dump these from what it is holding in RAM or does it flag them not to be considered again but still keeping them in RAM? It seems the latter would create a bit of a bottleneck as the number of remaining terms decreases.

rogue 2023-01-07 02:43

[QUOTE=storm5510;621885]Something I have been meaning to ask, if I can word it properly:

Example: You have a series or a .abcd file with 250,000 terms. As [I]srsieve2(cl)[/I] goes along, some terms are determined to be factors. Does the program dump these from what it is holding in RAM or does it flag them not to be considered again but still keeping them in RAM? It seems the latter would create a bit of a bottleneck as the number of remaining terms decreases.[/QUOTE]

There is a bitmap of n for each k. The bit is turned on if the n for the k has no factor, so once a factor is found the bit is turned off.

In short there is no impact to memory when factors are found.

storm5510 2023-01-07 15:15

[QUOTE=rogue;621892]There is a bitmap of n for each k. The bit is turned on if the n for the k has no factor, so once a factor is found the bit is turned off.

In short there is no impact to memory when factors are found.[/QUOTE]

A bitmap, as in a photo being two colors, black or white. Each bit (pixel) acts like a pointer to its corresponding [I]n[/I].

In the old days, this would have been done with a random-access file with three parts for each [I]n[/I]. The first part would be an index, the second being a binary one or zero, and the third would be the [I]n[/I]. By today's standards, super slow and super inefficient.

Citrix 2023-01-07 16:05

[QUOTE=Citrix;620831]The sr1sieve code seems correct though the first does produce a wrong subseq and the second one does not. I ran both with -Q4 option.

[CODE]
ABCD 21*2^$a+1 [1000] // Sieved to 1000000000 with srsieve
4004
2002
5005
3003
2002
5005
3003
2002
5005
3003
2002
5005
3003
[/CODE]

[CODE]
ABCD 21*2^$a+1 [1000] // Sieved to 1000000000 with srsieve
4
2
5
3
2
5
3
2
5
3
2
5
3
2
5
3
[/CODE][/QUOTE]

@rogue

Were you able to figure out why sr1sieve and mtsieve give different number of sub-sequences for the 2 input files above. The number of sub-sequences should be the same. Is it a bug? Are factors being missed?

rogue 2023-01-07 16:54

[QUOTE=Citrix;621924]@rogue

Were you able to figure out why sr1sieve and mtsieve give different number of sub-sequences for the 2 input files above. The number of sub-sequences should be the same. Is it a bug? Are factors being missed?[/QUOTE]

I will take a look when I have a chance.

rogue 2023-01-09 15:01

[QUOTE=Citrix;621924]Were you able to figure out why sr1sieve and mtsieve give different number of sub-sequences for the 2 input files above. The number of sub-sequences should be the same. Is it a bug? Are factors being missed?[/QUOTE]

I compared the possible q and number of subsequences for each q between sr1sieve and srsieve2. They are the same. -Q4 is not possible with sr1sieve because Q must be a multiple of 30.

Tyler Busby 2023-01-11 01:56

I'm attempting to sieve a quasi-repdigit sequence with srsieve2.exe, as LLR is starting to take about 17 seconds on my machine per (lightly trial divisioned) candidate at about (650*10^84853+43)/9. Not sure if that's an expected metric, but I figured trying to sieve the candidates down further couldn't hurt.

However I'm having a problem understanding if I'm doing something incorrectly or if there is just a bug, when I try to sieve via
[CODE]srsieve2.exe -P 2e6 -n 82977 -N 100000 -s "(650*10^n+43)/9"
srsieve2 v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Must use generic sieving logic because abs(c) != 1 for at least one sequence
Sieving with generic logic for p >= 3
Sieve started: 3 < p < 2e6 with 17024 terms (82977 < n < 100000, k*10^n+43) (expecting 15735 factors)
Fatal Error: Invalid factor: (650*10^82977+43)/9 mod 3 = 42[/CODE]

I also tried -p 5 to skip 3, I get [CODE]Fatal Error: pmin cannot be overridden when starting a new sieve[/CODE]

and sieving just the numerator or starting at a smaller -n doesn't seem to make a difference here either. So I'm a bit stuck for the moment! I thought about writing my own sieve but circumventing this error seems like a better use of time.

rogue 2023-01-11 02:53

[QUOTE=Tyler Busby;622211]I'm attempting to sieve a quasi-repdigit sequence with srsieve2.exe, as LLR is starting to take about 17 seconds on my machine per (lightly trial divisioned) candidate at about (650*10^84853+43)/9. Not sure if that's an expected metric, but I figured trying to sieve the candidates down further couldn't hurt.

However I'm having a problem understanding if I'm doing something incorrectly or if there is just a bug, when I try to sieve via
[CODE]srsieve2.exe -P 2e6 -n 82977 -N 100000 -s "(650*10^n+43)/9"
srsieve2 v1.6.8, a program to find factors of k*b^n+c numbers for fixed b and variable k and n
Must use generic sieving logic because abs(c) != 1 for at least one sequence
Sieving with generic logic for p >= 3
Sieve started: 3 < p < 2e6 with 17024 terms (82977 < n < 100000, k*10^n+43) (expecting 15735 factors)
Fatal Error: Invalid factor: (650*10^82977+43)/9 mod 3 = 42[/CODE]

I also tried -p 5 to skip 3, I get [CODE]Fatal Error: pmin cannot be overridden when starting a new sieve[/CODE]

and sieving just the numerator or starting at a smaller -n doesn't seem to make a difference here either. So I'm a bit stuck for the moment! I thought about writing my own sieve but circumventing this error seems like a better use of time.[/QUOTE]

(k*b^n+/-1)/d is not fully supported. I keep meaning to fix this, but have been getting sidetracked.

Batalov 2023-01-11 03:05

[QUOTE=Tyler Busby;622211]... I thought about writing my own sieve but circumventing this error seems like a better use of time.[/QUOTE]
1. Writing your own sieve is a good exercise. But you are right - if you [B]can [/B]circumvent it, that's the reasonable way to go. And ...
2. You [B]can[/B]! You drop the "/d" part from formula (you will need to take care of factor of 3 later); then you run Reynolds' 1-st generation srsieve on the "650*10^n+43" with -p 5 -P 1e7, for example, then from the resulting file you can remove all survivor n :: n%3==2, then you sieve some more (e.g. -p 1e7 -P 1e12); then you llr with
ABC ($a*$b^$c$d)/$e header (or you can run P95 just as well, you just will have to format input files in particular ways).

This workflow is a bit manual (and no, not always n%3==2, beware; you will have to figure it out) - but I've run that for several dozen series in Kamada's site without any problems over 15+ years, recently for something like [URL="https://stdkmd.net/nrr/4/44449.htm#prime"]this series[/URL]

Tyler Busby 2023-01-11 05:29

[QUOTE=Batalov;622220]1. Writing your own sieve is a good exercise. But you are right - if you [B]can [/B]circumvent it, that's the reasonable way to go. And ...
2. You [B]can[/B]! You drop the "/d" part from formula (you will need to take care of factor of 3 later); then you run Reynolds' 1-st generation srsieve on the "650*10^n+43" with -p 5 -P 1e7, for example, then from the resulting file you can remove all survivor n :: n%3==2, then you sieve some more (e.g. -p 1e7 -P 1e12); then you llr with
ABC ($a*$b^$c$d)/$e header (or you can run P95 just as well, you just will have to format input files in particular ways).

This workflow is a bit manual (and no, not always n%3==2, beware; you will have to figure it out) - but I've run that for several dozen series in Kamada's site without any problems over 15+ years, recently for something like [URL="https://stdkmd.net/nrr/4/44449.htm#prime"]this series[/URL][/QUOTE]

Thanks!! Your old post about sieving quasi-repdigits got me most of the way there, but I hadn't realized I needed an older srsieve for the time being (which was a bit hard to find until I knew exactly what I was looking for, managed to snag [URL="http://primegrid.com/download/sr2sieve/"]several older versions of srsieve[/URL] from primegrid). Already sieved up to 1e10 and counting, and I have a good workflow setup for other quasi-repdigits I feel like tackling.

pepi37 2023-01-11 22:10

MTsieve 2.4.0.7 ( latest)

e:\MTSIEVE\2022>twinsieve -P 1000000000 -W8 -w2500000 -k 2 -K 1500000 -b 10 -n 888888 -r -s
twinsieve v1.3, a program to find factors of k*b^n+1/-1 numbers for fixed b and n and variable k
Switching to ABC format since other formats are not supported when using -s
Sieve started: 1 < p < 1e9 with 2699998 terms (2 < k < 1500000, k*10^888888) (expecting 2609689 factors)
Sieve completed at p=0.
CPU time: 0.02 sec. (0.02 sieving) (0.00 cores)
2699998 terms written to k_b10_n888888.pfgw
Primes tested: 0. Factors found: 0. Remaining terms: 2699998. Time: 22.51 seconds.

rogue 2023-01-12 13:15

[QUOTE=pepi37;622290]MTsieve 2.4.0.7 ( latest)

e:\MTSIEVE\2022>twinsieve -P 1000000000 -W8 -w2500000 -k 2 -K 1500000 -b 10 -n 888888 -r -s
twinsieve v1.3, a program to find factors of k*b^n+1/-1 numbers for fixed b and n and variable k
Switching to ABC format since other formats are not supported when using -s
Sieve started: 1 < p < 1e9 with 2699998 terms (2 < k < 1500000, k*10^888888) (expecting 2609689 factors)
Sieve completed at p=0.
CPU time: 0.02 sec. (0.02 sieving) (0.00 cores)
2699998 terms written to k_b10_n888888.pfgw
Primes tested: 0. Factors found: 0. Remaining terms: 2699998. Time: 22.51 seconds.[/QUOTE]

I will look into it

hunson 2023-01-12 19:20

Today I tested several exponents with the new ccsieve. One of about 5 tests crashed with the following error. I am not sure if it matters, but the parameters of the successful tests differed in -k 10000000000 and -p 1e12 and -n XXXXX


[quote]
.\ccsieve.exe -b 2 -k 3 -K 100000000000 -f CC -n 49911 -W 6 -w 1e6 -P 1e13 -c 1 -t 1 -l 3
ccsieve v1.0, a program to eliminate terms for Cunningham Chain prime searches
Sieve started: 3 < p < 1e13 with 99999999998 terms (3 < k < 100000000000, first term of k*2^49911-1 for CC length 3) (expecting 96329836500 factors)
p=1051481, 25.44 p/sec, 99962056669 factors found at 8.314K f/sec (last 1 min), 0.0% done. ETC 2997-05-16 02:48
p=1116989, 26.40 p/sec, 99962551538 factors found at 8.316K f/sec (last 1 min), 0.0% done. ETC 2957-07-15 02:20
p=1188557, 27.46 p/sec, 99963050653 factors found at 8.264K f/sec (last 1 min), 0.0% done. ETC 2917-07-12 20:47
p=1266953, 28.60 p/sec, 99963551154 factors found at 8.288K f/sec (last 1 min), 0.0% done. ETC 2877-06-08 11:50
p=1351171, 29.81 p/sec, 99964043849 factors found at 8.279K f/sec (last 1 min), 0.0% done. ETC 2838-06-27 19:35
p=1443073, 31.13 p/sec, 99964543250 factors found at 8.261K f/sec (last 1 min), 0.0% done. ETC 2799-12-14 08:30
Fatal Error: Something is wrong. Counted terms (34576479) != expected terms (34189356)
[/quote]

regards

rogue 2023-01-12 20:39

[QUOTE=hunson;622367]Today I tested several exponents with the new ccsieve. One of about 5 tests crashed with the following error. I am not sure if it matters, but the parameters of the successful tests differed in -k 10000000000 and -p 1e12 and -n XXXXX[/QUOTE]

This will be fixed with the next release of ccsieve.

rogue 2023-01-13 19:48

I have posted mtsieve 2.4.1. Here are the changes:

[code]
twinsieve: version 1.4
Fix issue where no terms are removed.

gfndsieve/gfndsievecl: version 2.3
For each prime, only verify the first k is factored for that prime.
Avoid concurrency issues with term removal at same time output file is being written.

ccsieve: version 1.1
Added -N to allow splitting of k across multiple output files.
Fixed resuming from previous sieve (CC format).
For type 1 searches, only track odd k for base 2 and even k for odd bases.
Avoid concurrency issues with term removal at same time output file is being written.
[/code]

npg format is still not tested with ccsieve. It might work. It might fail. It just hasn't been tested.

kruoli 2023-01-14 17:58

How much memory is [C]ccsieve[/C] expected to allocate when I have a CC input file with 32M candidates? I have around 50 GB free memory, but it still says "bad alloc".

rogue 2023-01-14 20:01

[QUOTE=kruoli;622538]How much memory is [C]ccsieve[/C] expected to allocate when I have a CC input file with 32M candidates? I have around 50 GB free memory, but it still says "bad alloc".[/QUOTE]

It uses a bit map. What are the min and max k in the file?

kruoli 2023-01-14 20:51

So even if I have 99.9 % sieved out, it still falls back to a bitmap for all k? I guess this answers my question. I went from k=1e11 to K=1e12.

rogue 2023-01-15 03:20

[QUOTE=kruoli;622548]So even if I have 99.9 % sieved out, it still falls back to a bitmap for all k? I guess this answers my question. I went from k=1e11 to K=1e12.[/QUOTE]

I do need to modify to switch to a hashmap or ordered list when the density falls below a certain level, but I haven't don't that yet.

Tyler Busby 2023-01-17 18:24

Any idea why srsieve2cl.exe might not be closing gracefully when run (and consequently killed) as a python subprocess? I'm not sure of anything anymore, after debugging in circles for a long time last night. But my general experience is that I can find no way to get the process to gracefully close and save the output file. In the meantime I am choosing relatively small ranges to sieve so I don't lose too much progress if I need to stop the parent process for any reason.

But to give more concrete examples, my subprocess is generally started like this: (Popen() rather than call or run, as I don't want the process to block)
[CODE]
srsieve = subprocess.Popen([
os.path.join("..", "srsieve2cl.exe"), "-i", srsieve_abc_filename, "-f", "A",
"-o", srsieve_abc_filename, "-O", srsieve_factors_filename, "-P", sieve_level
], creationflags=subprocess.CREATE_NEW_PROCESS_GROUP)[/CODE]

And I have tried (seemingly) all combinations of the following flags and kill options, and it never seems to get the "control c accepted, quitting after <x>" type message you get if you send a control c to a console solely running srsieve2.exe

[CODE]
os.kill(srsieve.pid, signal.CTRL_C_EVENT)
os.kill(srsieve.pid, signal.SIGTERM)
os.kill(srsieve.pid, signal.SIGINT)
creationflags=0
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP
creationflags=subprocess.CREATE_NEW_CONSOLE
srsieve.send_signal(signal.CTRL_C_EVENT)
srsieve.send_signal(signal.SIGTERM)
srsieve.send_signal(signal.SIGINT)
[/CODE]

The [URL="https://docs.python.org/3/library/subprocess.html"]docs[/URL] have a lot of details about the python subprocess implementation specifics, but one strange thing I keep seeing is that sending ctrl c via send_signal interrupts the parent process in some cases, but even in that case it didn't seem like srsieve2cl.exe was quitting gracefully. And for a lot of these combinations the process would end up floating there long after the parent process was killed. Fwiw I didn't have much luck with killing srsieve2.exe either.


All times are UTC. The time now is 03:40.

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.