Combined Sieve

[SethTro]

Originally Posted by SethTro

The code uses gmp for numbers with < 8000 bits and pfgw for larger numbers. Sadly it's currently making one call per prime (e.g. hundreds per interval) I haven't found any easy way to make multiple calls (maybe I can write a list of all the offsets to check like Robert did in https://www.mersenneforum.org/showpo...&postcount=203 but I wasn't sure if pfgw stopped after the first prime.

Looks like I can do this

ABC $a*1009#/3018+$b // {number_primes,$a,1}
1 192
1 324
1 512
1 648
1 768
1 1296
1 2066
1 2126

Does anyone know if I can do something like {number_primes,1} instead of adding the dummy $a parameter? or maybe I'll just use m there

[henryzz]

Originally Posted by SethTro

Please do comment on issues, I'll write this (sqlite3 > 3.18 + suggestion of Ubuntu 20.04) up in README.md#setup later today. Does this mean you got some/all of it running? under WSL?

I have gotten most of it working. Python needs to be at least v3.7 as well which wasn't default on 18.04.

Are all the options available fully implemented? The calculate_expected_gaps call on line 831 of gap_test crashes so I can't set options that satisfy this if statement: if args.stats and (args.save_logs or args.num_plots):
I haven't produced any graphs yet.
Can the script cope with being run twice in the same directory on the same database(maybe different scripts)?

I am not aware of a way to not need a dummy variable for // {number_primes,$a,1}. If file size is an issue you could use the ABCD format but I would imagine the file format you posted is easier.

[SethTro]

Originally Posted by henryzz

I have gotten most of it working. Python needs to be at least v3.7 as well which wasn't default on 18.04.

Awesome that you got this working! Is this under WDL? you mentioned it before and it would be nice to confirm it can work there.

Hum, I'm running this on a Ubuntu 18.10 instance (with sqlite updated) using Python3.7, do you remember what broke python3.7?

Originally Posted by henryzz

Are all the options available fully implemented? The calculate_expected_gaps call on line 831 of gap_test crashes so I can't set options that satisfy this if statement: if args.stats and (args.save_logs or args.num_plots):
I haven't produced any graphs yet.
Can the script cope with being run twice in the same directory on the same database(maybe different scripts)?

for plotting '--stats' is unneeded (if you ran ./gap_stats) but I'll fix the python code. It should work if you pass `--no-one-side-skip --num-plots 3 --top-prp-percent=100`

Originally Posted by henryzz

Can the script cope with being run twice in the same directory on the same database(maybe different scripts)?

I'm not sure I've ever tested this but it should work (there might be DB contention but only if both are trying to submit lots of records per second). You can also use multiple databases by passing `--search-db` to the various tools / scripts.

[henryzz]

Originally Posted by SethTro

Awesome that you got this working! Is this under WDL? you mentioned it before and it would be nice to confirm it can work there.

Hum, I'm running this on a Ubuntu 18.10 instance (with sqlite updated) using Python3.7, do you remember what broke python3.7?

dataclasses doesn't exist before python 3.7,

Yes it is WSL I am using. Generally, I find it pretty much behaves like Linux(it is apart from system calls).

[SethTro]

Originally Posted by henryzz

dataclasses doesn't exist before python 3.7,

Yes it is WSL I am using. Generally, I find it pretty much behaves like Linux(it is apart from system calls).

I was misread your message and misunderstood. I thought you were saying it didn't work with 3.7 but it doesn't work before 3.7.

I've updated the README and I'm wrote a fix for `gap_test.py --stats` but tomorrow I'm going to remove `--stats` because it doesn't serve any use (it's not needed for plotting and only serves to validate / double check gap_stats).

It's late so I may have made some errors but I tested, briefly, before I pushed.

[SethTro]

Originally Posted by robert44444uk

I got quite a way with this Ubuntu thing, working through Microsoft Windows 10 subsystem WSL1. I think I got through almost all of stage 3 of Seth's idiot's guide "Now we get to the real project" - see post #1

...

 ...
robert44444uk@Robert:~/gmp/build/prime-gap/prime-gap$ make all
g++ -c -o gap_common.o gap_common.cpp -Wall -Werror -O3
make: g++: Command not found
make: *** [Makefile:39: gap_common.o] Error 127
robert44444uk@Robert:~/gmp/build/prime-gap/prime-gap$ $ head unknowns/907_53970_1_10000_s15000_l1000M.txt | cut -c1-100
$: command not found

Looks like g++ is not installed, I'd be slightly surprised if gmp installed (the first code block) you might need to redo that block if you get an error like "mpz_prevprime not found"

sudo apt install build-essential

should install g++ and you can continue from `make all`.

[robert44444uk]

Thanks Seth - I had actually figured that out in the end. There were several more issues on the way after that but it was a question that I was running everything from scratch and I needed to run some commands to install the missing bits.

The one that impact Ubuntu 20 is that pip does not work. Instead you need pip3.

That worked, and now I am away and exploring :)

UPDATE: All working like a dream. Now I can try to repeat everything on my main machine (I installed this on an old machine just in case).

As you are continuing to adapt the programme, and as gaps change, what are the scripts to run that can get this from you and the gap site?

[robert44444uk]

Trying to understand the input variables. In post #1 you write

sieve to sieve m * P#/d for m from a to a+b up to limit

./combined_sieve --save-unknowns -p <P> -d <d> --mstart <a> --minc <b> --sieve-length <X> --max-prime <limit>

# e.g. 907#/53970 sieving 10,000 in both directions up to 1B

./combined_sieve --save-unknowns -p 907 -d 53970 --mstart 1 --minc 10000 --sieve-length 15000 --max-prime 1000

I do not understand sieve-length - it seems to me that the program will sieve multiple m from 1 to 10000, that much understood.

You then write "sieving 10,000 in both directions" but sieve-length is 15000. Does this actually mean the program will sieve either side of the primorial centre point by 15000 or 10000 or 7500 or 5000?

max-prime - does this mean that the sieve will sieve up to a maximum prime? I take it that 1000 = 1000M, i.e. it will sieve up to the primes that is just lower than 1e9

[SethTro]

Originally Posted by robert44444uk

As you are continuing to adapt the programme, and as gaps change, what are the scripts to run that can get this from you and the gap site?

Both are git repositories which can be updated with `git pull`.
If you want to update the newest prime gap records do something like this (it would be easy to add to update_gap_repo.sh, then set executable with `chmod a+x update_gap_repo.sh` and run with `./update_gap_repo.sh`

cd prime-gap-list/
# This checks github for the last version and downloads it, this can  fail  if you've made local changes to allgaps.sql but you probably  haven't  (if it fails look up some git basics like "git status" and "git  reset")
git pull 
# Delete and recreate gaps.db
rm gaps.db
sqlite3 gaps.db < allgaps.sql
# Go back to the previous directory
cd -

for the prime-gap code it's the same basic thing

# This checks  github for the last version and downloads it
# This can fail if I've done bad things or if you've made local changes
# if it fails  look up some git basics like "git status" and "git reset origin/main"
git pull 
# delete and recompile binaries
make clean
make all

Originally Posted by robert44444uk

Trying to understand the input variables. In post #1 you write

sieve to sieve m * P#/d for m from a to a+b up to limit

./combined_sieve --save-unknowns -p <P> -d <d> --mstart   <a> --minc <b> --sieve-length <X> --max-prime   <limit>

# e.g. 907#/53970 sieving 10,000 in both directions up to 1B

./combined_sieve --save-unknowns -p 907 -d 53970 --mstart 1 --minc 10000 --sieve-length 15000 --max-prime 1000

I do not understand sieve-length - it seems to me that the program will sieve multiple m from 1 to 10000, that much understood.

You then write "sieving 10,000 in both directions" but sieve-length is 15000. Does this actually mean the program will sieve either side of the primorial centre point by 15000 or 10000 or 7500 or 5000?

15,000 vs 10,000 is a typo on my part, it should have been "15,000 in both directions".

You are correct `sieve-length` is the distance on either side of the center point that is sieved. I wish I had a better name for this than `sieve-length` maybe `half-sieve-interval`. I'm very open to suggestions.

Originally Posted by robert44444uk

max-prime - does this mean that the sieve will sieve up to a maximum prime? I take it that 1000 = 1000M, i.e. it will sieve up to the primes that is just lower than 1e9

Yes, this is the correct interpretation

[robert44444uk]

Thanks Seth

Installed on my big machine - effortless this time.
The last few questions for today question:

$ ./gap_test.py --unknown-filename 907_53970_1_10000_s15000_l1000M.txt --min-merit 15 --prp-top-percent 40 --threads 4
...
13818  15.9853  3937 * 907#/53970 -6686 to +7132
21164  24.4724  5801 * 907#/53970 -11364 to +9800    RECORD!

Where are the output items (in bold) kept?

I want to be able to access this file and play with it in MS Excel, which is in another operating system. How do I transfer the output items to a txt file in an MS Directory?

What command do I need to upload any records to the database? Or should I continue to use the primegaps/cloudy.go upload page?

[SethTro]

Originally Posted by robert44444uk

Thanks Seth

Installed on my big machine - effortless this time.
The last few questions for today question:

$ ./gap_test.py --unknown-filename 907_53970_1_10000_s15000_l1000M.txt --min-merit 15 --prp-top-percent 40 --threads 4
...
13818  15.9853  3937 * 907#/53970 -6686 to +7132
21164  24.4724  5801 * 907#/53970 -11364 to +9800    RECORD!

Where are the output items (in bold) kept?

I want to be able to access this file and play with it in MS Excel, which is in another operating system. How do I transfer the output items to a txt file in an MS Directory?

What command do I need to upload any records to the database? Or should I continue to use the primegaps/cloudy.go upload page?

All results (m,p,d,previous_prime_offset,next_prime_offset) are stored in a sql database (prime-gap-search.db).
the script `./misc/record_check.py` will retrieve all the record gaps. If you want a raw list you can start with something like `sqlite3 prime-gap-search.db -header -csv 'SELECT * FROM result WHERE merit > 20'`

If you pass --save-logs (requires a git pull as I just fixed this code) all screen output gets saved in the log directory under a sensible filename (which also gets logged)

There's no automated upload script, it's a good idea but I don't have the energy right now (and I kinda like the frontend warts and all)