Opinions/Suggestions for Data Collection thread

[kar_bon]

I think it's time to manage not only the search for primes (whatever form but mostly Sierpinski (k*2^n+1) and Riesel (k*2^n-1)) but also the results from these searches.

About 1 1/2 years ago a begun to collect all avalable data about Riesel-numbers/-primes.
The results you can see at www.rieselprime.org with many new features and data shown there on one website. There're many data missing or to be collect (i update these pages almost every day) and i think many people not involved in any project got their results without any publication.

So please tell me what do you think. Vote for such thread neded or not.
Perhaps there are other opinions or suggestions for creating such data collection.
And perhaps there's someone to collect such data about the Sierpinski side.

Thanks
K.Bonath

PS: Tell me your opinion, if we need such data collecting should it be an own Foum or Subforum and where to locate it.

[mdettweiler]

Yes, I agree that this would be a good idea. However, if we want to ensure that we get data from the widest possible range of sources (including users not associated with any particular project, as you mentioned), I think we need to arrange some sort of partnership with the top-5000 website on this--maybe they could have their system set up so that whenever someone reports a Riesel prime, on the "prime successfully reported" page it also gives a prominent link to our new data collection thread, saying "don't forget to report your progress!" or something like that?

[Flatlander]

Yes, a good idea. For instance, I've started testing 28 different ks but wasn't sure where to report this. Ok, I'll report it here!

I'm testing k*2^n-1 from n=350,000 upwards for the following ks:
1401, 1403, 1407, 1413, 1419, 1431, 1433, 1435, 1437, 1441, 1445, 1449, 1459, 1467, 1473, 1479, 1483, 1487, 1489, 1491, 1493, 1495, 1497
(Only tested to 351,387.)

3087, 3969, 4851, 5733, 6615
(Tested to 365,300.)

[fivemack]

I would go further than saying that results are needed; it's actually worth recording residues. This lets you do independent double-checks with a much higher precision than 'look, they found no prime in 100k..150k and neither did we'. At sixteen characters per core-devoted per half-hour it's not an unreasonable amount to store.

But this is a job for a database with a Web front-end rather than a thread on a forum.

[fivemack]

Originally Posted by mdettweiler

Yes, I agree that this would be a good idea. However, if we want to ensure that we get data from the widest possible range of sources (including users not associated with any particular project, as you mentioned), I think we need to arrange some sort of partnership with the top-5000 website on this--maybe they could have their system set up so that whenever someone reports a Riesel prime, on the "prime successfully reported" page it also gives a prominent link to our new data collection thread, saying "don't forget to report your progress!" or something like that?

I've had a prominent link saying "don't forget to report your factors to xilman" up on the homogeneous-Cunningham-numbers reservation server ever since I wrote it, and the success rate has been 90% but not 100%; in my experience nobody bothers looking at the report-received page.

[mdettweiler]

Originally Posted by fivemack

I would go further than saying that results are needed; it's actually worth recording residues. This lets you do independent double-checks with a much higher precision than 'look, they found no prime in 100k..150k and neither did we'. At sixteen characters per core-devoted per half-hour it's not an unreasonable amount to store.

But this is a job for a database with a Web front-end rather than a thread on a forum.

Indeed--definitely. It would be awesome if we could store residues for all this stuff--especially since NPLB plans to do some heavy doublechecking of various ranges in the future.

[Xyzzy]

We can set up a database if someone wants to take it and run with it. We can also provide web space.

How big would the database be? (Obviously it would grow over time, but say a year or 2 from now, how big could it grow to?)

[tnerual]

gimps use this kind of database ... with track of all factors, residues, single and double check.
rieselsieve.com has an automatic factor entry form

in fact, we need a complete automation of all process

- reserving ranges to factor/to LLR
- entering factors in the database
- entering residues in the database
- removing "completed ranges" from the active database
- checking single and double check (comparing them)
- answering llrnet request (by increasing values of kbn, like rieselsieve has done with his boinc part)
- exporting manual range to test
- ...

each kbn triplet has to go through test (factoring, first test, second test and maybe third test if test 1 and test 2 are different ...)

the database would be huge, the server cost to manage it out of range and the managing of it horrible ... but it would be the best case :D

[fivemack]

Originally Posted by Xyzzy

We can set up a database if someone wants to take it and run with it. We can also provide web space.

How big would the database be? (Obviously it would grow over time, but say a year or 2 from now, how big could it grow to?)

Quite big and relatively fast-growing, with the issues being

* which k are interesting?
* do you bother storing (k,n) pairs with factors <2^32 at all? Even storing a simple list of them would be huge.
* how large does n have to be to be non-trivial?

If you only store (k,n) pairs for which the LLR calculation took half an hour or more, I suspect there are have been less than five thousand core-years, or forty million half-hours, applied to the problems, so the database would fit in a couple of gigabytes and grow no more than a gigabyte a year.

[kar_bon]

wow! thanls to all for your suggestions. i did not expect so much response!!!

Originally Posted by mdettweiler

I think we need to arrange some sort of partnership with the top-5000 website on this

it's a good idea, perhaps a note/link at Top5000 is enough to remind people post their ranges here.
i'm checking every day the new primes reported to Top5000 and will include them in the database if they fit, so for now only k*2^n-1 primes.

Originally Posted by Flatlander

I've started testing 28 different ks

will include your ranges and mark the k's reserved by you!

Originally Posted by fivemack

I would go further than saying that results are needed; it's actually worth recording residues.

that's the idea i had for a long time: make a SQL-database with tables for k-values, n-values, ranges, contributors, results, residues,twins,Sophie-Germain,... and from that make the datapages by php-scripts automatically -> stats,checking easier, testing if a prime is a twin too, and many other features.
but there are some things i can't satisfy: my php-knowledge is minimal, SQL no problem, database creation too, time .... the thing i got not so much i want.
i'm playing with PHP/SQL but without much success. and time....
the next great change for the RieselPrimeDatabase will be such thing. i hope :-)

Originally Posted by Xyzzy

We can set up a database if someone wants to take it and run with it. We can also provide web space.

that's a great offer. i have to write a database definition to see if this works theoretically.
some ideas in mind but not the time to realise.

Originally Posted by tnerual

in fact, we need a complete automation of all process
the database would be huge, the server cost to manage it out of range and the managing of it horrible ... but it would be the best case :D

this is really the best way to collect results for doublechecking.
but the managing of such a big database is very time consuming, although it runs mostly per scripts for sending/receiving results...


so only for one day of this poll i think to speak for all, it's a good idea to make an own forum for such purposes here.

need to be cleared where to put such forum!?

[gd_barnes]

I have an unbelievable amount of results files from here and CRUS.

The reason why I haven't voted yet is I'm more in the 'whatever' category. lol Not that I'm apathetic to the idea. It's just that it's such a huge administrative effort to run 2 projects and I feel like this would add more to that.

What we need to get me on board more with the idea would be to have people step up and run with it. I'll be glad to forward what I have if someone will do it.

All of this said, I think such an effort is badly needed, especially at RPS, where results are rarely collected on individual reservations, ranges are almost never reported to Wilfred Keller's web page, and individual-reservation search ranges are hidden unless you are one of their 'inside' people. (lol) We attempt to and have been about 98-99% successful collecting all the results here. It's just a matter of what to do with them. I keep a backup of them but if I keel over, I think Max and Karsten would only have perhaps half of them. For that reason, I probably need to forward you guys what I have if I can dig it all up.


Gary