Finessing the Aliquot reservation page to sequences above 1 million - Page 3

yoyo · 2017-11-02, 11:55

I know where the AllSeqs.txt is located and the enhancements for 7 digit reservations I did already. So reservation in the forum and for yafu works already.

So I need to add the first batch of sequences.

Should we reserve all for ugly2dog ?

Dubslow · 2017-11-02, 12:01

Quote:

Originally Posted by yoyo

Should we reserve all for ugly2dog ?

He specified earlier the two ranges he would like reserved; they're available as the pastebin links in the previous post. Just copy and paste those sequences into a manual reservations.py command, and he will be (should be, at least) satisfied. Any of the rest should be fair game for yafu/the rest of us.

Quote:

Originally Posted by yoyo

I know where the AllSeqs.txt is located and the enhancements for 7 digit reservations I did already. So reservation in the forum and for yafu works already.

I don't see at least the html page changes live yet. Do I just need to wait for the next fdb update?

yoyo · 2017-11-02, 12:59

I just changed the .py files and will do the stuff this evening (my time). You might join our Slack Chat to discuss direct: https://rechenkraftnet.signup.team/

Batalov · 2017-11-02, 21:02

So why aren't merges still detected?
Example:

Code:

Sequence	Size	Index	Guide	Known Factors	Cofactor Size	Class	Last Updated	Recent Progress	Reservation
1001832	140	1294	2 * 3	2 * 3 * 65030071 * C131	131	-1	2017-10-31 19:28:41	1295	
667848	140	1295	2 * 3	2 * 3 * 65030071 * C131	131	-1	2017-11-01 22:48:41	7	yafu@home

I'd easily argue that any >1e6-starting sequence that is "miraculously" at the 139-140++ digit size is apparently a clear case of a merge. The only thing left is then to find with what.

There are standard tools for that: in addition to the aforementioned endpoint match, there is always the old approach of checking the "C90" database (all 90-digit elements attributed to all known open sequences, stored once, and periodically refreshed; if a new sequence is hitting any of these values, you know that there is a merge).

yoyo · 2017-11-02, 22:04

Range 1-2e6 is added, but it takes time until it appears on the webpage. The update scripts needs to fetch each sequence from factorDB which needs time and where we hit the limits since those sequences do not exist on factorDB and are created on the fly.
But I have a script running which fetches (and creates) all 1-2e6 sequences.

yoyo · 2017-11-02, 22:05

Quote:

Originally Posted by Batalov

So why aren't merges still detected?
Example:

Code:

Sequence	Size	Index	Guide	Known Factors	Cofactor Size	Class	Last Updated	Recent Progress	Reservation
1001832	140	1294	2 * 3	2 * 3 * 65030071 * C131	131	-1	2017-10-31 19:28:41	1295	
667848	140	1295	2 * 3	2 * 3 * 65030071 * C131	131	-1	2017-11-01 22:48:41	7	yafu@home

I'd easily argue that any >1e6-starting sequence that is "miraculously" at the 139-140++ digit size is apparently a clear case of a merge. The only thing left is then to find with what.

There are standard tools for that: in addition to the aforementioned endpoint match, there is always the old approach of checking the "C90" database (all 90-digit elements attributed to all known open sequences, stored once, and periodically refreshed; if a new sequence is hitting any of these values, you know that there is a merge).

They were already detected. But since all our scripts are busy with updating all sequences up to 2e6, we hadn't time to remove them.
So. please stay tuned.

ChristianB · 2017-11-03, 16:02

Removing merged sequences is not automated and needs to be done manually once a day. I deleted all the merges till now and tried to fix the script that updates the sequences from fdb. Because fdb needs to generate the sequences from u2d's list we spent a lot of CPU time and after about 180 sequences we hit the 750 cpu seconds per hour limit. This is currently not handled gracefully in the script. Dubslow you have a PM about this.

Because u2d is not so sure his list is really complete we also started to check 1e6-2e6 for open sequences using local resources. Once this is done we have a better picture on how to handle the remaining ranges till 10e6.

There are still ~8000 sequences to be added to the reservation page.

Dubslow · 2017-11-03, 23:02

Quote:

Originally Posted by ChristianB

Because u2d is not so sure his list is really complete we also started to check 1e6-2e6 for open sequences using local resources.

You mean without aid from FDB, correct? It sounds loaded enough already. I had been operating on the assumption that u2d's work involved using FDB's aliquot sequence function, but I guess that was a false assumption.

ugly2dog · 2017-11-04, 14:12

I had my own db with all the sequences in it that I used to find the open sequences, but I had a server failure and lost the work. I am in the process of designing a new db, and then I will have to enter in the 15+gigs of data I have locally. Then I will need to download some sequences from the FactorDB that I lost and ones that have been extended, terminated.
This will take some time.

Dubslow · 2017-11-04, 14:41

Quote:

Originally Posted by ugly2dog

I had my own db with all the sequences in it that I used to find the open sequences, but I had a server failure and lost the work. I am in the process of designing a new db, and then I will have to enter in the 15+gigs of data I have locally. Then I will need to download some sequences from the FactorDB that I lost and ones that have been extended, terminated.
This will take some time.

ChristianB has indicated to me that rechenkraft is undertaking a double-check of their own of your (1-2)e6 list of sequences. That too will take time and FDB resources. Might there be some duplication of work? I suppose I'm asking you to consider waiting for the "standard" page to get all the lost data in the range and then adding it into your DB afterwards, rather than querying the FDB for the same data in parallel.

ugly2dog · 2017-11-05, 12:15

Quote:

Originally Posted by Dubslow

ChristianB has indicated to me that rechenkraft is undertaking a double-check of their own of your (1-2)e6 list of sequences. That too will take time and FDB resources. Might there be some duplication of work? I suppose I'm asking you to consider waiting for the "standard" page to get all the lost data in the range and then adding it into your DB afterwards, rather than querying the FDB for the same data in parallel.

I have almost all the data in the 1e6-5e6 range stored locally. I will need to download a bunch of the sequences under 1e6 that I didn't have back-ups of, since these are already indexed, it won't be that much of a load on the FDB. I also spread my work out over several weeks to minimize the load.
There won't be much duplication in work at all, if any.

2017-11-02, 21:02	#26
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 10,061 Posts	So why aren't merges still detected? Example: Code: Sequence Size Index Guide Known Factors Cofactor Size Class Last Updated Recent Progress Reservation 1001832 140 1294 2 * 3 2 * 3 * 65030071 * C131 131 -1 2017-10-31 19:28:41 1295 667848 140 1295 2 * 3 2 * 3 * 65030071 * C131 131 -1 2017-11-01 22:48:41 7 yafu@home I'd easily argue that any >1e6-starting sequence that is "miraculously" at the 139-140++ digit size is apparently a clear case of a merge. The only thing left is then to find with what. There are standard tools for that: in addition to the aforementioned endpoint match, there is always the old approach of checking the "C90" database (all 90-digit elements attributed to all known open sequences, stored once, and periodically refreshed; if a new sequence is hitting any of these values, you know that there is a merge).

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Aliquot Sequences Summary Page	kar_bon	Aliquot Sequences	174	2022-03-19 13:07
Sequences from the 5 to 10 million range	Greebley	Aliquot Sequences	7	2015-09-04 08:22
Broken aliquot sequences	schickel	FactorDB	18	2013-06-12 16:09
A new theorem about aliquot sequences	garambois	Aliquot Sequences	34	2012-06-10 21:53
poaching aliquot sequences...	Andi47	FactorDB	21	2011-12-29 21:11

2017-11-02, 11:55	#23
yoyo Oct 2006 Berlin, Germany 659 Posts	I know where the AllSeqs.txt is located and the enhancements for 7 digit reservations I did already. So reservation in the forum and for yafu works already. So I need to add the first batch of sequences. Should we reserve all for ugly2dog ?

2017-11-02, 12:59	#25
yoyo Oct 2006 Berlin, Germany 293₁₆ Posts	I just changed the .py files and will do the stuff this evening (my time). You might join our Slack Chat to discuss direct: https://rechenkraftnet.signup.team/

2017-11-02, 22:04	#27
yoyo Oct 2006 Berlin, Germany 659 Posts	Range 1-2e6 is added, but it takes time until it appears on the webpage. The update scripts needs to fetch each sequence from factorDB which needs time and where we hit the limits since those sequences do not exist on factorDB and are created on the fly. But I have a script running which fetches (and creates) all 1-2e6 sequences.

2017-11-03, 16:02	#29
ChristianB Apr 2013 Germany 2×3×53 Posts	Removing merged sequences is not automated and needs to be done manually once a day. I deleted all the merges till now and tried to fix the script that updates the sequences from fdb. Because fdb needs to generate the sequences from u2d's list we spent a lot of CPU time and after about 180 sequences we hit the 750 cpu seconds per hour limit. This is currently not handled gracefully in the script. Dubslow you have a PM about this. Because u2d is not so sure his list is really complete we also started to check 1e6-2e6 for open sequences using local resources. Once this is done we have a better picture on how to handle the remaining ranges till 10e6. There are still ~8000 sequences to be added to the reservation page.

2017-11-04, 14:12	#31
ugly2dog May 2009 2·3³ Posts	I had my own db with all the sequences in it that I used to find the open sequences, but I had a server failure and lost the work. I am in the process of designing a new db, and then I will have to enter in the 15+gigs of data I have locally. Then I will need to download some sequences from the FactorDB that I lost and ones that have been extended, terminated. This will take some time.