Finessing the Aliquot reservation page to sequences above 1 million

[Dubslow]

Originally Posted by ChristianB

The goal of this extension is to get some more "low hanging fruit" for the yafu@home project to keep the volunteers happy with some shorter work.

We could add the 9449 sequences by ugyl2dog to the reservation list, wait 2 to 3 days for the scripts to detect new terminations and merges and let yafu@home work on the remaining.

This would not make those sequences available for reservation through the forums, although this can be changed too I think.

We could at first add manual reservations if there are any. A newer thread here only lists reservations outside of this extension. With the exception of 10077696 which LaurV reserved (http://www.mersenneforum.org/showpos...05&postcount=3).

Upon a ten minute review of code I've lost currency in, I believe that fixing the allseq and reservations scripts could be as simple as adjusting the '6' to '7' in these three lines, though in retrospect the last two probably should have been factored together to a common source.

I'm not sure that these three character changes would be sufficient, but I *think* they should be. We can always test it the fun way

[Dubslow]

Originally Posted by ugly2dog

Attached is what I have open in the 2e6-3e6 range.
Again all 9457 have been taken to at least 100 digit.
Same caveat as before.
I am currently showing 9601 open in the 3e6-4e6 range and 9634 in the 4e6-5e6 range.
I am still working in those range to get them all to 100 digits.

May we take this to mean that you have never worked above 5e6?

Given that this information is (presumably) available, might this forum consider extending to 5e6 rather than 2e6? Update intervals might be severely reduced, though IIRC there are already mods in place to the code that update the yoyo seqs faster.

[ugly2dog]

I haven't worked above 5e6, but Greebley has a thread about working on the 5e6-10e6 range. So no idea on what has been done. I was planning on moving above 5e6 after I get everything below that to at least 100 digits.
By extending the range up to 5e6, you will have another 38,141 sequences to check. To keep them updated, it might put a little bit of a strain on the FactorDB.

[Dubslow]

Originally Posted by ugly2dog

I was planning on moving above 5e6 after I get everything below that to at least 100 digits.

I suppose since no one's put the thought to paper yet, would you be okay with the forum co-opting your work and range?

Originally Posted by ugly2dog

By extending the range up to 5e6, you will have another 38,141 sequences to check. To keep them updated, it might put a little bit of a strain on the FactorDB.

I would propose that we keep the same current rate of updates (~3.5 days for ~9K sequences), accept the slower time for non-yoyo sequences (up to the order of two weeks), and continue to update the yoyo sequences more frequently, though within reason if the yoyo reservations extend into multiple thousands of sequences.

[ugly2dog]

I am currently have everything below 3461696 to at least 100 digits. So I don't care about those.
From 3461730 to 3999888, I am currently working on these to 100 digits.
From 4000050 to 4125144 are at least to 100 digits. So I don't care about those.
From 4125240 to 4157352 are at least to 97 digits. So I don't care about those.
From 4157460 to 4999980, I am currently working on these to 97 digits.
So have at it.
I am reserving 2005020.

Attached are 3e6-4e6 and 4e6-5e6.

I do believe that I have all the open sequences, but I might have had a few fall through the cracks in the past six years.

[yoyo]

Do you see a way how we can add them to the reservation page in a way that they are not yet checked against factordb to avoid factordb to overload? In this way the sequences go to the normal update cycle.

[Dubslow]

Originally Posted by yoyo

Do you see a way how we can add them to the reservation page in a way that they are not yet checked against factordb to avoid factordb to overload? In this way the sequences go to the normal update cycle.

It's not hard. ChristianB ought to be able to do it in a few minutes. Simply collate the files u2d has posted, copy and paste the selected ranges onto one line (each), and then he can use the reservations.py script in manual mode (there's a command to make it auto-spider, but before that was the manual "add" and "drop" commands that I used by hand back in... 2013 or so?).

[henryzz]

I quite like the idea of adding further ranges to the project. Originally this project was a great environment for people to initially learn about factorization. The length of factorizations has grown a bit beyond that now.

We do need to think about what this will add in terms of admin. A lot of sequences will terminate quickly in the new ranges. They will need adding to the terminating thread. I would suggest automating this.

In terms of how often sequences are updated I would suggest updating smaller sequences more often. Reserved sequences can be updated as frequently as sequences x digits smaller(30 maybe?). Maybe sequences should be updated when reserved or dropped. This will need balancing with server load at the factordb. We should possibly also make sure that we have an offline copy of all sequences beyond 100 digits as a backup.

[yoyo]

Originally Posted by Dubslow

It's not hard. ChristianB ought to be able to do it in a few minutes. Simply collate the files u2d has posted, copy and paste the selected ranges onto one line (each), and then he can use the reservations.py script in manual mode (there's a command to make it auto-spider, but before that was the manual "add" and "drop" commands that I used by hand back in... 2013 or so?).

Add requires a name to whom it should be reserved and it will add it only if it is a sequence which should be handled:

reservations.py add xxx 3000060
10:34:53 Read 2816 seqs
10:34:53 Add 1 seqs
10:34:53 Warning: 3000060 doesn't appear to be in the list
10:34:53 Wrote 2816 seqs

Afterwards the sequence is not in reservations and AllSeq.* files.

As I understood the python scripts I have to add them (with some dummy values) into AllSeq.json. Afterwards they go to the normal update handling.

[Dubslow]

Originally Posted by yoyo

Add requires a name to whom it should be reserved and it will add it only if it is a sequence which should be handled:

reservations.py add xxx 3000060
10:34:53 Read 2816 seqs
10:34:53 Add 1 seqs
10:34:53 Warning: 3000060 doesn't appear to be in the list
10:34:53 Wrote 2816 seqs

Afterwards the sequence is not in reservations and AllSeq.* files.

As I understood the python scripts I have to add them (with some dummy values) into AllSeq.json. Afterwards they go to the normal update handling.

I would recommend at this point that you wait and let ChristianB do the work. But, if you're even half as impatient as I am, here's the gist:

If you examine the allseq.py script, you will notice that it uses a master text file with a sequence-per-line list of all active sequences. Said file is called AllSeqs.txt, and is located.... well, somewhere. I think ChristianB changed its location on your server, but something like $ locate AllSeqs.txt ought to be able to find it. (Edit: Reviewing CB's web-rkn branch, it doesn't seem to have moved -- it can be found in website/html/AllSeqs.txt.)

To add new sequence to this "active" list, it suffices merely to take the four files that ugly2dog posted, and cat them onto AllSeqs.txt. That will (slowly) begin tracking them as the updates cycle through. (You might consider updating the per_hour variable in the script, or possibly modify the cronfile to launch allseq.py more frequently, but either way (or both) I would highly recommend no more than double the current update rate.

^ This should get the new sequences tracked, but to get reservations working you also need to make the changes I mentioned a few posts ago to allow 7 digit sequences.

Once you have made those changes so that the reservation scripts recognize 7 digits, the following procedure is an example of what I described in my previous post:

~/Downloads $ python3
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> with open('Open_4e6-5e6.txt') as f:
...     seqs = [int(line) for line in f]
... 
>>> len(seqs)
9634
>>> seqs[:10]
[4000050, 4000068, 4000272, 4000536, 4000542, 4000776, 4000800, 4000962, 4001046, 4001052]
>>> seqs[-1]
4999980
>>> seqs.index(4157460)
1536
>>> seqs = seqs[1536:]
>>> seqs[0]
4157460
>>> string = ' '.join(str(seq) for seq in seqs)
>>> len(string)
64783
>>> with open('many_seqs_reserved.txt', 'w') as f:
...     f.write(string)
... 
64783

I can then open that last text file with a graphical editor, whence I can copy and paste it: https://pastebin.com/73pngzde


Then the reservations.py command would look like this: reservations.py add ugly2dog <LongStringToBeCopiedFromPasteBin>.

Actually, I just made the string for his first batch reservation as well. https://pastebin.com/cGm2ykAc

Actually, I just remembered that ChristianB has pushed his changes for your live server to GitHub, so you can review what changes he's made relative to my base code here: https://github.com/ChristianBeer/Mer...t/tree/web-rkn

EditEdit: Be sure to only modify the master sequence list when scripts/allseq.py.lock does not exist, otherwise the changes will be overwritten and lost. And then after you've modified the master sequence list, also modify (when the lock doesn't exist!) scripts/allseq.py.conf to 9100 or so, so that it immediately starts updating the new sequences (instead of wherever it was in the current list).

PM me with any further problems.

[Dubslow]

Originally Posted by Dubslow

Upon a ten minute review of code I've lost currency in, I believe that fixing the allseq and reservations scripts could be as simple as adjusting the '6' to '7' in these three lines, though in retrospect the last two probably should have been factored together to a common source.

I'm not sure that these three character changes would be sufficient, but I *think* they should be. We can always test it the fun way

Not quite! One more line I caught on a second read through. I believe this commit should be all that's necessary to support 7 digit sequences.

@yoyo this post amends and aids the previous one.

Also of note: the forum god changed the date format for posts and edits sometime in the last several months!