Finessing the Aliquot reservation page to sequences above 1 million - Page 5

Dubslow · 2017-11-12, 01:39

Current issues with 1-2e6 integration notwithstanding (they are easily solvable), what is the general interest in extending the open range to 3e6 or 5e6?

I lean towards "we're fine as it is", but I'm curious as to the general opinion.

VBCurtis · 2017-11-12, 01:48

I am quite surprised at the burst of interest in the 1e6-2e6 range, and think that we should wait a while on 2e6-5e6; the better to rekindle interest in 2018 and beyond.
It'll take quite a while to get all these new ones up to dwindling-interest level, and only then should 2e6-3e6 be imported into the system.

ChristianB · 2017-11-12, 21:36

We are feeding the list with numbers that are not in u2d's list but survived our offline check (this was done using aliqueit to detect obvious merges with lower sequences that happened early but in order to go through the list quickly I stopped when the next iteration reached 50 digits, we then triggered the surviving sequences in FDB so the AllSeq page could detect merges that are happening "further up"). From time to time we're deleting the terminations manually. With the current rate this will take 4 more days to finish.

In the meantime you will also see some undetected mergers. If you want to help clean them up please post a space separated list of sequences (both) either here or in the terminations and mergers thread. I can quickly run this list through the script and it will automatically remove the mergers.

Batalov · 2017-11-12, 22:39

1734720 1646674
1725090 462918

Stargate38 · 2017-11-12, 23:03

1575450 453798
1731630 1702422

Batalov · 2017-11-12, 23:50

1766652 802284
1606724 802284
1734720 670516

Dubslow · 2017-11-13, 06:32

Quote:

Originally Posted by ChristianB

In the meantime you will also see some undetected mergers. If you want to help clean them up please post a space separated list of sequences (both) either here or in the terminations and mergers thread. I can quickly run this list through the script and it will automatically remove the mergers.

To elaborate: the update script does detect and remove mergers automatically; the problem is the same as its always been since I first added the mergefinder: synchronicity of data for the two seqs. In the chaos of the last week, a lot of "pending" mergers just have some delay between when each of the pair updated, allowing the merged sequence to extend between either update; they will all automatically eventually be found when the churn dies down. Perhaps a week or two if left alone.

But the script can "manually" update a specific set of sequences as well, which will eliminate the synchronization problem nicely. Given the amount of reservation and interest churn is as much as the data-itself churn, ChristianB was volunteering to take the time to run the manual updates and get the merges out. Any sequences in need of synchronizing can all be put on the same line, which is most convenient for copying into a terminal. Example:

1734720 1646674 1725090 462918 1575450 453798 1731630 1702422 1766652 802284 1606724 802284 1734720 670516

Allows ChristianB to do this:

Code:

bill@Gravemind ~/mfaliquot/scripts $ ./allseq.py 1734720 1646674 1725090 462918 1575450 453798 1731630 1702422 1766652 802284 1606724 802284 1734720 670516

2017-11-13 00:41:51
00:41:51 Initializing
00:41:52 Added 7 new sequences: [1734720, 1646674, 1725090, 1575450, 1731630, 1766652, 1606724]
00:41:52 Init complete, starting FDB queries
00:41:52 1 sequence  complete: 1734720
00:41:54 2 sequences complete: 1646674
00:41:55 3 sequences complete: 1725090
00:41:57 4 sequences complete: 462918
00:41:59 5 sequences complete: 1575450
00:42:01 6 sequences complete: 453798
00:42:03 7 sequences complete: 1731630
00:42:05 8 sequences complete: 1702422
00:42:07 9 sequences complete: 1766652
00:42:08 10 sequences complete: 802284
00:42:10 11 sequences complete: 1606724
00:42:11 12 sequences complete: 802284
00:42:13 13 sequences complete: 1734720
00:42:15 14 sequences complete: 670516
00:42:16 Loop complete, new data saved
00:42:16 Found merges!
00:42:16 1725090 seems to have merged with 462918
00:42:16 1840938 seems to have merged with 1839978
00:42:16 1847468 seems to have merged with 1600508
00:42:16 1840972 seems to have merged with 1839978
00:42:16 1734720 seems to have merged with 670516
00:42:16 1766652 seems to have merged with 802284
00:42:16 1840926 seems to have merged with 1839978
00:42:16 1839978 seems to have merged with 1839966
00:42:16 1766652 seems to have merged with 1606724
00:42:16 1840158 seems to have merged with 1839426
00:42:16 1844766 seems to have merged with 1839174
00:42:16 1734720 seems to have merged with 1646674
00:42:16 1844886 seems to have merged with 46758
00:42:16 1844898 seems to have merged with 46758
00:42:16 1844910 seems to have merged with 1839978
00:42:16 1575450 seems to have merged with 453798
00:42:16 1839786 seems to have merged with 1839426
00:42:16 1848840 seems to have merged with 1839324
00:42:16 1731630 seems to have merged with 1702422
00:42:17 alimerge email failed
00:42:17 Dropped 17 seqs from seq_list: {1734720, 1725090, 1844898, 1848840, 1840938, 1839978, 1840972, 1847468, 1844910, 1839786, 1731630, 1840926, 1844886, 1840158, 1575450, 1766652, 1844766}
00:42:17 Dropped 17 seqs from datadict: {1734720, 1725090, 1844898, 1848840, 1840938, 1839978, 1840972, 1847468, 1844910, 1839786, 1731630, 1840926, 1844886, 1840158, 1575450, 1766652, 1844766}
00:42:18 Written all data and HTML
00:42:18 Saved state and finalized.

Evidently, as of this moment, there's 10 more mergers ready to go in the next update!

ChristianB · 2017-11-17, 13:50

We have now finished adding sequences to the list. There seem to be around 2000 sequences more than expected fromjust adding u2d's list. Those are expected to merge with other sequences. I randomized the sequences that are fetched from FDB to hopefully find those mergers quick. I also increased the update interval to fetch 110 sequences every 15 minutes. The first update (as I write this) has already deleted 8 merged sequences.

ChristianB · 2017-11-19, 13:56

Thanks to contributions from forum members we have now cleared the remaining merges from the list. I started a complete update that is finished in 2 days.

We can now control which sequences to update more often and which to update less often based on the data in the list. I would like to hear from users of the list which sequences they would like to see updated more often. An example would be all sequences with a downdriver or all sequences with a low cofactor or ...

I would also like to start discussion on how to do the next extension of the list that is maybe happening early next year. We have the 2e6-3e6 list from u2d but he is not 100% sure that it is complete. So should we (the forum) do a doublecheck of the whole 2e6 region or not and if yes, how should we do that in order to find a lot of the merges quickly before adding the sequence to the list?

Dubslow · 2017-11-19, 17:58

Quote:

Originally Posted by ChristianB

So should we (the forum) do a doublecheck of the whole 2e6 region or not and if yes, how should we do that in order to find a lot of the merges quickly before adding the sequence to the list?

If and when we decide to extend, there's plenty of code just recently posted in the terminations thread. Code like that can be used first, feed the results of that into the 50-digit stuff you had been doing, and that should be significantly better than this time was.

Dubslow · 2018-01-15, 22:04

Okay, I think there's a fair bit of room with the new priority system to continue extending up to 5e6 with no additional strain on the FDB (with possibly 1e7 in the medium future). I'd like to reopen the discussion there and ask for opinions.

Also, I just verified that ugly2dog's 1e6-2e6 was missing no unique sequences:

Code:

bill@Gravemind ~/mfaliquot/scripts $ ./filter_seqs.py "ali.seq >= 1000000" "ali.seq" 10000 "
" > 1e6-2e6.txt
bill@Gravemind ~/mfaliquot/scripts $ comm -3 1e6-2e6.txt ~/Downloads/Open_1e6-2e6.txt
	1052310
	1053870
	1110258
	1164432
	1174086
	1183656
	1211160
	1280520
	1291620
	1322748
	1323088
	1338360
	1348410
	1356492
	1382232
	1431248
	1442016
	1455860
	1547640
	1596156
	1654488
	1684704
	1741632
	1766364
	1831464
	1855856
	1868148
	1913598
	1925148
	1934442
	1938060
	1952172
	1952670
	1974448
	1978340
	1983288
	1983780
	1989594

(Column 1 is blank, it would have any sequences in the current file that weren't in u2d's file. Column 2 contains those sequences which are in u2d's file but not the current list, i.e. those which have merged/termed since the extension.)

What are the opinions on extending to 5e6? (If we do, we would need to respect the reservations in post 16.)

2017-11-12, 23:50	#50
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 2741₁₆ Posts	1766652 802284 1606724 802284 1734720 670516 Last fiddled with by Batalov on 2017-11-13 at 01:05

2018-01-15, 22:04	#55
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 3·29·83 Posts	Okay, I think there's a fair bit of room with the new priority system to continue extending up to 5e6 with no additional strain on the FDB (with possibly 1e7 in the medium future). I'd like to reopen the discussion there and ask for opinions. Also, I just verified that ugly2dog's 1e6-2e6 was missing no unique sequences: Code: bill@Gravemind ~/mfaliquot/scripts $ ./filter_seqs.py "ali.seq >= 1000000" "ali.seq" 10000 " " > 1e6-2e6.txt bill@Gravemind ~/mfaliquot/scripts $ comm -3 1e6-2e6.txt ~/Downloads/Open_1e6-2e6.txt 1052310 1053870 1110258 1164432 1174086 1183656 1211160 1280520 1291620 1322748 1323088 1338360 1348410 1356492 1382232 1431248 1442016 1455860 1547640 1596156 1654488 1684704 1741632 1766364 1831464 1855856 1868148 1913598 1925148 1934442 1938060 1952172 1952670 1974448 1978340 1983288 1983780 1989594 (Column 1 is blank, it would have any sequences in the current file that weren't in u2d's file. Column 2 contains those sequences which are in u2d's file but not the current list, i.e. those which have merged/termed since the extension.) What are the opinions on extending to 5e6? (If we do, we would need to respect the reservations in post 16.)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Aliquot Sequences Summary Page	kar_bon	Aliquot Sequences	174	2022-03-19 13:07
Sequences from the 5 to 10 million range	Greebley	Aliquot Sequences	7	2015-09-04 08:22
Broken aliquot sequences	schickel	FactorDB	18	2013-06-12 16:09
A new theorem about aliquot sequences	garambois	Aliquot Sequences	34	2012-06-10 21:53
poaching aliquot sequences...	Andi47	FactorDB	21	2011-12-29 21:11

2017-11-12, 01:39	#45
Dubslow Basketry That Evening! "Bunslow the Bold" Jun 2011 40<A<43 -89<O<-88 3×29×83 Posts	Current issues with 1-2e6 integration notwithstanding (they are easily solvable), what is the general interest in extending the open range to 3e6 or 5e6? I lean towards "we're fine as it is", but I'm curious as to the general opinion.

2017-11-12, 01:48	#46
VBCurtis "Curtis" Feb 2005 Riverside, CA 17×331 Posts	I am quite surprised at the burst of interest in the 1e6-2e6 range, and think that we should wait a while on 2e6-5e6; the better to rekindle interest in 2018 and beyond. It'll take quite a while to get all these new ones up to dwindling-interest level, and only then should 2e6-3e6 be imported into the system.

2017-11-12, 21:36	#47
ChristianB Apr 2013 Germany 100111110₂ Posts	We are feeding the list with numbers that are not in u2d's list but survived our offline check (this was done using aliqueit to detect obvious merges with lower sequences that happened early but in order to go through the list quickly I stopped when the next iteration reached 50 digits, we then triggered the surviving sequences in FDB so the AllSeq page could detect merges that are happening "further up"). From time to time we're deleting the terminations manually. With the current rate this will take 4 more days to finish. In the meantime you will also see some undetected mergers. If you want to help clean them up please post a space separated list of sequences (both) either here or in the terminations and mergers thread. I can quickly run this list through the script and it will automatically remove the mergers.

2017-11-12, 22:39	#48
Batalov "Serge" Mar 2008 Phi(4,2^7658614+1)/2 13·773 Posts	1734720 1646674 1725090 462918

2017-11-12, 23:03	#49
Stargate38 "Daniel Jackson" May 2011 14285714285714285714 3·251 Posts	1575450 453798 1731630 1702422

2017-11-17, 13:50	#52
ChristianB Apr 2013 Germany 2·3·53 Posts	We have now finished adding sequences to the list. There seem to be around 2000 sequences more than expected fromjust adding u2d's list. Those are expected to merge with other sequences. I randomized the sequences that are fetched from FDB to hopefully find those mergers quick. I also increased the update interval to fetch 110 sequences every 15 minutes. The first update (as I write this) has already deleted 8 merged sequences.

2017-11-19, 13:56	#53
ChristianB Apr 2013 Germany 13E₁₆ Posts	Thanks to contributions from forum members we have now cleared the remaining merges from the list. I started a complete update that is finished in 2 days. We can now control which sequences to update more often and which to update less often based on the data in the list. I would like to hear from users of the list which sequences they would like to see updated more often. An example would be all sequences with a downdriver or all sequences with a low cofactor or ... I would also like to start discussion on how to do the next extension of the list that is maybe happening early next year. We have the 2e6-3e6 list from u2d but he is not 100% sure that it is complete. So should we (the forum) do a doublecheck of the whole 2e6 region or not and if yes, how should we do that in order to find a lot of the merges quickly before adding the sequence to the list?