mersenneforum.org  

Go Back   mersenneforum.org > Other Stuff > Archived Projects > NFSNET Discussion

 
 
Thread Tools
Old 2003-06-18, 09:43   #1
ColdFury
 
ColdFury's Avatar
 
Aug 2002

1010000002 Posts
Default Stats Flakiness?

One thing I've been noticing for a while is that occasionally the stats will flake out and display wierd values. The total lines sieved and relations found will drop dramatically from the day before, and the active computers number will drop as well. However, after the next day's stats run, everything will be back to normal.

Is this an outstanding bug in the stats?
ColdFury is offline  
Old 2003-06-20, 10:31   #2
xilman
Bamboozled!
 
xilman's Avatar
 
"๐’‰บ๐’ŒŒ๐’‡ท๐’†ท๐’€ญ"
May 2003
Down not across

71×163 Posts
Default Re: Stats Flakiness?

Quote:
Originally Posted by ColdFury
One thing I've been noticing for a while is that occasionally the stats will flake out and display wierd values. The total lines sieved and relations found will drop dramatically from the day before, and the active computers number will drop as well. However, after the next day's stats run, everything will be back to normal.

Is this an outstanding bug in the stats?
Sort of.

Behind the scenes, what happens is this: a server allocates a client a task; the client does some computation; the results of the computation are sent back to the server; the server stores the results on disk for later processing.

The results from the client includes both data (the relations) and metadata which is stuff like which client did the work, when the task was started, when it finished, how many lines were sieved, how many relations were found, how many cpu-seconds were taken and so on.

Once a day, just after midnight UTC, each server batches up all the relations and sends them to a "warehouse" site where they are stored until the accumulated relations is sufficient to complete the factorization. It also batches up all the metadata and sends it off to the stats processor. The stats processor would get confused by corrupted metadata so we attempt to filter out all the junk and send it off to a repair-shop to see whether anything can be salvaged from it.

By and large the mechanism outlined above works well. Every now and again some junk gets through the filter in error and confuses the stats server, resulting in the behaviour you described. As soon as we notice, we remove the junk by hand and re-run the stats generation.

To quantify the "by and large" comment: around a percent of the results sent in by clients is corrupted in one way or another. So each and every day, there is junk to be filtered out. Only once a week or less does an item of junk get through to confuse the stats server. We're always aiming to improve the process, but it's not perfect yet. If we're honest, it never will be perfect (nothing is) but were certainly working to improve it markedly.


Paul
xilman is online now  
 

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Overall Stats Fred PrimeNet 7 2016-02-10 14:48
CSVs for stats available + New combined stats opyrt Prime Sierpinski Project 3 2010-05-31 08:13
Stats em99010pepe Sierpinski/Riesel Base 5 66 2008-02-06 19:42
stats on 15k jocelynl 15k Search 5 2004-01-13 15:45
P4 On Stats HiddenWarrior Hardware 2 2003-08-13 14:39

All times are UTC. The time now is 19:06.


Sun Nov 27 19:06:11 UTC 2022 up 101 days, 16:34, 0 users, load averages: 1.43, 1.42, 1.21

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.

โ‰  ยฑ โˆ“ รท ร— ยท โˆ’ โˆš โ€ฐ โŠ— โŠ• โŠ– โŠ˜ โŠ™ โ‰ค โ‰ฅ โ‰ฆ โ‰ง โ‰จ โ‰ฉ โ‰บ โ‰ป โ‰ผ โ‰ฝ โŠ โŠ โŠ‘ โŠ’ ยฒ ยณ ยฐ
โˆ  โˆŸ ยฐ โ‰… ~ โ€– โŸ‚ โซ›
โ‰ก โ‰œ โ‰ˆ โˆ โˆž โ‰ช โ‰ซ โŒŠโŒ‹ โŒˆโŒ‰ โˆ˜ โˆ โˆ โˆ‘ โˆง โˆจ โˆฉ โˆช โจ€ โŠ• โŠ— ๐–• ๐–– ๐–— โŠฒ โŠณ
โˆ… โˆ– โˆ โ†ฆ โ†ฃ โˆฉ โˆช โŠ† โŠ‚ โŠ„ โŠŠ โŠ‡ โŠƒ โŠ… โŠ‹ โŠ– โˆˆ โˆ‰ โˆ‹ โˆŒ โ„• โ„ค โ„š โ„ โ„‚ โ„ต โ„ถ โ„ท โ„ธ ๐“Ÿ
ยฌ โˆจ โˆง โŠ• โ†’ โ† โ‡’ โ‡ โ‡” โˆ€ โˆƒ โˆ„ โˆด โˆต โŠค โŠฅ โŠข โŠจ โซค โŠฃ โ€ฆ โ‹ฏ โ‹ฎ โ‹ฐ โ‹ฑ
โˆซ โˆฌ โˆญ โˆฎ โˆฏ โˆฐ โˆ‡ โˆ† ฮด โˆ‚ โ„ฑ โ„’ โ„“
๐›ข๐›ผ ๐›ฃ๐›ฝ ๐›ค๐›พ ๐›ฅ๐›ฟ ๐›ฆ๐œ€๐œ– ๐›ง๐œ ๐›จ๐œ‚ ๐›ฉ๐œƒ๐œ— ๐›ช๐œ„ ๐›ซ๐œ… ๐›ฌ๐œ† ๐›ญ๐œ‡ ๐›ฎ๐œˆ ๐›ฏ๐œ‰ ๐›ฐ๐œŠ ๐›ฑ๐œ‹ ๐›ฒ๐œŒ ๐›ด๐œŽ๐œ ๐›ต๐œ ๐›ถ๐œ ๐›ท๐œ™๐œ‘ ๐›ธ๐œ’ ๐›น๐œ“ ๐›บ๐œ”