View Single Post
Old 2021-02-15, 13:53   #13
M344587487
 
M344587487's Avatar
 
"Composite as Heck"
Oct 2017

2×3×5×29 Posts
Default

An ordered list of moves for a position can be extracted from stockfish, by setting MultiPV to the number of valid moves of the position before telling SF to evaluate. If stockfish is used at all then using the full ordering is the way to go, otherwise we're throwing away a stronger evaluation for no reason. At depth 1 (where depth is a multiplier) it does require ~1TB of UCI communication to my code and ~35GB to stockfish for Caissabase, if this ~1TB is a problem stockfish can be modified to reduce this by an order of magnitude quite easily. At depth 1 it would take ~50 hours of a single Zen2 core to process Caissabase, so PGN splitting is implemented as a quick-and-dirty way of being able to parallelise, allowing an 8 core Zen2 to process Caissabase in ~8 hours.

The pgn can be converted to ila with stockfish sort but there's an as yet undiagnosed problem which means that converting from ila fails. Every sort method shares the same code path for non-sort operations, stockfish ordering appears to be deterministic, and the other sort algorithms can be encoded/decoded at will, so the bug hunting spear needs sharpening.

There's an extra way to squeeze more juice out of a generic compressor compressing ISA form, an ordering of the games increasing commonalities between adjacent games. Possibly as simple as ordering by the average index per game, next on the TODO list.
M344587487 is offline   Reply With Quote