Train for maia:2000 maia:2100 maia:2200

[jorditg]

It would be interesting to have other more powerful versions having the same human-like style.

[reidmcy]

Making higher rating Maia models is somewhat challenging as there are not enough games for us to use the same sampling methods we used for the published models. We have done some experiments with different strategies but the results have been unsatisfactory. Also as the ratings increases the players' moves approach those of Stockfish/Leela.

I'd also like to note that the goal of our project is not to create the best chess engine, we are trying to create models that can be used for learning/teaching.

[jorditg]

Thank you redmcy for your feed back. I understand the argument of not having enough data for training. In any case, if it was possible to train them for the proposed elos, you should have to take in to account that they will have also a great value. Think that leela and stockfish are 3500+ engines and that having a human like engine of a similar strength can be very valuable for training purposes. 2000- 2500 are the more common elos in OTB amateur chess, that's why it would be valuable to have human like engines of such elo ratings.

Thanks for your great work!

[FrugoFruit90]

Bumping @jorditg, as I myself would love to have a human-like sparring partner of this strenght. #2000-2500PlayerLivesMatter
What is the data requirement @reidmcy? In terms of number games for this to make sense.
For example, what if we just merged this whole segment?

[reidmcy]

We need about 12 million games, with good endgames so not fast time controls.

Mixed player ratings was one of the first things we tried after the first paper, it gave much weaker results.

[jorditg]

Do we have any performance values using maia9 for example with 16, 32, 64, 128, etc. tree search values instead of 1?

[reidmcy]

Figure 5 of the paper shows 10 rollouts, but we did others too with similar results

[wickeduk]

If number of games is a problem for >1900, why not use approx conversions and also import games from chess.com, FICS and ICC (the later two go back 25 years although ratings may have deflated / inflated during the course) into the training set. It might need a bit of tweaking to get it right, but I suspect the drop off of games between 1900 and 2000 is not that much that taking additional games from other servers into account should at least get you 2000 if not 2100.

However it does raise a bigger question which I think is an interesting one and why you might not get such good results above 1900, but worth investigating for both AI and chess. The higher the rating the more balancing intuition with calculation takes place. A master will say things like I feel this is the right move, or I'm not worried about that move, particularly at faster time controls which use of the clock is a big factor. When it does get a little more complicated they calculate, often 2 or 3 moves ahead (and as the time control increases conscious calculation becomes more a factor). By using depth 1 and a neural net, you're simulating which move would be played if calculating ahead were disallowed (although taking tactical patterns into account), and when you start going higher this won't predict the move made when calculation would have been made. This works well to simulate sub-2000 play, but not so much above that I believe. Thus to make a representative 2000 and above player (and there is a lot of interest here from improving players looking for a sparring/training opponent) you need to have an engine that balances choosing moves using intuition vs calculating when necessary. Using 'unnecessary' (or always to depth) calculation will cause evaluation of moves/positions the human wouldn't consider (even if it were finding objectively the best move). This might be impossible to train because humans don't really understand intuition well and the data where the player calculated or not isn't available except by guessing on how much clock time was used (not always an accurate indicator of tactical complexity), but it might be possible via some heuristic which sets the depth based on how tactical the position is and balances the tree / depth with the skill level being represented.

[reidmcy]

I did some experimenting with lowering the quality standards and didn't get good results. Using games from outside Lichess is tricky as most other servers don't have free archives available. I also can't go violating the terms of use for the sites with a scraper even if the data are available since I'm doing this as a part of my PhD.

I think your second point is interesting and we have a student looking into something similar since May, we frame it as more of an inverse RL task.

I do alos think your concerns about depth of search are a bit off, the neural network could be doing some kind of search internally. In fact the model is designed to extra information sequentially so is almost certainly do a some kind of search. So depth 1 search doesn't mean the same thing to Maia/Leela as to stockfish or a human.

[wickeduk]

It's possible to download games from chess.com (by player at least, I'm not sure about by rating) as well as FICS has a database ficsgames.org. FICS' database goes back more than 20 years - smaller numbers, but if you add it to the set it may get it above the number you need . Probably if you asked nicely at ICC (assuming there isn't already a download) they may be able to help you too and would have more games around the 2000 mark. The only problem would be standardising approximate rating across servers for which there are a number of surveys out there. David Sent from Yahoo Mail on Android On Wed, 15 Sep 2021 at 9:07, Reid ***@***.***> wrote: I did some experimenting with lowering the quality standards and didn't get good results. Using games from outside Lichess is tricky as most other servers don't have free archives available. I also can't go violating the terms of use for the sites with a scraper even if the data are available since I'm doing this as a part of my PhD. I think your second point is interesting and we have a student looking into something similar since May, we frame it as more of an inverse RL task. I do alos think your concerns about depth of search are a bit off, the neural network could be doing some kind of search internally. In fact the model is designed to extra information sequentially so is almost certainly do a some kind of search. So depth 1 search doesn't mean the same thing to Maia/Leela as to stockfish or a human. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[reidmcy]

The Chess.com per player downloads is what I was alluding to with the scraping comment

[codeisnotfunjk]

Making higher rating Maia models is somewhat challenging as there are not enough games for us to use the same sampling methods we used for the published models. We have done some experiments with different strategies but the results have been unsatisfactory. Also as the ratings increases the players' moves approach those of Stockfish/Leela.

I found some resources you can scrape off of(I already wrote this here: #43 (comment))

FICS Games: a free resource that offers a large selection of matches and allows you to sort by 2000-2199 ELO, 2200-2399 ELO, etc.- https://www.ficsgames.org/

Chessbase: a paid service that provides access to millions of matches played by higher ELO players- https://database.chessbase.com/

Lichess: another free option with over 8 million matches, including higher-elo games, sorted by month- https://database.lichess.org/

Chesstempo: offers over 2 million searchable games and allows you to sort by min and max ELO in advanced settings- https://old.chesstempo.com/game-database.html

EDIT: Some other resources I discovered-

https://www.kaggle.com/datasets/datasnaek/chess
https://www.kaggle.com/datasets/zq1200/world-chess-championships-1866-to-2021
https://www.kaggle.com/datasets/ronakbadhe/chess-evaluations
https://www.chess.com/games
https://www.chessabc.com/en/chessgames
https://gameknot.com/chess-games-database.pl
https://sourceforge.net/projects/codekiddy-chess/
http://www.chessgameslinks.lars-balzer.info/
https://www.openingmaster.com/chess-databases
https://www.365chess.com/chess-games.php

[codeisnotfunjk]

It would be interesting to have other more powerful versions having the same human-like style.

Someone made a higher ELO version of maia on Lichess. Link is below.

https://lichess.org/@/humanian

[reidmcy]

We can't easily combine games from different sources since Elo is not consistent between them. We also need about 12 million games after filtering so most of those sites are much too small. I also need to use sites that have licenses that allow use. I have been running experiments with Lichess, since there are many more games since the last paper, but I don't have anything to publicly release yet.

Also, keep in mind this is an academic project. The goal is not just to release better models, the goal is to do unique things and release new ideas. So releasing new models will need to be part of a larger project.

[reidmcy]

It would be interesting to have other more powerful versions having the same human-like style.

Someone made a higher ELO version of maia on Lichess. Link is below.

https://lichess.org/@/humanian

That appears to be our Maia-1800 weights using MCTS search. As we showed in the original paper, using search reduces humanness of the engine. So that's closer to a weak Leela model than our Maia models.

[hwuebben]

It would be interesting to have other more powerful versions having the same human-like style.

Someone made a higher ELO version of maia on Lichess. Link is below.

https://lichess.org/@/humanian

That appears to be our Maia-1800 weights using MCTS search. As we showed in the original paper, using search reduces humanness of the engine. So that's closer to a weak Leela model than our Maia models.

Hey :)
I'm interested whether there is data as to how the estimated elo of a maia model changes when using mcts (even if it loses humanness) . Like what for example would be the elo of the 1900 Maia with search depth 3? Any experiments there?