The policy-based version of Mortal has been open-sourced

[Nitasurin]

https://github.com/Nitasurin/Mortal-Policy
Its complete original implementation based on MortalV2 has been used in certain scenarios since 2022

[Equim-chan]

Excellent work!

Policy-based methods were what I started with at the very beginning (pre-v1). I experimented with various common methods and practices and spent a lot of time (several months) tuning it. However, they all failed to outperform the baseline. In the end, surprisingly and somewhat unreasonably, simple value-based methods turned out to be effective, so I went on building Mortal with value-based methods.

[Nitasurin]

Could you clarify which baseline you're referring to? Is it Akochan? According to my tests, this repository shows improvements over the baseline model I previously submitted to mjai.app

[Equim-chan]

The baseline was a supervised learning model.

[hyskylord]

Have you tried to compare the model with Akochan or Akagi? I think mjai.app doesn't provide enough samples to compare the performance of models.

Also, what's the size of your model?

[Nitasurin]

@hyskylord

1. From late 2022 to early 2023, I trained the Mortal-V2 and a policy version modified from Mortal-V2. The test results are as follows:
   Policy-based online + behavior cloning from value-based offline-to-online Mortal-V2 samples (now distillation might be a better choice) > value-based offline-to-online Mortal-V2 > value-based offline-only Mortal-V2.
   Among these, value-based online achieved only a very small performance improvement compared to offline (approximately +0.6PT in 500,000 Hanchan in 1v3 scenarios). However, after converting to Policy-based online, the performance improvement exceeded expectations, and the actual gameplay results on certain platforms were also satisfactory.
   In April 2024, I conducted tests using GPUs, accumulating several million Hanchan. By analyzing the historical performance of all players in the training dataset (including data obtained from some websites) and in-game statistics, I performed player-by-player cleaning of each dataset sample. Even the weakest value-based offline-only Mortal-V2 still outperformed the latest open weights in the community (approximately +1.8~2.0PT in 200,000 Hanchan against Akagi offline 240308 V4 best/V3 best in 1v3 scenarios).
   Due to the inability to obtain newer weights from the community, I remain cautiously confident about the above data as of February 2025.

2. In the policy version modified from Mortal-V2, I did not use ResNet-1D-CNN but instead opted for a more lightweight model network. Typically, online training does not require as large a parameter scale as offline reinforcement learning, while also improving training and sampling speeds.
   However, for specific reasons, I did not include the incremental code used for online training in the Mortal-policy library, such as opponent pools, replay buffers, multi-GPU parallelism, etc. (similar implementations exist in existing open-source libraries). Instead, I only modified the original Mortal repository code to implement the loss function and reward calculation.
   AWR (Advantage-Weighted Regression) is a new addition when I decided to open-source, replacing Behavior Cloning. I experimented with several hyperparameter sets, and under the best parameter group and a cleaned dataset, it provided performance slightly below CQL as initial parameters for online training. With appropriate hyperparameters, I also conducted preliminary online training, and the performance of Mortal-policy met my expectations.

3. I remain cautious about the results from mjai.app, and this attitude extends to actual gameplay. However, this is not due to the common viewpoint of "insufficient samples" (in the absence of parallelization conditions, conducting 100,000 games for a single weight version is unacceptable in terms of time cost).
   Instead, it is because I noticed that explicit or implicit opponent strategy modeling/estimation may have a greater impact on actual gameplay results than the differences shown in direct AI vs. AI matches. These influences cannot be adequately explained by insufficient samples or luck.