How to use MOTD3 algorithm for optimization

Currently MOTD3 algorithm is available in a branch and it is only tested and intended for purely optimization problems.
The plan is to update it in future to be compatible with controls problems.

Where is MOTD3?

MOTD3 algorithm is available in branch: https://github.com/JeffersonLab/SciOptControlToolkit/tree/14-add-motd3-for-optimization
It is registered as KerasMOTD3-v0
It uses mo_actor_fcnn registered as mo_actor_fcnn-v0 and mo_critic_fcnn mo_critic_fcnn-v0 registered as
Both mo_actor and mo_critic inherits from base actor and critic and only overwrites what's different
It also uses mo_er buffer registered as MO-ER-v0 to keep track of alphas in addition to traditional single objective tuples.

Clone the repository and - Switch to the branch that contains MOTD3 by running the following command

git checkout 14-add-motd3-for-optimization

Create the env by following the readme at the root directory.
- Note that pymoo is an additional package that is required to run MO algorithm, please install it with it's proper version if you already have the env.
You also need Multi-objective optimization environment (hint: use CEBAF environments if you have access to SCORE repository)

Once you have SOCT and SCORE repositories installed, you can run the following command to run the training

python mo_run_continuous.py --agent KerasMOTD3-v0 --env SCORE-MO-CEBAF-8D-VEC-v1

The driver will create a default results directory in the drivers directory and save all the results there. You can expect to see bunch of plots being generated as the training proceeds to monitor the progress.
If you would like to compare with the benchmark results, you can provide location (directory) where benchmark results are saved (Note: You can get the benchmark results by contacting us)

 python mo_run_continuous.py --agent KerasMOTD3-v0 --env SCORE-MO-CEBAF-8D-VEC-v1 --benchmark <benchmark_directory>

If you run the above command to run 8D env, you should see results similar to below figure at around 50K steps.