usage.rst

Usage

You need to provide the following for an AC run with Selector:

• A .pcs file with a parameter space definition according to PCS (PCS Manual, PCS Example) and the path to it.
• Instances: A problem instance set for training, a .txt with paths to the instances (one path per line) and the path to it.
• Instance features: A .txt file with the first line as headers, with INSTANCE_NAME,featureName1,featureName2..., and a line with the instance name and feature values for each instance in the training set and the path to it. If you cannot provide features, set up mock features, e.g. a few features with all values set to 1. However, providing meaningful features will increase the AC success.
• A parameterized target algorithm that can be called via shell and passed paraemeters to.
• A Python wrapper for the target algorithm, its name and the name of the class. It is of the following form:

import argparse
import ast


class Your_Wrapper():

    def get_command_line_args(self, runargs, config):

        instance = runargs["instance"]
        id = runargs["id"]

        configuration =""
        for param, value in config.items():
                configuration += f" --{param}={value}"

        exc = '/absolute/path/to/your/target_alorithm'
        # Example command
        cmd = f"stdbuf -oL {exc} {configuration} -instance {instance}"
        return cmd

if __name__ == "__main__":
    parser = argparse.ArgumentParser()

    parser.add_argument('--runargs',type=ast.literal_eval)
    parser.add_argument('--config',type=ast.literal_eval)
    args = vars(parser.parse_args())

    wrapper = Your_Wrapper()

• The value 'desktop' or 'cluster' set to ray_mode, if using the Selector facade (see Example section)
• The optimization objective ('runtime' or 'quality')
• The termination criterion ('runtime' or 'quality')
• The time limit for target algorithm runs (set to cutoff_time)
• The penalty value for target algorithm runs (if not solving instance within time limit, or crashing)
• How many configurations can be considered as winner in one tournament
• The instance set size
• The size of the subset of instances for the first tournament
• The path to the directory Selector ought to log to
• The time Selector has for the complete AC process

Below you will find a table describing all of Selectors hyperparameters.

Hyperparameters for Selector

Name	Type	Default	Description
--file	str	""	Path to the file containing selector arguments.
--check_path	bool	False	If True, validates the paths passed to Selector.
--seed	int	42	Sets all random generators in Selector.
--verbosity	int	0	Selector prints less, if 0, more if 1.
--log_folder	str	"latest"	Path to the directory Selector ought to log to.
--memory_limit	int	3069	Maximum allowed memory for a target algorithm run to use at once.
--wrapper_mod_name	str	""	Name of the target algorithm wrapper. Notation as in module import, e.g. module.sub_module.wrapper.
--wrapper_class_name	str	""	Name of the class in the wrapper module.
--quality_match	str	""	Regex for a line that signifies a solved instance in quality optimization.
--solve_match	list of str	[]	List of strings that signify a solved instance in runtime optimization, e.g. ['SAT', 'UNSAT'].
--quality_extract	str	""	Regex to extract the objective value.
--winners_per_tournament	int	1	Number of configurations to regard as winners in a tournament.
--tournament_size	int	5	Number of configurations to compete in a tournament.
--number_tournaments	int	2	Number of parallel tournaments.
--monitor	str	"tournament_level"	For runtime optimization ("tournament_level" or "instance_level"), level of capping.
--surrogate_amortized_time	int	30	If the GGA model requires more than the set seconds, it will be updated less frequently.
--termination_criterion	str	"runtime"	Termination criterion for AC run ("runtime" or "total_tournament_number").
--total_tournament_number	int	10	Total number of tournaments for the main loop.
--model_update_iteration	int	3	Sets the frequency of GGA updates if amortized time violated (# concluded tournaments).
--generator_multiple	int	5	Factor to multiply the number of suggestions from one method.
--initial_instance_set_size	int	5	Instance set size of the first tournaments.
--set_size	int	50	Size of the training instance set.
--smac_pca_dim	int	8	PCA dimension of SMAC.
--tn	int	100	Evaluation history is reduced each time this many tournaments concluded.
--cleanup	bool	False	If True, tmp directory is regularly, actively cleaned by selector.
--cpu_binding	bool	False	If True, target algorithm and all its child processes are bound to one CPU.
--scenario_file	str	None	Path to a file containing hyperparameters concerning the AC scenario, see example.
--run_obj	str	None	Optimization metric ("runtime" or "quality").
--overall_obj	str	None	PAR to use with the evaluation results.
--cutoff_time	str	None	Time limit for a target algorithm to run on one instance.
--crash_cost	float	10000000	Penalty for unfinished instance or crash.
--wallclock_limit	str	None	Total amount of time Selector can run.
--instance_file	str	None	Path to the file containing the instance paths.
--feature_file	str	None	Path to the file containing the instance features.
--paramfile	str	None	Path to the parameter space definition of the target algorithm.
--runtime_feedback	str	""	Regex to extract runtime if target algorithm reports it (runtime optimization).

Example Scenario File

paramfile = /your/path/to/params.pcs
execdir = .
run_obj = runtime
overall_obj = PAR10
cutoff_time = 30
wallclock_limit = 6400
instance_file = /your/path/to/instances.txt
feature_file = /your/path/to/features.txt

Example Selector Args file

--seed 44
--par 10

--winners_per_tournament 1

--tournament_size 4
--number_tournaments 2

--termination_criterion total_runtime
--monitor tournament_level

--initial_instance_set_size 5
--set_size 91

--generator_multiple 1

--ta_pid_name ""
--memory_limit 2048
--cutoff_time 30

--check_path False

--log_folder selector_log
--wrapper_mod_name selector.your_wrapper
--wrapper_class_name Your_Wrapper

You can add whichever arguments you like to these files.