80 hyperparameter

Introduction

Currently, the most widely used hyperparameter optimization technique is grid search. Recently, random search is proposed as an easy method which could obtain interesting good results (competitive with grid search, even better in some tasks) 2012 Bergstra and Bengio.

Random search hyperparameter optimization

In APRIL-ANN it is possible to do random search hyperparameter optimization using the script located at:

tools/trainable/random-search-hyperparemeter-optimization.lua

This scripts receives a configuration Lua file like this:

return {
  hyperparams = {
    { option="--o1=",  value=10,  tag="o1", sampling="fixed", hidden=true },
    { option="--o2=",  value=20,  tag="o2", sampling="fixed" },
    { option="--r1",   tag="r1", sampling = "log-uniform",
      type="real", prec=3,
      values= { { min=0.001, max=1 }, },
      filter = function(hyperparams) hyperparams["r1"] = "200" return true end },
    { option="--r2=", tag="r2", sampling = "uniform", values= { 1, 2, 3, 4, 5 } },
    { option="--r3=", tag="r3", prec=3,
      sampling = "gaussian", values= { mean=0, variance=0.1 } },
    { option="--r4=", tag="r4", sampling = "random",
      filter = function(hyperparams)
	if hyperparams["r2"] == "1" then hyperparams["r4"] = "0" end return true
      end },
    { option=nil, tag="r5", sampling="random" }
  },
  filter = function(hyperparams) hyperparams['r5'] = '0.4' return true end,
  script = "",
  exec   = "echo",
  working_dir = "/tmp/",
  -- seed = ANY_SEED_VALUE (if not given, take random from bash)
  n = 50 }

The configuration file returns a Lua table which contain some prior knowledge about each hyperparameter (a fully random optimization is unreliable). The table has this major fields:

• hyperparams: a table which describes the prior knowledge of each random searched hyperparameter (note that some of them could be 'fixed' instead of random). Each random hyperparemter is identified by a tag string, a unique option and fields which describe different prior distributions of hyperparameters. The sampling="fixed|uniform|log-uniform|gaussian|random" field indicates if the sampling distribution will be fixed (always the same value), uniform, log-uniform, gaussian, or totally random (this last one is not contrained). The fixed distribution needs a value=SOMETHING field which contains the value of this hyperparameter. The uniform distribution needs a values field which contains a table of values (values={1, 4, 8}) or an array of tables with min/max/step constrains (values={ {min=0, max=10, step=2}, {min=20, max=30, step=4} }). The log-uniform distribution needs a table with min/max constrains (not step). The field type="real|integer" is only useful for min/max/step values. The field prec=3 indicates the number of precission digits needed. All of them could be hidden=true, indicating that this hyperparameter won't be at the output filename string, but yes at the arguments list. Besides, the 'option' field could be option=nil, indicating that this hyperparameter is a metaparameter which won't be passed as argument to the script, but yes to the filter functions of each hyperparameter and the global filter function. The filter field is a function which returns true or false indicating if this set of hyperparameters is valid, and receives a table indexed by TAGs which contains all top hyperparameter values (it is possible to modify any value at this table).

• filter: is a function which received a dictionary table which associates each tag with its value (a string in all cases, even for integer or real numbers). This function is called just before run an experiment. It checks the validity of hyperparameters returning true, otherwise, the experiment won't be executed, and it modifies any hyperparameter value. NOTE that is recommended to write filter functions which use 'string' type for their modified hyperparameters.

• exec: the executable file, normally an april-ann binary file, but others are possible.

• script: it will be an script given as first argument of the executable.

• working_dir: where to store stdout of each experiment. Each experiment is stored at a filename "WORKING_DIR/output-TAG1:VALUE1_TAG2:VALUE2_TAG3:VALUE3_..._TAGM:VALUEM.log". Hyperparamteres marked as hidden=true won't be used to form this filename.

• seed: an optional random number generator seed.

• n: the number of experiments which will be executed.

The random search executes a script which receives non positional command line options. The option field indicates the string which will be concatenated as prefix of the value. So, if the script needs an option like this: --blah=NUMBER, the field may be: option="--blah=".

An option field could be nil, indicating that this hyperparameter is not used at the script, but it is needed in filter functions.

WARNING!!! variable params of each filter funtion always has string type, in order to ensure that the required number of precission digits is correct. So, you need to use tonumber(hyperparam[TAG]) in order to compare two numbers, and also is recommended to modify hyperaparams using string type values.

Command line execution

The execution of the procedure follows this syntax:

april-ann tools/trainable/random-search-hyperparemeter-optimization.lua configuration-script.lua [ARGS]

where ARGS follows this syntax:

ARGS : ARGS [ "all_hyperparams.TAG.value='blah'" | "global_vars.working_dir='blah'" | "global_vars.n=blah" ... ]

where all_hyperparams is a Lua table (associates tag names with hyperparmeter fields) which contains the fixed and randomized parameters of configuration-script.lua, so it is possible to modify any parameter field (option, value/s, sampling, prec, tag, values.min, values.max, ...) from the command line, and global_vars is a Lua table which contains the rest parameters of configuration-script.lua (seed, n, exec, script, working_dir, filter). All this command line arguments must be valid Lua code.