AutoML 2.0: regression throws The given key 'e4' was not present in the dictionary.

[torronen]

System Information (please complete the following information):

• OS & Version: windows 11
• ML.NET Version: ML.NET nightly 4th July 2023
• .NET Version: .NET 7.0

Describe the bug

When running regression experiment I got this error. It seems to appear at "random" experiment. I have got it on 2nd, 4th and now not yet for 21 trials.

System.Collections.Generic.KeyNotFoundException: The given key 'e4' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.ML.SearchSpace.SearchSpace.MappingToFeatureSpace(Parameter parameter)
   at Microsoft.ML.SearchSpace.SearchSpace.MappingToFeatureSpace(Parameter parameter)
   at Microsoft.ML.AutoML.CsvTrialResultManager.Save()
   at Microsoft.ML.AutoML.AutoMLExperiment.RunAsync(CancellationToken ct)
   at Kwork.MLTrainer2023Regression.Program.TrainRegression(List`1 trainingDataFiles, String LabelColumn, String saveTo, UInt32 trainingTimeSeconds) in C:\git\Kwork.MLTrainer2023\Kwork.MLTrainer2023Regression\Program.cs:line 422
   at Kwork.MLTrainer2023Regression.Program.Main(String[] args) in C:\git\Kwork.MLTrainer2023\Kwork.MLTrainer2023Regression\Program.cs:line 190
The given key 'e4' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at Microsoft.ML.SearchSpace.SearchSpace.MappingToFeatureSpace(Parameter parameter)
   at Microsoft.ML.SearchSpace.SearchSpace.MappingToFeatureSpace(Parameter parameter)
   at Microsoft.ML.AutoML.CsvTrialResultManager.Save()
   at Microsoft.ML.AutoML.AutoMLExperiment.RunAsync(CancellationToken ct)
   at Kwork.MLTrainer2023Regression.Program.TrainRegression(List`1 trainingDataFiles, String LabelColumn, String saveTo, UInt32 trainingTimeSeconds) in C:\git\Kwork.MLTrainer2023\Kwork.MLTrainer2023Regression\Program.cs:line 422
   at Kwork.MLTrainer2023Regression.Program.Main(String[] args) in C:\git\Kwork.MLTrainer2023\Kwork.MLTrainer2023Regression\

my code:

                pipeline = ctx.Transforms
               .SelectColumns(columnsToKeep) // Keep only the specified columns
               .Append(ctx.Auto().Featurizer(data, columnInformation: columnInference.ColumnInformation))
               .Append(ctx.Auto().Regression(labelColumnName: columnInference.ColumnInformation.LabelColumnName,
                useFastForest: false, useSdca: false, useLbfgsPoissonRegression: false, useLgbm: false,
               useFastTree: true
               )); //disable slow FastForest, run LightGBM on GPU machine

  ```

AutoMLExperiment.AutoMLExperimentSettings settings = new AutoMLExperiment.AutoMLExperimentSettings();
AutoMLExperiment experiment = ctx.Auto().CreateExperiment();

        // Configure experiment
        experiment
            .SetPipeline(pipeline)
            .SetRegressionMetric(RegressionMetric.RootMeanSquaredError, labelColumn: columnInference.ColumnInformation.LabelColumnName)
            .SetTrainingTimeInSeconds(trainingTimeSeconds)
            .SetDataset(data, fold: NumFolds, samplingKeyColumnName: SamplingKeyColumn)
            .SetEciCostFrugalTuner();

        var monitor = new AutoMLMonitor(pipeline);
        experiment.SetMonitor(monitor);

        var cts = new CancellationTokenSource();
        Console.WriteLine("Start experiment");
        TrialResult experimentResults = await experiment.RunAsync(cts.Token);

[LittleLittleCloud]

@torronen Did you call SetCheckPoint somewhere

[torronen]

@LittleLittleCloud
yes, I do set it. It may get reused on the later runs on different machines, because it is based on the training data file and label.

checkpointPath = @"X:\ai-checkpoints\models2023f\" + Path.GetFileNameWithoutExtension(trainingDataFiles[0]) + "-" + LabelColumn;
Directory.CreateDirectory(checkpointPath);
experiment.SetCheckpoint(checkpointPath);

[torronen]

@LittleLittleCloud Do you have ideas on how to possible debug the issue?

[LittleLittleCloud]

@torronen Sorry for the late reply. Could it be because you are training the model with different datasets?

The checkpoint is bound to the search space of a pipeline. This exception could be thrown when AutoMLExperiment tries to load/save the checkpoint file produced by a different dataset.

Could you share the existing checkpoints file under checkpointPath

[torronen]

@LittleLittleCloud I do not think so, at least not anymore. Originally it is possible I used different labels (and dropped other labels) but used the same checkpoint folder. Just in case I updated to add timestamp checkpoint folder to prevent it, no luck yet, below the code.

If you believe the problem is with checkpoint folder I will do some more testing. I am now trying without setting the checkpoint folder. Should I try run with full source and identify more, or is the above error enough?

checkpointPath = @"X:\ai-checkpoints\models2023f\" + Path.GetFileNameWithoutExtension(trainingDataFiles[0]) + "-" + LabelColumn + "-" + DateTime.UtcNow.ToString("ddMMyyyy-HHmm") + "\\";
Directory.CreateDirectory(checkpointPath);
experiment.SetCheckpoint(checkpointPath)

Now the checkpoint folder only includes this file, no hidden files etc. Only 1 trial should have completed.
trialResults-1710668321-Accuracy.csv

X: is a network drive (SAMBA share).

[torronen]

@LittleLittleCloud Yes, that seems to fix the issue, thanks. I will make a unique folder for every run. I am still quite sure I did not mix other datasets or other lables on the checkpoint folder because it is named based on the dataset and label. Is there other difference, I do not know.

For now I can run as unique runs, but it would be convenient to be able to give a quota for a dataset, let's say 12 hours. The later, if there are training clients available, they could pick from there and continue for a few days until new tasks arrive. Then again few hours after the tasks complete and so on.

Most optimal would be if checkpoint folder would allow checkpoints from multiple training agents at the same time.

In the coming months when I have time I may try to debug this issue, maybe add some information about the dataset together with the checkpoint. If you have any tips or advice, highly appreciated.

[LittleLittleCloud]

@torronen
I think I might found the bug. It's because Eci-cfo processes search space in a different way by splitting a nested search spaces into small pieces and therefore modify the schema. I'll work on on fix.

In the meantime, you should be able to use other tuners like cfo or smac, which doesn't modify search space schema for its trials.

[torronen]

Thanks, I'll do that! If I use one tuner would I be able to change the tuner later but use the same checkpoint? I suppose not?