OOB error

[datalorax]

Feature

Calculate out of bag error and use it for model performance estimates and hyper parameter tuning.

I did a fair amount of sleuthing on this and it looks like this used to be a feature that could be requested through control_bag(), but was removed in 9fae03c because of something related to C5.0.

I'm wondering if we can get this back for rpart models?

Extending this further, I wonder if it might be possible to use tune_grid() with a bagged model, using the OOB samples as the validation set, to tune hyperparameters. I also thought it might be worth thinking about a new function called something like fit_bagged() that would basically operate exactly like fit_resamples(), but would provide the metrics on the OOB samples.

[hardin47]

For anyone who finds this and wants a workaround, the randomForest engine (not as easy with ranger) allows you to pull the OOB models from the tidy fit. A full reproducible example is here.

[topepo]

We'll look back into this (it was a while back that I pulled that out) and maybe put it on the upcoming user survey.