Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

show_best tests for censored regression models #156

Merged
merged 10 commits into from
Jan 17, 2024
Merged

show_best tests for censored regression models #156

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
19e963e
initial tests for #118
Dec 18, 2023
64c6f19
return 5 configs
Dec 18, 2023
15081f2
missing %>%
Dec 18, 2023
18c4bce
reduce changes of ties affecting results
Dec 18, 2023
c147241
trying to avoid irreproducible results across runs and OS'es
Dec 18, 2023
30b3916
Apply suggestions from code review
topepo Jan 4, 2024
69043ac
update for latest tune version
Jan 11, 2024
ce40180
Merge branch 'main' into survival-show-best
topepo Jan 16, 2024
6e0e77d
updated snapshots
topepo Jan 16, 2024
4fdbeeb
updated snapshots with new current package versions
topepo Jan 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 183 additions & 0 deletions tests/testthat/_snaps/survival-tune-show-best.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
# show_best with censored data - integrated metric - grid

Code
show_best(grid_int_res)
Condition
Warning in `show_best()`:
No value of `metric` was given; "brier_survival_integrated" will be used.
Output
# A tibble: 5 x 7
cost_complexity .metric .estimator mean n std_err .config
<dbl> <chr> <chr> <dbl> <int> <dbl> <chr>
1 0.000126 brier_survival_integ~ standard 0.00791 10 0.00137 Prepro~
2 0.000251 brier_survival_integ~ standard 0.00793 10 0.00137 Prepro~
3 0.000200 brier_survival_integ~ standard 0.00794 10 0.00136 Prepro~
4 0.000158 brier_survival_integ~ standard 0.00796 10 0.00134 Prepro~
5 0.000316 brier_survival_integ~ standard 0.00797 10 0.00152 Prepro~

# show_best with censored data - dynamic metric - bayes

Code
show_best(bayes_dyn_res)
Condition
Warning in `show_best()`:
No value of `metric` was given; "brier_survival" will be used.
Output
# A tibble: 5 x 9
cost_complexity .metric .estimator .eval_time mean n std_err .config
<dbl> <chr> <chr> <dbl> <dbl> <int> <dbl> <chr>
1 1.26e- 9 brier_surv~ standard 100 0.0114 10 0.00312 Prepro~
2 1.31e- 6 brier_surv~ standard 100 0.0114 10 0.00312 Prepro~
3 3.55e- 8 brier_surv~ standard 100 0.0114 10 0.00312 Iter2
4 1.00e-10 brier_surv~ standard 100 0.0114 10 0.00312 Iter3
5 3.91e- 5 brier_surv~ standard 100 0.0114 10 0.00312 Prepro~
# i 1 more variable: .iter <int>

---

Code
show_best(bayes_dyn_res, metric = "brier_survival", eval_time = 1)
Condition
Error in `show_best()`:
! Evaluation time 1 is not in the results.

---

Code
show_best(bayes_dyn_res, metric = "brier_survival_integrated")
Condition
Error in `show_best()`:
! "brier_survival_integrated" was not in the metric set. Please choose from: "brier_survival".

# show_best with censored data - static metric - anova racing

Code
show_best(race_stc_res)
Condition
Warning in `show_best()`:
No value of `metric` was given; "concordance_survival" will be used.
Output
# A tibble: 1 x 7
cost_complexity .metric .estimator mean n std_err .config
<dbl> <chr> <chr> <dbl> <int> <dbl> <chr>
1 0.00001 concordance_survival standard 0.278 10 0.0147 Preproces~

---

Code
show_best(race_stc_res, metric = "concordance_survival", eval_time = 1)
Condition
Warning in `show_best()`:
An evaluation time is only required when a dynamic metric is selected (and `eval_time` will thus be ignored).
Output
# A tibble: 1 x 7
cost_complexity .metric .estimator mean n std_err .config
<dbl> <chr> <chr> <dbl> <int> <dbl> <chr>
1 0.00001 concordance_survival standard 0.278 10 0.0147 Preproces~

---

Code
show_best(race_stc_res, metric = "brier_survival_integrated")
Condition
Warning:
Metric "concordance_survival" was used to evaluate model candidates in the race but "brier_survival_integrated" has been chosen to rank the candidates. These results may not agree with the race.
Error in `show_best()`:
! "brier_survival_integrated" was not in the metric set. Please choose from: "concordance_survival".
Comment on lines +83 to +86
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just error here instead of warn about the metric and then error anyway. I'll open an issue if you agree.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open the issue on finetune tidymodels/finetune#89 -- this PR is ready to merge!


# show_best with censored data - static metric (+dyn) - W/L racing

Code
show_best(race_stc_res)
Condition
Warning in `show_best()`:
No value of `metric` was given; "concordance_survival" will be used.
Output
# A tibble: 5 x 8
cost_complexity .metric .estimator .eval_time mean n std_err .config
<dbl> <chr> <chr> <dbl> <dbl> <int> <dbl> <chr>
1 0.0706 concordance~ standard NA 0.297 10 0.0118 Prepro~
2 0.0000128 concordance~ standard NA 0.278 10 0.0147 Prepro~
3 0.0000591 concordance~ standard NA 0.275 10 0.0152 Prepro~
4 0.0000959 concordance~ standard NA 0.274 10 0.0149 Prepro~
5 0.000374 concordance~ standard NA 0.256 10 0.0133 Prepro~

---

Code
show_best(race_stc_res, metric = "concordance_survival", eval_time = 1)
Condition
Warning in `show_best()`:
An evaluation time is only required when a dynamic metric is selected (and `eval_time` will thus be ignored).
Output
# A tibble: 5 x 8
cost_complexity .metric .estimator .eval_time mean n std_err .config
<dbl> <chr> <chr> <dbl> <dbl> <int> <dbl> <chr>
1 0.0706 concordance~ standard NA 0.297 10 0.0118 Prepro~
2 0.0000128 concordance~ standard NA 0.278 10 0.0147 Prepro~
3 0.0000591 concordance~ standard NA 0.275 10 0.0152 Prepro~
4 0.0000959 concordance~ standard NA 0.274 10 0.0149 Prepro~
5 0.000374 concordance~ standard NA 0.256 10 0.0133 Prepro~

---

Code
show_best(race_stc_res, metric = "brier_survival_integrated")
Condition
Warning:
Metric "concordance_survival" was used to evaluate model candidates in the race but "brier_survival_integrated" has been chosen to rank the candidates. These results may not agree with the race.
Error in `show_best()`:
! "brier_survival_integrated" was not in the metric set. Please choose from: "concordance_survival" and "brier_survival".

# show_best with censored data - dyn metric (+stc) - W/L racing

Code
show_best(race_dyn_res)
Condition
Warning in `show_best()`:
No value of `metric` was given; "brier_survival" will be used.
Output
# A tibble: 5 x 8
cost_complexity .metric .estimator .eval_time mean n std_err .config
<dbl> <chr> <chr> <dbl> <dbl> <int> <dbl> <chr>
1 0.0000591 brier_surv~ standard 100 0.0110 10 0.00314 Prepro~
2 0.0000959 brier_surv~ standard 100 0.0110 10 0.00317 Prepro~
3 0.0000128 brier_surv~ standard 100 0.0114 10 0.00312 Prepro~
4 0.000374 brier_surv~ standard 100 0.0114 10 0.00340 Prepro~
5 0.000822 brier_surv~ standard 100 0.0124 10 0.00319 Prepro~

---

Code
show_best(race_dyn_res, metric = "concordance_survival")
Condition
Warning:
Metric "brier_survival" was used to evaluate model candidates in the race but "concordance_survival" has been chosen to rank the candidates. These results may not agree with the race.
Output
# A tibble: 5 x 8
cost_complexity .metric .estimator .eval_time mean n std_err .config
<dbl> <chr> <chr> <dbl> <dbl> <int> <dbl> <chr>
1 0.0000128 concordance~ standard NA 0.278 10 0.0147 Prepro~
2 0.0000591 concordance~ standard NA 0.275 10 0.0152 Prepro~
3 0.0000959 concordance~ standard NA 0.274 10 0.0149 Prepro~
4 0.000374 concordance~ standard NA 0.256 10 0.0133 Prepro~
5 0.000822 concordance~ standard NA 0.238 10 0.0182 Prepro~

---

Code
show_best(race_dyn_res, metric = "brier_survival", eval_time = 1)
Condition
Error in `show_best()`:
! Evaluation time 1 is not in the results.

---

Code
show_best(race_dyn_res, metric = "brier_survival_integrated")
Condition
Warning:
Metric "brier_survival" was used to evaluate model candidates in the race but "brier_survival_integrated" has been chosen to rank the candidates. These results may not agree with the race.
Error in `show_best()`:
! "brier_survival_integrated" was not in the metric set. Please choose from: "brier_survival" and "concordance_survival".

30 changes: 15 additions & 15 deletions tests/testthat/_snaps/survival-tune_race_anova.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@
# A tibble: 5 x 8
cost_complexity .metric .estimator .eval_time mean n std_err .config
<dbl> <chr> <chr> <dbl> <dbl> <int> <dbl> <chr>
1 0.0841 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
2 0.0891 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
3 0.0944 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
4 0.1 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
5 0.0794 brier_survi~ standard 1 0.202 30 0.00418 Prepro~
1 0.0000000001 brier_survi~ standard 1 0.177 30 0.00707 Prepro~
2 0.0841 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
3 0.0891 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
4 0.0944 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
5 0.1 brier_survi~ standard 1 0.201 30 0.00415 Prepro~

---

Expand All @@ -65,11 +65,11 @@
# A tibble: 5 x 8
cost_complexity .metric .estimator .eval_time mean n std_err .config
<dbl> <chr> <chr> <dbl> <dbl> <int> <dbl> <chr>
1 0.0841 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
2 0.0891 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
3 0.0944 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
4 0.1 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
5 0.0794 brier_survi~ standard 1 0.202 30 0.00418 Prepro~
1 0.0000000001 brier_survi~ standard 1 0.177 30 0.00707 Prepro~
2 0.0841 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
3 0.0891 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
4 0.0944 brier_survi~ standard 1 0.201 30 0.00415 Prepro~
5 0.1 brier_survi~ standard 1 0.201 30 0.00415 Prepro~

---

Expand All @@ -82,9 +82,9 @@
# A tibble: 5 x 8
cost_complexity .metric .estimator .eval_time mean n std_err .config
<dbl> <chr> <chr> <dbl> <dbl> <int> <dbl> <chr>
1 0.0794 brier_survi~ standard NA 0.338 30 0.00487 Prepro~
2 0.0841 brier_survi~ standard NA 0.338 30 0.00480 Prepro~
3 0.0891 brier_survi~ standard NA 0.338 30 0.00480 Prepro~
4 0.0944 brier_survi~ standard NA 0.338 30 0.00480 Prepro~
5 0.1 brier_survi~ standard NA 0.338 30 0.00480 Prepro~
1 0.0000000001 brier_survi~ standard NA 0.285 30 0.00426 Prepro~
2 0.0794 brier_survi~ standard NA 0.338 30 0.00487 Prepro~
3 0.0841 brier_survi~ standard NA 0.338 30 0.00480 Prepro~
4 0.0891 brier_survi~ standard NA 0.338 30 0.00480 Prepro~
5 0.0944 brier_survi~ standard NA 0.338 30 0.00480 Prepro~

Binary file modified tests/testthat/_snaps/survival-tune_race_anova/aov-race-plot.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified tests/testthat/_snaps/survival-tune_race_anova/dyn-aov-race-plot.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified tests/testthat/_snaps/survival-tune_race_anova/int-aov-race-plot.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/testthat/_snaps/survival-tune_race_anova/int-aov-racing.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
33 changes: 33 additions & 0 deletions tests/testthat/helper-cens-churn.R
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@

make_churn_cens_objects <- function(x) {
suppressPackageStartupMessages(require("tidymodels"))
suppressPackageStartupMessages(require("censored"))

data("mlc_churn")

mlc_churn <-
mlc_churn %>%
mutate(
churned = ifelse(churn == "yes", 1, 0),
event_time = survival::Surv(account_length, churned)
) %>%
select(event_time, account_length, area_code, total_eve_calls)

set.seed(6941)
churn_split <- initial_split(mlc_churn)
churn_tr <- training(churn_split)
churn_te <- testing(churn_split)
churn_rs <- vfold_cv(churn_tr)

eval_times <- c(50, 100, 150)

churn_rec <-
recipe(event_time ~ ., data = churn_tr) %>%
step_dummy(area_code) %>%
step_normalize(all_predictors())

list(split = churn_split, train = churn_tr, test = churn_te,
rs = churn_rs, times = eval_times, rec = churn_rec)

}

2 changes: 1 addition & 1 deletion tests/testthat/test-parsnip-survival-censoring-weights.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ test_that("error messages in context of .censoring_weights_graf()", {

test_that("error for .censoring_weights_graf.workflow()", {
# temporarily its own test, see above
skip_if_not_installed("workflows", minimum_version = "1.1.3.9001")
skip_if_not_installed("workflows", minimum_version = "1.1.3.9007")
expect_snapshot(error = TRUE, .censoring_weights_graf(workflows::workflow()))
})

Expand Down
Loading
Loading