Skip to content

Commit ee9bbef

Browse files
TimCookCountyDSTimothy SparerwrridgewayTim Sparer
authored
condo train test evaluation (#156)
* Add train_card row to file_dict.csv * Save training data as well as test in 01-train.R to new train_card * update dvc.yaml through evaluate * update dvs.yaml uploads * dvc.yaml fully updated * Add training data evaluations to the 03-evaluate.R pipeline * Fix headings * Add upload code for training set to 06-upload.R pipeline * Fix indentation * Lint * Correct line length * fix line length part 5 * Update pipeline/06-upload.R Co-authored-by: William Ridgeway <10358980+wrridgeway@users.noreply.github.com> --------- Co-authored-by: Timothy Sparer <tispare@cookcountyassessor.com> Co-authored-by: Sweaty Handshake <william.ridgeway@cookcountyil.gov> Co-authored-by: Tim Sparer <tispare@ccao-datals.ccao.local> Co-authored-by: William Ridgeway <10358980+wrridgeway@users.noreply.github.com>
1 parent b18d4a0 commit ee9bbef

6 files changed

Lines changed: 169 additions & 33 deletions

File tree

‎dvc.yaml‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ stages:
4444
cache: false
4545
- output/test_card/model_test_card.parquet:
4646
cache: false
47+
- output/train_card/model_train_card.parquet:
48+
cache: false
4749
- output/workflow/fit/model_workflow_fit.zip:
4850
cache: false
4951
- output/workflow/recipe/model_workflow_recipe.rds:
@@ -78,14 +80,16 @@ stages:
7880
evaluate:
7981
cmd: Rscript pipeline/03-evaluate.R
8082
desc: >
81-
Evaluate the model's performance using two methods:
83+
Evaluate the model's performance using three methods:
8284
1. The standard test set, in this case the most recent 10% of sales
83-
2. An assessor-specific ratio study comparing estimated assessments to
85+
2. The training set, and test set, used to detect overfitting.
86+
3. An assessor-specific ratio study comparing estimated assessments to
8487
the previous year's sales
8588
deps:
8689
- pipeline/03-evaluate.R
8790
- output/assessment_pin/model_assessment_pin.parquet
8891
- output/test_card/model_test_card.parquet
92+
- output/train_card/model_train_card.parquet
8993
params:
9094
- assessment
9195
- ratio_study
@@ -98,6 +102,14 @@ stages:
98102
cache: false
99103
- output/performance_quantile/model_performance_quantile_assessment.parquet:
100104
cache: false
105+
- output/performance/model_performance_train.parquet:
106+
cache: false
107+
- output/performance/model_performance_train_linear.parquet:
108+
cache: false
109+
- output/performance_quantile/model_performance_quantile_train.parquet:
110+
cache: false
111+
- output/performance_quantile/model_performance_quantile_train_linear.parquet:
112+
cache: false
101113
- output/intermediate/timing/model_timing_evaluate.parquet:
102114
cache: false
103115

@@ -168,12 +180,17 @@ stages:
168180
- output/workflow/fit/model_workflow_fit.zip
169181
- output/workflow/recipe/model_workflow_recipe.rds
170182
- output/test_card/model_test_card.parquet
183+
- output/train_card/model_train_card.parquet
171184
- output/assessment_card/model_assessment_card.parquet
172185
- output/assessment_pin/model_assessment_pin.parquet
173186
- output/performance/model_performance_test.parquet
174187
- output/performance_quantile/model_performance_quantile_test.parquet
175188
- output/performance/model_performance_assessment.parquet
176189
- output/performance_quantile/model_performance_quantile_assessment.parquet
190+
- output/performance/model_performance_train.parquet
191+
- output/performance/model_performance_train_linear.parquet
192+
- output/performance_quantile/model_performance_quantile_train.parquet
193+
- output/performance_quantile/model_performance_quantile_train_linear.parquet
177194
- output/shap/model_shap.parquet
178195
- output/feature_importance/model_feature_importance.parquet
179196
- output/metadata/model_metadata.parquet

‎misc/file_dict.csv‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,16 +12,21 @@ output,parameter_raw,1,train,ccao-model-results-us-east-1,output/parameter_searc
1212
output,parameter_final,1,train,ccao-model-results-us-east-1,output/parameter_final/model_parameter_final.parquet,parameter_final/year={year}/{run_id}.parquet,parameter_final,model run,"year, run_id",Chosen set of hyperparameters for each run,"As chosen by tune::select_best() if using CV, otherwise the default set of hyperparameters specified in params.yaml (model.hyperparameter.default)"
1313
output,parameter_range,1,train,ccao-model-results-us-east-1,output/parameter_range/model_parameter_range.parquet,parameter_range/year={year}/{run_id}.parquet,parameter_range,parameter,"year, run_id, parameter_name",Range of hyperparameters searched during CV tuning,As specified in params.yaml (model.hyperparameter.range)
1414
output,test_card,1,train,ccao-model-results-us-east-1,output/test_card/model_test_card.parquet,test_card/year={year}/{run_id}.parquet,test_card,card,"year, meta_pin, meta_card_num",Test set predictions at the card level,Only includes the minimal variables necessary to perform evaluation
15+
output,train_card,1,train,ccao-model-results-us-east-1,output/train_card/model_train_card.parquet,train_card/year={year}/{run_id}.parquet,train_card,card,"year, meta_pin, meta_card_num",Training set predictions at the card level,Only includes the minimal variables necessary to perform evaluation
1516
output,workflow_fit,1,train,ccao-model-results-us-east-1,output/workflow/fit/model_workflow_fit.zip,workflow/fit/year={year}/{run_id}.zip,,model run,,Trained LightGBM model object + Tidymodels specification,Can be loaded with lightsnip::lgbm_load() to produce predictions using new data
1617
output,workflow_recipe,1,train,ccao-model-results-us-east-1,output/workflow/recipe/model_workflow_recipe.rds,workflow/recipe/year={year}/{run_id}.rds,,model run,,Trained Tidymodels recipe object used for data preprocessing,Can be used to prepare new data in the same way as the original model training
1718
output,assessment_card,2,assess,ccao-model-results-us-east-1,output/assessment_card/model_assessment_card.parquet,assessment_card/,assessment_card,card,"year, run_id, township_code, meta_pin, meta_card_num",Assessment results at the card level AKA raw model output,Also includes card-level characteristics. Multi-card PINs will have more than one row. NOTE: Each run adds new partitions to S3 which must be added via a Glue crawler
1819
output,assessment_pin,2,assess,ccao-model-results-us-east-1,output/assessment_pin/model_assessment_pin.parquet,assessment_pin/,assessment_pin,pin,"year, run_id, township_code, meta_pin",Assessment results at the PIN level AKA aggregated and cleaned,"Aggregation depends on PIN specifics, see assess script for details. Includes PIN-level stats like YoY % changes, land, sales, etc. NOTE: Each run adds new partitions to S3 which must be added via a Glue crawler"
1920
output,performance_test,3,evaluate,ccao-model-results-us-east-1,output/performance/model_performance_test.parquet,performance/year={year}/stage=test/{run_id}.parquet,performance,geography [by class],"year, run_id, stage, geography_type, geography_id, by_class, class",Performance metrics (optionally) broken out by class for different levels of geography,Test set includes the most recent 10% of sales
2021
output,performance_test_linear,3,evaluate,ccao-model-results-us-east-1,output/performance/model_performance_test_linear.parquet,performance/year={year}/stage=test_linear/{run_id}.parquet,performance,geography [by class],"year, run_id, stage, geography_type, geography_id, by_class, class",Performance metrics (optionally) broken out by class for different levels of geography,Test set performance stats using a linear baseline model
2122
output,performance_assessment,3,evaluate,ccao-model-results-us-east-1,output/performance/model_performance_assessment.parquet,performance/year={year}/stage=assessment/{run_id}.parquet,performance,geography [by class],"year, run_id, stage, geography_type, geography_id, by_class, class",Performance metrics (optionally) broken out by class for different levels of geography,Assessment set uses the prior year sales to compare to the assessed value
23+
output,performance_train,3,evaluate,ccao-model-results-us-east-1,output/performance/model_performance_train.parquet,performance/year={year}/stage=train/{run_id}.parquet,performance,geography [by class],"year, run_id, stage, geography_type, geography_id, by_class, class",Performance metrics (optionally) broken out by class for different levels of geography,Training set includes the oldest 90% of sales
24+
output,performance_train_linear,3,evaluate,ccao-model-results-us-east-1,output/performance/model_performance_train_linear.parquet,performance/year={year}/stage=train_linear/{run_id}.parquet,performance,geography [by class],"year, run_id, stage, geography_type, geography_id, by_class, class",Performance metrics (optionally) broken out by class for different levels of geography,Training set performance stats using a linear baseline model
2225
output,performance_quantile_test,3,evaluate,ccao-model-results-us-east-1,output/performance_quantile/model_performance_quantile_test.parquet,performance_quantile/year={year}/stage=test/{run_id}.parquet,performance_quantile,geography [by class] by quantile,"year, run_id, stage, geography_type, geography_id, by_class, class, quantile",Performance metrics by quantile within class and geography,Test set includes the most recent 10% of sales
2326
output,performance_quantile_test_linear,3,evaluate,ccao-model-results-us-east-1,output/performance_quantile/model_performance_quantile_test_linear.parquet,performance_quantile/year={year}/stage=test_linear/{run_id}.parquet,performance_quantile,geography [by class] by quantile,"year, run_id, stage, geography_type, geography_id, by_class, class, quantile",Performance metrics by quantile within class and geography,Test set performance stats using a linear baseline model
2427
output,performance_quantile_assessment,3,evaluate,ccao-model-results-us-east-1,output/performance_quantile/model_performance_quantile_assessment.parquet,performance_quantile/year={year}/stage=assessment/{run_id}.parquet,performance_quantile,geography [by class] by quantile,"year, run_id, stage, geography_type, geography_id, by_class, class, quantile",Performance metrics by quantile within class and geography,Assessment set uses the prior year sales to compare to the assessed value
28+
output,performance_quantile_train,3,evaluate,ccao-model-results-us-east-1,output/performance_quantile/model_performance_quantile_train.parquet,performance_quantile/year={year}/stage=train/{run_id}.parquet,performance_quantile,geography [by class] by quantile,"year, run_id, stage, geography_type, geography_id, by_class, class, quantile",Performance metrics by quantile within class and geography,Training set includes the oldest 90% of sales
29+
output,performance_quantile_train_linear,3,evaluate,ccao-model-results-us-east-1,output/performance_quantile/model_performance_quantile_train_linear.parquet,performance_quantile/year={year}/stage=train_linear/{run_id}.parquet,performance_quantile,geography [by class] by quantile,"year, run_id, stage, geography_type, geography_id, by_class, class, quantile",Performance metrics by quantile within class and geography,Training set performance stats using a linear baseline model
2530
output,shap,4,interpret,ccao-model-results-us-east-1,output/shap/model_shap.parquet,shap/,shap,card,"year, run_id, township_code, meta_pin, meta_card_num",SHAP values for each feature for each card in the assessment data,NOTE: Each run adds new partitions to S3 which must be added via a Glue crawler
2631
output,feature_importance,4,interpret,ccao-model-results-us-east-1,output/feature_importance/model_feature_importance.parquet,feature_importance/year={year}/{run_id}.parquet,feature_importance,predictor,"year, run_id, model_predictor_all_name","Feature importance values (gain, cover, and frequency) for the run",
2732
output,report_performance,5,finalize,ccao-model-results-us-east-1,reports/performance/performance.html,report/year={year}/report_type=performance/{run_id}.html,,model run,,Rendered Quarto doc with model performance statistics,

‎output/train_card/.gitkeep‎

Whitespace-only changes.

‎pipeline/01-train.R‎

Lines changed: 40 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -381,39 +381,49 @@ lgbm_wflow_final_full_fit <- lgbm_wflow %>%
381381

382382

383383

384-
#- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
385-
# 5. Finalize Models -----------------------------------------------------------
386-
#- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
384+
#- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
385+
# 5. Finalize Models ---------------------------------------------------------
386+
#- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
387387
message("Finalizing and saving trained model")
388388

389-
# Get predictions on the test set using the training data model. These
390-
# predictions are used to evaluate model performance on the unseen test set.
389+
# Get predictions on both the test and training set using the
390+
# training data model.
391+
# These predictions are used to evaluate model
392+
# performance on the unseen test set, and to evaluate overfitting.
391393
# Keep only the variables necessary for evaluation
392-
test %>%
393-
mutate(
394-
pred_card_initial_fmv = predict(lgbm_wflow_final_fit, test)$.pred,
395-
pred_card_initial_fmv_lin = exp(predict(
396-
lin_wflow_final_fit,
397-
test %>% mutate(meta_sale_price = log(meta_sale_price))
398-
)$.pred)
399-
) %>%
400-
select(
401-
meta_year, meta_pin, meta_class, meta_card_num, meta_triad_code,
402-
all_of(params$ratio_study$geographies), char_building_sf, char_unit_sf,
403-
all_of(c(
404-
"prior_far_tot" = params$ratio_study$far_column,
405-
"prior_near_tot" = params$ratio_study$near_column
406-
)),
407-
pred_card_initial_fmv, pred_card_initial_fmv_lin,
408-
meta_sale_price, meta_sale_date, meta_sale_document_num, meta_modeling_group
409-
) %>%
410-
# Prior year values are AV, not FMV. Multiply by 10 to get FMV for residential
411-
mutate(
412-
prior_far_tot = prior_far_tot * 10,
413-
prior_near_tot = prior_near_tot * 10
414-
) %>%
415-
as_tibble() %>%
416-
write_parquet(paths$output$test_card$local)
394+
walk2(
395+
list(test, train),
396+
list(paths$output$test_card$local, paths$output$train_card$local),
397+
\(data, path) {
398+
data %>%
399+
mutate(
400+
pred_card_initial_fmv = predict(lgbm_wflow_final_fit, data)$.pred,
401+
pred_card_initial_fmv_lin = exp(predict(
402+
lin_wflow_final_fit,
403+
data %>% mutate(meta_sale_price = log(meta_sale_price))
404+
)$.pred)
405+
) %>%
406+
select(
407+
meta_year, meta_pin, meta_class, meta_card_num, meta_triad_code,
408+
all_of(params$ratio_study$geographies), char_building_sf, char_unit_sf,
409+
all_of(c(
410+
"prior_far_tot" = params$ratio_study$far_column,
411+
"prior_near_tot" = params$ratio_study$near_column
412+
)),
413+
pred_card_initial_fmv, pred_card_initial_fmv_lin,
414+
meta_sale_price, meta_sale_date, meta_sale_document_num,
415+
meta_modeling_group
416+
) %>%
417+
# Prior year values are AV, not FMV. Multiply by 10 to
418+
# get FMV for residential
419+
mutate(
420+
prior_far_tot = prior_far_tot * 10,
421+
prior_near_tot = prior_near_tot * 10
422+
) %>%
423+
as_tibble() %>%
424+
write_parquet(path)
425+
}
426+
)
417427

418428
# Save the finalized model object to file so it can be used elsewhere. Note the
419429
# lgbm_save() function, which uses lgb.save() rather than saveRDS(), since

‎pipeline/03-evaluate.R‎

Lines changed: 77 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,10 @@ message("Loading evaluation data")
4949
test_data_card <- read_parquet(paths$output$test_card$local) %>%
5050
filter(meta_modeling_group == "CONDO")
5151

52+
# Load train card: 90% of earliest sales - used to test for overfitting
53+
train_data_card <- read_parquet(paths$output$train_card$local) %>%
54+
filter(meta_modeling_group == "CONDO")
55+
5256
# Load the assessment results from the previous stage. This will include every
5357
# residential PIN that needs a value.
5458
assessment_data_pin <- read_parquet(paths$output$assessment_pin$local) %>%
@@ -430,8 +434,80 @@ pwalk(
430434
}
431435
)
432436

437+
## 4.2. Training Set ---------------------------------------------------------
438+
# Same as test set, but using training data to enable train vs. test
439+
# performance comparison and overfitting detection
440+
message("Calculating training set aggregate statistics")
441+
pwalk(
442+
list(
443+
rlang::quos(pred_card_initial_fmv, pred_card_initial_fmv_lin),
444+
list(
445+
paths$output$performance_train$local,
446+
paths$output$performance_train_linear$local
447+
)
448+
),
449+
function(pred, path) {
450+
future_pmap(
451+
geographies_list,
452+
function(geo, cls) {
453+
gen_agg_stats(
454+
data = train_data_card,
455+
truth = meta_sale_price,
456+
estimate = !!pred,
457+
bldg_sqft = char_unit_sf,
458+
rsn_col = prior_near_tot,
459+
rsf_col = prior_far_tot,
460+
triad = meta_triad_code,
461+
geography = !!geo,
462+
class = !!cls,
463+
col_dict = col_rename_dict,
464+
min_n = params$ratio_study$min_n_sales
465+
)
466+
},
467+
.options = furrr_options(seed = TRUE, stdout = FALSE),
468+
.progress = FALSE
469+
) %>%
470+
purrr::list_rbind() %>%
471+
write_parquet(path)
472+
}
473+
)
474+
475+
message("Calculating training set quantile statistics")
476+
pwalk(
477+
list(
478+
rlang::quos(pred_card_initial_fmv, pred_card_initial_fmv_lin),
479+
list(
480+
paths$output$performance_quantile_train$local,
481+
paths$output$performance_quantile_train_linear$local
482+
)
483+
),
484+
function(pred, path) {
485+
future_pmap(
486+
geographies_list_quantile,
487+
function(geo, cls, qnt) {
488+
gen_agg_stats_quantile(
489+
data = train_data_card,
490+
truth = meta_sale_price,
491+
estimate = !!pred,
492+
rsn_col = prior_near_tot,
493+
rsf_col = prior_far_tot,
494+
triad = meta_triad_code,
495+
geography = !!geo,
496+
class = !!cls,
497+
col_dict = col_rename_dict,
498+
num_quantile = qnt
499+
)
500+
},
501+
.options = furrr_options(seed = TRUE, stdout = FALSE),
502+
.progress = FALSE
503+
) %>%
504+
purrr::list_rbind() %>%
505+
write_parquet(path)
506+
}
507+
)
508+
433509

434-
## 4.2. Assessment Set ---------------------------------------------------------
510+
## 4.3. Assessment Set ---------------------------------------------------------
435511

436512
# Do the same thing for the assessment set. This will have accurate property
437513
# counts and proportions, since it also includes unsold properties

‎pipeline/06-upload.R‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,13 @@ if (upload_enable) {
6969
relocate(run_id) %>%
7070
write_parquet(paths$output$test_card$s3)
7171

72+
# Upload the training set predictions
73+
read_parquet(paths$output$train_card$local) %>%
74+
mutate(run_id = !!run_id) %>%
75+
relocate(run_id) %>%
76+
write_parquet(paths$output$train_card$s3)
77+
78+
7279
# Upload the parameter search objects if CV was enabled. Requires some
7380
# cleaning since the Tidymodels output is stored as a nested data frame
7481
if (cv_enable) {
@@ -167,6 +174,27 @@ if (upload_enable) {
167174
relocate(run_id) %>%
168175
write_parquet(paths$output$performance_quantile_test_linear$s3)
169176

177+
# Upload training set performance
178+
message("Uploading training set evaluation")
179+
read_parquet(paths$output$performance_train$local) %>%
180+
mutate(run_id = !!run_id) %>%
181+
relocate(run_id) %>%
182+
write_parquet(paths$output$performance_train$s3)
183+
read_parquet(paths$output$performance_quantile_train$local) %>%
184+
mutate(run_id = !!run_id) %>%
185+
relocate(run_id) %>%
186+
write_parquet(paths$output$performance_quantile_train$s3)
187+
188+
message("Uploading training linear baseline")
189+
read_parquet(paths$output$performance_train_linear$local) %>%
190+
mutate(run_id = !!run_id) %>%
191+
relocate(run_id) %>%
192+
write_parquet(paths$output$performance_train_linear$s3)
193+
read_parquet(paths$output$performance_quantile_train_linear$local) %>%
194+
mutate(run_id = !!run_id) %>%
195+
relocate(run_id) %>%
196+
write_parquet(paths$output$performance_quantile_train_linear$s3)
197+
170198
# Upload assessment set performance
171199
message("Uploading assessment set evaluation")
172200
read_parquet(paths$output$performance_assessment$local) %>%

0 commit comments

Comments
 (0)