condo train test evaluation - #156
Conversation
|
@wagnerlmichael can you try running this locally? just to see that workflow replicates in your environment. |
The code seems to work locally but errors on report generation, I have code in in the subset PR that fixes that. I also didn't test the upload stage since I don't have perms to upload locally. But the main logic changes in the new stat generation run without error |
|
I'll let you guys sort out the merge conflicts and then I'll take a look at this. |
…ct typos in misc/files_dict
@wrridgeway - merge conflicts should be resolved- we await your review :) |
wrridgeway
left a comment
wrridgeway
left a comment
There was a problem hiding this comment.
Great work here guys. Maybe I'm wrong, but it seems like we've been forgetting to track some of the test set performance with dvc. If that is the case, we might want to make sure to make sure we're not making the same mistake in the res avm as well.
| - output/performance_quantile/model_performance_quantile_train.parquet: | ||
| cache: false | ||
| - output/performance_quantile/model_performance_quantile_train_linear.parquet: | ||
| cache: false |
There was a problem hiding this comment.
Seems like we're missing test_linear and quantile_test_linear here... we should probably add them, yeah?
There was a problem hiding this comment.
Thanks! Just an oversight on my part - can add
Co-authored-by: William Ridgeway <10358980+wrridgeway@users.noreply.github.com>
3 participants
Adds evaluation statistics (RMSE, mDape, etc.) to the training set. These are the same performance stats that are currently calculated on the test set. (This also mirrors the new implementation in the residential model). The goal is to have more visible train/test metrics, so that we can better evaluate model overfitting.
More detail can be found in the linked issue (#146):
In addition to running the code, please validate by ensuring that there is now a new card (train_card) for your condo run, and that new stages, for train are added to the performance database). Check that the training data accounts for roughly 90% of the overall (train+test) data. Pick a geo or two at random and compare their train test stats on a metric of your choosing (e.g. RMSE - generally the train measure should be superior - although sometimes it is not). (See here for some examples: #146 (comment))
(general scaffold of the code is here- #146 (comment))