add build and run model workflow option to allow batch job to push to dvc (#181)

* test changes

* Update file_dict.csv

* Update pipeline/06-upload.R

Co-authored-by: Jean Cochrane <jeancochrane@users.noreply.github.com>

---------

Co-authored-by: Jean Cochrane <jeancochrane@users.noreply.github.com>

Author: Damonamajor

.github/workflows/build-and-run-model.yaml

@@ -68,7 +68,9 @@ jobs:
         id: set
         shell: bash
         run: |
-          if [[ "${{ inputs.repro_ingest }}" == "true" ]]; then
+          if [[ "${{ inputs.repro_ingest }}" == "true" && "${{ inputs.upload_enable }}" == "true" ]]; then
+            echo "command=dvc unfreeze ingest && dvc repro ingest && dvc push && dvc freeze ingest && dvc repro" >> $GITHUB_OUTPUT
+          elif [[ "${{ inputs.repro_ingest }}" == "true" ]]; then
             echo "command=dvc unfreeze ingest && dvc repro" >> $GITHUB_OUTPUT
           else
             echo "command=" >> $GITHUB_OUTPUT

misc/file_dict.csv

@@ -7,6 +7,7 @@ input,land_nbhd_rate,0,ingest,ccao-data-dvc-us-east-1,input/land_nbhd_rate_data.
 input,condo_strata,0,ingest,ccao-data-dvc-us-east-1,input/condo_strata_data.parquet,,,nbhd,"year, meta_pin",Condominium building strata cuts (ntile) or centers (k-means),Strata definitions by grouping variable. Each strata is a quantile boundary or k-means center recorded in log10 mean sale price of the condo building
 input,hie,0,ingest,ccao-data-dvc-us-east-1,input/hie_data.parquet,,,pin,"year, pin","HIE data used to build training and assessment data, used for report generation",A complete retrieval of ccao.hie
 input,char,0,ingest,ccao-data-dvc-us-east-1,input/char_data.parquet,,,card,"year, pin, card",Two years of characteristics data for the entire residential assessment set for report generation,Contains data for modeling year and one year prior
+input,dvc_lock,0,ingest,ccao-data-dvc-us-east-1,dvc.lock,model-condo-avm/lockfiles/year={year}/{run_id}.lock,,model run,"year, run_id",DVC lock file recording input data versions for the model run,lockfile must be manually copied from the S3 location
 output,parameter_search,1,train,ccao-model-results-us-east-1,output/parameter_search/model_parameter_search.parquet,parameter_search/year={year}/{run_id}.parquet,parameter_search,model cv fold,"year, run_id, configuration, fold_id",Tidymodels tuning output from cross-validation,Each row is the result from one fold assessment from one iteration
 output,parameter_raw,1,train,ccao-model-results-us-east-1,output/parameter_search/model_parameter_search.parquet,parameter_raw/year={year}/{run_id}.parquet,,model cv fold,"year, run_id, configuration, fold_id","Raw, nested Tidymodels tuning output",Not useful in Athena but needed to make models reproducible
 output,parameter_final,1,train,ccao-model-results-us-east-1,output/parameter_final/model_parameter_final.parquet,parameter_final/year={year}/{run_id}.parquet,parameter_final,model run,"year, run_id",Chosen set of hyperparameters for each run,"As chosen by tune::select_best() if using CV, otherwise the default set of hyperparameters specified in params.yaml (model.hyperparameter.default)"

pipeline/06-upload.R

@@ -273,6 +273,13 @@ if (upload_enable) {
       object = paths$output$report_model_features$s3
     )
   }
+
+  # 2.6. DVC lock file ---------------------------------------------------------
+  message("Pushing dvc.lock to ", paths$input$dvc_lock$s3)
+  aws.s3::put_object(
+    file = paths$input$dvc_lock$local,
+    object = paths$input$dvc_lock$s3
+  )
 }