Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Converted all 6 `dplyr::group_by()`/`dplyr::summarise()` grouped-aggregation call sites in `zi_aggregate.R` and the paired `group_by()`+`slice()`+`ungroup()` combo in `zi_prep_hud.R` to a new internal `group_summarise_base()` helper (`R/zi_utils.R`), as part of Epic L's dplyr removal; the helper matches `dplyr::group_by()`+`summarise()`'s output contract exactly (row/column ordering, `.groups` semantics, NA-as-a-real-level grouping, and NA-vs-NaN distinction in numeric grouping columns), verified via a 4-round adversarial code-review pass and 6 new regression tests; behavior is unchanged, no `rlang` introduced (#132)
- Converted all 7 `dplyr::left_join()` call sites in `zi_aggregate.R`, `zi_crosswalk.R`, `zi_label.R`, and `zi_prep_hud.R` to a new internal `left_join_base()` helper (`R/zi_utils.R`) wrapping Base R `merge()`, as part of Epic L's dplyr removal; the helper restores `merge()`'s row and column ordering to match `dplyr::left_join()`'s contract exactly (`x`'s original row order, `x` columns followed by new `y` columns) and disambiguates colliding non-key column names with `.x`/`.y` suffixes the same way `dplyr::left_join()` does, verified against NA-key matching, one-to-many duplicate-key expansion, and column-name collisions; behavior is unchanged, no `rlang` introduced (#131)
- Converted all `dplyr::filter()`, `select()`, `mutate()`, and `arrange()` call sites in `zi_aggregate.R`, `zi_crosswalk.R`, `zi_get_demographics.R`, `zi_get_geometry.R`, `zi_list_zctas.R`, `zi_load_crosswalk.R`, `zi_load_labels.R`, and `zi_prep_hud.R` to Base R equivalents, as part of Epic L's dplyr removal; behavior is unchanged, no `rlang` introduced (#130)
- Converted `dplyr::filter()`, `mutate()`, `arrange()`, and `bind_rows()` call sites in `data-raw/build_uds_crosswalk.R` and 18 `dplyr::filter()`/`select()` call sites in `tests/testthat/test_zi_aggregate.R` (discovered as an unstated test dependency during `R CMD check`) to Base R equivalents, and removed `dplyr (>= 1.0.0)` from `DESCRIPTION` Imports, completing Epic L's dplyr removal; `devtools::check()` now runs with 0 errors and 0 warnings (#134)
- Registered `[Epic K] Vendored Workflow Toolkit Adoption & Governance` (#123) on `docs/ROADMAP.md` under `Next`, with a preferred-but-not-blocking dependency edge to `[Epic J]` (#111) and a note that #120 gates #118 and #119 (#123)
- Added a `brew link --overwrite udunits gdal proj` step after the macOS `sf` dependency install in `R-CMD-check.yaml`, so a cache miss no longer silently leaves the runner-image-preinstalled formulae unlinked with a `##[warning] ... not linked` annotation (#122)
- Retired `[Epic I] CI & Development Infrastructure Health` (#110) from `docs/ROADMAP.md` to `Recently Shipped`; all five sub-issues (#102, #113, #114, #115, #122) shipped, resolving the caching-layer defects, checks-not-firing gap, Node.js 20 deprecation warnings, and macOS Homebrew warnings
Expand Down
1 change: 0 additions & 1 deletion DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ Language: en-US
Config/testthat/edition: 3
Imports:
cli (>= 3.0.0),
dplyr (>= 1.0.0),
httr2 (>= 1.0.0),
jsonlite (>= 1.7.0),
sf (>= 1.0-0),
Expand Down
4 changes: 4 additions & 0 deletions R/zi_prep_hud.R
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,10 @@ zi_prep_hud <- function(.data, by, return_max = TRUE){
# identify max
out <- out[order(out$zip5, out$state, out$geoid), ]

# NOTE: comments below intentionally reference dplyr by name to document
# the exact behavior each Base R construct reproduces (Epic L, #129).
# These are explanatory references, not functional dplyr usage, and are
# out of scope for "dplyr reference removal" (see #141).
# ave() groups by combining vectors via split(); unlike dplyr::group_by(),
# split() drops NA keys into per-row singleton groups instead of grouping
# them together. Coercing to factors with exclude = NULL preserves NA as a
Expand Down
5 changes: 5 additions & 0 deletions R/zi_utils.R
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
# NOTE: the comments in this file intentionally reference dplyr by name to
# document the exact behavior each Base R helper reproduces (Epic L, #129).
# These are explanatory references, not functional dplyr usage, and are
# out of scope for "dplyr reference removal" (see #141).

# Internal left join helper (replaces dplyr::left_join())
# Wraps base merge() to reproduce dplyr::left_join()'s row-order and
# column-order contract: all rows of x are kept in their original order,
Expand Down
38 changes: 32 additions & 6 deletions data-raw/build_uds_crosswalk.R
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
# Retrieved: run Sys.time() below to record date of build

# Dependencies ---------------------------------------------------------------
library(dplyr)
library(readr)
library(stringr)
library(tibble)
Expand Down Expand Up @@ -58,7 +57,7 @@ normalize_uds_year <- function(year) {

# remove rows with no ZCTA (must happen before zero-padding so "N/A" is caught)
if ("zcta" %in% names(df)) {
df <- dplyr::filter(df, !(zcta %in% c("N/A", NA, "No ZCTA", "")))
df <- df[!(df$zcta %in% c("N/A", NA, "No ZCTA", "")), , drop = FALSE]
}

# zero-pad ZIP and ZCTA to 5 digits
Expand All @@ -67,25 +66,52 @@ normalize_uds_year <- function(year) {

# remove military ZIPs
if ("zip_type" %in% names(df)) {
df <- dplyr::filter(df, !grepl("^M", zip_type))
df <- df[!grepl("^M", df$zip_type), , drop = FALSE]
}

# title-case post office names
if ("po_name" %in% names(df)) {
df <- dplyr::mutate(df, po_name = stringr::str_to_title(po_name))
df$po_name <- stringr::str_to_title(df$po_name)
}

# sort and add year tag
df <- dplyr::arrange(df, zip)
df <- df[order(df$zip), , drop = FALSE]
df$year <- as.integer(year)

cat(" OK (", nrow(df), "rows )\n", sep = "")
tibble::as_tibble(df)
}

# Build combined dataset -----------------------------------------------------
# NOTE: comments below intentionally reference dplyr by name to document
# the exact behavior this Base R helper reproduces (Epic L, #129). These
# are explanatory references, not functional dplyr usage, and are out of
# scope for "dplyr reference removal" (see #141).
# bind_rows_base(): row-binds two data frames by column name (matching
# dplyr::bind_rows()'s contract), since not every year's schema includes
# every standard column (see standard_cols intersection above). Missing
# columns are filled with NA for the frame that lacks them.
bind_rows_base <- function(x, y) {
x_cols <- names(x)
y_only_cols <- setdiff(names(y), x_cols)
all_cols <- c(x_cols, y_only_cols)

for (col in y_only_cols) {
x[[col]] <- NA
}
for (col in x_cols) {
if (!(col %in% names(y))) {
y[[col]] <- NA
}
}

out <- rbind(x[, all_cols, drop = FALSE], y[, all_cols, drop = FALSE])
rownames(out) <- NULL
out
}

all_years <- lapply(UDS_YEARS, normalize_uds_year)
crosswalk <- dplyr::bind_rows(all_years)
crosswalk <- Reduce(bind_rows_base, all_years)

cat("\nCombined rows:", nrow(crosswalk), "\n")
cat("Years present:", paste(sort(unique(crosswalk$year)), collapse = ", "), "\n")
Expand Down
36 changes: 18 additions & 18 deletions tests/testthat/test_zi_aggregate.R
Original file line number Diff line number Diff line change
Expand Up @@ -106,39 +106,39 @@ dec_weights <- tibble::tibble(
)

test_that("zi_census_extensive sums values by ZCTA3 and variable", {
input <- dplyr::filter(dec_fixture, variable == "P001001")
input <- dec_fixture[dec_fixture$variable == "P001001", ]
result <- zi_census_extensive(input)

expect_s3_class(result, "tbl_df")
expect_true(all(c("ZCTA3", "variable", "value") %in% names(result)))

r630 <- dplyr::filter(result, ZCTA3 == "630")
r631 <- dplyr::filter(result, ZCTA3 == "631")
r630 <- result[result$ZCTA3 == "630", ]
r631 <- result[result$ZCTA3 == "631", ]

expect_equal(r630$value, 600)
expect_equal(r631$value, 750)
})

test_that("zi_census_intensive computes weighted mean correctly", {
input <- dplyr::filter(dec_fixture, variable == "P013001")
input <- dec_fixture[dec_fixture$variable == "P013001", ]
result <- zi_census_intensive(input, weights = dec_weights, method = "mean")

expect_s3_class(result, "tbl_df")
expect_true(all(c("ZCTA3", "variable", "value") %in% names(result)))

# expected weighted mean for 630: (35.2*100 + 38.1*200 + 42.0*300) / 600
expected_630 <- stats::weighted.mean(c(35.2, 38.1, 42.0), c(100 / 600, 200 / 600, 300 / 600))
r630 <- dplyr::filter(result, ZCTA3 == "630")
r630 <- result[result$ZCTA3 == "630", ]
expect_equal(r630$value, expected_630, tolerance = 1e-6)

# expected weighted mean for 631: (29.5*150 + 33.7*250 + 40.1*350) / 750
expected_631 <- stats::weighted.mean(c(29.5, 33.7, 40.1), c(150 / 750, 250 / 750, 350 / 750))
r631 <- dplyr::filter(result, ZCTA3 == "631")
r631 <- result[result$ZCTA3 == "631", ]
expect_equal(r631$value, expected_631, tolerance = 1e-6)
})

test_that("zi_census_intensive computes weighted median correctly", {
input <- dplyr::filter(dec_fixture, variable == "P013001")
input <- dec_fixture[dec_fixture$variable == "P013001", ]
result <- zi_census_intensive(input, weights = dec_weights, method = "median")

expect_s3_class(result, "tbl_df")
Expand All @@ -148,7 +148,7 @@ test_that("zi_census_intensive computes weighted median correctly", {
# sorted x = c(35.2, 38.1, 42.0), cumsum(w)/sum(w) = c(1/6, 1/2, 1)
# first index where cumsum >= 0.5 is index 2 → value 38.1
expected_630 <- 38.1
r630 <- dplyr::filter(result, ZCTA3 == "630")
r630 <- result[result$ZCTA3 == "630", ]
expect_equal(r630$value, expected_630, tolerance = 1e-6)
})

Expand Down Expand Up @@ -181,24 +181,24 @@ acs_weights <- tibble::tibble(
)

test_that("zi_acs_extensive sums estimates and propagates MOE correctly", {
input <- dplyr::filter(acs_fixture, variable == "B01003_001")
input <- acs_fixture[acs_fixture$variable == "B01003_001", ]
result <- zi_acs_extensive(input)

expect_s3_class(result, "tbl_df")
expect_true(all(c("ZCTA3", "variable", "estimate", "moe") %in% names(result)))

r630 <- dplyr::filter(result, ZCTA3 == "630")
r630 <- result[result$ZCTA3 == "630", ]
expect_equal(r630$estimate, 6000)
# MOE = sqrt(100^2 + 150^2 + 200^2)
expect_equal(r630$moe, sqrt(100^2 + 150^2 + 200^2), tolerance = 1e-6)

r631 <- dplyr::filter(result, ZCTA3 == "631")
r631 <- result[result$ZCTA3 == "631", ]
expect_equal(r631$estimate, 7500)
expect_equal(r631$moe, sqrt(120^2 + 180^2 + 220^2), tolerance = 1e-6)
})

test_that("zi_acs_intensive computes weighted mean for estimate and moe", {
input <- dplyr::filter(acs_fixture, variable == "B19013_001")
input <- acs_fixture[acs_fixture$variable == "B19013_001", ]
result <- zi_acs_intensive(input, weights = acs_weights, method = "mean")

expect_s3_class(result, "tbl_df")
Expand All @@ -209,13 +209,13 @@ test_that("zi_acs_intensive computes weighted mean for estimate and moe", {
expected_est_630 <- stats::weighted.mean(c(55000, 62000, 48000), w630)
expected_moe_630 <- stats::weighted.mean(c(5000, 6000, 4500), w630)

r630 <- dplyr::filter(result, ZCTA3 == "630")
r630 <- result[result$ZCTA3 == "630", ]
expect_equal(r630$estimate, expected_est_630, tolerance = 1e-6)
expect_equal(r630$moe, expected_moe_630, tolerance = 1e-6)
})

test_that("zi_acs_intensive computes weighted median for estimate and moe", {
input <- dplyr::filter(acs_fixture, variable == "B19013_001")
input <- acs_fixture[acs_fixture$variable == "B19013_001", ]
result <- zi_acs_intensive(input, weights = acs_weights, method = "median")

expect_s3_class(result, "tbl_df")
Expand All @@ -228,7 +228,7 @@ test_that("zi_acs_intensive computes weighted median for estimate and moe", {
expected_est_630 <- 48000
expected_moe_630 <- 4500

r630 <- dplyr::filter(result, ZCTA3 == "630")
r630 <- result[result$ZCTA3 == "630", ]
expect_equal(r630$estimate, expected_est_630, tolerance = 1e-6)
expect_equal(r630$moe, expected_moe_630, tolerance = 1e-6)
})
Expand All @@ -250,10 +250,10 @@ test_that("zi_aggregate works with decennial extensive data", {
expect_true(all(c("ZCTA3", "variable", "value") %in% names(result)))
expect_equal(nrow(result), 2)

r630 <- dplyr::filter(result, ZCTA3 == "630")
r630 <- result[result$ZCTA3 == "630", ]
expect_equal(r630$value, 600)

r631 <- dplyr::filter(result, ZCTA3 == "631")
r631 <- result[result$ZCTA3 == "631", ]
expect_equal(r631$value, 750)
})

Expand Down Expand Up @@ -354,7 +354,7 @@ test_that("zi_aggregate works with live Census data", {
variables = c(medincome = "B19013_001"),
state = "VT",
year = 2020
) |> dplyr::select(-NAME)
) |> (\(d) d[, setdiff(names(d), "NAME"), drop = FALSE])()

expect_error(
zi_aggregate(year = 2020, survey = correct_survey,
Expand Down
4 changes: 4 additions & 0 deletions tests/testthat/test_zi_utils.R
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
# fixture data (no API calls) ------------------------------------------------
# NOTE: several test titles/comments below intentionally reference dplyr by
# name to document the exact behavior each Base R helper reproduces (Epic
# L, #129). These are explanatory references, not functional dplyr usage,
# and are out of scope for "dplyr reference removal" (see #141).

x_simple <- data.frame(
id = c(1, 2, NA, 3),
Expand Down