Update summaries in the clinic service before triggering the EHR sync

Author: toddkazakov

POSTPROCESS_MIGRATION_CHECKLIST.md

@@ -85,8 +85,9 @@ Consequences vs merging at enqueue: `JellyfishQuietDelayMaximum` is no longer ne
 a shared row, so nothing can be starved), and a long jellyfish upload recalculates roughly once per
 90s quiet window rather than once at the end.
 
-**Processor** (Quantity 15, Frequency 30s): `updateSummaries` (all 3 types) → `triggerEHRSync`
-(`SyncEHRDataForPatient`, at-least-once) → `Delete`.
+**Processor** (Quantity 15, Frequency 30s): `updateSummaries` (all 3 types) → `updateClinicSummaries`
+(cgm/bgm changes only, at-least-once — Step 16a) → `triggerEHRSync` (`SyncEHRDataForPatient`,
+at-least-once) → `Delete`.
 Failures → Failing (exponential 1m/5s jitter/1h cap).
 
 **Logging**: INFO — item lifecycle + data-shape changes (processor start/finish: userId, reasons,
@@ -667,6 +668,103 @@ bump the abbott plugin's platform pin (it pins the exact commit `9ee577a1c`, whi
       run the remediation (work user ids → `dates.outdatedSince`), confirm the runners pick them up
 - [ ] Done — **⏸ approval — PR1 checkpoint**
 
+### Step 16a — Clinic summary updates move from clinic-worker CDC into the work item (2026-08-26)
+
+The clinic service stores a copy of each patient's cgm/bgm summaries, and the EHR sync generates
+reports **from that copy** — so the copy must be current before the sync fires. Today clinic-worker's
+`patientsummary` CDC consumer maintains it (Mongo change stream → Kafka `data.summary` →
+`UpdatePatientSummary`/`DeletePatientSummary`). Moved into the processor as a step strictly between
+`updateSummaries` and `triggerElectronicHealthRecordSync`.
+
+- [x] **Clinic client bumped** `v0.0.0-20250122…` → `v0.0.0-20260814105914-911a077531db` (the old pin
+      predates `PatientSummaryV1`/`CgmStatsV1`/`DeletePatientSummary` and the ranges-based schema).
+      Generated names lost their unsuffixed aliases (`Clinic`→`ClinicV1`, `Patient`→`PatientV1`,
+      `EHRSettings`→`EhrSettingsV1`, …) — mechanical renames in clinics/ehr/prescription/notifications
+      + `clinics/test/clinics.go` rebuilt against the new `EhrSettingsV1` shape
+- [x] `clinics/summaries.go`: `NewPatientSummary(cgm, bgm) *clinic.PatientSummaryV1` — the
+      clinic-worker mapping (`patientsummary/models.go`) ported onto native `summary/types`, keeping
+      the UI eligibility rules verbatim (percentages only with records; TIR percent gated on >70%
+      single-day use or >24h CGM use; GMI only >70% use; percent deltas only when the reconstructed
+      previous period also qualifies; BGM stdev/CoV only ≥30 records and ≥7 days). Two clinic-worker
+      bugs fixed in the port: dead `IsZero()` guards on `time.UnixMilli` values (absent dates were
+      sent as 1970-01-01 with `Has*: true` — now omitted with `Has*: false`), and ~40 unguarded
+      `period.Delta` dereferences (nil delta panicked the consumer — now treated as a zero delta).
+      Type aliases `types.CGMSummary`/`BGMSummary`/`ContinuousSummary` added for the generic
+      instantiations
+- [x] `clinics.Client` gains `UpdatePatientSummary` (200/204/404 → success; 204 = not a patient of
+      any clinic, the common case) and `DeletePatientSummary` (200/204 → success), mirroring
+      clinic-worker's accepted statuses plus the `SyncEHRDataForPatient` 404 precedent
+- [x] **Only an actual change is reported** (requirement). The `Summarizers` seam returns
+      `SummariesUpdate{CGM, BGM, UpdatedTypes, Deleted}` (by value — a pointer invited nil checks its
+      callers never needed): each cgm/bgm update is classified against a pre-read (`GetSummary`
+      before `UpdateSummary`) — recalculation always advances `Dates.LastUpdatedDate`, the
+      no-data-modified branch never does, and deletion is a non-nil→nil transition (recording the
+      deleted summary's id for the clinic delete). The legacy outdated-flag clear (`SetNotOutdated`)
+      is deliberately classified as no change. Continuous summaries are calculated but never
+      classified or reported — the clinic model has no `con`
+- [x] **`LastUpdatedDate` discriminator audited (2026-08-26 review)**: the field is written in
+      exactly two places. `Dates.Update` sets it, and is reachable only past the
+      `status.EarliestModified.IsZero()` guard — i.e. only when the fetcher reports data modified
+      since the watermark. `Dates.Reset` zeroes it on the schema-migration path, but the zero can
+      never be persisted: the zero watermark makes the follow-on fetch either find data (full
+      recalculation → `Dates.Update` stamps a fresh value; classified updated, correctly — the
+      rebuild may change content) or find none (summary deleted; classified deleted). The
+      no-data-modified branch rewrites the summary without touching it. So "timestamp advanced" ⇔
+      "the summarizer was given new or modified data (or rebuilt in full)" — the invariant the
+      classification relies on holds
+- [x] **At-least-once across retries**: the pipeline retries from the start, and a retried
+      calculation reports no further change, so changes are persisted into the work metadata
+      (`pendingSummaryUpdates` types / `pendingSummaryDeletes` ids) BEFORE they are reported —
+      the absorb pattern. On the success path that is one extra `ProcessingUpdate` per changing item;
+      on the failing path the merged metadata rides the failing update for free (including changes
+      calculated before a mid-`UpdateSummaries` error — the seam returns partial results with the
+      error). Not cleared after reporting: the item is deleted at the end, and a re-report after an
+      EHR-sync failure stores the same summary again rather than twice. Remaining window, accepted: a
+      crash between the summary write and the metadata persist loses that report; the next actual
+      change reports the full current summary, so updates self-heal — a lost *delete* report does not
+- [x] **Retry-interleaving analysis (2026-08-26 review)**: deletions are reported before updates, and
+      a deletion matches only the summary id it names — never the id of a recreation — so every
+      single-pod retry interleaving converges (update-then-deleted → delete reported, update skipped
+      via nil content; deleted-then-recreated → old id unset, new stats stored; pending updates
+      re-send the CURRENT content re-read each attempt, so a sticky record never pushes stale stats;
+      the delete loop and the update re-send are idempotent). ONE genuine race was found: a processor
+      reaped mid-report (>5m stall) whose `UpdatePatientSummary` completes AFTER another pod's retry
+      pushed newer stats would regress the clinic copy until the user's next change — unlike CDC
+      there is no partition ordering. CLOSED clinic-side by **tidepool-org/clinic#265**
+      (2026-08-26): `UpdateSummaryInAllClinics` applies each stats type only when the incoming
+      `dates.lastUpdatedDate` is not older than the stored one (`$cond` pipeline update, per type,
+      per patient document; equal dates still apply so re-reports stay idempotent; BSON ordering
+      missing < null < date means absent stored stats always accept and an undated report replaces
+      only undated stats). Deploy #265 before or with this step — before it, the race stands as
+      originally accepted (requires a stalled-then-successful HTTP call spanning the reap grace plus
+      the retry's full recalculation)
+- [x] `updateClinicSummaries` step: deletes first (each id, tolerant of ids the clinic no longer
+      holds), then one `UpdatePatientSummary` carrying only the changed types' stats (the clinic
+      service merges per type, leaving the other untouched). An updated type whose summary was
+      deleted by a later attempt has its deletion recorded and no summary to report — skipped. Not
+      reason-gated, unlike the EHR sync: every actual change reaches the clinic
+- [x] `absorbPending` validates each sibling before absorbing it — `validateIdentity` (the same
+      check the claimed work gets) on top of the field validation `metadata.Decode` already runs —
+      so a row listed by group but naming another user in its metadata is left pending to fail on
+      its own pickup, rather than having that user's change merged into this one and destroyed
+- [x] Tests: processor — record→report→sync ordering pinned (`gomock.InOrder`); only changed types in
+      the payload; no clinic call when nothing changed; delete before update; pre-populated metadata
+      reports with a no-change recalculation (the retry case); failing metadata carries the
+      unreported changes (both after a report failure and after a partial calculation failure); three
+      new Failing entries. clinics — mapping (ids/config/dates, zero-date omission, empty-reason
+      replacement, every eligibility gate, nil delta, non-day period keys dropped) and both new
+      client methods (paths, tolerated statuses, error statuses). Metadata parse/validate/round-trip
+      entries for the new fields
+- [x] Done: `GOTEST_PKGS='./data/... ./summary/... ./clinics/... ./work/... ./ehr/... ./prescription/... ./notifications/...' make test-go`
+      green (84 packages); `go build ./...`/`go vet ./...`/`gofmt`/`goimports -local` clean
+- [ ] **Cross-repo consequence**: once this deploys, clinic-worker's `patientsummary` consumer and
+      the processor both push (benign duplicates — same content, idempotent upsert). Retire the
+      consumer in clinic-worker with the same gate as clinic-worker#123: the drain endpoint's inline
+      recalculations (alive until PR2/PR3) reach the clinic only via CDC, so the consumer must
+      outlive the legacy queue (`tidepool_summary_queue_length` → 0). The patient-create
+      `populateSummary` path in clinic-worker (`patients/consumer.go`) stays — it pulls, not CDC
+- **⏸ approval**
+
 ## PR2 — task service retirement
 
 ### Step 17 — Remove runners + task-row cleanup

clinics/clinician.go

@@ -6,7 +6,7 @@ import (
 	api "github.com/tidepool-org/clinic/client"
 )
 
-func IsPrescriber(clinician *api.Clinician) bool {
+func IsPrescriber(clinician *api.ClinicianV1) bool {
 	if clinician == nil {
 		return false
 	}

clinics/service.go

@@ -23,15 +23,17 @@ const ErrorCodeClinicClientFailure = "clinic-client-failure"
 var ClientModule = fx.Provide(NewClient)
 
 type Client interface {
-	GetClinic(ctx context.Context, clinicID string) (*clinic.Clinic, error)
-	GetClinician(ctx context.Context, clinicID, clinicianID string) (*clinic.Clinician, error)
-	GetEHRSettings(ctx context.Context, clinicId string) (*clinic.EHRSettings, error)
-	SharePatientAccount(ctx context.Context, clinicID, patientID string) (*clinic.Patient, error)
-	ListEHREnabledClinics(ctx context.Context) ([]clinic.Clinic, error)
+	GetClinic(ctx context.Context, clinicID string) (*clinic.ClinicV1, error)
+	GetClinician(ctx context.Context, clinicID, clinicianID string) (*clinic.ClinicianV1, error)
+	GetEHRSettings(ctx context.Context, clinicId string) (*clinic.EhrSettingsV1, error)
+	SharePatientAccount(ctx context.Context, clinicID, patientID string) (*clinic.PatientV1, error)
+	ListEHREnabledClinics(ctx context.Context) ([]clinic.ClinicV1, error)
 	SyncEHRData(ctx context.Context, clinicID string) error
 	SyncEHRDataForPatient(ctx context.Context, patientID string) error
-	GetPatients(ctx context.Context, clinicId string, userToken string, params *clinic.ListPatientsParams, injectedParams url.Values) ([]clinic.Patient, error)
-	GetPatient(ctx context.Context, clinicID, patientID string) (*clinic.Patient, error)
+	GetPatients(ctx context.Context, clinicId string, userToken string, params *clinic.ListPatientsParams, injectedParams url.Values) ([]clinic.PatientV1, error)
+	GetPatient(ctx context.Context, clinicID, patientID string) (*clinic.PatientV1, error)
+	UpdatePatientSummary(ctx context.Context, patientID string, patientSummary *clinic.PatientSummaryV1) error
+	DeletePatientSummary(ctx context.Context, summaryID string) error
 }
 
 type config struct {
@@ -76,7 +78,7 @@ func NewClient(authClient auth.ExternalAccessor) (Client, error) {
 	}, nil
 }
 
-func (d *defaultClient) GetClinician(ctx context.Context, clinicID, clinicianID string) (*clinic.Clinician, error) {
+func (d *defaultClient) GetClinician(ctx context.Context, clinicID, clinicianID string) (*clinic.ClinicianV1, error) {
 	response, err := d.httpClient.GetClinicianWithResponse(ctx, clinic.ClinicId(clinicID), clinic.ClinicianId(clinicianID))
 	if err != nil {
 		return nil, err
@@ -94,7 +96,7 @@ func (d *defaultClient) GetClinician(ctx context.Context, clinicID, clinicianID
 	return response.JSON200, nil
 }
 
-func (d *defaultClient) GetClinic(ctx context.Context, clinicID string) (*clinic.Clinic, error) {
+func (d *defaultClient) GetClinic(ctx context.Context, clinicID string) (*clinic.ClinicV1, error) {
 	response, err := d.httpClient.GetClinicWithResponse(ctx, clinic.ClinicId(clinicID))
 	if err != nil {
 		return nil, err
@@ -112,11 +114,11 @@ func (d *defaultClient) GetClinic(ctx context.Context, clinicID string) (*clinic
 	return response.JSON200, nil
 }
 
-func (d *defaultClient) ListEHREnabledClinics(ctx context.Context) ([]clinic.Clinic, error) {
+func (d *defaultClient) ListEHREnabledClinics(ctx context.Context) ([]clinic.ClinicV1, error) {
 	offset := 0
 	batchSize := 1000
 
-	clinics := make([]clinic.Clinic, 0)
+	clinics := make([]clinic.ClinicV1, 0)
 	for {
 		response, err := d.httpClient.ListClinicsWithResponse(ctx, &clinic.ListClinicsParams{
 			EhrEnabled: pointer.FromBool(true),
@@ -148,7 +150,7 @@ func (d *defaultClient) ListEHREnabledClinics(ctx context.Context) ([]clinic.Cli
 	return clinics, nil
 }
 
-func (d *defaultClient) GetEHRSettings(ctx context.Context, clinicId string) (*clinic.EHRSettings, error) {
+func (d *defaultClient) GetEHRSettings(ctx context.Context, clinicId string) (*clinic.EhrSettingsV1, error) {
 	response, err := d.httpClient.GetEHRSettingsWithResponse(ctx, clinicId)
 	if err != nil {
 		return nil, err
@@ -163,10 +165,10 @@ func (d *defaultClient) GetEHRSettings(ctx context.Context, clinicId string) (*c
 	return response.JSON200, nil
 }
 
-func (d *defaultClient) SharePatientAccount(ctx context.Context, clinicID, patientID string) (*clinic.Patient, error) {
+func (d *defaultClient) SharePatientAccount(ctx context.Context, clinicID, patientID string) (*clinic.PatientV1, error) {
 	permission := make(map[string]interface{}, 0)
 	body := clinic.CreatePatientFromUserJSONRequestBody{
-		Permissions: &clinic.PatientPermissions{
+		Permissions: &clinic.PatientPermissionsV1{
 			Note: &permission,
 			View: &permission,
 		},
@@ -222,7 +224,37 @@ func (d *defaultClient) SyncEHRDataForPatient(ctx context.Context, patientID str
 	return nil
 }
 
-func (d *defaultClient) GetPatient(ctx context.Context, clinicID, patientID string) (*clinic.Patient, error) {
+func (d *defaultClient) UpdatePatientSummary(ctx context.Context, patientID string, patientSummary *clinic.PatientSummaryV1) error {
+	response, err := d.httpClient.UpdatePatientSummaryWithResponse(ctx, clinic.PatientId(patientID), *patientSummary)
+	if err != nil {
+		return err
+	}
+	if response.StatusCode() != http.StatusOK && response.StatusCode() != http.StatusNoContent && response.StatusCode() != http.StatusNotFound {
+		err = errors.Preparedf(ErrorCodeClinicClientFailure,
+			"Unexpected status code from clinic service",
+			"unexpected response status code %v from %v", response.StatusCode(), response.HTTPResponse.Request.URL)
+		err = errors.WithMeta(err, response.HTTPResponse)
+		return err
+	}
+	return nil
+}
+
+func (d *defaultClient) DeletePatientSummary(ctx context.Context, summaryID string) error {
+	response, err := d.httpClient.DeletePatientSummaryWithResponse(ctx, clinic.SummaryId(summaryID))
+	if err != nil {
+		return err
+	}
+	if response.StatusCode() != http.StatusOK && response.StatusCode() != http.StatusNoContent {
+		err = errors.Preparedf(ErrorCodeClinicClientFailure,
+			"Unexpected status code from clinic service",
+			"unexpected response status code %v from %v", response.StatusCode(), response.HTTPResponse.Request.URL)
+		err = errors.WithMeta(err, response.HTTPResponse)
+		return err
+	}
+	return nil
+}
+
+func (d *defaultClient) GetPatient(ctx context.Context, clinicID, patientID string) (*clinic.PatientV1, error) {
 	response, err := d.httpClient.GetPatientWithResponse(ctx, clinic.ClinicId(clinicID), clinic.PatientId(patientID))
 	if err != nil {
 		return nil, err
@@ -237,7 +269,7 @@ func (d *defaultClient) GetPatient(ctx context.Context, clinicID, patientID stri
 	return response.JSON200, nil
 }
 
-func (d *defaultClient) GetPatients(ctx context.Context, clinicId string, userToken string, params *clinic.ListPatientsParams, injectedParams url.Values) ([]clinic.Patient, error) {
+func (d *defaultClient) GetPatients(ctx context.Context, clinicId string, userToken string, params *clinic.ListPatientsParams, injectedParams url.Values) ([]clinic.PatientV1, error) {
 	response, err := d.httpClient.ListPatientsWithResponse(ctx, clinicId, params, func(ctx context.Context, req *http.Request) error {
 		if len(injectedParams) != 0 {
 			q := req.URL.Query()