Skip to content

Commit 55d9dd4

Browse files
feat(spec,service-analytics): a dataset answer says which records each measure counted, so a drill lists what the number counted (#30596)
<!-- claude-projects-attribution: {"github_login":"os-sales"} --> _Requested by **Sales** · [project thread](https://claude.ai/code/project/chan_01D5bWFY3rFzgJcytD5gD4Uu?thread=cmsg_01D5bWFY3rFzgJcytD5gD4Uu6W6tm24ezCZ9jmVrXt12YA)_ domain:spec#2 · os-sales · session_01L6bhgXexAhb1ogpnymn5Jb Fixes #30526 Clause-②: yes (widening) Metadata-protocol: none ## 维护者速读 - 改了什么:数据集查询的回答多了一个可选字段 `drillScopes`,按度量说明“这个数字统计的是哪些记录”:数据集自己的筛选,加上度量自己的筛选,再加上被汇总的字段有值。 - 为什么改:点开一个带筛选的数字(例如“已付款发票 2 张”)时,现在的下钻只按分组和组件筛选去列记录,会列出 4 张。客户端读不到数据集和度量的定义,所以只能由服务端在回答里告诉它。 - 风险与回滚:只新增一个可选字段,不读它的客户端不受影响;查询算出的数字不变,唯一的行为改动是日期等占位符改为在查询入口解析一次,算数和下钻范围共用同一个时刻。回滚就是 revert 本 PR。 - 先看:① `packages/services/service-analytics/src/drill-scope.ts`(什么算范围、什么时候不给);② `packages/services/service-analytics/src/analytics-service.ts` 里提前解析占位符和写 `result.drillScopes` 的两处;③ `packages/spec/src/api/analytics.zod.ts` 的字段说明。 - 两分钟验证:`pnpm --filter @objectstack/service-analytics exec vitest run src/__tests__/dataset-drill-scopes.test.ts`,预期 20 个测试全部通过,其中标着 CONTROL 的一条表明:不带范围时列出 4 条记录,而数字是 2。 - 需要你做的:无,席位落地。 ## What changes The precedent is `drillRanges` (#27074), and before it `dimensionFields` / `drillRawRows` (#20700). Those keys bound the clicked **bucket**. This PR adds the fifth drill sidecar, which bounds the clicked **measure**. It sits on the same answer and follows the same rule (ADR-0021 D2: the server states the drill bounds, and the client never restates dataset semantics). - **`@objectstack/spec`**: `AnalyticsResult.drillScopes`, optional, a map from measure name to `FilterCondition`, declared on the `AnalyticsResultResponse` wire schema too. Each selected measure name maps to the list filter on the answer's `object` that selects the records the measure aggregates. The filter is built from three parts: - the dataset's `filter`; - AND the measure's `filter`; - AND, for a measure over a field, `{ field: { $null: false } }`, or both instants for a duration measure. Tokens are resolved as the query resolved them. The value is `{}` when no part applies. A measure is **left out** when no list filter can select its records, and a measure left out offers no drill (the #30080 ruling, option A). That happens in three cases: - a derived measure; - a scope naming a relationship path (`account.region` as a key, or a dotted `$field`); - a filter that is not a condition object. A `__compare` column has no entry. `drillScopes` is set on every dataset answer that has rows. It is absent on a cube query answer and on a draft-preview answer. - **`@objectstack/service-analytics`**: `queryDataset` computes the scope in `drill-scope.ts` (`measureDrillScopes`, a pure function). It takes the token-resolved filters from **before** the dataset door's field binding, because the binding's minted member names are request-local and no record list knows them. `resolveSelectionTokens` is now exported and called once in `answerDataset`. The executor and the scope read the same resolved trees, and the executor's own call is then the identity. `datasetMeasureSource` finds what a measure aggregates the same way the queries did: - derived, bare or `CUBE.`-qualified; - duration, through `durationMeasureIn`; - declared, through `scopedMeasureKey`; - caller-minted, through `mintCallerMeasures`. - **`@object-ui/data-objectstack`**: `queryDataset` hand-picks the keys it returns, so it dropped any sidecar it did not list. This was measured before the change: the key never reached a widget. It now passes `drillScopes` through, typed as the spec slot. - Generated: `content/docs/references/api/analytics.mdx` (`gen:docs`, the one artifact `check:generated` proved stale). Ledger: `packages/spec/dropped-refinements.baseline.json` gains one site; see the Acceptance notes. ### What each consumer must compose (not in this PR: #30080 and the ui card the seat files at landing) A drill of a clicked number on measure `M` lists `{ $and: [bucket, widgetFilter, drillScopes[M]] }` on `object`, leaving out any part that is absent: | Consumer | `bucket` | `widgetFilter` | scope | |---|---|---|---| | Report drill (`plugin-report` dataset renderer) | `buildDatasetDrillFilter` over `dimensionFields` + `drillRawRows[i]` + `drillRanges[i]` (a totals row: `drillRawTotals[i][j]`) | the report's `runtimeFilter` | `drillScopes[M]` | | Chart / table widget drill (`plugin-dashboard` `DatasetWidget`) | the same builder for the clicked row and series | the widget's `runtimeFilter` | `drillScopes[M]` of the clicked series | | Dashboard tile click (#30080) | none (a KPI has no dimensions) | the tile's `runtimeFilter` | `drillScopes[M]` | If `drillScopes` has no entry for `M`, the consumer offers no click. `{}` means "drillable, nothing to add", which is a different thing from a missing entry. ## Why A dataset number is computed under the dataset's `filter`, the measure's `filter` and the widget's `runtimeFilter`. A host that drills the number holds only the last one, and it has no read of the dataset definition. So every drill of a filtered measure listed a superset of what the number counted. On the showcase shape, the bucket listed 4 invoices behind `paid_count` 2. #30080 (the dashboard tile click, ruled option A) waits on this answer. ## Tests - `packages/services/service-analytics/src/__tests__/dataset-drill-scopes.test.ts` (new, 20 tests): - **Unit tier** (mocked ObjectQL face), on the real producer shape (`examples/app-showcase` revenue-pulse: `paid_count` = count with `filter: { status: 'paid' }`, and `paid_rate` derived). Pins: - measure filter only; - dataset filter only, and both; - count over a field; - nested-relation form and a same-record `$field` kept; - tokens resolved to the caller, and the engine was handed the same value; - a cube-qualified name; - a dimensionless KPI carries the scope, and a zero-row answer carries none; - the scope is a deep copy; - **contract vs wire**: the real service's answer, through `JSON.stringify`, parsed by `AnalyticsResultResponseSchema`, keeps every scope. - **Real-engine tier** (ObjectQL + `SqlDriver` on SQLite, both faces, with and without a widget filter). Every count's composed drill (`bucket ∧ widget ∧ scope`) lists as many records as the number, and the sum adds up. Further pins: - a CONTROL pin: the bucket alone lists 4 against `paid_count` 2; - a duration measure, where the mean over the listed records equals the number; - a nested-relation dataset filter; - a dotted measure filter and a dotted dataset filter leave the scope out; - a CONTROL pin: `engine.find` refuses the dotted key with `INVALID_FIELD`. - `packages/spec/src/api/analytics.test.ts`: the response schema keeps each scope verbatim. It refuses these malformed values at `data.drillScopes`: - a string; - a list in an equality slot; - `$and` given an object; - an array map. - `packages/spec/src/contracts/analytics-service.test.ts`: a typed `drillScopes` read on the contract, plus a `@ts-expect-error` on a string scope. - `ui/packages/data-objectstack/src/queryDataset.test.ts`: - the passthrough keeps scopes verbatim, `{}` included; - absent stays `undefined`; - an array, a string or `null` is dropped; - a compile-time pin of identity with `AnalyticsResult['drillScopes']`. ## Acceptance notes - **Shape choices beyond the brief, each deliberate.** Each one is pinned by a test above. 1. The not-null leg for a measure over a field. Every aggregate skips an empty value (`COUNT(field)`, #10298), so without the leg, `count of total` would list rows whose `total` is empty. 2. A duration measure requires both instants. 3. `{}` for a measure with nothing to add, so that "absent" keeps one meaning: no drill. 4. Tokens are resolved once and shared by the numbers and the scope, so the two cannot straddle a period boundary. The error path is unchanged: every token refusal is a declared 400 that the catch rethrows either way. 5. A caller-minted (undeclared) measure gets a scope too. - **Relationship paths.** The nested-relation form (`{ account: { tier: 'gold' } }`) is kept, because the record list serves it. A dotted path is left out, because the list refuses a dotted filter key. A later change could lower a one-hop dotted path to the nested form; this PR does not. - **Ledger growth, a decision for the seat.** `dropped-refinements.baseline.json` gains `api/AnalyticsResultResponse` → `data.drillScopes.valueType.lazy`. This is the same `FilterConditionSchema` lazy refinement already admitted at `api/AnalyticsQueryRequest` → `where.lazy`, and the spec build prints this entry. Header totals move 251 → 252 schemas and 1019 → 1020 sites. - **Found, not fixed here (in the report):** - The engine refuses the nested-relation form inside an aggregation's own `filter` ("the nested-relation form, which the engine serves in 'where' only"). So a dataset **measure** whose `filter` is `{ account: { tier: 'gold' } }` fails the whole answer, even though the dataset schema accepts it. This was measured at the service on SQLite, not at the HTTP door. - `data-objectstack` also drops `drillRawTotals`. No ui code reads it today, so the gap is not felt. - **Branch name.** The branch is the harness-assigned `claude/project-thread-mg9ioh`, not `claude/issue-30526-*`, as amended on the claim. - **Attribution link.** It names this card thread, where the brief named the seat thread: the platform writes this thread's own link. - **New lint warnings.** eslint on `data-objectstack` adds 1 + 3 `no-explicit-any` warnings. These are `(data as any)` and test `as any` casts in the files' existing pattern; there are 0 errors. ## Verification Head `ba62840b` (branch base `c23eafa8`, `origin/main` `f5aaa0bc` merged in), under `os-verify-lock`, `NODE_OPTIONS=--max-old-space-size=8192`. Every count below is a VERDICT `command-exit 0` unless it says otherwise. Every suite below ran twice, once at `25fc5696` before the merge and once at `ba62840b` after it, with the same counts both times. - `pnpm --filter @objectstack/spec build && pnpm --filter @objectstack/spec check:generated`: 14 of 14 artifacts up to date, after `gen:docs`, which was the one artifact it proved stale. At the default 4096 MB heap, `check:test-typecheck` stops with "tsc failed without diagnostics", an out-of-memory error and not a verdict. At 8192 MB it passes: 52 files, 240 errors, all 131 pinned signatures held. - `pnpm --filter @objectstack/spec test`: Test Files 815 passed (815), Tests 25648 passed, 1 todo. `pnpm --filter @objectstack/spec typecheck`: exit 0. - `@objectstack/service-analytics`: `test` gives Test Files 211 passed (211), Tests 4996 passed, 300 skipped. `typecheck`: exit 0. - `@object-ui/data-objectstack`: `test` gives Test Files 115 passed (115), Tests 1319 passed. `type-check`: exit 0. - Consumers that `git grep drillRanges` lists: `@object-ui/core` 233 files passed (4376 tests), `@object-ui/plugin-dashboard` 202 files passed (1932 tests), `@object-ui/plugin-report` 33 files passed (307 tests). - eslint on the 10 changed source files: 0 errors (warnings as in Acceptance notes). <details><summary>Ablation of the serving half (scripts/ablation-replace.mjs, restored to the HEAD blob bca6a05)</summary> - Mutation: `result.drillScopes = measureDrillScopes({` became `void measureDrillScopes({` in `analytics-service.ts`. The anchor count went 1 → 0, and the blob went `bca6a0582f42` → `98dc47836851`. - `dataset-drill-scopes.test.ts`: **17 failed | 3 passed (20)**. - Every service pin went red on both tiers, the bucket CONTROL and the contract-vs-wire pin included. - The 3 that stayed green are the two direct walk tests and the record-list CONTROL (`engine.find` refuses a dotted key), none of which goes through the service. - Restore: the blob equals HEAD, and `git diff HEAD` is empty. The test imports the service from `src`, so no rebuild was needed. </details> <details><summary>Gates: dispatch-gates --commands, 110 derived at ba62840, 110 run, 0 NOT-MEASURED</summary> - `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --ran ran.list` printed "110 derived famil(ies) accounted for — 110 run, 0 NOT-MEASURED (a DERIVED zero — all 110 recorded an exit code and none of them is 3)" and exited 0. Every line is `exit 0`. - On the first pass two gates exited 3 because built output was missing: - `check:skill-examples` lacked the `client-react` dist. - `check:dual-build-cjs-loads` lacked the dists of account, setup, studio, client-react, cloud-connection, connector-imap and connector-mcp. - After `turbo run build` of those 7 packages, both exit 0. - `check-plugin-teardown-shape --self-test` needed its pinned fixture commit `621a4876` fetched into the shallow clone first. The longest runs: `check:type-check-debt` 277 s, `check:query-options-erasure` 201 s, `ui` `vitest run scripts/` 192 s. - No `!` in the title and no `(narrowing)` arm, so there was no full dogfood run. CI owns the rest of the farm. </details> ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic (one optional field and its describe; the scope is computed in `service-analytics`). - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; eslint on the changed files has 0 errors (the full `pnpm lint` is the CI run). - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. - [x] 6. `packages/spec` touched: its generated artifacts are regenerated and committed. - [x] 7. No hand edits to `content/docs/references/` (regenerated by `gen:docs`), `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English (the 维护者速读 block excepted, per project convention). - [x] The *Before Submitting* list in CONTRIBUTING.md is done. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01L6bhgXexAhb1ogpnymn5Jb <!-- signed: domain:spec#2 · os-sales · session_01L6bhgVugtCCwBs98FeR27P --> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 5b748df commit 55d9dd4

13 files changed

Lines changed: 776 additions & 10 deletions

File tree

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
'@objectstack/spec': patch
3+
'@objectstack/service-analytics': patch
4+
'@object-ui/data-objectstack': patch
5+
---
6+
7+
feat(spec,service-analytics,data-objectstack): a dataset answer says, per measure, which records that measure counted (`drillScopes`)
8+
9+
Clause-②: yes (widening)
10+
Metadata-protocol: none
11+
12+
- **`@objectstack/spec`: `AnalyticsResult.drillScopes?: Record<string, FilterCondition>`**, also declared on the `AnalyticsResultResponse` wire schema. Each selected measure name maps to the list filter, on the answer's `object`, that selects the records the measure aggregates: the dataset's own `filter`, AND the measure's own `filter`, AND, for a measure over a field, that field holding a value (both instants for a duration measure), with each `{token}` resolved as the query resolved it. It is `{}` when none of these applies. A measure that no list filter can select is left out: a derived measure, or a scope that names a relationship path (`account.region`, as a key or a `$field` reference). A measure left out offers no drill. A `__compare` column has no entry.
13+
- **`@objectstack/service-analytics`: `queryDataset` sets `drillScopes` on every dataset answer that has rows**, on both query faces, from the same token-resolved filters the numbers were computed under. A drill that lists the clicked bucket (`dimensionFields` / `drillRawRows` / `drillRawTotals`, `drillRanges`) AND the widget's own filter AND this scope lists exactly the records the number counted. Before this, a drill built from the bucket and the widget filter alone listed every record in the bucket, so a filtered measure such as "paid invoices" showed 4 records behind the number 2.
14+
- **`@object-ui/data-objectstack`: `queryDataset` passes `drillScopes` through**, typed as the spec slot. The adapter lists the keys it returns, so until now the sidecar was dropped before it reached a widget.

‎content/docs/references/api/analytics.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,7 @@ const result = AnalyticsEndpoint.parse(data);
159159
| **drillRawRows** | `Record<string, any>[]` | optional | Drill-through sidecar, aligned to `rows` by index: each `dimensionFields` dimension name to the stored value that row was grouped by (a select option's value, a lookup's record id), captured before label resolution rewrites the row to its display label. An exact-match drill filter is built from these, not from the labels in `rows`. Set exactly when `dimensionFields` is. |
160160
| **drillRawTotals** | `Record<string, any>[][]` | optional | Drill-through sidecar for `totals`: entry `[i][j]` aligns to `totals[i].rows[j]` and holds the stored value of the `dimensionFields` dimensions that grouping groups by, so the grand-total grouping yields an empty map per row. Set only when `dimensionFields` is and the answer carries at least one `totals` grouping. |
161161
| **drillRanges** | `Record<string, { field: string; gte: string; lt: string }>[]` | optional | Drill-through sidecar for date ranges, aligned to `rows` by index: date dimension name to the half-open range `[gte, lt)` that row's records fall in on that dimension's field. A time bucket gets the span it covers; a `timeDimensions[].dateRange` window gets the window the query filtered by on every row, or narrows the bucket entry on the same field to the part inside the window. Bounds are `YYYY-MM-DD` calendar days, or ISO-8601 instants when the source field is `datetime` (a bucket's at midnight in the reference timezone); a row whose bucket value is null gets no bucket entry, and a window whose end is an instant or the last supported day gets none. Set only on a dataset answer that has rows and groups by at least one `type: 'date'` dimension with a `field` and a granularity, or carries a `dateRange` on a dimension with a `field`; independent of `dimensionFields`. Absent on a cube query answer. |
162+
| **drillScopes** | `Record<string, any>` | optional | Drill-through sidecar per measure: each selected measure name to the list filter, on the answer's `object`, that selects the records that measure aggregates: the dataset's own `filter`, AND the measure's own `filter`, AND, for a measure over a `field`, that field holding a value (`{ "$null": false }`; both instants for a duration measure), with each `{token}` resolved as the query resolved it; `{}` when none of these applies. A drill lists the clicked bucket (`dimensionFields` / `drillRawRows` / `drillRawTotals`, `drillRanges`) AND the widget's own filter AND this scope. A measure is left out when no list filter on `object` selects its records (a derived measure; a scope that names a relationship path such as `account.region`, as a key or as a `$field` reference), and a measure left out offers no drill. Keyed by measure name: a `__compare` column has no entry. Set on every dataset answer (`POST /analytics/dataset/query`) that has rows; absent on a cube query answer and on a draft-preview answer. |
162163

163164

164165
---

0 commit comments

Comments
 (0)